Files
sqlalchemy/test/ext/test_mutable.py
T
Mike Bayer f1e96cb087 reinvent xdist hooks in terms of pytest fixtures
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones.   Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them.   A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable.   New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.

From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow.  A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.

As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively.  The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.

Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session.   The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic.  A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function.  The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all.  Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.

As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.

An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2.  It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.

py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2.  For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.

Includes the following improvements:

Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`.  Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.

For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.

Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board.  the addition
of this flag has resolved the long-standing oracle "two task"
error problem.

For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically.  There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite.  Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.

Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.

Fixes: #5826
Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
2021-01-13 22:10:13 -05:00

1509 lines
37 KiB
Python

import copy
import pickle
from sqlalchemy import event
from sqlalchemy import ForeignKey
from sqlalchemy import func
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy import util
from sqlalchemy.ext.mutable import MutableComposite
from sqlalchemy.ext.mutable import MutableDict
from sqlalchemy.ext.mutable import MutableList
from sqlalchemy.ext.mutable import MutableSet
from sqlalchemy.orm import attributes
from sqlalchemy.orm import column_property
from sqlalchemy.orm import composite
from sqlalchemy.orm import mapper
from sqlalchemy.orm.instrumentation import ClassManager
from sqlalchemy.orm.mapper import Mapper
from sqlalchemy.testing import assert_raises
from sqlalchemy.testing import assert_raises_message
from sqlalchemy.testing import eq_
from sqlalchemy.testing import fixtures
from sqlalchemy.testing import mock
from sqlalchemy.testing.fixtures import fixture_session
from sqlalchemy.testing.schema import Column
from sqlalchemy.testing.schema import Table
from sqlalchemy.testing.util import picklers
from sqlalchemy.types import PickleType
from sqlalchemy.types import TypeDecorator
from sqlalchemy.types import VARCHAR
class Foo(fixtures.BasicEntity):
pass
class SubFoo(Foo):
pass
class FooWithEq(object):
def __init__(self, **kw):
for k in kw:
setattr(self, k, kw[k])
def __hash__(self):
return hash(self.id)
def __eq__(self, other):
return self.id == other.id
class Point(MutableComposite):
def __init__(self, x, y):
self.x = x
self.y = y
def __setattr__(self, key, value):
object.__setattr__(self, key, value)
self.changed()
def __composite_values__(self):
return self.x, self.y
def __getstate__(self):
return self.x, self.y
def __setstate__(self, state):
self.x, self.y = state
def __eq__(self, other):
return (
isinstance(other, Point)
and other.x == self.x
and other.y == self.y
)
class MyPoint(Point):
@classmethod
def coerce(cls, key, value):
if isinstance(value, tuple):
value = Point(*value)
return value
class _MutableDictTestFixture(object):
@classmethod
def _type_fixture(cls):
return MutableDict
def teardown_test(self):
# clear out mapper events
Mapper.dispatch._clear()
ClassManager.dispatch._clear()
class _MutableDictTestBase(_MutableDictTestFixture):
run_define_tables = "each"
def setup_mappers(cls):
foo = cls.tables.foo
mapper(Foo, foo)
def test_coerce_none(self):
sess = fixture_session()
f1 = Foo(data=None)
sess.add(f1)
sess.commit()
eq_(f1.data, None)
def test_coerce_raise(self):
assert_raises_message(
ValueError,
"Attribute 'data' does not accept objects of type",
Foo,
data=set([1, 2, 3]),
)
def test_in_place_mutation(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.commit()
f1.data["a"] = "c"
sess.commit()
eq_(f1.data, {"a": "c"})
def test_modified_event(self):
canary = mock.Mock()
event.listen(Foo.data, "modified", canary)
f1 = Foo(data={"a": "b"})
f1.data["a"] = "c"
eq_(
canary.mock_calls,
[
mock.call(
f1, attributes.Event(Foo.data.impl, attributes.OP_MODIFIED)
)
],
)
def test_clear(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.commit()
f1.data.clear()
sess.commit()
eq_(f1.data, {})
def test_update(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.commit()
f1.data.update({"a": "z"})
sess.commit()
eq_(f1.data, {"a": "z"})
def test_pop(self):
sess = fixture_session()
f1 = Foo(data={"a": "b", "c": "d"})
sess.add(f1)
sess.commit()
eq_(f1.data.pop("a"), "b")
sess.commit()
assert_raises(KeyError, f1.data.pop, "g")
eq_(f1.data, {"c": "d"})
def test_pop_default(self):
sess = fixture_session()
f1 = Foo(data={"a": "b", "c": "d"})
sess.add(f1)
sess.commit()
eq_(f1.data.pop("a", "q"), "b")
eq_(f1.data.pop("a", "q"), "q")
sess.commit()
eq_(f1.data, {"c": "d"})
def test_popitem(self):
sess = fixture_session()
orig = {"a": "b", "c": "d"}
# the orig dict remains unchanged when we assign,
# but just making this future-proof
data = dict(orig)
f1 = Foo(data=data)
sess.add(f1)
sess.commit()
k, v = f1.data.popitem()
assert k in ("a", "c")
orig.pop(k)
sess.commit()
eq_(f1.data, orig)
def test_setdefault(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.commit()
eq_(f1.data.setdefault("c", "d"), "d")
sess.commit()
eq_(f1.data, {"a": "b", "c": "d"})
eq_(f1.data.setdefault("c", "q"), "d")
sess.commit()
eq_(f1.data, {"a": "b", "c": "d"})
def test_replace(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.flush()
f1.data = {"b": "c"}
sess.commit()
eq_(f1.data, {"b": "c"})
def test_replace_itself_still_ok(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.flush()
f1.data = f1.data
f1.data["b"] = "c"
sess.commit()
eq_(f1.data, {"a": "b", "b": "c"})
def test_pickle_parent(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.commit()
f1.data
sess.close()
for loads, dumps in picklers():
sess = fixture_session()
f2 = loads(dumps(f1))
sess.add(f2)
f2.data["a"] = "c"
assert f2 in sess.dirty
def test_unrelated_flush(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"}, unrelated_data="unrelated")
sess.add(f1)
sess.flush()
f1.unrelated_data = "unrelated 2"
sess.flush()
f1.data["a"] = "c"
sess.commit()
eq_(f1.data["a"], "c")
def _test_non_mutable(self):
sess = fixture_session()
f1 = Foo(non_mutable_data={"a": "b"})
sess.add(f1)
sess.commit()
f1.non_mutable_data["a"] = "c"
sess.commit()
eq_(f1.non_mutable_data, {"a": "b"})
def test_copy(self):
f1 = Foo(data={"a": "b"})
f1.data = copy.copy(f1.data)
eq_(f1.data, {"a": "b"})
def test_deepcopy(self):
f1 = Foo(data={"a": "b"})
f1.data = copy.deepcopy(f1.data)
eq_(f1.data, {"a": "b"})
class _MutableListTestFixture(object):
@classmethod
def _type_fixture(cls):
return MutableList
def teardown_test(self):
# clear out mapper events
Mapper.dispatch._clear()
ClassManager.dispatch._clear()
class _MutableListTestBase(_MutableListTestFixture):
run_define_tables = "each"
def setup_mappers(cls):
foo = cls.tables.foo
mapper(Foo, foo)
def test_coerce_none(self):
sess = fixture_session()
f1 = Foo(data=None)
sess.add(f1)
sess.commit()
eq_(f1.data, None)
def test_coerce_raise(self):
assert_raises_message(
ValueError,
"Attribute 'data' does not accept objects of type",
Foo,
data=set([1, 2, 3]),
)
def test_in_place_mutation(self):
sess = fixture_session()
f1 = Foo(data=[1, 2])
sess.add(f1)
sess.commit()
f1.data[0] = 3
sess.commit()
eq_(f1.data, [3, 2])
def test_in_place_slice_mutation(self):
sess = fixture_session()
f1 = Foo(data=[1, 2, 3, 4])
sess.add(f1)
sess.commit()
f1.data[1:3] = 5, 6
sess.commit()
eq_(f1.data, [1, 5, 6, 4])
def test_del_slice(self):
sess = fixture_session()
f1 = Foo(data=[1, 2, 3, 4])
sess.add(f1)
sess.commit()
del f1.data[1:3]
sess.commit()
eq_(f1.data, [1, 4])
def test_clear(self):
if not hasattr(list, "clear"):
# py2 list doesn't have 'clear'
return
sess = fixture_session()
f1 = Foo(data=[1, 2])
sess.add(f1)
sess.commit()
f1.data.clear()
sess.commit()
eq_(f1.data, [])
def test_pop(self):
sess = fixture_session()
f1 = Foo(data=[1, 2, 3])
sess.add(f1)
sess.commit()
eq_(f1.data.pop(), 3)
eq_(f1.data.pop(0), 1)
sess.commit()
assert_raises(IndexError, f1.data.pop, 5)
eq_(f1.data, [2])
def test_append(self):
sess = fixture_session()
f1 = Foo(data=[1, 2])
sess.add(f1)
sess.commit()
f1.data.append(5)
sess.commit()
eq_(f1.data, [1, 2, 5])
def test_extend(self):
sess = fixture_session()
f1 = Foo(data=[1, 2])
sess.add(f1)
sess.commit()
f1.data.extend([5])
sess.commit()
eq_(f1.data, [1, 2, 5])
def test_operator_extend(self):
sess = fixture_session()
f1 = Foo(data=[1, 2])
sess.add(f1)
sess.commit()
f1.data += [5]
sess.commit()
eq_(f1.data, [1, 2, 5])
def test_insert(self):
sess = fixture_session()
f1 = Foo(data=[1, 2])
sess.add(f1)
sess.commit()
f1.data.insert(1, 5)
sess.commit()
eq_(f1.data, [1, 5, 2])
def test_remove(self):
sess = fixture_session()
f1 = Foo(data=[1, 2, 3])
sess.add(f1)
sess.commit()
f1.data.remove(2)
sess.commit()
eq_(f1.data, [1, 3])
def test_sort(self):
sess = fixture_session()
f1 = Foo(data=[1, 3, 2])
sess.add(f1)
sess.commit()
f1.data.sort()
sess.commit()
eq_(f1.data, [1, 2, 3])
def test_sort_w_key(self):
sess = fixture_session()
f1 = Foo(data=[1, 3, 2])
sess.add(f1)
sess.commit()
f1.data.sort(key=lambda elem: -1 * elem)
sess.commit()
eq_(f1.data, [3, 2, 1])
def test_sort_w_reverse_kwarg(self):
sess = fixture_session()
f1 = Foo(data=[1, 3, 2])
sess.add(f1)
sess.commit()
f1.data.sort(reverse=True)
sess.commit()
eq_(f1.data, [3, 2, 1])
def test_reverse(self):
sess = fixture_session()
f1 = Foo(data=[1, 3, 2])
sess.add(f1)
sess.commit()
f1.data.reverse()
sess.commit()
eq_(f1.data, [2, 3, 1])
def test_pickle_parent(self):
sess = fixture_session()
f1 = Foo(data=[1, 2])
sess.add(f1)
sess.commit()
f1.data
sess.close()
for loads, dumps in picklers():
sess = fixture_session()
f2 = loads(dumps(f1))
sess.add(f2)
f2.data[0] = 3
assert f2 in sess.dirty
def test_unrelated_flush(self):
sess = fixture_session()
f1 = Foo(data=[1, 2], unrelated_data="unrelated")
sess.add(f1)
sess.flush()
f1.unrelated_data = "unrelated 2"
sess.flush()
f1.data[0] = 3
sess.commit()
eq_(f1.data[0], 3)
def test_copy(self):
f1 = Foo(data=[1, 2])
f1.data = copy.copy(f1.data)
eq_(f1.data, [1, 2])
def test_deepcopy(self):
f1 = Foo(data=[1, 2])
f1.data = copy.deepcopy(f1.data)
eq_(f1.data, [1, 2])
def test_legacy_pickle_loads(self):
# due to an inconsistency between pickle and copy, we have to change
# MutableList to implement a __reduce_ex__ method. Which means we
# have to make sure all the old pickle formats are still
# deserializable since these can be used for persistence. these pickles
# were all generated using a MutableList that has only __getstate__ and
# __setstate__.
# f1 = Foo(data=[1, 2])
# pickles = [
# dumps(f1.data)
# for loads, dumps in picklers()
# ]
# print(repr(pickles))
# return
if util.py3k:
pickles = [
b"\x80\x04\x95<\x00\x00\x00\x00\x00\x00\x00\x8c\x16"
b"sqlalchemy.ext.mutable\x94\x8c\x0bMutableList\x94\x93\x94)"
b"\x81\x94(K\x01K\x02e]\x94(K\x01K\x02eb.",
b"ccopy_reg\n_reconstructor\np0\n(csqlalchemy.ext.mutable\n"
b"MutableList\np1\nc__builtin__\nlist\np2\n(lp3\nI1\naI2\n"
b"atp4\nRp5\n(lp6\nI1\naI2\nab.",
b"ccopy_reg\n_reconstructor\nq\x00(csqlalchemy.ext.mutable\n"
b"MutableList\nq\x01c__builtin__\nlist\nq\x02]q\x03(K\x01K"
b"\x02etq\x04Rq\x05]q\x06(K\x01K\x02eb.",
b"\x80\x02csqlalchemy.ext.mutable\nMutableList\nq\x00)\x81q"
b"\x01(K\x01K\x02e]q\x02(K\x01K\x02eb.",
b"\x80\x03csqlalchemy.ext.mutable\nMutableList\nq\x00)\x81q"
b"\x01(K\x01K\x02e]q\x02(K\x01K\x02eb.",
b"\x80\x04\x95<\x00\x00\x00\x00\x00\x00\x00\x8c\x16"
b"sqlalchemy.ext.mutable\x94\x8c\x0bMutableList\x94\x93\x94)"
b"\x81\x94(K\x01K\x02e]\x94(K\x01K\x02eb.",
]
else:
pickles = [
"\x80\x02csqlalchemy.ext.mutable\nMutableList\nq\x00]q\x01"
"(K\x01K\x02e\x85q\x02Rq\x03.",
"\x80\x02csqlalchemy.ext.mutable\nMutableList"
"\nq\x00]q\x01(K\x01K\x02e\x85q\x02Rq\x03.",
"csqlalchemy.ext.mutable\nMutableList\np0\n"
"((lp1\nI1\naI2\natp2\nRp3\n.",
"csqlalchemy.ext.mutable\nMutableList\nq\x00(]"
"q\x01(K\x01K\x02etq\x02Rq\x03.",
"\x80\x02csqlalchemy.ext.mutable\nMutableList"
"\nq\x01]q\x02(K\x01K\x02e\x85Rq\x03.",
"\x80\x02csqlalchemy.ext.mutable\nMutableList\n"
"q\x01]q\x02(K\x01K\x02e\x85Rq\x03.",
"csqlalchemy.ext.mutable\nMutableList\np1\n"
"((lp2\nI1\naI2\natRp3\n.",
"csqlalchemy.ext.mutable\nMutableList\nq\x01"
"(]q\x02(K\x01K\x02etRq\x03.",
]
for pickle_ in pickles:
obj = pickle.loads(pickle_)
eq_(obj, [1, 2])
assert isinstance(obj, MutableList)
class _MutableSetTestFixture(object):
@classmethod
def _type_fixture(cls):
return MutableSet
def teardown_test(self):
# clear out mapper events
Mapper.dispatch._clear()
ClassManager.dispatch._clear()
class _MutableSetTestBase(_MutableSetTestFixture):
run_define_tables = "each"
def setup_mappers(cls):
foo = cls.tables.foo
mapper(Foo, foo)
def test_coerce_none(self):
sess = fixture_session()
f1 = Foo(data=None)
sess.add(f1)
sess.commit()
eq_(f1.data, None)
def test_coerce_raise(self):
assert_raises_message(
ValueError,
"Attribute 'data' does not accept objects of type",
Foo,
data=[1, 2, 3],
)
def test_clear(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data.clear()
sess.commit()
eq_(f1.data, set())
def test_pop(self):
sess = fixture_session()
f1 = Foo(data=set([1]))
sess.add(f1)
sess.commit()
eq_(f1.data.pop(), 1)
sess.commit()
assert_raises(KeyError, f1.data.pop)
eq_(f1.data, set())
def test_add(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data.add(5)
sess.commit()
eq_(f1.data, set([1, 2, 5]))
def test_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data.update(set([2, 5]))
sess.commit()
eq_(f1.data, set([1, 2, 5]))
def test_binary_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data |= set([2, 5])
sess.commit()
eq_(f1.data, set([1, 2, 5]))
def test_intersection_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data.intersection_update(set([2, 5]))
sess.commit()
eq_(f1.data, set([2]))
def test_binary_intersection_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data &= set([2, 5])
sess.commit()
eq_(f1.data, set([2]))
def test_difference_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data.difference_update(set([2, 5]))
sess.commit()
eq_(f1.data, set([1]))
def test_operator_difference_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data -= set([2, 5])
sess.commit()
eq_(f1.data, set([1]))
def test_symmetric_difference_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data.symmetric_difference_update(set([2, 5]))
sess.commit()
eq_(f1.data, set([1, 5]))
def test_binary_symmetric_difference_update(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data ^= set([2, 5])
sess.commit()
eq_(f1.data, set([1, 5]))
def test_remove(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2, 3]))
sess.add(f1)
sess.commit()
f1.data.remove(2)
sess.commit()
eq_(f1.data, set([1, 3]))
def test_discard(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2, 3]))
sess.add(f1)
sess.commit()
f1.data.discard(2)
sess.commit()
eq_(f1.data, set([1, 3]))
f1.data.discard(2)
sess.commit()
eq_(f1.data, set([1, 3]))
def test_pickle_parent(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]))
sess.add(f1)
sess.commit()
f1.data
sess.close()
for loads, dumps in picklers():
sess = fixture_session()
f2 = loads(dumps(f1))
sess.add(f2)
f2.data.add(3)
assert f2 in sess.dirty
def test_unrelated_flush(self):
sess = fixture_session()
f1 = Foo(data=set([1, 2]), unrelated_data="unrelated")
sess.add(f1)
sess.flush()
f1.unrelated_data = "unrelated 2"
sess.flush()
f1.data.add(3)
sess.commit()
eq_(f1.data, set([1, 2, 3]))
def test_copy(self):
f1 = Foo(data=set([1, 2]))
f1.data = copy.copy(f1.data)
eq_(f1.data, set([1, 2]))
def test_deepcopy(self):
f1 = Foo(data=set([1, 2]))
f1.data = copy.deepcopy(f1.data)
eq_(f1.data, set([1, 2]))
class MutableColumnDefaultTest(_MutableDictTestFixture, fixtures.MappedTest):
@classmethod
def define_tables(cls, metadata):
MutableDict = cls._type_fixture()
mutable_pickle = MutableDict.as_mutable(PickleType)
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("data", mutable_pickle, default={}),
)
def setup_mappers(cls):
foo = cls.tables.foo
mapper(Foo, foo)
def test_evt_on_flush_refresh(self):
# test for #3427
sess = fixture_session()
f1 = Foo()
sess.add(f1)
sess.flush()
assert isinstance(f1.data, self._type_fixture())
assert f1 not in sess.dirty
f1.data["foo"] = "bar"
assert f1 in sess.dirty
class MutableWithScalarPickleTest(_MutableDictTestBase, fixtures.MappedTest):
@classmethod
def define_tables(cls, metadata):
MutableDict = cls._type_fixture()
mutable_pickle = MutableDict.as_mutable(PickleType)
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("skip", mutable_pickle),
Column("data", mutable_pickle),
Column("non_mutable_data", PickleType),
Column("unrelated_data", String(50)),
)
def test_non_mutable(self):
self._test_non_mutable()
class MutableWithScalarJSONTest(_MutableDictTestBase, fixtures.MappedTest):
@classmethod
def define_tables(cls, metadata):
import json
class JSONEncodedDict(TypeDecorator):
impl = VARCHAR(50)
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
MutableDict = cls._type_fixture()
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("data", MutableDict.as_mutable(JSONEncodedDict)),
Column("non_mutable_data", JSONEncodedDict),
Column("unrelated_data", String(50)),
)
def test_non_mutable(self):
self._test_non_mutable()
class MutableColumnCopyJSONTest(_MutableDictTestBase, fixtures.MappedTest):
@classmethod
def define_tables(cls, metadata):
import json
from sqlalchemy.ext.declarative import declarative_base
class JSONEncodedDict(TypeDecorator):
impl = VARCHAR(50)
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
MutableDict = cls._type_fixture()
Base = declarative_base(metadata=metadata)
class AbstractFoo(Base):
__abstract__ = True
id = Column(
Integer, primary_key=True, test_needs_autoincrement=True
)
data = Column(MutableDict.as_mutable(JSONEncodedDict))
non_mutable_data = Column(JSONEncodedDict)
unrelated_data = Column(String(50))
class Foo(AbstractFoo):
__tablename__ = "foo"
column_prop = column_property(
func.lower(AbstractFoo.unrelated_data)
)
assert Foo.data.property.columns[0].type is not AbstractFoo.data.type
def test_non_mutable(self):
self._test_non_mutable()
class MutableColumnCopyArrayTest(_MutableListTestBase, fixtures.MappedTest):
__requires__ = ("array_type",)
@classmethod
def define_tables(cls, metadata):
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.sql.sqltypes import ARRAY
MutableList = cls._type_fixture()
Base = declarative_base(metadata=metadata)
class Mixin(object):
data = Column(MutableList.as_mutable(ARRAY(Integer)))
class Foo(Mixin, Base):
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
class MutableListWithScalarPickleTest(
_MutableListTestBase, fixtures.MappedTest
):
@classmethod
def define_tables(cls, metadata):
MutableList = cls._type_fixture()
mutable_pickle = MutableList.as_mutable(PickleType)
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("skip", mutable_pickle),
Column("data", mutable_pickle),
Column("non_mutable_data", PickleType),
Column("unrelated_data", String(50)),
)
class MutableSetWithScalarPickleTest(_MutableSetTestBase, fixtures.MappedTest):
@classmethod
def define_tables(cls, metadata):
MutableSet = cls._type_fixture()
mutable_pickle = MutableSet.as_mutable(PickleType)
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("skip", mutable_pickle),
Column("data", mutable_pickle),
Column("non_mutable_data", PickleType),
Column("unrelated_data", String(50)),
)
class MutableAssocWithAttrInheritTest(
_MutableDictTestBase, fixtures.MappedTest
):
@classmethod
def define_tables(cls, metadata):
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("data", PickleType),
Column("non_mutable_data", PickleType),
Column("unrelated_data", String(50)),
)
Table(
"subfoo",
metadata,
Column("id", Integer, ForeignKey("foo.id"), primary_key=True),
)
def setup_mappers(cls):
foo = cls.tables.foo
subfoo = cls.tables.subfoo
mapper(Foo, foo)
mapper(SubFoo, subfoo, inherits=Foo)
MutableDict.associate_with_attribute(Foo.data)
def test_in_place_mutation(self):
sess = fixture_session()
f1 = SubFoo(data={"a": "b"})
sess.add(f1)
sess.commit()
f1.data["a"] = "c"
sess.commit()
eq_(f1.data, {"a": "c"})
def test_replace(self):
sess = fixture_session()
f1 = SubFoo(data={"a": "b"})
sess.add(f1)
sess.flush()
f1.data = {"b": "c"}
sess.commit()
eq_(f1.data, {"b": "c"})
class MutableAssociationScalarPickleTest(
_MutableDictTestBase, fixtures.MappedTest
):
@classmethod
def define_tables(cls, metadata):
MutableDict = cls._type_fixture()
MutableDict.associate_with(PickleType)
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("skip", PickleType),
Column("data", PickleType),
Column("unrelated_data", String(50)),
)
class MutableAssociationScalarJSONTest(
_MutableDictTestBase, fixtures.MappedTest
):
@classmethod
def define_tables(cls, metadata):
import json
class JSONEncodedDict(TypeDecorator):
impl = VARCHAR(50)
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
MutableDict = cls._type_fixture()
MutableDict.associate_with(JSONEncodedDict)
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("data", JSONEncodedDict),
Column("unrelated_data", String(50)),
)
class CustomMutableAssociationScalarJSONTest(
_MutableDictTestBase, fixtures.MappedTest
):
CustomMutableDict = None
@classmethod
def _type_fixture(cls):
if not (getattr(cls, "CustomMutableDict")):
MutableDict = super(
CustomMutableAssociationScalarJSONTest, cls
)._type_fixture()
class CustomMutableDict(MutableDict):
pass
cls.CustomMutableDict = CustomMutableDict
return cls.CustomMutableDict
@classmethod
def define_tables(cls, metadata):
import json
class JSONEncodedDict(TypeDecorator):
impl = VARCHAR(50)
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
CustomMutableDict = cls._type_fixture()
CustomMutableDict.associate_with(JSONEncodedDict)
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("data", JSONEncodedDict),
Column("unrelated_data", String(50)),
)
def test_pickle_parent(self):
# Picklers don't know how to pickle CustomMutableDict,
# but we aren't testing that here
pass
def test_coerce(self):
sess = fixture_session()
f1 = Foo(data={"a": "b"})
sess.add(f1)
sess.flush()
eq_(type(f1.data), self._type_fixture())
class _CompositeTestBase(object):
@classmethod
def define_tables(cls, metadata):
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("x", Integer),
Column("y", Integer),
Column("unrelated_data", String(50)),
)
def setup_test(self):
from sqlalchemy.ext import mutable
mutable._setup_composite_listener()
def teardown_test(self):
# clear out mapper events
Mapper.dispatch._clear()
ClassManager.dispatch._clear()
@classmethod
def _type_fixture(cls):
return Point
class MutableCompositeColumnDefaultTest(
_CompositeTestBase, fixtures.MappedTest
):
@classmethod
def define_tables(cls, metadata):
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("x", Integer, default=5),
Column("y", Integer, default=9),
Column("unrelated_data", String(50)),
)
@classmethod
def setup_mappers(cls):
foo = cls.tables.foo
cls.Point = cls._type_fixture()
mapper(
Foo,
foo,
properties={"data": composite(cls.Point, foo.c.x, foo.c.y)},
)
def test_evt_on_flush_refresh(self):
# this still worked prior to #3427 being fixed in any case
sess = fixture_session()
f1 = Foo(data=self.Point(None, None))
sess.add(f1)
sess.flush()
eq_(f1.data, self.Point(5, 9))
assert f1 not in sess.dirty
f1.data.x = 10
assert f1 in sess.dirty
class MutableCompositesUnpickleTest(_CompositeTestBase, fixtures.MappedTest):
@classmethod
def setup_mappers(cls):
foo = cls.tables.foo
cls.Point = cls._type_fixture()
mapper(
FooWithEq,
foo,
properties={"data": composite(cls.Point, foo.c.x, foo.c.y)},
)
def test_unpickle_modified_eq(self):
u1 = FooWithEq(data=self.Point(3, 5))
for loads, dumps in picklers():
loads(dumps(u1))
class MutableCompositesTest(_CompositeTestBase, fixtures.MappedTest):
@classmethod
def setup_mappers(cls):
foo = cls.tables.foo
Point = cls._type_fixture()
mapper(
Foo, foo, properties={"data": composite(Point, foo.c.x, foo.c.y)}
)
def test_in_place_mutation(self):
sess = fixture_session()
d = Point(3, 4)
f1 = Foo(data=d)
sess.add(f1)
sess.commit()
f1.data.y = 5
sess.commit()
eq_(f1.data, Point(3, 5))
def test_pickle_of_parent(self):
sess = fixture_session()
d = Point(3, 4)
f1 = Foo(data=d)
sess.add(f1)
sess.commit()
f1.data
assert "data" in f1.__dict__
sess.close()
for loads, dumps in picklers():
sess = fixture_session()
f2 = loads(dumps(f1))
sess.add(f2)
f2.data.y = 12
assert f2 in sess.dirty
def test_set_none(self):
sess = fixture_session()
f1 = Foo(data=None)
sess.add(f1)
sess.commit()
eq_(f1.data, Point(None, None))
f1.data.y = 5
sess.commit()
eq_(f1.data, Point(None, 5))
def test_set_illegal(self):
f1 = Foo()
assert_raises_message(
ValueError,
"Attribute 'data' does not accept objects",
setattr,
f1,
"data",
"foo",
)
def test_unrelated_flush(self):
sess = fixture_session()
f1 = Foo(data=Point(3, 4), unrelated_data="unrelated")
sess.add(f1)
sess.flush()
f1.unrelated_data = "unrelated 2"
sess.flush()
f1.data.x = 5
sess.commit()
eq_(f1.data.x, 5)
class MutableCompositeCallableTest(_CompositeTestBase, fixtures.MappedTest):
@classmethod
def setup_mappers(cls):
foo = cls.tables.foo
Point = cls._type_fixture()
# in this case, this is not actually a MutableComposite.
# so we don't expect it to track changes
mapper(
Foo,
foo,
properties={
"data": composite(lambda x, y: Point(x, y), foo.c.x, foo.c.y)
},
)
def test_basic(self):
sess = fixture_session()
f1 = Foo(data=Point(3, 4))
sess.add(f1)
sess.flush()
f1.data.x = 5
sess.commit()
# we didn't get the change.
eq_(f1.data.x, 3)
class MutableCompositeCustomCoerceTest(
_CompositeTestBase, fixtures.MappedTest
):
@classmethod
def _type_fixture(cls):
return MyPoint
@classmethod
def setup_mappers(cls):
foo = cls.tables.foo
Point = cls._type_fixture()
mapper(
Foo, foo, properties={"data": composite(Point, foo.c.x, foo.c.y)}
)
def test_custom_coerce(self):
f = Foo()
f.data = (3, 4)
eq_(f.data, Point(3, 4))
def test_round_trip_ok(self):
sess = fixture_session()
f = Foo()
f.data = (3, 4)
sess.add(f)
sess.commit()
eq_(f.data, Point(3, 4))
class MutableInheritedCompositesTest(_CompositeTestBase, fixtures.MappedTest):
@classmethod
def define_tables(cls, metadata):
Table(
"foo",
metadata,
Column(
"id", Integer, primary_key=True, test_needs_autoincrement=True
),
Column("x", Integer),
Column("y", Integer),
)
Table(
"subfoo",
metadata,
Column("id", Integer, ForeignKey("foo.id"), primary_key=True),
)
@classmethod
def setup_mappers(cls):
foo = cls.tables.foo
subfoo = cls.tables.subfoo
Point = cls._type_fixture()
mapper(
Foo, foo, properties={"data": composite(Point, foo.c.x, foo.c.y)}
)
mapper(SubFoo, subfoo, inherits=Foo)
def test_in_place_mutation_subclass(self):
sess = fixture_session()
d = Point(3, 4)
f1 = SubFoo(data=d)
sess.add(f1)
sess.commit()
f1.data.y = 5
sess.commit()
eq_(f1.data, Point(3, 5))
def test_pickle_of_parent_subclass(self):
sess = fixture_session()
d = Point(3, 4)
f1 = SubFoo(data=d)
sess.add(f1)
sess.commit()
f1.data
assert "data" in f1.__dict__
sess.close()
for loads, dumps in picklers():
sess = fixture_session()
f2 = loads(dumps(f1))
sess.add(f2)
f2.data.y = 12
assert f2 in sess.dirty