a few changes for py2k:
* map_imperatively() includes the check that a class
is being sent, this was only working for mapper() before
* the test suite didn't place the py2k "autouse" workaround
in the correct order, seemingly, tried to adjust the
per-test ordering setup in pytestplugin.py
Change-Id: I4cc39630724e810953cfda7b2afdadc8b948e3c2
The :class:`.TypeDecorator` class will now emit a warning when used in SQL
compilation with caching unless the ``.cache_ok`` flag is set to ``True``
or ``False``. ``.cache_ok`` indicates that all the parameters passed to the
object are safe to be used as a cache key, ``False`` means they are not.
Fixes: #6436
Change-Id: Ib1bb7dc4b124e38521d615c2e2e691e4915594fb
The ``sqlalchemy.ext.mutable`` extension now tracks the "parents"
collection using the :class:`.InstanceState` associated with objects,
rather than the object itself. The latter approach required that the object
be hashable so that it can be inside of a ``WeakKeyDictionary``, which goes
against the behavioral contract of the ORM overall which is that ORM mapped
objects do not need to provide any particular kind of ``__hash__()`` method
and that unhashable objects are supported.
Fixes: #6020
Change-Id: I414c8861bc5691f5f320dac3eb696f4b8c37d347
Fixed issue where the :class:`_mutable.MutableComposite` construct could be
placed into an invalid state when the parent object was already loaded, and
then covered by a subsequent query, due to the composite properties'
refresh handler replacing the object with a new one not handled by the
mutable extension.
Fixes: #6001
Change-Id: Ieebd8e6afe6b65f8902cc12dec1efb968f5438ef
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones. Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them. A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable. New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.
From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow. A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.
As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively. The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.
Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session. The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic. A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function. The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all. Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.
As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.
An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2. It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.
py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2. For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.
Includes the following improvements:
Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`. Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.
For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.
Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board. the addition
of this flag has resolved the long-standing oracle "two task"
error problem.
For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically. There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite. Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.
Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.
Fixes: #5826Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
in Iae6ab95938a7e92b6d42086aec534af27b5577d3 I missed
that the "bind" was being stuck onto the MetaData in
TablesTest, which led thousands of ORM tests to still use
bound metadata. Keep looking for bound metadata.
standardize all ORM tests on a single means of getting a
Session when the Session API isn't the thing we are directly
testing, using a new function fixture_session() that replaces
create_session() and uses modern defaults.
Change-Id: Iaf71206e9ee568151496d8bc213a069504bf65ef
Added keyword arguments to the :meth:`.MutableList.sort` function so that a
key function as well as the "reverse" keyword argument can be provided.
Fixes: #5114
Change-Id: Iefb29e1ccadfad6ecba558ce575029307001b88e
Fixed bug where using ``copy.copy()`` or ``copy.deepcopy()`` on
:class:`.MutableList` would cause the items within the list to be
duplicated, due to an inconsistency in how Python pickle and copy both make
use of ``__getstate__()`` and ``__setstate__()`` regarding lists. In order
to resolve, a ``__reduce_ex__`` method had to be added to
:class:`.MutableList`. In order to maintain backwards compatibility with
existing pickles based on ``__getstate__()``, the ``__setstate__()`` method
remains as well; the test suite asserts that pickles made against the old
version of the class can still be deserialized by the pickle module.
Also modified sqlalchemy.testing.util.picklers to return picklers all the way through
pickle.HIGHEST_PROTOCOL.
Fixes: #4603
Change-Id: I7f78b9cfb89d59a706248536c553dc5e1d987b88
Implemented a new feature whereby the :class:`.AliasedClass` construct can
now be used as the target of a :func:`.relationship`. This allows the
concept of "non primary mappers" to no longer be necessary, as the
:class:`.AliasedClass` is much easier to configure and automatically inherits
all the relationships of the mapped class, as well as preserves the
ability for loader options to work normally.
- introduce new name for mapped_table, "persist_selectable". this is
the selectable that selects against the local mapper and its superclasses,
but does not include columns local only to subclasses.
- relationship gains "entity" which is the mapper or aliasedinsp.
- clarfiy name "entity" vs. "query_entity" in loader strategies.
Fixes: #4423Fixes: #4422Fixes: #4421Fixes: #3348
Change-Id: Ic3609b43dc4ed115006da9ad9189e574dc0c72d9
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Fixed bug where using :meth:`.Mutable.associate_with` or
:meth:`.Mutable.as_mutable` in conjunction with a class that has non-
primary mappers set up with alternatively-named attributes would produce an
attribute error. Since non-primary mappers are not used for persistence,
the mutable extension now excludes non-primary mappers from its
instrumentation steps.
Change-Id: I2630d9f771a171aece03181ccf9159885f68f25e
Fixes: #4215
Fixed regression in :class:`.ARRAY` datatype caused by
🎫`3964`, which is essentially the same
issue as that of 🎫`3832`, where column attachment events
for :class:`.ARRAY` would not be invoked. This breaks the use case
of using declarative mixins that declare a :class:`.Column` which
makes use of :meth:`.MutableList.as_mutable`.
Change-Id: If8c57615860883837f6cf72661e46180a77778c1
Fixes: #4141
Implemented in-place mutation operators ``__ior__``, ``__iand__``,
``__ixor__`` and ``__isub__`` for :class:`.mutable.MutableSet`
and ``__iadd__`` for :class:`.mutable.MutableList` so that change
events are fired off when these mutator methods are used to alter the
collection.
Change-Id: Ib357a96d3b06c5deb6b53eb304a8b9f1dc9e9ede
Fixes: #3853
Added new event handler :meth:`.AttributeEvents.modified` which is
triggered when the func:`.attributes.flag_modified` function is
invoked, which is common when using the :mod:`sqlalchemy.ext.mutable`
extension module.
Change-Id: Ic152f1d5c53087d780b24ed7f1f1571527b9e8fc
Fixes: #3303
Fixed regression released in 1.1.8 due to 🎫`3950` where the
deeper search for information about column types in the case of a
"schema type" or a :class:`.TypeDecorator` would produce an attribute
error if the mapping also contained a :obj:`.column_property`.
Change-Id: I38254834d3d79c9b339289a8163eb4789ec4c931
Fixes: #3956
Fixed bug in :mod:`sqlalchemy.ext.mutable` where the
:meth:`.Mutable.as_mutable` method would not track a type that had
been copied using :meth:`.TypeEngine.copy`. This became more of
a regression in 1.1 compared to 1.0 because the :class:`.TypeDecorator`
class is now a subclass of :class:`.SchemaEventTarget`, which among
other things indicates to the parent :class:`.Column` that the type
should be copied when the :class:`.Column` is. These copies are
common when using declarative with mixins or abstract classes.
Change-Id: Ib04df862c58263185dbae686c548fea3e12c46f1
Fixes: #3950
as a result of the bugfix for 🎫`3167`,
where attribute and validation events are no longer
called within the flush process. The mutable
extension was relying upon this behavior in the case where a column
level Python-side default were responsible for generating the new value
on INSERT or UPDATE, or when a value were fetched from the RETURNING
clause for "eager defaults" mode. The new value would not be subject
to any event when populated and the mutable extension could not
establish proper coercion or history listening. A new event
:meth:`.InstanceEvents.refresh_flush` is added which the mutable
extension now makes use of for this use case.
fixes#3427
- Added new event :meth:`.InstanceEvents.refresh_flush`, invoked
when an INSERT or UPDATE level default value fetched via RETURNING
or Python-side default is invoked within the flush process. This
is to provide a hook that is no longer present as a result of
🎫`3167`, where attribute and validation events are no longer
called within the flush process.
- Added a new semi-public method to :class:`.MutableBase`
:meth:`.MutableBase._get_listen_keys`. Overriding this method
is needed in the case where a :class:`.MutableBase` subclass needs
events to propagate for attribute keys other than the key to which
the mutable type is associated with, when intercepting the
:meth:`.InstanceEvents.refresh` or
:meth:`.InstanceEvents.refresh_flush` events. The current example of
this is composites using :class:`.MutableComposite`.
If a class inherited from MutableDict (say, for instance, to add an update() method), coerce() would give back an instance of MutableDict instead of an instance of the derived class.
:meth:`.MutableBase.coerce` method to be used, even though
the code seemed to indicate this intent, so this now works
and a brief example is added. As a side-effect,
the mechanics of this event handler have been changed so that
new :class:`.MutableComposite` types no longer add per-type
global event handlers. Also in 0.7.10
[ticket:2624]
become an externally usable package but still remains within the main sqlalchemy parent package.
in this system, we use kind of an ugly hack to get the noseplugin imported outside of the
"sqlalchemy" package, while still making it available within sqlalchemy for usage by
third party libraries.
have enough of its state set up to work
correctly within the unpickle() event established
by the mutable object extension, if the object
needed ORM attribute access within
__eq__() or similar. [ticket:2362]
expired after an insert or update operation, instead
of regenerated in place. This ensures that a
column value which is expired within a flush
will be loaded first, before the composite
is regenerated using that value. [ticket:2309]
- [bug] The fix in [ticket:2309] also emits the
"refresh" event when the composite value is
loaded on access, even if all column
values were already present, as is appropriate.
This fixes the "mutable" extension which relies
upon the "load" event to ensure the _parents
dictionary is up to date, fixes [ticket:2308].
Thanks to Scott Torborg for the test case here.
if None or a non-corresponding type were set,
an error would be raised. None is now accepted
which assigns None to all attributes,
illegal values raise ValueError.
Mike Bayer