Fixed bug in :func:`.orm.selectinload` loading option where two or more
loaders that represent different relationships with the same string key
name as referenced from a single :func:`.orm.with_polymorphic` construct
with multiple subclass mappers would fail to invoke each subqueryload
separately, instead making use of a single string-based slot that would
prevent the other loaders from being invoked.
Fixes: #5228
Change-Id: Id0d1db8029ca88c13c0068115fe673adb7a68407
(cherry picked from commit a32c528c54)
Modified the queries used by subqueryload and selectinload to no longer
ORDER BY the primary key of the parent entity; this ordering was there to
allow the rows as they come in to be copied into lists directly with a
minimal level of Python-side collation. However, these ORDER BY clauses
can negatively impact the performance of the query as in many scenarios
these columns are derived from a subquery or are otherwise not actual
primary key columns such that SQL planners cannot make use of indexes. The
Python-side collation uses the native itertools.group_by() to collate the
incoming rows, and has been modified to allow multiple
row-groups-per-parent to be assembled together using list.extend(), which
should still allow for relatively fast Python-side performance. There will
still be an ORDER BY present for a relationship that includes an explicit
order_by parameter, however this is the only ORDER BY that will be added to
the query for both kinds of loading.
Fixes: #5162
Change-Id: I8befd1303c1af7cc24cbf005f39bc01c8b2745f3
(cherry picked from commit f86ee556add28afd4de31c10fce56b00a0014a4e)
Fixed issue where a lazyload that uses session-local "get" against a target
many-to-one relationship where an object with the correct primary key is
present, however it's an instance of a sibling class, does not correctly
return None as is the case when the lazy loader actually emits a load for
that row.
Fixes: #5210
Change-Id: I89f9946cfeba61d89a272435f76a5a082b1da30c
(cherry picked from commit 900402b9aa901bc9b1ae3f6b525f076076c52529)
Fixed regression in 1.3.14 due to 🎫`4849` where a sys.exc_info()
call failed to be invoked correctly when a flush error would occur. Test
coverage has been added for this exception case.
Fixes: #5196
Change-Id: Ib59a58a3a9d4c9c6f4b751201b794816a9f70225
(cherry picked from commit ceba13d4be)
Adjusted the error message emitted by :meth:`.Query.join` when a left hand
side can't be located that the :meth:`.Query.select_from` method is the
best way to resolve the issue. Also, within the 1.3 series, used a
deterministic ordering when determining the FROM clause from a given column
entity passed to :class:`.Query` so that the same expression is determined
each time.
Fixes: #5194
Change-Id: I2e4065fd31e98c57edf2f11d5e831be44d2c1ea2
(cherry picked from commit 7d8c64f908)
A viewonly=True relationship should not be mutated and ideally
mutation itself would raise an error, but we're not there yet.
Warn when a viewonly is to be the target of a back_populates
as this only means that it should be locally mutated, which
by definition will not work as expected because post-flush
it will deliver doubled results, due to its state not being
reset.
Setting a relationship to viewonly=True which is also the target of a
back_populates or backref configuration will now emit a warning and
eventually be disallowed. back_populates refers specifically to mutation
of an attribute or collection, which is disallowed when the attribute is
subject to viewonly=True. The viewonly attribute is not subject to
persistence behaviors which means it will not reflect correct results
when it is locally mutated.
Fixes: #5149
Change-Id: Ie51382a82e1a0ff5f3cf2cdbded780e77ace7f5f
(cherry picked from commit 4376c2e5b0)
The technique arrived at for doing exclusions inside of combinations
relies upon comparing all the arguments in a particular combination
to some set of combinations that were gathered as having
"exclusions". This logic is actually broken for the
case where the @testing.combinations has an "id", but if we fix
that, we still have the issue of all the arguments being
compared, which is complicated and also doesn't work for the
case of a py2/py3 incompatibility like a timezone that has
fractional minutes or seconds in it. It's also not clear
if a @testing.combinations that uses lambdas will work either
(maybe it does though because lambdax == lambdax compares...).
anyway, this patch reworks it so that we hit this on the decorator
side instead, where we add our own decorator and go through
the extra effort to create a decorator that accepts an extra
argument of "exclusions" which we can then check in a way that
is local to the whole pytest @combinations thing in the first place.
The only difficulty is that pytest is very sneaky about looking
at the test function so we need to make sure __wrapped__ isn't
set when doing this.
Change-Id: Ic57aae15b378e0f4ed009e4e82ae7ba73fb6dfc5
(cherry picked from commit 12ec0e0685)
Added a new flag :paramref:`.InstanceEvents.restore_load_context` and
:paramref:`.SessionEvents.restore_load_context` which apply to the
:meth:`.InstanceEvents.load`, :meth:`.InstanceEvents.refresh`, and
:meth:`.SessionEvents.loaded_as_persistent` events, which when set will
restore the "load context" of the object after the event hook has been
called. This ensures that the object remains within the "loader context"
of the load operation that is already ongoing, rather than the object being
transferred to a new load context due to refresh operations which may have
occurred in the event. A warning is now emitted when this condition occurs,
which recommends use of the flag to resolve this case. The flag is
"opt-in" so that there is no risk introduced to existing applications.
The change additionally adds support for the ``raw=True`` flag to
session lifecycle events.
Fixes: #5129
Change-Id: I2912f48ac8c5636297d63ed383454930e8e9a6a3
(cherry picked from commit 10b7937bb9)
Fixed an additional regression in the same area as that of 🎫`5080`
introduced in 1.3.0b3 via 🎫`4468` where the ability to create a
joined option across a :func:`.with_polymorphic` into a relationship
against the base class of that with_polymorphic, and then further into
regular mapped relationships would fail as the base class component would
not add itself to the load path in a way that could be located by the
loader strategy. The changes applied in 🎫`5080` have been further
refined to also accommodate this scenario.
Fixes: #5121
Change-Id: I9753dbbcb7b7640c995ad983a6d04b36fa18cf54
(cherry picked from commit c9406e58a4)
- Added pyproject.toml with black arguments
- Updated black version in precommit hook
- Reformatted the code
Fixes: #5100Closes: #5103
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5103
Pull-request-sha: 795fd5f896
Change-Id: I14eedbaa51fb531cbf90fcefe6a1e07c8a565625
(cherry picked from commit f49a3fc02c03e39b3d34da9dfba4fe46d71c4aff)
Fixed regression caused in 1.3.13 by 🎫`5056` where a refactor of the
ORM path registry system made it such that a path could no longer be
compared to an empty tuple, which can occur in a particular kind of joined
eager loading path. The "empty tuple" use case has been resolved so that
the path registry is compared to a path registry in all cases; the
:class:`.PathRegistry` object itself now implements ``__eq__()`` and
``__ne__()`` methods which will take place for all equality comparisons and
continue to succeed in the not anticipated case that a non-
:class:`.PathRegistry` object is compared, while emitting a warning that
this object should not be the subject of the comparison.
Fixes: #5110
Change-Id: I6cab6cd771c131d12b17939b369212f12c6bee16
(cherry picked from commit f5eeac3d18)
Fixed regression in loader options introduced in 1.3.0b3 via 🎫`4468`
where the ability to create a loader option using
:meth:`.PropComparator.of_type` targeting an aliased entity that is an
inheriting subclass of the entity which the preceding relationship refers
to would fail to produce a matching path. See also 🎫`5082` fixed
in this same release which involves a similar kind of issue.
Fixes: #5107
Change-Id: I5c6717b925060c3f8da42190d1f00d05248befd8
(cherry picked from commit c9bf876d93b9389d6d3b619565f6da166bab5ec2)
We must change the approach from 2734439 as the information
loss is breaking subquery eager loading.
Move the adjustment into a deeper set of logic inside
of path_regsitry. We can distinguish between a path
that will "naturally" build from an aliased entity
at the base, vs. one that will "naturally" build
on all raw mappers, based on if when we observe that
we are being given a with_polymorphic(), if the existing
parent path is already in progress or not.
In general, we prefer paths to have as much of the original
information as possible, and the "natural path" is supposed
to be where the loader lookup stuff happens.
Fixes: #5082
Change-Id: I3c0ee72993bae8a6f067bdef3dc9a57d83f64950
(cherry picked from commit 7cc2de880b)
One test in test_of_type was creating a cartesian product
because contains_eager() was used with "alias" to refer
to a with_polymorphic(), but the wp was not used with of_type(),
so the pathing did not know that additional entities were present.
while the docs indicate that of_type() should be used, there is no
reason to use "alias" when you are using of_type(). Attempts
to make this automatic don't work as the current usage contract
with "alias" is that the contains_eager() chain can continue
along in terms of the base entities, which is another example
of the implicit swapping of entities for an aliased version of
themselves that really should be entirely marked as deprecated
throughout 1.4 and removed in 2.0.
So instead, add test coverage for the of_type() versions of
things and begin to make the case that we can remove "alias"
entirely, where previously we thought we would only deprecate
the string form.
For the 1.3 backport we are also picking a little bit of the
lambda combinations improvements that landed in 217948f5c7 and for
some reason were not backported.
Fixes: #5096
Change-Id: Ia7b021c4044332ab3282267815f208da64410e95
(cherry picked from commit b54b6ff7c09d15cedec3d65b10cd383ef41f1fbc)
Fixed regression in joined eager loading introduced in 1.3.0b3 via
🎫`4468` where the ability to create a joined option across a
:func:`.with_polymorphic` into a polymorphic subclass using
:meth:`.RelationshipProperty.of_type` and then further along regular mapped
relationships would fail as the polymorphic subclass would not add itself
to the load path in a way that could be located by the loader strategy. A
tweak has been made to resolve this scenario.
Fixes: #5082
Change-Id: I1c7b8d70ed94436c655e433bf34394b13d384c35
(cherry picked from commit 2734439fff)
Repaired a warning in the ORM flush process that was not covered by test
coverage when deleting objects that use the "version_id" feature. This
warning is generally unreachable unless using a dialect that sets the
"supports_sane_rowcount" flag to False, which is not typically the case
however is possible for some MySQL configurations as well as older Firebird
drivers, and likely some third party dialects.
As part of this change, we would like to establish that versionining-
related warnings are not expected to emit for the core set of
test-covered dialects, and if they are we would like the check for
warning emitted to be explicit, rather than an ignore.
Fixes: #5068
Change-Id: Iee097554e43cbb54ced1be056635809e67cf30d6
(cherry picked from commit 7210595a79)
Likely due to new versions flake8-rst-docstrings or similar,
repair an incorrectly formatted code example in a doc string
causing all the gerrits to fail.
Change-Id: Ib2b242f435005a9e075315881c30c4a599d322e7
Added test support and repaired a wide variety of unnecessary reference
cycles created for short-lived objects, mostly in the area of ORM queries.
For the 1.3 backport, includes the prefix_anon_map() optimization
from 1.4 / master which inlines the anonymous symbol generation
into a single object. This removes a cycle from the compiler
that otherwise results in a signficantly higher number of
unreachable cycles.
Fixes: #5056
Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
(cherry picked from commit 492930ed572de5f5550d514bc2ca52a57f108350)
Fixed bug where usage of joined eager loading would not properly wrap the
query inside of a subquery when :meth:`.Query.group_by` were used against
the query. When any kind of result-limiting approach is used, such as
DISTINCT, LIMIT, OFFSET, joined eager loading embeds the row-limited query
inside of a subquery so that the collection results are not impacted. For
some reason, the presence of GROUP BY was never included in this criterion,
even though it has a similar effect as using DISTINCT. Additionally, the
bug would prevent using GROUP BY at all for a joined eager load query for
most database platforms which forbid non-aggregated, non-grouped columns
from being in the query, as the additional columns for the joined eager
load would not be accepted by the database.
Fixes: #5065
Change-Id: I9a2ed8196f83297ec38012138d1a5acdf9e88155
(cherry picked from commit 2d5fa22c7d)
Fixed issue where by if the "begin" of a transaction failed at the Core
engine/connection level, such as due to network error or database is locked
for some transactional recipes, within the context of the :class:`.Session`
procuring that connection from the connection pool and then immediately
returning it, the ORM :class:`.Session` would not close the connection
despite this connection not being stored within the state of that
:class:`.Session`. This would lead to the connection being cleaned out by
the connection pool weakref handler within garbage collection which is an
unpreferred codepath that in some special configurations can emit errors in
standard error.
Fixes: #5034
Change-Id: I6502a55791d86845f34bc10889c218f00765dfdc
(cherry picked from commit ff471152b6)
Needed to add tests to ensure this label reference is handled
correctly, so also modified the exception message to
be more clear if someone has this error within distinct().
Change-Id: I6e685e46ae336596272d14366445ac224c18d92c
(cherry picked from commit 55f66e430d18b8daa51121cbc18537e2dab1ad9f)
As the ORM's combinatoric tests mostly use entities and
table metadata that's defined in fixtures, we can't use
@testing.combinations directly as it takes place at the
module level. Instead we use lambdas, but to reduce
verbosity we use a code replacement so that the namespace
of the lambda can be provided at runtime rather than
module import time.
Change-Id: Ia63a510f9c1d08b055eef62cf047f1f427f0450c
(cherry picked from commit 1ab483ac54)
Setting persistence-related flags on :func:`.relationship` while also
setting viewonly=True will now emit a regular warning, as these flags do
not make sense for a viewonly=True relationship. In particular, the
"cascade" settings have their own warning that is generated based on the
individual values, such as "delete, delete-orphan", that should not apply
to a viewonly relationship. Note however that in the case of "cascade",
these settings are still erroneously taking effect even though the
relationship is set up as "viewonly". In 1.4, all persistence-related
cascade settings will be disallowed on a viewonly=True relationship in
order to resolve this issue.
Fixes: #4993
Change-Id: I4b607a96a7de2ffa15303a27fd93c162a681556d
(cherry picked from commit 22d1f0706b)
Fixed issue involving ``lazy="raise"`` strategy where an ORM delete of an
object would raise for a simple "use-get" style many-to-one relationship
that had lazy="raise" configured. This is inconsistent vs. the change
introduced in 1.3 as part of 🎫`4353`, where it was established that
a history operation that does not expect emit SQL should bypass the
``lazy="raise"`` check, and instead effectively treat it as
``lazy="raise_on_sql"`` for this case. The fix adjusts the lazy loader
strategy to not raise for the case where the lazy load was instructed that
it should not emit SQL if the object were not present.
Fixes: #4997
Change-Id: I4deb8c129900f28321c4a5c48301db5fe2aedf78
(cherry picked from commit ff4b63c2bc)
A helper for @testing.combinations when we just have lots of
true/false combinations as is the case with some ORM tests.
Change-Id: I9f2de97ce5b2487411ed610b8d41169c1052bd8f
(cherry picked from commit 431b019c43)
Fixed issue where when assigning a collection to itself as a slice, the
mutation operation would fail as it would first erase the assigned
collection inadvertently. As an assignment that does not change the
contents should not generate events, the operation is now a no-op. Note
that the fix only applies to Python 3; in Python 2, the ``__setitem__``
hook isn't called in this case; ``__setslice__`` is used instead which
recreates the list item-by-item in all cases.
Fixes: #4990
Change-Id: I08727880f70f4fe188de53a4dcd36746b62c7233
(cherry picked from commit 560044748a)
Fixed ORM bug where a "secondary" table that referred to a selectable which
in some way would refer to the local primary table would apply aliasing to
both sides of the join condition when a relationship-related join, either
via :meth:`.Query.join` or by :func:`.joinedload`, were generated. The
"local" side is now excluded.
Fixes: #4974
Change-Id: Ia43da747c22141b05439f4511ddeceb10248582e
(cherry picked from commit f3bfe0881726f1b3075400346f5987390ebe6c19)
Like py.test we need to be able to mark certain combination
elements with exclusion rules. Add additional logic
to pytestlplugin and exclusions so that the exclusion decorators
can be added to the combination tuples, where they will be applied
to the decorated function along with a qualifier that the test
arguments need to match what's given.
Change-Id: I15d2839954d77a252bab5aaf6e3fd9f388c99dd5
(cherry picked from commit bbe754784a)
The fails_on decorator was not being interpreted
correctly when multiple were present.
Remove obsolete fails_on from test_types that no longer
take place for MySQL, Oracle
Ensure test_types tests are using __backend__
mark currently failing Oracle interval tests
Change-Id: If8db0c02b31a8008fd1673c2380f1f974c3806a6
(cherry picked from commit 56dff403b7cbaf2de342c83d496d0937f5b719b1)
Added DDL support for "computed columns"; these are DDL column
specifications for columns that have a server-computed value, either upon
SELECT (known as "virtual") or at the point of which they are INSERTed or
UPDATEd (known as "stored"). Support is established for Postgresql, MySQL,
Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work
on this one.
ORM round trip tests included. The ORM makes use of existing
FetchedValue support and no additional ORM logic is present for
the basic feature.
It has been observed that Oracle RETURNING does not return the
new value of a computed column upon UPDATE; it returns the
prior value. As this is very dangerous, a warning is emitted
if a computed column is rendered into the RETURNING clause
of an UPDATE statement.
Fixes: #4894Closes: #4928
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4928
Pull-request-sha: d39c521d5a
Change-Id: I2610b2999a5b1b127ed927dcdaeee98b769643ce
(cherry picked from commit 602d1e6dfd538980bb8d513867b17dbc2b4b92dd)
The :paramref:`.relationship.omit_join` flag was not intended to be
manually set to True, and will now emit a warning when this occurs. The
omit_join optimization is detected automatically, and the ``omit_join``
flag was only intended to disable the optimization in the hypothetical case
that the optimization may have interfered with correct results, which has
not been observed with the modern version of this feature. Setting the
flag to True when it is not automatically detected may cause the selectin
load feature to not work correctly when a non-default primary join
condition is in use.
Fixes: #4954
Change-Id: I3afebefb13ddaf8f74c5c2b9e6e6a1a261ac5629
(cherry picked from commit 9b77474395)
A warning is emitted if a primary key value is passed to :meth:`.Query.get`
that consists of None for all primary key column positions. Previously,
passing a single None outside of a tuple would raise a ``TypeError`` and
passing a composite None (tuple of None values) would silently pass
through. The fix now coerces the single None into a tuple where it is
handled consistently with the other None conditions. Thanks to Lev
Izraelit for the help with this.
Fixes#4915Closes: #4917
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4917
Pull-request-sha: b388343c7c
Change-Id: Ibc6c27ccf50dfd4adbf15b6dbd393115c30d44fb
(cherry picked from commit 997f4b5f2b)
Added accessor :attr:`.Query.is_single_entity` to :class:`.Query`, which
will indicate if the results returned by this :class:`.Query` will be a
list of ORM entities, or a tuple of entities or column expressions.
SQLAlchemy hopes to improve upon the behavior of single entity / tuples in
future releases such that the behavior would be explicit up front, however
this attribute should be helpful with the current behavior. Pull request
courtesy Patrick Hayes.
Fixes: #4934Closes: #4935
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4935
Pull-request-sha: 98f72b40a8
Change-Id: If5db5db3ea46a93406d76ef90b5b588149ba2986
(cherry picked from commit e55e749e18)
The :class:`.BakedQuery` will not cache a query that was modified by a
:meth:`.QueryEvents.before_compile` event, so that compilation hooks that
may be applying ad-hoc modifications to queries will take effect on each
run. In particular this is helpful for events that modify queries used in
lazy loading as well as eager loading such as "select in" loading. In
order to re-enable caching for a query modified by this event, a new
flag ``bake_ok`` is added; see :ref:`baked_with_before_compile` for
details.
A longer term plan to provide a new form of SQL caching should solve this
kind of issue more comprehensively.
Fixes: #4947
Change-Id: I5823c4fa00e7b6d46a2e8461b02d8b16605a6ed0
(cherry picked from commit d6db28556b)
Add factilities to implement pytest.mark.parametrize and
pytest.fixtures patterns, which largely resemble things we are
already doing.
Ensure a facade is used, so that the test suite remains independent
of py.test, but also tailors the functions to the more limited
scope in which we are using them.
Additionally, create a class-based version that works from the
same facade.
Several old polymorphic tests as well as two of the sql test
are refactored to use the new features.
Change-Id: I6ef8af1dafff92534313016944d447f9439856cf
References: #4896
(cherry picked from commit 696ef9421da763a0ffcfd98083c0af78db3ea756)
A warning is emitted for a condition in which the :class:`.Session` may
implicitly swap an object out of the identity map for another one with the
same primary key, detaching the old one, which can be an observed result of
load operations which occur within the :meth:`.SessionEvents.after_flush`
hook. The warning is intended to notify the user that some special
condition has caused this to happen and that the previous object may not be
in the expected state.
Fixes: #4890
Change-Id: Ide11c6b9f21ca67ff5a96266c521d0c56fd6af8d
(cherry picked from commit 4aa43ecbd7)
Passing a plain string expression to :meth:`.Session.query` is deprecated,
as all string coercions were removed in 🎫`4481` and this one should
have been included. The :func:`.literal_column` function may be used to
produce a textual column expression.
Note this changeset is in the 1.3 branch only. The 1.4 (currently master)
branch will already have removed this behavior.
Fixes: #4873
Change-Id: I20067a6707c7e2f028c67b1fbf0e8e4d9d136233
Fixed regression in selectinload loader strategy caused by #4775 (released
in version 1.3.6) where a many-to-one attribute of None would no longer be
populated by the loader. While this was usually not noticeable due to the
lazyloader populating None upon get, it would lead to a detached instance
error if the object were detached.
Fixes: #4872
Change-Id: I18466841683111643810ebc4331376df38c4767b
(cherry picked from commit 9f3539b174)
Fixed unit test regression released in 1.3.8 that would cause failure for
Oracle, SQL Server and other non-native ENUM platforms due to new
enumeration tests added as part of 🎫`4285` enum sortability in the
unit of work; the enumerations created constraints that were duplicated on
name.
Fixes: #4285
Change-Id: I34f51e82be9b6bb43b0df8c54bb36822e6eeb73e
(cherry picked from commit 2df613243d)
Added support for the use of an :class:`.Enum` datatype using Python
pep-435 enumeration objects as values for use as a primary key column
mapped by the ORM. As these values are not inherently sortable, as
required by the ORM for primary keys, a new
:attr:`.TypeEngine.sort_key_function` attribute is added to the typing
system which allows any SQL type to implement a sorting for Python objects
of its type which is consulted by the unit of work. The :class:`.Enum`
type then defines this using the database value of a given enumeration.
The sorting scheme can be also be redefined by passing a callable to the
:paramref:`.Enum.sort_key_function` parameter. Pull request courtesy
Nicolas Caniart.
Fixes: #4285Closes: #4816
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4816
Pull-request-sha: 42266b766c
Change-Id: Iadcc16173c1ba26ffac5830db57743a4cb987c55
(cherry picked from commit 75b2518b26)
Fixed bug where :class:`.Load` objects were not pickleable due to
mapper/relationship state in the internal context dictionary. These
objects are now converted to picklable using similar techniques as that of
other elements within the loader option system that have long been
serializable.
Fixes: #4823
Change-Id: Id2a0d8b640ac475c86d6416ad540671e66d410e5
(cherry picked from commit cd2ccee9d8)
New tests added are failing on SQL Server which doesn't support
expressions in LIMIT, need better gerrit coverage
Fixes: #4803
Change-Id: I56bc44474d448a1faa1dfccff0b55763329e712d
(cherry picked from commit a34664040a)
Fixed bug where using :meth:`.Query.first` or a slice expression in
conjunction with a query that has an expression based "offset" applied
would raise TypeError, due to an "or" conditional against "offset" that did
not expect it to be a SQL expression as opposed to an integer or None.
Fixes: #4803
Change-Id: I56b97a5d23cb45427a27a90ab557fa1ac5c6739e
(cherry picked from commit 97d2a2091e)
Fixed regression caused by new selectinload for many-to-one logic where
a primaryjoin condition not based on real foreign keys would cause
KeyError if a related object did not exist for a given key value on the
parent object.
Fixes: #4777
Change-Id: I4ba96318302be68abe0e8c3611684087a04a2322
(cherry picked from commit 8f5b65f431)
Mike Bayer