Fixed issue in "lambda SQL" feature where the tracking of bound parameters
could be corrupted if the same lambda were evaluated across multiple
compile phases, including when using the same lambda across multiple engine
instances or with statement caching disabled.
Fixes: #12084
Change-Id: I327aa93ce7feb2326a22113164bd834b96b6b889
manually update the files to remove literal string concat on the same line,
since black does not seem to be making progress in handling these
Change-Id: I3c651374c5f3db5b8bc0c700328d67ca03743b7b
in I4e0b627bfa187f1780dc68ec81b94db1c78f846a the 1.4 version has more
changes than the main version, which failed to get the entire change,
yet the whole thing was merged. Restore the missing mutex related
code to the main version.
Fixed regression where the fix for 🎫`8098`, which was released in
the 1.4 series and provided a layer of concurrency-safe checks for the
lambda SQL API, included additional fixes in the patch that failed to be
applied to the main branch. These additional fixes have been applied.
Change-Id: Id172e09c421dafa6ef1d40b383aa4371de343864
References: #8098Fixes: #9461
Fixed issue in lambda SQL feature where the calculated type of a literal
value would not take into account the type coercion rules of the "compared
to type", leading to a lack of typing information for SQL expressions, such
as comparisons to :class:`.JSON` elements and similar.
Fixes: #9029
Change-Id: I381c8d7458d98ba762313dee9ec47a9c1881f74a
Added an informative error message when a method object is passed to a SQL
construct. Previously, when such a callable were passed, as is a common
typographical error when dealing with method-chained SQL constructs, they
were interpreted as "lambda SQL" targets to be invoked at compilation time,
which would lead to silent failures. As this feature was not intended to be
used with methods, method objects are now rejected.
Fixes: #7032
Change-Id: If714715bd8c11557ab769ee3b1a24264b0b06acc
All but one metaclass used internally can now
be replaced using __init_subclass__(). Within this
patch we remove:
* events._EventMeta
* sql.visitors.TraversibleType
* sql.visitors.InternalTraversibleType
* testing.fixtures.FindFixture
* testing.fixtures.FindFixtureDeclarative
* langhelpers.EnsureKWArgType
* sql.functions._GenericMeta
* sql.type_api.VisitableCheckKWArg (was a mixture of TraversibleType
and EnsureKWArgType)
The remaining internal class is MetaOptions used by the
sql.Options object which is in turn currently mostly for
ORM internal use, as this type implements class level overrides
for the ``+`` operator.
For declarative, removing DeclarativeMeta in place of
an `__init_subclass__()` class would not be fully feasible as
it would break backwards compatibility with applications that
refer to this class explicitly, but also DeclarativeMeta intercepts
class-level attribute set and delete operations which is a widely
used pattern. An option for declarative base to use
`__init_subclass__()` should be provided but this is out of
scope for this particular change.
Change-Id: I8aa898c7ab59d887739037d34b1cbab36521ab78
References: #6810
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Fixes: #7394
Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
Fixes: #7300
Change-Id: I6255c850b140522a4aba95085216d0bca18ce230
Fixed regression where the row objects returned for ORM queries, which are
now the normal :class:`_sql.Row` objects, would not be interpreted by the
:meth:`_sql.ColumnOperators.in_` operator as tuple values to be broken out
into individual bound parameters, and would instead pass them as single
values to the driver leading to failures. The change to the "expanding IN"
system now accommodates for the expression already being of type
:class:`.TupleType` and treats values accordingly if so. In the uncommon
case of using "tuple-in" with an untyped statement such as a textual
statement with no typing information, a tuple value is detected for values
that implement ``collections.abc.Sequence``, but that are not ``str`` or
``bytes``, as always when testing for ``Sequence``.
Added :class:`.TupleType` to the top level ``sqlalchemy`` import namespace.
Fixes: #7292
Change-Id: I8286387e3b3c3752b3bd4ae3560d4f31172acc22
Fixed issue in lambda caching system where an element of a query that
produces no cache key, like a custom option or clause element, would still
populate the expression in the "lambda cache" inappropriately.
This was discovered as part of 🎫`6887` but is a separate
issue.
References: #6887
Change-Id: I1665f4320254ddc63a0abf3088e9daeaffbd1840
Fixed an issue in the ``CacheKey.to_offline_string()`` method used by the
dogpile.caching example where attempting to create a proper cache key from
the special "lambda" query generated by the lazy loader would fail to
include the parameter values, leading to an incorrect cache key.
Fixes: #6858
Change-Id: Ice27087583c6f3ff79cf7d5b879e5dd0a4e58158
Adjusted the logic added as part of 🎫`6397` in 1.4.12 so that
internal mutation of the :class:`.BindParameter` object occurs within the
clause construction phase as it did before, rather than in the compilation
phase. In the latter case, the mutation still produced side effects against
the incoming construct and additionally could potentially interfere with
other internal mutation routines.
In order to solve the issue of the correct operator being present
on the BindParameter.expand_op, we necessarily have to expand the
BinaryExpression._negate() routine to flip the operator on the
BindParameter also.
Fixes: #6460
Change-Id: I1e53a9aeee4de4fc11af51d7593431532731561b
Fixed regression in ``selectinload`` loader strategy that would cause it to
cache its internal state incorrectly when handling relationships that join
across more than one column, such as when using a composite foreign key.
The invalid caching would then cause other loader operations to fail.
Fixes: #6410
Change-Id: I9f95ccca3553e7fd5794c619be4cf85c02b04626
Revised the "EMPTY IN" expression to no longer rely upon using a subquery,
as this was causing some compatibility and performance problems. The new
approach for selected databases takes advantage of using a NULL-returning
IN expression combined with the usual "1 != 1" or "1 = 1" expression
appended by AND or OR. The expression is now the default for all backends
other than SQLite, which still had some compatibility issues regarding
tuple "IN" for older SQLite versions.
Third party dialects can still override how the "empty set" expression
renders by implementing a new compiler method
``def visit_empty_set_op_expr(self, type_, expand_op)``, which takes
precedence over the existing
``def visit_empty_set_expr(self, element_types)`` which remains in place.
Fixes: #6258Fixes: #6397
Change-Id: I2df09eb00d2ad3b57039ae48128fdf94641b5e59
Fixed bug where combinations of the new "relationship with criteria"
feature could fail in conjunction with features that make use of the new
"lambda SQL" feature, including loader strategies such as selectinload and
lazyload, for more complicated scenarios such as polymorphic loading.
Fixes: #6131
Change-Id: I915dead6596866ae5fd1a7f593a90bce4b61d1af
Fixed a critical regression in the relationship lazy loader where the SQL
criteria used to fetch a related many-to-one object could go stale in
relation to other memoized structures within the loader if the mapper had
configuration changes, such as can occur when mappers are late configured
or configured on demand, producing a comparison to None and returning no
object. Huge thanks to Alan Hamlett for their help tracking this down late
into the night.
The primary change is that mapper._get_clause() uses a fixed name
for its bound parameters, which is memoized under a lambda statement
in the case of many-to-one lazy loading. This has implications for some other
logic namely the .compare() used by loader strategies to determine
use_get needed to be adjusted.
This change also repairs the lambda module's behavior of removing
the "required" flag from bound parameters, which caused this issue
to also fail silently rather than issuing an error for a required
bind parameter.
Fixes: #6055
Change-Id: I19e1aba9207a049873e0f13c19bad7541e223cfd
1. Improve coercions._deep_is_literal to check sequences
for clause elements, thus allowing a phrase like
lambda: col.in_([literal("x"), literal("y")]) to be handled
2. revise closure variable caching completely. All variables
entering must be part of a closure cache key or rejected.
only objects that can be resolved to HasCacheKey or FunctionType
are accepted; all other types are rejected. This adds a high
degree of strictness to lambdas and will make them a little more
awkward to use in some cases, however prevents several classes
of critical issues:
a. previously, a lambda that had an expression derived from
some kind of state, like "self.x", or "execution_context.session.foo"
would produce a closure cache key from "self" or "execution_context",
objects that can very well be per-execution and would therefore
cause a AnalyzedFunction objects to overflow. (memory won't leak
as it looks like an LRUCache is already used for these)
b. a lambda, such as one used within DeferredLamdaElement, that
produces different SQL expressions based on the arguments
(which is in fact what it's supposed to do), however it would
through the use of conditionals produce different bound parameter
combinations, leading to literal parameters not tracked properly.
These are now rejected as uncacheable whereas previously they would
again be part of the closure cache key, causing an overflow of
AnalyizedFunction objects.
3. Ensure non-mapped mixins are handled correctly by
with_loader_criteria().
4. Fixed bug in lambda SQL system where we are not supposed to allow a Python
function to be embedded in the lambda, since we can't predict a bound value
from it. While there was an error condition added for this, it was not
tested and wasn't working; an informative error is now raised.
5. new docs for lambdas
6. consolidated changelog for all of these
Fixes: #5760Fixes: #5765Fixes: #5766Fixes: #5768Fixes: #5770
Change-Id: Iedaa636c3225fad496df23b612c516c8ab247ab7
Fixed bug in lambda SQL feature, used by ORM
:meth:`_orm.with_loader_criteria` as well as available generally in the SQL
expression language, where assigning a boolean value True/False to a
variable would cause the query-time expression calculation to fail, as it
would produce a SQL expression not compatible with a bound value.
Fixed issue where the :attr:`_orm.ORMExecuteState.is_relationship_load`
parameter would not be set correctly for many lazy loads, all
selectinloads, etc. The flag is essential in order to test if options
should be added to statements or if they would already have been propagated
via relationship loads.
Fixes: #5763Fixes: #5764
Change-Id: I66aafbef193f892ff75ede0670698647b7475482
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function. it does not
yet fully address other issues such as keyword arguments passed
to the table.select().
Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.
Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
in order to accommodate relationship loaders
with lambda caching, a lot more is needed. This is
a full refactor of the lambda system such that it
now has two levels of caching; the first level caches what
can be known from the __code__ element, then the next level
of caching is against the lambda itself and the contents
of __closure__. This allows for the elements inside
the lambdas, like columns and entities, to change and
then be part of the cache key. Lazy/selectinloads' use of
baked queries had to add distinct cache key elements,
which was attempted here but overall things needed to be
more robust than that.
This commit is broken out from the very long and sprawling
commit at Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541 .
Change-Id: I29a513c98917b1d503abfdd61e6b6e8800851aa8
The coercions system allows us to add in lambdas as arguments
to Core and ORM elements without changing them at all. By allowing
the lambda to produce a deterministic cache key where we can also
cheat and yank out literal parameters means we can move towards
having 90% of "baked" functionality in a clearer way right in
Core / ORM.
As a second step, we can have whole statements inside the lambda,
and can then add generation with __add__(), so then we have
100% of "baked" functionality with full support of ad-hoc
literal values.
Adds some more short_selects tests for the moment for comparison.
Other tweaks inside cache key generation as we're trying to
approach a certain level of performance such that we can
remove the use of "baked" from the loader strategies.
As we have not yet closed#4639, however the caching feature
has been fully integrated as of
b0cfa7379c, we will also
add complete caching documentation here and close that issue
as well.
Closes: #4639Fixes: #5380
Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
Mike Bayer