Added support for Python 3.14+ template strings (t-strings) via the new
:func:`_sql.tstring` construct, as defined in :pep:`750`. This feature
allows for ergonomic SQL statement construction by automatically
interpolating Python values and SQLAlchemy expressions within template
strings.
Part of the challenge here is the syntax only works on py314, so we have
to exclude the test file at many levels when py314 is not used. not
sure yet how i want to adjust pep8 tests and rules for this.
Fixes: #12548
Change-Id: Ia060d1387ff452fe4f5d924f683529a22a8e1f72
Added new implementation for the :meth:`.Select.params` method and that of
similar statements, via a new statement-only
:meth:`.ExecutableStatement.params` method which works more efficiently and
correctly than the previous implementations available from
:class:`.ClauseElement`, by assocating the given parameter dictionary with
the statement overall rather than cloning the statement and rewriting its
bound parameters. The :meth:`_sql.ClauseElement.params` and
:meth:`_sql.ClauseElement.unique_params` methods, when called on an object
that does not implement :class:`.ExecutableStatement`, will continue to
work the old way of cloning the object, and will emit a deprecation
warning. This issue both resolves the architectural / performance
concerns of 🎫`7066` and also provides correct ORM compatibility for
functions like :func:`_orm.aliased`, reported by 🎫`12915`.
Fixes: #7066
Change-Id: I6543c7d0f4da3232b3641fb172c24c446f02f52a
use pyproject.toml so the options are safe from "rm .coverage*"
types of commands
omit some files
add test coverage for a big section of cache-key
Change-Id: Ia540c632d91ec09284e187e3edeb8ccf0214e1a6
Updated the :func:`_sql.over` clause to allow non-integer values in
:paramref:`_sql.over.range_` clause. Previously, only integer values
were allowed and any other values would lead to a failure.
To specify a non-integer value, use the new :class:`_sql.FrameClause`
construct along with the new :class:`_sql.FrameClauseType` enum to specify
the frame boundaries.
Fixes#12596.
Closes: #12695
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12695
Pull-request-sha: 8063cec65d
Change-Id: I248a938f6502d72555c005d86791c992822117d4
Fixed a caching issue where :func:`_orm.with_loader_criteria` would
incorrectly reuse cached bound parameter values when used with
:class:`_sql.CompoundSelect` constructs such as :func:`_sql.union`. The
issue was caused by the cache key for compound selects not including the
execution options that are part of the :class:`_sql.Executable` base class,
which :func:`_orm.with_loader_criteria` uses to apply its criteria
dynamically. The fix ensures that compound selects and other executable
constructs properly include execution options in their cache key traversal.
Fixes: #12905
Change-Id: I24bbd96233cddabe42eb716f078eea4c84b1af05
Added new generalized aggregate function ordering to functions via the
:func:`_functions.FunctionElement.aggregate_order_by` method, which
receives an expression and generates the appropriate embedded "ORDER BY" or
"WITHIN GROUP (ORDER BY)" phrase depending on backend database. This new
function supersedes the use of the PostgreSQL
:func:`_postgresql.aggregate_order_by` function, which remains present for
backward compatibility. To complement the new parameter, the
:paramref:`_functions.aggregate_strings.order_by` which adds ORDER BY
capability to the :class:`_functions.aggregate_strings` dialect-agnostic
function which works for all included backends. Thanks much to Reuven
Starodubski with help on this patch.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #12853Closes: #12856
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12856
Pull-request-sha: d93fb591751227eb1f96052ea3ad449f511f70b3
Change-Id: I8eb41ff2d57695963a358b5f0017ca9372f15f70
Improved the behavior of standalone "operators" like :func:`_sql.desc`,
:func:`_sql.asc`, :func:`_sql.all_`, etc. so that they consult the given
expression object for an overriding method for that operator, even if the
object is not itself a ``ClauseElement``, such as if it's an ORM attribute.
This allows custom comparators for things like :func:`_orm.composite` to
provide custom implementations of methods like ``desc()``, ``asc()``, etc.
Added default implementations of :meth:`.ColumnOperators.desc`,
:meth:`.ColumnOperators.asc`, :meth:`.ColumnOperators.nulls_first`,
:meth:`.ColumnOperators.nulls_last` to :func:`_orm.composite` attributes,
by default applying the modifier to all contained columns. Can be
overridden using a custom comparator.
Fixes: #12769
Change-Id: I055ce79bf7ac31fb61d48bc3ab34799d42fb6336
The :func:`_sql.values` construct gains a new method :meth:`_sql.Values.cte`,
which allows creation of a named, explicit-columns :class:`.CTE` against an
unnamed ``VALUES`` expression, producing a syntax that allows column-oriented
selection from a ``VALUES`` construct on modern versions of PostgreSQL, SQLite,
and MariaDB.
Fixes: #12734
Change-Id: I4a0146418420cce3cbbda4e50f5eb32206dc696b
Added new Core feature :func:`_sql.from_dml_column` that may be used in
expressions inside of :meth:`.UpdateBase.values` for INSERT or UPDATE; this
construct will copy whatever SQL expression is used for the given target
column in the statement to be used with additional columns. The construct
is mostly intended to be a helper with ORM :class:`.hybrid_property` within
DML hooks.
This is the Core side of the feature being added to the ORM
for #12496
Change-Id: Ic568638a8ce3607deea44af988b6451b30cde36c
Added new datatype :class:`_oracle.VECTOR` and accompanying DDL and DQL
support to fully support this type for Oracle Database. This change
includes the base :class:`_oracle.VECTOR` type that adds new type-specific
methods ``l2_distance``, ``cosine_distance``, ``inner_product`` as well as
new parameters ``oracle_vector`` for the :class:`.Index` construct,
allowing vector indexes to be configured, and ``oracle_fetch_approximate``
for the :meth:`.Select.fetch` clause. Pull request courtesy Suraj Shaw.
Fixes: #12317Closes: #12321
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12321
Pull-request-sha: a72a18a45c
Change-Id: I6f3af4623ce439d0820c14582cd129df293f0ba8
Implemented support for the GROUPS frame specification in window functions
by adding :paramref:`_sql.over.groups` option to :func:`_sql.over`
and :meth:`.FunctionElement.over`. Pull request courtesy Kaan Dikmen.
Fixes: #12450Closes: #12445
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12445
Pull-request-sha: c0808e135f
Change-Id: I9ff504a9c9650485830c4a0eaf44162898a3a2ad
When building a PostgreSQL ``ARRAY`` literal using
:class:`_postgresql.array` with an empty ``clauses`` argument, the
:paramref:`_postgresql.array.type_` parameter is now significant in that it
will be used to render the resulting ``ARRAY[]`` SQL expression with a
cast, such as ``ARRAY[]::INTEGER``. Pull request courtesy Denis Laxalde.
Fixes: #12432Closes: #12435
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12435
Pull-request-sha: 9633d3c15d
Change-Id: I29ed7bd0562b82351d22de0658fb46c31cfe44f6
Added the ability to create custom SQL constructs that can define new
clauses within SELECT, INSERT, UPDATE, and DELETE statements without
needing to modify the construction or compilation code of of
:class:`.Select`, :class:`.Insert`, :class:`.Update`, or :class:`.Delete`
directly. Support for testing these constructs, including caching support,
is present along with an example test suite. The use case for these
constructs is expected to be third party dialects for NewSQL or other novel
styles of database that introduce new clauses to these statements. A new
example suite is included which illustrates the ``QUALIFY`` SQL construct
used by several NewSQL databases which includes a cachable implementation
as well as a test suite.
Since these extensions start to make it a bit crowded with how many
kinds of "options" we have on statements, did some naming /
documentation changes with existing constructs on Executable, in
particular to distinguish ExecutableOption from SyntaxExtension.
Fixes: #12195
Change-Id: I4a44ee5bbc3d8b1b640837680c09d25b1b7077af
Fixed SQL composition bug which impacted caching where using a ``None``
value inside of an ``in_()`` expression would bypass the usual "expanded
bind parameter" logic used by the IN construct, which allows proper caching
to take place.
Fixes: #12314
References: #12312
Change-Id: I0d2fc4e15c73407379ba368dd4ee32660fc66259
Follow up of 🎫`11471` to fix caching issue where using the
:meth:`.CompoundSelectState.add_cte` method of the
:class:`.CompoundSelectState` construct would not set a correct cache key
which distinguished between different CTE expressions. Also added tests
that would detect issues similar to the one fixed in 🎫`11544`.
Fixes: #11471
Change-Id: Iae6a91077c987d83cd70ea826daff42855491330
Fixed caching issue where the
:paramref:`_sql.Select.with_for_update.key_share` element of
:meth:`_sql.Select.with_for_update` was not considered as part of the cache
key, leading to incorrect caching if different variations of this parameter
were used with an otherwise identical statement.
Also repairs a traversal issue where the ``of`` element of
``ForUpdateArg`` when set to ``None`` cannot be compared against a
non-None element because the traversal defines it as a clauselist.
Traversal in this case is adjusted to accommodate for this case so that
we dont need to create a risky-to-backport change to ``ForUpdateArg``
itself.
Fixes: #11544
Change-Id: Ie8a50716df06977af58b0c22a8c10e1b64d972b9
Enhanced the caching structure of the :paramref:`.over.rows` and
:paramref:`.over.range` so that different numerical values for the rows /
range fields are cached on the same cache key, to the extent that the
underlying SQL does not actually change (i.e. "unbounded", "current row",
negative/positive status will still change the cache key). This prevents
the use of many different numerical range/rows value for a query that is
otherwise identical from filling up the SQL cache.
Note that the semi-private compiler method ``_format_frame_clause()``
is removed by this fix, replaced with a new method
``visit_frame_clause()``. Third party dialects which may have referred
to this method will need to change the name and revise the approach to
rendering the correct SQL for that dialect.
This patch introduces a new ClauseElement called _FrameClause which
stores the integer range values separately and within cache-compatible
BindParameter objects from the "type" which
can be unbounded, current, preceding, or following, represented by
a _FrameClauseType enum. The negative
sign is also stripped from the integer and represented within the
_FrameClauseType. Tests from #11514 are adapted to include
a test for SQL Server's "literal_execute" flag taking effect so
that literal numeric values aren't stored in the cache.
Fixes: #11515
Change-Id: I8aad368ffef9f06cb5c3f8c4e971fadef029ffd5
Fixed caching issue where using the :meth:`.TextualSelect.add_cte` method
of the :class:`.TextualSelect` construct would not set a correct cache key
which distinguished between different CTE expressions.
Fixes: #11471
Change-Id: Ia9ce2c8cfd128f0f130aa9b26448dc23d994c324
Adjusted the fix made in 🎫`10570`, released in 2.0.23, where new
logic was added to reconcile possibly changing bound parameter values
across cache key generations used within the :func:`_orm.with_expression`
construct. The new logic changes the approach by which the new bound
parameter values are associated with the statement, avoiding the need to
deep-copy the statement which can result in a significant performance
penalty for very deep / complex SQL constructs. The new approach no longer
requires this deep-copy step.
Fixes: #11085
Change-Id: Ia51eb4e949c8f37af135399925a9916b9ed4ad2f
Fixed issue where the :meth:`_sql.ColumnOperators.regexp_match`
when using "flags" would not produce a "stable" cache key, that
is, the cache key would keep changing each time causing cache pollution.
The same issue existed for :meth:`_sql.ColumnOperators.regexp_replace`
with both the flags and the actual replacement expression.
The flags are now represented as fixed modifier strings rendered as
safestrings rather than bound parameters, and the replacement
expression is established within the primary portion of the "binary"
element so that it generates an appropriate cache key.
Note that as part of this change, the
:paramref:`_sql.ColumnOperators.regexp_match.flags` and
:paramref:`_sql.ColumnOperators.regexp_replace.flags` have been modified to
render as literal strings only, whereas previously they were rendered as
full SQL expressions, typically bound parameters. These parameters should
always be passed as plain Python strings and not as SQL expression
constructs; it's not expected that SQL expression constructs were used in
practice for this parameter, so this is a backwards-incompatible change.
The change also modifies the internal structure of the expression
generated, for :meth:`_sql.ColumnOperators.regexp_replace` with or without
flags, and for :meth:`_sql.ColumnOperators.regexp_match` with flags. Third
party dialects which may have implemented regexp implementations of their
own (no such dialects could be located in a search, so impact is expected
to be low) would need to adjust the traversal of the structure to
accommodate.
Fixed issue in mostly-internal :class:`.CacheKey` construct where the
``__ne__()`` operator were not properly implemented, leading to nonsensical
results when comparing :class:`.CacheKey` instances to each other.
Fixes: #10042
Change-Id: I2e245f81d7ee7136ad04cf77be35f9745c5da5e5
Fixed critical SQL caching issue where use of the :meth:`_sql.Operators.op`
custom operator function would not produce an appropriate cache key,
leading to reduce the effectiveness of the SQL cache.
Fixes: #9506
Change-Id: I3eab1ddb5e09a811ad717161a59df0884cdf70ed
Added :class:`_expression.ScalarValues` that can be used as a column
element allowing using :class:`_expression.Values` inside IN clauses
or in conjunction with ``ANY`` or ``ALL`` collection aggregates.
This new class is generated using the method
:meth:`_expression.Values.scalar_values`.
The :class:`_expression.Values` instance is now coerced to a
:class:`_expression.ScalarValues` when used in a ``IN`` or ``NOT IN``
operation.
Fixes: #6289
Change-Id: Iac22487ccb01553684b908e54d01c0687fa739f1
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
also adjusted CacheKeyFixture to be a general purpose
fixture so that sub-components / dialects can run
their own cache key tests.
Fixes: #8574
Change-Id: I6c66107856aee11e548d357cea77bceee3e316a0
Fixed issue where use of the :func:`_sql.table` construct, passing a string
for the :paramref:`_sql.table.schema` parameter, would fail to take the
"schema" string into account when producing a cache key, thus leading to
caching collisions if multiple, same-named :func:`_sql.table` constructs
with different schemas were used.
Fixes: #8441
Change-Id: Ic4b55b3e8ec53b4c88ba112691bdf60ea1d4c448
the pep484 task becomes more intense as there is mounting
pressure to come up with a consistency in how data moves
from end-user to instance variable.
current thinking is coming into:
1. there are _typing._XYZArgument objects that represent "what the
user sent"
2. there's the roles, which represent a kind of "filter" for different
kinds of objects. These are mostly important as the argument
we pass to coerce().
3. there's the thing that coerce() returns, which should be what the
construct uses as its internal representation of the thing.
This is _typing._XYZElement.
but there's some controversy over whether or
not we should pass actual ClauseElements around by their role
or not. I think we shouldn't at the moment, but this makes the
"role-ness" of something a little less portable. Like, we have
to set DMLTableRole for TableClause, Join, and Alias, but then
also we have to repeat those three types in order to set up
_DMLTableElement.
Other change introduced here, there was a deannotate=True
for the left/right of a sql.join(). All tests pass without that.
I'd rather not have that there as if we have a join(A, B) where
A, B are mapped classes, we want them inside of the _annotations.
The rationale seems to be performance, but this performance can
be illustrated to be on the compile side which we hope is cached
in the normal case.
CTEs now accommodate for text selects including recursive.
Get typing to accommodate "util.preloaded" cleanly; add "preloaded"
as a real module. This seemed like we would have needed
pep562 `__getattr__()` but we don't, just set names in
globals() as we import them.
References: #6810
Change-Id: I34d17f617de2fe2c086fc556bd55748dc782faf0
The ``literal_execute`` parameter now takes part of the cache
generation of a bindparam, since it changes the sql string generated
by the compiler.
Previously the correct bind values were used, but the ``literal_execute``
would be ignored on subsequent executions of the same query.
Fixes: #7876
Change-Id: I6bf887f1a2fe31f9d0ab68f5b4ff315004d006b2
Fixed a regression in the test suite where the test called
``CompareAndCopyTest::test_all_present`` would fail on some platforms due
to additional testing artifacts being detected. Pull request courtesy Nils
Philippsen.
In some circumstances, ephemeral class objects that are created within
the scope of a test method don't seem to be garbage collected directly
on exit. Filter out classes created in test modules.
Fixes: #7450Closes: #7451
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7451
Pull-request-sha: 135a8aaba2
Change-Id: I621967bd916089dc1e3f98625fd2a852cd9fd712
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Fixes: #7394
Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
Extended the ``cache_ok`` flag and corresponding warning message if this
flag is not defined, a behavior first established for
:class:`.TypeDecorator` as part of 🎫`6436`, to also take place for
:class:`.UserDefinedType`, by generalizing the flag and associated caching
logic to a new common base for these two types, :class:`.ExternalType`.
The change means any current :class:`.UserDefinedType` will now cause SQL
statement caching to no longer take place for statements which make use of
the datatype, along with a warning being emitted, unless the class defines
the :attr:`.UserDefinedType.cache_ok` flag as True. If the datatype cannot
form a deterministic, hashable cache key derived from its arguments, it may return
False which will continue to keep caching disabled but will suppress the
warning. In particular, custom datatypes currently used in packages such as
SQLAlchemy-utils will need to implement this flag. The issue was observed
as a result of a SQLAlchemy-utils datatype that is not currently cacheable.
Fixes: #7319
Change-Id: Ie0b5d4587df87bfe66d2fe7cd4585c3882584575
Repaired issue in new :paramref:`_sql.HasCTE.cte.nesting` parameter
introduced with 🎫`4123` where a recursive :class:`_sql.CTE` using
:paramref:`_sql.HasCTE.cte.recursive` in typical conjunction with UNION
would not compile correctly. Additionally makes some adjustments so that
the :class:`_sql.CTE` construct creates a correct cache key.
Pull request courtesy Eric Masseran.
Fixes: #4123
> This has not been caught by the tests because the nesting recursive
queries there did not union against itself, eg there was only the i
root clause...
- Now tests are real recursive queries
- Add tests on aliased nested CTEs (recursive or not)
- Adapt the `_restates` attribute to use it as a reference
- Add some docs around to explain some variables usage
Closes: #7133
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7133
Pull-request-sha: 2633f34f7f
Change-Id: I15512c94e1bc1f52afc619d82057ca647d274e92
Fixed critical caching issue where the ORM's persistence feature using
INSERT..RETURNING would cache an incorrect query when mixing the "bulk
save" and standard "flush" forms of INSERT.
Fixes: #6793
Change-Id: Ifeb61c1226d3fa6d5e1c2e29b6f5ff77a27d6a2d
Added new method :meth:`_sql.HasCTE.add_cte` to each of the
:func:`_sql.select`, :func:`_sql.insert`, :func:`_sql.update` and
:func:`_sql.delete` constructs. This method will add the given
:class:`_sql.CTE` as an "independent" CTE of the statement, meaning it
renders in the WITH clause above the statement unconditionally even if it
is not otherwise referenced in the primary statement. This is a popular use
case on the PostgreSQL database where a CTE is used for a DML statement
that runs against database rows independently of the primary statement.
Fixes: #6752
Change-Id: Ibf635763e40269cbd10f4c17e208850d8e8d0188
Fixed regression where the :func:`_sql.tablesample` construct would fail to
be executable when constructed given a floating-point sampling value not
embedded within a SQL function.
Fixes: #6735
Change-Id: I557bcd4bdbffc4329ad69d5659ba99b1c8deb554
Fixed further regressions in the same area as that of 🎫`6052` where
loader options as well as invocations of methods like
:meth:`_orm.Query.join` would fail if the left side of the statement for
which the option/join depends upon were replaced by using the
:meth:`_orm.Query.with_entities` method, or when using 2.0 style queries
when using the :meth:`_sql.Select.with_only_columns` method. A new set of
state has been added to the objects which tracks the "left" entities that
the options / join were made against which is memoized when the lead
entities are changed.
Fixes: #6503Fixes: #6253
Change-Id: I211b2af98b0b20d1263fb15dc513884dcc5de6a4
The :class:`.TypeDecorator` class will now emit a warning when used in SQL
compilation with caching unless the ``.cache_ok`` flag is set to ``True``
or ``False``. ``.cache_ok`` indicates that all the parameters passed to the
object are safe to be used as a cache key, ``False`` means they are not.
Fixes: #6436
Change-Id: Ib1bb7dc4b124e38521d615c2e2e691e4915594fb
Enhanced the "expanding" feature used for :meth:`_sql.ColumnOperators.in_`
operations to infer the type of expression from the right hand list of
elements, if the left hand side does not have any explicit type set up.
This allows the expression to support stringification among other things.
In 1.3, "expanding" was not automatically used for
:meth:`_sql.ColumnOperators.in_` expressions, so in that sense this change
fixes a behavioral regression.
Fixes: #6222
Change-Id: Icdfda1e2c226a21896cafd6d8f251547794451c2
Fixed issue where using a ``func`` that includes dotted packagenames would
fail to be cacheable by the SQL caching system due to a Python list of
names that needed to be a tuple.
Fixes: #6101
Change-Id: I1d4bb5bf230b83596c59b6a04aa498f18ecd9613
Fixed issue in new 1.4/2.0 style ORM queries where a statement-level label
style would not be preserved in the keys used by result rows; this has been
applied to all combinations of Core/ORM columns / session vs. connection
etc. so that the linkage from statement to result row is the same in all
cases.
also repairs a cache key bug where query.from_statement()
vs. select().from_statement() would not be disambiguated; the
compile options were not included in the cache key for
FromStatement.
Fixes: #5933
Change-Id: I22f6cf0f0b3360e55299cdcb2452cead2b2458ea
Mike Bayer