The :meth:`_types.ARRAY.Comparator.any` and
:meth:`_types.ARRAY.Comparator.all` methods for the :class:`_types.ARRAY`
type are now deprecated for removal; these two methods along with
:func:`_postgresql.Any` and :func:`_postgresql.All` have been legacy for
some time as they are superseded by the :func:`_sql.any_` and
:func:`_sql.all_` functions, which feature more intutive use.
Fixes: #10821
Change-Id: I8eb3bbcb98af4ee60a21767dc3bdac771cbc0b4c
Fixed issue where the :func:`_sql.any_` and :func:`_sql.all_` aggregation
operators would not correctly coerce the datatype of the compared value, in
those cases where the compared value were not a simple int/str etc., such
as a Python ``Enum`` or other custom value. This would lead to execution
time errors for these values. This issue is essentially the same as
🎫`6515` which was for the now-legacy :meth:`.ARRAY.any` and
:meth:`.ARRAY.all` methods.
Fixes: #12874
Change-Id: I980894c23b9974bc84d584a1a4c5fae72dded6d3
Added a new concept of "operator classes" to the SQL operators supported by
SQLAlchemy, represented within the enum :class:`.OperatorClass`. The
purpose of this structure is to provide an extra layer of validation when a
particular kind of SQL operation is used with a particular datatype, to
catch early the use of an operator that does not have any relevance to the
datatype in use; a simple example is an integer or numeric column used with
a "string match" operator.
Fixes: #12736
Change-Id: I44f46d7326aef6847dbf0cf7a325833f8e347da6
Improved the behavior of standalone "operators" like :func:`_sql.desc`,
:func:`_sql.asc`, :func:`_sql.all_`, etc. so that they consult the given
expression object for an overriding method for that operator, even if the
object is not itself a ``ClauseElement``, such as if it's an ORM attribute.
This allows custom comparators for things like :func:`_orm.composite` to
provide custom implementations of methods like ``desc()``, ``asc()``, etc.
Added default implementations of :meth:`.ColumnOperators.desc`,
:meth:`.ColumnOperators.asc`, :meth:`.ColumnOperators.nulls_first`,
:meth:`.ColumnOperators.nulls_last` to :func:`_orm.composite` attributes,
by default applying the modifier to all contained columns. Can be
overridden using a custom comparator.
Fixes: #12769
Change-Id: I055ce79bf7ac31fb61d48bc3ab34799d42fb6336
Fixed issue where :func:`.select` of a free-standing, unnamed scalar expression that
has a unary operator applied, such as negation, would not apply result
processors to the selected column even though the correct type remains in
place for the unary expression.
This change opened up a typing rabbithole where we were led to also
improve and harden the typing for the Exists element, in particular
in that the Exists now always refers to a ScalarSelect object, and
no longer a SelectStatementGrouping within the _regroup() cases; there
did not seem to be any reason for this inconsistency.
Fixes: #12681
Change-Id: If9131807941030c627ab31ede4ccbd86e44e707f
Allow custom operator systems to use the @ Python operator (#12479).
### Description
Add a dummy implementation for the `__matmul__` operator rasing `NotImplementedError` by default.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [X] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #12583
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12583
Pull-request-sha: 7e69d23610
Change-Id: Ia0d565decd437b940efd3b97478c16d7a0377bc6
Added support for the pow operator (``**``), with a default SQL
implementation of the ``POW()`` function. On Oracle Database, PostgreSQL
and MSSQL it renders as ``POWER()``. As part of this change, the operator
routes through a new first class ``func`` member :class:`_functions.pow`,
which renders on Oracle Database, PostgreSQL and MSSQL as ``POWER()``.
Fixes: #8579Closes: #8580
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8580
Pull-request-sha: 041b2ef474
Change-Id: I371bd44ed3e58f2d55ef705aeec7d04710c97f23
Fixed SQL composition bug which impacted caching where using a ``None``
value inside of an ``in_()`` expression would bypass the usual "expanded
bind parameter" logic used by the IN construct, which allows proper caching
to take place.
Fixes: #12314
References: #12312
Change-Id: I0d2fc4e15c73407379ba368dd4ee32660fc66259
the :class:`.Numeric` and :class:`.Float` SQL types have been separated out
so that :class:`.Float` no longer inherits from :class:`.Numeric`; instead,
they both extend from a common mixin :class:`.NumericCommon`. This
corrects for some architectural shortcomings where numeric and float types
are typically separate, and establishes more consistency with
:class:`.Integer` also being a distinct type. The change should not have
any end-user implications except for code that may be using
``isinstance()`` to test for the :class:`.Numeric` datatype; third party
dialects which rely upon specific implementation types for numeric and/or
float may also require adjustment to maintain compatibility.
Fixes: #5252
Change-Id: Iadc841340b3d97e3eb5f7e63f0a0cc3cb4e30f74
The ``.c`` and ``.columns`` attributes on the :class:`.Select` and
:class:`.TextualSelect` constructs, which are not instances of
:class:`.FromClause`, have been removed completely, in addition to the
``.select()`` method as well as other codepaths which would implicitly
generate a subquery from a :class:`.Select` without the need to explicitly
call the :meth:`.Select.subquery` method.
In the case of ``.c`` and ``.columns``, these attributes were never useful
in practice and have caused a great deal of confusion, hence were
deprecated back in version 1.4, and have emitted warnings since that
version. Accessing the columns that are specific to a :class:`.Select`
construct is done via the :attr:`.Select.selected_columns` attribute, which
was added in version 1.4 to suit the use case that users often expected
``.c`` to accomplish. In the larger sense, implicit production of
subqueries works against SQLAlchemy's modern practice of making SQL
structure as explicit as possible.
Note that this is **not related** to the usual :attr:`.FromClause.c` and
:attr:`.FromClause.columns` attributes, common to objects such as
:class:`.Table` and :class:`.Subquery`, which are unaffected by this
change.
Fixes: #10236
Change-Id: If241b8674ccacce7e860bfed25b5d266bfe1aca7
Fixed regression caused by an internal code change in response to recent
Mypy releases that caused the very unusual case of a list of ORM-mapped
attribute expressions passed to :meth:`.ColumnOperators.in_` to no longer
be accepted.
in this commit we had to revisit d8dd28c42e where mypy typing
didn't accept ColumnOperartors. the type here is the _HasClauseElement[_T]
protocol which means we need to use a duck type for a runtime check.
Fixes: #12019
Change-Id: Ib378e9cb8defb49d5ac4d726ec93d6bdc581b6a9
manually update the files to remove literal string concat on the same line,
since black does not seem to be making progress in handling these
Change-Id: I3c651374c5f3db5b8bc0c700328d67ca03743b7b
Improved compilation of :func:`_sql.any_` / :func:`_sql.all_` in the
context of a negation of boolean comparison, will now render ``NOT (expr)``
rather than reversing the equality operator to not equals, allowing
finer-grained control of negations for these non-typical operators.
Fixes: #10817
Change-Id: If0b324b1220ad3c7f053af91e8a61c81015f312a
Added support for :ref:`oracledb` in async mode.
The current implementation has some limitation, preventing
the support for :meth:`_asyncio.AsyncConnection.stream`.
Improved support if planned for the 2.1 release of SQLAlchemy.
Fixes: #10679
Change-Id: Iff123cf6241bcfa0fbac57529b80f933951be0a7
the expression clauselist feature added in #7744 failed to accommodate
this parameter that is used only by the PostgreSQL JSON
operators.
Fixed 2.0 regression caused by 🎫`7744` where chains of expressions
involving PostgreSQL JSON operators combined with other operators such as
string concatenation would lose correct parenthesization, due to an
implementation detail specific to the PostgreSQL dialect.
Fixes: #10479
Change-Id: Ic168bf6afd8bf1cfa648f2bad22fdd7254feaa34
Adjusted the operator precedence for the string concatenation operator to
be equal to that of string matching operators, such as
:meth:`.ColumnElement.like`, :meth:`.ColumnElement.regexp_match`,
:meth:`.ColumnElement.match`, etc., as well as plain ``==`` which has the
same precedence as string comparison operators, so that parenthesis will be
applied to a string concatenation expression that follows a string match
operator. This provides for backends such as PostgreSQL where the "regexp
match" operator is apparently of higher precedence than the string
concatenation operator.
Fixes: #9610
Change-Id: I73640e40e445375177340e1ed8f45b5da98d6dfb
Fixed issue where unpickling of a :class:`_schema.Column` or other
:class:`_sql.ColumnElement` would fail to restore the correct "comparator"
object, which is used to generate SQL expressions specific to the type
object.
Fixes: #10213
Change-Id: I74e805024bcc0d93d549bd94757c2865b3117d72
Fixed issue where the :meth:`_sql.ColumnOperators.regexp_match`
when using "flags" would not produce a "stable" cache key, that
is, the cache key would keep changing each time causing cache pollution.
The same issue existed for :meth:`_sql.ColumnOperators.regexp_replace`
with both the flags and the actual replacement expression.
The flags are now represented as fixed modifier strings rendered as
safestrings rather than bound parameters, and the replacement
expression is established within the primary portion of the "binary"
element so that it generates an appropriate cache key.
Note that as part of this change, the
:paramref:`_sql.ColumnOperators.regexp_match.flags` and
:paramref:`_sql.ColumnOperators.regexp_replace.flags` have been modified to
render as literal strings only, whereas previously they were rendered as
full SQL expressions, typically bound parameters. These parameters should
always be passed as plain Python strings and not as SQL expression
constructs; it's not expected that SQL expression constructs were used in
practice for this parameter, so this is a backwards-incompatible change.
The change also modifies the internal structure of the expression
generated, for :meth:`_sql.ColumnOperators.regexp_replace` with or without
flags, and for :meth:`_sql.ColumnOperators.regexp_match` with flags. Third
party dialects which may have implemented regexp implementations of their
own (no such dialects could be located in a search, so impact is expected
to be low) would need to adjust the traversal of the structure to
accommodate.
Fixed issue in mostly-internal :class:`.CacheKey` construct where the
``__ne__()`` operator were not properly implemented, leading to nonsensical
results when comparing :class:`.CacheKey` instances to each other.
Fixes: #10042
Change-Id: I2e245f81d7ee7136ad04cf77be35f9745c5da5e5
Fixed issue where the :paramref:`.ColumnOperators.like.escape` and similar
parameters did not allow an empty string as an argument that would be
passed through as the "escape" character; this is a supported syntax by
PostgreSQL. Pull requset courtesy Martin Caslavsky.
Fixes: #9907Closes: #9908
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9908
Pull-request-sha: d7ecc1778a
Change-Id: I39adb765a1b9650fe891883ed0973df66adc4e81
Fixed issue where element types of a tuple value would be hardcoded to take
on the types from a compared-to tuple, when the comparison were using the
:meth:`.ColumnOperators.in_` operator. This was inconsistent with the usual
way that types are determined for a binary expression, which is that the
actual element type on the right side is considered first before applying
the left-hand-side type.
Fixes: #9313
Change-Id: Ia8874c09682a6512fcf4084cf14481024959c461
Fixed critical regression in SQL expression formulation in the 2.0 series
due to 🎫`7744` which improved support for SQL expressions that
contained many elements against the same operator repeatedly; parenthesis
grouping would be lost with expression elements beyond the first two
elements.
Fixes: #9271
Change-Id: Ib6ed5b71efe0f6816dab75bda622297fc89e3b49
Added a full suite of new SQL bitwise operators, for performing
database-side bitwise expressions on appropriate data values such as
integers, bit-strings, and similar. Pull request courtesy Yegor Statkevich.
Fixes: #8780Closes: #9204
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9204
Pull-request-sha: a4541772a6
Change-Id: I4c70e80f9548dcc1b4e3dccd71bd59d51d3ed46e
The :meth:`_sql.ColumnOperators.in_` and
:meth:`_sql.ColumnOperators.not_in_` are typed to include
``Iterable[Any]`` rather than ``Sequence[Any]`` for more flexibility in
argument type.
The :func:`_sql.or_` and :func:`_sql.and_` from a typing perspective
require the first argument to be present, however these functions still
accept zero arguments which will emit a deprecation warning at runtime.
Typing is also added to support sending the fixed literal ``False`` for
:func:`_sql.or_` and ``True`` for :func:`_sql.and_` as the first argument
only, however the documentation now indicates sending the
:func:`_sql.false` and :func:`_sql.true` constructs in these cases as a
more explicit approach.
Fixed typing issue where iterating over a :class:`_orm.Query` object
was not correctly typed.
Fixes: #9122Fixes: #9123Fixes: #9125
Change-Id: I500e3e1b826717b3dd49afa1e682c3c8279c9226
Added :class:`_expression.ScalarValues` that can be used as a column
element allowing using :class:`_expression.Values` inside IN clauses
or in conjunction with ``ANY`` or ``ALL`` collection aggregates.
This new class is generated using the method
:meth:`_expression.Values.scalar_values`.
The :class:`_expression.Values` instance is now coerced to a
:class:`_expression.ScalarValues` when used in a ``IN`` or ``NOT IN``
operation.
Fixes: #6289
Change-Id: Iac22487ccb01553684b908e54d01c0687fa739f1
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
Added long-requested case-insensitive string operators
:meth:`_sql.ColumnOperators.icontains`,
:meth:`_sql.ColumnOperators.istartswith`,
:meth:`_sql.ColumnOperators.iendswith`, which produce case-insensitive
LIKE compositions (using ILIKE on PostgreSQL, and the LOWER() function on
all other backends) to complement the existing LIKE composition operators
:meth:`_sql.ColumnOperators.contains`,
:meth:`_sql.ColumnOperators.startswith`, etc. Huge thanks to Matias
Martinez Rebori for their meticulous and complete efforts in implementing
these new methods.
Fixes: #3482Closes: #8496
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8496
Pull-request-sha: 7287e2c436
Change-Id: I9fcdd603716218067547cc92a2b07bd02a2c366b
Adjusted the SQL compilation for string containment functions
``.contains()``, ``.startswith()``, ``.endswith()`` to force the use of the
string concatenation operator, rather than relying upon the overload of the
addition operator, so that non-standard use of these operators with for
example bytestrings still produces string concatenation operators.
To accommodate this, needed to add a new _rconcat operator function,
which is private, as well as a fallback in concat_op() that works
similarly to Python builtin ops.
Fixes: #8253
Change-Id: I2b7f56492f765742d88cb2a7834ded6a2892bd7e
The :meth:`.Operators.match` operator now uses ``plainto_tsquery()`` for
PostgreSQL full text search, rather than ``to_tsquery()``. The rationale
for this change is to provide better cross-compatibility with match on
other database backends. Full support for all PostgreSQL full text
functions remains available through the use of :data:`.func` in
conjunction with :meth:`.Operators.bool_op` (an improved version of
:meth:`.Operators.op` for boolean operators).
Additional doc updates here apply to 1.4 so will backport these
out to a separate commit.
Fixes: #7086
Change-Id: I1946075daf5d9c558e85f73f1bf852604b3b1b8c
Fixed an issue where using :func:`.bindparam` with no explicit data or type
given could be coerced into the incorrect type when used in expressions
such as when using :meth:`.ARRAY.comparator.any` and
:meth:`.ARRAY.comparator.all`.
Fixes: #7979
Change-Id: If7779e713c9a3a5fee496b66e417cfd3fca5b1f9
Fixed bug in :class:`.ARRAY` datatype in combination with :class:`.Enum` on
PostgreSQL where using the ``.any()`` method to render SQL ANY(), given
members of the Python enumeration as arguments, would produce a type
adaptation failure on all drivers.
Fixes: #6515
Change-Id: Ia1e3b4e10aaf264ed436ce6030d105fc60023433
Improved the construction of SQL binary expressions to allow for very long
expressions against the same associative operator without special steps
needed in order to avoid high memory use and excess recursion depth. A
particular binary operation ``A op B`` can now be joined against another
element ``op C`` and the resulting structure will be "flattened" so that
the representation as well as SQL compilation does not require recursion.
To implement this more cleanly, the biggest change here is that
column-oriented lists of things are broken away from ClauseList
in a new class ExpressionClauseList, that also forms the basis
of BooleanClauseList. ClauseList is still used for the generic
"comma-separated list" of things such as Tuple and things like
ORDER BY, as well as in some API endpoints.
Also adds __slots__ to the TypeEngine-bound Comparator
classes. Still can't really do __slots__ on ClauseElement.
Fixes: #7744
Change-Id: I81a8ceb6f8f3bb0fe52d58f3cb42e4b6c2bc9018
Improvements to the test suite's integration with pytest such that the
"warnings" plugin, if manually enabled, will not interfere with the test
suite, such that third parties can enable the warnings plugin or make use
of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass.
Additionally, modernized the detection of the "pytest-xdist" plugin so that
plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
without breaking the test suite if xdist were still installed. Warning
filters that promote deprecation warnings to errors are now localized to
SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for
general Python deprecation warnings, so that non-SQLAlchemy deprecation
warnings emitted from pytest plugins should also not impact the test suite.
Fixes: #7599
Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
Added new parameter :paramref:`_sql.Operators.op.python_impl`, available
from :meth:`_sql.Operators.op` and also when using the
:class:`_sql.Operators.custom_op` constructor directly, which allows an
in-Python evaluation function to be provided along with the custom SQL
operator. This evaluation function becomes the implementation used when the
operator object is used given plain Python objects as operands on both
sides, and in particular is compatible with the
``synchronize_session='evaluate'`` option used with
:ref:`orm_expression_update_delete`.
Fixes: #3162
Change-Id: If46ba6a0e303e2180a177ba418a8cafe9b42608e
### Description
Found via `codespell -q 3 -L ba,crate,datas,froms,gord,hist,inh,nd,selectin,strat,ue`
Also added codespell to the pep8 tox env
### Checklist
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
Closes: #7338
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7338
Pull-request-sha: 0deac22193
Change-Id: Icd61db31c8dc655d4a39d8a304194804d08555fe
Implemented full support for "truediv" and "floordiv" using the
"/" and "//" operators. A "truediv" operation between two expressions
using :class:`_types.Integer` now considers the result to be
:class:`_types.Numeric`, and the dialect-level compilation will cast
the right operand to a numeric type on a dialect-specific basis to ensure
truediv is achieved. For floordiv, conversion is also added for those
databases that don't already do floordiv by default (MySQL, Oracle) and
the ``FLOOR()`` function is rendered in this case, as well as for
cases where the right operand is not an integer (needed for PostgreSQL,
others).
The change resolves issues both with inconsistent behavior of the
division operator on different backends and also fixes an issue where
integer division on Oracle would fail to be able to fetch a result due
to inappropriate outputtypehandlers.
Fixes: #4926
Change-Id: Id54cc018c1fb7a49dd3ce1216d68d40f43fe2659
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Fixes: #7394
Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
Fixes: #7300
Change-Id: I6255c850b140522a4aba95085216d0bca18ce230
Fixed an inconsistency in the any_() / all_() functions / methods where the
special behavior these functions have of "flipping" the expression such
that the "ANY" / "ALL" expression is always on the right side would not
function if the comparison were against the None value, that is,
"column.any_() == None" should produce the same SQL expression as "null()
== column.any_()". Added more docs to clarify this as well, plus mentions
that any_() / all_() generally supersede the ARRAY version "any()" /
"all()".
Fixes: #7140
Change-Id: Ia5d55414ba40eb3fbda3598931fdd24c9b4a4411
Adjusted the logic added as part of 🎫`6397` in 1.4.12 so that
internal mutation of the :class:`.BindParameter` object occurs within the
clause construction phase as it did before, rather than in the compilation
phase. In the latter case, the mutation still produced side effects against
the incoming construct and additionally could potentially interfere with
other internal mutation routines.
In order to solve the issue of the correct operator being present
on the BindParameter.expand_op, we necessarily have to expand the
BinaryExpression._negate() routine to flip the operator on the
BindParameter also.
Fixes: #6460
Change-Id: I1e53a9aeee4de4fc11af51d7593431532731561b
Mike Bayer