Fixed issue where the :meth:`_sql.BindParameter.render_literal_execute`
method would fail when called on a parameter that also had ORM annotations
associated with it. In practice, this would be observed as a failure of SQL
compilation when using some combinations of a dialect that uses "FETCH
FIRST" such as Oracle along with a :class:`_sql.Select` construct that uses
:meth:`_sql.Select.limit`, within some ORM contexts, including if the
statement were embedded within a relationship primaryjoin expression.
Fixes: #9526
Change-Id: I2f512b6760a90293d274e60b06a891f10b276ecc
Fixed critical regression in PostgreSQL dialects such as asyncpg which rely
upon explicit casts in SQL in order for datatypes to be passed to the
driver correctly, where a :class:`.String` datatype would be cast along
with the exact column length being compared, leading to implicit truncation
when comparing a ``VARCHAR`` of a smaller length to a string of greater
length regardless of operator in use (e.g. LIKE, MATCH, etc.). The
PostgreSQL dialect now omits the length from ``VARCHAR`` when rendering
these casts.
Fixes: #9511
Change-Id: If094146d8cfd989a0b780872f38e86fd41ebfec2
Fixed critical SQL caching issue where use of the :meth:`_sql.Operators.op`
custom operator function would not produce an appropriate cache key,
leading to reduce the effectiveness of the SQL cache.
Fixes: #9506
Change-Id: I3eab1ddb5e09a811ad717161a59df0884cdf70ed
in I4e0b627bfa187f1780dc68ec81b94db1c78f846a the 1.4 version has more
changes than the main version, which failed to get the entire change,
yet the whole thing was merged. Restore the missing mutex related
code to the main version.
Fixed regression where the fix for 🎫`8098`, which was released in
the 1.4 series and provided a layer of concurrency-safe checks for the
lambda SQL API, included additional fixes in the patch that failed to be
applied to the main branch. These additional fixes have been applied.
Change-Id: Id172e09c421dafa6ef1d40b383aa4371de343864
References: #8098Fixes: #9461
Fixed regression involving pickling of Python rows between the cython and
pure Python implementations of :class:`.Row`, which occurred as part of
refactoring code for version 2.0 with typing. A particular constant were
turned into a string based ``Enum`` for the pure Python version of
:class:`.Row` whereas the cython version continued to use an integer
constant, leading to deserialization failures.
Regression occurred in a4bb502cf9Fixes: #9423
Change-Id: Icbd85cacb2d589cef7c246de7064249926146f2e
Fixed regression where the :func:`_sql.select` construct would not be able
to render if it were given no columns and then used in the context of an
EXISTS, raising an internal exception instead. While an empty "SELECT" is
not typically valid SQL, in the context of EXISTS databases such as
PostgreSQL allow it, and in any case the condition now no longer raises
an internal exception.
For this case, also add an extra whitespace trim step for the unusual
case that there are no columns to render. This is done in such a
way as to not interfere with other test cases that are involving
custom compilation schemes.
Fixes: #9440
Change-Id: If65ba9ce15d371f09b4342ad0669143b7b082a78
if .post1 fails to work out, we'll just do 2.0.6
The test_pickle_rows_other_process test is failing during wheel
builds as it seems that the "subprocess" run is not using the
cython extensions, leading to a pickle mismatch between the
cythonized and the pure python version of a row. comment
out this test and attempt to release as 2.0.5.post1 so that
wheels can build.
Fixes: #9429
Change-Id: I6e1e9f2b9c4ef8fa67a88ff86ebdacbeb02b90df
pymssql seems to be maintained again and seems to be working
completely, so let's try re-enabling it.
Fixed issue in the new :class:`.Uuid` datatype which prevented it from
working with the pymssql driver. As pymssql seems to be maintained again,
restored testing support for pymssql.
Tweaked the pymssql dialect to take better advantage of
RETURNING for INSERT statements in order to retrieve last inserted primary
key values, in the same way as occurs for the mssql+pyodbc dialect right
now.
Identified that the ``sqlite`` and ``mssql+pyodbc`` dialects are now
compatible with the SQLAlchemy ORM's "versioned rows" feature, since
SQLAlchemy now computes rowcount for a RETURNING statement in this specific
case by counting the rows returned, rather than relying upon
``cursor.rowcount``. In particular, the ORM versioned rows use case
(documented at :ref:`mapper_version_counter`) should now be fully
supported with the SQL Server pyodbc dialect.
Change-Id: I38a0666587212327aecf8f98e86031ab25d1f14d
References: #5321Fixes: #9414
Validate that when provided the :paramref:`_sql.MetaData.schema`
argument of :class:`_sql.MetaData` is a string.
Change-Id: I4237232d2ee0f5a4d0b9dbd9af5f5b57abf395b4
This adds the very small plugin flake8-import-single which
will prevent us from having an import with more than one symbol
on a line.
Flake8 by itself prevents this pattern with E401:
import collections, os, sys
However does not do anything with this:
from sqlalchemy import Column, text
Both statements have the same issues generating merge artifacts
as well as presenting a manual decision to be made. While
zimports generally cleans up such imports at the top level, we
don't enforce zimports / pre-commit use.
the plugin finds the same issue for imports that are inside of
test methods. We shouldn't usually have imports in test methods
so most of them here are moved to be top level.
The version is pinned at 0.1.5; the project seems to have no
activity since 2019, however there are three 0.1.6dev releases
on pypi which stopped in September 2019, they seem to be
experiments with packaging. The source for 0.1.5
is extremely simple and only reveals one method to flake8
(the run() method).
Change-Id: Icea894e43bad9c0b5d4feb5f49c6c666d6ea6aa1
To accommodate a change in column ordering used by ORM Declarative in
SQLAlchemy 2.0, a new parameter :paramref:`_orm.mapped_column.sort_order`
has been added that can be used to control the order of the columns defined
in the table by the ORM, for common use cases such as mixins with primary
key columns that should appear first in tables. The change notes at
:ref:`change_9297` illustrate the default change in ordering behavior
(which is part of all SQLAlchemy 2.0 releases) as well as use of the
:paramref:`_orm.mapped_column.sort_order` to control column ordering when
using mixins and multiple classes (new in 2.0.4).
Fixes: #9297
Change-Id: Ic7163d64efdc0eccb53d6ae0dd89ec83427fb675
Fixed issue where element types of a tuple value would be hardcoded to take
on the types from a compared-to tuple, when the comparison were using the
:meth:`.ColumnOperators.in_` operator. This was inconsistent with the usual
way that types are determined for a binary expression, which is that the
actual element type on the right side is considered first before applying
the left-hand-side type.
Fixes: #9313
Change-Id: Ia8874c09682a6512fcf4084cf14481024959c461
Added public property :attr:`_sql.Table.autoincrement_column` that
returns the column identified as autoincrementing in the column.
Fixes: #9277
Change-Id: If60d6f92e0df94f57d00ff6d89d285c61b02f5a4
The fix in #9217 opened up adapt_on_names to more kinds of
expressions than it was prepared for; adjust that logic
and also refine in the ORM where we are using it, as we
dont need it (yet) for the DML RETURNING use case.
Fixed regression introduced in version 2.0.2 due to 🎫`9217` where
using DML RETURNING statements, as well as
:meth:`_sql.Select.from_statement` constructs as was "fixed" in
🎫`9217`, in conjunction with ORM mapped classes that used
expressions such as with :func:`_orm.column_property`, would lead to an
internal error within Core where it would attempt to match the expression
by name. The fix repairs the Core issue, and also adjusts the fix in
🎫`9217` to not take effect for the DML RETURNING use case, where it
adds unnecessary overhead.
Fixes: #9273
Change-Id: Ie0344efb12ff7df48f21e71e62dc598c76a6a0de
Fixed critical regression in SQL expression formulation in the 2.0 series
due to 🎫`7744` which improved support for SQL expressions that
contained many elements against the same operator repeatedly; parenthesis
grouping would be lost with expression elements beyond the first two
elements.
Fixes: #9271
Change-Id: Ib6ed5b71efe0f6816dab75bda622297fc89e3b49
Added a full suite of new SQL bitwise operators, for performing
database-side bitwise expressions on appropriate data values such as
integers, bit-strings, and similar. Pull request courtesy Yegor Statkevich.
Fixes: #8780Closes: #9204
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9204
Pull-request-sha: a4541772a6
Change-Id: I4c70e80f9548dcc1b4e3dccd71bd59d51d3ed46e
Fixed regression related to the implementation for the new
"insertmanyvalues" feature where an internal ``TypeError`` would occur in
arrangements where a :func:`_sql.insert` would be referred towards inside
of another :func:`_sql.insert` via a CTE; made additional repairs for this
use case for positional dialects such as asyncpg when using
"insertmanyvalues".
at the core here is a change to positional insertmanyvalues
where we now get exactly the positions for the "manyvalues" within
the larger list, allowing non-"manyvalues" on the left and right
sides at the same time, not assuming anything about how RETURNING
renders etc., since CTEs are in the mix also.
Fixes: #9173
Change-Id: I5ff071fbef0d92a2d6046b9c4e609bb008438afd
Corrected the fix for 🎫`7664`, released in version 2.0.0, to also
include :class:`.DropSchema` which was inadvertently missed in this fix,
allowing stringification without a dialect. The fixes for both constructs
is backported to the 1.4 series as of 1.4.47.
Fixes: #7664
Change-Id: I509b7500ee496ac1e444ea2096c2a02520167e6d
Fixed stringify for a the :class:`.CreateSchema` DDL construct, which would
fail with an ``AttributeError`` when stringified without a dialect.
Fixes: #7664
Change-Id: Ifc1769604bc5219c060f5112f7bdea0f780f1a1c
The :meth:`_sql.ColumnOperators.in_` and
:meth:`_sql.ColumnOperators.not_in_` are typed to include
``Iterable[Any]`` rather than ``Sequence[Any]`` for more flexibility in
argument type.
The :func:`_sql.or_` and :func:`_sql.and_` from a typing perspective
require the first argument to be present, however these functions still
accept zero arguments which will emit a deprecation warning at runtime.
Typing is also added to support sending the fixed literal ``False`` for
:func:`_sql.or_` and ``True`` for :func:`_sql.and_` as the first argument
only, however the documentation now indicates sending the
:func:`_sql.false` and :func:`_sql.true` constructs in these cases as a
more explicit approach.
Fixed typing issue where iterating over a :class:`_orm.Query` object
was not correctly typed.
Fixes: #9122Fixes: #9123Fixes: #9125
Change-Id: I500e3e1b826717b3dd49afa1e682c3c8279c9226
Fixed bug / regression where using :func:`.bindparam()` with the same name
as a column in the :meth:`.Update.values` method of :class:`.Update`, as
well as the :meth:`.Insert.values` method of :class:`.Insert` in 2.0 only,
would in some cases silently fail to honor the SQL expression in which the
parameter were presented, replacing the expression with a new parameter of
the same name and discarding any other elements of the SQL expression, such
as SQL functions, etc. The specific case would be statements that were
constructed against ORM entities rather than plain :class:`.Table`
instances, but would occur if the statement were invoked with a
:class:`.Session` or a :class:`.Connection`.
:class:`.Update` part of the issue was present in both 2.0 and 1.4 and is
backported to 1.4.
Fixes: #9075
Change-Id: Ie954bc1f492ec6a566163588182ef4910c7ee452
Supported use case for foreign key constraints where the local column is
marked as "invisible". The errors normally generated when a
:class:`.ForeignKeyConstraint` is created that check for the target column
are disabled when reflecting, and the constraint is skipped with a warning
in the same way which already occurs for an :class:`.Index` with a similar
issue.
tests are added for indexes, unique constraints, and primary key
constraints, which were already working; indexes and uniques warn,
primary keys don't which we would assume is because we never see those
PK columns in the first place.
Constraints now raise an informative ConstraintColumnNotFoundError
in the general case for strings in the "pending colargs" collection
not being resolvable.
Fixes: #9059
Change-Id: I400cf0bff6abba0e0c75f38b07617be1a8ec3453
Fixed issue in the internal SQL traversal for DML statements like
:class:`_dml.Update` and :class:`_dml.Delete` which would cause among other
potential issues, a specific issue using lambda statements with the ORM
update/delete feature.
Fixes: #9033
Change-Id: I76428049cb767ba302fbea89555114bf63ab8687
Fixed issue in lambda SQL feature where the calculated type of a literal
value would not take into account the type coercion rules of the "compared
to type", leading to a lack of typing information for SQL expressions, such
as comparisons to :class:`.JSON` elements and similar.
Fixes: #9029
Change-Id: I381c8d7458d98ba762313dee9ec47a9c1881f74a
Fixed regression where newly revised PostgreSQL range types such as
:class:`_postgresql.INT4RANGE` could not be set up as the impl of a
:class:`.TypeDecorator` custom type, instead raising a ``TypeError``.
Fixes: #9020
Change-Id: Ib881c3c7f63d000f49a09185a8663659a9970aa9
Added parameter
:paramref:`.FunctionElement.column_valued.joins_implicitly`, which is
useful in preventing the "cartesian product" warning when making use of
table-valued or column-valued functions. This parameter was already
introduced for :meth:`.FunctionElement.table_valued` in 🎫`7845`,
however it failed to be added for :meth:`.FunctionElement.column_valued`
as well.
Fixes: #9009
Change-Id: Ifb72fbcb4f4d2998e730d6f85ec7280df3bf3d47
To accommodate for third party dialects with different character escaping
needs regarding bound parameters, the system by which SQLAlchemy "escapes"
(i.e., replaces with another character in its place) special characters in
bound parameter names has been made extensible for third party dialects,
using the :attr:`.SQLCompiler.bindname_escape_chars` dictionary which can
be overridden at the class declaration level on any :class:`.SQLCompiler`
subclass. As part of this change, also added the dot ``"."`` as a default
"escaped" character.
Fixes: #8994
Change-Id: I52fbbfa8c64497b123f57327113df3f022bd1419
Fixed bug where SQL compilation would fail to make use of
:meth:`_types.TypeEngine.bind_expression` on a given type when used in the
context of an "expanding" (i.e. "IN") parameter with the ``literal_binds``
compiler parameter in use.
Fixes: #8989
Change-Id: Ic9fd27b46381b488117295ea5a492d8fc158e39f
Fixed issue where :meth:`_engine.Result.freeze` method would not work for
textual SQL using either :func:`_sql.text` or
:meth:`_engine.Connection.exec_driver_sql`.
Fixes: #8963
Change-Id: Ia131c6ac41a4adf32eb1bf1abf23930ef395f16c
The :meth:`.SQLCompiler.construct_params` method, as well as the
:attr:`.SQLCompiler.params` accessor, will now return the
exact parameters that correspond to a compiled statement that used
the ``render_postcompile`` parameter to compile. Previously,
the method returned a parameter structure that by itself didn't correspond
to either the original parameters or the expanded ones.
Passing a new dictionary of parameters to
:meth:`.SQLCompiler.construct_params` for a :class:`.SQLCompiler` that was
constructed with ``render_postcompile`` is now disallowed; instead, to make
a new SQL string and parameter set for an alternate set of parameters, a
new method :meth:`.SQLCompiler.construct_expanded_state` is added which
will produce a new expanded form for the given parameter set, using the
:class:`.ExpandedState` container which includes a new SQL statement
and new parameter dictionary, as well as a positional parameter tuple.
Fixes: #6114
Change-Id: I9874905bb90f86799b82b244d57369558b18fd93
Changed how the positional compilation is performed. It's rendered by the compiler
the same as the pyformat compilation. The string is then processed to replace
the placeholders with the correct ones, and to obtain the correct order of the
parameters.
This vastly simplifies the computation of the order of the parameters, that in
case of nested CTE is very hard to compute correctly.
Reworked how numeric paramstyle behavers:
- added support for repeated parameter, without duplicating them like in normal
positional dialects
- implement insertmany support. This requires that the dialect supports out of
order placehoders, since all parameters that are not part of the VALUES clauses
are placed at the beginning of the parameter tuple
- support for different identifiers for a numeric parameter. It's for example
possible to use postgresql style placeholder $1, $2, etc
Added two new dialect based on sqlite to test "numeric" fully using
both :1 style and $1 style. Includes a workaround for SQLite's
not-really-correct numeric implementation.
Changed parmstyle of asyncpg dialect to use numeric, rendering with its native
$ identifiers
Fixes: #8926Fixes: #8849
Change-Id: I7c640467d49adfe6d795cc84296fc7403dcad4d6
Fixed issue where table reflection using :paramref:`.Table.extend_existing`
would fail to deduplicate a same-named column if the existing
:class:`.Table` used a separate key. The
:paramref:`.Table.autoload_replace` parameter would allow the column to be
skipped but under no circumstances should a :class:`.Table` ever have the
same-named column twice.
Additionally, changed deprecation warnings to exceptions
as were implemented in I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5 .
Fixes: #8925
Change-Id: I83d0f8658177a7ffbb06e01dbca91377d1a98d49
Fixed a series of issues regarding positionally rendered bound parameters,
such as those used for SQLite, asyncpg, MySQL and others. Some compiled
forms would not maintain the order of parameters correctly, such as the
PostgreSQL ``regexp_replace()`` function as well as within the "nesting"
feature of the :class:`.CTE` construct first introduced in 🎫`4123`.
Fixes: #8827
Change-Id: I9813ed7c358cc5c1e26725c48df546b209a442cb
Added additional type-detection for the new PostgreSQL
:class:`_postgresql.Range` type, where previous cases that allowed the
psycopg2-native range objects to be received directly by the DBAPI without
SQLAlchemy intercepting them stopped working, as we now have our own value
object. The :class:`_postgresql.Range` object has been enhanced such that
SQLAlchemy Core detects it in otherwise ambiguous situations (such as
comparison to dates) and applies appropriate bind handlers. Pull request
courtesy Lele Gaifax.
Fixes: #8884Closes: #8886
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8886
Pull-request-sha: 6e95e08a30
Change-Id: I3ca277c826dcf4b5644f44eb251345b439a84ee4
As I need dmypy to work without facing [1], I am
running the latest build of mypy which seems so far
to finally not have that issue.
update constructs that latest mypy is being more picky
about, including better typing for the _NONE_NAME
symbol used in constraints (porting those elements
from the Enum patch at
I15ac3daee770408b5795746f47c1bbd931b7d26d)
[1] https://github.com/python/mypy/issues/12744
Change-Id: Ib3f56787fa65ea9bb2e6a0bccc4d99f54c516dad
Added :class:`_expression.ScalarValues` that can be used as a column
element allowing using :class:`_expression.Values` inside IN clauses
or in conjunction with ``ANY`` or ``ALL`` collection aggregates.
This new class is generated using the method
:meth:`_expression.Values.scalar_values`.
The :class:`_expression.Values` instance is now coerced to a
:class:`_expression.ScalarValues` when used in a ``IN`` or ``NOT IN``
operation.
Fixes: #6289
Change-Id: Iac22487ccb01553684b908e54d01c0687fa739f1
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
commit two of two. this reorganizes ColumnCollection
to build a new index up front that's used to optimize
the corresponding_column() method.
Additional performance enhancements within ORM-enabled SQL statements,
specifically targeting callcounts within the construction of ORM
statements, using combinations of :func:`_orm.aliased` with
:func:`_sql.union` and similar "compound" constructs, in addition to direct
performance improvements to the ``corresponding_column()`` internal method
that is used heavily by the ORM by constructs like :func:`_orm.aliased` and
similar.
Fixes: #8796
Change-Id: I4a76788007d5a802b9a4081e6a0f6e4b52497b50
The ``aliased()`` constructor calls upon ``__clause_element__()``,
which internally annotates a ``FromClause``, like a subquery.
This became expensive as ``AnnotatedFromClause`` has for
many years called upon ``element.c`` so that the full ``.c``
collection is transferred to the Annotated.
Taking this out proved to be challenging. A straight remove
seemed to not break any tests except for the one that
tested the exact condition. Nevertheless this seemed
"spooky" so I instead moved the get of ``.c`` to be in a
memoized proxy method. However, that then exposed
a recursion issue related to loader_criteria; so the
source of that behavior, which was an accidental behavioral
artifact, is now made into an explcicit option that
loader_criteria uses directly.
The accidental behavioral artifact in question is still
kind of strange since I was not able to fully trace out
how it works, but the end result is that fixing the
artifact to be "correct" causes loader_criteria, within
the particular test for #7491, creates a select/
subquery structure with a cycle in it, so compilation fails
with recursion overflow.
The "solution" is to cause the artifact to occur in this
case, which is that the ``AnnotatedFromClause`` will have a
different ``.c`` collection than its element, which is a
subquery. It's not totally clear how a cycle is generated
when this is not done.
This is commit one of two, which goes through
some hoops to make essentially a one-line change.
The next commit will rework ColumnCollection to optimize
the corresponding_column() method significantly.
Fixes: #8796
Change-Id: Id58ae6554db62139462c11a8be7313a3677456ad
An informative re-raise is now thrown in the case where any "literal
bindparam" render operation fails, indicating the value itself and
the datatype in use, to assist in debugging when literal params
are being rendered in a statement.
Fixes: #8800
Change-Id: Id658f8b03359312353ddbb0c7563026239579f7b
The RETURNING clause now renders columns using the routine as that of the
:class:`.Select` to generate labels, which will include disambiguating
labels, as well as that a SQL function surrounding a named column will be
labeled using the column name itself. This is a more comprehensive change
than a similar one made for the 1.4 series that adjusted the function label
issue only.
includes 1.4's changelog for the backported version which also
fixes an Oracle issue independently of the 2.0 series.
Fixes: #8770
Change-Id: I2ab078a214a778ffe1720dbd864ae4c105a0691d
This change contains new features for 2.0 only as well as some
behaviors that will be backported to 1.4.
For 1.4 and 2.0:
Fixed issue where the underlying DBAPI cursor would not be closed when
using :class:`_orm.Query` with :meth:`_orm.Query.yield_per` and direct
iteration, if a user-defined exception case were raised within the
iteration process, interrupting the iterator. This would lead to the usual
MySQL-related issues with server side cursors out of sync.
For 1.4 only:
A similar scenario can occur when using :term:`2.x` executions with direct
use of :class:`.Result`, in that case the end-user code has access to the
:class:`.Result` itself and should call :meth:`.Result.close` directly.
Version 2.0 will feature context-manager calling patterns to address this
use case. However within the 1.4 scope, ensured that ``.close()`` methods
are available on all :class:`.Result` implementations including
:class:`.ScalarResult`, :class:`.MappingResult`.
For 2.0 only:
To better support the use case of iterating :class:`.Result` and
:class:`.AsyncResult` objects where user-defined exceptions may interrupt
the iteration, both objects as well as variants such as
:class:`.ScalarResult`, :class:`.MappingResult`,
:class:`.AsyncScalarResult`, :class:`.AsyncMappingResult` now support
context manager usage, where the result will be closed at the end of
iteration.
Corrected various typing issues within the engine and async engine
packages.
Fixes: #8710
Change-Id: I3166328bfd3900957eb33cbf1061d0495c9df670
Mike Bayer