Implemented initial support for free-threaded Python by adding new tests
and reworking the test harness and GitHub Actions to include Python 3.13t
and Python 3.14t in test runs. Two concurrency issues have been identified
and fixed: the first involves initialization of the ``.c`` collection on a
``FromClause``, a continuation of 🎫`12302`, where an optional mutex
under free-threading is added; the second involves synchronization of the
pool "first_connect" event, which first received thread synchronization in
🎫`2964`, however under free-threading the creation of the mutex
itself runs under the same free-threading mutex. Initial pull request and
test suite courtesy Lysandros Nikolaou.
py313t: yes
py314t: yes
Fixes: #12881Closes: #12882
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12882
Pull-request-sha: 53d65d96b9
Co-authored-by: Mike Bayer <mike_mp@zzzcomputng.com>
Change-Id: I2e4f2e9ac974ab6382cb0520cc446b396d9680a6
The "emulated" exception hierarchies for the asyncio
drivers such as asyncpg, aiomysql, aioodbc, etc. have been standardized
on a common base :class:`.EmulatedDBAPIException`, which is now what's
available from the :attr:`.StatementException.orig` attribute on a
SQLAlchemy :class:`.DBAPIException` object. Within :class:`.EmulatedDBAPIException`
and the subclasses in its hiearchy, the original driver-level exception is
also now avaliable via the :attr:`.EmulatedDBAPIException.orig` attribute,
and is also available from :class:`.DBAPIException` directly using the
:attr:`.DBAPIException.driver_exception` attribute.
Added additional emulated error classes for the subclasses of
``asyncpg.exception.IntegrityError`` including ``RestrictViolationError``,
``NotNullViolationError``, ``ForeignKeyViolationError``,
``UniqueViolationError`` ``CheckViolationError``,
``ExclusionViolationError``. These exceptions are not directly thrown by
SQLAlchemy's asyncio emulation, however are available from the
newly added :attr:`.DBAPIException.driver_exception` attribute when a
:class:`.IntegrityError` is caught.
Fixes: #8047
Change-Id: I6a34e85b055265c087b0615f7c573be8582b3486
on the theme of 2.0/2.1 are relatively similar right now, do
some more cleanup removing the py3k suffixes that we dont need anymore.
I'm not sure that the logic that would search for these suffixes is
even present anymore and I would guess we removed it when we removed
py2k support. However, I am planning on possibly bringing it back
for some of the py314 stuff, so I still want to be able to do
suffix stuff. that will be for another patch.
Change-Id: I929e6edd922f8d5f943acce77191fb1e3035b42c
Changes to the test suite to accommodate Python 3.14 as of version
3.14.0b1
Originally this included a major breaking change to how python 3.14
implemented :pep:`649`, however this was resolved by [1].
As of a7, greenlet is skipped due to issues in a7 and later b1
in [2].
1. the change to rewrite all conditionals in annotation related tests
is reverted.
2. test_memusage needed an explicit set_start_method() call so that
it can continue to use plain fork
3. unfortunately at the moment greenlet has to be re-disabled for 3.14.
4. Changes to tox overall, remove pysqlcipher which hasn't worked
in years, etc.
5. we need to support upcoming typing-extensions also, install the beta
6. 3.14.0a7 introduces major regressions to our runtime typing
utilities, unfortunately, it's not clear if these can be resolved
7. for 3.14.0b1, we have to vendor get_annotations to work around [3]
[1] https://github.com/python/cpython/issues/130881
[2] https://github.com/python-greenlet/greenlet/issues/440
[3] https://github.com/python/cpython/issues/133684
py314: yes
Fixes: #12405
References: #12399
Change-Id: I8715d02fae599472dd64a2a46ccf8986239ecd99
Fixed regression caused by ``typing_extension==4.13.0`` that introduced
a different implementation for ``TypeAliasType`` while SQLAlchemy assumed
that it would be equivalent to the ``typing`` version.
Added test regarding generic TypeAliasType
Fixes: #12473Closes: #12472
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12472
Pull-request-sha: 8861a5acfb
Change-Id: I053019a222546a625ed6d588314ae9f5b34c2f8a
Added the ability to create custom SQL constructs that can define new
clauses within SELECT, INSERT, UPDATE, and DELETE statements without
needing to modify the construction or compilation code of of
:class:`.Select`, :class:`.Insert`, :class:`.Update`, or :class:`.Delete`
directly. Support for testing these constructs, including caching support,
is present along with an example test suite. The use case for these
constructs is expected to be third party dialects for NewSQL or other novel
styles of database that introduce new clauses to these statements. A new
example suite is included which illustrates the ``QUALIFY`` SQL construct
used by several NewSQL databases which includes a cachable implementation
as well as a test suite.
Since these extensions start to make it a bit crowded with how many
kinds of "options" we have on statements, did some naming /
documentation changes with existing constructs on Executable, in
particular to distinguish ExecutableOption from SyntaxExtension.
Fixes: #12195
Change-Id: I4a44ee5bbc3d8b1b640837680c09d25b1b7077af
Fixed issue where creating an :class:`.Engine` using multiple calls to
:meth:`.Engine.execution_options` where a subsequent call involved certain
options such as ``isolation_level`` would lead to an internal error
involving event registration.
Fixes: #12289
Change-Id: Iec5fbc0eb0c5a92dda1ea762872ae992ca816685
Fixed issue in event system which prevented an event listener from being
attached and detached from multiple class-like objects, namely the
:class:`.sessionmaker` or :class:`.scoped_session` targets that assign to
:class:`.Session` subclasses.
Fixes: #12216
Change-Id: I3d8969fe604adbc23add07a13741938c7f4fc8ca
Fix issue that resulted in inconsistent handing of unions
depending on how they were declared
Consistently support TypeAliasType. This has required a revision
of the implementation added in #11305 to have a consistent
behavior.
References: #11944
References: #11955
References: #11305
Change-Id: Iffc34fd42b9769f73ddb4331bd59b6b37391635d
Fixed issue regarding ``Union`` types that would be present in the
:paramref:`_orm.registry.type_annotation_map` of a :class:`_orm.registry`
or declarative base class, where a ``Mapped[]`` element that included one
of the subtypes present in that ``Union`` would be matched to that entry,
potentially ignoring other entries that matched exactly. The correct
behavior now takes place such that an entry should only match in
``type_annotation_map`` exactly, as a ``Union`` type is a self-contained
type. For example, an attribute with ``Mapped[float]`` would previously
match to a ``type_annotation_map`` entry ``Union[float, Decimal]``; this
will no longer match and will now only match to an entry that states
``float``. Pull request courtesy Frazer McLean.
Fixes#11370Closes: #11942
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/11942
Pull-request-sha: 21a3d1971a
Change-Id: I3467be00f8fa8bd011dd4805a77a3b80ff74a215
hash_limit_string works by doing a modulus of a hash value so that
the range of possible numbers is 0-N. however, there's a chance
we might not populate every 0-N value in unusual cases on CI,
even after iterating 500 times apparently. Loosen the change by making
sure we got at least N/2 unique hash messages but not greater
than N.
Change-Id: I5cd2845697ec0a718ddca1c95fbc4867b06eabee
Replaces the pyx files with py files that can be both compiled
by cython or imported as is by python.
This avoids the need of duplicating the code to have a python
only fallback.
The cython files are also reorganized to be in the module they use
instead of all being in the cyextension package, that has been
removed.
The performance is pretty much equal between main and this change.
A detailed comparison is at this link
https://docs.google.com/spreadsheets/d/1jkmGpnCyEcPyy6aRK9alElGjxlNHu44Wxjr4VrD99so/edit?usp=sharing
Change-Id: Iaed232ea5dfb41534cc9f58f6ea2f912a93263af
The :class:`.Row` object now no longer makes use of an intermediary
``Tuple`` in order to represent its individual element types; instead,
the individual element types are present directly, via new :pep:`646`
integration, now available in more recent versions of Mypy. Mypy
1.7 or greater is now required for statements, results and rows
to be correctly typed. Pull request courtesy Yurii Karabas.
Fixes: #10635Closes: #10634
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10634
Pull-request-sha: 430785c8a0
Change-Id: Ibd0ae31a98b4ea69dcb89f970e640920b2be6c48
Removed the async_fallback mode and await_fallback function.
Replace get_event_loop with Runner.
Removed the internal function ``await_fallback()``.
Renamed the internal function ``await_only()`` to ``await_()``.
Change-Id: Ib43829be6ebdb59b6c4447f5a15b5d2b81403fa9
Fixed 2.0 regression in :class:`.MutableList` where a routine that detects
sequences would not correctly filter out string or bytes instances, making
it impossible to assign a string value to a specific index (while
non-sequence values would work fine).
Fixes: #10784
Change-Id: I829cd2a1ef555184de8e6a752f39df65f69f6943
This is the runtime raise when an async concurrency function
is called. in 2.0 this raises ValueError, however here we've
standardized on raising ``ImportError`.
continuing for #10747, add a test asserting we dont get an
endless loop and get a clean ImportError instead when greenlet not
installed and async functions are used.
Fixes: #10747
Change-Id: I54dffe8577025e2ef3a59f5ca9ab7f4362d4d91f
Added an initialize step to the import of
``sqlalchemy.ext.asyncio`` so that ``greenlet`` will
be imported only when the asyncio extension is first imported.
Alternatively, the ``greenlet`` library is still imported lazily on
first use to support use case that don't make direct use of the
SQLAlchemy asyncio extension.
Fixes: #10296
Change-Id: I97162a01aa29adb3e3fee97b718ab9567b2f6124
Initial fixes to test to accommodate py312
Here we are pulling in a current fork of greenlet that works
for python 3.12. It works though there is one issue with aiosqlite
that might be related.
a non-voting py312 job is added to gerrit / jenkins
Fixes: #9819
Change-Id: I91a51dcbad2902f7c4c7cec88ebbf42c2417b512
fix a handful of warnings that were emitting but not raising,
usually because they were inside an "expect_warnings" block.
modify "expect_warnings" to always use "raise_on_any_unexpected"
behavior; remove this parameter.
Fixed issue in semi-private ``await_only()`` and ``await_fallback()``
concurrency functions where the given awaitable would remain un-awaited if
the function threw a ``GreenletError``, which could cause "was not awaited"
warnings later on if the program continued. In this case, the given
awaitable is now cancelled before the exception is thrown.
Change-Id: I33668c5e8c670454a3d879e559096fb873b57244
Various performance improvements to Row instanciation
- avoid passing processors if they are all None
- improve processor logic in cython
- improve tuplegetter using slices when contiguous indexes are used
Some timing follow.
In particular [base_]row_new_proc that tests using processors has
a 25% improvement compared to before in cython.
Looking at the [b]row_new_proc_none that test a list of processors
all None, this has 50% improvement in cython when passing the none list,
but in this patch it would usually be disabled by passing None, so the
performance gain is actually 90%, since it would run the case
[base_]row_new.
Tuplegetter is a bit faster in the single item get and when getting
sequential indexes (like indexes 1,2,3,4) at the cost of a bit
longer creation time in python, cython is mostly the same.
Current times
| python | cython | cy / py |
base_row_new | 0.639817400 | 0.118265500 | 0.184842582 |
row_new | 0.680355100 | 0.129714600 | 0.190657202 |
base_row_new_proc | 3.076538900 | 1.488428600 | 0.483799701 |
row_new_proc | 3.119700100 | 1.532197500 | 0.491136151 |
brow_new_proc_none | 1.917702300 | 0.475511500 | 0.247958977 |
row_new_proc_none | 1.956253300 | 0.497803100 | 0.254467609 |
tuplegetter_one | 0.152512600 | 0.148523900 | 0.973846751 |
tuplegetter_many | 0.184394100 | 0.184511500 | 1.000636680 |
tuplegetter_seq | 0.154832800 | 0.156270100 | 1.009282917 |
tuplegetter_new_one | 0.523730000 | 0.343402200 | 0.655685563 |
tuplegetter_new_many| 0.738924400 | 0.420961400 | 0.569694816 |
tuplegetter_new_seq | 1.062036900 | 0.495462000 | 0.466520514 |
Parent commit times
| python | cython | cy / py |
base_row_new | 0.643890800 | 0.113548300 | 0.176347138 |
row_new | 0.674885900 | 0.124391800 | 0.184315304 |
base_row_new_proc | 3.072020400 | 2.017367000 | 0.656690626 |
row_new_proc | 3.109943400 | 2.048359400 | 0.658648450 |
brow_new_proc_none | 1.967133700 | 1.006326000 | 0.511569702 |
row_new_proc_none | 1.960814900 | 1.025217800 | 0.522852922 |
tuplegetter_one | 0.197359900 | 0.205999000 | 1.043773330 |
tuplegetter_many | 0.196575900 | 0.194888500 | 0.991416038 |
tuplegetter_seq | 0.192723900 | 0.205635000 | 1.066992729 |
tuplegetter_new_one | 0.534644500 | 0.414311700 | 0.774929322 |
tuplegetter_new_many| 0.479376500 | 0.417448100 | 0.870814694 |
tuplegetter_new_seq | 0.481580200 | 0.412697900 | 0.856966088 |
Change-Id: I2ca1f49dca2beff625c283f1363c29c8ccc0c3f7
Improved :class:`_engine.Row` implementation to optimize
``__getattr__`` performance.
The serialization of a :class:`_engine.Row` to pickle has changed with
this change. Pickle saved by older SQLAlchemy versions can still be loaded,
but new pickle saved by this version cannot be loaded by older ones.
Fixes: #9678Closes: #9668
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9668
Pull-request-sha: 86b8ccd195
Change-Id: Ia85c26a59e1a57ba2bf0d65578c6168f82a559f2
Fixed issue in "versioned history" example where using a declarative base
that is derived from :class:`_orm.DeclarativeBase` would fail to be mapped.
Additionally, repaired the given test suite so that the documented
instructions for running the example using Python unittest now work again.
Change-Id: I164a5b8dbdd01e3d815eb356f7b7cadf226ca296
References: #9546
Fixed regression involving pickling of Python rows between the cython and
pure Python implementations of :class:`.Row`, which occurred as part of
refactoring code for version 2.0 with typing. A particular constant were
turned into a string based ``Enum`` for the pure Python version of
:class:`.Row` whereas the cython version continued to use an integer
constant, leading to deserialization failures.
Regression occurred in a4bb502cf9Fixes: #9423
Change-Id: Icbd85cacb2d589cef7c246de7064249926146f2e
first change:
Reworked the :ref:`examples_versioned_history` to work with
version 2.0, while at the same time improving the overall working of
this example to use newer APIs, including a newly added hook
:meth:`_orm.MapperEvents.after_mapper_constructed`.
second change:
Added new event hook :meth:`_orm.MapperEvents.after_mapper_constructed`,
which supplies an event hook to take place right as the
:class:`_orm.Mapper` object has been fully constructed, but before the
:meth:`_orm.registry.configure` call has been called. This allows code that
can create additional mappings and table structures based on the initial
configuration of a :class:`_orm.Mapper`, which also integrates within
Declarative configuration. Previously, when using Declarative, where the
:class:`_orm.Mapper` object is created within the class creation process,
there was no documented means of running code at this point. The change
is to immediately benefit custom mapping schemes such as that
of the :ref:`examples_versioned_history` example, which generate additional
mappers and tables in response to the creation of mapped classes.
third change:
The infrequently used :attr:`_orm.Mapper.iterate_properties` attribute and
:meth:`_orm.Mapper.get_property` method, which are primarily used
internally, no longer implicitly invoke the :meth:`_orm.registry.configure`
process. Public access to these methods is extremely rare and the only
benefit to having :meth:`_orm.registry.configure` would have been allowing
"backref" properties be present in these collections. In order to support
the new :meth:`_orm.MapperEvents.after_mapper_constructed` event, iteration
and access to the internal :class:`_orm.MapperProperty` objects is now
possible without triggering an implicit configure of the mapper itself.
The more-public facing route to iteration of all mapper attributes, the
:attr:`_orm.Mapper.attrs` collection and similar, will still implicitly
invoke the :meth:`_orm.registry.configure` step thus making backref
attributes available.
In all cases, the :meth:`_orm.registry.configure` is always available to
be called directly.
fourth change:
Fixed obscure ORM inheritance issue caused by 🎫`8705` where some
scenarios of inheriting mappers that indicated groups of columns from the
local table and the inheriting table together under a
:func:`_orm.column_property` would nonetheless warn that properties of the
same name were being combined implicitly.
Fixes: #9220Fixes: #9232
Change-Id: Id335b8e8071c8ea509c057c389df9dcd2059437d
Supported use case for foreign key constraints where the local column is
marked as "invisible". The errors normally generated when a
:class:`.ForeignKeyConstraint` is created that check for the target column
are disabled when reflecting, and the constraint is skipped with a warning
in the same way which already occurs for an :class:`.Index` with a similar
issue.
tests are added for indexes, unique constraints, and primary key
constraints, which were already working; indexes and uniques warn,
primary keys don't which we would assume is because we never see those
PK columns in the first place.
Constraints now raise an informative ConstraintColumnNotFoundError
in the general case for strings in the "pending colargs" collection
not being resolvable.
Fixes: #9059
Change-Id: I400cf0bff6abba0e0c75f38b07617be1a8ec3453
Fixed issue where table reflection using :paramref:`.Table.extend_existing`
would fail to deduplicate a same-named column if the existing
:class:`.Table` used a separate key. The
:paramref:`.Table.autoload_replace` parameter would allow the column to be
skipped but under no circumstances should a :class:`.Table` ever have the
same-named column twice.
Additionally, changed deprecation warnings to exceptions
as were implemented in I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5 .
Fixes: #8925
Change-Id: I83d0f8658177a7ffbb06e01dbca91377d1a98d49
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
commit two of two. this reorganizes ColumnCollection
to build a new index up front that's used to optimize
the corresponding_column() method.
Additional performance enhancements within ORM-enabled SQL statements,
specifically targeting callcounts within the construction of ORM
statements, using combinations of :func:`_orm.aliased` with
:func:`_sql.union` and similar "compound" constructs, in addition to direct
performance improvements to the ``corresponding_column()`` internal method
that is used heavily by the ORM by constructs like :func:`_orm.aliased` and
similar.
Fixes: #8796
Change-Id: I4a76788007d5a802b9a4081e6a0f6e4b52497b50
in [1], Python 3.11 seems to have changed the behavior of
IntEnum. We didn't notice this because we have our own
workaround class already, but typing did. Ensure we remain
compatible with IntFlag.
This change also modifies FastIntFlag to no longer use
global symbols; this is unnecessary as we assign FastIntFlag
members explicitly. Use of ``symbol()`` should probably
be phased out.
[1] https://github.com/python/cpython/issues/99304Fixes: #8783
Change-Id: I8ae2e871ff1467ae5ca1f63e66b5dae45d4a6c93
This change contains new features for 2.0 only as well as some
behaviors that will be backported to 1.4.
For 1.4 and 2.0:
Fixed issue where the underlying DBAPI cursor would not be closed when
using :class:`_orm.Query` with :meth:`_orm.Query.yield_per` and direct
iteration, if a user-defined exception case were raised within the
iteration process, interrupting the iterator. This would lead to the usual
MySQL-related issues with server side cursors out of sync.
For 1.4 only:
A similar scenario can occur when using :term:`2.x` executions with direct
use of :class:`.Result`, in that case the end-user code has access to the
:class:`.Result` itself and should call :meth:`.Result.close` directly.
Version 2.0 will feature context-manager calling patterns to address this
use case. However within the 1.4 scope, ensured that ``.close()`` methods
are available on all :class:`.Result` implementations including
:class:`.ScalarResult`, :class:`.MappingResult`.
For 2.0 only:
To better support the use case of iterating :class:`.Result` and
:class:`.AsyncResult` objects where user-defined exceptions may interrupt
the iteration, both objects as well as variants such as
:class:`.ScalarResult`, :class:`.MappingResult`,
:class:`.AsyncScalarResult`, :class:`.AsyncMappingResult` now support
context manager usage, where the result will be closed at the end of
iteration.
Corrected various typing issues within the engine and async engine
packages.
Fixes: #8710
Change-Id: I3166328bfd3900957eb33cbf1061d0495c9df670
For 2.0, we provide a truly "larger than memory collection"
implementation, a write-only collection that will never
under any circumstances implicitly load the entire
collection, even during flush.
This is essentially a much more "strict" version
of the "dynamic" loader, which in fact has a lot of
scenarios that it loads the full backing collection
into memory, mostly defeating its purpose.
Typing constructs are added that support
both the new feature WriteOnlyMapping as well as the
legacy feature DynamicMapping. These have been
integrated with "annotion based mapping" so that
relationship() uses these annotations to configure
the loader strategy as well.
additional changes:
* the docs triggered a conflict in hybrid's
"transformers" section, this section is hard-coded
to Query using a pattern that doesnt seem to have
any use and isn't part of the current select()
interface, so just removed this section
* As the docs for WriteOnlyMapping are very long,
collections.rst is broken up into two pages now.
Fixes: #6229Fixes: #7123
Change-Id: I6929f3da6e441cad92285e7309030a9bac4e429d
ahead of trying to get everything formatted, some more
flexibility so that we can use doctest for all
python + sql code, while still being able to tell the
test suite to not run doctests on a sample. All of the
"non-console python with SQL" in the docs is because I was
showing an example that I didn't want tested.
Change-Id: Iae876ae1ffd93c36b096c6c2d6048843ae9698c8
the sqlite builds on github actions seem to be very
inconsistent about versions and many don't support
RETURNING. ensure any tests that depend on RETURNING present
are marked as such.
Change-Id: I7a60a81fa70b90642448cdd58eda33212c3afebc
reviewers: these docs publish periodically at:
https://docs.sqlalchemy.org/en/gerrit/4042/orm/queryguide/index.html
See the "last generated" timestamp near the bottom of the
page to ensure the latest version is up
Change includes some other adjustments:
* small typing fixes for end-user benefit
* removal of a bunch of old examples for patterns that nobody
uses or aren't really what we promote now
* modernization of some examples, including inheritance
Change-Id: I9929daab7797be9515f71c888b28af1209e789ff
Ensure that all cython extension are imported by the compied detection logic.
This is required since cython extensions moduels are marked as optional
in the install, so it's possible that only some of them are compiled.
The extensions are enabled only if all of them are correctly compiled
Change-Id: I355cbac06f5c7a47d35661f42ebab3b0156c1965
Fixed event listening issue where event listeners added to a superclass
would be lost if a subclass were created which then had its own listeners
associated. The practical example is that of the :class:`.sessionmaker`
class created after events have been associated with the
:class:`_orm.Session` class.
Fixes: #8467
Change-Id: I9bdba8769147e30110a09900d4a577e833ac3af9
Added new syntax to the ``.c`` collection on all :class:`.FromClause`
objects allowing tuples of keys to be passed to ``__getitem__()``, along
with support for ``select()`` handling of ``.c`` collections directly,
allowing the syntax ``select(table.c['a', 'b', 'c'])`` to be possible. The
sub-collection returned is itself a :class:`.ColumnCollection` which is
also directly consumable by :func:`_sql.select` and similar now.
Fixes: #8285
Change-Id: I2236662c477ffc50af079310589e213323c960d1
I screwed up a rebase or something so this was
temporarily in Ic51a12de3358f3a451bd7cf3542b375569499fc1
Change-Id: I847ee1336381221c0112b67854df022edf596b25
Repaired a deprecation warning class decorator that was preventing key
objects such as :class:`_engine.Connection` from having a proper
``__weakref__`` attribute, causing operations like Python standard library
``inspect.getmembers()`` to fail.
Fixes: #8115
Change-Id: Ifd0bc2325fb9dc9e1431998c308b7fc081968373
The :meth:`.DialectEvents.handle_error` event is now moved to the
:class:`.DialectEvents` suite from the :class:`.EngineEvents` suite, and
now participates in the connection pool "pre ping" event for those dialects
that make use of disconnect codes in order to detect if the database is
live. This allows end-user code to alter the state of "pre ping". Note that
this does not include dialects which contain a native "ping" method such as
that of psycopg2 or most MySQL dialects.
Fixes: #5648
Change-Id: I353d84a4f66f309d2467b7e67621db6b8c70411e
Lysandros Nikolaou