Improve handling of two phase transaction identifiers for PostgreSQL
when the identifier is provided by the user.
As part of this change the psycopg dialect was updated to use the DBAPI
two phase transaction API instead of executing the SQL directly.
Fixes: #13229
Change-Id: If8301a7253b4a0c88e5323c9a052c3a9fa258780
Added support for the ``mssql-python`` driver, Microsoft's official Python
driver for SQL Server.
Fixes: #12869
Change-Id: I9ce0fef1dd1105c20833cc6a46f24ac9580c4b39
Fixed a critical issue in :class:`.Engine` where connections created in
conjunction with the :meth:`.ConnectionEvents.do_connect` event listeners
would receive shared, mutable collections for the connection arguments,
leading to a variety of potential issues including unlimited growth of the
argument list as well as elements within the parameter dictionary being
shared among concurrent connection calls. In particular this could impact
do_connect routines making use of complex mutable authentication
structures.
Fixes: #13144
Change-Id: I1549dae36e8e7e6cf50fdaf796659b53e7b78234
The connection object returned by :meth:`_engine.Engine.raw_connection`
now supports the context manager protocol, automatically returning the
connection to the pool when exiting the context.
Fixes: #13116
Change-Id: I51eb1fd61b772368f12a787e5f60db153a839e70
The the parameter :paramref:`_schema.DropConstraint.isolate_from_table`
was deprecated since it has no effect on the drop table behavior.
Its default values was also changed to ``False``.
Fixes: #13006
Change-Id: Ibaa68e78da301a4f2a200f8c7abb2dc099fef37a
The default DBAPI driver for the Oracle Database dialect has been changed
to ``oracledb`` instead of ``cx_oracle``. The ``cx_oracle`` driver remains
fully supported and can be explicitly specified in the connection URL
using ``oracle+cx_oracle://``.
The default DBAPI driver for the PostgreSQL dialect has been changed to
``psycopg`` (psycopg version 3) instead of ``psycopg2``. The ``psycopg2``
driver remains fully supported and can be explicitly specified in the
connection URL using ``postgresql+psycopg2://``.
Fixes: #13010
Change-Id: Ie75810f4c3af609d20da63289d2662dfa2385ca2
Fixed issue in the :meth:`.ConnectionEvents.after_cursor_execute` method
where the SQL statement and parameter list for an "insertmanyvalues"
operation sent to the event would not be the actual SQL / parameters just
emitted on the cursor, instead being the non-batched form of the statement
that's used as a template to generate the batched statements.
Fixes: #13018
Change-Id: Ib5b6c576f2c7a9ed275de30290d619cc651b4816
this DB is extremely erratic in being able to connect. Add
a brute force connection retrier to all engines everywhere
(which for oracledb we can fortunately use their built-in feature
that also works).
This actually works and I can see it pausing under load, reconnecting,
and succeeding. the problem is that absolutely every engine everywhere
needs this routine otherwise an engine without a retrier in it will
crash. That then necessitates digging into testing_engine(),
making sure testing_engine() is used everywhere an engine that's going
to connect is used, then dealing with the fallout from that.
We also simplify some older workarounds for cx_oracle and
hack into config/provision to make oracledb seem like the primary
DBAPI for most tests.
testing_engine has been completely overhauled, making use of a new
post_configure_testing_engine() hook which moves and refines
the SQLite pool sharing and savepoint logic all into sqlite/provision.py
and also allows for cx_oracle to apply a retry event handler.
Change-Id: I4ea4c523effb878290d28b94d8925eb32fc5ae3b
one particular vector test wont run on oracle 23c free, so
just disable it.
added better skips for the rest of the vector tests and
fixed a deprecation issue.
this will be the first run on the new oracle23 on CI so we'll have to
see how this goes.
Also adjust for mariabdb12 being overly helpful with regards
to stale row updates.
as we are having trouble getting 23c to pass throug transaction
tests, i noted we have an explosion of tests due to the multiple
drivers, so this patch introduces __sparse_driver_backend__
for all tests where we want variety of
database server but there's no need to test every driver.
This should dramatically reduce the size of the test suite run
Change-Id: Ic8d3eb0a60e76b4c54c6bb4a721f90c81ede782b
the timing here is too sensitive to play out reliably
on CI machines particularly free threaded, so mock the time()
callable instead and ensure with subsecond clock intervals
we do the right math
Change-Id: Icc203ae2298eb4b64e3b45f063811e9527278d0c
this isn't used internally and only allows execute(stmt.compile())
to work, which is not something I want to support.
Since people are definitely using this [1] we might want to think about
either deprectation or having a recipe for people who really want
to still do this
[1] https://github.com/sqlalchemy/sqlalchemy/discussions/11108?converting=1
Change-Id: I5f0fe800e31aac052926cebe9206763eef9a804c
Implemented initial support for free-threaded Python by adding new tests
and reworking the test harness and GitHub Actions to include Python 3.13t
and Python 3.14t in test runs. Two concurrency issues have been identified
and fixed: the first involves initialization of the ``.c`` collection on a
``FromClause``, a continuation of 🎫`12302`, where an optional mutex
under free-threading is added; the second involves synchronization of the
pool "first_connect" event, which first received thread synchronization in
🎫`2964`, however under free-threading the creation of the mutex
itself runs under the same free-threading mutex. Initial pull request and
test suite courtesy Lysandros Nikolaou.
py313t: yes
py314t: yes
Fixes: #12881Closes: #12882
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12882
Pull-request-sha: 53d65d96b9
Co-authored-by: Mike Bayer <mike_mp@zzzcomputng.com>
Change-Id: I2e4f2e9ac974ab6382cb0520cc446b396d9680a6
Fixed issue where :meth:`_schema.MetaData.reflect` did not forward
dialect-specific keyword arguments to the :class:`_engine.Inspector`
methods, causing options like ``oracle_resolve_synonyms`` to be ignored
during reflection. The method now ensures that all extra kwargs passed to
:meth:`_schema.MetaData.reflect` are forwarded to
:meth:`_engine.Inspector.get_table_names` and related reflection methods.
Pull request courtesy Lukáš Kožušník.
Fixes: #12884Closes: #12885
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12885
Pull-request-sha: 9436812223
Change-Id: Idb1ca0bc624df8e175cbfa979a8f31e62de185fe
Named types such as :class:`_postgresql.ENUM` and
:class:`_postgresql.DOMAIN` (as well as the dialect-agnostic
:class:`_types.Enum` version) are now more strongly associated with the
:class:`_schema.MetaData` at the top of the table hierarchy and are
de-associated with any particular :class:`_schema.Table` they may be a part
of. This better represents how PostgreSQL named types exist independently
of any particular table, and that they may be used across many tables
simultaneously. The change impacts the behavior of the "default schema"
for a named type, as well as the CREATE/DROP behavior in relationship to
the :class:`.MetaData` and :class:`.Table` construct. The change also
includes a new :class:`.CheckFirst` enumeration which allows fine grained
control over "check" queries during DDL operations, as well as that the
:paramref:`_types.SchemaType.inherit_schema` parameter is deprecated and
will emit a deprecation warning when used. See the migration notes for
full details.
Fixes: #12690
Change-Id: I9752a9d52774ae760ec3448f62fc5046c58d68f5
The "emulated" exception hierarchies for the asyncio
drivers such as asyncpg, aiomysql, aioodbc, etc. have been standardized
on a common base :class:`.EmulatedDBAPIException`, which is now what's
available from the :attr:`.StatementException.orig` attribute on a
SQLAlchemy :class:`.DBAPIException` object. Within :class:`.EmulatedDBAPIException`
and the subclasses in its hiearchy, the original driver-level exception is
also now avaliable via the :attr:`.EmulatedDBAPIException.orig` attribute,
and is also available from :class:`.DBAPIException` directly using the
:attr:`.DBAPIException.driver_exception` attribute.
Added additional emulated error classes for the subclasses of
``asyncpg.exception.IntegrityError`` including ``RestrictViolationError``,
``NotNullViolationError``, ``ForeignKeyViolationError``,
``UniqueViolationError`` ``CheckViolationError``,
``ExclusionViolationError``. These exceptions are not directly thrown by
SQLAlchemy's asyncio emulation, however are available from the
newly added :attr:`.DBAPIException.driver_exception` attribute when a
:class:`.IntegrityError` is caught.
Fixes: #8047
Change-Id: I6a34e85b055265c087b0615f7c573be8582b3486
Added new parameter :paramref:`.create_engine.skip_autocommit_rollback`
which provides for a per-dialect feature of preventing the DBAPI
``.rollback()`` from being called under any circumstances, if the
connection is detected as being in "autocommit" mode. This improves upon
a critical performance issue identified in MySQL dialects where the network
overhead of the ``.rollback()`` call remains prohibitive even if autocommit
mode is set.
Fixes: #12784
Change-Id: I22b45ab2fc396c5aadeff5cdc5ce895144d00098
The function that validates the arguments in the normal execute flow
allowed by mistake list of tuples, that are not supported by the
code since the 2.0 series.
Change-Id: Ia401b0e19e72ed33b7d3d5718578cbed0d214c2a
Hardening of the compiler's actions for UPDATE statements that access
multiple tables to report more specifically when tables or aliases are
referenced in the SET clause; on cases where the backend does not support
secondary tables in the SET clause, an explicit error is raised, and on the
MySQL or similar backends that support such a SET clause, more specific
checking for not-properly-included tables is performed. Overall the change
is preventing these erroneous forms of UPDATE statements from being
compiled, whereas previously it was relied on the database to raise an
error, which was not always guaranteed to happen, or to be non-ambiguous,
due to cases where the parent table included the same column name as the
secondary table column being updated.
Fixed bug where the ORM would pull in the wrong column into an UPDATE when
a key name inside of the :meth:`.ValuesBase.values` method could be located
from an ORM entity mentioned in the statement, but where that ORM entity
was not the actual table that the statement was inserting or updating. An
extra check for this edge case is added to avoid this problem.
Fixes: #12692
Change-Id: I342832b09dda7ed494caaad0cbb81b93fc10fe18
The error message that is emitted when a URL cannot be parsed no longer
includes the URL itself within the error message.
Fixes: #12579
Change-Id: Icd17bd4fe0930036662b6a4fe0264cb13df04ba7
Support has been re-added for the MySQL-Connector/Python DBAPI using the
``mysql+mysqlconnector://`` URL scheme. The DBAPI now works against
modern MySQL versions as well as MariaDB versions (in the latter case it's
required to pass charset/collation explicitly). Note however that
server side cursor support is disabled due to unresolved issues with this
driver.
References: #12332
Change-Id: I81279478196e830d3c0d5f24ecb3fe2dc18d4ca6
Fixed issue where creating an :class:`.Engine` using multiple calls to
:meth:`.Engine.execution_options` where a subsequent call involved certain
options such as ``isolation_level`` would lead to an internal error
involving event registration.
Fixes: #12289
Change-Id: Iec5fbc0eb0c5a92dda1ea762872ae992ca816685
Adjusted URL parsing and stringification to apply url quoting to the
"database" portion of the URL. This allows a URL where the "database"
portion includes special characters such as question marks to be
accommodated.
Fixes: #11234
Change-Id: If868c96969b70f1090f0b474403d22fd3a2cc529
Added API support for server-side cursors for the oracledb async dialect,
allowing use of the :meth:`_asyncio.AsyncConnection.stream` and similar
stream methods.
Fixes: #10820
Change-Id: I861670ccc20a81ec5ee45132b8059fc2a0359087
still seeing failures
we may very well have to revert this and mark oracle as not supporting
recovery
Change-Id: I4d48607cb8579dc73c650f5232e4414a408735e2
Implemented two-phase transactions for the oracledb dialect. Historically,
this feature never worked with the cx_Oracle dialect, however recent
improvements to the oracledb successor now allow this to be possible. The
two phase transaction API is available at the Core level via the
:meth:`_engine.Connection.begin_twophase` method.
As part of this change, added new facility for testing that allows
a test to skip if a certain step takes too long, allowing for a
separate cleanup step. this is needed as oracle tpc wont allow
commit recovery if transaction is older than about 1 second, could not
find any docs on how to increase this timeout.
Fixed an execute call in the PostgreSQL dialect's provisioning that
drops old tpc transactions which was non-working, which indicates
that we've apparently never had any PG tpc transactions needing to
be cleaned up in CI for some years now, so that's good
Fixes: #11480
Change-Id: If3ad19cc29999e70f07f767b88afd330f6e5a4be
An empty sequence passed to any ``execute()`` method now
raised a deprecation warning, since such an executemany
is invalid.
Pull request courtesy of Carlos Sousa.
Fixes: #9647Closes: #10406
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10406
Pull-request-sha: 087ba2d88d
Change-Id: I482e91a047477c3156a3ca806e5c1eefb6224b95
Fixed issue in the
:paramref:`_engine.Connection.execution_options.logging_token` option,
where changing the value of ``logging_token`` on a connection that has
already logged messages would not be updated to reflect the new logging
token. This in particular prevented the use of
:meth:`_orm.Session.connection` to change the option on the connection,
since the BEGIN logging message would already have been emitted.
Fixes: #11210
Change-Id: I0ddade3778215259a6eacde3a67e09d30bc3257b
Replaces the pyx files with py files that can be both compiled
by cython or imported as is by python.
This avoids the need of duplicating the code to have a python
only fallback.
The cython files are also reorganized to be in the module they use
instead of all being in the cyextension package, that has been
removed.
The performance is pretty much equal between main and this change.
A detailed comparison is at this link
https://docs.google.com/spreadsheets/d/1jkmGpnCyEcPyy6aRK9alElGjxlNHu44Wxjr4VrD99so/edit?usp=sharing
Change-Id: Iaed232ea5dfb41534cc9f58f6ea2f912a93263af
An error is raised if a :class:`.QueuePool` or other non-asyncio pool class
is passed to :func:`_asyncio.create_async_engine`. This engine only
accepts asyncio-compatible pool classes including
:class:`.AsyncAdaptedQueuePool`. Other pool classes such as
:class:`.NullPool` are compatible with both synchronous and asynchronous
engines as they do not perform any locking.
Fixes: #8771
Change-Id: I5843ccea7d824488492d1a9d46207b9f05330ae3
manually update the files to remove literal string concat on the same line,
since black does not seem to be making progress in handling these
Change-Id: I3c651374c5f3db5b8bc0c700328d67ca03743b7b
Removed the async_fallback mode and await_fallback function.
Replace get_event_loop with Runner.
Removed the internal function ``await_fallback()``.
Renamed the internal function ``await_only()`` to ``await_()``.
Change-Id: Ib43829be6ebdb59b6c4447f5a15b5d2b81403fa9
Fixed URL-encoding of the username and password components of
:class:`.engine.URL` objects when converting them to string using the
:meth:`_engine.URL.render_as_string` method, by using Python standard
library ``urllib.parse.quote`` while allowing for plus signs and spaces to
remain unchanged as supported by SQLAlchemy's non-standard URL parsing,
rather than the legacy home-grown routine from many years ago. Pull request
courtesy of Xavier NUNN.
Fixes: #10662Closes: #10726
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10726
Pull-request-sha: 82219041b8
Change-Id: Iedca4929579d4d26ef8cce083252dcd1e476286b
Fixed URL-encoding of the username and password components of
:class:`.engine.URL` objects when converting them to string using the
:meth:`_engine.URL.render_as_string` method, by using Python standard
library ``urllib.parse.quote_plus``, rather than the legacy home-grown
routine from many years ago. Pull request courtesy of Xavier NUNN.
Fixes: #10662Closes: #10726
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10726
Pull-request-sha: 82219041b8
Change-Id: I90b7a9f4dfdb719082b4b178ad4e009a8531a18e
I updated flake8 in pre-commit a few days ago but forgot to do it in
tox.
this flake seems to be picking inside of f-strings, which black does
not fix, so fix these manually.
Change-Id: I9a641a99e280fbba9d893a6f1f051b5039d5d4eb
Fixed 2.0 regression where the :class:`.DDL` construct would no longer
``__repr__()`` due to the removed ``on`` attribute not being accommodated.
Pull request courtesy Iuri de Silvio.
Fixes: #10443Closes: #10442
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10442
Pull-request-sha: 76f0484091
Change-Id: Id49afd681a04e0f05014105b8f15fdb66e6594a0
Fixed issue where under some garbage collection / exception scenarios the
connection pool's cleanup routine would raise an error due to an unexpected
set of state, which can be reproduced under specific conditions.
Fixes: #10414
Change-Id: Ie732f23290d0d3d641f37cd2fee55aff5b9d0857
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [X] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Fixes#10079
**Have a nice day!**
Closes: #10080
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10080
Pull-request-sha: 133c7d6c35
Change-Id: Ie3d998385743680756bc10fbb4f4227669d57648
Adjusted the :paramref:`_sa.create_engine.schema_translate_map` feature
such that **all** schema names in the statement are now tokenized,
regardless of whether or not a specific name is in the immediate schema
translate map given, and to fallback to substituting the original name when
the key is not in the actual schema translate map at execution time. These
two changes allow for repeated use of a compiled object with schema
schema_translate_maps that include or dont include various keys on each
run, allowing cached SQL constructs to continue to function at runtime when
schema translate maps with different sets of keys are used each time. In
addition, added detection of schema_translate_map dictionaries which gain
or lose a ``None`` key across calls for the same statement, which affects
compilation of the statement and is not compatible with caching; an
exception is raised for these scenarios.
Fixes: #10025
Change-Id: I6f5e0c8e067d1702a3647b6251af483669ad854b
the cursor not being consumed leaves the database locked until
the program exits. i cant find any way to free up the file otherwise
Change-Id: Iaa723e217a9206c91e748cf9b0775a06209d599c
Initial fixes to test to accommodate py312
Here we are pulling in a current fork of greenlet that works
for python 3.12. It works though there is one issue with aiosqlite
that might be related.
a non-voting py312 job is added to gerrit / jenkins
Fixes: #9819
Change-Id: I91a51dcbad2902f7c4c7cec88ebbf42c2417b512
Léo Gallot