Support has been re-added for the MySQL-Connector/Python DBAPI using the
``mysql+mysqlconnector://`` URL scheme. The DBAPI now works against
modern MySQL versions as well as MariaDB versions (in the latter case it's
required to pass charset/collation explicitly). Note however that
server side cursor support is disabled due to unresolved issues with this
driver.
References: #12332
Change-Id: I81279478196e830d3c0d5f24ecb3fe2dc18d4ca6
Fixed issue where creating an :class:`.Engine` using multiple calls to
:meth:`.Engine.execution_options` where a subsequent call involved certain
options such as ``isolation_level`` would lead to an internal error
involving event registration.
Fixes: #12289
Change-Id: Iec5fbc0eb0c5a92dda1ea762872ae992ca816685
Adjusted URL parsing and stringification to apply url quoting to the
"database" portion of the URL. This allows a URL where the "database"
portion includes special characters such as question marks to be
accommodated.
Fixes: #11234
Change-Id: If868c96969b70f1090f0b474403d22fd3a2cc529
Added API support for server-side cursors for the oracledb async dialect,
allowing use of the :meth:`_asyncio.AsyncConnection.stream` and similar
stream methods.
Fixes: #10820
Change-Id: I861670ccc20a81ec5ee45132b8059fc2a0359087
still seeing failures
we may very well have to revert this and mark oracle as not supporting
recovery
Change-Id: I4d48607cb8579dc73c650f5232e4414a408735e2
Implemented two-phase transactions for the oracledb dialect. Historically,
this feature never worked with the cx_Oracle dialect, however recent
improvements to the oracledb successor now allow this to be possible. The
two phase transaction API is available at the Core level via the
:meth:`_engine.Connection.begin_twophase` method.
As part of this change, added new facility for testing that allows
a test to skip if a certain step takes too long, allowing for a
separate cleanup step. this is needed as oracle tpc wont allow
commit recovery if transaction is older than about 1 second, could not
find any docs on how to increase this timeout.
Fixed an execute call in the PostgreSQL dialect's provisioning that
drops old tpc transactions which was non-working, which indicates
that we've apparently never had any PG tpc transactions needing to
be cleaned up in CI for some years now, so that's good
Fixes: #11480
Change-Id: If3ad19cc29999e70f07f767b88afd330f6e5a4be
An empty sequence passed to any ``execute()`` method now
raised a deprecation warning, since such an executemany
is invalid.
Pull request courtesy of Carlos Sousa.
Fixes: #9647Closes: #10406
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10406
Pull-request-sha: 087ba2d88d
Change-Id: I482e91a047477c3156a3ca806e5c1eefb6224b95
Fixed issue in the
:paramref:`_engine.Connection.execution_options.logging_token` option,
where changing the value of ``logging_token`` on a connection that has
already logged messages would not be updated to reflect the new logging
token. This in particular prevented the use of
:meth:`_orm.Session.connection` to change the option on the connection,
since the BEGIN logging message would already have been emitted.
Fixes: #11210
Change-Id: I0ddade3778215259a6eacde3a67e09d30bc3257b
Replaces the pyx files with py files that can be both compiled
by cython or imported as is by python.
This avoids the need of duplicating the code to have a python
only fallback.
The cython files are also reorganized to be in the module they use
instead of all being in the cyextension package, that has been
removed.
The performance is pretty much equal between main and this change.
A detailed comparison is at this link
https://docs.google.com/spreadsheets/d/1jkmGpnCyEcPyy6aRK9alElGjxlNHu44Wxjr4VrD99so/edit?usp=sharing
Change-Id: Iaed232ea5dfb41534cc9f58f6ea2f912a93263af
An error is raised if a :class:`.QueuePool` or other non-asyncio pool class
is passed to :func:`_asyncio.create_async_engine`. This engine only
accepts asyncio-compatible pool classes including
:class:`.AsyncAdaptedQueuePool`. Other pool classes such as
:class:`.NullPool` are compatible with both synchronous and asynchronous
engines as they do not perform any locking.
Fixes: #8771
Change-Id: I5843ccea7d824488492d1a9d46207b9f05330ae3
manually update the files to remove literal string concat on the same line,
since black does not seem to be making progress in handling these
Change-Id: I3c651374c5f3db5b8bc0c700328d67ca03743b7b
Removed the async_fallback mode and await_fallback function.
Replace get_event_loop with Runner.
Removed the internal function ``await_fallback()``.
Renamed the internal function ``await_only()`` to ``await_()``.
Change-Id: Ib43829be6ebdb59b6c4447f5a15b5d2b81403fa9
Fixed URL-encoding of the username and password components of
:class:`.engine.URL` objects when converting them to string using the
:meth:`_engine.URL.render_as_string` method, by using Python standard
library ``urllib.parse.quote`` while allowing for plus signs and spaces to
remain unchanged as supported by SQLAlchemy's non-standard URL parsing,
rather than the legacy home-grown routine from many years ago. Pull request
courtesy of Xavier NUNN.
Fixes: #10662Closes: #10726
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10726
Pull-request-sha: 82219041b8
Change-Id: Iedca4929579d4d26ef8cce083252dcd1e476286b
Fixed URL-encoding of the username and password components of
:class:`.engine.URL` objects when converting them to string using the
:meth:`_engine.URL.render_as_string` method, by using Python standard
library ``urllib.parse.quote_plus``, rather than the legacy home-grown
routine from many years ago. Pull request courtesy of Xavier NUNN.
Fixes: #10662Closes: #10726
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10726
Pull-request-sha: 82219041b8
Change-Id: I90b7a9f4dfdb719082b4b178ad4e009a8531a18e
I updated flake8 in pre-commit a few days ago but forgot to do it in
tox.
this flake seems to be picking inside of f-strings, which black does
not fix, so fix these manually.
Change-Id: I9a641a99e280fbba9d893a6f1f051b5039d5d4eb
Fixed 2.0 regression where the :class:`.DDL` construct would no longer
``__repr__()`` due to the removed ``on`` attribute not being accommodated.
Pull request courtesy Iuri de Silvio.
Fixes: #10443Closes: #10442
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10442
Pull-request-sha: 76f0484091
Change-Id: Id49afd681a04e0f05014105b8f15fdb66e6594a0
Fixed issue where under some garbage collection / exception scenarios the
connection pool's cleanup routine would raise an error due to an unexpected
set of state, which can be reproduced under specific conditions.
Fixes: #10414
Change-Id: Ie732f23290d0d3d641f37cd2fee55aff5b9d0857
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [X] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Fixes#10079
**Have a nice day!**
Closes: #10080
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10080
Pull-request-sha: 133c7d6c35
Change-Id: Ie3d998385743680756bc10fbb4f4227669d57648
Adjusted the :paramref:`_sa.create_engine.schema_translate_map` feature
such that **all** schema names in the statement are now tokenized,
regardless of whether or not a specific name is in the immediate schema
translate map given, and to fallback to substituting the original name when
the key is not in the actual schema translate map at execution time. These
two changes allow for repeated use of a compiled object with schema
schema_translate_maps that include or dont include various keys on each
run, allowing cached SQL constructs to continue to function at runtime when
schema translate maps with different sets of keys are used each time. In
addition, added detection of schema_translate_map dictionaries which gain
or lose a ``None`` key across calls for the same statement, which affects
compilation of the statement and is not compatible with caching; an
exception is raised for these scenarios.
Fixes: #10025
Change-Id: I6f5e0c8e067d1702a3647b6251af483669ad854b
the cursor not being consumed leaves the database locked until
the program exits. i cant find any way to free up the file otherwise
Change-Id: Iaa723e217a9206c91e748cf9b0775a06209d599c
Initial fixes to test to accommodate py312
Here we are pulling in a current fork of greenlet that works
for python 3.12. It works though there is one issue with aiosqlite
that might be related.
a non-voting py312 job is added to gerrit / jenkins
Fixes: #9819
Change-Id: I91a51dcbad2902f7c4c7cec88ebbf42c2417b512
fix a handful of warnings that were emitting but not raising,
usually because they were inside an "expect_warnings" block.
modify "expect_warnings" to always use "raise_on_any_unexpected"
behavior; remove this parameter.
Fixed issue in semi-private ``await_only()`` and ``await_fallback()``
concurrency functions where the given awaitable would remain un-awaited if
the function threw a ``GreenletError``, which could cause "was not awaited"
warnings later on if the program continued. In this case, the given
awaitable is now cancelled before the exception is thrown.
Change-Id: I33668c5e8c670454a3d879e559096fb873b57244
Fixed a bug that prevented use of :attr:`_engine.URL.normalized_query` in
SQLAlchemy v2.
Fixes: #9682
Change-Id: I2704154af34f438b4cbb290602fc936c1184c074
Repaired a major shortcoming which was identified in the
:ref:`engine_insertmanyvalues` performance optimization feature first
introduced in the 2.0 series. This was a continuation of the change in
2.0.9 which disabled the SQL Server version of the feature due to a
reliance in the ORM on apparent row ordering that is not guaranteed to take
place. The fix applies new logic to all "insertmanyvalues" operations,
which takes effect when a new parameter
:paramref:`_dml.Insert.returning.sort_by_parameter_order` on the
:meth:`_dml.Insert.returning` or :meth:`_dml.UpdateBase.return_defaults`
methods, that through a combination of alternate SQL forms, direct
correspondence of client side parameters, and in some cases downgrading to
running row-at-a-time, will apply sorting to each batch of returned rows
using correspondence to primary key or other unique values in each row
which can be correlated to the input data.
Performance impact is expected to be minimal as nearly all common primary
key scenarios are suitable for parameter-ordered batching to be
achieved for all backends other than SQLite, while "row-at-a-time"
mode operates with a bare minimum of Python overhead compared to the very
heavyweight approaches used in the 1.x series. For SQLite, there is no
difference in performance when "row-at-a-time" mode is used.
It's anticipated that with an efficient "row-at-a-time" INSERT with
RETURNING batching capability, the "insertmanyvalues" feature can be later
be more easily generalized to third party backends that include RETURNING
support but not necessarily easy ways to guarantee a correspondence
with parameter order.
Fixes: #9618
References: #9603
Change-Id: I1d79353f5f19638f752936ba1c35e4dc235a8b7c
Added :func:`_sa.create_pool_from_url` and
:func:`_asyncio.create_async_pool_from_url` to create
a :class:`_pool.Pool` instance from an input url passed as string
or :class:`_sa.URL`.
Fixes: #9613
Change-Id: Icd8aa3f2849e6fd1bc5341114f3ef8d216a2c543
The support for pool ping listeners to receive exception events via the
:meth:`.ConnectionEvents.handle_error` event added in 2.0.0b1 for
🎫`5648` failed to take into account dialect-specific ping routines
such as that of MySQL and PostgreSQL. The dialect feature has been reworked
so that all dialects participate within event handling. Additionally,
a new boolean element :attr:`.ExceptionContext.is_pre_ping` is added
which identifies if this operation is occurring within the pre-ping
operation.
For this release, third party dialects which implement a custom
:meth:`_engine.Dialect.do_ping` method can opt in to the newly improved
behavior by having their method no longer catch exceptions or check
exceptions for "is_disconnect", instead just propagating all exceptions
outwards. Checking the exception for "is_disconnect" is now done by an
enclosing method on the default dialect, which ensures that the event hook
is invoked for all exception scenarios before testing the exception as a
"disconnect" exception. If an existing ``do_ping()`` method continues to
catch exceptions and check "is_disconnect", it will continue to work as it
did previously, but ``handle_error`` hooks will not have access to the
exception if it isn't propagated outwards.
Fixes: #5648
Change-Id: I6535d5cb389e1a761aad8c37cfeb332c548b876d
Repaired a regression caused by the fix for 🎫`8419` which caused
asyncpg connections to be reset (i.e. transaction ``rollback()`` called)
and returned to the pool normally in the case that the connection were not
explicitly returned to the connection pool and was instead being
intercepted by Python garbage collection, which would fail if the garbage
collection operation were being called outside of the asyncio event loop,
leading to a large amount of stack trace activity dumped into logging
and standard output.
The correct behavior is restored, which is that all asyncio connections
that are garbage collected due to not being explicitly returned to the
connection pool are detached from the pool and discarded, along with a
warning, rather than being returned the pool, as they cannot be reliably
reset. In the case of asyncpg connections, the asyncpg-specific
``terminate()`` method will be used to end the connection more gracefully
within this process as opposed to just dropping it.
This change includes a small behavioral change that is hoped to be useful
for debugging asyncio applications, where the warning that's emitted in the
case of asyncio connections being unexpectedly garbage collected has been
made slightly more aggressive by moving it outside of a ``try/except``
block and into a ``finally:`` block, where it will emit unconditionally
regardless of whether the detach/termination operation succeeded or not. It
will also have the effect that applications or test suites which promote
Python warnings to exceptions will see this as a full exception raise,
whereas previously it was not possible for this warning to actually
propagate as an exception. Applications and test suites which need to
tolerate this warning in the interim should adjust the Python warnings
filter to allow these warnings to not raise.
The behavior for traditional sync connections remains unchanged, that
garbage collected connections continue to be returned to the pool normally
without emitting a warning. This will likely be changed in a future major
release to at least emit a similar warning as is emitted for asyncio
drivers, as it is a usage error for pooled connections to be intercepted by
garbage collection without being properly returned to the pool.
Fixes: #9237
Change-Id: Ib35cfb2e628f2eb2da6d2b65674702556f55603a
Fixed a long-standing race condition in the connection pool which could
occur under eventlet/gevent monkeypatching schemes in conjunction with the
use of eventlet/gevent ``Timeout`` conditions, where a connection pool
checkout that's interrupted due to the timeout would fail to clean up the
failed state, causing the underlying connection record and sometimes the
database connection itself to "leak", leaving the pool in an invalid state
with unreachable entries. This issue was first identified and fixed in
SQLAlchemy 1.2 for 🎫`4225`, however the failure modes detected in
that fix failed to accommodate for ``BaseException``, rather than
``Exception``, which prevented eventlet/gevent ``Timeout`` from being
caught. In addition, a block within initial pool connect has also been
identified and hardened with a ``BaseException`` -> "clean failed connect"
block to accommodate for the same condition in this location.
Big thanks to Github user @niklaus for their tenacious efforts in
identifying and describing this intricate issue.
Fixes: #8974
Change-Id: I95a0e1f080d0cee6f1a66977432a586fdf87f686
Changed how the positional compilation is performed. It's rendered by the compiler
the same as the pyformat compilation. The string is then processed to replace
the placeholders with the correct ones, and to obtain the correct order of the
parameters.
This vastly simplifies the computation of the order of the parameters, that in
case of nested CTE is very hard to compute correctly.
Reworked how numeric paramstyle behavers:
- added support for repeated parameter, without duplicating them like in normal
positional dialects
- implement insertmany support. This requires that the dialect supports out of
order placehoders, since all parameters that are not part of the VALUES clauses
are placed at the beginning of the parameter tuple
- support for different identifiers for a numeric parameter. It's for example
possible to use postgresql style placeholder $1, $2, etc
Added two new dialect based on sqlite to test "numeric" fully using
both :1 style and $1 style. Includes a workaround for SQLite's
not-really-correct numeric implementation.
Changed parmstyle of asyncpg dialect to use numeric, rendering with its native
$ identifiers
Fixes: #8926Fixes: #8849
Change-Id: I7c640467d49adfe6d795cc84296fc7403dcad4d6
Fixed issue where table reflection using :paramref:`.Table.extend_existing`
would fail to deduplicate a same-named column if the existing
:class:`.Table` used a separate key. The
:paramref:`.Table.autoload_replace` parameter would allow the column to be
skipped but under no circumstances should a :class:`.Table` ever have the
same-named column twice.
Additionally, changed deprecation warnings to exceptions
as were implemented in I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5 .
Fixes: #8925
Change-Id: I83d0f8658177a7ffbb06e01dbca91377d1a98d49
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
Added new parameter :paramref:`.PoolEvents.reset.reset_state` parameter to
the :meth:`.PoolEvents.reset` event, with deprecation logic in place that
will continue to accept event hooks using the previous set of arguments.
This indicates various state information about how the reset is taking
place and is used to allow custom reset schemes to take place with full
context given.
Within this change a fix that's also backported to 1.4 is included which
re-enables the :meth:`.PoolEvents.reset` event to continue to take place
under all circumstances, including when :class:`.Connection` has already
"reset" the connection.
The two changes together allow custom reset schemes to be implemented using
the :meth:`.PoolEvents.reset` event, instead of the
:meth:`.PoolEvents.checkin` event (which continues to function as it always
has).
Change-Id: Ie17c4f55d02beb6f570b9de6b3044baffa7d6df6
Fixes: #8717
The :class:`.Sequence` construct restores itself to the DDL behavior it
had prior to the 1.4 series, where creating a :class:`.Sequence` with
no additional arguments will emit a simple ``CREATE SEQUENCE`` instruction
**without** any additional parameters for "start value". For most backends,
this is how things worked previously in any case; **however**, for
MS SQL Server, the default value on this database is
``-2**63``; to prevent this generally impractical default
from taking effect on SQL Server, the :paramref:`.Sequence.start` parameter
should be provided. As usage of :class:`.Sequence` is unusual
for SQL Server which for many years has standardized on ``IDENTITY``,
it is hoped that this change has minimal impact.
Fixes: #7211
Change-Id: I1207ea10c8cb1528a1519a0fb3581d9621c27b31
the sqlite builds on github actions seem to be very
inconsistent about versions and many don't support
RETURNING. ensure any tests that depend on RETURNING present
are marked as such.
Change-Id: I7a60a81fa70b90642448cdd58eda33212c3afebc
the feature is enabled for all built in backends
when RETURNING is used,
except for Oracle that doesn't need it, and on
psycopg2 and mssql+pyodbc it is used for all INSERT statements,
not just those that use RETURNING.
third party dialects would need to opt in to the new feature
by setting use_insertmanyvalues to True.
Also adds dialect-level guards against using returning
with executemany where we dont have an implementation to
suit it. execute single w/ returning still defers to the
server without us checking.
Fixes: #6047Fixes: #7907
Change-Id: I3936d3c00003f02e322f2e43fb949d0e6e568304
For improved security, the :class:`_url.URL` object will now use password
obfuscation by default when ``str(url)`` is called. To stringify a URL with
cleartext password, the :meth:`_url.URL.render_as_string` may be used,
passing the :paramref:`_url.URL.render_as_string.hide_password` parameter
as ``False``. Thanks to our contributors for this pull request.
Fixes: #8567Closes: #8563
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8563
Pull-request-sha: d1f1127f75
Change-Id: If756c8073ff99ac83876d9833c8fe1d7c76211f9
The :class:`_pool.QueuePool` now ignores ``max_overflow`` when
``pool_size=0``, properly making the pool unlimited in all cases.
Fixes: #8523
Change-Id: Ifc32eb47a281c4b3acf357352f07b9b8a73d1b6f
Due to the change for #5648 in ad14471bc9,
the mock in test_recycle_pool_no_race needs to implement
an empty handle_error dispatcher.
the exception was in a thread so did not interrupt the
test suite:
File "/home/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 2059, in _handle_dbapi_exception
for fn in self.dialect.dispatch.handle_error:
TypeError: 'Mock' object is not iterable
Change-Id: I764357c48aa1bf53a572d8ee24c89a7463505092
Integrated support for asyncpg's ``terminate()`` method call for cases
where the connection pool is recycling a possibly timed-out connection,
where a connection is being garbage collected that wasn't gracefully
closed, as well as when the connection has been invalidated. This allows
asyncpg to abandon the connection without waiting for a response that may
incur long timeouts.
Fixes: #8419
Change-Id: Ia575af779d5733b483a72dff3690b8bbbad2bb05
Implemented the DDL event hooks :meth:`.DDLEvents.before_create`,
:meth:`.DDLEvents.after_create`, :meth:`.DDLEvents.before_drop`,
:meth:`.DDLEvents.after_drop` for all :class:`.SchemaItem` objects that
include a distinct CREATE or DROP step, when that step is invoked as a
distinct SQL statement, including for :class:`.ForeignKeyConstraint`,
:class:`.Sequence`, :class:`.Index`, and PostgreSQL's
:class:`_postgresql.ENUM`.
Fixes: #8394
Change-Id: Iee6274984e794f50103451a04d089641d6ac443a
The PostgreSQL dialect now supports reflection of expression based indexes.
The reflection is supported both when using
:meth:`_engine.Inspector.get_indexes` and when reflecting a
:class:`_schema.Table` using :paramref:`_schema.Table.autoload_with`.
Thanks to immerrr and Aidan Kane for the help on this ticket.
Fixes: #7442
Change-Id: I3e36d557235286c0f7f6d8276272ff9225058d48
The warnings that are emitted regarding reflection of indexes or unique
constraints, when the :paramref:`.Table.include_columns` parameter is used
to exclude columns that are then found to be part of those constraints,
have been removed. When the :paramref:`.Table.include_columns` parameter is
used it should be expected that the resulting :class:`.Table` construct
will not include constraints that rely upon omitted columns. This change
was made in response to 🎫`8100` which repaired
:paramref:`.Table.include_columns` in conjunction with foreign key
constraints that rely upon omitted columns, where the use case became
clear that omitting such constraints should be expected.
Fixes: #8102
Change-Id: Id32f628def2d12499cd49d0b436ed345fe49dc6b
Rearchitected the schema reflection API to allow some dialects to make use
of high performing batch queries to reflect the schemas of many tables at
once using much fewer queries. The new performance features are targeted
first at the PostgreSQL and Oracle backends, and may be applied to any
dialect that makes use of SELECT queries against system catalog tables to
reflect tables (currently this omits the MySQL and SQLite dialects which
instead make use of parsing the "CREATE TABLE" statement, however these
dialects do not have a pre-existing performance issue with reflection. MS
SQL Server is still a TODO).
The new API is backwards compatible with the previous system, and should
require no changes to third party dialects to retain compatibility;
third party dialects can also opt into the new system by implementing
batched queries for schema reflection.
Along with this change is an updated reflection API that is fully
:pep:`484` typed, features many new methods and some changes.
Fixes: #4379
Change-Id: I897ec09843543aa7012bcdce758792ed3d415d08
Mike Bayer