Added new "lifo" mode to :class:`.QueuePool`, typically enabled by setting
the flag :paramref:`.create_engine.pool_use_lifo` to True. "lifo" mode
means the same connection just checked in will be the first to be checked
out again, allowing excess connections to be cleaned up from the server
side during periods of the pool being only partially utilized. Pull request
courtesy Taem Park.
Change-Id: Idb5e299c5082b3e6b547bd03022acf65fdc34f35
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/467
Fixed bug where a :class:`.Sequence` would be dropped explicitly before any
:class:`.Table` that refers to it, which breaks in the case when the
sequence is also involved in a server-side default for that table, when
using :meth:`.MetaData.drop_all`. The step which processes sequences
to be dropped via non server-side column default functions is now invoked
after the table itself is dropped.
Change-Id: I185f2cc76d2011ad4dd3ba9bde5d8aef0ec335ae
Fixes: #4300
for better separation of concerns split out the
core pooling constructs from the implementations. also
makes more room for new pool classes should we decide to
add them. The DBProxy feature is never used so remove it
from documentation and split it out.
Change-Id: I04c9d7ae1882b17f3cf5c37ed05b781b05ef88fa
Fixes: #4255
Fixed connection pool issue whereby if a disconnection error were raised
during the connection pool's "reset on return" sequence in conjunction with
an explicit transaction opened against the enclosing :class:`.Connection`
object (such as from calling :meth:`.Session.close` without a rollback or
commit, or calling :meth:`.Connection.close` without first closing a
transaction declared with :meth:`.Connection.begin`), a double-checkin would
result, which could then lead towards concurrent checkouts of the same
connection. The double-checkin condition is now prevented overall by an
assertion, as well as the specific double-checkin scenario has been
fixed.
Change-Id: If5bb6941e36326846b14918c33ebfdd5604f642e
Fixes: #4252
Fixed a reference leak issue where the values of the parameter dictionary
used in a statement execution would remain referenced by the "compiled
cache", as a result of storing the key view used by Python 3 dictionary
keys(). Pull request courtesy Olivier Grisel.
Change-Id: Icfb0f38111a165780f6dd3e4e3382a03df79ce26
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/441
Fixed 1.2 regression caused by 🎫`4060` where the query used to
reflect SQL Server cross-schema foreign keys was limiting the criteria
incorrectly.
Additionally, added some rework of the inter-schema reflection tests
so that MySQL, MSSQL can be included, breaking out some of the
Postgresql-specific behaviors into separate requirements.
Fixes: #4234
Change-Id: I20c8e70707075f1767b79127c2c27d4b313c6515
Fixed bug in connection pool where a connection could be present in the
pool without all of its "connect" event handlers called, if a previous
"connect" handler threw an exception; note that the dialects themselves
have connect handlers that emit SQL, such as those which set transaction
isolation, which can fail if the database is in a non-available state, but
still allows a connection. The connection is now invalidated first if any
of the connect handlers fail.
Change-Id: I61d6f4827a98ab8455f1c3e1c55d046eeccec09a
Fixes: #4225
Fixed regression caused in 1.2.3 due to fix from 🎫`4181` where
the changes to the event system involving :class:`.Engine` and
:class:`.OptionEngine` did not accommodate for event removals, which
would raise an ``AttributeError`` when invoked at the class
level.
Change-Id: I1c9083829d74dd710716d28b0eaca4fa15e86313
Fixes: #4190
The :class:`.URL` object now allows query keys to be specified multiple
times where their values will be joined into a list. This is to support
the plugins feature documented at :class:`.CreateEnginePlugin` which
documents that "plugin" can be passed multiple times. Additionally, the
plugin names can be passed to :func:`.create_engine` outside of the URL
using the new :paramref:`.create_engine.plugins` parameter.
Change-Id: Ifc48ad120bd6c6204eda567492caf79832aeeaa5
Fixes: #4170
Fixed a fairly serious connection pool bug where a connection that is
acquired after being refreshed as a result of a user-defined
:class:`.DisconnectionError` or due to the 1.2-released "pre_ping" feature
would not be correctly reset if the connection were returned to the pool by
weakref cleanup (e.g. the front-facing object is garbage collected); the
weakref would still refer to the previously invalidated DBAPI connection
which would have the reset operation erroneously called upon it instead.
This would lead to stack traces in the logs and a connection being checked
into the pool without being reset, which can cause locking issues.
Change-Id: Iabd9f3a63a1d0207d0de0054a6ced3560818cf9c
Fixes: #4184
Fixed bug where events associated with an :class:`Engine`
at the class level would be doubled when the
:meth:`.Engine.execution_options` method were used. To
achieve this, the semi-private class :class:`.OptionEngine`
no longer accepts events directly at the class level
and will raise an error; the class only propagates class-level
events from its parent :class:`.Engine`. Instance-level
events continue to work as before.
The comments present another way of doing this where we would
copy events from the parent engine at option time rather
than linking the event listeners, but this would be a behavioral
change that adding new events to the parent engine would not
take effect for an already-created OptionEngine.
Change-Id: Id128516f54103fbad9a2210d6571eceb59c8b0cb
Fixes: #4181
Fixed bug in cx_Oracle disconnect detection, used by pre_ping and other
features, where an error could be raised as DatabaseError which includes a
numeric error code; previously we weren't checking in this case for a
disconnect code.
Change-Id: I359bb5ede25a4726ea632b81af83c6391f405ae1
Fixes: #4182
This test often fails on CI under load and reveals a small
race condition within SingleonThreadPool's "cleanup" system.
As this pool's "cleanup" system is pretty much never used in
any case since this pool is no longer used for SQLite file
databases, this race should have little impact, especially
since the occasional non-closed connection will get closed
out when it's GC'ed in any case.
Change-Id: I68aefc0f9bbfc708c95af0497844e65366fc8429
The "password" attribute of the :class:`.url.URL` object can now be
any user-defined or user-subclassed string object that responds to the
Python ``str()`` builtin. The object passed will be maintained as the
datamember :attr:`.url.URL.password_original` and will be consulted
when the :attr:`.url.URL.password` attribute is read to produce the
string value.
Change-Id: I91d101c3b10e135ae7e4de60a5104b51776db84f
Fixes: #4089
In release 1.1, the :class:`.Boolean` type was broken in that
boolean coercion via ``bool()`` would occur for backends that did not
feature "native boolean", but would not occur for native boolean backends,
meaning the string ``"0"`` now behaved inconsistently. After a poll, a
consensus was reached that non-boolean values should be raising an error,
especially in the ambiguous case of string ``"0"``; so the :class:`.Boolean`
datatype will now raise ``ValueError`` if an incoming value is not
within the range ``None, True, False, 1, 0``.
Change-Id: If70c4f79c266f0dd1a0306c0ffe7acb9c66c4cc3
Fixes: #4102
SQL Server has an entirely different use for the TIMESTAMP
datatype that is unrelated to the SQL standard's version of this
type. It is a read-only type that returns an incrementing
binary value. The ROWVERSION name will supersede the TIMESTAMP
name. Implement datatype objects for both, separate from the
base DateTime/TIMESTAMP class hierarchy, and also implement
an optional integer coercion feature.
Change-Id: Ie2bd43b7aac57760b8ec6ff6e26460e2086a95eb
Fixes: #4086
Fixed issue where CURRENT_TIMESTAMP would not reflect correctly
in the MariaDB 10.2 series due to a syntax change, where the function
is now represented as ``current_timestamp()``.
Fixes: #4096
MariaDB 10.2 now supports CHECK constraints (warning: use version 10.2.9
or greater due to upstream issues noted in 🎫`4097`). Reflection
now takes these CHECK constraints into account when they are present in
the ``SHOW CREATE TABLE`` output.
Fixes: #4098
Change-Id: I8666d61814e8145ca12cbecad94019b44af868e3
Fixed more regressions caused by cx_Oracle 6.0; at the moment, the only
behavioral change for users is disconnect detection now detects for
cx_Oracle.DatabaseError in addition to cx_Oracle.InterfaceError, as
this behavior seems to have changed. Other issues regarding numeric
precision and uncloseable connections are pending with the upstream
cx_Oracle issue tracker.
Change-Id: Id61f1e33b21c155a598396340dfdecd28ff4066b
Fixes: #4045
Added support for views that are unreflectable due to stale
table definitions, when calling :meth:`.MetaData.reflect`; a warning
is emitted for the table that cannot respond to ``DESCRIBE``
but the operation succeeds. The MySQL dialect now
raises UnreflectableTableError which is in turn caught by
MetaData.reflect(). Reflecting the view standalone raises
this error directly.
Change-Id: Id8005219d8e073c154cc84a873df911b4a6cf4d6
Fixes: #3871
Fixed bug where a SQL-oriented Python-side column default could fail to
be executed properly upon INSERT in the "pre-execute" codepath, if the
SQL itself were an untyped expression, such as plain text. The "pre-
execute" codepath is fairly uncommon however can apply to non-integer
primary key columns with SQL defaults when RETURNING is not used.
Tests exist here to ensure typing is applied to
a typed expression for default, but in the case of
an untyped SQL value, we know the type from the column,
so apply this.
Change-Id: I5d8b391611c137b9f700115a50a2bf5b30abfe94
Fixes: #3923
Added an exception handler that will warn for the "cause" exception on
Py2K when the "autorollback" feature of :class:`.Connection` itself
raises an exception. In Py3K, the two exceptions are naturally reported
by the interpreter as one occurring during the handling of the other.
This is continuing with the series of changes for rollback failure
handling that were last visited as part of 🎫`2696` in 1.0.12.
Change-Id: I600ba455a14ebaea27c6189889181f97c632f179
Fixes: #3946
Added native "pessimistic disconnection" handling to the :class:`.Pool`
object. The new parameter :paramref:`.Pool.pre_ping`, available from
the engine as :paramref:`.create_engine.pool_pre_ping`, applies an
efficient form of the "pre-ping" recipe featured in the pooling
documentation, which upon each connection check out, emits a simple
statement, typically "SELECT 1", to test the connection for liveness.
If the existing connection is no longer able to respond to commands,
the connection is transparently recycled, and all other connections
made prior to the current timestamp are invalidated.
Change-Id: I89700d0075e60abd2250e54b9bd14daf03c71c00
Fixes: #3919
Added support for SQL comments on :class:`.Table` and :class:`.Column`
objects, via the new :paramref:`.Table.comment` and
:paramref:`.Column.comment` arguments. The comments are included
as part of DDL on table creation, either inline or via an appropriate
ALTER statement, and are also reflected back within table reflection,
as well as via the :class:`.Inspector`. Supported backends currently
include MySQL, Postgresql, and Oracle.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #1546
Change-Id: Ib90683850805a2b4ee198e420dc294f32f15d35d
Fixed bug where in the unusual case of passing a
:class:`.Compiled` object directly to :meth:`.Connection.execute`,
the dialect with which the :class:`.Compiled` object were generated
was not consulted for the paramstyle of the string statement, instead
assuming it would match the dialect-level paramstyle, causing
mismatches to occur.
Change-Id: I114e4db2183fbb75bb7c0b0641f5a161855696ee
Fixes: #3938
Fixed bug whereby the :meth:`.DDLEvents.column_reflect` event would not
allow a non-textual expression to be passed as the value of the
"default" for the new column, such as a :class:`.FetchedValue`
object to indicate a generic triggered default or a
:func:`.sql.expression.text` construct. Clarified the documentation
in this regard as well.
Fixes: #3905
Change-Id: I829796c3e9f87f375149bebee7eef133a6876d4d
Avoid putting engine password in the exception message in
`MetaData.reflect` (since exception messages often appear in logs).
Use the same redacted `__repr__` implementation in
`TLEngine` as in its base class `Engine`
Change-Id: Ic0a7baea917a9c8d87dffdd82ef566673ab08e02
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/327
Corrects some warnings and adds tox config. Adds DeprecationWarning
to the error category. Large sweep for string literals w/ backslashes
as this is common in docstrings
Co-authored-by: Andrii Soldatenko
Fixes: #3886
Change-Id: Ia7c838dfbbe70b262622ed0803d581edc736e085
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/337
The "extend_existing" option of :class:`.Table` reflection would
cause indexes and constraints to be doubled up in the case that the parameter
were used with :meth:`.MetaData.reflect` (as the automap extension does)
due to tables being reflected both within the foreign key path as well
as directly. A new de-duplicating set is passed through within the
:meth:`.MetaData.reflect` sequence to prevent double reflection in this
way.
Change-Id: Ibf6650c1e76a44ccbe15765fd79df2fa53d6bac7
Fixes: #3861
Tests illustrate that exceptions like GreenletExit and
even KeyboardInterrupt can corrupt the state of a DBAPI
connection like that of pymysql and mysqlclient. Intercept
BaseException errors within the handle_error scheme and
invalidate just the connection alone in this case, but not
the whole pool.
The change is backwards-incompatible with a program that
currently intercepts ctrl-C within a database transaction
and wants to continue working on that transaction. Ensure
the event hook can be used to reverse this behavior.
Change-Id: Ifaa013c13826d123eef34e32b7e79fff74f1b21b
Fixes: #3803
- add a connect=True key to connection record to support
pre-loading of _ConnectionRecord objects
- ensure _ConnectionRecord.close() leaves the record in a good
state for reopening
- add _ConnectionRecord.record_info for persistent storage
- add "in_use" accessor based on fairy_ref being present or not
- allow for the exclusions system and SuiteRequirements to be
usable without the full plugin_base setup.
- move some Python-env requirements to the importable
requirements.py module.
- allow starttime to be queried
- add additional events for engine plugins
- have "dialect" be a first-class parameter to the pool,
ensure the engine strategy supplies it up front
Change-Id: Ibf549f7a1766e49d335cd6f5e26bacfaef9a8229
Raise a more descriptive exception / message when ClauseElement
or non-SQLAlchemy objects that are not "executable" are erroneously
passed to ``.execute()``; a new exception ObjectNotExecutableError
is raised consistently in all cases.
Change-Id: I2dd393121e2c7e5b6b9e40286a2f25670876e8e4
Fixes: #3786
The processing performed by the :class:`.Boolean` datatype for backends
that only feature integer types has been made consistent between the
pure Python and C-extension versions, in that the C-extension version
will accept any integer value from the database as a boolean, not just
zero and one; additionally, non-boolean integer values being sent to
the database are coerced to exactly zero or one, instead of being
passed as the original integer value.
Change-Id: I01e647547fd7047bd549dd70e1fa202c51e8328b
Fixes: #3730
count() here is misleading in that it not only
counts from an arbitrary column in the table, it also
does not make accommodations for DISTINCT, JOIN, etc.
as the ORM-level function does. Core should not be
attempting to provide a function like this.
Change-Id: I9916fc51ef744389a92c54660ab08e9695b8afc2
Fixes: #3724
Previously, it was impossible to have a Table that has
None for a schema name when the "schema" parameter on
MetaData was set. A new symbol sqlalchemy.schema.BLANK_SCHEMA
is added which indicates that the schema name should unconditionally
be set to None. In particular, this value must be passed within
cross-schema foreign key reflection, so that a Table which
is in the "default" schema can be represented properly.
Fixes: #3716
Change-Id: I3d24f99c22cded206c5379fd32a225e74edb7a8e
Taem Park