this concept is not clear that we offer real
DBAPI autocommit everywhere. backport 1.3 with edits
as well
Change-Id: I2e8328b7fb6e1cdc5453ab29c94276f60c7ca149
(cherry picked from commit 28fbb0cb94)
Fixed issue in :class:`.URL` object where stringifying the object
would not URL encode special characters, preventing the URL from being
re-consumable as a real URL. Pull request courtesy Miguel Grinberg.
Fixes: #5341Closes: #5342
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5342
Pull-request-sha: 362ca33983
Change-Id: Ief6218122d1ec0c70479eb1a90e1c16433801924
(cherry picked from commit 2644693c0b)
The assumptions in _discard_transaction from
916e1fea25 were too narrow,
assuming that if the given transaction were not our
"current" one, that this would not be the reset agent. however
as the legacy behvaior is that even a "nested" transaction gets
set as "self._transaction", this did not accommodate for the nested
transaction being thrown away. We will attempt to refine all of this
logic in #5327 for 1.4 /master assuming this is feasible for the
full suite of current use cases.
Fixes: #5326
Change-Id: I6787e82c9e50c23317f87d0d094122c6a6f066da
(cherry picked from commit 79de84b25e)
Fixed fairly critical issue where the DBAPI connection could be returned to
the connection pool while still in an un-rolled-back state. The reset agent
responsible for rolling back the connection could be corrupted in the case
that the transaction was "closed" without being rolled back or committed,
which can occur in some scenarios when using ORM sessions and emitting
.close() in a certain pattern involving savepoints. The fix ensures that
the reset agent is always active.
Note for 1.3 the cherry-pick modifies the approach from master as
transaction handling has diverged.
Fixes: #5326
Change-Id: If056870ea70a2d9a1749768988d5e023f3061b31
(cherry picked from commit 4d161ac9c28986e1e022dfb93785767f28d6bfc8)
As part of this change Oracle also gets the concept of a
default isolation level, however since Oracle does not provide a
fixed method of knowing what the isolation level would be without a
server side transaction actually in progress, for now we hardcode
just to "READ COMMITTED".
Enhanced the test suite for isolation level testing in the dialect
test suite and added features to requirements so that the supported
isolation levels can be reported generically for dialects.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5200
Change-Id: I2c4d49da9ff80ccc228c21e196ec9a961de53478
(cherry picked from commit 1afca5823efe22c0f8cbc01455bafc573254ab55)
Added support for reflection of "computed" columns, which are now returned
as part of the structure returned by :meth:`.Inspector.get_columns`.
When reflecting full :class:`.Table` objects, computed columns will
be represented using the :class:`.Computed` construct.
Also improve the documentation in :meth:`Inspector.get_columns`, correctly
listing all the returned keys.
Fixes: #5063Fixes: #4051Closes: #5064
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5064
Pull-request-sha: ba00fc321ce468f8885aad23b3dd33c789e50fbe
Change-Id: I789986554fc8ac7f084270474d0b2c12046b1cc2
(cherry picked from commit 62b7dace0c)
1. ensure provision.py loads dialect implementations when running
reap_dbs.py. Reapers haven't been working since
598f2f7e55 .
2. add some exclusion rules to allow the sqlite_file target to work;
add to tox.
3. add reap dbs target for SQLite, repair SQLite drop_db routine
which also wasn't doing the right thing for memory databases
etc.
4. Fix logging in provision files, as the main provision logger
is the one that's enabled by reap_dbs and maybe others, have all
the provision files use the provision logger.
Fixes: #5180Fixes: #5168
Change-Id: Ibc1b0106394d20f5bcf847f37b09d185f26ac9b5
(cherry picked from commit 08fdf6da228aab54263abe043c21bcebf3513e72)
Observing a SQLite connection/cursor being hung on
test_resultset -> PositionalTextTest -> test_dupe_col_obj.
this uses connectionless execution and the result object
fails to be constructed. When that happens, there is no path
for the cursor or connection to be closed / released.
Recent changes with the exception assertions in #4849 seem to
be causing a cycle to last a little longer than usual which
is exposing this issue for one particular test on SQLite.
As we want to get rid of weakref cleanup, evaluate
why we dont have handle_dbapi_exception for this whole
block, as after_cursor_execute can raise, result construction
can raise, autocommit can raise, close can raise, there
does not seem to be a reason these things should be outside
of the block that gets cleaned up.
Fixes: #5182
Change-Id: I640ac55e8c5f39d287f779fbb5dc0ab727218ca3
(cherry picked from commit 4cd33f18a21d9e33b37ef7163822c327453d1e62)
Applied an explicit "cause" to most if not all internally raised exceptions
that are raised from within an internal exception catch, to avoid
misleading stacktraces that suggest an error within the handling of an
exception. While it would be preferable to suppress the internally caught
exception in the way that the ``__suppress_context__`` attribute would,
there does not as yet seem to be a way to do this without suppressing an
enclosing user constructed context, so for now it exposes the internally
caught exception as the cause so that full information about the context
of the error is maintained.
Fixes: #4849
Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
(cherry picked from commit 8be0dae77a7e0747f0d0fb4282db4aea7f41e03a)
For Windows, time.time() may only have 16 millisecond
accuracy, so invalidation routines which compare
the time.time() of invalidate() to the time.time() when
the ConnectionRecord last connected may fail in a unit test
environment that does not pause at least this much time
since the ConnectionRecord startup. Using >= for comparison
instead of > was considered but this only leads to more confusing
results as the ConnecitonRecord goes into a re-connect loop
as time continues to not pass.
Overall, while using routines such as Python 3.7's time_ns()
might be helpful, for now make sure tests which rely on this
are marked under timing intensive and add small sleeps.
Change-Id: I1a7162e67912d22c135fa517b687a073f8fd9151
(cherry picked from commit f1a22596e2)
The fails_on decorator was not being interpreted
correctly when multiple were present.
Remove obsolete fails_on from test_types that no longer
take place for MySQL, Oracle
Ensure test_types tests are using __backend__
mark currently failing Oracle interval tests
Change-Id: If8db0c02b31a8008fd1673c2380f1f974c3806a6
(cherry picked from commit 56dff403b7cbaf2de342c83d496d0937f5b719b1)
Fixed bug where parameter repr as used in logging and error reporting needs
additional context in order to distinguish between a list of parameters for
a single statement and a list of parameter lists, as the "list of lists"
structure could also indicate a single parameter list where the first
parameter itself is a list, such as for an array parameter. The
engine/connection now passes in an additional boolean indicating how the
parameters should be considered. The only SQLAlchemy backend that expects
arrays as parameters is that of psycopg2 which uses pyformat parameters,
so this issue has not been too apparent, however as other drivers that use
positional gain more features it is important that this be supported. It
also eliminates the need for the parameter repr function to guess based on
the parameter structure passed.
Fixes: #4902
Change-Id: I086246ee0eb51484adbefd83e07295fa56576c5f
(cherry picked from commit 9488480abe)
Added new parameter :paramref:`.create_engine.hide_parameters` which when
set to True will cause SQL parameters to no longer be logged, nor rendered
in the string representation of a :class:`.StatementError` object.
Fixes: #4815
Change-Id: Ib87f868b6936cf6b42b192644e9d732ec24266c2
(cherry picked from commit 4b321e8a5e)
Fixed an issue whereby if the dialect "initialize" process which occurs on
first connect would encounter an unexpected exception, the initialize
process would fail to complete and then no longer attempt on subsequent
connection attempts, leaving the dialect in an un-initialized, or partially
initialized state, within the scope of parameters that need to be
established based on inspection of a live connection. The "invoke once"
logic in the event system has been reworked to accommodate for this
occurrence using new, private API features that establish an "exec once"
hook that will continue to allow the initializer to fire off on subsequent
connections, until it completes without raising an exception. This does not
impact the behavior of the existing ``once=True`` flag within the event
system.
Fixes: #4807
Change-Id: Iec32999b61b6af4b38b6719e0c2651454619078c
(cherry picked from commit 2051fa2ce9)
Fixed bug where using reflection function such as :meth:`.MetaData.reflect`
with an :class:`.Engine` object that had execution options applied to it
would fail, as the resulting :class:`.OptionEngine` proxy object failed to
include a ``.engine`` attribute used within the reflection routines.
Fixes: #4754
Change-Id: I6c342af5c6db6fe362b9d25f3f26d6859f62f87a
(cherry picked from commit 3238d953b4)
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient. test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.
Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
(cherry picked from commit 190e0139e8)
Moved the "rollback" which occurs during dialect initialization so that it
occurs after additional dialect-specific initialize steps, in particular
those of the psycopg2 dialect which would inadvertently leave transactional
state on the first new connection, which could interfere with some
psycopg2-specific APIs which require that no transaction is started. Pull
request courtesy Matthew Wilkes.
Fixes: #4663Closes: #4664
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4664
Pull-request-sha: e544fe671d
Change-Id: If40a15a1679b4eec0b8b8222f678697728009c30
(cherry picked from commit f601791a91)
MySQL 8.0.16 introduces real CHECK constraints and MariaDB has also
added them into the 10.2 series sometime before 10.2.22.
Change-Id: Ia0f1be69f99df935aae069f63381bcc994f73cc7
(cherry picked from commit 1c3e926273)
Fixed behavioral regression as a result of deprecating the "use_threadlocal"
flag for :class:`.Pool`, where the :class:`.SingletonThreadPool` no longer
makes use of this option which causes the "rollback on return" logic to take
place when the same :class:`.Engine` is used multiple times in the context
of a transaction to connect or implicitly execute, thereby cancelling the
transaction. While this is not the recommended way to work with engines
and connections, it is nonetheless a confusing behavioral change as when
using :class:`.SingletonThreadPool`, the transaction should stay open
regardless of what else is done with the same engine in the same thread.
The ``use_threadlocal`` flag remains deprecated however the
:class:`.SingletonThreadPool` now implements its own version of the same
logic.
Fixes: #4585
Change-Id: I906293f2d0a5d14ed46cd9e64305a6481505a5a3
This test is non-critical as the SingletonThreadPool is
not production-level robust under multithreaded scenarios in
any case.
Fixes: #4527
Change-Id: Ie19ebd69438c97b2d8adb571f8f1b2c56894f7fb
Comparing two objects of :class:`.URL` using ``__eq__()`` did not take port
number into consideration, two objects differing only by port number were
considered equal. Port comparison is now added in ``__eq__()`` method of
:class:`.URL`, objects differing by port number are now not equal.
Additionally, ``__ne__()`` was not implemented for :class:`.URL` which
caused unexpected result when ``!=`` was used in Python2, since there are no
implied relationships among the comparison operators in Python2.
Fixes: #4406Closes: #4515
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4515
Pull-request-sha: 0f15b805f0
Change-Id: Iba7d224f1282dc3f4b884d1a746f2d46669f551e
Added new parameters :paramref:`.Table.resolve_fks` and
:paramref:`.MetaData.reflect.resolve_fks` which when set to False will
disable the automatic reflection of related tables encountered in
:class:`.ForeignKey` objects, which can both reduce SQL overhead for omitted
tables as well as avoid tables that can't be reflected for database-specific
reasons. Two :class:`.Table` objects present in the same :class:`.MetaData`
collection can still refer to each other even if the reflection of the two
tables occurred separately.
Fixes: #4517
Change-Id: I623baed42042a16c5109e4c8af6b2f64d2d00f95
Revised the formatting for :class:`.StatementError` when stringified. Each
error detail is broken up over multiple newlines instead of spaced out on a
single line. Additionally, the SQL representation now stringifies the SQL
statement rather than using ``repr()``, so that newlines are rendered as is.
Pull request courtesy Nate Clark.
Fixes: #4500Closes: #4501
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4501
Pull-request-sha: 60cc0ee68d
Change-Id: I79d8418b7495e5691c9a56f41e79495c26a967ff
Added accessors for execution options to Core and ORM, via
:meth:`.Query.get_execution_options`,
:meth:`.Connection.get_execution_options`,
:meth:`.Engine.get_execution_options`, and
:meth:`.Executable.get_execution_options`. PR courtesy Daniel Lister.
Fixes: #4406Closes: #4465
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4465
Pull-request-sha: 9674688bb5
Change-Id: I93ba51d7a2d687e255edd6938db15615e56dd237
A large change throughout the library has ensured that all objects, parameters,
and behaviors which have been noted as deprecated or legacy now emit
``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now
defaults to displaying deprecation warnings, as well as that modern test suites
based on tools like tox and pytest tend to display deprecation warnings,
this change should make it easier to note what API features are obsolete.
See the notes added to the changelog and migration notes for further
details.
Fixes: #4393
Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
These changes should be ported from 1.3 back to 1.0 or
possibly 0.9 to the extent they are relevant in each
version. In 1.3 we hope to turn all deprecation documentation
into warnings.
Change-Id: I205186cde161af9389af513a425c62ce90dd54d8
Fixed a regression introduced in version 1.2 where a refactor
of the :class:`.SQLAlchemyError` base exception class introduced an
inappropriate coercion of a plain string message into Unicode under
python 2k, which is not handled by the Python interpreter for characters
outside of the platform's encoding (typically ascii). The
:class:`.SQLAlchemyError` class now passes a bytestring through under
Py2K for ``__str__()`` as is the behavior of exception objects in general
under Py2K, does a safe coercion to unicode utf-8 with
backslash fallback for ``__unicode__()``. For Py3K the message is
typically unicode already, but if not is again safe-coerced with utf-8
with backslash fallback for the ``__str__()`` method.
Fixes: #4429
Change-Id: I2289da3f2c45c7d0041fa43d838958f7614defc3
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Added new "lifo" mode to :class:`.QueuePool`, typically enabled by setting
the flag :paramref:`.create_engine.pool_use_lifo` to True. "lifo" mode
means the same connection just checked in will be the first to be checked
out again, allowing excess connections to be cleaned up from the server
side during periods of the pool being only partially utilized. Pull request
courtesy Taem Park.
Change-Id: Idb5e299c5082b3e6b547bd03022acf65fdc34f35
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/467
Fixed bug where a :class:`.Sequence` would be dropped explicitly before any
:class:`.Table` that refers to it, which breaks in the case when the
sequence is also involved in a server-side default for that table, when
using :meth:`.MetaData.drop_all`. The step which processes sequences
to be dropped via non server-side column default functions is now invoked
after the table itself is dropped.
Change-Id: I185f2cc76d2011ad4dd3ba9bde5d8aef0ec335ae
Fixes: #4300
for better separation of concerns split out the
core pooling constructs from the implementations. also
makes more room for new pool classes should we decide to
add them. The DBProxy feature is never used so remove it
from documentation and split it out.
Change-Id: I04c9d7ae1882b17f3cf5c37ed05b781b05ef88fa
Fixes: #4255
Fixed connection pool issue whereby if a disconnection error were raised
during the connection pool's "reset on return" sequence in conjunction with
an explicit transaction opened against the enclosing :class:`.Connection`
object (such as from calling :meth:`.Session.close` without a rollback or
commit, or calling :meth:`.Connection.close` without first closing a
transaction declared with :meth:`.Connection.begin`), a double-checkin would
result, which could then lead towards concurrent checkouts of the same
connection. The double-checkin condition is now prevented overall by an
assertion, as well as the specific double-checkin scenario has been
fixed.
Change-Id: If5bb6941e36326846b14918c33ebfdd5604f642e
Fixes: #4252
Fixed a reference leak issue where the values of the parameter dictionary
used in a statement execution would remain referenced by the "compiled
cache", as a result of storing the key view used by Python 3 dictionary
keys(). Pull request courtesy Olivier Grisel.
Change-Id: Icfb0f38111a165780f6dd3e4e3382a03df79ce26
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/441
Fixed 1.2 regression caused by 🎫`4060` where the query used to
reflect SQL Server cross-schema foreign keys was limiting the criteria
incorrectly.
Additionally, added some rework of the inter-schema reflection tests
so that MySQL, MSSQL can be included, breaking out some of the
Postgresql-specific behaviors into separate requirements.
Fixes: #4234
Change-Id: I20c8e70707075f1767b79127c2c27d4b313c6515
Fixed bug in connection pool where a connection could be present in the
pool without all of its "connect" event handlers called, if a previous
"connect" handler threw an exception; note that the dialects themselves
have connect handlers that emit SQL, such as those which set transaction
isolation, which can fail if the database is in a non-available state, but
still allows a connection. The connection is now invalidated first if any
of the connect handlers fail.
Change-Id: I61d6f4827a98ab8455f1c3e1c55d046eeccec09a
Fixes: #4225
Fixed regression caused in 1.2.3 due to fix from 🎫`4181` where
the changes to the event system involving :class:`.Engine` and
:class:`.OptionEngine` did not accommodate for event removals, which
would raise an ``AttributeError`` when invoked at the class
level.
Change-Id: I1c9083829d74dd710716d28b0eaca4fa15e86313
Fixes: #4190
The :class:`.URL` object now allows query keys to be specified multiple
times where their values will be joined into a list. This is to support
the plugins feature documented at :class:`.CreateEnginePlugin` which
documents that "plugin" can be passed multiple times. Additionally, the
plugin names can be passed to :func:`.create_engine` outside of the URL
using the new :paramref:`.create_engine.plugins` parameter.
Change-Id: Ifc48ad120bd6c6204eda567492caf79832aeeaa5
Fixes: #4170
Fixed a fairly serious connection pool bug where a connection that is
acquired after being refreshed as a result of a user-defined
:class:`.DisconnectionError` or due to the 1.2-released "pre_ping" feature
would not be correctly reset if the connection were returned to the pool by
weakref cleanup (e.g. the front-facing object is garbage collected); the
weakref would still refer to the previously invalidated DBAPI connection
which would have the reset operation erroneously called upon it instead.
This would lead to stack traces in the logs and a connection being checked
into the pool without being reset, which can cause locking issues.
Change-Id: Iabd9f3a63a1d0207d0de0054a6ced3560818cf9c
Fixes: #4184
Fixed bug where events associated with an :class:`Engine`
at the class level would be doubled when the
:meth:`.Engine.execution_options` method were used. To
achieve this, the semi-private class :class:`.OptionEngine`
no longer accepts events directly at the class level
and will raise an error; the class only propagates class-level
events from its parent :class:`.Engine`. Instance-level
events continue to work as before.
The comments present another way of doing this where we would
copy events from the parent engine at option time rather
than linking the event listeners, but this would be a behavioral
change that adding new events to the parent engine would not
take effect for an already-created OptionEngine.
Change-Id: Id128516f54103fbad9a2210d6571eceb59c8b0cb
Fixes: #4181
Fixed bug in cx_Oracle disconnect detection, used by pre_ping and other
features, where an error could be raised as DatabaseError which includes a
numeric error code; previously we weren't checking in this case for a
disconnect code.
Change-Id: I359bb5ede25a4726ea632b81af83c6391f405ae1
Fixes: #4182
This test often fails on CI under load and reveals a small
race condition within SingleonThreadPool's "cleanup" system.
As this pool's "cleanup" system is pretty much never used in
any case since this pool is no longer used for SQLite file
databases, this race should have little impact, especially
since the occasional non-closed connection will get closed
out when it's GC'ed in any case.
Change-Id: I68aefc0f9bbfc708c95af0497844e65366fc8429
The "password" attribute of the :class:`.url.URL` object can now be
any user-defined or user-subclassed string object that responds to the
Python ``str()`` builtin. The object passed will be maintained as the
datamember :attr:`.url.URL.password_original` and will be consulted
when the :attr:`.url.URL.password` attribute is read to produce the
string value.
Change-Id: I91d101c3b10e135ae7e4de60a5104b51776db84f
Fixes: #4089
Mike Bayer