Fixed a fairly serious connection pool bug where a connection that is
acquired after being refreshed as a result of a user-defined
:class:`.DisconnectionError` or due to the 1.2-released "pre_ping" feature
would not be correctly reset if the connection were returned to the pool by
weakref cleanup (e.g. the front-facing object is garbage collected); the
weakref would still refer to the previously invalidated DBAPI connection
which would have the reset operation erroneously called upon it instead.
This would lead to stack traces in the logs and a connection being checked
into the pool without being reset, which can cause locking issues.
Change-Id: Iabd9f3a63a1d0207d0de0054a6ced3560818cf9c
Fixes: #4184
(cherry picked from commit ab1f524c355c0bbac68485a60cb99e7a9d0f944a)
- add a connect=True key to connection record to support
pre-loading of _ConnectionRecord objects
- ensure _ConnectionRecord.close() leaves the record in a good
state for reopening
- add _ConnectionRecord.record_info for persistent storage
- add "in_use" accessor based on fairy_ref being present or not
- allow for the exclusions system and SuiteRequirements to be
usable without the full plugin_base setup.
- move some Python-env requirements to the importable
requirements.py module.
- allow starttime to be queried
- add additional events for engine plugins
- have "dialect" be a first-class parameter to the pool,
ensure the engine strategy supplies it up front
Change-Id: Ibf549f7a1766e49d335cd6f5e26bacfaef9a8229
may be called against a stale connection without the "connect"
event handler having been called, in the case where the pool
attempted to reconnect after being invalidated and failed; the stale
connection would remain present and would be used on a subsequent
attempt. This issue has a greater impact in the 1.0 series subsequent
to 1.0.2, as it also delivers a blanked-out ``.info`` dictionary to
the event handler; prior to 1.0.2 the ``.info`` dictionary is still
the previous one.
fixes#3497
and the database can no longer be connected towards, that the checkout
handler failure is caught, the attempt to re-acquire the connection
also raises an exception, but the underlying connection record
is not immediately re-checked in before the exception is propagated
outwards, having the effect that the checked-out record does not close
itself until the stack trace it's associated with is garbage collected,
preventing that record from being used for a new checkout until we
leave the scope of the stack trace. This can lead to confusion
in the specific case of when the number of current stack traces
in memory exceeds the number of connections the pool can return,
as the pool will instead begin to raise errors about no more checkouts
available, rather than attempting a connection again. The fix
applies a checkin of the record before re-raising.
fixes#3419
functionality. Added a new "soft invalidate" feature to the
connection pool at the level of the checked out connection wrapper
as well as the :class:`._ConnectionRecord`. This works similarly
to a modern pool invalidation in that connections aren't actively
closed, but are recycled only on next checkout; this is essentially
a per-connection version of that feature. A new event
:class:`.PoolEvents.soft_invalidate` is added to complement it.
fixes#3379
- Added new flag
:attr:`.ExceptionContext.invalidate_pool_on_disconnect`.
Allows an error handler within :meth:`.ConnectionEvents.handle_error`
to maintain a "disconnect" condition, but to handle calling invalidate
on individual connections in a specific manner within the event.
- Added new event :class:`.DialectEvents.do_connect`, which allows
interception / replacement of when the :meth:`.Dialect.connect`
hook is called to create a DBAPI connection. Also added
dialect plugin hooks :meth:`.Dialect.get_dialect_cls` and
:meth:`.Dialect.engine_created` which allow external plugins to
add events to existing dialects using entry points.
fixes#3355
1. make sure pool._invalidate() sets the timestamp up before
invalidating the target connection. we can otherwise show how the
conn.invalidate() + pool._invalidate() can lead to an extra connection
being made.
2. to help with that, soften up the check on connection.invalidate()
when connection is already closed. a warning is fine here
3. add a mutex to test_max_overflow() when we connect, because the way
we're using mock depends on an iterator, that needs to be synchronized
recycles the connection pool when a "disconnect" condition is detected;
instead of discarding the pool and explicitly closing out connections,
the pool is retained and a "generational" timestamp is updated to
reflect the current time, thereby causing all existing connections
to be recycled when they are next checked out. This greatly simplifies
the recycle process, removes the need for "waking up" connect attempts
waiting on the old pool and eliminates the race condition that many
immediately-discarded "pool" objects could be created during the
recycle operation. fixes#2985
concurrent ability to return connections from the pool means that the
"first_connect" event is now no longer synchronized either, thus leading
to dialect mis-configurations under even minimal concurrency situations.
:class:`.RootTransaction` or :class:`.TwoPhaseTransaction`
with its immediate :class:`._ConnectionFairy` as a "reset handler"
for the span of that transaction, which takes over the task
of calling commit() or rollback() for the "reset on return" behavior
of :class:`.Pool` if the transaction was not otherwise completed.
This resolves the issue that a picky transaction
like that of MySQL two-phase will be
properly closed out when the connection is closed without an
explicit rollback or commit (e.g. no longer raises "XAER_RMFAIL"
in this case - note this only shows up in logging as the exception
is not propagated within pool reset).
This issue would arise e.g. when using an orm
:class:`.Session` with ``twophase`` set, and then
:meth:`.Session.close` is called without an explicit rollback or
commit. The change also has the effect that you will now see
an explicit "ROLLBACK" in the logs when using a :class:`.Session`
object in non-autocommit mode regardless of how that session was
discarded. Thanks to Jeff Dairiki and Laurence Rowe for isolating
the issue here. [ticket:2907]
events including auto-invalidation, which is useful both for tests here as well as
detecting failure conditions within the "reset" or "close" cases.
- rename the argument for PoolEvents.reset() to dbapi_connection and connection_record
to be consistent with everything else.
- add new documentation sections on invalidation, including auto-invalidation
and the invalidation process within the pool.
- add _ConnectionFairy and _ConnectionRecord to the pool documentation. Establish
docs for common _ConnectionFairy/_ConnectionRecord methods and accessors and
have PoolEvents docs refer to _ConnectionRecord,
since it is passed to all events. Rename a few _ConnectionFairy methods that are actually
private to pool such as _checkout(), _checkin() and _checkout_existing(); there should not
be any external code calling these
attempts when an existing connection attempt is blocking. Previously,
the production of new connections was serialized within the block
that monitored overflow; the overflow counter is now altered within
it's own critical section outside of the connection process itself.
[ticket:2880]
connection to be available, such that for a connection pool
with no timeout specified, it will every half a second break out of
the wait to check for the so-called "abort" flag, which allows the
waiter to break out in case the whole connection pool was dumped;
normally the waiter should break out due to a notify_all() but it's
possible this notify_all() is missed in very slim cases.
This is an extension of logic first introduced in 0.8.0, and the
issue has only been observed occasionally in stress tests.
make sure that the DBAPI connection passed to the reset-on-return events/dialect hooks
is also a "fairy", so that dictionaries like "info" are available. [ticket:2770]
- rework the execution_options system so that the dialect is given the job of making
any immediate adjustments based on a set event. move the "isolation level" logic to use
this new system. Also work things out so that even engine-level execution options
can be used for things like isolation level; the dialect attaches a connect-event
handler in this case to handle the task.
- to support this new system as well as further extensibiltiy of execution options
add events engine_connect(), set_connection_execution_options(), set_engine_execution_options()
actual mock objects from the mock library. I'd like to use mock
for new tests so we might as well use it in obvious places.
- use unittest.mock in py3.3
- changelog
- add a note to README.unittests
- add tests_require in setup.py
- have tests import from sqlalchemy.testing.mock
- apply usage of mock to one of the event tests. we can be using
this approach all over the place.
whereas the other do_rollback_twophase(), savepoint etc. work with
:class:`.Connection`. the context on these are different as twophase/savepoint
are available at the :class:`.Connection` level, whereas commit/rollback are needed
at a lower level as well. Rename the argument to "dbapi_connection" when the conneciton
is in fact the DBAPI interface.
- start thinking about being able to track "autocommit" vs. "commit", but not sure
we have a need for this yet.
- have Pool call out to a Dialect for all rollback/commit/close operations now. Pool
no longer calls DBAPI methods directly. May use this for a workaround for [ticket:2611]
- add a new Pool event reset() to allow the pool's reset of the connection to be intercepted.
- remove methods in Informix dialect which appear to be hardcoding some isolation
settings on new Transaction only; the isolation API should be implemented for Informix.
also removed "flag" for transaction commit/rollback being not available; this should
be based on server/DBAPI version and we will need someone with test access in order
to help determine how this should work
become an externally usable package but still remains within the main sqlalchemy parent package.
in this system, we use kind of an ugly hack to get the noseplugin imported outside of the
"sqlalchemy" package, while still making it available within sqlalchemy for usage by
third party libraries.
a disconnect detect + dispose that occurs
when the QueuePool has threads waiting
for connections would leave those
threads waiting for the duration of
the timeout on the old pool. The fix
now notifies those waiters with a special
exception case and has them move onto
the new pool. This fix may or may
not be ported to 0.7. [ticket:2522]
Mike Bayer