Commit Graph

220 Commits

Author SHA1 Message Date
mike bayer 3179f70408 Merge "Add deprecation for base Executable.bind" 2020-10-17 14:11:48 +00:00
Mike Bayer 5162f2bc5f Add deprecation for base Executable.bind
These attributes will be removed in SQLAlchemy 2.0.

Also alters the deprecation message to qualify the
type of object correctly.  this in turn requires changes
in the warnings filter and deprecation tests.

Change-Id: I5779d9813e88f42e5db0c7b5e3ffff1d1535c203
2020-10-16 19:15:58 -04:00
Mike Bayer 87c24c498c Genericize setinputsizes and support pyodbc
Reworked the "setinputsizes()" set of dialect hooks to be correctly
extensible for any arbirary DBAPI, by allowing dialects individual hooks
that may invoke cursor.setinputsizes() in the appropriate style for that
DBAPI.   In particular this is intended to support pyodbc's style of usage
which is fundamentally different from that of cx_Oracle.  Added support
for pyodbc.

Fixes: #5649
Change-Id: I9f1794f8368bf3663a286932cfe3992dae244a10
2020-10-16 14:28:04 -04:00
Mike Bayer c3f102c9fe upgrade to black 20.8b1
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.

Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
2020-09-28 15:17:26 -04:00
Andrzej Bartosiński c4f28fb414 Stringify correctly for non-str exception argument
Fixed issue where a non-string object sent to
:class:`_exc.SQLAlchemyError` or a subclass, as occurs with some third
party dialects, would fail to stringify correctly. Pull request
courtesy Andrzej Bartosiński.

Fixes: #5599
Closes: #5600
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5600
Pull-request-sha: cdccccc42a

Change-Id: Icd710d9015abc80f61a84893d75fbb33ee0fe46e
2020-09-20 14:09:23 -04:00
Mike Bayer 7e864fc7b1 Create a framework to allow all SQLALCHEMY_WARN_20 to pass
As the test suite has widespread use of many patterns
that are deprecated, enable SQLALCHEMY_WARN_20 globally
for the test suite but then break the warnings filter
out into a whole list of all the individual warnings
we are looking for.  this way individual changesets
can target a specific class of warning, as many of these
warnings will indivdidually affect dozens of files
and potentially hundreds of lines of code.

Many warnings are also resolved here as this
patch started out that way.   From this point
forward there should be changesets that target a
subset of the warnings at a time.

For expediency, updates some migration 2.0 docs
for ORM as well.

Change-Id: I98b8defdf7c37b818b3824d02f7668e3f5f31c94
2020-09-16 12:31:05 -04:00
Mike Bayer 25b8e89d3d Deprecate engine-wise ss cursors; repair mariadbconnector
The server_side_cursors engine-wide feature relies upon
regexp parsing of statements a well as general guessing as
to when the feature should be used.   This is not within the
2.0 way of doing things and should be removed.

Additionally, mariadbconnector defaults to unbuffered cursors;
add new cursor hooks so that mariadbconnector can specify
buffered or unbuffered cursors without too much difficulty.
This will also correctly default mariadbconnector to buffered
cursors which should repair the segfaults we've been getting.

Try to restore the assert_raises that was removed in
5b6dfc0c38 to see if mariadbconnector segfaults
are resolved.

Change-Id: I77f1c972c742e40694972f578140bb0cac8c39eb
2020-09-13 13:06:02 -04:00
mike bayer dad484e7fa Merge "Do not specify type on mssql by default" 2020-09-12 16:26:39 +00:00
Federico Caselli 1d40a4ae15 Do not specify type on mssql by default
Make optional sequences render as identity in mssql
Remove unused dialect option sequence_default_column_type

Change-Id: I821eeffcb442f8d1b69186a9b798b15c3d8d6ff3
2020-09-12 10:00:23 -04:00
Federico Caselli e860060866 Update select usage to use the new 1.4 format
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function.  it does not
yet fully address other issues such as keyword arguments passed
to the table.select().

Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.

Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
2020-09-08 17:13:48 -04:00
Georg Richter 603f7d30f6 MariaDB dialect implementation
Fixes: #5459

Closes: #5515
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5515
Pull-request-sha: 760090b906

Change-Id: I30e8fbc02b7b5329ca228cd39f6fb7cfd0e43092
2020-08-31 18:19:35 -04:00
mike bayer 406034d41a Merge "internal test framework files for standardization of is_not/not_in;" 2020-08-30 14:33:08 +00:00
jonathan vanasco 672087176e internal test framework files for standardization of is_not/not_in;
this is safe for 1.3.x

Change-Id: Icba38fdc20f5d8ac407383a4278ccb346e09af38
2020-08-29 12:05:58 -04:00
mike bayer 91100022ec Merge "make URL immutable" 2020-08-26 02:44:17 +00:00
Mike Bayer 3668b3a30c make URL immutable
it's not really correct that URL is mutable and doesn't do
any argument checking.   propose replacing it with an immutable
named tuple with rich copy-and-mutate methods.

At the moment this makes a hard change to the CreateEnginePlugin
docs that previously recommended url.query.pop().  I can't find
any plugins on github other than my own that are using this
feature, so see if we can just make a hard change on this one.

Fixes: #5526
Change-Id: I28a0a471d80792fa8c28f4fa573d6352966a4a79
2020-08-25 20:10:16 -04:00
Federico Caselli 9ab4da7018 Updates for MariaDB sequences
MariaDB should not run a Sequence if it has optional=True.
Additionally, rework the rules in crud.py to accommodate the
new combination MariaDB brings us, which is a dialect
that supports both cursor.lastrowid, explicit sequences,
*and* no support for returning.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5528
Change-Id: I9a8ea69a34983affa95dfd22186e2908fdf0d58c
2020-08-22 12:46:12 -04:00
Mike Bayer a1939719a6 normalize execute style for events, 2.0
The _execute_20 and exec_driver_sql methods should wrap
up the parameters so that they represent the single list / single
dictionary style of invocation into the legacy methods.  then
the before_ after_ execute event handlers should be receiving
the parameter dictionary as a single dictionary.   this requires
that we break out distill_params to work differently if event
handlers are present.

additionally, add deprecation warnings for old argument passing
styles.

Change-Id: I97cb4d06adfcc6b889f10d01cc7775925cffb116
2020-08-20 10:14:21 -04:00
Tony Locke 06f1929b86 Update dialect for pg8000 version 1.16.0
The pg8000 dialect has been revised and modernized for the most recent
version of the pg8000 driver for PostgreSQL.  Changes to the dialect
include:

* All data types are now sent as text rather than binary.

* Using adapters, custom types can be plugged in to pg8000.

* Previously, named prepared statements were used for all statements.
  Now unnamed prepared statements are used by default, and named
  prepared statements can be used explicitly by calling the
  Connection.prepare() method, which returns a PreparedStatement
  object.

Pull request courtesy Tony Locke.

Notes by Mike: to get this all working it was needed to break
up JSONIndexType into "str" and "int" subtypes; this will be
needed for any dialect that is dependent on setinputsizes().

also includes @caselit's idea to include query params
in the dbdriver parameter.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>

Closes: #5451
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5451
Pull-request-sha: 639751ca9c

Change-Id: I2869bc52c330916773a41d11d12c297aecc8fcd8
2020-08-18 11:12:16 -04:00
Mike Bayer 302e8dee82 Don't link on_connect to first_connect event handler
Adjusted the dialect initialization process such that the
:meth:`_engine.Dialect.on_connect` is not called a second time on the first
connection.   The hook is called first, then the
:meth:`_engine.Dialect.initialize` is called if that connection is the
first for that dialect, then no more events are called.   This eliminates
the two calls to the "on_connect" function which can produce very difficult
debugging situations.

Fixes: #5497
Change-Id: Icefc2e884e30ee7b4ac84b99dc54bf992a6085e3
2020-08-07 18:01:49 -04:00
Mike Bayer 91f376692d Add future=True to create_engine/Session; unify select()
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles.   This would make
migration simpler and reduce confusion.

However, confusion may be increased by the fact that select().join()
is different  Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs.  At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps.   Session.execute() will still
behave the old way as we are adding a future flag.

This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.

The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.

Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.

Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.

Fixes: #5379
Fixes: #5284
Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51
References: #5159
2020-07-08 11:05:11 -04:00
Mike Bayer b0cfa7379c Turn on caching everywhere, add logging
A variety of caching issues found by running
all tests with statement caching turned on.

The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.

Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.

Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *".   if a table re-ordered its columns
or added/removed, now that data is obsolete.

Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.

Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement.   Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.

Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.

mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.

lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly.   While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.

turn on cache stats in logging.

Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context.   This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.

DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced.  this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.

memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.

Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason.   Removed
these methods as well as the concept of a Compiler
holding onto a "bind".

Fixes: #5386

Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
2020-06-10 15:29:01 -04:00
Gord Thompson 668872fe01 Add support for "real" sequences in mssql
Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for
Microsoft SQL Server.  This removes the deprecated feature of using
:class:`.Sequence` objects to manipulate IDENTITY characteristics which
should now be performed using ``mssql_identity_start`` and
``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The
change includes a new parameter :paramref:`.Sequence.data_type` to
accommodate SQL Server's choice of datatype, which for that backend
includes INTEGER and BIGINT.   The default starting value for SQL Server's
version of :class:`.Sequence` has been set at 1; this default is now
emitted within the CREATE SEQUENCE DDL for all backends.

Fixes: #4235
Fixes: #4633
Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
2020-05-29 08:10:38 -06:00
Mike Bayer 77f1b7d236 callcount reductions and refinement for cached queries
This commit includes that we've removed the "_orm_query"
attribute from compile state as well as query context.
The attribute created reference cycles and also added
method call overhead.    As part of this change,
the interface for ORMExecuteState changes a bit, as well
as the interface for the horizontal sharding extension
which now deprecates the "query_chooser" callable
in favor of "execute_chooser", which receives the contextual
object.  This will also work more nicely when we implement
the new execution path for bulk updates and deletes.

Pre-merge execution options for statement, connection,
arguments all up front in Connection.  that way they
can be passed to the before_execute / after_execute events,
and the ExecutionContext doesn't have to merge as second
time.   Core execute is pretty close to 1.3 now.

baked wasn't using the new one()/first()/one_or_none() methods,
fixed that.

Convert non-buffered cursor strategy to be a stateless
singleton.  inline all the paths by which the strategy
gets chosen, oracle and SQL Server dialects make use of the
already-invoked post_exec() hook to establish the alternate
strategies, and this is actually much nicer than it was before.

Add caching to mapper instance processor for getters.

Identified a reference cycle per query that was showing
up as a lot of gc cleanup, fixed that.

After all that, performance not budging much.  Even
test_baked_query now runs with significantly fewer function
calls than 1.3, still 40% slower.

Basically something about the new patterns just makes
this slower and while I've walked a whole bunch of them
back, it hardly makes a dent.  that said, the performance
issues are relatively small, in the 20-40% time increase
range, and the new caching feature
does provide for regular ORM and Core queries that
are cached, and they are faster than non-cached.

Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
2020-05-28 14:38:56 -04:00
mike bayer ebceb618ef Merge "Add immutabledict C code" 2020-05-23 12:51:07 +00:00
Mike Bayer fcbd03e48a Add immutabledict C code
Start trying to convert fundamental objects to
C as we now rely on a fairly small core of things,
and 1.4 is having problems with complexity added being
slower than the performance gains we are trying to build in.

immutabledict here does seem to bench as twice as fast as the
Python one, see below.  However, it does not appear to be
used prominently enough to make any dent in the performance
tests.

at the very least it may provide us some more lift-and-copy
code for more C extensions.

import timeit

from sqlalchemy.util._collections import not_immutabledict, immutabledict

def run(dict_cls):
    for i in range(1000000):
        d1 = dict_cls({"x": 5, "y": 4})

        d2 = d1.union({"x": 17, "new key": "some other value"}, None)

        assert list(d2) == ["x", "y", "new key"]

print(
    timeit.timeit(
        "run(d)", "from __main__ import run, not_immutabledict as d", number=1
    )
)
print(
    timeit.timeit(
        "run(d)", "from __main__ import run, immutabledict as d", number=1
    )
)

output:

python: 1.8799766399897635
C code: 0.8880784640205093

Change-Id: I29e7104dc21dcc7cdf895bf274003af2e219bf6d
2020-05-23 00:05:13 -04:00
Mike Bayer 5b2abf4d26 Structural / performance refinements
* state connection schema_translate_map entirely in terms of
  execution options, support for per-execution options as well

* use slots for role impls, remove superclass of the roles
  themselves as this is not needed

* tighten loop in resolve, might become a C function

Change-Id: Ib98ac9b65022fbf976e49c6060e4c37573528c5f
2020-05-22 10:30:38 -04:00
Mike Bayer af0e59ff16 Disable "check unicode returns" under Python 3
Disabled the "unicode returns" check that runs on dialect startup when
running under Python 3, which for many years has occurred in order to test
the current DBAPI's behavior for whether or not it returns Python Unicode
or Py2K strings for the VARCHAR and NVARCHAR datatypes.  The check still
occurs by default under Python 2, however the mechanism to test the
behavior will be removed in SQLAlchemy 2.0 when Python 2 support is also
removed.

This logic was very effective when it was needed, however now that Python 3
is standard, all DBAPIs are expected to return Python 3 strings for
character datatypes.  In the unlikely case that a third party DBAPI does
not support this, the conversion logic within :class:`.String` is still
available and the third party dialect may specify this in its upfront
dialect flags by setting the dialect level flag ``returns_unicode_strings``
to one of :attr:`.String.RETURNS_CONDITIONAL` or
:attr:`.String.RETURNS_BYTES`, both of which will enable Unicode conversion
even under Python 3.

As part of this change, disabling testing of the doctest tutorials under
Python 2.

Fixes: #5315
Change-Id: I1260e894611409d3b7fe1a92bd90c52043bbcf19
2020-05-19 13:53:39 -04:00
Mike Bayer aded39f68c Propose Result as immediate replacement for ResultProxy
As progress is made on the _future.Result, including breaking
it out such that DBAPI behaviors are local to specific
implementations, it becomes apparent that the Result object
is a functional superset of ResultProxy and that basic
operations like fetchone(), fetchall(), and fetchmany()
behave pretty much exactly the same way on the new object.
Reorganize things so that ResultProxy is now referred to
as LegacyCursorResult, which subclasses CursorResult
that represents the DBAPI-cursor version of Result,
making use of a multiple inheritance pattern so that
the functionality of Result is also available in non-DBAPI
contexts, as will be necessary for some ORM
patterns.

Additionally propose the composition system for Result
that will form the basis for ORM-alternative result
systems such as horizontal sharding and dogpile cache.
As ORM results will soon be coming directly from
instances of Result, these extensions will instead
build their own ResultFetchStrategies that perform
the special steps to create composed or cached
result sets.

Also considering at the moment not emitting deprecation
warnings for fetchXYZ() methods; the immediate issue
is Keystone tests are calling upon it, but as the
implementations here are proving to be not in any
kind of conflict with how Result works, there's
not too much issue leaving them around and deprecating
at some later point.

References: #5087
References: #4395
Fixes: #4959
Change-Id: I8091919d45421e3f53029b8660427f844fee0228
2020-05-01 16:09:24 -04:00
Mike Bayer 2f617f56f2 Create initial 2.0 engine implementation
Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which
is used for forwards compatibility with SQLAlchemy 2.   This engine
features always-transactional behavior with autobegin.

Allow execution options per statement execution.  This includes
that the before_execute() and after_execute() events now accept
an additional dictionary with these options, empty if not
passed; a legacy event decorator is added for backwards compatibility
which now also emits a deprecation warning.

Add some basic tests for execution, transactions, and
the new result object.   Build out on a new testing fixture
that swaps in the future engine completely to start with.

Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a
Fixes: #4644
2020-04-16 13:35:55 -04:00
Federico Caselli a9b068ae56 Remove code deprecated before version 1.1
- Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and
  :class:`.Inspector` classes.
- Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`.
- Remove support for deprecated engine URLs of the form ``postgres://``.
- Remove deprecated dialect ``mysql+gaerdbms``.
- Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM`
  and :class:`.mysql.SET` in the ``mysql`` dialect.
- Remove deprecated function ``comparable_property``. and function
  ``comparable_using`` in the declarative extension.
- Remove deprecated function ``compile_mappers``.
- Remove deprecated method ``collection.linker``.
- Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``.
  This change also removes the class ``StrongInstanceDict``.
- Remove deprecated parameter ``mapper.order_by``.
- Remove deprecated parameter ``Session._enable_transaction_accounting`.
- Remove deprecated parameter ``Session.is_modified.passive``.
- Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`.
- Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and
  ``ClauseElement.__or__`` and attribute ``Over.func``.
- Remove deprecated ``FromClause.count`` method.
- Remove deprecated parameter ``Table.useexisting``.
- Remove deprecated parameters ``text.bindparams`` and ``text.typemap``.

- Remove boolean support for the ``passive`` parameter in ``get_history``.
- Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``.

Fixes: #4643
Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
2020-04-09 00:33:22 +02:00
Mike Bayer cadfc608d6 Convert schema_translate to a post compile
Revised the :paramref:`.Connection.execution_options.schema_translate_map`
feature such that the processing of the SQL statement to receive a specific
schema name occurs within the execution phase of the statement, rather than
at the compile phase.   This is to support the statement being efficiently
cached.   Previously, the current schema being rendered into the statement
for a particular run would be considered as part of the cache key itself,
meaning that for a run against hundreds of schemas, there would be hundreds
of cache keys, rendering the cache much less performant.  The new behavior
is that the rendering is done in a similar  manner as the "post compile"
rendering added in 1.4 as part of 🎫`4645`, 🎫`4808`.

Fixes: #5004
Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
2020-03-24 14:25:41 -04:00
Federico Caselli 9ec7588220 Deprecate plain string in execute and introduce exec_driver_sql
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.

Fixes: #4848
Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
2020-03-21 17:03:45 -04:00
Gord Thompson 79a53645ba Fix tests failing for SQLite file databases; repair provisioning
1. ensure provision.py loads dialect implementations when running
reap_dbs.py.   Reapers haven't been working since
598f2f7e55 .

2. add some exclusion rules to allow the sqlite_file target to work;
add to tox.

3. add reap dbs target for SQLite, repair SQLite drop_db routine
which also wasn't doing the right thing for memory databases
etc.

4. Fix logging in provision files, as the main provision logger
is the one that's enabled by reap_dbs and maybe others, have all
the provision files use the provision logger.

Fixes: #5180
Fixes: #5168

Change-Id: Ibc1b0106394d20f5bcf847f37b09d185f26ac9b5
2020-03-13 15:04:33 -04:00
Mike Bayer 3c682d225d Run handle_error for any exceptions raised in execute_context()
Observing a SQLite connection/cursor being hung on
test_resultset -> PositionalTextTest -> test_dupe_col_obj.
this uses connectionless execution and the result object
fails to be constructed.  When that happens, there is no path
for the cursor or connection to be closed / released.
Recent changes with the exception assertions in #4849 seem to
be causing a cycle to last a little longer than usual which
is exposing this issue for one particular test on SQLite.

As we want to get rid of weakref cleanup, evaluate
why we dont have handle_dbapi_exception for this whole
block, as after_cursor_execute can raise, result construction
can raise, autocommit can raise, close can raise, there
does not seem to be a reason these things should be outside
of the block that gets cleaned up.

Fixes: #5182
Change-Id: I640ac55e8c5f39d287f779fbb5dc0ab727218ca3
2020-03-03 20:00:07 -05:00
mike bayer b5050beb73 Merge "Ensure all nested exception throws have a cause" 2020-03-02 23:45:35 +00:00
Mike Bayer 57dc36a01b Ensure all nested exception throws have a cause
Applied an explicit "cause" to most if not all internally raised exceptions
that are raised from within an internal exception catch, to avoid
misleading stacktraces that suggest an error within the handling of an
exception.  While it would be preferable to suppress the internally caught
exception in the way that the ``__suppress_context__`` attribute would,
there does not as yet seem to be a way to do this without suppressing an
enclosing user constructed context, so for now it exposes the internally
caught exception as the cause so that full information about the context
of the error is maintained.

Fixes: #4849
Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
2020-03-02 17:24:19 -05:00
Gord Thompson 49d78c3354 Remove obsolete util.compat.nested
Fixes: #5177
Change-Id: Ie02b0c065e3833f43e056ad9c31f414871d9e8ee
2020-03-02 10:37:20 -07:00
Mike Bayer 93b7767d00 Deprecate connection branching
The :meth:`.Connection.connect` method is deprecated as is the concept of
"connection branching", which copies a :class:`.Connection` into a new one
that has a no-op ".close()" method.  This pattern is oriented around the
"connectionless execution" concept which is also being removed in 2.0.

As part of this change we begin to move the internals away from
"connectionless execution" overall.   Remove the "connectionless
execution" concept from the reflection internals and replace with
explicit patterns at the Inspector level.

Fixes: #5131
Change-Id: Id23d28a9889212ac5ae7329b85136157815d3e6f
2020-02-21 16:26:57 -05:00
Federico Caselli 1de64504d8 Deprecate empty or_() and and_()
Creating an :func:`.and_` or :func:`.or_` construct with no arguments or
empty ``*args`` will now emit a deprecation warning, as the SQL produced is
a no-op (i.e. it renders as a blank string). This behavior is considered to
be non-intuitive, so for empty or possibly empty :func:`.and_` or
:func:`.or_` constructs, an appropriate default boolean should be included,
such as ``and_(True, *args)`` or ``or_(False, *args)``.   As has been the
case for many major versions of SQLAlchemy, these particular boolean
values will not render if the ``*args`` portion is non-empty.

As there are some internal cases where an empty and_() construct is used
in order to build an optional WHERE expression, a private
utility function is added to suit this use case.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5054
Closes: #5062
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062
Pull-request-sha: 5ca2f27281

Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
2020-01-25 18:03:48 -05:00
Mike Bayer 9e31fc7408 Remove jython code, remove all jython / pypy symbols
Removed all dialect code related to support for Jython and zxJDBC. Jython
has not been supported by SQLAlchemy for many years and it is not expected
that the current zxJDBC code is at all functional; for the moment it just
takes up space and adds confusion by showing up in documentation. At the
moment, it appears that Jython has achieved Python 2.7 support in its
releases but not Python 3.   If Jython were to be supported again, the form
it should take is against the Python 3 version of Jython, and the various
zxJDBC stubs for various backends should be implemented as a third party
dialect.

Additionally modernized logic that distinguishes between "cpython"
and "pypy" to instead look at platform.python_distribution() which
reliably tells us if we are cPython or not; all booleans which
previously checked for pypy and sometimes jython are now converted
to be "not cpython", this impacts the test suite for tests that are
cPython centric.

Fixes: #5094
Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
2020-01-17 17:44:57 -05:00
Gord Thompson 6f99bdf013 Add sequence support for MariaDB 10.3+.
Added support for use of the :class:`.Sequence` construct with MariaDB 10.3
and greater, as this is now supported by this database.  The construct
integrates with the :class:`.Table` object in the same way that it does for
other databases like PostrgreSQL and Oracle; if is present on the integer
primary key "autoincrement" column, it is used to generate defaults.   For
backwards compatibility, to support a :class:`.Table` that has a
:class:`.Sequence` on it to support sequence only databases like Oracle,
while still not having the sequence fire off for MariaDB, the optional=True
flag should be set, which indicates the sequence should only be used to
generate the primary key if the target database offers no other option.

Fixes: #4976
Closes: #4996
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4996
Pull-request-sha: cb2e1426ea

Change-Id: I507bc405eee6cae2c5991345d0eac53a37fe7512
2019-11-26 11:06:37 -05:00
Mike Bayer 4b321e8a5e Add hide_parameters flag to create_engine
Added new parameter :paramref:`.create_engine.hide_parameters` which when
set to True will cause SQL parameters to no longer be logged, nor rendered
in the string representation of a :class:`.StatementError` object.

Fixes: #4815
Change-Id: Ib87f868b6936cf6b42b192644e9d732ec24266c2
2019-08-22 14:01:05 -04:00
Mike Bayer 00b5c10846 Modernize internal reflection
- Deprecated remaining engine-level introspection and utility methods
including :meth:`.Engine.run_callable`, :meth:`.Engine.transaction`,
:meth:`.Engine.table_names`, :meth:`.Engine.has_table`.   The utility
methods are superseded by modern context-manager patterns, and the table
introspection tasks are suited by the :class:`.Inspector` object.

- The internal dialect method ``Dialect.reflecttable`` has been removed.  A
review of third party dialects has not found any making use of this method,
as it was already documented as one that should not be used by external
dialects.  Additionally, the private ``Engine._run_visitor`` method
is also removed.

- The long-deprecated ``Inspector.get_table_names.order_by`` parameter has
been removed.

- The :paramref:`.Table.autoload_with` parameter now accepts an :class:`.Inspector` object
directly, as well as any :class:`.Engine` or :class:`.Connection` as was the case before.

Fixes: #4755
Change-Id: Iec3a8b0f3e298ba87d532b16fac1e1132f464e21
2019-08-06 18:05:45 -04:00
Mike Bayer dfb20f07d8 Remove threadlocal engine strategy, engine strategies pool threadlocal
The "threadlocal" execution strategy, deprecated in 1.3, has been
removed for 1.4, as well as the concept of "engine strategies" and the
``Engine.contextual_connect`` method.  The "strategy='mock'" keyword
argument is still accepted for now with a deprecation warning; use
:func:`.create_mock_engine` instead for this use case.

Fixes: #4632
Change-Id: I8a351f9fa1f7dfa2a56eec1cd2d1a4b9d65765a2
(cherry picked from commit b368c49b44c5716d93c7428ab22b6761c6ca7cf5)
2019-07-15 22:48:47 -04:00
Mike Bayer 3238d953b4 Ensure .engine is part of Connectable interface, implement as descriptor
Fixed bug where using reflection function such as :meth:`.MetaData.reflect`
with an :class:`.Engine` object that had execution options applied to it
would fail, as the resulting :class:`.OptionEngine` proxy object failed to
include a ``.engine`` attribute used within the reflection routines.

Fixes: #4754
Change-Id: I6c342af5c6db6fe362b9d25f3f26d6859f62f87a
2019-07-08 10:42:20 -04:00
Mike Bayer 190e0139e8 Enable F841
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient.  test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.

Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
2019-06-20 13:50:41 -04:00
Matthew Wilkes f601791a91 Move initialize do_rollback() outside of the dialect
Moved the "rollback" which occurs during dialect initialization so that it
occurs after additional dialect-specific initialize steps, in particular
those of the psycopg2 dialect which would inadvertently leave transactional
state on the first new connection, which could interfere with some
psycopg2-specific APIs which require that no transaction is started.  Pull
request courtesy Matthew Wilkes.

Fixes: #4663
Closes: #4664
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4664
Pull-request-sha: e544fe671d

Change-Id: If40a15a1679b4eec0b8b8222f678697728009c30
2019-05-09 21:50:59 -04:00
Nate Clark 8f318692d4 Include newlines in StatementError formatting
Revised the formatting for :class:`.StatementError` when stringified. Each
error detail is broken up over multiple newlines instead of spaced out on a
single line.  Additionally, the SQL representation now stringifies the SQL
statement rather than using ``repr()``, so that newlines are rendered as is.
Pull request courtesy Nate Clark.

Fixes: #4500

Closes: #4501
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4501
Pull-request-sha: 60cc0ee68d

Change-Id: I79d8418b7495e5691c9a56f41e79495c26a967ff
2019-02-20 18:56:47 -05:00
Daniel Lister 2bbd3ac1fb Add getters for all execution_options
Added accessors for execution options to Core and ORM, via
:meth:`.Query.get_execution_options`,
:meth:`.Connection.get_execution_options`,
:meth:`.Engine.get_execution_options`, and
:meth:`.Executable.get_execution_options`.  PR courtesy Daniel Lister.

Fixes: #4406
Closes: #4465
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4465
Pull-request-sha: 9674688bb5

Change-Id: I93ba51d7a2d687e255edd6938db15615e56dd237
2019-01-25 11:19:03 -05:00
Mike Bayer 4c2c2c40fd Add deprecation warnings to all deprecated APIs
A large change throughout the library has ensured that all objects, parameters,
and behaviors which have been noted as deprecated or legacy now emit
``DeprecationWarning`` warnings when invoked.   As the Python 3 interpreter now
defaults to displaying deprecation warnings, as well as that modern test suites
based on tools like tox and pytest tend to display deprecation warnings,
this change should make it easier to note what API features are obsolete.

See the notes added to the changelog and migration notes for further
details.

Fixes: #4393
Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
2019-01-23 18:10:06 -05:00