Added support for PostgreSQL "readonly" and "deferrable" flags for all of
psycopg2, asyncpg and pg8000 dialects. This takes advantage of a newly
generalized version of the "isolation level" API to support other kinds of
session attributes set via execution options that are reliably reset
when connections are returned to the connection pool.
Fixes: #5549
Change-Id: I0ad6d7a095e49d331618274c40ce75c76afdc7dd
fixed an issue where even though the method claims to be
matching up columns positionally, it was failing on that by
looking in "keymap" based on string name.
Adds a new member to the _keymap recs MD_RESULT_MAP_INDEX
so that we can efficiently link from the generated keymap
back to the compiled._result_columns structure without
any ambiguity.
Fixes: #5559
Change-Id: Ie2fa9165c16625ef860ffac1190e00575e96761f
The mariadbconnector driver is having sporadic interpreter
crashes within this test suite. hopefully the crashes
are specific to these tests else we'll have to remove
mariadb from CI.
Change-Id: Idb7a9521e795d39957bce45415013eb4e771e560
"Implicit autocommit", which is the COMMIT that occurs when a DML or DDL
statement is emitted on a connection, is deprecated and won't be part of
SQLAlchemy 2.0. A 2.0-style warning is emitted when autocommit takes
effect, so that the calling code may be adjusted to use an explicit
transaction.
As part of this change, DDL methods such as
:meth:`_schema.MetaData.create_all` when used against a
:class:`_engine.Engine` or :class:`_engine.Connection` will run the
operation in a BEGIN block if one is not started already.
The MySQL and MariaDB dialects now query from the information_schema.tables
system view in order to determine if a particular table exists or not.
Previously, the "DESCRIBE" command was used with an exception catch to
detect non-existent, which would have the undesirable effect of emitting a
ROLLBACK on the connection. There appeared to be legacy encoding issues
which prevented the use of "SHOW TABLES", for this, but as MySQL support is
now at 5.0.2 or above due to 🎫`4189`, the information_schema tables
are now available in all cases.
Fixes: #4846
Change-Id: I733a7e0e17477a63607fb9931c87c393bbd7ac57
Two operations have been defined:
* :meth:`~.ColumnOperators.regexp_match` implementing a regular
expression match like function.
* :meth:`~.ColumnOperators.regexp_replace` implementing a regular
expression string replace function.
Fixes: #1390
Change-Id: I44556846e4668ccf329023613bd26861d5c674e6
MariaDB should not run a Sequence if it has optional=True.
Additionally, rework the rules in crud.py to accommodate the
new combination MariaDB brings us, which is a dialect
that supports both cursor.lastrowid, explicit sequences,
*and* no support for returning.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5528
Change-Id: I9a8ea69a34983affa95dfd22186e2908fdf0d58c
The _execute_20 and exec_driver_sql methods should wrap
up the parameters so that they represent the single list / single
dictionary style of invocation into the legacy methods. then
the before_ after_ execute event handlers should be receiving
the parameter dictionary as a single dictionary. this requires
that we break out distill_params to work differently if event
handlers are present.
additionally, add deprecation warnings for old argument passing
styles.
Change-Id: I97cb4d06adfcc6b889f10d01cc7775925cffb116
Added support for the :class:`_types.JSON` datatype on the SQL Server
dialect using the :class:`_mssql.JSON` implementation, which implements SQL
Server's JSON functionality against the ``NVARCHAR(max)`` datatype as per
SQL Server documentation. Implementation courtesy Gord Thompson.
Fixes: #4384
Change-Id: I28af79a4d8fafaa68ea032228609bba727784f18
Fixed issue where the :class:`_postgresql.ENUM` type would not consult the
schema translate map when emitting a CREATE TYPE or DROP TYPE during the
test to see if the type exists or not. Additionally, repaired an issue
where if the same enum were encountered multiple times in a single DDL
sequence, the "check" query would run repeatedly rather than relying upon a
cached value.
Fixes: #5520
Change-Id: I79f46e29ac0168e873ff178c242f8d78f6679aeb
Added the :class:`_schema.Identity` construct that can be used to
configure identity columns rendered with GENERATED { ALWAYS |
BY DEFAULT } AS IDENTITY. Currently the supported backends are
PostgreSQL >= 10, Oracle >= 12 and MSSQL (with different syntax
and a subset of functionalities).
Fixes: #5362Fixes: #5324Fixes: #5360
Change-Id: Iecea6f3ceb36821e8b96f0b61049b580507a1875
Allow specifying the data type when creating a :class:`.Sequence` in
PostgreSQL by using the parameter :paramref:`.Sequence.data_type`.
Fixes: #5498
Change-Id: I2b4a80aa89b1503c56748dc3ecd2cf145faddd8b
Adjusted the :meth:`_types.ARRAY.Comparator.any` and
:meth:`_types.ARRAY.Comparator.all` methods to implement a straight "NOT"
operation for negation, rather than negating the comparison operator.
Fixes: #5518
Change-Id: I87ee9278c321aafe51a679fcfcbb5fbb11307fda
The pg8000 dialect has been revised and modernized for the most recent
version of the pg8000 driver for PostgreSQL. Changes to the dialect
include:
* All data types are now sent as text rather than binary.
* Using adapters, custom types can be plugged in to pg8000.
* Previously, named prepared statements were used for all statements.
Now unnamed prepared statements are used by default, and named
prepared statements can be used explicitly by calling the
Connection.prepare() method, which returns a PreparedStatement
object.
Pull request courtesy Tony Locke.
Notes by Mike: to get this all working it was needed to break
up JSONIndexType into "str" and "int" subtypes; this will be
needed for any dialect that is dependent on setinputsizes().
also includes @caselit's idea to include query params
in the dbdriver parameter.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Closes: #5451
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5451
Pull-request-sha: 639751ca9c
Change-Id: I2869bc52c330916773a41d11d12c297aecc8fcd8
MySQL dialect's server_version_info tuple is now all numeric. String
tokens like "MariaDB" are no longer present so that numeric comparison
works in all cases. The .is_mariadb flag on the dialect should be
consulted for whether or not mariadb was detected. Additionally removed
structures meant to support extremely old MySQL versions 3.x and 4.x;
the minimum MySQL version supported is now version 5.0.2.
In addition, as the "MariaDB" name goes away from server version,
expand upon the change in I330815ebe572b6a9818377da56621397335fa702
to support the name "mariadb" throughout the dialect and test suite
when mariadb-only mode is used. This changes the "name" field
on the MariaDB dialect to "mariadb", which then implies a change
throughout the testing requirements system as well as all the
dialect-specific DDL argument names such as "mysql_engine" is
now specified as "mariadb_engine", etc. Make use of the
recent additions to test suite URL provisioning so that we can
force MariaDB databases to have a "mariadb-only" dialect which
allows us to test this name change fully.
Update documentation to refer to MySQL / MariaDB explicitly
as well as indicating the "mariadb_" prefix used for options.
It seems likely that MySQL and MariaDB version numbers are going to
start colliding at some point so having the "mariadb" name
be available as a totally separate dialect name should give us
some options in this regard.
Currently also includes a date related fix to a test for
the postgresql dialect that was implicitly assuming a
non-UTC timezone
Fixes: #4189
Change-Id: I00e76d00f62971e1f067bd61915fa6cc1cf64e5e
Using the approach introduced at
https://gist.github.com/zzzeek/6287e28054d3baddc07fa21a7227904e
We can now create asyncio endpoints that are then handled
in "implicit IO" form within the majority of the Core internals.
Then coroutines are re-exposed at the point at which we call
into asyncpg methods.
Patch includes:
* asyncpg dialect
* asyncio package
* engine, result, ORM session classes
* new test fixtures, tests
* some work with pep-484 and a short plugin for the
pyannotate package, which seems to have so-so results
Change-Id: Idbcc0eff72c4cad572914acdd6f40ddb1aef1a7d
Fixes: #3414
The ``importlib_metadata`` library is used to scan for setuptools
entrypoints rather than pkg_resources. as importlib_metadata is a small
library that is included as of Python 3.8, the compatibility library is
installed as a dependency for Python versions older than 3.8.
Unfortunately setuptools "attr:" is broken because it tries to import
the module; seems like this is fixed as part of
https://github.com/pypa/setuptools/pull/1753 however this is too recent
to rely upon for now.
Added a new dialect token "mariadb" that may be used in place of "mysql" in
the :func:`_sa.create_engine` URL. This will deliver a MariaDB dialect
subclass of the MySQLDialect in use that forces the "is_mariadb" flag to
True. The dialect will raise an error if a server version string that does
not indicate MariaDB in use is received. This is useful for
MariaDB-specific testing scenarios as well as to support applications that
are hardcoding to MariaDB-only concepts. As MariaDB and MySQL featuresets
and usage patterns continue to diverge, this pattern may become more
prominent.
Fixes: #5400Fixes: #5496
Change-Id: I330815ebe572b6a9818377da56621397335fa702
MySQL 8.0.19 has some changes to how it reports on display widths
for int types (seems like it omits it in some cases), and also
no longer cares about the length for YEAR. Another adjustment
to the ordering of constraints reported in one case also.
At least one CI machine is at 8.0.21 now.
Change-Id: Ie2101bed3ad75dcbb62cd05abe95ef14ad895cf5
Added MariaDB code 1927 to the list of "disconnect" codes, as recent
MariaDB versions apparently use this code when the database server was
stopped.
Fixes: #5493
Change-Id: I63f1d692f36cb0411ead278556e3f8c64ab72ea4
Remove lookup logic that attempts to locate a dialect for a type,
just use StrSQLTypeCompiler.
Cleaned up the internal ``str()`` for datatypes so that all types produce a
string representation without any dialect present, including that it works
for third-party dialect types without that dialect being present. The
string representation defaults to being the UPPERCASE name of that type
with nothing else.
Fixes: #4262
Change-Id: I02149e8a1ba1e7336149e962939b07ae0df83c6b
Fixed issue where the return type for the various RANGE comparison
operators would itself be the same RANGE type rather than BOOLEAN, which
would cause an undesirable result in the case that a
:class:`.TypeDecorator` that defined result-processing behavior were in
use. Pull request courtesy Jim Bosch.
Fixes: #5476Closes: #5477
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5477
Pull-request-sha: 925b117e0c
Change-Id: I52ab4d4362d379c8253990f9d328a40990a64520
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles. This would make
migration simpler and reduce confusion.
However, confusion may be increased by the fact that select().join()
is different Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs. At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps. Session.execute() will still
behave the old way as we are adding a future flag.
This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.
The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.
Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.
Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.
Fixes: #5379Fixes: #5284
Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51
References: #5159
Renamed the :meth:`_schema.Table.tometadata` method to
:meth:`_schema.Table.to_metadata`. The previous name remains with a
deprecation warning.
Updated the "decorate" utility function to support decoration
of functions that include non-builtins as default values.
Moves test for deprecated "databases" package into
test/dialect/test_deprecations.py
Fixes: #5413Fixes: #5426
Change-Id: I6ed899871c935f9e46360127c17ccb7cf97cea6e
Build on #5401 to allow the ORM to take advanage
of executemany INSERT + RETURNING.
Implemented the feature
updated tests
to support INSERT DEFAULT VALUES, needed to come up with
a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT)
which can then be iterated out for executemany.
Added graceful degrade to plain executemany for PostgreSQL <= 8.2
Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN
Fix issue where unicode identifiers or parameter names wouldn't
work with execute_values() under Py2K, because we have to
encode the statement and therefore have to encode the
insert_single_values_expr too.
Correct issue from #5401 to support executemany + return_defaults
for a PK that is explicitly pre-generated, meaning we aren't actually
getting RETURNING but need to return it from compiled_parameters.
Fixes: #5263
Change-Id: Id68e5c158c4f9ebc33b61c06a448907921c2a657
The psycopg2 dialect now defaults to using the very performant
``execute_values()`` psycopg2 extension for compiled INSERT statements,
and also impements RETURNING support when this extension is used. This
allows INSERT statements that even include an autoincremented SERIAL
or IDENTITY value to run very fast while still being able to return the
newly generated primary key values. The ORM will then integrate this
new feature in a separate change.
Implements RETURNING for insert with executemany
Adds support to return_defaults() mode and inserted_primary_key
to support mutiple INSERTed rows, via return_defauls_rows
and inserted_primary_key_rows accessors.
within default execution context, new cached compiler
getters are used to fetch primary keys from rows
inserted_primary_key now returns a plain tuple. this
is not yet a row-like object however this can be
added.
Adds distinct "values_only" and "batch" modes, as
"values" has a lot of benefits but "batch" breaks
cursor.rowcount
psycopg2 minimum version 2.7 so we can remove the
large number of checks for very old versions of
psycopg2
simplify tests to no longer distinguish between
native and non-native json
Fixes: #5401
Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
Fixed bug in Oracle dialect where indexes that contain the full set of
primary key columns would be mistaken as the primary key index itself,
which is omitted, even if there were multiples. The check has been refined
to compare the name of the primary key constraint against the index name
itself, rather than trying to guess based on the columns present in the
index.
Fixes: #5421
Change-Id: I47c2ccdd0b13977cfd9ef249d4de06371c4fb241
Added "exists" to the list of reserved words for SQLite so that this word
will be quoted when used as a label or column name. Pull request courtesy
Thodoris Sotiropoulos.
Fixes: #5395Closes: #5396
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5396
Pull-request-sha: 5608317a38
Change-Id: Ia4769de2dec159dcf282eb4b30c11560da51a5c7
A variety of caching issues found by running
all tests with statement caching turned on.
The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.
Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.
Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *". if a table re-ordered its columns
or added/removed, now that data is obsolete.
Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.
Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement. Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.
Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.
mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.
lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly. While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.
turn on cache stats in logging.
Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context. This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.
DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced. this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.
memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.
Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason. Removed
these methods as well as the concept of a Compiler
holding onto a "bind".
Fixes: #5386
Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
The :paramref:`.Enum.create_constraint` and
:paramref:`.Boolean.create_constraint` parameters now default to False,
indicating when a so-called "non-native" version of these two datatypes is
created, a CHECK constraint will not be generated by default. These CHECK
constraints present schema-management maintenance complexities that should
be opted in to, rather than being turned on by default.
Fixes: #5367
Change-Id: I0a3fb608ce32143fa757546cc17ba2013e93272a
Refined the logic used by the SQL Server dialect to interpret multi-part
schema names that contain many dots, to not actually lose any dots if the
name does not have bracking or quoting used, and additionally to support a
"dbname" token that has many parts including that it may have multiple,
independently-bracketed sections.
This fix addresses #5364 to some degree but probably does not
resolve it fully.
References: #5364Fixes: #5366
Change-Id: I460cd74ce443efb35fb63b6864f00c6d81422688
Fixed an issue where the ``is_disconnect`` function in the SQL Server
pyodbc dialect was incorrectly reporting the disconnect state when the
exception messsage had a substring that matched a SQL Server ODBC error
code.
Fixes: #5359
Change-Id: I450c6818405a20f4daee20d58fce2d5ecb33e17f
Mike Bayer