Fixed issue in MySQL dialect when connecting to a psuedo-MySQL database
such as that provided by ProxySQL, the up front check for isolation level
when it returns no row will not prevent the dialect from continuing to
connect. A warning is emitted that the isolation level could not be
detected.
Fixes: #5239
Change-Id: I4a240386a0d38bd90733819495ce50e37fe2234c
(cherry picked from commit 978755e851)
First (baby) step at replacing engine.execute
calls in test code with the new preferred way
of executing. MSSQL was targeted because it was
the easiest for me to test locally.
Change-Id: Id2e02f0e39007cbfd28ca6a535115f53c6407015
(cherry picked from commit 60f627cbd0)
Fixed issue where a "covering" index, e.g. those which have an INCLUDE
clause, would be reflected including all the columns in INCLUDE clause as
regular columns. A warning is now emitted if these additional columns are
detected indicating that they are currently ignored. Note that full
support for "covering" indexes is part of 🎫`4458`. Pull request
courtesy Marat Sharafutdinov.
Fixes: #5205Closes: #5206
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5206
Pull-request-sha: 512a3817bb
Change-Id: I3196a2bf77dc5a6abd85b2fbf0ebff1b30d4fb00
(cherry picked from commit ee9bd719b7)
Added support for reflection of "computed" columns, which are now returned
as part of the structure returned by :meth:`.Inspector.get_columns`.
When reflecting full :class:`.Table` objects, computed columns will
be represented using the :class:`.Computed` construct.
Also improve the documentation in :meth:`Inspector.get_columns`, correctly
listing all the returned keys.
Fixes: #5063Fixes: #4051Closes: #5064
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5064
Pull-request-sha: ba00fc321ce468f8885aad23b3dd33c789e50fbe
Change-Id: I789986554fc8ac7f084270474d0b2c12046b1cc2
(cherry picked from commit 62b7dace0c)
Fixed regression / incorrect fix caused by fix for 🎫`5146` where the
Oracle dialect reads from the "all_tab_comments" view to get table comments
but fails to accommodate for the current owner of the table being
requested, causing it to read the wrong comment if multiple tables of the
same name exist in multiple schemas.
Fixes: #5146
Change-Id: Id79fbaa81b0e36cd4af60c48e4ab35c593ace057
(cherry picked from commit f1429823d3)
Fixed a reflection bug where table comments could only be retrieved for
tables actually owned by the user but not for tables visible to the user
but owned by someone else. Pull request courtesy Dave Hirschfeld.
Fixes: #5146Closes: #5147
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5147
Pull-request-sha: 0651e3bed0
Change-Id: If970fda10d6adf04d926d38df1a567df1de9f7b9
(cherry picked from commit 0c294d356f)
Fixed issue in MySQL :meth:`.mysql.Insert.on_duplicate_key_update` construct
where using a SQL function or other composed expression for a column argument
would not properly render the ``VALUES`` keyword surrounding the column
itself.
Fixes: #5173
Change-Id: I16d39c2fdb8bbb7f3d1b2ffdd20e1bf69359ab75
(cherry picked from commit 57b2aae0d9)
Fixed bug where PostgreSQL reflection of CHECK constraints would fail to
parse the constraint if the SQL text contained newline characters. The
regular expression has been adjusted to accommodate for this case. Pull
request courtesy Eric Borczuk.
Fixes: #5170Closes: #5172
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5172
Pull-request-sha: 5701b7f09f
Change-Id: If727e9140b645e8b685c3476fb0fa4417c1e6526
(cherry picked from commit 0fe528483a)
Fixed issue where the "schema_translate_map" feature would not work with a
PostgreSQL native enumeration type (i.e. :class:`.Enum`,
:class:`.postgresql.ENUM`) in that while the "CREATE TYPE" statement would
be emitted with the correct schema, the schema would not be rendered in
the CREATE TABLE statement at the point at which the enumeration was
referenced.
Fixes: #5158
Change-Id: I41529785de2e736c70a142c2ae5705060bfed73e
(cherry picked from commit 89b8c343ed)
in 55f6d61e85 we still
forgot to accommodate for Python 3 timezone constructor
rejecting fractional minutes so the test further needs lambdas
to prevent the constructor from invoking for Python versions
less than 3.7
Change-Id: I02a83888f5ffbbe53ed0ce16c46a70942add4a3c
(cherry picked from commit a8b4ea8819)
CI didn't notice that the fractional seconds in the new SQL Server
DATETIMEOFFSET test are not available on Python 3.6. It was
inadvertently assumed this was a Python 2 incompatibility.
Change-Id: I9839eafbf7c37512eef1ecf666846983f9651c02
(cherry picked from commit 55f6d61e85)
Fixed issue where the :class:`.mssql.DATETIMEOFFSET` type would not
accommodate for the ``None`` value, introduced as part of the series of
fixes for this type first introduced in 🎫`4983`, 🎫`5045`.
Additionally, added support for passing a backend-specific date formatted
string through this type, as is typically allowed for date/time types on
most other DBAPIs.
Fixes: #5132
Change-Id: Iab05d67382e0f550474d50e0c3c1c888521b678a
(cherry picked from commit 532ddb0fa7)
- Added pyproject.toml with black arguments
- Updated black version in precommit hook
- Reformatted the code
Fixes: #5100Closes: #5103
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5103
Pull-request-sha: 795fd5f896
Change-Id: I14eedbaa51fb531cbf90fcefe6a1e07c8a565625
(cherry picked from commit f49a3fc02c03e39b3d34da9dfba4fe46d71c4aff)
Fixed issue where the PostgreSQL dialect would fail to parse a reflected
CHECK constraint that was a boolean-valued function (as opposed to a
boolean-valued expression).
Fixes: #5039Closes: #5044
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5044
Pull-request-sha: b6903c6564
Change-Id: I7d39b104a8ce346cb593d541c1b4e5eab88867f9
(cherry picked from commit d8ac1e9e6b)
Fixed issue where a timezone-aware ``datetime`` value being converted to
string for use as a parameter value of a :class:`.mssql.DATETIMEOFFSET`
column was omitting the fractional seconds.
Fixes: #5045Closes: #5046
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5046
Pull-request-sha: 99dc7b23f6
Change-Id: I971629466fe0675536bbdf55693f0c1821dfb3cc
(cherry picked from commit 0d316e42f0)
Fixed a few test failures which would occur on Windows due to SQLite file
locking issues, as well as some timing issues in connection pool related
tests; pull request courtesy Federico Caselli.
Note the pool related issues were fixed by Mike in
I1a7162e67912d22c135fa517b687a073f8fd9151 but are being ticketed
here.
Fixes: #4946Closes: #5055
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5055
Pull-request-sha: 36925573af
Change-Id: Ic53ec82f5d588d0e26a2d033a17c6109900d7f63
(cherry picked from commit f461d710f9f848029603cf76f7c26725afdea11e)
Repaired support for the :class:`.mssql.DATETIMEOFFSET` datatype on PyODBC,
by adding PyODBC-level result handlers as it does not include native
support for this datatype. This includes usage of the Python 3 "timezone"
tzinfo subclass in order to set up a timezone, which on Python 2 makes
use of a minimal backport of "timezone" in sqlalchemy.util.
Fixes: #4983Closes: #4986
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4986
Pull-request-sha: e3a5a8dda0
Change-Id: Ia4c7c0d1971c6c0492515bd5fa0b1799f07fee1c
(cherry picked from commit f521577f6e)
Added "Connection was killed" message interpreted from the base
pymysql.Error class in order to detect closed connection, based on reports
that this message is arriving via a pymysql.InternalError() object which
indicates pymysql is not handling it correctly.
Change-Id: If6bbe0eb5993e1996c0c5de752eebaf7446cf93e
References: https://github.com/PyMySQL/PyMySQL/issues/816Fixes: #4945
(cherry picked from commit 31a66b5add)
Added new accessors to expressions of type :class:`.JSON` to allow for
specific datatype access and comparison, covering strings, integers,
numeric, boolean elements. This revises the documented approach of
CASTing to string when comparing values, instead adding specific
functionality into the PostgreSQL, SQlite, MySQL dialects to reliably
deliver these basic types in all cases.
The change also delivers a new feature to the test exclusions
system so that combinations and exclusions can be used together.
Fixes: #4276
Change-Id: Ica5a926c060feb40a0a7cd60b9d6e061d7825728
(cherry picked from commit 01cbf4d7b8)
Fixed issue in MSSQL dialect where an expression-based OFFSET value in a
SELECT would be rejected, even though the dialect can render this
expression inside of a ROW NUMBER-oriented LIMIT/OFFSET construct.
Fixes: #4973
Change-Id: I040d34f781791c4ed5a727e1b8fb98c68ddd0622
(cherry picked from commit b3c3562ecf)
Added DDL support for "computed columns"; these are DDL column
specifications for columns that have a server-computed value, either upon
SELECT (known as "virtual") or at the point of which they are INSERTed or
UPDATEd (known as "stored"). Support is established for Postgresql, MySQL,
Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work
on this one.
ORM round trip tests included. The ORM makes use of existing
FetchedValue support and no additional ORM logic is present for
the basic feature.
It has been observed that Oracle RETURNING does not return the
new value of a computed column upon UPDATE; it returns the
prior value. As this is very dangerous, a warning is emitted
if a computed column is rendered into the RETURNING clause
of an UPDATE statement.
Fixes: #4894Closes: #4928
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4928
Pull-request-sha: d39c521d5a
Change-Id: I2610b2999a5b1b127ed927dcdaeee98b769643ce
(cherry picked from commit 602d1e6dfd538980bb8d513867b17dbc2b4b92dd)
Fixed bug in :class:`.Inspector` where the cache key generation did not
take into account arguments passed in the form of tuples, such as the tuple
of view name styles to return for the PostgreSQL dialect. This would lead
the inspector to cache too generally for a more specific set of criteria.
The logic has been adjusted to include every keyword element in the cache,
as every argument is expected to be appropriate for a cache else the
caching decorator should be bypassed by the dialect.
Fixes: #4955
Change-Id: Icd97744323407977a3f7db26e8a63a1a5c576010
(cherry picked from commit 9fc54801c8)
Dialect tests tend to have a lot of lists of types,
SQL constructs etc, convert as many of these to @combinations
as possible.
This is exposing that we don't have per-combination
exclusion rules set up which is making things a little bit
cumbersome.
Also set up a fixture that does metadata + DDL.
Change-Id: Ief820e48c9202982b0b1e181b87862490cd7b0c3
(cherry picked from commit 240d9a60cc)
Added dialect-level flag ``encoding_errors`` to the cx_Oracle dialect,
which can be specified as part of :func:`.create_engine`. This is passed
to SQLAlchemy's unicode decoding converter under Python 2, and to
cx_Oracle's ``cursor.var()`` object as the ``encodingErrors`` parameter
under Python 3, for the very unusual case that broken encodings are present
in the target database which cannot be fetched unless error handling is
relaxed. The value is ultimately one of the Python "encoding errors"
parameters passed to ``decode()``.
Closes: #4801Fixes: #4799
Change-Id: I1d542ba367bcd187347c54db1fee815f7890e71c
(cherry picked from commit d76cb72135)
Fixed an issue in the :meth:`.Engine.table_names` method where it would
feed the dialect's default schema name back into the dialect level table
function, which in the case of SQL Server would interpret it as a
dot-tokenized schema name as viewed by the mssql dialect, which would
cause the method to fail in the case where the database username actually
had a dot inside of it. In 1.3, this method is still used by the
:meth:`.MetaData.reflect` function so is a prominent codepath. In 1.4,
which is the current master development branch, this issue doesn't exist,
both because :meth:`.MetaData.reflect` isn't using this method nor does the
method pass the default schema name explicitly. The fix nonetheless
guards against the default server name value returned by the dialect from
being interpreted as dot-tokenized name under any circumstances by
wrapping it in quoted_name().
Fixes: #4923
Change-Id: I821bd38ed89b767eaca0bdffee7f8ba3baf82560
(cherry picked from commit f50c6a04067acf2cd2fc5e42d5acaa9206d9a078)
The :class:`.sqltypes.NCHAR` datatype will now bind to the
``cx_Oracle.FIXED_NCHAR`` DBAPI data bindings when used in a bound
parameter, which supplies proper comparison behavior against a
variable-length string. Previously, the :class:`.sqltypes.NCHAR` datatype
would bind to ``cx_oracle.NCHAR`` which is not fixed length; the
:class:`.sqltypes.CHAR` datatype already binds to ``cx_Oracle.FIXED_CHAR``
so it is now consistent that :class:`.sqltypes.NCHAR` binds to
``cx_Oracle.FIXED_NCHAR``.
Fixes: #4913
Change-Id: I5bb111f2e06bbdd525bc5f716579baad31bbb3db
(cherry picked from commit e4a5da1c5a)
Fixed regression in Oracle dialect that was inadvertently using max
identifier length of 128 characters on Oracle server 12.2 and greater even
though the stated contract for the remainder of the 1.3 series is that
this value stays at 30 until version SQLAlchemy 1.4. Also repaired issues
with the retrieval of the "compatibility" version, and removed the warning
emitted when the "v$parameter" view was not accessible as this was causing
user confusion.
Fixes: #4898
Change-Id: Ieb7b3e093610896c5aa12d0789b63262e0ecf9d8
Added identifier quoting to the schema name applied to the "use" statement
which is invoked when a SQL Server multipart schema name is used within a
:class:`.Table` that is being reflected, as well as for :class:`.Inspector`
methods such as :meth:`.Inspector.get_table_names`; this accommodates for
special characters or spaces in the database name. Additionally, the "use"
statement is not emitted if the current database matches the target owner
database name being passed.
Fixes: #4883
Change-Id: I84419730e94aac3a88d331ad8c24d10aabbc34af
(cherry picked from commit 66a7befa0c)
Added new :func:`.create_engine` parameter
:paramref:`.create_engine.max_identifier_length`. This overrides the
dialect-coded "max identifier length" in order to accommodate for databases
that have recently changed this length and the SQLAlchemy dialect has
not yet been adjusted to detect for that version. This parameter interacts
with the existing :paramref:`.create_engine.label_length` parameter in that
it establishes the maximum (and default) value for anonymously generated
labels.
The Oracle dialect now emits a warning if Oracle version 12.2 or greater is
used, and the :paramref:`.create_engine.max_identifier_length` parameter is
not set. The version in this specific case defaults to that of the
"compatibility" version set in the Oracle server configuration, not the
actual server version. In version 1.4, the default max_identifier_length
for 12.2 or greater will move to 128 characters. In order to maintain
forwards compatibility, applications should set
:paramref:`.create_engine.max_identifier_length` to 30 in order to maintain
the same length behavior, or to 128 in order to test the upcoming behavior.
This length determines among other things how generated constraint names
are truncated for statements like ``CREATE CONSTRAINT`` and ``DROP
CONSTRAINT``, which means a the new length may produce a name-mismatch
against a name that was generated with the old length, impacting database
migrations.
Fixes: #4857
Change-Id: Ib62efb00c6180c375869029b57353d90385d7950
(cherry picked from commit 88761b8b0b0cfa67cdd6a4913e3a0ea5cba93cbb)
Added support for sqlite "URI" connections, which allow for sqlite-specific
flags to be passed in the query string such as "read only" for Python
sqlite3 drivers that support this.
Fixes: #4863
Change-Id: I7740b55ee8f2ede72a5c49ee94a7540e4d0250f2
(cherry picked from commit adb495503d)
Added support for reflection of CHECK constraints that include the special
PostgreSQL qualifier "NOT VALID", which can be present for CHECK
constraints that were added to an exsiting table with the directive that
they not be applied to existing data in the table. The PostgreSQL
dictionary for CHECK constraints as returned by
:meth:`.Inspector.get_check_constraints` may include an additional entry
``dialect_options`` which within will contain an entry ``"not_valid":
True`` if this symbol is detected. Pull request courtesy Bill Finn.
Fixes: #4824Closes: #4825
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4825
Pull-request-sha: a0e1ab133c
Change-Id: I78365f50055c95474c92124b85df66c5c80c00c8
(cherry picked from commit 3980a9a455)
Fixed bug where Postgresql operators such as
:meth:`.postgresql.ARRAY.Comparator.contains` and
:meth:`.postgresql.ARRAY.Comparator.contained_by` would fail to function
correctly for non-integer values when used against a
:class:`.postgresql.array` object, due to an erroneous assert statement.
Fixes: #4822
Change-Id: I886aca4c86dc7d64e9d0dfc1d910a0ae64d775a1
(cherry picked from commit 4b5d162ed0)
Fixed bug where a FOREIGN KEY that was set up to refer to the parent table
by table name only without the column names would not correctly be
reflected as far as setting up the "referred columns", since SQLite's
PRAGMA does not report on these columns if they weren't given explicitly.
For some reason this was harcoded to assume the name of the local column,
which might work for some cases but is not correct. The new approach
reflects the primary key of the referred table and uses the constraint
columns list as the referred columns list, if the remote column(s) aren't
present in the reflected pragma directly.
Fixes: #4810
Change-Id: I7789f83d68845ae197a782080af8ec64a7bf48cc
(cherry picked from commit f06c6ba673)
Revised the approach for the just added support for the psycopg2
"execute_values()" feature added in 1.3.7 for 🎫`4623`. The approach
relied upon a regular expression that would fail to match for a more
complex INSERT statement such as one which had subqueries involved. The
new approach matches exactly the string that was rendered as the VALUES
clause.
Fixes: #4623
Change-Id: Icaae0f7b6bcf87a2cf5c6290a839c8429dd5fac3
(cherry picked from commit 0a7ca00e04)
Added new dialect flag for the psycopg2 dialect, ``executemany_mode`` which
supersedes the previous experimental ``use_batch_mode`` flag.
``executemany_mode`` supports both the "execute batch" and "execute values"
functions provided by psycopg2, the latter which is used for compiled
:func:`.insert` constructs. Pull request courtesy Yuval Dinari.
.. seealso::
:ref:`executemany_mode`
Fixes: #4623Closes: #4764
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4764
Pull-request-sha: c3d3a36f7e
Change-Id: I77e26ca729f9317af1488a6c054c23fa1a6b622b
(cherry picked from commit 65f8edd458)
The dialects that support json are supposed to take arguments
``json_serializer`` and ``json_deserializer`` at the create_engine() level,
however the SQLite dialect calls them ``_json_serilizer`` and
``_json_deserilalizer``. The names have been corrected, the old names are
accepted with a change warning, and these parameters are now documented as
:paramref:`.create_engine.json_serializer` and
:paramref:`.create_engine.json_deserializer`.
Fixes: #4798
Change-Id: I1dbfe439b421fe9bb7ff3594ef455af8156f8851
(cherry picked from commit 104e690728)
Fixed bug where usage of "PRAGMA table_info" in SQLite dialect meant that
reflection features to detect for table existence, list of table columns,
and list of foreign keys, would default to any table in any attached
database, when no schema name was given and the table did not exist in the
base schema. The fix explicitly runs PRAGMA for the 'main' schema and then
the 'temp' schema if the 'main' returned no rows, to maintain the behavior
of tables + temp tables in the "no schema" namespace, attached tables only
in the "schema" namespace.
Fixes: #4793
Change-Id: I75bc03ef42581c46b98987510d2d2e701df07412
(cherry picked from commit e091775f1c)
Added another fix for an upstream MySQL 8 issue where a case sensitive
table name is reported incorrectly in foreign key constraint reflection,
this is an extension of the fix first added for 🎫`4344` which
affects a case sensitive column name. The new issue occurs through MySQL
8.0.17, so the general logic of the 88718 fix remains in place.
.. seealso::
https://bugs.mysql.com/bug.php?id=96365 - upstream bug
Fixes: #4751
Change-Id: I391903565db919b85b6b3c62c28f4b90ee596135
(cherry picked from commit 9a6654e3af)
Added support for reflection of indexes on PostgreSQL partitioned tables,
which was added to PostgreSQL as of version 11.
Fixes: #4771
Change-Id: I9e8e75c4d8a667b4d52d12afbd384e0a8db00466
(cherry picked from commit a739a3449f)
Added support for composite (tuple) IN operators with SQLite, by rendering
the VALUES keyword for this backend. As other backends such as DB2 are
known to use the same syntax, the syntax is enabled in the base compiler
using a dialect-level flag ``tuple_in_values``. The change also includes
support for "empty IN tuple" expressions for SQLite when using "in_()"
between a tuple value and an empty set.
Fixes: #4766
Change-Id: I416e1af29b31d78f9ae06ec3c3a48ef6d6e813f5
(cherry picked from commit 88168db8e9)
Fixed issue where the :class:`.array_agg` construct in combination with
:meth:`.FunctionElement.filter` would not produce the correct operator
precedence between the FILTER keyword and the array index operator.
Fixes: #4760
Change-Id: Ic662cd3da3330554ec673bafd80495b3f1506098
(cherry picked from commit 116faee662)
Added support for multidimensional Postgresql array literals via nesting
the :class:`.postgresql.array` object within another one. The
multidimensional array type is detected automatically.
Fixes: #4756
Change-Id: Ie2107ad3cf291112f6ca330dc90dc15a0a940cee
(cherry picked from commit 6bc7e0bfa9)
Enhanced MySQL/MariaDB version string parsing to accommodate for exotic
MariaDB version strings where the "MariaDB" word is embedded among other
alphanumeric characters such as "MariaDBV1". This detection is critical in
order to correctly accomodate for API features that have split between MySQL
and MariaDB such as the "transaction_isolation" system variable.
Fixes: #4624
Change-Id: Iba4b56535855629a974b1e24e012b07383d24199
(cherry picked from commit 95dd8768d4)
Ensured that the queries used to reflect indexes and view definitions will
explicitly CAST string parameters into NVARCHAR, as many SQL Server drivers
frequently treat string values, particularly those with non-ascii
characters or larger string values, as TEXT which often don't compare
correctly against VARCHAR characters in SQL Server's information schema
tables for some reason. These CAST operations already take place for
reflection queries against SQL Server ``information_schema.`` tables but
were missing from three additional queries that are against ``sys.``
tables.
Fixes: #4745
Change-Id: I3056533bf1a1e8ef17742879d369ab13f8b704ea
(cherry picked from commit 345f2eb05b)
Fixed bug where the special logic to render "NULL" for the
:class:`.TIMESTAMP` datatype when ``nullable=True`` would not work if the
column's datatype were a :class:`.TypeDecorator` or a :class:`.Variant`.
The logic now ensures that it unwraps down to the original
:class:`.TIMESTAMP` so that this special case NULL keyword is correctly
rendered when requested.
Fixes: #4743
Change-Id: I02b22dfa3db06daea37b044e2206a8569e2e5d22
(cherry picked from commit 401a2691fb)
Mike Bayer