Added support for reflection of indexes on PostgreSQL partitioned tables,
which was added to PostgreSQL as of version 11.
Fixes: #4771
Change-Id: I9e8e75c4d8a667b4d52d12afbd384e0a8db00466
(cherry picked from commit a739a3449f)
Added support for composite (tuple) IN operators with SQLite, by rendering
the VALUES keyword for this backend. As other backends such as DB2 are
known to use the same syntax, the syntax is enabled in the base compiler
using a dialect-level flag ``tuple_in_values``. The change also includes
support for "empty IN tuple" expressions for SQLite when using "in_()"
between a tuple value and an empty set.
Fixes: #4766
Change-Id: I416e1af29b31d78f9ae06ec3c3a48ef6d6e813f5
(cherry picked from commit 88168db8e9)
Fixed issue where the :class:`.array_agg` construct in combination with
:meth:`.FunctionElement.filter` would not produce the correct operator
precedence between the FILTER keyword and the array index operator.
Fixes: #4760
Change-Id: Ic662cd3da3330554ec673bafd80495b3f1506098
(cherry picked from commit 116faee662)
Added support for multidimensional Postgresql array literals via nesting
the :class:`.postgresql.array` object within another one. The
multidimensional array type is detected automatically.
Fixes: #4756
Change-Id: Ie2107ad3cf291112f6ca330dc90dc15a0a940cee
(cherry picked from commit 6bc7e0bfa9)
Enhanced MySQL/MariaDB version string parsing to accommodate for exotic
MariaDB version strings where the "MariaDB" word is embedded among other
alphanumeric characters such as "MariaDBV1". This detection is critical in
order to correctly accomodate for API features that have split between MySQL
and MariaDB such as the "transaction_isolation" system variable.
Fixes: #4624
Change-Id: Iba4b56535855629a974b1e24e012b07383d24199
(cherry picked from commit 95dd8768d4)
Ensured that the queries used to reflect indexes and view definitions will
explicitly CAST string parameters into NVARCHAR, as many SQL Server drivers
frequently treat string values, particularly those with non-ascii
characters or larger string values, as TEXT which often don't compare
correctly against VARCHAR characters in SQL Server's information schema
tables for some reason. These CAST operations already take place for
reflection queries against SQL Server ``information_schema.`` tables but
were missing from three additional queries that are against ``sys.``
tables.
Fixes: #4745
Change-Id: I3056533bf1a1e8ef17742879d369ab13f8b704ea
(cherry picked from commit 345f2eb05b)
Fixed bug where the special logic to render "NULL" for the
:class:`.TIMESTAMP` datatype when ``nullable=True`` would not work if the
column's datatype were a :class:`.TypeDecorator` or a :class:`.Variant`.
The logic now ensures that it unwraps down to the original
:class:`.TIMESTAMP` so that this special case NULL keyword is correctly
rendered when requested.
Fixes: #4743
Change-Id: I02b22dfa3db06daea37b044e2206a8569e2e5d22
(cherry picked from commit 401a2691fb)
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient. test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.
Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
(cherry picked from commit 190e0139e8)
Added support for column sorting flags when reflecting indexes for
PostgreSQL, including ASC, DESC, NULLSFIRST, NULLSLAST. Also adds this
facility to the reflection system in general which can be applied to other
dialects in future releases. Pull request courtesy Eli Collins.
Fixes: #4717Closes: #4725
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4725
Pull-request-sha: 3cbb067bd4
Change-Id: I8b0617d68580cfe4ff79d758a077263f33e852c2
(cherry picked from commit b4be7ceb86)
Fixed a series of quoting issues which all stemmed from the concept of the
:func:`.literal_column` construct, which when being "proxied" through a
subquery to be referred towards by a label that matches its text, the label
would not have quoting rules applied to it, even if the string in the
:class:`.Label` were set up as a :class:`.quoted_name` construct. Not
applying quoting to the text of the :class:`.Label` is a bug because this
text is strictly a SQL identifier name and not a SQL expression, and the
string should not have quotes embedded into it already unlike the
:func:`.literal_column` which it may be applied towards. The existing
behavior of a non-labeled :func:`.literal_column` being propagated as is on
the outside of a subquery is maintained in order to help with manual
quoting schemes, although it's not clear if valid SQL can be generated for
such a construct in any case.
Fixes: #4730
Change-Id: I300941f27872fc4298c74a1d1ed65aef1a5cdd82
(cherry picked from commit 009acc95b8)
Fixed bug where MySQL ON DUPLICATE KEY UPDATE would not accommodate setting
a column to the value NULL. Pull request courtesy Lukáš Banič.
Fixes: #4715Closes: #4716
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4716
Pull-request-sha: bfad6e6bdf
Change-Id: Ia8831cc171d131bf3824be8db4fd1d231231bba3
(cherry picked from commit 59e9276f04)
- don't call relatively expensive isEnabledFor(), just call _log_notices
- don't reset the list if it's empty
- fix the test to use a custom function to definitely create a notice, confirmed
that PG seems to no longer create the "implicit sequence" notices
- assert that the reset of the notices works too
- update the docs to illustrate for folks who haven't worked with logging before
Change-Id: I7291e647c177d338e0ad673f3106b4d503e4b3ea
(cherry picked from commit b0bf421f1b)
Fixed bug where PostgreSQL dialect could not correctly reflect an ENUM
datatype that has no members, returning a list with ``None`` for the
``get_enums()`` call and raising a TypeError when reflecting a column which
has such a datatype. The inspection now returns an empty list.
Fixes: #4701
Change-Id: I202bab19728862cbc64deae211d5ba6a103b8317
(cherry picked from commit 754e7f52cf)
Added support for DROP CHECK constraint which is required by MySQL 8.0.16
to drop a CHECK constraint; MariaDB supports plain DROP CONSTRAINT. The
logic distinguishes between the two syntaxes by checking the server version
string for MariaDB presence. Alembic migrations has already worked
around this issue by implementing its own DROP for MySQL / MariaDB CHECK
constraints, however this change implements it straight in Core so that its
available for general use. Pull request courtesy Hannes Hansen.
Fixes: #4650Closes: #4659
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4659
Pull-request-sha: 5b654a55e2
Change-Id: I967710f890722f11cf1f40406adbb17464d16194
(cherry picked from commit c55023641d)
Moved the "rollback" which occurs during dialect initialization so that it
occurs after additional dialect-specific initialize steps, in particular
those of the psycopg2 dialect which would inadvertently leave transactional
state on the first new connection, which could interfere with some
psycopg2-specific APIs which require that no transaction is started. Pull
request courtesy Matthew Wilkes.
Fixes: #4663Closes: #4664
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4664
Pull-request-sha: e544fe671d
Change-Id: If40a15a1679b4eec0b8b8222f678697728009c30
(cherry picked from commit f601791a91)
Added support for SQL Server filtered indexes, via the ``mssql_where``
parameter which works similarly to that of the ``postgresql_where`` index
function in the PostgreSQL dialect.
Fixes: #4657Closes: #4658
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4658
Pull-request-sha: cf609c19bc
Change-Id: I9c61b97d0b0cb6f6d417da7b1875b40f8f918a3c
(cherry picked from commit 174f2ae33b)
Fixed regression from release 1.3.2 caused by 🎫`4562` where a URL
that contained only a query string and no hostname, such as for the
purposes of specifying a service file with connection information, would no
longer be propagated to psycopg2 properly. The change in 🎫`4562`
has been adjusted to further suit psycopg2's exact requirements, which is
that if there are any connection parameters whatsoever, the "dsn" parameter
is no longer required, so in this case the query string parameters are
passed alone.
Fixes: #4601
Change-Id: Ic29a8b77bcf50ee996968bab25aaac7ae4bfc26f
Fixed issue in SQL Server dialect where if a bound parameter were present in
an ORDER BY expression that would ultimately not be rendered in the SQL
Server version of the statement, the parameters would still be part of the
execution parameters, leading to DBAPI-level errors. Pull request courtesy
Matt Lewellyn.
Fixes: #4587Closes: #4588
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4588
Pull-request-sha: 2992a473e0
Change-Id: Ie709aefdb1babf810bb81526289448f8cc7a4cb1
Added support for parameter-less connection URLs for the psycopg2 dialect,
meaning, the URL can be passed to :func:`.create_engine` as
``"postgresql+psycopg2://"`` with no additional arguments to indicate an
empty DSN passed to libpq, which indicates to connect to "localhost" with
no username, password, or database given. Pull request courtesy Julian
Mehnle.
Fixes: #4562Closes: #4563
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4563
Pull-request-sha: 8a05c96944
Change-Id: Ib6fca3c3c9eebeaf590d7f7fb0bc8cd4b6e4a55a
Modified the :paramref:`.Select.with_for_update.of` parameter so that if a
join or other composed selectable is passed, the individual :class:`.Table`
objects will be filtered from it, allowing one to pass a join() object to
the parameter, as occurs normally when using joined table inheritance with
the ORM. Pull request courtesy Raymond Lu.
Fixes: #4550
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Closes: #4551
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4551
Pull-request-sha: 452da77d15
Change-Id: If4b7c231f7b71190d7245543959fb5c3351125a1
A commit() is emitted after an isolation level change to SNAPSHOT, as both
pyodbc and pymssql open an implicit transaction which blocks subsequent SQL
from being emitted in the current transaction.
Fixes: #4536
Change-Id: If3ba70f495bce2a35a873a3a72d1b30406e678c8
Fixed regression in SQL Server reflection due to 🎫`4393` where the
removal of open-ended ``**kw`` from the :class:`.Float` datatype caused
reflection of this type to fail due to a "scale" argument being passed.
Fixes: #4525
Change-Id: Ief8bb535778055eff2ab0d71660f81e3676390a1
For very slow CI this test still fails so establish the date
as being between the range of what the operation took
Change-Id: I9204cd1915777ebc58bd598065b9015befb93345
Added support for reflection of the :class:`.NCHAR` datatype to the Oracle
dialect, and added :class:`.NCHAR` to the list of types exported by the
Oracle dialect.
Fixes: #4506
Change-Id: I3a120b9ad714cbd0ebd8064519f43cf57e99d920
Fixed bug where the SQL Server "IDENTITY_INSERT" logic that allows an INSERT
to proceed with an explicit value on an IDENTITY column was not detecting
the case where :meth:`.Insert.values` were used with a dictionary that
contained a :class:`.Column` as key and a SQL expression as a value.
Fixes: #4499
Change-Id: Ia61cd6524b030b40a665db9c20771f0c5aa5fcd7
Fixed a second regression caused by 🎫`4344` (the first was
🎫`4361`), which works around MySQL issue 88718, where the lower
casing function used was not correct for Python 2 with OSX/Windows casing
conventions, which would then raise ``TypeError``. Full coverage has been
added to this logic so that every codepath is exercised in a mock style for
all three casing conventions on all versions of Python. MySQL 8.0 has
meanwhile fixed issue 88718 so the workaround is only applies to a
particular span of MySQL 8.0 versions.
Fixes: #4492
Change-Id: I14e7237e0be4a9c21c58c921066304ae99ac4dc6
Fully removed the behavior of strings passed directly as components of a
:func:`.select` or :class:`.Query` object being coerced to :func:`.text`
constructs automatically; the warning that has been emitted is now an
ArgumentError or in the case of order_by() / group_by() a CompileError.
This has emitted a warning since version 1.0 however its presence continues
to create concerns for the potential of mis-use of this behavior.
Note that public CVEs have been posted for order_by() / group_by() which
are resolved by this commit: CVE-2019-7164 CVE-2019-7548
Added "SQL phrase validation" to key DDL phrases that are accepted as plain
strings, including :paramref:`.ForeignKeyConstraint.on_delete`,
:paramref:`.ForeignKeyConstraint.on_update`,
:paramref:`.ExcludeConstraint.using`,
:paramref:`.ForeignKeyConstraint.initially`, for areas where a series of SQL
keywords only are expected.Any non-space characters that suggest the phrase
would need to be quoted will raise a :class:`.CompileError`. This change
is related to the series of changes committed as part of 🎫`4481`.
Fixed issue where using an uppercase name for an index type (e.g. GIST,
BTREE, etc. ) or an EXCLUDE constraint would treat it as an identifier to
be quoted, rather than rendering it as is. The new behavior converts these
types to lowercase and ensures they contain only valid SQL characters.
Quoting is applied to :class:`.Function` names, those which are usually but
not necessarily generated from the :attr:`.sql.func` construct, at compile
time if they contain illegal characters, such as spaces or punctuation. The
names are as before treated as case insensitive however, meaning if the
names contain uppercase or mixed case characters, that alone does not
trigger quoting. The case insensitivity is currently maintained for
backwards compatibility.
Fixes: #4481Fixes: #4473Fixes: #4467
Change-Id: Ib22a27d62930e24702e2f0f7c74a0473385a08eb
Fixed bug in SQLite DDL where using an expression as a server side default
required that it be contained within parenthesis to be accepted by the
sqlite parser. Pull request courtesy Bartlomiej Biernacki.
Fixes: #4474Closes: #4475
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4475
Pull-request-sha: 0af3238c69
Change-Id: I888c69bfba5a5ab8ffa420f512557ac311a36b31
(cherry picked from commit f7ce37e9dd493d4b60b3156f08ef4c98fd63904d)
Revised the query used when reflecting CHECK constraints to make use of the
``pg_get_constraintdef`` function, as the ``consrc`` column is being
deprecated in PG 12. Thanks to John A Stevenson for the tip.
Fixes: #4463
Change-Id: Ie0ee9bdfddb0635db72b35c2e2e4b27f154162fd
A large change throughout the library has ensured that all objects, parameters,
and behaviors which have been noted as deprecated or legacy now emit
``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now
defaults to displaying deprecation warnings, as well as that modern test suites
based on tools like tox and pytest tend to display deprecation warnings,
this change should make it easier to note what API features are obsolete.
See the notes added to the changelog and migration notes for further
details.
Fixes: #4393
Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
Fixed regression in integer precision logic due to the refactor of the
cx_Oracle dialect in 1.2. We now no longer apply the cx_Oracle.NATIVE_INT
type to result columns sending integer values (detected as positive
precision with scale ==0) which encounters integer overflow issues with
values that go beyond the 32 bit boundary. Instead, the output variable
is left untyped so that cx_Oracle can choose the best option.
Fixes: #4457
Change-Id: I1e3114c2f37bf028fb1f521a3e9789a77e3a7491
The ``literal_processor`` for the :class:`.Unicode` and
:class:`.UnicodeText` datatypes now render an ``N`` character in front of
the literal string expression as required by SQL Server for Unicode string
values rendered in SQL expressions.
Note that this adds full unicode characters to the standard test suite,
which means we also need to bump MySQL provisioning up to utf8mb4.
Modern installs do not seem to be reproducing the 1271 issue locally,
if it reproduces in CI it would be better for us to skip those ORM-centric
tests for MySQL.
Also remove unused _StringType from SQL Server dialect
Fixes: #4442
Change-Id: Id55817b3e8a2d81ddc8b7b27f85e3f1dcc1cea7e
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Fixed issue where a :class:`.postgresql.ENUM` or a custom domain present
in a remote schema would not be recognized within column reflection if
the name of the enum/domain or the name of the schema required quoting.
A new parsing scheme now fully parses out quoted or non-quoted tokens
including support for SQL-escaped quotes.
Fixed issue where multiple :class:`.postgresql.ENUM` objects referred to
by the same :class:`.MetaData` object would fail to be created if
multiple objects had the same name under different schema names. The
internal memoization the Postgresql dialect uses to track if it has
created a particular :class:`.postgresql.ENUM` in the database during
a DDL creation sequence now takes schema name into account.
Fixes: #4416
Change-Id: I8cf03069e10b12f409e9b6796e24fc5850979955
test_compiler is mostly related to SELECT statements as well
as smaller SQL elements. While it still has some DDL related
tests, move out all the remaining insert/update tests into
the already present test_insert.py, test_update.py
Fixes: #2630
Change-Id: I4167618543fd1235d12d1717c8c629d2374b325a
Fixed issue where reflection of a PostgreSQL domain that is expressed as an
array would fail to be recognized. Pull request courtesy Jakub Synowiec.
Fixes: #4377
Change-Id: I252c79ca435b87d4d9172b1c84e0e74e789ef676
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4380
As MySQLdb support read_timeout and write_timeout option, and
sqlalchemy just support read_timeout option. So sqlalchemy need to add
write_timeout option.
Fixes: #4381
Change-Id: I2bea80bdd6f20fafc72b48fa0e5d795626d9d9b9
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4385
Updated the parameters that can be sent to the cx_Oracle DBAPI to both allow
for all current parameters as well as for future parameters not added yet.
In addition, removed unused parameters that were deprecated in version 1.2,
and additionally we are now defaulting "threaded" to False.
Fixes: #4369
Change-Id: I599668960e7b2d5bd1f5e6850e10b5b3ec215ed3
Deprecated the use of :class:`.Sequence` with SQL Server in order to affect
the "start" and "increment" of the IDENTITY value, in favor of new
parameters ``mssql_identity_start`` and ``mssql_identity_increment`` which
set these parameters directly. :class:`.Sequence` will be used to generate
real ``CREATE SEQUENCE`` DDL with SQL Server in a future release.
Fixes: #4362
Change-Id: I1e69378c5c960ff0bc28137c923589692f1a918f
Fixed regression caused by 🎫`4344` released in 1.2.13, where the fix
for MySQL 8.0's case sensitivity problem with referenced column names when
reflecting foreign key referents is worked around using the
``information_schema.columns`` view. The workaround was failing on OSX /
``lower_case_table_names=2`` which produces non-matching casing for the
``information_schema.columns`` vs. that of ``SHOW CREATE TABLE``, so in
case-insensitive SQL modes case-insensitive matching is now used.
Fixes: #4361
Change-Id: I748549bc4c27fad6394593f8ec93fc22bfd01f6c
Implemented the SQLite ``ON CONFLICT`` clause as understood at the DDL
level, e.g. for primary key, unique, and CHECK constraints as well as
specified on a :class:`.Column` to satisfy inline primary key and NOT NULL.
Pull request courtesy Denis Kataev.
Fixes: #4360
Change-Id: I4cd4bafa8fca41e3101c87dbbfe169741bbda3f4
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/431
Min ho Kim