Fixed issue where PostgreSQL dialect options such as ``postgresql_include``
on :class:`.PrimaryKeyConstraint` and :class:`.UniqueConstraint` were
rendered in the wrong position when combined with constraint deferrability
options like ``deferrable=True``. Pull request courtesy G Allajmi.
Fixes: #12867Closes: #13003
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13003
Pull-request-sha: 1a9216062f
Change-Id: I8c55d8faae25d56ff63c9126d569c01d8ee6c7dd
Added support for monotonic server-side functions such as PostgreSQL 18's
``uuidv7()`` to work with the :ref:`engine_insertmanyvalues` feature.
By passing ``monotonic=True`` to any :class:`.Function`, the function can
be used as a sentinel for tracking row order in batched INSERT operations
with RETURNING, allowing the ORM and Core to efficiently batch INSERT
statements while maintaining deterministic row ordering.
Fixes: #13014
Change-Id: I2fabf96c8fbdb6c1d255fd4781cbd31fed17e1e9
Fixed the structure of the SQL string used for the
:ref:`engine_insertmanyvalues` feature when an explicit sequence with
``nextval()`` is used. The SQL function invocation for the sequence has
been moved from being rendered inline within each tuple inside of VALUES to
being rendered once in the SELECT that reads from VALUES. This change
ensures the function is invoked in the correct order as rows are processed,
rather than assuming PostgreSQL will execute inline function calls within
VALUES in a particular order. While current PostgreSQL versions appear to
handle the previous approach correctly, the database does not guarantee
this behavior for future versions.
Fixes: #13015
Change-Id: Ia0a2a4e8f89e21852d7cb550dfa5d9ea9447b590
this DB is extremely erratic in being able to connect. Add
a brute force connection retrier to all engines everywhere
(which for oracledb we can fortunately use their built-in feature
that also works).
This actually works and I can see it pausing under load, reconnecting,
and succeeding. the problem is that absolutely every engine everywhere
needs this routine otherwise an engine without a retrier in it will
crash. That then necessitates digging into testing_engine(),
making sure testing_engine() is used everywhere an engine that's going
to connect is used, then dealing with the fallout from that.
We also simplify some older workarounds for cx_oracle and
hack into config/provision to make oracledb seem like the primary
DBAPI for most tests.
testing_engine has been completely overhauled, making use of a new
post_configure_testing_engine() hook which moves and refines
the SQLite pool sharing and savepoint logic all into sqlite/provision.py
and also allows for cx_oracle to apply a retry event handler.
Change-Id: I4ea4c523effb878290d28b94d8925eb32fc5ae3b
one particular vector test wont run on oracle 23c free, so
just disable it.
added better skips for the rest of the vector tests and
fixed a deprecation issue.
this will be the first run on the new oracle23 on CI so we'll have to
see how this goes.
Also adjust for mariabdb12 being overly helpful with regards
to stale row updates.
as we are having trouble getting 23c to pass throug transaction
tests, i noted we have an explosion of tests due to the multiple
drivers, so this patch introduces __sparse_driver_backend__
for all tests where we want variety of
database server but there's no need to test every driver.
This should dramatically reduce the size of the test suite run
Change-Id: Ic8d3eb0a60e76b4c54c6bb4a721f90c81ede782b
Added support for PostgreSQL 14+ HSTORE subscripting syntax. When connected
to PostgreSQL 14 or later, HSTORE columns now automatically use the native
subscript notation ``hstore_col['key']`` instead of the arrow operator
``hstore_col -> 'key'`` for both read and write operations. This provides
better compatibility with PostgreSQL's native HSTORE subscripting feature
while maintaining backward compatibility with older PostgreSQL versions.
as part of this change we add a new parameter to custom_op "visit_name"
which allows a custom op to refer to a specific visit method in a
dialect's compiler.
Fixes: #12948
Change-Id: Id98d333fe78e31d9c7679cb2902f1c7e458d6e11
Support for ``VIRTUAL`` computed columns on PostgreSQL 18 and later has
been added. The default behavior when :paramref:`.Computed.persisted` is
not specified has been changed to align with PostgreSQL 18's default of
``VIRTUAL``. When :paramref:`.Computed.persisted` is not specified, no
keyword is rendered on PostgreSQL 18 and later; on older versions a
warning is emitted and ``STORED`` is used as the default. To explicitly
request ``STORED`` behavior on all PostgreSQL versions, specify
``persisted=True``.
Fixes: #12866
Change-Id: Ic2ebdbe79e230a88370cf2b3503d2d1815f72a39
Support for storage parameters in ``CREATE TABLE`` using the ``WITH``
clause has been added. The ``postgresql_with`` dialect option of
:class:`_schema.Table` accepts a mapping of key/value options.
The PostgreSQL dialect now support reflection of table options, including
the storage parameters, table access method and table spaces. These options
are automatically reflected when autoloading a table, and are also
available via the :meth:`_engine.Inspector.get_table_options` and
:meth:`_engine.Inspector.get_multi_table_optionsmethod` methods.
Fixes: #10909Closes: #12584Closes: #12684
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12584
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12684
Pull-request-sha: a660459de9
Change-Id: I33e3d8b8eb0c02530933cb124e3d375ca4af7db2
Some improvements to the reflection of PostgreSQL to avoid
unnecessary queries when not needed.
Fixes: #12908
Change-Id: Ic3da50ee43670f26d3159f5ec9a235a9a1963b8a
Added support for reflection of collation in types for PostgreSQL.
The ``collation`` will be set only if different from the default
one for the type.
References #6511Closes: #12510
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12510
Pull-request-sha: c0a390314e
Change-Id: I269a194d526a0689a4b38f10456d28539c73cffb
Named types such as :class:`_postgresql.ENUM` and
:class:`_postgresql.DOMAIN` (as well as the dialect-agnostic
:class:`_types.Enum` version) are now more strongly associated with the
:class:`_schema.MetaData` at the top of the table hierarchy and are
de-associated with any particular :class:`_schema.Table` they may be a part
of. This better represents how PostgreSQL named types exist independently
of any particular table, and that they may be used across many tables
simultaneously. The change impacts the behavior of the "default schema"
for a named type, as well as the CREATE/DROP behavior in relationship to
the :class:`.MetaData` and :class:`.Table` construct. The change also
includes a new :class:`.CheckFirst` enumeration which allows fine grained
control over "check" queries during DDL operations, as well as that the
:paramref:`_types.SchemaType.inherit_schema` parameter is deprecated and
will emit a deprecation warning when used. See the migration notes for
full details.
Fixes: #12690
Change-Id: I9752a9d52774ae760ec3448f62fc5046c58d68f5
Added new parameter :paramref:`.Enum.create_type` to the Core
:class:`.Enum` class. This parameter is automatically passed to the
corresponding :class:`_postgresql.ENUM` native type during DDL operations,
allowing control over whether the PostgreSQL ENUM type is implicitly
created or dropped within DDL operations that are otherwise targeting
tables only. This provides control over the
:paramref:`_postgresql.ENUM.create_type` behavior without requiring
explicit creation of a :class:`_postgresql.ENUM` object.
Fixes: #10604
Change-Id: I450003ec2a2a65c119fe7ca8ff201392ce6b91e1
on the theme of 2.0/2.1 are relatively similar right now, do
some more cleanup removing the py3k suffixes that we dont need anymore.
I'm not sure that the logic that would search for these suffixes is
even present anymore and I would guess we removed it when we removed
py2k support. However, I am planning on possibly bringing it back
for some of the py314 stuff, so I still want to be able to do
suffix stuff. that will be for another patch.
Change-Id: I929e6edd922f8d5f943acce77191fb1e3035b42c
The :meth:`_types.ARRAY.Comparator.any` and
:meth:`_types.ARRAY.Comparator.all` methods for the :class:`_types.ARRAY`
type are now deprecated for removal; these two methods along with
:func:`_postgresql.Any` and :func:`_postgresql.All` have been legacy for
some time as they are superseded by the :func:`_sql.any_` and
:func:`_sql.all_` functions, which feature more intutive use.
Fixes: #10821
Change-Id: I8eb3bbcb98af4ee60a21767dc3bdac771cbc0b4c
Fixed issue where the :func:`_sql.any_` and :func:`_sql.all_` aggregation
operators would not correctly coerce the datatype of the compared value, in
those cases where the compared value were not a simple int/str etc., such
as a Python ``Enum`` or other custom value. This would lead to execution
time errors for these values. This issue is essentially the same as
🎫`6515` which was for the now-legacy :meth:`.ARRAY.any` and
:meth:`.ARRAY.all` methods.
Fixes: #12874
Change-Id: I980894c23b9974bc84d584a1a4c5fae72dded6d3
A :class:`.CompileError` is raised if attempting to create a PostgreSQL
:class:`_postgresql.ENUM` or :class:`_postgresql.DOMAIN` datatype using a
name that matches a known pg_catalog datatype name, and a default schema is
not specified. These types must be explicit within a schema in order to
be differentiated from the built-in pg_catalog type. The "public" or
otherwise default schema is not chosen by default here since the type can
only be reflected back using the explicit schema name as well (it is
otherwise not visible due to the pg_catalog name). Pull request courtesy
Kapil Dagur.
We originally thought we were going to do some default logic for the
default / "public" schema however this produces a type that is not
symmetric to its reflection, since the schema name must be explicit
for our current reflection queries.
So since it's an extremely bad idea to make an ENUM/DOMAIN with a
reserved type name anyway, we raise a compileerror if the type
has a known name. this is not robust against other names that
might exist in pg_catalog or other schemas that are in the search
path with these names. People just have to know what they're doing
here, the error here only covers a small subset of real world cases.
Fixes: #12761Closes: #12822
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12822
Pull-request-sha: 793cf27be2
Change-Id: I9df9c216ab4102f06685a64ae7e55db1b3a9535e
Added new generalized aggregate function ordering to functions via the
:func:`_functions.FunctionElement.aggregate_order_by` method, which
receives an expression and generates the appropriate embedded "ORDER BY" or
"WITHIN GROUP (ORDER BY)" phrase depending on backend database. This new
function supersedes the use of the PostgreSQL
:func:`_postgresql.aggregate_order_by` function, which remains present for
backward compatibility. To complement the new parameter, the
:paramref:`_functions.aggregate_strings.order_by` which adds ORDER BY
capability to the :class:`_functions.aggregate_strings` dialect-agnostic
function which works for all included backends. Thanks much to Reuven
Starodubski with help on this patch.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #12853Closes: #12856
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12856
Pull-request-sha: d93fb591751227eb1f96052ea3ad449f511f70b3
Change-Id: I8eb41ff2d57695963a358b5f0017ca9372f15f70
Fixed issue where selecting an enum array column containing NULL values
would fail to parse properly in the PostgreSQL dialect. The
:func:`._split_enum_values` function now correctly handles NULL entries by
converting them to Python ``None`` values.
Fixes: #12847
Change-Id: I39d10bc1be6b458da7e5d3f4b740f8faafd0adc5
Added a new concept of "operator classes" to the SQL operators supported by
SQLAlchemy, represented within the enum :class:`.OperatorClass`. The
purpose of this structure is to provide an extra layer of validation when a
particular kind of SQL operation is used with a particular datatype, to
catch early the use of an operator that does not have any relevance to the
datatype in use; a simple example is an integer or numeric column used with
a "string match" operator.
Fixes: #12736
Change-Id: I44f46d7326aef6847dbf0cf7a325833f8e347da6
Fixed regression in PostgreSQL dialect where JSONB subscription syntax
would generate incorrect SQL for JSONB-returning functions, causing syntax
errors. The dialect now properly wraps function calls and expressions in
parentheses when using the ``[]`` subscription syntax, generating
``(function_call)[index]`` instead of ``function_call[index]`` to comply
with PostgreSQL syntax requirements.
Fixes: #12778
Change-Id: If1238457e6bba6a933023b26519a41aa5de4dbcd
Fixes bug that would mistakenly interpret a domain or enum type
with name starting in ``interval`` as an ``INTERVAL`` type while
reflecting a table.
Fixes: #12744
Change-Id: I89ab287c3847ca545691afe73f26d86bf2337ae0
Added support for PostgreSQL 14+ JSONB subscripting syntax. When connected
to PostgreSQL 14 or later, JSONB columns now automatically use the native
subscript notation ``jsonb_col['key']`` instead of the arrow operator
``jsonb_col -> 'key'`` for both read and write operations. This provides
better compatibility with PostgreSQL's native JSONB subscripting feature
while maintaining backward compatibility with older PostgreSQL versions.
JSON columns continue to use the traditional arrow syntax regardless of
PostgreSQL version.
Fixes: #10927
Change-Id: I4b3a8a55a71f2ca3d95416a7b350b785574631eb
Adds a new ``str`` subclass :class:`dialects.postgresql.BitString`
representing PostgreSQL bitstrings in python, that includes
functionality for converting to and from ``int`` and ``bytes``, in
addition to implementing utility methods and operators for dealing
with bits.
This new class is returned automatically by the :class:`postgresql.BIT`
type.
Fixes: #10556Closes: #12594
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12594
Pull-request-sha: da47f40739
Change-Id: I64685660527c23666f7351b2c393fa86dfb643ea
Fill the `postgresql_ops` key of PostgreSQL's `dialect_options` returned by get_multi_indexes() with a mapping from column names to the operator class, if it's not the default for respective data type.
As we need to join on ``pg_catalog.pg_opclass``, the table definition is added to ``postgresql.pg_catalog``.
Fixes#8664.
Closes: #12504
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12504
Pull-request-sha: 8fdf93e1b2
Change-Id: I8789c1e9d15f8cc9a7205f492ec730570f19bbcc
Changed the default value of :paramref:`_types.Enum.inherit_schema` to
``True`` when :paramref:`_types.Enum.schema` and
:paramref:`_types.Enum.metadata` parameters are not provided.
The same behavior has been applied also to PostgreSQL
:class:`_postgresql.DOMAIN` type.
Fixes: #10594
Change-Id: Id3d819e3608974353e365cd063d9c5e40a071e73
Added support for the pow operator (``**``), with a default SQL
implementation of the ``POW()`` function. On Oracle Database, PostgreSQL
and MSSQL it renders as ``POWER()``. As part of this change, the operator
routes through a new first class ``func`` member :class:`_functions.pow`,
which renders on Oracle Database, PostgreSQL and MSSQL as ``POWER()``.
Fixes: #8579Closes: #8580
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8580
Pull-request-sha: 041b2ef474
Change-Id: I371bd44ed3e58f2d55ef705aeec7d04710c97f23
This is supported both for schema definition and reflection.
Fixes#10665.
Closes: #12485
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12485
Pull-request-sha: 1aabea7b55ece9fc0c6e069b777d4404ac01f964
Change-Id: I81d23966f84390dd1b03f0d13284ce6d883ee24e
When building a PostgreSQL ``ARRAY`` literal using
:class:`_postgresql.array` with an empty ``clauses`` argument, the
:paramref:`_postgresql.array.type_` parameter is now significant in that it
will be used to render the resulting ``ARRAY[]`` SQL expression with a
cast, such as ``ARRAY[]::INTEGER``. Pull request courtesy Denis Laxalde.
Fixes: #12432Closes: #12435
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12435
Pull-request-sha: 9633d3c15d
Change-Id: I29ed7bd0562b82351d22de0658fb46c31cfe44f6
Added syntax extension :func:`_postgresql.distinct_on` to build ``DISTINCT
ON`` clauses. The old api, that passed columns to
:meth:`_sql.Select.distinct`, is now deprecated.
Fixes: #12342
Change-Id: Ia6a7e647a11e57b6ac2f50848778c20dc55eaf54
Added support for specifying a list of columns for ``SET NULL`` and ``SET
DEFAULT`` actions of ``ON DELETE`` clause of foreign key definition on
PostgreSQL. Pull request courtesy Denis Laxalde.
Fixes: #11595Closes: #12421
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12421
Pull-request-sha: d0394db706
Change-Id: I036a559ae4a8efafe9ba64d776a840bd785a7397
Fixed issue in PostgreSQL network types :class:`_postgresql.INET`,
:class:`_postgresql.CIDR`, :class:`_postgresql.MACADDR`,
:class:`_postgresql.MACADDR8` where sending string values to compare to
these types would render an explicit CAST to VARCHAR, causing some SQL /
driver combinations to fail. Pull request courtesy Denis Laxalde.
Fixes: #12060Closes: #12412
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12412
Pull-request-sha: 029fda7f2d
Change-Id: Id4a502ebc119775567cacddbabef2ce9715c1a9f
Fixed compiler issue in the PostgreSQL dialect where incorrect keywords
would be passed when using "FOR UPDATE OF" inside of a subquery.
Fixes: #12417
Change-Id: I6255d165e8e719e1786e78aa60ee8e6a95af1dcb
Add SQL typing to reflection query used to retrieve a the structure
of IDENTITY columns, adding explicit JSON typing to the query to suit
unusual PostgreSQL driver configurations that don't support JSON natively.
Fixed issue affecting PostgreSQL 17.3 and greater where reflection of
domains with "NOT NULL" as part of their definition would include an
invalid constraint entry in the data returned by
:meth:`_postgresql.PGInspector.get_domains` corresponding to an additional
"NOT NULL" constraint that isn't a CHECK constraint; the existing
``"nullable"`` entry in the dictionary already indicates if the domain
includes a "not null" constraint. Note that such domains also cannot be
reflected on PostgreSQL 17.0 through 17.2 due to a bug on the PostgreSQL
side; if encountering errors in reflection of domains which include NOT
NULL, upgrade to PostgreSQL server 17.3 or greater.
Fixes: #11751
Change-Id: I8e69de51601dca3257186e38c6f699fbfd9014c6
Adjusted the asyncpg connection wrapper so that the asyncpg
``.transaction()`` call sends ``None`` for isolation_level if not otherwise
set in the SQLAlchemy dialect/wrapper, thereby allowing asyncpg to make use
of the server level setting for isolation_level in the absense of a
client-level setting. Previously, this behavior of asyncpg was blocked by a
hardcoded ``read_committed``.
Fixes: #12159
Change-Id: I2cd878a5059a8fefc9557a9b8e056fedaee2e9a4
Adjusted the asyncpg dialect so that an empty SQL string, which is valid
for PostgreSQL server, may be successfully processed at the dialect level,
such as when using :meth:`.Connection.exec_driver_sql`. Pull request
courtesy Andrew Jackson.
Closes: #12220
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12220
Pull-request-sha: 45c94febee
Change-Id: I870df9e31f4a229939e76c702724c25073329282
in 2.1 we want these structures to be cacheable, so start
by cleaning up types and adding coercions to enforce those types.
these will be more locked down in 2.1 as we will need to move
bound parameter coercion outside of compilation, but here
do some small starts and introduce in 2.0.
in one interest of cachability, a "literal_binds" that found
its way into SQLite's compiler is replaced with "literal_execute",
the difference being that the latter is cacheable. This literal
is apparently necessary to suit SQLite's query planner for
the "index criteria" portion of the on conflict clause that otherwise
can't work with a real bound parameter.
Change-Id: I4d66ec1473321616a1707da324a7dfe7a61ec94e
G Allajmi