<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
Issue: https://github.com/sqlalchemy/sqlalchemy/issues/10902
Hello! This is my first PR here, so please let me know what I may have missed in terms of having a valuable contribution. I was looking through issues to grab an easy first one, and found this. Looks like someone else was going to have a go at it, but never did.
I simply added a small change to the FK regex in for Postgres that allows anything not quotes alongside escaped double quotes. Test is included for the scenario mentioned in the issue. Alongside that, I didn't see a test for general quoted strings, so I added another one that includes spaces and dashes, in my experience common things to be used inside quoted identifiers.
A manual test as well:
DB setup:
```
austin_test_bug=# CREATE TABLE """test_parent_table-quoted""" (id SERIAL PRIMARY KEY, val INTEGER);
CREATE TABLE
austin_test_bug=# CREATE TABLE test_child_table_ref_quoted (id SERIAL, parent INTEGER, CONSTRAINT fk_parent FOREIGN KEY (parent) REFERENCES """test_parent_table-quoted"""(id));
CREATE TABLE
austin_test_bug=# \d+
List of relations
Schema | Name | Type | Owner | Persistence | Access method | Size | Description
--------+------------------------------------+----------+----------+-------------+---------------+------------+-------------
public | "test_parent_table-quoted" | table | postgres | permanent | heap | 0 bytes |
public | "test_parent_table-quoted"_id_seq | sequence | postgres | permanent | | 8192 bytes |
public | test_child_table_ref_quoted | table | postgres | permanent | heap | 0 bytes |
public | test_child_table_ref_quoted_id_seq | sequence | postgres | permanent | | 8192 bytes |
(4 rows)
```
And the python:
```
>>> from sqlalchemy import create_engine, inspect
>>> engine = create_engine('postgresql://scott:tiger@localhost:5432/austin_test_bug')
>>> connection = engine.connect()
>>> inspect(connection).get_multi_foreign_keys()
{(None, '"test_parent_table-quoted"'): [], (None, 'test_child_table_ref_quoted'): [{'name': 'fk_parent', 'constrained_columns': ['parent'], 'referred_schema': None, 'referred_table': '"test_parent_table-quoted"', 'referred_columns': ['id'], 'options': {}, 'comment': None}]}
```
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Fixes: #10902Closes: #13179
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13179
Pull-request-sha: 8890dc3250
Change-Id: I185c2cb062740551ab931368de602054eb5a4acd
Fixed issue where :meth:`_postgresql.Insert.on_conflict_do_update`
using parametrized bound parameters in the ``set_`` clause would fail
when used with executemany batching. For dialects that use the
``use_insertmanyvalues_wo_returning`` optimization (psycopg2),
insertmanyvalues is now disabled when there is an ON CONFLICT clause.
For cases with RETURNING, row-at-a-time mode is used when the SET
clause contains parametrized bindparams (bindparams that receive
values from the parameters dict), ensuring each row's parameters are
correctly applied. ON CONFLICT statements using expressions like
``excluded.<column>`` continue to batch normally.
Fixed issue where :meth:`_sqlite.Insert.on_conflict_do_update`
using parametrized bound parameters in the ``set_`` clause would fail
when used with executemany batching. Row-at-a-time mode is now used
for ON CONFLICT statements with RETURNING that contain parametrized
bindparams, ensuring each row's parameters are correctly applied. ON
CONFLICT statements using expressions like ``excluded.<column>``
continue to batch normally.
Fixes: #13130
Change-Id: I0c5a9142401c745d38e58d071c16e53610f9bfea
Fixed issue where :meth:`_postgresql.Insert.on_conflict_do_update`
as well as :meth:`_sqlite.Insert.on_conflict_do_update`
parameters were not respecting compilation options such as
``literal_binds=True``.
Pull request courtesy Loïc Simon.
Fixes: #13110Closes: #13111
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13111
Pull-request-sha: 9ca251610b
Change-Id: Ice21e508210d682098104c78e77bad8d24e6c93f
Fixed issue in the :ref:`engine_insertmanyvalues` feature where using
PostgreSQL's ``ON CONFLICT`` clause with
:paramref:`_dml.Insert.returning.sort_by_parameter_order` enabled would
generate invalid SQL when the insert used an implicit sentinel (server-side
autoincrement primary key). The generated SQL would incorrectly declare a
sentinel counter column in the ``imp_sen`` table alias without providing
corresponding values in the ``VALUES`` clause, leading to a
``ProgrammingError`` indicating column count mismatch. The fix allows batch
execution mode when ``embed_values_counter`` is active, as the embedded
counter provides the ordering capability needed even with upsert behaviors,
rather than unnecessarily downgrading to row-at-a-time execution.
Fixes: #13107
Change-Id: I382472b2cf2991b520344adea5783584e27425d0
Fixed an issue in the PostgreSQL dialect where foreign key constraint
reflection would incorrectly swap or fail to capture ``onupdate`` and
``ondelete`` values when these clauses appeared in a different order than
expected in the constraint definition. This issue primarily affected
PostgreSQL-compatible databases such as CockroachDB, which may return ``ON
DELETE`` before ``ON UPDATE`` in the constraint definition string. The
reflection logic now correctly parses both clauses regardless of their
ordering.
Fixes: #13105
Change-Id: I1331b433f713632e84ae6a34467806e080b8003e
Improved the foreign key reflection regular expression pattern used by the
PostgreSQL dialect to be more permissive in matching identifier characters,
allowing it to correctly handle unicode characters in table and column
names. This change improves compatibility with PostgreSQL variants such as
CockroachDB that may use different quoting patterns in combination with
unicode characters in their identifiers. Pull request courtesy Gord
Thompson.
Change-Id: Iaee340879400e01df2f776417e8b1018f1801cfe
The the parameter :paramref:`_schema.DropConstraint.isolate_from_table`
was deprecated since it has no effect on the drop table behavior.
Its default values was also changed to ``False``.
Fixes: #13006
Change-Id: Ibaa68e78da301a4f2a200f8c7abb2dc099fef37a
Fixed regression in PostgreSQL dialect where JSONB subscription syntax
would generate incorrect SQL for :func:`.cast` expressions returning JSONB,
causing syntax errors. The dialect now properly wraps cast expressions in
parentheses when using the ``[]`` subscription syntax, generating
``(CAST(...))[index]`` instead of ``CAST(...)[index]`` to comply with
PostgreSQL syntax requirements. This extends the fix from 🎫`12778`
which addressed the same issue for function calls.
This reverts how we did the fix for #12778 in Function.self_group()
and instead moves to a direct Grouping() applied in the PG compiler
based on isinstance of the left side.
in retrospect, when we first did #10927, we **definitely** made
the completely wrong choice in how to do this, the original idea
to detect when we were in an UPDATE and use [] only then was
by **far** what we should have done, given the fact that PG indexes
are based on exact syntax matches. but since we've made everyone
switch to [] format for their indexes now we can't keep going
back and forth. even though PG would like [] to be the defacto
syntax it simply is not. We should potentially pursue a dialect/
create_engine option to switch the use of [] back to -> for
all cases except UPDATE.
Fixes: #13067
Change-Id: I2e0d0f45ebb820d2a8f214550f1d1a500bae223b
Fixed issue where PostgreSQL JSONB operators
:meth:`_postgresql.JSONB.Comparator.path_match` and
:meth:`_postgresql.JSONB.Comparator.path_exists` were applying incorrect
``VARCHAR`` casts to the right-hand side operand when used with newer
PostgreSQL drivers such as psycopg. The operators now indicate the
right-hand type as ``JSONPATH``, which currently results in no casting
taking place, but is also compatible with explicit casts if the
implementation were require it at a later point.
Fixes: #13059
Change-Id: I8e1a58361456f7efabf4940339cb5ce2c5a1d5f9
The default DBAPI driver for the Oracle Database dialect has been changed
to ``oracledb`` instead of ``cx_oracle``. The ``cx_oracle`` driver remains
fully supported and can be explicitly specified in the connection URL
using ``oracle+cx_oracle://``.
The default DBAPI driver for the PostgreSQL dialect has been changed to
``psycopg`` (psycopg version 3) instead of ``psycopg2``. The ``psycopg2``
driver remains fully supported and can be explicitly specified in the
connection URL using ``postgresql+psycopg2://``.
Fixes: #13010
Change-Id: Ie75810f4c3af609d20da63289d2662dfa2385ca2
Fixed issue where PostgreSQL dialect options such as ``postgresql_include``
on :class:`.PrimaryKeyConstraint` and :class:`.UniqueConstraint` were
rendered in the wrong position when combined with constraint deferrability
options like ``deferrable=True``. Pull request courtesy G Allajmi.
Fixes: #12867Closes: #13003
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13003
Pull-request-sha: 1a9216062f
Change-Id: I8c55d8faae25d56ff63c9126d569c01d8ee6c7dd
Added support for monotonic server-side functions such as PostgreSQL 18's
``uuidv7()`` to work with the :ref:`engine_insertmanyvalues` feature.
By passing ``monotonic=True`` to any :class:`.Function`, the function can
be used as a sentinel for tracking row order in batched INSERT operations
with RETURNING, allowing the ORM and Core to efficiently batch INSERT
statements while maintaining deterministic row ordering.
Fixes: #13014
Change-Id: I2fabf96c8fbdb6c1d255fd4781cbd31fed17e1e9
Fixed the structure of the SQL string used for the
:ref:`engine_insertmanyvalues` feature when an explicit sequence with
``nextval()`` is used. The SQL function invocation for the sequence has
been moved from being rendered inline within each tuple inside of VALUES to
being rendered once in the SELECT that reads from VALUES. This change
ensures the function is invoked in the correct order as rows are processed,
rather than assuming PostgreSQL will execute inline function calls within
VALUES in a particular order. While current PostgreSQL versions appear to
handle the previous approach correctly, the database does not guarantee
this behavior for future versions.
Fixes: #13015
Change-Id: Ia0a2a4e8f89e21852d7cb550dfa5d9ea9447b590
this DB is extremely erratic in being able to connect. Add
a brute force connection retrier to all engines everywhere
(which for oracledb we can fortunately use their built-in feature
that also works).
This actually works and I can see it pausing under load, reconnecting,
and succeeding. the problem is that absolutely every engine everywhere
needs this routine otherwise an engine without a retrier in it will
crash. That then necessitates digging into testing_engine(),
making sure testing_engine() is used everywhere an engine that's going
to connect is used, then dealing with the fallout from that.
We also simplify some older workarounds for cx_oracle and
hack into config/provision to make oracledb seem like the primary
DBAPI for most tests.
testing_engine has been completely overhauled, making use of a new
post_configure_testing_engine() hook which moves and refines
the SQLite pool sharing and savepoint logic all into sqlite/provision.py
and also allows for cx_oracle to apply a retry event handler.
Change-Id: I4ea4c523effb878290d28b94d8925eb32fc5ae3b
one particular vector test wont run on oracle 23c free, so
just disable it.
added better skips for the rest of the vector tests and
fixed a deprecation issue.
this will be the first run on the new oracle23 on CI so we'll have to
see how this goes.
Also adjust for mariabdb12 being overly helpful with regards
to stale row updates.
as we are having trouble getting 23c to pass throug transaction
tests, i noted we have an explosion of tests due to the multiple
drivers, so this patch introduces __sparse_driver_backend__
for all tests where we want variety of
database server but there's no need to test every driver.
This should dramatically reduce the size of the test suite run
Change-Id: Ic8d3eb0a60e76b4c54c6bb4a721f90c81ede782b
Added support for PostgreSQL 14+ HSTORE subscripting syntax. When connected
to PostgreSQL 14 or later, HSTORE columns now automatically use the native
subscript notation ``hstore_col['key']`` instead of the arrow operator
``hstore_col -> 'key'`` for both read and write operations. This provides
better compatibility with PostgreSQL's native HSTORE subscripting feature
while maintaining backward compatibility with older PostgreSQL versions.
as part of this change we add a new parameter to custom_op "visit_name"
which allows a custom op to refer to a specific visit method in a
dialect's compiler.
Fixes: #12948
Change-Id: Id98d333fe78e31d9c7679cb2902f1c7e458d6e11
Support for ``VIRTUAL`` computed columns on PostgreSQL 18 and later has
been added. The default behavior when :paramref:`.Computed.persisted` is
not specified has been changed to align with PostgreSQL 18's default of
``VIRTUAL``. When :paramref:`.Computed.persisted` is not specified, no
keyword is rendered on PostgreSQL 18 and later; on older versions a
warning is emitted and ``STORED`` is used as the default. To explicitly
request ``STORED`` behavior on all PostgreSQL versions, specify
``persisted=True``.
Fixes: #12866
Change-Id: Ic2ebdbe79e230a88370cf2b3503d2d1815f72a39
Support for storage parameters in ``CREATE TABLE`` using the ``WITH``
clause has been added. The ``postgresql_with`` dialect option of
:class:`_schema.Table` accepts a mapping of key/value options.
The PostgreSQL dialect now support reflection of table options, including
the storage parameters, table access method and table spaces. These options
are automatically reflected when autoloading a table, and are also
available via the :meth:`_engine.Inspector.get_table_options` and
:meth:`_engine.Inspector.get_multi_table_optionsmethod` methods.
Fixes: #10909Closes: #12584Closes: #12684
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12584
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12684
Pull-request-sha: a660459de9
Change-Id: I33e3d8b8eb0c02530933cb124e3d375ca4af7db2
Some improvements to the reflection of PostgreSQL to avoid
unnecessary queries when not needed.
Fixes: #12908
Change-Id: Ic3da50ee43670f26d3159f5ec9a235a9a1963b8a
Added support for reflection of collation in types for PostgreSQL.
The ``collation`` will be set only if different from the default
one for the type.
References #6511Closes: #12510
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12510
Pull-request-sha: c0a390314e
Change-Id: I269a194d526a0689a4b38f10456d28539c73cffb
Named types such as :class:`_postgresql.ENUM` and
:class:`_postgresql.DOMAIN` (as well as the dialect-agnostic
:class:`_types.Enum` version) are now more strongly associated with the
:class:`_schema.MetaData` at the top of the table hierarchy and are
de-associated with any particular :class:`_schema.Table` they may be a part
of. This better represents how PostgreSQL named types exist independently
of any particular table, and that they may be used across many tables
simultaneously. The change impacts the behavior of the "default schema"
for a named type, as well as the CREATE/DROP behavior in relationship to
the :class:`.MetaData` and :class:`.Table` construct. The change also
includes a new :class:`.CheckFirst` enumeration which allows fine grained
control over "check" queries during DDL operations, as well as that the
:paramref:`_types.SchemaType.inherit_schema` parameter is deprecated and
will emit a deprecation warning when used. See the migration notes for
full details.
Fixes: #12690
Change-Id: I9752a9d52774ae760ec3448f62fc5046c58d68f5
Added new parameter :paramref:`.Enum.create_type` to the Core
:class:`.Enum` class. This parameter is automatically passed to the
corresponding :class:`_postgresql.ENUM` native type during DDL operations,
allowing control over whether the PostgreSQL ENUM type is implicitly
created or dropped within DDL operations that are otherwise targeting
tables only. This provides control over the
:paramref:`_postgresql.ENUM.create_type` behavior without requiring
explicit creation of a :class:`_postgresql.ENUM` object.
Fixes: #10604
Change-Id: I450003ec2a2a65c119fe7ca8ff201392ce6b91e1
on the theme of 2.0/2.1 are relatively similar right now, do
some more cleanup removing the py3k suffixes that we dont need anymore.
I'm not sure that the logic that would search for these suffixes is
even present anymore and I would guess we removed it when we removed
py2k support. However, I am planning on possibly bringing it back
for some of the py314 stuff, so I still want to be able to do
suffix stuff. that will be for another patch.
Change-Id: I929e6edd922f8d5f943acce77191fb1e3035b42c
The :meth:`_types.ARRAY.Comparator.any` and
:meth:`_types.ARRAY.Comparator.all` methods for the :class:`_types.ARRAY`
type are now deprecated for removal; these two methods along with
:func:`_postgresql.Any` and :func:`_postgresql.All` have been legacy for
some time as they are superseded by the :func:`_sql.any_` and
:func:`_sql.all_` functions, which feature more intutive use.
Fixes: #10821
Change-Id: I8eb3bbcb98af4ee60a21767dc3bdac771cbc0b4c
Fixed issue where the :func:`_sql.any_` and :func:`_sql.all_` aggregation
operators would not correctly coerce the datatype of the compared value, in
those cases where the compared value were not a simple int/str etc., such
as a Python ``Enum`` or other custom value. This would lead to execution
time errors for these values. This issue is essentially the same as
🎫`6515` which was for the now-legacy :meth:`.ARRAY.any` and
:meth:`.ARRAY.all` methods.
Fixes: #12874
Change-Id: I980894c23b9974bc84d584a1a4c5fae72dded6d3
A :class:`.CompileError` is raised if attempting to create a PostgreSQL
:class:`_postgresql.ENUM` or :class:`_postgresql.DOMAIN` datatype using a
name that matches a known pg_catalog datatype name, and a default schema is
not specified. These types must be explicit within a schema in order to
be differentiated from the built-in pg_catalog type. The "public" or
otherwise default schema is not chosen by default here since the type can
only be reflected back using the explicit schema name as well (it is
otherwise not visible due to the pg_catalog name). Pull request courtesy
Kapil Dagur.
We originally thought we were going to do some default logic for the
default / "public" schema however this produces a type that is not
symmetric to its reflection, since the schema name must be explicit
for our current reflection queries.
So since it's an extremely bad idea to make an ENUM/DOMAIN with a
reserved type name anyway, we raise a compileerror if the type
has a known name. this is not robust against other names that
might exist in pg_catalog or other schemas that are in the search
path with these names. People just have to know what they're doing
here, the error here only covers a small subset of real world cases.
Fixes: #12761Closes: #12822
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12822
Pull-request-sha: 793cf27be2
Change-Id: I9df9c216ab4102f06685a64ae7e55db1b3a9535e
Added new generalized aggregate function ordering to functions via the
:func:`_functions.FunctionElement.aggregate_order_by` method, which
receives an expression and generates the appropriate embedded "ORDER BY" or
"WITHIN GROUP (ORDER BY)" phrase depending on backend database. This new
function supersedes the use of the PostgreSQL
:func:`_postgresql.aggregate_order_by` function, which remains present for
backward compatibility. To complement the new parameter, the
:paramref:`_functions.aggregate_strings.order_by` which adds ORDER BY
capability to the :class:`_functions.aggregate_strings` dialect-agnostic
function which works for all included backends. Thanks much to Reuven
Starodubski with help on this patch.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #12853Closes: #12856
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12856
Pull-request-sha: d93fb591751227eb1f96052ea3ad449f511f70b3
Change-Id: I8eb41ff2d57695963a358b5f0017ca9372f15f70
Fixed issue where selecting an enum array column containing NULL values
would fail to parse properly in the PostgreSQL dialect. The
:func:`._split_enum_values` function now correctly handles NULL entries by
converting them to Python ``None`` values.
Fixes: #12847
Change-Id: I39d10bc1be6b458da7e5d3f4b740f8faafd0adc5
Added a new concept of "operator classes" to the SQL operators supported by
SQLAlchemy, represented within the enum :class:`.OperatorClass`. The
purpose of this structure is to provide an extra layer of validation when a
particular kind of SQL operation is used with a particular datatype, to
catch early the use of an operator that does not have any relevance to the
datatype in use; a simple example is an integer or numeric column used with
a "string match" operator.
Fixes: #12736
Change-Id: I44f46d7326aef6847dbf0cf7a325833f8e347da6
Fixed regression in PostgreSQL dialect where JSONB subscription syntax
would generate incorrect SQL for JSONB-returning functions, causing syntax
errors. The dialect now properly wraps function calls and expressions in
parentheses when using the ``[]`` subscription syntax, generating
``(function_call)[index]`` instead of ``function_call[index]`` to comply
with PostgreSQL syntax requirements.
Fixes: #12778
Change-Id: If1238457e6bba6a933023b26519a41aa5de4dbcd
Fixes bug that would mistakenly interpret a domain or enum type
with name starting in ``interval`` as an ``INTERVAL`` type while
reflecting a table.
Fixes: #12744
Change-Id: I89ab287c3847ca545691afe73f26d86bf2337ae0
Added support for PostgreSQL 14+ JSONB subscripting syntax. When connected
to PostgreSQL 14 or later, JSONB columns now automatically use the native
subscript notation ``jsonb_col['key']`` instead of the arrow operator
``jsonb_col -> 'key'`` for both read and write operations. This provides
better compatibility with PostgreSQL's native JSONB subscripting feature
while maintaining backward compatibility with older PostgreSQL versions.
JSON columns continue to use the traditional arrow syntax regardless of
PostgreSQL version.
Fixes: #10927
Change-Id: I4b3a8a55a71f2ca3d95416a7b350b785574631eb
Adds a new ``str`` subclass :class:`dialects.postgresql.BitString`
representing PostgreSQL bitstrings in python, that includes
functionality for converting to and from ``int`` and ``bytes``, in
addition to implementing utility methods and operators for dealing
with bits.
This new class is returned automatically by the :class:`postgresql.BIT`
type.
Fixes: #10556Closes: #12594
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12594
Pull-request-sha: da47f40739
Change-Id: I64685660527c23666f7351b2c393fa86dfb643ea
Fill the `postgresql_ops` key of PostgreSQL's `dialect_options` returned by get_multi_indexes() with a mapping from column names to the operator class, if it's not the default for respective data type.
As we need to join on ``pg_catalog.pg_opclass``, the table definition is added to ``postgresql.pg_catalog``.
Fixes#8664.
Closes: #12504
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12504
Pull-request-sha: 8fdf93e1b2
Change-Id: I8789c1e9d15f8cc9a7205f492ec730570f19bbcc
Changed the default value of :paramref:`_types.Enum.inherit_schema` to
``True`` when :paramref:`_types.Enum.schema` and
:paramref:`_types.Enum.metadata` parameters are not provided.
The same behavior has been applied also to PostgreSQL
:class:`_postgresql.DOMAIN` type.
Fixes: #10594
Change-Id: Id3d819e3608974353e365cd063d9c5e40a071e73
Added support for the pow operator (``**``), with a default SQL
implementation of the ``POW()`` function. On Oracle Database, PostgreSQL
and MSSQL it renders as ``POWER()``. As part of this change, the operator
routes through a new first class ``func`` member :class:`_functions.pow`,
which renders on Oracle Database, PostgreSQL and MSSQL as ``POWER()``.
Fixes: #8579Closes: #8580
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8580
Pull-request-sha: 041b2ef474
Change-Id: I371bd44ed3e58f2d55ef705aeec7d04710c97f23
Austin Graham