Optimized :func:`_orm.selectinload` to skip the ``.unique()`` call on inner
result sets when no nested :func:`_orm.joinedload` on a collection is
present. The uniquing pass is only required when a joined eager load
inflates rows due to a one-to-many or many-to-many JOIN; in the common case
of a leaf selectin load, rows are already unique by construction and the
per-row hashing overhead can be avoided. As a side effect, ``yield_per``
set in a ``do_orm_execute`` event for a :func:`_orm.selectinload`
relationship load no longer raises ``InvalidRequestError`` when no nested
collection joinedload is in effect, since ``.unique()`` is no longer called
in that path. Pull request courtesy Oliver Parker.
`_SelectInLoader._load_via_parent` and `_load_via_child` currently call
`.unique()` unconditionally on the inner `Result`. The uniqueness pass is only
required when a nested `joinedload` on a collection is in effect — in that case
the `JOIN` inflates rows (one row per `(child, grandchild)` instead of one per
child) and the outer `groupby` would produce duplicated entries without dedup.
`loading.instances()` already signals exactly this condition: it sets
`Result._unique_filter_state` to a `require_unique` guard when the inner
compile state has `multi_row_eager_loaders=True`. When that flag is unset (no
nested collection `joinedload`), `_unique_filter_state` stays `None`, meaning
the inner query produces unique rows by construction:
- `omit_join` 1:N — one row per child
- `omit_join` M2O — one row per parent
- `omit_join` M2M — one row per (parent, entity)
- `load_with_join` (non-omit) — one row per (parent, child)
This PR makes the `.unique()` call conditional, via a new `_has_unique_filter`
property on `Result` that exposes this state without reaching into the private
`_unique_filter_state` attribute directly:
```python
if result._has_unique_filter:
result = result.unique()
```
Per-row hashing in `_iterator_getter` is avoided in the common leaf-load case.
On an in-memory SQLite bench (2000 parents × 5 children + M:N tags, Python
3.14, n=1000 iterations):
| Workload | before avg | after avg | delta | before sd | after sd |
|---|---|---|---|---|---|
| selectin children (1:N) | 56.0 ms | 41.4 ms | **−26%** | 5.2 ms | 1.9 ms |
| selectin tags M2M | 25.6 ms | 20.0 ms | **−22%** | 4.2 ms | 3.1 ms |
| joined children (1:N) | 42.9 ms | 37.6 ms | ~0% | — | — |
| plain parent load | 4.2 ms | 3.6 ms | ~0% | — | — |
**Behaviour change:** `yield_per` set in a `do_orm_execute` event for a
relationship load no longer raises
`InvalidRequestError("Can't use yield_per in conjunction with unique")` for
`selectinload` without a nested collection `joinedload` — because `.unique()`
is no longer called in that path. `immediateload` is unaffected (it still calls
`.unique()` unconditionally).
Fixes: #13339Closes: #13341
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13341
Pull-request-sha: 980af0383b
Change-Id: I12e993f104d354ae894f81f9b09fbcf9a95b0027
Fixed issue where :meth:`.Result.freeze` would lose track of ambiguous
column names present in the original :class:`.CursorResult`, causing
key-based access on the thawed result to silently return a value instead of
raising :class:`.InvalidRequestError`. The
:class:`.SimpleResultMetaData` now accepts and propagates ambiguous key
information so that frozen, thawed, and pickled results raise consistently
for duplicate column names. Pull request courtesy Saurabh Kohli.
Fixes: #9427Closes: #13335
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13335
Pull-request-sha: c03904ece2
Change-Id: Ia184f77b442b069e6f9a4f94a967ead41a1704b6
Fixed regression caused by 🎫`8068` where a ``backref``
named ``'metadata'`` on a mapped class would cause an
``AssertionError`` when the class also used string-based
relationship references (e.g. ``secondary="some_table"``).
The ``_metadata_for_cls()`` helper now checks
``isinstance(meta, MetaData)`` as a condition rather than
asserting, falling back to ``registry.metadata`` when the
class attribute has been overwritten by a backref.
A warning is now emitted when a Declarative attribute name is named
``metadata`` or ``registry``. Previously, no warning was emitted for
``registry``, and using the name ``metadata`` would raise an
InvalidRequestError. Since these names can be used for attributes
that are mapped as backrefs or using imperative mappings, usage
under Declarative has been relaxed for ``metadata`` but also warns
for both names as they may have unintended interactions with the
Declarative reserved names.
References: https://bugs.launchpad.net/nova/+bug/2154165
References: https://github.com/sqlalchemy/sqlalchemy/discussions/8619Fixes: #13333
Change-Id: I0f3173bce9c8c8881bd6b8ea75102bb8ec92be8b
Fixed issue where :class:`_sql.StatementLambdaElement` would proxy
attribute access through the cached "expected" expression rather than the
resolved expression, causing stale closure-bound parameter values to be
used when a lambda statement was extended with non-lambda criteria such as
an additional ``.where()`` clause. Courtesy cjc0013.
Fixes: #10827Closes: #13327
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13327
Pull-request-sha: ec3e6735bf
Change-Id: I8a32c11f3da63109cf37c39541df8ebfee52b8c5
Fixed issue where :func:`_orm.subqueryload` combined with
:meth:`.PropComparator.of_type` and :meth:`.PropComparator.and_` would
silently drop the additional filter criteria, causing all related objects
to be loaded instead of only those matching the filter. The
:class:`.LoaderCriteriaOption` was being constructed against the base
entity rather than the effective entity indicated by
:meth:`.PropComparator.of_type`. Pull request courtesy Arya Rizky.
Fixes: #13207Closes: #13290
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13290
Pull-request-sha: b7a8617cde
Change-Id: I2c24652ec112511deaf39dbb9d6197e2097904ed
The :func:`.selectinload` loader strategy now selects the ``omit_join``
optimization for many-to-many non-self-referential relationships, reducing
the number of joins in the secondary SELECT by selecting from the secondary
table directly rather than joining back to the parent entity. ``omit_join``
is enabled automatically when the join condition determines that the
secondary table's foreign keys fully cover the parent's primary key. As
always, ``omit_join`` can be disabled by setting
:paramref:`.relationship.omit_join` to ``False``. Pull request courtesy
bekapono.
Fixes: #5987Closes: #13278
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13278
Pull-request-sha: fdd9847e7d
Change-Id: Ib68f8e2be1399222383cdd7b55793fe88402212c
Fixed issue where the presence of a :meth:`.SessionEvents.do_orm_execute`
event hook would cause internal execution options such as ``yield_per`` and
loader-specific state from the first ``orm_pre_session_exec`` pass to leak
into the second pass, leading to errors when using relationship loaders
such as :func:`.selectinload` and :func:`.immediateload`. The execution
options passed to the second compilation pass are now based on the original
options plus only the explicit updates made via
:meth:`.ORMExecuteState.update_execution_options` within the event hook.
Fixes: #13301
Change-Id: Ide64d7202102930b68a2ab903054d538cd2f99dd
Updated the attribute :attr:`_orm.ORMExecuteState.user_defined_options` to
include options that were added to the statement before calling
:meth:`.Select.with_only_columns` or :meth:`_orm.Query.with_entities`.
Fixes: #13309
Change-Id: Ie6e3f46662542010f4d524820ae697638f36d459
Fixed issue where using :func:`_orm.with_polymorphic` on a leaf class (a
subclass with no further descendants) or a non-inherited class would fail
with an ``AttributeError`` when used in an ORM statement, due to
:func:`_orm.configure_mappers` not being triggered implicitly. The fix
ensures that :class:`.AliasedInsp` participates in the ``_post_inspect``
hook, triggering mapper configuration during ORM statement compilation.
Fixes: #13319
Change-Id: Ic5910474676be41f8c815dc72c38fca8e20cdeb9
Fixed issue where the :class:`.ExcludeConstraint` construct did not
correctly forward the :paramref:`.ExcludeConstraint.info` parameter to
the superclass, causing user-defined metadata to be lost. Pull request
courtesy Wiktor Byrka.
Fixes: #13317Closes: #13316
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13316
Pull-request-sha: be7f4fee2c
Change-Id: Idc4846f02127d1d39a8c638cb03b0379932e9fd6
Fixed issue where using :func:`_orm.joinedload` with
:meth:`.PropComparator.of_type` targeting a joined-table subclass combined
with :meth:`.PropComparator.and_` referencing a column on that subclass
would generate invalid SQL, where the subclass column was not adapted to
the subquery alias. Pull request courtesy Joaquin Hui Gomez.
Fixes#13203Closes: #13206
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13206
Pull-request-sha: ba55b0c3e2
Change-Id: I78fe4672649d1d5498e3bc653e5d943ccb55dafd
Fixed issue where floor division (``//``) between a :class:`.Float` or
:class:`.Numeric` numerator and an :class:`.Integer` denominator would omit
the ``FLOOR()`` SQL wrapper on dialects where
:attr:`.Dialect.div_is_floordiv` is ``True`` (the default, including
PostgreSQL and SQLite). ``FLOOR()`` is now applied if either the
denominator or the numerator is a non-integer, so that expressions such as
``float_col // int_col`` render as ``FLOOR(float_col / int_col)`` instead
of the incorrect ``float_col / int_col``. Pull request courtesy r266-tech.
Fixes: #10528Closes: #13191
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13191
Pull-request-sha: c9cbc47c87
Change-Id: I5f9f02d966aa6ccee214a2c5cc27a73a4292da03
Fixed issue in aiomysql and asyncmy dialects that appears as of using
pymysql 1.2.0; the dialects were not properly taking into account logic
that detects the argument signature of pymysql's ``ping()`` method which
was added as part of 🎫`10492`.
We add a "does ping have reconnect" check for all three DBAPIs
individually. To suit asyncmy's use of cython we also needed to
adjust vendored getargspec() routines.
Fixes: #13306
Change-Id: Iad90ec6cfe9ee3b99736dd2153264090e7f76be1
Also resolved class-level MetaData not being consulted by the
declarative class registry when resolving string-based table
references. The registry now uses the same metadata resolution
logic as table creation, checking for a class-specific ``metadata``
attribute before falling back to ``registry.metadata``. The
``_metadata_for_cls`` function was factored into ``orm/util.py``
for shared use by both ``decl_base.py`` and ``clsregistry.py``.
Fixes: #8068Fixes: #13291
Change-Id: Ib846be0267f9295a5fee945dc6cf0a72c237bd2c
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
https://github.com/sqlalchemy/sqlalchemy/issues/8240 and https://github.com/sqlalchemy/sqlalchemy/pull/9834 added support for `NULLS NOT DISTINCT` to the PostgreSQL dialect, but didn't add it to the docs (only the change log). This adds a section to the "Constraint Options" section of the PostgreSQL dialect docs.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #13279
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13279
Pull-request-sha: cdc858cd88
Change-Id: I3f2c8fe346d3235fa8ba12c4d9ab712ddb840230
The unique() + yield_per combination was only blocked when yield_per
was set via execution_options(yield_per=N); calling these as methods
on the result (e.g. result.unique().yield_per(N)) bypassed the check
and silently produced incorrect results.
Restructured _unique_filters on SimpleResultMetaData to be a callable
_create_unique_filters that receives the Result, allowing it to check
the yield_per state regardless of how it was activated.
Fixes: #13293
Change-Id: I7e6a5e5b2e1d4c8f9a0b3d6e7f1c2a4d5b8e9f0a
The ``populate_existing`` execution option is now honored when passed
in the :paramref:`.Session.get.execution_options` dict by the method
:meth:`.Session.get` and analogous in other session kinds. The current
:paramref:`.Session.get.populate_existing` parameter will takes precedence
if specified, overriding the value of the execution options.
Fixes: #10610
Change-Id: I4ddc9a7c6dda8f31f4dd413b49a9196efb3edaa6
Added :class:`_sqlite.JSONB` type for SQLite's binary JSON storage
format, available as of SQLite version 3.45.0. Values are stored via
the ``jsonb()`` SQL function and retrieved via ``json()``, while the
Python-side behavior remains identical to :class:`_sqlite.JSON`.
Pull request courtesy Shamil Abdulaev.
Fixes: #13260
Adds `sqlalchemy.dialects.sqlite.JSONB` — a new dialect-specific type
for SQLite's binary JSON storage format, introduced in SQLite 3.45.0.
The type:
- renders `JSONB` in DDL (`CREATE TABLE t (col JSONB)`)
- wraps bind values with `jsonb()` on write, storing data as a BLOB
- wraps column reads with `json()`, returning standard text JSON to Python
- reflects back from the database as `sqlite.JSONB`
- behaves identically to `sqlite.JSON` on the Python side
SQLite 3.45.0 introduced a native binary JSON format (JSONB) that is
more compact and faster to parse than text JSON. Users who want to opt
into this storage format had no way to do so via SQLAlchemy.
- `JSONB` inherits from `sqlite.JSON` and defines `__visit_name__ = "JSONB"`
so the DDL compiler dispatches to `visit_JSONB` instead of `visit_JSON`
- Added `JSONB: JSONB` to `colspecs` so the type is not remapped to
`_SQliteJson` via the `sqltypes.JSON` MRO entry
- `bind_expression` / `column_expression` apply `jsonb()` / `json()`
transparently; existing `result_processor` from `sqltypes.JSON` handles
deserialization without changes
- Added `sqlite_jsonb` requirement (checks for SQLite >= 3.45 and that
`jsonb()` is available, since it is a loadable extension)
`test/dialect/sqlite/test_types.py::JSONBTest` (8 tests):
- DDL renders `JSONB`
- Reflection returns `sqlite.JSONB`
- Round-trip read/write including `None`
- Sub-object extraction via `[]` indexing
- Compiled SQL shows `jsonb(?)` on INSERT and `json(col)` on SELECT
- `typeof(col)` returns `blob` confirming binary storage
Closes: #13261
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13261
Pull-request-sha: 81b93af698
Change-Id: Ic38704674d30aa3d1bb5ce1e8ef5e4b0562ad91a
Changed the default backslash escape value in the PostgreSQL dialect to
``False`` to align it with the default value of
``standard_conforming_strings=on``. This change should not affect most users
since the value is set at driver initialization on first connect.
Fixes: #13268
Change-Id: I9b7986f1ee466fab3cab88e3f6117e313e3376cd
Added :paramref:`.selectinload.chunksize` parameter to :func`.selectinload`
allowing users to configure the number of primary keys sent per IN clause
when loading reltaionships. Pull request courtesy bekapono.
Fixes: #11450Closes: #13235
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13235
Pull-request-sha: 360585a48c
Change-Id: Id09776b7ba53c630a780f128fc67dfdc085a4062
Narrowed the scope of the internal workaround for MySQL bugs `#88718
<https://bugs.mysql.com/bug.php?id=88718>`_ and `#96365
<https://bugs.mysql.com/bug.php?id=96365>`_ so that it is only applied
where needed: MySQL 8.0.1 through 8.0.13 (where bug 88718 is present), and
on systems with ``lower_case_table_names=2`` (where bug 96365 applies,
typically macOS). Previously the workaround was applied unconditionally
for all MySQL 8.0+ versions, which caused a ``KeyError`` during foreign key
reflection when the database user lacked SELECT privileges on referred
tables.
Fixes: #13243
Change-Id: I7c29f67d1653c5cd32f29e098f038fea1d56117b
Fixed issue where the asyncpg driver could throw an insufficiently-handled
exception ``InternalClientError`` under some circumstances, leading to
connections not being properly marked as invalidated.
Fixes: #13241
References: https://github.com/MagicStack/asyncpg/issues/1069
Change-Id: Iaaf551b3d7b062cce62e13b441161583a484615f
Improve handling of two phase transaction identifiers for PostgreSQL
when the identifier is provided by the user.
As part of this change the psycopg dialect was updated to use the DBAPI
two phase transaction API instead of executing the SQL directly.
Fixes: #13229
Change-Id: If8301a7253b4a0c88e5323c9a052c3a9fa258780
Added support for the :class:`_sqltypes.JSON` datatype when using the
Oracle database with the oracledb dialect. JSON values are serialized and
deserialized using configurable strategies that accommodate Oracle's native
JSON type available as of Oracle 21c. Pull request courtesy Abdallah
Alhadad.
This fix also includes new dialect-level indicators for JSON support;
some attention given to issue #13213 indicates we can close that issue.
Fixes: #10375Closes: #13065
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13065
Pull-request-sha: 9a89237f4f
Change-Id: I8cbe35bc632dc9419642ddca8bf4ba9c20c0ae37
Federico Caselli