Optimized :func:`_orm.selectinload` to skip the ``.unique()`` call on inner
result sets when no nested :func:`_orm.joinedload` on a collection is
present. The uniquing pass is only required when a joined eager load
inflates rows due to a one-to-many or many-to-many JOIN; in the common case
of a leaf selectin load, rows are already unique by construction and the
per-row hashing overhead can be avoided. As a side effect, ``yield_per``
set in a ``do_orm_execute`` event for a :func:`_orm.selectinload`
relationship load no longer raises ``InvalidRequestError`` when no nested
collection joinedload is in effect, since ``.unique()`` is no longer called
in that path. Pull request courtesy Oliver Parker.
`_SelectInLoader._load_via_parent` and `_load_via_child` currently call
`.unique()` unconditionally on the inner `Result`. The uniqueness pass is only
required when a nested `joinedload` on a collection is in effect — in that case
the `JOIN` inflates rows (one row per `(child, grandchild)` instead of one per
child) and the outer `groupby` would produce duplicated entries without dedup.
`loading.instances()` already signals exactly this condition: it sets
`Result._unique_filter_state` to a `require_unique` guard when the inner
compile state has `multi_row_eager_loaders=True`. When that flag is unset (no
nested collection `joinedload`), `_unique_filter_state` stays `None`, meaning
the inner query produces unique rows by construction:
- `omit_join` 1:N — one row per child
- `omit_join` M2O — one row per parent
- `omit_join` M2M — one row per (parent, entity)
- `load_with_join` (non-omit) — one row per (parent, child)
This PR makes the `.unique()` call conditional, via a new `_has_unique_filter`
property on `Result` that exposes this state without reaching into the private
`_unique_filter_state` attribute directly:
```python
if result._has_unique_filter:
result = result.unique()
```
Per-row hashing in `_iterator_getter` is avoided in the common leaf-load case.
On an in-memory SQLite bench (2000 parents × 5 children + M:N tags, Python
3.14, n=1000 iterations):
| Workload | before avg | after avg | delta | before sd | after sd |
|---|---|---|---|---|---|
| selectin children (1:N) | 56.0 ms | 41.4 ms | **−26%** | 5.2 ms | 1.9 ms |
| selectin tags M2M | 25.6 ms | 20.0 ms | **−22%** | 4.2 ms | 3.1 ms |
| joined children (1:N) | 42.9 ms | 37.6 ms | ~0% | — | — |
| plain parent load | 4.2 ms | 3.6 ms | ~0% | — | — |
**Behaviour change:** `yield_per` set in a `do_orm_execute` event for a
relationship load no longer raises
`InvalidRequestError("Can't use yield_per in conjunction with unique")` for
`selectinload` without a nested collection `joinedload` — because `.unique()`
is no longer called in that path. `immediateload` is unaffected (it still calls
`.unique()` unconditionally).
Fixes: #13339Closes: #13341
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13341
Pull-request-sha: 980af0383b
Change-Id: I12e993f104d354ae894f81f9b09fbcf9a95b0027
Fixes#13336.
`SelectState._get_display_froms` recomputed a loop-invariant `_cloned_intersection(...)` once per FROM element in each of the three correlation comprehensions, making each branch O(N²) in the number of FROM elements. This hoists the call so it runs once, which is O(N).
`_cloned_intersection` / `_cloned_difference` are pure and return a set, and neither argument changes during the comprehension, so the result is identical. A function-level benchmark asserts `old == new` at every N (full numbers in #13336), and `test/sql/` plus the ORM compilation/query tests pass: 7442 passed, 359 skipped. Net -14 lines.
Per the issue discussion, no changelog entry is included.
### Checklist
This pull request is:
- [x] A short code fix
- Issue with a runnable demonstration: #13336
- Behavior-preserving (no logic change), so it is covered by the existing `test/sql/` and ORM compilation/query suites rather than adding new tests.
Closes: #13337
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13337
Pull-request-sha: beba43e77a
Change-Id: I95b02ffa66ef709c16bf5275924ec03244c19ecb
Fixed issue where :meth:`.Result.freeze` would lose track of ambiguous
column names present in the original :class:`.CursorResult`, causing
key-based access on the thawed result to silently return a value instead of
raising :class:`.InvalidRequestError`. The
:class:`.SimpleResultMetaData` now accepts and propagates ambiguous key
information so that frozen, thawed, and pickled results raise consistently
for duplicate column names. Pull request courtesy Saurabh Kohli.
Fixes: #9427Closes: #13335
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13335
Pull-request-sha: c03904ece2
Change-Id: Ia184f77b442b069e6f9a4f94a967ead41a1704b6
Fixed regression caused by 🎫`8068` where a ``backref``
named ``'metadata'`` on a mapped class would cause an
``AssertionError`` when the class also used string-based
relationship references (e.g. ``secondary="some_table"``).
The ``_metadata_for_cls()`` helper now checks
``isinstance(meta, MetaData)`` as a condition rather than
asserting, falling back to ``registry.metadata`` when the
class attribute has been overwritten by a backref.
A warning is now emitted when a Declarative attribute name is named
``metadata`` or ``registry``. Previously, no warning was emitted for
``registry``, and using the name ``metadata`` would raise an
InvalidRequestError. Since these names can be used for attributes
that are mapped as backrefs or using imperative mappings, usage
under Declarative has been relaxed for ``metadata`` but also warns
for both names as they may have unintended interactions with the
Declarative reserved names.
References: https://bugs.launchpad.net/nova/+bug/2154165
References: https://github.com/sqlalchemy/sqlalchemy/discussions/8619Fixes: #13333
Change-Id: I0f3173bce9c8c8881bd6b8ea75102bb8ec92be8b
Fixed issue where :class:`_sql.StatementLambdaElement` would proxy
attribute access through the cached "expected" expression rather than the
resolved expression, causing stale closure-bound parameter values to be
used when a lambda statement was extended with non-lambda criteria such as
an additional ``.where()`` clause. Courtesy cjc0013.
Fixes: #10827Closes: #13327
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13327
Pull-request-sha: ec3e6735bf
Change-Id: I8a32c11f3da63109cf37c39541df8ebfee52b8c5
Fixed issue where :func:`_orm.subqueryload` combined with
:meth:`.PropComparator.of_type` and :meth:`.PropComparator.and_` would
silently drop the additional filter criteria, causing all related objects
to be loaded instead of only those matching the filter. The
:class:`.LoaderCriteriaOption` was being constructed against the base
entity rather than the effective entity indicated by
:meth:`.PropComparator.of_type`. Pull request courtesy Arya Rizky.
Fixes: #13207Closes: #13290
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13290
Pull-request-sha: b7a8617cde
Change-Id: I2c24652ec112511deaf39dbb9d6197e2097904ed
The :func:`.selectinload` loader strategy now selects the ``omit_join``
optimization for many-to-many non-self-referential relationships, reducing
the number of joins in the secondary SELECT by selecting from the secondary
table directly rather than joining back to the parent entity. ``omit_join``
is enabled automatically when the join condition determines that the
secondary table's foreign keys fully cover the parent's primary key. As
always, ``omit_join`` can be disabled by setting
:paramref:`.relationship.omit_join` to ``False``. Pull request courtesy
bekapono.
Fixes: #5987Closes: #13278
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13278
Pull-request-sha: fdd9847e7d
Change-Id: Ib68f8e2be1399222383cdd7b55793fe88402212c
Fixes#9256.
This updates the annotations for Session.bulk_insert_mappings() and Session.bulk_update_mappings().
The docstrings and runtime behavior already allow either a mapped class or a Mapper object, but the previous annotations only accepted Mapper[Any].
This patch switches those arguments to the existing _EntityBindKey alias, which matches the inputs accepted by _class_to_mapper(): mapped classes and Mapper objects, but not AliasedClass or AliasedInsp.
I also updated the internal _bulk_save_mappings() annotation so the public methods and the private helper stay consistent. The scoped_session proxy output has been kept in sync with tools/generate_proxy_methods.py, and the generator check passes.
I added a typing regression test covering both mapped classes and Mapper objects for the two bulk mapping methods. I confirmed that the mapped-class cases fail with the old annotation and pass with this change.
Checked locally:
python -m pytest -m mypy test/typing/test_mypy.py -k "session.py" -q
python -m mypy ./lib/sqlalchemy
python tools/generate_proxy_methods.py --check
python -m pytest test/orm/dml/test_bulk.py -q
python -m pytest -m mypy test/typing/test_mypy.py -k "not typed_queries.py" -q
I could not run the full typing suite locally because my local Python 3.12 environment does not include string.templatelib. I only skipped typed_queries.py; that file is expected to be covered by SQLAlchemy's Python 3.14 mypy CI job.
Closes: #13322
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13322
Pull-request-sha: bb730f3427
Change-Id: I4c5d516b3933b4e7fae9c844881a61f557a8bb5e
Fixed issue where the presence of a :meth:`.SessionEvents.do_orm_execute`
event hook would cause internal execution options such as ``yield_per`` and
loader-specific state from the first ``orm_pre_session_exec`` pass to leak
into the second pass, leading to errors when using relationship loaders
such as :func:`.selectinload` and :func:`.immediateload`. The execution
options passed to the second compilation pass are now based on the original
options plus only the explicit updates made via
:meth:`.ORMExecuteState.update_execution_options` within the event hook.
Fixes: #13301
Change-Id: Ide64d7202102930b68a2ab903054d538cd2f99dd
Updated the attribute :attr:`_orm.ORMExecuteState.user_defined_options` to
include options that were added to the statement before calling
:meth:`.Select.with_only_columns` or :meth:`_orm.Query.with_entities`.
Fixes: #13309
Change-Id: Ie6e3f46662542010f4d524820ae697638f36d459
Fixed issue where using :func:`_orm.with_polymorphic` on a leaf class (a
subclass with no further descendants) or a non-inherited class would fail
with an ``AttributeError`` when used in an ORM statement, due to
:func:`_orm.configure_mappers` not being triggered implicitly. The fix
ensures that :class:`.AliasedInsp` participates in the ``_post_inspect``
hook, triggering mapper configuration during ORM statement compilation.
Fixes: #13319
Change-Id: Ic5910474676be41f8c815dc72c38fca8e20cdeb9
Fixed issue where the :class:`.ExcludeConstraint` construct did not
correctly forward the :paramref:`.ExcludeConstraint.info` parameter to
the superclass, causing user-defined metadata to be lost. Pull request
courtesy Wiktor Byrka.
Fixes: #13317Closes: #13316
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13316
Pull-request-sha: be7f4fee2c
Change-Id: Idc4846f02127d1d39a8c638cb03b0379932e9fd6
Fixed issue where using :func:`_orm.joinedload` with
:meth:`.PropComparator.of_type` targeting a joined-table subclass combined
with :meth:`.PropComparator.and_` referencing a column on that subclass
would generate invalid SQL, where the subclass column was not adapted to
the subquery alias. Pull request courtesy Joaquin Hui Gomez.
Fixes#13203Closes: #13206
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13206
Pull-request-sha: ba55b0c3e2
Change-Id: I78fe4672649d1d5498e3bc653e5d943ccb55dafd
Fixed issue where floor division (``//``) between a :class:`.Float` or
:class:`.Numeric` numerator and an :class:`.Integer` denominator would omit
the ``FLOOR()`` SQL wrapper on dialects where
:attr:`.Dialect.div_is_floordiv` is ``True`` (the default, including
PostgreSQL and SQLite). ``FLOOR()`` is now applied if either the
denominator or the numerator is a non-integer, so that expressions such as
``float_col // int_col`` render as ``FLOOR(float_col / int_col)`` instead
of the incorrect ``float_col / int_col``. Pull request courtesy r266-tech.
Fixes: #10528Closes: #13191
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13191
Pull-request-sha: c9cbc47c87
Change-Id: I5f9f02d966aa6ccee214a2c5cc27a73a4292da03
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
Fixes: #13274
References: #9154
There were two relatively causes to some of the ~21 failures on PyPy:
- weakrefs may be deleted but the objects not finalized on PyPy. This manifests as `ref.obj() is None` I added a test for the `release()` case that also failed on CPython before the fix.
- a condition added in 2022 for missing sqllite3 behaviour is no longer necessary, and is now causing a failure
In order to run the changes in CI, I added PyPy to the PR CI run. Before merging I will revert that change. There are still a number of failures with PyPy around different error messages, different inspect.signatures and one sticky problem with the pure-python datetime.py that actually comes from CPython. I will continue to work on them, but they are not specific to sqlalchemy.
Note the CI run is ~6 minutes where the CPython ones are ~3 minutes. This is expected, since PyPy's JIT does not kick in on short tests, and the base compiler is about 2x slower.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #13276
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13276
Pull-request-sha: 00472f32f6
Change-Id: Id5d4ba37cf8db2345a948f973d7b1710910359a1
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
The `.fromisoformat()` error message tested in `test_no_string()` changed in Python 3.15, this fixes the test.
See #13308 for the `rel_2_0` branch.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
- [x] A short code fix (in a test, therefore I didn’t create an issue)
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #13307
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13307
Pull-request-sha: 90bc13fc41
Change-Id: I461d71b0fad18fa4f108102bb1c22c0e980fc70e
Fixed issue in aiomysql and asyncmy dialects that appears as of using
pymysql 1.2.0; the dialects were not properly taking into account logic
that detects the argument signature of pymysql's ``ping()`` method which
was added as part of 🎫`10492`.
We add a "does ping have reconnect" check for all three DBAPIs
individually. To suit asyncmy's use of cython we also needed to
adjust vendored getargspec() routines.
Fixes: #13306
Change-Id: Iad90ec6cfe9ee3b99736dd2153264090e7f76be1
Also resolved class-level MetaData not being consulted by the
declarative class registry when resolving string-based table
references. The registry now uses the same metadata resolution
logic as table creation, checking for a class-specific ``metadata``
attribute before falling back to ``registry.metadata``. The
``_metadata_for_cls`` function was factored into ``orm/util.py``
for shared use by both ``decl_base.py`` and ``clsregistry.py``.
Fixes: #8068Fixes: #13291
Change-Id: Ib846be0267f9295a5fee945dc6cf0a72c237bd2c
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
https://github.com/sqlalchemy/sqlalchemy/issues/8240 and https://github.com/sqlalchemy/sqlalchemy/pull/9834 added support for `NULLS NOT DISTINCT` to the PostgreSQL dialect, but didn't add it to the docs (only the change log). This adds a section to the "Constraint Options" section of the PostgreSQL dialect docs.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #13279
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13279
Pull-request-sha: cdc858cd88
Change-Id: I3f2c8fe346d3235fa8ba12c4d9ab712ddb840230
The unique() + yield_per combination was only blocked when yield_per
was set via execution_options(yield_per=N); calling these as methods
on the result (e.g. result.unique().yield_per(N)) bypassed the check
and silently produced incorrect results.
Restructured _unique_filters on SimpleResultMetaData to be a callable
_create_unique_filters that receives the Result, allowing it to check
the yield_per state regardless of how it was activated.
Fixes: #13293
Change-Id: I7e6a5e5b2e1d4c8f9a0b3d6e7f1c2a4d5b8e9f0a
The ``populate_existing`` execution option is now honored when passed
in the :paramref:`.Session.get.execution_options` dict by the method
:meth:`.Session.get` and analogous in other session kinds. The current
:paramref:`.Session.get.populate_existing` parameter will takes precedence
if specified, overriding the value of the execution options.
Fixes: #10610
Change-Id: I4ddc9a7c6dda8f31f4dd413b49a9196efb3edaa6
run each command in a try/except (they print out error messages
regardless) and report at the end on all individual runs.
Change-Id: I347c04f5c49c69daadf9f5f9e7c6c488cdf27f35
Added :class:`_sqlite.JSONB` type for SQLite's binary JSON storage
format, available as of SQLite version 3.45.0. Values are stored via
the ``jsonb()`` SQL function and retrieved via ``json()``, while the
Python-side behavior remains identical to :class:`_sqlite.JSON`.
Pull request courtesy Shamil Abdulaev.
Fixes: #13260
Adds `sqlalchemy.dialects.sqlite.JSONB` — a new dialect-specific type
for SQLite's binary JSON storage format, introduced in SQLite 3.45.0.
The type:
- renders `JSONB` in DDL (`CREATE TABLE t (col JSONB)`)
- wraps bind values with `jsonb()` on write, storing data as a BLOB
- wraps column reads with `json()`, returning standard text JSON to Python
- reflects back from the database as `sqlite.JSONB`
- behaves identically to `sqlite.JSON` on the Python side
SQLite 3.45.0 introduced a native binary JSON format (JSONB) that is
more compact and faster to parse than text JSON. Users who want to opt
into this storage format had no way to do so via SQLAlchemy.
- `JSONB` inherits from `sqlite.JSON` and defines `__visit_name__ = "JSONB"`
so the DDL compiler dispatches to `visit_JSONB` instead of `visit_JSON`
- Added `JSONB: JSONB` to `colspecs` so the type is not remapped to
`_SQliteJson` via the `sqltypes.JSON` MRO entry
- `bind_expression` / `column_expression` apply `jsonb()` / `json()`
transparently; existing `result_processor` from `sqltypes.JSON` handles
deserialization without changes
- Added `sqlite_jsonb` requirement (checks for SQLite >= 3.45 and that
`jsonb()` is available, since it is a loadable extension)
`test/dialect/sqlite/test_types.py::JSONBTest` (8 tests):
- DDL renders `JSONB`
- Reflection returns `sqlite.JSONB`
- Round-trip read/write including `None`
- Sub-object extraction via `[]` indexing
- Compiled SQL shows `jsonb(?)` on INSERT and `json(col)` on SELECT
- `typeof(col)` returns `blob` confirming binary storage
Closes: #13261
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13261
Pull-request-sha: 81b93af698
Change-Id: Ic38704674d30aa3d1bb5ce1e8ef5e4b0562ad91a
mypy just went to 2.0.
and there seems to be...
exactly one "type: ignore" to remove and...that's it?
well OK!
Change-Id: I29f919641acc0e970b566c850063db7ecad70ed9
Federico Caselli