341 Commits

Author SHA1 Message Date
Kadir Can Ozden 0f74af5c33 Fix WeakSequence.__getitem__ catching KeyError instead of IndexError
### Description

`WeakSequence.__getitem__` catches `KeyError` but the internal `_storage` is a `list`, which raises `IndexError` for out-of-range access. This means the `except KeyError` handler never executes, and the custom error message is never shown.

### Current behavior

```python
def __getitem__(self, index):
    try:
        obj = self._storage[index]  # _storage is a list
    except KeyError:  # lists don't raise KeyError
        raise IndexError("Index %s out of range" % index)
    else:
        return obj()
```

On an out-of-range index, the raw `IndexError` from list access propagates directly (e.g., `list index out of range`) instead of the intended custom message.

### Fix

Changed `except KeyError` to `except IndexError` so the handler actually catches the exception raised by list indexing.

Closes: #13136
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13136
Pull-request-sha: ceb8f9fe46

Change-Id: Ia36c2b9b0f3aec97624bcd2a9e49f43b9ed786d9
2026-02-23 22:12:46 +01:00
Mike Bayer e0f13cb6c1 forwards-port cpython issue 141560 for getfullargspec
Fixed issue when using ORM mappings with Python 3.14's :pep:`649` feature
that no longer requires "future annotations", where the ORM's introspection
of the ``__init__`` method of mapped classes would fail if non-present
identifiers in annotations were present.  The vendored ``getfullargspec()``
method has been amended to use ``Format.FORWARDREF`` under Python 3.14 to
prevent resolution of names that aren't present.

Fixes: #13104
References: https://github.com/python/cpython/issues/141560
Change-Id: I6af8026a07131d4a1e28cd7fc2e90509194ae957
2026-02-03 10:22:11 -05:00
Mike Bayer eff50d9e9b split ReadOnlyColumnCollection from writeable methods
The :class:`.ColumnCollection` class hierarchy has been refactored to allow
column names such as ``add``, ``remove``, ``update``, ``extend``, and
``clear`` to be used without conflicts. :class:`.ColumnCollection` is now
an abstract base class, with mutation operations moved to
:class:`.WriteableColumnCollection` and :class:`.DedupeColumnCollection`
subclasses. The :class:`.ReadOnlyColumnCollection` exposed as attributes
such as :attr:`.Table.c` no longer includes mutation methods that raised
:class:`.NotImplementedError`, allowing these common column names to be
accessed naturally, e.g. ``table.c.add``, ``table.c.remove``,
``table.c.update``, etc.

Change-Id: I22da8314fe7c451003e948d774040d86901bbca4
2026-01-30 09:59:36 -05:00
LucasMalor 11cb95b1d3 improved immutabledict merge_with and union
Now `immutabledict.merge_with is an alias of `immutabledict.union`,
both accept multiple arguments.
The methods now avoid doing copies of not required: if the method is
called only one `immutabledict` that's not empty it's returned.

Fixes: #13043

Closes: #13042
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13042
Pull-request-sha: bd53e48843

Change-Id: I8078f239e1ca36994b488b15f2fac40facf7f249
2026-01-01 13:42:11 -05:00
Mike Bayer 9a0d004331 filter_by works across multiple entities
The :meth:`_sql.Select.filter_by`, :meth:`_sql.Update.filter_by` and
:meth:`_sql.Delete.filter_by` methods now search across all entities
present in the statement, rather than limiting their search to only the
last joined entity or the first FROM entity. This allows these methods
to locate attributes unambiguously across multiple joined tables,
resolving issues where changing the order of operations such as
:meth:`_sql.Select.with_only_columns` would cause the method to fail.

If an attribute name exists in more than one FROM clause entity, an
:class:`_exc.AmbiguousColumnError` is now raised, indicating that
:meth:`_sql.Select.filter` (or :meth:`_sql.Select.where`) should be used
instead with explicit table-qualified column references.

Fixes: #8601
Change-Id: I6a46b8f4784801f95f7980ca8ef92f1947653572
2025-12-04 13:53:27 +00:00
Michael Bayer ebd118a1a2 Merge "Add DDL association to Table, CreateView support" into main 2025-11-11 14:48:55 +00:00
Mike Bayer b3fa4fae23 be more aggressive in removing all references
python3.14t is showing connections being referenced and not finalized.
explicit closes/disposes and such dont seem to be working but
ensuring globs are cleared out is working better

Change-Id: I0c53f7702027e37ad11bd6e15e828900c8f52213
2025-11-09 13:14:48 -05:00
Mike Bayer 98361f19a8 Add DDL association to Table, CreateView support
Added support for the SQL ``CREATE VIEW`` statement via the new
:class:`.CreateView` DDL class. The new class allows creating database
views from SELECT statements, with support for options such as
``TEMPORARY``, ``IF NOT EXISTS``, and ``MATERIALIZED`` where supported by
the target database. Views defined with :class:`.CreateView` integrate with
:class:`.MetaData` for automated DDL generation and provide a
:class:`.Table` object for querying.

this alters the CreateTableAs commit:

* adds the ability for Table to be associated with Create and Drop
  DDL constructs
* Adds CreateView variant of CreateTableAs
* Both associate themselves with Table so they take place in
  create_all/create/drop_all/drop

Fixes: #181
Change-Id: If3e568d3d6a6ce19e3d15198c3fbbe06bd847c83
2025-11-08 13:42:15 -05:00
Lysandros Nikolaou 456727df50 Add explicit multi-threaded tests and support free-threaded build
Implemented initial support for free-threaded Python by adding new tests
and reworking the test harness and GitHub Actions to include Python 3.13t
and Python 3.14t in test runs. Two concurrency issues have been identified
and fixed: the first involves initialization of the ``.c`` collection on a
``FromClause``, a continuation of 🎫`12302`, where an optional mutex
under free-threading is added; the second involves synchronization of the
pool "first_connect" event, which first received thread synchronization in
🎫`2964`, however under free-threading the creation of the mutex
itself runs under the same free-threading mutex. Initial pull request and
test suite courtesy Lysandros Nikolaou.

py313t: yes
py314t: yes
Fixes: #12881
Closes: #12882
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12882
Pull-request-sha: 53d65d96b9

Co-authored-by: Mike Bayer <mike_mp@zzzcomputng.com>
Change-Id: I2e4f2e9ac974ab6382cb0520cc446b396d9680a6
2025-10-02 12:14:28 -04:00
Mike Bayer 89a4174a8d Improve asyncpg exception hierarchy and asyncio hierarchies overall
The "emulated" exception hierarchies for the asyncio
drivers such as asyncpg, aiomysql, aioodbc, etc. have been standardized
on a common base :class:`.EmulatedDBAPIException`, which is now what's
available from the :attr:`.StatementException.orig` attribute on a
SQLAlchemy :class:`.DBAPIException` object.   Within :class:`.EmulatedDBAPIException`
and the subclasses in its hiearchy, the original driver-level exception is
also now avaliable via the :attr:`.EmulatedDBAPIException.orig` attribute,
and is also available from :class:`.DBAPIException` directly using the
:attr:`.DBAPIException.driver_exception` attribute.

Added additional emulated error classes for the subclasses of
``asyncpg.exception.IntegrityError`` including ``RestrictViolationError``,
``NotNullViolationError``, ``ForeignKeyViolationError``,
``UniqueViolationError`` ``CheckViolationError``,
``ExclusionViolationError``.  These exceptions are not directly thrown by
SQLAlchemy's asyncio emulation, however are available from the
newly added :attr:`.DBAPIException.driver_exception` attribute when a
:class:`.IntegrityError` is caught.

Fixes: #8047
Change-Id: I6a34e85b055265c087b0615f7c573be8582b3486
2025-09-23 09:49:19 -04:00
Mike Bayer fb3243475e remove _py3k suffixes from test files
on the theme of 2.0/2.1 are relatively similar right now, do
some more cleanup removing the py3k suffixes that we dont need anymore.

I'm not sure that the logic that would search for these suffixes is
even present anymore and I would guess we removed it when we removed
py2k support.  However, I am planning on possibly bringing it back
for some of the py314 stuff, so I still want to be able to do
suffix stuff.  that will be for another patch.

Change-Id: I929e6edd922f8d5f943acce77191fb1e3035b42c
2025-09-20 18:31:25 -04:00
Federico Caselli bb3c24595d remove python 3.9 support since it's close to eol
Fixes: #12819
Change-Id: I60f848226c1fef2a769845938d6afa3d3c5d0509
2025-08-23 14:22:57 +02:00
Mike Bayer b8b07a2f28 implement pep-649 workarounds, test suite passing for python 3.14
Changes to the test suite to accommodate Python 3.14 as of version
3.14.0b1

Originally this included a major breaking change to how python 3.14
implemented :pep:`649`, however this was resolved by [1].

As of a7, greenlet is skipped due to issues in a7 and later b1
in [2].

1. the change to rewrite all conditionals in annotation related tests
   is reverted.
2. test_memusage needed an explicit set_start_method() call so that
   it can continue to use plain fork
3. unfortunately at the moment greenlet has to be re-disabled for 3.14.
4. Changes to tox overall, remove pysqlcipher which hasn't worked
   in years, etc.
5. we need to support upcoming typing-extensions also, install the beta
6. 3.14.0a7 introduces major regressions to our runtime typing
   utilities, unfortunately, it's not clear if these can be resolved
7. for 3.14.0b1, we have to vendor get_annotations to work around [3]

[1] https://github.com/python/cpython/issues/130881
[2] https://github.com/python-greenlet/greenlet/issues/440
[3] https://github.com/python/cpython/issues/133684

py314: yes
Fixes: #12405
References: #12399
Change-Id: I8715d02fae599472dd64a2a46ccf8986239ecd99
2025-05-08 18:58:58 -04:00
Daraan 690e754b65 compatibility with typing_extensions 4.13 and type statement
Fixed regression caused by ``typing_extension==4.13.0`` that introduced
a different implementation for ``TypeAliasType`` while SQLAlchemy assumed
that it would be equivalent to the ``typing`` version.

Added test regarding generic TypeAliasType

Fixes: #12473
Closes: #12472
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12472
Pull-request-sha: 8861a5acfb

Change-Id: I053019a222546a625ed6d588314ae9f5b34c2f8a
2025-03-27 00:36:23 +01:00
Mike Bayer 8be3b096a6 syntax extensions (patch 1)
Added the ability to create custom SQL constructs that can define new
clauses within SELECT, INSERT, UPDATE, and DELETE statements without
needing to modify the construction or compilation code of of
:class:`.Select`, :class:`.Insert`, :class:`.Update`, or :class:`.Delete`
directly.  Support for testing these constructs, including caching support,
is present along with an example test suite.  The use case for these
constructs is expected to be third party dialects for NewSQL or other novel
styles of database that introduce new clauses to these statements.   A new
example suite is included which illustrates the ``QUALIFY`` SQL construct
used by several NewSQL databases which includes a cachable implementation
as well as a test suite.

Since these extensions start to make it a bit crowded with how many
kinds of "options" we have on statements, did some naming /
documentation changes with existing constructs on Executable, in
particular to distinguish ExecutableOption from SyntaxExtension.

Fixes: #12195
Change-Id: I4a44ee5bbc3d8b1b640837680c09d25b1b7077af
2025-03-05 16:03:16 -05:00
Mike Bayer fc3623990e support accept for chains of joineddispatchers
Fixed issue where creating an :class:`.Engine` using multiple calls to
:meth:`.Engine.execution_options` where a subsequent call involved certain
options such as ``isolation_level`` would lead to an internal error
involving event registration.

Fixes: #12289
Change-Id: Iec5fbc0eb0c5a92dda1ea762872ae992ca816685
2025-01-29 10:10:09 -05:00
Mike Bayer e4f0afe06b guard against KeyError on subclass removal
Fixed issue in event system which prevented an event listener from being
attached and detached from multiple class-like objects, namely the
:class:`.sessionmaker` or :class:`.scoped_session` targets that assign to
:class:`.Session` subclasses.

Fixes: #12216
Change-Id: I3d8969fe604adbc23add07a13741938c7f4fc8ca
2025-01-03 12:19:27 -05:00
Federico Caselli e6b0b421d6 General improvement on annotated declarative
Fix issue that resulted in inconsistent handing of unions
depending on how they were declared

Consistently support TypeAliasType. This has required a revision
of the implementation added in #11305 to have a consistent
behavior.

References: #11944
References: #11955
References: #11305
Change-Id: Iffc34fd42b9769f73ddb4331bd59b6b37391635d
2024-12-12 22:06:48 +01:00
Frazer McLean 40c30ec446 dont match partial types in type_annotation_map
Fixed issue regarding ``Union`` types that would be present in the
:paramref:`_orm.registry.type_annotation_map` of a :class:`_orm.registry`
or declarative base class, where a ``Mapped[]`` element that included one
of the subtypes present in that ``Union`` would be matched to that entry,
potentially ignoring other entries that matched exactly.   The correct
behavior now takes place such that an entry should only match in
``type_annotation_map`` exactly, as a ``Union`` type is a self-contained
type. For example, an attribute with ``Mapped[float]`` would previously
match to a ``type_annotation_map`` entry ``Union[float, Decimal]``; this
will no longer match and will now only match to an entry that states
``float``. Pull request courtesy Frazer McLean.

Fixes #11370
Closes: #11942
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/11942
Pull-request-sha: 21a3d1971a

Change-Id: I3467be00f8fa8bd011dd4805a77a3b80ff74a215
2024-12-10 18:30:45 -05:00
Federico Caselli 2adc79c077 Removed support for Python 3.8 since it's EOL.
Fixes: #12029
Change-Id: Ibb4efec9bab0225d03f6bf3fed661a3f2fc72cc7
2024-11-05 21:23:34 +01:00
Mike Bayer ef04a40110 loosen up hash_limit test
hash_limit_string works by doing a modulus of a hash value so that
the range of possible numbers is 0-N.  however, there's a chance
we might not populate every 0-N value in unusual cases on CI,
even after iterating 500 times apparently.  Loosen the change by making
sure we got at least N/2 unique hash messages but not greater
than N.

Change-Id: I5cd2845697ec0a718ddca1c95fbc4867b06eabee
2024-06-11 09:16:26 -04:00
Federico Caselli 0be89aaa38 Switch to cython pure python mode
Replaces the pyx files with py files that can be both compiled
by cython or imported as is by python.
This avoids the need of duplicating the code to have a python
only fallback.
The cython files are also reorganized to be in the module they use
instead of all being in the cyextension package, that has been
removed.

The performance is pretty much equal between main and this change.
A detailed comparison is at this link
https://docs.google.com/spreadsheets/d/1jkmGpnCyEcPyy6aRK9alElGjxlNHu44Wxjr4VrD99so/edit?usp=sharing

Change-Id: Iaed232ea5dfb41534cc9f58f6ea2f912a93263af
2024-03-25 23:44:27 +01:00
Federico Caselli 53dad790db add doctests to asyncio tutorial
Change-Id: I28c94a7bc1e7ae572af0d206b8e63a110dc6fd7a
2024-03-11 23:57:29 +01:00
Yurii Karabas 00072000c5 Add PEP 646 integration
The :class:`.Row` object now no longer makes use of an intermediary
``Tuple`` in order to represent its individual element types; instead,
the individual element types are present directly, via new :pep:`646`
integration, now available in more recent versions of Mypy.  Mypy
1.7 or greater is now required for statements, results and rows
to be correctly typed.   Pull request courtesy Yurii Karabas.

Fixes: #10635
Closes: #10634
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10634
Pull-request-sha: 430785c8a0

Change-Id: Ibd0ae31a98b4ea69dcb89f970e640920b2be6c48
2024-01-22 19:49:05 +01:00
Michael Bayer f660c73e7d Merge "Remove async_fallback mode" into main 2023-12-27 21:34:57 +00:00
Federico Caselli 3976537274 Remove async_fallback mode
Removed the async_fallback mode and await_fallback function.
Replace get_event_loop with Runner.
Removed the internal function ``await_fallback()``.
Renamed the internal function ``await_only()`` to ``await_()``.

Change-Id: Ib43829be6ebdb59b6c4447f5a15b5d2b81403fa9
2023-12-20 22:54:28 +01:00
Mike Bayer 99da5ebab3 use a standard function to check for iterable collections
Fixed 2.0 regression in :class:`.MutableList` where a routine that detects
sequences would not correctly filter out string or bytes instances, making
it impossible to assign a string value to a specific index (while
non-sequence values would work fine).

Fixes: #10784
Change-Id: I829cd2a1ef555184de8e6a752f39df65f69f6943
2023-12-20 15:41:54 -05:00
Mike Bayer 54a4ac0eb9 raise ImportError when greenlet not installed
This is the runtime raise when an async concurrency function
is called.  in 2.0 this raises ValueError, however here we've
standardized on raising ``ImportError`.

continuing for #10747, add a test asserting we dont get an
endless loop and get a clean ImportError instead when greenlet not
installed and async functions are used.

Fixes: #10747
Change-Id: I54dffe8577025e2ef3a59f5ca9ab7f4362d4d91f
2023-12-11 13:21:52 -05:00
Federico Caselli f5f08c28fb Don't import greenlet at all until it's needed
Added an initialize step to the import of
``sqlalchemy.ext.asyncio`` so that ``greenlet`` will
be imported only when the asyncio extension is first imported.
Alternatively, the ``greenlet`` library is still imported lazily on
first use to support use case that don't make direct use of the
SQLAlchemy asyncio extension.

Fixes: #10296
Change-Id: I97162a01aa29adb3e3fee97b718ab9567b2f6124
2023-11-13 14:28:34 -05:00
Federico Caselli 27e80ff061 Modernize Self-Referential example
References: #10115
Change-Id: Iee85b1277cdfee93ffb096df8ba85af48d09f1bc
2023-08-05 09:51:53 +00:00
Federico Caselli 59521abcc0 add 3.12
Initial fixes to test to accommodate py312

Here we are pulling in a current fork of greenlet that works
for python 3.12.   It works though there is one issue with aiosqlite
that might be related.

a non-voting py312 job is added to gerrit / jenkins

Fixes: #9819
Change-Id: I91a51dcbad2902f7c4c7cec88ebbf42c2417b512
2023-06-05 20:49:38 +02:00
Federico Caselli 058c230cea Update black to v23.3.0 and flake8 to v6
This change could be added to .git-blame-ignore-revs

Change-Id: I7ba10052b26bc3c178d23fb50a1123d0aae965ca
2023-05-30 21:44:26 +02:00
Mike Bayer 60b3119831 fix test suite warnings
fix a handful of warnings that were emitting but not raising,
usually because they were inside an "expect_warnings" block.

modify "expect_warnings" to always use "raise_on_any_unexpected"
behavior; remove this parameter.

Fixed issue in semi-private ``await_only()`` and ``await_fallback()``
concurrency functions where the given awaitable would remain un-awaited if
the function threw a ``GreenletError``, which could cause "was not awaited"
warnings later on if the program continued. In this case, the given
awaitable is now cancelled before the exception is thrown.

Change-Id: I33668c5e8c670454a3d879e559096fb873b57244
2023-05-09 20:23:48 -04:00
Federico Caselli 105f18be35 Performance improvement in Row
Various performance improvements to Row instanciation
- avoid passing processors if they are all None
- improve processor logic in cython
- improve tuplegetter using slices when contiguous indexes are used

Some timing follow.

In particular [base_]row_new_proc that tests using processors has
a 25% improvement compared to before in cython.
Looking at the [b]row_new_proc_none that test a list of processors
all None, this has 50% improvement in cython when passing the none list,
but in this patch it would usually be disabled by passing None, so the
performance gain is actually 90%, since it would run the case
[base_]row_new.

Tuplegetter is a bit faster in the single item get and when getting
sequential indexes (like indexes 1,2,3,4) at the cost of a bit
longer creation time in python, cython is mostly the same.

Current times
                    | python      | cython      | cy / py     |
base_row_new        | 0.639817400 | 0.118265500 | 0.184842582 |
row_new             | 0.680355100 | 0.129714600 | 0.190657202 |
base_row_new_proc   | 3.076538900 | 1.488428600 | 0.483799701 |
row_new_proc        | 3.119700100 | 1.532197500 | 0.491136151 |
brow_new_proc_none  | 1.917702300 | 0.475511500 | 0.247958977 |
row_new_proc_none   | 1.956253300 | 0.497803100 | 0.254467609 |

tuplegetter_one     | 0.152512600 | 0.148523900 | 0.973846751 |
tuplegetter_many    | 0.184394100 | 0.184511500 | 1.000636680 |
tuplegetter_seq     | 0.154832800 | 0.156270100 | 1.009282917 |
tuplegetter_new_one | 0.523730000 | 0.343402200 | 0.655685563 |
tuplegetter_new_many| 0.738924400 | 0.420961400 | 0.569694816 |
tuplegetter_new_seq | 1.062036900 | 0.495462000 | 0.466520514 |

Parent commit times
                    | python      | cython      | cy / py     |
base_row_new        | 0.643890800 | 0.113548300 | 0.176347138 |
row_new             | 0.674885900 | 0.124391800 | 0.184315304 |
base_row_new_proc   | 3.072020400 | 2.017367000 | 0.656690626 |
row_new_proc        | 3.109943400 | 2.048359400 | 0.658648450 |
brow_new_proc_none  | 1.967133700 | 1.006326000 | 0.511569702 |
row_new_proc_none   | 1.960814900 | 1.025217800 | 0.522852922 |

tuplegetter_one     | 0.197359900 | 0.205999000 | 1.043773330 |
tuplegetter_many    | 0.196575900 | 0.194888500 | 0.991416038 |
tuplegetter_seq     | 0.192723900 | 0.205635000 | 1.066992729 |
tuplegetter_new_one | 0.534644500 | 0.414311700 | 0.774929322 |
tuplegetter_new_many| 0.479376500 | 0.417448100 | 0.870814694 |
tuplegetter_new_seq | 0.481580200 | 0.412697900 | 0.856966088 |

Change-Id: I2ca1f49dca2beff625c283f1363c29c8ccc0c3f7
2023-04-26 19:48:00 +00:00
J. Nick Koston ff198e35f0 Prebuild the row string to position lookup for Rows
Improved :class:`_engine.Row` implementation to optimize
``__getattr__`` performance.
The serialization of a :class:`_engine.Row` to pickle has changed with
this change. Pickle saved by older SQLAlchemy versions can still be loaded,
but new pickle saved by this version cannot be loaded by older ones.

Fixes: #9678
Closes: #9668
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9668
Pull-request-sha: 86b8ccd195

Change-Id: Ia85c26a59e1a57ba2bf0d65578c6168f82a559f2
2023-04-26 20:19:17 +02:00
Federico Caselli a979b6dc5e Add missing methods to OrderedSet.
Implemented missing method ``copy`` and ``pop`` in OrderedSet class.

Fixes: #9487
Change-Id: I1d2278b64939b44422e9d5857ec7d345fff53997
2023-03-30 22:18:11 +02:00
Mike Bayer 24dd3d8c90 support DeclarativeBase for versioned history example
Fixed issue in "versioned history" example where using a declarative base
that is derived from :class:`_orm.DeclarativeBase` would fail to be mapped.
Additionally, repaired the given test suite so that the documented
instructions for running the example using Python unittest now work again.

Change-Id: I164a5b8dbdd01e3d815eb356f7b7cadf226ca296
References: #9546
2023-03-27 09:55:57 -04:00
Federico Caselli fd9aa84792 Fix regression when deserializing python rows into cython
Fixed regression involving pickling of Python rows between the cython and
pure Python implementations of :class:`.Row`, which occurred as part of
refactoring code for version 2.0 with typing. A particular constant were
turned into a string based ``Enum`` for the pure Python version of
:class:`.Row` whereas the cython version continued to use an integer
constant, leading to deserialization failures.

Regression occurred in a4bb502cf9

Fixes: #9423
Change-Id: Icbd85cacb2d589cef7c246de7064249926146f2e
2023-03-10 10:26:26 -05:00
Mike Bayer 3c6acaba01 port history meta to 2.0
first change:

Reworked the :ref:`examples_versioned_history` to work with
version 2.0, while at the same time improving the overall working of
this example to use newer APIs, including a newly added hook
:meth:`_orm.MapperEvents.after_mapper_constructed`.

second change:

Added new event hook :meth:`_orm.MapperEvents.after_mapper_constructed`,
which supplies an event hook to take place right as the
:class:`_orm.Mapper` object has been fully constructed, but before the
:meth:`_orm.registry.configure` call has been called. This allows code that
can create additional mappings and table structures based on the initial
configuration of a :class:`_orm.Mapper`, which also integrates within
Declarative configuration. Previously, when using Declarative, where the
:class:`_orm.Mapper` object is created within the class creation process,
there was no documented means of running code at this point.  The change
is to immediately benefit custom mapping schemes such as that
of the :ref:`examples_versioned_history` example, which generate additional
mappers and tables in response to the creation of mapped classes.

third change:

The infrequently used :attr:`_orm.Mapper.iterate_properties` attribute and
:meth:`_orm.Mapper.get_property` method, which are primarily used
internally, no longer implicitly invoke the :meth:`_orm.registry.configure`
process. Public access to these methods is extremely rare and the only
benefit to having :meth:`_orm.registry.configure` would have been allowing
"backref" properties be present in these collections. In order to support
the new :meth:`_orm.MapperEvents.after_mapper_constructed` event, iteration
and access to the internal :class:`_orm.MapperProperty` objects is now
possible without triggering an implicit configure of the mapper itself.

The more-public facing route to iteration of all mapper attributes, the
:attr:`_orm.Mapper.attrs` collection and similar, will still implicitly
invoke the :meth:`_orm.registry.configure` step thus making backref
attributes available.

In all cases, the :meth:`_orm.registry.configure` is always available to
be called directly.

fourth change:

Fixed obscure ORM inheritance issue caused by 🎫`8705` where some
scenarios of inheriting mappers that indicated groups of columns from the
local table and the inheriting table together under a
:func:`_orm.column_property` would nonetheless warn that properties of the
same name were being combined implicitly.

Fixes: #9220
Fixes: #9232
Change-Id: Id335b8e8071c8ea509c057c389df9dcd2059437d
2023-02-06 11:08:50 -05:00
Federico Caselli dce11383f8 Improve sql formatting
change {opensql} to {printsql} in prints, add missing markers

Change-Id: I07b72e6620bb64e329d6b641afa27631e91c4f16
2023-01-11 20:24:29 +01:00
Mike Bayer 3aeb30ea10 warn and skip for FKs that refer to invisible cols for Oracle
Supported use case for foreign key constraints where the local column is
marked as "invisible". The errors normally generated when a
:class:`.ForeignKeyConstraint` is created that check for the target column
are disabled when reflecting, and the constraint is skipped with a warning
in the same way which already occurs for an :class:`.Index` with a similar
issue.

tests are added for indexes, unique constraints, and primary key
constraints, which were already working; indexes and uniques warn,
primary keys don't which we would assume is because we never see those
PK columns in the first place.

Constraints now raise an informative ConstraintColumnNotFoundError
in the general case for strings in the "pending colargs" collection
not being resolvable.

Fixes: #9059
Change-Id: I400cf0bff6abba0e0c75f38b07617be1a8ec3453
2023-01-09 09:51:03 -05:00
Mike Bayer 1284fa377e disallow same-named columns, unchecked replacement in Table
Fixed issue where table reflection using :paramref:`.Table.extend_existing`
would fail to deduplicate a same-named column if the existing
:class:`.Table` used a separate key. The
:paramref:`.Table.autoload_replace` parameter would allow the column to be
skipped but under no circumstances should a :class:`.Table` ever have the
same-named column twice.

Additionally, changed deprecation warnings to exceptions
as were implemented in I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5 .

Fixes: #8925
Change-Id: I83d0f8658177a7ffbb06e01dbca91377d1a98d49
2022-12-04 10:45:35 -05:00
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
Mike Bayer d5be2cc139 perf improvements related to corresponding_column (2)
commit two of two. this reorganizes ColumnCollection
to build a new index up front that's used to optimize
the corresponding_column() method.

Additional performance enhancements within ORM-enabled SQL statements,
specifically targeting callcounts within the construction of ORM
statements, using combinations of :func:`_orm.aliased` with
:func:`_sql.union` and similar "compound" constructs, in addition to direct
performance improvements to the ``corresponding_column()`` internal method
that is used heavily by the ORM by constructs like :func:`_orm.aliased` and
similar.

Fixes: #8796
Change-Id: I4a76788007d5a802b9a4081e6a0f6e4b52497b50
2022-11-15 14:16:06 -05:00
Mike Bayer e3a8d19891 work around Python 3.11 IntEnum issue; update FastIntFlag
in [1], Python 3.11 seems to have changed the behavior of
IntEnum.  We didn't notice this because we have our own
workaround class already, but typing did.   Ensure we remain
compatible with IntFlag.

This change also modifies FastIntFlag to no longer use
global symbols; this is unnecessary as we assign FastIntFlag
members explicitly.  Use of ``symbol()`` should probably
be phased out.

[1] https://github.com/python/cpython/issues/99304
Fixes: #8783

Change-Id: I8ae2e871ff1467ae5ca1f63e66b5dae45d4a6c93
2022-11-10 12:19:02 -05:00
Mike Bayer b96321ae79 Support result.close() for all iterator patterns
This change contains new features for 2.0 only as well as some
behaviors that will be backported to 1.4.

For 1.4 and 2.0:

Fixed issue where the underlying DBAPI cursor would not be closed when
using :class:`_orm.Query` with :meth:`_orm.Query.yield_per` and direct
iteration, if a user-defined exception case were raised within the
iteration process, interrupting the iterator. This would lead to the usual
MySQL-related issues with server side cursors out of sync.

For 1.4 only:

A similar scenario can occur when using :term:`2.x` executions with direct
use of :class:`.Result`, in that case the end-user code has access to the
:class:`.Result` itself and should call :meth:`.Result.close` directly.
Version 2.0 will feature context-manager calling patterns to address this
use case.  However within the 1.4 scope, ensured that ``.close()`` methods
are available on all :class:`.Result` implementations including
:class:`.ScalarResult`, :class:`.MappingResult`.

For 2.0 only:

To better support the use case of iterating :class:`.Result` and
:class:`.AsyncResult` objects where user-defined exceptions may interrupt
the iteration, both objects as well as variants such as
:class:`.ScalarResult`, :class:`.MappingResult`,
:class:`.AsyncScalarResult`, :class:`.AsyncMappingResult` now support
context manager usage, where the result will be closed at the end of
iteration.

Corrected various typing issues within the engine and async engine
packages.

Fixes: #8710
Change-Id: I3166328bfd3900957eb33cbf1061d0495c9df670
2022-11-03 18:42:52 -04:00
Federico Caselli 25619f2783 Add pep 584 to python immutabledict fallback
Fixes: #8695
Change-Id: Ie0412c3a7b2b1ba5bd5112f204318ff763cbb8f4
2022-10-22 09:55:49 +02:00
Mike Bayer 276349200c implement write-only colletions, typing for dynamic
For 2.0, we provide a truly "larger than memory collection"
implementation, a write-only collection that will never
under any circumstances implicitly load the entire
collection, even during flush.

This is essentially a much more "strict" version
of the "dynamic" loader, which in fact has a lot of
scenarios that it loads the full backing collection
into memory, mostly defeating its purpose.

Typing constructs are added that support
both the new feature WriteOnlyMapping as well as the
legacy feature DynamicMapping.  These have been
integrated with "annotion based mapping" so that
relationship() uses these annotations to configure
the loader strategy as well.

additional changes:

* the docs triggered a conflict in hybrid's
  "transformers" section, this section is hard-coded
  to Query using a pattern that doesnt seem to have
  any use and isn't part of the current select()
  interface, so just removed this section

* As the docs for WriteOnlyMapping are very long,
  collections.rst is broken up into two pages now.

Fixes: #6229
Fixes: #7123
Change-Id: I6929f3da6e441cad92285e7309030a9bac4e429d
2022-10-06 00:36:25 +00:00
Mike Bayer b2c20d8a53 add disable doctest tag for autodoc test suite
ahead of trying to get everything formatted, some more
flexibility so that we can use doctest for all
python + sql code, while still being able to tell the
test suite to not run doctests on a sample.    All of the
"non-console python with SQL" in the docs is because I was
showing an example that I didn't want tested.

Change-Id: Iae876ae1ffd93c36b096c6c2d6048843ae9698c8
2022-10-01 22:36:03 -04:00
Mike Bayer 066aed6817 adjust tests for sqlites w/o returning
the sqlite builds on github actions seem to be very
inconsistent about versions and many don't support
RETURNING.  ensure any tests that depend on RETURNING present
are marked as such.

Change-Id: I7a60a81fa70b90642448cdd58eda33212c3afebc
2022-09-26 08:56:24 -04:00