Use cython for the internal result methods. This improves the speed
of the row processing of about 15-30% depending on cases.
Change-Id: Ia19c1a61aa1a55405b8327f88b1ee02627c4217a
this DB is extremely erratic in being able to connect. Add
a brute force connection retrier to all engines everywhere
(which for oracledb we can fortunately use their built-in feature
that also works).
This actually works and I can see it pausing under load, reconnecting,
and succeeding. the problem is that absolutely every engine everywhere
needs this routine otherwise an engine without a retrier in it will
crash. That then necessitates digging into testing_engine(),
making sure testing_engine() is used everywhere an engine that's going
to connect is used, then dealing with the fallout from that.
We also simplify some older workarounds for cx_oracle and
hack into config/provision to make oracledb seem like the primary
DBAPI for most tests.
testing_engine has been completely overhauled, making use of a new
post_configure_testing_engine() hook which moves and refines
the SQLite pool sharing and savepoint logic all into sqlite/provision.py
and also allows for cx_oracle to apply a retry event handler.
Change-Id: I4ea4c523effb878290d28b94d8925eb32fc5ae3b
adds the ORM profiling tests to backend so that they run
on gerrits that use backendonly for nocext
fix CI only failures
Change-Id: Id36a9bcef1250af58f968a62edb3e7b7cb84f673
GC seems to be working in fewer cases across different interpreters,
so add a new step to force an InstanceState to act as though
it got dereferenced, for those cases where we have del'ed the object,
done a gc collect, and we know it should not be reachable.
Change-Id: Ie97af1d5739f8d5a24aabe1296f14ce99ec1b0e5
Implemented initial support for free-threaded Python by adding new tests
and reworking the test harness and GitHub Actions to include Python 3.13t
and Python 3.14t in test runs. Two concurrency issues have been identified
and fixed: the first involves initialization of the ``.c`` collection on a
``FromClause``, a continuation of 🎫`12302`, where an optional mutex
under free-threading is added; the second involves synchronization of the
pool "first_connect" event, which first received thread synchronization in
🎫`2964`, however under free-threading the creation of the mutex
itself runs under the same free-threading mutex. Initial pull request and
test suite courtesy Lysandros Nikolaou.
py313t: yes
py314t: yes
Fixes: #12881Closes: #12882
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12882
Pull-request-sha: 53d65d96b9
Co-authored-by: Mike Bayer <mike_mp@zzzcomputng.com>
Change-Id: I2e4f2e9ac974ab6382cb0520cc446b396d9680a6
Changes to the test suite to accommodate Python 3.14 as of version
3.14.0b1
Originally this included a major breaking change to how python 3.14
implemented :pep:`649`, however this was resolved by [1].
As of a7, greenlet is skipped due to issues in a7 and later b1
in [2].
1. the change to rewrite all conditionals in annotation related tests
is reverted.
2. test_memusage needed an explicit set_start_method() call so that
it can continue to use plain fork
3. unfortunately at the moment greenlet has to be re-disabled for 3.14.
4. Changes to tox overall, remove pysqlcipher which hasn't worked
in years, etc.
5. we need to support upcoming typing-extensions also, install the beta
6. 3.14.0a7 introduces major regressions to our runtime typing
utilities, unfortunately, it's not clear if these can be resolved
7. for 3.14.0b1, we have to vendor get_annotations to work around [3]
[1] https://github.com/python/cpython/issues/130881
[2] https://github.com/python-greenlet/greenlet/issues/440
[3] https://github.com/python/cpython/issues/133684
py314: yes
Fixes: #12405
References: #12399
Change-Id: I8715d02fae599472dd64a2a46ccf8986239ecd99
Identified some unnecessary cycles and overhead in how
this is implemented. since we want to add this to Select,
needs these improvements.
Change-Id: I4324db14aaf52ab87a8b7fa49ebf1b6624bc2dcb
The ``.c`` and ``.columns`` attributes on the :class:`.Select` and
:class:`.TextualSelect` constructs, which are not instances of
:class:`.FromClause`, have been removed completely, in addition to the
``.select()`` method as well as other codepaths which would implicitly
generate a subquery from a :class:`.Select` without the need to explicitly
call the :meth:`.Select.subquery` method.
In the case of ``.c`` and ``.columns``, these attributes were never useful
in practice and have caused a great deal of confusion, hence were
deprecated back in version 1.4, and have emitted warnings since that
version. Accessing the columns that are specific to a :class:`.Select`
construct is done via the :attr:`.Select.selected_columns` attribute, which
was added in version 1.4 to suit the use case that users often expected
``.c`` to accomplish. In the larger sense, implicit production of
subqueries works against SQLAlchemy's modern practice of making SQL
structure as explicit as possible.
Note that this is **not related** to the usual :attr:`.FromClause.c` and
:attr:`.FromClause.columns` attributes, common to objects such as
:class:`.Table` and :class:`.Subquery`, which are unaffected by this
change.
Fixes: #10236
Change-Id: If241b8674ccacce7e860bfed25b5d266bfe1aca7
Replaces the pyx files with py files that can be both compiled
by cython or imported as is by python.
This avoids the need of duplicating the code to have a python
only fallback.
The cython files are also reorganized to be in the module they use
instead of all being in the cyextension package, that has been
removed.
The performance is pretty much equal between main and this change.
A detailed comparison is at this link
https://docs.google.com/spreadsheets/d/1jkmGpnCyEcPyy6aRK9alElGjxlNHu44Wxjr4VrD99so/edit?usp=sharing
Change-Id: Iaed232ea5dfb41534cc9f58f6ea2f912a93263af
Adjusted the fix made in 🎫`10570`, released in 2.0.23, where new
logic was added to reconcile possibly changing bound parameter values
across cache key generations used within the :func:`_orm.with_expression`
construct. The new logic changes the approach by which the new bound
parameter values are associated with the statement, avoiding the need to
deep-copy the statement which can result in a significant performance
penalty for very deep / complex SQL constructs. The new approach no longer
requires this deep-copy step.
Fixes: #11085
Change-Id: Ia51eb4e949c8f37af135399925a9916b9ed4ad2f
Extract a fixture to run mypy on files
Move the plain files to test/typing
Move test files from stubs repository
Transform the fixture module in a package
Change-Id: I23acaecb84e7c4b9010259d44395dc1df83a9385
A deprecation warning is emitted whenever a property is added to a
:class:`_orm.Mapper` where an ORM mapped property were already configured,
or an attribute is already present on the class. Previously, there was a
non-deprecation warning for this case that did not emit consistently. The
logic for this warning has been improved so that it detects end-user
replacement of attribute while not having false positives for internal
Declarative and other cases where replacement of descriptors with new ones
is expected.
Fixes: #9841
Change-Id: I21ef25571466eeca3fedbec71612f7553fdbf2bf
fix a handful of warnings that were emitting but not raising,
usually because they were inside an "expect_warnings" block.
modify "expect_warnings" to always use "raise_on_any_unexpected"
behavior; remove this parameter.
Fixed issue in semi-private ``await_only()`` and ``await_fallback()``
concurrency functions where the given awaitable would remain un-awaited if
the function threw a ``GreenletError``, which could cause "was not awaited"
warnings later on if the program continued. In this case, the given
awaitable is now cancelled before the exception is thrown.
Change-Id: I33668c5e8c670454a3d879e559096fb873b57244
Improved :class:`_engine.Row` implementation to optimize
``__getattr__`` performance.
The serialization of a :class:`_engine.Row` to pickle has changed with
this change. Pickle saved by older SQLAlchemy versions can still be loaded,
but new pickle saved by this version cannot be loaded by older ones.
Fixes: #9678Closes: #9668
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9668
Pull-request-sha: 86b8ccd195
Change-Id: Ia85c26a59e1a57ba2bf0d65578c6168f82a559f2
This adds the very small plugin flake8-import-single which
will prevent us from having an import with more than one symbol
on a line.
Flake8 by itself prevents this pattern with E401:
import collections, os, sys
However does not do anything with this:
from sqlalchemy import Column, text
Both statements have the same issues generating merge artifacts
as well as presenting a manual decision to be made. While
zimports generally cleans up such imports at the top level, we
don't enforce zimports / pre-commit use.
the plugin finds the same issue for imports that are inside of
test methods. We shouldn't usually have imports in test methods
so most of them here are moved to be top level.
The version is pinned at 0.1.5; the project seems to have no
activity since 2019, however there are three 0.1.6dev releases
on pypi which stopped in September 2019, they seem to be
experiments with packaging. The source for 0.1.5
is extremely simple and only reveals one method to flake8
(the run() method).
Change-Id: Icea894e43bad9c0b5d4feb5f49c6c666d6ea6aa1
py311 may be more sensitive here, or maybe the machines are
acting differently these days, in any case move
memory / GC sensitive tests to test_memusage.
Change-Id: I218295150efc2f7ea88da9960ff10fda63dc60b1
sporadic (and at the moment persistent) test failures
related to aiosqlite seem to have in common that Python
gc stops working fully when we run a lot of tests with
aiosqlite. The failures are not limited to aiosqlite
as they are more involving places where we assume or
expect gc.collect() to get rid of things, and it doesn't.
Identify (based on reproducible case on the d3 CI runner)
the spots where this happens and add fixes.
test/orm/test_transaction.py test_gced_delete_on_rollback
has always been a very sensitive test with a lot of issues,
so here we move it to the test_memusage suite and limit
it only to when the memusage suite is running.
Change-Id: I683412d0effe8732c45980b40722e5bb63431177
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
commit two of two. this reorganizes ColumnCollection
to build a new index up front that's used to optimize
the corresponding_column() method.
Additional performance enhancements within ORM-enabled SQL statements,
specifically targeting callcounts within the construction of ORM
statements, using combinations of :func:`_orm.aliased` with
:func:`_sql.union` and similar "compound" constructs, in addition to direct
performance improvements to the ``corresponding_column()`` internal method
that is used heavily by the ORM by constructs like :func:`_orm.aliased` and
similar.
Fixes: #8796
Change-Id: I4a76788007d5a802b9a4081e6a0f6e4b52497b50
The ``aliased()`` constructor calls upon ``__clause_element__()``,
which internally annotates a ``FromClause``, like a subquery.
This became expensive as ``AnnotatedFromClause`` has for
many years called upon ``element.c`` so that the full ``.c``
collection is transferred to the Annotated.
Taking this out proved to be challenging. A straight remove
seemed to not break any tests except for the one that
tested the exact condition. Nevertheless this seemed
"spooky" so I instead moved the get of ``.c`` to be in a
memoized proxy method. However, that then exposed
a recursion issue related to loader_criteria; so the
source of that behavior, which was an accidental behavioral
artifact, is now made into an explcicit option that
loader_criteria uses directly.
The accidental behavioral artifact in question is still
kind of strange since I was not able to fully trace out
how it works, but the end result is that fixing the
artifact to be "correct" causes loader_criteria, within
the particular test for #7491, creates a select/
subquery structure with a cycle in it, so compilation fails
with recursion overflow.
The "solution" is to cause the artifact to occur in this
case, which is that the ``AnnotatedFromClause`` will have a
different ``.c`` collection than its element, which is a
subquery. It's not totally clear how a cycle is generated
when this is not done.
This is commit one of two, which goes through
some hoops to make essentially a one-line change.
The next commit will rework ColumnCollection to optimize
the corresponding_column() method significantly.
Fixes: #8796
Change-Id: Id58ae6554db62139462c11a8be7313a3677456ad
Added a new Postgresql :class:`_postgresql.DOMAIN` datatype, which follows
the same CREATE TYPE / DROP TYPE behaviors as that of PostgreSQL
:class:`_postgresql.ENUM`. Much thanks to David Baumgold for the efforts on
this.
Fixes: #7316Closes: #7317
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7317
Pull-request-sha: bc9a82f010
Change-Id: Id8d7e48843a896de17d20cc466b115b3cc065132
Fixed regression caused by 🎫`7823` which impacted the caching
system, such that bound parameters that had been "cloned" within ORM
operations, such as polymorphic loading, would in some cases not acquire
their correct execution-time value leading to incorrect bind values being
rendered.
Fixes: #7903
Change-Id: I61c802749b859bebeb127d24e66d6e77d13ce57a
the pep484 task becomes more intense as there is mounting
pressure to come up with a consistency in how data moves
from end-user to instance variable.
current thinking is coming into:
1. there are _typing._XYZArgument objects that represent "what the
user sent"
2. there's the roles, which represent a kind of "filter" for different
kinds of objects. These are mostly important as the argument
we pass to coerce().
3. there's the thing that coerce() returns, which should be what the
construct uses as its internal representation of the thing.
This is _typing._XYZElement.
but there's some controversy over whether or
not we should pass actual ClauseElements around by their role
or not. I think we shouldn't at the moment, but this makes the
"role-ness" of something a little less portable. Like, we have
to set DMLTableRole for TableClause, Join, and Alias, but then
also we have to repeat those three types in order to set up
_DMLTableElement.
Other change introduced here, there was a deannotate=True
for the left/right of a sql.join(). All tests pass without that.
I'd rather not have that there as if we have a join(A, B) where
A, B are mapped classes, we want them inside of the _annotations.
The rationale seems to be performance, but this performance can
be illustrated to be on the compile side which we hope is cached
in the normal case.
CTEs now accommodate for text selects including recursive.
Get typing to accommodate "util.preloaded" cleanly; add "preloaded"
as a real module. This seemed like we would have needed
pep562 `__getattr__()` but we don't, just set names in
globals() as we import them.
References: #6810
Change-Id: I34d17f617de2fe2c086fc556bd55748dc782faf0
Fixed bug in newly implemented
:paramref:`.FunctionElement.table_valued.joins_implicitly` feature where
the parameter would not automatically propagate from the original
:class:`.TableValuedAlias` object to the secondary object produced when
calling upon :meth:`.TableValuedAlias.render_derived` or
:meth:`.TableValuedAlias.alias`.
Additionally repaired these issues in :class:`.TableValuedAlias`:
* repaired a potential memory issue which could occur when
repeatedly calling :meth:`.TableValuedAlias.render_derived` against
successive copies of the same object (for .alias(), we currently
have to still continue chaining from the previous element. not sure
if this can be improved but this is standard behavior for .alias()
elsewhere)
* repaired issue where the individual element types would be lost when
calling upon :meth:`.TableValuedAlias.render_derived` or
:meth:`.TableValuedAlias.alias`.
Fixes: #7890
Change-Id: Ie5120c7ff1e5c1bba5aaf77c782a51c637860208
Improvements in memory usage by the ORM, removing a significant set of
intermediary expression objects that are typically stored when a copy of an
expression object is created. These clones have been greatly reduced,
reducing the number of total expression objects stored in memory by
ORM mappings by about 30%.
note this change causes the tests to have a bit of a harder time with
GC, which we would assume is because mappings now have a lot more
garbage to clean up after mappers are configured. it remains
to be seen what the long term effects of this are.
Fixes: #7823
Change-Id: If8729747ffb9bf27e8974f069a994b5a823ee095
Observed the tests here have different profiling
counts when run individually vs. as a group, and
this seems to be due to whether or not results of
each query are garbage collected or not. for
all but one test, ensuring results stay between
query runs seems to meet the current profiling
counts.
Change-Id: I5aca5db08936757ad2a6055c5fc077cc58979bdd
replaced the __tags__ class attribute and the
--exclude-tags / --include-tags test runner options
with regular pytest.mark names
so that we can take advantage of mark expressions.
options --nomemory, --notimingintensive, --backend-only,
--exclude-tags, --include-tags remain as legacy but
make use of pytest mark for implemementation.
Added a "mypy" mark for the section of tests that are doing mypy
integration tests.
The __backend__ and __sparse_backend__ class attributes also
use pytest marks for their implementation, which also allows
the marks "backend" and "sparse_backend" to be used explicitly.
Also removed the no longer used "--cdecimal" option as this was
python 2 specific.
in theory, the usage of pytest marks could expand such that
the whole exclusions system would be based on it, but this
does not seem to have any advantage at the moment.
Change-Id: Ideeb57d9d49f0efc7fc0b6b923b31207ab783025
the truediv test suite didn't have __backend__ so wasn't running
for every DB except in the main build. Repaired this as well
as truediv support to preserve the right-hand side type
when casting to numeric, if the right type is already a
numeric type.
also fixed a memusage test that relies on savepoints so was
not running under gerrit runs.
Change-Id: I3be223fdf697af9c1ed61b70d621f57cbbb7a92b
Finalize all remaining removed-in-2.0 changes so that we
can begin doing pep-484 typing without old things
getting in the way (we will also have to do public_factory).
note there are a few "moved_in_20()" and "became_legacy_in_20()"
warnings still in place. The SQLALCHEMY_WARN_20 variable
is now removed.
Also removed here are the legacy "in place mutators" for Select
statements, and some keyword-only argument signatures in Core
have been added.
Also in the big change department, the ORM mapper() function
is removed entirely; the Mapper class is otherwise unchanged,
just the public-facing API function. Mappers are now always
given a registry in which to participate, however the
argument signature of Mapper is not changed. ideally "registry"
would be the first positional argument.
Fixes: #7257
Change-Id: Ic70c57b9f1cf7eb996338af5183b11bdeb3e1623
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.
Also update Black and other pre-commit hooks and re-format with Black.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570
Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
These are small bits where cache_anon_map in particular
is part of the cache key generation scheme which is a key
target for cython.
changing such a tiny element of the cache key gen is
doing basically nothing yet, as the cython impl is
mostly the exact same speed as the python one. I guess for
cython to be effective we'd need to redo the whole cache key
generation and possibly not use the same kinds of structures,
which might not be very easy to do.
Additionally, some cython runtime import errors are being
observed on jenkins, add an upfront check to the test suite
to indicate if the expected build succeeded when REQUIRE_SQLALCHEMY_CEXT
is set.
Running case CacheAnonMap
Running python .... Done
Running cython .... Done
| python | cython | cy / py |
test_get_anon_non_present| 0.301266758 | 0.231203834 | 0.767438915 |
test_get_anon_present| 0.300919362 | 0.227336695 | 0.755473803 |
test_has_key_non_present| 0.152725077 | 0.133191719 | 0.872101171 |
test_has_key_present| 0.152689778 | 0.133673095 | 0.875455428 |
Running case PrefixAnonMap
Running python .. Done
Running cython .. Done
| python | cython | cy / py |
test_apply_non_present| 0.358715744 | 0.335245703 | 0.934572034 |
test_apply_present | 0.354434996 | 0.338579782 | 0.955266229 |
Change-Id: I0d3f1dd285c044afc234479141d831b2ee0455be
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet
Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.
See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharingFixes: #7256
Change-Id: I2930ef1894b5048210384728118e586e813f6a76
Signed-off-by: Federico Caselli <cfederico87@gmail.com>
* The :meth:`_orm.Query.join` method no longer accepts strings for
relationship names; the long-documented approach of using
``Class.attrname`` for join targets is now standard.
* Loader options no longer accept strings for attribute names. The
long-documented approach of using ``Class.attrname`` for loader option
targets is now standard.
It is hoped that a subsequent commit can refactor loader
options to no longer need "UnboundLoad" for most cases.
Change-Id: If4629882c40523dccbf4459256bf540fb468b618
References: #6986
Removed here includes:
* convert_unicode parameters
* encoding create_engine() parameter
* description encoding support
* "non-unicode fallback" modes under Python 2
* String symbols regarding Python 2 non-unicode fallbacks
* any concept of DBAPIs that don't accept unicode
statements, unicode bound parameters, or that return bytes
for strings anywhere except an explicit Binary / BLOB
type
* unicode processors in Python / C
Risk factors:
* Whether all DBAPIs do in fact return Unicode objects for
all entries in cursor.description now
* There was logic for mysql-connector trying to determine
description encoding. A quick test shows Unicode coming
back but it's not clear if there are still edge cases where
they return bytes. if so, these are bugs in that driver,
and at most we would only work around it in the mysql-connector
DBAPI itself (but we won't do that either).
* It seems like Oracle 8 was not expecting unicode bound parameters.
I'm assuming this was all Python 2 stuff and does not apply
for modern cx_Oracle under Python 3.
* third party dialects relying upon built in unicode encoding/decoding
but it's hard to imagine any non-SQLAlchemy database driver not
dealing exclusively in Python unicode strings in Python 3
Change-Id: I97d762ef6d4dd836487b714d57d8136d0310f28a
References: #7257
Federico Caselli