A memory growth issue was identified in this test which
caused the profiling results to be inaccurate.
Change-Id: I248dcce5833feada947bc91bdf09a8f925d31d65
(cherry picked from commit 139aac4aba)
1. ensure provision.py loads dialect implementations when running
reap_dbs.py. Reapers haven't been working since
598f2f7e55 .
2. add some exclusion rules to allow the sqlite_file target to work;
add to tox.
3. add reap dbs target for SQLite, repair SQLite drop_db routine
which also wasn't doing the right thing for memory databases
etc.
4. Fix logging in provision files, as the main provision logger
is the one that's enabled by reap_dbs and maybe others, have all
the provision files use the provision logger.
Fixes: #5180Fixes: #5168
Change-Id: Ibc1b0106394d20f5bcf847f37b09d185f26ac9b5
(cherry picked from commit 08fdf6da228aab54263abe043c21bcebf3513e72)
Applied an explicit "cause" to most if not all internally raised exceptions
that are raised from within an internal exception catch, to avoid
misleading stacktraces that suggest an error within the handling of an
exception. While it would be preferable to suppress the internally caught
exception in the way that the ``__suppress_context__`` attribute would,
there does not as yet seem to be a way to do this without suppressing an
enclosing user constructed context, so for now it exposes the internally
caught exception as the cause so that full information about the context
of the error is maintained.
Fixes: #4849
Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
(cherry picked from commit 8be0dae77a7e0747f0d0fb4282db4aea7f41e03a)
issues with backend-specific profiling should be limited
to tests that are explcitly against resultset, compiler, etc.
MySQL in particular has an often varying callcount that isn't
worth running these tests against nor is it worth profiling
them for other backends like Oracle and SQL Server.
Also add the REQUIRE_SQLALCHEMY_CEXT flag to
the regen_callcounts.tox.ini script, which is part of some review
somewhere but is needed here to generate callcounts correctly.
Add a "warmup" phase for some of the ORM tests for join conditions
that have varying profile counts based on whether mappings have been
used already or not; profiling should always be against the
"warmed up" version of a function.
Change-Id: If483820235fa4cc4360cbd067a9b68d83512d587
(cherry picked from commit 386012597b)
Added test support and repaired a wide variety of unnecessary reference
cycles created for short-lived objects, mostly in the area of ORM queries.
For the 1.3 backport, includes the prefix_anon_map() optimization
from 1.4 / master which inlines the anonymous symbol generation
into a single object. This removes a cycle from the compiler
that otherwise results in a signficantly higher number of
unreachable cycles.
Fixes: #5056
Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
(cherry picked from commit 492930ed572de5f5550d514bc2ca52a57f108350)
Identified a performance issue in the system by which a join is constructed
based on a mapped relationship. The clause adaption system would be used
for the majority of join expressions including in the common case where no
adaptation is needed. The conditions under which this adaptation occur
have been refined so that average non-aliased joins along a simple
relationship without a "secondary" table use about 70% less function calls.
Change-Id: Ifbe04214576e5a9fac86ca80c1dc7145c27cd50a
(cherry picked from commit 7e9f273835)
A warning is emitted for a condition in which the :class:`.Session` may
implicitly swap an object out of the identity map for another one with the
same primary key, detaching the old one, which can be an observed result of
load operations which occur within the :meth:`.SessionEvents.after_flush`
hook. The warning is intended to notify the user that some special
condition has caused this to happen and that the previous object may not be
in the expected state.
Fixes: #4890
Change-Id: Ide11c6b9f21ca67ff5a96266c521d0c56fd6af8d
(cherry picked from commit 4aa43ecbd7)
This test is very sensitive and fluctuates a lot, failing builds,
bump the variance to try and resolve.
Change-Id: Ia19bb8871b432059cb3917ca0050a68f75c0a0f2
(cherry picked from commit e429ef1d31)
Adjusted the initialization for :class:`.Enum` to minimize how often it
invokes the ``.__members__`` attribute of a given PEP-435 enumeration
object, to suit the case where this attribute is expensive to invoke, as is
the case for some popular third party enumeration libraries.
Fixes: #4758
Change-Id: Iffeb854c67393bdcb288944fc357a074e20e1325
(cherry picked from commit 2cc7308c96)
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient. test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.
Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
(cherry picked from commit 190e0139e8)
The memusage tests are extremely time and memory intensive,
and when CI runs against MySQL or Postgresql there are many
database/driver combinations for which the "backend" tests
repeatedly run; as these tests are more oriented towards
basic dialect interaction, add a new "sparse" backend option
that will run the tests only once per base dialect.
Change-Id: I312aa0332d7ec1ff4e2faa15f6b189d6f0f68393
(cherry picked from commit 0ad64da397)
Improved the behavior of :func:`.orm.with_polymorphic` in conjunction with
loader options, in particular wildcard operations as well as
:func:`.orm.load_only`. The polymorphic object will be more accurately
targeted so that column-level options on the entity will correctly take
effect.The issue is a continuation of the same kinds of things fixed in
🎫`4468`.
The path logic when using chained mapper options is improved
to be more accurate in terms of the entities being linked
in the path; when using :func:`.with_polymorphic`, mapper
options against this entity need to specify attributes
in terms of the with_polymorphic() object and not against the
base mappings. New error conditions are raised which were previously
more than likely silenty failures.
Fixes: #4469
Change-Id: Ie8d802879663b4ff6f6ac1438c885c06d78ae2a0
Implemented a new feature whereby the :class:`.AliasedClass` construct can
now be used as the target of a :func:`.relationship`. This allows the
concept of "non primary mappers" to no longer be necessary, as the
:class:`.AliasedClass` is much easier to configure and automatically inherits
all the relationships of the mapped class, as well as preserves the
ability for loader options to work normally.
- introduce new name for mapped_table, "persist_selectable". this is
the selectable that selects against the local mapper and its superclasses,
but does not include columns local only to subclasses.
- relationship gains "entity" which is the mapper or aliasedinsp.
- clarfiy name "entity" vs. "query_entity" in loader strategies.
Fixes: #4423Fixes: #4422Fixes: #4421Fixes: #3348
Change-Id: Ic3609b43dc4ed115006da9ad9189e574dc0c72d9
A large change throughout the library has ensured that all objects, parameters,
and behaviors which have been noted as deprecated or legacy now emit
``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now
defaults to displaying deprecation warnings, as well as that modern test suites
based on tools like tox and pytest tend to display deprecation warnings,
this change should make it easier to note what API features are obsolete.
See the notes added to the changelog and migration notes for further
details.
Fixes: #4393
Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Fixed a minor performance issue which could in some cases add unnecessary
overhead to result fetching, involving the use of ORM columns and entities
that include those same columns at the same time within a query. The issue
has to do with hash / eq overhead when referring to the column in different
ways.
Fixes: #4347
Change-Id: I191d4d1b1623898060a9accdfd186de16f89a6b7
:meth:`.TypeEngine.column_expression` methods where these methods would not
work if the target type were part of a :class:`.Variant`, or other target
type of a :class:`.TypeDecorator`. Additionally, the SQL compiler now
calls upon the dialect-level implementation when it renders these methods
so that dialects can now provide for SQL-level processing for built-in
types.
Change-Id: Ic7b39575184db582e628e6ecee48dcda7d03a817
Fixes: #3981
Fixed an issue that was both a performance regression in 1.2 as well as an
incorrect result regarding the "baked" lazy loader, involving the
generation of cache keys from the original :class:`.Query` object's loader
options. If the loader options were built up in a "branched" style using
common base elements for multiple options, the same options would be
rendered into the cache key repeatedly, causing both a performance issue as
well as generating the wrong cache key. This is fixed, along with a
performance improvement when such "branched" options are applied via
:meth:`.Query.options` to prevent the same option objects from being
applied repeatedly.
Change-Id: I955fe2f50186abd8e753ad490fd3eb8f017e26f9
Fixes: #4270
Drops support for cx_Oracle prior to version 5.x, reworks
numeric and binary support.
Fixes: #4064
Change-Id: Ib9ae9aba430c15cd2a6eeb4e5e3fd8e97b5fe480
Swing the biggest hammer, run multiprocessing.Process() for
each memusage test individually so that they are fully isolated
from the parent process and any side effects of pytest-xdist
Also add --nomemory as a shortcut to exclude_tags=memory-intensive
and add this to the setup.py test runner as the memory tests
should not be running for quick runs
Change-Id: I3c16c781e21b33deb939a64e77a6e0e41fb86922
the issue
- unblock pytest-xdist now that the upstream issue is fixed,
maybe this old version is the issue
Change-Id: I28dd7ae0872948a188651d42e2f4af60bcbafe81
Adding a new kind of relationship loader that is
a cross between the "immediateload" and the "subquery"
eager loader, using an IN criteria to load related items
in bulk immediately after the lead query result is loaded.
Change-Id: If13713fba9b465865aef8fd50b5b6b977fe3ef7d
Fixes: #3944
The ``lazy="select"`` loader strategy now makes used of the
:class:`.BakedQuery` query caching system in all cases. This
removes most overhead of generating a :class:`.Query` object and
running it into a :func:`.select` and then string SQL statement from
the process of lazy-loading related collections and objects. The
"baked" lazy loader has also been improved such that it can now
cache in most cases where query load options are used.
Change-Id: Ic96792fffaa045ae9aa0a4657d6d29235d3efb85
Fixes: #3954
when the suite runs, such as a background thread or something like that,
which is affecting these tests a bit.
Change-Id: I52d50a44778ec1eecb8e335ae59b1a4773e80a79
Added warnings to the LRU "compiled cache" used by the :class:`.Mapper`
(and ultimately will be for other ORM-based LRU caches) such that
when the cache starts hitting its size limits, the application will
emit a warning that this is a performance-degrading situation that
may require attention. The LRU caches can reach their size limits
primarily if an application is making use of an unbounded number
of :class:`.Engine` objects, which is an antipattern. Otherwise,
this may suggest an issue that should be brought to the SQLAlchemy
developer's attention.
Additionally, adjusted the test_memusage algorithm again as the
previous one could still allow a growing memory size to be missed.
Change-Id: I020d1ceafb7a08f6addfa990a1e7acd09f933240
Fixed bug in compiler where the string identifier of a savepoint would
be cached in the identifier quoting dictionary; as these identifiers
are arbitrary, a small memory leak could occur if a single
:class:`.Connection` had an unbounded number of savepoints used,
as well as if the savepoint clause constructs were used directly
with an unbounded umber of savepoint names. The memory leak does
**not** impact the vast majority of cases as normally the
:class:`.Connection`, which renders savepoint names with a simple
counter starting at "1", is used on a per-transaction or
per-fixed-number-of-transactions basis before being discarded.
The savepoint name in virtually all cases does not require quoting
at all, however to support potential third party use cases
the "check for quotes needed" logic is retained, at a small
performance cost. Uncondtionally quoting the name is another
option, but this would turn the name into a case sensitive name
which runs the risk of poor interactions with existing deployments
that may be looking at these names in other contexts.
Change-Id: I6b53c96abf7fdf1840592bbca5da81347911844c
Fixes: #3931
Continuing from Ie43beecf37945b2bb7fff0aaa597a597293daa18,
also observed is the overhead of PathRegsitry memoized token
functions, as these paths are not cached in the case of
long joinedloader chains. The memoizations here were made
with short paths in mind, and have been replaced with
an inlined straight create of these paths up front, producing
callcounts very similar to 0.8. Combined with the previous
optimizations, 1.1 now runs the "joined eager load of one row"
worst case test in about 40% fewer calls than 0.8 and 60%
fewer than 1.1.5.
Change-Id: Ib5e1c1345a1dd8edfbdb3fed06eb717d4e164d31
Fixes: #3915
Addressed some long unattended performance concerns within the
joined eager loader query construction system. The use of ad-hoc
:class:`.AliasedClass` objects per query, which produces lots of column
lookup overhead each time, has been replaced with a cached approach
that makes use of a small pool of :class:`.AliasedClass` objects
that are reused between invocations of joined eager loading.
Callcount reduction of SQL query generation for worst-case joined
loader scenarios (lots of joins, lots of columns) is reduced by
approximately 270%.
Change-Id: Ie43beecf37945b2bb7fff0aaa597a597293daa18
Fixes: #3915
This is an old parameter no longer relevant to how SQLAlchemy
works, once the Query object was introduced. By deprecating it
we establish that we aren't supporting non-working use cases
and that we encourage applications to move off of the use of this
parameter.
Fixes: #3394
Change-Id: I25b9a38142a1537bbcb27d3e8b66a8b265140072
non-recommended use case of replacing a relationship on a fixed
mapped class many times, referring to an arbitrarily growing number of
target mappers. A warning is emitted when the old relationship is
replaced, however if the mapping were already used for querying, the
old relationship would still be referenced within some registries.
fixes#3251
Mike Bayer