The :class:`_schema.Table` class now raises a deprecation warning
when columns with the same name are defined. To replace a column a new
parameter :paramref:`_schema.Table.append_column.replace_existing` was
added to the :meth:`_schema.Table.append_column` method.
The :meth:`_expression.ColumnCollection.contains_column` will now
raises an error when called with a string, suggesting the caller
to use ``in`` instead.
Co-authored-by: Federico Caselli <cfederico87@gmail.com>
Change-Id: I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.
Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function. it does not
yet fully address other issues such as keyword arguments passed
to the table.select().
Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.
Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
Using the approach introduced at
https://gist.github.com/zzzeek/6287e28054d3baddc07fa21a7227904e
We can now create asyncio endpoints that are then handled
in "implicit IO" form within the majority of the Core internals.
Then coroutines are re-exposed at the point at which we call
into asyncpg methods.
Patch includes:
* asyncpg dialect
* asyncio package
* engine, result, ORM session classes
* new test fixtures, tests
* some work with pep-484 and a short plugin for the
pyannotate package, which seems to have so-so results
Change-Id: Idbcc0eff72c4cad572914acdd6f40ddb1aef1a7d
Fixes: #3414
* major additions to 1.4 migration doc; removed additional
verbosity regarding caching methodology and reorganized the
doc to present itself more as a "what's changed" guide
* as we now have a path for asyncio, update that doc so that
we aren't spreading obsolete information
* updates to the 2.0 migration guide with latest info, however
this is still an architecture doc and not a migration guide
yet, will need further rework.
* start really talking about 1.x vs. 2.0 style everywhere. Querying
is most of the docs so this is going to be a prominent
theme, start getting it to fit in
* Add introductory documentation for ORM example sections as these
are too sparse
* new documentation for do_orm_execute(), many separate sections,
adding deprecation notes to before_compile() and similar
* new example suites to illustrate do_orm_execute(),
with_loader_criteria()
* modernized horizontal sharding examples and added a separate
example to distinguish between multiple databases and single
database w/ multiple tables use case
* introducing DEEP ALCHEMY, will use zzzeeksphinx 1.1.6
* no name for the alchemist yet however the dragon's name
is Flambé
Change-Id: Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541
The coercions system allows us to add in lambdas as arguments
to Core and ORM elements without changing them at all. By allowing
the lambda to produce a deterministic cache key where we can also
cheat and yank out literal parameters means we can move towards
having 90% of "baked" functionality in a clearer way right in
Core / ORM.
As a second step, we can have whole statements inside the lambda,
and can then add generation with __add__(), so then we have
100% of "baked" functionality with full support of ad-hoc
literal values.
Adds some more short_selects tests for the moment for comparison.
Other tweaks inside cache key generation as we're trying to
approach a certain level of performance such that we can
remove the use of "baked" from the loader strategies.
As we have not yet closed#4639, however the caching feature
has been fully integrated as of
b0cfa7379c, we will also
add complete caching documentation here and close that issue
as well.
Closes: #4639Fixes: #5380
Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
This patch makes several improvements in the area of
bulk updates and deletes as well as the new session mechanics.
RETURNING is now used for an UPDATE or DELETE statement
emitted for a diaelct that supports "full returning"
in order to satisfy the "fetch" strategy; this currently
includes PostgreSQL and SQL Server. The Oracle dialect
does not support RETURNING for more than one row,
so a new dialect capability "full_returning" is added
in addition to the existing "implicit_returning", indicating
this dialect supports RETURNING for zero or more rows,
not just a single identity row.
The "fetch" strategy will gracefully degrade to
the previous SELECT mechanics for dialects that do not
support RETURNING.
Additionally, the "fetch" strategy will attempt to use
evaluation for the VALUES that were UPDATEd, rather
than just expiring the updated attributes. Values should
be evalutable in all cases where the value is not
a SQL expression.
The new approach also incurs some changes in the
session.execute mechanics, where do_orm_execute() event
handlers can now be chained to each return results;
this is in turn used by the handler to detect on a
per-bind basis if the fetch strategy needs to
do a SELECT or if it can do RETURNING. A test suite is
added to test_horizontal_shard that breaks up a single
UPDATE or DELETE operation among multiple backends
where some are SQLite and don't support RETURNING and
others are PostgreSQL and do.
The session event mechanics are corrected
in terms of the "orm pre execute" hook, which now
receives a flag "is_reentrant" so that the two
ORM implementations for this can skip on their work
if they are being called inside of ORMExecuteState.invoke(),
where previously bulk update/delete were calling its
SELECT a second time.
In order for "fetch" to get the correct identity when
called as pre-execute, it also requests the identity_token
for each mapped instance which is now added as an optional
capability of a SELECT for ORM columns. the identity_token
that's placed by horizontal_sharding is now made available
within each result row, so that even when fetching a
merged result of plain rows we can tell which row belongs
to which identity token.
The evaluator that takes place within the ORM bulk update and delete for
synchronize_session="evaluate" now supports the IN and NOT IN operators.
Tuple IN is also supported.
Fixes: #1653
Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
This commit includes that we've removed the "_orm_query"
attribute from compile state as well as query context.
The attribute created reference cycles and also added
method call overhead. As part of this change,
the interface for ORMExecuteState changes a bit, as well
as the interface for the horizontal sharding extension
which now deprecates the "query_chooser" callable
in favor of "execute_chooser", which receives the contextual
object. This will also work more nicely when we implement
the new execution path for bulk updates and deletes.
Pre-merge execution options for statement, connection,
arguments all up front in Connection. that way they
can be passed to the before_execute / after_execute events,
and the ExecutionContext doesn't have to merge as second
time. Core execute is pretty close to 1.3 now.
baked wasn't using the new one()/first()/one_or_none() methods,
fixed that.
Convert non-buffered cursor strategy to be a stateless
singleton. inline all the paths by which the strategy
gets chosen, oracle and SQL Server dialects make use of the
already-invoked post_exec() hook to establish the alternate
strategies, and this is actually much nicer than it was before.
Add caching to mapper instance processor for getters.
Identified a reference cycle per query that was showing
up as a lot of gc cleanup, fixed that.
After all that, performance not budging much. Even
test_baked_query now runs with significantly fewer function
calls than 1.3, still 40% slower.
Basically something about the new patterns just makes
this slower and while I've walked a whole bunch of them
back, it hardly makes a dent. that said, the performance
issues are relatively small, in the 20-40% time increase
range, and the new caching feature
does provide for regular ORM and Core queries that
are cached, and they are faster than non-cached.
Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
This patch replaces the ORM execution flow with a
single pathway through Session.execute() for all queries,
including Core and ORM.
Currently included is full support for ORM Query,
Query.from_statement(), select(), as well as the
baked query and horizontal shard systems. Initial
changes have also been made to the dogpile caching
example, which like baked query makes use of a
new ORM-specific execution hook that replaces the
use of both QueryEvents.before_compile() as well
as Query._execute_and_instances() as the central
ORM interception hooks.
select() and Query() constructs alike can be passed to
Session.execute() where they will return ORM
results in a Results object. This API is currently
used internally by Query. Full support for
Session.execute()->results to behave in a fully
2.0 fashion will be in later changesets.
bulk update/delete with ORM support will also
be delivered via the update() and delete()
constructs, however these have not yet been adapted
to the new system and may follow in a subsequent
update.
Performance is also beginning to lag as of this
commit and some previous ones. It is hoped that
a few central functions such as the coercions
functions can be rewritten in C to re-gain
performance. Additionally, query caching
is now available and some subsequent patches
will attempt to cache more of the per-execution
work from the ORM layer, e.g. column getters
and adapters.
This patch also contains initial "turn on" of the
caching system enginewide via the query_cache_size
parameter to create_engine(). Still defaulting at
zero for "no caching". The caching system still
needs adjustments in order to gain adequate performance.
Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
Convert Query to do virtually all compile state computation
in the _compile_context() phase, and organize it all
such that a plain select() construct may also be used as the
source of information in order to generate ORM query state.
This makes it such that Query is not needed except for
its additional methods like from_self() which are all to
be deprecated.
The construction of ORM state will occur beyond the
caching boundary when the new execution model is integrated.
future select() gains a working join() and filter_by() method.
as we continue to rebase and merge each commit in the steps,
callcounts continue to bump around. will have to look at
the final result when it's all in.
References: #5159
References: #4705
References: #4639
References: #4871
References: #5010
Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
A few small mistakes led to huge callcounts. Additionally,
the warn-on-get behavior which is attempting to warn for
deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear
if its feasible to have this warning or to somehow alter how it
works.
Fixes: #5340
Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
For a 1.4 / 1.3 merge, rewrite the documentation for
Query.join() to indicate calling forms that are now considered
legacy, including the use of strings in join(), sending a
series of join paths in one call, and using the aliased=True
flag. update the elementtree examples as well to use aliased()
(they are much simpler to understand this way too) and update
other links.
Also improve docs for aliased() and some other ORM targets
such as PropComparator.
Change-Id: I636e3a9130dc5509e51c2cf60a52f38fcadffbc6
References: #4705
The next step in the 2.0 ORM changes is to have the
ORM integrate with the new Result object fully.
this patch uses Result to represent ORM objects rather
than lists. public API to get at this Result is not
added yet. dogpile.cache and horizontal sharding
recipe/extensions have small adjustments to accommodate
this change.
Callcounts have fluctuated, some slightly better and
some slightly worse. A few have gone up by a bit,
however as the codebase is still in flux it is anticipated
there will be some performance gains later on as
ORM fetching is refined to no longer need to accommodate
for extensive aliasing. The addition of caching
will then change the entire story.
References: #5087
References: #4395
Change-Id: If1a23824ffb77d8d58cf2338cf35dd6b5963b17f
Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf
Trying to see if by making the cache key memoized, we
still can have the older "identity" form of caching
which is the cheapest of all, at the same time as the
newer "cache key each time" version that is not nearly
as cheap; but still much cheaper than no caching at all.
Also needed is a per-execution update of _keymap when
we invoke from a cached select, so that Column objects
that are anonymous or otherwise adapted will match up.
this is analogous to the adaption of bound parameters
from the cache key.
Adds test coverage for the keymap / construct_params()
changes related to caching. Also hones performance
to a large extent for statement construction and
cache key generation.
Also includes a new memoized attribute
approach that vastly simplifies the previous approach
of "group_expirable_memoized_property" and finally
integrates cleanly with _clone(), _generate(), etc.
no more hardcoding of attributes is needed, as well
as that most _reset_memoization() calls are no longer
needed as the reset is inherent in a _generate() call;
this also has dramatic performance improvements.
Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
Introduced a modules registry to register modules that should be lazily loaded
in the package init. This ensures that they are in the system module cache,
avoiding potential thread safety issues as when importing them directly
in the function that uses them. The module registry is used to obtain
these modules directly, ensuring that the all the lazily loaded modules
are resolved at the proper time
This replaces dependency_for decorator and the dependencies decorator logic,
removing the need to pass the resolved modules as arguments of the
decodated functions and removes possible errors caused by linters.
Fixes: #4689Fixes: #4656
Change-Id: I2e291eba4297867fc0ddb5d875b9f7af34751d01
Remove the deprecated loader options ``joinedload_all``, ``subqueryload_all``,
``lazyload_all``, ``selectinload_all``. The normal version with method chaining
should be used in their place.
Fixes: #4642
Change-Id: I12eb4dfa7a86375911a570934ee662653d85d50a
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Remove print statements
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [X] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #5166
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5166
Pull-request-sha: 04a7394f71
Change-Id: Ib90a59fac929661a18748c6e44966fb87e3978c6
as the versioned_history example supplies an integer version
counter for the purposes of generating an audit trail, this
counter is also suited to be used for optimistic concurrency
detection using the version_id_col feature. Build upon
the test that was first added in ac54ba0f2d
to provide a flag and some basic documentation.
Fixes: #2861
Change-Id: I50236beae4c49b33ada8fdcc4c524273b4e21c75
The "expanding IN" feature, which generates IN expressions at query
execution time which are based on the particular parameters associated with
the statement execution, is now used for all IN expressions made against
lists of literal values. This allows IN expressions to be fully cacheable
independently of the list of values being passed, and also includes support
for empty lists. For any scenario where the IN expression contains
non-literal SQL expressions, the old behavior of pre-rendering for each
position in the IN is maintained. The change also completes support for
expanding IN with tuples, where previously type-specific bind processors
weren't taking effect.
As part of this change, a more explicit separation between
"literal execute" and "post compile" bound parameters is being made;
as the "ansi bind rules" feature is rendering bound parameters
inline, as we now support "postcompile" generically, these should
be used here, however we have to render literal values at
execution time even for "expanding" parameters. new test fixtures
etc. are added to assert everything goes to the right place.
Fixes: #4645
Change-Id: Iaa2b7bfbfaaf5b80799ee17c9b8507293cba6ed1
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient. test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.
Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
The latest flake8 seems to look for these and they are in fact
correctable with a backslash. Also need to add r to the strings
to avoid W605.
Change-Id: I8045309aa2ad29978ba7e99c45f75bc1457dff3d
Fixed bug in large_resultsets example case where a re-named "id" variable
due to code reformatting caused the test to fail. Pull request courtesy
Matt Schuchhardt.
Fixes: #4528Closes: #4529
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4529
Pull-request-sha: 8f52f10422
Change-Id: Iaca6fd06ec0ede8ff1745d627708ba492a2114c8
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
A modification to the "turn UPDATE into INSERT" recipe that
also UPDATEs the previous row. The example is using timestamps
to maintain a relationship between two objects.
Change-Id: Ifdb8ee73616190384263bbe88c71d9278d616f6b
Since I didn't even realize what this was for when reading the docs,
make it clearer that this is to mirror a Column default and remove
the extra verbiage about the mechanics of INSERTs.
Change-Id: Id2c6a29800f7b723573610e4707aec7e6ea38f5f
Added new attribute :attr:`.Query.lazy_loaded_from` which is populated
with an :class:`.InstanceState` that is using this :class:`.Query` in
order to lazy load a relationship. The rationale for this is that
it serves as a hint for the horizontal sharding feature to use, such that
the identity token of the state can be used as the default identity token
to use for the query within id_chooser().
Also repaired an issue in the :meth:`.Result.with_post_criteria`
method added in I899808734458e25a023142c2c5bb37cbed869479
for 🎫`4128` where the "unbake subquery loaders" version was calling
the post crtieria functions given the :class:`.Result` as the argument
rather than applying them to the :class:`.Query`.
Change-Id: I3c0919ce7fd151b80fe2f9b5f99f60df31c2d73d
Fixes: #4243
Updated the dogpile.caching example to include new structures that
accommodate for the "baked" query system, which is used by default within
lazy loaders and some eager relationship loaders. The dogpile.caching
"relationship_caching" and "advanced" examples were also broken due to
🎫`4256`. The issue here is also worked-around by the fix in
🎫`4128`.
Note that this recipe requires
I3f86fcb12a6a9a89aa308b335e75c25969bcc30e in order for the
"advanced" example to work.
Change-Id: I9d35417f1d6c1906555583b8225d3da7f81736f7
Fixes: #4258
For the purposes of assisting with sharded setups, add a new
member to the identity key that can be customized. this allows
sharding across databases where the primary key space is shared.
Change-Id: Iae3909f5d4c501b62c10d0371fbceb01abda51db
Fixes: #4137
remarkably, the examples for concrete and single were still
using classical mappings. Ensure all three examples use
modern declarative patterns, each illustrate the identical set
of query operations. Use back_populates, flat=True for joins,
etc. ensure flake8 linting, correct links and add a link back
from newly reworked inheritance documentation.
Change-Id: I8465a9badbb0eda804f457ccac599f051ee3c27c
Ensure that the history table sets autoincrement=False, since these values
are copied in all cases; the flag will emit an error as of 1.1 if the
primary key is composite. Additionally, use the sqlite_autoincrement flag
so that SQLite uses unique primary key identifiers for new rows even if
some rows have been deleted.
Fixes: #3872
Change-Id: I65912eb394b3b69d7f4e3c098f4f948b0a7a5374
Pull-request: https://bitbucket.org/zzzeek/sqlalchemy/pull-requests/93
This allows us to build default-setting recipes such
as one that allows us to actively read column-level
defaults. An example suite is also added.
Change-Id: I7b022d52cc89526132d5bc4201ac27fea4cf088d
Fixes: #1311
As per their email ‘Changes to project subdomains’:
> Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard.
Test Plan: Manually visited all the links I’ve modified.
dictlike-polymorphic was broken by the change
in 1714e0d6ef. Use the new
style of inheritance.
Fixes: #3704
Change-Id: I3509ef4bf7772dd6994daf600accf4a2c5eb6973
Changed the "directed graph" example to no longer consider
integer identifiers of nodes as significant; the "higher" / "lower"
references now allow mutual edges in both directions.
Change-Id: Ibfd5b420f0451a6fc746f0bcbbbb062f8f88dc1d
Fixes: #3698
(cherry picked from commit c3de4061fd)
simple but unusual system allows for a dramatic savings in Python
overhead for the construction and processing of orm :class:`.Query`
objects, from query construction up through rendering of a string
SQL statement.
fixes#3054
Mike Bayer