first change:
Reworked the :ref:`examples_versioned_history` to work with
version 2.0, while at the same time improving the overall working of
this example to use newer APIs, including a newly added hook
:meth:`_orm.MapperEvents.after_mapper_constructed`.
second change:
Added new event hook :meth:`_orm.MapperEvents.after_mapper_constructed`,
which supplies an event hook to take place right as the
:class:`_orm.Mapper` object has been fully constructed, but before the
:meth:`_orm.registry.configure` call has been called. This allows code that
can create additional mappings and table structures based on the initial
configuration of a :class:`_orm.Mapper`, which also integrates within
Declarative configuration. Previously, when using Declarative, where the
:class:`_orm.Mapper` object is created within the class creation process,
there was no documented means of running code at this point. The change
is to immediately benefit custom mapping schemes such as that
of the :ref:`examples_versioned_history` example, which generate additional
mappers and tables in response to the creation of mapped classes.
third change:
The infrequently used :attr:`_orm.Mapper.iterate_properties` attribute and
:meth:`_orm.Mapper.get_property` method, which are primarily used
internally, no longer implicitly invoke the :meth:`_orm.registry.configure`
process. Public access to these methods is extremely rare and the only
benefit to having :meth:`_orm.registry.configure` would have been allowing
"backref" properties be present in these collections. In order to support
the new :meth:`_orm.MapperEvents.after_mapper_constructed` event, iteration
and access to the internal :class:`_orm.MapperProperty` objects is now
possible without triggering an implicit configure of the mapper itself.
The more-public facing route to iteration of all mapper attributes, the
:attr:`_orm.Mapper.attrs` collection and similar, will still implicitly
invoke the :meth:`_orm.registry.configure` step thus making backref
attributes available.
In all cases, the :meth:`_orm.registry.configure` is always available to
be called directly.
fourth change:
Fixed obscure ORM inheritance issue caused by 🎫`8705` where some
scenarios of inheriting mappers that indicated groups of columns from the
local table and the inheriting table together under a
:func:`_orm.column_property` would nonetheless warn that properties of the
same name were being combined implicitly.
Fixes: #9220Fixes: #9232
Change-Id: Id335b8e8071c8ea509c057c389df9dcd2059437d
Added new option to horizontal sharding API
:class:`_horizontal.set_shard_id` which sets the effective shard identifier
to query against, for both the primary query as well as for all secondary
loaders including relationship eager loaders as well as relationship and
column lazy loaders.
Modernize sharding examples with new-style mappings, add new asyncio example.
Fixes: #7226Fixes: #7028
Change-Id: Ie69248060c305e8de04f75a529949777944ad511
Allow do_orm_execute() events to both receive the complete
state of bind_argments, load_options, update_delete_options
as they do already, but also allow them to *change* all those
things via new execution options. Options like autoflush,
populate_existing etc. can now be updated within a
do_orm_execute() hook and those changes will take effect
all the way through.
Took a few tries to get something that covers every case here,
in particular horizontal sharding which is consuming those
options as well as using context.invoke(), without excess
complexity. The good news seems to be that a simple
reorg and replacing the "reentrant" boolean with
"is this before do_orm_execute is invoked" was all that was
needed.
As part of this we add a new "identity_token" option allowing
this option to be controlled from do_orm_execute() as well
as from the outside.
WIP
Fixes: #7837
Change-Id: I087728215edec8d1b1712322ab389e3f52ff76ba
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
* requirements needs typing_extensions
* update all "future=True" references to be appropriate to 2.0
Change-Id: I2eeb0ae65afdb587f21aeb0020f1d8a292f67c21
For consistency with the prominent ORM concept :class:`_orm.Mapped`, the
names of the dictionary-oriented collections,
:func:`_orm.attribute_mapped_collection`,
:func:`_orm.column_mapped_collection`, and :class:`_orm.MappedCollection`,
are changed to :func:`_orm.attribute_keyed_dict`,
:func:`_orm.column_keyed_dict` and :class:`_orm.KeyFuncDict`, using the
phrase "dict" to minimize any confusion against the term "mapped". The old
names will remain indefinitely with no schedule for removal.
Docs here are also updated for typing as we can type
these collections as ``Mapped[dict[str, cls]]``, don't need
KeyFuncDict / MappedCollection for these
Fixes: #8608
Change-Id: Ib5cf63e0aef1c389e023a75e454bb21f9d779b54
reviewers: these docs publish periodically at:
https://docs.sqlalchemy.org/en/gerrit/4042/orm/queryguide/index.html
See the "last generated" timestamp near the bottom of the
page to ensure the latest version is up
Change includes some other adjustments:
* small typing fixes for end-user benefit
* removal of a bunch of old examples for patterns that nobody
uses or aren't really what we promote now
* modernization of some examples, including inheritance
Change-Id: I9929daab7797be9515f71c888b28af1209e789ff
the feature is enabled for all built in backends
when RETURNING is used,
except for Oracle that doesn't need it, and on
psycopg2 and mssql+pyodbc it is used for all INSERT statements,
not just those that use RETURNING.
third party dialects would need to opt in to the new feature
by setting use_insertmanyvalues to True.
Also adds dialect-level guards against using returning
with executemany where we dont have an implementation to
suit it. execute single w/ returning still defers to the
server without us checking.
Fixes: #6047Fixes: #7907
Change-Id: I3936d3c00003f02e322f2e43fb949d0e6e568304
This is so that custom Session and AsyncSession classes
can be typed for these factories. Added appropriate
typevars to `__call__()`, `__enter__()` and other methods
so that a custom Session or AsyncSession subclass is carried
through.
Fixes: #7656
Change-Id: Ia2b8c1f22b4410db26005c3285f6ba3d13d7f0e0
in this patch the asyncio/events.py module, which
existed only to raise errors when trying to attach event
listeners, is removed, as we were already coding an asyncio-specific
workaround in upstream Pool / Session to raise this error,
just moved the error out to the target and did the same thing
for Engine.
We also add an async_sessionmaker class. The initial rationale
here is because sessionmaker() is hardcoded to Session subclasses,
and there's not a way to get the use case of
sessionmaker(class_=AsyncSession) to type correctly without changing
the sessionmaker() symbol itself to be a function and not a class,
which gets too complicated for what this is. Additionally,
_SessionClassMethods has only three methods on it, one of which
is not usable with asyncio (close_all()), the others
not generally used from the session class.
Change-Id: I064a5fa5d91cc8d5bbe9597437536e37b4e801fe
both black and click were released in the past
few hours, and black 21.5b1 seems to suddenly
be failing on a missing symbol from click. just
update to the latest
Change-Id: Idf76732479a264f7f2245699a6bdaff018e3a123
the horizontal sharding API needs some work as it is
still exposing some legacy details, but in any case illustrate
how we can, for the moment, to use multiple schema translate
maps in a single session.
A lot more cleanup is needed in horizontal sharding, see #7837
Change-Id: Ia925e2226ecee9d747a8c4fc1772917f10bc505f
References: #7832
References: #7837
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
There's a few bits and pieces of code to support Python <= 3.6 which are no longer needed and can be removed, to slightly simplify the codebase.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #7544
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7544
Pull-request-sha: 282b4a9128
Change-Id: I9ddf15fcf72551d52e3f027f337c7fee4aa9083b
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.
Also update Black and other pre-commit hooks and re-format with Black.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570
Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Fixes: #7394
Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
Fixes: #6960
Even though a default driver still exists for
each dialect, remove most usages of `dialect://`
to encourage users to explicitly specify
`dialect+driver://`
Change-Id: I0ad42167582df509138fca64996bbb53e379b1af
- Fixed import to avoid MovedIn20Warning.
- Separated drop_all() and create_all() into separate
context managers to avoid dropping and creating the
same table within the same transaction. Apparently
some databases (e.g., CockroachDB) don't allow that.
Change-Id: Id26d7d719871a75ffb78c6af589658666802fb2f
Repaired the examples in examples/versioned_rows to use SQLAlchemy 1.4 APIs
correctly; these examples had been missed when API changes like removing
"passive" from ``Session.is_modified()`` were made as well as the
``do_orm_execute()`` event hook were added.
Fixes: #7169
Change-Id: I30930a3b185dc0f73be4467faa2f97c5ebf1a36d
Updated elementtree example to replace the deprecated classical mappings
interface with the current `registry.map_imperatively()` version.
The examples STILL use classical mappings, so the ticket remains open.
This update merely removes the deprecated API in favor of the (temporary?) replacement.
Change-Id: I8ddb86b76d82ac9d87a55edb0225e6b2f6ab4940
while I dont like this approach very much, people will likely
be asking for it a lot, so represent the most correct and
efficient form we can handle right now.
Added missing ``**kw`` arguments to the
:meth:`_asyncio.AsyncSession.connection` method.
Change-Id: Idadae2a02a4d96ecb96a5723ce64d017ab4c6217
References: https://github.com/sqlalchemy/sqlalchemy/discussions/6965
Fixed an issue in the ``CacheKey.to_offline_string()`` method used by the
dogpile.caching example where attempting to create a proper cache key from
the special "lambda" query generated by the lazy loader would fail to
include the parameter values, leading to an incorrect cache key.
Fixes: #6858
Change-Id: Ice27087583c6f3ff79cf7d5b879e5dd0a4e58158
I purposely didn't spend much documentation writing
about implicit IO when I first pushed out the asyncio
extension because I wanted to get a sense on what kinds
of issues people had. Now we know and the answer is
predictably "all of them". List out all the known
implicit IO points and how to avoid them. Also rename
the "adapting lazy loads" section, so that the title is less
suggestive that this is a necessary technique.
References: #5926
Change-Id: I3933b74bd37a5b06989531adbeade34347db679b
Added new attribute :attr:`_orm.ORMExecuteState.is_column_load` to indicate
that a :meth:`_orm.SessionEvents.do_orm_execute` handler that a particular
operation is a primary-key-directed column attribute load, such as from an
expiration or a deferred attribute, and that WHERE criteria or additional
loader options should not be added to the query. This has been added to
the examples which illustrate the :func:`_orm.with_loader_criteria` option.
The :func:`_orm.with_loader_criteria` option has been modified so that it
will never apply its criteria to the SELECT statement for an ORM refresh
operation, such as that invoked by :meth:`_orm.Session.refresh` or whenever
an expired attribute is loaded. These queries are only against the
primary key row of the object that is already present in memory so there
should not be additional criteria added.
Added doc caveats for using lambdas.
Added test coverage for most ORMExecuteState flags and fixed a few
basic access issues.
Change-Id: I6707e4cf0dc95cdfb8ce93e5ca22ead86074baa7
References: #5760Fixes: #5761Fixes: #5762
The :class:`_schema.Table` class now raises a deprecation warning
when columns with the same name are defined. To replace a column a new
parameter :paramref:`_schema.Table.append_column.replace_existing` was
added to the :meth:`_schema.Table.append_column` method.
The :meth:`_expression.ColumnCollection.contains_column` will now
raises an error when called with a string, suggesting the caller
to use ``in`` instead.
Co-authored-by: Federico Caselli <cfederico87@gmail.com>
Change-Id: I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.
Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function. it does not
yet fully address other issues such as keyword arguments passed
to the table.select().
Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.
Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
Using the approach introduced at
https://gist.github.com/zzzeek/6287e28054d3baddc07fa21a7227904e
We can now create asyncio endpoints that are then handled
in "implicit IO" form within the majority of the Core internals.
Then coroutines are re-exposed at the point at which we call
into asyncpg methods.
Patch includes:
* asyncpg dialect
* asyncio package
* engine, result, ORM session classes
* new test fixtures, tests
* some work with pep-484 and a short plugin for the
pyannotate package, which seems to have so-so results
Change-Id: Idbcc0eff72c4cad572914acdd6f40ddb1aef1a7d
Fixes: #3414
* major additions to 1.4 migration doc; removed additional
verbosity regarding caching methodology and reorganized the
doc to present itself more as a "what's changed" guide
* as we now have a path for asyncio, update that doc so that
we aren't spreading obsolete information
* updates to the 2.0 migration guide with latest info, however
this is still an architecture doc and not a migration guide
yet, will need further rework.
* start really talking about 1.x vs. 2.0 style everywhere. Querying
is most of the docs so this is going to be a prominent
theme, start getting it to fit in
* Add introductory documentation for ORM example sections as these
are too sparse
* new documentation for do_orm_execute(), many separate sections,
adding deprecation notes to before_compile() and similar
* new example suites to illustrate do_orm_execute(),
with_loader_criteria()
* modernized horizontal sharding examples and added a separate
example to distinguish between multiple databases and single
database w/ multiple tables use case
* introducing DEEP ALCHEMY, will use zzzeeksphinx 1.1.6
* no name for the alchemist yet however the dragon's name
is Flambé
Change-Id: Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541
The coercions system allows us to add in lambdas as arguments
to Core and ORM elements without changing them at all. By allowing
the lambda to produce a deterministic cache key where we can also
cheat and yank out literal parameters means we can move towards
having 90% of "baked" functionality in a clearer way right in
Core / ORM.
As a second step, we can have whole statements inside the lambda,
and can then add generation with __add__(), so then we have
100% of "baked" functionality with full support of ad-hoc
literal values.
Adds some more short_selects tests for the moment for comparison.
Other tweaks inside cache key generation as we're trying to
approach a certain level of performance such that we can
remove the use of "baked" from the loader strategies.
As we have not yet closed#4639, however the caching feature
has been fully integrated as of
b0cfa7379c, we will also
add complete caching documentation here and close that issue
as well.
Closes: #4639Fixes: #5380
Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
This patch makes several improvements in the area of
bulk updates and deletes as well as the new session mechanics.
RETURNING is now used for an UPDATE or DELETE statement
emitted for a diaelct that supports "full returning"
in order to satisfy the "fetch" strategy; this currently
includes PostgreSQL and SQL Server. The Oracle dialect
does not support RETURNING for more than one row,
so a new dialect capability "full_returning" is added
in addition to the existing "implicit_returning", indicating
this dialect supports RETURNING for zero or more rows,
not just a single identity row.
The "fetch" strategy will gracefully degrade to
the previous SELECT mechanics for dialects that do not
support RETURNING.
Additionally, the "fetch" strategy will attempt to use
evaluation for the VALUES that were UPDATEd, rather
than just expiring the updated attributes. Values should
be evalutable in all cases where the value is not
a SQL expression.
The new approach also incurs some changes in the
session.execute mechanics, where do_orm_execute() event
handlers can now be chained to each return results;
this is in turn used by the handler to detect on a
per-bind basis if the fetch strategy needs to
do a SELECT or if it can do RETURNING. A test suite is
added to test_horizontal_shard that breaks up a single
UPDATE or DELETE operation among multiple backends
where some are SQLite and don't support RETURNING and
others are PostgreSQL and do.
The session event mechanics are corrected
in terms of the "orm pre execute" hook, which now
receives a flag "is_reentrant" so that the two
ORM implementations for this can skip on their work
if they are being called inside of ORMExecuteState.invoke(),
where previously bulk update/delete were calling its
SELECT a second time.
In order for "fetch" to get the correct identity when
called as pre-execute, it also requests the identity_token
for each mapped instance which is now added as an optional
capability of a SELECT for ORM columns. the identity_token
that's placed by horizontal_sharding is now made available
within each result row, so that even when fetching a
merged result of plain rows we can tell which row belongs
to which identity token.
The evaluator that takes place within the ORM bulk update and delete for
synchronize_session="evaluate" now supports the IN and NOT IN operators.
Tuple IN is also supported.
Fixes: #1653
Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
This commit includes that we've removed the "_orm_query"
attribute from compile state as well as query context.
The attribute created reference cycles and also added
method call overhead. As part of this change,
the interface for ORMExecuteState changes a bit, as well
as the interface for the horizontal sharding extension
which now deprecates the "query_chooser" callable
in favor of "execute_chooser", which receives the contextual
object. This will also work more nicely when we implement
the new execution path for bulk updates and deletes.
Pre-merge execution options for statement, connection,
arguments all up front in Connection. that way they
can be passed to the before_execute / after_execute events,
and the ExecutionContext doesn't have to merge as second
time. Core execute is pretty close to 1.3 now.
baked wasn't using the new one()/first()/one_or_none() methods,
fixed that.
Convert non-buffered cursor strategy to be a stateless
singleton. inline all the paths by which the strategy
gets chosen, oracle and SQL Server dialects make use of the
already-invoked post_exec() hook to establish the alternate
strategies, and this is actually much nicer than it was before.
Add caching to mapper instance processor for getters.
Identified a reference cycle per query that was showing
up as a lot of gc cleanup, fixed that.
After all that, performance not budging much. Even
test_baked_query now runs with significantly fewer function
calls than 1.3, still 40% slower.
Basically something about the new patterns just makes
this slower and while I've walked a whole bunch of them
back, it hardly makes a dent. that said, the performance
issues are relatively small, in the 20-40% time increase
range, and the new caching feature
does provide for regular ORM and Core queries that
are cached, and they are faster than non-cached.
Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
This patch replaces the ORM execution flow with a
single pathway through Session.execute() for all queries,
including Core and ORM.
Currently included is full support for ORM Query,
Query.from_statement(), select(), as well as the
baked query and horizontal shard systems. Initial
changes have also been made to the dogpile caching
example, which like baked query makes use of a
new ORM-specific execution hook that replaces the
use of both QueryEvents.before_compile() as well
as Query._execute_and_instances() as the central
ORM interception hooks.
select() and Query() constructs alike can be passed to
Session.execute() where they will return ORM
results in a Results object. This API is currently
used internally by Query. Full support for
Session.execute()->results to behave in a fully
2.0 fashion will be in later changesets.
bulk update/delete with ORM support will also
be delivered via the update() and delete()
constructs, however these have not yet been adapted
to the new system and may follow in a subsequent
update.
Performance is also beginning to lag as of this
commit and some previous ones. It is hoped that
a few central functions such as the coercions
functions can be rewritten in C to re-gain
performance. Additionally, query caching
is now available and some subsequent patches
will attempt to cache more of the per-execution
work from the ORM layer, e.g. column getters
and adapters.
This patch also contains initial "turn on" of the
caching system enginewide via the query_cache_size
parameter to create_engine(). Still defaulting at
zero for "no caching". The caching system still
needs adjustments in order to gain adequate performance.
Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
Federico Caselli