Repaired a deprecation warning class decorator that was preventing key
objects such as :class:`_engine.Connection` from having a proper
``__weakref__`` attribute, causing operations like Python standard library
``inspect.getmembers()`` to fail.
Fixes: #8115
Change-Id: Ifd0bc2325fb9dc9e1431998c308b7fc081968373
The :meth:`.DialectEvents.handle_error` event is now moved to the
:class:`.DialectEvents` suite from the :class:`.EngineEvents` suite, and
now participates in the connection pool "pre ping" event for those dialects
that make use of disconnect codes in order to detect if the database is
live. This allows end-user code to alter the state of "pre ping". Note that
this does not include dialects which contain a native "ping" method such as
that of psycopg2 or most MySQL dialects.
Fixes: #5648
Change-Id: I353d84a4f66f309d2467b7e67621db6b8c70411e
after some experimentation it seems mypy is more amenable
to the generic types being fully integrated rather than
having separate spin-off types. so key structures
like Result, Row, Select become generic. For DML
Insert, Update, Delete, these are spun into type-specific
subclasses ReturningInsert, ReturningUpdate, ReturningDelete,
which is fine since the "row-ness" of these constructs
doesn't happen until returning() is called in any case.
a Tuple based model is then integrated so that these
objects can carry along information about their return
types. Overloads at the .execute() level carry through
the Tuple from the invoked object to the result.
To suit the issue of AliasedClass generating attributes
that are dynamic, experimented with a custom subclass
AsAliased, but then just settled on having aliased()
lie to the type checker and return `Type[_O]`, essentially.
will need some type-related accessors for with_polymorphic()
also.
Additionally, identified an issue in Update when used
"mysql style" against a join(), it basically doesn't work
if asked to UPDATE two tables on the same column name.
added an error message to the specific condition where
it happens with a very non-specific error message that we
hit a thing we can't do right now, suggest multi-table
update as a possible cause.
Change-Id: I5eff7eefe1d6166ee74160b2785c5e6a81fa8b95
Fixed issue in :meth:`.Result.columns` method where calling upon
:meth:`.Result.columns` with a single index could in some cases,
particularly ORM result object cases, cause the :class:`.Result` to yield
scalar objects rather than :class:`.Row` objects, as though the
:meth:`.Result.scalars` method had been called. In SQLAlchemy 1.4, this
scenario emits a warning that the behavior will change in SQLAlchemy 2.0.
Fixes: #7953
Change-Id: I3c4ca3eecc2bfc85ad1c38000e5990d6dde80d22
Allow setting contextvar values inside async adapted event handlers.
Previously the value set to the contextvar would not be properly
propagated.
Fixes: #7937
Change-Id: I787aa869f8d057579e13e32c749f05f184ffd02a
Also adds some fixes to annotation-based mapping
that have come up, as well as starts to add more
pep-484 test cases
Change-Id: Ia722bbbc7967a11b23b66c8084eb61df9d233fee
the pep484 task becomes more intense as there is mounting
pressure to come up with a consistency in how data moves
from end-user to instance variable.
current thinking is coming into:
1. there are _typing._XYZArgument objects that represent "what the
user sent"
2. there's the roles, which represent a kind of "filter" for different
kinds of objects. These are mostly important as the argument
we pass to coerce().
3. there's the thing that coerce() returns, which should be what the
construct uses as its internal representation of the thing.
This is _typing._XYZElement.
but there's some controversy over whether or
not we should pass actual ClauseElements around by their role
or not. I think we shouldn't at the moment, but this makes the
"role-ness" of something a little less portable. Like, we have
to set DMLTableRole for TableClause, Join, and Alias, but then
also we have to repeat those three types in order to set up
_DMLTableElement.
Other change introduced here, there was a deannotate=True
for the left/right of a sql.join(). All tests pass without that.
I'd rather not have that there as if we have a join(A, B) where
A, B are mapped classes, we want them inside of the _annotations.
The rationale seems to be performance, but this performance can
be illustrated to be on the compile side which we hope is cached
in the normal case.
CTEs now accommodate for text selects including recursive.
Get typing to accommodate "util.preloaded" cleanly; add "preloaded"
as a real module. This seemed like we would have needed
pep562 `__getattr__()` but we don't, just set names in
globals() as we import them.
References: #6810
Change-Id: I34d17f617de2fe2c086fc556bd55748dc782faf0
hitting DML which is causing us to open up the
ColumnCollection structure a bit, as we do put anonymous
column expressions with None here. However, we still want
Table /TableClause to have named column collections that
don't return None, so parametrize the "key" in this
collection also.
* rename some "immutable" elements to "readonly". we change
the contents of immutablecolumncollection underneath, so it's
not "immutable"
Change-Id: I2593995a4e5c6eae874bed5bf76117198be8ae97
This is done in 1.4 style so it can be backported to 1.4.
Will put this up as is, we can work on it. For 2.0, the
ORM mapping will be updated to mapped_column() style when
we do the full pass.
Change-Id: Icfdf81449973844dac244b3a107ce955a7d3b16c
Fixed issues where a descriptive error message was not raised for some
classes of event listening with an async engine, which should instead be a
sync engine instance.
Change-Id: I00b9f4fe9373ef5fd5464fac10651cc4024f648e
All modules in sqlalchemy.engine are strictly
typed with the exception of cursor, default, and
reflection. cursor and default pass with non-strict
typing, reflection is waiting on the multi-reflection
refactor.
Behavioral changes:
* create_connect_args() methods return a tuple of list,
dict, rather than a list of list, dict
* removed allow_chars parameter from
pyodbc connector ._get_server_version_info()
method
* the parameter list passed to do_executemany is now
a list in all cases. previously, this was being run
through dialect.execute_sequence_format, which
defaults to tuple and was only intended for individual
tuple params.
* broke up dialect.dbapi into dialect.import_dbapi
class method and dialect.dbapi module object. added
a deprecation path for legacy dialects. it's not
really feasible to type a single attr as a classmethod
vs. module type. The "type_compiler" attribute also
has this problem with greater ability to work around,
left that one for now.
* lots of constants changing to be Enum, so that we can
type them. for fixed tuple-position constants in
cursor.py / compiler.py (which are used to avoid the
speed overhead of namedtuple), using Literal[value]
which seems to work well
* some tightening up in Row regarding __getitem__, which
we can do since we are on full 2.0 style result use
* altered the set_connection_execution_options and
set_engine_execution_options event flows so that the
dictionary of options may be mutated within the event
hook, where it will then take effect as the actual
options used. Previously, changing the dict would
be silently ignored which seems counter-intuitive
and not very useful.
* A lot of DefaultDialect/DefaultExecutionContext
methods and attributes, including underscored ones, move
to interfaces. This is not fully ideal as it means
the Dialect/ExecutionContext interfaces aren't publicly
subclassable directly, but their current purpose
is more of documentation for dialect authors who should
(and certainly are) still be subclassing the DefaultXYZ
versions in all cases
Overall, Result was the most extremely difficult class
hierarchy to type here as this hierarchy passes through
largely amorphous "row" datatypes throughout, which
can in fact by all kinds of different things, like
raw DBAPI rows, or Row objects, or "scalar"/Any, but
at the same time these types have meaning so I tried still
maintaining some level of semantic markings for these,
it highlights how complex Result is now, as it's trying
to be extremely efficient and inlined while also being
very open-ended and extensible.
Change-Id: I98b75c0c09eab5355fc7a33ba41dd9874274f12a
large patch to get ORM / typing efforts started.
this is to support adding new test cases to mypy,
support dropping sqlalchemy2-stubs entirely from the
test suite, validate major ORM typing reorganization
to eliminate the need for the mypy plugin.
* New declarative approach which uses annotation
introspection, fixes: #7535
* Mapped[] is now at the base of all ORM constructs
that find themselves in classes, to support direct
typing without plugins
* Mypy plugin updated for new typing structures
* Mypy test suite broken out into "plugin" tests vs.
"plain" tests, and enhanced to better support test
structures where we assert that various objects are
introspected by the type checker as we expect.
as we go forward with typing, we will
add new use cases to "plain" where we can assert that
types are introspected as we expect.
* For typing support, users will be much more exposed to the
class names of things. Add these all to "sqlalchemy" import
space.
* Column(ForeignKey()) no longer needs to be `@declared_attr`
if the FK refers to a remote table
* composite() attributes mapped to a dataclass no longer
need to implement a `__composite_values__()` method
* with_variant() accepts multiple dialect names
Change-Id: I22797c0be73a8fbbd2d6f5e0c0b7258b17fe145d
Fixes: #7535Fixes: #7551
References: #6810
Starting to set up practices and conventions to
get the library typed.
Key goals for typing are:
1. whole library can pass mypy without any strict
turned on.
2. we can incrementally turn on some strict flags on a per-package/
module basis, as here we turn on more strictness for sqlalchemy.util, exc,
and log
3. mypy ORM plugin tests work fully without sqlalchemy2-stubs
installed
4. public facing methods all have return types, major parameter
signatures filled in also
5. Foundational elements like util etc. are typed enough so that
we can use them in fully typed internals higher up the stack.
Conventions set up here:
1. we can use lots of config in setup.cfg to limit where mypy
is throwing errors and how detailed it should be in different
packages / modules. We can use this to push up gerrits
that will pass tests fully without everything being typed.
2. a new tox target pep484 is added. this links to a new jenkins
pep484 job that works across all projects (alembic, dogpile, etc.)
We've worked around some mypy bugs that will likely
be around for awhile, and also set up some core practices
for how to deal with certain things such as public_factory
modules (mypy won't accept a module from a callable at all,
so need to use simple type checking conditionals).
References: #6810
Change-Id: I80be58029896a29fd9f491aa3215422a8b705e12
The warnings plugin lets us set the filters up
in the config, and as our filter requirements are now
simple we can just set this up.
additionally pytest now recommends pyproject.toml, since
we fully include this now, let's move it there.
the pytest logging plugin seems to not be any problem either
at the moment, so re-enable that. if it becomes apparent
whatever the problem was (which was probably that it was just
surprising, or something) we can disable it again and comment
what the reason was.
Change-Id: Ia9715533b01f72aa5fdcf6a27ce75b76f829fa43
start applying foundational annotations to key
elements.
two main elements addressed here:
1. removal of public_factory() and replacement with
explicit functions. this just works much better with
typing.
2. typing support for column expressions and operators.
The biggest part of this involves stubbing out all the
ColumnOperators methods under ColumnElement in a
TYPE_CHECKING section. Took me a while to see this
method vs. much more complicated things I thought
I needed.
Also for this version implementing #7519, ColumnElement
types against the Python type and not TypeEngine. it is
hoped this leads to easier transferrence between ORM/Core
as well as eventual support for result set typing.
Not clear yet how well this approach will work and what
new issues it may introduce.
given the current approach we now get full, rich typing for
scenarios like this:
from sqlalchemy import column, Integer, String, Boolean
c1 = column('a', String)
c2 = column('a', Integer)
expr1 = c2.in_([1, 2, 3])
expr2 = c2 / 5
expr3 = -c2
expr4_a = ~(c2 == 5)
expr4_b = ~column('q', Boolean)
expr5 = c1 + 'x'
expr6 = c2 + 10
Fixes: #7519Fixes: #6810
Change-Id: I078d9f57955549f6f7868314287175f6c61c44cb
Finalize all remaining removed-in-2.0 changes so that we
can begin doing pep-484 typing without old things
getting in the way (we will also have to do public_factory).
note there are a few "moved_in_20()" and "became_legacy_in_20()"
warnings still in place. The SQLALCHEMY_WARN_20 variable
is now removed.
Also removed here are the legacy "in place mutators" for Select
statements, and some keyword-only argument signatures in Core
have been added.
Also in the big change department, the ORM mapper() function
is removed entirely; the Mapper class is otherwise unchanged,
just the public-facing API function. Mappers are now always
given a registry in which to participate, however the
argument signature of Mapper is not changed. ideally "registry"
would be the first positional argument.
Fixes: #7257
Change-Id: Ic70c57b9f1cf7eb996338af5183b11bdeb3e1623
* :meth:`_orm.Query.join` no longer accepts the "aliased" and
"from_joinpoint" arguments
* :meth:`_orm.Query.join` no longer accepts chains of multiple join
targets in one method call.
* ``Query.from_self()`` and ``Query.with_polymorphic()``
are removed.
Change-Id: I534d04b53a538a4fc374966eb2bc8eb98a16497d
References: #7257
### Description
Found via `codespell -q 3 -L ba,crate,datas,froms,gord,hist,inh,nd,selectin,strat,ue`
Also added codespell to the pep8 tox env
### Checklist
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
Closes: #7338
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7338
Pull-request-sha: 0deac22193
Change-Id: Icd61db31c8dc655d4a39d8a304194804d08555fe
the discussion at #7387 refers to a condition that seems
to happen in the wild also, such as [1] [2] [3], it's not
entirely clear why this specific spot is how this occurs,
however it's maybe that when the connection is being acquired
from the pool, under load there might be a wait on the connection
pool, leading to more time for another errant thread to be
calling .close(), just a theory.
in this patch we propose using decorators and context managers
along with declarative state declarations to block reentrant
or concurrent calls to methods that conflict with expected
state changes.
The :class:`_orm.Session` (and by extension :class:`.AsyncSession`) now has
new state-tracking functionality that will proactively trap any unexpected
state changes which occur as a particular transactional method proceeds.
This is to allow situations where the :class:`_orm.Session` is being used
in a thread-unsafe manner, where event hooks or similar may be calling
unexpected methods within operations, as well as potentially under other
concurrency situations such as asyncio or gevent to raise an informative
message when the illegal access first occurs, rather than passing silently
leading to secondary failures due to the :class:`_orm.Session` being in an
invalid state.
[1] https://stackoverflow.com/questions/25768428/sqlalchemy-connection-errors
[2] https://groups.google.com/g/sqlalchemy/c/n5oVX3v4WOw
[3] https://github.com/cosmicpython/code/issues/23Fixes: #7433
Change-Id: I699b935c0ec4e5a63f12cf878af6f7a92a30a3aa
The architecture of Load is mostly rewritten here.
The change includes removal of the "pluggable" aspect
of the loader options, which would patch new methods onto
Load. This has been replaced by normal methods that
respond normally to typing annotations. As part of this
change, the bake_loaders() and unbake_loaders() options,
which have no effect since 1.4 and were unlikely to be
in any common use, have been removed.
Additionally, to support annotations for methods that
make use of @decorator, @generative etc., modified
format_argspec_plus to no longer return "args", instead
returns "grouped_args" which is always grouped and
allows return annotations to format correctly.
Fixes: #6986
Change-Id: I6117c642345cdde65a64389bba6057ddd5374427
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet
Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.
See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharingFixes: #7256
Change-Id: I2930ef1894b5048210384728118e586e813f6a76
Signed-off-by: Federico Caselli <cfederico87@gmail.com>
Implemented support for the test suite to run correctly under Pytest 7.
Previously, only Pytest 6.x was supported for Python 3, however the version
was not pinned on the upper bound in tox.ini. Pytest is not pinned in
tox.ini to be lower than version 8 so that SQLAlchemy versions released
with the current codebase will be able to be tested under tox without
changes to the environment. Much thanks to the Pytest developers for
their help with this issue.
Change-Id: I3b12166199be2b913ee16e78b3ebbff415654396
Fixed regression where the :meth:`_engine.CursorResult.fetchmany` method
would fail to autoclose a server-side cursor (i.e. when ``stream_results``
or ``yield_per`` is in use, either Core or ORM oriented results) when the
results were fully exhausted.
All :class:`_result.Result` objects will now consistently raise
:class:`_exc.ResourceClosedError` if they are used after a hard close,
which includes the "hard close" that occurs after calling "single row or
value" methods like :meth:`_result.Result.first` and
:meth:`_result.Result.scalar`. This was already the behavior of the most
common class of result objects returned for Core statement executions, i.e.
those based on :class:`_engine.CursorResult`, so this behavior is not new.
However, the change has been extended to properly accommodate for the ORM
"filtering" result objects returned when using 2.0 style ORM queries,
which would previously behave in "soft closed" style of returning empty
results, or wouldn't actually "soft close" at all and would continue
yielding from the underlying cursor.
As part of this change, also added :meth:`_result.Result.close` to the base
:class:`_result.Result` class and implemented it for the filtered result
implementations that are used by the ORM, so that it is possible to call
the :meth:`_engine.CursorResult.close` method on the underlying
:class:`_engine.CursorResult` when the the ``yield_per`` execution option
is in use to close a server side cursor before remaining ORM results have
been fetched. This was again already available for Core result sets but the
change makes it available for 2.0 style ORM results as well.
Fixes: #7274
Change-Id: Id4acdfedbcab891582a7f8edd2e2e7d20d868e53
most of the work for aliased / from_joinpoint has been done
already as I added all new tests for these and moved
most aliased/from_joinpoint to test/orm/test_deprecations.py
already
Change-Id: Ia23e332dec183de17b2fb9d89d946af8d5e89ae7
Implemented proper ``__reduce__()`` methods for all SQLAlchemy exception
objects to ensure they all support clean round trips when pickling, as
exception objects are often serialized for the purposes of various
debugging tools.
Fixes #7077Closes: #7078
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7078
Pull-request-sha: 8ba69f2653
Change-Id: Id62f8d351cd9180c441ffa9201efcf5f1876bf83
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
`__class_getitem__` to support generics was introduced in Python 3.7.
In 3.9 some built-ins were made generic but the functionality
for user-defined classes has been there since 3.7.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #7049
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7049
Pull-request-sha: 7bd16d416d
Change-Id: I4f45b9585d263636e613cc5c86e710fa31c7d283
The setup requirements have been modified such ``greenlet`` is a default
requirement only for those platforms that are well known for ``greenlet``
to be installable and for which there is already a pre-built binary on
pypi; the current list is ``x86_64 aarch64 ppc64le amd64 win32``. For other
platforms, greenlet will not install by default, which should enable
installation and test suite running of SQLAlchemy 1.4 on platforms that
don't support ``greenlet``, excluding any asyncio features. In order to
install with the ``greenlet`` dependency included on a machine architecture
outside of the above list, the ``[asyncio]`` extra may be included by
running ``pip install sqlalchemy[asyncio]`` which will then attempt to
install ``greenlet``.
Additionally, the test suite has been repaired so that tests can complete
fully when greenlet is not installed, with appropriate skips for
asyncio-related tests.
Fixes: #6136
Change-Id: I8f3a1c00a4a8b6a273484af1da1f7aaadf588ae7
Fixed bug in asyncio implementation where the greenlet adaptation system
failed to propagate ``BaseException`` subclasses, most notably including
``asyncio.CancelledError``, to the exception handling logic used by the
engine to invalidate and clean up the connection, thus preventing
connections from being correctly disposed when a task was cancelled.
Fixes: #6652
Change-Id: Id3809e6c9e7bced46a7a3b5a0d1906c4168dc4fc
Adjusted the means by which classes such as :class:`_orm.scoped_session`
and :class:`_asyncio.AsyncSession` are generated from the base
:class:`_orm.Session` class, such that custom :class:`_orm.Session`
subclasses such as that used by Flask-SQLAlchemy don't need to implement
positional arguments when they call into the superclass method, and can
continue using the same argument styles as in previous releases.
Fixes: #6285
References: https://github.com/pallets/flask-sqlalchemy/issues/953
Change-Id: I8612ab33743625e70eb158efceb0636d783c92a5
Python 3.10 fixes the issue where the loop would bind to a queue
on instantiation. Now an object binds to the loop only when first
needs it. I've looked at queues, but I'm assuming locks behave
the same,
Change-Id: Ibb8d263cdea230e9c85709528c21da31d0df2e65
Resolved various deprecation warnings which were appearing as of Python
version 3.10.0b1.
block aiomysql on python 3.10 as they are using the "loop" argument
that's removed
sqlcipher-binary has no builds on 3.10, block it for 3.10
Fixes: #6540Fixes: #6543
Change-Id: Iec1e3881fb289878881ae043b1a18c3ecdf5f077
This breaks data.rst into three separate sub-sections,
as SELECT is getting very long. It then adds sections
on select() + text/literal_column as well as unions
and set operations, and also tries to improve the
ORDER BY section a bit.
Change-Id: Id90e6b4ff3699b2bbcb6e2eebbd23193e2ede00a
Fixed an issue that prevented using ``scalar_one`` or
``scalar_one_or_none()`` after a call to ``unique``.
Additionally includes some clarifications in result.py
and also removes pep-484 annotations for now as these
are duplicate on top of sqlalchemy2-stubs.
Fixes: #6299
Change-Id: Ia04f3d078c7a4f0d8488745e43d2fd63b60de9a0
Tests here are failing for python 3.6 due to the lack
of asyncio.run(). It seems to be non-trivial to vendor
a working version of this in Python 3.6 as the tests here
are running it in alternate threads.
The python documentation imports everything directly from the
asyncio package, and it seems that py < 3.8 does not have the
asyncio.exception module
Closes: #5865
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5865
Pull-request-sha: 35cc1fa3f6
Change-Id: I9398c9fb2aa87f3228ce2f59277de732091bd541
Log an informative message if a connection is not closed
and the gc is reclaiming it when using an async dpapi, that
does not support running IO at that stage.
The ``AsyncAdaptedQueue`` used by default on async dpapis
should instantiate a queue only when it's first used
to avoid binding it to a possibly wrong event loop.
Fixes: #5823
Change-Id: Ibfc50e209b1937ae3d6599ae7997f028c7a92c33
Mike Bayer