python3.14t is showing connections being referenced and not finalized.
explicit closes/disposes and such dont seem to be working but
ensuring globs are cleared out is working better
Change-Id: I0c53f7702027e37ad11bd6e15e828900c8f52213
Added support for the SQL ``CREATE VIEW`` statement via the new
:class:`.CreateView` DDL class. The new class allows creating database
views from SELECT statements, with support for options such as
``TEMPORARY``, ``IF NOT EXISTS``, and ``MATERIALIZED`` where supported by
the target database. Views defined with :class:`.CreateView` integrate with
:class:`.MetaData` for automated DDL generation and provide a
:class:`.Table` object for querying.
this alters the CreateTableAs commit:
* adds the ability for Table to be associated with Create and Drop
DDL constructs
* Adds CreateView variant of CreateTableAs
* Both associate themselves with Table so they take place in
create_all/create/drop_all/drop
Fixes: #181
Change-Id: If3e568d3d6a6ce19e3d15198c3fbbe06bd847c83
Implemented initial support for free-threaded Python by adding new tests
and reworking the test harness and GitHub Actions to include Python 3.13t
and Python 3.14t in test runs. Two concurrency issues have been identified
and fixed: the first involves initialization of the ``.c`` collection on a
``FromClause``, a continuation of 🎫`12302`, where an optional mutex
under free-threading is added; the second involves synchronization of the
pool "first_connect" event, which first received thread synchronization in
🎫`2964`, however under free-threading the creation of the mutex
itself runs under the same free-threading mutex. Initial pull request and
test suite courtesy Lysandros Nikolaou.
py313t: yes
py314t: yes
Fixes: #12881Closes: #12882
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12882
Pull-request-sha: 53d65d96b9
Co-authored-by: Mike Bayer <mike_mp@zzzcomputng.com>
Change-Id: I2e4f2e9ac974ab6382cb0520cc446b396d9680a6
For 2.0, we provide a truly "larger than memory collection"
implementation, a write-only collection that will never
under any circumstances implicitly load the entire
collection, even during flush.
This is essentially a much more "strict" version
of the "dynamic" loader, which in fact has a lot of
scenarios that it loads the full backing collection
into memory, mostly defeating its purpose.
Typing constructs are added that support
both the new feature WriteOnlyMapping as well as the
legacy feature DynamicMapping. These have been
integrated with "annotion based mapping" so that
relationship() uses these annotations to configure
the loader strategy as well.
additional changes:
* the docs triggered a conflict in hybrid's
"transformers" section, this section is hard-coded
to Query using a pattern that doesnt seem to have
any use and isn't part of the current select()
interface, so just removed this section
* As the docs for WriteOnlyMapping are very long,
collections.rst is broken up into two pages now.
Fixes: #6229Fixes: #7123
Change-Id: I6929f3da6e441cad92285e7309030a9bac4e429d
ahead of trying to get everything formatted, some more
flexibility so that we can use doctest for all
python + sql code, while still being able to tell the
test suite to not run doctests on a sample. All of the
"non-console python with SQL" in the docs is because I was
showing an example that I didn't want tested.
Change-Id: Iae876ae1ffd93c36b096c6c2d6048843ae9698c8
the sqlite builds on github actions seem to be very
inconsistent about versions and many don't support
RETURNING. ensure any tests that depend on RETURNING present
are marked as such.
Change-Id: I7a60a81fa70b90642448cdd58eda33212c3afebc
reviewers: these docs publish periodically at:
https://docs.sqlalchemy.org/en/gerrit/4042/orm/queryguide/index.html
See the "last generated" timestamp near the bottom of the
page to ensure the latest version is up
Change includes some other adjustments:
* small typing fixes for end-user benefit
* removal of a bunch of old examples for patterns that nobody
uses or aren't really what we promote now
* modernization of some examples, including inheritance
Change-Id: I9929daab7797be9515f71c888b28af1209e789ff
I screwed up a rebase or something so this was
temporarily in Ic51a12de3358f3a451bd7cf3542b375569499fc1
Change-Id: I847ee1336381221c0112b67854df022edf596b25
This is done in 1.4 style so it can be backported to 1.4.
Will put this up as is, we can work on it. For 2.0, the
ORM mapping will be updated to mapped_column() style when
we do the full pass.
Change-Id: Icfdf81449973844dac244b3a107ce955a7d3b16c
This breaks data.rst into three separate sub-sections,
as SELECT is getting very long. It then adds sections
on select() + text/literal_column as well as unions
and set operations, and also tries to improve the
ORDER BY section a bit.
Change-Id: Id90e6b4ff3699b2bbcb6e2eebbd23193e2ede00a
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones. Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them. A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable. New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.
From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow. A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.
As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively. The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.
Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session. The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic. A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function. The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all. Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.
As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.
An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2. It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.
py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2. For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.
Includes the following improvements:
Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`. Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.
For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.
Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board. the addition
of this flag has resolved the long-standing oracle "two task"
error problem.
For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically. There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite. Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.
Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.
Fixes: #5826Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
Add SelectBase.exists() method as it seems strange this is
not available already. The Exists construct itself does
not provide full SELECT-building capabilities so it makes
sense this should be used more like a scalar_subquery.
Make sure stream_results is getting set up when yield_per
is used, for 2.0 style statements as well. this was
hardcoded inside of Query.yield_per() and is now moved
to take place within QueryContext.
Change-Id: Icafcd4fd9b708772343d56edf40995c9e8f835d6
"Implicit autocommit", which is the COMMIT that occurs when a DML or DDL
statement is emitted on a connection, is deprecated and won't be part of
SQLAlchemy 2.0. A 2.0-style warning is emitted when autocommit takes
effect, so that the calling code may be adjusted to use an explicit
transaction.
As part of this change, DDL methods such as
:meth:`_schema.MetaData.create_all` when used against a
:class:`_engine.Engine` or :class:`_engine.Connection` will run the
operation in a BEGIN block if one is not started already.
The MySQL and MariaDB dialects now query from the information_schema.tables
system view in order to determine if a particular table exists or not.
Previously, the "DESCRIBE" command was used with an exception catch to
detect non-existent, which would have the undesirable effect of emitting a
ROLLBACK on the connection. There appeared to be legacy encoding issues
which prevented the use of "SHOW TABLES", for this, but as MySQL support is
now at 5.0.2 or above due to 🎫`4189`, the information_schema tables
are now available in all cases.
Fixes: #4846
Change-Id: I733a7e0e17477a63607fb9931c87c393bbd7ac57
Disabled the "unicode returns" check that runs on dialect startup when
running under Python 3, which for many years has occurred in order to test
the current DBAPI's behavior for whether or not it returns Python Unicode
or Py2K strings for the VARCHAR and NVARCHAR datatypes. The check still
occurs by default under Python 2, however the mechanism to test the
behavior will be removed in SQLAlchemy 2.0 when Python 2 support is also
removed.
This logic was very effective when it was needed, however now that Python 3
is standard, all DBAPIs are expected to return Python 3 strings for
character datatypes. In the unlikely case that a third party DBAPI does
not support this, the conversion logic within :class:`.String` is still
available and the third party dialect may specify this in its upfront
dialect flags by setting the dialect level flag ``returns_unicode_strings``
to one of :attr:`.String.RETURNS_CONDITIONAL` or
:attr:`.String.RETURNS_BYTES`, both of which will enable Unicode conversion
even under Python 3.
As part of this change, disabling testing of the doctest tutorials under
Python 2.
Fixes: #5315
Change-Id: I1260e894611409d3b7fe1a92bd90c52043bbcf19
Fix doctest error caused by the missing order by clause in the
tests introduced in Ia1bbe3248b4f7c74fbee06fedf76a6ce94cd28a6
Change-Id: I55b45690879ebbaa57bc62765fcdab06f5e9c6f3
Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which
is used for forwards compatibility with SQLAlchemy 2. This engine
features always-transactional behavior with autobegin.
Allow execution options per statement execution. This includes
that the before_execute() and after_execute() events now accept
an additional dictionary with these options, empty if not
passed; a legacy event decorator is added for backwards compatibility
which now also emits a deprecation warning.
Add some basic tests for execution, transactions, and
the new result object. Build out on a new testing fixture
that swaps in the future engine completely to start with.
Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a
Fixes: #4644
Added "from linting" as a built-in feature to the SQL compiler. This
allows the compiler to maintain graph of all the FROM clauses in a
particular SELECT statement, linked by criteria in either the WHERE
or in JOIN clauses that link these FROM clauses together. If any two
FROM clauses have no path between them, a warning is emitted that the
query may be producing a cartesian product. As the Core expression
language as well as the ORM are built on an "implicit FROMs" model where
a particular FROM clause is automatically added if any part of the query
refers to it, it is easy for this to happen inadvertently and it is
hoped that the new feature helps with this issue.
The original recipe is from:
https://github.com/sqlalchemy/sqlalchemy/wiki/FromLinter
The linter is now enabled for all tests in the test suite as well.
This has necessitated that a lot of the queries be adjusted to
not include cartesian products. Part of the rationale for the
linter to not be enabled for statement compilation only was to reduce
the need for adjustment for the many test case statements throughout
the test suite that are not real-world statements.
This gerrit is adapted from Ib5946e57c9dba6da428c4d1dee6760b3e978dda0.
Fixes: #4737
Change-Id: Ic91fd9774379f895d021c3ad564db6062299211c
Closes: #4830
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4830
Pull-request-sha: f8a21aa626
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
method, and its interaction with result-row processing, now allows
the columns passed to the method to be positionally matched with the
result columns in the statement, rather than matching on name alone.
The advantage to this includes that when linking a textual SQL statement
to an ORM or Core table model, no system of labeling or de-duping of
common column names needs to occur, which also means there's no need
to worry about how label names match to ORM columns and so-forth. In
addition, the :class:`.ResultProxy` has been further enhanced to
map column and string keys to a row with greater precision in some
cases. fixes#3501
- reorganize the initialization of ResultMetaData for readability
and complexity; use the name "cursor_description", define the
task of "merging" cursor_description with compiled column information
as its own function, and also define "name extraction" as a separate task.
- fully change the name we use in the "ambiguous column" error to be the
actual name that was ambiguous, modify the C ext also
are now exercised within the normal unit test suite in both Python
2 and Python 3.
- remove the old testdocs.py runner and replace with test/base/test_tutorials.py
- use pytest's unicode fixer so that we can test for unicode strings
in both py2k/3k
- use py3k format overall for prints, exceptions
- add other fixers to guarantee deterministic results
- add skips and ellipses to outputs that aren't worth matching
Michael Bayer