Commit Graph

113 Commits

Author SHA1 Message Date
Mike Bayer 3d6a109c83 drop a 400 ton anvil on oracle 23c
this DB is extremely erratic in being able to connect.   Add
a brute force connection retrier to all engines everywhere
(which for oracledb we can fortunately use their built-in feature
that also works).
This actually works and I can see it pausing under load, reconnecting,
and succeeding.  the problem is that absolutely every engine everywhere
needs this routine otherwise an engine without a retrier in it will
crash.    That then necessitates digging into testing_engine(),
making sure testing_engine() is used everywhere an engine that's going
to connect is used, then dealing with the fallout from that.

We also simplify some older workarounds for cx_oracle and
hack into config/provision to make oracledb seem like the primary
DBAPI for most tests.

testing_engine has been completely overhauled, making use of a new
post_configure_testing_engine() hook which moves and refines
the SQLite pool sharing and savepoint logic all into sqlite/provision.py
and also allows for cx_oracle to apply a retry event handler.

Change-Id: I4ea4c523effb878290d28b94d8925eb32fc5ae3b
2025-11-28 13:16:13 -05:00
Mike Bayer cf62005639 test oracle 23c, mariadb12; reduce backend use
one particular vector test wont run on oracle 23c free, so
just disable it.

added better skips for the rest of the vector tests and
fixed a deprecation issue.

this will be the first run on the new oracle23 on CI so we'll have to
see how this goes.

Also adjust for mariabdb12 being overly helpful with regards
to stale row updates.

as we are having trouble getting 23c to pass throug transaction
tests, i noted we have an explosion of tests due to the multiple
drivers, so this patch introduces __sparse_driver_backend__
for all tests where we want variety of
database server but there's no need to test every driver.
This should dramatically reduce the size of the test suite run

Change-Id: Ic8d3eb0a60e76b4c54c6bb4a721f90c81ede782b
2025-11-25 14:50:56 -05:00
Federico Caselli 1dd96d274d postgresql dialect table options
Support for storage parameters in ``CREATE TABLE`` using the ``WITH``
clause has been added. The ``postgresql_with`` dialect option of
:class:`_schema.Table` accepts a mapping of key/value options.

The PostgreSQL dialect now support reflection of table options, including
the storage parameters, table access method and table spaces. These options
are automatically reflected when autoloading a table, and are also
available via the :meth:`_engine.Inspector.get_table_options` and
:meth:`_engine.Inspector.get_multi_table_optionsmethod` methods.

Fixes: #10909
Closes: #12584
Closes: #12684
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12584
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12684
Pull-request-sha: a660459de9

Change-Id: I33e3d8b8eb0c02530933cb124e3d375ca4af7db2
2025-10-14 19:47:26 +02:00
Federico Caselli 5b49b59ab7 Improve postgresql reflection
Some improvements to the reflection of PostgreSQL to avoid
unnecessary queries when not needed.

Fixes: #12908
Change-Id: Ic3da50ee43670f26d3159f5ec9a235a9a1963b8a
2025-10-14 13:27:18 +00:00
Denis Laxalde 78af7c4f04 Reflect collation in types on PostgreSQL
Added support for reflection of collation in types for PostgreSQL.
The ``collation`` will be set only if different from the default
one for the type.

References #6511
Closes: #12510
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12510
Pull-request-sha: c0a390314e

Change-Id: I269a194d526a0689a4b38f10456d28539c73cffb
2025-10-14 09:26:38 -04:00
Federico Caselli e688617c92 deprecate inherit_schema argument
Named types such as :class:`_postgresql.ENUM` and
:class:`_postgresql.DOMAIN` (as well as the dialect-agnostic
:class:`_types.Enum` version) are now more strongly associated with the
:class:`_schema.MetaData` at the top of the table hierarchy and are
de-associated with any particular :class:`_schema.Table` they may be a part
of. This better represents how PostgreSQL named types exist independently
of any particular table, and that they may be used across many tables
simultaneously.  The change impacts the behavior of the "default schema"
for a named type, as well as the CREATE/DROP behavior in relationship to
the :class:`.MetaData` and :class:`.Table` construct.  The change also
includes a new :class:`.CheckFirst` enumeration which allows fine grained
control over "check" queries during DDL operations, as well as that the
:paramref:`_types.SchemaType.inherit_schema` parameter is deprecated and
will emit a deprecation warning when used.  See the migration notes for
full details.

Fixes: #12690
Change-Id: I9752a9d52774ae760ec3448f62fc5046c58d68f5
2025-09-24 16:33:58 -04:00
Federico Caselli 0620f7d352 Fix reflection of enum named intervalsomething
Fixes bug that would mistakenly interpret a domain or enum type
with name starting in ``interval`` as an ``INTERVAL`` type while
reflecting a table.

Fixes: #12744
Change-Id: I89ab287c3847ca545691afe73f26d86bf2337ae0
2025-07-28 18:40:25 +02:00
Denis Laxalde 0642541c63 Reflect index's column operator class on PostgreSQL
Fill the `postgresql_ops` key of PostgreSQL's `dialect_options` returned by get_multi_indexes() with a mapping from column names to the operator class, if it's not the default for respective data type.

As we need to join on ``pg_catalog.pg_opclass``, the table definition is added to ``postgresql.pg_catalog``.

Fixes #8664.

Closes: #12504
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12504
Pull-request-sha: 8fdf93e1b2

Change-Id: I8789c1e9d15f8cc9a7205f492ec730570f19bbcc
2025-05-28 15:37:36 -04:00
Denis Laxalde 6154aa1b50 Add missing requires in the tests for older postgresql version
Follow up commit 39bb17442c.

Closes: #12612
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12612
Pull-request-sha: 894276ff23

Change-Id: Ib8d47f11e34d6bb40d9a88d5f411c2d5fee70823
2025-05-21 03:23:12 -04:00
Denis Laxalde 3b7725dd12 Support postgresql_include in UniqueConstraint and PrimaryKeyConstraint
This is supported both for schema definition and reflection.

Fixes #10665.

Closes: #12485
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12485
Pull-request-sha: 1aabea7b55ece9fc0c6e069b777d4404ac01f964

Change-Id: I81d23966f84390dd1b03f0d13284ce6d883ee24e
2025-04-01 23:20:32 +02:00
Denis Laxalde 39bb17442c Support column list for foreign key ON DELETE SET actions on PostgreSQL
Added support for specifying a list of columns for ``SET NULL`` and ``SET
DEFAULT`` actions of ``ON DELETE`` clause of foreign key definition on
PostgreSQL.  Pull request courtesy Denis Laxalde.

Fixes: #11595
Closes: #12421
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12421
Pull-request-sha: d0394db706

Change-Id: I036a559ae4a8efafe9ba64d776a840bd785a7397
2025-03-17 14:02:24 -04:00
Federico Caselli b23b6db14a Improve identity column reflection
Add SQL typing to reflection query used to retrieve a the structure
of IDENTITY columns, adding explicit JSON typing to the query to suit
unusual PostgreSQL driver configurations that don't support JSON natively.

Fixed issue affecting PostgreSQL 17.3 and greater where reflection of
domains with "NOT NULL" as part of their definition would include an
invalid constraint entry in the data returned by
:meth:`_postgresql.PGInspector.get_domains` corresponding to an additional
"NOT NULL" constraint that isn't a CHECK constraint; the existing
``"nullable"`` entry in the dictionary already indicates if the domain
includes a "not null" constraint.   Note that such domains also cannot be
reflected on PostgreSQL 17.0 through 17.2 due to a bug on the PostgreSQL
side; if encountering errors in reflection of domains which include NOT
NULL, upgrade to PostgreSQL server 17.3 or greater.

Fixes: #11751
Change-Id: I8e69de51601dca3257186e38c6f699fbfd9014c6
2025-03-06 19:26:19 -05:00
Mike Bayer 42fe1109c6 modernize PG domain reflection test and skip for pg17.2
Fixes: #12174
Change-Id: If4b1c29d7ee62b2858f1ef9d75fe1c4c41217706
2024-12-12 12:01:16 -05:00
Thomas Stephenson 0b6a54811d Add pg DOMAIN type reflection
The PostgreSQL dialect now returns :class:`_postgresql.DOMAIN` instances
when reflecting a column that has a domain as type.
Previously the domain data type was returned instead.
As part of this change, the domain reflection was improved to also
return the collation of the text types.

Fixes: #10693
Closes: #10729
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10729
Pull-request-sha: adac164d19

Change-Id: I8730840de2e7e9649067191430eefa086bcf5e7b
2024-03-22 09:26:53 -04:00
Ellis Valentiner 890b84e169 Support reflecting no inherit check constraint in pg.
Added support for reflection of PostgreSQL CHECK constraints marked with
"NO INHERIT", setting the key ``no_inherit=True`` in the reflected data.
Pull request courtesy Ellis Valentiner.

Fixes: #10777
Closes: #10778
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10778
Pull-request-sha: 058082ff62

Change-Id: Ia33e29c0c57cf0076e8819311f4628d712fdc332
2024-01-11 11:58:03 -05:00
Mike Bayer cc4c0dd962 adjust view definition reflection test for pg16
Change-Id: Idd45f8f9dab14bfb0e3f69e82e8c7898ab7d87ba
2023-09-20 13:43:21 -04:00
pavelserchenia f6dc8b872d PG nulls not distinct support
Added support for PostgreSQL 10 ``NULLS NOT DISTINCT`` feature of
unique indexes and unique constraint using the dialect option
postgresql_nulls_not_distinct``.
Updated the reflection logic to also correctly take this option
into account.

Fixes: #8240
Closes: #9834
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9834
Pull-request-sha: 825a4ff13a

Change-Id: I6585fbb05ad32a131cf9ba25a59f7b229bce5b52
2023-06-06 17:58:32 +00:00
Federico Caselli 058c230cea Update black to v23.3.0 and flake8 to v6
This change could be added to .git-blame-ignore-revs

Change-Id: I7ba10052b26bc3c178d23fb50a1123d0aae965ca
2023-05-30 21:44:26 +02:00
Federico Caselli f45f4a3afc Improve oracle index reflection
Added reflection support in the Oracle dialect to expression based indexes
and the ordering direction of index expressions.

Fixes: #9597
Change-Id: I40e163496789774e9930f46823d2208c35eab6f8
2023-04-28 20:38:24 +02:00
mike bayer 60cc662861 Merge "Try running pyupgrade on the code" into main 2022-11-17 14:35:10 +00:00
Mike Bayer 200e70b974 accommodate NULL format_type()
Made an adjustment to how the PostgreSQL dialect considers column types
when it reflects columns from a table, to accommodate for alternative
backends which may return NULL from the PG ``format_type()`` function.

Fixes: #8748
Change-Id: I6178287aac567210a76afaa5805b825daa7fa4db
2022-11-16 20:11:18 -05:00
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
Federico Caselli 68a3374d5a Reflect expression-based indexes on PostgreSQL
The PostgreSQL dialect now supports reflection of expression based indexes.
The reflection is supported both when using
:meth:`_engine.Inspector.get_indexes` and when reflecting a
:class:`_schema.Table` using :paramref:`_schema.Table.autoload_with`.
Thanks to immerrr and Aidan Kane for the help on this ticket.

Fixes: #7442
Change-Id: I3e36d557235286c0f7f6d8276272ff9225058d48
2022-07-28 19:27:23 +02:00
cheremnov 5fb63bc142 Comments on (named) constraints
Adds support for comments on named constraints, including `ForeignKeyConstraint`, `PrimaryKeyConstraint`, `CheckConstraint`, `UniqueConstraint`, solving the [Issue 5667](https://github.com/sqlalchemy/sqlalchemy/issues/5667).

Supports only PostgreSQL backend.

### Description

Following the example of [Issue 1546](https://github.com/sqlalchemy/sqlalchemy/issues/1546), supports comments on constraints. Specifically, enables comments on _named_ ones — as I get it, PostgreSQL prohibits comments on unnamed constraints.

Enables setting the comments for named constraints like this:
```
Table(
   'example', metadata,
   Column('id', Integer),
   Column('data', sa.String(30)),
   PrimaryKeyConstraint(
       "id", name="id_pk", comment="id_pk comment"
    ),
   CheckConstraint('id < 100', name="cc1", comment="Id value can't exceed 100"),
   UniqueConstraint(['data'], name="uc1", comment="Must have unique data field"),
)
```

Provides the DDL representation for constraint comments and routines to create and drop them. Class `.Inspector` reflects constraint comments via methods like `get_check_constraints` .
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
- [ ] A short code fix
- [x] A new feature implementation
	- Solves the issue 5667.
	- The commit message includes `Fixes: 5667`.
	- Includes tests based on comment reflection.

**Have a nice day!**

Fixes: #5667
Closes: #7742
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7742
Pull-request-sha: 42a5d3c3e9

Change-Id: Ia60f578595afdbd6089541c9a00e37997ef78ad3
2022-06-29 09:13:37 +00:00
David Baumgold 017fd9ae06 Domain type
Added a new Postgresql :class:`_postgresql.DOMAIN` datatype, which follows
the same CREATE TYPE / DROP TYPE behaviors as that of PostgreSQL
:class:`_postgresql.ENUM`. Much thanks to David Baumgold for the efforts on
this.

Fixes: #7316
Closes: #7317
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7317
Pull-request-sha: bc9a82f010

Change-Id: Id8d7e48843a896de17d20cc466b115b3cc065132
2022-06-21 10:17:40 -04:00
Federico Caselli be576e7d88 Merge "rearchitect reflection for batched performance" into main 2022-06-18 21:08:27 +00:00
Federico Caselli db08a69948 rearchitect reflection for batched performance
Rearchitected the schema reflection API to allow some dialects to make use
of high performing batch queries to reflect the schemas of many tables at
once using much fewer queries. The new performance features are targeted
first at the PostgreSQL and Oracle backends, and may be applied to any
dialect that makes use of SELECT queries against system catalog tables to
reflect tables (currently this omits the MySQL and SQLite dialects which
instead make use of parsing the "CREATE TABLE" statement, however these
dialects do not have a pre-existing performance issue with reflection. MS
SQL Server is still a TODO).

The new API is backwards compatible with the previous system, and should
require no changes to third party dialects to retain compatibility;
third party dialects can also opt into the new system by implementing
batched queries for schema reflection.

Along with this change is an updated reflection API that is fully
:pep:`484` typed, features many new methods and some changes.

Fixes: #4379
Change-Id: I897ec09843543aa7012bcdce758792ed3d415d08
2022-06-18 14:57:26 -04:00
Federico Caselli 3bde1eae31 Normalize postgresql docs links to point to current
Change-Id: Ib7d3ea7ff3356ff8a2f935892d904a69dbc25c3e
2022-06-17 21:53:32 +02:00
Mike Bayer c90396fbe7 bump black to 22.3.0
both black and click were released in the past
few hours, and black 21.5b1 seems to suddenly
be failing on a missing symbol from click.  just
update to the latest

Change-Id: Idf76732479a264f7f2245699a6bdaff018e3a123
2022-03-28 15:58:35 -04:00
Mike Bayer de0b4db838 dont use exception catches for warnings; modernize xdist detection
Improvements to the test suite's integration with pytest such that the
"warnings" plugin, if manually enabled, will not interfere with the test
suite, such that third parties can enable the warnings plugin or make use
of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass.
Additionally, modernized the detection of the "pytest-xdist" plugin so that
plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
without breaking the test suite if xdist were still installed. Warning
filters that promote deprecation warnings to errors are now localized to
SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for
general Python deprecation warnings, so that non-SQLAlchemy deprecation
warnings emitted from pytest plugins should also not impact the test suite.

Fixes: #7599
Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
2022-01-22 19:17:10 -05:00
Hugo van Kemenade 146a349d81 Update Black's target-version to py37
<!-- Provide a general summary of your proposed changes in the Title field above -->

### Description
<!-- Describe your changes in detail -->

Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.

Also update Black and other pre-commit hooks and re-format with Black.

### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

**Have a nice day!**

Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570

Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
2022-01-05 12:41:32 -05:00
Gord Thompson 5ada58954a Reflect included columns as dialect_options
Fixed reflection of covering indexes to report ``include_columns`` as part
of the ``dialect_options`` entry in the reflected index dictionary, thereby
enabling round trips from reflection->create to be complete. Included
columns continue to also be present under the ``include_columns`` key for
backwards compatibility.

Fixes: #7382
Change-Id: I4f16b65caed3a36d405481690a3a92432b5efd62
2021-12-27 14:32:39 -05:00
Federico Caselli 0b95f0055b Remove object in class definition
References: #4600
Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
2021-11-22 15:03:17 +00:00
Kevin Kirsche 3ea321ce22 Fix various lib / test / examples typos (#7017)
* fix: lib/sqlalchemy/sql/lambdas.py

* fix: lib/sqlalchemy/sql/compiler.py

* fix: lib/sqlalchemy/sql/selectable.py

* fix: lib/sqlalchemy/orm/relationships.py

* fix: lib/sqlalchemy/dialects/mssql/base.py

* fix: lib/sql/test_compiler.py

* fix: lib/sqlalchemy/testing/requirements.py

* fix: lib/sqlalchemy/orm/path_registry.py

* fix: lib/sqlalchemy/dialects/postgresql/psycopg2.py

* fix: lib/sqlalchemy/cextension/immutabledict.c

* fix: lib/sqlalchemy/cextension/resultproxy.c

* fix: ./lib/sqlalchemy/dialects/oracle/cx_oracle.py

* fix: examples/versioned_rows/versioned_rows_w_versionid.py

* fix: examples/elementtree/optimized_al.py

* fix: test/orm/test_attribute.py

* fix: test/sql/test_compare.py

* fix: test/sql/test_type_expression.py

* fix: capitalization in test/dialect/mysql/test_compiler.py

* fix: typos in test/dialect/postgresql/test_reflection.py

* fix: typo in tox.ini comment

* fix: typo in /lib/sqlalchemy/orm/decl_api.py

* fix: typo in test/orm/test_update_delete.py

* fix: self-induced typo

* fix: typo in test/orm/test_query.py

* fix: typos in test/dialect/mssql/test_types.py

* fix: typo in test/sql/test_types.py
2021-09-11 22:26:59 +02:00
Mike Bayer a92942a531 Repair PGInspector
Fixed issue where the PostgreSQL :class:`.PGInspector`, when generated
against an :class:`_engine.Engine`, would fail for ``.get_enums()``,
``.get_view_names()``, ``.get_foreign_table_names()`` and
``.get_table_oid()`` when used against a "future" style engine and not the
connection directly.

Fixes: #6170
Change-Id: I8c3abdfb758305c2f7a96002d3644729f29c998b
2021-03-31 11:50:26 -04:00
Mike Bayer ab61d66dee Refine domain nullable rules for PostgreSQL reflection
Fixed issue in PostgreSQL reflection where a column expressing "NOT NULL"
will supersede the nullability of a corresponding domain.

Fixes #6161

Change-Id: I1a3de49afcdb952f71bd7a7cc7b264513c93eff5
2021-03-30 14:49:46 -04:00
Federico Caselli 3fe8d618e3 Fixed reflection of identity columns in tables
with mixed case names in PostgreSQL.

Fixes: #6129
Change-Id: I50480cec03fcb44a668c9b0f9c72950803b771d9
2021-03-25 21:08:21 +01:00
Mike Bayer f1e96cb087 reinvent xdist hooks in terms of pytest fixtures
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones.   Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them.   A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable.   New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.

From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow.  A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.

As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively.  The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.

Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session.   The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic.  A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function.  The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all.  Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.

As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.

An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2.  It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.

py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2.  For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.

Includes the following improvements:

Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`.  Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.

For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.

Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board.  the addition
of this flag has resolved the long-standing oracle "two task"
error problem.

For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically.  There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite.  Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.

Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.

Fixes: #5826
Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
2021-01-13 22:10:13 -05:00
Mike Bayer fd3c063dd6 remove metadata.bind use from test suite
importantly this means we can remove bound metadata from
the fixtures that are used by Alembic's test suite.

hopefully this is the last one that has to happen to allow
Alembic to be fully 1.4/2.0.

Start moving from @testing.provide_metadata to a pytest
metadata fixture.  This does not seem to have any negative
effects even though TablesTest uses a "self.metadata" attribute.

Change-Id: Iae6ab95938a7e92b6d42086aec534af27b5577d3
2021-01-03 13:22:29 -05:00
Mike Bayer ba5cbf9366 correct for "autocommit" deprecation warning
Ensure no autocommit warnings occur internally or
within tests.

Also includes fixes for SQL Server full text tests
which apparently have not been working at all for a long
time, as it used long removed APIs.  CI has not had
fulltext running for some years and is now installed.

Change-Id: Id806e1856c9da9f0a9eac88cebc7a94ecc95eb96
2020-12-11 13:26:05 -05:00
Mike Bayer 80be40300d Convert to autoload_with internally
Fixed bug where the now-deprecated ``autoload`` parameter was being called
internally within the reflection routines when a related table were
reflected.

Fixes: #5684
Change-Id: I6ab439a2f49ff1ae2d3c7a15b531cbafbc3cf594
2020-11-07 11:13:28 -05:00
jonathan vanasco 9ddbd585a6 Apply underscore naming to several more operators
The operator changes are:

* `isfalse` is now `is_false`
* `isnot_distinct_from` is now `is_not_distinct_from`
* `istrue` is now `is_true`
* `notbetween` is now `not_between`
* `notcontains` is now `not_contains`
* `notendswith` is now `not_endswith`
* `notilike` is now `not_ilike`
* `notlike` is now `not_like`
* `notmatch` is now `not_match`
* `notstartswith` is now `not_startswith`
* `nullsfirst` is now `nulls_first`
* `nullslast` is now `nulls_last`

Because these are core operators, the internal migration strategy for this
change is to support legacy terms for an extended period of time -- if not
indefinitely -- but update all documentation, tutorials, and internal usage
to the new terms.  The new terms are used to define the functions, and
the legacy terms have been deprecated into aliases of the new terms.

Fixes: #5435
Change-Id: Ifbd7cb1cdda5981990243c4fc4b4ff467dc132ac
2020-10-30 10:02:29 -04:00
Federico Caselli 7362d454f4 Add reflection for Identity columns
Added support for reflecting "identity" columns, which are now returned
as part of the structure returned by :meth:`_reflection.Inspector.get_columns`.
When reflecting full :class:`_schema.Table` objects, identity columns will
be represented using the :class:`_schema.Identity` construct.

Fixed compilation error on oracle for sequence and identity column
``nominvalue`` and ``nomaxvalue`` options that require no space in them.

Improved test compatibility with oracle 18.

As part of the support for reflecting :class:`_schema.Identity` objects,
the method :meth:`_reflection.Inspector.get_columns` no longer returns
``mssql_identity_start`` and ``mssql_identity_increment`` as part of the
``dialect_options``. Use the information in the ``identity`` key instead.

The mssql dialect will assume that at least MSSQL 2005 is used.
There is no hard exception raised if a previous version is detected,
but operations may fail for older versions.

Fixes: #5527
Fixes: #5324
Change-Id: If039fe637c46b424499e6bac54a2cbc0dc54cb57
2020-09-28 18:11:12 -04:00
Mike Bayer c3f102c9fe upgrade to black 20.8b1
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.

Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
2020-09-28 15:17:26 -04:00
Mike Bayer 7e864fc7b1 Create a framework to allow all SQLALCHEMY_WARN_20 to pass
As the test suite has widespread use of many patterns
that are deprecated, enable SQLALCHEMY_WARN_20 globally
for the test suite but then break the warnings filter
out into a whole list of all the individual warnings
we are looking for.  this way individual changesets
can target a specific class of warning, as many of these
warnings will indivdidually affect dozens of files
and potentially hundreds of lines of code.

Many warnings are also resolved here as this
patch started out that way.   From this point
forward there should be changesets that target a
subset of the warnings at a time.

For expediency, updates some migration 2.0 docs
for ORM as well.

Change-Id: I98b8defdf7c37b818b3824d02f7668e3f5f31c94
2020-09-16 12:31:05 -04:00
mike bayer 645be4aa24 Merge "Improve handling of covering indexes" 2020-09-12 19:46:50 +00:00
Gord Thompson 1a08d1aade Improve handling of covering indexes
Improved support for covering indexes (with INCLUDE columns). Added the
ability for postgresql to render CREATE INDEX statements with an INCLUDE
clause from Core. Index reflection also report INCLUDE columns separately
for both mssql and postgresql (11+).

Fixes: #4458
Change-Id: If0b82103fbc898cdaeaf6a6d2d421c732744acd6
2020-09-12 13:00:16 -04:00
mike bayer a2f2863f5a Merge "Raname `mustexit to must_exist`" 2020-09-12 16:36:00 +00:00
RamonWill 69502725db Reflect mssql/postgresql filtered/partial indexes
Added support for inspection / reflection of partial indexes / filtered
indexes, i.e. those which use the ``mssql_where`` or ``postgresql_where``
parameters, with :class:`_schema.Index`.   The entry is both part of the
dictionary returned by :meth:`.Inspector.get_indexes` as well as part of a
reflected :class:`_schema.Index` construct that was reflected.  Pull
request courtesy Ramon Williams.

**Have a nice day!**
Fixes: #4966

Closes: #5504
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5504
Pull-request-sha: b3018bac98

Change-Id: Icbb2f93d1545700718ccb5222097185b815f5dbc
2020-09-12 10:30:43 -04:00
Federico Caselli bcfb12f813 Raname `mustexit to must_exist`
:class:`_schema.Table` parameter ``mustexist`` has been renamed
to :paramref:`_schema.Table.must_exist` and will now warn when used.

Change-Id: I0b0ca6021f9f7cfbe2040bbc1125a2236ac79f53
2020-09-03 23:46:35 +02:00