Fixed issue where SQLite table reflection would fail for tables using
``WITHOUT ROWID`` and/or ``STRICT`` table options when the table contained
generated columns. The regular expression used to parse ``CREATE TABLE``
statements for generated column detection has been updated to properly
handle these SQLite table options that appear after the column definitions.
Pull request courtesy Tip ten Brink.
Fixes: #12864Closes: #12865
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12865
Pull-request-sha: 833420afc7
Change-Id: Ia25c05746055a66234531b4f1275bd3a2f3b44ad
Added a new concept of "operator classes" to the SQL operators supported by
SQLAlchemy, represented within the enum :class:`.OperatorClass`. The
purpose of this structure is to provide an extra layer of validation when a
particular kind of SQL operation is used with a particular datatype, to
catch early the use of an operator that does not have any relevance to the
datatype in use; a simple example is an integer or numeric column used with
a "string match" operator.
Fixes: #12736
Change-Id: I44f46d7326aef6847dbf0cf7a325833f8e347da6
Fixed and added test support for a few SQLite SQL functions hardcoded into
the compiler most notably the "localtimestamp" function which rendered with
incorrect internal quoting.
Fixes: #12566
Change-Id: Id5bd8dc7841f0afab7df031ba5c0854dab845a1d
Expanded the rules for when to apply parenthesis to a server default in DDL
to suit the general case of a default string that contains non-word
characters such as spaces or operators and is not a string literal.
Fixed issue in MySQL server default reflection where a default that has
spaces would not be correctly reflected. Additionally, expanded the rules
for when to apply parenthesis to a server default in DDL to suit the
general case of a default string that contains non-word characters such as
spaces or operators and is not a string literal.
Fixes: #12425
Change-Id: Ie40703dcd5fdc135025d676c01baba57ff3b71ad
Fixed issue that omitted the comma between multiple SQLite table extension
clauses, currently ``WITHOUT ROWID`` and ``STRICT``, when both options
:paramref:`.Table.sqlite_with_rowid` and :paramref:`.Table.sqlite_strict`
were configured at their non-default settings at the same time. Pull
request courtesy david-fed.
Fixes: #12368Closes: #12369
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12369
Pull-request-sha: 3c9ceffe82
Change-Id: I1a44fd2d655d0e6eaad8213a360879daca9e4f11
in I4d66ec1473321616a1707da324a7dfe7a61ec94e we added new tests
in the sqlite suite but we forgot to extend from fixtures.TestBase,
so these tests did not get run at all. repair tests
Change-Id: Iaec17a754e0ab1d4b43f063706b512ed335a7465
in 2.1 we want these structures to be cacheable, so start
by cleaning up types and adding coercions to enforce those types.
these will be more locked down in 2.1 as we will need to move
bound parameter coercion outside of compilation, but here
do some small starts and introduce in 2.0.
in one interest of cachability, a "literal_binds" that found
its way into SQLite's compiler is replaced with "literal_execute",
the difference being that the latter is cacheable. This literal
is apparently necessary to suit SQLite's query planner for
the "index criteria" portion of the on conflict clause that otherwise
can't work with a real bound parameter.
Change-Id: I4d66ec1473321616a1707da324a7dfe7a61ec94e
The changes made for SQLite CHECK constraint reflection in versions 2.0.33
and 2.0.34 , 🎫`11832` and 🎫`11677`, have now been fully
reverted, as users continued to identify existing use cases that stopped
working after this change. For the moment, because SQLite does not
provide any consistent way of delivering information about CHECK
constraints, SQLAlchemy is limited in what CHECK constraint syntaxes can be
reflected, including that a CHECK constraint must be stated all on a
single, independent line (or inline on a column definition) without
newlines, tabs in the constraint definition or unusual characters in the
constraint name. Overall, reflection for SQLite is tailored towards being
able to reflect CREATE TABLE statements that were originally created by
SQLAlchemy DDL constructs. Long term work on a DDL parser that does not
rely upon regular expressions may eventually improve upon this situation.
A wide range of additional cross-dialect CHECK constraint reflection tests
have been added as it was also a bug that these changes did not trip any
existing tests.
Fixes: #11840
Change-Id: Iaa4f9651d0c3dd5dbb530ccaa6688169eb7f3bb8
Fixed regression in SQLite reflection caused by 🎫`11677` which
interfered with reflection for CHECK constraints that were followed
by other kinds of constraints within the same table definition. Pull
request courtesy Harutaka Kawamura.
Fixes: #11832Closes: #11834
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/11834
Pull-request-sha: a10fcef45c
Change-Id: I50bcc7aa727f73be235895d154cd859f19adda09
Improvements to the regex used by the SQLite dialect to reflect the name
and contents of a UNIQUE constraint that is defined inline within a column
definition inside of a SQLite CREATE TABLE statement, accommodating for tab
characters present within the column / constraint line. Pull request
courtesy John A Stevenson.
Fixes: #11746Closes: #11759
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/11759
Pull-request-sha: 5752491154
Change-Id: I048a90c992bfaf6857c1150f50bf3c6cc5697095
Improvements to the regex used by the SQLite dialect to reflect the name
and contents of a CHECK constraint. Constraints with newline, tab, or
space characters in either or both the constraint text and constraint name
are now properly reflected. Pull request courtesy Jeff Horemans.
Fixes: #11677Closes: #11701
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/11701
Pull-request-sha: b2d629f186
Change-Id: I2dd06c778e7c130848e418a80004032eb6144e6d
Fixed reflection of computed column in SQLite to properly account
for complex expressions.
Fixes: #11582
Change-Id: I8e9fdda3e47c04b376973ee245b3175374a08f56
manually update the files to remove literal string concat on the same line,
since black does not seem to be making progress in handling these
Change-Id: I3c651374c5f3db5b8bc0c700328d67ca03743b7b
This test is failing on windows due to the new SQlite
provisioning that seemed to be failing to delete schema files
as they are still used by the main connection.
Change-Id: I51093212ebfe1053f26b279c56c8fec0408806da
Fixed bug that prevented SQLAlchemy to connect when using a very old
sqlite version (before 3.9) on python 3.8+.
Fixes: #9379
Change-Id: I10ca347398221c952e1a572dc6ef80e491d1f5cf
Fixed regression caused by new support for reflection of partial indexes on
SQLite added in 1.4.45 for 🎫`8804`, where the ``index_list`` pragma
command in very old versions of SQLite (possibly prior to 3.8.9) does not
return the current expected number of columns, leading to exceptions raised
when reflecting tables and indexes.
Fixes: #8969
Change-Id: If317cdcfc6782f7e180df329b6ea0ddb48ce2269
Changed how the positional compilation is performed. It's rendered by the compiler
the same as the pyformat compilation. The string is then processed to replace
the placeholders with the correct ones, and to obtain the correct order of the
parameters.
This vastly simplifies the computation of the order of the parameters, that in
case of nested CTE is very hard to compute correctly.
Reworked how numeric paramstyle behavers:
- added support for repeated parameter, without duplicating them like in normal
positional dialects
- implement insertmany support. This requires that the dialect supports out of
order placehoders, since all parameters that are not part of the VALUES clauses
are placed at the beginning of the parameter tuple
- support for different identifiers for a numeric parameter. It's for example
possible to use postgresql style placeholder $1, $2, etc
Added two new dialect based on sqlite to test "numeric" fully using
both :1 style and $1 style. Includes a workaround for SQLite's
not-really-correct numeric implementation.
Changed parmstyle of asyncpg dialect to use numeric, rendering with its native
$ identifiers
Fixes: #8926Fixes: #8849
Change-Id: I7c640467d49adfe6d795cc84296fc7403dcad4d6
Added support for the SQLite backend to reflect the "DEFERRABLE" and
"INITIALLY" keywords which may be present on a foreign key construct. Pull
request courtesy Michael Gorven.
Fixes: #8903Closes: #8904
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8904
Pull-request-sha: 52aa4cf774
Change-Id: I713906db1a458d8f1be39625841ca3bbc03ec835
Added support for reflection of expression-oriented WHERE criteria included
in indexes on the SQLite dialect, in a manner similar to that of the
PostgreSQL dialect. Pull request courtesy Tobias Pfeiffer.
Fixes: #8804Closes: #8806
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8806
Pull-request-sha: 539dfcb372
Change-Id: I0e34d47dbe2b9c1da6fce531363084843e5127a3
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures
Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
just in my own testing, if I say insert().return_defaults()
and stringify, I should see it, so make sure all the dialects
default to "insert_returning" etc. , with downgrade on
server version check.
Change-Id: Id64e78fcb03c48b5dcb0feb21cb9cc495edd15e9
Added new parameter to SQLite for reflection methods called
``sqlite_include_internal=True``; when omitted, local tables that start
with the prefix ``sqlite_``, which per SQLite documentation are noted as
"internal schema" tables such as the ``sqlite_sequence`` table generated to
support "AUTOINCREMENT" columns, will not be included in reflection methods
that return lists of local objects. This prevents issues for example when
using Alembic autogenerate, which previously would consider these
SQLite-generated tables as being remove from the model.
Fixes: #8234
Change-Id: I36ee7a053e04b6c46c912aaa0d7e035a5b88a4f9
Adds support for comments on named constraints, including `ForeignKeyConstraint`, `PrimaryKeyConstraint`, `CheckConstraint`, `UniqueConstraint`, solving the [Issue 5667](https://github.com/sqlalchemy/sqlalchemy/issues/5667).
Supports only PostgreSQL backend.
### Description
Following the example of [Issue 1546](https://github.com/sqlalchemy/sqlalchemy/issues/1546), supports comments on constraints. Specifically, enables comments on _named_ ones — as I get it, PostgreSQL prohibits comments on unnamed constraints.
Enables setting the comments for named constraints like this:
```
Table(
'example', metadata,
Column('id', Integer),
Column('data', sa.String(30)),
PrimaryKeyConstraint(
"id", name="id_pk", comment="id_pk comment"
),
CheckConstraint('id < 100', name="cc1", comment="Id value can't exceed 100"),
UniqueConstraint(['data'], name="uc1", comment="Must have unique data field"),
)
```
Provides the DDL representation for constraint comments and routines to create and drop them. Class `.Inspector` reflects constraint comments via methods like `get_check_constraints` .
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- [ ] A short code fix
- [x] A new feature implementation
- Solves the issue 5667.
- The commit message includes `Fixes: 5667`.
- Includes tests based on comment reflection.
**Have a nice day!**
Fixes: #5667Closes: #7742
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7742
Pull-request-sha: 42a5d3c3e9
Change-Id: Ia60f578595afdbd6089541c9a00e37997ef78ad3
Rearchitected the schema reflection API to allow some dialects to make use
of high performing batch queries to reflect the schemas of many tables at
once using much fewer queries. The new performance features are targeted
first at the PostgreSQL and Oracle backends, and may be applied to any
dialect that makes use of SELECT queries against system catalog tables to
reflect tables (currently this omits the MySQL and SQLite dialects which
instead make use of parsing the "CREATE TABLE" statement, however these
dialects do not have a pre-existing performance issue with reflection. MS
SQL Server is still a TODO).
The new API is backwards compatible with the previous system, and should
require no changes to third party dialects to retain compatibility;
third party dialects can also opt into the new system by implementing
batched queries for schema reflection.
Along with this change is an updated reflection API that is fully
:pep:`484` typed, features many new methods and some changes.
Fixes: #4379
Change-Id: I897ec09843543aa7012bcdce758792ed3d415d08
Fixed bug where the PostgreSQL :meth:`_postgresql.Insert.on_conflict`
method and the SQLite :meth:`_sqlite.Insert.on_conflict` method would both
fail to correctly accommodate a column with a separate ".key" when
specifying the column using its key name in the dictionary passed to
``set_``, as well as if the :attr:`_sqlite.Insert.excluded` or
:attr:`_postgresql.Insert.excluded` collection were used as the dictionary
directly.
Fixes: #8014
Change-Id: I67226aeedcb2c683e22405af64720cc1f990f274
SQLite datetime, date, and time datatypes now use Python standard lib
``fromisoformat()`` methods in order to parse incoming datetime, date, and
time string values. This improves performance vs. the previous regular
expression-based approach, and also automatically accommodates for datetime
and time formats that contain either a six-digit "microseconds" format or a
three-digit "milliseconds" format.
Fixes: #7029
Change-Id: I67aab4fe5ee3055e5996050cf4564981413cc221
Fixed issue where SQLite unique constraint reflection would not work
for an inline UNIQUE constraint where the column name had an underscore
in its name.
Added support for reflecting SQLite inline unique constraints where
the column names are formatted with SQLite "escape quotes" ``[]``
or `` ` ``, which are discarded by the database when producing the
column name.
Fixes: #7736
Change-Id: I635003478dc27193995f7d7a6448f9333a498706
The SQLite dialect now defaults to :class:`_pool.QueuePool` when a file
based database is used. This is set along with setting the
``check_same_thread`` parameter to ``False``. It has been observed that the
previous approach of defaulting to :class:`_pool.NullPool`, which does not
hold onto database connections after they are released, did in fact have a
measurable negative performance impact. As always, the pool class is always
customizable via the :paramref:`_sa.create_engine.poolclass` parameter.
Fixes: #7490
Change-Id: I5f6c259def0ef43d401c6163dc99f651e519148d
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.
Also update Black and other pre-commit hooks and re-format with Black.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570
Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
### Description
Found via `codespell -q 3 -L ba,crate,datas,froms,gord,hist,inh,nd,selectin,strat,ue`
Also added codespell to the pep8 tox env
### Checklist
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
Closes: #7338
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7338
Pull-request-sha: 0deac22193
Change-Id: Icd61db31c8dc655d4a39d8a304194804d08555fe
Generalized the :paramref:`_sa.create_engine.isolation_level` parameter to
the base dialect so that it is no longer dependent on individual dialects
to be present. This parameter sets up the "isolation level" setting to
occur for all new database connections as soon as they are created by the
connection pool, where the value then stays set without being reset on
every checkin.
The :paramref:`_sa.create_engine.isolation_level` parameter is essentially
equivalent in functionality to using the
:paramref:`_engine.Engine.execution_options.isolation_level` parameter via
:meth:`_engine.Engine.execution_options` for an engine-wide setting. The
difference is in that the former setting assigns the isolation level just
once when a connection is created, the latter sets and resets the given
level on each connection checkout.
Fixes: #6342
Change-Id: Id81d6b1c1a94371d901ada728a610696e09e9741
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
Fixes: #7300
Change-Id: I6255c850b140522a4aba95085216d0bca18ce230
The major action here is to lift and move future.Connection
and future.Engine fully into sqlalchemy.engine.base. This
removes lots of engine concepts, including:
* autocommit
* Connection running without a transaction, autobegin
is now present in all cases
* most "autorollback" is obsolete
* Core-level subtransactions (i.e. MarkerTransaction)
* "branched" connections, copies of connections
* execution_options() returns self, not a new connection
* old argument formats, distill_params(), simplifies calling
scheme between engine methods
* before/after_execute() events (oriented towards compiled constructs)
don't emit for exec_driver_sql(). before/after_cursor_execute()
is still included for this
* old helper methods superseded by context managers, connection.transaction(),
engine.transaction() engine.run_callable()
* ancient engine-level reflection methods has_table(), table_names()
* sqlalchemy.testing.engines.proxying_engine
References: #7257
Change-Id: Ib20ed816642d873b84221378a9ec34480e01e82c
Fixed bug where the error message for SQLite invalid isolation level on the
pysqlite driver would fail to indicate that "AUTOCOMMIT" is one of the
valid isolation levels.
Change-Id: Icbceab9a28af6a560859761fa92320b5473269a9
References: #6959
The fix for pysqlcipher released in version 1.4.3 🎫`5848` was
unfortunately non-working, in that the new ``on_connect_url`` hook was
erroneously not receiving a ``URL`` object under normal usage of
:func:`_sa.create_engine` and instead received a string that was unhandled;
the test suite failed to fully set up the actual conditions under which
this hook is called. This has been fixed.
Fixes: #6586
Change-Id: I3bf738daec35877a10fdad740f08dca9e7420829
Mike Bayer