Commit Graph

149 Commits

Author SHA1 Message Date
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
Mike Bayer f684bb7659 ensure RETURNING renders in stringify w/ no server version
just in my own testing, if I say insert().return_defaults()
and stringify, I should see it, so make sure all the dialects
default to "insert_returning" etc. , with downgrade on
server version check.

Change-Id: Id64e78fcb03c48b5dcb0feb21cb9cc495edd15e9
2022-08-03 20:47:27 -04:00
Federico Caselli 63d90b0f44 SQLite reflection ignores schema internal names
Added new parameter to SQLite for reflection methods called
``sqlite_include_internal=True``; when omitted, local tables that start
with the prefix ``sqlite_``, which per SQLite documentation are noted as
"internal schema" tables such as the ``sqlite_sequence`` table generated to
support "AUTOINCREMENT" columns, will not be included in reflection methods
that return lists of local objects. This prevents issues for example when
using Alembic autogenerate, which previously would consider these
SQLite-generated tables as being remove from the model.

Fixes: #8234
Change-Id: I36ee7a053e04b6c46c912aaa0d7e035a5b88a4f9
2022-08-01 21:47:36 +00:00
cheremnov 5fb63bc142 Comments on (named) constraints
Adds support for comments on named constraints, including `ForeignKeyConstraint`, `PrimaryKeyConstraint`, `CheckConstraint`, `UniqueConstraint`, solving the [Issue 5667](https://github.com/sqlalchemy/sqlalchemy/issues/5667).

Supports only PostgreSQL backend.

### Description

Following the example of [Issue 1546](https://github.com/sqlalchemy/sqlalchemy/issues/1546), supports comments on constraints. Specifically, enables comments on _named_ ones — as I get it, PostgreSQL prohibits comments on unnamed constraints.

Enables setting the comments for named constraints like this:
```
Table(
   'example', metadata,
   Column('id', Integer),
   Column('data', sa.String(30)),
   PrimaryKeyConstraint(
       "id", name="id_pk", comment="id_pk comment"
    ),
   CheckConstraint('id < 100', name="cc1", comment="Id value can't exceed 100"),
   UniqueConstraint(['data'], name="uc1", comment="Must have unique data field"),
)
```

Provides the DDL representation for constraint comments and routines to create and drop them. Class `.Inspector` reflects constraint comments via methods like `get_check_constraints` .
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
- [ ] A short code fix
- [x] A new feature implementation
	- Solves the issue 5667.
	- The commit message includes `Fixes: 5667`.
	- Includes tests based on comment reflection.

**Have a nice day!**

Fixes: #5667
Closes: #7742
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7742
Pull-request-sha: 42a5d3c3e9

Change-Id: Ia60f578595afdbd6089541c9a00e37997ef78ad3
2022-06-29 09:13:37 +00:00
Federico Caselli db08a69948 rearchitect reflection for batched performance
Rearchitected the schema reflection API to allow some dialects to make use
of high performing batch queries to reflect the schemas of many tables at
once using much fewer queries. The new performance features are targeted
first at the PostgreSQL and Oracle backends, and may be applied to any
dialect that makes use of SELECT queries against system catalog tables to
reflect tables (currently this omits the MySQL and SQLite dialects which
instead make use of parsing the "CREATE TABLE" statement, however these
dialects do not have a pre-existing performance issue with reflection. MS
SQL Server is still a TODO).

The new API is backwards compatible with the previous system, and should
require no changes to third party dialects to retain compatibility;
third party dialects can also opt into the new system by implementing
batched queries for schema reflection.

Along with this change is an updated reflection API that is fully
:pep:`484` typed, features many new methods and some changes.

Fixes: #4379
Change-Id: I897ec09843543aa7012bcdce758792ed3d415d08
2022-06-18 14:57:26 -04:00
Mike Bayer 927abc3b33 render col name in on conflict set clause, not given key
Fixed bug where the PostgreSQL :meth:`_postgresql.Insert.on_conflict`
method and the SQLite :meth:`_sqlite.Insert.on_conflict` method would both
fail to correctly accommodate a column with a separate ".key" when
specifying the column using its key name in the dictionary passed to
``set_``, as well as if the :attr:`_sqlite.Insert.excluded` or
:attr:`_postgresql.Insert.excluded` collection were used as the dictionary
directly.

Fixes: #8014
Change-Id: I67226aeedcb2c683e22405af64720cc1f990f274
2022-05-15 17:27:27 -04:00
Mike Bayer ccadbec825 use .fromisoformat() for sqlite datetime, date, time parsing
SQLite datetime, date, and time datatypes now use Python standard lib
``fromisoformat()`` methods in order to parse incoming datetime, date, and
time string values. This improves performance vs. the previous regular
expression-based approach, and also automatically accommodates for datetime
and time formats that contain either a six-digit "microseconds" format or a
three-digit "milliseconds" format.

Fixes: #7029
Change-Id: I67aab4fe5ee3055e5996050cf4564981413cc221
2022-04-03 14:47:52 -04:00
Mike Bayer 834af17a46 improve reflection of inline UNIQUE constraints
Fixed issue where SQLite unique constraint reflection would not work
for an inline UNIQUE constraint where the column name had an underscore
in its name.

Added support for reflecting SQLite inline unique constraints where
the column names are formatted with SQLite "escape quotes" ``[]``
or `` ` ``, which are discarded by the database when producing the
column name.

Fixes: #7736
Change-Id: I635003478dc27193995f7d7a6448f9333a498706
2022-02-20 20:39:48 -05:00
Mike Bayer 22ed657827 use QueuePool for sqlite file databases
The SQLite dialect now defaults to :class:`_pool.QueuePool` when a file
based database is used. This is set along with setting the
``check_same_thread`` parameter to ``False``. It has been observed that the
previous approach of defaulting to :class:`_pool.NullPool`, which does not
hold onto database connections after they are released, did in fact have a
measurable negative performance impact. As always, the pool class is always
customizable via the :paramref:`_sa.create_engine.poolclass` parameter.

Fixes: #7490
Change-Id: I5f6c259def0ef43d401c6163dc99f651e519148d
2022-02-10 11:09:22 -05:00
mike bayer 5681d4e4da Merge "Fix various source comment/doc typos" into main 2022-01-07 16:42:18 +00:00
Hugo van Kemenade 146a349d81 Update Black's target-version to py37
<!-- Provide a general summary of your proposed changes in the Title field above -->

### Description
<!-- Describe your changes in detail -->

Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.

Also update Black and other pre-commit hooks and re-format with Black.

### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

**Have a nice day!**

Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570

Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
2022-01-05 12:41:32 -05:00
luz paz 56256b6d13 Fix various source comment/doc typos
### Description
Found via `codespell -q 3 -L ba,crate,datas,froms,gord,hist,inh,nd,selectin,strat,ue`
Also added codespell to the pep8 tox env

### Checklist

This pull request is:

- [x] A documentation / typographical error fix
	- Good to go, no issue or tests are needed

Closes: #7338
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7338
Pull-request-sha: 0deac22193

Change-Id: Icd61db31c8dc655d4a39d8a304194804d08555fe
2021-12-29 21:35:34 +01:00
Federico Caselli 31acba8ff7 Clean up most py3k compat
Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
2021-11-24 22:51:27 -05:00
Federico Caselli 0b95f0055b Remove object in class definition
References: #4600
Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
2021-11-22 15:03:17 +00:00
Mike Bayer af1b91626f fully support isolation_level parameter in base dialect
Generalized the :paramref:`_sa.create_engine.isolation_level` parameter to
the base dialect so that it is no longer dependent on individual dialects
to be present. This parameter sets up the "isolation level" setting to
occur for all new database connections as soon as they are created by the
connection pool, where the value then stays set without being reset on
every checkin.

The :paramref:`_sa.create_engine.isolation_level` parameter is essentially
equivalent in functionality to using the
:paramref:`_engine.Engine.execution_options.isolation_level` parameter via
:meth:`_engine.Engine.execution_options` for an engine-wide setting. The
difference is in that the former setting assigns the isolation level just
once when a connection is created, the latter sets and resets the given
level on each connection checkout.

Fixes: #6342
Change-Id: Id81d6b1c1a94371d901ada728a610696e09e9741
2021-11-18 13:11:43 -05:00
Mike Bayer b919a0a85a change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".

Fixes: #7300
Change-Id: I6255c850b140522a4aba95085216d0bca18ce230
2021-11-09 15:30:58 -05:00
Mike Bayer d050193daa fully implement future engine and remove legacy
The major action here is to lift and move future.Connection
and future.Engine fully into sqlalchemy.engine.base.   This
removes lots of engine concepts, including:

* autocommit
* Connection running without a transaction, autobegin
  is now present in all cases
* most "autorollback" is obsolete
* Core-level subtransactions (i.e. MarkerTransaction)
* "branched" connections, copies of connections
* execution_options() returns self, not a new connection
* old argument formats, distill_params(), simplifies calling
  scheme between engine methods
* before/after_execute() events (oriented towards compiled constructs)
  don't emit for exec_driver_sql().  before/after_cursor_execute()
  is still included for this
* old helper methods superseded by context managers, connection.transaction(),
  engine.transaction() engine.run_callable()
* ancient engine-level reflection methods has_table(), table_names()
* sqlalchemy.testing.engines.proxying_engine

References: #7257

Change-Id: Ib20ed816642d873b84221378a9ec34480e01e82c
2021-11-07 14:30:35 -05:00
Federico Caselli 36e7aebd8d First round of removal of python 2
References: #4600
Change-Id: I61e35bc93fe95610ae75b31c18a3282558cd4ffe
2021-11-01 15:11:25 -04:00
Mike Bayer 916bd20b6d ensure pysqlite dialect enumerates correct isolation levels
Fixed bug where the error message for SQLite invalid isolation level on the
pysqlite driver would fail to indicate that "AUTOCOMMIT" is one of the
valid isolation levels.

Change-Id: Icbceab9a28af6a560859761fa92320b5473269a9
References: #6959
2021-08-30 11:35:10 -04:00
Federico Caselli fb81f9c8d9 Replace all http:// links to https://
Also replace http://pypi.python.org/pypi with https://pypi.org/project

Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
2021-07-04 20:54:33 +02:00
Mike Bayer 03633a6462 Pass URL object, not the string, to on_connect_url
The fix for pysqlcipher released in version 1.4.3 🎫`5848` was
unfortunately non-working, in that the new ``on_connect_url`` hook was
erroneously not receiving a ``URL`` object under normal usage of
:func:`_sa.create_engine` and instead received a string that was unhandled;
the test suite failed to fully set up the actual conditions under which
this hook is called. This has been fixed.

Fixes: #6586
Change-Id: I3bf738daec35877a10fdad740f08dca9e7420829
2021-06-06 10:12:05 -04:00
Mike Bayer 6967b45020 don't cache TypeDecorator by default
The :class:`.TypeDecorator` class will now emit a warning when used in SQL
compilation with caching unless the ``.cache_ok`` flag is set to ``True``
or ``False``. ``.cache_ok`` indicates that all the parameters passed to the
object are safe to be used as a cache key, ``False`` means they are not.

Fixes: #6436
Change-Id: Ib1bb7dc4b124e38521d615c2e2e691e4915594fb
2021-05-06 13:57:43 -04:00
YongJieYongJie 1ee6332ba2 Use SingletonThreadPool for in-memory SQLite
database created using filename uri

Default to using ``SingletonThreadPool`` for in-memory SQLite databases
created using URI filenames. Previously the default pool used was the
``NullPool`` that precented sharing the same database between multiple
engines.

Fixes: #6379
Closes: #6380
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/6380
Pull-request-sha: 3b8024417a

Change-Id: Ice09622796455e796ede7711c98f3ceec13aa949
2021-04-28 20:52:34 +02:00
Mike Bayer 4476dca007 Repair pysqlcipher and use sqlcipher3
The ``pysqlcipher`` dialect now imports the ``sqlcipher3`` module
for Python 3 by default.    Regressions have been repaired such that
the connection routine was not working.

To better support the post-connection steps of the pysqlcipher
dialect, a new hook Dialect.on_connect_url() is added, which
supersedes Dialect.on_connect() and is passed the URL object.
The dialect now pulls the passphrase and other cipher args
from the URL directly without including them in the
"connect" args.  This will allow any user-defined extensibility
to connecting to work as it would for other dialects.

The commit also builds upon the extended routines in
sqlite/provisioning.py to better support running tests against
multiple simultaneous SQLite database files.  Additionally enables
backend for test_sqlite which was skipping everything
for aiosqlite too, fortunately everything there is passing.

Fixes: #5848
Change-Id: I43f53ebc62298a84a4abe149e1eb699a027b7915
2021-03-24 19:04:30 -04:00
Federico Caselli 502be87a0b Add support for aiosqlite
Added support for the aiosqlite database driver for use with the
SQLAlchemy asyncio extension.

Fixes: #5920
Change-Id: Id11a320516a44e886a6f518d2866a0f992413e55
2021-03-24 11:45:39 -04:00
Gord Thompson 857adaaf86 Accept ColumnCollection in update_on_conflict(set_=
Fixes: #5939
Change-Id: I21d7125765028e2a98d5ef4c32d8e7e457aa2d12
2021-02-15 11:16:38 -07:00
Mike Bayer 8fc36f3b56 Repair tests for older SQLite, timing, pypy
Change-Id: I9e425d2415d18a893342244755c7748b546fb20d
2021-02-03 16:47:53 -05:00
Mike Bayer 8860117c96 introduce generalized decorator to prevent invalid method calls
This introduces the ``_exclusive_against()`` utility decorator
that can be used to prevent repeated invocations of methods that
typically should only be called once.

An informative error message is now raised for a selected set of DML
methods (currently all part of :class:`_dml.Insert` constructs) if they are
called a second time, which would implicitly cancel out the previous
setting.  The methods altered include:
:class:`_sqlite.Insert.on_conflict_do_update`,
:class:`_sqlite.Insert.on_conflict_do_nothing` (SQLite),
:class:`_postgresql.Insert.on_conflict_do_update`,
:class:`_postgresql.Insert.on_conflict_do_nothing` (PostgreSQL),
:class:`_mysql.Insert.on_duplicate_key_update` (MySQL)

Fixes: #5169
Change-Id: I9278fa87cd3470dcf296ff96bb0fb17a3236d49d
2021-01-16 18:44:21 -05:00
Mike Bayer c5b4af959e Guard against re-entrant autobegin in Core, ORM
Fixed bug in "future" version of :class:`.Engine` where emitting SQL during
the :meth:`.EngineEvents.do_begin` event hook would cause a re-entrant
condition due to autobegin, including the recipe documented for SQLite to
allow for savepoints and serializable isolation support.

Fixed issue in new :class:`_orm.Session` similar to that of the
:class:`_engine.Connection` where the new "autobegin" logic could be
tripped into a re-entrant state if SQL were executed within the
:meth:`.SessionEvents.after_transaction_create` event hook.

Also repair the new "testing_engine" pytest fixture to
set up for "future" engine appropriately, which wasn't working
leading to the test_execute.py tests not using the future
engine since recent f1e96cb087.

Fixes: #5845
Change-Id: Ib2432d8c8bd753e24be60720ec47affb2df15a4a
2021-01-15 20:14:34 -05:00
Mike Bayer ebbbac0a76 update execute() arg formats in modules and tests
continuing with producing a SQLAlchemy 1.4.0b2 that internally
does not emit any of its own 2.0 deprecation warnings,
migrate the *args and **kwargs passed to execute() methods
that now must be a single list or dictionary.

Alembic 1.5 is again waiting on this internal consistency to
be present so that it can pass all tests with no 2.0
deprecation warnings.

Change-Id: If6b792e57c8c5dff205419644ab68e631575a2fa
2021-01-15 13:04:58 -05:00
Mike Bayer f1e96cb087 reinvent xdist hooks in terms of pytest fixtures
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones.   Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them.   A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable.   New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.

From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow.  A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.

As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively.  The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.

Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session.   The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic.  A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function.  The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all.  Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.

As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.

An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2.  It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.

py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2.  For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.

Includes the following improvements:

Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`.  Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.

For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.

Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board.  the addition
of this flag has resolved the long-standing oracle "two task"
error problem.

For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically.  There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite.  Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.

Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.

Fixes: #5826
Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
2021-01-13 22:10:13 -05:00
Mike Bayer fd3c063dd6 remove metadata.bind use from test suite
importantly this means we can remove bound metadata from
the fixtures that are used by Alembic's test suite.

hopefully this is the last one that has to happen to allow
Alembic to be fully 1.4/2.0.

Start moving from @testing.provide_metadata to a pytest
metadata fixture.  This does not seem to have any negative
effects even though TablesTest uses a "self.metadata" attribute.

Change-Id: Iae6ab95938a7e92b6d42086aec534af27b5577d3
2021-01-03 13:22:29 -05:00
Mike Bayer 31e7428b16 Repair mssql dep tests; have __only_on__ imply __backend__
CI missed a few SQL Server tests because we run mssql-backendonly
in the gerrit job.   As there was a test that was "only on"
mssql but didn't have backendonly, it never got run and then
fails in master where we run mssql fully.

Any suite that has an __only_on__ is inherently specific to
a backend, so if present this should imply __backend__ so that
it definitely runs when we have that backend present.

This in turn meant we had to fix a few sqlite_file tests that
weren't cleaning up or sharing well as they suddenly became
backend tests under sqlite_file.  Added a sqlite_file cleanup
to test class cleanup for now.

Change-Id: I9de1ceabd6596547a65c59059a55b7e5156103fd
2020-12-19 12:36:57 -05:00
Mike Bayer 628db9581c Emit 2.0 deprecation warning for sub-transactions
The nesting pattern will be removed in 2.0, so the use of the
MarkerTransaction should emit a 2.0 deprecation warning
unconditionally.

Change-Id: I96aed22c4e5db9b59e9b28a7f2d1283cd99a9cb6
2020-12-14 09:38:28 -05:00
Mike Bayer ba5cbf9366 correct for "autocommit" deprecation warning
Ensure no autocommit warnings occur internally or
within tests.

Also includes fixes for SQL Server full text tests
which apparently have not been working at all for a long
time, as it used long removed APIs.  CI has not had
fulltext running for some years and is now installed.

Change-Id: Id806e1856c9da9f0a9eac88cebc7a94ecc95eb96
2020-12-11 13:26:05 -05:00
Mike Bayer 584cabbf7e Support Column objects in the SET clause for upsert
Established support for :class:`_schema.Column` objects as well as ORM
instrumented attributes as keys in the ``set_`` dictionary passed to the
:meth:`_postgresql.Insert.on_conflict_do_update` and
:meth:`_sqlite.Insert.on_conflict_do_update` methods, which match to the
:class:`_schema.Column` objects in the ``.c`` collection of the target
:class:`_schema.Table`. Previously,  only string column names were
expected; a column expression would be assumed to be an out-of-table
expression that would render fully along with a warning.

Fixes: #5722
Change-Id: Ice73b501d721c28d978a0277a83cedc6aff756a9
2020-11-23 20:13:20 -05:00
mike bayer b3f2f08f48 Merge "Support SQLite WITHOUT ROWID tables" 2020-11-09 15:56:15 +00:00
RamonWill 89ddd0b897 Implement upsert for SQLite
Implemented INSERT... ON CONFLICT clause for SQLite. Pull request courtesy
Ramon Williams.

Fixes: #4010
Closes: #5580
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5580
Pull-request-sha: fb422e0749

Change-Id: Ibeea44f4c2cee8dab5dc22b7ec3ae1ab95c12b65
2020-11-08 13:34:24 -05:00
Sean Anderson 4b39b0f89d Support SQLite WITHOUT ROWID tables
This adds support for creating tables WITHOUT ROWID in the SQLite
dialect. WITHOUT ROWID tables were introduced in SQLite version 3.8.2
(2013-12-06). They do not use an implicit rowid column as the primary
key. This may result in space and performance savings for tables without
INTEGER primary keys and tables with composite primary keys. For more
information about this feature, see the sqlite documentation [1].

[1] https://www.sqlite.org/withoutrowid.html

Fixes: #5685

### Checklist
This pull request is:

- [x] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

Closes: #5686
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5686
Pull-request-sha: 2b44782d1b

Change-Id: Ifcf727b0c07c90e267b79828a8e3fd7a8260a074
2020-11-08 06:09:45 -07:00
Mike Bayer 80be40300d Convert to autoload_with internally
Fixed bug where the now-deprecated ``autoload`` parameter was being called
internally within the reflection routines when a related table were
reflected.

Fixes: #5684
Change-Id: I6ab439a2f49ff1ae2d3c7a15b531cbafbc3cf594
2020-11-07 11:13:28 -05:00
jonathan vanasco 9ddbd585a6 Apply underscore naming to several more operators
The operator changes are:

* `isfalse` is now `is_false`
* `isnot_distinct_from` is now `is_not_distinct_from`
* `istrue` is now `is_true`
* `notbetween` is now `not_between`
* `notcontains` is now `not_contains`
* `notendswith` is now `not_endswith`
* `notilike` is now `not_ilike`
* `notlike` is now `not_like`
* `notmatch` is now `not_match`
* `notstartswith` is now `not_startswith`
* `nullsfirst` is now `nulls_first`
* `nullslast` is now `nulls_last`

Because these are core operators, the internal migration strategy for this
change is to support legacy terms for an extended period of time -- if not
indefinitely -- but update all documentation, tutorials, and internal usage
to the new terms.  The new terms are used to define the functions, and
the legacy terms have been deprecated into aliases of the new terms.

Fixes: #5435
Change-Id: Ifbd7cb1cdda5981990243c4fc4b4ff467dc132ac
2020-10-30 10:02:29 -04:00
Mike Bayer c3f102c9fe upgrade to black 20.8b1
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.

Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
2020-09-28 15:17:26 -04:00
Federico Caselli e860060866 Update select usage to use the new 1.4 format
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function.  it does not
yet fully address other issues such as keyword arguments passed
to the table.select().

Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.

Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
2020-09-08 17:13:48 -04:00
Federico Caselli b1b97ed1fc Add support for regular expression on supported backend.
Two operations have been defined:

* :meth:`~.ColumnOperators.regexp_match` implementing a regular
  expression match like function.
* :meth:`~.ColumnOperators.regexp_replace` implementing a regular
  expression string replace function.

Fixes: #1390
Change-Id: I44556846e4668ccf329023613bd26861d5c674e6
2020-08-27 17:30:18 -04:00
Mike Bayer 91f376692d Add future=True to create_engine/Session; unify select()
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles.   This would make
migration simpler and reduce confusion.

However, confusion may be increased by the fact that select().join()
is different  Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs.  At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps.   Session.execute() will still
behave the old way as we are adding a future flag.

This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.

The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.

Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.

Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.

Fixes: #5379
Fixes: #5284
Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51
References: #5159
2020-07-08 11:05:11 -04:00
Thodoris Sotiropoulos ed4a4896ae Add 'exists' to SQLite's reserved keywords
Added "exists" to the list of reserved words for SQLite so that this word
will be quoted when used as a label or column name. Pull request courtesy
Thodoris Sotiropoulos.

Fixes: #5395
Closes: #5396
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5396
Pull-request-sha: 5608317a38

Change-Id: Ia4769de2dec159dcf282eb4b30c11560da51a5c7
2020-06-11 10:53:05 -04:00
Mike Bayer 97cd0a5db8 Default create_constraint to False
The :paramref:`.Enum.create_constraint` and
:paramref:`.Boolean.create_constraint` parameters now default to False,
indicating when a so-called "non-native" version of these two datatypes is
created, a CHECK constraint will not be generated by default.   These CHECK
constraints present schema-management maintenance complexities that should
be opted in to, rather than being turned on by default.

Fixes: #5367
Change-Id: I0a3fb608ce32143fa757546cc17ba2013e93272a
2020-06-03 16:19:40 -04:00
Federico Caselli 87949deb04 SQLite 3.31 added support for computed column.
This change enables their support in SQLAlchemy when targeting SQLite.

Fixes: #5297
Change-Id: Ia9f21a49e58fc977e3c669b8176036c95d93b9c8
2020-05-19 19:33:02 +02:00
Gord Thompson 57cf5371d0 Clean up (engine|db).execute for sqlite
Change-Id: Ie43e5b3a34ecc64ab77410e42f3e8de8bbc1e2d8
2020-03-31 07:19:13 -06:00
Federico Caselli 9ec7588220 Deprecate plain string in execute and introduce exec_driver_sql
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.

Fixes: #4848
Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
2020-03-21 17:03:45 -04:00