Commit Graph

264 Commits

Author SHA1 Message Date
mike bayer 28138ee6fb Merge "Update select usage to use the new 1.4 format" 2020-09-08 22:11:29 +00:00
Federico Caselli e860060866 Update select usage to use the new 1.4 format
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function.  it does not
yet fully address other issues such as keyword arguments passed
to the table.select().

Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.

Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
2020-09-08 17:13:48 -04:00
Federico Caselli 721b00e753 Fix typo in I0ad6d7a095e49d331618274c40ce75c76afdc7dd
Change-Id: I649662d440f83df379922e8c967d28f635f9c85b
2020-09-08 22:00:25 +02:00
Mike Bayer e3716012c5 Create connection characteristics API; implement postgresql flags
Added support for PostgreSQL "readonly" and "deferrable" flags for all of
psycopg2, asyncpg and pg8000 dialects.   This takes advantage of a newly
generalized version of the "isolation level" API to support other kinds of
session attributes set via execution options that are reliably reset
when connections are returned to the connection pool.

Fixes: #5549
Change-Id: I0ad6d7a095e49d331618274c40ce75c76afdc7dd
2020-09-08 11:16:53 -04:00
Federico Caselli b1b97ed1fc Add support for regular expression on supported backend.
Two operations have been defined:

* :meth:`~.ColumnOperators.regexp_match` implementing a regular
  expression match like function.
* :meth:`~.ColumnOperators.regexp_replace` implementing a regular
  expression string replace function.

Fixes: #1390
Change-Id: I44556846e4668ccf329023613bd26861d5c674e6
2020-08-27 17:30:18 -04:00
mike bayer 317f2e1be2 Merge "Add support for identity columns" 2020-08-21 21:39:40 +00:00
mike bayer 544ef23cd3 Merge "normalize execute style for events, 2.0" 2020-08-20 15:22:08 +00:00
Mike Bayer a1939719a6 normalize execute style for events, 2.0
The _execute_20 and exec_driver_sql methods should wrap
up the parameters so that they represent the single list / single
dictionary style of invocation into the legacy methods.  then
the before_ after_ execute event handlers should be receiving
the parameter dictionary as a single dictionary.   this requires
that we break out distill_params to work differently if event
handlers are present.

additionally, add deprecation warnings for old argument passing
styles.

Change-Id: I97cb4d06adfcc6b889f10d01cc7775925cffb116
2020-08-20 10:14:21 -04:00
mike bayer 6234792375 Merge "Implement DDL visitor for PG ENUM with schema translate support" 2020-08-19 17:38:07 +00:00
mike bayer 348afaf742 Merge "Add JSON support for mssql" 2020-08-19 17:25:06 +00:00
Gord Thompson 0c7a867ac7 Add JSON support for mssql
Added support for the :class:`_types.JSON` datatype on the SQL Server
dialect using the :class:`_mssql.JSON` implementation, which implements SQL
Server's JSON functionality against the ``NVARCHAR(max)`` datatype as per
SQL Server documentation. Implementation courtesy Gord Thompson.

Fixes: #4384
Change-Id: I28af79a4d8fafaa68ea032228609bba727784f18
2020-08-19 11:05:52 -04:00
Mike Bayer d1005e1305 Implement DDL visitor for PG ENUM with schema translate support
Fixed issue where the :class:`_postgresql.ENUM` type would not consult the
schema translate map when emitting a CREATE TYPE or DROP TYPE during the
test to see if the type exists or not.  Additionally, repaired an issue
where if the same enum were encountered multiple times in a single DDL
sequence, the "check" query would run repeatedly rather than relying upon a
cached value.

Fixes: #5520
Change-Id: I79f46e29ac0168e873ff178c242f8d78f6679aeb
2020-08-19 10:43:38 -04:00
Federico Caselli 26e8d3b5bd Add support for identity columns
Added the :class:`_schema.Identity` construct that can be used to
configure identity columns rendered with GENERATED { ALWAYS |
BY DEFAULT } AS IDENTITY. Currently the supported backends are
PostgreSQL >= 10, Oracle >= 12 and MSSQL (with different syntax
and a subset of functionalities).

Fixes: #5362
Fixes: #5324
Fixes: #5360

Change-Id: Iecea6f3ceb36821e8b96f0b61049b580507a1875
2020-08-19 00:34:23 +02:00
Federico Caselli 8e01a928d9 Support data types for CREATE SEQUENCE in PostgreSQL
Allow specifying the data type when creating a :class:`.Sequence` in
PostgreSQL by using the parameter :paramref:`.Sequence.data_type`.

Fixes: #5498

Change-Id: I2b4a80aa89b1503c56748dc3ecd2cf145faddd8b
2020-08-18 22:03:28 +00:00
mike bayer 0901190bb4 Merge "Deliver straight BinaryExpr w/ no negate for any() / all()" 2020-08-18 19:07:19 +00:00
Mike Bayer 8bc793c4db Deliver straight BinaryExpr w/ no negate for any() / all()
Adjusted the :meth:`_types.ARRAY.Comparator.any` and
:meth:`_types.ARRAY.Comparator.all` methods to implement a straight "NOT"
operation for negation, rather than negating the comparison operator.

Fixes: #5518
Change-Id: I87ee9278c321aafe51a679fcfcbb5fbb11307fda
2020-08-18 14:18:02 -04:00
Tony Locke 06f1929b86 Update dialect for pg8000 version 1.16.0
The pg8000 dialect has been revised and modernized for the most recent
version of the pg8000 driver for PostgreSQL.  Changes to the dialect
include:

* All data types are now sent as text rather than binary.

* Using adapters, custom types can be plugged in to pg8000.

* Previously, named prepared statements were used for all statements.
  Now unnamed prepared statements are used by default, and named
  prepared statements can be used explicitly by calling the
  Connection.prepare() method, which returns a PreparedStatement
  object.

Pull request courtesy Tony Locke.

Notes by Mike: to get this all working it was needed to break
up JSONIndexType into "str" and "int" subtypes; this will be
needed for any dialect that is dependent on setinputsizes().

also includes @caselit's idea to include query params
in the dbdriver parameter.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>

Closes: #5451
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5451
Pull-request-sha: 639751ca9c

Change-Id: I2869bc52c330916773a41d11d12c297aecc8fcd8
2020-08-18 11:12:16 -04:00
Mike Bayer fc97854f69 Bump minimum MySQL version to 5.0.2; use all-numeric server version
MySQL dialect's server_version_info tuple is now all numeric.  String
tokens like "MariaDB" are no longer present so that numeric comparison
works in all cases.  The .is_mariadb flag on the dialect should be
consulted for whether or not mariadb was detected.   Additionally removed
structures meant to support extremely old MySQL versions 3.x and 4.x;
the minimum MySQL version supported is now version 5.0.2.

In addition, as the "MariaDB" name goes away from server version,
expand upon the change in I330815ebe572b6a9818377da56621397335fa702
to support the name "mariadb" throughout the dialect and test suite
when mariadb-only mode is used.    This changes the "name" field
on the MariaDB dialect to "mariadb", which then implies a change
throughout the testing requirements system as well as all the
dialect-specific DDL argument names such as "mysql_engine" is
now specified as "mariadb_engine", etc.   Make use of the
recent additions to test suite URL provisioning so that we can
force MariaDB databases to have a "mariadb-only" dialect which
allows us to test this name change fully.

Update documentation to refer to MySQL / MariaDB explicitly
as well as indicating the "mariadb_" prefix used for options.

It seems likely that MySQL and MariaDB version numbers are going to
start colliding at some point so having the "mariadb" name
be available as a totally separate dialect name should give us
some options in this regard.

Currently also includes a date related fix to a test for
the postgresql dialect that was implicitly assuming a
non-UTC timezone

Fixes: #4189
Change-Id: I00e76d00f62971e1f067bd61915fa6cc1cf64e5e
2020-08-16 12:18:07 -04:00
Mike Bayer 5fb0138a32 Implement rudimentary asyncio support w/ asyncpg
Using the approach introduced at
https://gist.github.com/zzzeek/6287e28054d3baddc07fa21a7227904e

We can now create asyncio endpoints that are then handled
in "implicit IO" form within the majority of the Core internals.
Then coroutines are re-exposed at the point at which we call
into asyncpg methods.

Patch includes:

* asyncpg dialect

* asyncio package

* engine, result, ORM session classes

* new test fixtures, tests

* some work with pep-484 and a short plugin for the
  pyannotate package, which seems to have so-so results

Change-Id: Idbcc0eff72c4cad572914acdd6f40ddb1aef1a7d
Fixes: #3414
2020-08-13 18:41:53 -04:00
Vlastimil Zíma 47b51e5a43 Clean python UUID imports
Fixes: #5482
All supported python versions provide 'uuid' module.

Closes: #5483
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5483
Pull-request-sha: fc32498a8b

Change-Id: I8b41b811da7576f724353425dad5d6f581641b4b
2020-07-30 03:56:39 -04:00
Jim Bosch 07e57a0330 Ensure is_comparison passed for PG RANGE op() methods
Fixed issue where the return type for the various RANGE comparison
operators would itself be the same RANGE type rather than BOOLEAN, which
would cause an undesirable result in the case that a
:class:`.TypeDecorator` that defined result-processing behavior were in
use.  Pull request courtesy Jim Bosch.

Fixes: #5476
Closes: #5477
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5477
Pull-request-sha: 925b117e0c

Change-Id: I52ab4d4362d379c8253990f9d328a40990a64520
2020-07-26 16:57:13 -04:00
Mike Bayer 91f376692d Add future=True to create_engine/Session; unify select()
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles.   This would make
migration simpler and reduce confusion.

However, confusion may be increased by the fact that select().join()
is different  Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs.  At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps.   Session.execute() will still
behave the old way as we are adding a future flag.

This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.

The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.

Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.

Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.

Fixes: #5379
Fixes: #5284
Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51
References: #5159
2020-07-08 11:05:11 -04:00
Gord Thompson 2c1e517d03 Rename Table.tometadata to to_metadata
Renamed the :meth:`_schema.Table.tometadata` method to
:meth:`_schema.Table.to_metadata`.  The previous name remains with a
deprecation warning.

Updated the "decorate" utility function to support decoration
of functions that include non-builtins as default values.

Moves test for deprecated "databases" package into
test/dialect/test_deprecations.py

Fixes: #5413
Fixes: #5426
Change-Id: I6ed899871c935f9e46360127c17ccb7cf97cea6e
2020-06-29 13:56:21 -04:00
Mike Bayer 08c46eea92 ORM executemany returning
Build on #5401 to allow the ORM to take advanage
of executemany INSERT + RETURNING.

Implemented the feature

updated tests

to support INSERT DEFAULT VALUES, needed to come up with
a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT)
which can then be iterated out for executemany.

Added graceful degrade to plain executemany for PostgreSQL <= 8.2

Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN

Fix issue where unicode identifiers or parameter names wouldn't
work with execute_values() under Py2K, because we have to
encode the statement and therefore have to encode the
insert_single_values_expr too.

Correct issue from #5401 to support executemany + return_defaults
for a PK that is explicitly pre-generated, meaning we aren't actually
getting RETURNING but need to return it from compiled_parameters.

Fixes: #5263
Change-Id: Id68e5c158c4f9ebc33b61c06a448907921c2a657
2020-06-27 21:30:37 -04:00
Mike Bayer f1a3038f48 Default psycopg2 executemany mode to "values_only"
The psycopg2 dialect now defaults to using the very performant
``execute_values()`` psycopg2 extension for compiled INSERT statements,
and also impements RETURNING support when this extension is used.  This
allows INSERT statements that even include an autoincremented SERIAL
or IDENTITY value to run very fast while still being able to return the
newly generated primary key values.   The ORM will then integrate this
new feature in a separate change.

Implements RETURNING for insert with executemany

Adds support to return_defaults() mode and inserted_primary_key
to support mutiple INSERTed rows, via return_defauls_rows
and inserted_primary_key_rows accessors.

within default execution context, new cached compiler
getters are used to fetch primary keys from rows

inserted_primary_key now returns a plain tuple. this
is not yet a row-like object however this can be
added.

Adds distinct "values_only" and "batch" modes, as
"values" has a lot of benefits but "batch" breaks
cursor.rowcount

psycopg2 minimum version 2.7 so we can remove the
large number of checks for very old versions of
psycopg2

simplify tests to no longer distinguish between
native and non-native json

Fixes: #5401
Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
2020-06-25 18:58:34 -04:00
Mike Bayer b0cfa7379c Turn on caching everywhere, add logging
A variety of caching issues found by running
all tests with statement caching turned on.

The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.

Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.

Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *".   if a table re-ordered its columns
or added/removed, now that data is obsolete.

Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.

Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement.   Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.

Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.

mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.

lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly.   While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.

turn on cache stats in logging.

Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context.   This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.

DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced.  this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.

memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.

Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason.   Removed
these methods as well as the concept of a Compiler
holding onto a "bind".

Fixes: #5386

Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
2020-06-10 15:29:01 -04:00
Mike Bayer 97cd0a5db8 Default create_constraint to False
The :paramref:`.Enum.create_constraint` and
:paramref:`.Boolean.create_constraint` parameters now default to False,
indicating when a so-called "non-native" version of these two datatypes is
created, a CHECK constraint will not be generated by default.   These CHECK
constraints present schema-management maintenance complexities that should
be opted in to, rather than being turned on by default.

Fixes: #5367
Change-Id: I0a3fb608ce32143fa757546cc17ba2013e93272a
2020-06-03 16:19:40 -04:00
RobotScribe 103260ddb4 Add with_for_update mysql new functionalities
Fixes: #4860

# Description
Add nowait, skip_lock, of arguments to for_update_clause for mysql

### Checklist

This pull request is:

- [ ] A documentation / typographical error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [x] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

**Have a nice day!**

Closes: #5290
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5290
Pull-request-sha: 490e822e73

Change-Id: Ibd2acc47b538c601c69c8fb954776035ecab4c6c
2020-05-15 15:50:32 -06:00
Federico Caselli aaec1bdedf Support ARRAY of Enum, JSON or JSONB
Added support for columns or type :class:`.ARRAY` of :class:`.Enum`,
:class:`.JSON` or :class:`_postgresql.JSONB` in PostgreSQL.
Previously a workaround was required in these use cases.
Raise an explicit :class:`.exc.CompileError` when adding a table with a
column of type :class:`.ARRAY` of :class:`.Enum` configured with
:paramref:`.Enum.native_enum` set to ``False`` when
:paramref:`.Enum.create_constraint` is not set to ``False``

Fixes: #5265
Fixes: #5266
Change-Id: I83a2d20a599232b7066d0839f3e55ff8b78cd8fc
2020-04-20 11:54:20 -04:00
Mike Bayer 8725d89aba Pass connection to TablesTest.insert_data()
towards the goal of reducing verbosity and repetition
in test fixtures as well as that we are moving to
connection only for execution, move the insert_data()
classmethod to accept a connection and adjust all
fixtures to use it.

Change-Id: I3bf534acca0d5f4cda1d4da8ae91f1155b829b09
2020-04-15 11:12:59 -04:00
Federico Caselli a9b068ae56 Remove code deprecated before version 1.1
- Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and
  :class:`.Inspector` classes.
- Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`.
- Remove support for deprecated engine URLs of the form ``postgres://``.
- Remove deprecated dialect ``mysql+gaerdbms``.
- Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM`
  and :class:`.mysql.SET` in the ``mysql`` dialect.
- Remove deprecated function ``comparable_property``. and function
  ``comparable_using`` in the declarative extension.
- Remove deprecated function ``compile_mappers``.
- Remove deprecated method ``collection.linker``.
- Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``.
  This change also removes the class ``StrongInstanceDict``.
- Remove deprecated parameter ``mapper.order_by``.
- Remove deprecated parameter ``Session._enable_transaction_accounting`.
- Remove deprecated parameter ``Session.is_modified.passive``.
- Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`.
- Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and
  ``ClauseElement.__or__`` and attribute ``Over.func``.
- Remove deprecated ``FromClause.count`` method.
- Remove deprecated parameter ``Table.useexisting``.
- Remove deprecated parameters ``text.bindparams`` and ``text.typemap``.

- Remove boolean support for the ``passive`` parameter in ``get_history``.
- Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``.

Fixes: #4643
Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
2020-04-09 00:33:22 +02:00
Gord Thompson 83c8231ac1 Clean up .execute calls in PostgreSQL tests
Fixes: #5220
Change-Id: I789e45dffc2b177ebb15ea3268bb965be8b06397
2020-03-31 08:48:51 -06:00
Mike Bayer cadfc608d6 Convert schema_translate to a post compile
Revised the :paramref:`.Connection.execution_options.schema_translate_map`
feature such that the processing of the SQL statement to receive a specific
schema name occurs within the execution phase of the statement, rather than
at the compile phase.   This is to support the statement being efficiently
cached.   Previously, the current schema being rendered into the statement
for a particular run would be considered as part of the cache key itself,
meaning that for a run against hundreds of schemas, there would be hundreds
of cache keys, rendering the cache much less performant.  The new behavior
is that the rendering is done in a similar  manner as the "post compile"
rendering added in 1.4 as part of 🎫`4645`, 🎫`4808`.

Fixes: #5004
Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
2020-03-24 14:25:41 -04:00
Federico Caselli 9ec7588220 Deprecate plain string in execute and introduce exec_driver_sql
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.

Fixes: #4848
Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
2020-03-21 17:03:45 -04:00
mike bayer ee9bd719b7 Don't include PG INCLUDE columns as regular index columns
Fixed issue where a "covering" index, e.g. those which have an  INCLUDE
clause, would be reflected including all the columns in INCLUDE clause as
regular columns.  A warning is now emitted if these additional columns are
detected indicating that they are currently ignored.  Note that full
support for "covering" indexes is part of 🎫`4458`.  Pull request
courtesy Marat Sharafutdinov.

Fixes: #5205
Closes: #5206
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5206
Pull-request-sha: 512a3817bb

Change-Id: I3196a2bf77dc5a6abd85b2fbf0ebff1b30d4fb00
2020-03-18 19:28:49 -04:00
Federico Caselli 62b7dace0c Support inspection of computed column
Added support for reflection of "computed" columns, which are now returned
as part of the structure returned by :meth:`.Inspector.get_columns`.
When reflecting full :class:`.Table` objects, computed columns will
be represented using the :class:`.Computed` construct.

Also improve the documentation in :meth:`Inspector.get_columns`, correctly
listing all the returned keys.

Fixes: #5063
Fixes: #4051

Closes: #5064
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5064
Pull-request-sha: ba00fc321ce468f8885aad23b3dd33c789e50fbe

Change-Id: I789986554fc8ac7f084270474d0b2c12046b1cc2
2020-03-15 19:09:04 -04:00
Mike Bayer 851fb8f5a6 Decouple compiler state from DML objects; make cacheable
Targeting select / insert / update / delete, the goal
is to minimize overhead of construction and generative methods
so that only the raw arguments passed are handled.   An interim
stage that converts the raw state into more compiler-ready state
is added, which is analogous to the ORM QueryContext which will
also be rolled in to be a similar concept, as is currently
being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10.
the ORM update/delete BulkUD concept is also going to be rolled
onto this idea.   So while the compiler-ready state object,
here called DMLState, looks a little thin, it's the
base of a bigger pattern that will allow for ORM functionality
to embed itself directly into the compiler, execution
context, and result set objects.

This change targets the DML objects, primarily focused on the
values() method which is the most complex process.   The
work done by values() is minimized as much as possible
while still being able to create a cache key.   Additional
computation is then offloaded to a new object ValuesState
that is handled by the compiler.

Architecturally, a big change here is that insert.values()
and update.values() will generate BindParameter objects for
the values now, which are then carefully received by crud.py
so that they generate the expected names.   This is so that
the values() portion of these constructs is cacheable.
for the "multi-values" version of Insert, this is all skipped
and the plan right now is that a multi-values insert is
not worth caching (can always be revisited).

Using the
coercions system in values() also gets us nicer validation
for free, we can remove the NotAClauseElement thing from
schema, and we also now require scalar_subquery() is called
for an insert/update that uses a SELECT as a column value,
1.x deprecation path is added.

The traversal system is then applied to the DML objects
including tests so that they have traversal, cloning, and
cache key support.  cloning is not a use case for DML however
having it present allows better validation of the structure
within the tests.

Special per-dialect DML is explicitly not cacheable at the moment,
more as a proof of concept that third party DML constructs can
exist as gracefully not-cacheable rather than producing an
incomplete cache key.

A few selected performance improvements have been added as well,
simplifying the immutabledict.union() method and adding
a new SQLCompiler function that can generate delimeter-separated
clauses like WHERE and ORDER BY without having to build
a ClauseList object at all.   The use of ClauseList will
be removed from Select in an upcoming commit.  Overall,
ClaustList is unnecessary for internal use and only adds
overhead to statement construction and will likely be removed
as much as possible except for explcit use of conjunctions like
and_() and or_().

Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
2020-03-06 11:01:51 -05:00
mike bayer 3dd23a57b1 Merge "Deprecate the databases packages" 2020-03-02 16:35:47 +00:00
Eric Borczuk 0fe528483a While parsing for check constraints, ignore newline characters in the check condition
Fixed bug where PostgreSQL reflection of CHECK constraints would fail to
parse the constraint if the SQL text contained newline characters. The
regular expression has been adjusted to accommodate for this case. Pull
request courtesy Eric Borczuk.

Fixes: #5170
Closes: #5172
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5172
Pull-request-sha: 5701b7f09f

Change-Id: If727e9140b645e8b685c3476fb0fa4417c1e6526
2020-02-29 17:45:05 -05:00
Federico Caselli c155ae910b Deprecate the databases packages
The databases package is an old alias to the dialect package, and its usage
is now deprecated

Change-Id: I5343a2d270ed5a8c654b9fe13dff40cdf54649ed
2020-02-28 20:21:10 +01:00
Mike Bayer f559f378c4 Result initial introduction
This builds on cc718cccc0 which moved
RowProxy to Row, allowing Row to be more like a named tuple.

- KeyedTuple in ORM is replaced with Row

- ResultSetMetaData broken out into "simple" and "cursor" versions
  for ORM and Core, as well as LegacyCursor version.

- Row now has _mapping attribute that supplies full mapping behavior.
Row and SimpleRow both have named tuple behavior otherwise.
LegacyRow has some mapping features on the tuple which emit
deprecation warnings (e.g. keys(), values(), etc).   the biggest
change for mapping->tuple is the behavior of __contains__ which
moves from testing of "key in row" to "value in row".

- ResultProxy breaks into ResultProxy and FutureResult (interim),
the latter has the newer APIs.   Made available to dialects
using execution options.

- internal reflection methods and most tests move off of implicit
Row mapping behavior and move to row._mapping, result.mappings()
method using future result

- a new strategy system for cursor handling replaces the various
subclasses of RowProxy

- some execution context adjustments. We will leave EC in but
refined things like get_result_proxy() and out parameter handling.
Dialects for 1.4 will need to adjust from get_result_proxy()
to get_result_cursor_strategy(), if they are using this method

- out parameter handling now accommodated by get_out_parameter_values()
EC method.   Oracle changes for this.  external dialect for
DB2 for example will also need to adjust for this.

- deprecate case_insensitive flag for engine / result, this
feature is not used

mapping-methods on Row are deprecated, and replaced with
Row._mapping.<meth>, including:

   row.keys()  -> use row._mapping.keys()
   row.items()  -> use row._mapping.items()
   row.values() -> use row._mapping.values()
   key in row  -> use key in row._mapping
   int in row  -> use int < len(row)

Fixes: #4710
Fixes: #4878
Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
2020-02-21 17:53:33 -05:00
Mike Bayer 93b7767d00 Deprecate connection branching
The :meth:`.Connection.connect` method is deprecated as is the concept of
"connection branching", which copies a :class:`.Connection` into a new one
that has a no-op ".close()" method.  This pattern is oriented around the
"connectionless execution" concept which is also being removed in 2.0.

As part of this change we begin to move the internals away from
"connectionless execution" overall.   Remove the "connectionless
execution" concept from the reflection internals and replace with
explicit patterns at the Inspector level.

Fixes: #5131
Change-Id: Id23d28a9889212ac5ae7329b85136157815d3e6f
2020-02-21 16:26:57 -05:00
Mike Bayer 89b8c343ed Pass DDLCompiler IdentifierPreparer to visit_ENUM
Fixed issue where the "schema_translate_map" feature would not work with a
PostgreSQL native enumeration type (i.e. :class:`.Enum`,
:class:`.postgresql.ENUM`) in that while the "CREATE TYPE" statement would
be emitted with the correct schema, the schema would not be rendered in
the CREATE TABLE statement at the point at which the enumeration was
referenced.

Fixes: #5158
Change-Id: I41529785de2e736c70a142c2ae5705060bfed73e
2020-02-17 15:35:44 -05:00
Federico Caselli bfddff4c9f Add pyproject
- Added pyproject.toml with black arguments
- Updated black version in precommit hook
- Reformatted the code

Fixes: #5100
Closes: #5103
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5103
Pull-request-sha: 795fd5f896

Change-Id: I14eedbaa51fb531cbf90fcefe6a1e07c8a565625
2020-01-27 16:56:25 -05:00
Gord Thompson d8ac1e9e6b Improve regex parsing of CHECK constraints for PostgreSQL.
Fixed issue where the PostgreSQL dialect would fail to parse a reflected
CHECK constraint that was a boolean-valued function (as opposed to a
boolean-valued expression).

Fixes: #5039
Closes: #5044
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5044
Pull-request-sha: b6903c6564

Change-Id: I7d39b104a8ce346cb593d541c1b4e5eab88867f9
2020-01-18 09:31:31 -05:00
Mike Bayer 9e31fc7408 Remove jython code, remove all jython / pypy symbols
Removed all dialect code related to support for Jython and zxJDBC. Jython
has not been supported by SQLAlchemy for many years and it is not expected
that the current zxJDBC code is at all functional; for the moment it just
takes up space and adds confusion by showing up in documentation. At the
moment, it appears that Jython has achieved Python 2.7 support in its
releases but not Python 3.   If Jython were to be supported again, the form
it should take is against the Python 3 version of Jython, and the various
zxJDBC stubs for various backends should be implemented as a third party
dialect.

Additionally modernized logic that distinguishes between "cpython"
and "pypy" to instead look at platform.python_distribution() which
reliably tells us if we are cPython or not; all booleans which
previously checked for pypy and sometimes jython are now converted
to be "not cpython", this impacts the test suite for tests that are
cPython centric.

Fixes: #5094
Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
2020-01-17 17:44:57 -05:00
Mike Bayer 01cbf4d7b8 Add type accessors for JSON indexed/pathed element access
Added new accessors to expressions of type :class:`.JSON` to allow for
specific datatype access and comparison, covering strings, integers,
numeric, boolean elements.   This revises the documented approach of
CASTing to string when comparing values, instead adding specific
functionality into the PostgreSQL, SQlite, MySQL dialects to reliably
deliver these basic types in all cases.

The change also delivers a new feature to the test exclusions
system so that combinations and exclusions can be used together.

Fixes: #4276
Change-Id: Ica5a926c060feb40a0a7cd60b9d6e061d7825728
2019-11-11 14:37:55 -05:00
CaselIT 3a0e0531c1 Support for generated columns
Added DDL support for "computed columns"; these are DDL column
specifications for columns that have a server-computed value, either upon
SELECT (known as "virtual") or at the point of which they are INSERTed or
UPDATEd (known as "stored").  Support is established for Postgresql, MySQL,
Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work
on this one.

ORM round trip tests included.  The ORM makes use of existing
FetchedValue support and no additional ORM logic is present for
the basic feature.

It has been observed that Oracle RETURNING does not return the
new value of a computed column upon UPDATE; it returns the
prior value.  As this is very dangerous, a warning is emitted
if a computed column is rendered into the RETURNING clause
of an UPDATE statement.

Fixes: #4894
Closes: #4928
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4928
Pull-request-sha: d39c521d5a

Change-Id: I2610b2999a5b1b127ed927dcdaeee98b769643ce
2019-11-08 15:40:25 -05:00
Mike Bayer 9fc54801c8 Cache every key in reflection_cache
Fixed bug in :class:`.Inspector` where the cache key generation did not
take into account arguments passed in the form of tuples, such as the tuple
of view name styles to return for the PostgreSQL dialect. This would lead
the inspector to cache too generally for a more specific set of criteria.
The logic has been adjusted to include every keyword element in the cache,
as every argument is expected to be appropriate for a cache else the
caching decorator should be bypassed by the dialect.

Fixes: #4955
Change-Id: Icd97744323407977a3f7db26e8a63a1a5c576010
2019-10-31 09:33:58 -04:00
Mike Bayer 240d9a60cc Refactor dialect tests for combinations
Dialect tests tend to have a lot of lists of types,
SQL constructs etc, convert as many of these to @combinations
as possible.

This is exposing that we don't have per-combination
exclusion rules set up which is making things a little bit
cumbersome.

Also set up a fixture that does metadata + DDL.

Change-Id: Ief820e48c9202982b0b1e181b87862490cd7b0c3
2019-10-22 18:27:48 -04:00