Enhanced the disambiguating labels feature of the
:func:`~.sql.expression.select` construct such that when a select statement
is used in a subquery, repeated column names from different tables are now
automatically labeled with a unique label name, without the need to use the
full "apply_labels()" feature that conbines tablename plus column name.
The disambigated labels are available as plain string keys in the .c
collection of the subquery, and most importantly the feature allows an ORM
:func:`.orm.aliased` construct against the combination of an entity and an
arbitrary subquery to work correctly, targeting the correct columns despite
same-named columns in the source tables, without the need for an "apply
labels" warning.
The existing labeling style is now called
LABEL_STYLE_TABLENAME_PLUS_COL. This labeling style will remain used
throughout the ORM as has been the case for over a decade, however,
the new disambiguation scheme could theoretically replace this scheme
entirely. The new scheme would dramatically alter how SQL looks
when rendered from the ORM to be more succinct but arguably harder
to read.
The tablename_columnname scheme used by Join.c is unaffected here,
as that's still hardcoded to that scheme.
Fixes: #5221
Change-Id: Ib47d9e0f35046b3afc77bef6e65709b93d0c3026
Add ability to literal compile a :class:`DateTime`, :class:`Date`
or :class:"Time" when using the string dialect for debugging purposes.
This change does not impact real dialect implementation that retain
their current behavior.
Fixes: #5052
Change-Id: Ia3fad2be905c6d35b0106b9a2388c7508f067e90
Revised the :paramref:`.Connection.execution_options.schema_translate_map`
feature such that the processing of the SQL statement to receive a specific
schema name occurs within the execution phase of the statement, rather than
at the compile phase. This is to support the statement being efficiently
cached. Previously, the current schema being rendered into the statement
for a particular run would be considered as part of the cache key itself,
meaning that for a run against hundreds of schemas, there would be hundreds
of cache keys, rendering the cache much less performant. The new behavior
is that the rendering is done in a similar manner as the "post compile"
rendering added in 1.4 as part of 🎫`4645`, 🎫`4808`.
Fixes: #5004
Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
Added a core :class:`Values` object that enables a VALUES construct
to be used in the FROM clause of an SQL statement for databases that
support it (mainly PostgreSQL and SQL Server).
Fixes: #4868Closes: #5030
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5030
Pull-request-sha: 84684038a8
Change-Id: Ib8109b63bc1a9dc04ab987c5322ca3375f7e824d
Removed autocommit and legacy "for update" / "lockmode" elements
from selectable.py / query.py. lockmode was removed from
selectable in 693938dd6f
however was not removed from the ORM.
Also removes the ignore_nonexistent_tables option on
join().
Change-Id: I0cfcf9e6a8d4ef6432c9e25ef75173b3b3f5fd87
Partially-fixes: #4643
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.
Fixes: #4848Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9
- add an options() method to the base Generative construct.
this will be where ORM options can go
- Change Null, False_, True_ to be singletons, so that
we aren't instantiating them and having to use isinstance.
The previous issue with this was that they would produce dupe
labels in SELECT statements. Apply the duplicate column
logic, newly added in 1.4, to these objects as well as to
non-apply-labels SELECT statements in general as a means of
improving this.
- create a revised system for generating ClauseList compilation
constructs that simplfies up front creation to not actually
use ClauseList; a simple tuple is rendered by the compiler
using the same constrcution rules as what are used for
ClauseList but without creating the actual object. Apply
to Select, CompoundSelect, revise Update, Delete
- Select, CompoundSelect get an initial CompileState
implementation. All methods used only within compilation
are moved here
- refine update/insert/delete compile state to not require
an outside boolean
- refine and simplify Select._copy_internals
- rework bind(), which is going away, to not use some
of the internal traversal stuff
- remove "autocommit", "for_update" parameters from Select,
references #4643
- remove "autocommit" parameter from TextClause ,
references #4643
- add deprecation warnings for statement.execute(),
engine.execute(), statement.scalar(), engine.scalar().
Fixes: #5193
Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
Targeting select / insert / update / delete, the goal
is to minimize overhead of construction and generative methods
so that only the raw arguments passed are handled. An interim
stage that converts the raw state into more compiler-ready state
is added, which is analogous to the ORM QueryContext which will
also be rolled in to be a similar concept, as is currently
being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10.
the ORM update/delete BulkUD concept is also going to be rolled
onto this idea. So while the compiler-ready state object,
here called DMLState, looks a little thin, it's the
base of a bigger pattern that will allow for ORM functionality
to embed itself directly into the compiler, execution
context, and result set objects.
This change targets the DML objects, primarily focused on the
values() method which is the most complex process. The
work done by values() is minimized as much as possible
while still being able to create a cache key. Additional
computation is then offloaded to a new object ValuesState
that is handled by the compiler.
Architecturally, a big change here is that insert.values()
and update.values() will generate BindParameter objects for
the values now, which are then carefully received by crud.py
so that they generate the expected names. This is so that
the values() portion of these constructs is cacheable.
for the "multi-values" version of Insert, this is all skipped
and the plan right now is that a multi-values insert is
not worth caching (can always be revisited).
Using the
coercions system in values() also gets us nicer validation
for free, we can remove the NotAClauseElement thing from
schema, and we also now require scalar_subquery() is called
for an insert/update that uses a SELECT as a column value,
1.x deprecation path is added.
The traversal system is then applied to the DML objects
including tests so that they have traversal, cloning, and
cache key support. cloning is not a use case for DML however
having it present allows better validation of the structure
within the tests.
Special per-dialect DML is explicitly not cacheable at the moment,
more as a proof of concept that third party DML constructs can
exist as gracefully not-cacheable rather than producing an
incomplete cache key.
A few selected performance improvements have been added as well,
simplifying the immutabledict.union() method and adding
a new SQLCompiler function that can generate delimeter-separated
clauses like WHERE and ORDER BY without having to build
a ClauseList object at all. The use of ClauseList will
be removed from Select in an upcoming commit. Overall,
ClaustList is unnecessary for internal use and only adds
overhead to statement construction and will likely be removed
as much as possible except for explcit use of conjunctions like
and_() and or_().
Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
Fixed bug where a CTE of an INSERT/UPDATE/DELETE that also uses RETURNING
could then not be SELECTed from directly, as the internal state of the
compiler would try to treat the outer SELECT as a DELETE statement itself
and access nonexistent state.
Fixes: #5181
Change-Id: Icba76f2148c8344baa1c04bac4ab6c6d24f23072
Applied an explicit "cause" to most if not all internally raised exceptions
that are raised from within an internal exception catch, to avoid
misleading stacktraces that suggest an error within the handling of an
exception. While it would be preferable to suppress the internally caught
exception in the way that the ``__suppress_context__`` attribute would,
there does not as yet seem to be a way to do this without suppressing an
enclosing user constructed context, so for now it exposes the internally
caught exception as the cause so that full information about the context
of the error is maintained.
Fixes: #4849
Change-Id: I55a86b29023675d9e5e49bc7edc5a2dc0bcd4751
Removed very antiquated logic that checks if __visit_name__
is a property. There's no need for this as the compiler can handle
switching between implementations. Convert _compile_dispatch()
to be fully inlined.
Change-Id: Ic0c7247c2d7dfed93a27f09250a8ed6352370764
row.keys() is used by any use case that applies dict() to
a row. Access of elements by string key is also a 2.0 deprecation
not 1.4 so for rudimental dict(row) support make sure that is all
a 2.0 thing.
Fixes current Alembic test suite.
Change-Id: I895496324133d615676cd76bc5f2c5f4a83e9131
In 29330ec159 we ensured that annotations are part of cache keys.
However we failed to do so for the schema-level Table which
will definitely need to distinguish between ORM and non-ORM
annotated tables when caching, so ensure this is part of the
cache key.
Change-Id: I8d996873f2d7fa63230ef837db7e69a0101973b2
HasPrefix / HasSuffixes / SupportsCloneAnnotations exported
a _traverse_internals attribute that does not represent a
complete traversal, meaning non-traversible subclasses would
seem traversible. rename these attributes so that this
does not occur. DML is currently not traversible (will be soon).
Change-Id: I2605e61c8c3d49965335e66e09f4aeedc5e73bd3
In 9fca5d827d we attempted to deprecate the "inline=True" flag
and add a generative inline() method, however failed to include
any tests and the method was implemented incorrectly such that
it would get overwritten with the boolean flag immediately.
Rename the internal "inline" flag to "_inline" and add test
support both for the method as well as deprecated support
for the flag, including a fixture addition to assert the expected
value of the flag as it generally does not affect the
actual compiled SQL string.
Change-Id: I0450049f17f1f0d91e22d27f1a973a2b6c0e59f7
External dialects will definitely want to
be able to test their handling of Unicode
table/column names.
Change-Id: If1b67cf170dc9e4a42e3f51760ced8ddb7a34fcf
This builds on cc718cccc0 which moved
RowProxy to Row, allowing Row to be more like a named tuple.
- KeyedTuple in ORM is replaced with Row
- ResultSetMetaData broken out into "simple" and "cursor" versions
for ORM and Core, as well as LegacyCursor version.
- Row now has _mapping attribute that supplies full mapping behavior.
Row and SimpleRow both have named tuple behavior otherwise.
LegacyRow has some mapping features on the tuple which emit
deprecation warnings (e.g. keys(), values(), etc). the biggest
change for mapping->tuple is the behavior of __contains__ which
moves from testing of "key in row" to "value in row".
- ResultProxy breaks into ResultProxy and FutureResult (interim),
the latter has the newer APIs. Made available to dialects
using execution options.
- internal reflection methods and most tests move off of implicit
Row mapping behavior and move to row._mapping, result.mappings()
method using future result
- a new strategy system for cursor handling replaces the various
subclasses of RowProxy
- some execution context adjustments. We will leave EC in but
refined things like get_result_proxy() and out parameter handling.
Dialects for 1.4 will need to adjust from get_result_proxy()
to get_result_cursor_strategy(), if they are using this method
- out parameter handling now accommodated by get_out_parameter_values()
EC method. Oracle changes for this. external dialect for
DB2 for example will also need to adjust for this.
- deprecate case_insensitive flag for engine / result, this
feature is not used
mapping-methods on Row are deprecated, and replaced with
Row._mapping.<meth>, including:
row.keys() -> use row._mapping.keys()
row.items() -> use row._mapping.items()
row.values() -> use row._mapping.values()
key in row -> use key in row._mapping
int in row -> use int < len(row)
Fixes: #4710Fixes: #4878
Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
The :meth:`.Connection.connect` method is deprecated as is the concept of
"connection branching", which copies a :class:`.Connection` into a new one
that has a no-op ".close()" method. This pattern is oriented around the
"connectionless execution" concept which is also being removed in 2.0.
As part of this change we begin to move the internals away from
"connectionless execution" overall. Remove the "connectionless
execution" concept from the reflection internals and replace with
explicit patterns at the Inspector level.
Fixes: #5131
Change-Id: Id23d28a9889212ac5ae7329b85136157815d3e6f
Reorganization of Select() is the first major element
of the 2.0 restructuring. In order to start this we need
to first create the new Select constructor and apply legacy
elements to the old one. This in turn necessitates
starting up the RemovedIn20Warning concept which itself
need to refer to "sqlalchemy.future", so begin to establish
this basic framework. Additionally, update the
DML constructors with the newer no-keyword style. Remove
the use of the "pending deprecation" and fix Query.add_column()
deprecation which was not acting as deprecated.
Fixes: #4845Fixes: #4648
Change-Id: I0c7a22b2841a985e1c379a0bb6c94089aae6264c
SQL Server OFFSET and FETCH keywords are now used for limit/offset, rather
than using a window function, for SQL Server versions 11 and higher. TOP is
still used for a query that features only LIMIT. Pull request courtesy
Elkin.
Fixes: #5084Closes: #5125
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5125
Pull-request-sha: a45b7f7309
Change-Id: Id6a01ba30caac87d7d3d92c3903cdfd77fbcee5e
Creating an :func:`.and_` or :func:`.or_` construct with no arguments or
empty ``*args`` will now emit a deprecation warning, as the SQL produced is
a no-op (i.e. it renders as a blank string). This behavior is considered to
be non-intuitive, so for empty or possibly empty :func:`.and_` or
:func:`.or_` constructs, an appropriate default boolean should be included,
such as ``and_(True, *args)`` or ``or_(False, *args)``. As has been the
case for many major versions of SQLAlchemy, these particular boolean
values will not render if the ``*args`` portion is non-empty.
As there are some internal cases where an empty and_() construct is used
in order to build an optional WHERE expression, a private
utility function is added to suit this use case.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5054Closes: #5062
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062
Pull-request-sha: 5ca2f27281
Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
Added "from linting" as a built-in feature to the SQL compiler. This
allows the compiler to maintain graph of all the FROM clauses in a
particular SELECT statement, linked by criteria in either the WHERE
or in JOIN clauses that link these FROM clauses together. If any two
FROM clauses have no path between them, a warning is emitted that the
query may be producing a cartesian product. As the Core expression
language as well as the ORM are built on an "implicit FROMs" model where
a particular FROM clause is automatically added if any part of the query
refers to it, it is easy for this to happen inadvertently and it is
hoped that the new feature helps with this issue.
The original recipe is from:
https://github.com/sqlalchemy/sqlalchemy/wiki/FromLinter
The linter is now enabled for all tests in the test suite as well.
This has necessitated that a lot of the queries be adjusted to
not include cartesian products. Part of the rationale for the
linter to not be enabled for statement compilation only was to reduce
the need for adjustment for the many test case statements throughout
the test suite that are not real-world statements.
This gerrit is adapted from Ib5946e57c9dba6da428c4d1dee6760b3e978dda0.
Fixes: #4737
Change-Id: Ic91fd9774379f895d021c3ad564db6062299211c
Closes: #4830
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4830
Pull-request-sha: f8a21aa626
Removed all dialect code related to support for Jython and zxJDBC. Jython
has not been supported by SQLAlchemy for many years and it is not expected
that the current zxJDBC code is at all functional; for the moment it just
takes up space and adds confusion by showing up in documentation. At the
moment, it appears that Jython has achieved Python 2.7 support in its
releases but not Python 3. If Jython were to be supported again, the form
it should take is against the Python 3 version of Jython, and the various
zxJDBC stubs for various backends should be implemented as a third party
dialect.
Additionally modernized logic that distinguishes between "cpython"
and "pypy" to instead look at platform.python_distribution() which
reliably tells us if we are cPython or not; all booleans which
previously checked for pypy and sometimes jython are now converted
to be "not cpython", this impacts the test suite for tests that are
cPython centric.
Fixes: #5094
Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
The :func:`.true` and :func:`.false` operators may now be applied as the
"onclause" of a :func:`.sql.join` on a backend that does not support
"native boolean" expressions, e.g. Oracle or SQL Server, and the expression
will render as "1=1" for true and "1=0" false. This is the behavior that
was introduced many years ago in 🎫`2804` for and/or expressions.
Change-Id: I85311c31c22d6e226c618f8840f6b95eca611153
A function created using :class:`.GenericFunction` can now specify that the
name of the function should be rendered with or without quotes by assigning
the :class:`.quoted_name` construct to the .name element of the object.
Prior to 1.3.4, quoting was never applied to function names, and some
quoting was introduced in 🎫`4467` but no means to force quoting for
a mixed case name was available. Additionally, the :class:`.quoted_name`
construct when used as the name will properly register its lowercase name
in the function registry so that the name continues to be available via the
``func.`` registry.
Fixes: #5079
Change-Id: I0653ab8b16e75e628ce82dbbc3d0f77f8336c407
in trying to apply 2020 copyright to files, the pre-commit
hooks complain about random file issues.
- remove old corrections.py utility, this had something to do
with repairing refs in the sphinx docs
- run pre commit hooks on all files
- formatting adjustments to work around code formatting collisions
(long import lines that zimports can't rewrite correctly)
Change-Id: I260744866f69e902eb93665c7c728ee94d3371a2
Added test support and repaired a wide variety of unnecessary reference
cycles created for short-lived objects, mostly in the area of ORM queries.
Fixes: #5056
Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
The "expanding IN" feature, which generates IN expressions at query
execution time which are based on the particular parameters associated with
the statement execution, is now used for all IN expressions made against
lists of literal values. This allows IN expressions to be fully cacheable
independently of the list of values being passed, and also includes support
for empty lists. For any scenario where the IN expression contains
non-literal SQL expressions, the old behavior of pre-rendering for each
position in the IN is maintained. The change also completes support for
expanding IN with tuples, where previously type-specific bind processors
weren't taking effect.
As part of this change, a more explicit separation between
"literal execute" and "post compile" bound parameters is being made;
as the "ansi bind rules" feature is rendering bound parameters
inline, as we now support "postcompile" generically, these should
be used here, however we have to render literal values at
execution time even for "expanding" parameters. new test fixtures
etc. are added to assert everything goes to the right place.
Fixes: #4645
Change-Id: Iaa2b7bfbfaaf5b80799ee17c9b8507293cba6ed1
Fixed issue where the collection of value processors on a
:class:`.Compiled` object would be mutated when "expanding IN" parameters
were used with a datatype that has bind value processors; in particular,
this would mean that when using statement caching and/or baked queries, the
same compiled._bind_processors collection would be mutated concurrently.
Since these processors are the same function for a given bind parameter
namespace every time, there was no actual negative effect of this issue,
however, the execution of a :class:`.Compiled` object should never be
causing any changes in its state, especially given that they are intended
to be thread-safe and reusable once fully constructed.
Fixes: #5048
Change-Id: I876d16bd7484eb05ce590397420552ac36da6e52
upcoming changes for "expanding IN in all cases" and
"lambda elements" both rely upon comparisons that work
across changing bound values, so commit the testing fixture
ahead of time. Additionally, repair the feature itself
within traversals.
Change-Id: Ie65a512dc64745614180da77435f9f745ce78c71
Added support for prefixes to the :class:`.CTE` construct, to allow
support for Postgresql 12 "MATERIALIZED" and "NOT MATERIALIZED" phrases.
Pull request courtesy Marat Sharafutdinov.
Fixes: #5040Closes: #5043
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5043
Pull-request-sha: d1b9059a0b
Change-Id: I2e9cb5d7f85961ec98ee51965de5b3ec4a97be2f
Starting to go forward with the general idea of moving more
of Core / ORM construction into the compile phase. Bigger
initiatives like the refactor of Query will follow onto this.
Change-Id: I0f364d3182e21e32ed85ef34cfd11fd9d11cf653
Fixed bug where "distinct" keyword passed to :func:`.select` would not
treat a string value as a "label reference" in the same way that the
:meth:`.select.distinct` does; it would instead raise unconditionally. This
keyword argument and the others passed to :func:`.select` will ultimately
be deprecated for SQLAlchemy 2.0.
Fixes: #5028
Change-Id: Id36cfe477ed836c3248824ce1b81d0016dbe99f4
Needed to add tests to ensure this label reference is handled
correctly, so also modified the exception message to
be more clear if someone has this error within distinct().
Change-Id: I6e685e46ae336596272d14366445ac224c18d92c
Added support for use of the :class:`.Sequence` construct with MariaDB 10.3
and greater, as this is now supported by this database. The construct
integrates with the :class:`.Table` object in the same way that it does for
other databases like PostrgreSQL and Oracle; if is present on the integer
primary key "autoincrement" column, it is used to generate defaults. For
backwards compatibility, to support a :class:`.Table` that has a
:class:`.Sequence` on it to support sequence only databases like Oracle,
while still not having the sequence fire off for MariaDB, the optional=True
flag should be set, which indicates the sequence should only be used to
generate the primary key if the target database offers no other option.
Fixes: #4976Closes: #4996
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4996
Pull-request-sha: cb2e1426ea
Change-Id: I507bc405eee6cae2c5991345d0eac53a37fe7512
Fixed issue where when constructing constraints from ORM-bound columns,
primarily :class:`.ForeignKey` objects but also :class:`.UniqueConstraint`,
:class:`.CheckConstraint` and others, the ORM-level
:class:`.InstrumentedAttribute` is discarded entirely, and all ORM-level
annotations from the columns are removed; this is so that the constraints
are still fully pickleable without the ORM-level entities being pulled in.
These annotations are not necessary to be present at the schema/metadata
level.
Fully implemented coercions for constraint columns within
schema.py, including for FK referenced columns.
Fixes: #5001
Change-Id: I895400dd979310be034085d207f096707c635909
mike bayer