row.keys() is used by any use case that applies dict() to
a row. Access of elements by string key is also a 2.0 deprecation
not 1.4 so for rudimental dict(row) support make sure that is all
a 2.0 thing.
Fixes current Alembic test suite.
Change-Id: I895496324133d615676cd76bc5f2c5f4a83e9131
In 29330ec159 we ensured that annotations are part of cache keys.
However we failed to do so for the schema-level Table which
will definitely need to distinguish between ORM and non-ORM
annotated tables when caching, so ensure this is part of the
cache key.
Change-Id: I8d996873f2d7fa63230ef837db7e69a0101973b2
HasPrefix / HasSuffixes / SupportsCloneAnnotations exported
a _traverse_internals attribute that does not represent a
complete traversal, meaning non-traversible subclasses would
seem traversible. rename these attributes so that this
does not occur. DML is currently not traversible (will be soon).
Change-Id: I2605e61c8c3d49965335e66e09f4aeedc5e73bd3
In 9fca5d827d we attempted to deprecate the "inline=True" flag
and add a generative inline() method, however failed to include
any tests and the method was implemented incorrectly such that
it would get overwritten with the boolean flag immediately.
Rename the internal "inline" flag to "_inline" and add test
support both for the method as well as deprecated support
for the flag, including a fixture addition to assert the expected
value of the flag as it generally does not affect the
actual compiled SQL string.
Change-Id: I0450049f17f1f0d91e22d27f1a973a2b6c0e59f7
External dialects will definitely want to
be able to test their handling of Unicode
table/column names.
Change-Id: If1b67cf170dc9e4a42e3f51760ced8ddb7a34fcf
This builds on cc718cccc0 which moved
RowProxy to Row, allowing Row to be more like a named tuple.
- KeyedTuple in ORM is replaced with Row
- ResultSetMetaData broken out into "simple" and "cursor" versions
for ORM and Core, as well as LegacyCursor version.
- Row now has _mapping attribute that supplies full mapping behavior.
Row and SimpleRow both have named tuple behavior otherwise.
LegacyRow has some mapping features on the tuple which emit
deprecation warnings (e.g. keys(), values(), etc). the biggest
change for mapping->tuple is the behavior of __contains__ which
moves from testing of "key in row" to "value in row".
- ResultProxy breaks into ResultProxy and FutureResult (interim),
the latter has the newer APIs. Made available to dialects
using execution options.
- internal reflection methods and most tests move off of implicit
Row mapping behavior and move to row._mapping, result.mappings()
method using future result
- a new strategy system for cursor handling replaces the various
subclasses of RowProxy
- some execution context adjustments. We will leave EC in but
refined things like get_result_proxy() and out parameter handling.
Dialects for 1.4 will need to adjust from get_result_proxy()
to get_result_cursor_strategy(), if they are using this method
- out parameter handling now accommodated by get_out_parameter_values()
EC method. Oracle changes for this. external dialect for
DB2 for example will also need to adjust for this.
- deprecate case_insensitive flag for engine / result, this
feature is not used
mapping-methods on Row are deprecated, and replaced with
Row._mapping.<meth>, including:
row.keys() -> use row._mapping.keys()
row.items() -> use row._mapping.items()
row.values() -> use row._mapping.values()
key in row -> use key in row._mapping
int in row -> use int < len(row)
Fixes: #4710Fixes: #4878
Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
The :meth:`.Connection.connect` method is deprecated as is the concept of
"connection branching", which copies a :class:`.Connection` into a new one
that has a no-op ".close()" method. This pattern is oriented around the
"connectionless execution" concept which is also being removed in 2.0.
As part of this change we begin to move the internals away from
"connectionless execution" overall. Remove the "connectionless
execution" concept from the reflection internals and replace with
explicit patterns at the Inspector level.
Fixes: #5131
Change-Id: Id23d28a9889212ac5ae7329b85136157815d3e6f
Reorganization of Select() is the first major element
of the 2.0 restructuring. In order to start this we need
to first create the new Select constructor and apply legacy
elements to the old one. This in turn necessitates
starting up the RemovedIn20Warning concept which itself
need to refer to "sqlalchemy.future", so begin to establish
this basic framework. Additionally, update the
DML constructors with the newer no-keyword style. Remove
the use of the "pending deprecation" and fix Query.add_column()
deprecation which was not acting as deprecated.
Fixes: #4845Fixes: #4648
Change-Id: I0c7a22b2841a985e1c379a0bb6c94089aae6264c
SQL Server OFFSET and FETCH keywords are now used for limit/offset, rather
than using a window function, for SQL Server versions 11 and higher. TOP is
still used for a query that features only LIMIT. Pull request courtesy
Elkin.
Fixes: #5084Closes: #5125
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5125
Pull-request-sha: a45b7f7309
Change-Id: Id6a01ba30caac87d7d3d92c3903cdfd77fbcee5e
Creating an :func:`.and_` or :func:`.or_` construct with no arguments or
empty ``*args`` will now emit a deprecation warning, as the SQL produced is
a no-op (i.e. it renders as a blank string). This behavior is considered to
be non-intuitive, so for empty or possibly empty :func:`.and_` or
:func:`.or_` constructs, an appropriate default boolean should be included,
such as ``and_(True, *args)`` or ``or_(False, *args)``. As has been the
case for many major versions of SQLAlchemy, these particular boolean
values will not render if the ``*args`` portion is non-empty.
As there are some internal cases where an empty and_() construct is used
in order to build an optional WHERE expression, a private
utility function is added to suit this use case.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5054Closes: #5062
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062
Pull-request-sha: 5ca2f27281
Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
Added "from linting" as a built-in feature to the SQL compiler. This
allows the compiler to maintain graph of all the FROM clauses in a
particular SELECT statement, linked by criteria in either the WHERE
or in JOIN clauses that link these FROM clauses together. If any two
FROM clauses have no path between them, a warning is emitted that the
query may be producing a cartesian product. As the Core expression
language as well as the ORM are built on an "implicit FROMs" model where
a particular FROM clause is automatically added if any part of the query
refers to it, it is easy for this to happen inadvertently and it is
hoped that the new feature helps with this issue.
The original recipe is from:
https://github.com/sqlalchemy/sqlalchemy/wiki/FromLinter
The linter is now enabled for all tests in the test suite as well.
This has necessitated that a lot of the queries be adjusted to
not include cartesian products. Part of the rationale for the
linter to not be enabled for statement compilation only was to reduce
the need for adjustment for the many test case statements throughout
the test suite that are not real-world statements.
This gerrit is adapted from Ib5946e57c9dba6da428c4d1dee6760b3e978dda0.
Fixes: #4737
Change-Id: Ic91fd9774379f895d021c3ad564db6062299211c
Closes: #4830
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4830
Pull-request-sha: f8a21aa626
Removed all dialect code related to support for Jython and zxJDBC. Jython
has not been supported by SQLAlchemy for many years and it is not expected
that the current zxJDBC code is at all functional; for the moment it just
takes up space and adds confusion by showing up in documentation. At the
moment, it appears that Jython has achieved Python 2.7 support in its
releases but not Python 3. If Jython were to be supported again, the form
it should take is against the Python 3 version of Jython, and the various
zxJDBC stubs for various backends should be implemented as a third party
dialect.
Additionally modernized logic that distinguishes between "cpython"
and "pypy" to instead look at platform.python_distribution() which
reliably tells us if we are cPython or not; all booleans which
previously checked for pypy and sometimes jython are now converted
to be "not cpython", this impacts the test suite for tests that are
cPython centric.
Fixes: #5094
Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
The :func:`.true` and :func:`.false` operators may now be applied as the
"onclause" of a :func:`.sql.join` on a backend that does not support
"native boolean" expressions, e.g. Oracle or SQL Server, and the expression
will render as "1=1" for true and "1=0" false. This is the behavior that
was introduced many years ago in 🎫`2804` for and/or expressions.
Change-Id: I85311c31c22d6e226c618f8840f6b95eca611153
A function created using :class:`.GenericFunction` can now specify that the
name of the function should be rendered with or without quotes by assigning
the :class:`.quoted_name` construct to the .name element of the object.
Prior to 1.3.4, quoting was never applied to function names, and some
quoting was introduced in 🎫`4467` but no means to force quoting for
a mixed case name was available. Additionally, the :class:`.quoted_name`
construct when used as the name will properly register its lowercase name
in the function registry so that the name continues to be available via the
``func.`` registry.
Fixes: #5079
Change-Id: I0653ab8b16e75e628ce82dbbc3d0f77f8336c407
in trying to apply 2020 copyright to files, the pre-commit
hooks complain about random file issues.
- remove old corrections.py utility, this had something to do
with repairing refs in the sphinx docs
- run pre commit hooks on all files
- formatting adjustments to work around code formatting collisions
(long import lines that zimports can't rewrite correctly)
Change-Id: I260744866f69e902eb93665c7c728ee94d3371a2
Added test support and repaired a wide variety of unnecessary reference
cycles created for short-lived objects, mostly in the area of ORM queries.
Fixes: #5056
Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
The "expanding IN" feature, which generates IN expressions at query
execution time which are based on the particular parameters associated with
the statement execution, is now used for all IN expressions made against
lists of literal values. This allows IN expressions to be fully cacheable
independently of the list of values being passed, and also includes support
for empty lists. For any scenario where the IN expression contains
non-literal SQL expressions, the old behavior of pre-rendering for each
position in the IN is maintained. The change also completes support for
expanding IN with tuples, where previously type-specific bind processors
weren't taking effect.
As part of this change, a more explicit separation between
"literal execute" and "post compile" bound parameters is being made;
as the "ansi bind rules" feature is rendering bound parameters
inline, as we now support "postcompile" generically, these should
be used here, however we have to render literal values at
execution time even for "expanding" parameters. new test fixtures
etc. are added to assert everything goes to the right place.
Fixes: #4645
Change-Id: Iaa2b7bfbfaaf5b80799ee17c9b8507293cba6ed1
Fixed issue where the collection of value processors on a
:class:`.Compiled` object would be mutated when "expanding IN" parameters
were used with a datatype that has bind value processors; in particular,
this would mean that when using statement caching and/or baked queries, the
same compiled._bind_processors collection would be mutated concurrently.
Since these processors are the same function for a given bind parameter
namespace every time, there was no actual negative effect of this issue,
however, the execution of a :class:`.Compiled` object should never be
causing any changes in its state, especially given that they are intended
to be thread-safe and reusable once fully constructed.
Fixes: #5048
Change-Id: I876d16bd7484eb05ce590397420552ac36da6e52
upcoming changes for "expanding IN in all cases" and
"lambda elements" both rely upon comparisons that work
across changing bound values, so commit the testing fixture
ahead of time. Additionally, repair the feature itself
within traversals.
Change-Id: Ie65a512dc64745614180da77435f9f745ce78c71
Added support for prefixes to the :class:`.CTE` construct, to allow
support for Postgresql 12 "MATERIALIZED" and "NOT MATERIALIZED" phrases.
Pull request courtesy Marat Sharafutdinov.
Fixes: #5040Closes: #5043
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5043
Pull-request-sha: d1b9059a0b
Change-Id: I2e9cb5d7f85961ec98ee51965de5b3ec4a97be2f
Starting to go forward with the general idea of moving more
of Core / ORM construction into the compile phase. Bigger
initiatives like the refactor of Query will follow onto this.
Change-Id: I0f364d3182e21e32ed85ef34cfd11fd9d11cf653
Fixed bug where "distinct" keyword passed to :func:`.select` would not
treat a string value as a "label reference" in the same way that the
:meth:`.select.distinct` does; it would instead raise unconditionally. This
keyword argument and the others passed to :func:`.select` will ultimately
be deprecated for SQLAlchemy 2.0.
Fixes: #5028
Change-Id: Id36cfe477ed836c3248824ce1b81d0016dbe99f4
Needed to add tests to ensure this label reference is handled
correctly, so also modified the exception message to
be more clear if someone has this error within distinct().
Change-Id: I6e685e46ae336596272d14366445ac224c18d92c
Added support for use of the :class:`.Sequence` construct with MariaDB 10.3
and greater, as this is now supported by this database. The construct
integrates with the :class:`.Table` object in the same way that it does for
other databases like PostrgreSQL and Oracle; if is present on the integer
primary key "autoincrement" column, it is used to generate defaults. For
backwards compatibility, to support a :class:`.Table` that has a
:class:`.Sequence` on it to support sequence only databases like Oracle,
while still not having the sequence fire off for MariaDB, the optional=True
flag should be set, which indicates the sequence should only be used to
generate the primary key if the target database offers no other option.
Fixes: #4976Closes: #4996
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4996
Pull-request-sha: cb2e1426ea
Change-Id: I507bc405eee6cae2c5991345d0eac53a37fe7512
Fixed issue where when constructing constraints from ORM-bound columns,
primarily :class:`.ForeignKey` objects but also :class:`.UniqueConstraint`,
:class:`.CheckConstraint` and others, the ORM-level
:class:`.InstrumentedAttribute` is discarded entirely, and all ORM-level
annotations from the columns are removed; this is so that the constraints
are still fully pickleable without the ORM-level entities being pulled in.
These annotations are not necessary to be present at the schema/metadata
level.
Fully implemented coercions for constraint columns within
schema.py, including for FK referenced columns.
Fixes: #5001
Change-Id: I895400dd979310be034085d207f096707c635909
The :class:`.oracle.INTERVAL` class of the Oracle dialect is now correctly
a subclass of the abstract version of :class:`.Interval` as well as the
correct "emulated" base class, which allows for correct behavior under both
native and non-native modes; previously it was only based on
:class:`.TypeEngine`.
Fixes: #4971
Change-Id: I4400d9f090330388460cca930e4139e3bd21eb11
Added new accessors to expressions of type :class:`.JSON` to allow for
specific datatype access and comparison, covering strings, integers,
numeric, boolean elements. This revises the documented approach of
CASTing to string when comparing values, instead adding specific
functionality into the PostgreSQL, SQlite, MySQL dialects to reliably
deliver these basic types in all cases.
The change also delivers a new feature to the test exclusions
system so that combinations and exclusions can be used together.
Fixes: #4276
Change-Id: Ica5a926c060feb40a0a7cd60b9d6e061d7825728
The fails_on decorator was not being interpreted
correctly when multiple were present.
Remove obsolete fails_on from test_types that no longer
take place for MySQL, Oracle.
Ensure test_types tests are using __backend__
mark currently failing Oracle interval tests
Change-Id: If8db0c02b31a8008fd1673c2380f1f974c3806a6
Added DDL support for "computed columns"; these are DDL column
specifications for columns that have a server-computed value, either upon
SELECT (known as "virtual") or at the point of which they are INSERTed or
UPDATEd (known as "stored"). Support is established for Postgresql, MySQL,
Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work
on this one.
ORM round trip tests included. The ORM makes use of existing
FetchedValue support and no additional ORM logic is present for
the basic feature.
It has been observed that Oracle RETURNING does not return the
new value of a computed column upon UPDATE; it returns the
prior value. As this is very dangerous, a warning is emitted
if a computed column is rendered into the RETURNING clause
of an UPDATE statement.
Fixes: #4894Closes: #4928
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4928
Pull-request-sha: d39c521d5a
Change-Id: I2610b2999a5b1b127ed927dcdaeee98b769643ce
Created new visitor system called "internal traversal" that
applies a data driven approach to the concept of a class that
defines its own traversal steps, in contrast to the existing
style of traversal now known as "external traversal" where
the visitor class defines the traversal, i.e. the SQLCompiler.
The internal traversal system now implements get_children(),
_copy_internals(), compare() and _cache_key() for most Core elements.
Core elements with special needs like Select still implement
some of these methods directly however most of these methods
are no longer explicitly implemented.
The data-driven system is also applied to ORM elements that
take part in SQL expressions so that these objects, like mappers,
aliasedclass, query options, etc. can all participate in the
cache key process.
Still not considered is that this approach to defining traversibility
will be used to create some kind of generic introspection system
that works across Core / ORM. It's also not clear if
real statement caching using the _cache_key() method is feasible,
if it is shown that running _cache_key() is nearly as expensive as
compiling in any case. Because it is data driven, it is more
straightforward to optimize using inlined code, as is the case now,
as well as potentially using C code to speed it up.
In addition, the caching sytem now accommodates for anonymous
name labels, which is essential so that constructs which have
anonymous labels can be cacheable, that is, their position
within a statement in relation to other anonymous names causes
them to generate an integer counter relative to that construct
which will be the same every time. Gathering of bound parameters
from any cache key generation is also now required as there is
no use case for a cache key that does not extract bound parameter
values.
Applies-to: #4639
Change-Id: I0660584def8627cad566719ee98d3be045db4b8d
The maximum buffer size for the :class:`.BufferedRowResultProxy`, which
is used by dialects such as PostgreSQL when ``stream_results=True``, can
now be set to a number greater than 1000 and the buffer will grow to
that size. Previously, the buffer would not go beyond 1000 even if the
value were set larger. The growth of the buffer is also now based
on a simple multiplying factor currently set to 5. Pull request courtesy
Soumaya Mauthoor.
Fixes: #4914Closes: #4930
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4930
Pull-request-sha: 66841f56e9
Change-Id: I6286220bd9d488027fadc444039421a410e19a19
Changed the ``repr()`` of the :class:`.quoted_name` construct to use
regular string repr() under Python 3, rather than running it through
"backslashreplace" escaping, which can be misleading.
Modified the approach of "name normalization" for the Oracle and Firebird
dialects, which converts from the UPPERCASE-as-case-insensitive convention
of these dialects into lowercase-as-case-insensitive for SQLAlchemy, to not
automatically apply the :class:`.quoted_name` construct to a name that
matches itself under upper or lower case conversion, as is the case for
many non-european characters. All names used within metadata structures
are converted to :class:`.quoted_name` objects in any case; the change
here would only affect the output of some inspection functions.
Moved name normalize to be under default dialect, added test coverage
in test/sql/test_quote.py
Fixes: #4931
Change-Id: Ic121b20e07249824710a54423e321d94a425362f
Add factilities to implement pytest.mark.parametrize and
pytest.fixtures patterns, which largely resemble things we are
already doing.
Ensure a facade is used, so that the test suite remains independent
of py.test, but also tailors the functions to the more limited
scope in which we are using them.
Additionally, create a class-based version that works from the
same facade.
Several old polymorphic tests as well as two of the sql test
are refactored to use the new features.
Change-Id: I6ef8af1dafff92534313016944d447f9439856cf
References: #4896
Fixed bug where a table that would have a column label overlap with a plain
column name, such as "foo.id AS foo_id" vs. "foo.foo_id", would prematurely
generate the ``._label`` attribute for a column before this overlap could
be detected due to the use of the ``index=True`` or ``unique=True`` flag on
the column in conjunction with the default naming convention of
``"column_0_label"``. This would then lead to failures when ``._label``
were used later to generate a bound parameter name, in particular those
used by the ORM when generating the WHERE clause for an UPDATE statement.
The issue has been fixed by using an alternate ``._label`` accessor for DDL
generation that does not affect the state of the :class:`.Column`. The
accessor also bypasses the key-deduplication step as it is not necessary
for DDL, the naming is now consistently ``"<tablename>_<columnname>"``
without any subsequent numeric symbols when used in DDL.
Fixes: #4911
Change-Id: Iabf5fd3250738d800d6e41a2a3a27a7ce2405e7d
Alembic needs a portable way of getting at the name of an
index without quoting being applied. As we would like the
indexes created by the Column index=True flag to support
deferred index names, supply a function that delivers this
for Alembic without it having to dig too deeply into the
internals. the _alembic_quote flag may be made public
at a later time, however as we've been through many quoting
flags that are difficult to get rid of, try to be conservative
to start.
Change-Id: I184adaeae26c2e75093aaea5ebe01a3815cadb08
The :class:`.Join` construct no longer considers the "onclause" as a source
of additional FROM objects to be omitted from the FROM list of an enclosing
:class:`.Select` object as standalone FROM objects. This applies to an ON
clause that includes a reference to another FROM object outside the JOIN;
while this is usually not correct from a SQL perspective, it's also
incorrect for it to be omitted, and the behavioral change makes the
:class:`.Select` / :class:`.Join` behave a bit more intuitively.
Fixes: #4621
Change-Id: Iaa1e75b7c59b21e9701ab3c9b69e66930feaf8ee
In order for text(), custom compiled objects, etc. to be usable
by Query(), they are all targeted by object key in the result map.
As we no longer want Query to implicitly label these, as well as that
text() has no label feature, support adding entries to the result
map that have no name, key, or type, only the object itself, and
then ensure that the compiler sets up for positional targeting
when this condition is detected.
Allows for more flexible ORM query usage with custom expressions
and text() while having less special logic in query itself.
Fixes: #4887
Change-Id: Ie073da127d292d43cb132a2b31bc90af88bfe2fd
Mike Bayer