Fixed bug where :meth:`.TypeEngine.column_expression` method would not be
applied to subsequent SELECT statements inside of a UNION or other
:class:`.CompoundSelect`, even though the SELECT statements are rendered at
the topmost level of the statement. New logic now differentiates between
rendering the column expression, which is needed for all SELECTs in the
list, vs. gathering the returned data type for the result row, which is
needed only for the first SELECT.
Fixes: #4787
Change-Id: Iceb63e430e76d2365649aa25ead09c4e2a062e10
Fixed issue where :class:`.Index` object which contained a mixture of
functional expressions which were not resolvable to a particular column,
in combination with string-based column names, would fail to initialize
its internal state correctly leading to failures during DDL compilation.
Fixes: #4778
Change-Id: I0fa9c627a1fde92ba8b9ed10af167c156012bd5d
Added support for composite (tuple) IN operators with SQLite, by rendering
the VALUES keyword for this backend. As other backends such as DB2 are
known to use the same syntax, the syntax is enabled in the base compiler
using a dialect-level flag ``tuple_in_values``. The change also includes
support for "empty IN tuple" expressions for SQLite when using "in_()"
between a tuple value and an empty set.
Fixes: #4766
Change-Id: I416e1af29b31d78f9ae06ec3c3a48ef6d6e813f5
Fixed issue where the :class:`.array_agg` construct in combination with
:meth:`.FunctionElement.filter` would not produce the correct operator
precedence between the FILTER keyword and the array index operator.
Fixes: #4760
Change-Id: Ic662cd3da3330554ec673bafd80495b3f1506098
The :func:`.select` construct and related constructs now allow for
duplication of column labels and columns themselves in the columns clause,
mirroring exactly how column expressions were passed in. This allows
the tuples returned by an executed result to match what was SELECTed
for in the first place, which is how the ORM :class:`.Query` works, so
this establishes better cross-compatibility between the two constructs.
Additionally, it allows column-positioning-sensitive structures such as
UNIONs (i.e. :class:`.CompoundSelect`) to be more intuitively constructed
in those cases where a particular column might appear in more than one
place. To support this change, the :class:`.ColumnCollection` has been
revised to support duplicate columns as well as to allow integer index
access.
Fixes: #4753
Change-Id: Ie09a8116f05c367995c1e43623c51e07971d3bf0
As part of the SQLAlchemy 2.0 migration project, a conceptual change has
been made to the role of the :class:`.SelectBase` class hierarchy,
which is the root of all "SELECT" statement constructs, in that they no
longer serve directly as FROM clauses, that is, they no longer subclass
:class:`.FromClause`. For end users, the change mostly means that any
placement of a :func:`.select` construct in the FROM clause of another
:func:`.select` requires first that it be wrapped in a subquery first,
which historically is through the use of the :meth:`.SelectBase.alias`
method, and is now also available through the use of
:meth:`.SelectBase.subquery`. This was usually a requirement in any
case since several databases don't accept unnamed SELECT subqueries
in their FROM clause in any case.
See the documentation in this change for lots more detail.
Fixes: #4617
Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
Fixed an unlikely issue where the "corresponding column" routine for unions
and other :class:`.CompoundSelect` objects could return the wrong column in
some overlapping column situtations, thus potentially impacting some ORM
operations when set operations are in use, if the underlying
:func:`.select` constructs were used previously in other similar kinds of
routines, due to a cached value not being cleared.
Fixes: #4747
Change-Id: I7fb134cac3604f8fe62e220fb24a0945d0a1c56f
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient. test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.
Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
Fixed a series of quoting issues which all stemmed from the concept of the
:func:`.literal_column` construct, which when being "proxied" through a
subquery to be referred towards by a label that matches its text, the label
would not have quoting rules applied to it, even if the string in the
:class:`.Label` were set up as a :class:`.quoted_name` construct. Not
applying quoting to the text of the :class:`.Label` is a bug because this
text is strictly a SQL identifier name and not a SQL expression, and the
string should not have quotes embedded into it already unlike the
:func:`.literal_column` which it may be applied towards. The existing
behavior of a non-labeled :func:`.literal_column` being propagated as is on
the outside of a subquery is maintained in order to help with manual
quoting schemes, although it's not clear if valid SQL can be generated for
such a construct in any case.
Fixes: #4730
Change-Id: I300941f27872fc4298c74a1d1ed65aef1a5cdd82
The Alias object no longer has "element" and "original", it now
has "wrapped" and "element" (the name .original is also left
as a descriptor for legacy access by third party dialects).
These two data members refer to the
dual roles Alias needs to play, where in the Python sense it needs
to refer to the thing it was applied against directly, whereas in the
SQL sense it needs to refer to the ultimate "non-alias" thing it
refers towards. Both are necessary to maintain. However, the change
here has each Alias object access the non-Alias object immediately
so that the "unwrapping" is simpler and does not need any special
logic.
In the SQL sense, Alias objects don't nest, the only potential
was that of the CTE, however there is no such thing as
a nested CTE, see link below.
This change is an interim change along the way to breaking Alias
into more classes and breaking away Select objects from being
FromClause objects.
Change-Id: Ie7a0d064226cb074ca745505129b5ec7d879e389
References: https://stackoverflow.com/questions/1413516/can-you-create-nested-with-clauses-for-common-table-expressions
as SELECT statements will have subquery() and not alias(),
start getting ready for the places where the ORM coerces SELECTs
into subqueries and be ready to warn about it
Change-Id: I90d4b6cae2c72816c6b192016ce074589caf4731
A major refactoring of all the functions handle all detection of
Core argument types as well as perform coercions into a new class hierarchy
based on "roles", each of which identify a syntactical location within a
SQL statement. In contrast to the ClauseElement hierarchy that identifies
"what" each object is syntactically, the SQLRole hierarchy identifies
the "where does it go" of each object syntactically. From this we define
a consistent type checking and coercion system that establishes well
defined behviors.
This is a breakout of the patch that is reorganizing select()
constructs to no longer be in the FromClause hierarchy.
Also includes a rename of as_scalar() into scalar_subquery(); deprecates
automatic coercion to scalar_subquery().
Partially-fixes: #4617
Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
Fixed that the :class:`.GenericFunction` class was inadvertently
registering itself as one of the named functions. Pull request courtesy
Adrien Berchet.
Fixes: #4653Closes: #4654
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4654
Pull-request-sha: 1112b89f0d
Change-Id: Ia0d366d3bff44a763aa496287814278dff732a19
Registered function names based on :class:`.GenericFunction` are now
retrieved in a case-insensitive fashion in all cases, removing the
deprecation logic from 1.3 which temporarily allowed multiple
:class:`.GenericFunction` objects to exist with differing cases. A
:class:`.GenericFunction` that replaces another on the same name whether or
not it's case sensitive emits a warning before replacing the object.
Fixes: #4649
Change-Id: I265ae19833132db07ed5b5ae40c4d24f659b1ab3
The :class:`.GenericFunction` namespace is being migrated so that function
names are looked up in a case-insensitive manner, as SQL functions do not
collide on case sensitive differences nor is this something which would
occur with user-defined functions or stored procedures. Lookups for
functions declared with :class:`.GenericFunction` now use a case
insensitive scheme, however a deprecation case is supported which allows
two or more :class:`.GenericFunction` objects with the same name of
different cases to exist, which will cause case sensitive lookups to occur
for that particular name, while emitting a warning at function registration
time. Thanks to Adrien Berchet for a lot of work on this complicated
feature.
Fixes: #4569Closes: #4570
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4570
Pull-request-sha: 37d4f3322b
Change-Id: Ief07c6eb55bf398f6aad85b60ef13ee6d1173109
This leverages the work started in #4336 to allow ClauseElement
structures to be cachable based on structure, not just identity.
Change-Id: Ia99ddeb5353496dd7d61243245685f02b98d8100
Reworked the :meth:`.ClauseElement.compare` methods in terms of a new
visitor-based approach, and additionally added test coverage ensuring that
all :class:`.ClauseElement` subclasses can be accurately compared
against each other in terms of structure. Structural comparison
capability is used to a small degree within the ORM currently, however
it also may form the basis for new caching features.
Fixes: #4336
Change-Id: I581b667d8e1642a6c27165cc9f4aded1c66effc6
Fixed issue where double negation of a boolean column wouldn't reset
the "NOT" operator.
Fixes: #4618
Change-Id: Ica280a0d6b5b0870aa2d05c4d059a1e559e6b12a
(cherry picked from commit 18f25f50353d9736e6638266585b2cb3ef7b0ea4)
Apparently the BIND_PARAMS regex passes over double colons,
it just doesn't accommodate for a bound parameter in that case.
add this use case to current tests as people can be relying upon it.
Change-Id: I6555621b1bb05d09b17428f4b4094ff7b219b460
Fixed bug where use of :func:`.with_polymorphic` or other aliased construct
would not properly adapt when the aliased target were used as the
:meth:`.Select.correlate_except` target of a subquery used inside of a
:func:`.column_property`. This required a fix to the clause adaption
mechanics to properly handle a selectable that shows up in the "correlate
except" list, in a similar manner as which occurs for selectables that show
up in the "correlate" list. This is ultimately a fairly fundamental bug
that has lasted for a long time but it is hard to come across it.
Fixes: #4537
Change-Id: Ibb97d4eea18b3c452aad519dd14919bfb84d422f
The :class:`.Alias` class and related subclasses :class:`.CTE`,
:class:`.Lateral` and :class:`.TableSample` have been reworked so that it is
not possible for a user to construct the objects directly. These constructs
require that the standalone construction function or selectable-bound method
be used to instantiate new objects.
Fixes: #4509
Change-Id: I74ae4786cb3ae625dab33b00bfd6bdc4e1219139
Revised the formatting for :class:`.StatementError` when stringified. Each
error detail is broken up over multiple newlines instead of spaced out on a
single line. Additionally, the SQL representation now stringifies the SQL
statement rather than using ``repr()``, so that newlines are rendered as is.
Pull request courtesy Nate Clark.
Fixes: #4500Closes: #4501
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4501
Pull-request-sha: 60cc0ee68d
Change-Id: I79d8418b7495e5691c9a56f41e79495c26a967ff
Fixed issue where the :class:`.JSON` type had a read-only
:attr:`.JSON.should_evaluate_none` attribute, which would cause failures
when making use of the :meth:`.TypeEngine.evaluates_none` method in
conjunction with this type. Pull request courtesy Sanjana S.
Fixes: #4485Closes: #4496
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4496
Pull-request-sha: 044beb2398
Change-Id: I1f3e1d7dec9d2ceb6ccaaa8cac158a062cf02710
A SQL expression can now be assigned to a primary key attribute for an ORM
flush in the same manner as ordinary attributes as described in
:ref:`flush_embedded_sql_expressions` where the expression will be evaulated
and then returned to the ORM using RETURNING, or in the case of pysqlite,
works using the cursor.lastrowid attribute.Requires either a database that
supports RETURNING (e.g. Postgresql, Oracle, SQL Server) or pysqlite.
Fixes: #3133Fixes: #4494
Change-Id: I83da8357354de002cb04fa4a553f2a2f90c5157d
Fully removed the behavior of strings passed directly as components of a
:func:`.select` or :class:`.Query` object being coerced to :func:`.text`
constructs automatically; the warning that has been emitted is now an
ArgumentError or in the case of order_by() / group_by() a CompileError.
This has emitted a warning since version 1.0 however its presence continues
to create concerns for the potential of mis-use of this behavior.
Note that public CVEs have been posted for order_by() / group_by() which
are resolved by this commit: CVE-2019-7164 CVE-2019-7548
Added "SQL phrase validation" to key DDL phrases that are accepted as plain
strings, including :paramref:`.ForeignKeyConstraint.on_delete`,
:paramref:`.ForeignKeyConstraint.on_update`,
:paramref:`.ExcludeConstraint.using`,
:paramref:`.ForeignKeyConstraint.initially`, for areas where a series of SQL
keywords only are expected.Any non-space characters that suggest the phrase
would need to be quoted will raise a :class:`.CompileError`. This change
is related to the series of changes committed as part of 🎫`4481`.
Fixed issue where using an uppercase name for an index type (e.g. GIST,
BTREE, etc. ) or an EXCLUDE constraint would treat it as an identifier to
be quoted, rather than rendering it as is. The new behavior converts these
types to lowercase and ensures they contain only valid SQL characters.
Quoting is applied to :class:`.Function` names, those which are usually but
not necessarily generated from the :attr:`.sql.func` construct, at compile
time if they contain illegal characters, such as spaces or punctuation. The
names are as before treated as case insensitive however, meaning if the
names contain uppercase or mixed case characters, that alone does not
trigger quoting. The case insensitivity is currently maintained for
backwards compatibility.
Fixes: #4481Fixes: #4473Fixes: #4467
Change-Id: Ib22a27d62930e24702e2f0f7c74a0473385a08eb
This affects mostly docstrings, except in orm/events.py::dispose_collection()
where one parameter gets renamed: given that the method is
empty, it seemed reasonable to me to fix that too.
Closes: #4440
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4440
Pull-request-sha: 779ed75acb
Change-Id: Ic0553fe97853054b09c2453af76d96363de6eb0e
The deprecations review didn't include tests of identifier_preparer.quote.force
for backends, so MSSQL slipped through. We have to fully reimplement
the deprecation warning here so that it passes tests which are now
enabled for all backends.
Change-Id: I9d07e6766e16b5a35b7f7566f1daf94b04346270
Added accessors for execution options to Core and ORM, via
:meth:`.Query.get_execution_options`,
:meth:`.Connection.get_execution_options`,
:meth:`.Engine.get_execution_options`, and
:meth:`.Executable.get_execution_options`. PR courtesy Daniel Lister.
Fixes: #4406Closes: #4465
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4465
Pull-request-sha: 9674688bb5
Change-Id: I93ba51d7a2d687e255edd6938db15615e56dd237
A large change throughout the library has ensured that all objects, parameters,
and behaviors which have been noted as deprecated or legacy now emit
``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now
defaults to displaying deprecation warnings, as well as that modern test suites
based on tools like tox and pytest tend to display deprecation warnings,
this change should make it easier to note what API features are obsolete.
See the notes added to the changelog and migration notes for further
details.
Fixes: #4393
Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Fixes to the test suite, a few errant imports, and setup.py:
- mysql and postgresql have unused 'json' imports; remove
- postgresql is exporting the 'json' symbol, remove
- make sure setup.py can find __version__ using " or '
- retry logic in provision create database for postgresql fixed
- refactor test_magazine to use cls.tables rather than globals
- remove unused class in test_scoping
- add a comment to test_deprecations that this test suite itself
is deprecated
- don't use mapper() and orm_mapper() in test_unitofwork, just
use mapper()
- remove dupe test_scalar_set_None test in test_attributes
- Python 2.7 and above includes unittest.SkipTest, remove pre-2.7
fallback
- use imported SkipTest in profiling
- declarative test_reflection tests with "reflectable_autoincrement"
already don't run on oracle or firebird; remove conditional logic
for these, which also removes an "id" symbol
- clean up test in test_functions, remove print statement
- remove dupe test_literal_processor_coercion_native_int_out_of_range
in test/sql/test_types.py
- fix psycopg2_hstore ref
Change-Id: I7b3444f8546aac82be81cd1e7b6d8b2ad6834fe6
a few code changes ahead of time to handle some __all__
issues better. also include new flake8 rules, since the
existing flake8 doesn't pass in any case.
Change-Id: I1efdf75124ae7bcac719c22e505bb5b13db06c04
Fixed issue in "expanding IN" feature where using the same bound parameter
name more than once in a query would lead to a KeyError within the process
of rewriting the parameters in the query.
Fixes: #4394
Change-Id: Ibcadce9fefbcb060266d9447c2044ee6efeccf5a
test_compiler is mostly related to SELECT statements as well
as smaller SQL elements. While it still has some DDL related
tests, move out all the remaining insert/update tests into
the already present test_insert.py, test_update.py
Fixes: #2630
Change-Id: I4167618543fd1235d12d1717c8c629d2374b325a
Amended the :class:`.AnsiFunction` class, the base of common SQL
functions like ``CURRENT_TIMESTAMP``, to accept positional arguments
like a regular ad-hoc function. This to suit the case that many of
these functions on specific backends accept arguments such as
"fractional seconds" precision and such. If the function is created
with arguments, it renders the the parenthesis and the arguments. If
no arguents are present, the compiler generates the non-parenthesized form.
Fixes: #4386
Change-Id: Ic492ef177e4987cec99ec4d95f55292be8daa087
Added :class:`.Sequence` to the "string SQL" system that will render a
meaningful string expression (``"<next sequence value: my_sequence>"``)
when stringifying without a dialect a statement that includes a "sequence
nextvalue" expression, rather than raising a compilation error.
Fixes: #4144
Change-Id: Ia910f0e22008a7cde7597365954ede324101cf4d
Added new naming convention tokens ``column_0N_name``, ``column_0_N_name``,
etc., which will render the names / keys / labels for all columns referenced
by a particular constraint in a sequence. In order to accommodate for the
length of such a naming convention, the SQL compiler's auto-truncation
feature now applies itself to constraint names as well, which creates a
shortened, deterministically generated name for the constraint that will
apply to a target backend without going over the character limit of that
backend.
Additional notes:
1. the SQLite dialect had a format_index method that was apparently not
used, removed.
2. the naming convention logic has been applying the foreign key
remote column spec to the naming convention, and not the actual
column name. In the case where the referenced Table object uses
.key inside the columns and these are what ForeignKey() references,
the naming convention was doing the wrong thing. The patch here
fixes this, however this isn't noted in the migration notes.
Fixes: #3989
Change-Id: Ib24f4754b886676096c480fc54b2e5c2463ac99a
this test was using sysdate() and current_timestamp() together
in conjunction with a truncation to DAY, however for four hours
on saturday night (see commit time :) ) these two values will
have a different value if one side is EDT and the other is UTC.
tox does not transmit environment variables including TZ by
default, so even if the server is set up for EDT, running tox
will not set TZ and at least Oracle client seems to use this
value, producing UTC for session time but the database on CI
was configured for EDT, producing EDT for sysdate.
Change-Id: I56602d2402a475a0c4fdf61c1c5fc2618c82f915
Fixed additional warnings generated by Python 3.7 due to changes in the
organization of the Python ``collections`` and ``collections.abc`` packages.
Previous ``collections`` warnings were fixed in version 1.2.11. Pull request
courtesy xtreak.
See I2d1c0ef97c8ecac7af152cc56263422a40faa6bb for the original collections.abc
fixes.
Fixes: #4339
Change-Id: Ia92d2461f20309fb33ea6c6f592f7d4e7e32ae7a
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/475
Fixed bug where the :paramref:`.Enum.create_constraint` flag on the
:class:`.Enum` datatype would not be propagated to copies of the type, which
affects use cases such as declarative mixins and abstract bases.
Fixes: #4341
Change-Id: I978be65f33a616fe4d5f5de03fb3eaab6f6a2272
Mike Bayer