Adjusted the "column disambiguation" logic that's new in 1.4, where the
same expression repeated gets an "extra anonymous" label, so that the logic
more aggressively deduplicates those labels when the repeated element
is the same Python expression object each time, as occurs in cases like
when using "singleton" values like :func:`_sql.null`. This is based on
the observation that at least some databases (e.g. MySQL, but not SQLite)
will raise an error if the same label is repeated inside of a subquery.
Related to 🎫`7153`, fixed an issue where result column lookups
would fail for "adapted" SELECT statements that selected for
"constant" value expressions most typically the NULL expression,
as would occur in such places as joined eager loading in conjunction
with limit/offset. This was overall a regression due to issue
🎫`6259` which removed all "adaption" for constants like NULL,
"true", and "false", but this broke the case where the same adaption
logic were used to match the constant to a labeled expression referring
to the constant in a subquery.
Fixes: #7153Fixes: #7154
Change-Id: I43823343721b9e70524ea3f5e8f39dd543a3e92b
Fixed an inconsistency in the any_() / all_() functions / methods where the
special behavior these functions have of "flipping" the expression such
that the "ANY" / "ALL" expression is always on the right side would not
function if the comparison were against the None value, that is,
"column.any_() == None" should produce the same SQL expression as "null()
== column.any_()". Added more docs to clarify this as well, plus mentions
that any_() / all_() generally supersede the ARRAY version "any()" /
"all()".
Fixes: #7140
Change-Id: Ia5d55414ba40eb3fbda3598931fdd24c9b4a4411
Fixed issue where using ORM column expressions as keys in the list of
dictionaries passed to :meth:`_sql.Insert.values` for "multi-valued insert"
would not be processed correctly into the correct column expressions.
Fixes: #7060
Change-Id: I1c4c286c33ea6eeaafba617996828f5c88ff0a1c
Fixed a two issues where combinations of ``select()`` and ``join()`` when
adapted to form a copy of the element would not completely copy the state
of all column objects associated with subqueries. A key problem this caused
is that usage of the :meth:`_sql.ClauseElement.params` method (which should
probably be moved into a legacy category as it is inefficient and error
prone) would leave copies of the old :class:`_sql.BindParameter` objects
around, leading to issues in correctly setting the parameters at execution
time.
Fixes: #7055
Change-Id: Ib822a978a99561b4402da3fb727b370f5c58210b
Added new parameter :meth:`_sql.HasCte.cte.nesting` to the
:class:`_sql.CTE` constructor and :meth:`_sql.HasCTE.cte` method, which
flags the CTE as one which should remain nested within an enclosing CTE,
rather than being moved to the top level of the outermost SELECT. While in
the vast majority of cases there is no difference in SQL functionality,
users have identified various edge-cases where true nesting of CTE
constructs is desirable. Much thanks to Eric Masseran for lots of work on
this intricate feature.
Fixes: #4123Closes: #6709
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/6709
Pull-request-sha: 64ab2f6ea2
Change-Id: Ic4dc25ab763af96d96632369e01527d48a654149
Fixed issue related to new ``add_cte()`` feature where pairing two
"INSERT..FROM SELECT" statements simultaneously would lose track of the two
independent SELECT statements, leading to the wrong SQL.
Fixes: #7036
Change-Id: I90fe47eb203bc5c1ea5810db0edba08250b2b7e6
Fixed issue where an engine that had ``implicit_returning`` set to False
would fail to function when PostgreSQL's "fast insertmany" feature were
used in conjunction with a ``Sequence``, as well as if any kind of
"executemany" with "return_defaults()" were used in conjunction with a
``Sequence``. Note that PostgreSQL "fast insertmany" uses "RETURNING" by
definition, when the SQL statement is passed to the driver; overall, the
``implicit_returning`` flag is legacy and has no real use in modern
SQLAlchemy, and will be deprecated in a separate change.
Fixes: #6963
Change-Id: Id8e3dd50a21b9124f338067b0fdb57b8f608dca8
Fixed issue in lambda caching system where an element of a query that
produces no cache key, like a custom option or clause element, would still
populate the expression in the "lambda cache" inappropriately.
This was discovered as part of 🎫`6887` but is a separate
issue.
References: #6887
Change-Id: I1665f4320254ddc63a0abf3088e9daeaffbd1840
Adjusted the "from linter" warning feature to accommodate for a chain of
joins more than one level deep where the ON clauses don't explicitly match
up the targets, such as an expression such as "ON TRUE". This mode of use
is intended to cancel the cartesian product warning simply by the fact that
there's a JOIN from "a to b", which was not working for the case where the
chain of joins had more than one element.
this incurs a bit more compiler overhead that comes out in profiling
but is not extensive.
Added the "is_comparison" flag to the PostgreSQL "overlaps",
"contained_by", "contains" operators, so that they work in relevant ORM
contexts as well as in conjunction with the "from linter" feature.
Fixes: #6886
Change-Id: I078dc3fe6d4f7871ffe4ebac3e71e62f3f213d12
Fix issue in :class:`_sql.CTE` where new :meth:`_sql.HasCTE.add_cte` method
added in version 1.4.21 / 🎫`6752` failed to function correctly for
"compound select" structures such as :func:`_sql.union`,
:func:`_sql.union_all`, :func:`_sql.except`, etc. Pull request courtesy
Eric Masseran.
Fixes: #6752Closes: #6849
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/6849
Pull-request-sha: 1c4b4d72b2
Change-Id: I49a16a4fc2af8299502011f3a02d8a2ad93255e3
Added new attribute :attr:`_sql.Select.columns_clause_froms` that will
retrieve the FROM list implied by the columns clause of the
:class:`_sql.Select` statement. This differs from the old
:attr:`_sql.Select.froms` collection in that it does not perform any ORM
compilation steps, which necessarily deannotate the FROM elements and do
things like compute joinedloads etc., which makes it not an appropriate
candidate for the :meth:`_sql.Select.select_from` method. Additionally adds
a new parameter
:paramref:`_sql.Select.with_only_columns.maintain_column_froms` that
transfers this collection to :meth:`_sql.Select.select_from` before
replacing the columns collection.
In addition, the :attr:`_sql.Select.froms` is renamed to
:meth:`_sql.Select.get_final_froms`, to stress that this collection is not
a simple accessor and is instead calculated given the full state of the
object, which can be an expensive call when used in an ORM context.
Additionally fixes a regression involving the
:func:`_orm.with_only_columns` function to support applying criteria to
column elements that were replaced with either
:meth:`_sql.Select.with_only_columns` or :meth:`_orm.Query.with_entities` ,
which had broken as part of 🎫`6503` released in 1.4.19.
Fixes: #6808
Change-Id: Ib5d66cce488bbaca06dab4f68fb5cdaa73e8823e
this appears to be unnecessary and prevents end-user
literal_binds case from working.
Fixed issue where the ``literal_binds`` compiler flag, as used externally
to render bound parameters inline, would fail to work when used with a
certain class of parameters known as "literal_execute", which covers things
like LIMIT and OFFSET values for dialects where the drivers don't allow a
bound parameter, such as SQL Server's "TOP" clause. The issue locally
seemed to affect only the MSSQL dialect.
Fixes: #6863
Change-Id: Ia74cff5b0107b129a11b9b965883552b2962e449
Fixed an issue in the ``CacheKey.to_offline_string()`` method used by the
dogpile.caching example where attempting to create a proper cache key from
the special "lambda" query generated by the lazy loader would fail to
include the parameter values, leading to an incorrect cache key.
Fixes: #6858
Change-Id: Ice27087583c6f3ff79cf7d5b879e5dd0a4e58158
Fixed issue where a bound parameter object that was "cloned" would cause a
name conflict in the compiler, if more than one clone of this parameter
were used at the same time in a single statement. This could occur in
particular with things like ORM single table inheritance queries that
indicated the same "discriminator" value multiple times in one query.
Fixes: #6824
Change-Id: Iba7a786fc5a2341ff7d07fc666d24ed790ad4fe8
Fixed issue where the unit of work would internally use a 2.0-deprecated
SQL expression form, emitting a deprecation warning when SQLALCHEMY_WARN_20
were enabled.
Fixes: #6812
Change-Id: I0a031e728527a1c3382848b6ddc793939362b128
Fixed issue where use of the :paramref:`_sql.case.whens` parameter passing
a dictionary positionally and not as a keyword argument would emit a 2.0
deprecation warning, referring to the deprecation of passing a list
positionally. The dictionary format of "whens", passed positionally, is
still supported and was accidentally marked as deprecated.
Removes warning filter for case statement.
Fixes: #6786
Change-Id: I8efd1882563773bec89ae5e34f0dfede77fc4683
Fixed critical caching issue where the ORM's persistence feature using
INSERT..RETURNING would cache an incorrect query when mixing the "bulk
save" and standard "flush" forms of INSERT.
Fixes: #6793
Change-Id: Ifeb61c1226d3fa6d5e1c2e29b6f5ff77a27d6a2d
Fixed issue in new :meth:`_schema.Table.table_valued` method where the
resulting :class:`_sql.TableValuedColumn` construct would not respond
correctly to alias adaptation as is used throughout the ORM, such as for
eager loading, polymorphic loading, etc.
Fixes: #6775
Change-Id: I77cec4b6e1b1003f2b6be242b54ada8e4a435250
Fixed issue where type-specific bound parameter handlers would not be
called upon in the case of using the :meth:`_sql.Insert.values` method with
the Python ``None`` value; in particular, this would be noticed when using
the :class:`_types.JSON` datatype as well as related PostgreSQL specific
types such as :class:`_postgresql.JSONB` which would fail to encode the
Python ``None`` value into JSON null, however the issue was generalized to
any bound parameter handler in conjunction with this specific method of
:class:`_sql.Insert`.
The issue with coercions forcing out ``null()`` may still impact
SQL expression usage as well; the change here is limited to crud
as the behavior there is relevant to some use cases, which may
need to be evaluated separately.
Fixes: #6770
Change-Id: If53edad811b37dada7578a89daf395628db058a6
Fixed regression which appeared in version 1.4.3 due to 🎫`6060`
where rules that limit ORM adaptation of derived selectables interfered
with other ORM-adaptation based cases, in this case when applying
adaptations for a :func:`_orm.with_polymorphic` against a mapping which
uses a :func:`_orm.column_property` which in turn makes use of a scalar
select that includes a :func:`_orm.aliased` object of the mapped table.
Fixes: #6762
Change-Id: Ice3dc34b97d12b59f044bdc0c5faaefcc4015227
Fixed issue in CTE constructs where a recursive CTE that referred to a
SELECT that has duplicate column names, which are typically deduplicated
using labeling logic in 1.4, would fail to refer to the deduplicated label
name correctly within the WITH clause.
As part of this change we are also attempting to remove the
behavior of SelectStatementGrouping forcing off the "asfrom"
contextual flag, which will have the result of additional labeling
being applied to some UNION and similar statements when they are
interpreted as subqueries. To maintain compatibility with
"grouping", the Grouping/SelectStatementGrouping are now broken
out into two separate compiler cases, as the "asfrom" logic appears
to be tailored towards table valued SELECTS as column expressions.
Fixes: #6710
Change-Id: I8af07a5c670dbe5736cd9f16084ef82f5e4c8642
To service #6718 and #6710, the system by which columns are
given labels in a SELECT statement as well as the system that
gives them keys in a .c or .selected_columns collection have
been refactored to provide a single source of truth for
both, in constrast to the previous approach that included
similar logic repeated in slightly different ways.
Main ideas:
1. ColumnElement attributes ._label, ._anon_label, ._key_label
are renamed to include the letters "tq", meaning
"table-qualified" - these labels are only used when rendering
a SELECT that has LABEL_STYLE_TABLENAME_PLUS_COL for its
label style; as this label style is primarily legacy, the
"tq" names should be isolated so that in a 2.0 style application
these aren't being used at all
2. The means by which the "labels" and "proxy keys" for the elements
of a SELECT has been centralized to a single source of truth;
previously, the three of _generate_columns_plus_names,
_generate_fromclause_column_proxies, and _column_naming_convention
all had duplicated rules between them, as well as that there
were a little bit of labeling rules in compiler._label_select_column
as well; by this we mean that the various "anon_label" "anon_key"
methods on ColumnElement were called by all four of these methods,
where there were many cases where it was necessary that one method
comes up with the same answer as another of the methods. This
has all been centralized into _generate_columns_plus_names
for all the names except the "proxy key", which is generated
by _column_naming_convention.
3. compiler._label_select_column has been rewritten to both not make
any naming decisions nor any "proxy key" decisions, only whether
to label or not to label; the _generate_columns_plus_names method
gives it the information, where the proxy keys come from
_column_naming_convention; previously, these proxy keys were matched
based on restatement of similar (but not really the same) logic in
two places. The heuristics of "whether to label or not to label"
are also reorganized to be much easier to read and understand.
4. a new method compiler._label_returning_column is added for dialects
to use in their "generate returning columns" methods. A
github search reveals a small number of third party dialects also
doing this using the prior _label_select_column method so we
try to make sure _label_select_column continues to work the
exact same way for that specific use case; for the "SELECT" use
case it now needs
5. After some attempts to do it different ways, for the case where
_proxy_key is giving us some kind of anon label, we are hard
changing it to "_no_label" right now, as there's not currently
a way to fully match anonymized labels from stmt.c or
stmt.selected_columns to what will be in the result map. The
idea of "_no_label" is to encourage the user to use label('name')
for columns they want to be able to target by string name that
don't have a natural name.
Change-Id: I7a92a66f3a7e459ccf32587ac0a3c306650daf11
Added new method :meth:`_sql.HasCTE.add_cte` to each of the
:func:`_sql.select`, :func:`_sql.insert`, :func:`_sql.update` and
:func:`_sql.delete` constructs. This method will add the given
:class:`_sql.CTE` as an "independent" CTE of the statement, meaning it
renders in the WITH clause above the statement unconditionally even if it
is not otherwise referenced in the primary statement. This is a popular use
case on the PostgreSQL database where a CTE is used for a DML statement
that runs against database rows independently of the primary statement.
Fixes: #6752
Change-Id: Ibf635763e40269cbd10f4c17e208850d8e8d0188
Fixed regression where the :func:`_sql.tablesample` construct would fail to
be executable when constructed given a floating-point sampling value not
embedded within a SQL function.
Fixes: #6735
Change-Id: I557bcd4bdbffc4329ad69d5659ba99b1c8deb554
Fixed regression caused in 1.4.19 due to #6503 and related involving
:meth:`_orm.Query.with_entities` where the new structure used would be
inappropriately transferred to an enclosing :class:`_orm.Query` when making
use of set operations such as :meth:`_orm.Query.union`, causing the JOIN
instructions within to be applied to the outside query as well.
Fixes: #6698
Change-Id: Ia9f294215ebc01330d142a0a3e5be9d02be9380f
Fixed issue where passing ``None`` for the value of
:paramref:`_schema.Table.prefixes` would not store an empty list, but
rather the constant ``None``, which may be unexpected by third party
dialects. The issue is revealed by a usage in recent versions of Alembic
that are passing ``None`` for this value. Pull request courtesy Kai
Mueller.
Fixes: #6685Closes: #6672
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/6672
Pull-request-sha: b79aca0ee4
Change-Id: I758641c6fbde6f2607d074fecea7efa6728aeea0
Add a impl parameter to :class:`_types.PickleType` constructor, allowing
any arbitary type to be used in place of the default implementation of
:class:`_types.LargeBinary`. Pull request courtesy jason3gb.
Fixes: #6646Closes: #6657
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/6657
Pull-request-sha: e49bcd368d
Change-Id: Ib79f3b0ebbc94393f673f5a5ba6558260083d0cf
Fixed old issue where a :func:`_sql.select()` made against the token "*",
which then yielded exactly one column, would fail to correctly organize the
``cursor.description`` column name into the keys of the result object.
Fixes: #6665
Change-Id: Ie8c00f62998972ad4a19a750d2642d00fde006f6
Refined the behavior of ORM subquery rendering with regards to deferred
columns and column properties to be more compatible with that of 1.3 while
also providing for 1.4's newer features. As a subquery in 1.4 does not make
use of loader options, including :func:`_orm.deferred`, a subquery that is
against an ORM entity with deferred attributes will now render those
deferred attributes that refer directly to mapped table columns, as these
are needed in the outer SELECT if that outer SELECT makes use of these
columns; however a deferred attribute that refers to a composed SQL
expression as we normally do with :func:`_orm.column_property` will not be
part of the subquery, as these can be selected explicitly if needed in the
subquery. If the entity is being SELECTed from this subquery, the column
expression can still render on "the outside" in terms of the derived
subquery columns. This produces essentially the same behavior as when
working with 1.3. However in this case the fix has to also make sure that
the ``.selected_columns`` collection of an ORM-enabled :func:`_sql.select`
also follows these rules, which in particular allows recursive CTEs to
render correctly in this scenario, which were previously failing to render
correctly due to this issue.
As part of this change the _exported_columns_iterator() method has been
removed and logic simplified to use ._all_selected_columns from any
SelectBase object where _exported_columns_iterator() was used before.
Additionally sets up UpdateBase to include ReturnsRows in its hierarchy;
the literal point of ReturnsRows was to be a common base for UpdateBase
and SelectBase so it was kind of weird it wasn't there.
Fixes: #6661
Fixed issue in CTE constructs mostly relevant to ORM use cases where a
recursive CTE against "anonymous" labels such as those seen in ORM
``column_property()`` mappings would render in the
``WITH RECURSIVE xyz(...)`` section as their raw internal label and not a
cleanly anonymized name.
Fixes: #6663
Change-Id: I26219d4d8e6c0915b641426e9885540f74fae4d2
Fixed further regressions in the same area as that of 🎫`6052` where
loader options as well as invocations of methods like
:meth:`_orm.Query.join` would fail if the left side of the statement for
which the option/join depends upon were replaced by using the
:meth:`_orm.Query.with_entities` method, or when using 2.0 style queries
when using the :meth:`_sql.Select.with_only_columns` method. A new set of
state has been added to the objects which tracks the "left" entities that
the options / join were made against which is memoized when the lead
entities are changed.
Fixes: #6503Fixes: #6253
Change-Id: I211b2af98b0b20d1263fb15dc513884dcc5de6a4
Fixed issue where query production for joinedload against a complex left
hand side involving joined-table inheritance could fail to produce a
correct query, due to a clause adaption issue.
Fixes: #6595
Change-Id: Id4b839d52447cdc103b392dd8946c4cfa7a829e1
Fixed regression involving clause adaption of labeled ORM compound
elements, such as single-table inheritance discriminator expressions with
conditionals or CASE expressions, which could cause aliased expressions
such as those used in ORM join / joinedload operations to not be adapted
correctly, such as referring to the wrong table in the ON clause in a join.
This change also improves a performance bump that was located within the
process of invoking :meth:`_sql.Select.join` given an ORM attribute
as a target.
Fixes: #6550
Change-Id: I98906476f0cce6f41ea00b77c789baa818e9d167
Mike Bayer