Commit Graph

1863 Commits

Author SHA1 Message Date
mike bayer 5763c97780 Merge "apply basic escaping to anon_labels unconditionally" into rel_1_4 2022-10-28 13:59:16 +00:00
Mike Bayer 6d77db96f7 open some compound tests for mysql 8.0.31
Not sure of exact version but as we have done a major
rebuild of CI, newer mysql 8.0 is passing on these.

Change-Id: Ibcfe0ce519ab6a2941ca514b4254944769b60df4
(cherry picked from commit 50d3b85c69)
2022-10-28 02:11:34 -04:00
Mike Bayer d8910c442e apply basic escaping to anon_labels unconditionally
Fixed issue which prevented the :func:`_sql.literal_column` construct from
working properly within the context of a :class:`.Select` construct as well
as other potential places where "anonymized labels" might be generated, if
the literal expression contained characters which could interfere with
format strings, such as open parenthesis, due to an implementation detail
of the "anonymous label" structure.

Fixes: #8724
Change-Id: I3089124fbd055a011c8a245964258503b717d941
(cherry picked from commit caa9f0ff98)
2022-10-27 10:25:49 -04:00
Federico Caselli e3a71aadd7 aggregate_order_by now supports cache generation.
also adjusted CacheKeyFixture to be a general purpose
fixture so that sub-components / dialects can run
their own cache key tests.

Fixes: #8574
Change-Id: I6c66107856aee11e548d357cea77bceee3e316a0
(cherry picked from commit 7980b677085fc759a0406f6778b9729955f3c7f6)
2022-09-25 21:16:41 -04:00
Mike Bayer 4d6d402d73 remove should_nest behavior for contains_eager()
Fixed regression for 1.4 in :func:`_orm.contains_eager` where the "wrap in
subquery" logic of :func:`_orm.joinedload` would be inadvertently triggered
for use of the :func:`_orm.contains_eager` function with similar statements
(e.g. those that use ``distinct()``, ``limit()`` or ``offset()``). This is
not appropriate for :func:`_orm.contains_eager` which has always had the
contract that the user-defined SQL statement is unmodified with the
exception of adding the appropriate columns.

Also includes an adjustment to the assertion in Label._make_proxy()
which was there to prevent a fixed label name from being anonymized;
if the label is already anonymous, the change should proceed.
This logic was being hit before the contains_eager behavior was
adjusted. With the adjustment, this code is not used.

Fixes: #8569
Change-Id: I161e65041c0162fd2b83cbef40f57a50fcfaf0fd
(cherry picked from commit 57b400f079)
2022-09-23 17:18:37 -04:00
Mike Bayer bbd90a987e break out text() from TextualSelect for col matching
Fixed issue where mixing "*" with additional explicitly-named column
expressions within the columns clause of a :func:`_sql.select` construct
would cause result-column targeting to sometimes consider the label name or
other non-repeated names to be an ambiguous target.

Fixes: #8536
Change-Id: I3c845eaf571033e54c9208762344f67f4351ac3a
(cherry picked from commit 78327d98be9236c61f950526470f29b184dabba6)
2022-09-19 21:50:03 -04:00
Mike Bayer 12547f9633 include TableClause.schema in cache key
Fixed issue where use of the :func:`_sql.table` construct, passing a string
for the :paramref:`_sql.table.schema` parameter, would fail to take the
"schema" string into account when producing a cache key, thus leading to
caching collisions if multiple, same-named :func:`_sql.table` constructs
with different schemas were used.

Fixes: #8441
Change-Id: Ic4b55b3e8ec53b4c88ba112691bdf60ea1d4c448
(cherry picked from commit 613642d963)
2022-08-30 10:47:59 -04:00
Mike Bayer 9cd7cca2c7 fill out all distinguising fields for AliasedInsp
Hardened the cache key strategy for the :func:`_orm.aliased` and
:func:`_orm.with_polymorphic` constructs. While no issue involving actual
statements being cached can easily be demonstrated (if at all), these two
constructs were not including enough of what makes them unique in their
cache keys for caching on the aliased construct alone to be accurate.

Fixes: #8401
Change-Id: I13d14985b6965f398edd9494601d8ae89ac641f2
(cherry picked from commit a58f1b9c69)
2022-08-17 16:53:38 -04:00
Mike Bayer 7914b6491b deep compare CTEs before considering them conflicting
Fixed issue where referencing a CTE multiple times in conjunction with a
polymorphic SELECT could result in multiple "clones" of the same CTE being
constructed, which would then trigger these two CTEs as duplicates. To
resolve, the two CTEs are deep-compared when this occurs to ensure that
they are equivalent, then are treated as equivalent.

Fixes: #8357
Change-Id: I1f634a9cf7a6c4256912aac1a00506aecea3b0e2
(cherry picked from commit 85fa363c84)
2022-08-05 17:26:49 -04:00
Mike Bayer 81d20ae7f2 check for TypeDecorator when handling getitem
Fixed issue where :class:`.TypeDecorator` would not correctly proxy the
``__getitem__()`` operator when decorating the :class:`.ARRAY` datatype,
without explicit workarounds.

Fixes: #7249
Change-Id: I3273572b4757e41fb5952639cb867314227d368a
(cherry picked from commit 1e01fab7e600c53284eabceceab5706e4074eb2e)
2022-07-19 12:37:57 -04:00
Mike Bayer 0052827c44 use concat() directly for contains, startswith, endswith
Adjusted the SQL compilation for string containment functions
``.contains()``, ``.startswith()``, ``.endswith()`` to force the use of the
string concatenation operator, rather than relying upon the overload of the
addition operator, so that non-standard use of these operators with for
example bytestrings still produces string concatenation operators.

To accommodate this, needed to add a new _rconcat operator function,
which is private, as well as a fallback in concat_op() that works
similarly to Python builtin ops.

Fixes: #8253
Change-Id: I2b7f56492f765742d88cb2a7834ded6a2892bd7e
(cherry picked from commit 85a88df13a)
2022-07-17 11:35:19 -04:00
Mike Bayer 844365266a generalize sql server check for id col to accommodate ORM cases
Fixed issues that prevented the new usage patterns for using DML with ORM
objects presented at :ref:`orm_dml_returning_objects` from working
correctly with the SQL Server pyodbc dialect.

Here we add a step to look in compile_state._dict_values more thoroughly
for the keys we need to determine "identity insert" or not, and also
add a new compiler variable dml_compile_state so that we can skip the
ORM's compile_state if present.

Fixes: #8210
Change-Id: Idbd76bb3eb075c647dc6c1cb78f7315c821e15f7
(cherry picked from commit 5806428800d2f1ac775156f90497a2fc3a644f35)
2022-07-05 22:02:43 -04:00
Mike Bayer 06685f392d repair yield_per for non-SS dialects and add new options
Implemented new :paramref:`_engine.Connection.execution_options.yield_per`
execution option for :class:`_engine.Connection` in Core, to mirror that of
the same :ref:`yield_per <orm_queryguide_yield_per>` option available in
the ORM. The option sets both the
:paramref:`_engine.Connection.execution_options.stream_results` option at
the same time as invoking :meth:`_engine.Result.yield_per`, to provide the
most common streaming result configuration which also mirrors that of the
ORM use case in its usage pattern.

Fixed bug in :class:`_engine.Result` where the usage of a buffered result
strategy would not be used if the dialect in use did not support an
explicit "server side cursor" setting, when using
:paramref:`_engine.Connection.execution_options.stream_results`. This is in
error as DBAPIs such as that of SQLite and Oracle already use a
non-buffered result fetching scheme, which still benefits from usage of
partial result fetching.   The "buffered" strategy is now used in all
cases where :paramref:`_engine.Connection.execution_options.stream_results`
is set.

Added :meth:`.FilterResult.yield_per` so that result implementations
such as :class:`.MappingResult`, :class:`.ScalarResult` and
:class:`.AsyncResult` have access to this method.

Fixes: #8199
Change-Id: I6dde3cbe483a1bf81e945561b60f4b7d1c434750
(cherry picked from commit e5a0cdb2eaa1d7f381e93d0529a7f8e6d5888877)
2022-07-01 12:14:58 -04:00
Mike Bayer 49bd3b0046 restore parameter escaping for public methods
Adjusted the fix made for 🎫`8056` which adjusted the escaping of
bound parameter names with special characters such that the escaped names
were translated after the SQL compilation step, which broke a published
recipe on the FAQ illustrating how to merge parameter names into the string
output of a compiled SQL string. The change restores the escaped names that
come from ``compiled.params`` and adds a conditional parameter to
:meth:`.SQLCompiler.construct_params` named ``escape_names`` that defaults
to ``True``, restoring the old behavior by default.

Fixes: #8113
Change-Id: I9cbedb1080bc06d51f287fd2cbf26aaab1c74653
(cherry picked from commit 105cd180856309cf5abf24f59b782a1bcd8210d6)
2022-06-09 10:15:25 -04:00
Mike Bayer b698d512ad graceful degrade for FKs not reflectable
Fixed bugs involving the :paramref:`.Table.include_columns` and the
:paramref:`.Table.resolve_fks` parameters on :class:`.Table`; these
little-used parameters were apparently not working for columns that refer
to foreign key constraints.

In the first case, not-included columns that refer to foreign keys would
still attempt to create a :class:`.ForeignKey` object, producing errors
when attempting to resolve the columns for the foreign key constraint
within reflection; foreign key constraints that refer to skipped columns
are now omitted from the table reflection process in the same way as
occurs for :class:`.Index` and :class:`.UniqueConstraint` objects with the
same conditions. No warning is produced however, as we likely want to
remove the include_columns warnings for all constraints in 2.0.

In the latter case, the production of table aliases or subqueries would
fail on an FK related table not found despite the presence of
``resolve_fks=False``; the logic has been repaired so that if a related
table is not found, the :class:`.ForeignKey` object is still proxied to the
aliased table or subquery (these :class:`.ForeignKey` objects are normally
used in the production of join conditions), but it is sent with a flag that
it's not resolvable. The aliased table / subquery will then work normally,
with the exception that it cannot be used to generate a join condition
automatically, as the foreign key information is missing. This was already
the behavior for such foreign key constraints produced using non-reflection
methods, such as joining :class:`.Table` objects from different
:class:`.MetaData` collections.

Fixes: #8100
Fixes: #8101

Change-Id: Ifa37a91bd1f1785fca85ef163eec031660d9ea4d
(cherry picked from commit 40e3c0da5be7dd526866bfc63590fc5621a9bd6e)
2022-06-07 16:27:28 -04:00
Mike Bayer c1b82a9b4f propagate proxy_key from WrapsColumnExpression
this allows cast() of a label() to propagate the
proxy key outwards in the same way that it apparently
works at the SQL level.

This is stuffing even more rules into naming so basically
seeing how far we can go without other cases starting
to fail.

Fixes: #8084
Change-Id: I20bd97dae798fee6492334c06934e807d0f269ef
(cherry picked from commit 14250f2668)
2022-06-01 12:46:46 -04:00
Mike Bayer 96d40575ec raise informative error when selectable can't be extended
An informative error is raised for the use case where
:meth:`.Insert.from_select` is being passed a "compound select" object such
as a UNION, yet the INSERT statement needs to append additional columns to
support Python-side or explicit SQL defaults from the table metadata. In
this case a subquery of the compound object should be passed.

Fixes: #8073
Change-Id: Ic4a5dbf84ec49d2451901be05cb9cf6ae93f02b7
(cherry picked from commit 7474df2159f89d684d32aabb15014ef95cea1641)
2022-05-31 13:01:09 -04:00
Mike Bayer 03f886aaf5 move bindparam quote application from compiler to default
in 296c84313a for #5653 we generalized Oracle's
parameter escaping feature into the compiler, so that it could also
work for PostgreSQL.  The compiler used quoted names within parameter
dictionaries, which then led to the complexity that all functions
which interpreted keys from the compiled_params dict had to
also quote the param names to use the dictionary.  This
extra complexity was not added to the ORM peristence.py however,
which led to the versioning id feature being broken as well as
other areas where persistence.py relies on naming schemes present
in context.compiled_params.  It also was not added to the
"processors" lookup which led to #8053, that added this escaping
to that part of the compiler.

To both solve the whole problem as well as simplify the compiler
quite a bit, move the actual application of the escaped names
to be as late as possible, when default.py builds the final list
of parameters.  This is more similar to how it worked previously
where OracleExecutionContext would be late-applying these
escaped names.   This re-establishes context.compiled_params as
deterministically named regardless of dialect in use and moves
out the complexity of the quoted param names to be only at the
cursor.execute stage.

Fixed bug, likely a regression from 1.3, where usage of column names that
require bound parameter escaping, more concretely when using Oracle with
column names that require quoting such as those that start with an
underscore, or in less common cases with some PostgreSQL drivers when using
column names that contain percent signs, would cause the ORM versioning
feature to not work correctly if the versioning column itself had such a
name, as the ORM assumes certain bound parameter naming conventions that
were being interfered with via the quotes. This issue is related to
🎫`8053` and essentially revises the approach towards fixing this,
revising the original issue 🎫`5653` that created the initial
implementation for generalized bound-parameter name quoting.

Fixes: #8056
Change-Id: I57b064e8f0d070e328b65789c30076f6a0ca0fef
(cherry picked from commit a48b597d0cafa1dd7fc46be99eb808fd4cb0a347)
2022-05-29 14:34:25 -04:00
Mike Bayer 853d6be759 apply bindparam escape name to processors dictionary
Fixed SQL compiler issue where the "bind processing" function for a bound
parameter would not be correctly applied to a bound value if the bound
parameter's name were "escaped". Concretely, this applies, among other
cases, to Oracle when a :class:`.Column` has a name that itself requires
quoting, such that the quoting-required name is then used for the bound
parameters generated within DML statements, and the datatype in use
requires bind processing, such as the :class:`.Enum` datatype.

Fixes: #8053
Change-Id: I39d060a87e240b4ebcfccaa9c535e971b7255d99
(cherry picked from commit 4d58ca05e83048e999059a8c2c2e67cb77abf976)
2022-05-25 08:53:47 -04:00
Mike Bayer 405f6afaaa raise for same param name in expanding + non expanding
An informative error is raised if two individual :class:`.BindParameter`
objects share the same name, yet one is used within an "expanding" context
(typically an IN expression) and the other is not; mixing the same name in
these two different styles of usage is not supported and typically the
``expanding=True`` parameter should be set on the parameters that are to
receive list values outside of IN expressions (where ``expanding`` is set
by default).

Fixes: #8018
Change-Id: Ie707f29680eea16b9e421af93560ac1958e11a54
(cherry picked from commit f9fccdeeb6)
2022-05-15 17:26:57 -04:00
Mike Bayer 55ac391ef9 use bindparam_type in BinaryElementImpl._post_coercion if available
Fixed an issue where using :func:`.bindparam` with no explicit data or type
given could be coerced into the incorrect type when used in expressions
such as when using :meth:`.ARRAY.comparator.any` and
:meth:`.ARRAY.comparator.all`.

Fixes: #7979
Change-Id: If7779e713c9a3a5fee496b66e417cfd3fca5b1f9
(cherry picked from commit 889cbe5312)
2022-05-01 12:29:29 -04:00
Mike Bayer e32937fa6a backport 6f02d5edd8 to 1.4
in 6f02d5edd8 some cleanup
to ForeignKey repaired the use case of ForeignKey objects
referring to table name alone, by adding more robust
column resolution logic.  This change also fixes an issue
where the "referred column" naming convention key uses the
resolved referred column earlier than usual when a
ForeignKey is setting up its constraint.

Fixed bug where :class:`.ForeignKeyConstraint` naming conventions using the
``referred_column_0`` naming convention key would not work if the foreign
key constraint were set up as a :class:`.ForeignKey` object rather than an
explicit :class:`.ForeignKeyConstraint` object. As this change makes use of
a backport of some fixes from version 2.0, an additional little-known
feature that has likely been broken for many years is also fixed which is
that a :class:`.ForeignKey` object may refer to a referred table by name of
the table alone without using a column name, if the name of the referent
column is the same as that of the referred column.

The ``referred_column_0`` naming convention key was not previously not
tested with the :class:`.ForeignKey` object, only
:class:`.ForeignKeyConstraint`, and this bug reveals that the feature has
never worked correctly unless :class:`.ForeignKeyConstraint` is used for
all FK constraints. This bug traces back to the original introduction of
the feature introduced for 🎫`3989`.

Fixes: #7958
Change-Id: I230d43e9deba5dff889b9e7fee6cd4d3aa2496d3
2022-04-24 16:40:42 -04:00
Mike Bayer eb7061ea7d properly type array element in any() / all()
Fixed bug in :class:`.ARRAY` datatype in combination with :class:`.Enum` on
PostgreSQL where using the ``.any()`` method to render SQL ANY(), given
members of the Python enumeration as arguments, would produce a type
adaptation failure on all drivers.

Fixes: #6515
Change-Id: Ia1e3b4e10aaf264ed436ce6030d105fc60023433
(cherry picked from commit d023c8e1c7ad82fb249fab5155eb83dee17a160c)
2022-04-22 12:53:27 -04:00
Federico Caselli 7097abc391 update flake8 noqa skips with proper syntax
Change-Id: I42ed77f559e3ee5b8c600d98457ee37803ef0ea6
(cherry picked from commit 139c6ec0fb1f930be9b64545262d2580f6cbc83e)
2022-04-11 17:29:40 -04:00
Mike Bayer d1f97828fd maintain complete cloned_set for BindParameter
Fixed regression caused by 🎫`7823` which impacted the caching
system, such that bound parameters that had been "cloned" within ORM
operations, such as polymorphic loading, would in some cases not acquire
their correct execution-time value leading to incorrect bind values being
rendered.

Fixes: #7903
Change-Id: I61c802749b859bebeb127d24e66d6e77d13ce57a
(cherry picked from commit 2168a64aff)
2022-04-06 10:37:09 -04:00
Mike Bayer 270ba157a1 TableValuedAlias generation fixes
Fixed bug in newly implemented
:paramref:`.FunctionElement.table_valued.joins_implicitly` feature where
the parameter would not automatically propagate from the original
:class:`.TableValuedAlias` object to the secondary object produced when
calling upon :meth:`.TableValuedAlias.render_derived` or
:meth:`.TableValuedAlias.alias`.

Additionally repaired these issues in :class:`.TableValuedAlias`:

* repaired a potential memory issue which could occur when
repeatedly calling :meth:`.TableValuedAlias.render_derived` against
successive copies of the same object (for .alias(), we currently
have to still continue chaining from the previous element.  not sure
if this can be improved but this is standard behavior for .alias()
elsewhere)
* repaired issue where the individual element types would be lost when
calling upon :meth:`.TableValuedAlias.render_derived` or
:meth:`.TableValuedAlias.alias`.

Fixes: #7890
Change-Id: Ie5120c7ff1e5c1bba5aaf77c782a51c637860208
(cherry picked from commit c315c7401a)
2022-04-03 12:31:59 -04:00
Federico Caselli b1aaf0a29f Update bindparam cache key
The ``literal_execute`` parameter now takes part of the cache
generation of a bindparam, since it changes the sql string generated
by the compiler.
Previously the correct bind values were used, but the ``literal_execute``
would be ignored on subsequent executions of the same query.

Fixes: #7876
Change-Id: I6bf887f1a2fe31f9d0ab68f5b4ff315004d006b2
(cherry picked from commit 429512d55e814b03854bc12ec541dbeee9e3b94e)
2022-03-30 23:17:31 +02:00
Mike Bayer f4654ed333 column_descriptions or equiv for DML, core select
Added new attributes :attr:`.ValuesBase.returning_column_descriptions` and
:attr:`.ValuesBase.entity_description` to allow for inspection of ORM
attributes and entities that are installed as part of an :class:`.Insert`,
:class:`.Update`, or :class:`.Delete` construct. The
:attr:`.Select.column_descriptions` accessor is also now implemented for
Core-only selectables.

Fixes: #7861
Change-Id: Ia6a1cd24c798ba61f4e8e8eac90a0fd00d738342
(cherry picked from commit 2f1df5f9105149d6cb01c8b6ab6b9ccffa020780)
2022-03-28 16:51:01 -04:00
Mike Bayer 1a196d5b7d support BLANK_SCHEMA, RETAIN_SCHEMA FK schema on copy
Added support so that the :paramref:`.Table.tometadata.referred_schema_fn`
callable passed to :meth:`.Table.to_metadata` may return the value
:data:`.BLANK_SCHEMA` to indicate that the referenced foreign key should be
reset to None. The :data.`RETAIN_SCHEMA` symbol may also be returned from
this function to indicate "no change", which will behave the same as
``None`` currently does which also indicates no change.

Fixes: #7860
Change-Id: I82a45988d534295d8356453f68001b21d4ff706d
(cherry picked from commit 74d3f3965b)
2022-03-26 10:11:47 -04:00
Mike Bayer 58fad09720 Add option to disable from linting for table valued function
Added new parameter
:paramref:`.FunctionElement.table_valued.joins_implicitly`, for the
:meth:`.FunctionElement.table_valued` construct. This parameter indicates
that the given table-valued function implicitly joins to the table it
refers towards, essentially disabling the "from linting" feature, i.e. the
"cartesian product" warning, from taking effect due to the presence of this
parameter. May be used for functions such as ``func.json_each()``.

Fixes: #7845
Change-Id: I80edcb74efbd4417172132c0db4d9c756fdd5eae
(cherry picked from commit 04dcc5c704)
2022-03-23 10:26:54 -04:00
Mike Bayer 26b819e534 pop the stack that we pushed
Fixed regression caused by 🎫`7760` where the new capabilities of
:class:`.TextualSelect` were not fully implemented within the compiler
properly, leading to issues with composed INSERT constructs such as "INSERT
FROM SELECT" and "INSERT...ON CONFLICT" when combined with CTE and textual
statements.

Fixes: #7798
Change-Id: Ia2ce92507e574dd36fd26dd38ec9dd2713584467
(cherry picked from commit c36965ab21)
2022-03-08 13:41:29 -05:00
mike bayer 6b3401bdbf Merge "warn for enum length silently ignored" into rel_1_4 2022-03-04 23:58:17 +00:00
mike bayer b17367905e Merge "fix type string formatting calls" into rel_1_4 2022-03-04 23:31:58 +00:00
Mike Bayer 4955b6f53f warn for enum length silently ignored
the "length" parameter is silently ignored when native_enum
is not passed as False.  if native_enum is True, a non-native
VARCHAR can still be generated.   Warn for this silent ignore
right now, consider having "length" used in all cases where
non-native enum is rendered likely in 2.0.

Change-Id: Ibceedd4e3aa3926f3268c0c39d94ab73d17a9bdc
(cherry picked from commit 18683f474b)
2022-03-04 18:29:50 -05:00
Mike Bayer efe4359052 fix type string formatting calls
Fixed type-related error messages that would fail for values that were
tuples, due to string formatting syntax, including compile of unsupported
literal values and invalid boolean values.

Fixes: #7721
Change-Id: I6775721486ef2db2d0738b9aa08b9f2570f55659
(cherry picked from commit a261a78894)
2022-03-04 17:48:53 -05:00
Mike Bayer d8c293145f add length to enum repr params
This amends the fix for #7789.

Fixes: #7598
Change-Id: I067a081d743f1efaf8288601bec0400712012265
(cherry picked from commit a26a522648)
2022-03-04 17:31:39 -05:00
petit87 3cc9448ce3 Fix repr for MySQL SET, generic Enum
Fixed issues in :class:`_mysql.SET` datatype as well as :class:`.Enum`
where the ``__repr__()`` method would not render all optional parameters in
the string output, impacting the use of these types in Alembic
autogenerate. Pull request for MySQL courtesy Yuki Nishimine.

Fixes: #7720
Fixes: #7789
Closes: #7772
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7772
Pull-request-sha: d58845479f
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>

Change-Id: Idcec23eab4258511d9f32f4e3d78e511ea6021f1
(cherry picked from commit a926dea6b7)
2022-03-04 13:01:30 -05:00
Mike Bayer 2b72c09573 support add_cte() for TextualSelect
Fixed issue where the :meth:`.HasCTE.add_cte` method as called upon a
:class:`.TextualSelect` instance was not being accommodated by the SQL
compiler. The fix additionally adds more "SELECT"-like compiler behavior to
:class:`.TextualSelect` including that DML CTEs such as UPDATE and INSERT
may be accommodated.

Fixes: #7760
Change-Id: Id97062d882e9b2a81b8e31c2bfaa9cfc5f77d5c1
(cherry picked from commit bef67e5812)
2022-02-23 16:58:08 -05:00
Mike Bayer 4a99fc26ab dont use exception catches for warnings; modernize xdist detection
Improvements to the test suite's integration with pytest such that the
"warnings" plugin, if manually enabled, will not interfere with the test
suite, such that third parties can enable the warnings plugin or make use
of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass.
Additionally, modernized the detection of the "pytest-xdist" plugin so that
plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
without breaking the test suite if xdist were still installed. Warning
filters that promote deprecation warnings to errors are now localized to
SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for
general Python deprecation warnings, so that non-SQLAlchemy deprecation
warnings emitted from pytest plugins should also not impact the test suite.

Identified a bit of cleanup for the PostgreSQL provisioning
as a result.

Fixes: #7599
Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
(cherry picked from commit a0f1914b903de6c130ab1c3267138b8ad208e144)
2022-01-22 19:18:37 -05:00
Mike Bayer 692d05a5cc reject methods as lambda SQL callables
Added an informative error message when a method object is passed to a SQL
construct. Previously, when such a callable were passed, as is a common
typographical error when dealing with method-chained SQL constructs, they
were interpreted as "lambda SQL" targets to be invoked at compilation time,
which would lead to silent failures. As this feature was not intended to be
used with methods, method objects are now rejected.

Fixes: #7032
Change-Id: If714715bd8c11557ab769ee3b1a24264b0b06acc
(cherry picked from commit e28ec27b59)
2022-01-18 17:21:59 -05:00
Mike Bayer 6b646139bb ensure correlate_except is checked for empty tuple
Fixed issue where :meth:`_sql.Select.correlate_except` method, when passed
either the ``None`` value or no arguments, would not correlate any elements
when used in an ORM context (that is, passing ORM entities as FROM
clauses), rather than causing all FROM elements to be considered as
"correlated" in the same way which occurs when using Core-only constructs.

Fixes: #7514
Change-Id: Ic4a5252c8f3c1140aba6c308264948f3a91f33f5
(cherry picked from commit 709239f4a6)
2022-01-03 17:39:51 -05:00
Mike Bayer a0cfc934dc accommodate for "clone" of ColumnClause
for use with the ClauseElement.params() method,
altered ColumnClause._clone() so that while the element
stays immutable, if the column is associated with a subquery,
it returns a new version of itself as corresponding to a
clone of the subquery.  this allows processing functions
to access the parameters in the subquery and produce a
copy of it.  The use case here is the expanded use of
.params() within loader strategies that use
HasCacheKey._apply_params_to_element().

Fixed issue in new "loader criteria" method
:meth:`_orm.PropComparator.and_` where usage with a loader strategy like
:func:`_orm.selectinload` against a column that was a member of the ``.c.``
collection of a subquery object, where the subquery would be dynamically
added to the FROM clause of the statement, would be subject to stale
parameter values within the subquery in the SQL statement cache, as the
process used by the loader strategy to replace the parameters at execution
time would fail to accommodate the subquery when received in this form.

Fixes: #7489
Change-Id: Ibb3b6af140b8a62a2c8d05b2ac92e86ca3013c46
(cherry picked from commit 267e9cbf6e)
2021-12-21 20:11:35 -05:00
Nils Philippsen 4f0bf86eec Ignore ephemeral classes in test_all_present()
Fixed a regression in the test suite where the test called
``CompareAndCopyTest::test_all_present`` would fail on some platforms due
to additional testing artifacts being detected. Pull request courtesy Nils
Philippsen.

In some circumstances, ephemeral class objects that are created within
the scope of a test method don't seem to be garbage collected directly
on exit. Filter out classes created in test modules.

Fixes: #7450
Closes: #7451
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7451
Pull-request-sha: 135a8aaba2

Change-Id: I621967bd916089dc1e3f98625fd2a852cd9fd712
(cherry picked from commit 4291e3c235569031948ebaacec4dde8776255e0e)
2021-12-14 08:50:06 -05:00
Mike Bayer 5ae1c14f4c implement correct errors for Row immutability
Corrected the error message for the ``AttributeError`` that's raised when
attempting to write to an attribute on the :class:`_result.Row` class,
which is immutable. The previous message claimed the column didn't exist
which is misleading.

Fixes: #7432
Change-Id: If0e2cbd3f763dca6c99a18aa42252c69f1207d59
(cherry picked from commit f113e97921)
2021-12-09 14:24:29 -05:00
Mike Bayer 8d5ed4c39e Warn when caching is disabled / document
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.

As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``.  These have been amended to use inherit
cache where appropriate.  "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.

The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.

Fixes: #7394
Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
(cherry picked from commit 22deafe152)
2021-12-06 18:28:50 -05:00
Federico Caselli 70d355358d The where method of exists now accepts multiple cluase.
Support multiple clause elements in the :meth:`_sql.Exists.where` method,
unifying the api with the on presented by a normal :func:`_sql.select`
construct.

Fixes: #7386
Change-Id: I5df20478008cd5167053d357cbfad8a641c62b44
(cherry picked from commit b2bc0c8e41)
2021-12-04 23:09:38 +01:00
Mike Bayer 77f987022c generalize cache_ok to UserDefinedType
Extended the ``cache_ok`` flag and corresponding warning message if this
flag is not defined, a behavior first established for
:class:`.TypeDecorator` as part of 🎫`6436`, to also take place for
:class:`.UserDefinedType`, by generalizing the flag and associated caching
logic to a new common base for these two types, :class:`.ExternalType`.

The change means any current :class:`.UserDefinedType` will now cause SQL
statement caching to no longer take place for statements which make use of
the datatype, along with a warning being emitted, unless the class defines
the :attr:`.UserDefinedType.cache_ok` flag as True. If the datatype cannot
form a deterministic, hashable cache key derived from its arguments, it may return
False which will continue to keep caching disabled but will suppress the
warning. In particular, custom datatypes currently used in packages such as
SQLAlchemy-utils will need to implement this flag. The issue was observed
as a result of a SQLAlchemy-utils datatype that is not currently cacheable.

Fixes: #7319
Change-Id: Ie0b5d4587df87bfe66d2fe7cd4585c3882584575
(cherry picked from commit 4761e6878b)
2021-11-17 15:55:21 -05:00
Eric Masseran 24a53fd8fc Add Non linear CTE support
"Compound select" methods like :meth:`_sql.Select.union`,
:meth:`_sql.Select.intersect_all` etc. now accept ``*other`` as an argument
rather than ``other`` to allow for multiple additional SELECTs to be
compounded with the parent statement at once. In particular, the change as
applied to :meth:`_sql.CTE.union` and :meth:`_sql.CTE.union_all` now allow
for a so-called "non-linear CTE" to be created with the :class:`_sql.CTE`
construct, whereas previously there was no way to have more than two CTE
sub-elements in a UNION together while still correctly calling upon the CTE
in recursive fashion. Pull request courtesy Eric Masseran.

Allow:

```sql
WITH RECURSIVE nodes(x) AS (
   SELECT 59
   UNION
   SELECT aa FROM edge JOIN nodes ON bb=x
   UNION
   SELECT bb FROM edge JOIN nodes ON aa=x
)
SELECT x FROM nodes;
```

Based on @zzzeek suggestion: https://github.com/sqlalchemy/sqlalchemy/pull/7133#issuecomment-933882348

Fixes: #7259
Closes: #7260
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7260
Pull-request-sha: 2565a5fd4b

Change-Id: I685c8379762b5fb6ab4107ff8f4d8a4de70c0ca6
(cherry picked from commit 958f902b1f)
2021-11-16 14:45:35 -05:00
mike bayer 411de1f1cf Merge "set within_columns_clause=False for all sub-elements of select()" into rel_1_4 2021-11-09 23:56:20 +00:00
mike bayer 6678a3fa7f Merge "change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting" into rel_1_4 2021-11-09 23:52:38 +00:00