Commit Graph

1973 Commits

Author SHA1 Message Date
Mike Bayer 1284fa377e disallow same-named columns, unchecked replacement in Table
Fixed issue where table reflection using :paramref:`.Table.extend_existing`
would fail to deduplicate a same-named column if the existing
:class:`.Table` used a separate key. The
:paramref:`.Table.autoload_replace` parameter would allow the column to be
skipped but under no circumstances should a :class:`.Table` ever have the
same-named column twice.

Additionally, changed deprecation warnings to exceptions
as were implemented in I1d58c8ebe081079cb669e7ead60886ffc1b1a7f5 .

Fixes: #8925
Change-Id: I83d0f8658177a7ffbb06e01dbca91377d1a98d49
2022-12-04 10:45:35 -05:00
Federico Caselli 0f2baae6bf Fix positional compiling bugs
Fixed a series of issues regarding positionally rendered bound parameters,
such as those used for SQLite, asyncpg, MySQL and others. Some compiled
forms would not maintain the order of parameters correctly, such as the
PostgreSQL ``regexp_replace()`` function as well as within the "nesting"
feature of the :class:`.CTE` construct first introduced in 🎫`4123`.

Fixes: #8827
Change-Id: I9813ed7c358cc5c1e26725c48df546b209a442cb
2022-12-01 23:50:30 +01:00
mike bayer 7857a1de32 Merge "Add value-level hooks for SQL type detection; apply to Range" into main 2022-11-29 23:35:16 +00:00
Lele Gaifax 0b239579f0 Add value-level hooks for SQL type detection; apply to Range
Added additional type-detection for the new PostgreSQL
:class:`_postgresql.Range` type, where previous cases that allowed the
psycopg2-native range objects to be received directly by the DBAPI without
SQLAlchemy intercepting them stopped working, as we now have our own value
object. The :class:`_postgresql.Range` object has been enhanced such that
SQLAlchemy Core detects it in otherwise ambiguous situations (such as
comparison to dates) and applies appropriate bind handlers. Pull request
courtesy Lele Gaifax.

Fixes: #8884
Closes: #8886
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8886
Pull-request-sha: 6e95e08a30

Change-Id: I3ca277c826dcf4b5644f44eb251345b439a84ee4
2022-11-29 17:11:38 -05:00
Federico Caselli 78833af4e6 update for mypy 1.0 dev
As I need dmypy to work without facing [1], I am
running the latest build of mypy which seems so far
to finally not have that issue.

update constructs that latest mypy is being more picky
about, including better typing for the _NONE_NAME
symbol used in constraints (porting those elements
from the Enum patch at
I15ac3daee770408b5795746f47c1bbd931b7d26d)

[1] https://github.com/python/mypy/issues/12744

Change-Id: Ib3f56787fa65ea9bb2e6a0bccc4d99f54c516dad
2022-11-29 14:15:16 -05:00
Federico Caselli 61443aa62b Implement ScalarValue
Added :class:`_expression.ScalarValues` that can be used as a column
element allowing using :class:`_expression.Values` inside IN clauses
or in conjunction with ``ANY`` or ``ALL`` collection aggregates.
This new class is generated using the method
:meth:`_expression.Values.scalar_values`.
The :class:`_expression.Values` instance is now coerced to a
:class:`_expression.ScalarValues` when used in a ``IN`` or ``NOT IN``
operation.

Fixes: #6289
Change-Id: Iac22487ccb01553684b908e54d01c0687fa739f1
2022-11-26 18:49:06 -05:00
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
mike bayer 3fc6c40ea7 Merge "perf improvements related to corresponding_column (2)" into main 2022-11-16 00:03:08 +00:00
mike bayer 073553be44 Merge "don't invoke fromclause.c when creating an annotated" into main 2022-11-16 00:01:33 +00:00
Mike Bayer d5be2cc139 perf improvements related to corresponding_column (2)
commit two of two. this reorganizes ColumnCollection
to build a new index up front that's used to optimize
the corresponding_column() method.

Additional performance enhancements within ORM-enabled SQL statements,
specifically targeting callcounts within the construction of ORM
statements, using combinations of :func:`_orm.aliased` with
:func:`_sql.union` and similar "compound" constructs, in addition to direct
performance improvements to the ``corresponding_column()`` internal method
that is used heavily by the ORM by constructs like :func:`_orm.aliased` and
similar.

Fixes: #8796
Change-Id: I4a76788007d5a802b9a4081e6a0f6e4b52497b50
2022-11-15 14:16:06 -05:00
Mike Bayer 93dc7ea150 don't invoke fromclause.c when creating an annotated
The ``aliased()`` constructor calls upon ``__clause_element__()``,
which internally annotates a ``FromClause``, like a subquery.
This became expensive as ``AnnotatedFromClause`` has for
many years called upon ``element.c`` so that the full ``.c``
collection is transferred to the Annotated.

Taking this out proved to be challenging. A straight remove
seemed to not break any tests except for the one that
tested the exact condition.  Nevertheless this seemed
"spooky" so I instead moved the get of ``.c`` to be in a
memoized proxy method.   However, that then exposed
a recursion issue related to loader_criteria; so the
source of that behavior, which was an accidental behavioral
artifact, is now made into an explcicit option that
loader_criteria uses directly.

The accidental behavioral artifact in question is still
kind of strange since I was not able to fully trace out
how it works, but the end result is that fixing the
artifact to be "correct" causes loader_criteria, within
the particular test for #7491, creates a select/
subquery structure with a cycle in it, so compilation fails
with recursion overflow.
The "solution" is to cause the artifact to occur in this
case, which is that the ``AnnotatedFromClause`` will have a
different ``.c`` collection than its element, which is a
subquery.  It's not totally clear how a cycle is generated
when this is not done.

This is commit one of two, which goes through
some hoops to make essentially a one-line change.

The next commit will rework ColumnCollection to optimize
the corresponding_column() method significantly.

Fixes: #8796
Change-Id: Id58ae6554db62139462c11a8be7313a3677456ad
2022-11-15 13:46:02 -05:00
Mike Bayer eea0f44bbd add informative exception context for literal render
An informative re-raise is now thrown in the case where any "literal
bindparam" render operation fails, indicating the value itself and
the datatype in use, to assist in debugging when literal params
are being rendered in a statement.

Fixes: #8800
Change-Id: Id658f8b03359312353ddbb0c7563026239579f7b
2022-11-14 10:28:40 -05:00
Mike Bayer 8e91cfe529 establish consistency for RETURNING column labels
The RETURNING clause now renders columns using the routine as that of the
:class:`.Select` to generate labels, which will include disambiguating
labels, as well as that a SQL function surrounding a named column will be
labeled using the column name itself. This is a more comprehensive change
than a similar one made for the 1.4 series that adjusted the function label
issue only.

includes 1.4's changelog for the backported version which also
fixes an Oracle issue independently of the 2.0 series.

Fixes: #8770
Change-Id: I2ab078a214a778ffe1720dbd864ae4c105a0691d
2022-11-11 16:20:00 +00:00
Mike Bayer b96321ae79 Support result.close() for all iterator patterns
This change contains new features for 2.0 only as well as some
behaviors that will be backported to 1.4.

For 1.4 and 2.0:

Fixed issue where the underlying DBAPI cursor would not be closed when
using :class:`_orm.Query` with :meth:`_orm.Query.yield_per` and direct
iteration, if a user-defined exception case were raised within the
iteration process, interrupting the iterator. This would lead to the usual
MySQL-related issues with server side cursors out of sync.

For 1.4 only:

A similar scenario can occur when using :term:`2.x` executions with direct
use of :class:`.Result`, in that case the end-user code has access to the
:class:`.Result` itself and should call :meth:`.Result.close` directly.
Version 2.0 will feature context-manager calling patterns to address this
use case.  However within the 1.4 scope, ensured that ``.close()`` methods
are available on all :class:`.Result` implementations including
:class:`.ScalarResult`, :class:`.MappingResult`.

For 2.0 only:

To better support the use case of iterating :class:`.Result` and
:class:`.AsyncResult` objects where user-defined exceptions may interrupt
the iteration, both objects as well as variants such as
:class:`.ScalarResult`, :class:`.MappingResult`,
:class:`.AsyncScalarResult`, :class:`.AsyncMappingResult` now support
context manager usage, where the result will be closed at the end of
iteration.

Corrected various typing issues within the engine and async engine
packages.

Fixes: #8710
Change-Id: I3166328bfd3900957eb33cbf1061d0495c9df670
2022-11-03 18:42:52 -04:00
mike bayer 2833aa2afb Merge "apply basic escaping to anon_labels unconditionally" into main 2022-10-28 13:59:05 +00:00
Mike Bayer 50d3b85c69 open some compound tests for mysql 8.0.31
Not sure of exact version but as we have done a major
rebuild of CI, newer mysql 8.0 is passing on these.

Change-Id: Ibcfe0ce519ab6a2941ca514b4254944769b60df4
2022-10-28 01:49:48 -04:00
Mike Bayer caa9f0ff98 apply basic escaping to anon_labels unconditionally
Fixed issue which prevented the :func:`_sql.literal_column` construct from
working properly within the context of a :class:`.Select` construct as well
as other potential places where "anonymized labels" might be generated, if
the literal expression contained characters which could interfere with
format strings, such as open parenthesis, due to an implementation detail
of the "anonymous label" structure.

Fixes: #8724
Change-Id: I3089124fbd055a011c8a245964258503b717d941
2022-10-27 09:28:02 -04:00
Federico Caselli 974b1bd0fc Revert automatic set of sequence start to 1
The :class:`.Sequence` construct restores itself to the DDL behavior it
had prior to the 1.4 series, where creating a :class:`.Sequence` with
no additional arguments will emit a simple ``CREATE SEQUENCE`` instruction
**without** any additional parameters for "start value".   For most backends,
this is how things worked previously in any case; **however**, for
MS SQL Server, the default value on this database is
``-2**63``; to prevent this generally impractical default
from taking effect on SQL Server, the :paramref:`.Sequence.start` parameter
should be provided.   As usage of :class:`.Sequence` is unusual
for SQL Server which for many years has standardized on ``IDENTITY``,
it is hoped that this change has minimal impact.

Fixes: #7211
Change-Id: I1207ea10c8cb1528a1519a0fb3581d9621c27b31
2022-10-17 15:36:25 -04:00
Mike Bayer 2b966de419 accommodate arbitrary embedded params in insertmanyvalues
Fixed bug in new "insertmanyvalues" feature where INSERT that included a
subquery with :func:`_sql.bindparam` inside of it would fail to render
correctly in "insertmanyvalues" format. This affected psycopg2 most
directly as "insertmanyvalues" is used unconditionally with this driver.

Fixes: #8639
Change-Id: I67903fa86afe208899d4f23f940e0727d1be2ce3
2022-10-16 08:47:47 -04:00
Mike Bayer b580b425c3 doc edits
this is addressing comments still remaining on
I9929daab7797be9515f71c888b28af1209e789ff

removes "asyncio" wording discussed in #7659

Change-Id: I1bab2a6fde330b83ef34602956c2988ee6331b21
2022-10-10 21:27:50 +02:00
Jesse Bakker b6eed88ef4 Make if_exists and if_not_exists flags on ddl statements match compiler
Added ``if_exists`` and ``if_not_exists`` parameters for all "Create" /
"Drop" constructs including :class:`.CreateSequence`,
:class:`.DropSequence`, :class:`.CreateIndex`, :class:`.DropIndex`, etc.
allowing generic "IF EXISTS" / "IF NOT EXISTS" phrases to be rendered
within DDL. Pull request courtesy Jesse Bakker.

Fixes: #7354
Closes: #8492
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8492
Pull-request-sha: d107c6ce55

Change-Id: I367e57b2d9216f5180bcc44e86ca6f3dc794e5ca
2022-10-04 02:40:20 +00:00
Mike Bayer 4df1f8393d add "index", "unique" to Column.merge() attrs
Fixes: #8578
Change-Id: Ic79c19748d5bb00353d0a97f3a4b4f5eb9fdbb0c
2022-09-26 22:31:05 -04:00
mike bayer 1657cea73d Merge "aggregate_order_by now supports cache generation." into main 2022-09-26 02:33:19 +00:00
Federico Caselli c86ec8f8c9 aggregate_order_by now supports cache generation.
also adjusted CacheKeyFixture to be a general purpose
fixture so that sub-components / dialects can run
their own cache key tests.

Fixes: #8574
Change-Id: I6c66107856aee11e548d357cea77bceee3e316a0
2022-09-25 21:14:48 -04:00
Mike Bayer a8029f5a7e ORM bulk insert via execute
* ORM Insert now includes "bulk" mode that will run
  essentially the same process as session.bulk_insert_mappings;
  interprets the given list of values as ORM attributes for
  key names
* ORM UPDATE has a similar feature, without RETURNING support,
  for session.bulk_update_mappings
* Added support for upserts to do RETURNING ORM objects as well
* ORM UPDATE/DELETE with list of parameters + WHERE criteria
  is a not implemented; use connection
* ORM UPDATE/DELETE defaults to "auto" synchronize_session;
  use fetch if RETURNING is present, evaluate if not, as
  "fetch" is much more efficient (no expired object SELECT problem)
  and less error prone if RETURNING is available
  UPDATE: howver this is inefficient!   please continue to
  use evaluate for simple cases, auto can move to fetch
  if criteria not evaluable
* "Evaluate" criteria will now not preemptively
  unexpire and SELECT attributes that were individually
  expired. Instead, if evaluation of the criteria indicates that
  the necessary attrs were expired, we expire the object
  completely (delete) or expire the SET attrs unconditionally
  (update). This keeps the object in the same unloaded state
  where it will refresh those attrs on the next pass, for
  this generally unusual case.  (originally #5664)
* Core change! update/delete rowcount comes from len(rows)
  if RETURNING was used.  SQLite at least otherwise did not
  support this.  adjusted test_rowcount accordingly
* ORM DELETE with a list of parameters at all is also a not
  implemented as this would imply "bulk", and there is no
  bulk_delete_mappings (could be, but we dont have that)
* ORM insert().values() with single or multi-values translates
  key names based on ORM attribute names
* ORM returning() implemented for insert, update, delete;
  explcit returning clauses now interpret rows in an ORM
  context, with support for qualifying loader options as well
* session.bulk_insert_mappings() assigns polymorphic identity
  if not set.
* explicit RETURNING + synchronize_session='fetch' is now
  supported with UPDATE and DELETE.
* expanded return_defaults() to work with DELETE also.
* added support for composite attributes to be present
  in the dictionaries used by bulk_insert_mappings and
  bulk_update_mappings, which is also the new ORM bulk
  insert/update feature, that will expand the composite
  values into their individual mapped attributes the way they'd
  be on a mapped instance.
* bulk UPDATE supports "synchronize_session=evaluate", is the
  default.  this does not apply to session.bulk_update_mappings,
  just the new version
* both bulk UPDATE and bulk INSERT, the latter with or without
  RETURNING, support *heterogenous* parameter sets.
  session.bulk_insert/update_mappings did this, so this feature
  is maintained.  now cursor result can be both horizontally
  and vertically spliced :)

This is now a long story with a lot of options, which in
itself is a problem to be able to document all of this
in some way that makes sense.  raising exceptions for
use cases we haven't supported is pretty important here
too, the tradition of letting unsupported things just not work
is likely not a good idea at this point, though there
are still many cases that aren't easily avoidable

Fixes: #8360
Fixes: #7864
Fixes: #7865
Change-Id: Idf28379f8705e403a3c6a937f6a798a042ef2540
2022-09-24 11:18:01 -04:00
Mike Bayer 2bcc97da42 implement batched INSERT..VALUES () () for executemany
the feature is enabled for all built in backends
when RETURNING is used,
except for Oracle that doesn't need it,  and on
psycopg2 and mssql+pyodbc it is used for all INSERT statements,
not just those that use RETURNING.

third party dialects would need to opt in to the new feature
by setting use_insertmanyvalues to True.

Also adds dialect-level guards against using returning
with executemany where we dont have an implementation to
suit it.   execute single w/ returning still defers to the
server without us checking.

Fixes: #6047
Fixes: #7907
Change-Id: I3936d3c00003f02e322f2e43fb949d0e6e568304
2022-09-24 11:15:32 -04:00
Mike Bayer 57b400f079 remove should_nest behavior for contains_eager()
Fixed regression for 1.4 in :func:`_orm.contains_eager` where the "wrap in
subquery" logic of :func:`_orm.joinedload` would be inadvertently triggered
for use of the :func:`_orm.contains_eager` function with similar statements
(e.g. those that use ``distinct()``, ``limit()`` or ``offset()``). This is
not appropriate for :func:`_orm.contains_eager` which has always had the
contract that the user-defined SQL statement is unmodified with the
exception of adding the appropriate columns.

Also includes an adjustment to the assertion in Label._make_proxy()
which was there to prevent a fixed label name from being anonymized;
if the label is already anonymous, the change should proceed.
This logic was being hit before the contains_eager behavior was
adjusted. With the adjustment, this code is not used.

Fixes: #8569
Change-Id: I161e65041c0162fd2b83cbef40f57a50fcfaf0fd
2022-09-23 17:17:02 -04:00
mike bayer 214d1ad7c3 Merge "break out text() from TextualSelect for col matching" into main 2022-09-20 02:34:31 +00:00
Mike Bayer c9af2ebf5e break out text() from TextualSelect for col matching
Fixed issue where mixing "*" with additional explicitly-named column
expressions within the columns clause of a :func:`_sql.select` construct
would cause result-column targeting to sometimes consider the label name or
other non-repeated names to be an ambiguous target.

Fixes: #8536
Change-Id: I3c845eaf571033e54c9208762344f67f4351ac3a
2022-09-19 18:39:18 -04:00
Federico Caselli 02fe382d6b Improve array_agg and Array processing
The :class:`_functions.array_agg` will now set the array dimensions to 1.
Improved :class:`_types.ARRAY` processing to accept ``None`` values as
value of a multi-array.

Fixes: #7083
Change-Id: Iafec4f77fde9719ccc7c8535bf6235dbfbc62102
2022-09-17 13:12:35 +02:00
Matias Martinez Rebori 93aaf16727 implement icontains, istartswith, iendswith operators
Added long-requested case-insensitive string operators
:meth:`_sql.ColumnOperators.icontains`,
:meth:`_sql.ColumnOperators.istartswith`,
:meth:`_sql.ColumnOperators.iendswith`, which produce case-insensitive
LIKE compositions (using ILIKE on PostgreSQL, and the LOWER() function on
all other backends) to complement the existing LIKE composition operators
:meth:`_sql.ColumnOperators.contains`,
:meth:`_sql.ColumnOperators.startswith`, etc. Huge thanks to Matias
Martinez Rebori for their meticulous and complete efforts in implementing
these new methods.

Fixes: #3482
Closes: #8496
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8496
Pull-request-sha: 7287e2c436

Change-Id: I9fcdd603716218067547cc92a2b07bd02a2c366b
2022-09-08 12:15:23 -04:00
Mike Bayer 613642d963 include TableClause.schema in cache key
Fixed issue where use of the :func:`_sql.table` construct, passing a string
for the :paramref:`_sql.table.schema` parameter, would fail to take the
"schema" string into account when producing a cache key, thus leading to
caching collisions if multiple, same-named :func:`_sql.table` constructs
with different schemas were used.

Fixes: #8441
Change-Id: Ic4b55b3e8ec53b4c88ba112691bdf60ea1d4c448
2022-08-30 10:47:24 -04:00
Mike Bayer 7842678484 Column._copy() duplicates "user defined" nullable state exactly
To accommodate how mapped_column() works, after many
attempts to get this working it became clear that _copy()
should just transfer "nullable" state exactly as it was,
including the state where .nullable was set but user_defined_nullable
remains at not user set.

additionally, added a similar step to _merge() that was needed
to preserve the nullability behavior when Identity is present.

server / client default objects are not copied within column._copy()
and this should be fixed.

Fixes: #8410
Change-Id: Ib09df52b71f3e58e67e9f19b893d40a6cc4eec5c
2022-08-21 11:58:20 -04:00
Mike Bayer 85fa363c84 deep compare CTEs before considering them conflicting
Fixed issue where referencing a CTE multiple times in conjunction with a
polymorphic SELECT could result in multiple "clones" of the same CTE being
constructed, which would then trigger these two CTEs as duplicates. To
resolve, the two CTEs are deep-compared when this occurs to ensure that
they are equivalent, then are treated as equivalent.

Fixes: #8357
Change-Id: I1f634a9cf7a6c4256912aac1a00506aecea3b0e2
2022-08-05 17:26:00 -04:00
mike bayer f085762744 Merge "translate joined inheritance cols in UPDATE/DELETE" into main 2022-08-05 19:29:26 +00:00
Mike Bayer 9f992e4218 translate joined inheritance cols in UPDATE/DELETE
Fixed issue in ORM enabled UPDATE when the statement is created against a
joined-inheritance subclass, updating only local table columns, where the
"fetch" synchronization strategy would not render the correct RETURNING
clause for databases that use RETURNING for fetch synchronization.
Also adjusts the strategy used for RETURNING in UPDATE FROM and
DELETE FROM statements.

Also fixes MariaDB which does not support RETURNING with
DELETE..USING.  this was not caught in tests because
"fetch" strategy wasn't tested.  so also adjust the ORMDMLState
classes to look for "extra froms" first before adding
RETURNING, add new parameters to interfaces for
"update_returning_multitable" and "delete_returning_multitable".
A new execution option is_delete_using=True, described in the
changelog message, is added to allow the ORM to know up front
if a certain statement should have a SELECT up front
for "fetch" strategy.

Fixes: #8344
Change-Id: I3dcdb68e6e97ab0807a573c2fdb3d53c16d063ba
2022-08-05 12:53:35 -04:00
Mike Bayer 82a1d4096f include column.default, column.onupdate in eager_defaults
Fixed bug in the behavior of the :paramref:`_orm.Mapper.eager_defaults`
parameter such that client-side SQL default or onupdate expressions in the
table definition alone will trigger a fetch operation using RETURNING or
SELECT when the ORM emits an INSERT or UPDATE for the row. Previously, only
server side defaults established as part of table DDL and/or server-side
onupdate expressions would trigger this fetch, even though client-side SQL
expressions would be included when the fetch was rendered.

Fixes: #7438
Change-Id: Iba719298ba4a26d185edec97ba77d2d54585e5a4
2022-08-05 10:07:15 -04:00
Mike Bayer f684bb7659 ensure RETURNING renders in stringify w/ no server version
just in my own testing, if I say insert().return_defaults()
and stringify, I should see it, so make sure all the dialects
default to "insert_returning" etc. , with downgrade on
server version check.

Change-Id: Id64e78fcb03c48b5dcb0feb21cb9cc495edd15e9
2022-08-03 20:47:27 -04:00
Mike Bayer 1ecbf14cc2 implement tuple-slices from .c collections
Added new syntax to the ``.c`` collection on all :class:`.FromClause`
objects allowing tuples of keys to be passed to ``__getitem__()``, along
with support for ``select()`` handling of ``.c`` collections directly,
allowing the syntax ``select(table.c['a', 'b', 'c'])`` to be possible. The
sub-collection returned is itself a :class:`.ColumnCollection` which is
also directly consumable by :func:`_sql.select` and similar now.

Fixes: #8285
Change-Id: I2236662c477ffc50af079310589e213323c960d1
2022-08-01 21:46:33 +00:00
mike bayer 9a8d039716 Merge "feat: add drop constraint if exists to compiler" into main 2022-07-28 18:19:44 +00:00
Mike Bayer 807d4945d2 check for TypeDecorator when handling getitem
Fixed issue where :class:`.TypeDecorator` would not correctly proxy the
``__getitem__()`` operator when decorating the :class:`.ARRAY` datatype,
without explicit workarounds.

Fixes: #7249
Change-Id: I3273572b4757e41fb5952639cb867314227d368a
2022-07-19 12:37:26 -04:00
Mike Bayer 85a88df13a use concat() directly for contains, startswith, endswith
Adjusted the SQL compilation for string containment functions
``.contains()``, ``.startswith()``, ``.endswith()`` to force the use of the
string concatenation operator, rather than relying upon the overload of the
addition operator, so that non-standard use of these operators with for
example bytestrings still produces string concatenation operators.

To accommodate this, needed to add a new _rconcat operator function,
which is private, as well as a fallback in concat_op() that works
similarly to Python builtin ops.

Fixes: #8253
Change-Id: I2b7f56492f765742d88cb2a7834ded6a2892bd7e
2022-07-17 11:32:27 -04:00
Mike Bayer 26c0e8e184 implement column._merge()
this takes the user-defined args of one Column and merges
them into the not-user-defined args of another Column.

Implemented within the pep-593 column transfer operation
to begin to make this new feature more robust.

work may still be needed for constraints etc. but
in theory everything from the left side annotated column
should take effect for the right side if not otherwise
specified on the right.

Change-Id: I57eb37ed6ceb4b60979a35cfc4b63731d990911d
2022-07-16 17:41:09 -04:00
Mike Bayer 370d73b4e4 generalize sql server check for id col to accommodate ORM cases
Fixed issues that prevented the new usage patterns for using DML with ORM
objects presented at :ref:`orm_dml_returning_objects` from working
correctly with the SQL Server pyodbc dialect.

Here we add a step to look in compile_state._dict_values more thoroughly
for the keys we need to determine "identity insert" or not, and also
add a new compiler variable dml_compile_state so that we can skip the
ORM's compile_state if present.

Fixes: #8210
Change-Id: Idbd76bb3eb075c647dc6c1cb78f7315c821e15f7
2022-07-06 09:58:11 -04:00
Mike Fiedler 41656f830c feat: add drop constraint if exists to compiler
### Description

Add `DROP CONSTRAINT ... IF EXISTS` behavior to the compiler.

Fixes https://github.com/sqlalchemy/sqlalchemy/issues/8141

### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [x] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

**Have a nice day!**

Closes: #8161
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8161
Pull-request-sha: 43276e29fa

Change-Id: I18bae3cf013159b6fffde4413fb59ce19ff83c16
2022-07-03 17:56:07 -04:00
Mike Bayer 741af02893 repair yield_per for non-SS dialects and add new options
Implemented new :paramref:`_engine.Connection.execution_options.yield_per`
execution option for :class:`_engine.Connection` in Core, to mirror that of
the same :ref:`yield_per <orm_queryguide_yield_per>` option available in
the ORM. The option sets both the
:paramref:`_engine.Connection.execution_options.stream_results` option at
the same time as invoking :meth:`_engine.Result.yield_per`, to provide the
most common streaming result configuration which also mirrors that of the
ORM use case in its usage pattern.

Fixed bug in :class:`_engine.Result` where the usage of a buffered result
strategy would not be used if the dialect in use did not support an
explicit "server side cursor" setting, when using
:paramref:`_engine.Connection.execution_options.stream_results`. This is in
error as DBAPIs such as that of SQLite and Oracle already use a
non-buffered result fetching scheme, which still benefits from usage of
partial result fetching.   The "buffered" strategy is now used in all
cases where :paramref:`_engine.Connection.execution_options.stream_results`
is set.

Added :meth:`.FilterResult.yield_per` so that result implementations
such as :class:`.MappingResult`, :class:`.ScalarResult` and
:class:`.AsyncResult` have access to this method.

Fixes: #8199

Change-Id: I6dde3cbe483a1bf81e945561b60f4b7d1c434750
2022-07-01 12:14:02 -04:00
cheremnov 5fb63bc142 Comments on (named) constraints
Adds support for comments on named constraints, including `ForeignKeyConstraint`, `PrimaryKeyConstraint`, `CheckConstraint`, `UniqueConstraint`, solving the [Issue 5667](https://github.com/sqlalchemy/sqlalchemy/issues/5667).

Supports only PostgreSQL backend.

### Description

Following the example of [Issue 1546](https://github.com/sqlalchemy/sqlalchemy/issues/1546), supports comments on constraints. Specifically, enables comments on _named_ ones — as I get it, PostgreSQL prohibits comments on unnamed constraints.

Enables setting the comments for named constraints like this:
```
Table(
   'example', metadata,
   Column('id', Integer),
   Column('data', sa.String(30)),
   PrimaryKeyConstraint(
       "id", name="id_pk", comment="id_pk comment"
    ),
   CheckConstraint('id < 100', name="cc1", comment="Id value can't exceed 100"),
   UniqueConstraint(['data'], name="uc1", comment="Must have unique data field"),
)
```

Provides the DDL representation for constraint comments and routines to create and drop them. Class `.Inspector` reflects constraint comments via methods like `get_check_constraints` .
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
- [ ] A short code fix
- [x] A new feature implementation
	- Solves the issue 5667.
	- The commit message includes `Fixes: 5667`.
	- Includes tests based on comment reflection.

**Have a nice day!**

Fixes: #5667
Closes: #7742
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7742
Pull-request-sha: 42a5d3c3e9

Change-Id: Ia60f578595afdbd6089541c9a00e37997ef78ad3
2022-06-29 09:13:37 +00:00
Mike Bayer 815de6c343 require at least one dialect name for variant
the call doesn't make sense otherwise

Fixes: #8179
Change-Id: I0e5dd584dc7090b536f9732cbfc6f3a5c8846dc5
2022-06-26 16:29:04 +02:00
David Baumgold 017fd9ae06 Domain type
Added a new Postgresql :class:`_postgresql.DOMAIN` datatype, which follows
the same CREATE TYPE / DROP TYPE behaviors as that of PostgreSQL
:class:`_postgresql.ENUM`. Much thanks to David Baumgold for the efforts on
this.

Fixes: #7316
Closes: #7317
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7317
Pull-request-sha: bc9a82f010

Change-Id: Id8d7e48843a896de17d20cc466b115b3cc065132
2022-06-21 10:17:40 -04:00
Mike Bayer 46c0fa56e9 implement literal stringification for arrays
as we already implement stringification for the contents,
provide a bracketed syntax for default and ARRAY literal
for PG specifically.   ARRAY literal seems much simpler to
render than their quoted syntax which requires double quotes
for strings.

also open up testing for pg8000 which has likely been
fine with arrays for awhile now, bump the version pin
also.

Fixes: #8138
Change-Id: Id85b052b0a9564d6aa1489160e58b7359f130fdd
2022-06-15 14:32:53 -04:00