Commit Graph

1940 Commits

Author SHA1 Message Date
Mike Bayer 85fa363c84 deep compare CTEs before considering them conflicting
Fixed issue where referencing a CTE multiple times in conjunction with a
polymorphic SELECT could result in multiple "clones" of the same CTE being
constructed, which would then trigger these two CTEs as duplicates. To
resolve, the two CTEs are deep-compared when this occurs to ensure that
they are equivalent, then are treated as equivalent.

Fixes: #8357
Change-Id: I1f634a9cf7a6c4256912aac1a00506aecea3b0e2
2022-08-05 17:26:00 -04:00
mike bayer f085762744 Merge "translate joined inheritance cols in UPDATE/DELETE" into main 2022-08-05 19:29:26 +00:00
Mike Bayer 9f992e4218 translate joined inheritance cols in UPDATE/DELETE
Fixed issue in ORM enabled UPDATE when the statement is created against a
joined-inheritance subclass, updating only local table columns, where the
"fetch" synchronization strategy would not render the correct RETURNING
clause for databases that use RETURNING for fetch synchronization.
Also adjusts the strategy used for RETURNING in UPDATE FROM and
DELETE FROM statements.

Also fixes MariaDB which does not support RETURNING with
DELETE..USING.  this was not caught in tests because
"fetch" strategy wasn't tested.  so also adjust the ORMDMLState
classes to look for "extra froms" first before adding
RETURNING, add new parameters to interfaces for
"update_returning_multitable" and "delete_returning_multitable".
A new execution option is_delete_using=True, described in the
changelog message, is added to allow the ORM to know up front
if a certain statement should have a SELECT up front
for "fetch" strategy.

Fixes: #8344
Change-Id: I3dcdb68e6e97ab0807a573c2fdb3d53c16d063ba
2022-08-05 12:53:35 -04:00
Mike Bayer 82a1d4096f include column.default, column.onupdate in eager_defaults
Fixed bug in the behavior of the :paramref:`_orm.Mapper.eager_defaults`
parameter such that client-side SQL default or onupdate expressions in the
table definition alone will trigger a fetch operation using RETURNING or
SELECT when the ORM emits an INSERT or UPDATE for the row. Previously, only
server side defaults established as part of table DDL and/or server-side
onupdate expressions would trigger this fetch, even though client-side SQL
expressions would be included when the fetch was rendered.

Fixes: #7438
Change-Id: Iba719298ba4a26d185edec97ba77d2d54585e5a4
2022-08-05 10:07:15 -04:00
Mike Bayer f684bb7659 ensure RETURNING renders in stringify w/ no server version
just in my own testing, if I say insert().return_defaults()
and stringify, I should see it, so make sure all the dialects
default to "insert_returning" etc. , with downgrade on
server version check.

Change-Id: Id64e78fcb03c48b5dcb0feb21cb9cc495edd15e9
2022-08-03 20:47:27 -04:00
Mike Bayer 1ecbf14cc2 implement tuple-slices from .c collections
Added new syntax to the ``.c`` collection on all :class:`.FromClause`
objects allowing tuples of keys to be passed to ``__getitem__()``, along
with support for ``select()`` handling of ``.c`` collections directly,
allowing the syntax ``select(table.c['a', 'b', 'c'])`` to be possible. The
sub-collection returned is itself a :class:`.ColumnCollection` which is
also directly consumable by :func:`_sql.select` and similar now.

Fixes: #8285
Change-Id: I2236662c477ffc50af079310589e213323c960d1
2022-08-01 21:46:33 +00:00
mike bayer 9a8d039716 Merge "feat: add drop constraint if exists to compiler" into main 2022-07-28 18:19:44 +00:00
Mike Bayer 807d4945d2 check for TypeDecorator when handling getitem
Fixed issue where :class:`.TypeDecorator` would not correctly proxy the
``__getitem__()`` operator when decorating the :class:`.ARRAY` datatype,
without explicit workarounds.

Fixes: #7249
Change-Id: I3273572b4757e41fb5952639cb867314227d368a
2022-07-19 12:37:26 -04:00
Mike Bayer 85a88df13a use concat() directly for contains, startswith, endswith
Adjusted the SQL compilation for string containment functions
``.contains()``, ``.startswith()``, ``.endswith()`` to force the use of the
string concatenation operator, rather than relying upon the overload of the
addition operator, so that non-standard use of these operators with for
example bytestrings still produces string concatenation operators.

To accommodate this, needed to add a new _rconcat operator function,
which is private, as well as a fallback in concat_op() that works
similarly to Python builtin ops.

Fixes: #8253
Change-Id: I2b7f56492f765742d88cb2a7834ded6a2892bd7e
2022-07-17 11:32:27 -04:00
Mike Bayer 26c0e8e184 implement column._merge()
this takes the user-defined args of one Column and merges
them into the not-user-defined args of another Column.

Implemented within the pep-593 column transfer operation
to begin to make this new feature more robust.

work may still be needed for constraints etc. but
in theory everything from the left side annotated column
should take effect for the right side if not otherwise
specified on the right.

Change-Id: I57eb37ed6ceb4b60979a35cfc4b63731d990911d
2022-07-16 17:41:09 -04:00
Mike Bayer 370d73b4e4 generalize sql server check for id col to accommodate ORM cases
Fixed issues that prevented the new usage patterns for using DML with ORM
objects presented at :ref:`orm_dml_returning_objects` from working
correctly with the SQL Server pyodbc dialect.

Here we add a step to look in compile_state._dict_values more thoroughly
for the keys we need to determine "identity insert" or not, and also
add a new compiler variable dml_compile_state so that we can skip the
ORM's compile_state if present.

Fixes: #8210
Change-Id: Idbd76bb3eb075c647dc6c1cb78f7315c821e15f7
2022-07-06 09:58:11 -04:00
Mike Fiedler 41656f830c feat: add drop constraint if exists to compiler
### Description

Add `DROP CONSTRAINT ... IF EXISTS` behavior to the compiler.

Fixes https://github.com/sqlalchemy/sqlalchemy/issues/8141

### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [x] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

**Have a nice day!**

Closes: #8161
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8161
Pull-request-sha: 43276e29fa

Change-Id: I18bae3cf013159b6fffde4413fb59ce19ff83c16
2022-07-03 17:56:07 -04:00
Mike Bayer 741af02893 repair yield_per for non-SS dialects and add new options
Implemented new :paramref:`_engine.Connection.execution_options.yield_per`
execution option for :class:`_engine.Connection` in Core, to mirror that of
the same :ref:`yield_per <orm_queryguide_yield_per>` option available in
the ORM. The option sets both the
:paramref:`_engine.Connection.execution_options.stream_results` option at
the same time as invoking :meth:`_engine.Result.yield_per`, to provide the
most common streaming result configuration which also mirrors that of the
ORM use case in its usage pattern.

Fixed bug in :class:`_engine.Result` where the usage of a buffered result
strategy would not be used if the dialect in use did not support an
explicit "server side cursor" setting, when using
:paramref:`_engine.Connection.execution_options.stream_results`. This is in
error as DBAPIs such as that of SQLite and Oracle already use a
non-buffered result fetching scheme, which still benefits from usage of
partial result fetching.   The "buffered" strategy is now used in all
cases where :paramref:`_engine.Connection.execution_options.stream_results`
is set.

Added :meth:`.FilterResult.yield_per` so that result implementations
such as :class:`.MappingResult`, :class:`.ScalarResult` and
:class:`.AsyncResult` have access to this method.

Fixes: #8199

Change-Id: I6dde3cbe483a1bf81e945561b60f4b7d1c434750
2022-07-01 12:14:02 -04:00
cheremnov 5fb63bc142 Comments on (named) constraints
Adds support for comments on named constraints, including `ForeignKeyConstraint`, `PrimaryKeyConstraint`, `CheckConstraint`, `UniqueConstraint`, solving the [Issue 5667](https://github.com/sqlalchemy/sqlalchemy/issues/5667).

Supports only PostgreSQL backend.

### Description

Following the example of [Issue 1546](https://github.com/sqlalchemy/sqlalchemy/issues/1546), supports comments on constraints. Specifically, enables comments on _named_ ones — as I get it, PostgreSQL prohibits comments on unnamed constraints.

Enables setting the comments for named constraints like this:
```
Table(
   'example', metadata,
   Column('id', Integer),
   Column('data', sa.String(30)),
   PrimaryKeyConstraint(
       "id", name="id_pk", comment="id_pk comment"
    ),
   CheckConstraint('id < 100', name="cc1", comment="Id value can't exceed 100"),
   UniqueConstraint(['data'], name="uc1", comment="Must have unique data field"),
)
```

Provides the DDL representation for constraint comments and routines to create and drop them. Class `.Inspector` reflects constraint comments via methods like `get_check_constraints` .
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
- [ ] A short code fix
- [x] A new feature implementation
	- Solves the issue 5667.
	- The commit message includes `Fixes: 5667`.
	- Includes tests based on comment reflection.

**Have a nice day!**

Fixes: #5667
Closes: #7742
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7742
Pull-request-sha: 42a5d3c3e9

Change-Id: Ia60f578595afdbd6089541c9a00e37997ef78ad3
2022-06-29 09:13:37 +00:00
Mike Bayer 815de6c343 require at least one dialect name for variant
the call doesn't make sense otherwise

Fixes: #8179
Change-Id: I0e5dd584dc7090b536f9732cbfc6f3a5c8846dc5
2022-06-26 16:29:04 +02:00
David Baumgold 017fd9ae06 Domain type
Added a new Postgresql :class:`_postgresql.DOMAIN` datatype, which follows
the same CREATE TYPE / DROP TYPE behaviors as that of PostgreSQL
:class:`_postgresql.ENUM`. Much thanks to David Baumgold for the efforts on
this.

Fixes: #7316
Closes: #7317
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7317
Pull-request-sha: bc9a82f010

Change-Id: Id8d7e48843a896de17d20cc466b115b3cc065132
2022-06-21 10:17:40 -04:00
Mike Bayer 46c0fa56e9 implement literal stringification for arrays
as we already implement stringification for the contents,
provide a bracketed syntax for default and ARRAY literal
for PG specifically.   ARRAY literal seems much simpler to
render than their quoted syntax which requires double quotes
for strings.

also open up testing for pg8000 which has likely been
fine with arrays for awhile now, bump the version pin
also.

Fixes: #8138
Change-Id: Id85b052b0a9564d6aa1489160e58b7359f130fdd
2022-06-15 14:32:53 -04:00
Mike Bayer 4274981156 honor enum length in all cases
The :paramref:`.Enum.length` parameter, which sets the length of the
``VARCHAR`` column for non-native enumeration types, is now used
unconditionally when emitting DDL for the ``VARCHAR`` datatype, including
when the :paramref:`.Enum.native_enum` parameter is set to ``True`` for
target backends that continue to use ``VARCHAR``. Previously the parameter
would be erroneously ignored in this case. The warning previously emitted
for this case is now removed.

Fixes: #7791
Change-Id: I91764546b56e9416479949be8a118cdc91ac5ed9
2022-06-10 13:21:26 -04:00
Mike Bayer ab8b4f25ee restore parameter escaping for public methods
Adjusted the fix made for 🎫`8056` which adjusted the escaping of
bound parameter names with special characters such that the escaped names
were translated after the SQL compilation step, which broke a published
recipe on the FAQ illustrating how to merge parameter names into the string
output of a compiled SQL string. The change restores the escaped names that
come from ``compiled.params`` and adds a conditional parameter to
:meth:`.SQLCompiler.construct_params` named ``escape_names`` that defaults
to ``True``, restoring the old behavior by default.

Fixes: #8113
Change-Id: I9cbedb1080bc06d51f287fd2cbf26aaab1c74653
2022-06-09 10:14:13 -04:00
Mike Bayer 93bc7ed534 graceful degrade for FKs not reflectable
Fixed bugs involving the :paramref:`.Table.include_columns` and the
:paramref:`.Table.resolve_fks` parameters on :class:`.Table`; these
little-used parameters were apparently not working for columns that refer
to foreign key constraints.

In the first case, not-included columns that refer to foreign keys would
still attempt to create a :class:`.ForeignKey` object, producing errors
when attempting to resolve the columns for the foreign key constraint
within reflection; foreign key constraints that refer to skipped columns
are now omitted from the table reflection process in the same way as
occurs for :class:`.Index` and :class:`.UniqueConstraint` objects with the
same conditions. No warning is produced however, as we likely want to
remove the include_columns warnings for all constraints in 2.0.

In the latter case, the production of table aliases or subqueries would
fail on an FK related table not found despite the presence of
``resolve_fks=False``; the logic has been repaired so that if a related
table is not found, the :class:`.ForeignKey` object is still proxied to the
aliased table or subquery (these :class:`.ForeignKey` objects are normally
used in the production of join conditions), but it is sent with a flag that
it's not resolvable. The aliased table / subquery will then work normally,
with the exception that it cannot be used to generate a join condition
automatically, as the foreign key information is missing. This was already
the behavior for such foreign key constraints produced using non-reflection
methods, such as joining :class:`.Table` objects from different
:class:`.MetaData` collections.

Fixes: #8100
Fixes: #8101

Change-Id: Ifa37a91bd1f1785fca85ef163eec031660d9ea4d
2022-06-07 12:34:47 -04:00
mike bayer 6e538cf64a Merge "Generalize RETURNING and suppor for MariaDB / SQLite" into main 2022-06-05 20:40:41 +00:00
cui fliter 8131ef51ef fix some typos (#8093)
Signed-off-by: cuishuang <imcusg@gmail.com>
2022-06-04 12:16:27 +02:00
Daniel Black 466ed5b53a Generalize RETURNING and suppor for MariaDB / SQLite
As almost every dialect supports RETURNING now, RETURNING
is also made more of a default assumption.

* the default compiler generates a RETURNING clause now
  when specified; CompileError is no longer raised.
* The dialect-level implicit_returning parameter now has
  no effect.   It's not fully clear if there are real world
  cases relying on the dialect-level parameter, so we will see
  once 2.0 is released.   ORM-level RETURNING can be disabled
  at the table level, and perhaps "implicit returning" should
  become an ORM-level option at some point as that's where
  it applies.
* Altered ORM update() / delete() to respect table-level
  implicit returning for fetch.
* Since MariaDB doesnt support UPDATE returning, "full_returning"
  is now split into insert_returning, update_returning, delete_returning
* Crazy new thing.  Dialects that have *both* cursor.lastrowid
  *and* returning.   so now we can pick between them for SQLite
  and mariadb.  so, we are trying to keep it on .lastrowid for
  simple inserts with an autoincrement column, this helps with
  some edge case test scenarios and i bet .lastrowid is faster
  anyway.  any return_defaults() / multiparams etc then we
  use returning
* SQLite decided they dont want to return rows that match in
  ON CONFLICT.  this is flat out wrong, but for now we need to
  work with it.

Fixes: #6195
Fixes: #7011
Closes: #7047
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7047
Pull-request-sha: d25d5ea3ab

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I9908ce0ff7bdc50bd5b27722081767c31c19a950
2022-06-02 12:51:20 -04:00
mike bayer 6e8bc369d2 Merge "propagate proxy_key from WrapsColumnExpression" into main 2022-06-01 17:27:13 +00:00
mike bayer 7b6fb299bb Merge "add backend agnostic UUID datatype" into main 2022-06-01 16:13:36 +00:00
Mike Bayer 14250f2668 propagate proxy_key from WrapsColumnExpression
this allows cast() of a label() to propagate the
proxy key outwards in the same way that it apparently
works at the SQL level.

This is stuffing even more rules into naming so basically
seeing how far we can go without other cases starting
to fail.

Fixes: #8084
Change-Id: I20bd97dae798fee6492334c06934e807d0f269ef
2022-06-01 12:11:19 -04:00
Mike Bayer 349a7c5e0e add backend agnostic UUID datatype
Added new backend-agnostic :class:`_types.Uuid` datatype generalized from
the PostgreSQL dialects to now be a core type, as well as migrated
:class:`_types.UUID` from the PostgreSQL dialect. Thanks to Trevor Gross
for the help on this.

also includes:

* corrects some missing behaviors in the suite literal fixtures
  test where row round trips weren't being correctly asserted.
* fixes some of the ISO literal date rendering added in
  952383f9ee for #5052 to truncate datetime strings for date/time
  datatypes in the same way that drivers typically do for bound
  parameters; this was not working fully and wasn't caught by the
  broken test fixture

Fixes: #7212
Change-Id: I981ac6d34d278c18281c144430a528764c241b04
2022-06-01 11:40:56 -04:00
Mike Bayer d1142c6e9f raise informative error when selectable can't be extended
An informative error is raised for the use case where
:meth:`.Insert.from_select` is being passed a "compound select" object such
as a UNION, yet the INSERT statement needs to append additional columns to
support Python-side or explicit SQL defaults from the table metadata. In
this case a subquery of the compound object should be passed.

Fixes: #8073
Change-Id: Ic4a5dbf84ec49d2451901be05cb9cf6ae93f02b7
2022-05-31 13:00:14 -04:00
Mike Bayer cac8de9ab2 move bindparam quote application from compiler to default
in 296c84313a for #5653 we generalized Oracle's
parameter escaping feature into the compiler, so that it could also
work for PostgreSQL.  The compiler used quoted names within parameter
dictionaries, which then led to the complexity that all functions
which interpreted keys from the compiled_params dict had to
also quote the param names to use the dictionary.  This
extra complexity was not added to the ORM peristence.py however,
which led to the versioning id feature being broken as well as
other areas where persistence.py relies on naming schemes present
in context.compiled_params.  It also was not added to the
"processors" lookup which led to #8053, that added this escaping
to that part of the compiler.

To both solve the whole problem as well as simplify the compiler
quite a bit, move the actual application of the escaped names
to be as late as possible, when default.py builds the final list
of parameters.  This is more similar to how it worked previously
where OracleExecutionContext would be late-applying these
escaped names.   This re-establishes context.compiled_params as
deterministically named regardless of dialect in use and moves
out the complexity of the quoted param names to be only at the
cursor.execute stage.

Fixed bug, likely a regression from 1.3, where usage of column names that
require bound parameter escaping, more concretely when using Oracle with
column names that require quoting such as those that start with an
underscore, or in less common cases with some PostgreSQL drivers when using
column names that contain percent signs, would cause the ORM versioning
feature to not work correctly if the versioning column itself had such a
name, as the ORM assumes certain bound parameter naming conventions that
were being interfered with via the quotes. This issue is related to
🎫`8053` and essentially revises the approach towards fixing this,
revising the original issue 🎫`5653` that created the initial
implementation for generalized bound-parameter name quoting.

Fixes: #8056
Change-Id: I57b064e8f0d070e328b65789c30076f6a0ca0fef
2022-05-29 14:36:20 -04:00
Mike Bayer a5d481eaa5 apply bindparam escape name to processors dictionary
Fixed SQL compiler issue where the "bind processing" function for a bound
parameter would not be correctly applied to a bound value if the bound
parameter's name were "escaped". Concretely, this applies, among other
cases, to Oracle when a :class:`.Column` has a name that itself requires
quoting, such that the quoting-required name is then used for the bound
parameters generated within DML statements, and the datatype in use
requires bind processing, such as the :class:`.Enum` datatype.

Fixes: #8053
Change-Id: I39d060a87e240b4ebcfccaa9c535e971b7255d99
2022-05-25 10:07:31 -04:00
mike bayer 3d00de4cca Merge "render select froms first" into main 2022-05-23 14:58:55 +00:00
Mike Bayer 719197dd93 use plainto_tsquery for PG match
The :meth:`.Operators.match` operator now uses ``plainto_tsquery()`` for
PostgreSQL full text search, rather than ``to_tsquery()``. The rationale
for this change is to provide better cross-compatibility with match on
other database backends.    Full support for all PostgreSQL full text
functions remains available through the use of :data:`.func` in
conjunction with :meth:`.Operators.bool_op` (an improved version of
:meth:`.Operators.op` for boolean operators).

Additional doc updates here apply to 1.4 so will backport these
out to a separate commit.

Fixes: #7086
Change-Id: I1946075daf5d9c558e85f73f1bf852604b3b1b8c
2022-05-22 15:25:58 -04:00
Mike Bayer fbdf825192 render select froms first
The FROM clauses that are established on a :func:`_sql.select` construct
when using the :meth:`_sql.Select.select_from` method will now render first
in the FROM clause of the rendered SELECT, which serves to maintain the
ordering of clauses as was passed to the :meth:`_sql.Select.select_from`
method itself without being affected by the presence of those clauses also
being mentioned in other parts of the query. If other elements of the
:class:`_sql.Select` also generate FROM clauses, such as the columns clause
or WHERE clause, these will render after the clauses delivered by
:meth:`_sql.Select.select_from` assuming they were not explictly passed to
:meth:`_sql.Select.select_from` also. This improvement is useful in those
cases where a particular database generates a desirable query plan based on
a particular ordering of FROM clauses and allows full control over the
ordering of FROM clauses.

Fixes: #7888
Change-Id: I740f262a3841f829239011120a59b5e58452db5b
2022-05-22 19:24:18 +00:00
Mike Bayer f9fccdeeb6 raise for same param name in expanding + non expanding
An informative error is raised if two individual :class:`.BindParameter`
objects share the same name, yet one is used within an "expanding" context
(typically an IN expression) and the other is not; mixing the same name in
these two different styles of usage is not supported and typically the
``expanding=True`` parameter should be set on the parameters that are to
receive list values outside of IN expressions (where ``expanding`` is set
by default).

Fixes: #8018
Change-Id: Ie707f29680eea16b9e421af93560ac1958e11a54
2022-05-15 17:26:09 -04:00
Mike Bayer 9f0db34563 update for flake8-future-imports 0.0.5
a whole bunch of errors were apparently blocked by 0.0.4
being installed.

Fixes: #8020
Change-Id: I22a0faeaabe03de501897893391946d677c2df7e
2022-05-14 12:28:01 -04:00
Mike Bayer d972b0f4ed explicitly fetch inserted pk for values(pkcol=None)
Altered the compilation mechanics of the :class:`.Insert` construct such
that the "autoincrement primary key" column value will be fetched via
``cursor.lastrowid`` or RETURNING even if present in the parameter set or
within the :meth:`.Insert.values` method as a plain bound value, for
single-row INSERT statements on specific backends that are known to
generate autoincrementing values even when explicit NULL is passed. This
restores a behavior that was in the 1.3 series for both the use case of
separate parameter set as well as :meth:`.Insert.values`. In 1.4, the
parameter set behavior unintentionally changed to no longer do this, but
the :meth:`.Insert.values` method would still fetch autoincrement values up
until 1.4.21 where 🎫`6770` changed the behavior yet again again
unintentionally as this use case was never covered.

The behavior is now defined as "working" to suit the case where databases
such as SQLite, MySQL and MariaDB will ignore an explicit NULL primary key
value and nonetheless invoke an autoincrement generator.

Fixes: #7998
Change-Id: I5d4105a14217945f87fbe9a6f2a3c87f6ef20529
2022-05-09 09:49:50 -04:00
Mike Bayer 889cbe5312 use bindparam_type in BinaryElementImpl._post_coercion if available
Fixed an issue where using :func:`.bindparam` with no explicit data or type
given could be coerced into the incorrect type when used in expressions
such as when using :meth:`.ARRAY.comparator.any` and
:meth:`.ARRAY.comparator.all`.

Fixes: #7979
Change-Id: If7779e713c9a3a5fee496b66e417cfd3fca5b1f9
2022-05-01 12:28:36 -04:00
Mike Bayer ad11c482e2 pep484 ORM / SQL result support
after some experimentation it seems mypy is more amenable
to the generic types being fully integrated rather than
having separate spin-off types.   so key structures
like Result, Row, Select become generic.  For DML
Insert, Update, Delete, these are spun into type-specific
subclasses ReturningInsert, ReturningUpdate, ReturningDelete,
which is fine since the "row-ness" of these constructs
doesn't happen until returning() is called in any case.

a Tuple based model is then integrated so that these
objects can carry along information about their return
types.  Overloads at the .execute() level carry through
the Tuple from the invoked object to the result.

To suit the issue of AliasedClass generating attributes
that are dynamic, experimented with a custom subclass
AsAliased, but then just settled on having aliased()
lie to the type checker and return `Type[_O]`, essentially.
will need some type-related accessors for with_polymorphic()
also.

Additionally, identified an issue in Update when used
"mysql style" against a join(), it basically doesn't work
if asked to UPDATE two tables on the same column name.
added an error message to the specific condition where
it happens with a very non-specific error message that we
hit a thing we can't do right now, suggest multi-table
update as a possible cause.

Change-Id: I5eff7eefe1d6166ee74160b2785c5e6a81fa8b95
2022-04-27 14:46:36 -04:00
Mike Bayer 3da3322ec4 forwards port test cases and changelog for #7958 to 2.0
in 6f02d5edd8 some cleanup
to ForeignKey repaired the use case of ForeignKey objects
referring to table name alone, by adding more robust
column resolution logic.  This change also fixes an issue
where the "referred column" naming convention key uses the
resolved referred column earlier than usual when a
ForeignKey is setting up its constraint.

change message for 1.4:

Fixed bug where :class:`.ForeignKeyConstraint` naming conventions using the
``referred_column_0`` naming convention key would not work if the foreign
key constraint were set up as a :class:`.ForeignKey` object rather than an
explicit :class:`.ForeignKeyConstraint` object. As this change makes use of
a backport of some fixes from version 2.0, an additional little-known
feature that has likely been broken for many years is also fixed which is
that a :class:`.ForeignKey` object may refer to a referred table by name of
the table alone without using a column name, if the name of the referent
column is the same as that of the referred column.

The ``referred_column_0`` naming convention key was not previously not
tested with the :class:`.ForeignKey` object, only
:class:`.ForeignKeyConstraint`, and this bug reveals that the feature has
never worked correctly unless :class:`.ForeignKeyConstraint` is used for
all FK constraints. This bug traces back to the original introduction of
the feature introduced for 🎫`3989`.

Fixes: #7958
Change-Id: I230d43e9deba5dff889b9e7fee6cd4d3aa2496d3
(cherry picked from commit e32937fa6a)
2022-04-24 16:42:52 -04:00
Mike Bayer 63191fbef6 properly type array element in any() / all()
Fixed bug in :class:`.ARRAY` datatype in combination with :class:`.Enum` on
PostgreSQL where using the ``.any()`` method to render SQL ANY(), given
members of the Python enumeration as arguments, would produce a type
adaptation failure on all drivers.

Fixes: #6515
Change-Id: Ia1e3b4e10aaf264ed436ce6030d105fc60023433
2022-04-22 23:29:11 -04:00
Mike Bayer aeeff72e80 pep-484: ORM public API, constructors
for the moment, abandoning using @overload with
relationship() and mapped_column().  The overloads
are very difficult to get working at all, and
the overloads that were there all wouldn't pass on
mypy.  various techniques of getting them to
"work", meaning having right hand side dictate
what's legal on the left, have mixed success
and wont give consistent results; additionally,
it's legal to have Optional / non-optional
independent of nullable in any case for columns.
relationship cases are less ambiguous but mypy
was not going along with things.

we have a comprehensive system of allowing
left side annotations to drive the right side,
in the absense of explicit settings on the right.
so type-centric SQLAlchemy will be left-side
driven just like dataclasses, and the various flags
and switches on the right side will just not be
needed very much.

in other matters, one surprise, forgot to remove string support
from orm.join(A, B, "somename") or do deprecations
for it in 1.4.   This is a really not-directly-used
structure barely
mentioned in the docs for many years, the example
shows a relationship being used, not a string, so
we will just change it to raise the usual error here.

Change-Id: Iefbbb8d34548b538023890ab8b7c9a5d9496ec6e
2022-04-20 15:14:09 -04:00
mike bayer 07b9788690 Merge "pep484: schema API" into main 2022-04-15 14:58:04 +00:00
Mike Bayer c932123bac pep484: schema API
implement strict typing for schema.py

this module has lots of public API, lots of old decisions
and very hard to follow construction sequences in many
cases, and is also where we get a lot of new feature requests,
so strict typing should help keep things clean.

among improvements here, fixed the pool .info getters
and also figured out how to get ColumnCollection and
related to be covariant so that we may set them up
as returning Column or ColumnClause without any conflicts.

DDL was affected, noting that superclasses of DDLElement
(_DDLCompiles, added recently) can now be passed into
"ddl_if" callables; reorganized ddl into ExecutableDDLElement
as a new name for DDLElement and _DDLCompiles renamed to
BaseDDLElement.

setting up strict also located an API use case that
is completely broken, which is connection.execute(some_default)
returns a scalar value.   This case has been deprecated
and new paths have been set up so that connection.scalar()
may be used.  This likely wasn't possible in previous
versions because scalar() would assume a CursorResult.

The scalar() change also impacts Session as we have explicit
support (since someone had reported it as a regression)
for session.execute(Sequence()) to work.  They will get the
same deprecation message (which omits the word "Connection",
just uses ".execute()" and ".scalar()") and they can then
use Session.scalar() as well.  Getting this to type
correctly while still supporting ORM use cases required
some refactoring, and I also set up a keyword only delimeter
for Session.execute() and related as execution_options /
bind_arguments should always be keyword only, applied these
changes to AsyncSession as well.

Additionally simpify Table __init__ now that we are Python
3 only, we can have positional plus explicit kwargs finally.
Simplify Column.__init__ as well again taking advantage
of kw only arguments.

Fill in most/all __init__ methods in sqltypes.py as
the constructor for types is most of the API.   should
likely do this for dialect-specific types as well.

Apply _InfoType for all info attributes as should have been
done originally and update descriptor decorators.

Change-Id: I3f9f8ff3f1c8858471ff4545ac83d68c88107527
2022-04-15 10:29:23 -04:00
mike bayer 997e4dfae9 Merge "implement multi-element expression constructs" into main 2022-04-14 12:38:43 +00:00
Mike Bayer 428262a2d5 implement multi-element expression constructs
Improved the construction of SQL binary expressions to allow for very long
expressions against the same associative operator without special steps
needed in order to avoid high memory use and excess recursion depth. A
particular binary operation ``A op B`` can now be joined against another
element ``op C`` and the resulting structure will be "flattened" so that
the representation as well as SQL compilation does not require recursion.

To implement this more cleanly, the biggest change here is that
column-oriented lists of things are broken away from ClauseList
in a new class ExpressionClauseList, that also forms the basis
of BooleanClauseList. ClauseList is still used for the generic
"comma-separated list" of things such as Tuple and things like
ORDER BY, as well as in some API endpoints.

Also adds __slots__ to the TypeEngine-bound Comparator
classes.   Still can't really do __slots__ on ClauseElement.

Fixes: #7744
Change-Id: I81a8ceb6f8f3bb0fe52d58f3cb42e4b6c2bc9018
2022-04-13 17:19:31 -04:00
Mike Bayer 0bfb620009 Don't pass vistor to immutables in cloned traverse
Saw someone using cloned_traverse to move columns around
(changing their .table) and not surprisingly having poor results.
As cloned traversal is to provide a hook for in-place mutation
of elements, it should not be given Immutable objects as these
should not be changed once they are structurally composed.

Change-Id: I43b22f52f243ef481a75d2cf5ecc73d50f110a81
2022-04-12 10:09:56 -04:00
Federico Caselli 11bf824474 update flake8 noqa skips with proper syntax
Change-Id: I42ed77f559e3ee5b8c600d98457ee37803ef0ea6
2022-04-11 22:48:23 +02:00
mike bayer 735792d756 Merge "cx_Oracle modernize" into main 2022-04-07 15:19:10 +00:00
Mike Bayer 2acc9ec128 cx_Oracle modernize
Full "RETURNING" support is implemented for the cx_Oracle dialect, meaning
multiple RETURNING rows are now recived for DML statements that produce
more than one row for RETURNING.

cx_Oracle 7 is now the minimum version for cx_Oracle.

Getting Oracle to do multirow returning took about 5 minutes.  however,
getting Oracle's RETURNING system to integrate with ORM-enabled
insert, update, delete, is a big deal because that architecture wasn't
really working very robustly, including some recent changes in 1.4
for FromStatement were done in a hurry, so this patch also cleans up
the FromStatement situation and begins to establish it more concretely
as the base for all ReturnsRows / TextClause ORM scenarios.

Fixes: #6245
Change-Id: I2b4e6007affa51ce311d2d5baa3917f356ab961f
2022-04-07 10:47:53 -04:00
Mike Bayer 2168a64aff maintain complete cloned_set for BindParameter
Fixed regression caused by 🎫`7823` which impacted the caching
system, such that bound parameters that had been "cloned" within ORM
operations, such as polymorphic loading, would in some cases not acquire
their correct execution-time value leading to incorrect bind values being
rendered.

Fixes: #7903
Change-Id: I61c802749b859bebeb127d24e66d6e77d13ce57a
2022-04-06 10:01:43 -04:00