64 Commits

Author SHA1 Message Date
Mike Bayer 9a0d004331 filter_by works across multiple entities
The :meth:`_sql.Select.filter_by`, :meth:`_sql.Update.filter_by` and
:meth:`_sql.Delete.filter_by` methods now search across all entities
present in the statement, rather than limiting their search to only the
last joined entity or the first FROM entity. This allows these methods
to locate attributes unambiguously across multiple joined tables,
resolving issues where changing the order of operations such as
:meth:`_sql.Select.with_only_columns` would cause the method to fail.

If an attribute name exists in more than one FROM clause entity, an
:class:`_exc.AmbiguousColumnError` is now raised, indicating that
:meth:`_sql.Select.filter` (or :meth:`_sql.Select.where`) should be used
instead with explicit table-qualified column references.

Fixes: #8601
Change-Id: I6a46b8f4784801f95f7980ca8ef92f1947653572
2025-12-04 13:53:27 +00:00
Mike Bayer cf62005639 test oracle 23c, mariadb12; reduce backend use
one particular vector test wont run on oracle 23c free, so
just disable it.

added better skips for the rest of the vector tests and
fixed a deprecation issue.

this will be the first run on the new oracle23 on CI so we'll have to
see how this goes.

Also adjust for mariabdb12 being overly helpful with regards
to stale row updates.

as we are having trouble getting 23c to pass throug transaction
tests, i noted we have an explosion of tests due to the multiple
drivers, so this patch introduces __sparse_driver_backend__
for all tests where we want variety of
database server but there's no need to test every driver.
This should dramatically reduce the size of the test suite run

Change-Id: Ic8d3eb0a60e76b4c54c6bb4a721f90c81ede782b
2025-11-25 14:50:56 -05:00
Mike Bayer 89b81ec8c4 add CRUD column marker
Added new Core feature :func:`_sql.from_dml_column` that may be used in
expressions inside of :meth:`.UpdateBase.values` for INSERT or UPDATE; this
construct will copy whatever SQL expression is used for the given target
column in the statement to be used with additional columns. The construct
is mostly intended to be a helper with ORM :class:`.hybrid_property` within
DML hooks.

This is the Core side of the feature being added to the ORM
for #12496

Change-Id: Ic568638a8ce3607deea44af988b6451b30cde36c
2025-07-03 12:11:45 -04:00
Mike Bayer dc0d081762 hardening against inappropriate multi-table updates
Hardening of the compiler's actions for UPDATE statements that access
multiple tables to report more specifically when tables or aliases are
referenced in the SET clause; on cases where the backend does not support
secondary tables in the SET clause, an explicit error is raised, and on the
MySQL or similar backends that support such a SET clause, more specific
checking for not-properly-included tables is performed.  Overall the change
is preventing these erroneous forms of UPDATE statements from being
compiled, whereas previously it was relied on the database to raise an
error, which was not always guaranteed to happen, or to be non-ambiguous,
due to cases where the parent table included the same column name as the
secondary table column being updated.

Fixed bug where the ORM would pull in the wrong column into an UPDATE when
a key name inside of the :meth:`.ValuesBase.values` method could be located
from an ORM entity mentioned in the statement, but where that ORM entity
was not the actual table that the statement was inserting or updating.  An
extra check for this edge case is added to avoid this problem.

Fixes: #12692
Change-Id: I342832b09dda7ed494caaad0cbb81b93fc10fe18
2025-06-24 10:19:48 -04:00
Mike Bayer 359f2ef702 simplify internal storage of DML ordered values
towards some refactorings I will need to do for #12496, this
factors out the "_ordered_values" list of tuples that was used to
track UPDATE VALUES in a specific order.   The rationale for this
separate collection was due to Python dictionaries not maintaining
insert order.   Now that this is standard behavior in Python 3
we can use the same `statement._values` for param-ordered and
table-column-ordered UPDATE rendering.

Change-Id: Id6024ab06e5e3ba427174e7ba3630ff83d81f603
2025-04-10 10:28:56 -04:00
Mike Bayer 7083214539 adjust concat precedence to match that of string comparison
Adjusted the operator precedence for the string concatenation operator to
be equal to that of string matching operators, such as
:meth:`.ColumnElement.like`, :meth:`.ColumnElement.regexp_match`,
:meth:`.ColumnElement.match`, etc., as well as plain ``==`` which has the
same precedence as string comparison operators, so that parenthesis will be
applied to a string concatenation expression that follows a string match
operator. This provides for backends such as PostgreSQL where the "regexp
match" operator is apparently of higher precedence than the string
concatenation operator.

Fixes: #9610
Change-Id: I73640e40e445375177340e1ed8f45b5da98d6dfb
2023-08-21 11:30:53 -04:00
Federico Caselli 058c230cea Update black to v23.3.0 and flake8 to v6
This change could be added to .git-blame-ignore-revs

Change-Id: I7ba10052b26bc3c178d23fb50a1123d0aae965ca
2023-05-30 21:44:26 +02:00
Mike Bayer 4a62625d99 implement FromLinter for UPDATE, DELETE statements
Implemented the "cartesian product warning" for UPDATE and DELETE
statements, those which include multiple tables that are not correlated
together in some way.

Fixed issue where :func:`_dml.update` construct that included multiple
tables and no VALUES clause would raise with an internal error. Current
behavior for :class:`_dml.Update` with no values is to generate a SQL
UPDATE statement with an empty "set" clause, so this has been made
consistent for this specific sub-case.

Fixes: #9721
Change-Id: I556639811cc930d2e37532965d2ae751882af921
2023-05-09 10:08:52 -04:00
Mike Bayer e636917a72 fix ORM support for column-named bindparam() in crud .values()
Fixed bug / regression where using :func:`.bindparam()` with the same name
as a column in the :meth:`.Update.values` method of :class:`.Update`, as
well as the :meth:`.Insert.values` method of :class:`.Insert` in 2.0 only,
would in some cases silently fail to honor the SQL expression in which the
parameter were presented, replacing the expression with a new parameter of
the same name and discarding any other elements of the SQL expression, such
as SQL functions, etc. The specific case would be statements that were
constructed against ORM entities rather than plain :class:`.Table`
instances, but would occur if the statement were invoked with a
:class:`.Session` or a :class:`.Connection`.

:class:`.Update` part of the issue was present in both 2.0 and 1.4 and is
backported to 1.4.

Fixes: #9075
Change-Id: Ie954bc1f492ec6a566163588182ef4910c7ee452
2023-01-11 11:47:02 -05:00
Federico Caselli 06c234d037 Rewrite positional handling, test for "numeric"
Changed how the positional compilation is performed. It's rendered by the compiler
the same as the pyformat compilation. The string is then processed to replace
the placeholders with the correct ones, and to obtain the correct order of the
parameters.
This vastly simplifies the computation of the order of the parameters, that in
case of nested CTE is very hard to compute correctly.

Reworked how numeric paramstyle behavers:
- added support for repeated parameter, without duplicating them like in normal
positional dialects
- implement insertmany support. This requires that the dialect supports out of
order placehoders, since all parameters that are not part of the VALUES clauses
are placed at the beginning of the parameter tuple
- support for different identifiers for a numeric parameter. It's for example
possible to use postgresql style placeholder $1, $2, etc

Added two new dialect based on sqlite to test "numeric" fully using
both :1 style and $1 style. Includes a workaround for SQLite's
not-really-correct numeric implementation.

Changed parmstyle of asyncpg dialect to use numeric, rendering with its native
$ identifiers

Fixes: #8926
Fixes: #8849
Change-Id: I7c640467d49adfe6d795cc84296fc7403dcad4d6
2022-12-05 09:59:01 -05:00
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
Mike Bayer ad11c482e2 pep484 ORM / SQL result support
after some experimentation it seems mypy is more amenable
to the generic types being fully integrated rather than
having separate spin-off types.   so key structures
like Result, Row, Select become generic.  For DML
Insert, Update, Delete, these are spun into type-specific
subclasses ReturningInsert, ReturningUpdate, ReturningDelete,
which is fine since the "row-ness" of these constructs
doesn't happen until returning() is called in any case.

a Tuple based model is then integrated so that these
objects can carry along information about their return
types.  Overloads at the .execute() level carry through
the Tuple from the invoked object to the result.

To suit the issue of AliasedClass generating attributes
that are dynamic, experimented with a custom subclass
AsAliased, but then just settled on having aliased()
lie to the type checker and return `Type[_O]`, essentially.
will need some type-related accessors for with_polymorphic()
also.

Additionally, identified an issue in Update when used
"mysql style" against a join(), it basically doesn't work
if asked to UPDATE two tables on the same column name.
added an error message to the specific condition where
it happens with a very non-specific error message that we
hit a thing we can't do right now, suggest multi-table
update as a possible cause.

Change-Id: I5eff7eefe1d6166ee74160b2785c5e6a81fa8b95
2022-04-27 14:46:36 -04:00
Hugo van Kemenade 146a349d81 Update Black's target-version to py37
<!-- Provide a general summary of your proposed changes in the Title field above -->

### Description
<!-- Describe your changes in detail -->

Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.

Also update Black and other pre-commit hooks and re-format with Black.

### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

**Have a nice day!**

Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570

Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
2022-01-05 12:41:32 -05:00
Federico Caselli 0b95f0055b Remove object in class definition
References: #4600
Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
2021-11-22 15:03:17 +00:00
Mike Bayer 18b4a3437a process bulk_update_tuples before cache key or compilation
Fixed regression where the use of a :class:`_orm.hybrid_property` attribute
or a mapped :func:`_orm.composite` attribute as a key passed to the
:meth:`_dml.Update.values` method for an ORM-enabled :class:`_dml.Update`
statement, as well as when using it via the legacy
:meth:`_orm.Query.update` method, would be processed for incoming
ORM/hybrid/composite values within the compilation stage of the UPDATE
statement, which meant that in those cases where caching occurred,
subsequent invocations of the same statement would no longer receive the
correct values. This would include not only hybrids that use the
:meth:`_orm.hybrid_property.update_expression` method, but any use of a
plain hybrid attribute as well. For composites, the issue instead caused a
non-repeatable cache key to be generated, which would break caching and
could fill up the statement cache with repeated statements.

The :class:`_dml.Update` construct now handles the processing of key/value
pairs passed to :meth:`_dml.Update.values` and
:meth:`_dml.Update.ordered_values` up front when the construct is first
generated, before the cache key has been generated so that the key/value
pairs are processed each time, and so that the cache key is generated
against the individual column/value pairs that will ultimately be
used in the statement.

Fixes: #7209
Change-Id: I08f248d1d60ea9690b014c21439b775d951fb9e5
2021-10-19 15:10:14 -04:00
Federico Caselli debeea97ee Update black flak8 and zimports
Change-Id: I488c9557eda390e4a88319affd4c8813ee274f80
2021-05-12 22:10:19 +02:00
Mike Bayer 8860117c96 introduce generalized decorator to prevent invalid method calls
This introduces the ``_exclusive_against()`` utility decorator
that can be used to prevent repeated invocations of methods that
typically should only be called once.

An informative error message is now raised for a selected set of DML
methods (currently all part of :class:`_dml.Insert` constructs) if they are
called a second time, which would implicitly cancel out the previous
setting.  The methods altered include:
:class:`_sqlite.Insert.on_conflict_do_update`,
:class:`_sqlite.Insert.on_conflict_do_nothing` (SQLite),
:class:`_postgresql.Insert.on_conflict_do_update`,
:class:`_postgresql.Insert.on_conflict_do_nothing` (PostgreSQL),
:class:`_mysql.Insert.on_duplicate_key_update` (MySQL)

Fixes: #5169
Change-Id: I9278fa87cd3470dcf296ff96bb0fb17a3236d49d
2021-01-16 18:44:21 -05:00
Mike Bayer ba5cbf9366 correct for "autocommit" deprecation warning
Ensure no autocommit warnings occur internally or
within tests.

Also includes fixes for SQL Server full text tests
which apparently have not been working at all for a long
time, as it used long removed APIs.  CI has not had
fulltext running for some years and is now installed.

Change-Id: Id806e1856c9da9f0a9eac88cebc7a94ecc95eb96
2020-12-11 13:26:05 -05:00
Mike Bayer f483573aa6 Scan for tables without relying upon whereclause
Fixed bug where an UPDATE statement against a JOIN using MySQL multi-table
format would fail to include the table prefix for the target table if the
statement had no WHERE clause, as only the WHERE clause were scanned to
detect a "multi table update" at that particular point.  The target
is now also scanned if it's a JOIN to get the leftmost table as the
primary table and the additional entries as additional FROM entries.

Fixes: #5617
Change-Id: I26d74afebe06e28af28acf960258f170a1627823
2020-09-29 16:46:56 -04:00
Mike Bayer c3f102c9fe upgrade to black 20.8b1
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.

Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
2020-09-28 15:17:26 -04:00
Mike Bayer 7e864fc7b1 Create a framework to allow all SQLALCHEMY_WARN_20 to pass
As the test suite has widespread use of many patterns
that are deprecated, enable SQLALCHEMY_WARN_20 globally
for the test suite but then break the warnings filter
out into a whole list of all the individual warnings
we are looking for.  this way individual changesets
can target a specific class of warning, as many of these
warnings will indivdidually affect dozens of files
and potentially hundreds of lines of code.

Many warnings are also resolved here as this
patch started out that way.   From this point
forward there should be changesets that target a
subset of the warnings at a time.

For expediency, updates some migration 2.0 docs
for ORM as well.

Change-Id: I98b8defdf7c37b818b3824d02f7668e3f5f31c94
2020-09-16 12:31:05 -04:00
Federico Caselli e860060866 Update select usage to use the new 1.4 format
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function.  it does not
yet fully address other issues such as keyword arguments passed
to the table.select().

Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.

Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
2020-09-08 17:13:48 -04:00
Mike Bayer 65da699109 Sweep through UPDATE ordered_values a second time
The fix in 180ae7c1a5
didn't do much as the code was not preserving parameter
order at all, in fact.    Reworked stmt_parameters to be
delivered in the correct order up front and preserve
throughout crud.py which was not being done at all
before.

Fixes: #5510
Change-Id: I0795c71df73005a25d1bbf216732d41b41e11a5f
2020-08-12 22:00:43 -04:00
Mike Bayer 180ae7c1a5 Correct for update.ordered_values() and use in new test
The test for the parameter fix in c0685e5f41 was not working
deterministically on Python 2, so use ordered_values(), however
a second issue in ordered_values() was assuming each element was a
column, so also test for array-assignment expressions with
ordered_values.

Change-Id: I944c72a52700ffb4ab5ae1a83ae21f1efc84b505
2020-08-11 17:46:11 -04:00
Mike Bayer c0685e5f41 render INSERT/UPDATE column expressions up front; pass state
Fixes related to rendering of complex UPDATE DML
which was not correctly preserving positional parameter
order in conjunction with DML features that are only known
to work on the PostgreSQL database.    Both pg8000
and asyncpg use positional parameters which is why these
issues are suddenly apparent.

crud.py now takes on the task of rendering the column
expressions for SET or VALUES so that for the very unusual
case that the column expression is a compound expression
that includes a bound parameter (namely an array index),
the bound parameter order is preserved.

Additionally, crud.py passes through the positional_names
keyword argument into bindparam_string() which is necessary
when CTEs are being rendered, as PG supports complex
CTE / INSERT / UPDATE scenarios.

Change-Id: I7f03920500e19b721636b84594de78a5bfdcbc82
2020-08-08 13:34:27 -04:00
Mike Bayer f1a3038f48 Default psycopg2 executemany mode to "values_only"
The psycopg2 dialect now defaults to using the very performant
``execute_values()`` psycopg2 extension for compiled INSERT statements,
and also impements RETURNING support when this extension is used.  This
allows INSERT statements that even include an autoincremented SERIAL
or IDENTITY value to run very fast while still being able to return the
newly generated primary key values.   The ORM will then integrate this
new feature in a separate change.

Implements RETURNING for insert with executemany

Adds support to return_defaults() mode and inserted_primary_key
to support mutiple INSERTed rows, via return_defauls_rows
and inserted_primary_key_rows accessors.

within default execution context, new cached compiler
getters are used to fetch primary keys from rows

inserted_primary_key now returns a plain tuple. this
is not yet a row-like object however this can be
added.

Adds distinct "values_only" and "batch" modes, as
"values" has a lot of benefits but "batch" breaks
cursor.rowcount

psycopg2 minimum version 2.7 so we can remove the
large number of checks for very old versions of
psycopg2

simplify tests to no longer distinguish between
native and non-native json

Fixes: #5401
Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
2020-06-25 18:58:34 -04:00
Mike Bayer b0cfa7379c Turn on caching everywhere, add logging
A variety of caching issues found by running
all tests with statement caching turned on.

The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.

Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.

Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *".   if a table re-ordered its columns
or added/removed, now that data is obsolete.

Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.

Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement.   Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.

Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.

mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.

lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly.   While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.

turn on cache stats in logging.

Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context.   This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.

DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced.  this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.

memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.

Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason.   Removed
these methods as well as the concept of a Compiler
holding onto a "bind".

Fixes: #5386

Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
2020-06-10 15:29:01 -04:00
Mike Bayer 851fb8f5a6 Decouple compiler state from DML objects; make cacheable
Targeting select / insert / update / delete, the goal
is to minimize overhead of construction and generative methods
so that only the raw arguments passed are handled.   An interim
stage that converts the raw state into more compiler-ready state
is added, which is analogous to the ORM QueryContext which will
also be rolled in to be a similar concept, as is currently
being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10.
the ORM update/delete BulkUD concept is also going to be rolled
onto this idea.   So while the compiler-ready state object,
here called DMLState, looks a little thin, it's the
base of a bigger pattern that will allow for ORM functionality
to embed itself directly into the compiler, execution
context, and result set objects.

This change targets the DML objects, primarily focused on the
values() method which is the most complex process.   The
work done by values() is minimized as much as possible
while still being able to create a cache key.   Additional
computation is then offloaded to a new object ValuesState
that is handled by the compiler.

Architecturally, a big change here is that insert.values()
and update.values() will generate BindParameter objects for
the values now, which are then carefully received by crud.py
so that they generate the expected names.   This is so that
the values() portion of these constructs is cacheable.
for the "multi-values" version of Insert, this is all skipped
and the plan right now is that a multi-values insert is
not worth caching (can always be revisited).

Using the
coercions system in values() also gets us nicer validation
for free, we can remove the NotAClauseElement thing from
schema, and we also now require scalar_subquery() is called
for an insert/update that uses a SELECT as a column value,
1.x deprecation path is added.

The traversal system is then applied to the DML objects
including tests so that they have traversal, cloning, and
cache key support.  cloning is not a use case for DML however
having it present allows better validation of the structure
within the tests.

Special per-dialect DML is explicitly not cacheable at the moment,
more as a proof of concept that third party DML constructs can
exist as gracefully not-cacheable rather than producing an
incomplete cache key.

A few selected performance improvements have been added as well,
simplifying the immutabledict.union() method and adding
a new SQLCompiler function that can generate delimeter-separated
clauses like WHERE and ORDER BY without having to build
a ClauseList object at all.   The use of ClauseList will
be removed from Select in an upcoming commit.  Overall,
ClaustList is unnecessary for internal use and only adds
overhead to statement construction and will likely be removed
as much as possible except for explcit use of conjunctions like
and_() and or_().

Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
2020-03-06 11:01:51 -05:00
Mike Bayer ffc375d1f0 Repair inline flag
In 9fca5d827d we attempted to deprecate the "inline=True" flag
and add a generative inline() method, however failed to include
any tests and the method was implemented incorrectly such that
it would get overwritten with the boolean flag immediately.

Rename the internal "inline" flag to "_inline" and add test
support both for the method as well as deprecated support
for the flag, including a fixture addition to assert the expected
value of the flag as it generally does not affect the
actual compiled SQL string.

Change-Id: I0450049f17f1f0d91e22d27f1a973a2b6c0e59f7
2020-02-22 11:02:56 -05:00
Mike Bayer 9fca5d827d Create initial future package, RemovedIn20Warning
Reorganization of Select() is the first major element
of the 2.0 restructuring.   In order to start this we need
to first create the new Select constructor and apply legacy
elements to the old one.    This in turn necessitates
starting up the RemovedIn20Warning concept which itself
need to refer to "sqlalchemy.future", so begin to establish
this basic framework.   Additionally, update the
DML constructors with the newer no-keyword style.  Remove
the use of the "pending deprecation" and fix Query.add_column()
deprecation which was not acting as deprecated.

Fixes: #4845
Fixes: #4648
Change-Id: I0c7a22b2841a985e1c379a0bb6c94089aae6264c
2020-02-12 12:44:47 -05:00
Federico Caselli 1de64504d8 Deprecate empty or_() and and_()
Creating an :func:`.and_` or :func:`.or_` construct with no arguments or
empty ``*args`` will now emit a deprecation warning, as the SQL produced is
a no-op (i.e. it renders as a blank string). This behavior is considered to
be non-intuitive, so for empty or possibly empty :func:`.and_` or
:func:`.or_` constructs, an appropriate default boolean should be included,
such as ``and_(True, *args)`` or ``or_(False, *args)``.   As has been the
case for many major versions of SQLAlchemy, these particular boolean
values will not render if the ``*args`` portion is non-empty.

As there are some internal cases where an empty and_() construct is used
in order to build an optional WHERE expression, a private
utility function is added to suit this use case.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5054
Closes: #5062
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062
Pull-request-sha: 5ca2f27281

Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
2020-01-25 18:03:48 -05:00
Mike Bayer 41dc71ad2f Use separate label generator for column_label naming convention
Fixed bug where a table that would have a column label overlap with a plain
column name, such as "foo.id AS foo_id" vs. "foo.foo_id", would prematurely
generate the ``._label`` attribute for a column before this overlap could
be detected due to the use of the ``index=True`` or ``unique=True`` flag on
the column in conjunction with the default naming convention of
``"column_0_label"``.  This would then lead to failures when ``._label``
were used later to generate a bound parameter name, in particular those
used by the ORM when generating the WHERE clause for an UPDATE statement.
The issue has been fixed by using an alternate ``._label`` accessor for DDL
generation that does not affect the state of the :class:`.Column`.   The
accessor also bypasses the key-deduplication step as it is not necessary
for DDL, the naming is now consistently ``"<tablename>_<columnname>"``
without any subsequent numeric symbols when used in DDL.

Fixes: #4911

Change-Id: Iabf5fd3250738d800d6e41a2a3a27a7ce2405e7d
2019-10-14 12:51:14 -04:00
Mike Bayer f07e050c9c Implement new ClauseElement role and coercion system
A major refactoring of all the functions handle all detection of
Core argument types as well as perform coercions into a new class hierarchy
based on "roles", each of which identify a syntactical location within a
SQL statement.  In contrast to the ClauseElement hierarchy that identifies
"what" each object is syntactically, the SQLRole hierarchy identifies
the "where does it go" of each object syntactically.   From this we define
a consistent type checking and coercion system that establishes well
defined behviors.

This is a breakout of the patch that is reorganizing select()
constructs to no longer be in the FromClause hierarchy.

Also includes a rename of as_scalar() into scalar_subquery(); deprecates
automatic coercion to scalar_subquery().

Partially-fixes: #4617
Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
2019-05-18 17:46:10 -04:00
Mike Bayer 1e278de4cc Post black reformatting
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9,  this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.

Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.

Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
2019-01-06 18:23:11 -05:00
Mike Bayer 1e1a38e780 Run black -l 79 against all source files
This is a straight reformat run using black as is, with no edits
applied at all.

The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines.   The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.

Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
2019-01-06 17:34:50 +00:00
Mike Bayer 87cdda0086 Move CRUDTest, InlineDefaultTest from test_compiler
test_compiler is mostly related to SELECT statements as well
as smaller SQL elements.  While it still has some DDL related
tests, move out all the remaining insert/update tests into
the already present test_insert.py, test_update.py

Fixes: #2630
Change-Id: I4167618543fd1235d12d1717c8c629d2374b325a
2018-12-01 14:28:57 -05:00
Mike Bayer abeea1d82d Include UPDATE/DELETE extra_froms in correlation
Fixed bug where the multi-table support for UPDATE and DELETE statements
did not consider the additional FROM elements as targets for correlation,
when a correlated SELECT were also combined with the statement.  This
change now includes that a SELECT statement in the WHERE clause for such a
statement will try to auto-correlate back to these additional tables in the
parent UPDATE/DELETE or unconditionally correlate if
:meth:`.Select.correlate` is used.  Note that auto-correlation raises an
error if the SELECT statement would have no FROM clauses as a result, which
can now occur if the parent UPDATE/DELETE specifies the same tables in its
additional set of tables ; specify :meth:`.Select.correlate` explicitly to
resolve.

Change-Id: Ie11eaad7e49af3f59df11691b104d6359341bdae
Fixes: #4313
2018-08-04 13:46:19 -04:00
Mike Bayer 58540ae93d Support JOIN in UPDATE..FROM
The :class:`.Update` construct now accommodates a :class:`.Join` object
as supported by MySQL for UPDATE..FROM.  As the construct already
accepted an alias object for a similar purpose, the feature of UPDATE
against a non-table was already implied so this has been added.

Change-Id: I7b2bca627849384d5377abb0c94626463e4fad04
Fixes: #3645
2018-06-13 21:38:20 -04:00
inytar d12b37f90e Allow delete where clause to refer multiple tables.
Implemented "DELETE..FROM" syntax for Postgresql, MySQL, MS SQL Server
(as well as within the unsupported Sybase dialect) in a manner similar
to how "UPDATE..FROM" works.  A DELETE statement that refers to more than
one table will switch into "multi-table" mode and render the appropriate
"USING" or multi-table "FROM" clause as understood by the database.
Pull request courtesy Pieter Mulder.

For SQL syntaxes see:

Postgresql: https://www.postgresql.org/docs/current/static/sql-delete.html
MySQL: https://dev.mysql.com/doc/refman/5.7/en/delete.html#multiple-table_syntax
MSSQL: https://docs.microsoft.com/en-us/sql/t-sql/statements/delete-transact-sql
Sybase: http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc00801.1510/html/iqrefso/X315721.htm

Co-authored by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I6dfd57b49e44a095d076dc493cd2360bb5d920d3
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/392
Fixes: #959
2017-12-05 14:26:28 -05:00
Khairi Hafsham 772374735d Make all tests to be PEP8 compliant
tested using pycodestyle version 2.2.0

Fixes: #3885
Change-Id: I5df43adc3aefe318f9eeab72a078247a548ec566
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/343
2017-02-07 11:21:56 -05:00
Mike Bayer fa6dd376bb Support python3.6
Corrects some warnings and adds tox config.  Adds DeprecationWarning
to the error category.   Large sweep for string literals w/ backslashes
as this is common in docstrings

Co-authored-by: Andrii Soldatenko
Fixes: #3886
Change-Id: Ia7c838dfbbe70b262622ed0803d581edc736e085
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/337
2017-01-13 10:57:41 -05:00
Ville Skyttä 8c2c464cb8 spelling: Postgresql -> PostgreSQL 2016-10-08 20:42:50 +03:00
Mike Bayer 02190234a2 - a variety of test adjustments to accomodate for MySQL 5.7
Change-Id: Ied4245433d0d7b469dae6e7394c4931d8405f387
2016-07-24 16:36:27 -04:00
Mike Bayer 6b7c207801 - move out unconsumed names tests from test_compiler out to test_insert, test_update
- establish consistent names between existing unconsumed names tests and new ones
added per ref #3666
2016-03-02 20:24:49 -05:00
Mike Bayer c90f0a49f3 - Added support for parameter-ordered SET clauses in an UPDATE
statement.  This feature is available by passing the
:paramref:`~.sqlalchemy.sql.expression.update.preserve_parameter_order`
flag either to the core :class:`.Update` construct or alternatively
adding it to the :paramref:`.Query.update.update_args` dictionary at
the ORM-level, also passing the parameters themselves as a list of 2-tuples.
Thanks to Gorka Eguileor for implementation and tests.
adapted from pullreq github:200
2015-11-28 14:30:05 -05:00
Mike Bayer bb5f4392a4 - update the flake8 rules again
- apply autopep8 + manual fixes to most of test/sql/
2014-07-18 17:40:58 -04:00
Mike Bayer 55eacc8dbe - Fixed bug where :meth:.Table.update and :meth:.Table.delete
would produce an empty WHERE clause when an empty :func:`.and_()`
or :func:`.or_()` or other blank expression were applied.  This is
now consistent with that of :func:`.select`.
fixes #3045
2014-05-08 22:44:06 -04:00
Mike Bayer 2f150bee28 - rename __multiple__ to __backend__, and apply __backend__ to a large number of tests.
- move out logging tests from test_execute to test_logging
2014-03-24 11:33:53 -04:00
Mike Bayer 5d2bfc4df4 - Fixed a few errant `u''` strings that would prevent tests from passing
in Py3.2.  Patch courtesy Arfrever Frehtes Taifersar Arahesis. fixes #2980
2014-03-22 18:48:59 -04:00
Mike Bayer 6aeec027a0 - Adjusted the logic which applies names to the .c collection when
a no-name :class:`.BindParameter` is received, e.g. via :func:`.sql.literal`
or similar; the "key" of the bind param is used as the key within
.c. rather than the rendered name.  Since these binds have "anonymous"
names in any case, this allows individual bound parameters to
have their own name within a selectable if they are otherwise unlabeled.
fixes #2974
2014-02-26 15:45:52 -05:00