Commit Graph

98 Commits

Author SHA1 Message Date
Mike Bayer 9a6854223c fix tests for aggregate_order_by
forgot this doesnt work on sqlite older than 3.44

Change-Id: Icebc2ffaceb078a436a8f61ba583bc05b77f680d
2025-09-19 15:18:39 -04:00
Reuven Starodubski 459ebc668a Add FunctionElement.aggregate_order_by
Added new generalized aggregate function ordering to functions via the
:func:`_functions.FunctionElement.aggregate_order_by` method, which
receives an expression and generates the appropriate embedded "ORDER BY" or
"WITHIN GROUP (ORDER BY)" phrase depending on backend database.  This new
function supersedes the use of the PostgreSQL
:func:`_postgresql.aggregate_order_by` function, which remains present for
backward compatibility.   To complement the new parameter, the
:paramref:`_functions.aggregate_strings.order_by` which adds ORDER BY
capability to the :class:`_functions.aggregate_strings` dialect-agnostic
function which works for all included backends. Thanks much to Reuven
Starodubski with help on this patch.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #12853
Closes: #12856
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12856
Pull-request-sha: d93fb591751227eb1f96052ea3ad449f511f70b3

Change-Id: I8eb41ff2d57695963a358b5f0017ca9372f15f70
2025-09-19 09:38:57 -04:00
Kaan 7e28adbe0c Implement GROUPS frame spec for window functions
Implemented support for the GROUPS frame specification in window functions
by adding :paramref:`_sql.over.groups` option to :func:`_sql.over`
and :meth:`.FunctionElement.over`. Pull request courtesy Kaan Dikmen.

Fixes: #12450
Closes: #12445
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12445
Pull-request-sha: c0808e135f

Change-Id: I9ff504a9c9650485830c4a0eaf44162898a3a2ad
2025-03-27 13:34:52 +00:00
Federico Caselli d539bc3a0f update the format_docs_code to also work on python files
Change-Id: I0a6c9610b3fd85365ed4c2c199e3cad87ee64022
2024-11-30 21:10:21 +01:00
Michael Bayer 6f704cac01 Merge "Add missing function element methods" into main 2024-06-03 14:24:46 +00:00
Federico Caselli 346e7ea0d0 Make FunctionFilter.filter generative
Fixed bug in :meth:`_sql.FunctionFilter.filter` that would mutate
the existing function in-place. It now behaves like the rest of the
SQLAlchemy API, returning a new instance instead of mutating the
original one.

Fixes: #11426
Change-Id: I46ffebaed82426cfb1623db066686cfb911055a1
2024-05-30 22:25:33 +02:00
Federico Caselli 57bba09659 Add missing function element methods
Added missing methods :meth:`_sql.FunctionFilter.within_group`
and :meth:`_sql.WithinGroup.filter`

Fixes: #11423
Change-Id: I4bafd9e3cab5883b28b2b997269df239739a2212
2024-05-30 22:17:48 +02:00
Joshua Morris d24048a8fb Add support for SQL string aggregation function aggregate_strings.
Add support for SQL string aggregation function :class:`.aggregate_strings`.
Pull request curtesy Joshua Morris.

Fixes #9873
Closes: #9892
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/9892
Pull-request-sha: 0f4f83bade

Change-Id: I6e4afc83664a142e3b1e245978b08a200b6d03d9
2023-09-14 15:49:29 -04:00
Federico Caselli 058c230cea Update black to v23.3.0 and flake8 to v6
This change could be added to .git-blame-ignore-revs

Change-Id: I7ba10052b26bc3c178d23fb50a1123d0aae965ca
2023-05-30 21:44:26 +02:00
Mike Bayer da70478eb2 ensure single import per line
This adds the very small plugin flake8-import-single which
will prevent us from having an import with more than one symbol
on a line.

Flake8 by itself prevents this pattern with E401:

import collections, os, sys

However does not do anything with this:

from sqlalchemy import Column, text

Both statements have the same issues generating merge artifacts
as well as presenting a manual decision to be made.   While
zimports generally cleans up such imports at the top level, we
don't enforce zimports / pre-commit use.

the plugin finds the same issue for imports that are inside of
test methods.   We shouldn't usually have imports in test methods
so most of them here are moved to be top level.

The version is pinned at 0.1.5; the project seems to have no
activity since 2019, however there are three 0.1.6dev releases
on pypi which stopped in September 2019, they seem to be
experiments with packaging.  The source for 0.1.5
is extremely simple and only reveals one method to flake8
(the run() method).

Change-Id: Icea894e43bad9c0b5d4feb5f49c6c666d6ea6aa1
2023-02-28 11:50:54 -05:00
Federico Caselli 0f2baae6bf Fix positional compiling bugs
Fixed a series of issues regarding positionally rendered bound parameters,
such as those used for SQLite, asyncpg, MySQL and others. Some compiled
forms would not maintain the order of parameters correctly, such as the
PostgreSQL ``regexp_replace()`` function as well as within the "nesting"
feature of the :class:`.CTE` construct first introduced in 🎫`4123`.

Fixes: #8827
Change-Id: I9813ed7c358cc5c1e26725c48df546b209a442cb
2022-12-01 23:50:30 +01:00
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
Federico Caselli 974b1bd0fc Revert automatic set of sequence start to 1
The :class:`.Sequence` construct restores itself to the DDL behavior it
had prior to the 1.4 series, where creating a :class:`.Sequence` with
no additional arguments will emit a simple ``CREATE SEQUENCE`` instruction
**without** any additional parameters for "start value".   For most backends,
this is how things worked previously in any case; **however**, for
MS SQL Server, the default value on this database is
``-2**63``; to prevent this generally impractical default
from taking effect on SQL Server, the :paramref:`.Sequence.start` parameter
should be provided.   As usage of :class:`.Sequence` is unusual
for SQL Server which for many years has standardized on ``IDENTITY``,
it is hoped that this change has minimal impact.

Fixes: #7211
Change-Id: I1207ea10c8cb1528a1519a0fb3581d9621c27b31
2022-10-17 15:36:25 -04:00
Federico Caselli 02fe382d6b Improve array_agg and Array processing
The :class:`_functions.array_agg` will now set the array dimensions to 1.
Improved :class:`_types.ARRAY` processing to accept ``None`` values as
value of a multi-array.

Fixes: #7083
Change-Id: Iafec4f77fde9719ccc7c8535bf6235dbfbc62102
2022-09-17 13:12:35 +02:00
Mike Bayer fbdf825192 render select froms first
The FROM clauses that are established on a :func:`_sql.select` construct
when using the :meth:`_sql.Select.select_from` method will now render first
in the FROM clause of the rendered SELECT, which serves to maintain the
ordering of clauses as was passed to the :meth:`_sql.Select.select_from`
method itself without being affected by the presence of those clauses also
being mentioned in other parts of the query. If other elements of the
:class:`_sql.Select` also generate FROM clauses, such as the columns clause
or WHERE clause, these will render after the clauses delivered by
:meth:`_sql.Select.select_from` assuming they were not explictly passed to
:meth:`_sql.Select.select_from` also. This improvement is useful in those
cases where a particular database generates a desirable query plan based on
a particular ordering of FROM clauses and allows full control over the
ordering of FROM clauses.

Fixes: #7888
Change-Id: I740f262a3841f829239011120a59b5e58452db5b
2022-05-22 19:24:18 +00:00
Federico Caselli 11bf824474 update flake8 noqa skips with proper syntax
Change-Id: I42ed77f559e3ee5b8c600d98457ee37803ef0ea6
2022-04-11 22:48:23 +02:00
Mike Bayer c315c7401a TableValuedAlias generation fixes
Fixed bug in newly implemented
:paramref:`.FunctionElement.table_valued.joins_implicitly` feature where
the parameter would not automatically propagate from the original
:class:`.TableValuedAlias` object to the secondary object produced when
calling upon :meth:`.TableValuedAlias.render_derived` or
:meth:`.TableValuedAlias.alias`.

Additionally repaired these issues in :class:`.TableValuedAlias`:

* repaired a potential memory issue which could occur when
repeatedly calling :meth:`.TableValuedAlias.render_derived` against
successive copies of the same object (for .alias(), we currently
have to still continue chaining from the previous element.  not sure
if this can be improved but this is standard behavior for .alias()
elsewhere)
* repaired issue where the individual element types would be lost when
calling upon :meth:`.TableValuedAlias.render_derived` or
:meth:`.TableValuedAlias.alias`.

Fixes: #7890
Change-Id: Ie5120c7ff1e5c1bba5aaf77c782a51c637860208
2022-04-03 11:30:09 -04:00
Mike Bayer 22deafe152 Warn when caching is disabled / document
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.

As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``.  These have been amended to use inherit
cache where appropriate.  "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.

The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.

Fixes: #7394
Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
2021-12-06 18:27:19 -05:00
Federico Caselli 31acba8ff7 Clean up most py3k compat
Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
2021-11-24 22:51:27 -05:00
Federico Caselli ed78e679ea Remove deprecated dialects and drivers
Fixes: #7258
Change-Id: I3577f665eca04f2632b69bcb090f0a4ec9271db9
2021-10-31 12:31:56 -04:00
Mike Bayer 53ad3cf4e9 Pickling fixes for ORM / Core
Fixed regression where ORM loaded objects could not be pickled in cases
where loader options making use of ``"*"`` were used in certain
combinations, such as combining the :func:`_orm.joinedload` loader strategy
with ``raiseload('*')`` of sub-elements.

Fixes: #7134

Fixed issue where SQL queries using the
:meth:`_functions.FunctionElement.within_group` construct could not be
pickled, typically when using the ``sqlalchemy.ext.serializer`` extension
but also for general generic pickling.

Fixes: #6520

Change-Id: Ib73fd49c875e6da9898493c190f610e68b88ec72
2021-10-13 13:52:23 -04:00
Mike Bayer 115e4d1a49 Apply quoting to render_derived() names
Fixed bug in new :meth:`_functions.FunctionElement.render_derived` feature
where column names rendered out explicitly in the alias SQL would not have
proper quoting applied for case sensitive names and other non-alphanumeric
names.

Fixes: #6183
Change-Id: I33e2534affc6e1f449f564750028fd027cb0f352
2021-04-01 13:41:44 -04:00
Mike Bayer e67d1b79c5 Deannoate functions before matching .__class__
Fixed regression where the SQL compilation of a :class:`.Function` would
not work correctly if the object had been "annotated", which is an internal
memoization process used mostly by the ORM. In particular it could affect
ORM lazy loads which make greater use of this feature in 1.4.

Fixes: #6095
Change-Id: I7a6527df651f440a04d911ba78ee0b0dd4436dcd
2021-03-19 01:19:32 -04:00
Mike Bayer 05f34fc29c test Function(?:Element)._bind_param() with in_()
Fixed 1.4 regression where the :meth:`_functions.Function.in_` method was
not covered by tests and failed to function properly in all cases.

Fixes: #5934
Change-Id: I93423a296e391aabd5594cb670d36b91ced0231d
2021-02-11 17:53:29 -05:00
Mike Bayer afcab5edf6 Implement support for functions as FROM with columns clause support
Implemented support for "table valued functions" along with additional
syntaxes supported by PostgreSQL, one of the most commonly requested
features. Table valued functions are SQL functions that return lists of
values or rows, and are prevalent in PostgreSQL in the area of JSON
functions, where the "table value" is commonly referred towards as the
"record" datatype. Table valued functions are also supported by Oracle and
SQL Server.

Moved from I5b093b72533ef695293e737eb75850b9713e5e03 due
to accidental push

Fixes: #3566
Change-Id: Iea36d04c80a5ed3509dcdd9ebf0701687143fef5
2021-02-03 15:52:17 -05:00
Gord Thompson 22f65156bb Replace with_labels() and apply_labels() in ORM/Core
Replace :meth:`_orm.Query.with_labels` and
:meth:`_sql.GenerativeSelect.apply_labels` with explicit getters and
setters ``get_label_style`` and ``set_label_style`` to accommodate the
three supported label styles: ``LABEL_STYLE_DISAMBIGUATE_ONLY`` (default),
``LABEL_STYLE_TABLENAME_PLUS_COL``, and ``LABEL_STYLE_NONE``.

In addition, for Core and "future style" ORM queries,
``LABEL_STYLE_DISAMBIGUATE_ONLY`` is now the default label style. This
style differs from the existing "no labels" style in that labeling is
applied in the case of column name conflicts; with ``LABEL_STYLE_NONE``, a
duplicate column name is not accessible via name in any case.

For legacy ORM queries using :class:`_query.Query`, the table-plus-column
names labeling style applied by ``LABEL_STYLE_TABLENAME_PLUS_COL``
continues to be used so that existing test suites and logging facilities
see no change in behavior by default, however this style of labeling is no
longer required for SQLAlchemy queries to function, as result sets are
commonly matched to columns using a positional approach since SQLAlchemy
1.0.

Within test suites, all use of apply_labels()  / use_labels
now uses the new methods.    New tests added to
test/sql/test_deprecations.py nad test/orm/test_deprecations.py
to cover just the old apply_labels() method call.  Tests
in ORM that made explicit use apply_labels()/ etc. where it isn't needed
for the ORM to work correctly use default label style now.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #4757
Change-Id: I5fdcd2ed4ae8c7fe62f8be2b6d0e8f66409b6a54
2021-01-26 16:52:30 -05:00
Mike Bayer 851a3a362e Revert "Implement support for functions as FROM with columns clause support"
This reverts commit 05a31f2708.

Atom has this little button called "push" and just pushes to master,
I wasn't even *on* master.  oops
2021-01-21 11:15:06 -05:00
Mike Bayer 05a31f2708 Implement support for functions as FROM with columns clause support
WIP

Fixes: #3566
Change-Id: I5b093b72533ef695293e737eb75850b9713e5e03
2021-01-20 23:30:21 -05:00
Mike Bayer ebbbac0a76 update execute() arg formats in modules and tests
continuing with producing a SQLAlchemy 1.4.0b2 that internally
does not emit any of its own 2.0 deprecation warnings,
migrate the *args and **kwargs passed to execute() methods
that now must be a single list or dictionary.

Alembic 1.5 is again waiting on this internal consistency to
be present so that it can pass all tests with no 2.0
deprecation warnings.

Change-Id: If6b792e57c8c5dff205419644ab68e631575a2fa
2021-01-15 13:04:58 -05:00
Mike Bayer f1e96cb087 reinvent xdist hooks in terms of pytest fixtures
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones.   Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them.   A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable.   New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.

From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow.  A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.

As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively.  The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.

Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session.   The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic.  A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function.  The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all.  Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.

As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.

An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2.  It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.

py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2.  For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.

Includes the following improvements:

Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`.  Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.

For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.

Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board.  the addition
of this flag has resolved the long-standing oracle "two task"
error problem.

For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically.  There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite.  Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.

Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.

Fixes: #5826
Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
2021-01-13 22:10:13 -05:00
Mike Bayer 6fccdf4a28 remove more bound metadata
in Iae6ab95938a7e92b6d42086aec534af27b5577d3 I missed
that the "bind" was being stuck onto the MetaData in
TablesTest, which led thousands of ORM tests to still use
bound metadata.  Keep looking for bound metadata.

standardize all ORM tests on a single means of getting a
Session when the Session API isn't the thing we are directly
testing, using a new function fixture_session() that replaces
create_session() and uses modern defaults.

Change-Id: Iaf71206e9ee568151496d8bc213a069504bf65ef
2021-01-05 23:46:02 -05:00
Mike Bayer 9322de12f5 Repair reduction in Over
Fixed issue where a plain pickle dumps call of the :class:`_sql.Over`
construct didn't work.

Fixes: #5644
Change-Id: I4b07f74ecd5d52f0794128585367012200a38a36
2020-10-12 12:48:17 -04:00
Mike Bayer c3f102c9fe upgrade to black 20.8b1
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.

Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
2020-09-28 15:17:26 -04:00
Mike Bayer 7e864fc7b1 Create a framework to allow all SQLALCHEMY_WARN_20 to pass
As the test suite has widespread use of many patterns
that are deprecated, enable SQLALCHEMY_WARN_20 globally
for the test suite but then break the warnings filter
out into a whole list of all the individual warnings
we are looking for.  this way individual changesets
can target a specific class of warning, as many of these
warnings will indivdidually affect dozens of files
and potentially hundreds of lines of code.

Many warnings are also resolved here as this
patch started out that way.   From this point
forward there should be changesets that target a
subset of the warnings at a time.

For expediency, updates some migration 2.0 docs
for ORM as well.

Change-Id: I98b8defdf7c37b818b3824d02f7668e3f5f31c94
2020-09-16 12:31:05 -04:00
mike bayer dad484e7fa Merge "Do not specify type on mssql by default" 2020-09-12 16:26:39 +00:00
Federico Caselli 1d40a4ae15 Do not specify type on mssql by default
Make optional sequences render as identity in mssql
Remove unused dialect option sequence_default_column_type

Change-Id: I821eeffcb442f8d1b69186a9b798b15c3d8d6ff3
2020-09-12 10:00:23 -04:00
Federico Caselli e860060866 Update select usage to use the new 1.4 format
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function.  it does not
yet fully address other issues such as keyword arguments passed
to the table.select().

Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.

Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
2020-09-08 17:13:48 -04:00
mike bayer ba047cc8ca Merge "Fix a wide variety of typos and broken links" 2020-06-26 15:30:57 +00:00
aplatkouski 2a1a9f5f5a Fix a wide variety of typos and broken links
Note the PR has a few remaining doc linking issues
listed in the comment that must be addressed separately.

Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com>
Closes: #5371
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371
Pull-request-sha: 7e7d233cf3

Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
2020-06-25 19:42:28 -04:00
Mike Bayer f1a3038f48 Default psycopg2 executemany mode to "values_only"
The psycopg2 dialect now defaults to using the very performant
``execute_values()`` psycopg2 extension for compiled INSERT statements,
and also impements RETURNING support when this extension is used.  This
allows INSERT statements that even include an autoincremented SERIAL
or IDENTITY value to run very fast while still being able to return the
newly generated primary key values.   The ORM will then integrate this
new feature in a separate change.

Implements RETURNING for insert with executemany

Adds support to return_defaults() mode and inserted_primary_key
to support mutiple INSERTed rows, via return_defauls_rows
and inserted_primary_key_rows accessors.

within default execution context, new cached compiler
getters are used to fetch primary keys from rows

inserted_primary_key now returns a plain tuple. this
is not yet a row-like object however this can be
added.

Adds distinct "values_only" and "batch" modes, as
"values" has a lot of benefits but "batch" breaks
cursor.rowcount

psycopg2 minimum version 2.7 so we can remove the
large number of checks for very old versions of
psycopg2

simplify tests to no longer distinguish between
native and non-native json

Fixes: #5401
Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
2020-06-25 18:58:34 -04:00
Gord Thompson 668872fe01 Add support for "real" sequences in mssql
Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for
Microsoft SQL Server.  This removes the deprecated feature of using
:class:`.Sequence` objects to manipulate IDENTITY characteristics which
should now be performed using ``mssql_identity_start`` and
``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The
change includes a new parameter :paramref:`.Sequence.data_type` to
accommodate SQL Server's choice of datatype, which for that backend
includes INTEGER and BIGINT.   The default starting value for SQL Server's
version of :class:`.Sequence` has been set at 1; this default is now
emitted within the CREATE SEQUENCE DDL for all backends.

Fixes: #4235
Fixes: #4633
Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
2020-05-29 08:10:38 -06:00
Gord Thompson 50cee43fbc Clean up .execute in test/sql/test_functions.py
Change-Id: I2bc7a50893f90c6ea7e119a8558731ee32965871
2020-04-20 07:46:24 -06:00
Mike Bayer a9b62055bf Try to measure new style caching in the ORM, take two
Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf

Trying to see if by making the cache key memoized, we
still can have the older "identity" form of caching
which is the cheapest of all, at the same time as the
newer "cache key each time" version that is not nearly
as cheap; but still much cheaper than no caching at all.

Also needed is a per-execution update of _keymap when
we invoke from a cached select, so that Column objects
that are anonymous or otherwise adapted will match up.
this is analogous to the adaption of bound parameters
from the cache key.

Adds test coverage for the keymap / construct_params()
 changes related to caching.  Also hones performance
to a large extent for statement construction and
cache key generation.

Also includes a new memoized attribute
approach that vastly simplifies the previous approach
of "group_expirable_memoized_property" and finally
integrates cleanly with _clone(), _generate(), etc.
no more hardcoding of attributes is needed, as well
as that most _reset_memoization() calls are no longer
needed as the reset is inherent in a _generate() call;
this also has dramatic performance improvements.

Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
2020-04-01 16:12:23 -04:00
Mike Bayer f559f378c4 Result initial introduction
This builds on cc718cccc0 which moved
RowProxy to Row, allowing Row to be more like a named tuple.

- KeyedTuple in ORM is replaced with Row

- ResultSetMetaData broken out into "simple" and "cursor" versions
  for ORM and Core, as well as LegacyCursor version.

- Row now has _mapping attribute that supplies full mapping behavior.
Row and SimpleRow both have named tuple behavior otherwise.
LegacyRow has some mapping features on the tuple which emit
deprecation warnings (e.g. keys(), values(), etc).   the biggest
change for mapping->tuple is the behavior of __contains__ which
moves from testing of "key in row" to "value in row".

- ResultProxy breaks into ResultProxy and FutureResult (interim),
the latter has the newer APIs.   Made available to dialects
using execution options.

- internal reflection methods and most tests move off of implicit
Row mapping behavior and move to row._mapping, result.mappings()
method using future result

- a new strategy system for cursor handling replaces the various
subclasses of RowProxy

- some execution context adjustments. We will leave EC in but
refined things like get_result_proxy() and out parameter handling.
Dialects for 1.4 will need to adjust from get_result_proxy()
to get_result_cursor_strategy(), if they are using this method

- out parameter handling now accommodated by get_out_parameter_values()
EC method.   Oracle changes for this.  external dialect for
DB2 for example will also need to adjust for this.

- deprecate case_insensitive flag for engine / result, this
feature is not used

mapping-methods on Row are deprecated, and replaced with
Row._mapping.<meth>, including:

   row.keys()  -> use row._mapping.keys()
   row.items()  -> use row._mapping.items()
   row.values() -> use row._mapping.values()
   key in row  -> use key in row._mapping
   int in row  -> use int < len(row)

Fixes: #4710
Fixes: #4878
Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
2020-02-21 17:53:33 -05:00
Mike Bayer a697fcc1cb Support GenericFunction.name passed as a quoted_name
A function created using :class:`.GenericFunction` can now specify that the
name of the function should be rendered with or without quotes by assigning
the :class:`.quoted_name` construct to the .name element of the object.
Prior to 1.3.4, quoting was never applied to function names, and some
quoting was introduced in 🎫`4467` but no means to force quoting for
a mixed case name was available.  Additionally, the :class:`.quoted_name`
construct when used as the name will properly register its lowercase name
in the function registry so that the name continues to be available via the
``func.`` registry.

Fixes: #5079
Change-Id: I0653ab8b16e75e628ce82dbbc3d0f77f8336c407
2020-01-06 14:11:18 -05:00
Mike Bayer 65aee6cce5 Add result map targeting for custom compiled, text objects
In order for text(), custom compiled objects, etc. to be usable
by Query(), they are all targeted by object key in the result map.
As we no longer want Query to implicitly label these, as well as that
text() has no label feature, support adding entries to the result
map that have no name, key, or type, only the object itself, and
then ensure that the compiler sets up for positional targeting
when this condition is detected.

Allows for more flexible ORM query usage with custom expressions
and text() while having less special logic in query itself.

Fixes: #4887
Change-Id: Ie073da127d292d43cb132a2b31bc90af88bfe2fd
2019-10-07 23:06:06 -04:00
Mike Bayer 116faee662 self_group() for FunctionFilter
Fixed issue where the :class:`.array_agg` construct in combination with
:meth:`.FunctionElement.filter` would not produce the correct operator
precedence between the FILTER keyword and the array index operator.

Fixes: #4760
Change-Id: Ic662cd3da3330554ec673bafd80495b3f1506098
2019-07-12 22:44:27 -04:00
Mike Bayer ef7ff058eb SelectBase no longer a FromClause
As part of the SQLAlchemy 2.0 migration project, a conceptual change has
been made to the role of the :class:`.SelectBase` class hierarchy,
which is the root of all "SELECT" statement constructs, in that they no
longer serve directly as FROM clauses, that is, they no longer subclass
:class:`.FromClause`.  For end users, the change mostly means that any
placement of a :func:`.select` construct in the FROM clause of another
:func:`.select` requires first that it be wrapped in a subquery first,
which historically is through the use of the :meth:`.SelectBase.alias`
method, and is now also available through the use of
:meth:`.SelectBase.subquery`.    This was usually a requirement in any
case since several databases don't accept unnamed SELECT subqueries
in their FROM clause in any case.

See the documentation in this change for lots more detail.

Fixes: #4617
Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
2019-07-06 13:02:22 -04:00
Adrien Berchet 61c47cbdc6 Do not register the GenericFunction in sql.functions._registry
Fixed that the :class:`.GenericFunction` class was inadvertently
registering itself as one of the named functions.  Pull request courtesy
Adrien Berchet.

Fixes: #4653

Closes: #4654
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4654
Pull-request-sha: 1112b89f0d

Change-Id: Ia0d366d3bff44a763aa496287814278dff732a19
2019-05-06 17:41:34 -04:00
Mike Bayer ae14861a31 Make the GenericFunction registry fully case insensitive
Registered function names based on :class:`.GenericFunction` are now
retrieved in a case-insensitive fashion in all cases, removing the
deprecation logic from 1.3 which temporarily allowed multiple
:class:`.GenericFunction` objects to exist with differing cases.   A
:class:`.GenericFunction` that replaces another on the same name whether or
not it's case sensitive emits a warning before replacing the object.

Fixes: #4649
Change-Id: I265ae19833132db07ed5b5ae40c4d24f659b1ab3
2019-04-30 14:01:52 -04:00