Fixed critical regression where bound parameter tracking as used in the SQL
caching system could fail to track all parameters for the case where the
same SQL expression containing a parameter were used in an ORM-related
query using a feature such as class inheritance, which was then embedded in
an enclosing expression which would make use of that same expression
multiple times, such as a UNION. The ORM would individually copy the
individual SELECT statements as part of compilation with class inheritance,
which then embedded in the enclosing statement would fail to accommodate
for all parameters. The logic that tracks this condition has been adjusted
to work for multiple copies of a parameter.
Fixes: #6391
Change-Id: I6db5dee0d361a3bb58d753a2d27ef2eee2b369c5
Repaired and solidified issues regarding custom functions and other
arbitrary expression constructs which within SQLAlchemy's column labeling
mechanics would seek to use ``str(obj)`` to get a string representation to
use as an anonymous column name in the ``.c`` collection of a subquery.
This is a very legacy behavior that performs poorly and leads to lots of
issues, so has been revised to no longer perform any compilation by
establishing specific methods on :class:`.FunctionElement` to handle this
case, as SQL functions are the only use case that it came into play. An
effect of this behavior is that an unlabeled column expression with no
derivable name will be given an arbitrary label starting with the prefix
``"_no_label"`` in the ``.c`` collection of a subquery; these were
previously being represented either as the generic stringification of that
expression, or as an internal symbol.
This change seeks to make the concept of "anon name" more private
and renames anon_label and anon_key_label to _anon_name_label
and _anon_key_label. There's no end-user utility to these accessors
and we need to be able to reorganize these as well.
Fixes: #6256
Change-Id: Ie63c86b20ca45873affea78500388da94cf8bf94
Fixed the "stringify" compiler to support a basic stringification
of a "multirow" INSERT statement, i.e. one with multiple tuples
following the VALUES keyword.
Change-Id: I1fe38d204d9965275d3a72157d5a72a53bec4b11
Fixed regression where usage of a token in the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
dictionary which contained special characters such as braces would fail to
be substituted properly. Use of square bracket characters ``[]`` is now
explicitly disallowed as these are used as a delimiter character in the
current implementation.
Fixes: #6216
Change-Id: I7ccfc2292b17340054cedf485ed1adf3119b96c8
Executing a :class:`_sql.Subquery` using :meth:`_engine.Connection.execute`
is deprecated and will emit a deprecation warning; this use case was an
oversight that should have been removed from 1.4. The operation will now
execute the underlying :class:`_sql.Select` object directly for backwards
compatibility. Similarly, the :class:`_sql.CTE` class is also not
appropriate for execution. In 1.3, attempting to execute a CTE would result
in an invalid "blank" SQL statement being executed; since this use case was
not working it now raises :class:`_exc.ObjectNotExecutableError`.
Previously, 1.4 was attempting to execute the CTE as a statement however it
was working only erratically.
The change also breaks out StatementRole from ReturnsRowsRole, as these
roles should not be in the same lineage (some statements don't return
rows, the whole class of ReturnsRows that are from clauses are
not statements). Consolidate StatementRole and
CoerceTextStatementRole as there's no usage difference between
these. Simplify some old tests that were trying to make
sure that "execution options" didn't transmit from a cte/subquery
out to a select; as cte/subuqery() aren't executable in any case
the options are removed.
Fixes: #6204
Change-Id: I62613b7ab418afdd22c409eae75659e3f52fb65f
Fixed further issues in the same area as that of 🎫`6173` released in
1.4.5, where a "postcompile" parameter, again most typically those used for
LIMIT/OFFSET rendering in Oracle and SQL Server, would fail to be processed
correctly if the same parameter rendered in multiple places in the
statement.
Fixes: #6202
Change-Id: I95c355aa52a7546fe579ad67f9a8402a213cb79d
Fixed regression where usage of the standalone :func:`_sql.distinct()` used
in the form of being directly SELECTed would fail to be locatable in the
result set by column identity, which is how the ORM locates columns. While
standalone :func:`_sql.distinct()` is not oriented towards being directly
SELECTed (use :meth:`_sql.select.distinct` for a regular
``SELECT DISTINCT..``) , it was usable to a limited extent in this way
previously (but wouldn't work in subqueries, for example). The column
targeting for unary expressions such as "DISTINCT <col>" has been improved
so that this case works again, and an additional improvement has been made
so that usage of this form in a subquery at least generates valid SQL which
was not the case previously.
The change additionally enhances the ability to target elements in
``row._mapping`` based on SQL expression objects in ORM-enabled
SELECT statements, including whether the statement was invoked by
``connection.execute()`` or ``session.execute()``.
Fixes: #6008
Change-Id: I5cfa39435f5418861d70a7db8f52ab4ced6a792e
Altered the compilation for the :class:`.CTE` construct so that a string is
returned representing the inner SELECT statement if the :class:`.CTE` is
stringified directly, outside of the context of an enclosing SELECT; This
is the same behavior of :meth:`_FromClause.alias` and
:meth:`_SelectStatement.subquery`. Previously, a blank string would be
returned as the CTE is normally placed above a SELECT after that SELECT has
been generated, which is generally misleading when debugging.
Change-Id: Id3007c28e4a7a56d867e850bb890752946bd8f6f
References: #5988
Fixed regression where the "unsupported compilation error" for unknown
datatypes would fail to raise correctly.
Fixes: #5979
Change-Id: I984fe95666813832ab5bdfc568322e2aa7cc3db0
Forked from I22f6cf0f0b3360e55299cdcb2452cead2b2458ea
we are attempting to decide the case for columns mapped
under a different name. since the .key feature of
Column seems to support this fully, see if an annotation
can be used to indicate an effective .key for a column.
The effective change is that the labeling of column expressions
in rows has been improved to retain the original name of the ORM
attribute even if used in a subquery.
References: #5933
Change-Id: If251f556f7d723f50d349f765f1690d6c679d2ef
Adjusted the "literal_binds" feature of :class:`_sql.Compiler` to render
NULL for a bound parameter that has ``None`` as the value, either
explicitly passed or omitted. The previous error message "bind parameter
without a renderable value" is removed, and a missing or ``None`` value
will now render NULL in all cases. Previously, rendering of NULL was
starting to happen for DML statements due to internal refactorings, but was
not explicitly part of test coverage, which it now is.
While no error is raised, when the context is within that of a column
comparison, and the operator is not "IS"/"IS NOT", a warning is emitted
that this is not generally useful from a SQL perspective.
Fixes: #5888
Change-Id: Id5939d8dbfb1156a9f8a7f7e76cf18327155331a
Replace :meth:`_orm.Query.with_labels` and
:meth:`_sql.GenerativeSelect.apply_labels` with explicit getters and
setters ``get_label_style`` and ``set_label_style`` to accommodate the
three supported label styles: ``LABEL_STYLE_DISAMBIGUATE_ONLY`` (default),
``LABEL_STYLE_TABLENAME_PLUS_COL``, and ``LABEL_STYLE_NONE``.
In addition, for Core and "future style" ORM queries,
``LABEL_STYLE_DISAMBIGUATE_ONLY`` is now the default label style. This
style differs from the existing "no labels" style in that labeling is
applied in the case of column name conflicts; with ``LABEL_STYLE_NONE``, a
duplicate column name is not accessible via name in any case.
For legacy ORM queries using :class:`_query.Query`, the table-plus-column
names labeling style applied by ``LABEL_STYLE_TABLENAME_PLUS_COL``
continues to be used so that existing test suites and logging facilities
see no change in behavior by default, however this style of labeling is no
longer required for SQLAlchemy queries to function, as result sets are
commonly matched to columns using a positional approach since SQLAlchemy
1.0.
Within test suites, all use of apply_labels() / use_labels
now uses the new methods. New tests added to
test/sql/test_deprecations.py nad test/orm/test_deprecations.py
to cover just the old apply_labels() method call. Tests
in ORM that made explicit use apply_labels()/ etc. where it isn't needed
for the ORM to work correctly use default label style now.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #4757
Change-Id: I5fdcd2ed4ae8c7fe62f8be2b6d0e8f66409b6a54
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones. Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them. A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable. New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.
From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow. A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.
As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively. The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.
Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session. The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic. A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function. The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all. Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.
As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.
An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2. It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.
py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2. For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.
Includes the following improvements:
Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`. Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.
For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.
Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board. the addition
of this flag has resolved the long-standing oracle "two task"
error problem.
For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically. There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite. Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.
Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.
Fixes: #5826Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
Added parameters :paramref:`_ddl.CreateTable.if_not_exists`,
:paramref:`_ddl.CreateIndex.if_not_exists`,
:paramref:`_ddl.DropTable.if_exists` and
:paramref:`_ddl.DropIndex.if_exists` to the :class:`_ddl.CreateTable`,
:class:`_ddl.DropTable`, :class:`_ddl.CreateIndex` and
:class:`_ddl.DropIndex` constructs which result in "IF NOT EXISTS" / "IF
EXISTS" DDL being added to the CREATE/DROP. These phrases are not accepted
by all databases and the operation will fail on a database that does not
support it as there is no similarly compatible fallback within the scope of
a single DDL statement. Pull request courtesy Ramon Williams.
Fixes: #2843Closes: #5663
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5663
Pull-request-sha: 748b847234
Change-Id: I6a2b1f697993ed49c31584f0a31887fb0a868ed3
Dialect-specific constructs such as
:meth:`_postgresql.Insert.on_conflict_do_update` can now stringify in-place
without the need to specify an explicit dialect object. The constructs,
when called upon for ``str()``, ``print()``, etc. now have internal
direction to call upon their appropriate dialect rather than the
"default"dialect which doesn't know how to stringify these. The approach
is also adapted to generic schema-level create/drop such as
:class:`_schema.AddConstraint`, which will adapt its stringify dialect to
one indicated by the element within it, such as the
:class:`_postgresql.ExcludeConstraint` object.
mostly towards being able to provide doctest-style
examples for "on conflict" constructs using print statements.
Change-Id: I4b855516fe6dee2df77744c1bb21a373d7fbab93
Add SelectBase.exists() method as it seems strange this is
not available already. The Exists construct itself does
not provide full SELECT-building capabilities so it makes
sense this should be used more like a scalar_subquery.
Make sure stream_results is getting set up when yield_per
is used, for 2.0 style statements as well. this was
hardcoded inside of Query.yield_per() and is now moved
to take place within QueryContext.
Change-Id: Icafcd4fd9b708772343d56edf40995c9e8f835d6
The operator changes are:
* `isfalse` is now `is_false`
* `isnot_distinct_from` is now `is_not_distinct_from`
* `istrue` is now `is_true`
* `notbetween` is now `not_between`
* `notcontains` is now `not_contains`
* `notendswith` is now `not_endswith`
* `notilike` is now `not_ilike`
* `notlike` is now `not_like`
* `notmatch` is now `not_match`
* `notstartswith` is now `not_startswith`
* `nullsfirst` is now `nulls_first`
* `nullslast` is now `nulls_last`
Because these are core operators, the internal migration strategy for this
change is to support legacy terms for an extended period of time -- if not
indefinitely -- but update all documentation, tutorials, and internal usage
to the new terms. The new terms are used to define the functions, and
the legacy terms have been deprecated into aliases of the new terms.
Fixes: #5435
Change-Id: Ifbd7cb1cdda5981990243c4fc4b4ff467dc132ac
Fixed structural compiler issue where some constructs such as MySQL /
PostgreSQL "on conflict / on duplicate key" would rely upon the state of
the :class:`_sql.Compiler` object being fixed against their statement as
the top level statement, which would fail in cases where those statements
are branched from a different context, such as a DDL construct linked to a
SQL statement.
Fixes: #5656
Change-Id: I568bf40adc7edcf72ea6c7fd6eb9d07790de189e
Improved support for column names that contain percent signs in the string,
including repaired issues involving anoymous labels that also embedded a
column name with a percent sign in it, as well as re-established support
for bound parameter names with percent signs embedded on the psycopg2
dialect, using a late-escaping process similar to that used by the
cx_Oracle dialect.
* Added new constructor for _anonymous_label() that ensures incoming
string tokens based on column or table names will have percent
signs escaped; abstracts away the format of the label.
* generalized cx_Oracle's quoted_bind_names facility into the compiler
itself, and leveraged this for the psycopg2 dialect's issue with
percent signs in names as well. the parameter substitution is now
integrated with compiler.construct_parameters() as well as the
recently reworked set_input_sizes(), reducing verbosity in the
cx_Oracle dialect.
Fixes: #5653
Change-Id: Ia2ad13ea68b4b0558d410026e5a33f5cb3fbab2c
Add support to ``FETCH {FIRST | NEXT} [ count ] {ROW | ROWS}
{ONLY | WITH TIES}`` in the select for the supported backends,
currently PostgreSQL, Oracle and MSSQL.
Fixes: #5576
Change-Id: Ibb5871a457c0555f82b37e354e7787d15575f1f7
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.
Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function. it does not
yet fully address other issues such as keyword arguments passed
to the table.select().
Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.
Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
fixed an issue where even though the method claims to be
matching up columns positionally, it was failing on that by
looking in "keymap" based on string name.
Adds a new member to the _keymap recs MD_RESULT_MAP_INDEX
so that we can efficiently link from the generated keymap
back to the compiled._result_columns structure without
any ambiguity.
Fixes: #5559
Change-Id: Ie2fa9165c16625ef860ffac1190e00575e96761f
Fixed issue where the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature would not take effect when the :meth:`_schema.Sequence.next_value`
function function for a :class:`_schema.Sequence` were used in the
:paramref:`_schema.Column.server_default` parameter and the create table
DDL were emitted.
Fixes: #5500
Change-Id: I74a9fa13d22749d06c8202669f9ea220d9d984d9
Remove lookup logic that attempts to locate a dialect for a type,
just use StrSQLTypeCompiler.
Cleaned up the internal ``str()`` for datatypes so that all types produce a
string representation without any dialect present, including that it works
for third-party dialect types without that dialect being present. The
string representation defaults to being the UPPERCASE name of that type
with nothing else.
Fixes: #4262
Change-Id: I02149e8a1ba1e7336149e962939b07ae0df83c6b
This is kind of a mixed bag of all kinds to help get us
to 1.4 betas. The documentation stuff is a work in
progress. Lots of other relatively small changes to
APIs and things. More commits will follow to continue
improving the documentation and transitioning to the
1.4/2.0 hybrid documentation. In particular some refinements
to Session usage models so that it can match Engine's
scoping / transactional patterns, and a decision to
start moving away from "subtransactions" completely.
* add select().from_statement() to produce FromStatement in an
ORM context
* begin referring to select() that has "plugins" for the few edge
cases where select() will have ORM-only behaviors
* convert dynamic.AppenderQuery to its own object that can use
select(), though at the moment it uses Query to support legacy
join calling forms.
* custom query classes for AppenderQuery are replaced by
do_orm_execute() hooks for custom actions, a separate gerrit
will document this
* add Session.get() to replace query.get()
* Deprecate session.begin->subtransaction. propose within the
test suite a hypothetical recipe for apps that rely on this
pattern
* introduce Session construction level context manager,
sessionmaker context manager, rewrite the whole top of the
session_transaction.rst documentation. Establish context manager
patterns for Session that are identical to engine
* ensure same begin_nested() / commit() behavior as engine
* devise all new "join into an external transaction" recipe,
add test support for it, add rules into Session so it
just works, write new docs. need to ensure this doesn't
break anything
* vastly reduce the verbosity of lots of session docs as
I dont think people read this stuff and it's difficult
to keep current in any case
* constructs like case(), with_only_columns() really need
to move to *columns, add a coercion rule to just change
these.
* docs need changes everywhere I look. in_() is not in
the Core tutorial? how do people even know about it?
Remove tons of cruft from Select docs, etc.
* build a system for common ORM options like populate_existing
and autoflush to populate from execution options.
* others?
Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles. This would make
migration simpler and reduce confusion.
However, confusion may be increased by the fact that select().join()
is different Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs. At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps. Session.execute() will still
behave the old way as we are adding a future flag.
This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.
The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.
Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.
Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.
Fixes: #5379Fixes: #5284
Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51
References: #5159
Renamed the :meth:`_schema.Table.tometadata` method to
:meth:`_schema.Table.to_metadata`. The previous name remains with a
deprecation warning.
Updated the "decorate" utility function to support decoration
of functions that include non-builtins as default values.
Moves test for deprecated "databases" package into
test/dialect/test_deprecations.py
Fixes: #5413Fixes: #5426
Change-Id: I6ed899871c935f9e46360127c17ccb7cf97cea6e
A variety of caching issues found by running
all tests with statement caching turned on.
The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.
Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.
Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *". if a table re-ordered its columns
or added/removed, now that data is obsolete.
Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.
Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement. Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.
Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.
mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.
lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly. While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.
turn on cache stats in logging.
Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context. This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.
DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced. this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.
memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.
Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason. Removed
these methods as well as the concept of a Compiler
holding onto a "bind".
Fixes: #5386
Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
This patch contains a variety of ORM and expression layer
tweaks to support ORM constructs in select() statements,
without the 1.3.x requiremnt in Query that a full
_compile_context() + new select() is needed in order to
get a working statement object.
Includes such tweaks as the ability to implement
aliased class of an aliased class,
as we are looking to fully support ACs against subqueries,
as well as the ability to access anonymously-labeled
ColumnProperty expressions within subqueries by
naming the ".key" of the label after the property
key. Some tuning to query.join() as well
as ORMJoin internals to allow things to work more
smoothly.
Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for
Microsoft SQL Server. This removes the deprecated feature of using
:class:`.Sequence` objects to manipulate IDENTITY characteristics which
should now be performed using ``mssql_identity_start`` and
``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The
change includes a new parameter :paramref:`.Sequence.data_type` to
accommodate SQL Server's choice of datatype, which for that backend
includes INTEGER and BIGINT. The default starting value for SQL Server's
version of :class:`.Sequence` has been set at 1; this default is now
emitted within the CREATE SEQUENCE DDL for all backends.
Fixes: #4235Fixes: #4633
Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
Added :meth:`.Select.with_hint` output to the generic SQL string that is
produced when calling ``str()`` on a statement. Previously, this clause
would be omitted under the assumption that it was dialect specific.
The hint text is presented within brackets to indicate the rendering
of such hints varies among backends.
Fixes: #5353
References: #4667
Change-Id: I01d97d6baa993e495519036ec7ecd5ae62856c16
This patch replaces the ORM execution flow with a
single pathway through Session.execute() for all queries,
including Core and ORM.
Currently included is full support for ORM Query,
Query.from_statement(), select(), as well as the
baked query and horizontal shard systems. Initial
changes have also been made to the dogpile caching
example, which like baked query makes use of a
new ORM-specific execution hook that replaces the
use of both QueryEvents.before_compile() as well
as Query._execute_and_instances() as the central
ORM interception hooks.
select() and Query() constructs alike can be passed to
Session.execute() where they will return ORM
results in a Results object. This API is currently
used internally by Query. Full support for
Session.execute()->results to behave in a fully
2.0 fashion will be in later changesets.
bulk update/delete with ORM support will also
be delivered via the update() and delete()
constructs, however these have not yet been adapted
to the new system and may follow in a subsequent
update.
Performance is also beginning to lag as of this
commit and some previous ones. It is hoped that
a few central functions such as the coercions
functions can be rewritten in C to re-gain
performance. Additionally, query caching
is now available and some subsequent patches
will attempt to cache more of the per-execution
work from the ORM layer, e.g. column getters
and adapters.
This patch also contains initial "turn on" of the
caching system enginewide via the query_cache_size
parameter to create_engine(). Still defaulting at
zero for "no caching". The caching system still
needs adjustments in order to gain adequate performance.
Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
Convert Query to do virtually all compile state computation
in the _compile_context() phase, and organize it all
such that a plain select() construct may also be used as the
source of information in order to generate ORM query state.
This makes it such that Query is not needed except for
its additional methods like from_self() which are all to
be deprecated.
The construction of ORM state will occur beyond the
caching boundary when the new execution model is integrated.
future select() gains a working join() and filter_by() method.
as we continue to rebase and merge each commit in the steps,
callcounts continue to bump around. will have to look at
the final result when it's all in.
References: #5159
References: #4705
References: #4639
References: #4871
References: #5010
Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
Added a "schema" parameter to the :func:`_expression.table` construct,
allowing ad-hoc table expressions to also include a schema name.
Pull request courtesy Dylan Modesitt.
Fixes: #5309Closes: #5310
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5310
Pull-request-sha: ce85681050
Change-Id: I32015d593e1ee1121c7426fbffdcc565d025fad1
As progress is made on the _future.Result, including breaking
it out such that DBAPI behaviors are local to specific
implementations, it becomes apparent that the Result object
is a functional superset of ResultProxy and that basic
operations like fetchone(), fetchall(), and fetchmany()
behave pretty much exactly the same way on the new object.
Reorganize things so that ResultProxy is now referred to
as LegacyCursorResult, which subclasses CursorResult
that represents the DBAPI-cursor version of Result,
making use of a multiple inheritance pattern so that
the functionality of Result is also available in non-DBAPI
contexts, as will be necessary for some ORM
patterns.
Additionally propose the composition system for Result
that will form the basis for ORM-alternative result
systems such as horizontal sharding and dogpile cache.
As ORM results will soon be coming directly from
instances of Result, these extensions will instead
build their own ResultFetchStrategies that perform
the special steps to create composed or cached
result sets.
Also considering at the moment not emitting deprecation
warnings for fetchXYZ() methods; the immediate issue
is Keystone tests are calling upon it, but as the
implementations here are proving to be not in any
kind of conflict with how Result works, there's
not too much issue leaving them around and deprecating
at some later point.
References: #5087
References: #4395Fixes: #4959
Change-Id: I8091919d45421e3f53029b8660427f844fee0228
Deprecate usage of ``DISTINCT ON`` in dialect other than PostgreSQL.
Previously this was silently ignored.
Deprecate old usage of string distinct in MySQL dialect
Fixes: #4002
Change-Id: I38fc64aef75e77748083c11d388ec831f161c9c9
Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf
Trying to see if by making the cache key memoized, we
still can have the older "identity" form of caching
which is the cheapest of all, at the same time as the
newer "cache key each time" version that is not nearly
as cheap; but still much cheaper than no caching at all.
Also needed is a per-execution update of _keymap when
we invoke from a cached select, so that Column objects
that are anonymous or otherwise adapted will match up.
this is analogous to the adaption of bound parameters
from the cache key.
Adds test coverage for the keymap / construct_params()
changes related to caching. Also hones performance
to a large extent for statement construction and
cache key generation.
Also includes a new memoized attribute
approach that vastly simplifies the previous approach
of "group_expirable_memoized_property" and finally
integrates cleanly with _clone(), _generate(), etc.
no more hardcoding of attributes is needed, as well
as that most _reset_memoization() calls are no longer
needed as the reset is inherent in a _generate() call;
this also has dramatic performance improvements.
Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
Add ability to literal compile a :class:`DateTime`, :class:`Date`
or :class:"Time" when using the string dialect for debugging purposes.
This change does not impact real dialect implementation that retain
their current behavior.
Fixes: #5052
Change-Id: Ia3fad2be905c6d35b0106b9a2388c7508f067e90
Revised the :paramref:`.Connection.execution_options.schema_translate_map`
feature such that the processing of the SQL statement to receive a specific
schema name occurs within the execution phase of the statement, rather than
at the compile phase. This is to support the statement being efficiently
cached. Previously, the current schema being rendered into the statement
for a particular run would be considered as part of the cache key itself,
meaning that for a run against hundreds of schemas, there would be hundreds
of cache keys, rendering the cache much less performant. The new behavior
is that the rendering is done in a similar manner as the "post compile"
rendering added in 1.4 as part of 🎫`4645`, 🎫`4808`.
Fixes: #5004
Change-Id: Ia5c89eb27cc8dc2c5b8e76d6c07c46290a7901b6
Continuation of I408e0b8be91fddd77cf279da97f55020871f75a9
- add an options() method to the base Generative construct.
this will be where ORM options can go
- Change Null, False_, True_ to be singletons, so that
we aren't instantiating them and having to use isinstance.
The previous issue with this was that they would produce dupe
labels in SELECT statements. Apply the duplicate column
logic, newly added in 1.4, to these objects as well as to
non-apply-labels SELECT statements in general as a means of
improving this.
- create a revised system for generating ClauseList compilation
constructs that simplfies up front creation to not actually
use ClauseList; a simple tuple is rendered by the compiler
using the same constrcution rules as what are used for
ClauseList but without creating the actual object. Apply
to Select, CompoundSelect, revise Update, Delete
- Select, CompoundSelect get an initial CompileState
implementation. All methods used only within compilation
are moved here
- refine update/insert/delete compile state to not require
an outside boolean
- refine and simplify Select._copy_internals
- rework bind(), which is going away, to not use some
of the internal traversal stuff
- remove "autocommit", "for_update" parameters from Select,
references #4643
- remove "autocommit" parameter from TextClause ,
references #4643
- add deprecation warnings for statement.execute(),
engine.execute(), statement.scalar(), engine.scalar().
Fixes: #5193
Change-Id: I04ca0152b046fd42c5054ba10f37e43fc6e5a57b
Targeting select / insert / update / delete, the goal
is to minimize overhead of construction and generative methods
so that only the raw arguments passed are handled. An interim
stage that converts the raw state into more compiler-ready state
is added, which is analogous to the ORM QueryContext which will
also be rolled in to be a similar concept, as is currently
being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10.
the ORM update/delete BulkUD concept is also going to be rolled
onto this idea. So while the compiler-ready state object,
here called DMLState, looks a little thin, it's the
base of a bigger pattern that will allow for ORM functionality
to embed itself directly into the compiler, execution
context, and result set objects.
This change targets the DML objects, primarily focused on the
values() method which is the most complex process. The
work done by values() is minimized as much as possible
while still being able to create a cache key. Additional
computation is then offloaded to a new object ValuesState
that is handled by the compiler.
Architecturally, a big change here is that insert.values()
and update.values() will generate BindParameter objects for
the values now, which are then carefully received by crud.py
so that they generate the expected names. This is so that
the values() portion of these constructs is cacheable.
for the "multi-values" version of Insert, this is all skipped
and the plan right now is that a multi-values insert is
not worth caching (can always be revisited).
Using the
coercions system in values() also gets us nicer validation
for free, we can remove the NotAClauseElement thing from
schema, and we also now require scalar_subquery() is called
for an insert/update that uses a SELECT as a column value,
1.x deprecation path is added.
The traversal system is then applied to the DML objects
including tests so that they have traversal, cloning, and
cache key support. cloning is not a use case for DML however
having it present allows better validation of the structure
within the tests.
Special per-dialect DML is explicitly not cacheable at the moment,
more as a proof of concept that third party DML constructs can
exist as gracefully not-cacheable rather than producing an
incomplete cache key.
A few selected performance improvements have been added as well,
simplifying the immutabledict.union() method and adding
a new SQLCompiler function that can generate delimeter-separated
clauses like WHERE and ORDER BY without having to build
a ClauseList object at all. The use of ClauseList will
be removed from Select in an upcoming commit. Overall,
ClaustList is unnecessary for internal use and only adds
overhead to statement construction and will likely be removed
as much as possible except for explcit use of conjunctions like
and_() and or_().
Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
Removed very antiquated logic that checks if __visit_name__
is a property. There's no need for this as the compiler can handle
switching between implementations. Convert _compile_dispatch()
to be fully inlined.
Change-Id: Ic0c7247c2d7dfed93a27f09250a8ed6352370764
This builds on cc718cccc0 which moved
RowProxy to Row, allowing Row to be more like a named tuple.
- KeyedTuple in ORM is replaced with Row
- ResultSetMetaData broken out into "simple" and "cursor" versions
for ORM and Core, as well as LegacyCursor version.
- Row now has _mapping attribute that supplies full mapping behavior.
Row and SimpleRow both have named tuple behavior otherwise.
LegacyRow has some mapping features on the tuple which emit
deprecation warnings (e.g. keys(), values(), etc). the biggest
change for mapping->tuple is the behavior of __contains__ which
moves from testing of "key in row" to "value in row".
- ResultProxy breaks into ResultProxy and FutureResult (interim),
the latter has the newer APIs. Made available to dialects
using execution options.
- internal reflection methods and most tests move off of implicit
Row mapping behavior and move to row._mapping, result.mappings()
method using future result
- a new strategy system for cursor handling replaces the various
subclasses of RowProxy
- some execution context adjustments. We will leave EC in but
refined things like get_result_proxy() and out parameter handling.
Dialects for 1.4 will need to adjust from get_result_proxy()
to get_result_cursor_strategy(), if they are using this method
- out parameter handling now accommodated by get_out_parameter_values()
EC method. Oracle changes for this. external dialect for
DB2 for example will also need to adjust for this.
- deprecate case_insensitive flag for engine / result, this
feature is not used
mapping-methods on Row are deprecated, and replaced with
Row._mapping.<meth>, including:
row.keys() -> use row._mapping.keys()
row.items() -> use row._mapping.items()
row.values() -> use row._mapping.values()
key in row -> use key in row._mapping
int in row -> use int < len(row)
Fixes: #4710Fixes: #4878
Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
Reorganization of Select() is the first major element
of the 2.0 restructuring. In order to start this we need
to first create the new Select constructor and apply legacy
elements to the old one. This in turn necessitates
starting up the RemovedIn20Warning concept which itself
need to refer to "sqlalchemy.future", so begin to establish
this basic framework. Additionally, update the
DML constructors with the newer no-keyword style. Remove
the use of the "pending deprecation" and fix Query.add_column()
deprecation which was not acting as deprecated.
Fixes: #4845Fixes: #4648
Change-Id: I0c7a22b2841a985e1c379a0bb6c94089aae6264c
Creating an :func:`.and_` or :func:`.or_` construct with no arguments or
empty ``*args`` will now emit a deprecation warning, as the SQL produced is
a no-op (i.e. it renders as a blank string). This behavior is considered to
be non-intuitive, so for empty or possibly empty :func:`.and_` or
:func:`.or_` constructs, an appropriate default boolean should be included,
such as ``and_(True, *args)`` or ``or_(False, *args)``. As has been the
case for many major versions of SQLAlchemy, these particular boolean
values will not render if the ``*args`` portion is non-empty.
As there are some internal cases where an empty and_() construct is used
in order to build an optional WHERE expression, a private
utility function is added to suit this use case.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #5054Closes: #5062
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5062
Pull-request-sha: 5ca2f27281
Change-Id: I599b9c8befa64d9a59a35ad7dd84ff400e3aa647
Mike Bayer