The maximum buffer size for the :class:`.BufferedRowResultProxy`, which
is used by dialects such as PostgreSQL when ``stream_results=True``, can
now be set to a number greater than 1000 and the buffer will grow to
that size. Previously, the buffer would not go beyond 1000 even if the
value were set larger. The growth of the buffer is also now based
on a simple multiplying factor currently set to 5. Pull request courtesy
Soumaya Mauthoor.
Fixes: #4914Closes: #4930
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4930
Pull-request-sha: 66841f56e9
Change-Id: I6286220bd9d488027fadc444039421a410e19a19
Changed the ``repr()`` of the :class:`.quoted_name` construct to use
regular string repr() under Python 3, rather than running it through
"backslashreplace" escaping, which can be misleading.
Modified the approach of "name normalization" for the Oracle and Firebird
dialects, which converts from the UPPERCASE-as-case-insensitive convention
of these dialects into lowercase-as-case-insensitive for SQLAlchemy, to not
automatically apply the :class:`.quoted_name` construct to a name that
matches itself under upper or lower case conversion, as is the case for
many non-european characters. All names used within metadata structures
are converted to :class:`.quoted_name` objects in any case; the change
here would only affect the output of some inspection functions.
Moved name normalize to be under default dialect, added test coverage
in test/sql/test_quote.py
Fixes: #4931
Change-Id: Ic121b20e07249824710a54423e321d94a425362f
Add factilities to implement pytest.mark.parametrize and
pytest.fixtures patterns, which largely resemble things we are
already doing.
Ensure a facade is used, so that the test suite remains independent
of py.test, but also tailors the functions to the more limited
scope in which we are using them.
Additionally, create a class-based version that works from the
same facade.
Several old polymorphic tests as well as two of the sql test
are refactored to use the new features.
Change-Id: I6ef8af1dafff92534313016944d447f9439856cf
References: #4896
Fixed bug where a table that would have a column label overlap with a plain
column name, such as "foo.id AS foo_id" vs. "foo.foo_id", would prematurely
generate the ``._label`` attribute for a column before this overlap could
be detected due to the use of the ``index=True`` or ``unique=True`` flag on
the column in conjunction with the default naming convention of
``"column_0_label"``. This would then lead to failures when ``._label``
were used later to generate a bound parameter name, in particular those
used by the ORM when generating the WHERE clause for an UPDATE statement.
The issue has been fixed by using an alternate ``._label`` accessor for DDL
generation that does not affect the state of the :class:`.Column`. The
accessor also bypasses the key-deduplication step as it is not necessary
for DDL, the naming is now consistently ``"<tablename>_<columnname>"``
without any subsequent numeric symbols when used in DDL.
Fixes: #4911
Change-Id: Iabf5fd3250738d800d6e41a2a3a27a7ce2405e7d
Alembic needs a portable way of getting at the name of an
index without quoting being applied. As we would like the
indexes created by the Column index=True flag to support
deferred index names, supply a function that delivers this
for Alembic without it having to dig too deeply into the
internals. the _alembic_quote flag may be made public
at a later time, however as we've been through many quoting
flags that are difficult to get rid of, try to be conservative
to start.
Change-Id: I184adaeae26c2e75093aaea5ebe01a3815cadb08
The :class:`.Join` construct no longer considers the "onclause" as a source
of additional FROM objects to be omitted from the FROM list of an enclosing
:class:`.Select` object as standalone FROM objects. This applies to an ON
clause that includes a reference to another FROM object outside the JOIN;
while this is usually not correct from a SQL perspective, it's also
incorrect for it to be omitted, and the behavioral change makes the
:class:`.Select` / :class:`.Join` behave a bit more intuitively.
Fixes: #4621
Change-Id: Iaa1e75b7c59b21e9701ab3c9b69e66930feaf8ee
In order for text(), custom compiled objects, etc. to be usable
by Query(), they are all targeted by object key in the result map.
As we no longer want Query to implicitly label these, as well as that
text() has no label feature, support adding entries to the result
map that have no name, key, or type, only the object itself, and
then ensure that the compiler sets up for positional targeting
when this condition is detected.
Allows for more flexible ORM query usage with custom expressions
and text() while having less special logic in query itself.
Fixes: #4887
Change-Id: Ie073da127d292d43cb132a2b31bc90af88bfe2fd
Fixed bug in SQL Server dialect with new "max_identifier_length" feature
where the mssql dialect already featured this flag, and the implementation
did not accommodate for the new initialization hook correctly.
Fixes: #4857
Change-Id: I96a9c6ca9549d8f6fb167c0333f684e8d922a3bf
Dropped support for right-nested join rewriting to support old SQLite
versions prior to 3.7.16, released in 2013. It is expected that
all modern Python versions among those now supported should all include
much newer versions of SQLite.
Fixes: #4895
Change-Id: I7f0cfc2b7d988ff147b9a4c6d5e2adec87e27029
As of #4753 we allow duplicate columns. This creates some new
problems that there can be duplicate columns in a subquery
which are then not addressible on the outside because they
are ambiguous (Postgresql has this behavior at least). Additionally
it creates situations where we are making an anon label of an
anon label which is leaking into the query.
New logic for generating anon labels handles this situation and
also alters the .c collection
of a subquery such that we are only getting the first column
from the derived selectable that has that name, the subsequent ones
have a new deduping label with two underscores and are not exposed
in .c. The dedupe logic when rendering the columns will handle
duplicate label names for different columns, vs. the same column
repeated, as separate cases.
Fixes: #4892
Change-Id: I929fbd8da14bcc239e0481c24bbd9b5ce826e8fa
Deprecate query.instances() without a context
Deprecate string alias with contains_eager()
Deprecated the behavior by which a :class:`.Column` can be used as the key
in a result set row lookup, when that :class:`.Column` is not part of the
SQL selectable that is being selected; that is, it is only matched on name.
A deprecation warning is now emitted for this case. Various ORM use
cases, such as those involving :func:`.text` constructs, have been improved
so that this fallback logic is avoided in most cases.
Calling the :meth:`.Query.instances` method without passing a
:class:`.QueryContext` is deprecated. The original use case for this was
that a :class:`.Query` could yield ORM objects when given only the entities
to be selected as well as a DBAPI cursor object. However, for this to work
correctly there is essential metadata that is passed from a SQLAlchemy
:class:`.ResultProxy` that is derived from the mapped column expressions,
which comes originally from the :class:`.QueryContext`. To retrieve ORM
results from arbitrary SELECT statements, the :meth:`.Query.from_statement`
method should be used.
Note there is a small bump in test_zoomark because the
column._label is being calculated for each of those columns within
baseline_3_properties, as it is now part of the result map.
This label can't be calculated when the column is attached
to the table because it needs to have all the columns present
to do this correctly. Another approach here would be to
pre-load the _label before the test runs however the zoomark
tests don't have an easy place for this to happen and it's
not really worth it.
Fixes: #4877Fixes: #4719
Change-Id: I9bd29e72e6dce7c855651d69ba68d7383469acbc
Added new :func:`.create_engine` parameter
:paramref:`.create_engine.max_identifier_length`. This overrides the
dialect-coded "max identifier length" in order to accommodate for databases
that have recently changed this length and the SQLAlchemy dialect has
not yet been adjusted to detect for that version. This parameter interacts
with the existing :paramref:`.create_engine.label_length` parameter in that
it establishes the maximum (and default) value for anonymously generated
labels.
The Oracle dialect now emits a warning if Oracle version 12.2 or greater is
used, and the :paramref:`.create_engine.max_identifier_length` parameter is
not set. The version in this specific case defaults to that of the
"compatibility" version set in the Oracle server configuration, not the
actual server version. In version 1.4, the default max_identifier_length
for 12.2 or greater will move to 128 characters. In order to maintain
forwards compatibility, applications should set
:paramref:`.create_engine.max_identifier_length` to 30 in order to maintain
the same length behavior, or to 128 in order to test the upcoming behavior.
This length determines among other things how generated constraint names
are truncated for statements like ``CREATE CONSTRAINT`` and ``DROP
CONSTRAINT``, which means a the new length may produce a name-mismatch
against a name that was generated with the old length, impacting database
migrations.
Fixes: #4857
Change-Id: Ib62efb00c6180c375869029b57353d90385d7950
as part of a larger series of changes to generalize row-tuples,
RowProxy becomes plain Row and is no longer a "proxy"; the
DBAPI row is now copied directly into the Row when constructed,
result handling occurs at once.
Subsequent changes will break out Row into a new version that
behaves fully a tuple.
Change-Id: I2ffa156afce5d21c38f28e54c3a531f361345dd5
generation is to be enhanced to include caching
functionality, so ensure that Query and all generative in Core
(e.g. select, DML etc) are using the same generations system.
Additionally, deprecate Select.append methods and state
Select methods independently of their append versions.
Mutability of expression objects is a special case only when
generating new objects during a visit.
Fixes: #4637
Change-Id: I3dfac00d5e0f710c833b236f7a0913e1ca24dde4
Apparently Alias had an .as_scalar() method, so restore an
equivalent to Subquery with an appropriate deprecation warning.
Fixes: #4854
Change-Id: I6255d61b7d82487ca90ba8ee79d4b3a74e7cbe38
Added an explicit error message for the case when objects passed to
:class:`.Table` are not :class:`.SchemaItem` objects, rather than resolving
to an attribute error.
Fixes: #4847
Change-Id: I4dcdcee86b64c85ccf12e2ddc3d638563d307991
We are looking to build a generalization of copy_internals(),
so move out any special logic from these methods. Re-implement
and clarify rationale for the Alias doesnt copy a TableClause rule as
part of the adaption
traversal, establish that we forgot to build out comparison and cache
key for CTE, remove incomplete _copy_internals() from GenerativeSelect
(it doesn't handle the order_by_clause or group_by_clause, so is incomplete)
Change-Id: I95039f042503171aade4ba0fabc9b1598e3c49cf
Characters that interfere with "pyformat" or "named" formats in bound
parameters, namely ``%, (, )`` and the space character, as well as a few
other typically undesirable characters, are stripped early for a
:func:`.bindparam` that is using an anonymized name, which is typically
generated automatically from a named column which itself includes these
characters in its name and does not use a ``.key``, so that they do not
interfere either with the SQLAlchemy compiler's use of string formatting or
with the driver-level parsing of the parameter, both of which could be
demonstrated before the fix. The change only applies to anonymized
parameter names that are generated and consumed internally, not end-user
defined names, so the change should have no impact on any existing code.
Applies in particular to the psycopg2 driver which does not otherwise quote
special parameter names, but also strips leading underscores to suit Oracle
(but not yet leading numbers, as some anon parameters are currently
entirely numeric/underscore based); Oracle in any case continues to quote
parameter names that include special characters.
Fixes: #4837
Change-Id: I21cb654c3e4ef786114160b8b4295242720bf3f9
Added new entity-targeting capabilities to the :class:`.Query` object to
help with the case where the :class:`.Session` is using a bind dictionary
against mapped classes, rather than a single bind, and the :class:`.Query`
is against a Core statement that was ultimately generated from a method
such as :meth:`.Query.subquery`; a deep search is performed to locate
any ORM entity related to the query in order to locate a mapper if
one is not otherwise present.
Fixes: #4829
Change-Id: I95cf325a5aba21baec4b313246c6f4d692284820
Added new "post compile parameters" feature. This feature allows a
:func:`.bindparam` construct to have its value rendered into the SQL string
before being passed to the DBAPI driver, but after the compilation step,
using the "literal render" feature of the compiler. The immediate
rationale for this feature is to support LIMIT/OFFSET schemes that don't
work or perform well as bound parameters handled by the database driver,
while still allowing for SQLAlchemy SQL constructs to be cacheable in their
compiled form. The immediate targets for the new feature are the "TOP
N" clause used by SQL Server (and Sybase) which does not support a bound
parameter, as well as the "ROWNUM" and optional "FIRST_ROWS()" schemes used
by the Oracle dialect, the former of which has been known to perform better
without bound parameters and the latter of which does not support a bound
parameter. The feature builds upon the mechanisms first developed to
support "expanding" parameters for IN expressions. As part of this
feature, the Oracle ``use_binds_for_limits`` feature is turned on
unconditionally and this flag is now deprecated.
- adds limited support for "unique" bound parameters within
a text() construct.
- adds an additional int() check within the literal render
function of the Integer datatype and tests that non-int values
raise ValueError.
Fixes: #4808
Change-Id: Iace97d544d1a7351ee07db970c6bc06a19c712c6
Additional logic has been added such that certain SQL expressions which
typically wrap a single database column will use the name of that column as
their "anonymous label" name within a SELECT statement, potentially making
key-based lookups in result tuples more intutive. The primary example of
this is that of a CAST expression, e.g. ``CAST(table.colname AS INTEGER)``,
which will export its default name as "colname", rather than the usual
"anon_1" label, that is, ``CAST(table.colname AS INTEGER) AS colname``.
If the inner expression doesn't have a name, then the previous "anonymous
label" logic is used. When using SELECT statements that make use of
:meth:`.Select.apply_labels`, such as those emitted by the ORM, the
labeling logic will produce ``<tablename>_<inner column name>`` in the same
was as if the column were named alone. The logic applies right now to the
:func:`.cast` and :func:`.type_coerce` constructs as well as some
single-element boolean expressions.
Fixes: #4449
Change-Id: Ie3b73470e3bea53f2386cd86514cdc556491564e
Added support for the use of an :class:`.Enum` datatype using Python
pep-435 enumeration objects as values for use as a primary key column
mapped by the ORM. As these values are not inherently sortable, as
required by the ORM for primary keys, a new
:attr:`.TypeEngine.sort_key_function` attribute is added to the typing
system which allows any SQL type to implement a sorting for Python objects
of its type which is consulted by the unit of work. The :class:`.Enum`
type then defines this using the database value of a given enumeration.
The sorting scheme can be also be redefined by passing a callable to the
:paramref:`.Enum.sort_key_function` parameter. Pull request courtesy
Nicolas Caniart.
Fixes: #4285Closes: #4816
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4816
Pull-request-sha: 42266b766c
Change-Id: Iadcc16173c1ba26ffac5830db57743a4cb987c55
The :meth:`.Index.create` and :meth:`.Index.drop` methods now have a
parameter :paramref:`.Index.create.checkfirst`, in the same way as that of
:class:`.Table` and :class:`.Sequence`, which when enabled will cause the
operation to detect if the index exists (or not) before performing a create
or drop operation.
Fixes: #527
Change-Id: Idf994bc016359d0ae86cc64ccb20378115cb66d6
- Deprecated remaining engine-level introspection and utility methods
including :meth:`.Engine.run_callable`, :meth:`.Engine.transaction`,
:meth:`.Engine.table_names`, :meth:`.Engine.has_table`. The utility
methods are superseded by modern context-manager patterns, and the table
introspection tasks are suited by the :class:`.Inspector` object.
- The internal dialect method ``Dialect.reflecttable`` has been removed. A
review of third party dialects has not found any making use of this method,
as it was already documented as one that should not be used by external
dialects. Additionally, the private ``Engine._run_visitor`` method
is also removed.
- The long-deprecated ``Inspector.get_table_names.order_by`` parameter has
been removed.
- The :paramref:`.Table.autoload_with` parameter now accepts an :class:`.Inspector` object
directly, as well as any :class:`.Engine` or :class:`.Connection` as was the case before.
Fixes: #4755
Change-Id: Iec3a8b0f3e298ba87d532b16fac1e1132f464e21
The commit message in 896d47f318 is not accurate,
we do in fact still need _is_clone_of when a clone of a column is
created so that we can implement _cloned_set(). Additionally,
the linkage of _is_clone_of within make_proxy() also suits
an additional use case that seems to be related to [ticket:2419].
Adjust one of the tests which likely got changed within 1.4's
refactoring of Select to test this correctly.
Change-Id: I124c7c6b02498e2dfad9797816df42a5b6f91901
Fixed issue where internal cloning of SELECT constructs could lead to a key
error if the copy of the SELECT changed its state such that its list of
columns changed. This was observed to be occurring in some ORM scenarios
which may be unique to 1.3 and above, so is partially a regression fix.
For 1.4, the _is_clone_of key will be removed entirely as it seems to
have no purpose. This commit is the initial backport to 1.3 which
includes tests.
Fixes: #4780
Change-Id: I0c64962a2eba3763bea3107fc7c7d7aed8244430
Fixed bug where :meth:`.TypeEngine.column_expression` method would not be
applied to subsequent SELECT statements inside of a UNION or other
:class:`.CompoundSelect`, even though the SELECT statements are rendered at
the topmost level of the statement. New logic now differentiates between
rendering the column expression, which is needed for all SELECTs in the
list, vs. gathering the returned data type for the result row, which is
needed only for the first SELECT.
Fixes: #4787
Change-Id: Iceb63e430e76d2365649aa25ead09c4e2a062e10
Fixed issue where :class:`.Index` object which contained a mixture of
functional expressions which were not resolvable to a particular column,
in combination with string-based column names, would fail to initialize
its internal state correctly leading to failures during DDL compilation.
Fixes: #4778
Change-Id: I0fa9c627a1fde92ba8b9ed10af167c156012bd5d
Added support for composite (tuple) IN operators with SQLite, by rendering
the VALUES keyword for this backend. As other backends such as DB2 are
known to use the same syntax, the syntax is enabled in the base compiler
using a dialect-level flag ``tuple_in_values``. The change also includes
support for "empty IN tuple" expressions for SQLite when using "in_()"
between a tuple value and an empty set.
Fixes: #4766
Change-Id: I416e1af29b31d78f9ae06ec3c3a48ef6d6e813f5
Fixed issue where the :class:`.array_agg` construct in combination with
:meth:`.FunctionElement.filter` would not produce the correct operator
precedence between the FILTER keyword and the array index operator.
Fixes: #4760
Change-Id: Ic662cd3da3330554ec673bafd80495b3f1506098
The :func:`.select` construct and related constructs now allow for
duplication of column labels and columns themselves in the columns clause,
mirroring exactly how column expressions were passed in. This allows
the tuples returned by an executed result to match what was SELECTed
for in the first place, which is how the ORM :class:`.Query` works, so
this establishes better cross-compatibility between the two constructs.
Additionally, it allows column-positioning-sensitive structures such as
UNIONs (i.e. :class:`.CompoundSelect`) to be more intuitively constructed
in those cases where a particular column might appear in more than one
place. To support this change, the :class:`.ColumnCollection` has been
revised to support duplicate columns as well as to allow integer index
access.
Fixes: #4753
Change-Id: Ie09a8116f05c367995c1e43623c51e07971d3bf0
As part of the SQLAlchemy 2.0 migration project, a conceptual change has
been made to the role of the :class:`.SelectBase` class hierarchy,
which is the root of all "SELECT" statement constructs, in that they no
longer serve directly as FROM clauses, that is, they no longer subclass
:class:`.FromClause`. For end users, the change mostly means that any
placement of a :func:`.select` construct in the FROM clause of another
:func:`.select` requires first that it be wrapped in a subquery first,
which historically is through the use of the :meth:`.SelectBase.alias`
method, and is now also available through the use of
:meth:`.SelectBase.subquery`. This was usually a requirement in any
case since several databases don't accept unnamed SELECT subqueries
in their FROM clause in any case.
See the documentation in this change for lots more detail.
Fixes: #4617
Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
Fixed an unlikely issue where the "corresponding column" routine for unions
and other :class:`.CompoundSelect` objects could return the wrong column in
some overlapping column situtations, thus potentially impacting some ORM
operations when set operations are in use, if the underlying
:func:`.select` constructs were used previously in other similar kinds of
routines, due to a cached value not being cleared.
Fixes: #4747
Change-Id: I7fb134cac3604f8fe62e220fb24a0945d0a1c56f
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient. test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.
Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
Fixed a series of quoting issues which all stemmed from the concept of the
:func:`.literal_column` construct, which when being "proxied" through a
subquery to be referred towards by a label that matches its text, the label
would not have quoting rules applied to it, even if the string in the
:class:`.Label` were set up as a :class:`.quoted_name` construct. Not
applying quoting to the text of the :class:`.Label` is a bug because this
text is strictly a SQL identifier name and not a SQL expression, and the
string should not have quotes embedded into it already unlike the
:func:`.literal_column` which it may be applied towards. The existing
behavior of a non-labeled :func:`.literal_column` being propagated as is on
the outside of a subquery is maintained in order to help with manual
quoting schemes, although it's not clear if valid SQL can be generated for
such a construct in any case.
Fixes: #4730
Change-Id: I300941f27872fc4298c74a1d1ed65aef1a5cdd82
The Alias object no longer has "element" and "original", it now
has "wrapped" and "element" (the name .original is also left
as a descriptor for legacy access by third party dialects).
These two data members refer to the
dual roles Alias needs to play, where in the Python sense it needs
to refer to the thing it was applied against directly, whereas in the
SQL sense it needs to refer to the ultimate "non-alias" thing it
refers towards. Both are necessary to maintain. However, the change
here has each Alias object access the non-Alias object immediately
so that the "unwrapping" is simpler and does not need any special
logic.
In the SQL sense, Alias objects don't nest, the only potential
was that of the CTE, however there is no such thing as
a nested CTE, see link below.
This change is an interim change along the way to breaking Alias
into more classes and breaking away Select objects from being
FromClause objects.
Change-Id: Ie7a0d064226cb074ca745505129b5ec7d879e389
References: https://stackoverflow.com/questions/1413516/can-you-create-nested-with-clauses-for-common-table-expressions
as SELECT statements will have subquery() and not alias(),
start getting ready for the places where the ORM coerces SELECTs
into subqueries and be ready to warn about it
Change-Id: I90d4b6cae2c72816c6b192016ce074589caf4731
A major refactoring of all the functions handle all detection of
Core argument types as well as perform coercions into a new class hierarchy
based on "roles", each of which identify a syntactical location within a
SQL statement. In contrast to the ClauseElement hierarchy that identifies
"what" each object is syntactically, the SQLRole hierarchy identifies
the "where does it go" of each object syntactically. From this we define
a consistent type checking and coercion system that establishes well
defined behviors.
This is a breakout of the patch that is reorganizing select()
constructs to no longer be in the FromClause hierarchy.
Also includes a rename of as_scalar() into scalar_subquery(); deprecates
automatic coercion to scalar_subquery().
Partially-fixes: #4617
Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
Fixed that the :class:`.GenericFunction` class was inadvertently
registering itself as one of the named functions. Pull request courtesy
Adrien Berchet.
Fixes: #4653Closes: #4654
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4654
Pull-request-sha: 1112b89f0d
Change-Id: Ia0d366d3bff44a763aa496287814278dff732a19
Registered function names based on :class:`.GenericFunction` are now
retrieved in a case-insensitive fashion in all cases, removing the
deprecation logic from 1.3 which temporarily allowed multiple
:class:`.GenericFunction` objects to exist with differing cases. A
:class:`.GenericFunction` that replaces another on the same name whether or
not it's case sensitive emits a warning before replacing the object.
Fixes: #4649
Change-Id: I265ae19833132db07ed5b5ae40c4d24f659b1ab3
sumau