Fixed issue where a plain pickle dumps call of the :class:`_sql.Over`
construct didn't work.
Fixes: #5644
Change-Id: I4b07f74ecd5d52f0794128585367012200a38a36
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.
Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
As the test suite has widespread use of many patterns
that are deprecated, enable SQLALCHEMY_WARN_20 globally
for the test suite but then break the warnings filter
out into a whole list of all the individual warnings
we are looking for. this way individual changesets
can target a specific class of warning, as many of these
warnings will indivdidually affect dozens of files
and potentially hundreds of lines of code.
Many warnings are also resolved here as this
patch started out that way. From this point
forward there should be changesets that target a
subset of the warnings at a time.
For expediency, updates some migration 2.0 docs
for ORM as well.
Change-Id: I98b8defdf7c37b818b3824d02f7668e3f5f31c94
Make optional sequences render as identity in mssql
Remove unused dialect option sequence_default_column_type
Change-Id: I821eeffcb442f8d1b69186a9b798b15c3d8d6ff3
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function. it does not
yet fully address other issues such as keyword arguments passed
to the table.select().
Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.
Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
The psycopg2 dialect now defaults to using the very performant
``execute_values()`` psycopg2 extension for compiled INSERT statements,
and also impements RETURNING support when this extension is used. This
allows INSERT statements that even include an autoincremented SERIAL
or IDENTITY value to run very fast while still being able to return the
newly generated primary key values. The ORM will then integrate this
new feature in a separate change.
Implements RETURNING for insert with executemany
Adds support to return_defaults() mode and inserted_primary_key
to support mutiple INSERTed rows, via return_defauls_rows
and inserted_primary_key_rows accessors.
within default execution context, new cached compiler
getters are used to fetch primary keys from rows
inserted_primary_key now returns a plain tuple. this
is not yet a row-like object however this can be
added.
Adds distinct "values_only" and "batch" modes, as
"values" has a lot of benefits but "batch" breaks
cursor.rowcount
psycopg2 minimum version 2.7 so we can remove the
large number of checks for very old versions of
psycopg2
simplify tests to no longer distinguish between
native and non-native json
Fixes: #5401
Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
Added support for "CREATE SEQUENCE" and full :class:`.Sequence` support for
Microsoft SQL Server. This removes the deprecated feature of using
:class:`.Sequence` objects to manipulate IDENTITY characteristics which
should now be performed using ``mssql_identity_start`` and
``mssql_identity_increment`` as documented at :ref:`mssql_identity`. The
change includes a new parameter :paramref:`.Sequence.data_type` to
accommodate SQL Server's choice of datatype, which for that backend
includes INTEGER and BIGINT. The default starting value for SQL Server's
version of :class:`.Sequence` has been set at 1; this default is now
emitted within the CREATE SEQUENCE DDL for all backends.
Fixes: #4235Fixes: #4633
Change-Id: I6aa55c441e8146c2f002e2e201a7f645e667b916
Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf
Trying to see if by making the cache key memoized, we
still can have the older "identity" form of caching
which is the cheapest of all, at the same time as the
newer "cache key each time" version that is not nearly
as cheap; but still much cheaper than no caching at all.
Also needed is a per-execution update of _keymap when
we invoke from a cached select, so that Column objects
that are anonymous or otherwise adapted will match up.
this is analogous to the adaption of bound parameters
from the cache key.
Adds test coverage for the keymap / construct_params()
changes related to caching. Also hones performance
to a large extent for statement construction and
cache key generation.
Also includes a new memoized attribute
approach that vastly simplifies the previous approach
of "group_expirable_memoized_property" and finally
integrates cleanly with _clone(), _generate(), etc.
no more hardcoding of attributes is needed, as well
as that most _reset_memoization() calls are no longer
needed as the reset is inherent in a _generate() call;
this also has dramatic performance improvements.
Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
This builds on cc718cccc0 which moved
RowProxy to Row, allowing Row to be more like a named tuple.
- KeyedTuple in ORM is replaced with Row
- ResultSetMetaData broken out into "simple" and "cursor" versions
for ORM and Core, as well as LegacyCursor version.
- Row now has _mapping attribute that supplies full mapping behavior.
Row and SimpleRow both have named tuple behavior otherwise.
LegacyRow has some mapping features on the tuple which emit
deprecation warnings (e.g. keys(), values(), etc). the biggest
change for mapping->tuple is the behavior of __contains__ which
moves from testing of "key in row" to "value in row".
- ResultProxy breaks into ResultProxy and FutureResult (interim),
the latter has the newer APIs. Made available to dialects
using execution options.
- internal reflection methods and most tests move off of implicit
Row mapping behavior and move to row._mapping, result.mappings()
method using future result
- a new strategy system for cursor handling replaces the various
subclasses of RowProxy
- some execution context adjustments. We will leave EC in but
refined things like get_result_proxy() and out parameter handling.
Dialects for 1.4 will need to adjust from get_result_proxy()
to get_result_cursor_strategy(), if they are using this method
- out parameter handling now accommodated by get_out_parameter_values()
EC method. Oracle changes for this. external dialect for
DB2 for example will also need to adjust for this.
- deprecate case_insensitive flag for engine / result, this
feature is not used
mapping-methods on Row are deprecated, and replaced with
Row._mapping.<meth>, including:
row.keys() -> use row._mapping.keys()
row.items() -> use row._mapping.items()
row.values() -> use row._mapping.values()
key in row -> use key in row._mapping
int in row -> use int < len(row)
Fixes: #4710Fixes: #4878
Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
A function created using :class:`.GenericFunction` can now specify that the
name of the function should be rendered with or without quotes by assigning
the :class:`.quoted_name` construct to the .name element of the object.
Prior to 1.3.4, quoting was never applied to function names, and some
quoting was introduced in 🎫`4467` but no means to force quoting for
a mixed case name was available. Additionally, the :class:`.quoted_name`
construct when used as the name will properly register its lowercase name
in the function registry so that the name continues to be available via the
``func.`` registry.
Fixes: #5079
Change-Id: I0653ab8b16e75e628ce82dbbc3d0f77f8336c407
In order for text(), custom compiled objects, etc. to be usable
by Query(), they are all targeted by object key in the result map.
As we no longer want Query to implicitly label these, as well as that
text() has no label feature, support adding entries to the result
map that have no name, key, or type, only the object itself, and
then ensure that the compiler sets up for positional targeting
when this condition is detected.
Allows for more flexible ORM query usage with custom expressions
and text() while having less special logic in query itself.
Fixes: #4887
Change-Id: Ie073da127d292d43cb132a2b31bc90af88bfe2fd
Fixed issue where the :class:`.array_agg` construct in combination with
:meth:`.FunctionElement.filter` would not produce the correct operator
precedence between the FILTER keyword and the array index operator.
Fixes: #4760
Change-Id: Ic662cd3da3330554ec673bafd80495b3f1506098
As part of the SQLAlchemy 2.0 migration project, a conceptual change has
been made to the role of the :class:`.SelectBase` class hierarchy,
which is the root of all "SELECT" statement constructs, in that they no
longer serve directly as FROM clauses, that is, they no longer subclass
:class:`.FromClause`. For end users, the change mostly means that any
placement of a :func:`.select` construct in the FROM clause of another
:func:`.select` requires first that it be wrapped in a subquery first,
which historically is through the use of the :meth:`.SelectBase.alias`
method, and is now also available through the use of
:meth:`.SelectBase.subquery`. This was usually a requirement in any
case since several databases don't accept unnamed SELECT subqueries
in their FROM clause in any case.
See the documentation in this change for lots more detail.
Fixes: #4617
Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
Fixed that the :class:`.GenericFunction` class was inadvertently
registering itself as one of the named functions. Pull request courtesy
Adrien Berchet.
Fixes: #4653Closes: #4654
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4654
Pull-request-sha: 1112b89f0d
Change-Id: Ia0d366d3bff44a763aa496287814278dff732a19
Registered function names based on :class:`.GenericFunction` are now
retrieved in a case-insensitive fashion in all cases, removing the
deprecation logic from 1.3 which temporarily allowed multiple
:class:`.GenericFunction` objects to exist with differing cases. A
:class:`.GenericFunction` that replaces another on the same name whether or
not it's case sensitive emits a warning before replacing the object.
Fixes: #4649
Change-Id: I265ae19833132db07ed5b5ae40c4d24f659b1ab3
The :class:`.GenericFunction` namespace is being migrated so that function
names are looked up in a case-insensitive manner, as SQL functions do not
collide on case sensitive differences nor is this something which would
occur with user-defined functions or stored procedures. Lookups for
functions declared with :class:`.GenericFunction` now use a case
insensitive scheme, however a deprecation case is supported which allows
two or more :class:`.GenericFunction` objects with the same name of
different cases to exist, which will cause case sensitive lookups to occur
for that particular name, while emitting a warning at function registration
time. Thanks to Adrien Berchet for a lot of work on this complicated
feature.
Fixes: #4569Closes: #4570
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4570
Pull-request-sha: 37d4f3322b
Change-Id: Ief07c6eb55bf398f6aad85b60ef13ee6d1173109
Fully removed the behavior of strings passed directly as components of a
:func:`.select` or :class:`.Query` object being coerced to :func:`.text`
constructs automatically; the warning that has been emitted is now an
ArgumentError or in the case of order_by() / group_by() a CompileError.
This has emitted a warning since version 1.0 however its presence continues
to create concerns for the potential of mis-use of this behavior.
Note that public CVEs have been posted for order_by() / group_by() which
are resolved by this commit: CVE-2019-7164 CVE-2019-7548
Added "SQL phrase validation" to key DDL phrases that are accepted as plain
strings, including :paramref:`.ForeignKeyConstraint.on_delete`,
:paramref:`.ForeignKeyConstraint.on_update`,
:paramref:`.ExcludeConstraint.using`,
:paramref:`.ForeignKeyConstraint.initially`, for areas where a series of SQL
keywords only are expected.Any non-space characters that suggest the phrase
would need to be quoted will raise a :class:`.CompileError`. This change
is related to the series of changes committed as part of 🎫`4481`.
Fixed issue where using an uppercase name for an index type (e.g. GIST,
BTREE, etc. ) or an EXCLUDE constraint would treat it as an identifier to
be quoted, rather than rendering it as is. The new behavior converts these
types to lowercase and ensures they contain only valid SQL characters.
Quoting is applied to :class:`.Function` names, those which are usually but
not necessarily generated from the :attr:`.sql.func` construct, at compile
time if they contain illegal characters, such as spaces or punctuation. The
names are as before treated as case insensitive however, meaning if the
names contain uppercase or mixed case characters, that alone does not
trigger quoting. The case insensitivity is currently maintained for
backwards compatibility.
Fixes: #4481Fixes: #4473Fixes: #4467
Change-Id: Ib22a27d62930e24702e2f0f7c74a0473385a08eb
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Fixes to the test suite, a few errant imports, and setup.py:
- mysql and postgresql have unused 'json' imports; remove
- postgresql is exporting the 'json' symbol, remove
- make sure setup.py can find __version__ using " or '
- retry logic in provision create database for postgresql fixed
- refactor test_magazine to use cls.tables rather than globals
- remove unused class in test_scoping
- add a comment to test_deprecations that this test suite itself
is deprecated
- don't use mapper() and orm_mapper() in test_unitofwork, just
use mapper()
- remove dupe test_scalar_set_None test in test_attributes
- Python 2.7 and above includes unittest.SkipTest, remove pre-2.7
fallback
- use imported SkipTest in profiling
- declarative test_reflection tests with "reflectable_autoincrement"
already don't run on oracle or firebird; remove conditional logic
for these, which also removes an "id" symbol
- clean up test in test_functions, remove print statement
- remove dupe test_literal_processor_coercion_native_int_out_of_range
in test/sql/test_types.py
- fix psycopg2_hstore ref
Change-Id: I7b3444f8546aac82be81cd1e7b6d8b2ad6834fe6
Amended the :class:`.AnsiFunction` class, the base of common SQL
functions like ``CURRENT_TIMESTAMP``, to accept positional arguments
like a regular ad-hoc function. This to suit the case that many of
these functions on specific backends accept arguments such as
"fractional seconds" precision and such. If the function is created
with arguments, it renders the the parenthesis and the arguments. If
no arguents are present, the compiler generates the non-parenthesized form.
Fixes: #4386
Change-Id: Ic492ef177e4987cec99ec4d95f55292be8daa087
Added missing window function parameters
:paramref:`.WithinGroup.over.range_` and :paramref:`.WithinGroup.over.rows`
parameters to the :meth:`.WithinGroup.over` and
:meth:`.FunctionFilter.over` methods, to correspond to the range/rows
feature added to the "over" method of SQL functions as part of
🎫`3049` in version 1.1.
Fixes: #4322
Change-Id: I77dcdac65c699a4b52a3fc3ee09a100ffb4fc20e
Added new feature :meth:`.FunctionElement.as_comparison` which allows a SQL
function to act as a binary comparison operation that can work within the
ORM.
Change-Id: I07018e2065d09775c0406cabdd35fc38cc0da699
Fixes: #3831
Fixed bug in :func:`.array_agg` function where passing an argument
that is already of type :class:`.ARRAY`, such as a Postgresql
:obj:`.postgresql.array` construct, would produce a ``ValueError``, due
to the function attempting to nest the arrays.
Change-Id: Ibe5f6275d90e4868e6ef8a733de05acd44c05d78
Fixes: #4107
One test appears to use some awkward calling style
with the current_date function that isn't working in pg10
anymore, this looks like an extremely
old test that can be removed
Change-Id: I5f8aee0f5ed423461be5a9060c812eb0acdc7df5
persistence of JSON values in MySQL as well as basic operator support
of "getitem" and "getpath", making use of the ``JSON_EXTRACT``
function in order to refer to individual paths in a JSON structure.
fixes#3547
- Added a new type to core :class:`.types.JSON`. This is the
base of the PostgreSQL :class:`.postgresql.JSON` type as well as that
of the new :class:`.mysql.JSON` type, so that a PG/MySQL-agnostic
JSON column may be used. The type features basic index and path
searching support.
fixes#3619
- reorganization of migration docs etc. to try to refer both to
the fixes to JSON that helps Postgresql while at the same time
indicating these are new features of the new base JSON type.
- a rework of the Array/Indexable system some more, moving things
that are specific to Array out of Indexable.
- new operators for JSON indexing added to core so that these can
be compiled by the PG and MySQL dialects individually
- rename sqltypes.Array to sqltypes.ARRAY - as there is no generic
Array implementation, this is an uppercase type for now, consistent
with the new sqltypes.JSON type that is also not a generic implementation.
There may need to be some convention change to handle the case of
datatypes that aren't generic, rely upon DB-native implementations,
but aren't necessarily all named the same thing.
``<function> WITHIN GROUP (ORDER BY <criteria>)``, using the
method :class:`.FunctionElement.within_group`. A series of common
set-aggregate functions with return types derived from the set have
been added. This includes functions like :class:`.percentile_cont`,
:class:`.dense_rank` and others.
fixes#1370
- make sure we use func.name for all _literal_as_binds in functions.py
so we get consistent naming behavior for parameters.
which automatically returns an :class:`.Array` of the correct type
and supports index / slice operations. As arrays are only
supported on Postgresql at the moment, only actually works on
Postgresql. fixes#3132
of :class:`.FunctionElement` or other column element that incorrectly
states 'None' or any other invalid object as the ``.type``
attribute will report this exception instead of recursion overflow.
fixes#3485
constructs are now importable from the "from sqlalchemy" namespace,
just like every other Core construct.
- The implicit conversion of strings to :func:`.text` constructs
when passed to most builder methods of :func:`.select` as
well as :class:`.Query` now emits a warning with just the
plain string sent. The textual conversion still proceeds normally,
however. The only method that accepts a string without a warning
are the "label reference" methods like order_by(), group_by();
these functions will now at compile time attempt to resolve a single
string argument to a column or label expression present in the
selectable; if none is located, the expression still renders, but
you get the warning again. The rationale here is that the implicit
conversion from string to text is more unexpected than not these days,
and it is better that the user send more direction to the Core / ORM
when passing a raw string as to what direction should be taken.
Core/ORM tutorials have been updated to go more in depth as to how text
is handled.
fixes#2992
e.g. the ``func`` construct. Previously, behavior for this method
was undefined. The current behavior mimics that of pre-0.9.4,
which is that the function is turned into a single-column FROM
clause with the given alias name, where the column itself is
anonymously named.
fixes#3137
is currently being supported in addition to nose, and will likely
be preferred to nose going forward. The nose plugin system used
by SQLAlchemy has been split out so that it works under pytest as
well. There are no plans to drop support for nose at the moment
and we hope that the test suite itself can continue to remain as
agnostic of testing platform as possible. See the file
README.unittests.rst for updated information on running tests
with pytest.
The test plugin system has also been enhanced to support running
tests against mutiple database URLs at once, by specifying the ``--db``
and/or ``--dburi`` flags multiple times. This does not run the entire test
suite for each database, but instead allows test cases that are specific
to certain backends make use of that backend as the test is run.
When using pytest as the test runner, the system will also run
specific test suites multiple times, once for each database, particularly
those tests within the "dialect suite". The plan is that the enhanced
system will also be used by Alembic, and allow Alembic to run
migration operation tests against multiple backends in one run, including
third-party backends not included within Alembic itself.
Third party dialects and extensions are also encouraged to standardize
on SQLAlchemy's test suite as a basis; see the file README.dialects.rst
for background on building out from SQLAlchemy's test platform.
into the names used by standard functions in :mod:`sqlalchemy.sql.functions`,
such as ``func.coalesce()`` and ``func.max()``. Using these functions
in ORM attributes and thus producing annotated versions of them could
corrupt the actual function name rendered in the SQL. [ticket:2927]
become an externally usable package but still remains within the main sqlalchemy parent package.
in this system, we use kind of an ugly hack to get the noseplugin imported outside of the
"sqlalchemy" package, while still making it available within sqlalchemy for usage by
third party libraries.
will now be produced via the func.* accessor
as well, as users naturally try to access these
names from func.* they might as well do
what's expected, even though the returned
object is not a FunctionElement.
[ticket:2562]
True by default, if not passed explicitly,
on bindparam() if the "value" or "callable"
parameters are not passed.
This will cause statement execution to check
for the parameter being present in the final
collection of bound parameters, rather than
implicitly assigning None. [ticket:2556]
Mike Bayer