Adjustments made to the BLOB / CLOB / NCLOB datatypes in the cx_Oracle and
oracledb dialects, to improve performance based on recommendations from
Oracle developers.
References: https://github.com/oracle/python-cx_Oracle/issues/596Fixes: #7494
Change-Id: I0d8cc3579140aa65cacf5b7d3373f7e1929a8f85
As almost every dialect supports RETURNING now, RETURNING
is also made more of a default assumption.
* the default compiler generates a RETURNING clause now
when specified; CompileError is no longer raised.
* The dialect-level implicit_returning parameter now has
no effect. It's not fully clear if there are real world
cases relying on the dialect-level parameter, so we will see
once 2.0 is released. ORM-level RETURNING can be disabled
at the table level, and perhaps "implicit returning" should
become an ORM-level option at some point as that's where
it applies.
* Altered ORM update() / delete() to respect table-level
implicit returning for fetch.
* Since MariaDB doesnt support UPDATE returning, "full_returning"
is now split into insert_returning, update_returning, delete_returning
* Crazy new thing. Dialects that have *both* cursor.lastrowid
*and* returning. so now we can pick between them for SQLite
and mariadb. so, we are trying to keep it on .lastrowid for
simple inserts with an autoincrement column, this helps with
some edge case test scenarios and i bet .lastrowid is faster
anyway. any return_defaults() / multiparams etc then we
use returning
* SQLite decided they dont want to return rows that match in
ON CONFLICT. this is flat out wrong, but for now we need to
work with it.
Fixes: #6195Fixes: #7011Closes: #7047
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7047
Pull-request-sha: d25d5ea3ab
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I9908ce0ff7bdc50bd5b27722081767c31c19a950
Added new backend-agnostic :class:`_types.Uuid` datatype generalized from
the PostgreSQL dialects to now be a core type, as well as migrated
:class:`_types.UUID` from the PostgreSQL dialect. Thanks to Trevor Gross
for the help on this.
also includes:
* corrects some missing behaviors in the suite literal fixtures
test where row round trips weren't being correctly asserted.
* fixes some of the ISO literal date rendering added in
952383f9ee for #5052 to truncate datetime strings for date/time
datatypes in the same way that drivers typically do for bound
parameters; this was not working fully and wasn't caught by the
broken test fixture
Fixes: #7212
Change-Id: I981ac6d34d278c18281c144430a528764c241b04
Fixed SQL compiler issue where the "bind processing" function for a bound
parameter would not be correctly applied to a bound value if the bound
parameter's name were "escaped". Concretely, this applies, among other
cases, to Oracle when a :class:`.Column` has a name that itself requires
quoting, such that the quoting-required name is then used for the bound
parameters generated within DML statements, and the datatype in use
requires bind processing, such as the :class:`.Enum` datatype.
Fixes: #8053
Change-Id: I39d060a87e240b4ebcfccaa9c535e971b7255d99
this test is failing on CI with "##foo does not exist",
so hypothesize there's some kind of race condition with
global temp table names.
Change-Id: I8c6c26a7fda70f67735ce20af67373c311e48731
The :meth:`.Operators.match` operator now uses ``plainto_tsquery()`` for
PostgreSQL full text search, rather than ``to_tsquery()``. The rationale
for this change is to provide better cross-compatibility with match on
other database backends. Full support for all PostgreSQL full text
functions remains available through the use of :data:`.func` in
conjunction with :meth:`.Operators.bool_op` (an improved version of
:meth:`.Operators.op` for boolean operators).
Additional doc updates here apply to 1.4 so will backport these
out to a separate commit.
Fixes: #7086
Change-Id: I1946075daf5d9c558e85f73f1bf852604b3b1b8c
These are replaced by the read-only ManagesConnection.dbapi_connection
attribute.
For some reason both of these objects had "setter" for .connection
as well; there's no use case for that at all so just remove
setter logic entirely.
Fixes: #6981
Change-Id: I6425de4a017f6370e1a7476cd491cabc55e55e67
Explicitly specify the collation when reflecting table columns using
MSSQL to prevent "collation conflict" errors.
Fixes: #8035
Change-Id: I4239a5ca8b041f56d7b3bba67b3357c176db31ee
Added disconnect code for MySQL error 4031, introduced in MySQL >= 8.0.24,
indicating connection idle timeout exceeded. In particular this repairs an
issue where pre-ping could not reconnect on a timed-out connection. Pull
request courtesy valievkarim.
Fixes: #8036Closes: #8037
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8037
Pull-request-sha: 7ab605c2d2
Change-Id: I21249c9d8acb305ac43ce61b90b41daf7fabdfe8
Fixed bug where the PostgreSQL :meth:`_postgresql.Insert.on_conflict`
method and the SQLite :meth:`_sqlite.Insert.on_conflict` method would both
fail to correctly accommodate a column with a separate ".key" when
specifying the column using its key name in the dictionary passed to
``set_``, as well as if the :attr:`_sqlite.Insert.excluded` or
:attr:`_postgresql.Insert.excluded` collection were used as the dictionary
directly.
Fixes: #8014
Change-Id: I67226aeedcb2c683e22405af64720cc1f990f274
Fixed bug in :class:`.ARRAY` datatype in combination with :class:`.Enum` on
PostgreSQL where using the ``.any()`` method to render SQL ANY(), given
members of the Python enumeration as arguments, would produce a type
adaptation failure on all drivers.
Fixes: #6515
Change-Id: Ia1e3b4e10aaf264ed436ce6030d105fc60023433
Implemented :attr:`_postgresql.UUID.python_type` attribute for the
:class:`_postgresql.UUID` type object. The attribute will return either
``str`` or ``uuid.UUID`` based on the :paramref:`_postgresql.UUID.as_uuid`
parameter setting. Previously, this attribute was unimplemented. Pull
request courtesy Alex Grönholm.
Fixes: #7943Closes: #7944
Change-Id: Ic4fbaeee134d586b08339801968e787cc7e14285
implement strict typing for schema.py
this module has lots of public API, lots of old decisions
and very hard to follow construction sequences in many
cases, and is also where we get a lot of new feature requests,
so strict typing should help keep things clean.
among improvements here, fixed the pool .info getters
and also figured out how to get ColumnCollection and
related to be covariant so that we may set them up
as returning Column or ColumnClause without any conflicts.
DDL was affected, noting that superclasses of DDLElement
(_DDLCompiles, added recently) can now be passed into
"ddl_if" callables; reorganized ddl into ExecutableDDLElement
as a new name for DDLElement and _DDLCompiles renamed to
BaseDDLElement.
setting up strict also located an API use case that
is completely broken, which is connection.execute(some_default)
returns a scalar value. This case has been deprecated
and new paths have been set up so that connection.scalar()
may be used. This likely wasn't possible in previous
versions because scalar() would assume a CursorResult.
The scalar() change also impacts Session as we have explicit
support (since someone had reported it as a regression)
for session.execute(Sequence()) to work. They will get the
same deprecation message (which omits the word "Connection",
just uses ".execute()" and ".scalar()") and they can then
use Session.scalar() as well. Getting this to type
correctly while still supporting ORM use cases required
some refactoring, and I also set up a keyword only delimeter
for Session.execute() and related as execution_options /
bind_arguments should always be keyword only, applied these
changes to AsyncSession as well.
Additionally simpify Table __init__ now that we are Python
3 only, we can have positional plus explicit kwargs finally.
Simplify Column.__init__ as well again taking advantage
of kw only arguments.
Fill in most/all __init__ methods in sqltypes.py as
the constructor for types is most of the API. should
likely do this for dialect-specific types as well.
Apply _InfoType for all info attributes as should have been
done originally and update descriptor decorators.
Change-Id: I3f9f8ff3f1c8858471ff4545ac83d68c88107527
Improved the construction of SQL binary expressions to allow for very long
expressions against the same associative operator without special steps
needed in order to avoid high memory use and excess recursion depth. A
particular binary operation ``A op B`` can now be joined against another
element ``op C`` and the resulting structure will be "flattened" so that
the representation as well as SQL compilation does not require recursion.
To implement this more cleanly, the biggest change here is that
column-oriented lists of things are broken away from ClauseList
in a new class ExpressionClauseList, that also forms the basis
of BooleanClauseList. ClauseList is still used for the generic
"comma-separated list" of things such as Tuple and things like
ORDER BY, as well as in some API endpoints.
Also adds __slots__ to the TypeEngine-bound Comparator
classes. Still can't really do __slots__ on ClauseElement.
Fixes: #7744
Change-Id: I81a8ceb6f8f3bb0fe52d58f3cb42e4b6c2bc9018
Fixed an issue what would cause autocommit mode to be reset
when using pre_ping in conjunction engine level autocommit
on the psycopg2 driver.
Fixes: #7930
Change-Id: I4cccaf1b7f8cbacd853689458080784114fcc390
Added modified ISO-8601 rendering (i.e. ISO-8601 with the T converted to a
space) when using ``literal_binds`` with the SQL compilers provided by the
PostgreSQL, MySQL, MariaDB, MSSQL, Oracle dialects. For Oracle, the ISO
format is wrapped inside of an appropriate TO_DATE() function call.
Previously this rendering was not implemented for dialect-specific
compilation.
Fixes: #5052
Change-Id: I7af15a51fedf5c5a8e76e645f7c3be997ece35f0
Full "RETURNING" support is implemented for the cx_Oracle dialect, meaning
multiple RETURNING rows are now recived for DML statements that produce
more than one row for RETURNING.
cx_Oracle 7 is now the minimum version for cx_Oracle.
Getting Oracle to do multirow returning took about 5 minutes. however,
getting Oracle's RETURNING system to integrate with ORM-enabled
insert, update, delete, is a big deal because that architecture wasn't
really working very robustly, including some recent changes in 1.4
for FromStatement were done in a hurry, so this patch also cleans up
the FromStatement situation and begins to establish it more concretely
as the base for all ReturnsRows / TextClause ORM scenarios.
Fixes: #6245
Change-Id: I2b4e6007affa51ce311d2d5baa3917f356ab961f
SQLite datetime, date, and time datatypes now use Python standard lib
``fromisoformat()`` methods in order to parse incoming datetime, date, and
time string values. This improves performance vs. the previous regular
expression-based approach, and also automatically accommodates for datetime
and time formats that contain either a six-digit "microseconds" format or a
three-digit "milliseconds" format.
Fixes: #7029
Change-Id: I67aab4fe5ee3055e5996050cf4564981413cc221
Scaled back a fix made for 🎫`6581` where "executemany values" mode
for psycopg2 were disabled for all "ON CONFLICT" styles of INSERT, to
not apply to the "ON CONFLICT DO NOTHING" clause, which does not include
any parameters and is safe for "executemany values" mode. "ON CONFLICT
DO UPDATE" is still blocked from "executemany values" as there may
be additional parameters in the DO UPDATE clause that cannot be batched
(which is the original issue fixed by 🎫`6581`).
Fixes: #7880
Change-Id: Id3e23a0c6699333409a50148fa8923cb8e564bdc
both black and click were released in the past
few hours, and black 21.5b1 seems to suddenly
be failing on a missing symbol from click. just
update to the latest
Change-Id: Idf76732479a264f7f2245699a6bdaff018e3a123
Fixed regression caused by 🎫`7760` where the new capabilities of
:class:`.TextualSelect` were not fully implemented within the compiler
properly, leading to issues with composed INSERT constructs such as "INSERT
FROM SELECT" and "INSERT...ON CONFLICT" when combined with CTE and textual
statements.
Fixes: #7798
Change-Id: Ia2ce92507e574dd36fd26dd38ec9dd2713584467
Fixed issues in :class:`_mysql.SET` datatype as well as :class:`.Enum`
where the ``__repr__()`` method would not render all optional parameters in
the string output, impacting the use of these types in Alembic
autogenerate. Pull request for MySQL courtesy Yuki Nishimine.
Fixes: #7720Fixes: #7789Closes: #7772
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7772
Pull-request-sha: d58845479f
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: Idcec23eab4258511d9f32f4e3d78e511ea6021f1
All modules in sqlalchemy.engine are strictly
typed with the exception of cursor, default, and
reflection. cursor and default pass with non-strict
typing, reflection is waiting on the multi-reflection
refactor.
Behavioral changes:
* create_connect_args() methods return a tuple of list,
dict, rather than a list of list, dict
* removed allow_chars parameter from
pyodbc connector ._get_server_version_info()
method
* the parameter list passed to do_executemany is now
a list in all cases. previously, this was being run
through dialect.execute_sequence_format, which
defaults to tuple and was only intended for individual
tuple params.
* broke up dialect.dbapi into dialect.import_dbapi
class method and dialect.dbapi module object. added
a deprecation path for legacy dialects. it's not
really feasible to type a single attr as a classmethod
vs. module type. The "type_compiler" attribute also
has this problem with greater ability to work around,
left that one for now.
* lots of constants changing to be Enum, so that we can
type them. for fixed tuple-position constants in
cursor.py / compiler.py (which are used to avoid the
speed overhead of namedtuple), using Literal[value]
which seems to work well
* some tightening up in Row regarding __getitem__, which
we can do since we are on full 2.0 style result use
* altered the set_connection_execution_options and
set_engine_execution_options event flows so that the
dictionary of options may be mutated within the event
hook, where it will then take effect as the actual
options used. Previously, changing the dict would
be silently ignored which seems counter-intuitive
and not very useful.
* A lot of DefaultDialect/DefaultExecutionContext
methods and attributes, including underscored ones, move
to interfaces. This is not fully ideal as it means
the Dialect/ExecutionContext interfaces aren't publicly
subclassable directly, but their current purpose
is more of documentation for dialect authors who should
(and certainly are) still be subclassing the DefaultXYZ
versions in all cases
Overall, Result was the most extremely difficult class
hierarchy to type here as this hierarchy passes through
largely amorphous "row" datatypes throughout, which
can in fact by all kinds of different things, like
raw DBAPI rows, or Row objects, or "scalar"/Any, but
at the same time these types have meaning so I tried still
maintaining some level of semantic markings for these,
it highlights how complex Result is now, as it's trying
to be extremely efficient and inlined while also being
very open-ended and extensible.
Change-Id: I98b75c0c09eab5355fc7a33ba41dd9874274f12a
Added :class:`.Double`, :class:`.DOUBLE`, :class:`.DOUBLE_PRECISION`
datatypes to the base ``sqlalchemy.`` module namespace, for explicit use of
double/double precision as well as generic "double" datatypes. Use
:class:`.Double` for generic support that will resolve to DOUBLE/DOUBLE
PRECISION/FLOAT as needed for different backends.
Implemented DDL and reflection support for ``FLOAT`` datatypes which
include an explicit "binary_precision" value. Using the Oracle-specific
:class:`_oracle.FLOAT` datatype, the new parameter
:paramref:`_oracle.FLOAT.binary_precision` may be specified which will
render Oracle's precision for floating point types directly. This value is
interpreted during reflection. Upon reflecting back a ``FLOAT`` datatype,
the datatype returned is one of :class:`_types.DOUBLE_PRECISION` for a
``FLOAT`` for a precision of 126 (this is also Oracle's default precision
for ``FLOAT``), :class:`_types.REAL` for a precision of 63, and
:class:`_oracle.FLOAT` for a custom precision, as per Oracle documentation.
As part of this change, the generic :paramref:`_sqltypes.Float.precision`
value is explicitly rejected when generating DDL for Oracle, as this
precision cannot be accurately converted to "binary precision"; instead, an
error message encourages the use of
:meth:`_sqltypes.TypeEngine.with_variant` so that Oracle's specific form of
precision may be chosen exactly. This is a backwards-incompatible change in
behavior, as the previous "precision" value was silently ignored for
Oracle.
Fixes: #5465Closes: #7674
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7674
Pull-request-sha: 5c68419e5a
Change-Id: I831f4af3ee3b23fde02e8f6393c83e23dd7cd34d
Added support to parse "DPI" error codes from cx_Oracle exception objects
such as ``DPI-1080`` and ``DPI-1010``, both of which now indicate a
disconnect scenario as of cx_Oracle 8.3.
Fixes: #7748
Change-Id: I4a10d606d512c0d7f9b4653c47ea5734afffb8a5
Fixed issue where SQLite unique constraint reflection would not work
for an inline UNIQUE constraint where the column name had an underscore
in its name.
Added support for reflecting SQLite inline unique constraints where
the column names are formatted with SQLite "escape quotes" ``[]``
or `` ` ``, which are discarded by the database when producing the
column name.
Fixes: #7736
Change-Id: I635003478dc27193995f7d7a6448f9333a498706
large patch to get ORM / typing efforts started.
this is to support adding new test cases to mypy,
support dropping sqlalchemy2-stubs entirely from the
test suite, validate major ORM typing reorganization
to eliminate the need for the mypy plugin.
* New declarative approach which uses annotation
introspection, fixes: #7535
* Mapped[] is now at the base of all ORM constructs
that find themselves in classes, to support direct
typing without plugins
* Mypy plugin updated for new typing structures
* Mypy test suite broken out into "plugin" tests vs.
"plain" tests, and enhanced to better support test
structures where we assert that various objects are
introspected by the type checker as we expect.
as we go forward with typing, we will
add new use cases to "plain" where we can assert that
types are introspected as we expect.
* For typing support, users will be much more exposed to the
class names of things. Add these all to "sqlalchemy" import
space.
* Column(ForeignKey()) no longer needs to be `@declared_attr`
if the FK refers to a remote table
* composite() attributes mapped to a dataclass no longer
need to implement a `__composite_values__()` method
* with_variant() accepts multiple dialect names
Change-Id: I22797c0be73a8fbbd2d6f5e0c0b7258b17fe145d
Fixes: #7535Fixes: #7551
References: #6810
The SQLite dialect now defaults to :class:`_pool.QueuePool` when a file
based database is used. This is set along with setting the
``check_same_thread`` parameter to ``False``. It has been observed that the
previous approach of defaulting to :class:`_pool.NullPool`, which does not
hold onto database connections after they are released, did in fact have a
measurable negative performance impact. As always, the pool class is always
customizable via the :paramref:`_sa.create_engine.poolclass` parameter.
Fixes: #7490
Change-Id: I5f6c259def0ef43d401c6163dc99f651e519148d
Fixed issue in Oracle dialect where using a column name that requires
quoting when written as a bound parameter, such as ``"_id"``, would not
correctly track a Python generated default value due to the bound-parameter
rewriting missing this value, causing an Oracle error to be raised.
Fixes: #7676
Change-Id: I5a54426d24f2f9b336e3597d5595fb3e031aad97
Python string values for which a SQL type is determined from the type of
the value, mainly when using :func:`_sql.literal`, will now apply the
:class:`_types.String` type, rather than the :class:`_types.Unicode`
datatype, for Python string values that test as "ascii only" using Python
``str.isascii()``. If the string is not ``isascii()``, the
:class:`_types.Unicode` datatype will be bound instead, which was used in
all string detection previously. This behavior **only applies to in-place
detection of datatypes when using ``literal()`` or other contexts that have
no existing datatype**, which is not usually the case under normal
:class:`_schema.Column` comparison operations, where the type of the
:class:`_schema.Column` being compared always takes precedence.
Use of the :class:`_types.Unicode` datatype can determine literal string
formatting on backends such as SQL Server, where a literal value (i.e.
using ``literal_binds``) will be rendered as ``N'<value>'`` instead of
``'value'``. For normal bound value handling, the :class:`_types.Unicode`
datatype also may have implications for passing values to the DBAPI, again
in the case of SQL Server, the pyodbc driver supports the use of
:ref:`setinputsizes mode <mssql_pyodbc_setinputsizes>` which will handle
:class:`_types.String` versus :class:`_types.Unicode` differently.
Fixes: #7551
Change-Id: I4f8de63e36532ae8ce4c630ee59211349ce95361
Fixed regression caused by 🎫`7518` where changing the syntax "SHOW
VARIABLES" to "SELECT @@" broke compatibility with MySQL versions older
than 5.6, including early 5.0 releases. While these are very old MySQL
versions, a change in compatibility was not planned, so version-specific
logic has been restored to fall back to "SHOW VARIABLES" for MySQL server
versions < 5.6.
includes unrelated orm/test_expire ordering issue , only showing
up on 1.4 / py2.7 but seems to be passing by luck otherwise
Fixes: #7518
Change-Id: Ia554080af742f2c3437f88cf3f7a4827b5e55da8
Added compiler support for the PostgreSQL ``NOT VALID`` phrase when rendering
DDL for the :class:`.CheckConstraint`, :class:`.ForeignKeyConstraint`
and :class:`.ForeignKey` schema constructs. Pull request courtesy
Gilbert Gilb's.
Fixes: #7600Closes: #7601
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7601
Pull-request-sha: 78eecd55fd
Change-Id: I84bfe84596856eeea2bcca45c04ad23d980a75ec
Improvements to the test suite's integration with pytest such that the
"warnings" plugin, if manually enabled, will not interfere with the test
suite, such that third parties can enable the warnings plugin or make use
of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass.
Additionally, modernized the detection of the "pytest-xdist" plugin so that
plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
without breaking the test suite if xdist were still installed. Warning
filters that promote deprecation warnings to errors are now localized to
SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for
general Python deprecation warnings, so that non-SQLAlchemy deprecation
warnings emitted from pytest plugins should also not impact the test suite.
Fixes: #7599
Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
Fixed regression where the change in 🎫`7148` to repair ENUM handling
in PostgreSQL broke the use case of an empty ARRAY of ENUM, preventing rows
that contained an empty array from being handled correctly when fetching
results.
Fixes: #7590
Change-Id: I43a35ef25281a6e0a26b698efebef6ba12a63e8c
Added string rendering to the :class:`.postgresql.UUID` datatype, so that
stringifying a statement with "literal_binds" that uses this type will
render an appropriate string value for the PostgreSQL backend. Pull request
courtesy José Duarte.
Fixes: #7561Closes: #7563
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7563
Pull-request-sha: cf6fe73265
Change-Id: I4b162bdcdce2293a90683e36da54e4a891a3c684
mike bayer