Scaled back a fix made for 🎫`6581` where "executemany values" mode
for psycopg2 were disabled for all "ON CONFLICT" styles of INSERT, to
not apply to the "ON CONFLICT DO NOTHING" clause, which does not include
any parameters and is safe for "executemany values" mode. "ON CONFLICT
DO UPDATE" is still blocked from "executemany values" as there may
be additional parameters in the DO UPDATE clause that cannot be batched
(which is the original issue fixed by 🎫`6581`).
Fixes: #7880
Change-Id: Id3e23a0c6699333409a50148fa8923cb8e564bdc
both black and click were released in the past
few hours, and black 21.5b1 seems to suddenly
be failing on a missing symbol from click. just
update to the latest
Change-Id: Idf76732479a264f7f2245699a6bdaff018e3a123
Fixed regression caused by 🎫`7760` where the new capabilities of
:class:`.TextualSelect` were not fully implemented within the compiler
properly, leading to issues with composed INSERT constructs such as "INSERT
FROM SELECT" and "INSERT...ON CONFLICT" when combined with CTE and textual
statements.
Fixes: #7798
Change-Id: Ia2ce92507e574dd36fd26dd38ec9dd2713584467
All modules in sqlalchemy.engine are strictly
typed with the exception of cursor, default, and
reflection. cursor and default pass with non-strict
typing, reflection is waiting on the multi-reflection
refactor.
Behavioral changes:
* create_connect_args() methods return a tuple of list,
dict, rather than a list of list, dict
* removed allow_chars parameter from
pyodbc connector ._get_server_version_info()
method
* the parameter list passed to do_executemany is now
a list in all cases. previously, this was being run
through dialect.execute_sequence_format, which
defaults to tuple and was only intended for individual
tuple params.
* broke up dialect.dbapi into dialect.import_dbapi
class method and dialect.dbapi module object. added
a deprecation path for legacy dialects. it's not
really feasible to type a single attr as a classmethod
vs. module type. The "type_compiler" attribute also
has this problem with greater ability to work around,
left that one for now.
* lots of constants changing to be Enum, so that we can
type them. for fixed tuple-position constants in
cursor.py / compiler.py (which are used to avoid the
speed overhead of namedtuple), using Literal[value]
which seems to work well
* some tightening up in Row regarding __getitem__, which
we can do since we are on full 2.0 style result use
* altered the set_connection_execution_options and
set_engine_execution_options event flows so that the
dictionary of options may be mutated within the event
hook, where it will then take effect as the actual
options used. Previously, changing the dict would
be silently ignored which seems counter-intuitive
and not very useful.
* A lot of DefaultDialect/DefaultExecutionContext
methods and attributes, including underscored ones, move
to interfaces. This is not fully ideal as it means
the Dialect/ExecutionContext interfaces aren't publicly
subclassable directly, but their current purpose
is more of documentation for dialect authors who should
(and certainly are) still be subclassing the DefaultXYZ
versions in all cases
Overall, Result was the most extremely difficult class
hierarchy to type here as this hierarchy passes through
largely amorphous "row" datatypes throughout, which
can in fact by all kinds of different things, like
raw DBAPI rows, or Row objects, or "scalar"/Any, but
at the same time these types have meaning so I tried still
maintaining some level of semantic markings for these,
it highlights how complex Result is now, as it's trying
to be extremely efficient and inlined while also being
very open-ended and extensible.
Change-Id: I98b75c0c09eab5355fc7a33ba41dd9874274f12a
Added :class:`.Double`, :class:`.DOUBLE`, :class:`.DOUBLE_PRECISION`
datatypes to the base ``sqlalchemy.`` module namespace, for explicit use of
double/double precision as well as generic "double" datatypes. Use
:class:`.Double` for generic support that will resolve to DOUBLE/DOUBLE
PRECISION/FLOAT as needed for different backends.
Implemented DDL and reflection support for ``FLOAT`` datatypes which
include an explicit "binary_precision" value. Using the Oracle-specific
:class:`_oracle.FLOAT` datatype, the new parameter
:paramref:`_oracle.FLOAT.binary_precision` may be specified which will
render Oracle's precision for floating point types directly. This value is
interpreted during reflection. Upon reflecting back a ``FLOAT`` datatype,
the datatype returned is one of :class:`_types.DOUBLE_PRECISION` for a
``FLOAT`` for a precision of 126 (this is also Oracle's default precision
for ``FLOAT``), :class:`_types.REAL` for a precision of 63, and
:class:`_oracle.FLOAT` for a custom precision, as per Oracle documentation.
As part of this change, the generic :paramref:`_sqltypes.Float.precision`
value is explicitly rejected when generating DDL for Oracle, as this
precision cannot be accurately converted to "binary precision"; instead, an
error message encourages the use of
:meth:`_sqltypes.TypeEngine.with_variant` so that Oracle's specific form of
precision may be chosen exactly. This is a backwards-incompatible change in
behavior, as the previous "precision" value was silently ignored for
Oracle.
Fixes: #5465Closes: #7674
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7674
Pull-request-sha: 5c68419e5a
Change-Id: I831f4af3ee3b23fde02e8f6393c83e23dd7cd34d
large patch to get ORM / typing efforts started.
this is to support adding new test cases to mypy,
support dropping sqlalchemy2-stubs entirely from the
test suite, validate major ORM typing reorganization
to eliminate the need for the mypy plugin.
* New declarative approach which uses annotation
introspection, fixes: #7535
* Mapped[] is now at the base of all ORM constructs
that find themselves in classes, to support direct
typing without plugins
* Mypy plugin updated for new typing structures
* Mypy test suite broken out into "plugin" tests vs.
"plain" tests, and enhanced to better support test
structures where we assert that various objects are
introspected by the type checker as we expect.
as we go forward with typing, we will
add new use cases to "plain" where we can assert that
types are introspected as we expect.
* For typing support, users will be much more exposed to the
class names of things. Add these all to "sqlalchemy" import
space.
* Column(ForeignKey()) no longer needs to be `@declared_attr`
if the FK refers to a remote table
* composite() attributes mapped to a dataclass no longer
need to implement a `__composite_values__()` method
* with_variant() accepts multiple dialect names
Change-Id: I22797c0be73a8fbbd2d6f5e0c0b7258b17fe145d
Fixes: #7535Fixes: #7551
References: #6810
Added compiler support for the PostgreSQL ``NOT VALID`` phrase when rendering
DDL for the :class:`.CheckConstraint`, :class:`.ForeignKeyConstraint`
and :class:`.ForeignKey` schema constructs. Pull request courtesy
Gilbert Gilb's.
Fixes: #7600Closes: #7601
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7601
Pull-request-sha: 78eecd55fd
Change-Id: I84bfe84596856eeea2bcca45c04ad23d980a75ec
Improvements to the test suite's integration with pytest such that the
"warnings" plugin, if manually enabled, will not interfere with the test
suite, such that third parties can enable the warnings plugin or make use
of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass.
Additionally, modernized the detection of the "pytest-xdist" plugin so that
plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
without breaking the test suite if xdist were still installed. Warning
filters that promote deprecation warnings to errors are now localized to
SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for
general Python deprecation warnings, so that non-SQLAlchemy deprecation
warnings emitted from pytest plugins should also not impact the test suite.
Fixes: #7599
Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
Fixed regression where the change in 🎫`7148` to repair ENUM handling
in PostgreSQL broke the use case of an empty ARRAY of ENUM, preventing rows
that contained an empty array from being handled correctly when fetching
results.
Fixes: #7590
Change-Id: I43a35ef25281a6e0a26b698efebef6ba12a63e8c
Added string rendering to the :class:`.postgresql.UUID` datatype, so that
stringifying a statement with "literal_binds" that uses this type will
render an appropriate string value for the PostgreSQL backend. Pull request
courtesy José Duarte.
Fixes: #7561Closes: #7563
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7563
Pull-request-sha: cf6fe73265
Change-Id: I4b162bdcdce2293a90683e36da54e4a891a3c684
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.
Also update Black and other pre-commit hooks and re-format with Black.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570
Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
### Description
Found via `codespell -q 3 -L ba,crate,datas,froms,gord,hist,inh,nd,selectin,strat,ue`
Also added codespell to the pep8 tox env
### Checklist
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
Closes: #7338
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7338
Pull-request-sha: 0deac22193
Change-Id: Icd61db31c8dc655d4a39d8a304194804d08555fe
The :meth:`_sqltypes.TypeEngine.with_variant` method now returns a copy of
the original :class:`_sqltypes.TypeEngine` object, rather than wrapping it
inside the ``Variant`` class, which is effectively removed (the import
symbol remains for backwards compatibility with code that may be testing
for this symbol). While the previous approach maintained in-Python
behaviors, maintaining the original type allows for clearer type checking
and debugging.
Fixes: #6980
Change-Id: I158c7e56306b886b5b82b040205c428a5c4a242c
Fixed reflection of covering indexes to report ``include_columns`` as part
of the ``dialect_options`` entry in the reflected index dictionary, thereby
enabling round trips from reflection->create to be complete. Included
columns continue to also be present under the ``include_columns`` key for
backwards compatibility.
Fixes: #7382
Change-Id: I4f16b65caed3a36d405481690a3a92432b5efd62
Implemented full support for "truediv" and "floordiv" using the
"/" and "//" operators. A "truediv" operation between two expressions
using :class:`_types.Integer` now considers the result to be
:class:`_types.Numeric`, and the dialect-level compilation will cast
the right operand to a numeric type on a dialect-specific basis to ensure
truediv is achieved. For floordiv, conversion is also added for those
databases that don't already do floordiv by default (MySQL, Oracle) and
the ``FLOOR()`` function is rendered in this case, as well as for
cases where the right operand is not an integer (needed for PostgreSQL,
others).
The change resolves issues both with inconsistent behavior of the
division operator on different backends and also fixes an issue where
integer division on Oracle would fail to be able to fetch a result due
to inappropriate outputtypehandlers.
Fixes: #4926
Change-Id: Id54cc018c1fb7a49dd3ce1216d68d40f43fe2659
Add a new system so that PostgreSQL and other dialects have a
reliable way to add casts to bound parameters in SQL statements,
replacing previous use of setinputsizes() for PG dialects.
rationale:
1. psycopg3 will be using the same SQLAlchemy-side "setinputsizes"
as asyncpg, so we will be seeing a lot more of this
2. the full rendering that SQLAlchemy's compilation is performing
is in the engine log as well as error messages. Without this,
we introduce three levels of SQL rendering, the compiler, the
hidden "setinputsizes" in SQLAlchemy, and then whatever the DBAPI
driver does. With this new approach, users reporting bugs etc.
will be less confused that there are as many as two separate
layers of "hidden rendering"; SQLAlchemy's rendering is again
fully transparent
3. calling upon a setinputsizes() method for every statement execution
is expensive. this way, the work is done behind the caching layer
4. for "fast insertmany()", I also want there to be a fast approach
towards setinputsizes. As it was, we were going to be taking
a SQL INSERT with thousands of bound parameter placeholders and
running a whole second pass on it to apply typecasts. this way,
we will at least be able to build the SQL string once without a huge
second pass over the whole string
5. psycopg2 can use this same system for its ARRAY casts
6. the general need for PostgreSQL to have lots of type casts
is now mostly in the base PostgreSQL dialect and works independently
of a DBAPI being present. dependence on DBAPI symbols that aren't
complete / consistent / hashable is removed
I was originally going to try to build this into bind_expression(),
but it was revealed this worked poorly with custom bind_expression()
as well as empty sets. the current impl also doesn't need to
run a second expression pass over the POSTCOMPILE sections, which
came out better than I originally thought it would.
Change-Id: I363e6d593d059add7bcc6d1f6c3f91dd2e683c0c
the _CompileLabel class included ``__slots__`` but these
weren't used as the superclasses included slots.
Create a ``__slots__`` superclass for ``ClauseElement``,
creating a new class of compilable SQL elements that don't
include heavier features like caching, annotations and
cloning, which are meant to be used only in an ad-hoc
compiler fashion. Create new ``CompilerColumnElement``
from that which serves in column-oriented contexts, but
similarly does not include any expression operator support
as it is intended to be used only to generate a string.
Apply this to both
``_CompileLabel`` as well as PostgreSQL ``_ColonCast``,
which does not actually subclass ``ColumnElement`` as this
class has memoized attributes that aren't worth changing,
and does not include SQL operator capabilities as these
are not needed for these compiler-only objects.
this allows us to more inexpensively add new ad-hoc
labels / casts etc. at compile time, as we will be seeking
to expand out the typecasts that are needed for PostgreSQL
dialects in a subsequent patch.
Change-Id: I52973ae3295cb6e2eb0d7adc816c678a626643ed
Removed here includes:
* convert_unicode parameters
* encoding create_engine() parameter
* description encoding support
* "non-unicode fallback" modes under Python 2
* String symbols regarding Python 2 non-unicode fallbacks
* any concept of DBAPIs that don't accept unicode
statements, unicode bound parameters, or that return bytes
for strings anywhere except an explicit Binary / BLOB
type
* unicode processors in Python / C
Risk factors:
* Whether all DBAPIs do in fact return Unicode objects for
all entries in cursor.description now
* There was logic for mysql-connector trying to determine
description encoding. A quick test shows Unicode coming
back but it's not clear if there are still edge cases where
they return bytes. if so, these are bugs in that driver,
and at most we would only work around it in the mysql-connector
DBAPI itself (but we won't do that either).
* It seems like Oracle 8 was not expecting unicode bound parameters.
I'm assuming this was all Python 2 stuff and does not apply
for modern cx_Oracle under Python 3.
* third party dialects relying upon built in unicode encoding/decoding
but it's hard to imagine any non-SQLAlchemy database driver not
dealing exclusively in Python unicode strings in Python 3
Change-Id: I97d762ef6d4dd836487b714d57d8136d0310f28a
References: #7257
Getting
TypeError: object MagicMock can't be used in 'await' expression
for Python 3.7 and earlier. this test is not needed
on all platforms it's confirming that two methods
are present.
Change-Id: If918add023c98c062ea0c1cd132a999647a2d35f
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
Fixes: #7300
Change-Id: I6255c850b140522a4aba95085216d0bca18ce230
The :paramref:`_sa.create_engine.implicit_returning` parameter is
deprecated on the :func:`_sa.create_engine` function only; the parameter
remains available on the :class:`_schema.Table` object. This parameter was
originally intended to enable the "implicit returning" feature of
SQLAlchemy when it was first developed and was not enabled by default.
Under modern use, there's no reason this parameter should be disabled, and
it has been observed to cause confusion as it degrades performance and
makes it more difficult for the ORM to retrieve recently inserted server
defaults. The parameter remains available on :class:`_schema.Table` to
specifically suit database-level edge cases which make RETURNING
infeasible, the sole example currently being SQL Server's limitation that
INSERT RETURNING may not be used on a table that has INSERT triggers on it.
Also removed from the Oracle dialect some logic that would upgrade
an Oracle 8/8i server version to use implicit returning if the
parameter were explictly passed; these versions of Oracle
still support RETURNING so the feature is now enabled for all
Oracle versions.
Fixes: #6962
Change-Id: Ib338e300cd7c8026c3083043f645084a8211aed8
Fixes: #6960
Even though a default driver still exists for
each dialect, remove most usages of `dialect://`
to encourage users to explicitly specify
`dialect+driver://`
Change-Id: I0ad42167582df509138fca64996bbb53e379b1af
The major action here is to lift and move future.Connection
and future.Engine fully into sqlalchemy.engine.base. This
removes lots of engine concepts, including:
* autocommit
* Connection running without a transaction, autobegin
is now present in all cases
* most "autorollback" is obsolete
* Core-level subtransactions (i.e. MarkerTransaction)
* "branched" connections, copies of connections
* execution_options() returns self, not a new connection
* old argument formats, distill_params(), simplifies calling
scheme between engine methods
* before/after_execute() events (oriented towards compiled constructs)
don't emit for exec_driver_sql(). before/after_cursor_execute()
is still included for this
* old helper methods superseded by context managers, connection.transaction(),
engine.transaction() engine.run_callable()
* ancient engine-level reflection methods has_table(), table_names()
* sqlalchemy.testing.engines.proxying_engine
References: #7257
Change-Id: Ib20ed816642d873b84221378a9ec34480e01e82c
Added overridable methods ``PGDialect_asyncpg.setup_asyncpg_json_codec``
and ``PGDialect_asyncpg.setup_asyncpg_jsonb_codec`` codec, which handle the
required task of registering JSON/JSONB codecs for these datatypes when
using asyncpg. The change is that methods are broken out as individual,
overridable methods to support third party dialects that need to alter or
disable how these particular codecs are set up.
Fixes: #7284
Change-Id: I3eac258fea61f3975bd03c428747f788813ce45e
Fixed issue where "expanding IN" would fail to function correctly with
datatypes that use the :meth:`_types.TypeEngine.bind_expression` method,
where the method would need to be applied to each element of the
IN expression rather than the overall IN expression itself.
Fixed issue where IN expressions against a series of array elements, as can
be done with PostgreSQL, would fail to function correctly due to multiple
issues within the "expanding IN" feature of SQLAlchemy Core that was
standardized in version 1.4. The psycopg2 dialect now makes use of the
:meth:`_types.TypeEngine.bind_expression` method with :class:`_types.ARRAY`
to portably apply the correct casts to elements. The asyncpg dialect was
not affected by this issue as it applies bind-level casts at the driver
level rather than at the compiler level.
as part of this commit the "bind translate" feature has been
simplified and also applies to the names in the POSTCOMPILE tag to
accommodate for brackets.
Fixes: #7177
Change-Id: I08c703adb0a9bd6f5aeee5de3ff6f03cccdccdc5
a few changes for py2k:
* map_imperatively() includes the check that a class
is being sent, this was only working for mapper() before
* the test suite didn't place the py2k "autouse" workaround
in the correct order, seemingly, tried to adjust the
per-test ordering setup in pytestplugin.py
Change-Id: I4cc39630724e810953cfda7b2afdadc8b948e3c2
Improve the interface used by adapted drivers, like the asyncio ones,
to access the actual connection object returned by the driver.
The :class:`_engine._ConnectionRecord` and
:class:`_engine._ConnectionFairy` now have two new attributes:
* ``dbapi_connection`` always represents a DBAPI compatible
object. For pep-249 drivers, this is the DBAPI connection as it always
has been, previously accessed under the ``.connection`` attribute.
For asyncio drivers that SQLAlchemy adapts into a pep-249 interface,
the returned object will normally be a SQLAlchemy adaption object
called :class:`_engine.AdaptedConnection`.
* ``driver_connection`` always represents the actual connection object
maintained by the third party pep-249 DBAPI or async driver in use.
For standard pep-249 DBAPIs, this will always be the same object
as that of the ``dbapi_connection``. For an asyncio driver, it will be
the underlying asyncio-only connection object.
The ``.connection`` attribute remains available and is now a legacy alias
of ``.dbapi_connection``.
Fixes: #6832
Change-Id: Ib72f97deefca96dce4e61e7c38ba430068d6a82e
any UPPERCASE datatype refers to that exact type name rendered
on the database. So PG's ENUM must render "ENUM" and is
"native" by definition. warn if this flag is passed.
The :class:`_postgresql.ENUM` datatype is PostgreSQL-native and therefore
should not be used with the ``native_enum=False`` flag. This flag is now
ignored if passed to the :class:`_postgresql.ENUM` datatype and a warning
is emitted; previously the flag would cause the type object to fail to
function correctly.
Fixes: #6106
Change-Id: I08e0ec6fcfafd068e1eaf6aec13c8010f09ce94a
Adjusted the "from linter" warning feature to accommodate for a chain of
joins more than one level deep where the ON clauses don't explicitly match
up the targets, such as an expression such as "ON TRUE". This mode of use
is intended to cancel the cartesian product warning simply by the fact that
there's a JOIN from "a to b", which was not working for the case where the
chain of joins had more than one element.
this incurs a bit more compiler overhead that comes out in profiling
but is not extensive.
Added the "is_comparison" flag to the PostgreSQL "overlaps",
"contained_by", "contains" operators, so that they work in relevant ORM
contexts as well as in conjunction with the "from linter" feature.
Fixes: #6886
Change-Id: I078dc3fe6d4f7871ffe4ebac3e71e62f3f213d12
Fixed issue where the unit of work would internally use a 2.0-deprecated
SQL expression form, emitting a deprecation warning when SQLALCHEMY_WARN_20
were enabled.
Fixes: #6812
Change-Id: I0a031e728527a1c3382848b6ddc793939362b128
Mike Bayer