Commit Graph

117 Commits

Author SHA1 Message Date
Mike Bayer 719197dd93 use plainto_tsquery for PG match
The :meth:`.Operators.match` operator now uses ``plainto_tsquery()`` for
PostgreSQL full text search, rather than ``to_tsquery()``. The rationale
for this change is to provide better cross-compatibility with match on
other database backends.    Full support for all PostgreSQL full text
functions remains available through the use of :data:`.func` in
conjunction with :meth:`.Operators.bool_op` (an improved version of
:meth:`.Operators.op` for boolean operators).

Additional doc updates here apply to 1.4 so will backport these
out to a separate commit.

Fixes: #7086
Change-Id: I1946075daf5d9c558e85f73f1bf852604b3b1b8c
2022-05-22 15:25:58 -04:00
Mike Bayer 927abc3b33 render col name in on conflict set clause, not given key
Fixed bug where the PostgreSQL :meth:`_postgresql.Insert.on_conflict`
method and the SQLite :meth:`_sqlite.Insert.on_conflict` method would both
fail to correctly accommodate a column with a separate ".key" when
specifying the column using its key name in the dictionary passed to
``set_``, as well as if the :attr:`_sqlite.Insert.excluded` or
:attr:`_postgresql.Insert.excluded` collection were used as the dictionary
directly.

Fixes: #8014
Change-Id: I67226aeedcb2c683e22405af64720cc1f990f274
2022-05-15 17:27:27 -04:00
Mike Bayer 63191fbef6 properly type array element in any() / all()
Fixed bug in :class:`.ARRAY` datatype in combination with :class:`.Enum` on
PostgreSQL where using the ``.any()`` method to render SQL ANY(), given
members of the Python enumeration as arguments, would produce a type
adaptation failure on all drivers.

Fixes: #6515
Change-Id: Ia1e3b4e10aaf264ed436ce6030d105fc60023433
2022-04-22 23:29:11 -04:00
Mike Bayer c36965ab21 pop the stack that we pushed
Fixed regression caused by 🎫`7760` where the new capabilities of
:class:`.TextualSelect` were not fully implemented within the compiler
properly, leading to issues with composed INSERT constructs such as "INSERT
FROM SELECT" and "INSERT...ON CONFLICT" when combined with CTE and textual
statements.

Fixes: #7798
Change-Id: Ia2ce92507e574dd36fd26dd38ec9dd2713584467
2022-03-08 13:40:12 -05:00
zeeeeeb b9d231869d Implement generic Double and related fixed types
Added :class:`.Double`, :class:`.DOUBLE`, :class:`.DOUBLE_PRECISION`
datatypes to the base ``sqlalchemy.`` module namespace, for explicit use of
double/double precision as well as generic "double" datatypes. Use
:class:`.Double` for generic support that will resolve to DOUBLE/DOUBLE
PRECISION/FLOAT as needed for different backends.

Implemented DDL and reflection support for ``FLOAT`` datatypes which
include an explicit "binary_precision" value. Using the Oracle-specific
:class:`_oracle.FLOAT` datatype, the new parameter
:paramref:`_oracle.FLOAT.binary_precision` may be specified which will
render Oracle's precision for floating point types directly. This value is
interpreted during reflection. Upon reflecting back a ``FLOAT`` datatype,
the datatype returned is one of :class:`_types.DOUBLE_PRECISION` for a
``FLOAT`` for a precision of 126 (this is also Oracle's default precision
for ``FLOAT``), :class:`_types.REAL` for a precision of 63, and
:class:`_oracle.FLOAT` for a custom precision, as per Oracle documentation.

As part of this change, the generic :paramref:`_sqltypes.Float.precision`
value is explicitly rejected when generating DDL for Oracle, as this
precision cannot be accurately converted to "binary precision"; instead, an
error message encourages the use of
:meth:`_sqltypes.TypeEngine.with_variant` so that Oracle's specific form of
precision may be chosen exactly. This is a backwards-incompatible change in
behavior, as the previous "precision" value was silently ignored for
Oracle.

Fixes: #5465
Closes: #7674
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7674
Pull-request-sha: 5c68419e5a

Change-Id: I831f4af3ee3b23fde02e8f6393c83e23dd7cd34d
2022-02-25 00:51:32 +00:00
Gilbert Gilb's 77dd6808f2 Add compiler support for PostgreSQL "NOT VALID" constraints.
Added compiler support for the PostgreSQL ``NOT VALID`` phrase when rendering
DDL for the :class:`.CheckConstraint`, :class:`.ForeignKeyConstraint`
and :class:`.ForeignKey` schema constructs.  Pull request courtesy
Gilbert Gilb's.

Fixes: #7600
Closes: #7601
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7601
Pull-request-sha: 78eecd55fd

Change-Id: I84bfe84596856eeea2bcca45c04ad23d980a75ec
2022-01-26 22:22:14 -05:00
Mike Bayer 6d589ffbb5 consider truediv as truediv; support floordiv operator
Implemented full support for "truediv" and "floordiv" using the
"/" and "//" operators.  A "truediv" operation between two expressions
using :class:`_types.Integer` now considers the result to be
:class:`_types.Numeric`, and the dialect-level compilation will cast
the right operand to a numeric type on a dialect-specific basis to ensure
truediv is achieved.  For floordiv, conversion is also added for those
databases that don't already do floordiv by default (MySQL, Oracle) and
the ``FLOOR()`` function is rendered in this case, as well as for
cases where the right operand is not an integer (needed for PostgreSQL,
others).

The change resolves issues both with inconsistent behavior of the
division operator on different backends and also fixes an issue where
integer division on Oracle would fail to be able to fetch a result due
to inappropriate outputtypehandlers.

Fixes: #4926
Change-Id: Id54cc018c1fb7a49dd3ce1216d68d40f43fe2659
2021-12-26 19:32:53 -05:00
mike bayer 8ddb3ef165 Merge "propose emulated setinputsizes embedded in the compiler" into main 2021-11-25 18:22:59 +00:00
Federico Caselli 31acba8ff7 Clean up most py3k compat
Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
2021-11-24 22:51:27 -05:00
Mike Bayer 939de240d3 propose emulated setinputsizes embedded in the compiler
Add a new system so that PostgreSQL and other dialects have a
reliable way to add casts to bound parameters in SQL statements,
replacing previous use of setinputsizes() for PG dialects.

rationale:

1. psycopg3 will be using the same SQLAlchemy-side "setinputsizes"
   as asyncpg, so we will be seeing a lot more of this

2. the full rendering that SQLAlchemy's compilation is performing
   is in the engine log as well as error messages.   Without this,
   we introduce three levels of SQL rendering, the compiler, the
   hidden "setinputsizes" in SQLAlchemy, and then whatever the DBAPI
   driver does.  With this new approach, users reporting bugs etc.
   will be less confused that there are as many as two separate
   layers of "hidden rendering"; SQLAlchemy's rendering is again
   fully transparent

3. calling upon a setinputsizes() method for every statement execution
   is expensive.  this way, the work is done behind the caching layer

4. for "fast insertmany()", I also want there to be a fast approach
   towards setinputsizes.  As it was, we were going to be taking
   a SQL INSERT with thousands of bound parameter placeholders and
   running a whole second pass on it to apply typecasts.    this way,
   we will at least be able to build the SQL string once without a huge
   second pass over the whole string

5. psycopg2 can use this same system for its ARRAY casts

6. the general need for PostgreSQL to have lots of type casts
   is now mostly in the base PostgreSQL dialect and works independently
   of a DBAPI being present.   dependence on DBAPI symbols that aren't
   complete / consistent / hashable is removed

I was originally going to try to build this into bind_expression(),
but it was revealed this worked poorly with custom bind_expression()
as well as empty sets.   the current impl also doesn't need to
run a second expression pass over the POSTCOMPILE sections, which
came out better than I originally thought it would.

Change-Id: I363e6d593d059add7bcc6d1f6c3f91dd2e683c0c
2021-11-23 16:52:55 -05:00
Mike Bayer d3a4e96196 Support lightweight compiler column elements w/ slots
the _CompileLabel class included ``__slots__`` but these
weren't used as the superclasses included slots.

Create a ``__slots__`` superclass for ``ClauseElement``,
creating a new class of compilable SQL elements that don't
include heavier features like caching, annotations and
cloning, which are meant to be used only in an ad-hoc
compiler fashion.   Create new ``CompilerColumnElement``
from that which serves in column-oriented contexts, but
similarly does not include any expression operator support
as it is intended to be used only to generate a string.

Apply this to both
``_CompileLabel`` as well as PostgreSQL ``_ColonCast``,
which does not actually subclass ``ColumnElement`` as this
class has memoized attributes that aren't worth changing,
and does not include SQL operator capabilities as these
are not needed for these compiler-only objects.

this allows us to more inexpensively add new ad-hoc
labels / casts etc. at compile time, as we will be seeking
to expand out the typecasts that are needed for PostgreSQL
dialects in a subsequent patch.

Change-Id: I52973ae3295cb6e2eb0d7adc816c678a626643ed
2021-11-22 11:26:33 -05:00
Federico Caselli 0b95f0055b Remove object in class definition
References: #4600
Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
2021-11-22 15:03:17 +00:00
jonathan vanasco b2df5be7ee Deprecate create_engine.implicit_returning
The :paramref:`_sa.create_engine.implicit_returning` parameter is
deprecated on the :func:`_sa.create_engine` function only; the parameter
remains available on the :class:`_schema.Table` object. This parameter was
originally intended to enable the "implicit returning" feature of
SQLAlchemy when it was first developed and was not enabled by default.
Under modern use, there's no reason this parameter should be disabled, and
it has been observed to cause confusion as it degrades performance and
makes it more difficult for the ORM to retrieve recently inserted server
defaults. The parameter remains available on :class:`_schema.Table` to
specifically suit database-level edge cases which make RETURNING
infeasible, the sole example currently being SQL Server's limitation that
INSERT RETURNING may not be used on a table that has INSERT triggers on it.

Also removed from the Oracle dialect some logic that would upgrade
an Oracle 8/8i server version to use implicit returning if the
parameter were explictly passed; these versions of Oracle
still support RETURNING so the feature is now enabled for all
Oracle versions.

Fixes: #6962
Change-Id: Ib338e300cd7c8026c3083043f645084a8211aed8
2021-11-09 11:34:48 -05:00
Gord Thompson 257f9130c3 Modernize tests - calling_mapper_directly
a few changes for py2k:

* map_imperatively() includes the check that a class
  is being sent, this was only working for mapper() before

* the test suite didn't place the py2k "autouse" workaround
  in the correct order, seemingly, tried to adjust the
  per-test ordering setup in pytestplugin.py

Change-Id: I4cc39630724e810953cfda7b2afdadc8b948e3c2
2021-09-30 10:10:16 -04:00
Mike Bayer b987465332 fix linter JOIN logic; fix PostgreSQL ARRAY op comparison
Adjusted the "from linter" warning feature to accommodate for a chain of
joins more than one level deep where the ON clauses don't explicitly match
up the targets, such as an expression such as "ON TRUE". This mode of use
is intended to cancel the cartesian product warning simply by the fact that
there's a JOIN from "a to b", which was not working for the case where the
chain of joins had more than one element.

this incurs a bit more compiler overhead that comes out in profiling
but is not extensive.

Added the "is_comparison" flag to the PostgreSQL "overlaps",
"contained_by", "contains" operators, so that they work in relevant ORM
contexts as well as in conjunction with the "from linter" feature.

Fixes: #6886
Change-Id: I078dc3fe6d4f7871ffe4ebac3e71e62f3f213d12
2021-08-15 23:14:59 -04:00
Gord Thompson 2c4a497a0c Modernize tests - dml_values
Change-Id: Ida86ed40c43d91813151621b847376976773a5f9
2021-07-26 14:00:53 -06:00
mike bayer b64ecb03a5 Merge "Extract format_constraint truncation rules to ON CONFLICT" 2021-07-13 15:09:05 +00:00
Mike Bayer 0e46359cb0 Extract format_constraint truncation rules to ON CONFLICT
Fixed issue where a too-long constraint name rendered as part of the "ON
CONFLICT ON CONSTRAINT" element of the :class:`_postgresql.Insert`
construct due to naming convention generation would not correctly truncate
the name in the same way that it normally renders within a CREATE TABLE
statement, thus producing a non-matching and too-long constraint name.

Fixes: #6755
Change-Id: Ib27014a5ecbc9cd5861a396f8bb49fbc60bf49fe
2021-07-13 10:15:07 -04:00
mike bayer 18dba87757 Merge "implement independent CTEs" 2021-07-12 21:53:11 +00:00
mike bayer b7791290ed Merge "Modernize tests - select(whereclause)" 2021-07-12 21:52:09 +00:00
Mike Bayer 204ff1f60c implement independent CTEs
Added new method :meth:`_sql.HasCTE.add_cte` to each of the
:func:`_sql.select`, :func:`_sql.insert`, :func:`_sql.update` and
:func:`_sql.delete` constructs. This method will add the given
:class:`_sql.CTE` as an "independent" CTE of the statement, meaning it
renders in the WITH clause above the statement unconditionally even if it
is not otherwise referenced in the primary statement. This is a popular use
case on the PostgreSQL database where a CTE is used for a DML statement
that runs against database rows independently of the primary statement.

Fixes: #6752
Change-Id: Ibf635763e40269cbd10f4c17e208850d8e8d0188
2021-07-12 17:10:58 -04:00
Mike Bayer 520b808db2 repair schema_translate_map for schema type use cases
Fixed issue where the PostgreSQL ``ENUM`` datatype as embedded in the
``ARRAY`` datatype would fail to emit correctly in create/drop when the
``schema_translate_map`` feature were also in use. Additionally repairs a
related issue where the same ``schema_translate_map`` feature would not
work for the ``ENUM`` datatype in combination with a ``CAST``, that's also
intrinsic to how the ``ARRAY(ENUM)`` combination works on the PostgreSQL
dialect.

Fixes: #6739
Change-Id: I44b1ad4db4af3acbf639aa422c46c22dd3b0d3a6
2021-07-11 19:23:40 -04:00
Gord Thompson 4ff4760fad Modernize tests - select(whereclause)
Change-Id: I306cfbea9920b35100e3087dcc21d7ffa6c39c55
2021-07-04 15:56:40 -06:00
Mike Bayer dee5747788 apply quoting to "ON CONSTRAINT" symbol
Fixed issue in :meth:`_postgresql.Insert.on_conflict_do_nothing` and
:meth:`_postgresql.Insert.on_conflict_do_update` where the name of a unique
constraint passed as the ``constraint`` parameter would not be properly
quoted if it contained characters which required quoting.

Fixes: #6696
Change-Id: I4ffca9b8c72cef4ed39e2de96831ccc11a620422
2021-06-30 09:23:02 -04:00
Federico Caselli 85d66c5adf Test some complex update cases in the pg dialect
Change-Id: I2323e155e78aa8e1e00359b103974fb8d27d80eb
2021-05-27 20:49:44 +02:00
Federico Caselli d3c73ad801 Propertly ignore `Identity` in MySQL and MariaDb.
Ensure that the MySQL and MariaDB dialect ignore the
:class:`_sql.Identity` construct while rendering the
``AUTO_INCREMENT`` keyword in a create table.

The Oracle and PostgreSQL compiler was updated to not render
:class:`_sql.Identity` if the database version does not support it
(Oracle < 12 and PostgreSQL < 10). Previously it was rendered regardless
of the database version.

Fixes: #6338
Change-Id: I2ca0902fdd7b4be4fc1a563cf5585504cbea9360
2021-04-28 20:03:27 -04:00
Federico Caselli dfa1d3b28f CAST the elements in ARRAYs when using psycopg2
Adjusted the psycopg2 dialect to emit an explicit PostgreSQL-style cast for
bound parameters that contain ARRAY elements. This allows the full range of
datatypes to function correctly within arrays. The asyncpg dialect already
generated these internal casts in the final statement. This also includes
support for array slice updates as well as the PostgreSQL-specific
:meth:`_postgresql.ARRAY.contains` method.

Fixes: #6023
Change-Id: Ia7519ac4371a635f05ac69a3a4d0f4e6d2f04cad
2021-03-15 20:11:20 -04:00
Gord Thompson deaefbe748 Deprecate and rename schema .copy() methods
Fixes: #5953
Change-Id: I1e69a1628e408f06b43efbc0cc52fc0ad1e8cbc4
2021-02-18 15:25:08 -05:00
Gord Thompson 09e999808a Use schema._copy_expression() fully in column collection constraints
Fixed issue where using :meth:`_schema.Table.to_metadata` (called
:meth:`_schema.Table.tometadata` in 1.3) in conjunction with a PostgreSQL
:class:`_postgresql.ExcludeConstraint` that made use of ad-hoc column
expressions would fail to copy correctly.

Fixes: #5850
Change-Id: I062480afb23f6f60962b7b55bc93f5e4e6ff05e4
2021-01-29 16:48:00 -05:00
Gord Thompson 22f65156bb Replace with_labels() and apply_labels() in ORM/Core
Replace :meth:`_orm.Query.with_labels` and
:meth:`_sql.GenerativeSelect.apply_labels` with explicit getters and
setters ``get_label_style`` and ``set_label_style`` to accommodate the
three supported label styles: ``LABEL_STYLE_DISAMBIGUATE_ONLY`` (default),
``LABEL_STYLE_TABLENAME_PLUS_COL``, and ``LABEL_STYLE_NONE``.

In addition, for Core and "future style" ORM queries,
``LABEL_STYLE_DISAMBIGUATE_ONLY`` is now the default label style. This
style differs from the existing "no labels" style in that labeling is
applied in the case of column name conflicts; with ``LABEL_STYLE_NONE``, a
duplicate column name is not accessible via name in any case.

For legacy ORM queries using :class:`_query.Query`, the table-plus-column
names labeling style applied by ``LABEL_STYLE_TABLENAME_PLUS_COL``
continues to be used so that existing test suites and logging facilities
see no change in behavior by default, however this style of labeling is no
longer required for SQLAlchemy queries to function, as result sets are
commonly matched to columns using a positional approach since SQLAlchemy
1.0.

Within test suites, all use of apply_labels()  / use_labels
now uses the new methods.    New tests added to
test/sql/test_deprecations.py nad test/orm/test_deprecations.py
to cover just the old apply_labels() method call.  Tests
in ORM that made explicit use apply_labels()/ etc. where it isn't needed
for the ORM to work correctly use default label style now.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Fixes: #4757
Change-Id: I5fdcd2ed4ae8c7fe62f8be2b6d0e8f66409b6a54
2021-01-26 16:52:30 -05:00
mike bayer 6c97f0eceb Merge "`Identity implies nullable=False`." 2021-01-17 01:31:03 +00:00
Mike Bayer 8860117c96 introduce generalized decorator to prevent invalid method calls
This introduces the ``_exclusive_against()`` utility decorator
that can be used to prevent repeated invocations of methods that
typically should only be called once.

An informative error message is now raised for a selected set of DML
methods (currently all part of :class:`_dml.Insert` constructs) if they are
called a second time, which would implicitly cancel out the previous
setting.  The methods altered include:
:class:`_sqlite.Insert.on_conflict_do_update`,
:class:`_sqlite.Insert.on_conflict_do_nothing` (SQLite),
:class:`_postgresql.Insert.on_conflict_do_update`,
:class:`_postgresql.Insert.on_conflict_do_nothing` (PostgreSQL),
:class:`_mysql.Insert.on_duplicate_key_update` (MySQL)

Fixes: #5169
Change-Id: I9278fa87cd3470dcf296ff96bb0fb17a3236d49d
2021-01-16 18:44:21 -05:00
Federico Caselli 3ebc3710a7 `Identity implies nullable=False`.
Altered the behavior of the :class:`_schema.Identity` construct such that
when applied to a :class:`_schema.Column`, it will automatically imply that
the value of :paramref:`_sql.Column.nullable` should default to ``False``,
in a similar manner as when the :paramref:`_sql.Column.primary_key`
parameter is set to ``True``.   This matches the default behavior of all
supporting databases where ``IDENTITY`` implies ``NOT NULL``.  The
PostgreSQL backend is the only one that supports adding ``NULL`` to an
``IDENTITY`` column, which is here supported by passing a ``True`` value
for the :paramref:`_sql.Column.nullable` parameter at the same time.

Fixes: #5775
Change-Id: I0516d506ff327cff35cda605e8897a27440e0373
2021-01-16 18:39:11 -05:00
Mike Bayer f1e96cb087 reinvent xdist hooks in terms of pytest fixtures
To allow the "connection" pytest fixture and others work
correctly in conjunction with setup/teardown that expects
to be external to the transaction, remove and prevent any usage
of "xdist" style names that are hardcoded by pytest to run
inside of fixtures, even function level ones.   Instead use
pytest autouse fixtures to implement our own
r"setup|teardown_test(?:_class)?" methods so that we can ensure
function-scoped fixtures are run within them.   A new more
explicit flow is set up within plugin_base and pytestplugin
such that the order of setup/teardown steps, which there are now
many, is fully documented and controllable.   New granularity
has been added to the test teardown phase to distinguish
between "end of the test" when lock-holding structures on
connections should be released to allow for table drops,
vs. "end of the test plus its teardown steps" when we can
perform final cleanup on connections and run assertions
that everything is closed out.

From there we can remove most of the defensive "tear down everything"
logic inside of engines which for many years would frequently dispose
of pools over and over again, creating for a broken and expensive
connection flow.  A quick test shows that running test/sql/ against
a single Postgresql engine with the new approach uses 75% fewer new
connections, creating 42 new connections total, vs. 164 new
connections total with the previous system.

As part of this, the new fixtures metadata/connection/future_connection
have been integrated such that they can be combined together
effectively.  The fixture_session(), provide_metadata() fixtures
have been improved, including that fixture_session() now strongly
references sessions which are explicitly torn down before
table drops occur afer a test.

Major changes have been made to the
ConnectionKiller such that it now features different "scopes" for
testing engines and will limit its cleanup to those testing
engines corresponding to end of test, end of test class, or
end of test session.   The system by which it tracks DBAPI
connections has been reworked, is ultimately somewhat similar to
how it worked before but is organized more clearly along
with the proxy-tracking logic.  A "testing_engine" fixture
is also added that works as a pytest fixture rather than a
standalone function.  The connection cleanup logic should
now be very robust, as we now can use the same global
connection pools for the whole suite without ever disposing
them, while also running a query for PostgreSQL
locks remaining after every test and assert there are no open
transactions leaking between tests at all.  Additional steps
are added that also accommodate for asyncio connections not
explicitly closed, as is the case for legacy sync-style
tests as well as the async tests themselves.

As always, hundreds of tests are further refined to use the
new fixtures where problems with loose connections were identified,
largely as a result of the new PostgreSQL assertions,
many more tests have moved from legacy patterns into the newest.

An unfortunate discovery during the creation of this system is that
autouse fixtures (as well as if they are set up by
@pytest.mark.usefixtures) are not usable at our current scale with pytest
4.6.11 running under Python 2.  It's unclear if this is due
to the older version of pytest or how it implements itself for
Python 2, as well as if the issue is CPU slowness or just large
memory use, but collecting the full span of tests takes over
a minute for a single process when any autouse fixtures are in
place and on CI the jobs just time out after ten minutes.
So at the moment this patch also reinvents a small version of
"autouse" fixtures when py2k is running, which skips generating
the real fixture and instead uses two global pytest fixtures
(which don't seem to impact performance) to invoke the
"autouse" fixtures ourselves outside of pytest.
This will limit our ability to do more with fixtures
until we can remove py2k support.

py.test is still observed to be much slower in collection in the
4.6.11 version compared to modern 6.2 versions, so add support for new
TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that
will run the suite for fewer backends under Python 2.  For Python 3
pin pytest to modern 6.2 versions where performance for collection
has been improved greatly.

Includes the following improvements:

Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would
be raised rather than :class:`.exc.TimeoutError`.  Also repaired the
:paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using
the async engine, which previously would ignore the timeout and block
rather than timing out immediately as is the behavior with regular
:class:`.QueuePool`.

For asyncio the connection pool will now also not interact
at all with an asyncio connection whose ConnectionFairy is
being garbage collected; a warning that the connection was
not properly closed is emitted and the connection is discarded.
Within the test suite the ConnectionKiller is now maintaining
strong references to all DBAPI connections and ensuring they
are released when tests end, including those whose ConnectionFairy
proxies are GCed.

Identified cx_Oracle.stmtcachesize as a major factor in Oracle
test scalability issues, this can be reset on a per-test basis
rather than setting it to zero across the board.  the addition
of this flag has resolved the long-standing oracle "two task"
error problem.

For SQL Server, changed the temp table style used by the
"suite" tests to be the double-pound-sign, i.e. global,
variety, which is much easier to test generically.  There
are already reflection tests that are more finely tuned
to both styles of temp table within the mssql test
suite.  Additionally, added an extra step to the
"dropfirst" mechanism for SQL Server that will remove
all foreign key constraints first as some issues were
observed when using this flag when multiple schemas
had not been torn down.

Identified and fixed two subtle failure modes in the
engine, when commit/rollback fails in a begin()
context manager, the connection is explicitly closed,
and when "initialize()" fails on the first new connection
of a dialect, the transactional state on that connection
is still rolled back.

Fixes: #5826
Fixes: #5827
Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
2021-01-13 22:10:13 -05:00
Mike Bayer 4beecfb904 Emit deprecation warnings for plain text under Session
Deprecation warnings are emitted under "SQLALCHEMY_WARN_20" mode when
passing a plain string to :meth:`_orm.Session.execute`.

It was also considered to have DDL string expressions to include
this as well, however this leaves us with no backwards-compatible
way of handling reflection of elemens, such as an Index() which
reflects "postgresql_where='x > 5'", there's no place for a rule
that will turn those into text() within the reflection process
that would be separate from when the user passes postgresql_where
to the Index.  Not worth it right now.

Fixes: #5754
Change-Id: I8673a79f0e87de0df576b655f39dad0351725ca8
2020-12-11 15:39:46 -05:00
mike bayer 92ed93fedc Merge "Don't discard leftovers from surface_selectables" 2020-11-27 18:04:13 +00:00
Mike Bayer 6125dfff91 Don't discard leftovers from surface_selectables
Fixed regression introduced in 1.3.2 for the PostgreSQL dialect, also
copied out to the MySQL dialect's feature in 1.3.18, where usage of a non
:class:`_schema.Table` construct such as :func:`_sql.text` as the argument
to :paramref:`_sql.Select.with_for_update.of` would fail to be accommodated
correctly within the PostgreSQL or MySQL compilers.

Fixes: #5729
Change-Id: I265bcc171f0eb865ac3910ee805b162f3b70e2c1
2020-11-26 12:21:22 -05:00
Mike Bayer 584cabbf7e Support Column objects in the SET clause for upsert
Established support for :class:`_schema.Column` objects as well as ORM
instrumented attributes as keys in the ``set_`` dictionary passed to the
:meth:`_postgresql.Insert.on_conflict_do_update` and
:meth:`_sqlite.Insert.on_conflict_do_update` methods, which match to the
:class:`_schema.Column` objects in the ``.c`` collection of the target
:class:`_schema.Table`. Previously,  only string column names were
expected; a column expression would be assumed to be an out-of-table
expression that would render fully along with a warning.

Fixes: #5722
Change-Id: Ice73b501d721c28d978a0277a83cedc6aff756a9
2020-11-23 20:13:20 -05:00
AlonM 018aa98701 Add opsclass to exclusion constraint
Added new parameter :paramref:`_postgresql.ExcludeConstraint.ops` to the
:class:`_postgresql.ExcludeConstraint` object, to support operator class
specification with this constraint.  Pull request courtesy Alon Menczer.

Fixes: #5604
Closes: #5700
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5700
Pull-request-sha: f8613e1007

Change-Id: Iaf664131ec84f8c2fb05edf799198b8d3bb83597
2020-11-16 14:02:49 -05:00
Federico Caselli 34e6b732a1 Fetch first support
Add support to ``FETCH {FIRST | NEXT} [ count ] {ROW | ROWS}
 {ONLY | WITH TIES}`` in the select for the supported backends,
currently PostgreSQL, Oracle and MSSQL.

Fixes: #5576
Change-Id: Ibb5871a457c0555f82b37e354e7787d15575f1f7
2020-10-02 21:34:24 +02:00
Mike Bayer c3f102c9fe upgrade to black 20.8b1
It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.

Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
2020-09-28 15:17:26 -04:00
Mike Bayer f0f08db571 Complete deprecation of from_self()
For most from_self() tests, move them into
test/orm/test_deprecated.py and replace the existing
test with one that uses aliased() plus a subquery.
This then revealed a few more issues.

Related items:

* Added slice() method to GenerativeSelect, to match that
  of orm.Query and to make possible migration of one of the
  from_self() tests.  moved the utility functions used for this
  from orm/util into sql/util.

* repairs a caching issue related to subqueryload
  where information being derived from the cached path info
  was mixing up with query information based on the per-query
  state, specifically an AliasedClass that is per query.

* for the above issue, it seemed like path_registry maybe
  had to change so that it represents AliasedClass objects
  as their cache key rather than on identity, but it wasn't
  needed.  still seems like it would be more correct.

* enhances the error message raised by coercions for a case
  such as when an AliasedClass holds onto a select() object
  and not a subquery(); will name the original and resolved
  object for clarity  (although how is AliasedClass able to
  accept a Select() object in the first place?)

* Added _set_propagate_attrs() to Query so that again if
  it's passed to AliasedClass, it doesn't raise an error
  during coercion, but again maybe that should also be
  rejected up front

Fixes: #5368
Change-Id: I5912aa611d899acc87a75eb5ee9f95990592f210
2020-09-18 22:35:48 -04:00
mike bayer 645be4aa24 Merge "Improve handling of covering indexes" 2020-09-12 19:46:50 +00:00
Gord Thompson 1a08d1aade Improve handling of covering indexes
Improved support for covering indexes (with INCLUDE columns). Added the
ability for postgresql to render CREATE INDEX statements with an INCLUDE
clause from Core. Index reflection also report INCLUDE columns separately
for both mssql and postgresql (11+).

Fixes: #4458
Change-Id: If0b82103fbc898cdaeaf6a6d2d421c732744acd6
2020-09-12 13:00:16 -04:00
mike bayer f53e5fc7cb Merge "Reflect mssql/postgresql filtered/partial indexes" 2020-09-12 16:26:28 +00:00
RamonWill 69502725db Reflect mssql/postgresql filtered/partial indexes
Added support for inspection / reflection of partial indexes / filtered
indexes, i.e. those which use the ``mssql_where`` or ``postgresql_where``
parameters, with :class:`_schema.Index`.   The entry is both part of the
dictionary returned by :meth:`.Inspector.get_indexes` as well as part of a
reflected :class:`_schema.Index` construct that was reflected.  Pull
request courtesy Ramon Williams.

**Have a nice day!**
Fixes: #4966

Closes: #5504
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5504
Pull-request-sha: b3018bac98

Change-Id: Icbb2f93d1545700718ccb5222097185b815f5dbc
2020-09-12 10:30:43 -04:00
Federico Caselli e860060866 Update select usage to use the new 1.4 format
This change includes mainly that the bracketed use within
select() is moved to positional, and keyword arguments are
removed from calls to the select() function.  it does not
yet fully address other issues such as keyword arguments passed
to the table.select().

Additionally, allows False / None to both be considered
as "disable" for all of select.correlate(), select.correlate_except(),
query.correlate(), which establishes consistency with
passing of ``False`` for the legact select(correlate=False)
argument.

Change-Id: Ie6c6e6abfbd3d75d4c8de504c0cf0159e6999108
2020-09-08 17:13:48 -04:00
Federico Caselli b1b97ed1fc Add support for regular expression on supported backend.
Two operations have been defined:

* :meth:`~.ColumnOperators.regexp_match` implementing a regular
  expression match like function.
* :meth:`~.ColumnOperators.regexp_replace` implementing a regular
  expression string replace function.

Fixes: #1390
Change-Id: I44556846e4668ccf329023613bd26861d5c674e6
2020-08-27 17:30:18 -04:00
mike bayer 317f2e1be2 Merge "Add support for identity columns" 2020-08-21 21:39:40 +00:00
Federico Caselli 26e8d3b5bd Add support for identity columns
Added the :class:`_schema.Identity` construct that can be used to
configure identity columns rendered with GENERATED { ALWAYS |
BY DEFAULT } AS IDENTITY. Currently the supported backends are
PostgreSQL >= 10, Oracle >= 12 and MSSQL (with different syntax
and a subset of functionalities).

Fixes: #5362
Fixes: #5324
Fixes: #5360

Change-Id: Iecea6f3ceb36821e8b96f0b61049b580507a1875
2020-08-19 00:34:23 +02:00