Commit Graph

280 Commits

Author SHA1 Message Date
Mike Bayer 87e27693ad proposal: remove _execute_compiled()
this isn't used internally and only allows execute(stmt.compile())
to work, which is not something I want to support.

Since people are definitely using this [1] we might want to think about
either deprectation or having a recipe for people who really want
to still do this

[1] https://github.com/sqlalchemy/sqlalchemy/discussions/11108?converting=1

Change-Id: I5f0fe800e31aac052926cebe9206763eef9a804c
2025-10-23 21:38:15 +02:00
Mike Bayer 89a4174a8d Improve asyncpg exception hierarchy and asyncio hierarchies overall
The "emulated" exception hierarchies for the asyncio
drivers such as asyncpg, aiomysql, aioodbc, etc. have been standardized
on a common base :class:`.EmulatedDBAPIException`, which is now what's
available from the :attr:`.StatementException.orig` attribute on a
SQLAlchemy :class:`.DBAPIException` object.   Within :class:`.EmulatedDBAPIException`
and the subclasses in its hiearchy, the original driver-level exception is
also now avaliable via the :attr:`.EmulatedDBAPIException.orig` attribute,
and is also available from :class:`.DBAPIException` directly using the
:attr:`.DBAPIException.driver_exception` attribute.

Added additional emulated error classes for the subclasses of
``asyncpg.exception.IntegrityError`` including ``RestrictViolationError``,
``NotNullViolationError``, ``ForeignKeyViolationError``,
``UniqueViolationError`` ``CheckViolationError``,
``ExclusionViolationError``.  These exceptions are not directly thrown by
SQLAlchemy's asyncio emulation, however are available from the
newly added :attr:`.DBAPIException.driver_exception` attribute when a
:class:`.IntegrityError` is caught.

Fixes: #8047
Change-Id: I6a34e85b055265c087b0615f7c573be8582b3486
2025-09-23 09:49:19 -04:00
Mike Bayer b056dd2c5a re-support mysql-connector python
Support has been re-added for the MySQL-Connector/Python DBAPI using the
``mysql+mysqlconnector://`` URL scheme.   The DBAPI now works against
modern MySQL versions as well as MariaDB versions (in the latter case it's
required to pass charset/collation explicitly).   Note however that
server side cursor support is disabled due to unresolved issues with this
driver.

References: #12332
Change-Id: I81279478196e830d3c0d5f24ecb3fe2dc18d4ca6
2025-03-10 14:04:21 -04:00
Mike Bayer fc3623990e support accept for chains of joineddispatchers
Fixed issue where creating an :class:`.Engine` using multiple calls to
:meth:`.Engine.execution_options` where a subsequent call involved certain
options such as ``isolation_level`` would lead to an internal error
involving event registration.

Fixes: #12289
Change-Id: Iec5fbc0eb0c5a92dda1ea762872ae992ca816685
2025-01-29 10:10:09 -05:00
Federico Caselli 2adc79c077 Removed support for Python 3.8 since it's EOL.
Fixes: #12029
Change-Id: Ibb4efec9bab0225d03f6bf3fed661a3f2fc72cc7
2024-11-05 21:23:34 +01:00
Federico Caselli ffb2e2d033 Added support for server-side cursor in oracledb async dialect.
Added API support for server-side cursors for the oracledb async dialect,
allowing use of the :meth:`_asyncio.AsyncConnection.stream` and similar
stream methods.

Fixes: #10820

Change-Id: I861670ccc20a81ec5ee45132b8059fc2a0359087
2024-08-05 12:16:47 -04:00
Carlos Sousa af655e55d3 Warn in execute when parameter is an empty list
An empty sequence passed to any ``execute()`` method now
raised a deprecation warning, since such an executemany
is invalid.
Pull request courtesy of Carlos Sousa.

Fixes: #9647
Closes: #10406
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/10406
Pull-request-sha: 087ba2d88d

Change-Id: I482e91a047477c3156a3ca806e5c1eefb6224b95
2024-05-03 20:17:00 +02:00
Mike Bayer c449505f65 raise for asyncio-incompatible pool classes
An error is raised if a :class:`.QueuePool` or other non-asyncio pool class
is passed to :func:`_asyncio.create_async_engine`.  This engine only
accepts asyncio-compatible pool classes including
:class:`.AsyncAdaptedQueuePool`. Other pool classes such as
:class:`.NullPool` are compatible with both synchronous and asynchronous
engines as they do not perform any locking.

Fixes: #8771
Change-Id: I5843ccea7d824488492d1a9d46207b9f05330ae3
2024-02-16 10:02:07 -05:00
Federico Caselli 9b153ff18f Update black to 24.1.1
Change-Id: Iadaea7b798d8e99302e1acb430dc7b758ca61137
2024-01-31 21:54:59 +01:00
Mike Bayer 51b65467e1 accomodate schema translate keys present or not present
Adjusted the :paramref:`_sa.create_engine.schema_translate_map` feature
such that **all** schema names in the statement are now tokenized,
regardless of whether or not a specific name is in the immediate schema
translate map given, and to fallback to substituting the original name when
the key is not in the actual schema translate map at execution time.  These
two changes allow for repeated use of a compiled object with schema
schema_translate_maps that include or dont include various keys on each
run, allowing cached SQL constructs to continue to function at runtime when
schema translate maps with different sets of keys are used each time. In
addition, added detection of schema_translate_map dictionaries which gain
or lose a ``None`` key across calls for the same statement, which affects
compilation of the statement and is not compatible with caching; an
exception is raised for these scenarios.

Fixes: #10025
Change-Id: I6f5e0c8e067d1702a3647b6251af483669ad854b
2023-06-27 12:59:05 -04:00
Federico Caselli 058c230cea Update black to v23.3.0 and flake8 to v6
This change could be added to .git-blame-ignore-revs

Change-Id: I7ba10052b26bc3c178d23fb50a1123d0aae965ca
2023-05-30 21:44:26 +02:00
Mike Bayer 60b3119831 fix test suite warnings
fix a handful of warnings that were emitting but not raising,
usually because they were inside an "expect_warnings" block.

modify "expect_warnings" to always use "raise_on_any_unexpected"
behavior; remove this parameter.

Fixed issue in semi-private ``await_only()`` and ``await_fallback()``
concurrency functions where the given awaitable would remain un-awaited if
the function threw a ``GreenletError``, which could cause "was not awaited"
warnings later on if the program continued. In this case, the given
awaitable is now cancelled before the exception is thrown.

Change-Id: I33668c5e8c670454a3d879e559096fb873b57244
2023-05-09 20:23:48 -04:00
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
Federico Caselli 974b1bd0fc Revert automatic set of sequence start to 1
The :class:`.Sequence` construct restores itself to the DDL behavior it
had prior to the 1.4 series, where creating a :class:`.Sequence` with
no additional arguments will emit a simple ``CREATE SEQUENCE`` instruction
**without** any additional parameters for "start value".   For most backends,
this is how things worked previously in any case; **however**, for
MS SQL Server, the default value on this database is
``-2**63``; to prevent this generally impractical default
from taking effect on SQL Server, the :paramref:`.Sequence.start` parameter
should be provided.   As usage of :class:`.Sequence` is unusual
for SQL Server which for many years has standardized on ``IDENTITY``,
it is hoped that this change has minimal impact.

Fixes: #7211
Change-Id: I1207ea10c8cb1528a1519a0fb3581d9621c27b31
2022-10-17 15:36:25 -04:00
Mike Bayer 2bcc97da42 implement batched INSERT..VALUES () () for executemany
the feature is enabled for all built in backends
when RETURNING is used,
except for Oracle that doesn't need it,  and on
psycopg2 and mssql+pyodbc it is used for all INSERT statements,
not just those that use RETURNING.

third party dialects would need to opt in to the new feature
by setting use_insertmanyvalues to True.

Also adds dialect-level guards against using returning
with executemany where we dont have an implementation to
suit it.   execute single w/ returning still defers to the
server without us checking.

Fixes: #6047
Fixes: #7907
Change-Id: I3936d3c00003f02e322f2e43fb949d0e6e568304
2022-09-24 11:15:32 -04:00
Federico Caselli fcbdae075b Add support for the new oracle driver `oracledb`.
Fixes: #8054
Change-Id: Idd7c1bbb7ca39499f53bdf59a63a6a9d65f144a5
2022-06-07 12:59:57 -04:00
Mike Bayer ad14471bc9 Support handle_error for pre_ping
The :meth:`.DialectEvents.handle_error` event is now moved to the
:class:`.DialectEvents` suite from the :class:`.EngineEvents` suite, and
now participates in the connection pool "pre ping" event for those dialects
that make use of disconnect codes in order to detect if the database is
live. This allows end-user code to alter the state of "pre ping". Note that
this does not include dialects which contain a native "ping" method such as
that of psycopg2 or most MySQL dialects.

Fixes: #5648
Change-Id: I353d84a4f66f309d2467b7e67621db6b8c70411e
2022-05-31 15:06:28 -04:00
Mike Bayer 9f0db34563 update for flake8-future-imports 0.0.5
a whole bunch of errors were apparently blocked by 0.0.4
being installed.

Fixes: #8020
Change-Id: I22a0faeaabe03de501897893391946d677c2df7e
2022-05-14 12:28:01 -04:00
Mike Bayer c932123bac pep484: schema API
implement strict typing for schema.py

this module has lots of public API, lots of old decisions
and very hard to follow construction sequences in many
cases, and is also where we get a lot of new feature requests,
so strict typing should help keep things clean.

among improvements here, fixed the pool .info getters
and also figured out how to get ColumnCollection and
related to be covariant so that we may set them up
as returning Column or ColumnClause without any conflicts.

DDL was affected, noting that superclasses of DDLElement
(_DDLCompiles, added recently) can now be passed into
"ddl_if" callables; reorganized ddl into ExecutableDDLElement
as a new name for DDLElement and _DDLCompiles renamed to
BaseDDLElement.

setting up strict also located an API use case that
is completely broken, which is connection.execute(some_default)
returns a scalar value.   This case has been deprecated
and new paths have been set up so that connection.scalar()
may be used.  This likely wasn't possible in previous
versions because scalar() would assume a CursorResult.

The scalar() change also impacts Session as we have explicit
support (since someone had reported it as a regression)
for session.execute(Sequence()) to work.  They will get the
same deprecation message (which omits the word "Connection",
just uses ".execute()" and ".scalar()") and they can then
use Session.scalar() as well.  Getting this to type
correctly while still supporting ORM use cases required
some refactoring, and I also set up a keyword only delimeter
for Session.execute() and related as execution_options /
bind_arguments should always be keyword only, applied these
changes to AsyncSession as well.

Additionally simpify Table __init__ now that we are Python
3 only, we can have positional plus explicit kwargs finally.
Simplify Column.__init__ as well again taking advantage
of kw only arguments.

Fill in most/all __init__ methods in sqltypes.py as
the constructor for types is most of the API.   should
likely do this for dialect-specific types as well.

Apply _InfoType for all info attributes as should have been
done originally and update descriptor decorators.

Change-Id: I3f9f8ff3f1c8858471ff4545ac83d68c88107527
2022-04-15 10:29:23 -04:00
Mike Bayer 169ab546a9 add close=False parameter to engine.dispose()
Added new parameter :paramref:`.Engine.dispose.close`, defaulting to True.
When False, the engine disposal does not touch the connections in the old
pool at all, simply dropping the pool and replacing it. This use case is so
that when the original pool is transferred from a parent process, the
parent process may continue to use those connections.

Fixes: #7877
Change-Id: I88b0808442381ba5e50674787cdb64f0e77d8b54
2022-03-31 09:19:46 -04:00
mike bayer 678e7399e5 Merge "test #7820" into main 2022-03-21 19:57:52 +00:00
Mike Bayer 6c3d738757 pep 484 for types
strict types type_api.py, including TypeDecorator,
NativeForEmulated, etc.

Change-Id: Ib2eba26de0981324a83733954cb7044a29bbd7db
2022-03-19 23:15:15 -04:00
Mike Bayer fc5ec4fa97 test #7820
There was an apparent improvement in the distill params
methodology used in exec_driver_sql which allows raw tuples to
pass through.  In 1.4 there seems to be a _distill_cursor_params()
function that says it can handle this kind of parameter, but it isn't
used and when I tried to substitute it in for exec_driver_sql(),
things still fail.

In any case, add coverage here for the use case of passing
direct tuple params to exec_driver_sql including as the first
param, to note that it isn't mis-interpreted the way it is
in 1.x.

Change-Id: I27b875c0f874aee3f6f0d3e28c4c858dd39344e9
2022-03-14 17:03:59 -04:00
Mike Bayer 4c28867f94 additional mypy strictness
enable type checking within untyped defs.  This allowed
some more internals to be fixed up with assertions etc.

some internals that were unnecessary or not even used
at all were removed.  BaseCursorResult was no longer
necessary since we only have one kind of CursorResult
now.  The different ResultProxy subclasses that had
alternate "strategies" dont appear to be used at all
even in 1.4.x, as there's no code that accesses the
_cursor_strategy_cls attribute, which is also removed.
As these were mostly private constructs that weren't
even functioning correctly in any case,
it's fine to remove these over the 2.0 boundary.

Change-Id: Ifd536987d104b1cd8b546cefdbd5c1e5d1801082
2022-03-12 11:42:50 -05:00
Mike Bayer a4bb502cf9 pep-484 for engine
All modules in sqlalchemy.engine are strictly
typed with the exception of cursor, default, and
reflection.  cursor and default pass with non-strict
typing, reflection is waiting on the multi-reflection
refactor.

Behavioral changes:

* create_connect_args() methods return a tuple of list,
  dict, rather than a list of list, dict
* removed allow_chars parameter from
  pyodbc connector ._get_server_version_info()
  method
* the parameter list passed to do_executemany is now
  a list in all cases. previously, this was being run
  through dialect.execute_sequence_format, which
  defaults to tuple and was only intended for individual
  tuple params.
* broke up dialect.dbapi into dialect.import_dbapi
  class method and dialect.dbapi module object.  added
  a deprecation path for legacy dialects.  it's not
  really feasible to type a single attr as a classmethod
  vs. module type.  The "type_compiler" attribute also
  has this problem with greater ability to work around,
  left that one for now.
* lots of constants changing to be Enum, so that we can
  type them.  for fixed tuple-position constants in
  cursor.py / compiler.py (which are used to avoid the
  speed overhead of namedtuple), using Literal[value]
  which seems to work well
* some tightening up in Row regarding __getitem__, which
  we can do since we are on full 2.0 style result use
* altered the set_connection_execution_options and
  set_engine_execution_options event flows so that the
  dictionary of options may be mutated within the event
  hook, where it will then take effect as the actual
  options used.  Previously, changing the dict would
  be silently ignored which seems counter-intuitive
  and not very useful.
* A lot of DefaultDialect/DefaultExecutionContext
  methods and attributes, including underscored ones, move
  to interfaces.  This is not fully ideal as it means
  the Dialect/ExecutionContext interfaces aren't publicly
  subclassable directly, but their current purpose
  is more of documentation for dialect authors who should
  (and certainly are) still be subclassing the DefaultXYZ
  versions in all cases

Overall, Result was the most extremely difficult class
hierarchy to type here as this hierarchy passes through
largely amorphous "row" datatypes throughout, which
can in fact by all kinds of different things, like
raw DBAPI rows, or Row objects, or "scalar"/Any, but
at the same time these types have meaning so I tried still
maintaining some level of semantic markings for these,
it highlights how complex Result is now, as it's trying
to be extremely efficient and inlined while also being
very open-ended and extensible.

Change-Id: I98b75c0c09eab5355fc7a33ba41dd9874274f12a
2022-03-01 09:09:02 -05:00
Mike Bayer ff1ab665cb mypy: sqlalchemy.util
Starting to set up practices and conventions to
get the library typed.

Key goals for typing are:

1. whole library can pass mypy without any strict
   turned on.
2. we can incrementally turn on some strict flags on a per-package/
   module basis, as here we turn on more strictness for sqlalchemy.util, exc,
   and log
3. mypy ORM plugin tests work fully without sqlalchemy2-stubs
   installed
4. public facing methods all have return types, major parameter
   signatures filled in also
5. Foundational elements like util etc. are typed enough so that
   we can use them in fully typed internals higher up the stack.

Conventions set up here:

1. we can use lots of config in setup.cfg to limit where mypy
   is throwing errors and how detailed it should be in different
   packages / modules.  We can use this to push up gerrits
   that will pass tests fully without everything being typed.
2. a new tox target pep484 is added.  this links to a new jenkins
   pep484 job that works across all projects (alembic, dogpile, etc.)

We've worked around some mypy bugs that will likely
be around for awhile, and also set up some core practices
for how to deal with certain things such as public_factory
modules (mypy won't accept a module from a callable at all,
so need to use simple type checking conditionals).

References: #6810
Change-Id: I80be58029896a29fd9f491aa3215422a8b705e12
2022-01-24 15:14:01 -05:00
Federico Caselli 76fa211620 Replace c extension with cython versions.
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet

Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.

See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharing

Fixes: #7256
Change-Id: I2930ef1894b5048210384728118e586e813f6a76
Signed-off-by: Federico Caselli <cfederico87@gmail.com>
2021-12-17 21:29:05 +01:00
Mike Bayer 22deafe152 Warn when caching is disabled / document
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.

As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``.  These have been amended to use inherit
cache where appropriate.  "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.

The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.

Fixes: #7394
Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
2021-12-06 18:27:19 -05:00
mike bayer 8ddb3ef165 Merge "propose emulated setinputsizes embedded in the compiler" into main 2021-11-25 18:22:59 +00:00
Federico Caselli 31acba8ff7 Clean up most py3k compat
Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
2021-11-24 22:51:27 -05:00
Mike Bayer 939de240d3 propose emulated setinputsizes embedded in the compiler
Add a new system so that PostgreSQL and other dialects have a
reliable way to add casts to bound parameters in SQL statements,
replacing previous use of setinputsizes() for PG dialects.

rationale:

1. psycopg3 will be using the same SQLAlchemy-side "setinputsizes"
   as asyncpg, so we will be seeing a lot more of this

2. the full rendering that SQLAlchemy's compilation is performing
   is in the engine log as well as error messages.   Without this,
   we introduce three levels of SQL rendering, the compiler, the
   hidden "setinputsizes" in SQLAlchemy, and then whatever the DBAPI
   driver does.  With this new approach, users reporting bugs etc.
   will be less confused that there are as many as two separate
   layers of "hidden rendering"; SQLAlchemy's rendering is again
   fully transparent

3. calling upon a setinputsizes() method for every statement execution
   is expensive.  this way, the work is done behind the caching layer

4. for "fast insertmany()", I also want there to be a fast approach
   towards setinputsizes.  As it was, we were going to be taking
   a SQL INSERT with thousands of bound parameter placeholders and
   running a whole second pass on it to apply typecasts.    this way,
   we will at least be able to build the SQL string once without a huge
   second pass over the whole string

5. psycopg2 can use this same system for its ARRAY casts

6. the general need for PostgreSQL to have lots of type casts
   is now mostly in the base PostgreSQL dialect and works independently
   of a DBAPI being present.   dependence on DBAPI symbols that aren't
   complete / consistent / hashable is removed

I was originally going to try to build this into bind_expression(),
but it was revealed this worked poorly with custom bind_expression()
as well as empty sets.   the current impl also doesn't need to
run a second expression pass over the POSTCOMPILE sections, which
came out better than I originally thought it would.

Change-Id: I363e6d593d059add7bcc6d1f6c3f91dd2e683c0c
2021-11-23 16:52:55 -05:00
Federico Caselli 0b95f0055b Remove object in class definition
References: #4600
Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
2021-11-22 15:03:17 +00:00
mike bayer c0b0bf8ab8 Merge "Deprecate create_engine.implicit_returning" into main 2021-11-18 16:11:35 +00:00
mike bayer 863a048790 Merge "change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting" into main 2021-11-09 23:52:28 +00:00
Mike Bayer b919a0a85a change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".

Fixes: #7300
Change-Id: I6255c850b140522a4aba95085216d0bca18ce230
2021-11-09 15:30:58 -05:00
jonathan vanasco b2df5be7ee Deprecate create_engine.implicit_returning
The :paramref:`_sa.create_engine.implicit_returning` parameter is
deprecated on the :func:`_sa.create_engine` function only; the parameter
remains available on the :class:`_schema.Table` object. This parameter was
originally intended to enable the "implicit returning" feature of
SQLAlchemy when it was first developed and was not enabled by default.
Under modern use, there's no reason this parameter should be disabled, and
it has been observed to cause confusion as it degrades performance and
makes it more difficult for the ORM to retrieve recently inserted server
defaults. The parameter remains available on :class:`_schema.Table` to
specifically suit database-level edge cases which make RETURNING
infeasible, the sole example currently being SQL Server's limitation that
INSERT RETURNING may not be used on a table that has INSERT triggers on it.

Also removed from the Oracle dialect some logic that would upgrade
an Oracle 8/8i server version to use implicit returning if the
parameter were explictly passed; these versions of Oracle
still support RETURNING so the feature is now enabled for all
Oracle versions.

Fixes: #6962
Change-Id: Ib338e300cd7c8026c3083043f645084a8211aed8
2021-11-09 11:34:48 -05:00
Gord Thompson bd1be0b7e0 De-emphasize notion of "default driver" (DBAPI)
Fixes: #6960

Even though a default driver still exists for
each dialect, remove most usages of `dialect://`
to encourage users to explicitly specify
`dialect+driver://`

Change-Id: I0ad42167582df509138fca64996bbb53e379b1af
2021-11-09 06:12:39 -07:00
Mike Bayer d050193daa fully implement future engine and remove legacy
The major action here is to lift and move future.Connection
and future.Engine fully into sqlalchemy.engine.base.   This
removes lots of engine concepts, including:

* autocommit
* Connection running without a transaction, autobegin
  is now present in all cases
* most "autorollback" is obsolete
* Core-level subtransactions (i.e. MarkerTransaction)
* "branched" connections, copies of connections
* execution_options() returns self, not a new connection
* old argument formats, distill_params(), simplifies calling
  scheme between engine methods
* before/after_execute() events (oriented towards compiled constructs)
  don't emit for exec_driver_sql().  before/after_cursor_execute()
  is still included for this
* old helper methods superseded by context managers, connection.transaction(),
  engine.transaction() engine.run_callable()
* ancient engine-level reflection methods has_table(), table_names()
* sqlalchemy.testing.engines.proxying_engine

References: #7257

Change-Id: Ib20ed816642d873b84221378a9ec34480e01e82c
2021-11-07 14:30:35 -05:00
Mike Bayer 248d232459 Check for Mapping explicitly in 2.0 params
Fixed issue in future :class:`_future.Connection` object where the
:meth:`_future.Connection.execute` method would not accept a non-dict
mapping object, such as SQLAlchemy's own :class:`.RowMapping` or other
``abc.collections.Mapping`` object as a parameter dictionary.

Fixes: #7291
Change-Id: I819f079d86d19d1d81c570e0680f987e51e34b84
2021-11-04 18:07:21 -04:00
mike bayer 72780bae33 Merge "First round of removal of python 2" into main 2021-11-02 20:51:28 +00:00
Mike Bayer c4abf5a442 use full context manager flow for future.Engine.begin()
Fixed issue in future :class:`_future.Engine` where calling upon
:meth:`_future.Engine.begin` and entering the context manager would not
close the connection if the actual BEGIN operation failed for some reason,
such as an event handler raising an exception; this use case failed to be
tested for the future version of the engine. Note that the "future" context
managers which handle ``begin()`` blocks in Core and ORM don't actually run
the "BEGIN" operation until the context managers are actually entered. This
is different from the legacy version which runs the "BEGIN" operation up
front.

Fixes: #7272
Change-Id: I9667ac0861a9e007c4b3dfcf0fcf0829038a8711
2021-11-01 16:37:31 -04:00
Federico Caselli 36e7aebd8d First round of removal of python 2
References: #4600
Change-Id: I61e35bc93fe95610ae75b31c18a3282558cd4ffe
2021-11-01 15:11:25 -04:00
Federico Caselli ed78e679ea Remove deprecated dialects and drivers
Fixes: #7258
Change-Id: I3577f665eca04f2632b69bcb090f0a4ec9271db9
2021-10-31 12:31:56 -04:00
Federico Caselli 26140c0811 Surface driver connection object when using a proxied dialect
Improve the interface used by adapted drivers, like the asyncio ones,
to access the actual connection object returned by the driver.

The :class:`_engine._ConnectionRecord` and
:class:`_engine._ConnectionFairy` now have two new attributes:

* ``dbapi_connection`` always represents a DBAPI compatible
object.  For pep-249 drivers, this is the DBAPI connection as it always
has been, previously accessed under the ``.connection`` attribute.
For asyncio drivers that SQLAlchemy adapts into a pep-249 interface,
the returned object will normally be a SQLAlchemy adaption object
called :class:`_engine.AdaptedConnection`.
* ``driver_connection`` always represents the actual connection object
maintained by the third party pep-249 DBAPI or async driver in use.
For standard pep-249 DBAPIs, this will always be the same object
as that of the ``dbapi_connection``.  For an asyncio driver, it will be
the underlying asyncio-only connection object.

The ``.connection`` attribute remains available and is now a legacy alias
of ``.dbapi_connection``.

Fixes: #6832
Change-Id: Ib72f97deefca96dce4e61e7c38ba430068d6a82e
2021-09-17 18:08:42 -04:00
Miguel Grinberg 4127482cc5 Add scalars method to connection and session classes
Added new methods :meth:`_orm.Session.scalars`,
:meth:`_engine.Connection.scalars`, :meth:`_asyncio.AsyncSession.scalars`
and :meth:`_asyncio.AsyncSession.stream_scalars`, which provide a short cut
to the use case of receiving a row-oriented :class:`_result.Result` object
and converting it to a :class:`_result.ScalarResult` object via the
:meth:`_engine.Result.scalars` method, to return a list of values rather
than a list of rows. The new methods are analogous to the long existing
:meth:`_orm.Session.scalar` and :meth:`_engine.Connection.scalar` methods
used to return a single value from the first row only. Pull request
courtesy Miguel Grinberg.

Fixes: #6990
Closes: #6991
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/6991
Pull-request-sha: b3e0bb3042

Change-Id: Ia445775e24ca964b0162c2c8e5ca67dd1e39199f
2021-09-14 12:58:37 -04:00
Mike Bayer f9ba830ec0 restore statement substitution to before_execute()
Fixed issue where the ability of the
:meth:`_engine.ConnectionEvents.before_execute` method to alter the SQL
statement object passed, returning the new object to be invoked, was
inadvertently removed. This behavior has been restored.

The refactor in a1939719a6 removed this
feature for some reason and there were no tests in place to detect
it.  I don't see any indication this was planned.

Fixes: #6913
Change-Id: Ia77ca08aa91ab9403f19a8eb61e2a0e41aad138a
2021-08-20 22:17:55 -04:00
Mike Bayer 707e5d70fc labeling refactor
To service #6718 and #6710, the system by which columns are
given labels in a SELECT statement as well as the system that
gives them keys in a .c or .selected_columns collection have
been refactored to provide a single source of truth for
both, in constrast to the previous approach that included
similar logic repeated in slightly different ways.

Main ideas:

1. ColumnElement attributes ._label, ._anon_label, ._key_label
   are renamed to include the letters "tq", meaning
   "table-qualified" - these labels are only used when rendering
   a SELECT that has LABEL_STYLE_TABLENAME_PLUS_COL for its
   label style; as this label style is primarily legacy, the
   "tq" names should be isolated so that in a 2.0 style application
   these aren't being used at all

2. The means by which the "labels" and "proxy keys" for the elements
   of a SELECT has been centralized to a single source of truth;
   previously, the three of _generate_columns_plus_names,
   _generate_fromclause_column_proxies, and _column_naming_convention
   all had duplicated rules between them, as well as that there
   were a little bit of labeling rules in compiler._label_select_column
   as well; by this we mean that the various "anon_label" "anon_key"
   methods on ColumnElement were called by all four of these methods,
   where there were many cases where it was necessary that one method
   comes up with the same answer as another of the methods.  This
   has all been centralized into _generate_columns_plus_names
   for all the names except the "proxy key", which is generated
   by _column_naming_convention.

3. compiler._label_select_column has been rewritten to both not make
   any naming decisions nor any "proxy key" decisions, only whether
   to label or not to label; the _generate_columns_plus_names method
   gives it the information, where the proxy keys come from
   _column_naming_convention; previously, these proxy keys were matched
   based on restatement of similar (but not really the same) logic in
   two places.   The heuristics of "whether to label or not to label"
   are also reorganized to be much easier to read and understand.

4. a new method compiler._label_returning_column is added for dialects
   to use in their "generate returning columns" methods.   A
   github search reveals a small number of third party dialects also
   doing this using the prior _label_select_column method so we
   try to make sure _label_select_column continues to work the
   exact same way for that specific use case; for the "SELECT" use
   case it now needs

5. After some attempts to do it different ways, for the case where
   _proxy_key is giving us some kind of anon label, we are hard
   changing it to "_no_label" right now, as there's not currently
   a way to fully match anonymized labels from stmt.c or
   stmt.selected_columns to what will be in the result map.  The
   idea of "_no_label" is to encourage the user to use label('name')
   for columns they want to be able to target by string name that
   don't have a natural name.

Change-Id: I7a92a66f3a7e459ccf32587ac0a3c306650daf11
2021-07-12 18:50:29 -04:00
Mike Bayer 6967b45020 don't cache TypeDecorator by default
The :class:`.TypeDecorator` class will now emit a warning when used in SQL
compilation with caching unless the ``.cache_ok`` flag is set to ``True``
or ``False``. ``.cache_ok`` indicates that all the parameters passed to the
object are safe to be used as a cache key, ``False`` means they are not.

Fixes: #6436
Change-Id: Ib1bb7dc4b124e38521d615c2e2e691e4915594fb
2021-05-06 13:57:43 -04:00
Mike Bayer 37414a752b Add new "sync once" mode for pool.connect
Fixed critical regression caused by the change in :ticket`5497` where the
connection pool "init" phase no longer occurred within mutexed isolation,
allowing other threads to proceed with the dialect uninitialized, which
could then impact the compilation of SQL statements.

This issue is essentially the same regression which was fixed many years
ago in 🎫`2964` in dd32540dab,
which was missed this time as the test suite fo
that issue only tested the pool in isolation, and assumed the
"first_connect" event would be used by the Engine.  However
🎫`5497` stopped using "first_connect" and no test detected
the lack of mutexing, that has been resolved here through
the addition of more tests.

This fix also identifies what is probably a bug in earlier versions
of SQLAlchemy where the "first_connect" handler would be cancelled
if the initializer failed; this is evidenced by
test_explode_in_initializer which was doing a reconnect due to
c.rollback() yet wasn't hanging.  We now solve this issue by
preventing the manufactured Connection from ever reconnecting
inside the first_connect handler.

Also remove the "_sqla_unwrap" test attribute; this is almost
not used anymore however we can use a more targeted
wrapper supplied by the testing.engines.proxying_engine
function.

See if we can also open up Oracle for "ad hoc engines" tests
now that we have better connection management logic.

Fixes: #6337
Change-Id: I4a3476625c4606f1a304dbc940d500325e8adc1a
2021-04-21 22:31:42 -04:00
Mike Bayer b26cf96462 Explicitly test for Connection in dialect.has_table()
The :meth:`_engine.Dialect.has_table` method now raises an informative
exception if a non-Connection is passed to it, as this incorrect behavior
seems to be common.  This method is not intended for external use outside
of a dialect.  Please use the :meth:`.Inspector.has_table` method
or for cross-compatibility with older SQLAlchemy versions, the
:meth:`_engine.Engine.has_table` method.

Fixes: #5780
Fixes: #6062
Fixes: #6260
Change-Id: I9b2439675167019b68d682edee3dcdcfce836987
2021-04-14 12:14:12 -04:00