Commit Graph

935 Commits

Author SHA1 Message Date
Federico Caselli 4eb4ceca36 Try running pyupgrade on the code
command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>"
pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not
exists in sqlalchemy fixtures

Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
2022-11-16 23:03:04 +01:00
Mike Bayer b4261c45ab ensure pool.reset event always called for reset
Added new parameter :paramref:`.PoolEvents.reset.reset_state` parameter to
the :meth:`.PoolEvents.reset` event, with deprecation logic in place that
will continue to accept event hooks using the previous set of arguments.
This indicates various state information about how the reset is taking
place and is used to allow custom reset schemes to take place with full
context given.

Within this change a fix that's also backported to 1.4 is included which
re-enables the :meth:`.PoolEvents.reset` event to continue to take place
under all circumstances, including when :class:`.Connection` has already
"reset" the connection.

The two changes together allow custom reset schemes to be implemented using
the :meth:`.PoolEvents.reset` event, instead of the
:meth:`.PoolEvents.checkin` event (which continues to function as it always
has).

Change-Id: Ie17c4f55d02beb6f570b9de6b3044baffa7d6df6
Fixes: #8717
2022-10-30 15:43:19 -04:00
Federico Caselli 974b1bd0fc Revert automatic set of sequence start to 1
The :class:`.Sequence` construct restores itself to the DDL behavior it
had prior to the 1.4 series, where creating a :class:`.Sequence` with
no additional arguments will emit a simple ``CREATE SEQUENCE`` instruction
**without** any additional parameters for "start value".   For most backends,
this is how things worked previously in any case; **however**, for
MS SQL Server, the default value on this database is
``-2**63``; to prevent this generally impractical default
from taking effect on SQL Server, the :paramref:`.Sequence.start` parameter
should be provided.   As usage of :class:`.Sequence` is unusual
for SQL Server which for many years has standardized on ``IDENTITY``,
it is hoped that this change has minimal impact.

Fixes: #7211
Change-Id: I1207ea10c8cb1528a1519a0fb3581d9621c27b31
2022-10-17 15:36:25 -04:00
Mike Bayer 066aed6817 adjust tests for sqlites w/o returning
the sqlite builds on github actions seem to be very
inconsistent about versions and many don't support
RETURNING.  ensure any tests that depend on RETURNING present
are marked as such.

Change-Id: I7a60a81fa70b90642448cdd58eda33212c3afebc
2022-09-26 08:56:24 -04:00
Mike Bayer 2bcc97da42 implement batched INSERT..VALUES () () for executemany
the feature is enabled for all built in backends
when RETURNING is used,
except for Oracle that doesn't need it,  and on
psycopg2 and mssql+pyodbc it is used for all INSERT statements,
not just those that use RETURNING.

third party dialects would need to opt in to the new feature
by setting use_insertmanyvalues to True.

Also adds dialect-level guards against using returning
with executemany where we dont have an implementation to
suit it.   execute single w/ returning still defers to the
server without us checking.

Fixes: #6047
Fixes: #7907
Change-Id: I3936d3c00003f02e322f2e43fb949d0e6e568304
2022-09-24 11:15:32 -04:00
Yassen Damyanov 3333c6623f Tighten password security by removing URL.__str__
For improved security, the :class:`_url.URL` object will now use password
obfuscation by default when ``str(url)`` is called. To stringify a URL with
cleartext password, the :meth:`_url.URL.render_as_string` may be used,
passing the :paramref:`_url.URL.render_as_string.hide_password` parameter
as ``False``. Thanks to our contributors for this pull request.

Fixes: #8567
Closes: #8563
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/8563
Pull-request-sha: d1f1127f75

Change-Id: If756c8073ff99ac83876d9833c8fe1d7c76211f9
2022-09-23 17:27:30 -04:00
Federico Caselli 20917c5918 Ignore max_overflow when pool_size is 0
The :class:`_pool.QueuePool` now ignores ``max_overflow`` when
``pool_size=0``, properly making the pool unlimited in all cases.

Fixes: #8523
Change-Id: Ifc32eb47a281c4b3acf357352f07b9b8a73d1b6f
2022-09-15 22:42:08 +02:00
mike bayer 1ea3c783b6 Merge "integrate connection.terminate() for supporting dialects" into main 2022-08-24 18:08:17 +00:00
Mike Bayer 1c9da1e1d7 fix mock issue in pool test
Due to the change for #5648 in ad14471bc9,
the mock in test_recycle_pool_no_race needs to implement
an empty handle_error dispatcher.

the exception was in a thread so did not interrupt the
test suite:

File "/home/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 2059, in _handle_dbapi_exception
      for fn in self.dialect.dispatch.handle_error:
  TypeError: 'Mock' object is not iterable

Change-Id: I764357c48aa1bf53a572d8ee24c89a7463505092
2022-08-24 13:06:09 -04:00
Mike Bayer 776abf43d7 integrate connection.terminate() for supporting dialects
Integrated support for asyncpg's ``terminate()`` method call for cases
where the connection pool is recycling a possibly timed-out connection,
where a connection is being garbage collected that wasn't gracefully
closed, as well as when the connection has been invalidated. This allows
asyncpg to abandon the connection without waiting for a response that may
incur long timeouts.

Fixes: #8419
Change-Id: Ia575af779d5733b483a72dff3690b8bbbad2bb05
2022-08-24 13:02:23 -04:00
Mike Bayer c5a316053d support create/drop events for all CREATE/DROP
Implemented the DDL event hooks :meth:`.DDLEvents.before_create`,
:meth:`.DDLEvents.after_create`, :meth:`.DDLEvents.before_drop`,
:meth:`.DDLEvents.after_drop` for all :class:`.SchemaItem` objects that
include a distinct CREATE or DROP step, when that step is invoked as a
distinct SQL statement, including for :class:`.ForeignKeyConstraint`,
:class:`.Sequence`, :class:`.Index`, and PostgreSQL's
:class:`_postgresql.ENUM`.

Fixes: #8394
Change-Id: Iee6274984e794f50103451a04d089641d6ac443a
2022-08-20 11:10:33 -04:00
Federico Caselli 68a3374d5a Reflect expression-based indexes on PostgreSQL
The PostgreSQL dialect now supports reflection of expression based indexes.
The reflection is supported both when using
:meth:`_engine.Inspector.get_indexes` and when reflecting a
:class:`_schema.Table` using :paramref:`_schema.Table.autoload_with`.
Thanks to immerrr and Aidan Kane for the help on this ticket.

Fixes: #7442
Change-Id: I3e36d557235286c0f7f6d8276272ff9225058d48
2022-07-28 19:27:23 +02:00
Mike Bayer 4e2a89c41b remove warnings for index/unique skipped due to exclude_cols
The warnings that are emitted regarding reflection of indexes or unique
constraints, when the :paramref:`.Table.include_columns` parameter is used
to exclude columns that are then found to be part of those constraints,
have been removed. When the :paramref:`.Table.include_columns` parameter is
used it should be expected that the resulting :class:`.Table` construct
will not include constraints that rely upon omitted columns. This change
was made in response to 🎫`8100` which repaired
:paramref:`.Table.include_columns` in conjunction with foreign key
constraints that rely upon omitted columns, where the use case became
clear that omitting such constraints should be expected.

Fixes: #8102
Change-Id: Id32f628def2d12499cd49d0b436ed345fe49dc6b
2022-06-20 11:07:47 -04:00
Federico Caselli db08a69948 rearchitect reflection for batched performance
Rearchitected the schema reflection API to allow some dialects to make use
of high performing batch queries to reflect the schemas of many tables at
once using much fewer queries. The new performance features are targeted
first at the PostgreSQL and Oracle backends, and may be applied to any
dialect that makes use of SELECT queries against system catalog tables to
reflect tables (currently this omits the MySQL and SQLite dialects which
instead make use of parsing the "CREATE TABLE" statement, however these
dialects do not have a pre-existing performance issue with reflection. MS
SQL Server is still a TODO).

The new API is backwards compatible with the previous system, and should
require no changes to third party dialects to retain compatibility;
third party dialects can also opt into the new system by implementing
batched queries for schema reflection.

Along with this change is an updated reflection API that is fully
:pep:`484` typed, features many new methods and some changes.

Fixes: #4379
Change-Id: I897ec09843543aa7012bcdce758792ed3d415d08
2022-06-18 14:57:26 -04:00
mike bayer 73f867d996 Merge "graceful degrade for FKs not reflectable" into main 2022-06-08 17:13:28 +00:00
mike bayer 1961e13214 Merge "Add support for the new oracle driver `oracledb`." into main 2022-06-07 18:46:36 +00:00
Federico Caselli fcbdae075b Add support for the new oracle driver `oracledb`.
Fixes: #8054
Change-Id: Idd7c1bbb7ca39499f53bdf59a63a6a9d65f144a5
2022-06-07 12:59:57 -04:00
Mike Bayer 93bc7ed534 graceful degrade for FKs not reflectable
Fixed bugs involving the :paramref:`.Table.include_columns` and the
:paramref:`.Table.resolve_fks` parameters on :class:`.Table`; these
little-used parameters were apparently not working for columns that refer
to foreign key constraints.

In the first case, not-included columns that refer to foreign keys would
still attempt to create a :class:`.ForeignKey` object, producing errors
when attempting to resolve the columns for the foreign key constraint
within reflection; foreign key constraints that refer to skipped columns
are now omitted from the table reflection process in the same way as
occurs for :class:`.Index` and :class:`.UniqueConstraint` objects with the
same conditions. No warning is produced however, as we likely want to
remove the include_columns warnings for all constraints in 2.0.

In the latter case, the production of table aliases or subqueries would
fail on an FK related table not found despite the presence of
``resolve_fks=False``; the logic has been repaired so that if a related
table is not found, the :class:`.ForeignKey` object is still proxied to the
aliased table or subquery (these :class:`.ForeignKey` objects are normally
used in the production of join conditions), but it is sent with a flag that
it's not resolvable. The aliased table / subquery will then work normally,
with the exception that it cannot be used to generate a join condition
automatically, as the foreign key information is missing. This was already
the behavior for such foreign key constraints produced using non-reflection
methods, such as joining :class:`.Table` objects from different
:class:`.MetaData` collections.

Fixes: #8100
Fixes: #8101

Change-Id: Ifa37a91bd1f1785fca85ef163eec031660d9ea4d
2022-06-07 12:34:47 -04:00
Daniel Black 466ed5b53a Generalize RETURNING and suppor for MariaDB / SQLite
As almost every dialect supports RETURNING now, RETURNING
is also made more of a default assumption.

* the default compiler generates a RETURNING clause now
  when specified; CompileError is no longer raised.
* The dialect-level implicit_returning parameter now has
  no effect.   It's not fully clear if there are real world
  cases relying on the dialect-level parameter, so we will see
  once 2.0 is released.   ORM-level RETURNING can be disabled
  at the table level, and perhaps "implicit returning" should
  become an ORM-level option at some point as that's where
  it applies.
* Altered ORM update() / delete() to respect table-level
  implicit returning for fetch.
* Since MariaDB doesnt support UPDATE returning, "full_returning"
  is now split into insert_returning, update_returning, delete_returning
* Crazy new thing.  Dialects that have *both* cursor.lastrowid
  *and* returning.   so now we can pick between them for SQLite
  and mariadb.  so, we are trying to keep it on .lastrowid for
  simple inserts with an autoincrement column, this helps with
  some edge case test scenarios and i bet .lastrowid is faster
  anyway.  any return_defaults() / multiparams etc then we
  use returning
* SQLite decided they dont want to return rows that match in
  ON CONFLICT.  this is flat out wrong, but for now we need to
  work with it.

Fixes: #6195
Fixes: #7011
Closes: #7047
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7047
Pull-request-sha: d25d5ea3ab

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I9908ce0ff7bdc50bd5b27722081767c31c19a950
2022-06-02 12:51:20 -04:00
Mike Bayer ad14471bc9 Support handle_error for pre_ping
The :meth:`.DialectEvents.handle_error` event is now moved to the
:class:`.DialectEvents` suite from the :class:`.EngineEvents` suite, and
now participates in the connection pool "pre ping" event for those dialects
that make use of disconnect codes in order to detect if the database is
live. This allows end-user code to alter the state of "pre ping". Note that
this does not include dialects which contain a native "ping" method such as
that of psycopg2 or most MySQL dialects.

Fixes: #5648
Change-Id: I353d84a4f66f309d2467b7e67621db6b8c70411e
2022-05-31 15:06:28 -04:00
mike bayer e6792a3db7 Merge "enable pg8000 for 1.29.1 and above" into main 2022-05-25 14:12:41 +00:00
Mike Bayer c0612f8166 enable pg8000 for 1.29.1 and above
ROLLBACK TO SAVEPOINT is re-enabled
in https://github.com/tlocke/pg8000/issues/111.

we still have to add savepoint support to our fixture that
deletes from tables without checking for them.
this is inconvenient but not incorrect.

Change-Id: I2f4a0a3e18db93c3e6794ade9b0fee33d2e4b7dc
2022-05-23 14:53:26 -04:00
Mike Bayer 656fd0d73a deprecate .connection on _ConnectionFairy, _ConnectionRecord
These are replaced by the read-only ManagesConnection.dbapi_connection
attribute.

For some reason both of these objects had "setter" for .connection
as well; there's no use case for that at all so just remove
setter logic entirely.

Fixes: #6981
Change-Id: I6425de4a017f6370e1a7476cd491cabc55e55e67
2022-05-22 15:24:05 -04:00
Mike Bayer 9f0db34563 update for flake8-future-imports 0.0.5
a whole bunch of errors were apparently blocked by 0.0.4
being installed.

Fixes: #8020
Change-Id: I22a0faeaabe03de501897893391946d677c2df7e
2022-05-14 12:28:01 -04:00
Mike Bayer c932123bac pep484: schema API
implement strict typing for schema.py

this module has lots of public API, lots of old decisions
and very hard to follow construction sequences in many
cases, and is also where we get a lot of new feature requests,
so strict typing should help keep things clean.

among improvements here, fixed the pool .info getters
and also figured out how to get ColumnCollection and
related to be covariant so that we may set them up
as returning Column or ColumnClause without any conflicts.

DDL was affected, noting that superclasses of DDLElement
(_DDLCompiles, added recently) can now be passed into
"ddl_if" callables; reorganized ddl into ExecutableDDLElement
as a new name for DDLElement and _DDLCompiles renamed to
BaseDDLElement.

setting up strict also located an API use case that
is completely broken, which is connection.execute(some_default)
returns a scalar value.   This case has been deprecated
and new paths have been set up so that connection.scalar()
may be used.  This likely wasn't possible in previous
versions because scalar() would assume a CursorResult.

The scalar() change also impacts Session as we have explicit
support (since someone had reported it as a regression)
for session.execute(Sequence()) to work.  They will get the
same deprecation message (which omits the word "Connection",
just uses ".execute()" and ".scalar()") and they can then
use Session.scalar() as well.  Getting this to type
correctly while still supporting ORM use cases required
some refactoring, and I also set up a keyword only delimeter
for Session.execute() and related as execution_options /
bind_arguments should always be keyword only, applied these
changes to AsyncSession as well.

Additionally simpify Table __init__ now that we are Python
3 only, we can have positional plus explicit kwargs finally.
Simplify Column.__init__ as well again taking advantage
of kw only arguments.

Fill in most/all __init__ methods in sqltypes.py as
the constructor for types is most of the API.   should
likely do this for dialect-specific types as well.

Apply _InfoType for all info attributes as should have been
done originally and update descriptor decorators.

Change-Id: I3f9f8ff3f1c8858471ff4545ac83d68c88107527
2022-04-15 10:29:23 -04:00
Mike Bayer 2acc9ec128 cx_Oracle modernize
Full "RETURNING" support is implemented for the cx_Oracle dialect, meaning
multiple RETURNING rows are now recived for DML statements that produce
more than one row for RETURNING.

cx_Oracle 7 is now the minimum version for cx_Oracle.

Getting Oracle to do multirow returning took about 5 minutes.  however,
getting Oracle's RETURNING system to integrate with ORM-enabled
insert, update, delete, is a big deal because that architecture wasn't
really working very robustly, including some recent changes in 1.4
for FromStatement were done in a hurry, so this patch also cleans up
the FromStatement situation and begins to establish it more concretely
as the base for all ReturnsRows / TextClause ORM scenarios.

Fixes: #6245
Change-Id: I2b4e6007affa51ce311d2d5baa3917f356ab961f
2022-04-07 10:47:53 -04:00
Mike Bayer ccadbec825 use .fromisoformat() for sqlite datetime, date, time parsing
SQLite datetime, date, and time datatypes now use Python standard lib
``fromisoformat()`` methods in order to parse incoming datetime, date, and
time string values. This improves performance vs. the previous regular
expression-based approach, and also automatically accommodates for datetime
and time formats that contain either a six-digit "microseconds" format or a
three-digit "milliseconds" format.

Fixes: #7029
Change-Id: I67aab4fe5ee3055e5996050cf4564981413cc221
2022-04-03 14:47:52 -04:00
Mike Bayer 169ab546a9 add close=False parameter to engine.dispose()
Added new parameter :paramref:`.Engine.dispose.close`, defaulting to True.
When False, the engine disposal does not touch the connections in the old
pool at all, simply dropping the pool and replacing it. This use case is so
that when the original pool is transferred from a parent process, the
parent process may continue to use those connections.

Fixes: #7877
Change-Id: I88b0808442381ba5e50674787cdb64f0e77d8b54
2022-03-31 09:19:46 -04:00
mike bayer 59308fd60f Merge "generalize conditional DDL throughout schema / DDL" into main 2022-03-25 19:35:32 +00:00
Mike Bayer 57877461c1 generalize conditional DDL throughout schema / DDL
Expanded on the "conditional DDL" system implemented by the
:class:`_schema.DDLElement` class to be directly available on
:class:`_schema.SchemaItem` constructs such as :class:`_schema.Index`,
:class:`_schema.ForeignKeyConstraint`, etc. such that the conditional logic
for generating these elements is included within the default DDL emitting
process. This system can also be accommodated by a future release of
Alembic to support conditional DDL elements within all schema-management
systems.

Fixes: #7631
Change-Id: I9457524d7f66f49696187cf7d2b37dbb44f0e20b
2022-03-25 10:16:36 -04:00
Mike Bayer 5636692467 more autocommit messaging
Further clarified connection-level logging to indicate the BEGIN, ROLLBACK
and COMMIT log messages do not actually indicate a real transaction when
the AUTOCOMMIT isolation level is in use; messaging has been extended to
include the BEGIN message itself, and the messaging has also been fixed to
accommodate when the :class:`.Engine` level
:paramref:`.create_engine.isolation_level` parameter was used directly.

Fixes: #7853
Change-Id: Iafc78070737ad117f84262e4bde84b81a81e4ea1
2022-03-24 13:59:02 -04:00
mike bayer 678e7399e5 Merge "test #7820" into main 2022-03-21 19:57:52 +00:00
Mike Bayer 6c3d738757 pep 484 for types
strict types type_api.py, including TypeDecorator,
NativeForEmulated, etc.

Change-Id: Ib2eba26de0981324a83733954cb7044a29bbd7db
2022-03-19 23:15:15 -04:00
Mike Bayer fc5ec4fa97 test #7820
There was an apparent improvement in the distill params
methodology used in exec_driver_sql which allows raw tuples to
pass through.  In 1.4 there seems to be a _distill_cursor_params()
function that says it can handle this kind of parameter, but it isn't
used and when I tried to substitute it in for exec_driver_sql(),
things still fail.

In any case, add coverage here for the use case of passing
direct tuple params to exec_driver_sql including as the first
param, to note that it isn't mis-interpreted the way it is
in 1.x.

Change-Id: I27b875c0f874aee3f6f0d3e28c4c858dd39344e9
2022-03-14 17:03:59 -04:00
Mike Bayer 4c28867f94 additional mypy strictness
enable type checking within untyped defs.  This allowed
some more internals to be fixed up with assertions etc.

some internals that were unnecessary or not even used
at all were removed.  BaseCursorResult was no longer
necessary since we only have one kind of CursorResult
now.  The different ResultProxy subclasses that had
alternate "strategies" dont appear to be used at all
even in 1.4.x, as there's no code that accesses the
_cursor_strategy_cls attribute, which is also removed.
As these were mostly private constructs that weren't
even functioning correctly in any case,
it's fine to remove these over the 2.0 boundary.

Change-Id: Ifd536987d104b1cd8b546cefdbd5c1e5d1801082
2022-03-12 11:42:50 -05:00
Mike Bayer a4bb502cf9 pep-484 for engine
All modules in sqlalchemy.engine are strictly
typed with the exception of cursor, default, and
reflection.  cursor and default pass with non-strict
typing, reflection is waiting on the multi-reflection
refactor.

Behavioral changes:

* create_connect_args() methods return a tuple of list,
  dict, rather than a list of list, dict
* removed allow_chars parameter from
  pyodbc connector ._get_server_version_info()
  method
* the parameter list passed to do_executemany is now
  a list in all cases. previously, this was being run
  through dialect.execute_sequence_format, which
  defaults to tuple and was only intended for individual
  tuple params.
* broke up dialect.dbapi into dialect.import_dbapi
  class method and dialect.dbapi module object.  added
  a deprecation path for legacy dialects.  it's not
  really feasible to type a single attr as a classmethod
  vs. module type.  The "type_compiler" attribute also
  has this problem with greater ability to work around,
  left that one for now.
* lots of constants changing to be Enum, so that we can
  type them.  for fixed tuple-position constants in
  cursor.py / compiler.py (which are used to avoid the
  speed overhead of namedtuple), using Literal[value]
  which seems to work well
* some tightening up in Row regarding __getitem__, which
  we can do since we are on full 2.0 style result use
* altered the set_connection_execution_options and
  set_engine_execution_options event flows so that the
  dictionary of options may be mutated within the event
  hook, where it will then take effect as the actual
  options used.  Previously, changing the dict would
  be silently ignored which seems counter-intuitive
  and not very useful.
* A lot of DefaultDialect/DefaultExecutionContext
  methods and attributes, including underscored ones, move
  to interfaces.  This is not fully ideal as it means
  the Dialect/ExecutionContext interfaces aren't publicly
  subclassable directly, but their current purpose
  is more of documentation for dialect authors who should
  (and certainly are) still be subclassing the DefaultXYZ
  versions in all cases

Overall, Result was the most extremely difficult class
hierarchy to type here as this hierarchy passes through
largely amorphous "row" datatypes throughout, which
can in fact by all kinds of different things, like
raw DBAPI rows, or Row objects, or "scalar"/Any, but
at the same time these types have meaning so I tried still
maintaining some level of semantic markings for these,
it highlights how complex Result is now, as it's trying
to be extremely efficient and inlined while also being
very open-ended and extensible.

Change-Id: I98b75c0c09eab5355fc7a33ba41dd9874274f12a
2022-03-01 09:09:02 -05:00
zeeeeeb b9d231869d Implement generic Double and related fixed types
Added :class:`.Double`, :class:`.DOUBLE`, :class:`.DOUBLE_PRECISION`
datatypes to the base ``sqlalchemy.`` module namespace, for explicit use of
double/double precision as well as generic "double" datatypes. Use
:class:`.Double` for generic support that will resolve to DOUBLE/DOUBLE
PRECISION/FLOAT as needed for different backends.

Implemented DDL and reflection support for ``FLOAT`` datatypes which
include an explicit "binary_precision" value. Using the Oracle-specific
:class:`_oracle.FLOAT` datatype, the new parameter
:paramref:`_oracle.FLOAT.binary_precision` may be specified which will
render Oracle's precision for floating point types directly. This value is
interpreted during reflection. Upon reflecting back a ``FLOAT`` datatype,
the datatype returned is one of :class:`_types.DOUBLE_PRECISION` for a
``FLOAT`` for a precision of 126 (this is also Oracle's default precision
for ``FLOAT``), :class:`_types.REAL` for a precision of 63, and
:class:`_oracle.FLOAT` for a custom precision, as per Oracle documentation.

As part of this change, the generic :paramref:`_sqltypes.Float.precision`
value is explicitly rejected when generating DDL for Oracle, as this
precision cannot be accurately converted to "binary precision"; instead, an
error message encourages the use of
:meth:`_sqltypes.TypeEngine.with_variant` so that Oracle's specific form of
precision may be chosen exactly. This is a backwards-incompatible change in
behavior, as the previous "precision" value was silently ignored for
Oracle.

Fixes: #5465
Closes: #7674
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7674
Pull-request-sha: 5c68419e5a

Change-Id: I831f4af3ee3b23fde02e8f6393c83e23dd7cd34d
2022-02-25 00:51:32 +00:00
Mike Bayer e120837b68 updates for mariadb connector 1.0.10
Fixed regression in mariadbconnector dialect as of mariadb connector 1.0.10
where the DBAPI no longer pre-buffers cursor.lastrowid. The dialect now
fetches this value proactively for situations where it applies.

test_invalidate_on_results seems to pass for mariadbconnector now.
the driver has likely changed how it buffers result sets.  This is
a major change for them to make in a point release so we might
want to watch this in case they reverse course again.

Fixes: #7738
Change-Id: I9610aae01d1ae42fa92ffbc7123a6948e40ec9dd
2022-02-19 14:11:19 -05:00
Mike Bayer 5157e0aa54 pep-484 for pool
also extends into some areas of utils, events and others
as needed.

Formalizes a public hierarchy for pool API,
with ManagesConnection -> PoolProxiedConnection /
ConnectionPoolEntry for connectionfairy / connectionrecord,
which are now what's exposed in the event API and other
APIs.  all public API docs moved to the new objects.

Corrects the mypy plugin's check for sqlalchemy-stubs
not being insatlled, which has to be imported using the
dash in the name to be effective.

Change-Id: I16c2cb43b2e840d28e70a015f370a768e70f3581
2022-02-17 14:45:04 -05:00
Markus Gerstel f24a34140f Fix up Python logging metadata
Adjusted the logging for key SQLAlchemy components including
:class:`_engine.Engine`, :class:`_engine.Connection` to establish an
appropriate stack level parameter, so that the Python logging tokens
``funcName`` and ``lineno`` when used in custom logging formatters will
report the correct information, which can be useful when filtering log
output; supported on Python 3.8 and above. Pull request courtesy Markus
Gerstel.

Fixes: #7612
Closes: #7615
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7615
Pull-request-sha: cf9567beb0

Change-Id: Iff23c92ef3453ac93cbd0d190e7efbf8ea4457a2
2022-01-27 15:54:57 -05:00
Mike Bayer ff1ab665cb mypy: sqlalchemy.util
Starting to set up practices and conventions to
get the library typed.

Key goals for typing are:

1. whole library can pass mypy without any strict
   turned on.
2. we can incrementally turn on some strict flags on a per-package/
   module basis, as here we turn on more strictness for sqlalchemy.util, exc,
   and log
3. mypy ORM plugin tests work fully without sqlalchemy2-stubs
   installed
4. public facing methods all have return types, major parameter
   signatures filled in also
5. Foundational elements like util etc. are typed enough so that
   we can use them in fully typed internals higher up the stack.

Conventions set up here:

1. we can use lots of config in setup.cfg to limit where mypy
   is throwing errors and how detailed it should be in different
   packages / modules.  We can use this to push up gerrits
   that will pass tests fully without everything being typed.
2. a new tox target pep484 is added.  this links to a new jenkins
   pep484 job that works across all projects (alembic, dogpile, etc.)

We've worked around some mypy bugs that will likely
be around for awhile, and also set up some core practices
for how to deal with certain things such as public_factory
modules (mypy won't accept a module from a callable at all,
so need to use simple type checking conditionals).

References: #6810
Change-Id: I80be58029896a29fd9f491aa3215422a8b705e12
2022-01-24 15:14:01 -05:00
Mike Bayer de0b4db838 dont use exception catches for warnings; modernize xdist detection
Improvements to the test suite's integration with pytest such that the
"warnings" plugin, if manually enabled, will not interfere with the test
suite, such that third parties can enable the warnings plugin or make use
of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass.
Additionally, modernized the detection of the "pytest-xdist" plugin so that
plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
without breaking the test suite if xdist were still installed. Warning
filters that promote deprecation warnings to errors are now localized to
SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for
general Python deprecation warnings, so that non-SQLAlchemy deprecation
warnings emitted from pytest plugins should also not impact the test suite.

Fixes: #7599
Change-Id: Ibcf09af25228d39ee5a943fda82d8a9302433726
2022-01-22 19:17:10 -05:00
mike bayer 18c6b74562 Merge "remove dbapi_proxy module" into main 2022-01-15 00:13:34 +00:00
Mike Bayer e058d25cf0 remove dbapi_proxy module
This module was not documented nor part of any test suite,
and it's unlikely it was working correctly.   It's not likely
that this module was ever used after the first year or so
of SQLAlchemy, and it's stayed around because it is so
obscure that I never remembered to remove it.

Change-Id: I0ed9030438982e935add87c51abbfff50e7382be
References: #7257
2022-01-14 17:45:59 -05:00
Mike Bayer 4999784664 Initial ORM typing layout
introduces:

1. new mapped_column() helper
2. DeclarativeBase helper
3. declared_attr has been re-typed
4. rework of Mapped[] to return InstrumentedAtribute for
   class get, so works without Mapped itself having expression
   methods
5. ORM constructs now generic on [_T]

also includes some early typing work, most of which will
be in later commits:

1. URL and History become typing.NamedTuple
2. come up with type-checking friendly way of type
   checking cy extensions, where type checking will be applied
   to the py versions, just needed to come up with a succinct
   conditional pattern for the imports

References: #6810
References: #7535
References: #7562
Change-Id: Ie5d9a44631626c021d130ca4ce395aba623c71fb
2022-01-14 16:30:41 -05:00
Mike Bayer 01c50c64e3 Remove all remaining removed_in_20 warnings slated for removal
Finalize all remaining removed-in-2.0 changes so that we
can begin doing pep-484 typing without old things
getting in the way (we will also have to do public_factory).

note there are a few "moved_in_20()" and "became_legacy_in_20()"
warnings still in place.  The SQLALCHEMY_WARN_20 variable
is now removed.

Also removed here are the legacy "in place mutators" for Select
statements, and some keyword-only argument signatures in Core
have been added.

Also in the big change department, the ORM mapper() function
is removed entirely; the Mapper class is otherwise unchanged,
just the public-facing API function.  Mappers are now always
given a registry in which to participate, however the
argument signature of Mapper is not changed. ideally "registry"
would be the first positional argument.

Fixes: #7257

Change-Id: Ic70c57b9f1cf7eb996338af5183b11bdeb3e1623
2022-01-05 19:28:49 -05:00
Hugo van Kemenade 146a349d81 Update Black's target-version to py37
<!-- Provide a general summary of your proposed changes in the Title field above -->

### Description
<!-- Describe your changes in detail -->

Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.

Also update Black and other pre-commit hooks and re-format with Black.

### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)

-->

This pull request is:

- [ ] A documentation / typographical error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

**Have a nice day!**

Closes: #7536
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536
Pull-request-sha: b3aedf5570

Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
2022-01-05 12:41:32 -05:00
Federico Caselli 76fa211620 Replace c extension with cython versions.
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet

Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.

See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharing

Fixes: #7256
Change-Id: I2930ef1894b5048210384728118e586e813f6a76
Signed-off-by: Federico Caselli <cfederico87@gmail.com>
2021-12-17 21:29:05 +01:00
mike bayer ca186ff935 Merge "Warn when caching is disabled / document" into main 2021-12-07 15:46:54 +00:00
mike bayer 1411f0bc48 Merge "contextmanager skips rollback if trans says to skip it" into main 2021-12-07 14:52:55 +00:00