It's better, the majority of these changes look more readable to me.
also found some docstrings that had formatting / quoting issues.
Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
Provide support for multiple hosts in the PostgreSQL connection string.
A user requested for SQLAlchemy to support multiple hosts within a PostgreSQL URL string. The proposed fix allows this. In the event that the url contains multiple hosts the proposed code will convert the query["hosts"] tuple into a single string. This allows the hosts to then get converted into a valid dsn variable in the psycopg2 connect function.
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [X ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Fixes: #4392Closes: #5554
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5554
Pull-request-sha: 3f7a0ab8df
Change-Id: I3f3768d51b8331de786ffdc025b7ecfc662eafe5
As the test suite has widespread use of many patterns
that are deprecated, enable SQLALCHEMY_WARN_20 globally
for the test suite but then break the warnings filter
out into a whole list of all the individual warnings
we are looking for. this way individual changesets
can target a specific class of warning, as many of these
warnings will indivdidually affect dozens of files
and potentially hundreds of lines of code.
Many warnings are also resolved here as this
patch started out that way. From this point
forward there should be changesets that target a
subset of the warnings at a time.
For expediency, updates some migration 2.0 docs
for ORM as well.
Change-Id: I98b8defdf7c37b818b3824d02f7668e3f5f31c94
Added support for PostgreSQL "readonly" and "deferrable" flags for all of
psycopg2, asyncpg and pg8000 dialects. This takes advantage of a newly
generalized version of the "isolation level" API to support other kinds of
session attributes set via execution options that are reliably reset
when connections are returned to the connection pool.
Fixes: #5549
Change-Id: I0ad6d7a095e49d331618274c40ce75c76afdc7dd
The pg8000 dialect has been revised and modernized for the most recent
version of the pg8000 driver for PostgreSQL. Changes to the dialect
include:
* All data types are now sent as text rather than binary.
* Using adapters, custom types can be plugged in to pg8000.
* Previously, named prepared statements were used for all statements.
Now unnamed prepared statements are used by default, and named
prepared statements can be used explicitly by calling the
Connection.prepare() method, which returns a PreparedStatement
object.
Pull request courtesy Tony Locke.
Notes by Mike: to get this all working it was needed to break
up JSONIndexType into "str" and "int" subtypes; this will be
needed for any dialect that is dependent on setinputsizes().
also includes @caselit's idea to include query params
in the dbdriver parameter.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Closes: #5451
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5451
Pull-request-sha: 639751ca9c
Change-Id: I2869bc52c330916773a41d11d12c297aecc8fcd8
MySQL dialect's server_version_info tuple is now all numeric. String
tokens like "MariaDB" are no longer present so that numeric comparison
works in all cases. The .is_mariadb flag on the dialect should be
consulted for whether or not mariadb was detected. Additionally removed
structures meant to support extremely old MySQL versions 3.x and 4.x;
the minimum MySQL version supported is now version 5.0.2.
In addition, as the "MariaDB" name goes away from server version,
expand upon the change in I330815ebe572b6a9818377da56621397335fa702
to support the name "mariadb" throughout the dialect and test suite
when mariadb-only mode is used. This changes the "name" field
on the MariaDB dialect to "mariadb", which then implies a change
throughout the testing requirements system as well as all the
dialect-specific DDL argument names such as "mysql_engine" is
now specified as "mariadb_engine", etc. Make use of the
recent additions to test suite URL provisioning so that we can
force MariaDB databases to have a "mariadb-only" dialect which
allows us to test this name change fully.
Update documentation to refer to MySQL / MariaDB explicitly
as well as indicating the "mariadb_" prefix used for options.
It seems likely that MySQL and MariaDB version numbers are going to
start colliding at some point so having the "mariadb" name
be available as a totally separate dialect name should give us
some options in this regard.
Currently also includes a date related fix to a test for
the postgresql dialect that was implicitly assuming a
non-UTC timezone
Fixes: #4189
Change-Id: I00e76d00f62971e1f067bd61915fa6cc1cf64e5e
Using the approach introduced at
https://gist.github.com/zzzeek/6287e28054d3baddc07fa21a7227904e
We can now create asyncio endpoints that are then handled
in "implicit IO" form within the majority of the Core internals.
Then coroutines are re-exposed at the point at which we call
into asyncpg methods.
Patch includes:
* asyncpg dialect
* asyncio package
* engine, result, ORM session classes
* new test fixtures, tests
* some work with pep-484 and a short plugin for the
pyannotate package, which seems to have so-so results
Change-Id: Idbcc0eff72c4cad572914acdd6f40ddb1aef1a7d
Fixes: #3414
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles. This would make
migration simpler and reduce confusion.
However, confusion may be increased by the fact that select().join()
is different Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs. At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps. Session.execute() will still
behave the old way as we are adding a future flag.
This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.
The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.
Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.
Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.
Fixes: #5379Fixes: #5284
Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51
References: #5159
Build on #5401 to allow the ORM to take advanage
of executemany INSERT + RETURNING.
Implemented the feature
updated tests
to support INSERT DEFAULT VALUES, needed to come up with
a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT)
which can then be iterated out for executemany.
Added graceful degrade to plain executemany for PostgreSQL <= 8.2
Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN
Fix issue where unicode identifiers or parameter names wouldn't
work with execute_values() under Py2K, because we have to
encode the statement and therefore have to encode the
insert_single_values_expr too.
Correct issue from #5401 to support executemany + return_defaults
for a PK that is explicitly pre-generated, meaning we aren't actually
getting RETURNING but need to return it from compiled_parameters.
Fixes: #5263
Change-Id: Id68e5c158c4f9ebc33b61c06a448907921c2a657
The psycopg2 dialect now defaults to using the very performant
``execute_values()`` psycopg2 extension for compiled INSERT statements,
and also impements RETURNING support when this extension is used. This
allows INSERT statements that even include an autoincremented SERIAL
or IDENTITY value to run very fast while still being able to return the
newly generated primary key values. The ORM will then integrate this
new feature in a separate change.
Implements RETURNING for insert with executemany
Adds support to return_defaults() mode and inserted_primary_key
to support mutiple INSERTed rows, via return_defauls_rows
and inserted_primary_key_rows accessors.
within default execution context, new cached compiler
getters are used to fetch primary keys from rows
inserted_primary_key now returns a plain tuple. this
is not yet a row-like object however this can be
added.
Adds distinct "values_only" and "batch" modes, as
"values" has a lot of benefits but "batch" breaks
cursor.rowcount
psycopg2 minimum version 2.7 so we can remove the
large number of checks for very old versions of
psycopg2
simplify tests to no longer distinguish between
native and non-native json
Fixes: #5401
Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
- Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and
:class:`.Inspector` classes.
- Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`.
- Remove support for deprecated engine URLs of the form ``postgres://``.
- Remove deprecated dialect ``mysql+gaerdbms``.
- Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM`
and :class:`.mysql.SET` in the ``mysql`` dialect.
- Remove deprecated function ``comparable_property``. and function
``comparable_using`` in the declarative extension.
- Remove deprecated function ``compile_mappers``.
- Remove deprecated method ``collection.linker``.
- Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``.
This change also removes the class ``StrongInstanceDict``.
- Remove deprecated parameter ``mapper.order_by``.
- Remove deprecated parameter ``Session._enable_transaction_accounting`.
- Remove deprecated parameter ``Session.is_modified.passive``.
- Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`.
- Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and
``ClauseElement.__or__`` and attribute ``Over.func``.
- Remove deprecated ``FromClause.count`` method.
- Remove deprecated parameter ``Table.useexisting``.
- Remove deprecated parameters ``text.bindparams`` and ``text.typemap``.
- Remove boolean support for the ``passive`` parameter in ``get_history``.
- Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``.
Fixes: #4643
Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.
Fixes: #4848Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
Targeting select / insert / update / delete, the goal
is to minimize overhead of construction and generative methods
so that only the raw arguments passed are handled. An interim
stage that converts the raw state into more compiler-ready state
is added, which is analogous to the ORM QueryContext which will
also be rolled in to be a similar concept, as is currently
being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10.
the ORM update/delete BulkUD concept is also going to be rolled
onto this idea. So while the compiler-ready state object,
here called DMLState, looks a little thin, it's the
base of a bigger pattern that will allow for ORM functionality
to embed itself directly into the compiler, execution
context, and result set objects.
This change targets the DML objects, primarily focused on the
values() method which is the most complex process. The
work done by values() is minimized as much as possible
while still being able to create a cache key. Additional
computation is then offloaded to a new object ValuesState
that is handled by the compiler.
Architecturally, a big change here is that insert.values()
and update.values() will generate BindParameter objects for
the values now, which are then carefully received by crud.py
so that they generate the expected names. This is so that
the values() portion of these constructs is cacheable.
for the "multi-values" version of Insert, this is all skipped
and the plan right now is that a multi-values insert is
not worth caching (can always be revisited).
Using the
coercions system in values() also gets us nicer validation
for free, we can remove the NotAClauseElement thing from
schema, and we also now require scalar_subquery() is called
for an insert/update that uses a SELECT as a column value,
1.x deprecation path is added.
The traversal system is then applied to the DML objects
including tests so that they have traversal, cloning, and
cache key support. cloning is not a use case for DML however
having it present allows better validation of the structure
within the tests.
Special per-dialect DML is explicitly not cacheable at the moment,
more as a proof of concept that third party DML constructs can
exist as gracefully not-cacheable rather than producing an
incomplete cache key.
A few selected performance improvements have been added as well,
simplifying the immutabledict.union() method and adding
a new SQLCompiler function that can generate delimeter-separated
clauses like WHERE and ORDER BY without having to build
a ClauseList object at all. The use of ClauseList will
be removed from Select in an upcoming commit. Overall,
ClaustList is unnecessary for internal use and only adds
overhead to statement construction and will likely be removed
as much as possible except for explcit use of conjunctions like
and_() and or_().
Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
Removed all dialect code related to support for Jython and zxJDBC. Jython
has not been supported by SQLAlchemy for many years and it is not expected
that the current zxJDBC code is at all functional; for the moment it just
takes up space and adds confusion by showing up in documentation. At the
moment, it appears that Jython has achieved Python 2.7 support in its
releases but not Python 3. If Jython were to be supported again, the form
it should take is against the Python 3 version of Jython, and the various
zxJDBC stubs for various backends should be implemented as a third party
dialect.
Additionally modernized logic that distinguishes between "cpython"
and "pypy" to instead look at platform.python_distribution() which
reliably tells us if we are cPython or not; all booleans which
previously checked for pypy and sometimes jython are now converted
to be "not cpython", this impacts the test suite for tests that are
cPython centric.
Fixes: #5094
Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
Revised the approach for the just added support for the psycopg2
"execute_values()" feature added in 1.3.7 for 🎫`4623`. The approach
relied upon a regular expression that would fail to match for a more
complex INSERT statement such as one which had subqueries involved. The
new approach matches exactly the string that was rendered as the VALUES
clause.
Fixes: #4623
Change-Id: Icaae0f7b6bcf87a2cf5c6290a839c8429dd5fac3
Added new dialect flag for the psycopg2 dialect, ``executemany_mode`` which
supersedes the previous experimental ``use_batch_mode`` flag.
``executemany_mode`` supports both the "execute batch" and "execute values"
functions provided by psycopg2, the latter which is used for compiled
:func:`.insert` constructs. Pull request courtesy Yuval Dinari.
.. seealso::
:ref:`executemany_mode`
Fixes: #4623Closes: #4764
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4764
Pull-request-sha: c3d3a36f7e
Change-Id: I77e26ca729f9317af1488a6c054c23fa1a6b622b
- don't call relatively expensive isEnabledFor(), just call _log_notices
- don't reset the list if it's empty
- fix the test to use a custom function to definitely create a notice, confirmed
that PG seems to no longer create the "implicit sequence" notices
- assert that the reset of the notices works too
- update the docs to illustrate for folks who haven't worked with logging before
Change-Id: I7291e647c177d338e0ad673f3106b4d503e4b3ea
Moved the "rollback" which occurs during dialect initialization so that it
occurs after additional dialect-specific initialize steps, in particular
those of the psycopg2 dialect which would inadvertently leave transactional
state on the first new connection, which could interfere with some
psycopg2-specific APIs which require that no transaction is started. Pull
request courtesy Matthew Wilkes.
Fixes: #4663Closes: #4664
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4664
Pull-request-sha: e544fe671d
Change-Id: If40a15a1679b4eec0b8b8222f678697728009c30
Fixed regression from release 1.3.2 caused by 🎫`4562` where a URL
that contained only a query string and no hostname, such as for the
purposes of specifying a service file with connection information, would no
longer be propagated to psycopg2 properly. The change in 🎫`4562`
has been adjusted to further suit psycopg2's exact requirements, which is
that if there are any connection parameters whatsoever, the "dsn" parameter
is no longer required, so in this case the query string parameters are
passed alone.
Fixes: #4601
Change-Id: Ic29a8b77bcf50ee996968bab25aaac7ae4bfc26f
Added support for parameter-less connection URLs for the psycopg2 dialect,
meaning, the URL can be passed to :func:`.create_engine` as
``"postgresql+psycopg2://"`` with no additional arguments to indicate an
empty DSN passed to libpq, which indicates to connect to "localhost" with
no username, password, or database given. Pull request courtesy Julian
Mehnle.
Fixes: #4562Closes: #4563
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4563
Pull-request-sha: 8a05c96944
Change-Id: Ib6fca3c3c9eebeaf590d7f7fb0bc8cd4b6e4a55a
A large change throughout the library has ensured that all objects, parameters,
and behaviors which have been noted as deprecated or legacy now emit
``DeprecationWarning`` warnings when invoked. As the Python 3 interpreter now
defaults to displaying deprecation warnings, as well as that modern test suites
based on tools like tox and pytest tend to display deprecation warnings,
this change should make it easier to note what API features are obsolete.
See the notes added to the changelog and migration notes for further
details.
Fixes: #4393
Change-Id: If0ea11a1fc24f9a8029352eeadfc49a7a54c0a1b
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Extends AUTOCOMMIT_REGEXP for the postgres dialect to include `TRUNCATE`.
Change-Id: I315e03674b89bb89aae669b8655481e4d890491e
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/407
Added a new flag ``use_batch_mode`` to the psycopg2 dialect. This flag
enables the use of psycopg2's ``psycopg2.extras.execute_batch``
extension when the :class:`.Engine` calls upon ``cursor.executemany()``.
This extension provides a critical performance increase by over an order of magnitude
when running INSERT statements in batch. The flag is False by default
as it is considered to be experimental for now.
Change-Id: Ib88d28bc792958d47109f644ff1d08c897db4ff7
Fixes: #4109
Make sure we clear the plugin registry before testing
that the "postgres" name raises. Also move non-backend
tests out of MiscTest into a new suite.
Change-Id: Icd1bb4745aa07f52d585fcf959f76fcd8bdc7f24
Fixed bug where the pg8000 driver would fail if using
:meth:`.MetaData.reflect` with a schema name, since the schema name would
be sent as a "quoted_name" object that's a string subclass, which pg8000
doesn't recognize. The quoted_name type is added to pg8000's
py_types collection on connect.
Change-Id: Id0f838320cb66563685e094e4eae2d5116100d27
Fixes: #4041
Continuing with the fix that correctly handles Postgresql
version string "10devel" released in 1.1.8, an additional regexp
bump to handle version strings of the form "10beta1". While
Postgresql now offers better ways to get this information, we
are sticking w/ the regexp at least through 1.1.x for the least
amount of risk to compatibility w/ older or alternate Postgresql
databases.
Change-Id: I12ddb06465f7dcf80563c27632441ef5963f60d4
Fixes: #4005
Extends `AUTOCOMMIT_REGEXP` for the postgres dialect to include `GRANT` and `REVOKE`.
Change-Id: Iba15f1ebf5bd7bc0fc1193fdf561417e53bf5d57
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/357
Added support for parsing the Postgresql version string for
a development version like "PostgreSQL 10devel". Pull request
courtesy Sean McCully.
Change-Id: I7bc18bc4d290349c23e9796367b7d694d0873096
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/351
Added regular expressions for the "IMPORT FOREIGN SCHEMA",
"REFRESH MATERIALIZED VIEW" Postgresql statements so that they
autocommit when invoked via a connection or engine without
an explicit transaction. Pull requests courtesy Frazer McLean
and Paweł Stiasny.
Fixes: #3840
Co-authored-by: Frazer McLean
Co-authored-by: Paweł Stiasny
Change-Id: I92b2b61683d29d57fa23a66a3559120cb1241c2f
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/323
Some of the dialect impl memoization for TypeDecorator
necessarily keeps the top-level TypeDecorator type
around, since a user-defined type will have bind and result
set processing behavior. For both TypeDecorator and Variant,
PG dialect needs to ensure it's looking at the SQLAlchemy
type to check for SmallInteger / BigInteger.
Fixes: 3739
Change-Id: I2d45fb997f17c6272d6bb826a77d2dba665adae7
(cherry picked from commit 421fa6b8bf)
removed; this has emitted a warning for many years and projects
should be calling upon ``sqlalchemy.dialects.postgresql``.
Engine URLs of the form ``postgres://`` will still continue to function,
however.
psycopg2cffi dialect, in particular that the current 2.7.0 version
does not have native support for the JSONB type. The version detection
for psycopg2 features has been tuned into a specific sub-version
for psycopg2cffi. Additionally, test coverage has been enabled
for the full series of psycopg2 features under psycopg2cffi.
fixes#3439
:func:`.engine_from_config` were not being parsed correctly;
these included ``pool_threadlocal`` and the psycopg2 argument
``use_native_unicode``. fixes#3435
- add legacy_schema_aliasing config parsing for mssql
- move use_native_unicode config arg to the psycopg2 dialect
non-standard DBAPI exceptions, such as the psycopg2
TransactionRollbackError. These exceptions will now be raised
using the closest available subclass in ``sqlalchemy.exc``, in the
case of TransactionRollbackError, ``sqlalchemy.exc.OperationalError``.
fixes#3075
As with postgresql+psycopg2,
execution_options(isolation_level='AUTOCOMMIT') now works for the
postgresql+pg8000 dialect.
Also enabled the autocommit test in test_dialect.py for pg8000.
mike bayer