Use cython for the internal result methods. This improves the speed
of the row processing of about 15-30% depending on cases.
Change-Id: Ia19c1a61aa1a55405b8327f88b1ee02627c4217a
Implemented initial support for free-threaded Python by adding new tests
and reworking the test harness and GitHub Actions to include Python 3.13t
and Python 3.14t in test runs. Two concurrency issues have been identified
and fixed: the first involves initialization of the ``.c`` collection on a
``FromClause``, a continuation of 🎫`12302`, where an optional mutex
under free-threading is added; the second involves synchronization of the
pool "first_connect" event, which first received thread synchronization in
🎫`2964`, however under free-threading the creation of the mutex
itself runs under the same free-threading mutex. Initial pull request and
test suite courtesy Lysandros Nikolaou.
py313t: yes
py314t: yes
Fixes: #12881Closes: #12882
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/12882
Pull-request-sha: 53d65d96b9
Co-authored-by: Mike Bayer <mike_mp@zzzcomputng.com>
Change-Id: I2e4f2e9ac974ab6382cb0520cc446b396d9680a6
Replaces the pyx files with py files that can be both compiled
by cython or imported as is by python.
This avoids the need of duplicating the code to have a python
only fallback.
The cython files are also reorganized to be in the module they use
instead of all being in the cyextension package, that has been
removed.
The performance is pretty much equal between main and this change.
A detailed comparison is at this link
https://docs.google.com/spreadsheets/d/1jkmGpnCyEcPyy6aRK9alElGjxlNHu44Wxjr4VrD99so/edit?usp=sharing
Change-Id: Iaed232ea5dfb41534cc9f58f6ea2f912a93263af
Ensure that all cython extension are imported by the compied detection logic.
This is required since cython extensions moduels are marked as optional
in the install, so it's possible that only some of them are compiled.
The extensions are enabled only if all of them are correctly compiled
Change-Id: I355cbac06f5c7a47d35661f42ebab3b0156c1965
This greatly simplifies the setup.py file and makes it compatible with
setuptools 64+ change to editable install.
Change-Id: I16a776b175bcf67e26f4bbb8f80b1b473f585e2a
Using cibuildwheel the following wheels are created:
- windows x64 and x84
- macos x64 and arm
- linux x64 and arm on manylinux and mosulinux (for alpine)
- create a pure python wheel (for pypy and other archs)
Fixes: #6702Fixes: #7607Closes: #7992
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7992
Pull-request-sha: 61d5e24e5b
Change-Id: If0c0b353766e0b61d421789d619eb2b940c08ad0
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet
Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.
See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharingFixes: #7256
Change-Id: I2930ef1894b5048210384728118e586e813f6a76
Signed-off-by: Federico Caselli <cfederico87@gmail.com>
Python 3.10 has deprecated "distutils" in favor of explicit use of
"setuptools" in :pep:`632`; SQLAlchemy's setup.py has replaced imports
accordingly. However, since setuptools itself only recently added the
replacement symbols mentioned in pep-632 as of November of 2022 in version
59.0.1, ``setup.py`` still has fallback imports to distutils, as SQLAlchemy
1.4 does not have a hard setuptools versioning requirement at this time.
SQLAlchemy 2.0 is expected to use a full :pep:`517` installation layout
which will indicate appropriate setuptools versioning up front.
Fixes: #7311
Change-Id: I215ef3c3b226a38266f59d181214aea462c4664d
* remove the c version of distill params since it's actually slower than
the python one
* add a function to langhelpers to check if the cextensions are active
* minor cleanup to the OrderedSet implementation
Change-Id: Iec3d0c3f0f42cdf51f802aaca342ba37b8783b85
* ensure on python 2 correct cflags to fail on undefined
symbols take effect
* fail for implicit function declaration
* python 2 does not publish PyDict_GetItemWithError but has
it as _PyDict_GetItemWIthError but only as of Python 2.7.12
Change-Id: I007509afddf7f44ca64e52fa9140be39f815fa7a
The ``importlib_metadata`` library is used to scan for setuptools
entrypoints rather than pkg_resources. as importlib_metadata is a small
library that is included as of Python 3.8, the compatibility library is
installed as a dependency for Python versions older than 3.8.
Unfortunately setuptools "attr:" is broken because it tries to import
the module; seems like this is fixed as part of
https://github.com/pypa/setuptools/pull/1753 however this is too recent
to rely upon for now.
Added a new dialect token "mariadb" that may be used in place of "mysql" in
the :func:`_sa.create_engine` URL. This will deliver a MariaDB dialect
subclass of the MySQLDialect in use that forces the "is_mariadb" flag to
True. The dialect will raise an error if a server version string that does
not indicate MariaDB in use is received. This is useful for
MariaDB-specific testing scenarios as well as to support applications that
are hardcoding to MariaDB-only concepts. As MariaDB and MySQL featuresets
and usage patterns continue to diverge, this pattern may become more
prominent.
Fixes: #5400Fixes: #5496
Change-Id: I330815ebe572b6a9818377da56621397335fa702
Start trying to convert fundamental objects to
C as we now rely on a fairly small core of things,
and 1.4 is having problems with complexity added being
slower than the performance gains we are trying to build in.
immutabledict here does seem to bench as twice as fast as the
Python one, see below. However, it does not appear to be
used prominently enough to make any dent in the performance
tests.
at the very least it may provide us some more lift-and-copy
code for more C extensions.
import timeit
from sqlalchemy.util._collections import not_immutabledict, immutabledict
def run(dict_cls):
for i in range(1000000):
d1 = dict_cls({"x": 5, "y": 4})
d2 = d1.union({"x": 17, "new key": "some other value"}, None)
assert list(d2) == ["x", "y", "new key"]
print(
timeit.timeit(
"run(d)", "from __main__ import run, not_immutabledict as d", number=1
)
)
print(
timeit.timeit(
"run(d)", "from __main__ import run, immutabledict as d", number=1
)
)
output:
python: 1.8799766399897635
C code: 0.8880784640205093
Change-Id: I29e7104dc21dcc7cdf895bf274003af2e219bf6d
- Fix typo in resultproxy.c that would error on windows.
- add -Wundef to C flags when linux is detected so that undefined
symbols emit a warning
- a few adjustments for tests to succeed on python 3.5
- note minimum version still documented here as 3.4 but this should
move to at least 3.5 if not 3.6 for SQLAlchemy 1.4
Change-Id: Ia93ee1cb5c52e51e72eb0a24c100421c5157d04b
Fixes: #5113
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
Add py3.8 and py3.9 tokens to setup.py
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
**Have a nice day!**
Closes: #5115
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5115
Pull-request-sha: 5ae662f1ec
Change-Id: I1313df9bd117b1356c1d06d588e8e04a07ba4e39
The C extensions have been broken since cc718cccc0
however CI did not find this, because the build degraded to
non-C extensions without failing. Ensure that if cext is set,
there is no fallback to non-cext build if the C extension build
fails.
Repair C related issues introduced in cc718cccc0.
As C extensions have been silently failing on 2.7 for some commits,
the callcounts also needed to be adjusted for recent performance-related
changes. That in turn required a fix to the profiling decorator
to use signature rewriting in order to support py.test's
fixture mechanism under Python 2, usage introduced under profiling
in 89bf6d80a9.
Fixes: #5076
Change-Id: Id968f10c85d6bf489298b1c318a1f869ad3e7d80
Added a workaround for a setuptools-related failure that has been observed
as occurring on Windows installations, where setuptools is not correctly
reporting a build error when the MSVC build dependencies are not installed
and therefore not allowing graceful degradation into non C extensions
builds.
Setuptools issue https://github.com/pypa/setuptools/issues/1902Fixes: #4967
Change-Id: I5e21e7e78cb6d927b18afce64cacf8643b98354e
While we have the OSI classifier for "MIT License", it looks
like for the "license" field, this is normally just the word
"MIT" and not "MIT License". While the pypa docs suggest we
only need it as the OSI classifier, keep it also in "license"
in order to appease common tooling.
Change-Id: Ife51bbc74d6c1b8ab9a736024818fbba35316e17
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.
Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
This is a straight reformat run using black as is, with no edits
applied at all.
The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines. The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.
Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
Fixes to the test suite, a few errant imports, and setup.py:
- mysql and postgresql have unused 'json' imports; remove
- postgresql is exporting the 'json' symbol, remove
- make sure setup.py can find __version__ using " or '
- retry logic in provision create database for postgresql fixed
- refactor test_magazine to use cls.tables rather than globals
- remove unused class in test_scoping
- add a comment to test_deprecations that this test suite itself
is deprecated
- don't use mapper() and orm_mapper() in test_unitofwork, just
use mapper()
- remove dupe test_scalar_set_None test in test_attributes
- Python 2.7 and above includes unittest.SkipTest, remove pre-2.7
fallback
- use imported SkipTest in profiling
- declarative test_reflection tests with "reflectable_autoincrement"
already don't run on oracle or firebird; remove conditional logic
for these, which also removes an "id" symbol
- clean up test in test_functions, remove print statement
- remove dupe test_literal_processor_coercion_native_int_out_of_range
in test/sql/test_types.py
- fix psycopg2_hstore ref
Change-Id: I7b3444f8546aac82be81cd1e7b6d8b2ad6834fe6
Unfortunately they have released
3.9.2 before fixing issue 4181, which means we will have to
re-pin a third time after they eventually release with
a fix
Change-Id: I9b6fab1b535f883a035f002264c41ad0166790ab
Reference: https://github.com/pytest-dev/pytest/issues/4181
As we are changing Oracle typing *again*, we should start tracking
if we trip over a big performance hit on the unicode stuff.
Change-Id: I72719cca4b9424171b32e1f2e58d655426e3bbba
Swing the biggest hammer, run multiprocessing.Process() for
each memusage test individually so that they are fully isolated
from the parent process and any side effects of pytest-xdist
Also add --nomemory as a shortcut to exclude_tags=memory-intensive
and add this to the setup.py test runner as the memory tests
should not be running for quick runs
Change-Id: I3c16c781e21b33deb939a64e77a6e0e41fb86922
By telling wheel that we have extension modules, even though we
have none, wheel will create a Wheel which is platform and
interpreter specific. This will ensure that the pure Python wheels
on PyPy do not trigger installs on CPython without the C speedups.
- add first-class "pytest" runner to setup.py, fixes#3489
- remove references to setuptools Feature and mark that "--without-cextensions"
is removed, fixes#3500
removal of the ``setuptools.Feature`` extension from setuptools.
If this keyword isn't present, the setup will still succeed
with setuptools rather than falling back to distutils. C extension
building can be disabled now also by setting the
DISABLE_SQLALCHEMY_CEXT environment variable. This variable works
whether or not setuptools is even available. fixes#2986
- using platform.python_implementation() in setup.py to detect CPython.
I've tested this function on OSX and linux on Python 2.6 through 3.4,
including 3.1, 3.2, 3.3.
Unfortunately, on OSX + 3.2 only, it seems to segfault. I've tried
installing 3.2.5 from the python.org .dmg, building it from source,
and also blew away the whole 3.2 directory, something seems to be wrong
with the "platform" module on that platform only, and there's also no
issue on bugs.python.org; however, I'm going with
it anyway. If someone is using 3.2 on OSX they really should be upgrading.
is currently being supported in addition to nose, and will likely
be preferred to nose going forward. The nose plugin system used
by SQLAlchemy has been split out so that it works under pytest as
well. There are no plans to drop support for nose at the moment
and we hope that the test suite itself can continue to remain as
agnostic of testing platform as possible. See the file
README.unittests.rst for updated information on running tests
with pytest.
The test plugin system has also been enhanced to support running
tests against mutiple database URLs at once, by specifying the ``--db``
and/or ``--dburi`` flags multiple times. This does not run the entire test
suite for each database, but instead allows test cases that are specific
to certain backends make use of that backend as the test is run.
When using pytest as the test runner, the system will also run
specific test suites multiple times, once for each database, particularly
those tests within the "dialect suite". The plan is that the enhanced
system will also be used by Alembic, and allow Alembic to run
migration operation tests against multiple backends in one run, including
third-party backends not included within Alembic itself.
Third party dialects and extensions are also encouraged to standardize
on SQLAlchemy's test suite as a basis; see the file README.dialects.rst
for background on building out from SQLAlchemy's test platform.
actual mock objects from the mock library. I'd like to use mock
for new tests so we might as well use it in obvious places.
- use unittest.mock in py3.3
- changelog
- add a note to README.unittests
- add tests_require in setup.py
- have tests import from sqlalchemy.testing.mock
- apply usage of mock to one of the event tests. we can be using
this approach all over the place.
Federico Caselli