Commit Graph

246 Commits

Author SHA1 Message Date
Mike Bayer 31f80b9eae Refactor for cx_Oracle version 6
Drops support for cx_Oracle prior to version 5.x, reworks
numeric and binary support.

Fixes: #4064

Change-Id: Ib9ae9aba430c15cd2a6eeb4e5e3fd8e97b5fe480
2017-09-11 14:17:10 -04:00
Mike Bayer 2efd89d029 Add SQL Server CI coverage
Change-Id: Ida0d01ae9bcc0573b86e24fddea620a38c962822
2017-08-31 17:20:26 -04:00
Mike Bayer 2c594da214 Isolate memory tests in forks
Swing the biggest hammer, run multiprocessing.Process() for
each memusage test individually so that they are fully isolated
from the parent process and any side effects of pytest-xdist

Also add --nomemory as a shortcut to exclude_tags=memory-intensive
and add this to the setup.py test runner as the memory tests
should not be running for quick runs

Change-Id: I3c16c781e21b33deb939a64e77a6e0e41fb86922
2017-08-14 12:41:58 -04:00
Mike Bayer f1fd7f1e01 - try adding the prints back in and re-test
Change-Id: I3a82b06ecc3c065240cc05871dee2881f20e414e
2017-08-11 09:54:16 -04:00
Mike Bayer 3568e508d0 - take more print statements out, maybe this is not
the issue
- unblock pytest-xdist now that the upstream issue is fixed,
maybe this old version is the issue

Change-Id: I28dd7ae0872948a188651d42e2f4af60bcbafe81
2017-08-10 15:46:37 -04:00
Mike Bayer 725298ea4b - dont print samples, this appears like it may be
itself causing the memory leak in conjunction with pytest-xdist

Change-Id: Ia8704e54186e6dd60ea0e32a246fcf1419686663
2017-08-08 17:24:23 -04:00
Mike Bayer 0a23654649 - allow the shrink phase for memusage to go until zeroed,
some MySQL-env element is causing memory growth that goes very
far before stopping

Change-Id: Ic0882dd78636067980fceba4e3a969de78d5b26a
2017-08-08 14:02:16 -04:00
Mike Bayer 19d2424e05 Add selectin loading
Adding a new kind of relationship loader that is
a cross between the "immediateload" and the "subquery"
eager loader, using an IN criteria to load related items
in bulk immediately after the lead query result is loaded.

Change-Id: If13713fba9b465865aef8fd50b5b6b977fe3ef7d
Fixes: #3944
2017-04-26 16:08:57 -04:00
Mike Bayer b7644319e8 Use baked lazyloading by default
The ``lazy="select"`` loader strategy now makes used of the
:class:`.BakedQuery` query caching system in all cases.  This
removes most overhead of generating a :class:`.Query` object and
running it into a :func:`.select` and then string SQL statement from
the process of lazy-loading related collections and objects.  The
"baked" lazy loader has also been improved such that it can now
cache in most cases where query load options are used.

Change-Id: Ic96792fffaa045ae9aa0a4657d6d29235d3efb85
Fixes: #3954
2017-04-13 14:22:59 -04:00
Mike Bayer bfaef1eaf5 - try one more test, then we're likely going to give up on cx_Oracle
Change-Id: I7f9a1265664b0368ee7a771d01c7ca1612156d1f
2017-04-12 15:28:05 -04:00
Mike Bayer b1f8d46fca - move a few memusage tests out of "backend". something is up w/ cx_Oracle
when the suite runs, such as a background thread or something like that,
which is affecting these tests a bit.

Change-Id: I52d50a44778ec1eecb8e335ae59b1a4773e80a79
2017-04-12 15:15:16 -04:00
Mike Bayer cef4e5ff38 Warn on _compiled_cache growth
Added warnings to the LRU "compiled cache" used by the :class:`.Mapper`
(and ultimately will be for other ORM-based LRU caches) such that
when the cache starts hitting its size limits, the application will
emit a warning that this is a performance-degrading situation that
may require attention.   The LRU caches can reach their size limits
primarily if an application is making use of an unbounded number
of :class:`.Engine` objects, which is an antipattern.  Otherwise,
this may suggest an issue that should be brought to the SQLAlchemy
developer's attention.

Additionally, adjusted the test_memusage algorithm again as the
previous one could still allow a growing memory size to be missed.

Change-Id: I020d1ceafb7a08f6addfa990a1e7acd09f933240
2017-04-12 12:53:40 -04:00
Mike Bayer f4c4f784cd Don't cache savepoint identifiers
Fixed bug in compiler where the string identifier of a savepoint would
be cached in the identifier quoting dictionary; as these identifiers
are arbitrary, a small memory leak could occur if a single
:class:`.Connection` had an unbounded number of savepoints used,
as well as if the savepoint clause constructs were used directly
with an unbounded umber of savepoint names.   The memory leak does
**not** impact the vast majority of cases as normally the
:class:`.Connection`, which renders savepoint names with a simple
counter starting at "1", is used on a per-transaction or
per-fixed-number-of-transactions basis before being discarded.

The savepoint name in virtually all cases does not require quoting
at all, however to support potential third party use cases
the "check for quotes needed" logic is retained, at a small
performance cost.   Uncondtionally quoting the name is another
option, but this would turn the name into a case sensitive name
which runs the risk of poor interactions with existing deployments
that may be looking at these names in other contexts.

Change-Id: I6b53c96abf7fdf1840592bbca5da81347911844c
Fixes: #3931
2017-03-06 17:20:06 -05:00
Mike Bayer fe1cabb88a Performance within instances()
Continuing from Ie43beecf37945b2bb7fff0aaa597a597293daa18,
also observed is the overhead of PathRegsitry memoized token
functions, as these paths are not cached in the case of
long joinedloader chains.  The memoizations here were made
with short paths in mind, and have been replaced with
an inlined straight create of these paths up front, producing
callcounts very similar to 0.8. Combined with the previous
optimizations, 1.1 now runs the "joined eager load of one row"
worst case test in about 40% fewer calls than 0.8 and 60%
fewer than 1.1.5.

Change-Id: Ib5e1c1345a1dd8edfbdb3fed06eb717d4e164d31
Fixes: #3915
2017-02-23 16:56:20 -05:00
Mike Bayer 1a7e8d77ba Memoize AliasedClass, ClauseAdapter objects in joined eager load
Addressed some long unattended performance concerns within the
joined eager loader query construction system.  The use of ad-hoc
:class:`.AliasedClass` objects per query, which produces lots of column
lookup overhead each time, has been replaced with a cached approach
that makes use of a small pool of :class:`.AliasedClass` objects
that are reused between invocations of joined eager loading.
Callcount reduction of SQL query generation for worst-case joined
loader scenarios (lots of joins, lots of columns) is reduced by
approximately 270%.

Change-Id: Ie43beecf37945b2bb7fff0aaa597a597293daa18
Fixes: #3915
2017-02-16 11:31:26 -05:00
Khairi Hafsham 772374735d Make all tests to be PEP8 compliant
tested using pycodestyle version 2.2.0

Fixes: #3885
Change-Id: I5df43adc3aefe318f9eeab72a078247a548ec566
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/343
2017-02-07 11:21:56 -05:00
Mike Bayer 23825a2591 Deprecate Mapper.order_by
This is an old parameter no longer relevant to how SQLAlchemy
works, once the Query object was introduced.  By deprecating it
we establish that we aren't supporting non-working use cases
and that we encourage applications to move off of the use of this
parameter.

Fixes: #3394
Change-Id: I25b9a38142a1537bbcb27d3e8b66a8b265140072
2016-04-19 18:17:45 -04:00
Mike Bayer 2419bfe34c - fix stray connection 2016-01-23 17:31:02 -05:00
Mike Bayer 8712ef2f81 - Added new checks for the common error case of passing mapped classes
or mapped instances into contexts where they are interpreted as
SQL bound parameters; a new exception is raised for this.
fixes #3321
2015-08-22 12:47:13 -04:00
Mike Bayer 92ac4a9f88 - revert this change which was inadvertently included in the
fix for #3451
2015-06-13 20:47:18 -04:00
Mike Bayer 2497d559dd - add changelog for #3451, with 09485d7331 fixes #3451
- also add a bulk_insert_mappings test
2015-06-13 20:15:17 -04:00
Eric Streeper 45876612d2 PEP8 cleanup in /test/aaa_profiling 2015-03-18 22:56:18 -07:00
Mike Bayer fcfa62bd76 - start locking down querying for cols after observing yesterdays
callcount bump due to the slots thing
- rewrite profiles using new technique
2015-03-09 13:07:09 -04:00
Mike Bayer 3eff76c4b0 - add the test_expire_lots test for comparison 2015-02-18 15:50:37 -05:00
Mike Bayer 026449c15f - Fixed a leak which would occur in the unsupported and highly
non-recommended use case of replacing a relationship on a fixed
mapped class many times, referring to an arbitrarily growing number of
target mappers.  A warning is emitted when the old relationship is
replaced, however if the mapping were already used for querying, the
old relationship would still be referenced within some registries.
fixes #3251
2014-11-13 13:17:38 -05:00
Mike Bayer 7c6a45c480 - The :func:~.expression.column and :func:~.expression.table
constructs are now importable from the "from sqlalchemy" namespace,
just like every other Core construct.
- The implicit conversion of strings to :func:`.text` constructs
when passed to most builder methods of :func:`.select` as
well as :class:`.Query` now emits a warning with just the
plain string sent.   The textual conversion still proceeds normally,
however.  The only method that accepts a string without a warning
are the "label reference" methods like order_by(), group_by();
these functions will now at compile time attempt to resolve a single
string argument to a column or label expression present in the
selectable; if none is located, the expression still renders, but
you get the warning again. The rationale here is that the implicit
conversion from string to text is more unexpected than not these days,
and it is better that the user send more direction to the Core / ORM
when passing a raw string as to what direction should be taken.
Core/ORM tutorials have been updated to go more in depth as to how text
is handled.
fixes #2992
2014-09-01 20:19:54 -04:00
Mike Bayer 3c60d3b1ca - A new style of warning can be emitted which will "filter" up to
N occurrences of a parameterized string.   This allows parameterized
warnings that can refer to their arguments to be delivered a fixed
number of times until allowing Python warning filters to squelch them,
and prevents memory from growing unbounded within Python's
warning registries.
fixes #3178
2014-08-31 15:22:00 -04:00
Mike Bayer 51c7005dc5 - clean up zoomark a little and try to get new profiles written 2014-08-30 00:11:32 -04:00
Mike Bayer b577afcb2b - rework profiling, zoomark tests into single tests so that
they can be used under xdist
2014-08-16 19:49:07 -04:00
Mike Bayer ab7c785d96 - turn off the testing reaper here, that's the source of the leaks
when we are running with pydist, and even when we are running without it
in fact...
2014-08-08 13:06:41 -04:00
Mike Bayer 54592942c4 - add support for tags, including include/exclude support.
simplify tox again now that we can exclude tests more easily
2014-07-27 18:46:20 -04:00
Mike Bayer a007c6e616 put a greater variance into this test to prevent sporadic failures 2014-07-21 12:09:12 -04:00
Mike Bayer ef968470c9 imports gone bad 2014-07-10 11:36:57 -04:00
Mike Bayer 1fc440393a - fully flake8 test/aaa_profiling 2014-07-09 18:59:16 -04:00
Matthias Urlichs f341cfbfe1 Documentation fix-up: "its" vs. "it's"
Removed ungrammatical apostrophes from documentation, replacing
"it's" with "its" where appropriate (but in a few cases with "it is"
when that read better).

While doing that, I also fixed a couple of minor typos etc.
as I noticed them.
2014-05-11 16:49:56 +02:00
Alex Gaynor 1caa7fafbd Fix many typos throughout the codebase
Found using: https://github.com/intgr/topy
2014-04-26 13:13:13 -07:00
Mike Bayer b6411ee3cf fixes to get profiling tests working again 2014-03-26 20:43:20 -04:00
Mike Bayer a465f3d031 use integer division here 2014-03-26 16:39:43 -04:00
Mike Bayer dc0d581d5d - rework memusage tests so that it only runs five iterations at a time, if it sees success within
that period, it's done.  memusage tests have become very slow
2014-03-26 15:18:29 -04:00
Mike Bayer 2f150bee28 - rename __multiple__ to __backend__, and apply __backend__ to a large number of tests.
- move out logging tests from test_execute to test_logging
2014-03-24 11:33:53 -04:00
Mike Bayer 30a50cc46a - the wrapped memoized_property here was not working, as the attribute name
didn't match.  use straight memoized_props here for now, add a perf test to check it
2013-11-26 23:24:13 -05:00
Mike Bayer 02f21ffcf3 - The :class:.RowProxy object is now sortable in Python as a regular
tuple is; this is accomplished via ensuring tuple() conversion on
both sides within the ``__eq__()`` method as well as
the addition of a ``__lt__()`` method. [ticket:2848]
2013-11-19 19:29:18 -05:00
Mike Bayer 08a6a8b519 - Removed some now unneeded version checks [ticket:2829] courtesy alex gaynor 2013-09-22 20:35:40 -04:00
Mike Bayer b83a1564f1 Improved support for the cymysql driver, supporting version 0.6.5,
courtesy Hajime Nakagami.
2013-08-17 00:31:57 -04:00
Mike Bayer 4505425a38 - Removal of event listeners is now implemented. The feature is
provided via the :func:`.event.remove` function.
[ticket:2268]
- reorganization of event.py module into a package; with the addition of the
docstring work as well as the new registry for removal, there's a lot more code now.
the package separates concerns and provides a top-level doc for each subsection
of functionality
- the remove feature works by providing the EventKey object which associates
the user-provided arguments to listen() with a global, weak-referencing registry.
This registry stores a collection of _ListenerCollection and _DispatchDescriptor
objects associated with each set of arguments, as well as the wrapped function
which was applied to that collection.  The EventKey can then be recreated for
a removal, all the _ListenerCollection and _DispatchDescriptor objects are located,
and the correct wrapped function is removed from each one.
2013-07-26 14:21:58 -04:00
Mike Bayer 731a4daf63 A performance fix related to the usage of the :func:.defer option
when loading mapped entities.   The function overhead of applying
a per-object deferred callable to an instance at load time was
significantly higher than that of just loading the data from the row
(note that ``defer()`` is meant to reduce DB/network overhead, not
necessarily function call count); the function call overhead is now
less than that of loading data from the column in all cases.  There
is also a reduction in the number of "lazy callable" objects created
per load from N (total deferred values in the result) to 1 (total
number of deferred cols).
[ticket:2778]
2013-07-13 16:29:07 -04:00
Mike Bayer b38a76cd1d - replace most explicitly-named test objects called "Mock..." with
actual mock objects from the mock library.  I'd like to use mock
for new tests so we might as well use it in obvious places.
- use unittest.mock in py3.3
- changelog
- add a note to README.unittests
- add tests_require in setup.py
- have tests import from sqlalchemy.testing.mock
- apply usage of mock to one of the event tests.  we can be using
this approach all over the place.
2013-06-30 18:35:12 -04:00
Mike Bayer 3b4168875d - zoomark tests
- rewrite all profiles, we'll review the diffs to see if anything is too far out
2013-05-26 19:47:13 -04:00
Mike Bayer 6cde27fe91 a pass where we try to squash down as many list()/keys() combinations
as possible
2013-05-26 19:06:13 -04:00
Mike Bayer 4b614b9b35 - the raw 2to3 run
- went through examples/ and cleaned out excess list() calls
2013-04-27 19:53:57 -04:00