Threads are gone after fork, so clear the queues too. Otherwise the
child process (here created via multiprocessing.Process) crashes on
interpreter exit.
(cherry picked from commit 1848ce61f3)
Co-authored-by: Andrei Bodrov <Drino@users.noreply.github.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
There was a deadlock when `ProcessPoolExecutor` shuts down at the same
time that a queueing thread handles an error processing a task.
Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use
an internal lock instead. This fixes the ordering deadlock where the
`ExecutorManagerThread` holds the `_shutdown_lock` and joins the
queueing thread, while the queueing thread is attempting to acquire the
`_shutdown_lock` while closing the `_ThreadWakeup`.
(cherry picked from commit 760872efec)
This was left out of the 3.12 backport for three related issues:
- gh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- gh-109370 (which changes this to be only called on Windows)
- gh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)
Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Follow-up of gh-107219.
* Only close the connection writer on Windows.
* Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of
WSA_OPERATION_ABORTED.
(cherry picked from commit 0b4e090422)
gh-109047: concurrent.futures catches PythonFinalizationError (#109810)
concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.
Add test_python_finalization_error() to test_concurrent_futures.
concurrent.futures._ExecutorManagerThread changes:
* terminate_broken() no longer calls shutdown_workers() since the
call queue is no longer working anymore (read and write ends of
the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
to prevent race conditons with ProcessPoolExecutor.submit().
multiprocessing.Queue changes:
* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
leaking "dangling threads" even if the thread was not started
yet.
(cherry picked from commit 6351842121)
gh-107219: Fix concurrent.futures terminate_broken() (GH-109244)
Fix a race condition in concurrent.futures. When a process in the
process pool was terminated abruptly (while the future was running or
pending), close the connection write end. If the call queue is
blocked on sending bytes to a worker process, closing the connection
write end interrupts the send, so the queue can be closed.
Changes:
* _ExecutorManagerThread.terminate_broken() now closes
call_queue._writer.
* multiprocessing PipeConnection.close() now interrupts
WaitForMultipleObjects() in _send_bytes() by cancelling the
overlapped operation.
(cherry picked from commit a9b1f84790)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-94777: Fix deadlock in ProcessPoolExecutor (GH-94784)
Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe. See GH-94777 for more details.
(cherry picked from commit 6782fc0502)
Co-authored-by: Louis Paulot <55740424+lpaulot@users.noreply.github.com>
gh-102024: Reduced _idle_semaphore.release calls (GH-102025)
Reduced _idle_semaphore.release calls in concurrent.futures.thread._worker
_idle_semaphore.release() is now only called if only work_queue is empty.
---------
(cherry picked from commit 0242e9a57a)
Co-authored-by: Andrii Kuzmin <jack.cvr@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
This starts the process. Users who don't specify their own start method
and use the default on platforms where it is 'fork' will see a
DeprecationWarning upon multiprocessing.Pool() construction or upon
multiprocessing.Process.start() or concurrent.futures.ProcessPool use.
See the related issue and documentation within this change for details.
Do not spawn ProcessPool workers on demand when they spawn via fork.
This avoids potential deadlocks in the child processes due to forking from
a multithreaded process.
Prevent `max_tasks_per_child` use with a "fork" mp_context to avoid deadlocks.
Also defaults to "spawn" when no mp_context is supplied for safe convenience.
* bpo-20369: concurrent.futures.wait() now deduplicates futures given as arg.
* 📜🤖 Added by blurb_it.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().
Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.
Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.
Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.
Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.
As reported initially by @rad-pat in #6084, the following script causes a deadlock.
```
from concurrent.futures import ProcessPoolExecutor
class ObjectWithPickleError():
"""Triggers a RuntimeError when sending job to the workers"""
def __reduce__(self):
raise RuntimeError()
if __name__ == "__main__":
e = ProcessPoolExecutor()
f = e.submit(id, ObjectWithPickleError())
e.shutdown(wait=False)
f.result() # Deadlock on get
```
This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing.
https://bugs.python.org/issue39104
Automerge-Triggered-By: @pitrou
* Fixes issue 24882
* Add news file entry for change.
* Change test_concurrent_futures.ThreadPoolShutdownTest
Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.
If idle threads are being recycled properly, then we should have
less than three threads.
* Switched idle count to semaphor, Updated tests
As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.
Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).
Also added a new test to confirm the thread pool can be saturated.
* Updates tests as requested by pitrou.
* Correct minor whitespace error.
* Make test_saturation faster
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
Deprecated passing the following arguments as keyword arguments:
- "func" in functools.partialmethod(), weakref.finalize(),
profile.Profile.runcall(), cProfile.Profile.runcall(),
bdb.Bdb.runcall(), trace.Trace.runfunc() and
curses.wrapper().
- "function" in unittest.addModuleCleanup() and
unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
contextlib.AsyncExitStack.callback() and
contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in the create() method of multiprocessing.managers.Server
and multiprocessing.managers.SharedMemoryServer.
- "obj" in weakref.finalize().
Also allowed to pass arbitrary keyword arguments (even "self" and "func")
if the above arguments are passed as positional argument.
Miss Islington (bot)
