mpi4py.util.pool

New in version 4.0.0.

See also

This module intends to be a drop-in replacement for the multiprocessing.pool interface from the Python standard library. The Pool class exposed here is implemented as a thin wrapper around MPIPoolExecutor.

Note

The mpi4py.futures package offers a higher level interface for asynchronously pushing tasks to MPI worker process, allowing for a clear separation between submitting tasks and waiting for the results.

class mpi4py.util.pool.Pool

Pool using MPI processes as workers.

__init__(processes=None, initializer=None, initargs=(), **kwargs)

Initialize a new Pool instance.

Parameters:
  • processes – Number of worker processes.

  • initializer – An callable used to initialize workers processes.

  • initargs – A tuple of arguments to pass to the initializer.

Note

Additional keyword arguments are passed down to the MPIPoolExecutor constructor.

Warning

The maxtasksperchild and context arguments of multiprocessing.pool.Pool are not supported. Specifying maxtasksperchild or context with a value other than None will issue a warning of category UserWarning.

apply(func, args=(), kwds={})

Call func with arguments args and keyword arguments kwds.

Equivalent to func(*args, **kwds).

apply_async(func, args=(), kwds={}, callback=None, error_callback=None)

Asynchronous version of apply() returning ApplyResult.

map(func, iterable, chunksize=None)

Apply func to each element in iterable.

Equivalent to list(map(func, iterable)).

Block until all results are ready and return them in a list.

The iterable is choped into a number of chunks which are submitted as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

Consider using imap() or imap_unordered() with explicit chunksize for better efficiency.

map_async(func, iterable, chunksize=None, callback=None, error_callback=None)

Asynchronous version of map() returning MapResult.

imap(func, iterable, chunksize=1)

Like map() but return an iterator.

Equivalent to map(func, iterable).

imap_unordered(func, iterable, chunksize=1)

Like imap() but ordering of results is arbitrary.

starmap(func, iterable, chunksize=None)

Apply func to each argument tuple in iterable.

Equivalent to list(itertools.starmap(func, iterable)).

Block until all results are ready and return them in a list.

The iterable is choped into a number of chunks which are submitted as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

Consider using istarmap() or istarmap_unordered() with explicit chunksize for better efficiency.

starmap_async(func, iterable, chunksize=None, callback=None, error_callback=None)

Asynchronous version of starmap() returning MapResult.

istarmap(func, iterable, chunksize=1)

Like starmap() but return an iterator.

Equivalent to itertools.starmap(func, iterable).

istarmap_unordered(func, iterable, chunksize=1)

Like istarmap() but ordering of results is arbitrary.

close()

Prevent any more tasks from being submitted to the pool.

terminate()

Stop the worker processes without completing pending tasks.

join()

Wait for the worker processes to exit.

class mpi4py.util.pool.ThreadPool

Bases: Pool

Pool using threads as workers.

class mpi4py.util.pool.AsyncResult

Asynchronous result.

get(timeout=None)

Return the result when it arrives.

If timeout is not None and the result does not arrive within timeout seconds then raise TimeoutError.

If the remote call raised an exception then that exception will be reraised.

wait(timeout=None)

Wait until the result is available or timeout seconds pass.

ready()

Return whether the call has completed.

successful()

Return whether the call completed without raising an exception.

If the result is not ready then raise ValueError.

class mpi4py.util.pool.ApplyResult

Bases: AsyncResult

Result type of apply_async().

class mpi4py.util.pool.MapResult

Bases: AsyncResult

Result type of map_async() and starmap_async().