API Reference

Communicators

chainermn.create_communicator(communicator_name='hierarchical', mpi_comm=None)

Create a ChainerMN communicator.

Different communicators provide different approaches of communication, so they have different performance charasteristics. The default communicator hierarchical is expected to generally perform well on a variety of environments, so one need not to change communicators in most cases. However, choosing proper communicator may give better performance. The following communicators are available.

Name CPU GPU NCCL Recommended Use Cases
naive OK OK   Testing on CPU mode
hierarchical   OK Required Each node has a single NIC or HCA
two_dimensional   OK Required Each node has multiple NICs or HCAs
single_node   OK Required Single node with multiple GPUs
flat   OK   N/A
Parameters:
  • communicator_name – The name of communicator (naive, flat, hierarchical, two_dimensional, or single_node)
  • mpi_comm – MPI4py communicator
Returns:

ChainerMN communicator

Optimizers and Evaluators

chainermn.create_multi_node_optimizer(actual_optimizer, communicator)

Create a multi node optimizer from a Chainer optimizer.

Parameters:
  • actual_optimizer – Chainer optimizer (e.g., chainer.optimizers.Adam).
  • communicator – ChainerMN communicator.
Returns:

The multi node optimizer based on actual_optimizer.

chainermn.create_multi_node_evaluator(actual_evaluator, communicator)

Create a multi node evaluator from a normal evaluator.

Parameters:
  • actual_evaluator – evaluator (e.g., chainer.training.extensions.Evaluator)
  • communicator – ChainerMN communicator
Returns:

The multi node evaluator based on actual_evaluator.

Dataset Utilities

chainermn.scatter_dataset(dataset, comm)

Scatter the given dataset to the workers in the communicator.

The dataset of worker 0 (i.e., the worker whose comm.rank is 0) is scattered to all workers. The given dataset of other workers are ignored. The dataset is split to sub datasets of almost equal sizes and scattered to workers. To create a sub dataset, chainer.datasets.SubDataset is used.

Parameters:
  • dataset – A dataset (e.g., list, numpy.ndarray, chainer.datasets.TupleDataset, ...).
  • comm – ChainerMN communicator or MPI4py communicator.
Returns:

Scattered dataset.