Tutorial

• Overview
  • Data Parallelism
  • Training Iterations
  • MPI
• Step 1: Communicators and Optimizers
  • Creating a Communicator
  • Creating a Multi-Node Optimizer
  • Run
  • Multi-node execution
  • Copying datasets
  • If you have trouble
  • Next Steps
• Step 2: Datasets and Evaluators
  • Scattering Datasets
  • Creating A Multi-Node Evaluator
  • Suppressing Unnecessary Extensions
• Tips and FAQs
  • Using MultiprocessIterator
  • Using Your Own Evaluator
  • Using MPI4py Communicator
  • Using FP16
  • MPI process hangs after an unhandled Python exception.