zoom_in Click to enlarge

Horovod

▲ ▼

language

description Horovod Overview

Horovod is an open-source distributed deep learning framework designed to scale training across multiple GPUs, machines, and even clusters. It provides a simple API that wraps around MPI (Message Passing Interface), NCCL, and Gloo backends. Horovod allows developers to take existing PyTorch or TensorFlow code and distribute it with minimal changes, making it highly effective for large-scale model training.

help Horovod FAQ

Does Horovod work with both TensorFlow and PyTorch?

Yes. Horovod supports TensorFlow, Keras, PyTorch, and Apache MXNet, and its installation extras can be selected for the frameworks a project uses.

What does Horovod's allreduce operation actually do?

Allreduce combines gradients calculated by multiple workers and distributes the result back to each worker. Horovod became known for an efficient ring-allreduce approach that lets existing single-GPU training code scale with relatively few changes.

Does Horovod require MPI?

MPI is a common Horovod controller and communication option, but Horovod can also use Gloo in supported configurations. GPU tensor operations are commonly handled through NVIDIA NCCL, while MPI or Gloo coordinates processes.

How does Horovod differ from PyTorch DistributedDataParallel?

Horovod offers one distributed-training interface across several frameworks and can integrate with environments such as Spark and Kubernetes. PyTorch DistributedDataParallel is built directly into PyTorch, so teams using only PyTorch often prefer it for tighter framework integration.

swap_horiz

Looking for Horovod alternatives? Compare top competitors ranked & scored

arrow_forward