Accelerate

Accelerate is a popular, easy-to-use library built to work with PyTorch models. It provides a very straightforward API that wraps with PyTorch and transformers code.

Accelerate can automatically allocate GPU memory usage across nodes, and is used for both training and inference. It is built on top of Megatron and can interface with DeepSpeed.