Identifying Bottlenecks

Accelerate and DeepSpeed offer methods for identifying GPU usage.

DeepSpeed’s tools are considerably more in-depth, so consider using it for real fine-grained monitoring of your code usage.