Skip to main content

Search

RC Learning Portal

RC Learning Portal

Home
Short Courses
Tutorials
People
Contact

Light Dark Automatic

Multi-GPU LLM Inference

GPU Memory Requirements

UVA HPC

GPUs on UVA HPC
GPU access on UVA HPC
UVA BasePOD
POD Access
Wait Time in the Queue
Viewing Available GPUs
GPU Dashboard
Calculating GPU Memory Requirements
Gemma 3 Memory Requirements
CPU Resource Allocation

Multi-GPU Strategies

Data Parallelism
Pipeline Parallelism (Inter-Layer)
Tensor Parallelism (Intra-Layer)
Key Tools and Frameworks

Accelerate

Advantages of Accelerate
Accelerate Code Demo

DeepSpeed

Transformer-specific Latency Advantages
Advantages of DeepSpeed

vLLM

Advantages of vLLM

Identifying Bottlenecks

Wrap Up

Accelerate Code Demo

The notebook is available on Rivanna/Afton in the folder /project/hpc_training/multigpu_llm_inf.

Previous

Advantages of Accelerate

Next

Last updated on Jan 1, 0001

© 2025 The Rector and Visitors of the University of Virginia

Cite