Skip to main content

Search

Type to search the site. Results will appear below as you type.

Search

RC Learning Portal

RC Learning Portal

Home
Programming Courses
Tutorials
People
Contact

Light Dark Automatic

Multi-GPU LLM Inference

GPU Memory Requirements

UVA HPC

GPUs on UVA HPC GPU access on UVA HPC UVA BasePOD POD Access Wait Time in the Queue Viewing Available GPUs GPU Dashboard Calculating GPU Memory Requirements Gemma 3 Memory Requirements CPU Resource Allocation

Multi-GPU Strategies

Data Parallelism Pipeline Parallelism (Inter-Layer) Tensor Parallelism (Intra-Layer) Key Tools and Frameworks

Accelerate

Advantages of Accelerate Accelerate Code Demo

DeepSpeed

Transformer-specific Latency Advantages Advantages of DeepSpeed

vLLM

Advantages of vLLM

Identifying Bottlenecks

Wrap Up

Accelerate Code Demo

The notebook is available on Rivanna/Afton in the folder /project/hpc_training/multigpu_llm_inf.

Previous

Advantages of Accelerate

Next

Last updated on Jul 8, 2025

© 2026 The Rector and Visitors of the University of Virginia

Cite