How Much VRAM Do You Need for LLM Inference?
Choose a GPU for LLM inference based on model size
How much VRAM do you need for LLM inference? Start with an example. An 8B model in BF16 with four simultaneous 4,096-token requests needs about 17 GiB for weights and cache. A 16 GiB card is already too small. A 24 GiB card is a candidate for testing because it leaves some room for vLLM.
What uses GPU memory
Weights are the model parameters loaded onto the GPU. Their footprint depends on the model size and storage format. An 8B model needs about 14.9 GiB for BF16 weights. With ideal 4-bit packing, the estimate drops to 3.7 GiB. Actual 4-bit model files typically take more space, so check the weight size of the exact model variant before renting.
The KV cache holds intermediate data for tokens the model has processed, so it can reuse that work as it generates an answer. Longer prompts and answers, or more requests at once, need more cache. The serving engine also needs memory. If the estimate is close to the card’s capacity, choose a configuration with more headroom or reduce the workload.
Read the table and choose a GPU
The table uses representative 8B, 32B and 70B models. Pick the model row, select the weight format and add the cache. The first number in each cache cell is for one request, and the second is for four simultaneous requests. Length includes the input and the answer. All numbers are in GiB and assume a BF16 cache. Runtime overhead is additional.
| Class | BF16 weights | 4-bit weights | KV 4,096 tokens 1 / 4 | KV 16,384 tokens 1 / 4 |
|---|---|---|---|---|
| 8B | 14.9 | 3.7 | 0.50 / 2.00 | 2.00 / 8.00 |
| 32B | 59.6 | 14.9 | 1.00 / 4.00 | 4.00 / 16.00 |
| 70B | 130.4 | 32.6 | 1.25 / 5.00 | 5.00 / 20.00 |
For example, a 4-bit 32B model with four short requests needs about 18.9 GiB in the table (14.9 + 4). At 16,384 tokens per request, that rises to roughly 30.9 GiB (14.9 + 16). A 24 GiB card is worth testing for the short workload. It is too small for the longer workload, where a 48 GiB card is a candidate with more headroom. In both cases, check the actual weight footprint and runtime overhead.
What to check before a longer rental
Before paying, decide which model variant you’ll serve, how long requests and answers can get, and how many requests may overlap at peak. On Qudata.ai, pick a GPU with enough VRAM and validate it in a short run. Check the available KV cache capacity in the server logs, then send long requests at your expected peak concurrency. If memory is tight, reduce context length or concurrency. To move to a different GPU on Qudata.ai, rent a suitable instance separately.
Recalculate for another model
For weights, multiply the parameter count by bytes per parameter and divide by 2³⁰. BF16 and FP16 use 2 bytes per parameter, and ideal 4-bit packing uses 0.5. For a full-attention model, estimate KV cache bytes as 2 × layer count × KV head count × head dimension × bytes per cache element × total tokens across active requests. The factor of two accounts for keys and values.
The table uses Llama 3.1 8B, Qwen2.5 32B and Llama 3.1 70B architectures, with 32, 64 and 80 layers respectively. Each has eight KV heads and a head dimension of 128. Weight quantization alone leaves the cache element size unchanged. If you separately quantize the KV cache in vLLM, use the new element size in the formula and check answer quality.