GPU rental for running and fine-tuning LLMs
Open-weight language models removed the dependency on external APIs but handed the hardware question to the developer. Llama, Qwen, DeepSeek and Mistral need an amount of video memory dictated directly by model size and chosen precision — and they need it only while the model is actually working.
QuData gives access to cards of the right class for the duration of the task: H200 and H100 for large models in full precision, A100 and L40S for quantized versions and fine-tuning. Ollama and vLLM deploy from a ready-made image, and root access over SSH lets you assemble any environment of your own. Data is encrypted for the session and deleted once it ends — the very scenario local deployment is usually chosen for.
Ready-made images
Deploying an LLM from scratch means matching driver versions, building an inference engine and dealing with weight formats. Ready-made images start in about 30 seconds and hand you a working endpoint immediately.
Ollama
The fastest way to bring a model up. Weights load with a single command, the API is compatible — handy for prototypes and internal tools.vLLM
A production inference engine with high throughput. Handles dozens of parallel requests on a single card.text-generation-webui
A web interface for manual experiments with prompts, generation parameters and side-by-side model comparison.Fine-tuning environment
A prepared stack for LoRA and QLoRA with dataset handling and quantization libraries.Who it suits
AI product developers
Test several models on your own task and pick the best quality-to-inference-cost ratio without paying a subscription for every option.Companies that must keep data inside their perimeter
Run the model on a rented card, process sensitive data and end the session without sending anything to external services.System integrators
Spin up a demo stand for a specific client for the duration of a pilot and shut it down after the deal closes or falls through.Researchers
Fine-tune an open model on a narrow domain sample and compare it against the base version on your own metrics.How to pick a GPU for a language model
LLM video memory requirements are almost pure arithmetic, and that is the main difference from other workloads: you can tell in advance whether the model will fit on a card.
How much video memory an LLM needs
A baseline estimate for inference: multiply the number of parameters by the number of bytes per parameter. In fp16 that is two bytes, so a 7-billion-parameter model takes about 14 GB, 13 billion about 26 GB, and 70 billion over 140 GB. Add memory for the KV cache, which grows with context length and the number of concurrent requests — budget a 15–25% margin.
Quantization changes the picture radically. At 8 bits consumption halves, at 4 bits it drops fourfold. A 70-billion-parameter model in 4-bit quantization fits into 40–48 GB, that is a single L40S- or A100-class card. Quality degrades moderately, and for most applied tasks the difference is imperceptible.
Inference and fine-tuning have different demands
Fine-tuning is always more memory-hungry than serving. Full fine-tuning needs room for gradients and optimizer states — effectively several times more than the model itself. That is exactly why LoRA and QLoRA dominate in practice: they train a small set of additional weights on top of a frozen base model.
LoRA for a mid-sized model fits on a single 48–80 GB card. QLoRA, where the base model is stored in 4 bits, lowers the bar further. Full fine-tuning of large models is already a multi-GPU configuration and a separate budget conversation.
Throughput versus latency
A chat scenario cares about response time for a single request; batch processing cares about the total number of tokens processed. These are different optimizations. If you are pushing an array of documents through the model, an engine like vLLM with batching will deliver several times more output from the same card than a naive sequential run.
Measure cost not in rental hours but in cost per thousand processed tokens — only then does the comparison with an external API become fair.
When local deployment is justified
Three situations where your own model on a rented card beats an external API: the data cannot leave your perimeter; the processing volume is so large that per-request pricing stops adding up; you need a model fine-tuned for a narrow domain. In every other case, compare the total cost honestly — sometimes the API turns out cheaper.
Frequently asked questions
Which GPU do I need to run an LLM?
Go by the model size in video memory: models up to 13B parameters run on a single 24 GB card; for 70B in full precision you need an H100 or H200, or a multi-GPU configuration.
How much video memory does Llama 70B need?
In 4-bit quantization the model fits into roughly 40–48 GB, in 8-bit around 70–80 GB, and in fp16 it requires over 140 GB — that is an H200 or several cards.
Are there ready-made images with Ollama and vLLM?
Yes, both are available in the list of ready-made solutions and deploy in about 30 seconds without manual dependency installation.
Can I fine-tune a model on my own data?
Yes. LoRA and QLoRA need a single 48–80 GB card; full fine-tuning of large models requires a multi-GPU configuration.
Will my data stay confidential?
Data is encrypted for the session and deleted once it ends. This is the main scenario for companies that cannot send data to external APIs.