Private AI inference means running language models on hardware you control, usually because the data can't leave your environment or because a steady workload has made cloud GPU pricing painful. The sizing is more tractable than the discourse suggests, because inference is bounded by two knowable numbers: the memory footprint of the model you serve, and how many requests hit it at once.
The memory math
A model's weights occupy roughly its parameter count multiplied by the bytes per parameter at your chosen precision. That arithmetic sizes the GPU before any benchmark does:
| Model size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 8B parameters | ≈16 GB | ≈8 GB | ≈4 GB |
| 70B parameters | ≈140 GB | ≈70 GB | ≈35 GB |
On top of the weights, the serving runtime needs working memory for the KV cache, which grows with context length and concurrent requests; leaving meaningful headroom above the weights is the rule, not the exception. The practical consequences: an 8B-class model at reduced precision fits on a single mainstream datacenter GPU with room to serve; a 70B-class model at full precision needs multiple GPUs working together. Quantization is the lever that moves a model down a hardware class, at some cost in quality that your own evaluation should judge.
What nobody budgets for
The GPU is the headline, but the host around it does real work. Retrieval-augmented generation, which is what most enterprise AI actually is, needs fast storage for the document index, memory for embedding and retrieval, and CPU for everything that isn't the model. A serious inference host is a balanced server that happens to carry GPUs, which is why ours start from a proper platform build, not a GPU strapped to whatever was free.
The second unbudgeted item is operations. An inference platform is a production system: model updates, runtime patching, monitoring, and capacity decisions when the pilot succeeds and usage triples. Decide who runs it before it arrives.
When the cloud is still right
Bursty experimentation, models you swap weekly, and loads that spike rarely are what cloud GPUs are for, and renting is the honest recommendation in those cases. Owned inference hardware wins when the load is steady, the data is sensitive, or the monthly cloud bill has crossed what the hardware would cost in a year. Run that comparison with your real numbers; if you send them with a quote request, we'll run it with you and show the work.