Local AI Hardware

Short answer: plan for model weights, the context's KV cache, runtime working buffers, and memory used by the OS and other applications. RAM, dedicated GPU VRAM, and unified memory are not interchangeable numbers. Quantization reduces weight storage, often with a quality tradeoff, but doesn't remove context costs or guarantee a particular speed.

Read model and runtime foundations and context budgeting. These estimates concern inference, not training.

A machining workshop

A workshop keeps reusable tooling in cabinets, brings it to a workbench, and may move a job onto a powered machine's bed. The cabinets are disk storage, the general bench is system RAM, and the machine bed is dedicated VRAM. The learned weights are the tooling needed for the job. A large cabinet doesn't create more working space on the machine.

The current job also needs fixtures and room for work in progress. That's the KV cache and runtime buffers. Running four jobs at once requires more space even when they share the same tooling. On a unified-memory machine, CPU and GPU share a physical pool, more like two operators using the same bench. They don't each get the full advertised capacity independently.

Which memory does what?

On Windows, Task Manager's Performance view separates Memory and GPU memory. On an NVIDIA system, nvidia-smi can report device memory and driver information. In Ollama, ollama ps shows loaded models and CPU/GPU placement. A displayed 100% GPU describes placement, not 100% GPU utilization.

First estimate: the weights

For a simplified dense model, raw weight bytes = parameter count × bits per parameter ÷ 8. Here is an illustrative 8-billion-parameter model, assuming every parameter uses the listed width:

One GB is 1,000,000,000 bytes; one GiB is 1,073,741,824 bytes. These are arithmetic estimates, not model download sizes or complete memory requirements. Quantization schemes store scales and other metadata, may keep selected tensors at higher precision, and can require extra runtime buffers. A Q4_K_M artifact isn't exactly four bits for every parameter.

Quantization replaces higher-precision values with a smaller representation and associated scaling. It is like bringing a more compact set of tooling, with some precision sacrificed. It can preserve useful quality for many tasks, but aggressive quantization can damage results. A smaller artifact is not necessarily faster on a backend without efficient support for that format.

GGUF is a model-file format used by llama.cpp-compatible runtimes, not a quantization level. FP16 and BF16 are different 16-bit floating-point formats. Model architecture, file format, numerical format, and inference backend must all be compatible. A mixture-of-experts model may activate only some parameters per token but still need storage for its other experts; don't size it from the active count alone.

Second estimate: context memory

For a conventional full-attention transformer with equally sized key and value heads, an approximate KV-cache size is: 2 × layers × tokens × KV heads × head dimension × bytes per element × concurrent sequences. The factor 2 accounts for keys and values.

Illustrative architecture, one sequence:
32 layers, 8 KV heads, head dimension 128
4,096 cached tokens, 2 bytes per cache element

2 x 32 x 4096 x 8 x 128 x 2 = 536,870,912 bytes
536,870,912 / 1,073,741,824 = 0.5 GiB

At 8,192 tokens: 1 GiB
Four such 8,192-token sequences: 4 GiB

These are hypothetical architecture values, not the specification of every 8B model. Grouped-query attention (GQA) can use fewer KV heads than query heads. Sliding-window attention, cache quantization, allocation strategy, and other architectures change the calculation. Weight quantization and cache quantization are separate settings.

Fitting is not the same as being fast

Prefill processes the input prompt; decode generates subsequent tokens. Loading a cold model adds another delay. A long prompt can make time to first token poor even if subsequent generation is fast. A single interactive decode stream often depends heavily on memory bandwidth, while batching and prefill change the balance of compute and memory work.

A CPU can run supported models without a dedicated GPU, but latency depends on model size, memory bandwidth, thread configuration, and instructions supported by the CPU. A compatible GPU can accelerate inference. Check the runtime's current support matrix for the exact GPU, operating system, and driver; a product label like "AI PC" or an NPU TOPS number doesn't establish support.

Some runtimes split work between CPU and GPU when the whole model doesn't fit in VRAM. The workshop can use both benches, but moving work and waiting for the slower station has a cost. Adding 8 GiB of VRAM to 16 GiB of RAM doesn't create a single 24 GiB pool with uniform speed. Two discrete GPUs aren't automatically one larger GPU either.

Choose conservatively, then measure

1. Choose a task and model: use an instruction-tuned text model for basic chat. Check its license and exact artifact, not just the family name.
2. Estimate memory: use weight size as a starting point, add context/cache and runtime overhead, and leave headroom. Start with one request and a short context.
3. Verify placement: inspect runtime logs and ollama ps while loaded. Check actual available RAM/VRAM under the intended workload.
4. Measure separately: record cold-load latency, warm-request latency, output rate, and peak memory. Keep input/output lengths and concurrency comparable.
5. Check quality: run the same answer-key cases after changing quantization or model size. Faster wrong answers are still wrong.

Ollama reports generation duration in nanoseconds. For a synthetic result of 120 output tokens in 6,000,000,000 ns, the decode rate is 120 / (6000000000 / 1000000000) = 20 tokens/s. That excludes loading and prompt processing, so it isn't the end-to-end request rate or time to first token. Record the runtime version, model ID, context, settings, hardware, and competing workload before comparing measurements.

When the job doesn't fit

Continue reading

Next: Run Your First Local Model. Revisit context budgeting before increasing limits; retrieval can reduce irrelevant input without making memory costs disappear.

Primary sources

• Transformers: cache strategies and memory tradeoffs
• llama.cpp: supported formats, quantization, and backends
• Ollama: hardware support
• Ollama: context allocation
• Ollama: timing and token-count fields