AI infrastructure terms, decoded

Quick reference

• TTFT — Time to first token: how long the user stares at nothing before the answer starts. It covers the queue and the prefill pass.
• tokens/s — Decode throughput: how fast tokens arrive after the first one. Batching raises the total while a single request slows down.
• batching — Packing several requests into one forward pass. Static batching waits for a full batch; continuous swaps requests every step.
• KV cache — The attention keys and values kept from earlier tokens so they are not recomputed. It buys speed and costs memory, one per request.
• quantisation — Weights stored in fewer bits: FP16 down to INT8 or FP8. Less memory, faster maths, and an accuracy loss to measure.
• MIG — Multi-Instance GPU: one card split into isolated slices, each with its own memory and compute. For sharing, not for one big job.

Explained

TTFT

Time to first token is how long the user stares at nothing before the answer starts. It covers the queue plus prefill, the pass over the prompt, so it grows with prompt length.

tokens/s

Decode throughput is how fast tokens arrive after the first one. It is measured per request or across the batch, and the two move in opposite directions: more requests raise the total while one request slows down.

Batching

Packing several requests into one forward pass. Static batching waits for a full batch; continuous batching adds and removes requests at every step, which is why serving engines hold a queue.

KV cache

The attention keys and values kept from earlier tokens so they are not recomputed. It buys speed and costs memory, one cache per concurrent request, which is the real ceiling on users per card.

Quantisation

Storing weights in fewer bits, from FP16 down to INT8 or FP8. Less memory and faster matrix maths, at the risk of a small accuracy loss you measure rather than assume.

MIG

Multi-Instance GPU splits one card into isolated slices, each with its own memory and compute. Good for sharing a card, not for making one large job fit.

In one line

Watch one card

$nvidia-smi dmon -s mu -d 2