Llama 3 70B at FP16 weighs 140 gigabytes and one GPU holds 80 — the model does not fit, and that mismatch is the entire interview question. Here is the memory math, the Megatron split, the replicate-it-instead alternative, and how requests get routed and batched across a full node.
Imagine eight friends trying to move one enormous pizza that will not fit through any single door. Option one: cut the pizza into eight slices, hand each friend a slice, and have them walk in step so the pizza arrives whole — that is splitting the model. Option two: if the pizza is small enough, bake eight whole pizzas, hand one to each friend, and let everyone carry their own — that is copying the model.
A language model is a pile of numbers — its weights — and serving it means loading those numbers onto GPUs and answering requests. The whole game is arithmetic: do the numbers fit on one card? If yes, copy the model to every card and split the traffic. If no, slice the numbers across cards and make the cards cooperate on every single request. A serving system is the person at the door making that call, then handing out delivery orders so no friend ever stands idle. Everything in this guide is that decision, worked out.
The single-GPU serving question is a warm-up. The real interview question hands you a node — eight GPUs, one model, thousands of requests — and asks you to keep all eight cards fed.
Models outgrew cards years ago, and interviewers at the frontier labs know it. When they ask about inference at scale, they are testing whether you think in clusters: can you look at a parameter count, do the memory math on the whiteboard, and name the layout before you touch anything else? The answer has two axes. If the model does not fit one GPU, you split it — tensor parallelism — and the cards cooperate on every request. If it fits, you copy it — data parallelism — and the cards split the traffic. Then, on top of whichever layout you chose, you need the serving layer: a router that sends each request to the right group of cards, and a scheduler that batches requests inside each group so the GPUs never idle.
Notice the order of operations. Memory math first, layout second, routing and batching third. Candidates who jump straight to “vLLM with tensor parallel 8” without showing the arithmetic are telling the interviewer they memorized a flag instead of understanding the constraint. The math is the interview.
Too big to fit — split with tensor parallelism and pay the all-reduce tax. Small enough — replicate for throughput. Route by load, batch inside the group.
Every layout decision starts from one formula: parameters × bytes per parameter. At FP16 each parameter costs 2 bytes, so Llama 3’s 70 billion parameters weigh about 140 GB. A flagship GPU holds 80 GB. The model does not fit — you cannot just load it and serve. That mismatch is the entire problem, and saying it out loud in the first thirty seconds is what separates a strong answer from a rambling one.
Two refinements before you pick a layout. First, weights are not the whole story: the KV cache, activations, and CUDA context all want memory too, so treat roughly 70–75% of a card as usable for weights. Second, precision is a dial: FP8 halves the bill to 1 byte per parameter, which is exactly how 400B-class models get served on 80 GB cards at all. The worked numbers are in the next section — the habit to build is doing this math before naming any framework.
The classic solution comes from Megatron-LM (Shoeybi et al., 2019): instead of putting whole layers on different GPUs, split inside each layer’s weight matrices. Column parallelism slices a matrix along its output dimension — each GPU computes a slice of the outputs, and the nonlinearity applies locally with no communication at all. Row parallelism slices along the input dimension — each GPU computes a partial sum, and then an all-reduce adds the partial sums together across the cards.
The elegant bit is the pairing: feed a column-parallel layer into a row-parallel layer, and each attention block needs exactly one all-reduce, each MLP block exactly one — two per transformer layer, the minimal communication the split allows. Self-attention splits the same way: the Q/K/V projections go column-parallel across heads, the output projection goes row-parallel, one all-reduce to combine. One constraint to name in the interview: the attention head count must divide evenly by the tensor-parallel degree, or the heads do not split cleanly.
Left: a 70B-class model sharded TP=8 — all eight cards collaborate on every request. Right: a 7B-class model replicated DP=8 — each card is an independent server.
Those all-reduces run on the critical path of every forward pass — after every layer, the cards stop and sync. The traffic moves over NVLink inside a node or InfiniBand between nodes, and the collective’s volume scales with batch size × sequence length × hidden dimension, not with the weights. That is why tensor parallelism is communication-hungry: make the batch big and the sync dominates; make the link slow and the whole thing chokes.
The interview rule to state plainly: keep the tensor-parallel group inside one node. TP=8 means one eight-GPU node, period. Spreading a TP group across nodes puts the all-reduce on a slower fabric and the per-token latency falls off a cliff. If the model needs more than eight cards’ worth of memory, you do not stretch TP across nodes — you mix in data parallelism or pipeline parallelism instead.
Now the other axis. A 7B model at FP16 is 14 GB — it fits on one card with room to spare. Splitting it would be pure overhead: all the all-reduce tax with no memory problem to solve. So you never split at all. Run one full copy per GPU, and every card serves its own requests independently. Throughput scales roughly eight times on eight cards, and there is zero communication in the forward pass.
This is the answer most candidates forget to give, and interviewers notice. Data parallelism for inference is embarrassingly parallel — the only shared piece is the router in front deciding which copy gets each request. When the model fits, replication beats sharding on simplicity, on latency (no sync per layer), and on fault tolerance (a dead card takes out one eighth of capacity, not the whole model).
Put the two axes together and the decision is clean. A tensor-parallel group is one big fast lane: all eight cards collaborate on each request, so per-request latency is low — but total throughput is bounded by that one lane. Data-parallel replicas are many lanes: each card works alone, so total throughput is high — but no single request goes faster. Tensor parallelism buys fit and latency; data parallelism buys throughput.
Production answers mix them. A 70B model at TP=2 (70 GB per card — tight) or TP=4 (35 GB per card — comfortable) leaves cards free to replicate: TP=2 × DP=4 gives four fast lanes, each handling its own traffic. TP=4 × DP=2 gives two even faster lanes. The knob you turn is the workload: latency-sensitive chatbot, lean toward tensor parallelism; batch API that cares about tokens per second per dollar, lean toward data parallelism. Say that sentence and you have answered the question underneath the question.
Fit the weights first (tensor parallelism, smallest degree that fits), then spend the remaining cards on throughput (data parallelism). Never pay the all-reduce tax for a model that fits on one card.
The layout answers “where does the model live”. Serving still needs “where does each request go”. Requests hit a router, which picks a replica group: least-loaded for raw throughput, consistent hashing on a session key when you want the same user landing on the same group, or sticky-by-session so repeated prompts reuse the group’s prefix cache. Inside each group, requests get continuously batched — new prompts join mid-stream at every decode step instead of waiting for the slowest request, which is the scheduling trick from the continuous batching guide.
The layering matters: route to a group, batch inside the group. The router thinks in groups and load; the scheduler inside each group thinks in tokens and freed slots. Mixing the two — one global batch across all eight cards — couples every request’s latency to the slowest group and throws away the isolation that data parallelism bought you.
The column/row split is a 2019 paper, the engine flags are one CLI argument, and the managed API is one config field — the same two axes at three levels of abstraction.
The paper that named the split. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism” (Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro — NVIDIA, arXiv 1909.08053, 2019) showed how to partition a transformer’s matrix multiplications across GPUs with minimal communication: column-parallel input, row-parallel output, one all-reduce per attention block and one per MLP block. Every serving engine’s tensor-parallel implementation is this layout, repurposed from training to inference.
The engine that productized it. vLLM exposes the two axes as two flags: --tensor-parallel-size shards one model copy across the cards (the split), --data-parallel-size runs one full copy per card (the replication). Underneath, it pairs the layout with continuous batching inside each group, so the cards stay fed — the router-and-batcher layering from the mechanism section, as an implementation rather than a diagram.
The managed version. The Inference Endpoints API exposes the same decision as config: set tensorParallelSize to shard one model copy across the instance’s accelerators, or dataParallelSize (on the vLLM engine) to run one copy per accelerator — the API even validates the parallelism config against the instance’s accelerator count. The canonical example is an 8×H200 instance serving a 120B model with tensor-parallel size 8: the memory math from this guide’s first section, answered by one config field.
These are the projects’ and papers’ own documented designs and reported figures — the wording here is ours, the ideas are credited above.
The pattern is always the same: fit the weights first, then spend the remaining hardware on throughput. A paper gave us the minimal-communication split, an engine turned it into flags, and a managed API turned the flags into config — but the whiteboard math underneath never changed. When your interviewer says “eight GPUs,” reach for the arithmetic before the API.
Three models, 80 GB cards, ~60 GB usable per card once you reserve headroom for the KV cache and kernels. FP16 unless noted.
Same node, three models, three layouts — the arithmetic picks before you do.
A memory-fit planner that picks the layout from the parameter count, and a least-loaded router with sticky sessions — the two pieces of code behind almost every “how would you serve it” follow-up.
def fit_plan(params_b, bytes_per=2, gpu_gb=80, headroom=0.75, cards=8):
# weights first, then spend the remaining cards on throughput
weight_gb = params_b * bytes_per # FP16 -> 2 bytes/param
usable = gpu_gb * headroom # reserve room for KV cache + kernels
if weight_gb <= usable:
return {"mode": "data-parallel", "tp": 1, "dp": cards}
tp = 1
while weight_gb / tp > usable:
tp *= 2 # TP degrees are powers of two
return {"mode": "tensor-parallel", "tp": tp,
"dp": cards // tp, "gb_per_gpu": round(weight_gb / tp, 1)}
fit_plan(7) # 14 GB -> data-parallel, 8 copies
fit_plan(70) # 140 GB -> TP=2 (70 GB/card, tight) or TP=4
fit_plan(405) # 810 GB -> TP=8 is 101 GB/card: quantize to FP8 firstimport hashlib
from bisect import bisect
class Router:
# route to a replica group: sticky sessions stay put,
# everyone else goes to the least-loaded group
def __init__(self, groups, ring_replicas=200):
self.groups = groups
self.inflight = {g: 0 for g in groups}
self.ring = {}
for g in groups:
for i in range(ring_replicas):
h = int(hashlib.md5(f"{g}#{i}".encode()).hexdigest(), 16)
self.ring[h] = g
self.keys = sorted(self.ring)
def route(self, session_id=None):
if session_id: # sticky: same session, same group (prefix-cache reuse)
h = int(hashlib.md5(session_id.encode()).hexdigest(), 16)
return self.ring[self.keys[bisect(self.keys, h) % len(self.keys)]]
g = min(self.inflight, key=self.inflight.get) # least-loaded
self.inflight[g] += 1
return g
def release(self, group):
self.inflight[group] -= 1The planner encodes the decision rule from the mechanism section — weights first, throughput second — and the router encodes the layering: groups are the unit of routing, batching happens inside each group. In production the router also watches queue depth and per-group latency, but the skeleton above is what the interviewer wants to see.
What to say (≈90 sec): “First the memory math: 70B parameters at FP16 is 140 GB, one card holds 80, so it doesn’t fit — that forces tensor parallelism. I’d shard it TP=4, 35 GB per card, which leaves comfortable headroom for the KV cache, and run DP=2 across the two groups — two fast lanes splitting the traffic. The split is Megatron-style: column-parallel then row-parallel per block, one all-reduce per attention block and one per MLP block, all inside one node on NVLink — never across nodes. Requests hit a router that picks a group by least-loaded, or sticky-by-session when I want prefix-cache reuse, and inside each group requests are continuously batched so new prompts join every decode step. If the workload is a latency-sensitive chatbot I’d lean the split toward TP=8; for a batch API I’d lean toward more data-parallel replicas.”
Likely follow-up: “Why TP=4 and not TP=2?” → TP=2 is 70 GB per card against ~60 GB of comfortable headroom — one long-context burst OOMs you. TP=4 buys the headroom and still leaves two data-parallel lanes.
The answer that sinks you: “I’d set tensor-parallel-size to 8.” Why it fails: it skips the arithmetic entirely — 70B only needs TP=2 to fit, so TP=8 pays the all-reduce tax on eight cards for no memory reason and throws away the data-parallel throughput.
What to say (≈90 sec): “It’s the latency/throughput tradeoff. TP=8 is one big fast lane: all eight cards collaborate on every request, so per-request latency is minimal — but the whole node serves one request stream, and every layer pays the all-reduce. TP=2 × DP=4 is four lanes: each pair serves its own traffic, so total tokens per second is far higher, at the cost of higher per-request latency. Pick TP=8 for an interactive chatbot where time-to-first-token is the SLO; pick TP=2 × DP=4 for a batch or async API where throughput per dollar wins. And the memory math constrains you first — you can only choose among layouts where the TP degree actually fits the weights.”
Likely follow-up: “What breaks if you push TP=8 with a huge batch?” → All-reduce volume scales with batch × sequence × hidden, so the sync dominates and per-token latency regresses — TP wants modest batches on a fast fabric.
The answer that sinks you: “TP=8 is always better — more parallelism.” Why it fails: more parallelism of the wrong kind — TP buys latency, not throughput, and you just spent all eight cards on one lane.
What to say (≈90 sec): “Because the all-reduce is on the critical path of every forward pass — two per layer, every layer, every token. Inside a node that traffic rides NVLink; across nodes it drops to InfiniBand or worse, and the sync time balloons. The cards spend more time waiting on each other than computing, and per-token latency falls off a cliff. The rule is: TP groups stay intra-node. If the model needs more memory than one node, you add pipeline parallelism or data parallelism across nodes — collectives that are rare or point-to-point — never a wider TP group.”
Likely follow-up: “So how would you serve a model that needs 16 cards of memory?” → TP=8 inside each node, pipeline-parallel across the two nodes (one point-to-point handoff per stage boundary), or data-parallel replicas per node.
The answer that sinks you: “InfiniBand is fast enough.” Why it fails: hand-waving past the actual constraint — the all-reduce happens ~160 times per forward pass for an 80-layer model, and “fast enough” is not a number.
What to say (≈90 sec): “7B at FP16 is 14 GB — it fits on one card five times over, so I’d never split it. Data-parallel: one full copy per GPU, eight independent servers behind a least-loaded router, throughput scales roughly 8×. No all-reduce anywhere, no NVLink dependency, and a dead card takes out one eighth of capacity instead of the whole model. Inside each copy, continuous batching keeps the card fed. The only subtlety is the router: least-loaded for raw throughput, sticky-by-session if I want prefix-cache hits on repeated prompts.”
Likely follow-up: “When would you tensor-parallelize a 7B model anyway?” → Only for latency: if a single request’s time-to-first-token must beat what one card can do — rare, and you’d say so explicitly rather than defaulting to it.
The answer that sinks you: “TP=8, same as the 70B setup.” Why it fails: it applies the tool without the problem — all the communication tax, none of the memory need.
What to say (≈90 sec): “The all-reduce volume scales with batch × sequence length × hidden dimension — it’s activation traffic, not weights. Double the batch and every one of the ~160 all-reduces per forward pass moves twice the bytes, while the compute per card barely changes because decode is memory-bound anyway. The sync starts dominating the step: the cards wait on communication instead of computing. Fixes: cap the batch (TP wants modest batches), check you’re actually on NVLink and not falling back to a slower fabric, or restructure to TP=4 × DP=2 so each all-reduce spans four cards and the batch splits across two independent lanes.”
Likely follow-up: “How would you confirm it’s the all-reduce and not the kernels?” → Profile with NCCL timing or Nsight: if the step time tracks the collective time as batch grows, it’s communication; if kernel time grows, it’s compute.
The answer that sinks you: “Bigger batches are always faster — add more GPUs.” Why it fails: it misdiagnoses a communication bottleneck as a capacity problem — more cards just add more all-reduce participants.
What to say (≈90 sec): “Least-loaded routing maximizes throughput but scatters a user’s requests across groups — every group recomputes the shared prompt prefix from scratch, wasting the prefill each time. Sticky-by-session routing sends the same session to the same group, so the group’s KV cache already holds the conversation prefix and only the new tokens need prefilling. The tradeoff is utilization versus locality: sticky routing can strand capacity when sessions are uneven, while least-loaded keeps every group busy but repeats work. In practice: sticky for chat workloads with long shared contexts, least-loaded for stateless single-shot traffic — and consistent hashing for the sticky mapping so adding a group doesn’t reshuffle every session.”
Likely follow-up: “What breaks when a group dies under sticky routing?” → Its sessions rehash to surviving groups and lose their prefix caches — a cold-start burst. Mitigate with bounded retries and cache-aware re-admission.
The answer that sinks you: “The router doesn’t matter — the scheduler handles everything.” Why it fails: it collapses two layers — the router picks the group, the scheduler batches inside it, and prefix locality lives or dies at the router.