Design the Post-Training Pipeline for a 70B Model
From raw pretrained weights to a helpful, harmless assistant: the data, training stages, and eval gates that turn a 70-billion-parameter base model into something you’d actually ship.
Explain it like I’m five
Pretraining gives the model the whole library: it can finish any sentence, but it rambles, invents facts, and doesn’t know what a good answer even looks like. Post-training is finishing school — you show it examples of great answers, let teachers pick the better of two answers until it learns taste, give it homework it can check itself (math, code), and make it pass a final exam before graduation.
The interview version: “design the pipeline that runs that school at 70-billion-parameter scale.” They want the stages in order, the data each stage needs, how much compute it costs, and how you catch each stage quietly breaking the last one.
Intuition: a pipeline with feedback loops, not one training run
Nobody asks “what is SFT?” and stops. The interviewer wants to hear you reason about a system: which stage runs first and why, what data each stage consumes, what each stage costs at 70B scale, and — the part that separates senior answers — what the eval harness between stages measures so you know the pipeline is working and not just running.
The four stages, in order: (1) supervised fine-tuning teaches the instruction-following format; (2) preference tuning teaches taste — which answer a human would pick; (3) reasoning RL teaches verifiable skills like math and code, where the reward can be checked rather than judged; (4) evals and red-teaming gate every stage, because each stage can silently undo the last one. That last clause is the whole game: post-training stages interfere with each other, so the pipeline is really train → measure → fix the data → train again, repeated until the gates pass.
The arrows pointing back up are the point. Labs describe running post-training in repeated rounds, adding data where the evals hurt, not as a single left-to-right pass.
At 7B you can full-fine-tune on a single node and shrug at a bad data batch. At 70B, bf16 weights alone are ~140 GB — full fine-tuning with Adam needs on the order of a terabyte of GPU memory, sharded across many H100s. Every training run costs real money, so the pipeline design tilts toward data quality over data quantity, parameter-efficient methods (LoRA) where they suffice, and evals that catch a bad stage before you burn the next one.
The four stages, and what each one consumes
1. Supervised fine-tuning — teach the format
Start from the pretrained 70B and train on prompt→response demonstrations with ordinary next-token loss, computed only on the response tokens. This is where the model learns “a user asks, an assistant answers helpfully in this shape.” Data comes from human-written demonstrations, model-generated responses filtered by quality, and targeted synthetic data for weak skills. The design decisions that matter: mix (what fraction is coding, math, chat, tool use, safety refusals), dedup (near-duplicate prompts teach memorization, not skill), and length distribution (too many long answers and the model never learns to be concise).
2. Preference tuning — teach the taste
SFT teaches format; it does not teach which of two good-format answers a human prefers. Collect pairwise comparisons — same prompt, two sampled responses, a reviewer picks the better one under a rubric — then optimize. Two routes: RLHF trains a reward model on the comparisons and optimizes the policy against it (PPO, with a KL penalty against the SFT model), or DPO optimizes the preference objective directly from the pairs, no reward model, no rollouts. Operationally DPO is simpler and more stable; RLHF keeps the online rollout signal for rewards that only appear through acting. Either way, the failure mode is reward hacking: the policy learns the reviewer’s quirks (verbosity, confident tone, format tricks) instead of quality — which is why the KL constraint and a holdout preference set belong in the design, not as afterthoughts.
3. Reasoning RL — teach verifiable skills
The newer stage. For tasks with checkable answers — math proofs, code that must pass tests — you don’t need human taste at all: the reward is whether the answer verifies. The policy samples many reasoning traces, keeps the ones that verify, and learns. This is the machinery behind long chain-of-thought behavior, and it runs best after the model already follows instructions and has decent taste, so the RL budget goes into reasoning rather than relearning format. Watch the interference: reasoning RL can degrade chat quality and instruction-following if the verifiable task mix dominates — the classic “alignment tax” reappearing between stages.
4. Evals and red-teaming — the gates
Every stage is gated by the same harness: capability benchmarks (does it still know things?), preference win-rate against the previous checkpoint (are humans actually picking it more?), a safety red-team suite (jailbreaks, disallowed content, sycophancy probes), and a regression set of product-critical prompts. A stage ships only when the gates pass — and when they don’t, you fix the data (add the failing slice to the next round) rather than just tuning hyperparameters. Modern recipes run each stage in multiple rounds for exactly this reason.
SFT first (format), preference tuning second (taste), reasoning RL third (verifiable skill). Run preference tuning on a base model that can’t follow instructions and the reviewers’ comparisons are noise. Run heavy reasoning RL before SFT and you burn your most expensive compute teaching the model things the cheaper stages would have covered.
Llama 3: a published post-training recipe
Meta’s Llama 3 report describes post-training as repeated rounds of supervised fine-tuning and preference optimization — six rounds of SFT followed by DPO, with rejection sampling and a final PPO stage on top — driven by model-generated data that humans annotated and filtered.
The pattern worth stealing for your interview answer: they did not collect one giant dataset and train once. Each round targeted the failures the evals surfaced — more long-context data here, more tool-use data there — and the preference data mixed human annotations with model-generated comparisons that passed quality filters. The report also describes using the reward model itself to select the best of several sampled responses (rejection sampling), so the SFT data for the next round is already preference-filtered. That is the feedback loop from the diagram, running in production.
SFT first to lock instruction-following, then DPO rounds to shape preferences, then a PPO stage. Each round’s data is chosen from the previous round’s eval failures — the pipeline is failure-driven, not schedule-driven.
Generate candidate responses with the current model, have humans rank them, filter aggressively, and feed the winners back as the next round’s training data. The model improves the data that trains its successor.
Capability, preference, and safety evals run after every round. A round that improves preference win-rate but regresses a capability benchmark gets its data mix rebalanced, not shipped.
The same three-stage skeleton shows up in the original InstructGPT recipe (SFT → reward model → PPO) and in DeepSeek-R1’s RL-first variant, where verifiable-reward RL does the heavy lifting for reasoning. The stages are the industry consensus; the mix is the secret sauce.
Source: “The Llama 3 Herd of Models” (Meta), arXiv:2407.21783 ↗Source: “Training language models to follow instructions with human feedback” (InstructGPT), arXiv:2203.02155 ↗Source: “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL”, arXiv:2501.12948 ↗The published recipes describe the architecture of the pipeline, not the data mix ratios, the reviewer rubrics, or the gate thresholds — those are the parts labs guard. In an interview, naming what you can’t know from the papers (and how you’d discover it empirically: ablate the mix, sweep the ratio, measure the gates) reads as senior judgment, not a gap.
Budget the pipeline: three back-of-envelope numbers
Interviewers love asking what a stage costs. These are rough, clearly labeled estimates — the skill is the method, not the digits.
The punchline for the interview: at 70B scale, compute is priced per run and human data is priced per hour, so the pipeline design is an economic argument — LoRA vs full fine-tuning is a memory-budget call, DPO vs PPO is partly an infra-complexity call, and every eval gate exists to avoid paying for the next stage twice.
The stage-gate loop in code
def post_train(model, stages, eval_harness, data_budget):
# stages: ordered list of (name, train_fn, data_mixer).
# The loop that matters is not train() — it is what happens on gate failure.
for round_idx in range(MAX_ROUNDS):
for name, train_fn, mixer in stages:
data = mixer.sample(data_budget[name]) # mix: coding/math/chat/safety/...
model = train_fn(model, data) # SFT, DPO, or reasoning-RL step
report = eval_harness.run(model) # capability + win-rate + safety gates
if not report.gates_pass:
# Fix the DATA: add the failing slice, rebalance the mix,
# then re-run THIS stage — not the next one.
mixer.add_failure_slices(report.failures)
break # restart rounds with the corrected mix
else:
return model # every gate passed: ship it
raise PipelineStalled("gates still failing after max rounds")Conceptual code, not a training script. The interview skill is defending the two design choices it encodes: failures route back to the data mixer (not to hyperparameter twiddling), and a failing stage blocks the pipeline instead of letting a later stage paper over it.
What interviewers actually ask
They rarely ask for definitions. They ask for the design decision — and they listen for the tradeoff.
What to say (≈90 sec): “I’d run SFT first, preference tuning second, reasoning RL third — and the order is load-bearing. SFT teaches the instruction format cheaply; without it, preference reviewers are comparing two models that can’t follow the rubric, so the comparison signal is noise. Preference tuning comes next because it shapes taste on top of a model that already behaves like an assistant. Reasoning RL goes last because verifiable rewards are the most expensive signal per unit of behavior change, and I want that budget spent on reasoning, not on teaching the model to say ‘Sure, here’s…’. I’d also expect to repeat the loop — the Llama 3 recipe ran six rounds of SFT+DPO — because each round’s eval failures tell me what data the next round needs.”
Likely follow-up: “What breaks if you swap preference tuning and reasoning RL?” → You’d spend your most expensive compute teaching reasoning to a model whose taste isn’t set, then preference-tune afterward and risk washing out the reasoning gains — the classic stage-interference problem. The cheaper, broader stages should always set the foundation the expensive, narrow stages build on.
The answer that sinks you: “Run them all at once in a multi-task setup to save time.” Why it fails: The stages optimize different objectives on different data with different failure modes; mixing them removes the eval gates that tell you which stage broke. The pipeline exists precisely so each stage is measurable.
What to say (≈90 sec): “That’s the preference stage rewarding the wrong thing — reviewers consistently picked agreeable, flattering answers, so the policy learned that agreement scores well. It’s reward hacking through the human labelers rather than through a reward model bug. I’d fix it in the data, not the optimizer: first confirm with a sycophancy probe set — prompts where the user states a wrong opinion and we measure agreement rate versus the pre-preference checkpoint. Then rebalance: add contrastive pairs where the preferred answer politely corrects the user, rewrite the rubric to penalize unearned agreement, and re-run the preference stage. I’d also add a sycophancy probe to the standing eval gates so the next data refresh can’t reintroduce it silently.”
Likely follow-up: “Why not just add a penalty term against sycophancy in the loss?” → Because I can’t write down sycophancy as a differentiable term without a classifier, and training against a classifier is just reward hacking one level down — the policy learns to fool the sycophancy detector. The preference data is the ground truth here; fix the ground truth.
The answer that sinks you: “Increase the KL penalty to pull it back toward the SFT model.” Why it fails: That suppresses everything the preference stage learned, including the genuine helpfulness gains. It treats the symptom by undoing the treatment.
What to say (≈90 sec): “It depends on which stage. For SFT, I’d seriously consider LoRA: the format-following behavior lives in a low-rank subspace empirically, it trains a few hundred million parameters instead of 70B, and it fits on a single 8×H100 node instead of needing FSDP across many — which means faster iteration on data mixes, and data iteration is where SFT quality comes from. For preference tuning I’d want full fine-tuning, or at least a high-rank adapter, because taste changes are distributed across the model and LoRA can underfit them — plus the KL constraint against the reference is cleaner to reason about with full weights. For reasoning RL, full fine-tuning: you’re asking the model to develop genuinely new behaviors, and the empirical record is that low-rank adapters struggle there. So the call is stage-dependent, and the deciding variables are memory budget, iteration speed, and how distributed the target behavior is.”
Likely follow-up: “LoRA SFT underperforms on your long-context evals. Now what?” → I’d check whether the failure is capacity or data: raise the rank and see if the eval moves — if it moves, it was capacity; if it doesn’t, the data mix is wrong. Long-context failures are usually data (not enough long examples in the mix), not rank. I’d fix the mix first because it’s cheaper than the full-FT alternative.
The answer that sinks you: “Always full fine-tune — LoRA is just a hack for people without GPUs.” Why it fails: At 70B, full fine-tuning costs ~1TB of GPU memory and slows your data iteration to a crawl — and data iteration is the actual driver of post-training quality. Dismissing the cost/iteration tradeoff shows you’ve never budgeted a real run.
What to say (≈90 sec): “Four suites, and each stage must pass all four to advance. One: capability benchmarks — MMLU-style knowledge, coding, math — to catch the stage degrading what the base model knew. Two: preference win-rate against the previous checkpoint on a held-out prompt set judged blind, which is the only direct measure of the preference stage working. Three: a safety red-team suite — jailbreaks, disallowed content, sycophancy and bias probes — because later stages love to erode earlier safety behavior. Four: a product regression set of prompts the product actually cares about, since benchmarks don’t cover your users. The gate rule: no suite may regress beyond a threshold, and the suite the current stage targets must improve. A preference-tuning round that raises win-rate but drops the capability benchmark fails — the data mix gets rebalanced and the stage re-runs.”
Likely follow-up: “Your win-rate improved but human judges disagree with each other 35% of the time. Trust the gate?” → Not yet. I’d stratify the disagreement: if it’s genuine ambiguity (both answers defensible), the win-rate is noisy but unbiased, and I’d widen the judge pool and the prompt set. If disagreement concentrates on a slice — say, coding answers — the rubric is underspecified there, and I’d fix the rubric and re-judge that slice before trusting the gate.
The answer that sinks you: “Track the training loss — if it’s going down, the stage is working.” Why it fails: Training loss measures fit to the training data, which is exactly what reward hacking also minimizes. The gates exist because loss curves can’t distinguish learning from exploitation.
What to say (≈90 sec): “Three mechanisms. First, tiered reviewers: qualification tests and calibration on gold tasks before anyone touches production data, with continuous gold-task injection so per-reviewer accuracy is measured, not assumed — reviewers who drift get retrained or removed. Second, spend human hours only where they’re scarce: use the current model to pre-filter candidate pairs and route only the ambiguous, high-disagreement ones to humans; the obvious pairs get model-generated labels with spot audits. Third, version everything: prompts, rubric version, reviewer cohort, and dataset snapshot are pinned together, so when a later eval regresses I can trace it to the exact data batch and rubric that caused it. And I’d watch the leading indicators — inter-annotator agreement by cohort, gold-task accuracy trends, and the rate at which adjudication queues grow — because quality collapses show up in those metrics weeks before they show up in the model.”
Likely follow-up: “Your preference data is now 10x bigger but win-rate gains flattened. What happened?” → Diminishing returns on the easy distribution — the extra pairs are probably near-duplicates of what the model already learned. I’d dedup by embedding similarity, then shift the sampling toward the model’s current failures: hard prompts, long-tail skills, and the slices where the eval gates are weakest. More of the same data is the most expensive way to learn nothing.
The answer that sinks you: “Hire more annotators and pay per label to keep throughput up.” Why it fails: Piece-rate pay optimizes for speed over judgment, which is exactly how you get 10x the labels at 1x the quality. Throughput without calibration buys noise at scale.
What to say (≈90 sec): “Distillation, but done as a post-training pipeline in miniature. The 70B becomes the teacher: generate SFT demonstrations from it across the skill distribution the phone model needs, generate preference pairs by having the 70B produce candidate answers and ranking them (with human spot-checks), and train the 3B student on both. The key design point is that the student gets the teacher’s post-trained behavior — the taste and the reasoning patterns — not just its knowledge, which is why you distill from the final checkpoint, not the base model. I’d keep a capability gap analysis: run the same eval gates on the student and diff against the teacher to find which skills didn’t transfer — usually long reasoning chains and rare knowledge — and either accept the gap, add targeted teacher data for it, or route those queries to the server model. And the privacy invariant: no user data used for the teacher’s training crosses into the student’s data without the same consent and scrubbing pipeline.”
Likely follow-up: “The student matches the teacher on benchmarks but users prefer the teacher in blind tests. Why?” → Benchmarks measure correctness; blind preference measures the intangibles — tone, formatting, when to be concise — that the preference stage taught the teacher. The student probably underfit the preference signal, so I’d upweight the DPO stage in the distillation mix and check the student’s win-rate against the teacher specifically, not just against benchmarks.
The answer that sinks you: “Just quantize the 70B to 4-bit and ship that.” Why it fails: A 4-bit 70B is still ~40GB — it doesn’t fit in a phone’s memory budget, and quantization doesn’t change the serving economics. The constraint is the device, not the precision.
Key takeaways
- Post-training is a gated pipeline — SFT, preference tuning, reasoning RL, evals — not a single training run.
- Stage order is load-bearing: format first, taste second, verifiable skill third.
- At 70B scale, every design choice is also a budget choice: memory, compute, and annotator-hours.
- Stages interfere: each one can undo the last, so eval gates — not loss curves — decide what ships.
- When a gate fails, fix the data and re-run the stage. The pipeline is failure-driven.
Sources & further reading
Every claim in this guide traces to one of these — the wording is ours, the ideas are credited.
- “The Llama 3 Herd of Models” (Meta), arXiv:2407.21783 (six rounds of SFT+DPO, rejection sampling, PPO; model-generated data with human annotation)
- “Training language models to follow instructions with human feedback” (InstructGPT), arXiv:2203.02155 (the original SFT → reward model → PPO recipe)
- “Direct Preference Optimization” (Rafailov et al.), arXiv:2305.18290 (preference optimization without an explicit reward model)
- “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL”, arXiv:2501.12948 (verifiable-reward RL for reasoning behavior)
- “Constitutional AI: Harmlessness from AI Feedback” (Anthropic), arXiv:2212.08073 (AI-generated preference signal as an alternative to pure human labeling)