RLHF Explained: How Human Feedback Trains LLMs
From pairwise human labels to reward models and policy optimization: how reinforcement learning from human feedback turns preferences into a safer language model.
Explain it like I’m five
Imagine a robot gives two answers and a teacher points to the better one. After many choices, another little “judge” learns what people usually prefer, and the robot practices making answers that earn higher scores.
That is RLHF. People do not write every future answer; their comparisons teach a reward signal. The robot then improves against that signal while staying close to what it already knew. If the judge rewards the wrong thing, the robot can learn the wrong lesson.
Intuition: teach with comparisons
You fine-tune your model on human preferences and it gets weirdly sycophantic — agreeing with everything, writing longer answers nobody asked for, gaming your reward model instead of being helpful. That’s reward hacking: the policy found a loophole in your reward signal. A KL penalty is the fix: it keeps the policy close to the model you trusted while it learns what humans prefer. In interviews, nobody asks ‘what is RLHF?’ They ask ‘why is a KL penalty necessary, and how would you catch reward hacking before launch?’ By the end of this guide you’ll explain the full pipeline — reward model, policy optimization, and the guardrails — like you’ve run it.
A pretrained model predicts likely text, not necessarily helpful or harmless text. RLHF turns human preference judgments into a learning signal. Reviewers compare responses; a reward model learns the pattern behind those choices; the language-model policy is then optimized to earn higher reward without drifting too far from a stable reference.
“RLHF” is a family of pipelines, not one optimizer. Modern systems may replace PPO with preference objectives such as DPO while keeping the core idea: learn from comparisons.
Humans provide a limited set of demonstrations or pairwise preferences. Models generalize that signal to new prompts. This is powerful—and exactly why label quality and reward misspecification matter.
The three-stage pipeline
1. Supervised fine-tuning
Start from a pretrained model and train on high-quality prompt–response demonstrations. This creates a policy that follows instructions well enough to generate useful candidate responses.
2. Preference modeling
For one prompt, sample answers A and B. A reviewer picks the better one under a rubric. A reward model assigns scalar scores and is trained so the chosen answer scores above the rejected answer.
3. Policy optimization
Sample responses from the current policy, score them, and update the policy toward higher reward. A KL penalty discourages it from moving too far from the reference policy. PPO is one route; direct preference methods such as DPO optimize from comparisons without an explicit reinforcement-learning loop.
The failure modes
A policy may exploit quirks in the reward model: unnecessary verbosity, confident style, or rubric-specific phrases. This is reward hacking. Distribution shift, inconsistent labels, preference diversity, and over-optimization all complicate the neat diagram.
ChatGPT: turning comparisons into behavior
OpenAI documented training ChatGPT with reinforcement learning from human feedback: people compared candidate answers, and those preferences shaped how the assistant responds.
Human AI trainers first created supervised conversations by playing both sides. OpenAI then collected comparison data: the model produced alternative answers to a prompt, trainers ranked them, and those rankings trained a reward model. The policy was optimized with PPO, and the process was repeated.
Trainer-written conversations give the pretrained model an initial pattern for following instructions and participating in dialogue.
Rankings say which of several answers is more helpful, truthful, or appropriate. The reward model learns to predict those preferences beyond the exact prompts reviewers saw.
PPO updates the assistant toward higher predicted reward while constraints keep it from drifting arbitrarily far from the reference behavior.
OpenAI described the resulting product behavior as able to answer follow-ups, admit mistakes, challenge incorrect premises, and reject inappropriate requests. The launch post reports qualitative improvements, not a single numeric RLHF score.
Source: OpenAI, “Introducing ChatGPT,” Nov. 30, 2022 ↗If reviewers consistently reward polished length, the model can learn to sound helpful without becoming more correct. Preference data must represent the behavior the product actually values.
A pairwise loss by hand
Suppose the reward model scores the chosen response r⁺ = 1.4 and the rejected response r⁻ = 0.6.
Now imagine the policy gets reward 2.0 but incurs estimated KL drift 3.0 with β=0.1. The regularized objective is 2.0 − 0.1×3.0 = 1.7. High reward still wins, but drift has a price.
The two ideas in code
import torch
import torch.nn.functional as F
def preference_loss(reward_chosen, reward_rejected):
# Maximize P(chosen beats rejected).
margin = reward_chosen - reward_rejected
return -F.logsigmoid(margin).mean()
def regularized_objective(reward, logp, ref_logp, beta=0.1):
# Approximate per-token KL penalty keeps policy near reference.
kl_estimate = logp - ref_logp
return (reward - beta * kl_estimate).mean()This is conceptual code, not a production PPO implementation. The interview skill is explaining what the objective rewards, what the reference constrains, and how you would detect exploitation.
What interviewers actually ask
Strong answers connect optimization details to human and product consequences.
What to say (≈90 sec): “I’d start from why the reward model can’t be trusted blindly: it’s trained on a fixed dataset sampled from the reference policy, so its scores are only reliable near that distribution — push the policy far away and you’re optimizing a hallucinated signal, which is Goodhart’s law in action. The KL penalty, usually β · KL(π ∥ π_ref) against the SFT reference, is a trust region: it lets the policy chase higher reward while staying where the reward model generalizes. In practice I’d sweep β and plot mean reward against KL — pick the knee where reward plateaus before KL explodes — and watch two dashboards during training: KL per batch and win-rate on a holdout prompt set. If KL climbs while holdout quality flattens, the optimizer found an exploit, not an improvement.”
Likely follow-up: “How do you pick the β value in practice?” → I wouldn’t guess it. I’d run a small-scale sweep of β values, plot mean reward against KL divergence per batch, and pick the knee of the curve — the point where additional KL buys almost no extra reward. Then I’d confirm it in a short full-scale run, watching holdout win-rate: if the policy moves a lot on KL but holdout quality stalls, β is too low and the reward model is being gamed.
The answer that sinks you: “The KL penalty makes the model’s answers more accurate.” Why it fails: The KL term never touches accuracy or the reward signal — it constrains how far the policy moves from the reference. Its job is trust-region enforcement, because the reward model is only reliable near the distribution it trained on.
What to say (≈90 sec): “I’d run five checks before launch. First, adversarial red-teaming aimed at known exploits — prompts that reward verbosity, sycophancy, or format tricks. Second, blinded human audits on gold tasks where we know the right answer, so judges can’t tell which output came from the new policy. Third, distribution slices: track reward versus actual quality by length, topic, and safety category — reward climbing while quality is flat is the smoke signal for hacking. Fourth, policy–reward disagreement: train a second reward model on a different seed or data split and flag samples where they strongly disagree, since those are the policy’s likeliest exploits. Fifth, a holdout rubric the optimizer never saw — a separate judge with different criteria — as an independent check. And I’d plant canary probes: deliberately inject a spurious feature, like always ending with a summary, and see whether the policy learns to game it.”
Likely follow-up: “Your holdout rubric disagrees with the reward model on 20% of samples. Now what?” → I wouldn’t average them and move on. I’d sample the disagreements and have humans adjudicate to separate genuine ambiguity from systematic error. If the reward model is consistently wrong on a slice — say, it rewards confident-but-wrong medical answers — that slice gets relabeled and the reward model retrained. Ship only when the remaining disagreement looks like noise, not a pattern the policy can exploit.
The answer that sinks you: “I’d just raise the KL penalty until the hacking stops.” Why it fails: Cranking up KL also cranks down learning — it masks the exploit without fixing the reward model’s blind spot, and it sacrifices the alignment gains the training was for.
What to say (≈90 sec): “Operationally they’re different animals. PPO runs three models — policy, frozen reference, reward model — plus a value head, with online rollouts generating fresh samples during training. That means rollout infrastructure, careful batch scheduling, and a KL controller to tune; it’s powerful but finicky and can diverge. DPO skips the reward model entirely: it derives an implicit reward from the log-probability ratio between the policy and the reference on each preferred–rejected pair, so training is a stable supervised-style objective on offline data — one model, no rollouts, far simpler infra. The tradeoff is that DPO can only learn from the pairs you already collected; it can’t incorporate online rewards from verifiable outcomes or explore states the offline data never covered.”
Likely follow-up: “When would you still choose PPO over DPO?” → Whenever the reward can’t be expressed as a pairwise preference. Verifiable outcomes — unit tests passing, math proofs checking out — multi-turn rollouts where the agent explores states offline data never covered, or tool-use traces where reward comes from the environment. DPO is simpler, but it can’t learn from a signal you can only get by acting.
The answer that sinks you: “DPO is just a better PPO.” Why it fails: That erases the actual tradeoff. DPO drops the reward model entirely — that’s what makes it simpler and also what removes online, rollout-based reward signals. Calling it ‘better’ without the tradeoff shows you don’t know what was traded away.
What to say (≈90 sec): “Five main sources. Annotator demographics — who labels decides what ‘helpful’ means, so a narrow pool bakes in a narrow worldview. Rubric interpretation — vague guidelines let annotators substitute personal judgment. Presentation order — position bias means the first answer wins disproportionately. Verbosity preference — longer answers score higher even when they’re worse. And cultural assumptions about politeness, humor, or political framing. I’d model and measure disagreement rather than pretend it away: overlap tasks across annotators, track inter-annotator agreement, stratify pools by demographics, A/B the presentation order to quantify position bias, and normalize for length when analyzing scores.”
Likely follow-up: “Your annotators disagree 30% of the time. Keep the labels?” → Not blindly. I’d separate disagreement into ambiguity and error using gold standards — pairs where calibrated annotators disagree but both answers are defensible are genuine preference ambiguity, and keeping them teaches the model uncertainty. Pairs where gold-task performance shows one side was simply wrong get re-adjudicated or dropped. High disagreement with high gold accuracy is signal; high disagreement with low gold accuracy is noise.
The answer that sinks you: “Hire more annotators — scale fixes bias.” Why it fails: More annotators from the same pool just amplify the same skew. Scale without diversity and calibration turns bias into a statistically significant result.
What to say (≈90 sec): “I’d build it in stages. Prompt sampling first: stratify across domains, difficulty, and safety categories, and deduplicate with embedding clustering so a million pairs aren’t ten thousand unique prompts repeated. Then response generation from multiple model snapshots and temperatures for diversity. Reviewer routing: qualification tests, calibration on gold tasks, and continuous gold-check injection so per-reviewer accuracy is measured, not assumed — with blinded, order-randomized presentation to kill position bias. Quality control: inter-annotator disagreement metrics, adjudication queues for low-agreement pairs, and immutable versioned dataset snapshots with the rubric version pinned. Privacy: PII scrubbing on prompts, retention policies, access controls. And monitoring for feedback loops — watch whether the policy drifts into regions the labels were never collected under, and track reward-score distributions over time for dataset staleness.”
Likely follow-up: “How do you keep quality up when you 10x the reviewer pool?” → Tiered onboarding. New reviewers start on gold tasks and low-stakes pairs, and graduate to production queues only above an accuracy threshold. The gold-injection rate stays constant regardless of pool size, so quality is measured continuously rather than assumed — and I’d track per-cohort agreement metrics to catch a bad hiring batch before it contaminates the dataset.
The answer that sinks you: “Hire more labelers and run everything through them once.” Why it fails: One-pass labeling at scale with no calibration, gold checks, or dedup buys millions of noisy, duplicated pairs — and no way to tell which ones are wrong.
What to say (≈90 sec): “I’d split alignment into two layers: a global model trained in the cloud, and local adaptation that never crosses the trust boundary. On-device ranking keeps the user’s preference history on the phone and fine-tunes a small adapter — like a LoRA — or a local reranker, so raw conversations never leave the device. For learning across users, federated signals: each device computes updates locally and only uploads aggregated updates with differential-privacy noise added, so no individual’s data is recoverable. And explicit controls: user-visible toggles for personalization, per-feature opt-in, and on-device deletion. The invariant is that only differentially private aggregates ever leave the device — never raw text.”
Likely follow-up: “Does on-device personalization risk undoing global safety alignment?” → Yes, if it’s unconstrained — a local adapter trained on a user’s preferences could amplify exactly the behaviors the global model was taught to refuse. I’d keep safety as a frozen base layer the adapter can’t override: constrain what the adapter is allowed to change, and run personalized outputs through the same on-device safety classifier. And personalization updates get validated against the safety eval suite before they ever reach the device.
The answer that sinks you: “Encrypt the conversations before uploading them.” Why it fails: Encryption protects data in transit, but the server still decrypts it to train. The privacy risk was never the wire — it was who gets to read the plaintext.
Key takeaways
- Preferences are easier to collect than perfect demonstrations.
- The reward model approximates human judgment; it is not the true objective.
- A reference-policy constraint helps prevent destructive drift.
- PPO and DPO are different optimization routes around the same preference signal.
- Reward hacking and label bias are core engineering problems, not footnotes.