Evaluating LLMs: Metrics, LLM Judges & Interview Questions
From recall@k to LLM judges: how to measure whether an LLM system is actually getting better, with a real evaluation flywheel from production.
Explain it like I’m five
Imagine two robots both say, “I am better.” Instead of guessing, you give them the same set of question cards and decide the rules before they begin.
Those tests are evals. They check whether answers are correct and safe, and also whether they are fast and affordable. When a real user finds a new mistake, you add that kind of mistake to the test pile so it does not quietly return later.
Intuition: define “better” before optimizing
You ship a new model because the benchmark score went up — then users in one language get worse answers and nobody noticed until the complaints arrived. The aggregate hid a regression. Evals are the fix: small, repeatable experiments that turn ‘it feels better’ into evidence, slice by slice, before anything ships. In interviews, nobody asks ‘what is an eval?’ They ask ‘your aggregate score rose but one slice regressed — do you ship?’ By the end of this guide you’ll design evals for a coding assistant, calibrate an LLM judge, and know exactly when to hold the launch.
“The answers feel better” is not a launch criterion. A useful evaluation names the task, supplies representative inputs, defines a grader or rubric, records operating constraints, and compares a candidate against a baseline.
LLM systems are stochastic and open-ended, so one number is rarely enough. You need a portfolio: deterministic checks, expert labels, model-based graders, safety probes, latency and cost metrics, plus live monitoring after launch.
A good eval suite is a living specification. Production failures become regression cases; launch decisions combine quality with safety, latency, and cost.
Evaluate the system users experience: model, prompt, retrieval, tools, policies, and UI. A benchmark score on the base model cannot certify the product.
Build an eval that survives reality
Start with a decision
Write the decision the eval informs: “Ship candidate B if grounded-answer pass rate improves by 3 points, no safety slice regresses, p95 latency stays under 1.8 seconds, and cost rises less than 10%.” Metrics now have a job.
Choose cases and slices
Combine curated normal cases, known production failures, adversarial inputs, and long-tail examples. Slice results by language, prompt length, user intent, tool path, risk level, and any product-relevant category. Averages hide failures.
Choose graders
Use exact match for deterministic outputs, executable tests for code, reference-based rubrics for constrained tasks, and calibrated human or LLM judges for open-ended quality. Model judges need position-bias tests, blind comparisons, and human agreement checks.
Protect the test set
Version cases and graders together. Keep a holdout set. Prevent prompt authors from repeatedly tuning to every eval item. Track confidence intervals; a tiny score movement may be noise.
DoorDash: an eval gate before production
DoorDash documented a simulation-and-evaluation flywheel for support chatbots that runs multi-turn conversations offline and uses quality measurements as launch criteria.
The challenge is non-determinism: a prompt or model change can improve a demo while creating regressions elsewhere. DoorDash pairs an LLM-driven user simulator with an evaluation framework, then calibrates LLM-as-a-judge results against expert human judgment.
Generated users carry realistic support goals through multi-turn conversations, exercising branches that single-turn test prompts miss.
Automated judges score the conversations against defined quality criteria. Calibration checks that those scores track how expert reviewers judge the same behavior.
Eval pass rate becomes a north-star metric and exit criterion. Teams iterate offline until the candidate clears the bar, then compare production behavior and feed new failures back into the suite.
In one documented hallucination-reduction case study, DoorDash reports a 90% reduction in hallucinations in simulation and says that result carried into production.
Source: DoorDash Engineering, Jan. 26, 2026 ↗A hand-picked prompt can make almost any candidate look good. An eval suite forces variants through the same representative conversations, graders, and decision rule.
Precision, recall, and a launch threshold
An unsafe-content classifier flags 30 responses. Review finds 24 are truly unsafe; it missed 6 other unsafe responses.
Those equal numbers do not mean the risks are equal. For a high-harm category, six misses may be unacceptable even if F1 looks respectable. Thresholds must reflect consequence.
Begin with graders you can debug
from dataclasses import dataclass
@dataclass
class Case:
prompt: str
must_include: tuple[str, ...]
must_not_include: tuple[str, ...] = ()
def grade(answer, case):
text = answer.lower()
good = all(term.lower() in text for term in case.must_include)
safe = all(term.lower() not in text for term in case.must_not_include)
return {"pass": good and safe, "correct": good, "safe": safe}
# Pin model + prompt versions, then record latency and token cost too.Store the raw output, grader rationale, model and prompt version, timing, token counts, and dataset version. Without traceability, a failing score is only a warning light with no engine diagnostics.
What interviewers actually ask
A strong answer begins with the product decision and failure cost, not a favorite metric.
What to say (≈90 sec): “I'd build the eval in four layers. First, functional correctness on held-out tasks — run generated code against hidden test suites and measure pass@k, since text similarity says nothing about whether the code runs. Second, repository context — multi-file tasks that force the model to read across the codebase, scoring whether it resolves real dependencies rather than completing an isolated snippet. Third, acceptance and task completion — in production, track edit acceptance rate and whether the accepted code survives code review unchanged. Fourth, security and latency as hard gates — static analysis for common vulnerability patterns, plus tail latency, because a correct answer that introduces an injection flaw or arrives too late is a failed evaluation. The headline metric is task completion rate on held-out work, with security and latency as launch gates.”
Likely follow-up: “Pass@k went up but edit acceptance didn't move. What do you do?” → That gap means the benchmark doesn't reflect real usage, so I'd mine production sessions for the tasks users abandon or heavily edit, add those as new eval cases, and recheck whether the task distribution matches actual workflow.
The answer that sinks you: “I'd compare model output against reference solutions with BLEU or exact match.” Why it fails: text similarity doesn't measure whether code runs — it rewards superficial matches and punishes correct solutions that differ from the reference.
What to say (≈90 sec): “I'd earn trust in four steps. First, calibrate against blinded human labels — collect a few hundred gold judgments where raters don't know which model produced the output, then measure agreement with accuracy and Cohen's kappa as the baseline. Second, probe the known biases explicitly — swap answer order to test position bias, compare long versus short answers for verbosity bias, and run self-preference probes where the judge might favor its own outputs. Third, measure agreement by slice — by task type, difficulty, and language — because a judge that's 90 percent right overall can be a coin flip on the edge cases that matter. Fourth, audit every disagreement — sample the judge-versus-human conflicts, label them by hand, and use the failures to fix the rubric or retire the judge on that slice.”
Likely follow-up: “The judge agrees with humans 92 percent of the time but favors its own model's outputs. Now what?” → That's self-preference bias: I'd blind the judge to model identity, switch to order-randomized paired comparisons, and rerun calibration — if the bias survives, the judge gets replaced by human review on that slice.
The answer that sinks you: “I'd trust it because it's a strong model.” Why it fails: capability doesn't imply impartiality — strong models still carry position, verbosity, and self-preference biases that only explicit bias probes can reveal.
What to say (≈90 sec): “Not automatically — I'd work through four checks. First, sample size and confidence: is the regression statistically significant, or noise on a small slice? I'd look at confidence intervals, not just the point estimate. Second, severity: what actually regressed — a formatting nit, or factual answers in a safety-critical domain? Third, product exposure: how many users in that language, and is this a launch-blocking market? Fourth, and most important, the predeclared gate: a good eval plan states no-regression criteria per slice before you ever run it. If the gate said no slice may drop more than two points, the aggregate doesn't overrule it. My default is hold and investigate, shipping only if the regression is noise-level on a low-exposure slice with a follow-up fix already scheduled.”
Likely follow-up: “The team wants to ship anyway and fix it next sprint. How do you push back?” → I'd translate the slice into user terms — how many people are affected, with concrete degraded examples — and propose a staged rollout that excludes the regressed locale until the fix lands.
The answer that sinks you: “The aggregate went up, so ship it.” Why it fails: averages hide slice-level regressions — shipping punishes exactly the users the aggregate hid, which is the failure evals exist to prevent.
What to say (≈90 sec): “I'd attack it in three layers. First, prevention by construction: keep private holdouts that never touch the internet, and generate fresh or synthetic variants of public benchmarks — same difficulty distribution, brand-new items — so memorization can't score. Second, direct detection: plant canary strings in the dataset and check whether the model reproduces them, run memorization probes like showing the first half of an item and checking for verbatim completion, and use temporal splits comparing performance on items dated before versus after the model's training cutoff. Third, pattern analysis: look for suspicious item-level signatures — near-perfect scores on the most memorization-friendly items combined with poor performance on paraphrased twins. If a model nails the exact benchmark but collapses on a rephrased variant, that's contamination, not capability.”
Likely follow-up: “The vendor says the data wasn't in training. How do you verify?” → You don't trust the claim — you run the paraphrase-twin test and a temporal split yourself; contaminated models show a sharp gap between memorized items and fresh variants.
The answer that sinks you: “I'd ask the training team to confirm it wasn't in the data.” Why it fails: training provenance is often unverifiable, especially with third-party models — you need empirical detection, not attestations.
What to say (≈90 sec): “I'd structure it in three tiers. First, guardrails before anything else: safety and policy violations, hallucination rate on product facts, and revenue-protection metrics like return rate — these get alerting with auto-revert, not just dashboards. Second, the core metrics: task success — did the user find and buy what they needed — measured through conversion and session-level resolution; deflection, whether the assistant resolved the session without a human agent; and conversion quality, not just conversion, because a pushy assistant that drives regretted purchases shows up in returns. Third, the experiment infrastructure: counterfactual logging so every ranking decision is recorded for offline replay, interleaving or shadow experiments before full A/B tests, and latency and cost per session as hard budget constraints. I'd start on a small traffic slice with guardrail alerts wired to auto-revert.”
Likely follow-up: “Conversion is up but returns are up too. What's your read?” → The assistant is optimizing purchases but not fit — it's likely pushing the wrong products. I'd switch to return-adjusted conversion and check whether recommendations respect the user's stated constraints before calling it a win.
The answer that sinks you: “Run an A/B test on click-through rate.” Why it fails: CTR rewards engagement, not task success — it can't distinguish a helpful assistant from a manipulative one, and it ignores the guardrails entirely.
What to say (≈90 sec): “When you can't see production behavior, you front-load rigor in three places. First, rich offline suites covering the full operating envelope: accuracy benchmarks, synthetic stress tests for adversarial and out-of-distribution inputs, and slice-by-device-tier evaluation — because an on-device model behaves differently on older chips with less memory. Second, privacy-preserving measurement: differentially private aggregates and on-device computed metrics that report only anonymized summaries, plus an opt-in diagnostics cohort whose volunteered data gives ground truth on real usage. Third, conservative launch gates: staged rollouts by device tier with automated rollback triggers on the aggregates you do have, and wider tolerance bands — since the signal is sparse, you demand larger effect sizes before declaring a win. The rule is simple: compensate for thin telemetry with thicker offline evidence and slower rollouts.”
Likely follow-up: “The opt-in cohort loves it but the aggregates are flat. Do you trust the cohort?” → No — opt-in users are enthusiasts, not a representative sample. I'd treat cohort feedback as qualitative signal only, and require the privacy-preserving aggregates to move before expanding the rollout.
The answer that sinks you: “Ship it and watch the crash reports.” Why it fails: crash reports are the thinnest possible signal — launching without offline rigor and staged gates means you discover regressions from user complaints.
Key takeaways
- An eval exists to support a concrete decision.
- System-level quality includes safety, latency, cost, and tool behavior.
- Use multiple graders and calibrate subjective ones.
- Slice metrics; averages hide concentrated failures.
- Turn production failures into versioned regression tests.