Skip to content

CCAR-P : Evaluation & Testing (Domain 4)

Domain 4 : Evaluation, Testing & Optimization Quiz

20 questionsmedium

The Evaluation, Testing, and Optimization domain focus on the systematic measurement and refinement of Claude-powered solutions. Achieving production-grade reliability requires moving beyond anecdotal “vibe checks” to a rigorous framework centered on objective metrics, representative datasets, and a diagnostic approach to system failures. This domain covers the selection of performance indicators across five critical pillars, the implementation of mixed-methodology testing, and the execution of cost and latency optimization strategies.

Performance Metric Selection: The Five Pillars of CCAR-P Evaluation

Effective evaluation begins with matching the correct metric to the specific failure mode or business constraint of the solution. Performance is assessed across five primary pillars:

PillarFocus AreaPrimary Metrics
AccuracyCorrectness and groundingFactuality scores against labeled datasets; accuracy on task completion.
LatencyResponse speedp95/p99 percentiles; Time to First Token (TTFT) for streaming applications.
CostFinancial efficiencyCost per request; total tokens per task; cache hit rates.
SafetyPolicy adherenceSafety violation rates on adversarial sets; refusal accuracy.
SecurityRobustnessPass rates against prompt injection and data leakage attack patterns.

Designing Robust Evaluation Datasets for Enterprise AI

The integrity of an evaluation framework is entirely dependent on the quality of the dataset used. A robust evaluation dataset must adhere to the following architectural principles:

  • Production Mirroring: The dataset must mirror the actual distribution of live traffic. Using “clean” or simplified inputs leads to inflated performance scores that collapse upon deployment.
  • Edge Case Integration: Datasets should deliberately include edge cases and adversarial examples in known proportions. This allows for stratified scoring to understand how the system handles complex or non-standard inputs.
  • Held-Out Data: Evaluation sets must be strictly held out from the prompt development process. Tuning instructions against a test set results in “overfitting,” where the model appears to perform well but fails to generalize to new data.
  • Continuous Refresh: As production traffic patterns drift over time, the evaluation dataset must be updated to remain a representative benchmark of reality.

Mixed-Methodology Testing Strategies for Claude

A comprehensive testing strategy utilizes a tiered approach to grading, moving from automated checks to nuanced human calibration.

1. Code-Based Graders for Deterministic Checks

These are deterministic scripts used to validate objective properties. They are ideal for checking schema validity, the presence of required JSON fields, exact string matches, and adherence to latency budgets.

2. Using LLM-as-Judge with Objective Rubrics

For open-ended quality assessments such as tone, helpfulness, and reasoning depth, a separate, highly capable model (often a higher-tier model than the one being tested) acts as a judge. This judge evaluates outputs against a written, objective rubric to provide consistent, scalable scoring.

3. Human Review and LLM-as-Judge Calibration

Human oversight is the “gold standard” used to calibrate the LLM-as-judge. Architects should periodically sample judge-graded outputs and compare them against human ratings. If the judge and human scores diverge, the rubric or the grading prompt must be recalibrated.

A/B Testing Best Practices for Architectural Changes

A/B testing is the standard for rolling out architectural changes. To ensure statistical significance and avoid regressions:

  • Single Variable Isolation: Only change one component per experiment (e.g., the model tier, the prompt version, or the retrieval depth). Changing multiple variables makes it impossible to attribute performance shifts.
  • Predefined Success Metrics: Metrics and target sample sizes must be established before the test begins.
  • Evidence-Based Rollout: Decisions to promote a variant should be based on statistically significant data rather than anecdotal “feeling.”
  • Live Traffic Splitting: Run variants in parallel on identical slices of live traffic to account for temporal variables.

Root-Cause Failure Diagnosis in AI Systems

When evaluation scores drop, architects must diagnose the specific failure mode to apply the correct lever.

SymptomIdentified Root CausePrimary Resolution
Documents are correct, but the answer is wrong or ungrounded.Hallucination / Weak GroundingTighten grounding instructions; mandate citations to retrieved text.
The retrieved documents are irrelevant or missing.Retrieval FailureImprove chunking/embeddings; add a re-ranking stage.
Failure occurs only on complex, multi-step reasoning.Model MismatchUpgrade to a higher-capability model tier (e.g., Opus tier).
Failure is uniform across all types of inputs.Prompt FailureRewrite system prompts to include explicit, testable criteria.

AI Optimization Levers: Token Usage, Latency, and Cost

Optimization involves balancing quality, speed, and spend. The primary levers include:

  • Token Usage: Prune unnecessary tool outputs, trim conversation history, and utilize prompt caching for stable prefixes.
  • Latency: Implement streaming for user-facing paths and route high-volume, low-complexity tasks to the fastest available model tier.
  • Cost-Performance: Reserve premium model tiers for reasoning-heavy slices of the workload. For non-urgent, high-volume tasks, leverage asynchronous processing.

Leveraging the Message Batches API for Cost Savings

The Message Batches API is a critical tool for cost-sensitive, high-volume workloads. It offers a 50% discount compared to standard API pricing.

  • Correlation: Uses a custom_id to map requests to responses.
  • SLA: Processing is completed within a 24-hour window.
  • Limitations: Does not support multi-turn tool calling.
  • Turnaround Math: To calculate the maximum expected turnaround time ($T_{turnaround}$) for a batch job, use the formula: $$T_{turnaround} = T_{wait} + 24h$$ In most enterprise scheduling, architects aim for a total window where $T_{turnaround} \le 30h$, accounting for queue wait times and the 24h processing SLA.

Domain 4 Scenario-Based Practice Questions

Evaluation and Testing Questions

  1. A team is testing a RAG system. The evaluation shows that while the correct chunks are being retrieved from the database, Claude frequently ignores the context and provides answers based on its internal training data. What is the correct diagnostic and fix?
  2. You are optimizing a high-traffic support bot. Currently, every call includes a 5,000-token instruction set. Evals show that the bot performs well with Sonnet, but the latency is slightly above the p95 goal. What is the most effective optimization lever here?
  3. A financial services firm requires a deterministic check to ensure that the model never outputs a credit card number in its response. Which testing methodology should be prioritized for this specific requirement?
  4. You are designing an A/B test to compare a new prompt version. Your colleague suggests changing the prompt and upgrading the model tier simultaneously to see the “maximum possible improvement.” Why is this an anti-pattern?
  5. A batch processing job for sentiment analysis on 1 million reviews needs to be scheduled. The team wants to minimize costs. Which API should be used, and what is the maximum processing time they must budget for?
  6. During evaluation, a model consistently fails on a specific demographic slice of data, though its aggregate accuracy is 92%. What does this indicate about the dataset design?
  7. An architect observes that a judge model (LLM-as-judge) is consistently giving higher scores than the human reviewers for the same set of responses. What is the required next step?
  8. A developer suggests using a “monolithic context” approach, feeding the entire tool catalog (100+ tools) into every request to ensure Claude always has what it needs. What is the architectural risk?
  9. A system is failing on a complex logical puzzle that requires five distinct steps of deduction. The current model is a fast-tier model. What is the likely root cause?
  10. A prompt caching strategy is implemented, but the team finds that the cache hit rate is 0%. They notice that a “Current Timestamp” is included at the very beginning of every system prompt. What is the error?

Domain 4 Practice Questions Answer Key

  1. Root Cause: Weak Grounding. The retrieval is successful, but the model is not adhering to the context. Fix: Tighten grounding instructions in the system prompt and require the model to provide citations for every claim.
  2. Optimization: Prompt Caching. Since the 5,000-token instruction set is stable and reused across high-traffic calls, caching the prefix will reduce both latency and cost.
  3. Methodology: Code-based Graders. Checking for a specific pattern (like a credit card number) is a deterministic, objective task that a script/regex can perform more reliably and cheaply than an LLM.
  4. Reasoning: Single Variable Isolation. Changing both the prompt and the model tier violates the principle of isolating a single variable, making it impossible to know which change caused a performance shift.
  5. API & Math: Message Batches API. It provides a 50% discount. The budget must account for a 24-hour SLA.
  6. Design Issue: Lack of Stratified Evaluation. High aggregate accuracy can hide “quality cliffs” in specific slices. The dataset design must include enough demographic-specific examples to make per-slice scores meaningful.
  7. Action: Calibration. The architect must recalibrate the rubric used by the judge model to align more closely with human judgment.
  8. Risk: Capability Bloat. Too many tools burn unnecessary tokens and degrade selection accuracy, as the model may struggle to choose the correct tool from an overlapping or oversized list.
  9. Root Cause: Model Mismatch. Fast-tier models often lack the reasoning depth for multi-step logic. The fix is to test the prompt on a higher-capability tier (Sonnet or Opus).
  10. Error: Cache-Breaking. Any change in the prompt prefix (like a shifting timestamp) breaks the cache from that point forward. Variables must be placed at the end of the prompt to keep the cached prefix stable.

Evaluation and Optimization Reflection Questions

  1. Metric Alignment: Design a metric dashboard for a healthcare AI agent. Which metrics would you prioritize to satisfy a HIPAA-regulated environment, and how would they differ from a standard creative writing tool?
  2. Dataset Integrity: Explain the trade-offs between using a “synthetic” evaluation dataset generated by Claude vs. a dataset consisting purely of real human-user logs. How does this impact “production mirroring”?
  3. Optimization Strategy: You have a fixed budget and a latency SLA of 2 seconds. Describe a routing architecture that uses model tiering and Batch API to maximize quality without violating these constraints.
  4. The Human Element: In a high-volume system, where is the most “risk-aware” placement for a Human-in-the-Loop (HITL)? Define the criteria for when an agent should escalate to a human reviewer vs. providing an automated response.
  5. Failure Analysis: If a system passes 99% of its code-based grader checks but fails 40% of its “LLM-as-judge” helpfulness checks, what does this tell you about the current state of the prompt and the integration?

Essential CCAR-P Evaluation and Testing Glossary

  1. A/B Testing: An experimental methodology where two versions of a system (A and B) are compared by splitting live traffic to determine which performs better based on predefined metrics.
  2. Accuracy: A pillar of evaluation measuring how often the model provides correct, grounded, or factual information.
  3. Batch API: An Anthropic API feature providing a 50% discount for non-urgent tasks processed within a 24-hour window.
  4. Capability Bloat: A condition where an agent is given more tools or permissions than necessary, leading to reduced accuracy and higher costs.
  5. Code-Based Grader: An automated script used to validate deterministic properties of an output, such as JSON structure or latency.
  6. Edge Case: An input that is at the extreme or non-standard end of the expected distribution, often used to test system robustness.
  7. Grounding: The technique of ensuring a model’s response is based strictly on provided context rather than internal training data.
  8. Hallucination: A failure mode where the model generates plausible-sounding but incorrect or ungrounded information.
  9. Held-Out Dataset: A set of evaluation data that is never shown to the model or used during the prompt engineering phase to ensure unbiased testing.
  10. LLM-as-Judge: The use of a highly capable model to evaluate the quality of another model’s output based on a specific rubric.
  11. Model Mismatch: A failure where the selected model tier lacks the reasoning capability required for the complexity of the task.
  12. Progressive Discovery: An integration pattern where tool definitions or data are loaded on-demand rather than all at once.
  13. Prompt Caching: A feature that stores the processed form of a stable prompt prefix to reduce latency and costs for repeated calls.
  14. Re-ranking: A second-stage retrieval process that re-orders search results to ensure only the most relevant chunks are sent to the model.
  15. Time to First Token (TTFT): A latency metric measuring the duration between a request and the first character of the streaming response.

Leaderboard

No scores saved yet. Be the first!

20 Questions — Domain 4 : Evaluation, Testing & Optimization Quiz

Expand any question to reveal the correct answer and explanation.

  1. 1 A RAG system that previously performed well begins returning confident but incorrect answers immediately following a document index refresh. The model version and temperature settings remain unchanged. Which component should an architect investigate first?

    Consider which part of the system architecture was modified just before the performance drop occurred.

    The retrieval and indexing pipeline

    When symptoms are triggered specifically by a document refresh, the most likely cause is a broken re-indexing process or mismatched embeddings feeding poor context to the model.

    • ✗ The model's context window limit

      Context window limits affect the volume of data processed, but a sudden shift in accuracy after a data refresh specifically points to the retrieval pipeline.

    • ✗ The system prompt's grounding instructions

      While grounding instructions help, they are unlikely to be the root cause of a sudden performance drop that correlates exactly with a data update.

    • ✗ The model's internal weights

      Model weights are fixed in production; changes in behavior are driven by inputs, which in RAG systems are provided by the retrieval step.

  2. 2 You are designing an evaluation suite for a Claude Sonnet-powered agent. Which model selection for the 'LLM-as-judge' component adheres to architectural best practices for detecting subtle reasoning errors?

    Recall the relationship between the reasoning capability of the judge and the model being judged.

    Claude Opus, as it is a more capable model than the candidate

    Professional evaluation methodology requires using a stronger model as a judge to identify mistakes that the candidate model might miss.

    • ✗ Claude Sonnet, to ensure the judge understands the candidate's logic

      Using the same model as the one being evaluated often leads to an under-detection of errors that the model itself is prone to making.

    • ✗ Claude Haiku, to minimize the cost and latency of the evaluation process

      Judges must be at least as capable as the candidate; a lower-tier model may miss complex reasoning failures in a higher-tier candidate.

    • ✗ An ensemble of three Haiku models to provide a majority-vote score

      While ensembles provide stability, the individual reasoning capacity of the judge must still exceed or match the candidate to ensure high-quality rubric grading.

  3. 3 Your team is preparing to process $50,000$ legacy documents using the Message Batches API. Initial tests show that $18\%$ of documents require multiple prompt refinements. What is the most cost-efficient strategy for scaling this workload?

    Think about how to maximize first-pass success rates before committing to high-volume API calls.

    Refine prompts on a representative sample before starting the full batch run

    Maximizing first-pass success on a smaller sample ensures the prompt is robust before committing to large-scale processing, minimizing expensive resubmissions.

    • ✗ Run the entire volume through the Batch API and resubmit failures as they occur

      Iterative resubmissions at scale result in high costs that can negate the savings provided by the Batch API.

    • ✗ Switch to the synchronous API to allow for real-time retry logic

      This approach sacrifices the $50\%$ cost discount of the Batch API for a workload that is likely latency-tolerant.

    • ✗ Increase the max_tokens limit for the full batch to handle complex documents

      Increasing token limits does not address underlying prompt failures and may lead to higher costs without improving accuracy.

  4. 4 A production system must comply with a strict business SLA requiring results within $30$ hours of user submission. Using the Message Batches API (which has a $24$-hour processing window), what is the longest scheduling interval that still guarantees compliance?

    Use the formula $T_{turnaround} = T_{wait} + T_{process}$ where $T_{process}$ is the maximum processing time of the API.

    Submit batches every $6$ hours

    The turnaround is calculated as $T_{wait} + T_{process}$. With $T_{process} = 24h$, $T_{wait}$ must be $\le 6h$ to satisfy a $30h$ SLA.

    • ✗ Submit batches every $4$ hours

      While this meets the SLA, it is not the most cost-efficient as it requires more frequent orchestration than the maximum allowed interval.

    • ✗ Submit batches every $12$ hours

      A $12$-hour wait plus a $24$-hour processing time totals $36$ hours, which violates the $30$-hour SLA.

    • ✗ Submit one batch every $24$ hours

      This schedule could lead to a total turnaround time of up to $48$ hours, significantly exceeding the business requirement.

  5. 5 After a batch run of $10,000$ documents, $300$ documents fail with a 'context_length_exceeded' error. How should an architect proceed to minimize costs while completing the task?

    Focus on the mechanism provided by the Batch API to correlate requests and responses.

    Identify failures via custom_id, chunk them, and resubmit as a new batch

    Isolating only the failed requests using the provided ID mapping and adjusting their size for a new batch is the most cost-effective recovery method.

    • ✗ Resubmit all $10,000$ documents with a smaller chunking size

      Reprocessing successful documents is a waste of budget; only the failures need attention.

    • ✗ Switch the failed $300$ documents to the synchronous API

      The synchronous API is more expensive and does not solve the underlying context limit issue without additional chunking.

    • ✗ Request a global increase to the model's context window

      Context limits are fixed constraints of the model architecture and cannot be adjusted as a configuration setting.

  6. 6 When diagnosing quality failures, you find that correct documents are being retrieved, but the model produces answers that are ungrounded or contain hallucinations. What is the most effective architectural fix?

    Determine whether the failure lies in finding the information or in how the model uses the information provided.

    Tighten grounding instructions and require citations to the retrieved text

    Explicit instructions to use only provided text and to cite sources help anchor the model's output to the retrieved facts, reducing fabrication.

    • ✗ Increase the number of retrieved chunks (top-k)

      Adding more chunks increases noise if the model is already failing to ground its answers in the currently provided relevant context.

    • ✗ Upgrade the model tier to a higher-capability model

      While a larger model might help, the first step should be optimizing the prompt instructions specifically for grounding and citation.

    • ✗ Add a re-ranking step to the retrieval pipeline

      Re-ranking improves retrieval precision, but in this scenario, the correct documents are already being successfully retrieved.

  7. 7 An architect notices that refusal rates have increased by $30\%$ week-over-week despite no changes to the application code. What should be the first hypothesis for investigation?

    Consider factors external to the code that could influence how the model evaluates incoming requests.

    A shift in the distribution of incoming user traffic (drift)

    Sudden changes in performance metrics like refusal rates often signal a change in the types of inputs being received from users, known as data drift.

    • ✗ The model has reached its internal training knowledge cutoff

      Knowledge cutoffs are static properties of the model and would not cause a sudden week-over-week spike in refusal rates.

    • ✗ A failure in the prompt caching layer

      Caching failures impact cost and latency rather than the semantic content or the model's willingness to answer.

    • ✗ Exceeding the organization's token-per-minute (TPM) rate limit

      Rate limits result in $429$ errors rather than policy-based refusals within the model's generated response.

  8. 8 Which evaluation methodology is specifically designed to score open-ended quality characteristics like 'helpfulness' or 'tone' at scale?

    Identify the approach that utilizes a model's reasoning capabilities to evaluate non-objective criteria.

    LLM-as-judge using a written rubric

    Using a separate model to grade outputs against a rubric allows for the automated assessment of qualitative traits that are difficult to code.

    • ✗ Code-based graders using exact-match logic

      Programmatic graders are best for objective criteria like schema validity or latency, not subjective quality.

    • ✗ Property-based invariant checks

      Invariant checks ensure certain rules are never broken (e.g., no PII), but they do not measure the relative quality of tone or helpfulness.

    • ✗ Unit-style regression testing

      Regression testing focuses on ensuring previous behaviors haven't changed, rather than assessing the nuance of subjective quality.

  9. 9 How should an evaluation dataset be designed to ensure it validly represents production readiness?

    Think about the relationship between the test data and the actual inputs the system will handle.

    By mirroring the actual distribution of production traffic, including edge cases

    A representative dataset must include the same proportion of easy, hard, and adversarial inputs seen in real-world usage to provide meaningful scores.

    • ✗ By using synthetically generated, perfectly formatted queries

      Synthetic data often lacks the messiness and noise of real traffic, leading to an overestimation of system accuracy.

    • ✗ By selecting only the 'golden' cases where the model is known to succeed

      Focusing only on successful cases creates a 'cherry-picking' bias and fails to identify where the system might fail.

    • ✗ By maximizing the number of tokens in each query to test context limits

      While testing limits is useful, it does not represent the standard traffic distribution needed for a general evaluation of production readiness.

  10. 10 You want to implement a change to your system prompt. What is the minimum responsible process for verifying that this does not regress quality?

    Consider a systematic approach that happens before the code reaches any live users.

    Run a regression suite of $50$-$200$ prompts gated in the CI/CD pipeline

    Automated regression testing using a significant sample size is the production baseline for ensuring changes don't negatively impact performance.

    • ✗ Perform a spot-check on five representative prompts

      Small sample sizes are insufficient to detect regressions and can lead to shipping flawed updates based on 'vibes' rather than data.

    • ✗ A/B test the new prompt in production for one week

      While A/B testing is a valid final step, it should not be the first check, as it exposes live users to potentially regressed behavior.

    • ✗ Ask the model to self-evaluate if its new instructions are clearer

      Models cannot objectively evaluate the quality of their own instructions; this provides no verifiable data on task accuracy.

  11. 11 In the context of observability, what is the most important metric for making high-level architectural decisions regarding system efficiency?

    Think about a metric that combines operational cost with successful business outcomes.

    Cost-per-completed-task

    Tracking the total resources and costs required to finish a specific unit of work (like a support ticket) is the best driver for architectural choices.

    • ✗ Latency per request

      Latency is critical for user experience, but it doesn't provide the full picture of architectural efficiency regarding value delivered.

    • ✗ Tokens per request

      Tokens per request is an intermediate metric; the focus should be on the total cost to achieve the successful outcome.

    • ✗ Cache hit percentage

      Cache hit rates help optimize a specific feature but don't represent the overall business value or task-level efficiency.

  12. 12 When configuring A/B testing for a Claude-powered application, which practice is essential for accurately identifying the cause of an improvement?

    Focus on the principle of variable isolation in experimental design.

    Changing only one variable per experiment (e.g., prompt version)

    Isolating a single variable ensures that any detected change in metrics like accuracy or latency is caused by that specific modification.

    • ✗ Testing multiple prompt versions and a model tier change simultaneously

      Changing multiple variables at once makes it impossible to attribute a performance shift to a specific change.

    • ✗ Relying on qualitative user feedback to determine the winner

      Qualitative feedback is subjective; A/B tests should be anchored to predefined, objective success metrics.

    • ✗ Ending the test as soon as one variant appears to be leading

      Tests must run until they reach a predefined sample size or statistical significance to avoid making decisions based on random noise.

  13. 13 An architect is evaluating a RAG system where fact lookups are consistently failing despite relevant documents being in the corpus. Which adjustment to chunking strategy is most appropriate?

    Match the granularity of the chunks to the specific nature of the user's information needs.

    Decreasing chunk size to the sentence or paragraph level

    Small chunks provide precise semantic matching for specific fact lookups, which is ideal for pinpointing information in a large corpus.

    • ✗ Increasing chunk size to include full sections

      Larger chunks can dilute the relevance score for specific facts and increase the noise passed to the model.

    • ✗ Removing all chunk overlap to maximize distinct information

      Removing overlap can cause facts that span boundaries to be lost, worsening retrieval quality.

    • ✗ Switching from semantic search to keyword-only search

      Keyword search misses semantic nuances; it is rarely a solution for general fact lookup failures compared to optimizing chunk granularity.

  14. 14 To maintain developer trust in an automated code review system with a high false positive rate in certain categories, what is the best immediate architectural action?

    Think about how to preserve the perceived value of the system's accurate findings.

    Temporarily disable the high false-positive categories while improving prompts

    Prioritizing precision over recall by removing unreliable categories protects developer trust while improvements are made offline.

    • ✗ Instruct developers to ignore the high-noise categories

      Leaving high-noise categories active undermines confidence in the entire system's accuracy.

    • ✗ Add a 'be conservative' instruction to the general system prompt

      Vague instructions like 'be conservative' fail to improve precision and are less effective than specific categorical criteria.

    • ✗ Implement a human-in-the-loop step for every single finding

      Requiring human review for every routine action destroys the efficiency gains that justify the system's existence.

  15. 15 You are implementing a self-correction validation flow for an extraction task. Which design pattern provides the best feedback to the model for retries?

    Look for a method that uses internal consistency checks to identify failures.

    Extracting 'calculated_total' and 'stated_total' to flag discrepancies

    Designing fields that allow for internal cross-validation gives the system a deterministic way to detect errors and request corrections.

    • ✗ Resubmitting the same prompt and expecting a better result

      Retrying without providing specific error feedback rarely results in a correction of the original mistake.

    • ✗ Instructing the model to 'be more careful' on the next attempt

      Instructional modifiers are too vague; effective retries require structural or semantic error feedback.

    • ✗ Increasing the model's temperature setting for the retry

      Increasing temperature increases randomness, which is generally counter-productive for data extraction tasks requiring precision.

  16. 16 What is the primary risk of relying on aggregate accuracy metrics (e.g., $97\%$ overall) to validate a production deployment?

    Consider how a single number might hide failures in localized subsets of the data.

    Aggregate metrics can mask poor performance on specific document types or fields

    A high overall average can hide the fact that the system fails $50\%$ of the time on a specific, high-risk category of data.

    • ✗ The metric does not account for the cost of the tokens used

      Cost is important, but the primary quality risk of aggregate metrics is related to performance gaps in specific areas.

    • ✗ Aggregate metrics are only applicable to binary classification tasks

      Aggregate metrics can be calculated for many tasks, but their breadth is what makes them potentially misleading.

    • ✗ High accuracy numbers may indicate that the test set is too small

      While small test sets are an issue, the danger of an 'aggregate' metric is specifically its failure to show per-slice performance.

  17. 17 When using 'LLM-as-judge', how is the reliability of the automated judge typically verified and maintained?

    Identify the 'gold standard' used to check the quality of automated assessments.

    By periodic human review to calibrate judge scores against human ratings

    Comparing automated scores with human judgment helps ensure the judge remains aligned with expected quality standards.

    • ✗ By ensuring the judge uses the same prompt as the candidate

      The judge needs a grading rubric, which is a completely different prompt than the one used by the candidate.

    • ✗ By calculating the F1 score of the judge's responses

      F1 scores are for classification; judging open-ended quality requires calibration against a gold standard (humans).

    • ✗ By checking the judge's latency against a predefined SLA

      Latency is an operational metric, but it does not verify the reliability or accuracy of the judge's qualitative scores.

  18. 18 In a diagnostic scenario where a model fails only on complex multi-step reasoning while succeeding on simple tasks, which fix should be tested first?

    Consider whether the task requirements are out-stripping the current model's processing depth.

    Test the same prompt on a higher-capability model tier

    Model mismatch is a common cause of reasoning failures; verifying performance on a stronger tier confirms if the task exceeds the current model's capacity.

    • ✗ Rewrite the prompt to include more few-shot examples

      Few-shot examples help with format and edge cases but don't always provide the fundamental reasoning boost needed for complex logic.

    • ✗ Increase the retrieval top-k to provide more context

      Adding context doesn't solve a reasoning deficiency and may actually increase the complexity the model must manage.

    • ✗ Implement prompt caching to reduce token overhead

      Caching is for cost and latency optimization and has no impact on the model's reasoning capabilities.

  19. 19 An architect is tracking the performance of a high-volume extraction agent. Which sampling technique is recommended for detecting novel error patterns in high-confidence outputs?

    Think about how to ensure that diverse categories of data are checked, regardless of their overall frequency.

    Stratified random sampling across different document types

    Segmenting the data ensures that even rare or difficult document categories are represented in the audit, revealing novel errors.

    • ✗ Random sampling of all incoming requests

      While simple, random sampling may not be efficient enough to catch specific failures in high-confidence slices.

    • ✗ Reviewing only the lowest-confidence outputs

      Low-confidence reviews are good for routing, but you must also sample high-confidence outputs to detect silent failures (hallucinations).

    • ✗ Selecting the first $100$ requests of each day for review

      Time-based sampling can be biased by patterns in traffic at different times of day and is less robust than stratified sampling.

  20. 20 Which optimization lever should be applied to reduce perceived latency in a user-facing chat application without changing the model tier?

    Consider a technique that changes how data is delivered to the user rather than how it is calculated.

    Implement response streaming

    Streaming sends the output token-by-token as it's generated, significantly reducing 'time-to-first-token' and improving the user's experience of speed.

    • ✗ Switch to non-streaming response mode

      Non-streaming increases perceived latency because the user must wait for the entire response to be generated before seeing anything.

    • ✗ Use the Message Batches API for all requests

      The Batch API can take up to $24$ hours to process and is inappropriate for interactive, real-time user chat.

    • ✗ Increase the system prompt length to include more safety guardrails

      Longer prompts increase processing time and input token costs, which generally increases latency.