20 Questions — Domain 2 : Claude Models, Prompting & Context Engineering Quiz

Expand any question to reveal the correct answer and explanation.

1. 1 A solution architect is optimizing a production system using Claude 3.5 Sonnet that processes a static 800-token system prompt and a 150-token tool definition schema before each user request. Despite enabling prompt caching, the logs indicate a $0\%$ cache hit rate. What is the most likely cause?

   Check the specific token requirements for the Sonnet tier's caching feature.

   The total stable prefix length is below the minimum cacheable threshold for the Sonnet tier.

   For the Sonnet and Haiku tiers, the minimum cacheable prefix length is $1024$ tokens; the combined $950$ tokens in this scenario fail to trigger the caching mechanism.

   • ✗ Prompt caching is only supported for the Claude 3 Opus model tier.

     Caching is supported across Opus, Sonnet, and Haiku tiers, though the minimum token requirements vary between them.

   • ✗ The default TTL of 5 minutes is likely expiring between infrequent user requests.

     While TTL can cause misses, the prompt size itself is below the architectural requirement for the specific model tier to begin caching.

   • ✗ Tool definitions cannot be cached and must be placed after the cache breakpoint.

     Tool definitions are highly recommended for caching and should be placed early in the stable prefix to maximize cost savings.

2. 2 An enterprise is designing a high-volume triage system where latency is the primary success pillar. The current architecture uses a single Claude 3 Opus instance. Which architectural refinement would best optimize for both cost and speed without a quality cliff?

   Consider how different model tiers can be combined to handle varying task complexities.

   Implement a routing layer using Haiku to escalate only complex queries to a more capable tier.

   Routing high-volume, well-bounded tasks to a fast tier while reserving premium tiers for difficult slices maintains quality while significantly reducing latency and spend.

   • ✗ Switch the entire workload to Sonnet and disable streaming to reduce overhead.

     Disabling streaming actually increases perceived latency for users, and a single-model approach misses the optimization benefits of a tiered routing strategy.

   • ✗ Increase the retrieval count in the RAG pipeline to provide Opus with more context.

     Adding more context to a premium model increases token costs and processing time, which contradicts the goal of optimizing for speed and cost.

   • ✗ Use the Message Batches API to process user-facing triage requests in parallel.

     The Batch API has a $24$-hour processing window, making it unsuitable for real-time triage where immediate response is required.

3. 3 When configuring prompt caching for a multi-turn agentic loop, which ordering of prompt components is considered an architectural best practice to ensure the highest cache hit rate?

   The most stable information must be processed before the data that changes with every turn.

   System Instructions $\rightarrow$ Tool Definitions $\rightarrow$ Static Knowledge $\rightarrow$ Dynamic History.

   Structuring the prompt with the most stable content first ensures that the prefix remains identical across calls, allowing the cache to persist even as dynamic history grows.

   • ✗ User Message $\rightarrow$ System Instructions $\rightarrow$ Tool Definitions $\rightarrow$ Static Knowledge.

     Placing the variable user message at the beginning of the prompt invalidates the cache for all subsequent static content due to the exact-prefix matching rule.

   • ✗ Tool Definitions $\rightarrow$ Dynamic History $\rightarrow$ System Instructions $\rightarrow$ User Message.

     Mixing dynamic content into the middle of the prefix prevents the caching of stable instructions that follow it.

   • ✗ Static Knowledge $\rightarrow$ User Message $\rightarrow$ Dynamic History $\rightarrow$ Tool Definitions.

     This structure fragments the stable content, meaning only the first block would be eligible for caching before the user message breaks the prefix.

4. 4 A developer is using few-shot prompting to improve a Claude-based data extraction tool. To minimize hallucinations when specific information is missing from a source document, which type of example should be included in the few-shot block?

   Think about how to show the model what to do when it cannot find the requested information.

   A 'correct rejection' example where the model returns a null or empty value for absent data.

   Including examples that demonstrate what to leave alone or how to handle missing data teaches the model to generalize judgment and reduces the likelihood of fabrication.

   • ✗ An example using Chain-of-Thought to guess the most likely missing value based on context.

     Encouraging the model to guess missing information directly increases the risk of hallucination, which is the failure mode being addressed.

   • ✗ A summary of the document's main themes to provide a broader context window.

     Broad context does not address the specific behavior of how to handle a missing field during a structured extraction task.

   • ✗ Multiple examples of similar valid data to reinforce the expected data types.

     Reinforcing only successful extractions doesn't provide the model with a boundary for when it should refrain from generating an answer.

5. 5 In a Claude 3 Opus-powered system, a stable $3000$-token prefix is cached. If a developer accidentally inserts a dynamic 'Current Timestamp' variable into the middle of the system prompt (at token $500$), what is the immediate impact on prompt caching performance?

   Recall the exact-prefix matching rule and how modifications affect subsequent tokens.

   The entire cache downstream from token $500$ is invalidated for every request.

   Any modification occurring before a cache breakpoint changes the exact prefix, which invalidates all subsequent cached content regardless of its length.

   • ✗ The system will automatically create a new cache sub-layer for the timestamp.

     Claude's caching system relies on strict prefix matching and does not support automatic sub-layering for dynamic variables inserted within a prefix.

   • ✗ Only the specific $500$-token block containing the timestamp will be billed at the full rate.

     Caching is not block-based in a way that allows subsequent identical tokens to remain cached if the preceding prefix has changed.

   • ✗ The cache hit rate will remain unaffected if the timestamp is less than $10$ tokens.

     The length of the change is irrelevant; any change to the prefix, even a single character, results in a cache miss.

6. 6 Which prompting technique is specifically recommended when a task involves multi-step logic and the architect is willing to trade higher output token counts and latency for improved reasoning accuracy?

   This technique involves the model explaining its work before providing the final answer.

   Chain-of-thought prompting.

   Asking the model to reason step-by-step before answering improves performance on complex logic, though it increases the number of generated tokens and overall response time.

   • ✗ Zero-shot prompting with explicit criteria.

     Zero-shot is best for simple, unambiguous tasks and does not provide the model with a structured reasoning path for complex logic.

   • ✗ Few-shot prompting with formatted examples.

     Few-shot is primarily used for teaching format and edge-case judgment rather than guiding the model through a step-by-step reasoning process.

   • ✗ Modular system prompts using Claude Skills.

     Claude Skills are for packaging reusable instructions and do not inherently force the model to perform intermediate reasoning steps for a specific query.

7. 7 An architect notices that a Claude-based system frequently fails to follow instructions located in the middle of a $150,000$-token context window. This phenomenon is known as the:

   The term describes where the model's attention is weakest in a long document.

   'Lost in the middle' effect.

   Models reliably process information at the beginning and end of long inputs but often exhibit degraded attention and omit findings from the middle sections.

   • ✗ Context window saturation.

     Saturation implies the window is full, whereas this specific issue describes the loss of focus on content regardless of remaining capacity.

   • ✗ Token dilution error.

     This is not a standard term used in Claude documentation to describe attention decay within a large context window.

   • ✗ Progressive summarization risk.

     While summarization can lose details, it is a manual process or strategy, not the inherent model behavior of missing information in large inputs.

8. 8 When designing system prompts for high-stakes enterprise applications, which type of instruction is most effective for improving precision and developer trust?

   Avoid vague adjectives and focus on instructions that can be clearly verified.

   Explicit, categorical, and testable criteria.

   Defining specific boundaries and testable rules is superior to vague instructions, as it allows for verifiable and reliable model behavior.

   • ✗ General boundary statements like 'be conservative'.

     Vague instructions like 'be conservative' are subjective and fail to provide the model with actionable guidance to reduce false positives.

   • ✗ Implicit role-play definitions without constraints.

     Defining a role without setting explicit constraints leads to unpredictable behavior and lack of reliability in production environments.

   • ✗ High-level guidance to 'only report high-confidence findings'.

     Self-reported confidence is often poorly calibrated in models, making this an unreliable instruction for systemic precision.

9. 9 To standardize recurring tasks and prevent 'prompt bloat' where thousands of tokens of instructions are copy-pasted into every request, which Claude feature should an architect utilize?

   This feature packages specialized rules into reusable, modular sets.

   Claude Skills.

   Skills provide modular, reusable instruction sets that help Claude perform specialized tasks consistently while keeping individual request prompts lean.

   • ✗ Few-shot examples with structured JSON.

     While few-shot improves consistency, copy-pasting them into every request actually contributes to prompt bloat rather than solving it.

   • ✗ The Message Batches API.

     The Batches API is for bulk asynchronous processing and does not provide a mechanism for modularizing or reusing prompt instructions.

   • ✗ Dynamic Context Truncation.

     Truncation is a method for managing long conversation histories, not for standardizing or modularizing system-level instructions.

10. 10 A nightly job processes $100,000$ extractions using Claude 3 Sonnet. The job is not time-sensitive but must stay within a strict budget. Which API feature provides the highest cost reduction for this specific scenario?

    This API feature is specifically designed for high-volume, non-urgent tasks with a major price discount.

    Message Batches API.

    The Batch API offers a $50\%$ cost discount for non-real-time workloads that can be processed within a $24$-hour window.

    • ✗ Prompt Caching.

      While caching saves money on repeated prefixes, the $50\%$ flat discount from the Batch API is typically the highest impact for bulk, non-real-time extractions.

    • ✗ Haiku-to-Sonnet Routing.

      Routing to a smaller model saves money but the Batch API provides a significant discount for the Sonnet model itself.

    • ✗ Context Window Truncation.

      Truncation reduces token count but doesn't offer a direct per-token pricing discount like the specialized Batch API.

11. 11 What is a major risk associated with 'progressive summarization' as a context management strategy in a complex customer support application?

    Consider what might happen to specific, factual details when a long conversation is turned into a short summary.

    Condensing critical values like order numbers and dates into vague prose references.

    Summarizing conversation history can strip out specific numerical data, causing the model to give confident but incorrect answers in later turns.

    • ✗ Exceeding the maximum token limit of the Claude 3.5 Sonnet tier.

      Summarization is intended to reduce token usage and prevent hitting the limit, so exceeding it is not the primary risk of the strategy itself.

    • ✗ Increasing the latency of the first-token response due to summary overhead.

      Summaries are generally shorter than raw history, which typically reduces rather than increases the processing time for the input prefix.

    • ✗ Violating the exact-prefix matching rule for prompt caching.

      Summarization happens at the dynamic end of the prompt; it doesn't inherently break the caching of the stable system prompt at the beginning.

12. 12 An architect is selecting a model for an LLM-as-judge evaluation framework to score the helpfulness of a Claude 3 Sonnet agent. According to best practices, which tier should the judge model belong to?

    To catch mistakes made by a certain model, you typically need a model with superior reasoning abilities.

    A stronger model tier, such as Claude 3 Opus.

    A judge must be at least as capable as the model it is evaluating to reliably identify errors and nuanced failures.

    • ✗ The same tier (Claude 3 Sonnet) to ensure parity.

      Using the same model often leads to 'self-evaluation bias' where the judge misses the same types of errors the candidate makes.

    • ✗ A faster, lower-cost tier like Claude 3 Haiku.

      Lower-tier models may lack the reasoning depth required to grade the output of a more sophisticated model accurately.

    • ✗ The choice of tier does not matter if the rubric is clear.

      Model reasoning capability is a critical factor in the effectiveness of an LLM-as-judge, regardless of the quality of the prompt rubric.

13. 13 Claude 3.5 Sonnet supports context windows up to $200,000$ tokens standard. If a use case requires processing a massive corpus of $800,000$ tokens, what is the primary architectural consideration regarding performance?

    Check the documentation regarding the $1$M token context window and its associated trade-offs.

    Pricing and attention dynamics may change on tiers supporting the $1$M context window.

    While $1$M windows are available on supported tiers, pricing can change above the standard $200$K threshold, and attention decay must be managed.

    • ✗ Claude cannot process more than $200,000$ tokens in any configuration.

      Claude supports up to $1$M tokens on specific tiers, so $200,000$ is not an absolute hardware limit for the model family.

    • ✗ Latency will decrease as the model uses parallel processing for large windows.

      Processing larger context windows significantly increases latency because the model must attend to more tokens simultaneously.

    • ✗ Prompt caching is disabled for any request exceeding $200,000$ tokens.

      Caching is an effective tool for large windows, allowing the reuse of massive prefixes and actually mitigating some of the cost of large requests.

14. 14 A developer needs to measure the exact number of tokens in a complex prompt to ensure it meets the minimum caching requirement. Which method is most reliable for Claude?

    Look for the specific API tool provided by Anthropic for this purpose.

    Using the dedicated 'count_tokens' endpoint.

    Token counting should always use the official endpoint, as character-based heuristics are inaccurate for LLM tokenization.

    • ✗ A standard character-to-token ratio (e.g., $4$ chars per token).

      Heuristics like $4$ characters per token are estimates and cannot be used for precise architectural planning or caching threshold validation.

    • ✗ Checking the word count in a text editor.

      Word counts are not equivalent to token counts, as tokenization often splits words or combines punctuation in unique ways.

    • ✗ Dividing the payload size in bytes by $8$.

      Byte size is related to encoding (like UTF-8) and does not correlate accurately with model-specific tokenization patterns.

15. 15 When implementing a retry strategy for a Claude-based production application, which HTTP error code should NOT trigger an automatic retry?

    This error category usually implies that there is a problem with the formatting or content of your request.

    $400$ - Bad Request.

    $4xx$ errors (excluding $429$) are client-side errors indicating the request itself is malformed; retrying without modification will result in the same failure.

    • ✗ $529$ - Overloaded.

      $529$ is a transient server error, and retrying with exponential backoff is a recommended strategy for handling it.

    • ✗ $429$ - Rate Limit.

      While it's a client error, a $429$ indicates a temporary threshold hit, making it a candidate for retrying after a jittered backoff period.

    • ✗ $500$ - Internal Server Error.

      Server errors are often transient and can be resolved by retrying the request after a short delay.

16. 16 Which header should be utilized to ensure safe retries on streaming requests and to enable better observability across a multi-step Claude solution?

    This header helps identify and track individual requests throughout their lifecycle.

    'anthropic-request-id'.

    Using unique request IDs allows for tracing and safe retries, ensuring idempotency and better debugging in complex production pipelines.

    • ✗ 'x-api-key'.

      The API key is for authentication only and does not provide session tracing or idempotency for specific request retries.

    • ✗ 'Content-Type: application/json'.

      This header defines the data format and does not contribute to request tracing or idempotent retry logic.

    • ✗ 'Cache-Control: max-age=3600'.

      This is a standard web caching header and is not used to manage Claude's internal prompt caching or request idempotency.

17. 17 In a RAG-based system, an architect finds that the model returns confident but incorrect answers following a document refresh. If the model version and latency are unchanged, what is the most likely root cause?

    Map the symptom (confident but wrong) to the component that was recently modified (the data).

    The retrieval/indexing step is returning irrelevant or stale chunks.

    Confidence combined with inaccuracy after a document update suggests the model is being fed poor context from the retrieval pipeline, likely due to a broken re-index.

    • ✗ The model weights have silently shifted during an update.

      Model weights are static for a specific version; unpinned aliases might cause drift, but 'document refresh' points directly to the data pipeline.

    • ✗ The temperature setting is suddenly too low for creative retrieval.

      Low temperature usually increases determinism and grounding; it wouldn't cause a sudden shift to 'confident but wrong' specifically after a data update.

    • ✗ The prompt caching TTL has expired.

      Expired TTL results in cache misses and higher costs/latency, but it does not affect the factual accuracy of the model's response.

18. 18 To minimize risk during the deployment of a new prompt version, an architect wants to run a 'regression suite'. Which evaluation methodology is most suitable for detecting subtle quality drops at scale?

    This methodology uses a second, more powerful model to grade the candidate's responses.

    LLM-as-judge using a rubric on a golden dataset.

    Automated scoring by a more capable model against a stable rubric is the most scalable way to catch nuanced regressions across large test sets.

    • ✗ Manual human review of every output in the suite.

      While human review is the gold standard, it is not scalable for frequent regression testing of large datasets during a deployment cycle.

    • ✗ Simple string-matching (regex) for expected keywords.

      String-matching is too brittle and cannot evaluate the 'subtle quality' or helpfulness that is critical in sophisticated AI systems.

    • ✗ Checking if the model refusal rate has decreased.

      Refusal rates are a safety metric, not a general quality metric; a decrease in refusals doesn't necessarily mean the quality of helpful responses has improved.

19. 19 A developer observes that their Claude-based chat application's refusal rate has climbed $30\%$ week-over-week without any changes to the system prompt. What should be their first investigative hypothesis?

    Check how the model is identified in the API request (e.g., is it using a generic name or a specific date?).

    An unpinned model alias (e.g., 'claude-3-sonnet-latest') was updated to a newer dated version.

    Using latest aliases can cause silent behavioral drift when Anthropic releases updates; pinning to a dated version ensures reproducible behavior.

    • ✗ The prompt cache TTL was accidentally reduced.

      TTL affects cost and performance, not the model's tendency to refuse requests based on safety or policy filters.

    • ✗ Users have collectively started submitting more abusive content.

      While possible, it is statistically less likely than a version update as the primary cause for a sudden, sharp, system-wide shift in behavior.

    • ✗ The context window has become too small to process the requests.

      Context window size is determined by the model and request, not a setting that would cause a $30\%$ increase in refusals without crashing the request.

20. 20 Which of the following is a critical 'trap' when designing multi-agent systems for latency-sensitive applications?

    Avoid redundant steps that move data back and forth between agents unnecessarily.

    Spawning a subagent to process data that is already available in the coordinator's context.

    Unnecessary delegation creates a bottleneck through the spawning and token transfer process, adding significant latency for no architectural benefit.

    • ✗ Using Claude 3 Opus as the coordinator for all tasks.

      While expensive, using a capable coordinator is a valid design choice; the 'trap' specifically refers to redundant and inefficient task spawning.

    • ✗ Configuring the 'allowedTools' parameter to include 'Task'.

      Including 'Task' is the required technical mechanism to enable subagent delegation, not an architectural trap.

    • ✗ Assigning 'citation_id' tags to source documents at the earliest stage.

      This is a best practice for maintaining provenance and attribution in multi-agent pipelines, not a latency-increasing trap.