20 Questions — Domain 5 : Governance, Safety & Risk Management Quiz

Expand any question to reveal the correct answer and explanation.

1. 1 A financial services firm is deploying a Claude-powered agent to assist account managers. The agent is configured with the `execute_trade` and `update_customer_tier` tools. To mitigate the risk of unauthorized trades, the lead developer proposes adding a mandatory confirmation prompt to the user interface. According to the principle of least privilege, what is the most effective architectural action?

   Consider the difference between managing a risk through observation versus eliminating the risk's source.

   Restrict the agent's toolset to only include capabilities required for its specific role, removing `execute_trade` if it is not essential.

   Least privilege is achieved by removing unnecessary capabilities entirely at the configuration level, effectively eliminating the risk of misuse.

   • ✗ Implement comprehensive audit logging for every trade executed by the agent to enable forensic review.

     Logging is a detective control that records events after the fact rather than preventing unauthorized access or reducing the attack surface.

   • ✗ Add explicit instructions to the system prompt telling Claude never to execute a trade without verified verbal consent.

     Prompt-based constraints are probabilistic and can be bypassed by injection or model errors, making them insufficient for critical security enforcement.

   • ✗ Configure a human-in-the-loop gate that requires a senior manager to approve every trade requested by the agent.

     While effective for high-impact actions, a confirmation step is a compensating control that does not address the underlying issue of an over-permissioned agent.

2. 2 In a highly regulated legal research application, the development team reports that Claude occasionally ignores system instructions when processing large, user-provided PDF files containing adversarial text like 'Ignore all previous rules.' Which structural defense is most robust against this type of prompt injection?

   Think about how to transform a flat stream of tokens into a structured format that distinguishes instructions from data.

   Wrapping all untrusted user content in clear XML boundary delimiters and instructing the model to treat the enclosed text as data only.

   Boundary markers act as a programmatic sandbox, explicitly signaling to the model that the content within them is untrusted and should not be executed as instructions.

   • ✗ Increasing the model temperature to ensure the agent remains creative enough to identify and ignore conflicting instructions.

     Higher temperature increases randomness and unpredictability, which typically worsens the model's adherence to safety constraints and increases injection risk.

   • ✗ Appending a high-priority safety reminder to the end of every user message to reinforce the system prompt's authority.

     While recency bias helps, simply repeating instructions does not create the structural isolation needed to prevent a flat token stream from being misinterpreted.

   • ✗ Pre-scanning the uploaded text with a regex-based filter to detect and redact phrases like 'ignore previous instructions.'

     Keyword filtering is easily bypassed by paraphrasing or creative encoding and does not address the fundamental vulnerability of instruction-data confusion.

3. 3 A healthcare organization is building an automated summary tool for patient records. Initial testing shows a $97\%$ overall accuracy rate across all fields. Why should the architect insist on a stratified analysis of these metrics before deploying the solution?

   A high average score can hide a failing grade on the most important section of an exam.

   To identify if specific document types or critical fields like 'Allergies' suffer from catastrophic failure rates masked by high overall performance.

   Aggregate metrics can mask poor performance on critical subsets; a stratified review ensures reliability across every specific segment before automation.

   • ✗ To ensure that the latency for processing minority document types does not exceed the defined service-level agreement.

     While latency is important, the primary risk with aggregate accuracy is the hidden failure rate in specific data segments, not speed.

   • ✗ To calculate the total token consumption per document type to optimize the operating cost of the model tier.

     Cost optimization is a separate objective and does not justify the need for segmented accuracy validation for safety purposes.

   • ✗ To determine the optimal chunking granularity for the RAG pipeline based on the length of different patient files.

     Chunking strategies improve retrieval but do not address the governance requirement of verifying accuracy across demographic or document-type slices.

4. 4 An architect is designing an agent for an insurance company that can process claims and issue payments. Which of the following scenarios absolutely requires a Human-in-the-Loop (HITL) gate rather than an automated check?

   Identify which action carries the highest risk and cannot be easily undone if the model makes a mistake.

   Executing an irreversible final payment of $\$25,000$ to a third-party vendor after a claim is approved.

   High-impact, irreversible actions require human oversight to prevent financial loss and ensure accountability in the event of an error.

   • ✗ Validating that the claimant's address in the policy matches the address provided in the current claim form.

     Address validation is a deterministic task that can be handled through programmatic comparisons or low-risk model checks.

   • ✗ Summarizing a ten-page accident report to extract the names of involved witnesses and vehicle identification numbers.

     Summarization is a routine cognitive task where automated evaluation and sampling are generally sufficient to manage quality.

   • ✗ Routing a claim to the correct internal department based on the detected category of the insurance policy.

     Routing is a low-risk classification task that can be automated, with errors being easily corrected by the receiving department.

5. 5 When deploying a Claude solution in a US federal government environment, which compliance framework is most critical for authorizing the cloud service for operation?

   This framework is a prerequisite for any cloud service provider selling to the US federal government.

   FedRAMP

   FedRAMP is the mandatory US federal program providing a standardized approach to security assessment and authorization for cloud products and services.

   • ✗ GDPR

     GDPR is a European regulation focused on the personal data of EU residents, not the authorization of cloud services for US federal agencies.

   • ✗ HIPAA

     HIPAA is a US law protecting health information, relevant for healthcare entities but not the primary authorization for federal cloud deployments.

   • ✗ SOC 2 Type II

     While SOC 2 is a common enterprise security standard, it does not replace the specific FedRAMP requirement for US federal agency operations.

6. 6 A multi-agent research system uses a coordinator to delegate tasks to specialized subagents. During an audit, it is found that sensitive user preferences are being leaked into the logs of a subagent that only requires access to public web data. What is the root cause of this security gap?

   Look for a systemic failure in managing permissions and boundaries between the primary agent and its workers.

   The coordinator is using an over-privileged service account that shares all user session data with every spawned subagent.

   Using a shared, over-privileged account erases per-user access boundaries and prevents meaningful audit trails for individual subagent actions.

   • ✗ The subagent is running on a model tier that is too high, leading to excessive reasoning over sensitive data.

     Model tier capability is unrelated to the data isolation and context management strategies of the orchestration layer.

   • ✗ The subagent does not have a comprehensive system prompt instructing it to ignore sensitive user information.

     Instructions are probabilistic; the root cause is a failure in structural authorization and data isolation at the infrastructure level.

   • ✗ The RAG pipeline is incorrectly chunking private documents and storing them in the public vector database.

     While possible, the scenario specifies that the leak occurs through the subagent's session context during orchestration, not the database.

7. 7 An architect is configuring safety guardrails for an AI chatbot. Which layer is considered the final deterministic check before a response is delivered to the end user?

   Consider the order of operations and which layer has the 'last look' at the content.

   Output filtering and moderation layers.

   Output filtering acts as a programmatic gate that scans the model's generated response to block harmful content before it reaches the user.

   • ✗ Input validation and injection screening.

     Input validation occurs before the model processes the request and cannot account for harmful content generated by the model itself.

   • ✗ System prompt constraints and refusal behavior.

     System prompts are part of the model's generation process and remain probabilistic, meaning the model may still produce harmful output.

   • ✗ Tool permission scoping and least privilege.

     Tool scoping limits what an agent can *do* in the external world but does not filter the text the agent *says* to the user.

8. 8 A developer wants to reduce false positives in a code-review agent that flags security vulnerabilities. Which prompting strategy will most reliably improve precision?

   Identify the difference between asking for a 'vibe' of caution versus providing a 'rubric' for success.

   Providing explicit categorical criteria and examples of what constitutes a 'valid flag' versus a 'non-vulnerable pattern.'

   Explicit, testable criteria replace ambiguity with clear boundaries, allowing the model to apply more precise logic and reduce noise.

   • ✗ Adding the instruction 'be conservative and only report findings you are very confident in' to the system prompt.

     Vague instructions like 'be conservative' are subjective and fail to provide the model with the clear logic needed to filter out false positives.

   • ✗ Enabling chain-of-thought reasoning so the model explains its logic before every flag it raises.

     Chain-of-thought can improve reasoning but does not inherently reduce false positives if the underlying criteria for what to flag remain vague.

   • ✗ Running the code through three identical Claude instances and only flagging vulnerabilities that all three identify.

     Consensus between identical models can still result in common hallucinations or failures if the prompt itself is flawed or ambiguous.

9. 9 In the context of ethical AI and bias management, why is it critical to validate accuracy by document type and field segment before automating a document extraction workflow?

   Think about how a single high score might hide a 'failing grade' for a smaller, different-looking group of documents.

   To prevent aggregate accuracy metrics from masking significant performance disparities that may unfairly impact certain groups or data formats.

   Bias often manifests in specific subsets of data; validating per segment ensures that the system performs equitably across all document variations.

   • ✗ To comply with GDPR's right to erasure by ensuring that all fields can be successfully deleted upon request.

     Segmented accuracy is a quality and fairness concern, whereas the right to erasure is a data management and compliance requirement.

   • ✗ To ensure the RAG pipeline utilizes the most efficient embedding model for each specific document category.

     Embedding efficiency is a performance optimization and does not directly address the fairness or accuracy risks associated with automation.

   • ✗ To determine the exact token limit required for the context window based on the most complex document type.

     Calculating context window size is a technical constraint and is unrelated to the ethical imperative of measuring disparate performance outcomes.

10. 10 An architect is building a RAG-based support system. The team discovers that an attacker can embed malicious instructions in a public support forum post that the system then retrieves and executes. What is this vulnerability called?

    Consider the source of the malicious instructions and how they reached the model.

    Indirect Prompt Injection

    Indirect injection happens when the model retrieves untrusted content (via tools or RAG) that contains hidden instructions designed to manipulate it.

    • ✗ Direct Prompt Injection

      Direct injection occurs when the user themselves submits instructions to the model; this scenario involves a third-party source.

    • ✗ Data Leakage

      Data leakage is the unauthorized exposure of sensitive information, which is a potential *result* of an injection, not the injection itself.

    • ✗ Recursive Jailbreaking

      Jailbreaking refers to bypassing safety guardrails, but 'indirect injection' specifically describes the delivery mechanism of the attack.

11. 11 A system requires an audit trail of every model interaction to meet compliance standards. Which logging strategy is the most complete and defensible for an auditor?

    What information would an investigator need to perfectly recreate a specific past event?

    Recording the full request and response bodies, including system prompts, tool calls, model versions, and metadata.

    Full payload logging is essential for reconstruction and forensic analysis, ensuring that auditors can see exactly what the model was told and what it did.

    • ✗ Logging the user's initial question and the final response provided by the agent.

      This ignores intermediate tool calls, system prompts, and model versions, which are necessary to reconstruct the full context of an action.

    • ✗ Storing the unique Request IDs and timestamps for every interaction to correlate with system performance logs.

      While useful for performance, IDs and timestamps alone do not provide the content needed to verify what actually occurred during the interaction.

    • ✗ Hashing all prompts and responses before storage to protect user privacy while maintaining a record of the interaction.

      Hashing prevents the content from being read, which defeats the purpose of an audit trail that requires investigation into the actual behavior of the system.

12. 12 An agent is responsible for processing sensitive European customer data. To adhere to GDPR's 'data minimization' principle, what should be the architect's primary strategy when designing the system context?

    This principle focuses on processing only the 'bare essentials' needed for a specific job.

    Send only the specific fields required for the current task to the model and redact or exclude irrelevant PII from the context.

    Data minimization requires limiting personal data processing to what is strictly necessary; excluding irrelevant fields directly fulfills this principle.

    • ✗ Encrypt all customer data using a private key before appending it to the conversation history.

      Encryption protects data but does not meet the goal of minimization, which is to limit the amount of data processed in the first place.

    • ✗ Ensure that the agent is running on the largest available context window to avoid truncating any customer information.

      Processing *more* information is the opposite of minimization and increases the risk and token cost of the interaction.

    • ✗ Automatically summarize all customer documents into a single paragraph before they reach the model.

      Summarization can lose critical details (the 'progressive summarization trap') and doesn't necessarily remove sensitive PII if it's included in the summary.

13. 13 A team is building an agent to automate corporate expense approvals. They decide that any expense over $\$500$ must be routed to a human manager. Where should this logic be enforced to be considered a 'deterministic gate'?

    Differentiate between a rule the model 'should' follow versus a wall it 'cannot' pass.

    Inside the code logic of the `approve_expense` tool, which returns an error or a 'pending approval' state if the amount is too high.

    Code-level enforcement is deterministic; the model cannot 'reason' its way past a hard-coded check in the tool's backend logic.

    • ✗ In the system prompt, as a strict rule: 'You MUST NOT approve any expense over $\$500$.'

      Prompt rules are probabilistic and can be bypassed by the model or by adversarial user input.

    • ✗ By providing few-shot examples of the agent correctly escalating expenses above the threshold.

      Examples guide behavior but do not create a hard barrier that prevents the model from failing in novel or complex scenarios.

    • ✗ By using a second Claude instance to review the first agent's decisions before they are finalized.

      While a useful check, using another LLM layer remains probabilistic and is less reliable than a simple programmatic check.

14. 14 A model's self-reported confidence score for an extraction task is $99\%$, but the actual accuracy on that field is $75\%$. What does this indicate about the system's human-in-the-loop strategy?

    Consider why an 'expert' who is always sure but often wrong is dangerous for automation.

    The architect should implement deterministic confidence thresholds using a separate validation set for calibration.

    Since models are often poorly calibrated, review routing should be based on thresholds validated against labeled data, not the model's raw self-report.

    • ✗ The model is well-calibrated and can be trusted to route its own review attention.

      The disparity between self-reported confidence and actual accuracy shows the model is poorly calibrated and overconfident.

    • ✗ The system prompt should be updated to tell the model to 'be more honest' about its mistakes.

      Subjective instructions do not solve the underlying mathematical miscalibration of the model's internal probability distributions.

    • ✗ The agent should be downgraded to a smaller model to reduce the cost of the incorrect extractions.

      Downgrading doesn't fix the calibration problem and would likely result in even lower accuracy.

15. 15 Which layer of the safety stack is primarily responsible for ensuring that an agent cannot take unauthorized actions like deleting a database, even if it is successfully 'jailbroken' through prompt injection?

    Think about the difference between 'telling' someone not to do something and 'locking the door' so they can't.

    Programmatic tool permission scoping and service-level access controls.

    If the agent's credentials do not have permission to delete the database, no amount of prompt manipulation can force the action to succeed.

    • ✗ The system prompt's list of prohibited actions.

      A jailbreak by definition bypasses the system prompt, making this layer ineffective in such a scenario.

    • ✗ Output filtering for keywords like 'DELETE' or 'DROP TABLE.'

      Filtering keywords can be bypassed by creative phrasing or encoding and does not restrict the actual permissions of the underlying connection.

    • ✗ A human-in-the-loop confirmation message displayed after the injection occurs.

      If the action is already triggered by a tool call, a UI-level confirmation may be bypassed or ignored by a malicious autonomous loop.

16. 16 When designing an AI system for a healthcare provider, the architect must ensure that no Protected Health Information (PHI) is retained in the model's logs or training data. Which control is most relevant for meeting this HIPAA requirement?

    How can you ensure a secret is never kept if you never share it in the first place?

    Implementing a redaction pipeline to scrub PII/PHI before payloads are sent to the model API.

    Removing sensitive data before it reaches the external provider is the most direct way to ensure it is not logged or retained improperly.

    • ✗ Using $256$-bit AES encryption for all data at rest and in transit.

      Encryption is a standard security requirement but does not address the specific mandate to prevent unauthorized *retention* or use for training.

    • ✗ Ensuring the system prompt contains a strict 'do not log' command for the model to follow.

      Models do not control the logging infrastructure of the API provider; this is an architectural configuration, not a prompt instruction.

    • ✗ Requiring all users to sign a Non-Disclosure Agreement (NDA) before using the application.

      NDAs are legal agreements for humans and do not provide technical controls over how data is processed by an LLM.

17. 17 An architect is reviewing a design where a customer support agent has tools to `check_order_status`, `update_shipping_address`, and `delete_user_account`. The architect recommends removing the `delete_user_account` tool entirely for this role. What is the best justification for this change?

    Consider the impact of a 'jailbroken' agent that still has access to that specific tool.

    It implements true least privilege by removing a high-risk capability that the agent's role doesn't require.

    Removing unused, high-risk tools shrinks the attack surface and prevents accidental or malicious execution of irreversible actions.

    • ✗ It reduces the token cost of the system prompt by shortening the tool catalog.

      While true, cost reduction is a minor benefit compared to the fundamental security and risk management reasons for the change.

    • ✗ It improves the model's selection accuracy by reducing the number of overlapping tool descriptions.

      The scenario doesn't suggest that the shipping tool and delete tool overlap; the change is primarily about security, not routing accuracy.

    • ✗ It simplifies the logging requirement since the system no longer needs to track account deletions.

      Logging is a detective control; the goal of removal is prevention of the risk itself, not the simplification of the logs.

18. 18 A multi-agent system uses a synthesis agent to combine results from five different subagents. To ensure transparency and prevent hallucinations during the final step, what is the best architectural practice?

    Think about how an author proves their work is accurate by using footnotes and references.

    Require subagents to return structured JSON metadata including citations and source IDs, which the synthesis agent must include in its response.

    Structured metadata allows for explicit attribution and verification, preventing the synthesis agent from 'collapsing' details into vague or fabricated summaries.

    • ✗ Instructions to the synthesis agent to 'be as detailed as possible' and avoid making things up.

      Vague instructions do not provide the structural grounding needed to prevent hallucinations in a long multi-agent pipeline.

    • ✗ Using a faster, smaller model for synthesis to minimize the time available for the model to hallucinate.

      Smaller models typically have *lower* reasoning capabilities and are more prone to errors and hallucinations than larger, more capable ones.

    • ✗ Allowing the synthesis agent to access the raw conversation history of all subagents to ensure it has the 'full context.'

      This results in monolithic context bloat, which degrades attention and actually *increases* the risk of the model missing key instructions or findings.

19. 19 An agent that helps employees draft internal emails is found to be leaking confidential project names into general responses. Investigation reveals that these names were present in the tool outputs of a previous search task. What is the recommended fix?

    This fix involves 'cleaning up' the information the model is allowed to remember.

    Trim tool outputs before appending them to the history, keeping only the specific fields required for the current task.

    Aggressive context management ensures that sensitive but irrelevant data never stays in the conversation history, reducing both token cost and risk.

    • ✗ Implement output filtering to redact any text that matches a list of known project code names.

      Filtering is a detective/compensating control that is difficult to maintain and does not address the source of the bloat in the context history.

    • ✗ Add a disclaimer to every email drafted by the AI stating that it may contain confidential information.

      A disclaimer is a legal fallback and does not provide a technical control to prevent the actual leak of data.

    • ✗ Upgrade the model to a tier with better 'reasoning' to help it understand which projects are confidential.

      Models struggle to distinguish 'sensitive' from 'relevant' unless the context is explicitly managed or the data is excluded entirely.

20. 20 Which ethical AI checkpoint is being addressed when an architect conducts a red-teaming exercise to see if an agent can be manipulated into providing biased advice depending on the regional dialect used in the user's prompt?

    Think about the goal of ensuring the system works equally well for all types of users.

    Fairness and Bias detection

    Testing for variations in quality or advice across demographic or linguistic slices is a core component of evaluating AI fairness.

    • ✗ Transparency

      Transparency focuses on explaining how decisions are made, not necessarily testing for disparate outcomes across dialects.

    • ✗ Data Privacy

      Bias testing does not directly address data privacy concerns like PII protection or encryption.

    • ✗ Accountability

      While related, accountability usually refers to the human-in-the-loop structures and legal responsibility for the system's actions.