Domain 5 — Mixed Scenario Quiz
Context Management & Reliability — Exam Style
Each question has one best answer.
Question 1: Progressive Summarization Risk
A support agent summarizes a long conversation as:
Later, it forgets the exact refund amount, delivery date, and customer-stated expectation.
What is the best fix?
A) Make the prose summary slightly longer.
B) Extract exact transactional facts into a persistent case_facts block included in each prompt.
C) Drop older messages entirely.
D) Ask the model to “remember important details.”
Answer
**B.** Critical facts like amounts, dates, order IDs, statuses, and customer commitments should be preserved in structured form outside the summarized history.Question 2: Multi-Issue Session
A customer discusses three orders: one delayed delivery, one missing item, and one refund dispute. The agent later mixes up which order had which problem.
What should the system maintain?
A) One paragraph summarizing the whole session.
B) A structured issues[] layer with order IDs, issue types, amounts, statuses, and open questions.
C) Only the latest user message.
D) Full raw tool output for every order lookup.
Answer
**B.** Multi-issue sessions need structured issue data so facts from separate issues do not merge.Question 3: Tool Output Bloat
An order lookup returns 40 fields, including warehouse routing, campaign IDs, debug data, and internal logistics metadata. The current task is refund eligibility.
What should the agent do?
A) Append all 40 fields to the conversation.
B) Trim the tool result to refund-relevant fields before adding it to context.
C) Discard the order lookup entirely.
D) Convert the result into “order seems refundable.”
Answer
**B.** Verbose tool outputs consume context disproportionately. Keep only fields relevant to the current task.Question 4: Lost in the Middle
A synthesis prompt contains 80 pages of subagent findings. The final answer omits an important issue buried around page 45.
What is the best mitigation?
A) Add more raw findings.
B) Put key findings at the beginning and organize detailed evidence under explicit section headers.
C) Remove all section headers.
D) Shuffle findings randomly.
Answer
**B.** Important findings should be front-loaded, with detailed evidence organized below to reduce position-effect omissions.Question 5: Stateless API Context
A developer sends only the latest user message in each API request and expects Claude to remember the prior conversation.
What is wrong?
A) Nothing; the API automatically remembers all prior messages.
B) The application must send the relevant conversation history or equivalent state on each request.
C) Claude cannot do multi-turn interactions.
D) Tool use prevents conversation history.
Answer
**B.** Conversational coherence requires passing the relevant history/state forward.Question 6: Subagent Metadata
A subagent returns:
The synthesizer cannot verify the claim.
What should the subagent have returned?
A) Only more prose.
B) Structured metadata: files, source locations, evidence, method, confidence, and relevance.
C) Nothing, because uncertainty is normal.
D) Only its internal reasoning chain.
Answer
**B.** Downstream synthesis needs source-grounded structured facts, not vague claims.Question 7: Explicit Human Request
A customer says:
The issue looks simple and could probably be resolved by the bot.
What should the agent do?
A) Try to resolve it first.
B) Escalate immediately.
C) Ask why they want a human.
D) Run sentiment analysis first.
Answer
**B.** An explicit request for a human is an immediate escalation trigger.Question 8: Frustration Without Human Request
A customer says:
They have not asked for a human, and order lookup is available.
What should the agent do?
A) Escalate based on negative sentiment.
B) Acknowledge frustration and offer to check the order status.
C) Refuse to continue.
D) Ask the customer to calm down.
Answer
**B.** Frustration alone is not an escalation trigger if the issue is straightforward and within the agent’s capability.Question 9: Policy Gap
A company policy covers price adjustments only when the company’s own website drops the price within 7 days. A customer asks for a competitor price match.
What should the agent do?
A) Apply the own-site policy to the competitor price.
B) Escalate because the policy is silent on competitor price matching.
C) Deny without explanation.
D) Guess based on customer sentiment.
Answer
**B.** Policy gaps and exception requests should be escalated rather than handled by heuristic extension.Question 10: Multiple Customer Matches
A customer lookup returns three accounts with the same name.
What should the agent do?
A) Pick the most recent account.
B) Pick the account with the largest order.
C) Ask for an additional identifier such as order number, email, phone, or billing ZIP/postal code.
D) Randomly choose one.
Answer
**C.** Multiple matches require clarification, not heuristic selection.Question 11: Confidence as Escalation Signal
The model says it is only 45% confident, but the policy clearly covers the request and all required facts are verified.
What should happen?
A) Escalate solely because confidence is low.
B) Resolve autonomously using the clear policy.
C) Refuse the request.
D) Ask the customer to restart the case.
Answer
**B.** Self-reported confidence is not a reliable proxy for actual case complexity.Question 12: Unresolved Ambiguity
The customer cannot provide enough identifying information to distinguish between multiple matching accounts after reasonable clarification attempts.
What should the agent do?
A) Guess the most likely account.
B) Escalate because ambiguity remains unresolved and progress is blocked.
C) Continue asking the same question indefinitely.
D) Choose the first result.
Answer
**B.** If identity ambiguity cannot be resolved, escalation is appropriate.Question 13: Generic Error Propagation
A subagent returns:
The coordinator cannot decide whether to retry, use another source, or synthesize partial results.
What is the best fix?
A) Terminate the entire workflow.
B) Return structured error context including failure type, attempted query, attempts made, partial results, and alternatives.
C) Hide the error and return empty results.
D) Ask the coordinator to guess.
Answer
**B.** Structured error context enables intelligent recovery decisions.Question 14: Empty Result vs Access Failure
A refund history query succeeds and returns zero matches.
How should the subagent report this?
A) As an access failure.
B) As a valid empty result with result_count: 0.
C) As a timeout.
D) As a policy gap.
Answer
**B.** A successful query with no matches is not an error.Question 15: Timeout
A policy database times out before returning results.
How should this be reported?
A) As result_count: 0.
B) As an access failure with retryability and attempts made.
C) As proof that no policy exists.
D) As success.
Answer
**B.** A timeout means the truth is unknown, not that no matching policy exists.Question 16: Partial Failure
A subagent searched three sources. Two succeeded and one failed.
What should it return?
A) Only the failure.
B) Only the successes, hiding the failure.
C) Partial results plus structured error details for the failed source.
D) Nothing.
Answer
**C.** Partial results and unresolved errors should both be propagated.Question 17: Local Recovery
A transient source timeout occurs.
What should the subagent do before propagating the error?
A) Immediately terminate the whole workflow.
B) Attempt local recovery, such as retrying with backoff or trying an approved alternate source.
C) Return success with empty results.
D) Invent a likely result.
Answer
**B.** Subagents should locally recover from transient failures when reasonable, then propagate only unresolved errors.Question 18: Synthesis Coverage
Some subagent findings are well-supported, while others depend on sources that failed.
What should the synthesizer include?
A) A confident answer with no caveats.
B) Coverage annotations distinguishing well-supported findings from gaps caused by unavailable sources.
C) Only raw logs.
D) No answer.
Answer
**B.** Synthesis should clearly separate verified findings from coverage gaps.Question 19: Context Degradation in Codebase Exploration
During a long codebase exploration, Claude starts saying:
But earlier it discovered the exact classes and dependency chain.
What is the best response?
A) Continue in the same bloated session.
B) Re-anchor on scratchpad files, compact with focus instructions, and reference confirmed findings.
C) Ask Claude to infer from common patterns.
D) Read the entire codebase again in the main conversation.
Answer
**B.** Scratchpads and focused compaction counter context degradation by preserving exact discovered facts.Question 20: Subagents for Large Exploration
A task requires finding all tests related to refunds across a large monorepo.
What is the best architecture?
A) Main agent reads every file.
B) Spawn a focused subagent to search tests and return a structured summary.
C) Ask the model to infer test names from conventions.
D) Skip test discovery.
Answer
**B.** Focused subagents isolate verbose exploration output and keep the main context clean.Question 21: Phase Summaries
Before spawning Phase 2 subagents, what should the coordinator do?
A) Send no context so they start fresh.
B) Summarize Phase 1 findings and inject that summary into Phase 2 prompts.
C) Delete the scratchpad.
D) Ask each subagent to rediscover everything.
Answer
**B.** Phase summaries help later agents continue from confirmed facts instead of restarting or guessing.Question 22: Crash Recovery
A long exploration session crashes.
What should the coordinator load on resume?
A) Only the latest user message.
B) Manifest, scratchpad, and structured agent state exports.
C) Random source files.
D) All raw terminal logs.
Answer
**B.** Structured state exports and a manifest enable reliable recovery.Question 23: /compact
The context window is filling with verbose exploration output, and the next task is still part of the same investigation.
What should you use?
A) /clear
B) /compact with focus instructions
C) Delete all scratchpads
D) Start over with no summary
Answer
**B.** Use `/compact` when continuing the same investigation but reducing context pressure.Question 24: /clear
The user is switching from refund-flow exploration to an unrelated UI styling task.
What is the better command?
A) /clear
B) /compact focus on refunds
C) Spawn more refund subagents
D) Keep all old context
Answer
**A.** For unrelated work, clear the old context rather than carrying irrelevant investigation state.Question 25: Aggregate Accuracy
A document extraction system reports 97% overall accuracy. The team wants to stop human review for all high-confidence outputs.
What is the best next step?
A) Stop review immediately.
B) Analyze accuracy by document type and field before reducing review.
C) Trust all confidence scores.
D) Review only short documents.
Answer
**B.** Aggregate accuracy can hide weak performance on specific document types or fields.Question 26: Field-Level Confidence
The model reports document-level confidence of 0.98, but payment_terms is often wrong.
What should the system use?
A) Only document-level confidence.
B) Field-level confidence and field-level validation metrics.
C) Random sampling only.
D) No confidence scores.
Answer
**B.** Review routing should happen at field level because fields vary in difficulty and risk.Question 27: Calibration
The model reports confidence ≥ 0.95 for scanned contract renewal dates, but validation shows those fields are correct only 84% of the time.
What does this mean?
A) Confidence is reliable for that segment.
B) Confidence is overconfident for that segment and should not be used directly for automation.
C) Human labels are irrelevant.
D) All scanned contracts should be auto-accepted.
Answer
**B.** Confidence must be calibrated using labeled validation data, and calibration can vary by segment.Question 28: High-Confidence Sampling
The team reviews only low-confidence extractions. Later, they discover a new vendor template where the model confidently extracts the wrong date field.
What sampling strategy would have caught this earlier?
A) Stratified random sampling of high-confidence extractions.
B) Reviewing only low-confidence cases.
C) No sampling.
D) Sampling only short documents.
Answer
**A.** High-confidence sampling detects confidently wrong novel error patterns.Question 29: Human Review Routing
Which is the strongest routing rule?
A) Review only fields with confidence below 0.8.
B) Review fields with low confidence, ambiguous or contradictory source evidence, poor segment performance, high business risk, or new/unvalidated templates.
C) Review every field forever.
D) Review no high-confidence fields.
Answer
**B.** Good routing combines confidence, ambiguity, contradiction, segment performance, risk, and template novelty.Question 30: Progressive Automation
Validation shows:
invoice.total_amount at confidence ≥ 0.98 = 99.5% accurate
purchase_order.payment_terms at confidence ≥ 0.98 = 82% accurate
What should the automation policy do?
A) Auto-accept both because confidence is high.
B) Auto-accept invoice totals with audit sampling, but keep purchase order payment terms in human review.
C) Reject all outputs.
D) Use only document-level accuracy.
Answer
**B.** Automation should be segment-specific, not based on a universal confidence threshold.Question 31: Lost Provenance
A subagent summarizes three documents into:
It omits source names, values, dates, and excerpts.
What is the best fix?
A) Ask for a shorter summary.
B) Require structured claim-source mappings with source names, URLs/document IDs, dates, values, and relevant excerpts.
C) Trust the summary.
D) Remove all numbers.
Answer
**B.** The issue is lost provenance. Claims need source mappings that survive synthesis.Question 32: Conflicting Statistics
Two credible sources report different churn rates: 6.9% and 8.1%.
What should the synthesis agent do?
A) Pick the lower number.
B) Pick the higher number.
C) Include both values with source attribution, dates, and methodology context, then mark the finding contested or methodology-dependent.
D) Average them.
Answer
**C.** Conflicting statistics should be preserved and annotated, not arbitrarily resolved.Question 33: Temporal Interpretation
One market report says market share was 18%, and another says 22%. The first used 2024 data; the second used 2026 data.
What is the best interpretation?
A) The sources necessarily contradict.
B) The difference may reflect temporal change, so preserve collection dates and interpret as a time-series difference unless definitions also conflict.
C) Use only the older report.
D) Delete both values.
Answer
**B.** Publication and collection dates help distinguish temporal change from true contradiction.Question 34: Contract Conflict
A legal contract subagent finds two renewal dates in different documents.
What should it do?
A) Choose one silently.
B) Return both values with document names, excerpts, sections, dates, and conflict annotation for the coordinator.
C) Ignore renewal dates.
D) Invent a compromise date.
Answer
**B.** Subagents should preserve conflicts and let the coordinator apply reconciliation policy.Question 35: Source Characterization
A study says:
The synthesis says:
What went wrong?
A) The source characterization was distorted.
B) The report used too many citations.
C) The result should have been a table.
D) Nothing.
Answer
**A.** Synthesis must preserve whether a source claims association, causation, estimate, projection, allegation, or confirmed fact.Question 36: Rendering by Content Type
A synthesis contains financial metrics, news events, and technical bug findings.
What is the best rendering strategy?
A) Convert everything into identical prose bullets.
B) Use tables for financial metrics, prose/timelines for news, and structured lists for technical findings.
C) Use only JSON for everything.
D) Remove all formatting.
Answer
**B.** Format should match content type to preserve meaning and readability.Answer Key
1. B
2. B
3. B
4. B
5. B
6. B
7. B
8. B
9. B
10. C
11. B
12. B
13. B
14. B
15. B
16. C
17. B
18. B
19. B
20. B
21. B
22. B
23. B
24. A
25. B
26. B
27. B
28. A
29. B
30. B
31. B
32. C
33. B
34. B
35. A
36. B
Scoring Guide
33–36 correct: Strong Domain 5 readiness.
28–32 correct: Good readiness; review missed modules.
22–27 correct: Solid basics, but scenario tradeoffs need more practice.
Below 22: Revisit D5.1–D5.6 before moving on.
Highest-Yield Domain 5 Traps
1. Vague summaries lose exact amounts, dates, IDs, and customer expectations.
2. Multi-issue sessions need structured issue data.
3. Tool outputs should be trimmed before accumulating in context.
4. Key findings should be front-loaded to reduce lost-in-the-middle risk.
5. Explicit human requests require immediate escalation.
6. Frustration alone is not an escalation trigger.
7. Multiple customer matches require clarification, not heuristic selection.
8. Empty result is not the same as access failure.
9. Subagents should propagate structured errors with partial results and attempted actions.
10. Large-codebase exploration needs scratchpads, subagents, phase summaries, and manifests.
11. Overall accuracy can hide weak fields or document types.
12. Confidence must be calibrated by segment using labeled validation sets.
13. High-confidence outputs still need stratified sampling.
14. Claim-source mappings must survive summarization.
15. Conflicting credible sources should be annotated, not arbitrarily resolved.
16. Temporal metadata prevents false contradictions.