Domain 4 — Mixed Scenario Quiz
Prompt Engineering & Structured Output — Exam Style
Each question has one best answer.
Question 1: Vague Review Criteria
A review prompt says:
Claude reports many comments that are incomplete, vague, or slightly outdated, even though they do not contradict the code.
What is the best improvement?
A) Repeat “be conservative” several times.
B) Define reportable comment issues as concrete behavioral contradictions between the comment and current code behavior, and explicitly skip incomplete/style-only comments.
C) Increase the model’s max tokens.
D) Ask Claude to report fewer comments.
Answer
**B.** General instructions like “be conservative” do not define the reporting boundary. The prompt should specify exactly which comment issues are reportable and which should be skipped.Question 2: High False Positives
A code review bot correctly catches security issues, but its performance and comment-accuracy categories produce many false positives. Developers begin ignoring all review comments.
What should the team do?
A) Disable the entire review bot.
B) Temporarily disable or sharply narrow the noisy categories while preserving accurate categories.
C) Mark every finding as Important.
D) Ask Claude to “only report high-confidence findings.”
Answer
**B.** Noisy categories reduce trust in the entire system. Temporarily disabling or narrowing false-positive-prone categories protects trust while prompts are improved.Question 3: Severity Criteria
Which severity definition is strongest?
A) Important means “Claude is very confident.”
B) Important means “the issue should block merge because it can cause a production correctness bug, security issue, data corruption, tenant isolation failure, or crash on a common path.”
C) Important means “the code could be cleaner.”
D) Important means “any missing test.”
Answer
**B.** Severity should be based on concrete impact and merge-blocking criteria, not confidence or subjective judgment.Question 4: Missing Tests
Claude reports missing tests on every PR, including copy-only changes and pure refactors.
What prompt improvement best reduces false positives?
A) “Be careful about test comments.”
B) “Report missing tests only when the diff changes behavior, adds business logic, fixes a bug, or introduces an API path requiring coverage. Skip copy-only, formatting-only, or refactor-only changes with no behavior change.”
C) “Always report missing tests.”
D) “Never report missing tests.”
Answer
**B.** Specific categorical criteria are better than vague caution. This defines when missing-test findings are valid and when they should be skipped.Question 5: Evidence Gate
A review finding says:
It does not include a changed line, execution path, expected behavior, or actual behavior.
What should the prompt require?
A) Post the finding if it sounds serious.
B) Require code evidence, realistic failure path, expected vs actual behavior, and proof the issue was introduced by the diff.
C) Convert it into a Nit.
D) Ask the developer to investigate.
Answer
**B.** Evidence requirements reduce false positives. Findings should not be posted unless they can explain the concrete failure.Question 6: Few-Shot for Output Format
Detailed instructions tell Claude to output findings as:
But Claude’s output format remains inconsistent.
What is the best fix?
A) Add “be consistent.”
B) Provide 2–4 few-shot examples using the exact desired output format.
C) Remove the structure requirement.
D) Ask for shorter output.
Answer
**B.** Few-shot examples are especially effective when detailed instructions alone do not produce consistent formatting.Question 7: Ambiguous Tool Selection
An agent sometimes calls get_customer and sometimes lookup_order for order-related requests. Some requests are ambiguous because an order lookup is safe, but refunds require verified identity.
What few-shot examples would help most?
A) Only obvious examples where the user says “look up order.”
B) Examples showing ambiguous requests and explaining why one tool or sequence is chosen over plausible alternatives.
C) Examples unrelated to tools.
D) Examples with no expected action.
Answer
**B.** Few-shot examples should teach ambiguous-case handling. The best examples show the decision boundary and rationale.Question 8: False Positive Reduction with Examples
Claude flags every loop as a possible performance issue.
What few-shot set would best reduce false positives?
A) Only examples of serious performance bugs.
B) One acceptable in-memory loop with “No finding,” and one genuine N+1 database query with a finding and rationale.
C) Ten examples of unrelated syntax errors.
D) A single instruction: “Be conservative.”
Answer
**B.** Contrastive examples teach the difference between acceptable patterns and genuine issues.Question 9: Extraction Hallucination
A document extraction prompt hallucinates values when fields are absent.
What should the few-shot examples include?
A) Only examples where all fields are present.
B) Examples where required fields are missing and the correct output is null.
C) Examples that always force a value.
D) No examples.
Answer
**B.** Null examples teach Claude not to invent missing values.Question 10: Varied Document Structures
Claude extracts methodology only when a document has a “Methodology” heading, but misses methodology embedded in narrative text.
What should the examples demonstrate?
A) Only documents with perfect headings.
B) Dedicated methodology sections, embedded methodology details, and documents with no extractable methodology.
C) Only bibliographies.
D) Only short documents.
Answer
**B.** Few-shot examples should cover varied structures so Claude generalizes beyond one layout.Question 11: Tool Use for Structured Output
A developer prompts:
Sometimes Claude adds prose or returns malformed JSON.
What is the most reliable fix?
A) Add “valid JSON only” to the prompt.
B) Use tool use with a JSON schema and require Claude to call the extraction tool.
C) Increase temperature.
D) Parse the prose with regex.
Answer
**B.** Tool use with JSON schemas is the most reliable method for schema-compliant structured output.Question 12: tool_choice: auto
A pipeline requires structured output, but Claude sometimes answers in text instead of calling the extraction tool.
Which setting is likely causing the issue?
A) tool_choice: {"type": "auto"}
B) tool_choice: {"type": "any"}
C) tool_choice: {"type": "tool", "name": "extract_invoice"}
D) A forced tool choice
Answer
**A.** With `auto`, Claude may choose text instead of a tool call. Use `any` or a forced tool when structured output is mandatory.Question 13: Unknown Document Type
A system receives unknown documents that may be invoices, receipts, or contracts. It must always produce structured output.
What is the best configuration?
A) Force extract_invoice every time.
B) Provide the three extraction tools and use tool_choice: {"type": "any"}.
C) Use no tools and ask for JSON.
D) Use tool_choice: {"type": "auto"}.
Answer
**B.** `any` guarantees a tool call while allowing Claude to select among relevant extraction schemas. Clear tool descriptions are still needed for correct selection.Question 14: Forced Extraction Step
A workflow must always extract metadata before any enrichment step.
What should be used?
A) tool_choice: {"type": "auto"}
B) tool_choice: {"type": "tool", "name": "extract_metadata"}
C) No schema
D) A vague instruction to “start with metadata”
Answer
**B.** Forced tool selection guarantees the specific extraction tool runs first.Question 15: Schema Valid but Semantically Wrong
A strict schema returns valid JSON:
What does this show?
A) Strict schemas guarantee semantic correctness.
B) Strict schemas eliminate syntax/schema errors but not semantic errors.
C) JSON schemas are not useful.
D) The output cannot be parsed.
Answer
**B.** The structure is valid, but the values are semantically inconsistent.Question 16: Nullable Fields
A contract extraction schema requires:
Many contracts do not contain a termination date, and Claude fabricates one.
What is the best fix?
A) Make termination_date nullable or optional.
B) Add more required fields.
C) Force a default termination date.
D) Remove extraction.
Answer
**A.** When the source may not contain a field, the schema should allow absence. This reduces fabrication pressure.Question 17: Enum Design
A classifier enum is:
The system also receives quotes and ambiguous documents.
What is the best schema design?
A) Force the closest category.
B) Add other, unclear, and an other_detail field.
C) Remove all categories.
D) Treat every ambiguous document as a contract.
Answer
**B.** `other` handles clear out-of-taxonomy cases. `unclear` handles ambiguity. A detail field preserves useful information.Question 18: Format Normalization
A schema defines:
Source documents contain dates like 01/06/26, June 1, 2026, and 2026.06.01.
What else should be added?
A) Prompt rules for date normalization, such as YYYY-MM-DD, and ambiguity handling.
B) Nothing; the schema alone handles normalization.
C) A random default date.
D) A broader enum.
Answer
**A.** The schema defines shape. Prompt rules define normalization semantics.Question 19: Retry With Feedback
An extraction fails validation. The retry prompt says:
What is the best improvement?
A) Include the original document, failed extraction, and specific validation errors.
B) Remove the schema.
C) Increase temperature.
D) Ask Claude to guess missing fields.
Answer
**A.** Retry-with-error-feedback works because it gives Claude specific correction targets.Question 20: Retry Limits
A document says:
Appendix B is not provided, but the extraction requires itemized pricing.
What should the system do?
A) Retry until itemized pricing appears.
B) Return null / missing_from_source and request the missing appendix if needed.
C) Guess line items from the total.
D) Use the invoice total as every line item.
Answer
**B.** Retries cannot extract information absent from the provided source.Question 21: Correctable Retry
The source says:
The extraction places this value in invoice_date.
Should a retry help?
A) Yes, because the value exists but is in the wrong field.
B) No, because the information is absent.
C) No, because schema validation already passed.
D) Only if the document is chunked.
Answer
**A.** Wrong field placement is a semantic error that can often be corrected with specific validation feedback.Question 22: Feedback Loop Metadata
Developers dismiss many review findings. The team wants to identify which code patterns trigger false positives.
What field should structured findings include?
A) detected_pattern
B) temperature
C) font_size
D) model_name_only
Answer
**A.** `detected_pattern` enables systematic analysis of dismissal patterns.Question 23: Self-Correction Schema
An invoice extraction schema includes both stated_total and calculated_total.
Why is this useful?
A) It removes the need for validation.
B) It allows the system to detect discrepancies between the source’s stated total and arithmetic over extracted components.
C) It guarantees all totals are correct.
D) It makes output less structured.
Answer
**B.** Separating stated and calculated values supports semantic validation.Question 24: Conflicting Source Data
A contract source contains two conflicting renewal dates.
What should the schema support?
A) conflict_detected and conflict_details.
B) Silently choose one date.
C) Invent a compromise date.
D) Return malformed JSON.
Answer
**A.** Conflicting source data should be surfaced explicitly rather than hidden.Question 25: Batch API Fit
A team wants to run a weekly audit over 100,000 historical documents. Results are needed within two days, not immediately.
What should they use?
A) Synchronous API only
B) Message Batches API
C) Interactive chat
D) Pre-merge CI check
Answer
**B.** This is high-volume, non-blocking, and latency-tolerant, making it a good fit for batch processing.Question 26: Blocking Workflow
A pre-merge CI check must finish before developers can merge a PR.
Which API approach is appropriate?
A) Message Batches API because it is cheaper.
B) Synchronous API because the workflow is blocking.
C) Weekly batch job.
D) Overnight audit.
Answer
**B.** Batch processing can take up to 24 hours and is inappropriate for blocking pre-merge workflows.Question 27: Batch Cost
A nightly non-blocking analysis costs $4,000 using synchronous calls. Ignoring other discounts, what is the expected Batch API cost?
A) $4,000
B) $2,000
C) $8,000
D) $0
Answer
**B.** The Message Batches API provides 50% cost savings relative to standard API prices.Question 28: Batch SLA Calculation
A batch may take up to 24 hours to process. The business requires results within 30 hours of document arrival.
How often must batches be submitted?
A) Every 24 hours
B) Every 12 hours
C) Every 6 hours or more frequently
D) Every 30 hours
Answer
**C.** Worst-case completion is submission interval plus 24 hours. To meet a 30-hour SLA, the interval must be ≤ 6 hours.Question 29: Batch Correlation
Batch responses arrive out of order.
What should be used to map responses back to requests?
A) Array position
B) custom_id
C) Prompt length
D) Model name
Answer
**B.** `custom_id` is used to correlate request/response pairs. Do not rely on response order.Question 30: Batch Failure Handling
A batch has 10,000 documents. 9,800 succeed, 100 expire, and 100 fail due to context length.
What is the best response?
A) Resubmit all 10,000 documents unchanged.
B) Resubmit only the failed/expired documents; retry expired ones and chunk oversized documents before resubmission.
C) Ignore all failures.
D) Switch every future job to synchronous API.
Answer
**B.** Batch failures should be handled selectively. Resubmit only failed items and modify requests when the failure cause requires it.Question 31: Batch Prompt Refinement
A team plans to batch-process 200,000 documents using a new extraction prompt that has never been tested.
What is the best first step?
A) Submit all documents immediately.
B) Test and refine the prompt on a representative sample before large-scale batch submission.
C) Remove schemas.
D) Use the same custom_id for every request.
Answer
**B.** Prompt refinement on a sample set improves first-pass success and reduces expensive resubmission.Question 32: Batch and Tool Loops
A workflow requires Claude to call an internal database tool, your application to execute it, then Claude to use the tool result within the same request.
What is the best interpretation?
A) This is a poor fit for a single batch request; use synchronous orchestration, pre-fetch data, or split into stages.
B) Batch always supports arbitrary client-side interactive tool loops.
C) custom_id executes tools automatically.
D) Use no context.
Answer
**A.** Batch is asynchronous and not suitable for interactive client-side tool execution inside a single request.Question 33: Self-Review Limitation
Claude implemented a feature in a long session. The user asks the same session to review its own work.
What is the best improvement?
A) Ask the same session to think longer.
B) Use an independent Claude instance or fresh subagent to review the diff without the generator’s reasoning context.
C) Skip review.
D) Only check formatting.
Answer
**B.** The generator retains assumptions from implementation. A fresh reviewer is more likely to question the final behavior.Question 34: Large Multi-File PR
A PR changes 60 files across backend, frontend, shared types, and migrations. A single review pass gives shallow and contradictory findings.
What architecture is best?
A) One even larger review prompt.
B) Per-file local passes, a cross-file integration pass, and a verification/deduplication pass.
C) Only style review.
D) No review.
Answer
**B.** Large reviews should be split into focused passes to avoid attention dilution and contradictory findings.Question 35: Local vs Integration Review
A backend change adds a new status value, but the frontend switch statement does not handle it.
Which pass is most likely to catch this?
A) Per-file local pass only
B) Cross-file integration pass
C) Spellcheck pass
D) Formatting pass
Answer
**B.** The bug exists in the contract between files. Integration review is designed for cross-file mismatches.Question 36: Verification Pass
Several review agents produce candidate findings, including duplicates and speculative comments.
What should happen before posting PR comments?
A) Post all findings.
B) Run a verification pass that checks evidence, realistic failure path, severity, duplicates, and confidence.
C) Convert all findings to Important.
D) Remove severity.
Answer
**B.** Verification filters false positives, deduplicates findings, and improves developer trust.Question 37: Confidence Routing
A verification pass marks a finding as valid but medium confidence because it depends on runtime configuration not shown in the diff.
What is the best routing?
A) Automatically post as a blocking comment.
B) Route to human review or summarize as needs-review.
C) Suppress all medium-confidence findings permanently.
D) Treat confidence as proof.
Answer
**B.** Confidence is a routing signal. Medium-confidence findings may need human review rather than automatic posting.Question 38: Over-Engineering Review
A PR fixes a typo in one error message.
What review architecture is appropriate?
A) Twenty independent reviewers plus integration analysis.
B) A simple direct check or normal review.
C) Cross-file data-flow review.
D) Dynamic multi-agent workflow.
Answer
**B.** Multi-instance and multi-pass review has overhead and is best for larger or riskier changes.Answer Key
1. B
2. B
3. B
4. B
5. B
6. B
7. B
8. B
9. B
10. B
11. B
12. A
13. B
14. B
15. B
16. A
17. B
18. A
19. A
20. B
21. A
22. A
23. B
24. A
25. B
26. B
27. B
28. C
29. B
30. B
31. B
32. A
33. B
34. B
35. B
36. B
37. B
38. B
Scoring Guide
34–38 correct: Strong Domain 4 readiness.
29–33 correct: Good readiness; review missed modules.
23–28 correct: Solid basics, but scenario tradeoffs need practice.
Below 23: Revisit D4.1–D4.6 before moving on.
Highest-Yield Domain 4 Traps
1. “Be conservative” is weaker than explicit report/skip criteria.
2. Few-shot examples should include both report and skip cases.
3. Tool use with schemas solves structured output syntax, not semantic correctness.
4. auto may return text; any forces some tool; forced tool forces a specific tool.
5. Required non-null fields can cause fabricated extraction values.
6. Retry works for correctable extraction errors, not absent source information.
7. detected_pattern enables systematic false-positive analysis.
8. Batch API is for latency-tolerant workloads, not blocking pre-merge checks.
9. custom_id is required for correlating batch results.
10. Same-session self-review is weaker than independent review.
11. Large reviews need per-file, integration, and verification passes.
12. Confidence is a routing signal, not proof.