Apply Few-Shot Prompting to Improve Output Consistency and Quality
This module is about using examples to teach Claude the judgment pattern you want.
In D4.1, we focused on explicit criteria. In D4.2, the key idea is:
Anthropic’s current prompt engineering docs describe examples as one of the most reliable ways to steer Claude’s output format, tone, and structure, and say well-crafted few-shot or multishot examples improve accuracy and consistency. The same docs recommend examples that are relevant, diverse, and structured, ideally wrapped in <example> / <examples> tags so Claude can distinguish examples from instructions. (Claude)
1. Core mental model
Few-shot prompting means providing a small set of examples before the actual task.
Each example usually shows:
The exam shortcut:
Use few-shot examples when:
Detailed instructions are still producing inconsistent output.
The output format must be consistent.
The task requires judgment.
The model confuses similar categories.
The model over-reports false positives.
The model under-extracts required fields.
The input documents vary in structure.
Do not use examples only to cover every possible case. The point is to teach a generalizable decision pattern.
2. Few-shot vs zero-shot
Zero-shot prompt
This may work, but output can vary:
Some findings include location; others do not.
Severity labels drift.
Suggested fixes are vague.
False positives appear.
Few-shot prompt
Then you show 2–4 examples of good findings and skipped cases.
Few-shot examples help Claude infer:
What counts as a real issue.
What should be skipped.
How severe each issue is.
What the output should look like.
How much explanation is expected.
Anthropic’s prompt engineering overview says good prompt engineering starts with clear success criteria and a way to test against those criteria; few-shot examples are useful because they make those success criteria concrete. (Claude)
3. How many examples?
For this exam, remember:
Anthropic’s general docs recommend 3–5 examples for best results, but the exam statement specifically emphasizes 2–4 targeted examples for ambiguous scenarios. The practical point is not the exact number; it is that the examples should be high-signal, not repetitive. (Claude)
Good few-shot set:
1 clear positive example
1 clear negative / skip example
1 ambiguous boundary example
1 edge case example
Bad few-shot set:
4 near-identical examples
Only happy paths
Examples with inconsistent formatting
Examples that contradict the written rules
Examples that accidentally teach the wrong pattern
4. Anatomy of a strong few-shot example
A good example should include:
Use a concise rationale, not a long hidden reasoning trace. The goal is to show the classification basis.
Example structure:
<examples>
<example>
<input>
User asks: "Can you check whether order #123 shipped?"
Available tools: get_customer, lookup_order
</input>
<expected_action>
Use lookup_order
</expected_action>
<rationale>
The user provided an order ID and asks about order status. Customer identity lookup is not needed before retrieving non-sensitive shipment status.
</rationale>
</example>
</examples>
Anthropic’s docs recommend XML-style tags when prompts combine instructions, context, examples, and variable inputs, because tags help Claude parse complex prompts unambiguously. (Claude)
5. Few-shot for consistent review output
The exam specifically mentions output format like:
Weak prompt
Better prompt with examples
<instructions>
Review the diff. Report only correctness or security issues introduced by the diff.
For each finding, use this exact format:
- location:
- issue:
- severity: Important | Nit
- suggested_fix:
Skip style-only issues, formatting, lint, and speculative concerns.
</instructions>
<examples>
<example>
<input>
Changed code:
if (user.role = "admin") {
grantAdminAccess(user.id)
}
</input>
<expected_output>
- location: auth/permissions.ts:42
- issue: The condition assigns "admin" instead of comparing the user role, so every user can be treated as admin.
- severity: Important
- suggested_fix: Use a strict comparison, for example user.role === "admin".
</expected_output>
<rationale>
This is a production-impacting authorization bug introduced by the changed condition.
</rationale>
</example>
<example>
<input>
Changed code:
const res = await fetchSubscription(userId)
</input>
<expected_output>
No finding.
</expected_output>
<rationale>
The variable name is vague, but this is only a style issue and does not affect behavior.
</rationale>
</example>
</examples>
<input>
{{DIFF}}
</input>
Why this works:
The examples demonstrate both reportable and non-reportable cases.
The expected output fixes the structure.
The rationale teaches the boundary between real bugs and style noise.
Claude Code’s review docs show that review findings are surfaced with severity and file/line information, and that Code Review focuses by default on production-breaking correctness bugs rather than formatting preferences. This aligns well with few-shot examples that teach consistent location/severity/reporting behavior. (Claude)
6. Few-shot for ambiguous tool selection
Few-shot examples are especially useful when two actions are plausible.
Scenario
The agent has tools:
The model sometimes calls get_customer for every order-related question, even when lookup_order is sufficient.
Few-shot examples
<examples>
<example>
<input>
User: "Can you check the status of order #A123?"
</input>
<expected_action>
lookup_order
</expected_action>
<rationale>
The user provided an order ID and asks for order status. The order tool directly matches the request.
</rationale>
</example>
<example>
<input>
User: "I need a refund. My name is Priya Sharma."
</input>
<expected_action>
get_customer first
</expected_action>
<rationale>
Refunds are sensitive and require verified customer identity before order or refund operations.
</rationale>
</example>
<example>
<input>
User: "Order #B778 arrived damaged. Can you refund it?"
</input>
<expected_action>
get_customer first, then lookup_order after identity is verified
</expected_action>
<rationale>
The user provided an order ID, but refund processing requires verified customer identity before taking financial action.
</rationale>
</example>
</examples>
Exam point:
Few-shot examples should show the ambiguous boundary, not just easy cases.
The third example is the most valuable because both lookup_order and get_customer seem plausible. It teaches the model that financial action changes the sequence.
7. Few-shot for branch-level test coverage gaps
The official guide mentions branch-level test coverage gaps.
This is a subtle review task. A vague prompt like “check tests” often causes noisy findings. Few-shot examples can teach when a missing test is real.
Prompt pattern
<instructions>
Review whether this diff introduces meaningful untested behavior.
Report missing tests only when:
1. The diff adds or changes a branch of behavior.
2. That branch has realistic production impact.
3. Existing tests do not already cover the branch.
Do not report missing tests for pure refactors, copy changes, or behavior already covered by existing tests.
</instructions>
<examples>
<example>
<input>
Diff adds:
if (payment.status === "chargeback") {
lockAccount(userId)
}
Existing tests cover: "paid", "failed", "refunded"
Existing tests do not cover: "chargeback"
</input>
<expected_output>
- location: billing/paymentStatus.ts
- issue: The new chargeback branch locks accounts but has no test coverage.
- severity: Important
- suggested_fix: Add a test for chargeback status verifying the account is locked.
</expected_output>
<rationale>
The diff adds a behaviorally significant branch with account-impacting side effects.
</rationale>
</example>
<example>
<input>
Diff renames helper formatUserName() to formatDisplayName().
Existing tests cover the same behavior through public API tests.
</input>
<expected_output>
No finding.
</expected_output>
<rationale>
This is a refactor with existing behavioral coverage. A duplicate test would be low-value.
</rationale>
</example>
<example>
<input>
Diff changes:
if (!email) return null
to:
if (!email) return ""
Existing tests cover valid emails only.
</input>
<expected_output>
- location: users/emailFormatter.ts
- issue: The empty-email branch changed behavior from null to empty string without test coverage.
- severity: Nit
- suggested_fix: Add a test for missing email to lock down the intended return value.
</expected_output>
<rationale>
The changed branch affects behavior, but the impact is lower than account locking.
</rationale>
</example>
</examples>
This set teaches:
Important missing test: high-impact branch.
Skip: refactor already covered.
Nit: lower-risk behavior branch.
8. Few-shot for reducing false positives
The exam specifically calls out distinguishing acceptable code patterns from genuine issues.
This is important because examples should not only show what to report. They should also show what to skip.
Example: acceptable vs genuine issue
<examples>
<example>
<input>
Changed code:
const users = await db.users.findMany({ where: { orgId } })
return users.map(toDto)
</input>
<expected_output>
No finding.
</expected_output>
<rationale>
This is a single bounded query followed by an in-memory mapping. There is no evidence of an N+1 query or correctness bug.
</rationale>
</example>
<example>
<input>
Changed code:
for (const user of users) {
const orders = await db.orders.findMany({ where: { userId: user.id } })
results.push({ user, orders })
}
</input>
<expected_output>
- location: reports/userOrders.ts
- issue: The diff introduces one database query per user on an unbounded list, creating an N+1 query path.
- severity: Important
- suggested_fix: Batch-load orders for all user IDs in one query and group them by userId.
</expected_output>
<rationale>
This is a real performance issue because the repeated database call is inside a loop over a production-sized collection.
</rationale>
</example>
</examples>
Why this works:
It prevents Claude from flagging every loop or every query.
It teaches the concrete difference between acceptable linear work and reportable N+1 behavior.
This is the D4.2 extension of D4.1: criteria plus examples reduce false positives.
9. Few-shot for extraction tasks
Few-shot prompting is very useful when extraction inputs vary.
Extraction prompts often fail in two ways:
1. Hallucination: Claude fills missing fields with guessed values.
2. Under-extraction: Claude returns null because the field appears in an unusual format.
Few-shot examples can teach both:
Anthropic’s long-context prompting guidance recommends structuring documents with tags and, for document tasks, grounding responses in quotes before answering so the model can focus on relevant evidence. (Claude)
9.1 Example: informal measurements
Task:
Weak prompt:
Better few-shot prompt:
<instructions>
Extract dimensions from the supplier description.
Return JSON:
{
"height_cm": number | null,
"width_cm": number | null,
"depth_cm": number | null,
"evidence": string
}
Rules:
- Convert meters to centimeters.
- Accept informal formats like "about 2m tall" when clearly referring to height.
- Do not guess missing dimensions.
- If a dimension is absent, return null.
</instructions>
<examples>
<example>
<input>
"Cabinet is about 2m tall, 80 cm wide, and 45 cm deep."
</input>
<expected_output>
{
"height_cm": 200,
"width_cm": 80,
"depth_cm": 45,
"evidence": "about 2m tall, 80 cm wide, and 45 cm deep"
}
</expected_output>
<rationale>
The informal phrase "about 2m tall" clearly refers to height and can be converted to 200 cm.
</rationale>
</example>
<example>
<input>
"Compact shelf, 60cm across. Depth not specified."
</input>
<expected_output>
{
"height_cm": null,
"width_cm": 60,
"depth_cm": null,
"evidence": "60cm across. Depth not specified."
}
</expected_output>
<rationale>
"Across" is treated as width. Height and depth are not present, so they remain null.
</rationale>
</example>
<example>
<input>
"Tall narrow frame, suitable for hallway use."
</input>
<expected_output>
{
"height_cm": null,
"width_cm": null,
"depth_cm": null,
"evidence": ""
}
</expected_output>
<rationale>
Descriptive words like "tall" and "narrow" are not measurements. Do not infer numbers.
</rationale>
</example>
</examples>
<input>
{{SUPPLIER_DESCRIPTION}}
</input>
This teaches:
How to extract informal measurements.
How to convert units.
How to avoid hallucinating missing fields.
10. Few-shot for varied document structures
The official guide mentions:
inline citations vs bibliographies
methodology sections vs embedded details
varied document structures
This is a classic extraction problem. If the model expects one structure, it may return null when the information appears elsewhere.
Example: extracting methodology details
<instructions>
Extract the study methodology.
Return JSON:
{
"methodology": string | null,
"sample_size": number | null,
"citation_style": "inline" | "bibliography" | "none",
"evidence": string
}
Rules:
- Use only information present in the document.
- If methodology is in a dedicated Methodology section, extract it.
- If methodology is embedded in the introduction or results narrative, extract it if explicit.
- If citations appear inline, mark citation_style as "inline".
- If citations appear only in a references section, mark citation_style as "bibliography".
- Do not invent sample size.
</instructions>
<examples>
<example>
<input>
"Methodology: We surveyed 240 participants using a 12-question online questionnaire. Results are reported in Table 2 (Chen, 2024)."
</input>
<expected_output>
{
"methodology": "Surveyed participants using a 12-question online questionnaire.",
"sample_size": 240,
"citation_style": "inline",
"evidence": "We surveyed 240 participants using a 12-question online questionnaire. ... (Chen, 2024)."
}
</expected_output>
<rationale>
The methodology and sample size are explicit, and the citation appears inline.
</rationale>
</example>
<example>
<input>
"This report compares onboarding outcomes across three cohorts. Participants completed onboarding tasks while observers recorded time-to-completion. References: [1] Chen, 2024. [2] Rao, 2023."
</input>
<expected_output>
{
"methodology": "Participants completed onboarding tasks while observers recorded time-to-completion.",
"sample_size": null,
"citation_style": "bibliography",
"evidence": "Participants completed onboarding tasks while observers recorded time-to-completion. References: [1] Chen, 2024. [2] Rao, 2023."
}
</expected_output>
<rationale>
The methodology is embedded in narrative text rather than a Methodology section. No sample size is provided.
</rationale>
</example>
<example>
<input>
"The new workflow improved satisfaction. Teams reported fewer support tickets after rollout."
</input>
<expected_output>
{
"methodology": null,
"sample_size": null,
"citation_style": "none",
"evidence": ""
}
</expected_output>
<rationale>
The document reports conclusions but does not describe how the evidence was collected.
</rationale>
</example>
</examples>
<input>
{{DOCUMENT}}
</input>
This example set teaches Claude to generalize across structure:
11. Few-shot for empty/null extraction failures
The official guide also mentions adding examples when required fields are coming back empty/null because the document structure varies.
Problem
The prompt says:
Claude returns:
even when the contract says:
This agreement automatically extends for another 12 months on the anniversary of the Effective Date.
Effective Date: March 1, 2026.
Few-shot fix
<examples>
<example>
<input>
"Renewal Date: July 15, 2026."
</input>
<expected_output>
{
"renewal_date": "2026-07-15",
"derivation": "explicit",
"evidence": "Renewal Date: July 15, 2026."
}
</expected_output>
<rationale>
The renewal date is explicitly stated.
</rationale>
</example>
<example>
<input>
"Effective Date: March 1, 2026. This agreement automatically extends for another 12 months on the anniversary of the Effective Date."
</input>
<expected_output>
{
"renewal_date": "2027-03-01",
"derivation": "derived_from_effective_date",
"evidence": "Effective Date: March 1, 2026. ... extends for another 12 months on the anniversary of the Effective Date."
}
</expected_output>
<rationale>
The renewal date is not labeled directly, but it is derivable from explicit contract language.
</rationale>
</example>
<example>
<input>
"The parties may renew the agreement by mutual written consent."
</input>
<expected_output>
{
"renewal_date": null,
"derivation": "not_available",
"evidence": ""
}
</expected_output>
<rationale>
The document mentions renewal but provides no date or computable rule.
</rationale>
</example>
</examples>
This teaches:
Extract explicit fields.
Derive only when the derivation rule is explicit.
Return null when neither explicit nor derivable.
That reduces both false nulls and hallucinated dates.
12. Few-shot examples should teach generalization, not memorization
A bad few-shot set accidentally teaches keyword matching.
Bad examples:
Every reportable bug example contains the word "crash."
Every security example contains the word "token."
Every extraction example uses the same heading.
The model may overfit to surface patterns.
Better examples vary:
One crash bug.
One wrong-output bug.
One authorization bug.
One skip case with scary-looking but acceptable code.
The official guide’s key idea is that examples should help Claude generalize judgment to novel patterns, not simply match pre-specified cases.
13. How to choose few-shot examples
Pick examples that are:
Relevant: close to the real task.
Diverse: cover different shapes and edge cases.
Contrasting: include both report and skip cases.
Boundary-focused: show ambiguous cases.
Format-complete: exactly match desired output.
Non-contradictory: do not conflict with instructions.
Anthropic’s docs explicitly recommend relevant, diverse, structured examples, and warn that diversity helps avoid unintended patterns. (Claude)
14. Common exam traps
Trap 1: More prose instead of examples
Scenario:
Weak fix:
Better fix:
Trap 2: Only positive examples
Bad:
Better:
Why?
Trap 3: Examples without rationale
Bad:
Better for judgment tasks:
For classification, tool selection, and review, the rationale teaches the category boundary.
Trap 4: Too many repetitive examples
Bad:
Better:
Trap 5: Examples contradict instructions
Bad:
This teaches the wrong behavior.
Trap 6: Extraction examples do not include null cases
Bad:
Better:
This reduces hallucination.
15. Prompt templates
Template 1: Review output consistency
<instructions>
Review the diff. Report only issues introduced by the diff.
Use this exact format:
- location:
- issue:
- severity: Important | Nit
- suggested_fix:
Do not report style, formatting, lint, or speculative issues.
</instructions>
<examples>
<example>
<input>...</input>
<expected_output>...</expected_output>
<rationale>...</rationale>
</example>
<example>
<input>...</input>
<expected_output>No finding.</expected_output>
<rationale>...</rationale>
</example>
</examples>
<input>
{{DIFF}}
</input>
Template 2: Ambiguous tool selection
<instructions>
Choose the correct tool or tool sequence. Use the examples to resolve ambiguous cases.
</instructions>
<tools>
- get_customer: verifies customer identity.
- lookup_order: retrieves order status by order ID.
- process_refund: issues refund after verified identity and validated order.
</tools>
<examples>
<example>
<input>...</input>
<expected_action>...</expected_action>
<rationale>...</rationale>
</example>
</examples>
<input>
{{USER_REQUEST}}
</input>
Template 3: Extraction with varied document formats
<instructions>
Extract the requested fields as JSON.
Rules:
- Use only evidence in the document.
- Return null for unavailable fields.
- Derive fields only when the derivation rule is explicit.
- Include evidence for each non-null field.
</instructions>
<examples>
<example>
<input>...</input>
<expected_output>...</expected_output>
<rationale>...</rationale>
</example>
</examples>
<document>
{{DOCUMENT}}
</document>
Template 4: False-positive reduction
<instructions>
Report only genuine issues. Use the examples to distinguish acceptable patterns from reportable problems.
</instructions>
<examples>
<example>
<input>Acceptable pattern...</input>
<expected_output>No finding.</expected_output>
<rationale>Why this is acceptable.</rationale>
</example>
<example>
<input>Genuine issue...</input>
<expected_output>Finding...</expected_output>
<rationale>Why this is reportable.</rationale>
</example>
</examples>
<input>
{{CODE_OR_DIFF}}
</input>
16. Scenario-based practice questions
Question 1
A review prompt gives detailed formatting instructions, but Claude’s findings still vary: sometimes it omits severity, sometimes it omits suggested fixes, and sometimes it writes paragraphs.
What is the best improvement?
A) Add “be consistent” to the prompt.
B) Provide 2–4 few-shot examples using the exact desired output format.
C) Increase the number of tools.
D) Remove all formatting instructions.
Answer
B. Few-shot examples are especially effective when detailed instructions alone do not produce consistent formatting.
Question 2
A model reports many false positives for performance issues. You want it to distinguish acceptable in-memory loops from genuine N+1 database queries.
What should you add?
A) Only the instruction “be conservative.”
B) Few-shot examples showing one acceptable loop with no finding and one N+1 query with a finding.
C) More severity labels.
D) A longer introduction.
Answer
B. Contrastive examples teach the boundary between acceptable patterns and genuine issues.
Question 3
An agent sometimes calls get_customer and sometimes lookup_order for order-related questions. Some requests are ambiguous.
What kind of few-shot example is most valuable?
A) Only easy examples where the tool choice is obvious.
B) Examples showing ambiguous requests and explaining why one tool or sequence is chosen over plausible alternatives.
C) Examples with no rationale.
D) Examples unrelated to tools.
Answer
B. Few-shot examples are most useful when they clarify ambiguous boundary cases.
Question 4
Claude extracts null for a required contract field when the field appears in a nonstandard format.
What is the best fix?
A) Tell Claude “try harder.”
B) Add few-shot examples showing the field in varied document structures, including explicit, derived, and unavailable cases.
C) Remove the field from the schema.
D) Always force a value.
Answer
B. Examples teach Claude to extract from varied structures while still returning null when evidence is absent.
Question 5
A document extraction task suffers from hallucinated values when documents omit a field.
What should your few-shot examples include?
A) Only examples where all fields are present.
B) Examples where fields are missing and the correct output is null.
C) No examples.
D) A command to never output null.
Answer
B. Null examples teach the model not to invent missing values.
Question 6
A prompt includes examples, but all examples are nearly identical happy paths.
What is the problem?
A) There are too many XML tags.
B) The examples may cause overfitting to a narrow pattern and fail on edge cases.
C) Few-shot examples never work.
D) The examples should not include outputs.
Answer
B. Examples should be diverse enough to demonstrate the general judgment pattern.
Question 7
A code review prompt reports missing tests for every change, including pure refactors. What few-shot set would help most?
A) Examples where every change gets a missing-test finding.
B) One example of a behavior-changing branch without coverage, one pure refactor with no finding, and one lower-risk branch classified as Nit.
C) Only style examples.
D) No examples because tests are objective.
Answer
B. This teaches branch-level test coverage judgment and skip behavior.
Question 8
A prompt says to extract methodology details from research summaries. Some documents have a Methodology section, while others embed methodology in narrative text.
What should the examples demonstrate?
A) Only documents with a Methodology heading.
B) Dedicated methodology sections, embedded methodology details, and documents with no extractable methodology.
C) Only bibliography extraction.
D) Only documents with perfect formatting.
Answer
B. Few-shot examples should cover varied document structures so Claude generalizes beyond one layout.
Question 9
Which is the best few-shot example for a judgment task?
A)
B)
Input: ...
Expected output: ...
Rationale: This is reportable because it changes runtime behavior on a realistic path.
C)
D)
Answer
B. For judgment tasks, a concise rationale teaches why the action was chosen over alternatives.
Question 10
A prompt has 12 few-shot examples, but all are long and repetitive. The model starts copying irrelevant phrasing from examples.
What is the best improvement?
A) Replace them with 2–4 targeted, diverse examples focused on the ambiguous cases.
B) Add 20 more examples.
C) Remove all structure.
D) Use only negative examples.
Answer
A. Few-shot examples should be targeted and diverse. More examples are not automatically better if they are repetitive or noisy.
17. Final D4.2 checklist
Memorize this:
1. Use few-shot examples when instructions alone produce inconsistent results.
2. Few-shot examples are especially strong for output format consistency.
3. Use 2–4 targeted examples for exam scenarios.
4. Good examples are relevant, diverse, structured, and non-contradictory.
5. Include both report and skip examples.
6. Use examples to demonstrate ambiguous-case handling.
7. Include concise rationales for judgment tasks.
8. Few-shot examples help Claude generalize to novel patterns.
9. Avoid repetitive examples that teach accidental keyword matching.
10. Use examples to distinguish acceptable code from genuine issues.
11. Use examples to calibrate severity and suggested fixes.
12. Use examples for branch-level test coverage judgment.
13. Use extraction examples with varied document structures.
14. Include null/missing-field examples to reduce hallucination.
15. Show explicit, derived, and unavailable extraction cases.
D4.2 mental model
Instructions tell Claude the rule.
Few-shot examples show Claude the pattern.
Positive examples show what to do.
Negative examples show what to skip.
Boundary examples teach judgment.
Extraction examples teach when to extract, derive, or return null.
The exam’s favorite distinction is:
When detailed instructions still produce inconsistent or noisy outputs, add targeted few-shot examples that demonstrate both the desired format and the judgment boundary.