Implement Validation, Retry, and Feedback Loops for Extraction Quality
This module is the natural follow-up to D4.3 structured output.
D4.3 said:
D4.4 adds:
Now validate whether the structured values are actually correct, retry when the error is correctable, and collect feedback data when findings are dismissed.
The most important exam distinction is:
Schema validity is not the same as extraction quality.
Strict tool use can guarantee that Claude’s tool inputs match a JSON Schema, with correctly typed arguments and no missing required schema fields, but this only solves structural conformance. It does not prove that the extracted values are semantically correct. (Claude)
1. Core mental model
Use this pipeline:
1. Extract with tool use / structured schema.
2. Validate schema and semantics.
3. If validation errors are correctable, retry with specific feedback.
4. If information is absent, do not retry blindly.
5. Track recurring false-positive / failure patterns.
6. Improve prompts, schemas, examples, and validators over time.
Exam shortcut:
Schema catches shape errors.
Semantic validation catches meaning errors.
Retry fixes correctable extraction mistakes.
Retry cannot extract information that was never provided.
Feedback fields reveal recurring failure patterns.
Anthropic’s docs emphasize giving Claude a way to verify its work, such as tests, builds, or other executable checks; the same principle applies to extraction pipelines: validation gives Claude a concrete signal for correction. (Claude)
2. Schema syntax errors vs semantic validation errors
This is one of the highest-yield D4.4 concepts.
2.1 Schema syntax / structural errors
These are errors like:
Malformed JSON
Missing required field
Wrong type: "100" instead of 100
Invalid enum value
Extra field not allowed by schema
With strict tool use, these are largely eliminated because the model’s tool input is constrained to match the JSON Schema. Anthropic’s strict tool-use docs say strict: true guarantees that Claude’s tool inputs match the schema using grammar-constrained sampling. (Claude)
One caveat: strict schemas have complexity limits. Very deep nesting or very large schemas may not be fully enforceable in strict mode, so the practical workaround is to split or flatten the schema (for example, a separate extraction tool per document section, merged afterward) rather than relying on one giant nested schema. This does not change the core point: even when strict mode is in effect, it removes syntax/schema errors but not the semantic errors covered next.
2.2 Semantic validation errors
These are errors where the output is structurally valid but wrong.
Examples:
The schema is valid, but:
Other semantic errors:
Invoice due date placed in invoice_date.
Vendor name placed in customer_name.
Line items do not sum to total.
Extracted date is normalized incorrectly.
Required value is fabricated even though source is silent.
Source document contains conflicting values.
Review finding points to a line but the issue is actually pre-existing.
Exam phrase to remember:
3. Retry-with-error-feedback
Retry-with-error-feedback means you do not simply say:
Instead, you send Claude:
1. The original document
2. The failed extraction
3. The exact validation errors
4. The correction rules
5. The same schema/tool requirement
This gives Claude a concrete repair target.
3.1 Weak retry
Why weak:
Claude does not know which fields failed.
It may change correct fields.
It may introduce new mistakes.
It may fabricate values to satisfy the correction.
3.2 Strong retry
Your previous extraction failed validation.
Original document:
<document>
Invoice INV-1042
Subtotal: $100.00
Tax: $8.00
Shipping: $5.00
Total: $113.00
</document>
Previous extraction:
{
"invoice_number": "INV-1042",
"subtotal": 100,
"tax": 8,
"shipping": 5,
"stated_total": 200,
"currency": "USD"
}
Validation errors:
1. stated_total is inconsistent with the source document. Source says Total: $113.00, but extracted stated_total is 200.
2. calculated_total should equal subtotal + tax + shipping = 113.
Retry the extraction.
Rules:
- Use only values supported by the original document.
- Preserve correct fields when possible.
- Do not fabricate missing values.
- Return null for unavailable fields.
This is the canonical D4.4 pattern.
4. Follow-up request structure
A good retry prompt has a repeatable shape:
Original document:
...
Failed extraction:
...
Validation errors:
...
Correction instructions:
...
Re-extract using the same schema.
Example
<original_document>
{{DOCUMENT}}
</original_document>
<failed_extraction>
{{FAILED_JSON}}
</failed_extraction>
<validation_errors>
- invoice_date was extracted as "2026-07-01", but the source labels that date as the due date.
- invoice_date is absent from the source. It should be null.
- due_date should be "2026-07-01".
</validation_errors>
<retry_instructions>
Correct only the fields implicated by validation errors unless another field is directly affected.
Use only source-supported values.
Return null when the source does not contain the requested information.
</retry_instructions>
Exam point:
A retry should be diagnostic, not generic.
5. When retries work
Retries are useful when the information exists in the source and the model made a correctable mistake.
Good retry candidates:
Wrong field placement
Wrong date normalization
Wrong enum selection
Missed value that is present in source
Line items do not sum correctly
Conflicting source values need to be flagged
Output omitted an evidence quote
Ambiguous value should be marked unclear instead of guessed
Example: wrong field placement
Source:
Failed extraction:
Validation error:
A retry can fix this because the correct values are present.
6. When retries do not work
Retries are ineffective when the required information is absent from the provided source.
Bad retry candidate:
But the document does not contain a tax ID.
A retry cannot recover information that is not in the context.
Correct behavior:
Not:
Exam distinction
Information present but extracted incorrectly → retry may help.
Information absent from source → retry will not help.
Examples where retry is ineffective:
The answer exists only in an external document not provided.
The document references an appendix that was not included.
The source says “see attached schedule,” but the schedule is absent.
The field is genuinely not present.
The source is illegible or redacted.
The extraction requires database lookup but no tool/context is provided.
Best response:
7. Designing semantic validators
A validator is application code that checks whether the extraction makes sense.
7.1 Invoice total validator
def validate_invoice(extraction: dict) -> list[dict]:
errors = []
subtotal = extraction.get("subtotal")
tax = extraction.get("tax")
shipping = extraction.get("shipping")
stated_total = extraction.get("stated_total")
if None not in (subtotal, tax, shipping):
calculated_total = round(subtotal + tax + shipping, 2)
if stated_total is not None and round(stated_total, 2) != calculated_total:
errors.append({
"field": "stated_total",
"code": "TOTAL_MISMATCH",
"message": (
f"stated_total={stated_total} does not equal "
f"subtotal + tax + shipping = {calculated_total}"
),
"retryable": True
})
return errors
This catches semantic inconsistency after schema-valid extraction.
7.2 Contract date validator
def validate_contract_dates(extraction: dict) -> list[dict]:
errors = []
effective_date = extraction.get("effective_date")
termination_date = extraction.get("termination_date")
renewal_date = extraction.get("renewal_date")
if effective_date and termination_date and termination_date < effective_date:
errors.append({
"field": "termination_date",
"code": "DATE_ORDER_INVALID",
"message": "termination_date is earlier than effective_date.",
"retryable": True
})
if renewal_date is None and extraction.get("renewal_rule_evidence"):
errors.append({
"field": "renewal_date",
"code": "DERIVABLE_DATE_MISSING",
"message": "renewal_date may be derivable from renewal_rule_evidence.",
"retryable": True
})
return errors
8. Self-correction schema patterns
The official task statement calls out:
These are schema designs that make validation easier.
8.1 calculated_total and stated_total
Instead of extracting only:
extract:
{
"subtotal": 100,
"tax": 8,
"shipping": 5,
"calculated_total": 113,
"stated_total": 113,
"total_discrepancy_detected": false
}
Schema idea:
{
"subtotal": { "type": ["number", "null"] },
"tax": { "type": ["number", "null"] },
"shipping": { "type": ["number", "null"] },
"calculated_total": {
"type": ["number", "null"],
"description": "subtotal + tax + shipping when all components are present; otherwise null."
},
"stated_total": {
"type": ["number", "null"],
"description": "The total amount explicitly stated in the source document."
},
"total_discrepancy_detected": {
"type": "boolean",
"description": "True when calculated_total and stated_total are both present and differ."
}
}
Why this helps:
It separates what the document states from what the numbers imply.
It allows downstream validation.
It avoids hiding discrepancies by forcing one total field.
8.2 conflict_detected
Use this when the source may contain inconsistent data.
Example source:
A 12-month term from January 1, 2026 would end around January 1, 2027, but the stated termination date says 2028.
Output:
{
"effective_date": "2026-01-01",
"term_months": 12,
"termination_date": "2028-12-31",
"conflict_detected": true,
"conflict_details": "The stated 12-month term conflicts with the stated termination date of 2028-12-31.",
"evidence": "Contract Term: 12 months ... Termination Date: December 31, 2028 ... Effective Date: January 1, 2026"
}
Why this is better than silently choosing one value:
It preserves source uncertainty.
It avoids pretending conflicting data is clean.
It gives the downstream system a review flag.
9. Retry decision table
| Validation result | Retry? | Reason |
|---|---|---|
| Malformed JSON | Usually no if strict tool use; otherwise yes | Structural issue |
| Missing required schema field | Usually no if strict tool use; otherwise yes | Structural issue |
| Date in wrong field | Yes | Correct value is present |
| Total mismatch | Yes | Model may correct arithmetic or field selection |
Ambiguous source date 01/02/2026 |
Maybe no | Better return null/unclear unless locale evidence exists |
| Tax ID absent from document | No | Information unavailable |
| Referenced appendix missing | No | Need missing document |
| Vendor/customer swapped | Yes | Source likely contains both labels |
| Source contains two contradictory totals | Retry only to set conflict flag | Do not force a clean answer |
| External lookup required but no tool/context | No | Cannot infer external facts |
10. Retry loop design
10.1 Basic algorithm
MAX_RETRIES = 2
def extract_with_validation(document: str) -> dict:
extraction = call_extraction_tool(document)
validation_errors = validate_extraction(extraction)
attempt = 0
while validation_errors and attempt < MAX_RETRIES:
retryable_errors = [
err for err in validation_errors
if err.get("retryable") is True
]
if not retryable_errors:
break
extraction = call_extraction_tool_with_feedback(
document=document,
failed_extraction=extraction,
validation_errors=retryable_errors
)
validation_errors = validate_extraction(extraction)
attempt += 1
return {
"extraction": extraction,
"validation_errors": validation_errors,
"status": "valid" if not validation_errors else "needs_review"
}
Key design:
Retry only retryable errors.
Cap retries.
Preserve validation errors after final failure.
Route unresolved cases to review.
10.2 Retry prompt builder
def build_retry_prompt(document: str, failed_extraction: dict, validation_errors: list[dict]) -> str:
return f"""
Your previous extraction failed validation.
<original_document>
{document}
</original_document>
<failed_extraction>
{failed_extraction}
</failed_extraction>
<validation_errors>
{validation_errors}
</validation_errors>
Please re-extract using the same schema.
Rules:
- Use only information present in the original document.
- Correct fields implicated by validation errors.
- Preserve correct fields unless they conflict with the validation errors.
- Return null for information absent from the document.
- If the source contains conflicting values, set conflict_detected=true and explain the conflict.
"""
11. Feedback loop design for review findings
D4.4 also moves beyond extraction into review-quality improvement.
The official statement says:
This is how you analyze false positives systematically.
Claude Code’s Code Review docs say review findings can be rated as useful or wrong/noisy, and that Anthropic uses reaction counts to tune the reviewer. They also describe a verification step that checks candidate findings against code behavior to filter false positives. (Claude)
In your own pipeline, you can build a similar feedback loop by storing structured metadata for each finding.
11.1 Add detected_pattern
Example structured finding schema:
{
"type": "object",
"properties": {
"file": { "type": "string" },
"line": { "type": "integer" },
"severity": {
"type": "string",
"enum": ["important", "nit"]
},
"category": {
"type": "string",
"enum": ["correctness", "security", "performance", "test_gap", "comment_mismatch", "other"]
},
"detected_pattern": {
"type": "string",
"enum": [
"null_dereference",
"n_plus_one_query",
"missing_auth_check",
"missing_branch_test",
"comment_code_contradiction",
"speculative_performance",
"other"
]
},
"detected_pattern_detail": {
"type": ["string", "null"]
},
"message": { "type": "string" },
"suggested_fix": { "type": "string" }
},
"required": [
"file",
"line",
"severity",
"category",
"detected_pattern",
"detected_pattern_detail",
"message",
"suggested_fix"
]
}
Why this matters:
You can aggregate dismissed findings by detected_pattern.
You can see which patterns cause false positives.
You can tune only the noisy category instead of disabling everything.
11.2 Example analysis
Suppose developers dismiss findings like this:
n_plus_one_query: 8 dismissed, 7 accepted
missing_auth_check: 1 dismissed, 12 accepted
speculative_performance: 22 dismissed, 1 accepted
comment_code_contradiction: 15 dismissed, 2 accepted
What should you do?
Keep missing_auth_check.
Keep or lightly tune n_plus_one_query.
Disable or sharply narrow speculative_performance.
Improve comment_code_contradiction prompt with stricter criteria and examples.
Exam point:
detected_patternturns subjective dismissal feedback into prompt-engineering data.
12. Using feedback to tune prompts
If dismissed findings cluster around:
Do not just say:
Instead tune the category:
Do not report performance findings unless all are true:
1. The diff introduces repeated I/O, database calls, network calls, or expensive computation inside a loop.
2. The loop iterates over an unbounded production-sized collection.
3. The previous code avoided the repeated operation.
4. The finding includes the loop, the repeated operation, and the production path.
Skip:
- in-memory map/filter/reduce over already-loaded arrays
- speculative caching suggestions
- small constant-size loops
- performance comments without source evidence
This connects directly to D4.1 and D4.2:
D4.1: Add explicit criteria.
D4.2: Add few-shot examples.
D4.4: Use dismissal data to identify which criteria/examples need improvement.
13. Extraction schema with validation-friendly fields
Here is a strong invoice extraction schema for D4.4.
{
"type": "object",
"properties": {
"invoice_number": {
"type": ["string", "null"]
},
"currency": {
"type": ["string", "null"],
"description": "ISO 4217 currency code, or null if absent."
},
"subtotal": {
"type": ["number", "null"]
},
"tax": {
"type": ["number", "null"]
},
"shipping": {
"type": ["number", "null"]
},
"discount": {
"type": ["number", "null"]
},
"calculated_total": {
"type": ["number", "null"],
"description": "subtotal + tax + shipping - discount when all required components are available."
},
"stated_total": {
"type": ["number", "null"],
"description": "Total explicitly stated in the invoice."
},
"total_discrepancy_detected": {
"type": "boolean"
},
"conflict_detected": {
"type": "boolean"
},
"conflict_details": {
"type": ["string", "null"]
},
"missing_fields": {
"type": "array",
"items": {
"type": "string"
}
},
"evidence": {
"type": "string"
}
},
"required": [
"invoice_number",
"currency",
"subtotal",
"tax",
"shipping",
"discount",
"calculated_total",
"stated_total",
"total_discrepancy_detected",
"conflict_detected",
"conflict_details",
"missing_fields",
"evidence"
],
"additionalProperties": false
}
Note: Strict schemas have complexity limits, especially around optional fields and union types like ["string", "null"]; Anthropic’s structured-output docs recommend simplifying schemas, flattening nested objects, and splitting into multiple requests if schemas become too complex. (Claude)
14. Worked example: extraction + validation + retry
Source document
Invoice INV-9001
Subtotal: $120.00
Tax: $9.60
Shipping: $10.00
Total: $139.60
Due Date: July 15, 2026
Failed extraction
{
"invoice_number": "INV-9001",
"subtotal": 120,
"tax": 9.6,
"shipping": 10,
"calculated_total": 139.6,
"stated_total": 149.6,
"invoice_date": "2026-07-15",
"due_date": null
}
Validation errors
[
{
"field": "stated_total",
"code": "TOTAL_MISMATCH",
"message": "Source states Total: $139.60 but extraction has stated_total=149.60.",
"retryable": true
},
{
"field": "invoice_date",
"code": "WRONG_FIELD_PLACEMENT",
"message": "July 15, 2026 is labeled Due Date, not Invoice Date.",
"retryable": true
}
]
Retry prompt
Your previous extraction failed validation.
Original document:
Invoice INV-9001
Subtotal: $120.00
Tax: $9.60
Shipping: $10.00
Total: $139.60
Due Date: July 15, 2026
Previous extraction:
{
"invoice_number": "INV-9001",
"subtotal": 120,
"tax": 9.6,
"shipping": 10,
"calculated_total": 139.6,
"stated_total": 149.6,
"invoice_date": "2026-07-15",
"due_date": null
}
Validation errors:
1. Source states Total: $139.60 but extraction has stated_total=149.60.
2. July 15, 2026 is labeled Due Date, not Invoice Date.
Retry extraction using the same schema.
Use only source-supported values.
Return null for invoice_date because no invoice date is present.
Set due_date to 2026-07-15.
Corrected extraction
{
"invoice_number": "INV-9001",
"subtotal": 120,
"tax": 9.6,
"shipping": 10,
"calculated_total": 139.6,
"stated_total": 139.6,
"invoice_date": null,
"due_date": "2026-07-15"
}
15. Common exam traps
Trap 1: Retrying without error details
Wrong:
Right:
Trap 2: Retrying absent information
Wrong:
Right:
Trap 3: Treating strict schema as full validation
Wrong:
Right:
Trap 4: No feedback metadata
Wrong finding:
Better finding:
{
"category": "performance",
"detected_pattern": "speculative_performance",
"message": "This may be slow.",
"evidence": "...",
"severity": "nit"
}
Why:
Trap 5: Hiding source conflicts
Wrong:
when the source contains conflicting term data.
Better:
{
"termination_date": "2028-12-31",
"conflict_detected": true,
"conflict_details": "12-month term conflicts with stated 2028 termination date."
}
16. Scenario-based practice questions
Question 1
An extraction returns valid JSON matching the schema, but the invoice line items sum to $113.00 while the extracted total is $200.00.
What kind of error is this?
A) Schema syntax error
B) Semantic validation error
C) Tool selection error only
D) JSON parsing error
Answer
B. The output is structurally valid but semantically inconsistent.
Question 2
A retry prompt says only:
What is the best improvement?
A) Include the original document, failed extraction, and specific validation errors.
B) Remove the schema.
C) Increase temperature.
D) Force Claude to invent missing fields.
Answer
A. Retry-with-error-feedback works because the model receives precise correction signals.
Question 3
A contract does not contain a termination date. The validator reports termination_date_missing.
Should the system retry?
A) Yes, always retry missing fields.
B) No, if the information is absent from the source. Return null or missing_from_source.
C) Yes, because retry can create the missing value.
D) Only if the schema is removed.
Answer
B. Retries cannot extract information that is not in the provided source.
Question 4
The source says:
The extraction puts this value into invoice_date.
Should retry help?
A) Yes, because the value exists but is in the wrong field.
B) No, because the date is absent.
C) No, because schemas cannot contain dates.
D) Only if the field is required.
Answer
A. Wrong field placement is a correctable semantic error.
Question 5
A review system has many dismissed findings. The team wants to know which code constructs trigger false positives.
What field should the structured finding include?
A) detected_pattern
B) temperature
C) random_seed
D) font_size
Answer
A. detected_pattern enables aggregation and analysis of false-positive patterns.
Question 6
Developers dismiss 90% of findings with:
What is the best response?
A) Disable all code review.
B) Temporarily disable or narrow speculative performance findings and add stricter criteria/examples.
C) Mark all performance findings Important.
D) Remove detected_pattern.
Answer
B. Feedback data should be used to tune noisy categories without disabling accurate ones.
Question 7
An invoice schema has both stated_total and calculated_total.
Why is this useful?
A) It makes the schema shorter.
B) It lets the system detect discrepancies between the document’s stated value and arithmetic over line items.
C) It prevents all extraction errors.
D) It removes the need for validation.
Answer
B. Separate stated and calculated values make semantic validation possible.
Question 8
A source document contains two conflicting renewal dates. What should the extraction schema support?
A) A conflict_detected boolean and conflict details.
B) Choosing one date silently.
C) Fabricating a third date.
D) Returning malformed JSON.
Answer
A. Conflicts should be surfaced explicitly rather than hidden.
Question 9
Strict tool use eliminates which class of problem?
A) Semantic inconsistency
B) Missing source information
C) Schema syntax/structural errors such as malformed JSON and wrong primitive types
D) Wrong business interpretation in all cases
Answer
C. Strict tool use guarantees schema-conformant tool input, not semantic correctness.
Question 10
A document says:
Appendix B is not provided. The extraction needs itemized pricing.
What should the system do?
A) Retry until pricing appears.
B) Return null/missing_from_source and request Appendix B if needed.
C) Guess pricing from context.
D) Use the invoice total as every line item.
Answer
B. The required information is outside the provided context; retry is ineffective.
17. Final D4.4 checklist
Memorize this:
1. Structured output solves syntax/schema problems, not all quality problems.
2. Semantic validators catch wrong values in valid JSON.
3. Retry with specific error feedback, not generic “try again.”
4. Retry prompts should include original document, failed extraction, and validation errors.
5. Retry works when information is present but mis-extracted.
6. Retry does not work when information is absent from the source.
7. Return null, unclear, or missing_from_source for absent data.
8. Cap retries to avoid loops.
9. Separate retryable and non-retryable validation errors.
10. Use stated_total and calculated_total to detect arithmetic discrepancies.
11. Use conflict_detected for inconsistent source data.
12. Preserve evidence fields to support validation.
13. Add detected_pattern to structured findings.
14. Aggregate dismissed findings by detected_pattern.
15. Use feedback patterns to tune prompts, schemas, and examples.
D4.4 mental model
Tool schema makes output parseable.
Semantic validation checks whether it is correct.
Retry feedback tells Claude exactly what to repair.
Absent information should become null, not a hallucination.
detected_pattern turns developer dismissals into prompt-improvement data.
The exam’s favorite distinction is:
Retry is useful for correctable extraction mistakes, not for missing source information. Validation must check meaning, not just JSON shape.