Design Human Review Workflows and Confidence Calibration
This module is about deciding when AI-extracted data can be trusted automatically and when humans must review it.
The exam’s core idea:
Do not reduce human review based on aggregate accuracy alone.
Measure accuracy by document type, field, confidence band, and source condition.
Then route human review where it reduces the most risk.
Anthropic’s evaluation guidance starts from the same principle: define success criteria, build evaluations, and test against those criteria rather than relying on vague impressions of quality. Their Console Evaluation tool is designed to compare prompt outputs across test cases, grade quality, and rerun suites after prompt changes. (Claude)
1. Core mental model
Use this workflow:
1. Extract structured data.
2. Have the model output field-level confidence and evidence.
3. Validate against labeled human-reviewed data.
4. Measure accuracy by document type and field.
5. Calibrate confidence thresholds using validation results.
6. Route uncertain, ambiguous, contradictory, or high-risk cases to humans.
7. Continue sampling high-confidence items to detect hidden error patterns.
Exam shortcut:
Aggregate accuracy tells you whether the system looks good overall.
Segmented accuracy tells you where it is safe to automate.
Stratified sampling tells you whether “high confidence” is still trustworthy over time.
2. Why aggregate accuracy is dangerous
A model may report:
That sounds excellent. But it may hide failures like:
Invoices: 99% accurate
Receipts: 98% accurate
Purchase orders: 82% accurate
Vendor name: 99% accurate
Invoice date: 98% accurate
Total amount: 96% accurate
Payment terms: 71% accurate
Google’s ML guidance makes the same general evaluation point: metrics over an entire validation set can hide poor performance on subgroups, so auditing should break results out by subgroup rather than relying only on overall accuracy. (Google for Developers)
Exam implication
Do not say:
Say:
3. Segment-level evaluation
For document extraction, useful segments include:
document_type: invoice, receipt, purchase_order, contract, bank_statement
field_name: vendor_name, invoice_date, due_date, total_amount, tax_id, payment_terms
source_format: scanned_pdf, native_pdf, email_body, spreadsheet, photo
vendor/customer template: known_vendor, unknown_vendor, new_template
confidence_band: 0.95–1.00, 0.90–0.95, 0.80–0.90, below_0.80
risk_level: low, medium, high
ambiguity_flag: clear, ambiguous, contradictory
A good evaluation table:
| Segment | Accuracy | Decision |
|---|---|---|
| Invoice / vendor_name | 99.3% | Candidate for automation |
| Invoice / total_amount | 98.8% | Candidate for automation with audit sampling |
| Purchase order / payment_terms | 81.5% | Keep human review |
| Contract / renewal_date | 88.0% | Keep human review; improve prompt/schema |
| Scanned receipt / tax_amount | 76.2% | Human review required |
Exam rule:
4. Field-level confidence
A single document-level confidence score is not enough.
Bad:
{
"document_confidence": 0.96,
"vendor_name": "Acme Corp",
"invoice_date": "2026-06-01",
"total_amount": 1299.50,
"payment_terms": "Net 30"
}
Problem:
Some fields may be easy and others uncertain.
The document can be high-confidence overall while one field is wrong.
Better:
{
"document_type": "invoice",
"fields": {
"vendor_name": {
"value": "Acme Corp",
"confidence": 0.98,
"evidence": "Vendor: Acme Corp",
"needs_review": false
},
"invoice_date": {
"value": "2026-06-01",
"confidence": 0.96,
"evidence": "Invoice Date: June 1, 2026",
"needs_review": false
},
"payment_terms": {
"value": "Net 30",
"confidence": 0.61,
"evidence": "Payment due within thirty days unless otherwise agreed",
"needs_review": true
}
},
"document_flags": {
"ambiguous_source": false,
"contradiction_detected": false
}
}
Exam shortcut:
5. Confidence is not automatically calibrated
A model saying:
does not automatically mean:
Confidence must be calibrated against labeled validation data. Anthropic’s eval guidance emphasizes empirical testing against success criteria, and Google’s ML glossary describes validation/test sets as the basis for judging supervised model behavior; for routing workflows, the same idea applies to confidence: compare predicted confidence to human-labeled correctness. (Claude)
Calibration table example
| Model confidence band | Actual field accuracy | Interpretation |
|---|---|---|
| 0.95–1.00 | 99.2% | Well calibrated for this segment |
| 0.90–0.95 | 96.0% | Mostly usable with sampling |
| 0.80–0.90 | 89.5% | Human review for high-risk fields |
| 0.70–0.80 | 78.0% | Human review |
| below 0.70 | 61.0% | Human review + prompt/schema improvement |
But calibration may differ by segment:
| Segment | Confidence ≥ 0.95 actual accuracy |
|---|---|
| Native invoice / total_amount | 99.5% |
| Scanned receipt / tax_amount | 91.0% |
| Contract / renewal_date | 86.0% |
| Purchase order / payment_terms | 80.0% |
Exam point:
6. Labeled validation sets
A labeled validation set contains human-reviewed ground truth.
For extraction, each labeled example should include:
document_id
document_type
source_format
field_name
model_extracted_value
human_labeled_value
model_confidence
correct/incorrect
error_type
reviewer_notes
Example row:
{
"document_id": "INV-00091",
"document_type": "invoice",
"source_format": "native_pdf",
"field_name": "invoice_date",
"model_value": "2026-06-01",
"human_value": "2026-06-01",
"model_confidence": 0.97,
"correct": true,
"error_type": null
}
Incorrect example:
{
"document_id": "PO-00412",
"document_type": "purchase_order",
"source_format": "scanned_pdf",
"field_name": "payment_terms",
"model_value": "Net 30",
"human_value": "Net 45",
"model_confidence": 0.94,
"correct": false,
"error_type": "wrong_field_extraction"
}
Use this labeled data to decide thresholds.
7. Stratified random sampling
Stratified sampling means:
A general definition of stratified sampling is to divide the population into homogeneous subgroups, then randomly sample within those groups so evaluation is more representative of those subgroups. (Baeldung on Kotlin)
For this exam, the important twist is:
Why?
Good stratification dimensions
document_type
field_name
confidence_band
source_format
vendor/template family
risk tier
new vs known document layout
ambiguous vs clean source
Example sampling plan
Every week, sample:
- 2% of high-confidence invoice totals
- 5% of high-confidence purchase order payment terms
- 10% of scanned receipt tax fields
- 100% of low-confidence high-risk fields
- 100% of contradictory or ambiguous-source documents
- a fixed minimum sample count per rare document type
The goal is not only error-rate measurement. It is also novel pattern detection.
8. Review routing with limited human capacity
Human review capacity is limited, so route intelligently.
Priority order:
1. Contradictory source documents
2. Ambiguous documents
3. Low-confidence high-risk fields
4. Underperforming document-type/field segments
5. New document templates or vendors
6. Random sample of high-confidence items
7. Low-risk high-confidence segments
Routing table
| Case | Route |
|---|---|
| Low confidence total_amount | Human review |
| High confidence total_amount in validated invoice segment | Auto-accept + sample audit |
| High confidence payment_terms in poor-performing PO segment | Human review or higher sampling |
| Contradictory totals in source | Human review |
| New vendor template | Human review until segment validated |
Ambiguous date 01/02/2026 |
Human review or return ambiguous |
| High confidence but from rare doc type | Sample or review until enough data |
Confidence thresholding is commonly used as a workflow-routing technique, but it should be backed by validation data and not treated as a universal truth. Google’s classification material describes thresholds as decision boundaries, and this module applies the same idea to human-review routing thresholds. (Google for Developers)
9. Progressive automation
Do not go directly from 100% human review to full automation.
Use progressive automation:
Phase 1: 100% human review
Phase 2: Collect labeled validation set
Phase 3: Analyze accuracy by document type and field
Phase 4: Calibrate field-level confidence thresholds
Phase 5: Auto-accept only validated high-confidence segments
Phase 6: Continue stratified random sampling
Phase 7: Expand automation only when segment metrics remain stable
Example
Start:
After validation:
Auto-accept:
- invoice.vendor_name when confidence ≥ 0.95
- invoice.invoice_date when confidence ≥ 0.95
- invoice.total_amount when confidence ≥ 0.98
Still human-review:
- purchase_order.payment_terms
- contract.renewal_date
- any field with contradiction_detected = true
- any new document template
Exam point:
10. Confidence + evidence + flags
A good extraction should provide enough data for routing.
{
"document_id": "INV-1042",
"document_type": "invoice",
"document_type_confidence": 0.99,
"source_format": "native_pdf",
"fields": {
"invoice_number": {
"value": "INV-1042",
"confidence": 0.99,
"evidence": "Invoice No: INV-1042",
"ambiguity": false,
"contradiction": false
},
"invoice_date": {
"value": "2026-06-01",
"confidence": 0.93,
"evidence": "Date: 06/01/2026",
"ambiguity": true,
"ambiguity_reason": "Date could be MM/DD/YYYY or DD/MM/YYYY without locale evidence.",
"contradiction": false
},
"total_amount": {
"value": 1299.5,
"confidence": 0.97,
"evidence": "Total Due: $1,299.50",
"ambiguity": false,
"contradiction": false
}
}
}
Routing result:
{
"needs_human_review": true,
"review_reasons": [
"invoice_date is ambiguous despite moderate confidence"
],
"auto_accepted_fields": [
"invoice_number",
"total_amount"
],
"fields_for_review": [
"invoice_date"
]
}
Exam shortcut:
Confidence alone is not enough.
Use confidence + ambiguity + contradiction + segment performance + field risk.
11. Field risk levels
Some fields deserve stricter thresholds.
| Field | Risk | Suggested routing |
|---|---|---|
| invoice_number | Medium | Auto if validated high confidence |
| vendor_name | Medium | Auto if segment validated |
| total_amount | High | Higher threshold + audit sample |
| bank_account | Very high | Human review or dual validation |
| tax_id | High | Human review until highly validated |
| renewal_date | High | Human review if derived/ambiguous |
| memo/description | Low | Lower threshold acceptable |
| payment_terms | Medium/high | Segment-specific threshold |
Exam point:
12. Novel error pattern detection
If you only review low-confidence outputs, you may never catch a high-confidence systematic error.
Example:
Confidence:
If high-confidence outputs bypass all review, this error pattern may persist at scale.
That is why the exam emphasizes:
Review finding
{
"error_pattern": "ship_date_extracted_as_invoice_date",
"document_type": "invoice",
"vendor_template": "new_vendor_A",
"confidence_band": "0.95-1.00",
"detected_by": "high_confidence_stratified_sample",
"action": "route all new_vendor_A invoice_date fields to human review until fixed"
}
13. Human review queue design
A good queue is prioritized, not first-in-first-out.
{
"review_item_id": "review-91822",
"document_id": "PO-00412",
"document_type": "purchase_order",
"field_name": "payment_terms",
"model_value": "Net 30",
"model_confidence": 0.94,
"segment_accuracy": 0.81,
"risk_level": "medium",
"ambiguity_detected": false,
"contradiction_detected": false,
"priority_score": 0.86,
"review_reason": [
"Segment purchase_order/payment_terms under threshold",
"High confidence but historically error-prone field"
]
}
Priority score can consider:
low confidence
high business risk
poor segment performance
new template/vendor
ambiguous or contradictory source
random audit sampling requirement
customer impact
regulatory impact
14. Example: threshold calibration
Suppose validation data shows:
| Segment | Confidence threshold | Actual accuracy | Automation decision |
|---|---|---|---|
| invoice.total_amount | ≥ 0.98 | 99.4% | Auto + 2% audit |
| invoice.invoice_date | ≥ 0.95 | 97.5% | Auto + 5% audit |
| receipt.tax_amount | ≥ 0.95 | 91.2% | Keep review |
| contract.renewal_date | ≥ 0.95 | 88.0% | Keep review |
| purchase_order.payment_terms | ≥ 0.95 | 80.5% | Keep review + prompt fix |
A bad policy:
A good policy:
Auto-accept only field/document-type segments whose labeled validation accuracy meets the required threshold, and continue stratified sampling.
15. Common exam traps
Trap 1: Aggregate accuracy
Wrong:
Right:
Trap 2: Confidence without calibration
Wrong:
Right:
Trap 3: Review only low-confidence cases
Wrong:
Right:
Continue stratified random sampling of high-confidence outputs to detect confidently wrong patterns.
Trap 4: Document-level confidence only
Wrong:
Right:
Trap 5: Automate all fields in a validated document type
Wrong:
Right:
Trap 6: Random sampling without strata
Wrong:
Right:
16. Scenario-based practice questions
Question 1
A document extraction system reports 97% overall accuracy. The team wants to stop human review for all high-confidence outputs.
What is the best next step?
A) Stop review immediately.
B) Analyze accuracy by document type and field before reducing review.
C) Trust all confidence scores.
D) Review only documents with low total token count.
Answer
B. Aggregate accuracy can hide poor performance on specific fields or document types.
Question 2
A model outputs document-level confidence of 0.98, but one field, payment_terms, is often wrong.
What should the system use?
A) Only document-level confidence.
B) Field-level confidence and field-level validation metrics.
C) Random guessing.
D) No confidence scores.
Answer
B. Review routing should happen at the field level, especially when fields vary in difficulty and risk.
Question 3
The model says confidence: 0.95 for extracted dates. Validation shows those dates are correct only 84% of the time for scanned contracts.
What does this mean?
A) Confidence is calibrated.
B) Confidence is overconfident for that segment and should not be used directly for automation.
C) Human labels are irrelevant.
D) All scanned contracts should be auto-accepted.
Answer
B. Confidence must be calibrated by segment using labeled validation data.
Question 4
The team reviews only low-confidence extractions. Later, they discover a new vendor template where the model confidently extracts the wrong date field.
What sampling strategy would have caught this earlier?
A) Stratified random sampling of high-confidence extractions.
B) Reviewing only low-confidence cases.
C) No sampling.
D) Sampling only short documents.
Answer
A. High-confidence sampling detects confidently wrong novel error patterns.
Question 5
Which is the best human review routing rule?
A) Review only fields with confidence below 0.8.
B) Review fields with low confidence, ambiguous or contradictory source evidence, poor segment performance, high business risk, or new/unvalidated templates.
C) Review every field forever.
D) Review no high-confidence fields.
Answer
B. Good routing combines confidence, ambiguity, contradiction, segment performance, and risk.
Question 6
A validation set shows:
invoice.total_amount at confidence ≥ 0.98 = 99.5% accurate
purchase_order.payment_terms at confidence ≥ 0.98 = 82% accurate
What should the automation policy do?
A) Auto-accept both because confidence is ≥ 0.98.
B) Auto-accept invoice totals with audit sampling, but keep purchase order payment terms in human review.
C) Reject all outputs.
D) Use document-level accuracy only.
Answer
B. Thresholds must be calibrated by document type and field.
Question 7
A source document contains two contradictory totals.
What should happen?
A) Auto-accept the model’s higher-confidence value.
B) Route to human review because the source is contradictory.
C) Ignore the contradiction.
D) Average the totals.
Answer
B. Contradictory source documents should be routed to human review.
Question 8
What should a labeled validation set include?
A) Only model outputs.
B) Human-labeled ground truth, model values, confidence scores, document type, field name, correctness, and error type.
C) Only timestamps.
D) Only document IDs.
Answer
B. Calibration requires comparing model outputs and confidence against human-labeled truth.
Question 9
The team has limited reviewer capacity. Which items should be prioritized?
A) Random low-risk high-confidence fields only.
B) Low-confidence, high-risk, ambiguous/contradictory, poor-segment, and new-template cases.
C) The shortest documents.
D) The oldest documents only.
Answer
B. Review capacity should be spent where risk and uncertainty are highest, while still sampling high-confidence segments.
Question 10
Which policy is safest before reducing human review?
A) Validate only overall accuracy.
B) Validate accuracy by document type and field segment, calibrate confidence thresholds, then progressively automate validated segments while continuing stratified sampling.
C) Trust confidence scores immediately.
D) Automate everything above 0.9 confidence.
Answer
B. This is the full D5.5 workflow.
17. Final D5.5 checklist
Memorize this:
1. Aggregate accuracy can hide weak document types or fields.
2. Analyze accuracy by document type.
3. Analyze accuracy by field.
4. Analyze accuracy by confidence band.
5. Do not automate based on overall accuracy alone.
6. Use field-level confidence, not only document-level confidence.
7. Calibrate confidence with labeled validation sets.
8. A confidence score is not automatically a probability.
9. Review thresholds must be segment-specific.
10. Route low-confidence fields to human review.
11. Route ambiguous or contradictory source documents to human review.
12. Route poor-performing segments to human review.
13. Prioritize limited reviewer capacity by risk and uncertainty.
14. Continue stratified random sampling of high-confidence outputs.
15. Use high-confidence samples to detect novel error patterns.
16. Reduce human review progressively, segment by segment.
D5.5 mental model
Overall accuracy tells you the average.
Segment accuracy tells you the risk.
Field confidence tells you where to look.
Calibration tells you whether confidence means anything.
Stratified sampling tells you whether automation remains safe.
Human review goes where risk, ambiguity, and uncertainty are highest.
The exam’s favorite distinction is:
High overall accuracy and high model confidence are not enough. Validate and calibrate by document type and field before reducing human review.