Design Efficient Batch Processing Strategies
This module is about choosing between synchronous API calls and the Message Batches API based on latency, cost, scale, and retry behavior.
The exam’s core distinction:
The Message Batches API is designed for asynchronous large-volume processing when immediate responses are not required; Anthropic’s docs describe it as suitable for large-scale evaluations, moderation, analysis, summarization, and bulk content generation. It provides 50% pricing compared with standard API prices, but batches are processed asynchronously and may take up to 24 hours, so it is not appropriate when a user or CI pipeline is blocked waiting for the answer. (Claude)
1. Core mental model
Use this decision rule:
Need result now? Use synchronous API.
Can wait hours? Use Message Batches API.
Need lower cost at scale? Use Message Batches API.
Need interactive tool loop or blocking CI? Use synchronous API.
| Workflow | Best API |
|---|---|
| Pre-merge PR check that must finish before merge | Synchronous API |
| Inline code review comment while developer waits | Synchronous API |
| Overnight security audit of 20,000 files | Batch API |
| Weekly contract extraction job | Batch API |
| Nightly test-generation suggestions | Batch API |
| Offline eval run across 50,000 examples | Batch API |
| User-facing chat response | Synchronous API |
| CI gate with strict latency expectation | Synchronous API |
The Message Batches API reduces cost and increases throughput for non-immediate workloads, but Anthropic’s docs say batches are processed asynchronously, results are available when all messages complete or after 24 hours, and processing may slow depending on demand and request volume. (Claude)
2. Message Batches API basics
A batch request contains many independent Messages API requests. Each item has:
Example shape:
{
"requests": [
{
"custom_id": "doc_0001",
"params": {
"model": "claude-sonnet-4-6",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Extract structured metadata from document 0001..."
}
]
}
},
{
"custom_id": "doc_0002",
"params": {
"model": "claude-sonnet-4-6",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Extract structured metadata from document 0002..."
}
]
}
}
]
}
Anthropic’s docs state that each batch request needs a unique custom_id and a params object with the usual Messages API parameters; custom_id must be 1–64 characters and match the allowed alphanumeric, hyphen, and underscore pattern. (Claude)
3. Why custom_id matters
Batch responses are not guaranteed to return in the same order as requests. Anthropic’s docs explicitly say results may be returned in any order and that you should use custom_id to match responses back to inputs. (Claude)
Bad:
Good:
documents_by_id = {
"doc_0001": doc1,
"doc_0002": doc2,
}
for result in batch_results:
original_doc = documents_by_id[result["custom_id"]]
Exam shortcut:
4. When to use batch processing
Use batch processing for workloads that are:
Good examples:
Overnight report generation
Weekly audit of code review findings
Nightly test generation suggestions
Large-scale document classification
Bulk invoice extraction
Offline eval runs
Dataset labeling
Post-merge quality analysis
Why these fit:
The user is not waiting.
The results can arrive later.
Each document/test case/report can be processed independently.
50% cost reduction matters at volume.
Anthropic describes batch processing as useful when immediate responses are not required and when optimizing for cost efficiency or running large-scale evaluations or analyses. (Claude)
5. When not to use batch processing
Do not use the Batch API for blocking workflows.
Bad batch use cases:
Pre-merge CI gate
Interactive PR review requested by a developer
User-facing request-response workflow
Real-time fraud decision
Checkout flow
On-call incident triage where humans need immediate output
Blocking deployment approval
Why:
Batch processing can take up to 24 hours.
There is no low-latency guarantee.
The workflow is blocked until the result returns.
For a pre-merge check, the correct answer is almost always:
6. Cost vs latency tradeoff
The Batch API gives a large cost advantage: Anthropic’s docs state that all batch usage is charged at 50% of standard API prices. (Claude)
But the tradeoff is latency:
Synchronous API:
Higher cost, immediate response.
Batch API:
Lower cost, asynchronous response, up to 24-hour window.
Example cost reasoning
Suppose a weekly audit costs $1,000 using synchronous calls.
With batch pricing:
If the audit runs weekly and no one needs the result immediately, batch is a strong fit.
But if the same analysis blocks merge:
7. Batch window and SLA calculations
This is a likely exam calculation pattern.
Assume:
Batch processing may take up to 24 hours.
New documents arrive continuously.
You submit batches every N hours.
Worst-case completion time for a newly arrived item:
So:
A 4-hour batch submission cadence gives:
So a 4-hour cadence satisfies a 30-hour SLA with 2 hours of buffer.
Exam pattern
Question:
You need results within 30 hours. Batch processing can take 24 hours. How often should you submit batches?
Best answer:
8. Batch API and tool use limitations
The official study guide says the Batch API does not support an interactive multi-turn tool-calling loop inside a single request where your application executes a client-side tool mid-request and returns the result.
Current Anthropic docs add nuance: batch requests can include many Messages API features, including tool use, server tools, system messages, multi-turn conversation history, and extended thinking; server tools can run in the batch worker’s server-side agentic loop. However, because there is no open client connection during asynchronous processing, if a batch result pauses, you continue it by submitting a follow-up request. (Claude)
Exam-safe interpretation:
Batch is fine for single-shot extraction/classification/generation.
Batch is not the right choice when the request depends on your application executing tools interactively between model turns inside the same batch item.
Bad batch design:
Batch request:
1. Claude asks to call internal database tool.
2. Your app executes the tool.
3. Claude uses tool result.
4. All inside one batch request.
Better design:
Pre-fetch required data before submitting the batch.
Include the needed context in each batch request.
Or run the interactive tool loop synchronously.
Or split into separate batch stages with persisted intermediate outputs.
9. Designing a batch workflow
A robust batch processing pipeline looks like this:
1. Sample and refine prompt on a small set.
2. Validate request shape synchronously.
3. Create batch with custom_id per item.
4. Poll batch status.
5. Stream results.
6. Map results back using custom_id.
7. Separate succeeded, errored, expired, and validation-failed items.
8. Resubmit only failed/recoverable items.
9. Store final outputs and failure reasons.
Anthropic recommends testing batch request shapes with the standard Messages API first because validation errors in batch params are returned asynchronously after the batch ends; docs also recommend monitoring status, implementing retry logic, using meaningful custom_ids, and breaking very large datasets into multiple batches. (Claude)
10. Prompt refinement before large batches
This is a major skill item in the task statement.
Bad workflow:
Submit 100,000 documents with an untested prompt.
Discover 20% failed or low-quality outputs.
Resubmit thousands of documents.
Pay again and lose time.
Good workflow:
1. Select 50–100 representative documents.
2. Run synchronously or as a tiny batch.
3. Inspect outputs and failures.
4. Improve prompt, schema, chunking, and examples.
5. Run a medium pilot batch.
6. Only then submit the large batch.
Why this matters:
Batch is cheap per token, but large mistakes are still expensive.
First-pass success rate matters.
Retries consume time and cost.
Exam shortcut:
11. Failure handling
Batch failure handling should be selective.
Do not resubmit the entire batch if only a few items failed. Anthropic’s docs note that one request’s failure does not affect the processing of other requests, and the results stream includes each item’s custom_id and result type. (Claude)
Result categories
Typical handling pattern:
succeeded → store output
errored: invalid_request_error → fix request before resubmitting
errored: server error → retry same request
expired → resubmit if still needed
too_large/context issue → chunk or summarize first
Example from the docs shows handling succeeded, errored, and expired result types by custom_id, and distinguishes validation errors from server errors. (Claude)
Example failure-handling table
| Failure type | Cause | Resubmission strategy |
|---|---|---|
| Context length exceeded | Document too large | Chunk document, summarize sections, or reduce prompt |
| Invalid request | Bad schema, unsupported param, malformed params | Fix request shape, test synchronously, resubmit |
| Server error | Transient backend issue | Resubmit same item |
| Expired | Not completed within processing window | Resubmit item, maybe smaller batch or reduced workload |
| Low-quality extraction | Prompt/schema issue | Refine prompt on sample, then resubmit affected items |
| Missing source info | Source lacks required data | Do not retry blindly; mark missing or request source |
12. Resubmitting only failed documents
Suppose your original batch has:
Bad:
Good:
But with different handling:
200 expired → resubmit directly or in smaller batches.
80 context-length errors → chunk documents before resubmitting.
20 invalid request errors → fix schema/request shape before resubmitting.
Example resubmission logic:
failed_requests = []
for result in batch_results:
custom_id = result["custom_id"]
result_type = result["result"]["type"]
if result_type == "succeeded":
save_success(custom_id, result["result"]["message"])
elif result_type == "expired":
failed_requests.append(original_requests[custom_id])
elif result_type == "errored":
error_type = result["result"]["error"]["error"]["type"]
if error_type == "invalid_request_error":
fixed_request = fix_request_shape(original_requests[custom_id])
failed_requests.append(fixed_request)
else:
failed_requests.append(original_requests[custom_id])
submit_new_batch(failed_requests)
13. Chunking oversized documents
If a document exceeds context limits or makes the request too large, do not resubmit unchanged.
Bad:
Better:
Example chunking plan:
custom_id: contract_123_part_01
custom_id: contract_123_part_02
custom_id: contract_123_part_03
custom_id: contract_123_merge
Important design:
Example custom IDs:
Then merge outputs by original document ID.
14. Synchronous vs batch examples
Example A: pre-merge check
Correct API:
Reason:
Example B: nightly test generation
Correct API:
Reason:
Example C: weekly audit
Correct API:
Reason:
Example D: interactive assistant
Correct API:
Reason:
15. Batch submission cadence examples
Scenario 1: 30-hour SLA
A 4-hour cadence is safe:
Scenario 2: 48-hour SLA
Daily batch submission is acceptable.
Scenario 3: 25-hour SLA
This is risky. If your SLA is that tight, the synchronous API may be safer unless the workload can tolerate occasional misses.
Exam shortcut:
16. Common exam traps
Trap 1: Batch for blocking PR checks
Wrong:
Right:
Trap 2: Ignoring custom_id
Wrong:
Right:
Trap 3: Retrying the whole batch
Wrong:
Right:
Trap 4: Retrying oversized documents unchanged
Wrong:
Right:
Trap 5: Scaling before prompt validation
Wrong:
Right:
Trap 6: Assuming batch supports interactive client-side tool loops
Wrong:
Right:
Precompute tool context, use server-side tools where supported, split into stages, or use synchronous orchestration.
17. Scenario-based practice questions
Question 1
A CI pipeline must block merge until Claude reviews the changed migration file. The team wants the cheapest option.
What should they use?
A) Message Batches API
B) Synchronous Messages API
C) Weekly batch job
D) No API
Answer
B. Pre-merge checks are blocking. Batch processing can take up to 24 hours and is not appropriate for latency-sensitive CI gates.
Question 2
A team wants to run a weekly audit over 80,000 historical support tickets. The results are needed by Monday morning, not immediately.
What should they use?
A) Synchronous API only
B) Message Batches API
C) Interactive Claude Code session
D) Manual review
Answer
B. This is high-volume, non-blocking, and latency-tolerant, making it a good fit for batch processing.
Question 3
Batch results arrive in a different order than the input requests.
What field should be used to correlate responses?
A) line_number
B) custom_id
C) temperature
D) role
Answer
B. Batch results may be returned out of order. Use custom_id to match outputs to inputs.
Question 4
A batch has 10,000 documents. 9,850 succeed, 100 expire, and 50 fail because the documents exceed context limits.
What should the system do?
A) Resubmit all 10,000 documents.
B) Resubmit only the 150 failed documents; directly retry expired ones and chunk the oversized ones.
C) Ignore all failures.
D) Switch all future workloads to synchronous.
Answer
B. Failures are per request. Resubmit only failed items, and modify requests when the failure requires it.
Question 5
A prompt has not been tested, but the team plans to submit 100,000 documents in one batch.
What is the best first step?
A) Submit immediately to save time.
B) Run the prompt on a representative sample and refine it before large-scale batching.
C) Remove custom_id.
D) Use the same custom ID for all documents.
Answer
B. Prompt refinement on a sample set improves first-pass success and reduces costly resubmissions.
Question 6
A batch workload has a 30-hour SLA. Batch processing may take up to 24 hours. What submission cadence guarantees the SLA?
A) Every 12 hours
B) Every 8 hours
C) Every 6 hours or more frequently
D) Every 30 hours
Answer
C. Worst case is submission interval plus 24 hours. To meet 30 hours, submit at least every 6 hours. Every 4 hours gives buffer.
Question 7
A batch request needs to call an internal application database tool, wait for your application to execute it, then pass the tool result back to Claude inside the same request.
What is the best interpretation?
A) This is a poor fit for a single batch request; use synchronous orchestration, pre-fetch data, or split into stages.
B) Batch always supports arbitrary interactive client-side tool loops.
C) Use custom_id to execute the database tool.
D) Remove all context.
Answer
A. Batch is asynchronous and does not maintain an open client connection for your application to execute tools mid-request.
Question 8
A nightly report generation job costs $2,000 with synchronous calls and can wait until the next day.
What is the expected batch cost, ignoring other discounts?
A) $2,000
B) $1,000
C) $4,000
D) $0
Answer
B. Batch usage is charged at 50% of standard API prices.
Question 9
A failed batch request has invalid_request_error due to unsupported parameters.
What should happen before resubmission?
A) Resubmit unchanged.
B) Fix the request shape and test it before resubmitting.
C) Retry indefinitely.
D) Ignore custom_id.
Answer
B. Invalid request errors require fixing the request; unchanged retries will usually fail again.
Question 10
A user-facing chatbot must answer within seconds.
What API approach is appropriate?
A) Message Batches API
B) Synchronous API
C) Weekly batch processing
D) Overnight queue
Answer
B. User-facing interactive workflows require low latency, so synchronous calls are appropriate.
18. Final D4.5 checklist
Memorize this:
1. Message Batches API is asynchronous.
2. Batch usage is charged at 50% of standard API prices.
3. Batch processing can take up to 24 hours.
4. Batch is for non-blocking, latency-tolerant workloads.
5. Use synchronous API for blocking workflows.
6. Pre-merge checks should use synchronous API.
7. Overnight reports, weekly audits, nightly test generation, and offline evals are batch-friendly.
8. Each batch request needs a unique custom_id.
9. Results may return out of order; use custom_id for correlation.
10. Do not assume one failed request breaks the entire batch.
11. Resubmit only failed or expired documents.
12. Modify failed requests when needed, such as chunking oversized documents.
13. Test request shape and prompt quality on a sample before large batches.
14. Batch is not suitable for interactive client-side tool loops inside a single request.
15. Submission cadence should satisfy: interval + 24 hours ≤ SLA.
D4.5 mental model
Synchronous API = fast, blocking, higher cost.
Batch API = cheap, asynchronous, latency-tolerant.
custom_id = correlation key.
Sample first = fewer expensive retries.
Failed only = efficient resubmission.
Chunk oversized = fix the cause, not just retry.
The exam’s favorite distinction is:
Use the Batch API when time is flexible and volume is high; use the synchronous API when the workflow blocks a human, merge, deployment, or user response.