OpenAI Batch API Savings

OpenAI Batch API Architecture: 50% Cost Arbitrage vs Asynchronous Queueing

The OpenAI Batch API delivers a flat 50% discount on all token usage (prompt and completion) across models including GPT-4o, GPT-4o-mini, and o1/o3-mini. In exchange for non-real-time completion (guaranteed 24-hour turnaround, typical delivery within 45 to 120 minutes), teams process bulk embedding indexing, synthetic data generation, document summarization, and historical log analysis at half standard API retail pricing. Additionally, batch workloads operate on dedicated batch rate limits separate from interactive Tier 1–5 organization quotas.

Production Pipeline: Resilient Batch Submission & Status Polling Worker

Submitting large JSONL files to the Batch API requires uploading via the Files API, creating the batch execution job, and polling with backoff. Here is a production Python automation script:

import time
from openai import OpenAI

client = OpenAI()

def execute_batch_job(jsonl_filepath: str) -> str:
    # Step 1: Upload batch JSONL payload
    with open(jsonl_filepath, "rb") as file_data:
        uploaded_file = client.files.create(
            file=file_data,
            purpose="batch"
        )
    print(f"[Batch] Uploaded file ID: {uploaded_file.id}")

    # Step 2: Trigger Batch Job with 24h completion window
    batch_job = client.batches.create(
        input_file_id=uploaded_file.id,
        endpoint="/v1/chat/completions",
        completion_window="24h",
        metadata={"project": "rag_corpus_indexing"}
    )
    print(f"[Batch] Submitted job ID: {batch_job.id}")

    # Step 3: Exponential backoff polling
    poll_interval = 60
    while True:
        status = client.batches.retrieve(batch_job.id)
        print(f"[Batch] Status: {status.status} ({status.request_counts.completed}/{status.request_counts.total})")
        
        if status.status in ("completed", "failed", "cancelled", "expired"):
            if status.status == "completed":
                print(f"[Batch] Success! Output File ID: {status.output_file_id}")
                return status.output_file_id
            raise RuntimeError(f"Batch execution failed with status: {status.status}")
        
        time.sleep(poll_interval)
        poll_interval = min(poll_interval * 1.5, 600)

Production Governance & Failure Recovery Rules

• Strict 100MB File Size Guardrail: Individual JSONL batch files must not exceed 100MB or 50,000 lines. Production batch splitters should slice large million-row datasets into 25,000-line chunks before uploading.
• Custom ID Traceability: Always set `custom_id` in each JSONL row to a UUID or database primary key. The Batch API returns responses unordered, making input index matching impossible without explicit identifiers.
• Partial Failure Extraction: Even when a batch status reports `completed`, individual line items can fail due to content policy flags or context length limits. Always fetch both `output_file_id` and `error_file_id` to log failed records.

Frequently Asked Batch Processing & API Economics Questions

Q: Does the 50% discount apply to prompt context tokens, completion tokens, or both?
A: The 50% discount applies across both input prompt tokens and output completion tokens for all supported models, including GPT-4o, GPT-4o-mini, and o1/o3-mini reasoning models.

Q: What happens if a batch job takes longer than 24 hours to complete?
A: If OpenAI fails to process the batch within the 24-hour completion window, the batch status transitions to `expired`. Unprocessed line items are not billed, and the system emits an error log allowing automated re-submission.

Q: Are batch rate limits shared with interactive real-time API production quotas?
A: No. Batch workloads have dedicated Enqueued Token Limits (separate from your interactive Tier 1–5 Requests Per Minute and Tokens Per Minute quotas), preventing bulk indexing jobs from causing rate limit errors on customer-facing apps.

Automated Retry Orchestration & Chunked Resubmission

When processing millions of historical customer records, individual line-item JSONL parsing errors should never abort the overarching batch. Implement automated error extraction that parses `error_file_id`, isolates rejected records into a quarantine bucket, and automatically resubmits valid payloads to preserve continuous 24-hour pipeline throughput.