DOCS — PRODUCT · BATCH INFERENCE

Batch Inference¶

OpenMed can submit a large set of independent inference requests to a remote batch provider, return immediately, and resume the job from another process. Version 0.5.0 supports Doubleword. OpenMed keeps the batch lifecycle provider-neutral so providers with a different wire format can be added without changing the CLI or local job format.

For interactive streaming or individual async requests, see Doubleword real-time and Responses. All modes share the same key.

Batch inference is separate from the interactive Agent TUI. It does not run OpenMed tools or a multi-turn agent loop. Each JSONL row is one direct chat-completions or embeddings request.

There is no Doubleword batch screen or /batch command in the TUI in 0.5.0. Run openmed batch in a second terminal while using the TUI. Its existing /privacy-batch command is a separate clinical privacy workflow.

Why use a 24-hour batch?¶

Use batching when you have many independent records and can trade response time for a lower inference price. Your submission returns a job ID immediately; the provider schedules the work and you collect the results later. 24h is the completion window for each submitted batch, not a mandatory 24-hour sleep or a promise that a multi-stage workflow will finish overnight.

Workload	One request per	What you collect later
Retrospective coding-quality review	Closed encounter plus prepared NLP/ICD-10/HCC evidence	Draft discrepancies and a human-review queue
Scheduled research-corpus screening	Abstract or document chunk	Inclusion labels, evidence quotes, and reasons for review
Backlog document enrichment	Previously collected document	Structured metadata or a draft summary
Offline regression evaluation	Fixed synthetic test case	Model outputs to grade and compare with earlier runs

These workloads tolerate delayed results. A live consultation, urgent triage, an interactive TUI turn, or an action waiting on the next model response usually needs a synchronous provider instead.

For a concrete price illustration, Doubleword's DeepSeek V4.1 Flash model page listed 24-hour batch input/output at $0.08/$0.30 per million tokens on 2026-09-28, versus $0.15/$0.60 for realtime where available. A workload with 1 million uncached input tokens and 100,000 output tokens would therefore be about $0.11 in batch versus $0.21 realtime. This is an illustration, not a quote: check current prices, availability, cached tokens, and actual usage. Native NLP and coding service calls have their own costs; batching does not discount them.

A useful overnight workflow¶

For a chart backlog, prepare each encounter's permitted source text and run the NLP, code-validation, and HCC reference lookups first. Keep their real outputs, failures, model year, and provenance. Package that evidence into one independent DeepSeek request per encounter asking for a source-grounded audit draft. Submit the file, close your terminal, and collect the review queue the next day. Separate unsupported diagnoses and missing service evidence for human review; do not automatically turn model drafts into claims or clinical decisions. Pin the coding model, payment year, and service reference-data version for the workload. Check representative HCC mappings against the corresponding official reference release: a successful tool response or echoed model name does not verify the underlying mapping data or RAF calculation. Keep that validation separate from checks that the model followed its instructions.

The source checkout has a runnable example of this exact single-window pattern:

These demo scripts require the development checkout, python-dotenv, a Doubleword key, and working native medical services for the NLP/coding steps. Doubleword credentials alone do not provision those services. Without them, the receipts show missing evidence; they do not become a successful end-to-end medical-service demonstration.

python3 scripts/demo/run_doubleword_overnight_audit.py
# Native-service calls only; inspect evidence before paying for generation:
python3 scripts/demo/run_doubleword_overnight_audit.py --prepare
# Or prepare and submit six synthetic encounter audits, then exit:
python3 scripts/demo/run_doubleword_overnight_audit.py --submit
# Later, collect the saved run into review-queue.json and REPORT.md:
python3 scripts/demo/run_doubleword_overnight_audit.py --collect artifacts/doubleword-overnight-audit/<run>

The runner calls configured NLP, ICD-10 lookup/validation, HCC, RAF, CCSR and comorbidity services first. Its HCC/RAF results are explicitly labeled mechanical references for unreviewed imported codes, including suspect codes, so the model can audit the import rather than mistake a mapping for clinical support. It saves every service receipt and the six independent requests under artifacts/doubleword-overnight-audit/. Service failures remain visible in the evidence. --submit prints a job ID and returns; it does not keep a model/tool loop running. For a prepared file, submit the printed JSONL path directly to avoid collecting the same service evidence again. When using the CLI to submit a prepared file, associate its returned job ID on the first collection:

python3 scripts/demo/run_doubleword_overnight_audit.py --collect artifacts/doubleword-overnight-audit/<run> --job-id <job-id>

The collector verifies that the job's input digest matches the prepared file. --collect refreshes the existing job, downloads available rows, and builds a review queue without resubmitting. Its report includes row status, finish reasons, selected synthetic-fixture checks, token usage, and an estimated generation cost. It also spot-checks six HCC mapping entries against a pinned CMS 2025 V28 sample, separately from the model's audit checks. Zero checked entries means no matching reference results were available; even a passing spot check does not validate the full service dataset, hierarchy, coefficients, or RAF score. Pending rows remain pending; a partial report is not a completed audit.

openmed batch validate overnight-audit.jsonl
openmed batch submit overnight-audit.jsonl --description "closed-encounter review"
# Save the returned job ID. Later, or from a scheduled job:
openmed batch status <job-id>
openmed batch results <job-id> --json

Build each row with body.model set to deepseek-ai/DeepSeek-V4.1-Flash, stable custom_id values, and a suitable output-token limit. Join results to source records by custom_id, never by output order. Inspect row errors as well as the overall job status, and retry only explicitly failed records after reconciliation. The CLI sends the JSONL you prepare; it does not automatically gather medical evidence or execute tools.

A model-driven tool loop needs a local orchestrator to execute returned tool calls and then submit another batch with the receipts. Every dependent batch has its own completion window: nine serial stages could span nine such windows. For economical overnight processing, prepare deterministic service evidence before the batch and minimize dependent model stages. Use the multi-stage demo below to test failure handling, not as a next-morning completion guarantee.

Configure Doubleword¶

Set the key in the environment:

export DOUBLEWORD_API_KEY="sk-..."

Or store it in OpenMed's private provider-secret file:

openmed config provider-set doubleword --api-key "sk-..."
openmed config provider-show

Alternatively, this ~/.openmed/provider_secrets.json entry references the exported environment variable (replace the reference with a literal key to store it in the file):

{
  "provider": {
    "doubleword": {
      "options": {
        "apiKey": "{env:DOUBLEWORD_API_KEY}"
      }
    }
  }
}

The environment variable takes precedence over the stored value. The Doubleword endpoint is fixed to https://api.doubleword.ai/v1; there is no base-URL setting for this hosted provider. OpenMed sends this key only to that host and never writes it into batch state, result files, or CLI output. Batch commands use this file for credentials. Its optional interactive models list does not rewrite the models in your JSONL. openmed batch submit selects batch processing; an individual flex request uses openmed responses submit instead. There is no async flag to set in provider secrets. Merge the doubleword entry inside an existing provider object when other providers are configured.

Create a JSONL input file¶

Save the following two lines as requests.jsonl. Each line is an independent OpenAI-style request. custom_id must be unique in the file and is how you join a result back to its source record.

{"custom_id":"case-001","method":"POST","url":"/v1/chat/completions","body":{"model":"deepseek-ai/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"Return only the word OK."}],"max_tokens":4096}}
{"custom_id":"case-002","method":"POST","url":"/v1/chat/completions","body":{"model":"deepseek-ai/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"Return only the word READY."}],"max_tokens":4096}}

Use a model ID available to your Doubleword account. OpenMed does not replace, route, or silently correct model IDs inside the request body. For reasoning models, the output budget also covers reasoning tokens. A tiny budget can produce finish_reason: "length" with no final text even when the request succeeds; inspect each result's finish reason and message content.

Validate before spending credits:

openmed batch validate requests.jsonl

Validation is local and does not require a key or network connection. It checks UTF-8 JSONL structure, unique IDs, request method, endpoint consistency, body.model, file size, request count, and disallows nested streaming.

Submit and resume¶

Submission uploads the file, creates the remote batch, saves the returned IDs, and exits. It does not wait for the provider to finish.

openmed batch submit requests.jsonl --description "nightly extraction"

The command prints an OpenMed local job ID such as batch-20260927T210000Z-12ab34cd. Use either that local ID or the Doubleword batch ID with subsequent commands:

openmed batch status  batch-20260927T210000Z-12ab34cd
openmed batch results batch-20260927T210000Z-12ab34cd
openmed batch wait    batch-20260927T210000Z-12ab34cd
openmed batch cancel  batch-20260927T210000Z-12ab34cd
openmed batch list

status refreshes provider state. results refreshes state and downloads any new output and error records. wait polls, downloads partial results as they become available, and stops at completed, failed, expired, or cancelled. Stopping wait with Ctrl+C affects only the local waiter; the remote job remains resumable. Cancellation is a best-effort provider request, and results already processed remain downloadable.

To submit and wait in one command:

openmed batch submit requests.jsonl --wait --poll-interval 10

For automation, every command that returns a job accepts --json. A bounded wait can be resumed later:

openmed batch wait <job-id> --timeout 900 --json

Wait intervals must be finite and at least one second; timeouts must be finite and positive. Omitting --timeout waits without a local deadline. A single HTTP request can outlast that deadline until its network timeout expires. Invalid wait options are rejected before submission. submit --wait prints the saved job ID to stderr before waiting, including when --json is selected.

wait and submit --wait exit 0 for a completed job with no reported failed rows, 1 for unsuccessful terminal outcomes or row failures, 2 for validation, provider, or wait-timeout errors, and 130 on Ctrl+C. Commands such as status return 0 when the inspection succeeds, even if the inspected job has failed. Always inspect both the batch status and per-row errors.

Partial results and local files¶

Doubleword exposes output and error files while a job is still processing. OpenMed uses the provider's X-Last-Line cursor to request later chunks and also deduplicates records by custom_id, so repeating results does not append the same row twice.

One process at a time can update a job. Concurrent updates return a busy error; retry after the other command finishes its current request. Result files are replaced atomically before cursors are saved, so interrupted local writes can be retried without leaving partial JSON lines.

By default, one job lives at:

~/.openmed/batches/<local-job-id>/
├── job.json
├── input.jsonl
├── output.jsonl     # appears after the first successful result
└── errors.jsonl     # appears after the first failed result

The job directory is mode 0700; state, input, and result files are mode 0600. job.json contains provider IDs, status, counts, result cursors, timestamps, models, and the input digest. It never contains the API key or an authorization header. Override the root with OPENMED_BATCH_DIR when a job store belongs on a different encrypted or managed volume.

Keep the local job directory until you have retrieved the records you need. Remote result files have an expiry timestamp in the provider response and may not remain downloadable indefinitely.

If submission fails after the provider may have accepted it, check Doubleword's dashboard before submitting again. A local submission_failed state does not prove no remote job exists; paid creation requests are never retried automatically.

Limits and endpoints¶

The Doubleword adapter enforces the provider's published input limits before upload:

Contract	Current value
Maximum input file	200 MiB
Maximum requests	50,000 JSONL rows
Maximum `custom_id` length	64 characters
Completion window	`24h`
Supported row URLs	`/v1/chat/completions`, `/v1/embeddings`

Select embeddings explicitly, and use the same URL in every row:

openmed batch validate embeddings.jsonl --endpoint /v1/embeddings
openmed batch submit embeddings.jsonl --endpoint /v1/embeddings

See Doubleword's Batch Inference documentation for its live service contract and model catalog for account-visible model IDs.

Data boundary¶

Submitting a file sends every row body to Doubleword and to the infrastructure serving the selected model. The local copy is private on disk, but it is not automatically de-identified. Do not assume TUI PHI mode or the file-privacy workflow transforms batch JSONL; prepare the body content for the remote provider before submission and apply your organization's data-use, retention, region, and provider-agreement requirements.

Provider adapters¶

The durable job store and CLI use a small adapter contract: capabilities, submit, status, cancel, and incremental result retrieval. Doubleword implements the file-backed OpenAI Batch shape. Doubleword is the only enabled batch provider in 0.5.0; OpenRouter batching is outside this release's scope.

Repository demos¶

From a source checkout, the demo runner defaults to offline validation:

python3 scripts/demo/run_doubleword_batch_demo.py
python3 scripts/demo/run_doubleword_batch_demo.py --submit --wait
python3 scripts/demo/run_doubleword_batch_demo.py --kind embeddings --submit --wait
python3 scripts/demo/run_doubleword_batch_demo.py --resume <job-id>

Each submitted demo has two synthetic rows. The runner saves each input and a submission receipt under a fresh artifacts/doubleword-batch-demo/ directory. It uses the actual CLI and prints model answers or embedding dimensions. Its wait is bounded to 300 seconds by default; resume a pending job later instead of submitting it again. The full walkthrough lives in the repository demo library at PLANS/DEMOS/13-doubleword-batching.md.

For maintainers, mocked contract tests run with the ordinary unit suite. The opt-in real-service test spends credits on one synthetic chat row and one embedding row:

OPENMED_DOUBLEWORD_BATCH_LIVE=1 \
  python3 -m pytest -q tests/integration/test_doubleword_batch_live.py

Override its default model IDs with DOUBLEWORD_BATCH_CHAT_MODEL and DOUBLEWORD_BATCH_EMBEDDING_MODEL when the account catalog changes.

DeepSeek workflow stress demo¶

When interpreting service evidence, distinguish request success from supported coding. NLP entity spans may include negated or family-history mentions. A code validator's valid field may describe syntax while exists and billable are false. A successful HCC lookup only says how a code maps; it does not establish that the encounter supports that code. Keep these fields and the original note in the review evidence instead of converting every successful service call into an accepted diagnosis.

The source checkout includes a separate synthetic retrospective coding-audit stress run using DeepSeek V4.1 Flash, with real configured NLP and coding services. It exercises nine dependent batches across six encounters: NLP, ICD-10 search, exact lookup, validation, HCC, RAF, CCSR/comorbidity, audit, and an adversarial recheck. The model chooses tool arguments; the local demo executes only the stage's allowed read-only tools and returns actual receipts.

# Preview the bounded run without network calls
python3 scripts/demo/run_doubleword_workflow_stress.py
# Run the model batches and configured medical services
python3 scripts/demo/run_doubleword_workflow_stress.py --run
# Continue a pending run without submitting completed stages again
python3 scripts/demo/run_doubleword_workflow_stress.py --resume artifacts/doubleword-workflow-stress/<run>

This harness requires the existing native-service credentials in addition to DOUBLEWORD_API_KEY. It records missing credentials, outages, invalid arguments, row errors, truncated outputs, omitted calls, and selected fixture failures. The default wait limit is 600 seconds per stage; timing out preserves the job for a later resume. --timeout changes the local wait only. The local orchestrator must run again for dependent stages to progress after it exits.

The cap is 54 model requests with at most 4096 generated tokens per request, 100 KB per request body, and 160 tool calls. Receipts, raw result files, final audits, and REPORT.md live under artifacts/doubleword-workflow-stress/. This is a bounded failure-discovery exercise over synthetic notes, not a clinical evaluation or a guarantee of correctness on a production chart backlog.