Batch

Batch parses up to 1,000 documents with one config, so you can process a folder or a list of files and follow them as a group. POST /parse/batch takes the documents and a config or config_id. Each document becomes its own parse job, and the batch keeps a manifest of the jobs that have ended. A batch always runs async. To run more steps on each document, start them from its parse.completed webhook.

Start a batch

This request parses two documents from URLs with one config:

batch = client.parse.batch.run(
    file_urls=["https://example.com/a.pdf", "https://example.com/b.pdf"],
    config={"model": "ultra-2"},
)
print(batch.batch_id)

const { batchId } = await client.parse.batch.run({
  fileUrls: ["https://example.com/a.pdf", "https://example.com/b.pdf"],
  config: { model: "ultra-2" },
});

curl -X POST https://api.runpulse.com/parse/batch \
  -H "x-api-key: $PULSE_API_KEY" \
  -H "Pulse-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "file_urls": ["https://example.com/a.pdf", "https://example.com/b.pdf"],
    "config": {"model": "ultra-2"}
  }'

client is the one from SDKs. Send the documents in exactly one of these fields:

file_urls: up to 1,000 URLs. Pulse fetches each URL when its job runs, up to 48 hours later, so a signed URL must still be valid then. A URL that returns no document by then fails only its own job.
folder_url: a folder in Amazon S3, Google Cloud Storage or Azure Blob Storage. Pulse lists it when you send the batch, without credentials, so anyone must be able to list it and read its files. A folder with more than 1,000 supported files or more than 10,000 objects is refused whole, so narrow it to a subfolder or set max_depth to include fewer subfolder levels.
files: up to 1,000 uploads, each its own multipart part named files, stored when you send the batch. If no part has a supported type, the batch is refused.

A folder or an upload takes .pdf, .jpg, .jpeg, .png, .tif, .tiff, .docx, .pptx, .xlsx, .xlsm, .xls, .html and .htm files. Pulse skips every other type, hidden files and, in a folder, archived ones. To parse a skipped type, such as .csv, send it in file_urls or to /parse.

Follow the batch

GET /parse/batch/{batch_id} returns the batch’s status and a summary that counts its documents by state. A batch’s status is never queued or expired:

Status	Meaning
`processing`	Its jobs are still running.
`completed`	Every job has ended and at least one didn’t fail.
`failed`	Every job failed, the batch was still running 48 hours after you sent it, or it was interrupted. `error.detail` says which.
`canceled`	You canceled it.

To cancel a batch, send DELETE /parse/batch/{batch_id}. Its unfinished jobs are canceled within seconds, and a job that finishes first keeps its result. Anyone in your organization can cancel a batch. Each job is checked against your plan and credits, and billed, when it runs. The batch adds no charge of its own; see Pricing. Each job’s result follows your organization’s retention, but the batch and its manifest never expire.

Read the manifest

GET /parse/batch/{batch_id}/result returns the manifest: a JSON array with one entry for each job that has ended, in the order the jobs ended. You can read it while the batch runs; a job that isn’t listed hasn’t ended yet. To page through the manifest, send limit (1 to 1,000). Every page but the last has a Link header with the next page’s URL, which you read from the raw response, as below. Key your records by parse_id. The manifest doesn’t say which document each job parsed, and a result’s history.file_name can repeat across URLs. To match each result to its source, send one async /parse per document instead.

Drain the manifest

This helper hands each ended job to handle once. It waits while the batch runs and stops when the batch has ended and the last page is read.

import time
from urllib.parse import parse_qs, urlparse

from pulse.core.api_error import ApiError

ENDED = ("completed", "failed", "canceled")


def next_cursor(headers) -> str | None:
    """The cursor in a manifest page's Link header, or None on the last page."""
    link = headers.get("link")
    if not link:
        return None
    return parse_qs(urlparse(link.split(";")[0].strip(" <>")).query)["cursor"][0]


def drain(batch_id: str, handle, seen: set[str]) -> None:
    """Call handle(job) once for each ended job, until the batch has ended and every page is read."""
    cursor = None
    while True:
        ended = client.parse.batch.get(batch_id).status in ENDED
        page = client.parse.batch.with_raw_response.get_result(batch_id, limit=500, cursor=cursor)
        for job in page.data:
            if job.parse_id not in seen:
                handle(job)
                seen.add(job.parse_id)  # only after handle returns
        if found := next_cursor(page.headers):
            cursor = found
        elif ended:
            return
        else:
            time.sleep(30)


def handle(job) -> None:
    if job.status != "completed":
        record_failure(job.parse_id, job.error)
        return
    try:
        result = client.parse.get_result(job.parse_id)
    except ApiError as error:
        if error.status_code != 410:
            raise
        record_failure(job.parse_id, error.body)  # expired, or already read once
        return
    save(job.parse_id, result)


drain(batch.batch_id, handle, seen=set())

save and record_failure are yours to write. A job joins seen only after handle returns, so if drain stops on an error, run it again with the same seen: no job is skipped and no result is read twice. A 410 means the result expired, or was already read where your organization deletes results on first read, so handle records it instead of raising. After a cancel or a failed batch, jobs can still end. Run drain again later with the same seen to pick them up. A document that a job couldn’t be created for counts in summary.failed but has no manifest entry.

If the batch was interrupted, the jobs it already created keep running. Resend the documents missing from its manifest only once summary.queued and summary.processing are both 0, or you may parse and bill a document twice.

Next steps

Cloud intake

Parse a bucket and write each result back to your storage.

Get started

Endpoints

Run at scale

Essentials

Account

Start a batch

Follow the batch

Read the manifest

Drain the manifest

Next steps

Cloud intake

​Start a batch

​Follow the batch

​Read the manifest

​Drain the manifest

​Next steps

Cloud intake

Start a batch

Follow the batch

Read the manifest

Drain the manifest

Next steps