RAG ingestion

Turn a document into retrieval-ready chunks, embed them, and keep the metadata that ties each answer to its file and pages.

What you need

The Python SDK and an API key, set up as in Cookbooks.
LangChain and an embedding model: pip install langchain-core langchain-huggingface sentence-transformers.
A document to index. This recipe uses the Attention Is All You Need sample, which has 15 pages.

Build it

Parse with semantic chunking

This call parses the paper with semantic chunking and reads the result once the parse finishes. Choose a chunk size explains max_characters.

import os

from pulse import Pulse

client = Pulse(api_key=os.environ["PULSE_API_KEY"])
PAPER = "https://platform.runpulse.com/api/examples/3be15d23-d622-4f27-9843-ec2929140eec/pdf"

parse_id = client.parse.run(
    file_url=PAPER,
    config={
        "model": "ultra-2",
        "extensions": {"chunking": {"strategies": [{"strategy": "semantic", "max_characters": 1000}]}},
    },
).parse_id
result = client.parse.get_result(parse_id)
if any(warning.code == "CHUNKS_SKIPPED" for warning in result.warnings or []):
    raise RuntimeError(f"Chunking failed on {parse_id}: parse the document again")

If chunking can’t finish, the parse still completes, with an empty chunk list and a CHUNKS_SKIPPED warning.

Turn chunks into documents

This code makes one LangChain document per chunk. The metadata holds what you filter on at query time (tenant_id) and what you cite (file_name and pages).

from langchain_core.documents import Document


def documents(result, *, tenant_id, document_id, source_url):
    """One LangChain document per chunk, with the metadata to filter and cite it."""
    chunking = result.extensions.chunking if result.extensions else None
    return [
        Document(
            id=f"{result.parse_id}:{chunk.id}",  # the same each time you index this parse
            page_content=chunk.content,
            metadata={
                "tenant_id": tenant_id,
                "document_id": document_id,
                "source_url": source_url,
                "file_name": result.history.file_name,
                "parse_id": result.parse_id,
                "chunk_id": chunk.id,
                "pages": chunk.pages,
            },
        )
        for chunk in (chunking.semantic if chunking else None) or []
    ]


docs = documents(result, tenant_id="tenant-a", document_id="paper-17", source_url=PAPER)

A chunk’s id is its place in reading order, such as semantic-0. Its content is Markdown, with page breaks and figure images removed.

Embed and index

This code embeds every chunk into an in-memory vector store. Any LangChain embedding model and vector store take the same documents.

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_huggingface import HuggingFaceEmbeddings

store = InMemoryVectorStore(HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2"))
store.add_documents(docs)

Search with provenance

This code searches one tenant’s chunks and prints the file and pages of each hit.

def search(question, tenant_id, k=3):
    """The k chunks closest to the question, from one tenant's documents only."""
    return store.similarity_search(question, k=k, filter=lambda doc: doc.metadata["tenant_id"] == tenant_id)


for doc in search("What does multi-head attention do?", tenant_id="tenant-a"):
    print(doc.metadata["file_name"], doc.metadata["pages"], doc.page_content[:200])

Filter by tenant before any chunk reaches a model. Pass each hit to the model with its file name and pages, so the answer can cite them.

How it works

This diagram shows where each part of a chunk goes in the vector store: At query time, the metadata does the work: The vector id combines the parse_id and the chunk’s id, so indexing the same parse again reuses its ids, and a store that writes by id, such as InMemoryVectorStore, replaces those vectors instead of adding copies.

Choose a chunk size

A chunk longer than max_characters is cut again, at paragraph breaks first, then at line, sentence and word breaks.

Retrieval need	`max_characters`	Why
Broad questions	Leave it out	Each chunk runs as long as its topic.
An embedding model with an input limit	Below the limit, in characters	A chunk past the limit is cut short by the model. `all-MiniLM-L6-v2` reads 256 tokens, about 1,000 characters of English.
Precise answers with page citations	Smaller, such as `600`	Each chunk covers fewer pages, so citations are tighter.

Chunking is included in Parse’s price.

Keep the index current

Re-parsing a document

A new parse of a document has a new parse_id and new chunk boundaries. Delete the document’s old vectors by document_id, then index the new ones.

Keeping the result

Read each result once, and keep it before you index it. A result can expire, and some organizations delete a result the first time it’s read.

One chunk per page

To index one chunk per page instead, join each page’s elements and leave out page headers, footers and numbers:

from collections import defaultdict

SKIP = {"Page-header", "Page-footer", "Page-number"}
by_page = defaultdict(list)
for element in result.elements:
    if element.category not in SKIP and getattr(element, "content", None):
        by_page[element.page].append(element.content)
page_texts = {page: "\n\n".join(parts) for page, parts in sorted(by_page.items())}

Cookbooks

What you need

Build it

How it works

Choose a chunk size

Keep the index current

Next steps

Parse options

Elements and bounding boxes

​What you need

​Build it

​How it works

​Choose a chunk size

​Keep the index current

​Next steps

Parse options

Elements and bounding boxes

What you need

Build it

How it works

Choose a chunk size

Keep the index current

Next steps