> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runpulse.com/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG ingestion

> Chunk a parse, embed the chunks, and keep the metadata that traces every answer to its pages.

Turn a document into retrieval-ready chunks, embed them, and keep the metadata that ties each answer to its file and pages.

## What you need

* The Python SDK and an API key, set up as in [Cookbooks](https://docs.runpulse.com/v2/cookbooks/overview.md#before-you-start).
* LangChain and an embedding model: `pip install langchain-core langchain-huggingface sentence-transformers`.
* A document to index. This recipe uses the [Attention Is All You Need](https://platform.runpulse.com/dashboard/examples/3be15d23-d622-4f27-9843-ec2929140eec) sample, which has 15 pages.

## Build it

**1. Parse with semantic chunking**

This call parses the paper with semantic chunking and reads the result once the parse finishes. [Choose a chunk size](#choose-a-chunk-size) explains `max_characters`.

```python theme={null}
import os

from pulse import Pulse

client = Pulse(api_key=os.environ["PULSE_API_KEY"])
PAPER = "https://platform.runpulse.com/api/examples/3be15d23-d622-4f27-9843-ec2929140eec/pdf"

parse_id = client.parse.run(
    file_url=PAPER,
    config={
        "model": "ultra-2",
        "extensions": {"chunking": {"strategies": [{"strategy": "semantic", "max_characters": 1000}]}},
    },
).parse_id
result = client.parse.get_result(parse_id)
if any(warning.code == "CHUNKS_SKIPPED" for warning in result.warnings or []):
    raise RuntimeError(f"Chunking failed on {parse_id}: parse the document again")
```

If chunking can't finish, the parse still completes, with an empty chunk list and a `CHUNKS_SKIPPED` warning.

**2. Turn chunks into documents**

This code makes one LangChain document per chunk. The metadata holds what you filter on at query time (`tenant_id`) and what you cite (`file_name` and `pages`).

```python theme={null}
from langchain_core.documents import Document


def documents(result, *, tenant_id, document_id, source_url):
    """One LangChain document per chunk, with the metadata to filter and cite it."""
    chunking = result.extensions.chunking if result.extensions else None
    return [
        Document(
            id=f"{result.parse_id}:{chunk.id}",  # the same each time you index this parse
            page_content=chunk.content,
            metadata={
                "tenant_id": tenant_id,
                "document_id": document_id,
                "source_url": source_url,
                "file_name": result.history.file_name,
                "parse_id": result.parse_id,
                "chunk_id": chunk.id,
                "pages": chunk.pages,
            },
        )
        for chunk in (chunking.semantic if chunking else None) or []
    ]


docs = documents(result, tenant_id="tenant-a", document_id="paper-17", source_url=PAPER)
```

A chunk's `id` is its place in reading order, such as `semantic-0`. Its `content` is Markdown, with page breaks and figure images removed.

**3. Embed and index**

This code embeds every chunk into an in-memory vector store. Any LangChain embedding model and vector store take the same documents.

```python theme={null}
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_huggingface import HuggingFaceEmbeddings

store = InMemoryVectorStore(HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2"))
store.add_documents(docs)
```

**4. Search with provenance**

This code searches one tenant's chunks and prints the file and pages of each hit.

```python theme={null}
def search(question, tenant_id, k=3):
    """The k chunks closest to the question, from one tenant's documents only."""
    return store.similarity_search(question, k=k, filter=lambda doc: doc.metadata["tenant_id"] == tenant_id)


for doc in search("What does multi-head attention do?", tenant_id="tenant-a"):
    print(doc.metadata["file_name"], doc.metadata["pages"], doc.page_content[:200])
```

Filter by tenant before any chunk reaches a model. Pass each hit to the model with its file name and pages, so the answer can cite them.

***

## How it works

This diagram shows where each part of a chunk goes in the vector store:

```mermaid theme={null}
%%{init: {"themeVariables": {"fontSize": "14px"}, "flowchart": {"nodeSpacing": 24, "rankSpacing": 32, "padding": 8}}}%%
flowchart LR
    P["/parse + chunking"] --> C[Chunk]
    C -->|content| E[Embedding]
    C -->|parse_id + id| I[Vector id]
    C -->|pages + your ids| M[Metadata]
    E & I & M --> V[Vector store]
```

At query time, the metadata does the work:

```mermaid theme={null}
%%{init: {"themeVariables": {"fontSize": "14px"}, "sequence": {"mirrorActors": false, "messageMargin": 28, "height": 44, "boxMargin": 6}}}%%
sequenceDiagram
    participant User
    participant App
    participant Store as Vector store
    User->>App: Question
    App->>Store: Search one tenant
    Store-->>App: Chunks + metadata
    App-->>User: Answer citing pages
```

The vector id combines the `parse_id` and the chunk's `id`, so indexing the same parse again reuses its ids, and a store that writes by id, such as `InMemoryVectorStore`, replaces those vectors instead of adding copies.

## Choose a chunk size

A chunk longer than `max_characters` is cut again, at paragraph breaks first, then at line, sentence and word breaks.

| Retrieval need | `max_characters` | Why |
| - | - | - |
| Broad questions | Leave it out | Each chunk runs as long as its topic. |
| An embedding model with an input limit | Below the limit, in characters | A chunk past the limit is cut short by the model. `all-MiniLM-L6-v2` reads 256 tokens, about 1,000 characters of English. |
| Precise answers with page citations | Smaller, such as `600` | Each chunk covers fewer pages, so citations are tighter. |

Chunking is included in Parse's [price](https://docs.runpulse.com/v2/pricing.md).

## Keep the index current

> [accordion] Re-parsing a document
>
> A new parse of a document has a new `parse_id` and new chunk boundaries. Delete the document's old vectors by `document_id`, then index the new ones.

> [accordion] Keeping the result
>
> Read each result once, and keep it before you index it. A result can [expire](https://docs.runpulse.com/v2/jobs.md#retention), and some organizations delete a result the first time it's read.

> [accordion] One chunk per page
>
> To index one chunk per page instead, join each page's elements and leave out page headers, footers and numbers:
>
> ```python theme={null}
> from collections import defaultdict
>
> SKIP = {"Page-header", "Page-footer", "Page-number"}
> by_page = defaultdict(list)
> for element in result.elements:
>     if element.category not in SKIP and getattr(element, "content", None):
>         by_page[element.page].append(element.content)
> page_texts = {page: "\n\n".join(parts) for page, parts in sorted(by_page.items())}
> ```

## Next steps

> [card] [Parse options](https://docs.runpulse.com/v2/parse.md)
>
> Chunking, word boxes, footnotes and more.

> [card] [Elements and bounding boxes](https://docs.runpulse.com/v2/elements.md)
>
> Element categories, pages and boxes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.