Highlight sources

Show a reviewer exactly where an extracted value came from: the page and box of each word behind it, ready to draw over the page.

Bank statement with word-level bounding boxes overlaid — Word-level boxes drawn over a bank statement.

What you need

The Python SDK and an API key, set up as in Cookbooks.
Word boxes enabled for your organization. Without them, a parse that asks for word boxes is refused with a 400; ask support to turn them on.
A document to review. This recipe uses the Bank Statement sample, which has 2 pages.

Build it

Parse with word boxes

This call parses the statement and waits for it to finish. word_level_bounding_boxes adds every word, with its box, to the result.

import os
import re

from pulse import Pulse

client = Pulse(api_key=os.environ["PULSE_API_KEY"])
STATEMENT = "https://platform.runpulse.com/api/examples/637e5678-30b1-45fa-acc4-877f2d636419/pdf"

parse_id = client.parse.run(
    file_url=STATEMENT,
    config={"model": "ultra-2", "extensions": {"word_level_bounding_boxes": True}},
).parse_id

Word boxes make a result much larger. Ask for them only on documents someone will review.

Extract the value

This code fills a one-field schema from the parse and reads the value with its citation.

schema = {"type": "object", "properties": {"account_holder": {"type": "string"}}}
extract_id = client.extract.run(parse_id=parse_id, config={"input_schema": schema}).extract_id
leaf = client.extract.get_result(extract_id).extraction.enriched["account_holder"]
print(leaf.value, leaf.citation)

Read the parse

This code reads the parse, indexes its elements and table cells by id, and keeps its word boxes.

parse = client.parse.get_result(parse_id)
located = {element.id: element for element in parse.elements}
for element in parse.elements:
    for cell in getattr(element, "cells", None) or []:
        located[cell.id] = cell
words = (parse.extensions.word_level_bounding_boxes if parse.extensions else None) or []

Read the parse only after every step on it has finished: where results are deleted on first read, a parse you’ve already read can’t feed another step.

Find the value's words

highlight returns the words a value was read from. For a cited table cell, that’s every word in the cell. For any other element, it’s the element’s words that match the value’s text.

def norm(text):
    """Lowercase, without punctuation, so "$1,234.50" matches "1234.50" and "Smith," matches "smith"."""
    return re.sub(r"[^\w.]", "", str(text).lower()).strip(".")


def inside(word, box):
    """Whether the center of the word's box falls inside the element's box, on the same page."""
    if word.page != box.page or word.bbox is None:
        return False
    xs, ys = box.bbox[0::2], box.bbox[1::2]
    x, y = sum(word.bbox[0::2]) / 4, sum(word.bbox[1::2]) / 4
    return min(xs) <= x <= max(xs) and min(ys) <= y <= max(ys)


def highlight(leaf):
    """The words behind an extracted value, from its citations."""
    if leaf.value is None:
        return []
    tokens = {norm(token) for token in str(leaf.value).split()}
    hits = []
    for cite in leaf.citation:
        box = located.get(cite.sub_element_id or cite.element_id)
        if box is None or box.bbox is None:  # spreadsheets and some captions have no box
            continue
        found = [word for word in words if inside(word, box)]
        hits += found if cite.sub_element_id else [word for word in found if norm(word.content) in tokens]
    return hits

Convert the boxes to pixels

A bbox is the four corners of a box, clockwise from the top-left, as fractions of the page’s width and height. This code converts each word’s box to pixels on a page image.

def to_pixels(bbox, width, height):
    """A bbox's corners in pixels, on a page image `width` by `height` pixels."""
    return [round(v * (width if i % 2 == 0 else height)) for i, v in enumerate(bbox)]


for word in highlight(leaf):
    print(word.page, word.content, to_pixels(word.bbox, 1700, 2200))  # a US Letter page at 200 DPI

Draw each box over page word.page in your viewer. Store the parse_id, page and box with each review note to draw it again later.

How it works

A value’s citation names a parse element, and that element’s box picks out the word boxes inside it:

citation[].element_id is the id of one of the parse’s elements.
citation[].sub_element_id, when present, is the id of a cell in that table’s cells.
Word boxes are Word elements, with a content, page and bbox, in extensions.word_level_bounding_boxes.

Edge cases

A value with no highlight

A value with an empty citation list can’t be placed. Neither can a value read from a spreadsheet, whose elements have no bbox, or from a caption without one. Show these values without a highlight.

Numbers that don't match their words

Matching compares text, so a value can be written differently from its words, such as 1234.5 for “1,234.50”. When no word matches, highlight the whole cited element instead.

Cookbooks

What you need

Build it

How it works

Edge cases

Next steps

Elements and bounding boxes

Extract

​What you need

​Build it

​How it works

​Edge cases

​Next steps

Elements and bounding boxes

Extract

What you need

Build it

How it works

Edge cases

Next steps