Document pipelines for RAG: OCR fallback, canonical text and chunk IDs you can diff

Most retrieval problems in document-heavy products start before the embedding model ever runs. A contract exported from an ERP has invisible text layered over a scan, every page repeats a confidentiality footer, page 7 switches to two columns, and one appendix has a broken font map, so the extractor returns (cid:31)(cid:52) instead of words. If ingestion is "extract text, chunk, embed", all of that goes into the index, and because embeddings make even broken text look searchable, nothing fails. Recall just drifts down while everyone blames the prompt or the retriever.

The fix is to treat ingestion like a build: deterministic steps, pinned versions, artifacts you can compare between runs, and checks that fail the run when the output changes for no reason. This post walks through the four places where pipelines usually go wrong (extraction, canonicalization, chunk size and chunk identity) and ends with a manifest that makes re-ingestion diffable. The chunking code below runs as-is with uv run.

Most business PDFs have a usable text layer on most pages, and OCR on a page that already has good text throws signal away. So decide per page. Extract native text first, then check each page: how many characters it produced relative to its area, how much of the output is (cid:N) placeholders or replacement characters, and whether the characters form words in the languages you expect. Pages that pass keep their native text. Pages that fail get rendered to an image and sent to OCR, and the manifest records which pages took that path.

For native extraction, pdfminer.six (MIT) is still maintained and gives you characters with coordinates. pypdfium2 wraps PDFium, the PDF engine from Chromium, under BSD and Apache licenses. PyMuPDF is excellent but dual-licensed under AGPL or a commercial license, which matters if you ship it inside a product. When you render a page for OCR, fix the DPI (300 is a sensible default) and record it, because OCR output changes with rasterization and you don't want a library default deciding what your corpus says.

Reading order is the next trap. Two-column pages extracted naively interleave the columns line by line, and tables come out as a soup of cells. Layout-aware tools fix this by detecting regions and their order first. If you find older tutorials built on layoutparser, note that its last release, 0.3.4, dates from April 2022; for new work, Docling (MIT) covers page layout, reading order, table structure and OCR for scanned pages, and it can also run vision-language models. For OCR itself, Tesseract 5.5 is the free on-premise option and handles clean scans well, while open VLM-based OCR models such as olmOCR (Apache 2.0) and the cloud providers' hosted document AI services are the ones to evaluate for bad scans, handwriting and dense tables, at a real cost per page in GPU time or fees. Route by page and document type instead of sending everything to the most expensive engine.

Whatever you choose, keep the raw output of each stage per page, together with the tool and its version. When retrieval gets worse in three months, you'll want to see whether extraction changed before you look at anything else.

Extracted text has noise that should never reach the embedder: running headers and footers, page numbers, soft hyphens and words broken across lines. Strip it before chunking, in one deterministic function, so the same input always produces the same text:

 1import hashlib
 2import re
 3import unicodedata
 4from collections import Counter
 5from dataclasses import dataclass
 6
 7from tokenizers import Tokenizer
 8
 9MODEL = "sentence-transformers/all-MiniLM-L6-v2"
10REVISION = "1110a243fdf4706b3f48f1d95db1a4f5529b4d41"
11MAX_SEQ_LENGTH = 256  # from the model's sentence_bert_config.json
12MAX_TOKENS = MAX_SEQ_LENGTH - 2  # leave room for [CLS] and [SEP]
13
14tok = Tokenizer.from_pretrained(MODEL, revision=REVISION)
15# tokenizer.json ships with truncation and padding at 128; counting through
16# it would silently cap every count at 128
17tok.no_truncation()
18tok.no_padding()
19
20
21def n_tokens(text: str) -> int:
22    return len(tok.encode(text, add_special_tokens=False).ids)
23
24
25def mask(line: str) -> str:
26    return re.sub(r"\d+", "#", " ".join(line.lower().split()))
27
28
29def strip_repeated_lines(pages: list[str], edge: int = 3, threshold: float = 0.6) -> list[str]:
30    """Drop lines near the top or bottom of a page that repeat on most pages."""
31    if len(pages) < 3:
32        return pages  # too few pages to tell a running header from content
33    edges = [p.splitlines()[:edge] + p.splitlines()[-edge:] for p in pages]
34    counts = Counter(m for lines in edges for m in {mask(l) for l in lines if l.strip()})
35    repeated = {m for m, c in counts.items() if c / len(pages) >= threshold}
36    out = []
37    for p in pages:
38        lines = p.splitlines()
39        keep = [
40            l for i, l in enumerate(lines)
41            if not ((i < edge or i >= len(lines) - edge) and mask(l) in repeated)
42        ]
43        out.append("\n".join(keep))
44    return out
45
46
47def canonicalize(pages: list[str]) -> list[str]:
48    """Return paragraphs: NFC, soft hyphens and line-end hyphenation removed, whitespace collapsed."""
49    text = unicodedata.normalize("NFC", "\n".join(strip_repeated_lines(pages)))
50    text = text.replace("­", "")
51    text = re.sub(r"(\w)-\n(\w)", r"\1\2", text)
52    paragraphs = re.split(r"\n\s*\n", text)
53    return [" ".join(p.split()) for p in paragraphs if p.strip()]

The header and footer stripper looks only at the first and last few lines of each page and removes lines that appear on at least 60% of pages once digits are masked, so "Page 3 of 42" and "Page 4 of 42" count as the same line. That catches confidentiality notices, page counters and ERP export banners without touching body text that happens to repeat. Unicode normalization to NFC matters more than it looks for Czech and other accented languages, because the same "é" can arrive as one code point or two depending on the PDF producer, and the two forms hash and tokenize differently.

The hyphenation rule is a trade-off you should know about. Joining pay- at a line end with ment on the next line is right, but the same rule turns a compound like well- / known into wellknown. If your documents are full of hyphenated compounds, check the joined word against a dictionary before removing the hyphen. And if your layout tool labels headers, footers and headings itself, as Docling does, use its labels and keep the repetition rule as a fallback for the native-text path.

Chunk size is usually set in round numbers, 350 or 512 tokens, without checking what the embedding model accepts. all-MiniLM-L6-v2 is a common default, and its model card says input longer than 256 word pieces is truncated, so with 350-token chunks the last quarter of every chunk is silently dropped at embedding time. It's still in the database and still shown to the LLM, but it can never be found by vector search. all-mpnet-base-v2 stops at 384, and hosted models have their own limits. Read the limit from the model's configuration, subtract the special tokens, and count with the model's own tokenizer.

There's a second trap in that tokenizer. The tokenizer.json shipped with all-MiniLM-L6-v2 has truncation and padding enabled at 128 tokens, so if you load it with the tokenizers library and count tokens without turning that off, every count tops out at 128 and every chunk looks like it fits. The symptom is easy to miss: while testing the code below, every chunk came out at exactly 128 tokens until truncation was switched off.

The chunker itself stays inside sections, packs whole paragraphs up to the budget, and only splits a paragraph at sentence boundaries, or as a last resort at a token offset, when it doesn't fit on its own:

 1HEADING = re.compile(r"^\d+(\.\d+)*\s+\S")
 2
 3
 4def sections(paragraphs: list[str]) -> list[tuple[str, list[str]]]:
 5    out: list[tuple[str, list[str]]] = [("", [])]
 6    for p in paragraphs:
 7        if HEADING.match(p) and len(p) < 120 and not p.endswith((".", "!", "?")):
 8            out.append((p, []))
 9        else:
10            out[-1][1].append(p)
11    return [s for s in out if s[1]]
12
13
14def split_long(paragraph: str, budget: int) -> list[str]:
15    """Split a paragraph that exceeds the budget at sentence ends, then hard-split by tokens."""
16    pieces, cur = [], ""
17    for sentence in re.split(r"(?<=[.!?])\s+", paragraph):
18        cand = f"{cur} {sentence}".strip()
19        if n_tokens(cand) <= budget:
20            cur = cand
21            continue
22        if cur:
23            pieces.append(cur)
24        enc = tok.encode(sentence, add_special_tokens=False)
25        while len(enc.ids) > budget:
26            end = enc.offsets[budget - 1][1]
27            pieces.append(sentence[:end].strip())
28            sentence = sentence[end:].strip()
29            enc = tok.encode(sentence, add_special_tokens=False)
30        cur = sentence
31    if cur:
32        pieces.append(cur)
33    return pieces
34
35
36@dataclass
37class Chunk:
38    chunk_id: str
39    ordinal: int
40    section: str
41    text: str
42    tokens: int
43
44
45def chunk_document(doc_id: str, pages: list[str]) -> list[Chunk]:
46    chunks: list[Chunk] = []
47    seen: Counter[str] = Counter()
48    for heading, paras in sections(canonicalize(pages)):
49        prefix = f"{heading}\n" if heading else ""
50        budget = MAX_TOKENS - n_tokens(prefix)
51        units = [u for p in paras for u in (split_long(p, budget) if n_tokens(p) > budget else [p])]
52        cur: list[str] = []
53        for u in units + [None]:
54            if u is not None and n_tokens("\n".join(cur + [u])) <= budget:
55                cur.append(u)
56                continue
57            if cur:
58                text = prefix + "\n".join(cur)
59                norm = " ".join(text.lower().split())
60                seen[norm] += 1  # identical text twice in one doc still gets distinct IDs
61                digest = hashlib.sha256(f"{doc_id}\x00{seen[norm]}\x00{norm}".encode()).hexdigest()[:32]
62                chunks.append(Chunk(digest, len(chunks), heading, text, n_tokens(text)))
63            cur = [u] if u is not None else []
64    return chunks
65
66
67def diff(old: list[Chunk], new: list[Chunk]) -> dict[str, int]:
68    a, b = {c.chunk_id for c in old}, {c.chunk_id for c in new}
69    return {"kept": len(a & b), "added": len(b - a), "removed": len(a - b)}

Every chunk starts with its section heading, which gives both the embedder and the LLM context that a bare paragraph lacks, and the heading's tokens come out of the budget. The numbered-heading regex is a stand-in that works for contracts and policies; with a layout tool, use its heading labels instead. On a synthetic four-page contract with six sections, running headers, page footers and a word hyphenated across a line, the chunker produced 16 chunks with a maximum of 253 tokens against a budget of 254, and no header or footer text survived into the canonical paragraphs.

The chunk ID decides whether you can tell what changed between two ingestion runs. A common scheme hashes the document ID, page, chunk ordinal and text together. The ordinal is the problem: insert one paragraph near the start of a document, and every chunk after it gets a new ordinal and therefore a new ID, even though its text didn't change. A diff between runs then says the whole document changed, which is useless, and your pipeline re-embeds everything after the edit.

The code above derives the ID from the document ID and the normalized chunk text only, and keeps the ordinal as a separate field for ordering. An occurrence counter keeps two identical chunks in one document, such as a repeated boilerplate clause, from colliding. To see the difference, the test inserts a twelve-sentence paragraph into section 2 of the synthetic contract and diffs the chunk IDs of the two versions:

 1content IDs: {'kept': 15, 'added': 2, 'removed': 1}
 2ordinal IDs: {'kept': 2, 'added': 15, 'removed': 14}

With content IDs, the diff shows exactly the local change: one chunk in section 2 was replaced by two, and the other fifteen kept their IDs, so their embeddings can be reused. With ordinals in the hash, fourteen of sixteen chunks look new. Because packing is greedy, an edit can still shift chunk boundaries until the end of its section, but sections stop the cascade, and in practice that's the difference between re-embedding a handful of chunks and re-embedding the document.

Store the ID with the page range, the section, the character offsets into the canonical text and the version of the chunking code. That's what a RAG answer needs to cite a passage precisely, and it's what the post on why RAG fails and how to fix it builds its provenance schema on.

Every ingestion run should write a manifest per document version: the hash of the raw file, the extraction engine and version, which pages went through OCR and at what DPI, the tokenizer and its exact revision, the chunk budget, a hash of the chunking code, per-page hashes of the canonical text and the list of chunk IDs. Here's one produced by the code above, trimmed:

 1{
 2  "document_id": "contract-42",
 3  "raw_sha256": "3321d5388c291c176d552345cd4e1fa529e2d727f9d0bf4a9c98749765d19105",
 4  "extraction": {
 5    "engine": "pdfminer.six",
 6    "version": "20260107",
 7    "ocr_pages": [],
 8    "dpi": 300
 9  },
10  "chunking": {
11    "tokenizer": "sentence-transformers/all-MiniLM-L6-v2",
12    "tokenizer_revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
13    "tokenizers": "0.23.3",
14    "max_tokens": 254,
15    "code_sha256": "25e79d1288dd0bda"
16  },
17  "pages": [
18    {"page": 1, "chars": 5598, "sha256": "d41c82ebcde85688"},
19    {"page": 2, "chars": 5758, "sha256": "1ddaccd255cdb3e4"}
20  ],
21  "chunks": ["ef4db40d66a255515f1bc6bbb905f12a", "77c4745a790478da771d4a7d4eec4c42"]
22}

With manifests in place, re-ingestion becomes a comparison, and the comparison can fail the job. If the raw file is byte-identical but a large share of chunk IDs changed, something in the pipeline changed, and that should stop publication until someone looks:

 1def check(old, new, max_changed=0.15):
 2    a, b = set(old["chunks"]), set(new["chunks"])
 3    changed = 1 - len(a & b) / max(len(a | b), 1)
 4    if old["raw_sha256"] == new["raw_sha256"] and changed > max_changed:
 5        raise SystemExit(f"FAIL: same raw file, {changed:.0%} of chunks changed")
 6    return round(changed, 3)

Add the same kind of check per batch for the OCR fallback rate, and per page for character counts: a page that produced 1,800 characters last week and 600 today is either a parser regression or a bad render, and either way you want to know before it reaches the index. The 15% threshold is a starting point; set it from what a normal re-run of your own corpus looks like. A replay command that reruns one document version with its historical manifest and prints the page-level differences turns most of these investigations into a few minutes of reading.

Pin everything that touches text: the extraction library, the OCR engine and its language data, the container image, the tokenizer revision and the embedding model. Changing any of them is a corpus migration, not a config tweak, because it changes what's in the index. Re-embed deliberately, compare retrieval on a fixed evaluation set, and keep the old vectors until the new ones have proven themselves, which is the process the post on rolling out a new embedding model in pgvector describes.