Past Character Recognition: Why Document Structure Is the Harder OCR Problem

The character accuracy problem in OCR is largely solved for Latin scripts and, with enough training data, manageable for CJK scripts too. The harder problem, the one that quietly breaks downstream pipelines, is structural: given a page with two columns, a sidebar, a table, and a caption, what order should the text come out in, which lines form a logical paragraph, and which numbers belong to which table header?

NVIDIA’s Nemotron OCR v2 is getting attention mostly for its throughput numbers (34.7 pages per second on a single A100, 28x faster than PaddleOCR v5) and its multilingual coverage (English, Simplified Chinese, Traditional Chinese, Japanese, Korean, and Russian from a single 84M-parameter model). The architectural decision worth examining more carefully is the third component of the pipeline: a compact Transformer encoder that predicts reading order and logical groupings across detected text regions.

What the Relational Model Actually Does

The architecture has three parts. A RegNetX-8GF convolutional backbone processes the image once and produces feature maps that all three components reuse. A pre-norm Transformer handles the actual character recognition from detected regions. Then there is the relational model, a compact Transformer encoder that operates on the layout of detected text to predict structural relationships.

Specifically, the relational model predicts which text regions belong to the same logical unit (word to line, line to paragraph), which direction reading proceeds, and how multi-column layouts should be linearized. For tables, it predicts which cells share rows or columns and how headers relate to data cells.

This work is inspired by the HierText dataset, which introduced hierarchical paragraph-line-word annotations to the OCR research community. HierText acknowledged that raw bounding boxes and transcribed strings are not enough for practical document understanding. Nemotron v2 takes that further by making structural prediction a first-class output of the model rather than a postprocessing step.

Why Synthetic Data Enables This

The reason structural prediction is hard is not the model architecture; compact Transformers are well understood. The bottleneck is ground-truth annotations at scale. Annotating reading order and logical groupings on real documents requires human judgment, and at the scale needed to train a robust model across six languages and multiple layout types, manual annotation is not viable.

Synthetic rendering changes the equation. When you procedurally generate a document, you know everything about its structure before you render it. You know that the left column comes before the right column. You know that the third row of a table belongs to the same row as the fourth cell. You know that vertical Japanese text should be read top-to-bottom, right-to-left across columns. This information is free at generation time but prohibitively expensive to label post-hoc on real documents.

The training dataset is 12.2 million samples, of which 9.8 million are synthetic and around 680,000 are real-world images. That 9.8 million includes not just simple flowing text but a diverse set of layout modes: multi-column flowing text, scattered scene text, vertical CJK columns, tables with headers and borders, table-of-contents layouts with dot leaders, and PowerPoint-style slides. Each is rendered with complete ground-truth structural annotations, including the relation graphs that train the relational model.

For CJK languages, the scale of the character set compounds the annotation problem. The English model handles 855 characters; the multilingual model handles 14,244. Getting balanced coverage across that character set in real annotated data would require documents that collectively exercise every character in context, at multiple font weights, sizes, and degradation levels. The synthetic pipeline handles this through a font pool of 165 to 1,258 fonts per language, combined with augmentation passes at the text, image, and page level.

What the Structural Output Enables Downstream

For anyone building a document understanding pipeline, the structural output matters more than character accuracy numbers. Consider a few concrete cases.

A retrieval-augmented generation system ingesting PDFs needs coherent text chunks. If OCR output is a flat list of bounding-box strings in undefined order, chunking requires a separate layout analysis step. When the OCR model itself outputs a hierarchy with reading-order indices, that step is already done. The difference between a well-ordered chunk and a garbled one is the difference between a RAG system that works and one that fails on documents with any non-trivial layout.

A financial document processor parsing tables needs to know which cells are in the same row. Without structural prediction, you typically fall back to heuristic bounding-box overlap calculations, which break on rotated tables, merged cells, or documents that were scanned slightly skewed. A relational model trained on diverse synthetic tables has seen enough variation to generalize better.

Multi-column text is a persistent failure mode for older OCR systems because the natural reading order cuts across the physical vertical ordering of bounding boxes. A two-column page has text in the left column at y-coordinate 100 that should be read before text in the right column at y-coordinate 100. Sorting by y-coordinate gives alternating fragments from both columns, which destroys sentence coherence. The relational model’s explicit column grouping prediction is the correct solution to this problem.

The Benchmark Gap Worth Watching

The SynthDoG benchmark results for Nemotron v2 are strong. Normalized edit distance for Japanese drops from 0.723 in v1 to 0.046 in v2. Korean goes from 0.923 to 0.047. These are not incremental gains; they represent the difference between a model that functionally fails on a script and one that handles it competently.

The real-world OmniDocBench results tell a more cautious story. On Chinese text, Nemotron v2 multilingual posts a NED of 0.072 against PaddleOCR v5’s 0.037. On mixed English-Chinese content, the gap widens to 0.142 versus 0.041. The throughput advantage is still dramatic (34.7 pages/second versus 1.2), but accuracy on real-world documents lags behind systems trained with more real annotated data.

This gap reflects a known challenge with synthetic training data: the distribution of document styles, scanning artifacts, and degradation patterns in real documents diverges from what even a well-designed synthetic pipeline can generate. The augmentation suite includes morphological operations, median blur, elastic distortion, contrast and brightness jitter, and Gaussian noise, but real documents find ways to be stranger than any augmentation set anticipates.

For the relational model specifically, this gap deserves careful monitoring. Character recognition errors are measurable and well-understood. Structural prediction errors are harder to quantify in standard benchmarks but have larger consequences downstream. A character transcribed slightly wrong can often be corrected by a language model in the pipeline. A reading order that scrambles two columns is much harder to recover from programmatically.

Where This Fits in the Broader Ecosystem

The dominant approach to document structure understanding before dedicated OCR structural models was to chain separate systems: an OCR engine for character recognition, then a layout analysis model (LayoutLM, DocFormer, or similar) operating on the OCR output plus visual features to predict structure. These two-stage pipelines are effective but slow and require coordination between systems with different failure modes.

LayoutLMv3 and its successors treat document understanding as a multimodal problem, fusing text tokens with visual position embeddings. They perform well on tasks like form key-value extraction and document classification, but they operate on pre-extracted text and require a separate OCR pass anyway. The integration is at the representation level, not the inference level.

Nemotron v2’s approach of predicting structure inside the OCR pipeline, using a shared convolutional backbone to avoid redundant computation, is a cleaner architecture for pure information extraction tasks. The shared backbone is also why the throughput numbers are achievable: rather than running a separate layout model over OCR output, the structural prediction adds relatively small overhead on top of detection and recognition that are already running.

Whether that architectural cleanliness translates to better structural accuracy on diverse real-world documents is still an open question. OmniDocBench, which covers scientific papers, financial reports, textbooks, and administrative documents, is a reasonable stress test of real-world generalization. The current numbers suggest there is room to improve, particularly on documents with complex mixed layouts.

Using the Model

The model is available on Hugging Face under the NVIDIA Open Model License, and the synthetic training dataset is released separately as nvidia/OCR-Synthetic-Multilingual-v1 under CC-BY-4.0. There is also a live demo space for quick evaluation without spinning up the model locally.

The output format includes word, line, and paragraph-level bounding boxes with reading-order indices, which makes it straightforward to reconstruct linearized text respecting the structural predictions:

from PIL import Image
import torch

# Load model and processor from Hugging Face Hub
# nvidia/nemotron-ocr-v2
image = Image.open("document.png")
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

# outputs include:
# - per-region transcribed text
# - bounding boxes at word/line/paragraph level
# - reading_order: integer indices per region
# - relation_groups: logical groupings (same line, same paragraph)

# Sort regions by reading order to get linearized text
regions = zip(outputs.text, outputs.reading_order)
linearized = " ".join(text for text, _ in sorted(regions, key=lambda x: x[1]))

The reading-order indices are the output that matters most for downstream applications. A pipeline that uses them to sort and group text before chunking will behave meaningfully better on multi-column and table-heavy documents than one that ignores structural predictions.

The 84M parameter footprint is small enough to run alongside other models in a document processing pipeline without significant memory pressure. At 34.7 pages per second on an A100, a million-page corpus processes in roughly eight hours, which changes the economics of large-scale document processing considerably.

The dataset itself, 12.2 million samples with full hierarchical structural annotations across six language families, may turn out to be as valuable as the model weights. Training structural layout models typically requires expensive human annotation; having a large synthetically generated dataset with ground-truth reading order and relation graphs provides a foundation that other researchers can fine-tune against or use to bootstrap real-world annotation pipelines.