NVIDIA Nemotron Parse 1.1 | Summary

Contents

Summary
1. Introduction
2. Nemotron-Parse
- 2.1. Model Architecture
- 2.2. Prompts and Output Format
3. Training and Data
- 3.1. Datasets
4. Experimental results
5. Implementation
6. Contributors
- Infrastructure, Data, Benchmarking, and Product
8. Examples
Appendix
Brief Thoughts

This article explains the key points of NVIDIA Nemotron Parse 1.1.

2025-11-25 (arXiv)
Chumachenko, Kateryna, Deshmukh, Amala Sanjay, Seppanen, Jarno, Karmanov, Ilia, Chen, Chia-Chih, Voegtle, Lukas, Fischer, Philipp, Wawrzos, Marek, Motiian, Saeid, Ageev, Roman, et al.
NVIDIA
Paper
Project Page

Summary

NVIDIA Nemotron-Parse 1.1 addresses document parsing: extracting text from page images while retaining formatting, table structure, element locations, and reading order. On the GOT document-recognition benchmark, its 885M-parameter model achieves an optical character recognition (OCR, recognition of text in images) F1 score, a text-recognition comparison score, of 0.9785, below Gemini Flash 2.0 at 0.9915 and above the other extractors listed in Table 3.
An image encoder and text decoder convert page images into formatted text, bounding boxes—rectangles locating page elements—and semantic classes, labels identifying roles such as titles and captions. The token compression (TC) variant reduces the image representation from 3201 to 833 tokens, the feature units passed to the decoder, and incorporates floating elements such as tables and captions into the page’s natural reading order.
Training combines synthetic documents, public datasets, and human annotations, with prompts matched to the annotations available for each sample. NVpdftex, a pipeline coupling LaTeX document compilation with structured annotation extraction, supplies aligned text, character locations, semantic labels, and reading order.
On the English subset of OmniDocBench v1.0, a document-parsing benchmark, TC improves the overall error metric from 0.131 to 0.129 and the reading-order metric from 0.066 to 0.048, while slightly worsening text, formula, and table metrics. On one H100 graphics processing unit (GPU) using bf16, a 16-bit numerical format, throughput increases from 3800 to 4500 tokens/second over 10,000 pages containing 1000 tokens each.
The reported accuracy and throughput favor different variants depending on the workload. Multilingual testing uses generated scientific documents; Chinese, Japanese, and Korean support is limited for images and documents outside scientific and standard PDF settings, and the paper provides no numerical ablations isolating decoder positional embeddings or multi-token training.

1. Introduction

Document extraction for language models, retrieval, and question answering requires layout, reading order, formulas, tables, and element roles in addition to plain text. The paper identifies two practical difficulties: multi-stage pipelines sacrifice throughput, while end-to-end vision-language models—models processing images and generating language—can perform unevenly across extraction subtasks.

Nemotron Parse 1.1 succeeds Nemoretriever-Parse 1.0 and Eclair, preserving formatted text and block-level layout information within one model. Markdown supplies text formatting, while LaTeX represents mathematical and tabular content.
TC retains the core extraction features and places floating elements, including tables and captions, within the natural ordering of the page.

2. Nemotron-Parse

Nemotron-Parse combines an encoder-decoder architecture with a conditional output interface, allowing prompts to select the requested annotations. Its two variants differ in image-token compression and reading-order behavior, while both support text extraction, element localization, and semantic classification.

2.1. Model Architecture

The vision encoder is initialized from RADIO, the cited vision foundation model, and uses ViT-H/16, a vision transformer architecture, with 657M parameters. A vision neck, the component connecting encoder features to the decoder, uses horizontal convolutional kernels of size 1 × 4 and stride 1 × 4; for a 1648 × 2048 image, it produces 3200 tokens before adding RADIO’s summary token.

TC applies pixel-shuffle, an additional rearrangement of image features used here to shorten the sequence, reducing it to 833 tokens; the paper describes a total ×16 reduction relative to the encoder representation.
The text decoder uses mBART, the cited multilingual encoder-decoder model, reduced to 10 layers with tied weights, meaning shared model weights. The abstract reports a 256M-parameter language decoder and 885M parameters overall.
Each output token is conditioned on the compressed image representation and preceding text tokens, including the input prompt.

2.1.1. Positional embeddings: the decoder is trained and evaluated without explicit positional embeddings, which supply token-position information. The authors report comparable accuracy and longer-context inference, arguing that causal attention—attention restricted to preceding tokens—and spatial information in image features already provide positional cues.

The authors also suggest that removing an additional one-dimensional position signal avoids interference with the two-dimensional spatial information in image features.
The paper does not specify the tested context-length limits or provide a numerical comparison with a decoder using positional embeddings.

2.1.2. Multi-token inference: additional linear layers predict several future tokens from the current hidden state and the preceding prediction’s embedding. Training uses teacher forcing, supplying reference token embeddings for additional predictions; inference uses greedy decoding, selecting the highest-scoring token, without token verification.

The additional prediction head computes l_head(l_1(h_n + l_2(e_{n+1}))), where h_n is the current hidden state and e_{n+1} is the embedding of the next token.
The authors also report improved accuracy when multi-token training is followed by ordinary single-token inference, but provide no numerical ablation or separate multi-token throughput measurement.

2.2. Prompts and Output Format

2.2.1. Input prompts: three prompt choices control text format, bounding boxes, and semantic classes, yielding eight valid combinations. Training assigns each sample the combination supported by its annotations, excluding requests for classes without boxes and the trivial case requesting no output.

Text options are <output_markdown>, <output_plain>, and <output_no_text>. Markdown output uses LaTeX for tables and formulas requiring that syntax; simpler inline formulas remain in Markdown.
Location options are <predict_bbox> and <no_bbox>; class options are <predict_classes> and <no_classes>.
The maximal-information prompt (MIP), requesting formatted text, locations, and classes together, is <output_markdown><predict_bbox><predict_classes>.

2.2.2. Output format: each semantic block contains its top-left coordinates, text, bottom-right coordinates, and class label. The paper describes relative coordinates on a 1024 × 1280 scale and gives the matching pattern <x_(\d+)><y_(\d+)>(.*?)<x_(\d+)><y_(\d+)><class_([^>]+)>.

The base model puts Page-Header elements first, followed by Text, Section-Header, List-Item, Title, and Formula elements in reading order, and then Footnotes, Page-Footers, Tables, Pictures, and Captions.
TC integrates these floating elements into the natural page ordering.
The source example is <x_0.1152><y_0.2586># NVIDIA Nemotron-Parse 1.1<x_0.8799><y_0.2797> <class_Title>. Its decimal coordinates differ from the integer-matching pattern printed in the paper.

3. Training and Data

Training combines synthetic, public, and human-annotated data with different available output labels. This section documents the data sources and annotation construction; it does not specify a complete optimization schedule, training compute budget, or sampling proportions.

3.1. Datasets

NVpdftex couples LaTeX compilation and structured-output extraction in one pass, preserving alignment between rendered pages and text down to character-level bounding boxes. It extends the TeX Live compilation toolchain to intercept character and node creation, box allocations, token reads, and page-output events.

Unlike the cited Nougat pipeline, which converts LaTeX through HTML to Markdown, NVpdftex extracts annotations during compilation. The pipeline is released at https://github.com/NVIDIA-NeMo/Curator/tree/experimental/experimental/nvpdftex.
Machine translation adds data in 6 languages, while LaTeX-level font, color, and layout changes diversify rendering.

DocLayNet, a public layout-analysis dataset, receives additional automatic labels for reading order, text within images, Markdown formatting, tables, and formulas. Common Crawl samples have human-provided plain text, bounding boxes, and classes, supplemented with automatic labels for text within images and formatting.

For Common Crawl, a stage-1 trained model processes individual box crops; predictions are filtered using edit distance, a measure of text changes, between formatting-stripped predictions and reference plain text.
Low-quality regions are blanked in the image. Full-page predictions supply heading formatting, which is aligned with individual blocks using heuristics.

The largest sources in Table 1 are Multilingual arXiv at 8.3M, Multilingual Synthetic OCR data at 3.5M, and Multilingual Wikipedia OCR data at 9.5M. Smaller sources contribute diverse layouts and tables: SynthTabNet 480K, DocLayNet 56K, Common Crawl samples 255k, Synthetic tables 26K, Pubtables 585K, Fintabnet 91.5K, and TabRecSet 38.2K.

Synthetic tables are generated in HTML, a markup format for structured content, converted to LaTeX, and rendered with varied layouts, formatting, sparsity, and checkboxes.
Dense OCR data includes random words, characters, and symbols in 6 languages; Wikipedia text is rendered with font, background, and color augmentations.
Public table datasets are converted from HTML to LaTeX, with remaining page elements automatically labeled. Released training data is available at https://huggingface.co/datasets/nvidia/Nemotron-VLM-Dataset-v2.

Training dataset sizes, annotation modalities, and language coverage.

4. Experimental results

Experiments cover text recognition and reading order, structured table extraction, and multilingual OCR. Evaluation conditions differ across datasets, including language coverage, normalization rules, and required output conversions.

4.1. OCR and reading order evaluation

The internal evaluation uses 789 human-labeled PDF pages from magazines, books, and Common Crawl, with explicit reading-order annotations. Outputs are normalized, and tables, equations, and TeX commands are excluded; headers and footers are also masked for GOT’s Markdown mode.

Without masking, Nemotron-Parse-MIP records word error rate (WER), the reported word-level error metric, of 0.109 and F1 of 0.958; TC records 0.111 and 0.953.
With masking, the base model records WER 0.102 and F1 0.957, compared with Kosmos-2.5 OCR mode at 0.195 and 0.937. TC records 0.121 and 0.949.
Table 2 marks all baseline rows as masked, while the accompanying text specifically discusses masking GOT Markdown mode. This discrepancy leaves the baseline preprocessing description incomplete.

Internal test-set WER and F1 under masked and unmasked conditions.

On GOT, the base model reaches OCR/F1 0.9785 and TC reaches 0.9755, compared with Gemini Flash 2.0 at 0.9915. Both variants have text-only reading-order edit distance 0.014, compared with 0.0125 for Gemini Flash 2.0.

The base model’s reading-order METEOR and BLEU, text-comparison scores applied to the extracted reading-order sequence, are 0.9858 and 0.9623; TC scores 0.9838 and 0.9582.
The base model exceeds all listed extractors except Gemini Flash 2.0 across the four reported metrics.

OCR and text-only reading-order metrics on the GOT benchmark.

On OmniDocBench v1.0’s English subset, the base model reports overall, text, formula, table, and order metrics of 0.131, 0.052, 0.288, 0.118, and 0.066. TC reports 0.129, 0.055, 0.295, 0.121, and 0.048, respectively, with lower values indicating better performance.

TC’s overall improvement accompanies better reading order and slightly worse text, formula, and table metrics.
The authors explain that correctly represented simple formulas can be penalized when Markdown output lacks the LaTeX math delimiters expected by the evaluator.
Competitor results come from the DeepSeek-OCR paper. DeepSeek-OCR-Gundam-M†200dpi has overall 0.123, while dots.ocr†200dpi has table 0.099 and order 0.04; the marked variants use the table’s 200dpi image-resolution condition.

English OmniDocBench results and image-token counts for pipeline and end-to-end models.

4.2. Table extraction

Tables are predicted in LaTeX and converted to HTML or Markdown when required for evaluation. Table 5 reports TEDS and S-TEDS, two table-extraction comparison scores whose definitions are not given in this paper, while Table 6 reports a separate table-similarity score on RD-TableBench’s diverse real-world tables.

Base/TC TEDS values are 86.2/85.3 on RD-TableBench, 81.3/80.9 on PubTabNet, and 82.68/84.73 on OmniDocBench 1.0 en. Corresponding S-TEDS values are 79.9/79.6, 93.99/93.6, and 89.06/91.44.
RD-TableBench table similarity is 85.8 for the base model and 85.4 for TC, below Reducto’s 90.2 and above Azure’s 82.7.
Table 6 takes competitor scores from public Reducto results; its table-similarity values are distinct from Table 5’s TEDS values.

TEDS and S-TEDS results on three public table-extraction benchmarks.

RD-TableBench table-similarity comparison.

4.3. Multilingual OCR

Multilingual evaluation uses 10,000 dense scientific documents per language generated through NVpdftex with font and color augmentations. Table 7 reports WER/F1 of 0.03/0.98 for English, 0.06/0.96 for German, 0.05/0.97 for French and Italian, 0.04/0.97 for Spanish, and 0.03/0.98 for Chinese and Japanese.

Although the text states F1 > 0.96 for all languages, the table reports German at 0.96.
The authors describe Chinese, Japanese, and Korean support as effective in scientific and standard PDF settings, with limited support for real-world images and documents. Table 7 contains no Korean result.

Multilingual NVpdftex OCR results measured with WER and F1.

5. Implementation

Both variants are released in fp32, a 32-bit floating-point format, and bf16 formats with VLLM inference support at https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.1 and https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.1-TC. An optimized NIM deployment container, a packaged model-serving release, is available at https://build.nvidia.com/nvidia/nemotron-parse.

Using a bf16 model on one H100 GPU, end-to-end throughput averaged over 10,000 pages of 1000 tokens each is 3800 tokens/second for the base model and 4500 tokens/second for TC.
The abstract describes a 20% speed improvement; the table’s rounded values imply approximately 18.4% higher throughput. The implementation text approximates these rates as 4 and 5 pages/second.
Batch size, concurrency, and latency distributions are not reported, limiting direct translation of these throughput results to interactive serving.

End-to-end generation throughput for the base and token-compressed variants.

6. Contributors

Core Model Development credits Kateryna Chumachenko, Amala Sanjay Deshmukh, Jarno Seppänen, Ilia Karmanov, Chia-Chih Chen, Lukas Vögtle, Philipp Fischer, Marek Wawrzos, Timo Roman, Karan Sapra, Andrew Tao, and Bryan Catanzaro.

Infrastructure, Data, Benchmarking, and Product

Infrastructure, Data, Benchmarking, and Product credits Saeid Motiian, Roman Ageev, Kedi Wu, Alexandre Milesi, Maryam Moosaei, Krzysztof Pawelec, Padmavathy Subramanian, Mehrzad Samadi, Xin Yu, Celina Dear, Sarah Stoddard, Jenna Diamond, Jesse Oliver, Leanna Chraghchian, Patrick Skelly, Tom Balough, Yao Xu, Jane Polak Scowcroft, Daniel Korzekwa, Darragh Hanley, and Sandip Bhaskar.

8. Examples

The examples illustrate three outputs: page-element locations and classes, formatted mathematical text, and complex tables reconstructed from predicted LaTeX. These selected examples show the output representation without measuring aggregate accuracy or failure frequency.

Figure 1 labels pictures, text, a section header, list items, a caption, and a page footer on a visually varied page.
Figure 2 pairs predicted classes and bounding boxes with rendered mathematical content from a two-column document.
Figure 3 shows detected table regions, raw predicted LaTeX, and rendered tables.

Predicted bounding boxes and semantic classes for document layout analysis.

Semantic regions and rendered text, formatting, and mathematical equations.

Complex table extraction from detected regions through LaTeX output to rendered tables.

Appendix

The supplied 14-page paper contains no separate appendix. References follow the examples, and no additional experimental appendix or supplementary ablation tables are included.

Brief Thoughts

TC merits evaluation for workloads that prioritize throughput and integrated reading order: it uses 833 image tokens, improves OmniDocBench’s order metric to 0.048, and reaches 4500 tokens/second under the reported H100 conditions. Its OCR and table regressions make the choice workload-dependent, and its table results vary across benchmarks and metrics. Released weights, training data, and the NVpdftex pipeline support inspection and reuse. The paper does not provide optimization settings or mixture proportions sufficient to reproduce the complete training run. The individual effects of positional-embedding removal and multi-token training remain unquantified. Multilingual evaluation on the intended document domain would address the gap between generated scientific test pages and deployment data.