Document AI Shouldn’t Guess: Introducing MonkeyOCRv2, a Document-Native Vision Encoder
Pretrained on 113M document-image samples across 17 languages, MonkeyOCRv2 combines text generation with pixel reconstruction to preserve character-level visual evidence.
A vision model can understand that an image contains a financial statistic and still read “700,000” as “100,000.”
A single digit can alter a payment. A decimal point can alter a medical measurement. A missing minus sign can invalidate a formula. A confused character can change a person’s name, a legal clause, or an experimental result.
This failure reveals a deeper mismatch in today’s document AI systems.
Most widely used visual encoders are pretrained on natural images. They are designed to recognize objects, scenes, global concepts, or region boundaries, and they often learn to ignore local variations that do not change high-level semantics.
Documents demand almost the opposite.
A reliable document model must remain sensitive to character strokes, punctuation, digits, superscripts, subscripts, table boundaries, reading order, and other fine-grained visual details. What looks like a negligible pixel-level variation to a natural-image encoder may carry the most important information on a page.
We built MonkeyOCRv2 around this premise: document intelligence needs a visual foundation model designed for documents from the beginning.
MonkeyOCRv2 is a family of document-native vision encoders that learns both what a document says and what the document actually looks like.
The hidden bottleneck in document AI
Modern document systems often focus on the language model, decoder, prompting strategy, or post-training pipeline. The visual encoder is frequently treated as an interchangeable front end.
But the decoder can only reason over information that survives visual encoding.
When an encoder fails to preserve the difference between a “7” and a “1,” or between a decimal point and background noise, a language model may fill the gap using context. The result may sound plausible while disagreeing with the page.
This is especially dangerous in documents because plausible output is not always faithful output.
Document AI therefore needs more than semantic understanding. It needs a visual representation that preserves the evidence required for exact recognition.
Instead of adapting a natural-image encoder after the fact, we asked a more fundamental question:
What happens when the vision encoder itself is pretrained from scratch for document images?
MonkeyDoc v2: document-oriented pretraining at scale
To train such an encoder, we constructed MonkeyDoc v2, a corpus containing 113 million document-image samples across 17 languages.
The corpus contains approximately:
- 8 million page-level document images;
- 105 million cropped document elements;
- 61 million real-world samples;
- 52 million synthetic samples.
Its content covers printed papers, scanned books, newspapers, magazines, financial reports, handwritten notes, presentation slides, rendered documents, tables, formulas, scene text, and other document types.
To the best of our knowledge, MonkeyDoc v2 is the largest visual-text pretraining corpus built specifically for document scenarios.
For real-world documents, we use a multi-expert annotation pipeline rather than relying on the output of a single OCR system. Multiple complementary expert models independently recognize each document region. Low-agreement regions are filtered, and page-level checks are applied to detect missing regions and unreasonable reading order.
Synthetic data expands multilingual coverage, rare-character coverage, table structures, formulas, fonts, resolutions, and visual styles.
The goal is not simply to collect more pages. It is to provide the dense, character-level supervision that document-oriented visual representations require.
One representation, two complementary objectives
MonkeyOCRv2 uses two pretraining objectives:
- Image-to-text generation
- Pixel-level document reconstruction
The image-to-text objective teaches the visual representation to align with the textual content shown in the image.
The reconstruction objective asks the model to recover the original document image from its visual tokens. This encourages the encoder to retain character strokes, glyph shapes, punctuation, local structures, and layout details that may be discarded when training only for text generation.
A simple way to describe the design is:
Text generation teaches the encoder what the page says. Reconstruction teaches it what the page looks like.
These objectives provide complementary supervision.
Generation gives the representation semantic meaning. Reconstruction makes it harder for the encoder to learn semantic shortcuts while discarding fine-grained visual evidence.
The result is not just another end-to-end OCR model. MonkeyOCRv2 is designed as a reusable visual backbone that can be inserted into different document AI systems.
A 0.7B parser with a 0.1B vision encoder
We first evaluate MonkeyOCRv2 on multilingual document parsing.
MonkeyOCRv2-B-Parsing combines:
- a 0.1B MonkeyOCRv2 vision encoder;
- a lightweight 0.6B language model;
- 0.7B total parameters.
On MDPBench, which evaluates digital-born and photographed documents across 17 languages, MonkeyOCRv2-B-Parsing reaches an overall score of 83.3.
It exceeds the previous best evaluated open-source model, the 3B dots.mocr, by 2.8 absolute points, while using a vision encoder that is approximately 11 times smaller.
The 0.6B MonkeyOCRv2-S-Parsing model also reaches 82.5, showing that the performance is not limited to the larger encoder variant.
The important point is not simply that a smaller model achieves a higher benchmark score. It is that investing in the document-specific visual representation can reduce how much downstream model capacity is required to recover information that the encoder failed to preserve.
Controlled comparisons: change only the vision encoder
Document systems are difficult to compare because papers often change the data, decoder, training schedule, prompts, and architecture at the same time.
To isolate the contribution of the vision encoder, we conduct a controlled document-understanding comparison.
We keep the vision encoders frozen and pair them with the same language model, the same training data, and the same downstream settings. Under this protocol, MonkeyOCRv2 is compared with general-purpose and document-oriented encoders including CLIP, SigLIP 2, DINOv3, SAM, RADIO, OpenVision, oCLIP, and DiT.
MonkeyOCRv2-B achieves an average score of 57.2 across eight document-understanding benchmarks, compared with 44.0 for the strongest non-MonkeyOCRv2 encoder in this controlled table.
The improvement is especially visible in tasks that require precise perception of names, numbers, dense text, telephone numbers, charts, and multilingual document content.
This supports the central hypothesis behind the project: document-oriented visual pretraining is valuable independently of decoder scale.
One encoder family, seven document AI tasks
We also evaluate MonkeyOCRv2 as a backbone replacement across a broader set of document tasks:
- multilingual document parsing;
- document understanding;
- text recognition;
- formula recognition;
- scene-text detection;
- document tampering detection;
- overlapping-text segmentation.
Replacing existing visual backbones with MonkeyOCRv2 produces consistent gains across these task families.
For example, MonkeyOCRv2 raises the overall recognition result of a representative CRNN system from 58.7 to 67.3.
In formula recognition, a 110M UniMERNet-T model equipped with MonkeyOCRv2 outperforms the much larger 325M UniMERNet-B model.
MonkeyOCRv2 also improves systems for text detection, tampering localization, and overlapping-text segmentation.
These tasks require very different outputs: sequences, formulas, bounding regions, masks, tampering maps, structured pages, and answers. Consistent transfer across them suggests that MonkeyOCRv2 is learning a reusable document representation rather than optimizing for one parsing benchmark.
Does the model read the page, or autocomplete it?
High accuracy on normal text does not necessarily prove strong visual perception.
When a word is partially unclear, a decoder may recover the expected word from linguistic context. That is useful in some situations, but it also makes it difficult to determine whether a model is reading the visible characters or predicting what is statistically likely.
We therefore conduct a controlled scrambled-text experiment.
Characters are shuffled to remove most normal linguistic context, forcing the system to rely more heavily on visual evidence. We then reduce the input resolution and compare models trained with and without pixel reconstruction.
At the lowest tested resolution, adding reconstruction raises scrambled-text recognition accuracy from 55.4% to 72.1%. It also reduces the accuracy gap between semantically coherent and scrambled text from 29.3 points to 15.3 points.
This experiment is an operational proxy rather than a complete measurement of hallucination. Scrambled text introduces distribution shift and can interact with tokenization and decoding. Nevertheless, the controlled comparison provides evidence that reconstruction helps retain useful character-level information when linguistic context is unavailable.
Faithfulness when language and vision disagree
We further evaluate MonkeyOCRv2 on CHAOS-Bench, which intentionally modifies characters in document images to produce visually observable but semantically meaningless words.
A model relying heavily on language priors may “correct” these words into more plausible text. A faithful model should reproduce what is visibly present.
MonkeyOCRv2-B-Parsing achieves a page-average recall of 17.9, compared with 14.2 for HunyuanOCR-1.5 and 6.0 for PaddleOCR-VL-1.6 under the reported evaluation.
For the smaller MonkeyOCRv2-S-Parsing model, adding image reconstruction improves the result from 12.1 to 14.7, an absolute gain of 2.6 points.
These results do not mean document hallucination is solved. They show that pretraining the encoder to preserve visible information can make downstream output more faithful when language expectations conflict with the page.
The public release
The MonkeyOCRv2 release includes three vision-encoder variants:
- MonkeyOCRv2-S: 28M parameters;
- MonkeyOCRv2-B: 113M parameters;
- MonkeyOCRv2-AS: 21M parameters, designed for tasks benefiting from multi-scale visual features.
We also provide:
- 0.6B and 0.7B multilingual document parsing models;
- document-understanding models;
- model weights on Hugging Face and ModelScope;
- local inference;
- vLLM deployment;
- a Gradio interface;
- a FastAPI service;
- evaluation results and examples.
The project is released under the Apache License 2.0 and is available for both research and commercial use.
- Code: https://github.com/Yuliang-Liu/MonkeyOCRv2
- Paper: https://arxiv.org/abs/2607.11562
- Models: https://huggingface.co/collections/zenosai/monkeyocrv2
- ModelScope: https://modelscope.cn/datasets/zenosai/MonkeyDocv2
- Demo: http://vlrlabmonkey.xyz:8891/
What MonkeyOCRv2 does not claim
We want to be explicit about the current limitations.
MonkeyOCRv2-Parsing uses a deliberately simple supervised fine-tuning pipeline with a frozen encoder. It does not yet include the progressive post-training used by some specialized document parsers.
Its current architecture predicts layout autoregressively and recognizes individual elements in separate passes. This favors parsing accuracy and structural fidelity over minimum latency.
The current work also does not fully isolate every contribution from decoder design and reconstruction-loss scheduling.
Finally, although MonkeyDoc v2 spans 17 languages, its distribution remains biased toward higher-resource scripts. Better coverage of low-resource and historical writing systems remains an important direction.
MonkeyOCRv2 should therefore be understood as a strong document-native visual foundation and a starting point for further systems work—not as a claim that every document problem has been solved.
Help us find where it fails
Benchmarks are useful, but they do not cover the full diversity of real documents.
We are particularly interested in tests involving:
- photographed and distorted documents;
- Arabic and other right-to-left layouts;
- dense tables and scientific formulas;
- low-resolution scans;
- visually similar characters;
- unusual fonts and historical scripts;
- documents where visible text conflicts with linguistic expectations;
- complicated multilingual reading orders.
Run MonkeyOCRv2 on your own documents and share both successful and failed cases.
The most valuable community contribution is not simply a star. It is a document that reveals what the model still cannot see.
A document model should not produce the most plausible answer. It should produce the answer on the page.
Document AI shouldn’t guess. It should see.