MonkeyOCRv2: A Visual-Text Foundation Model for Document AI
Most document-AI systems still run on vision encoders — CLIP, SigLIP, SAM, DINO — trained on natural images that never learned to see a stroke, a decimal point, or a reading order. MonkeyOCRv2 is our attempt at the visual encoder documents actually deserve: pretrained on MonkeyDoc v2 (113M images, 17 languages) by jointly learning image-to-text generation and pixel-level reconstruction. Frozen and paired with a 0.6B language model, its 0.1B encoder sets a new open-source state of the art on multilingual document parsing — matching or beating models an order of magnitude larger.
For years my group has worked on teaching machines to read — scene text, handwriting, tables, formulas, even oracle bones. Underneath almost everything we built sat one quiet assumption: that the vision backbone was a solved problem, something you download and fine-tune. We used CLIP, we used SAM, we used whatever encoder shipped inside the strongest multimodal model that month. MonkeyOCRv2 began the day we stopped believing that assumption. A document is not a photograph, and I have come to think the encoder is precisely where document AI has been quietly losing accuracy all along.
The mismatch we kept paying for
The vision models we all rely on — CLIP and SigLIP for alignment, DINO for self-supervised features, SAM for regions — were trained on natural images. Their objectives reward the gist of a scene and are built to be invariant to small local changes: a cat is a cat whether it shifts a little or the light changes. Documents demand the opposite discipline. Here a single stroke separates 未 from 末, a decimal point moves a number by an order of magnitude, a superscript rewrites an exponent. The cover figure lays out the two gaps we hit at once — a distribution gap (these encoders have barely seen a document) and a task gap (they were never asked to preserve character-level detail). Bolting a natural-image encoder onto a document model asks it to unlearn exactly the invariances it was trained to have.
MonkeyDoc v2: data built for documents, not borrowed from the web
You cannot pretrain a document-native encoder on data that does not exist, so we built it. MonkeyDoc v2 is, to our knowledge, the largest visual-text pretraining corpus for documents: 113 million densely annotated images across 17 languages and more than a dozen genres — academic papers, bids and forms, textbooks, exam papers, financial reports, government files, magazines, newspapers, manuals, posters, scene text, handwritten notes. The contrast with prior work is the whole point (Table 1). General encoders train on hundreds of millions to tens of billions of natural images; document-oriented predecessors — oCLIP, DiG, DiT, Donut, Pix2Struct — are smaller, thinner in annotation, and almost always confined to one or two languages. Coverage and dense labels are not glamorous, but they are what let an encoder learn a script no one ever told it the name of.
Two objectives, pulling against each other on purpose
The encoder is trained with two objectives at once. The first, image-to-text generation, aligns what the encoder sees with what the document says — dense, direct supervision that forces the features to carry linguistic content. The second, pixel-level reconstruction, asks the encoder to redraw the page from its own features, which forces it to keep the fine visual evidence — stroke shapes, glyph geometry, layout — that a pure text objective is happy to throw away. The two pull in different directions: generation abstracts toward meaning, reconstruction holds on to appearance. That tension is the design, not a compromise. When the language is familiar, generation carries the load; when it is a script the model has never been taught, reconstruction keeps it honest to the pixels.
What reconstruction actually buys — and how we measured it
It is easy to add a reconstruction loss and claim it helps; I wanted evidence. So we ran a controlled study that removes the crutch models usually lean on — language. We scramble the characters in a line so that context can no longer rescue a weak reading, then lower the resolution until the pixels themselves get hard to read (Figure 5). Two things fall out. First, at the lowest resolution our full model recognizes scrambled text 16.7 points more accurately than the same model trained without reconstruction — precisely the regime where you cannot guess from meaning and must actually see the strokes. Second, the gap between reading coherent text and reading scrambled text — a proxy for how much a model leans on language rather than vision — shrinks by 14 points. Reconstruction is what keeps the encoder looking when guessing would be easier. For low-resource scripts and degraded scans, that is the whole ballgame.
Does it transfer? Five tasks, one frozen recipe
A representation is only as good as what it carries into other problems. We first treat MonkeyOCRv2 as a drop-in backbone and simply swap it for the original encoder, keeping everything downstream fixed. On text recognition it lifts a plain CRNN's overall accuracy from 58.7% to 67.3% and pushes a PARSeq recognizer from 82.2 to 84.3, with the largest jumps on the hardest splits — curved, multi-oriented, occluded (Table 2). On formula recognition it lets a 110M model beat the 325M UniMERNet-B (Table 3): the same accuracy from a third of the parameters, which is what a better representation should buy you. On scene-text detection it improves across all six detector-benchmark combinations — most strikingly turning 67.9 into 78.1 on the hard ArT set (Figure 4). Document tampering detection and overlapping-text segmentation rise by +7.5 and +5.3 as well. Five tasks, one encoder, consistent gains — the sign that the features are genuinely document-native rather than tuned to a single benchmark.
From a good encoder to a small, strong system
The result I am most pleased with is what happens when we stop fine-tuning the encoder at all. We freeze the 0.1B MonkeyOCRv2 encoder, attach a 0.6B language model, and train a 0.7B document-parsing model, MonkeyOCRv2-Parsing. On MDPBench — a demanding multilingual benchmark spanning digital-born and photographed pages across 17 languages — it scores 83.3, a new open-source state of the art: +2.8 over the 3B dots.mocr with a vision encoder roughly 11× smaller, and +8.3 over PaddleOCR-VL-1.6 (Figure 2, Table 6). On OmniDocBench it outperforms general-purpose VLMs many times its size, including Qwen3-VL-235B and GPT-5.2. Kept frozen for document understanding and compared head-to-head against CLIP, SigLIP 2, DINOv3, SAM and OpenVision under identical training, it wins across eight VQA and KIE benchmarks (Table 8) — an overall of 57.2 where the CLIP-B baseline reaches 16.0. The through-line: put the visual burden where it belongs, in an encoder built to read, and a small language model is enough.
Reading order and structure, not just characters
Two failure cases make the point better than any average. In a densely typeset Arabic page (Figure 6), competing systems recognize the words but scramble the reading order — a right-to-left, multi-column layout is exactly the kind of global structure a natural-image encoder never had to learn; MonkeyOCRv2-Parsing keeps the order intact. In a photographed Chinese product sheet (Figure 7), the challenge is a nested specification table; other parsers flatten or mis-merge its cells, while ours reconstructs the structure faithfully. Getting characters right is table stakes. Getting order and structure right is what makes a parse usable downstream — and it is where a document-native encoder shows its hand.
Where it stops
I would rather state the limits than have a reader find them. MonkeyOCRv2 is a visual encoder, not a reasoning engine — it makes the pixels legible, but the language model on top still owns the reasoning, and on the hardest MDPBench pages the best closed models (Gemini-3-Pro) remain ahead. Seventeen languages is broad but not universal; scripts outside the corpus will need more data before the same guarantees hold. And reconstruction costs compute at pretraining time — a price worth paying once, so that everyone downstream inherits a stronger, smaller encoder for free. These are the honest edges of the current model, and they are also our roadmap.
Why I think this matters
For a decade, progress in document AI has mostly come from bigger language models bolted onto borrowed eyes. MonkeyOCRv2 is an argument that the eyes deserve their own research — that document-oriented visual pretraining can be a foundation in its own right, not an afterthought. A 0.1B encoder that reads well enough to carry a 0.6B model past 3B and 235B competitors is, to me, the more interesting kind of scaling. We are releasing the code, the weights, and MonkeyDoc v2 in full, because a foundation is only useful if others can build on it. If your document model is still running on a natural-image encoder, I hope this gives you a reason to reconsider.