The majority of visual document retrievers on the market today are second-hand. ColPali The models which followed this model repurposed a vision-language generative model as an encoder. It still has a vision tower that is separately pretrained and a decoder with a causality, but it never produces a token. This is a lot of parameter overhead and computation for a job that requires only representations.
H Company Has been released NeoMMEThe family is made up of bidirectional encoders with 260M or 800M that do not require both components. One Transformer processes multilingual text tokens and raw 32×32 RGB image patches through the same layers, trained from random initialization. NeoMME Retriever is able to reach 0.523 nDCG@10 in the retrieval fine tune. ViDoRe v3 At 260M parameter.
Is it deployable? Yes. Apache 2.0 is the default version of every checkpoint. Hugging Face Transformers. The 260M models index 51.3 pages in seconds on one NVIDIA GPU L40S, and encode a query within 78.3 milliseconds when using only CPUs.
Two towers, one tower
The text is entered using an ALBERT factorized embedding. This involves a lookup of 256 dimensions projected onto the model width. Images are split into non-overlapping 32×32 patches and projected by a 2-layer MLP trained from scratch. There is no patch-merging modules, and there are also no SigLIP2 tower.
Both models support a 16,384-token context, enough for two standard 3,840×2,160 4K UHD images after patching. All layers pay global attention. This stack utilizes grouped-query-attention, query-key-normalization, gated-attention, 2D-rotary-position embeddings and squared ReLU MLPs. The exact parameter counts are 793,715,032 and 262,937906
The tokenizer is a whitespace-unconstrained BPE with a 131,072-entry vocabulary, trained from scratch. In the FLORES 200 devtest it produced 44.4% less tokens across 14 target languages than ModernBERT.
The training as a diffusion denoiser masked
The pretraining is a discrete diffusion masking the text. It can be conditioned by visible patches. The corruption rates for text-only segments are uniformly ranging from 0 to 1 Multimodal segments have a corruption rate of 0.30 to 1 which forces the model into reading the page.
Cross-modal ablation is shown to work. Visible page patches increase masked token accuracy for both the 260M and 800M models by up to 38.4 and 40.5% points respectively at 90% masking. On each H100 accelerator, the run process approximately 524 trillion packed input tokens. Approximately 290 billion are text only.
Search results
NeoMME Retriever has added two joint trained heads to the backbone. A dense mean-pooled head that projects each token or patch in all 128 dimensions. One forward pass returns both.
ViDoRe 3 scores the 260M nDCG@10 model at 0.523% and the 800-M model at 0.556. The result for the 260M is within 0.002. ColQwen2.5-v0.2 At 3.75B, the 800M model is 26.1 points higher than any other sub-300M. This 800M model is 0.9 points less than the similar sized Vultron Retriever Flash. The models on ViDoRe V1 and V2 reach 0.861/0.522 and 0.873/0.559 nDCG@5.
The text retrieval algorithm is less effective. Late interaction on BEIR-15 reaches 0.4881 – 0.5126, compared to 0.5722. LateOn The authors attribute this partly to the supervision scale: NeoMME saw roughly 430K text query examples, compared with approximately 660M contrastive examples for mLateOn. NeoMME only saw 430K query text examples, while mLateOn had 660M.
Storage and Throughput
The cost of late-interaction indexes is high. A 2048×2048 page yields 4,162 vectors, about 1.5 MB per ViDoRe v3 document in float32. There are two ways to reduce that. Hierarchical token pooling at factor 10 with int8 queries and documents gives 39.0 kB per page, a 39.4× reduction retaining 99.16% of baseline nDCG@10. Pool factor 8 with int8 queries and binary documents gives 6.0 kB, a 255.5× reduction retaining 95.19%.
The indexing process is quick for vector counts. At a matched 2048×2048 input on one L40S, NeoMME-260M encodes 51.3 pages per second against ColModernVBERT’s 26.0, a 1.97× gap.
Interactive explainer
The Key Takeaways
- A bidirectional Transformer is all you need to handle text and image patches. No vision tower or decoder required.
- NeoMME-Retriever-260M scores 0.523 nDCG@10 on ViDoRe v3, beating every evaluated model below 800M.
- It matches 3.75B-parameter ColQwen2.5 on ViDoRe v3 while being 14.4× smaller.
- By combining token pooling with asymmetric quantization, the index is reduced by approximately 1.5MB to only 6kB per sheet.
- The weakest points are the text-only retrieval of images and natural image transfer frozen.
Click here to find out more Paper, Model Collection You can also find out more about the following: Demo. Also, feel free to follow us on Twitter Don’t forget about our 150k+ML SubReddit Subscribe Now our Newsletter. Wait! What? now you can join us on telegram as well.
Want to promote your GitHub repo, Hugging Face page, Product release or Webinar?? Connect with us
Asif Razzaq serves as the CEO at Marktechpost Media Inc. As an entrepreneur, Asif has a passion for harnessing Artificial Intelligence to benefit society. Marktechpost was his most recent venture. This platform, which focuses on machine learning and deep-learning news, is both technical and understandable to a broad audience. Over 2 million views per month are a testament to the platform’s popularity.
