In order to predict and design genomics, we need models that link local motifs and megabase scale regulatory context across multiple organisms. InstaDeep has developed a new foundation model, Nucleotide Converter v3, which is suited to multi-species genomics. The model, which runs in 1 Mb contexts with single nucleotide precision, unifies representational learning, functional tracking and genome annotation predictions, as well as controllable sequencing.
Nucleotide Transformer earlier models have shown that pretraining with thousands of genomes under self-supervision yields powerful features to predict molecular traits. Models from 50M-2.5B parameters were trained using 3,200 genomes, including 850 from other species. This sequence-only training idea is retained in NTv3, but it has been extended to larger contexts. It also includes explicit functional supervision, a generation mode and longer contexts.
Architectural Framework for Genomic Windows of 1 Mb
NTv3 is based on a U-Net type architecture, which targets genomic windows that are very long. Convolutional downsampling compresses input sequences, transformer stacks model long-range dependencies within that compressed space and deconvolution towers restore base resolution to predict and generate. Tokenization is done at character level, over A,T,C,G,N with tokens like , , , , ” . To enforce this requirement, the reference implementation applies padding. The vocabulary size for all public checkpoints is 11 tokens.
The public model NTv3 8M has 7.69M parameter values with hidden dimension of 256. FFN dimensions are 1,024. There are 2 transformer layers. 8 attention heads and 7 stages for downsampling. On the other end of the spectrum, NTv3650M has hidden dimensions 1,536 and 6,144 FFN, as well as 12 transformers layers, 24 Attention Heads, 7 Downsample Stages, and conditioning layers to add species-specific prediction heads.
Training data
The NTv3 is pretrained using language masking based on base resolution for 9 trillion base pair data from OpenGenome2. This stage is followed by a post training phase that integrates self-directed learning with supervised instruction on 16,000 annotation labels and functional tracks from 24 plant and animal species.
Performance Benchmark for Ntv3
After post-training, NTv3 achieves the highest accuracy in functional track predictions and genome annotations across all species. On public benchmarks it outperforms both strong sequence to functions models and prior genomic foundation models.
Ntv3 Benchmark is currently comprised of 106 tasks, which are long-range, single nucleotide cross assays, and cross species. NTv3 learns a common regulatory grammar by observing thousands of tracks in 24 species. This allows the model to transfer between organisms, assays and functions.
Prediction to sequence control
NTv3 is able to be tuned beyond prediction into a controllable generative engine via masked language diffusion modeling. The model is conditioned with desired levels of enhancer activation and promoter selection. It then uses masked DNA spans to fill in gaps that are consistent with these conditions.
Stark Lab, in collaboration with Stark Lab, validates 1,000 sequences of enhancers that have been designed with specific activity and promoter-specificity in vitro. Results show that the generated enhancers achieve more than two times better promoter-specificity when compared to baselines.
Comparison Table
| Measurement | Nucleotide Transformer (NTv3) | GENA-LM |
|---|---|---|
| Goals | Model for representational learning, sequence-to-function prediction and controlled sequence generation. | Transfer learning of long DNA sequences based on a family of language models. |
| Architecture | Post-trained versions include multi species conditioning, task specific heads, and a transformer stack in U-Net style. | BigBird models and variants of BERT-based encoding with 12 or 24-layers. |
| The Parameter Scale | The family spans parameters of 8M,100M and 650M | Large models, such as BigBirds, have 336M parameter values. |
| Native context length | Pre-trained and Post-trained models can both use inputs up to 1 Mb at single nucleotide level. | For BERT models, you can get up to 4500 BPE with just 512 tokens and for BigBirds models, it’s as high as 36000 BPE with 4096 tokens |
| Extended context mechanisms | The context is a single-base resolution, and the U-Net tower aggregates long range context prior to transformer layers. | The recurrent Memory Transformer and sparse attention of BigBird versions are used to increase the context of hundreds of thousands of bases pairs. |
| Tokenization | Tokenizer for A, T C G N special nucleotides | BPE tokenizer that maps DNA data into 4500 bp (512 tokens); two tokenizers used; one for T2T alone and another on 1000G plus Multispecies plus T2T. |
| Size of pretraining corpus | OpenGenome2: First-stage pre-training with approximately 9 trillion base pair from over 128000 species | Human only models trained on pre processed human T2T v2 plus 1000 Genomes SNPs, about 480 × 10^9 base pairs, multispecies models trained on combined human and multispecies data, about 1072 × 10^9 base pairs |
| Species coverage | OpenGenome2 has trained and supervised more than 128000 animal and plant species. | Models that are human-focused plus models specific to yeast, Arabidopsis, Drosophila as well as multispecies models from ENSEMBL |
| Post-training signals supervised | 16000 tracks, spanning 10 types of assays and 2700 tissues from 24 species. Used to label the backbone and train functional heads. | All tasks are fine tuned, such as Drosophila promoters (Drosophila enhancers), chromatin profiles or polyadenylation site, by adding task-specific heads to the LM. |
| Generative abilities | Can be fine tuned into a controllable generative model using masked diffusion language modeling, used to design 1000 promoter specific enhancers that achieved more than 2× increased specificity in STARR seq assays | The MLM model is used primarily as an extractor of features and a language mask. It supports the completion of sequences through MLM, but its main focus is on prediction tasks instead. |
What you need to know
- The NTv3 model is multi-species, long range genomic foundation.This architecture unifies representation-learning, functional track predictions, genome annotation and controllable sequencing in one U Net. It supports contexts of 1 Mb nucleotide for 24 species, including animal and plants.
- The model has been trained using 9 trillion base pairs, with both self-supervised and monitored objectives.Post-training is done on 16,000 annotations and functional tracks from 24 species. This objective combines self-supervision with supervised training.
- NTv3 shows state-of-the art performance in the Ntv3 BenchmarkAfter training, NTv3 reaches the state-of-the art for functional track predictions and genome annotations across species. It also outperforms the previous genomics foundation and sequence to function model on public benchmarks, as well as the Ntv3 Benchmark which includes 106 standardized downstream long range tasks, with inputs of 32 kb and outputs at base resolution.
- The backbone is the same for both controllable and enhancer designs that have been validated using STARR.NTv3 is a model that can be controlled using diffusion language masking to create enhancer sequences of specified levels of activity and selectivity for promoters. These designs are then validated with STARR-seq assays, which confirms the desired activity order and improves promoter specificity.
Click here to find out more Repo, Model on HF The following are some examples of how to get started: Technical details. Also, feel free to follow us on Twitter Join our Facebook group! 100k+ ML SubReddit Subscribe Now our Newsletter. Wait! What? now you can join us on telegram as well.
Asif Razzaq, CEO of Marktechpost Media Inc. is a visionary engineer and entrepreneur who is dedicated to using Artificial Intelligence (AI) for the greater good. Marktechpost is his latest venture, a media platform that focuses on Artificial Intelligence. It is known for providing in-depth news coverage about machine learning, deep learning, and other topics. The content is technically accurate and easy to understand by an audience of all backgrounds. This platform has over 2,000,000 monthly views which shows its popularity.

