🔍 Read the full analysis: From Data Generation To Pretraining: Scaling Indic AI on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
NVIDIA and AI4Bharat at IIT Madras say they plan to publish Nemotron-pretraining-Indic-v1, a corpus with about 2.8 trillion retained tokens across 22 Indian languages, plus the GPU-accelerated workflow used to generate it. The release date, licensing terms and detailed quality results have not been specified, and the project reports no independent evidence yet of improved model performance.
NVIDIA and AI4Bharat at IIT Madras say they plan to release Nemotron-pretraining-Indic-v1, a corpus containing about 2.8 trillion retained tokens across 22 Indian languages, along with the GPU-accelerated pipeline used to produce it. The resource is aimed at model developers who need more material for pretraining in Indic languages; the announcement does not say when the corpus or pipeline will be available.
The reported inventory contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. Native-script material is included for all 22 languages, while romanized text—Indic-language content written in Latin letters—is included for 12. The project describes these as 34 language-script configurations. It cautions that records may include translations and script variants of shared source documents, so the total should not be read as a count of unique documents.
The workflow begins with English documents from 15 Nemotron-CC-v2 subsets, including web text, question-and-answer material, mathematics and code. Using NVIDIA’s NeMo Data Designer framework, it translates documents, scores translation quality, repairs outputs that receive low scores, and converts some accepted translations into Latin script. The announcement says records retain source identifiers and that outputs are checked against inputs, with checks intended to preserve elements such as equations, tables and code.
The teams say developers will be able to use the corpus directly or adapt the prompts, workflows and quality checks to their own source material. They also report that continued pretraining for Nemotron 3.5 Super VL used 10 billion Indic tokens from a 12-language subset of the corpus alongside naturally authored Indic text. That training figure is separate from the full corpus inventory: it is not evidence that the model trained on all 2.8 trillion retained tokens.
A Larger Data Source for Indic Models
A large corpus could give teams a broader starting point for training models in languages that have had less available digital material than English. The source mix includes technical topics, code and question-and-answer content, which may be useful for models designed to respond to questions or explain technical subjects in Indic languages. These are potential uses, not demonstrated results: the announcement supplies no independent performance evaluation showing how much the corpus improves a model.
Publishing the generation workflow could also let researchers examine or adapt the process, rather than relying only on a finished collection of text. Source identifiers and input-output checks may support traceability, while the ability to use other source documents could make the process reusable. The value of those features will depend on the released code, documentation, licenses and quality measurements.
The corpus is based on translations from English, not a substitute for naturally authored Indic writing. Translation can carry over the source material’s subject choices and framing, and may not capture the full range of local knowledge and cultural context. The project says naturally authored Indic data remains essential, a limitation relevant to anyone building training mixtures from the new resource.
As an affiliate, we earn on qualifying purchases.
From Sangraha to Synthetic Data
AI4Bharat, an Indian-language research lab at IIT Madras, develops open datasets and models. The announcement cites Sangraha, its open pretraining corpus for 22 Indian languages, as part of the foundation for open Indic-language model research. Nemotron-pretraining-Indic-v1 would add translated material and a documented generation workflow to that work.
The project says its 22 languages are those listed in India’s Constitution and describes them as the first languages of 97% of Indians, or nearly 1.2 billion people, citing the 2011 Census. That population figure refers to the cited census, not a current estimate. The project also says a 12-language subset contributed 10 billion Indic tokens to continued pretraining for Nemotron 3.5 Super VL, alongside naturally authored text.
Those figures describe different things: the 2.8 trillion figure is a retained corpus count using the Gemma-4 tokenizer, while 10 billion is the amount the announcement says was used in a particular training effort. Neither number establishes the quantity of unique source documents or, by itself, the effect on model quality.
“We do not have enough data for Indian languages.”
— Mitesh Khapra, head of AI4Bharat
large language model training corpus
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Release Terms and Quality Still Pending
The announcement describes the release as planned but does not provide a publication date or hosting location. It also does not specify the precise licensing terms for use, redistribution or adaptation, or give a complete account of the filtering rules and acceptance thresholds used to retain translated text.
Detailed quality scores, per-language token counts and independent assessments of translation fidelity are also absent from the source material. The project says it evaluates translations and repairs low-scoring outputs, but the reported information does not show how those scores are calibrated or how well the process performs across different languages and subjects. No comparative results establish whether models trained with the data perform better.
It is also not clear how future users should balance this material with naturally authored Indic text. The reported total includes accepted rewrites and script variants, and the announcement cautions that records may be translations or variants of shared source documents. The corpus count therefore should not be interpreted as an equal volume of distinct, locally authored material.
GPU-accelerated data pipeline tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Publication and Independent Review
The next milestone is publication of the corpus and its generation pipeline. Once released, developers and researchers can examine the data, documentation and code, and assess the terms for adapting or redistributing the materials. The project’s plans, as described in the announcement, do not include a specific release date.
Researchers will also be able to evaluate translation quality, language-by-language coverage and the preservation of structured content such as equations and code. Comparisons of models trained with different data mixtures could help establish whether the corpus contributes measurable gains, and how it works alongside naturally authored Indic text. Until those materials and evaluations are available, the reported scale and workflow are project claims rather than independently assessed evidence of downstream performance.
multilingual text translation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is Nemotron-pretraining-Indic-v1?
It is a planned corpus from NVIDIA and AI4Bharat at IIT Madras, reported to contain about 2.8 trillion retained tokens across 22 Indian languages. The teams also plan to release the workflow used to generate the data.
How is the corpus being generated?
The described pipeline starts with English documents from 15 Nemotron-CC-v2 subsets, translates them using NVIDIA’s NeMo Data Designer framework, evaluates and repairs some translations, and converts a portion into Latin-script romanizations.
When will the corpus be available?
No release date has been specified in the announcement. The corpus and pipeline are described as planned releases, not as resources already available.
Does the 2.8 trillion-token count mean 2.8 trillion unique documents?
No. The figure is a token count using the Gemma-4 tokenizer. The announcement says the records can include translations and script variants of shared source documents, so the total is not a count of unique documents.
Has the corpus been shown to improve model performance?
The announcement reports use of 10 billion Indic tokens from a 12-language subset in continued pretraining for Nemotron 3.5 Super VL, alongside naturally authored Indic text. It gives no independent performance comparison establishing the corpus’s effect on model quality.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
