AIHugging Faceabout 1 hour ago
NVIDIA Releases 2.8 Trillion Token Indic Corpus
11 min read
NVIDIA and IIT Madras launch Nemotron-pretraining-Indic-v1, a 2.8 trillion token dataset covering 22 Indian languages. The open pipeline uses NeMo Data Designer to translate English documents, retaining outputs scoring 75 out of 100. This resource aims to address data scarcity for Indian languages, supporting 1.2 billion citizens. Future Nemotron models will build on this expanded corpus.
Level
Hype check
Tap to vote and see what everyone thinks.
In this storyNvidia
