Samanantar is the largest publicly available parallel corpora collection for Indic languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu. The corpus has 49.6M sentence pairs between English to Indian Languages.
Update 04-11-2021
Samanantar v0.3 along with LaBSE scores metadata is available for download. Go to Downloads
Dataset Format
The publicly released version is randomly shuffled, untokenized, and deduplicated.
Downloads
Benchmarks
The testsets used to benchmark IndicTrans can be found here
STS Benchmark
The Semantic Textual Similarity (STS) benchmark can be downloaded from here
En-Indic
The entire dataset can be downloaded from Samanantar v0.3.
The folder has 2 directories
- existing – all existing data compiled before samanantar
- created – mined as part of samanantar
- We have separate sub-dir for each source
Mirror Links
- Please use this mirror gdrive link to download the v0.3 data
- Please use this mirror gdrive link to download the benchmarks
Indic-Indic
The entire Indic-Indic data can be downloaded from here
Change Log
- [Will be updated soon] 06 July 2021, v0.2.1 data with metadata of source and Labse Alignment Score (LAS) was made available here
- 09 June 2021, The Semantic Textual Similarity (STS) benchmark is now available for download
- 05 June 2021, The benchmarking testsets are now available for download
- 15 May 2021, The language wise splits are now available for download
- 02 May 2021, Indic-Indic v0.2 data has been updated with super strict overlap removal
- 30 April 2021, v0.2 uses super strict overlap removal of validation and test data with train data
Contributors
- Gowtham Ramesh, (RBCDSAI, IITM)
- Sumanth Doddapaneni, (RBCDSAI, IITM)
- Aravinth Bheemaraj, (Tarento, EkStep)
- Mayank Jobanputra, (IITM)
- Raghavan AK, (AI4Bharat)
- Ajitesh Sharma, (Tarento, EkStep)
- Sujit Sahoo, (Tarento, EkStep)
- Harshita Diddee, (AI4Bharat)
- Mahalakshmi J, (AI4Bharat)
- Divyanshu Kakwani, (IITM, AI4Bharat)
- Navneet Kumar, (Tarento, EkStep)
- Aswin Pradeep, (Tarento, EkStep)
- Srihari Nagaraj, (Tarento, EkStep)
- Kumar Deepak, (Tarento, EkStep)
- Vivek Raghavan, (EkStep)
- Anoop Kunchukuttan, (Microsoft, AI4Bharat)
- Pratyush Kumar, (RBCDSAI, AI4Bharat, IITM)
- Mitesh Shantadevi Khapra, (RBCDSAI, AI4Bharat, IITM)
Citing
If you are using any of the resources, please cite the following article:
@misc{ramesh2021samanantar,
title={Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages},
author={Gowtham Ramesh and Sumanth Doddapaneni and Aravinth Bheemaraj and Mayank Jobanputra and Raghavan AK and Ajitesh Sharma and Sujit Sahoo and Harshita Diddee and Mahalakshmi J and Divyanshu Kakwani and Navneet Kumar and Aswin Pradeep and Srihari Nagaraj and Kumar Deepak and Vivek Raghavan and Anoop Kunchukuttan and Pratyush Kumar and Mitesh Shantadevi Khapra},
year={2021},
eprint={2104.05596},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
The bibtex entries for the existing data sources is available here
License
Samanantar is released under this licensing scheme:
- We do not own any of the text from which this data has been extracted.
- We license the actual packaging of this data under the Creative Commons CC0 license (“no rights reserved”).
- To the extent possible under law, AI4Bharat has waived all copyright and related or neighboring rights to Samanantar
- This work is published from: India.