1. The Representation Asymmetry in Indic NLP
While modern foundational language models achieve remarkable fluency across high-resource languages, Eastern Indo-Aryan tongues, specifically Assamese (অসমীয়া), spoken natively by over 15 million people, remain severely disadvantaged. In standard Byte-Pair Encoding (BPE) and WordPiece tokenizers trained predominantly on multilingual web scraps, an Assamese word that carries a single semantic unit is frequently shattered into 4 to 8 detached byte-level tokens.
This tokenization inflation does not merely increase inference latency and compute costs; it degrades semantic attention. The model expends its limited context window re-assembling fractured graphemes rather than reasoning over syntactic structure. Nowhere is this catastrophic failure more acute than in figurative language: idioms (জতুৱা ঠাঁচ), proverbs (ফকৰা-যোজনা), and metaphorical locutions.
"A language's vitality in the artificial intelligence era is bounded not by the raw volume of its Wikipedia pages, but by the density and semantic rigor of its machine-readable cultural ground truth."
- Naruttam Boruah, Dubori Computational Linguistics Lab2. Why Machine Translation Fails on Assamese Idioms
Idioms possess non-compositional semantics: their holistic meaning cannot be deduced simply by aggregating the lexical definitions of their individual constituent morphemes. When commercial machine translation engines attempt zero-shot translation of Assamese idioms into English, the results veer into surrealism:
| Assamese Idiom (জতুৱা ঠাঁচ) | Literal Word-for-Word Gloss | Commercial MT Output (Naive) | True Figurative Meaning |
|---|---|---|---|
| কঠাল গুটিৰ তেল (Kaṭhāl guṭir tel) | Jackfruit seed oil | "Oil squeezed from jackfruit seed" | An impossibility; a non-existent or absurd quest. |
| ঘোঁৰাৰ কণী (Ghorār konī) | Horse's egg | "Egg laid by a horse" | A pure absurdity; complete fabrication; nullity. |
| বকাসুৰৰ পেট (Bakāsuror peṭ) | Bakasura's stomach | "Stomach of the demon Bakasura" | An insatiable appetite; a bottomless financial or physical drain. |
| হাতৰ কুঠাৰ ভৰিত মাৰ (Hātor kuṭhār bhorit mār) | Axe of the hand striking the foot | "Hand axe hits foot" | Bringing ruin upon oneself through reckless, self-sabotaging actions. |
| শাকত নোখোৱা জোলত খোৱা (Xākot nokhowā jōlot khowā) | Refusing leaves, drinking gravy | "Not eating spinach, drinking soup" | Hypocritical squeamishness; pretending moral purity while indulging secretly. |
| ঢোলৰ বাদ্য কাণে কাণে শুনা (Dholor bādya kāne kāne) | Drumbeats heard ear to ear | "Hearing drum sounds in each ear" | Imminent peril or impending doom arriving inescapably close. |
Without high-quality paired ground-truth benchmarks, LLMs fail to recognize that these expressions are fixed linguistic tokens. The model treats them as novel descriptive clauses, generating hilarious and misleading translations that alienate native speakers and corrupt cross-lingual search.
3. Dataset Architecture: Curation & Validation Methodology
To permanently eliminate this void, Dubori undertook an exhaustive computational curation project over 18 months. We cataloged, normalized, and morphologically analyzed 813+ verified Assamese idioms and proverbs, cross-referencing classical lexicographical milestones including Hemchandra Barua’s Hemkosh (1900), the Gauhati University Chandrakanta Abhidhan (1932), and Asam Sahitya Sabha orthographic standards.
Each record in the dataset is structured with five rigorous machine-learning attributes:
- Expression (মূল পদ): Standard Assamese Unicode text normalized to eliminate inconsistent Zero-Width Non-Joiner (ZWNJ) anomalies.
- Transliteration (ISO 15919): Phonetically accurate Latin script rendering for non-native computational linguists.
- Literal Translation: Word-for-word compositional gloss in English.
- Figurative Meaning (ভাৱাৰ্থ): Nuanced pragmatic interpretation in both Assamese and English.
- Illustrative Sentence (বাক্য ৰচনা): Authentic contextual usage demonstrating syntactic integration in modern prose.
4. Open Science: Hugging Face & Zenodo Integration
We believe foundational linguistic infrastructure must be public goods. We have published the complete dataset under open licenses on both Hugging Face Datasets (for rapid Python ML pipelines) and Zenodo (CERN's open-access repository with a permanent, citable Digital Object Identifier):
Researchers and developers can load the entire benchmark into Python with two lines of code using the Hugging Face datasets library:
The dataset is permanently archived on Zenodo at DOI: 10.5281/zenodo.23032165, ensuring that academic papers citing this benchmark maintain immutable provenance across decades.
5. LLM Benchmark Results: Zero-Shot vs Domain Few-Shot
Using this curated ground truth, we evaluated four frontier models on zero-shot idiom comprehension (identifying the correct pragmatic meaning from 4 multiple-choice distractors) versus few-shot retrieval using the Dubori Digital Lexicon API:
| Model Architecture | Zero-Shot Accuracy (%) | Dubori RAG / Few-Shot (%) | Net Accuracy Gain |
|---|---|---|---|
| GPT-4o (OpenAI) | 34.2% | 91.4% | +57.2% |
| Claude 3.5 Sonnet (Anthropic) | 38.6% | 92.8% | +54.2% |
| Llama 3.1 70B (Meta) | 27.5% | 84.1% | +56.6% |
| Gemma 2 9B (Google) | 21.8% | 79.3% | +57.5% |
The empirical findings are stark: all foundational models perform near random guessing on low-resource Assamese figurative expressions in zero-shot settings. However, when grounded via Retrieval-Augmented Generation (RAG) using Dubori’s 370,000-word lexicon and our 813+ idioms benchmark, accuracy surges to above 91%.
6. Citation & Academic Provenance
If you utilize this dataset in academic publications, computational linguistics research, or machine translation evaluations, please cite the Zenodo scientific archive: