NeuralBhasha / D2AR
co-authored a research benchmark comparing discrete diffusion and autoregressive language models for Hindi NLP at IIT Kanpur.
most major NLP benchmarks and open-source datasets are built entirely around english. if you want to evaluate how well a model handles hindi, the resources are extremely sparse and often just poorly machine-translated versions of english benchmarks, which completely ruins the grammar and nuances. pavitra and i co-authored D2AR (diffusion vs. autoregressive) at IIT Kanpur to change this. we built a rigorous benchmark specifically designed to test hindi NLP across 8 state-of-the-art models. our benchmark evaluates models across 4 core linguistic tasks: text classification, named entity recognition (NER), question answering, and text generation. we set up a 5-step pipeline that scores the models not just on basic accuracy, but also on syntactic coherence and semantic preservation in hindi. the research specifically focuses on comparing discrete diffusion language models (which generate tokens in parallel by denoising) against traditional autoregressive models (which generate tokens sequentially). building the tokenizers and clean training datasets for hindi was a massive challenge because devanagari script handles compound characters and ligatures in a way that standard tokenizers completely break. our paper outlines how these architectures perform when forced to compute hindi's morphosyllabic structure.