The growing deployment of artificial intelligence (AI) systems in India is creating demand for a new layer of testing focused on how these systems perform in Indian languages and real-world conditions, giving rise to an emerging market for AI evaluation services.
Unlike conventional benchmarks that primarily compare the accuracy of AI models, companies are increasingly looking to test whether an AI system can reliably perform tasks for Indian users, including understanding regional accents, handling code-switching between languages and functioning in noisy environments.
The development has prompted companies to build evaluation platforms specifically around Indian use cases. AI4Bharat and Josh Talks AI recently launched Voice of India, an independent multimodal platform designed to evaluate AI systems across Indian languages and real-world conditions. Other players in the space include Krutrim, with its Indic BharatBench, Maxim AI, which develops agent evaluation and observability tools, and Athina AI, which has built an LLM evaluation and monitoring platform used by companies including Perplexity and Meesho.
“The biggest gap is that we are still evaluating complex AI systems with evaluation frameworks designed for simpler models,” Supriya Paul, co-founder, Josh Talks AI, said. She added that AI systems now need to be assessed based on how they perform an entire task rather than simply whether their final response is correct.
This is particularly relevant for voice-based applications being deployed in India. A banking AI agent, for instance, would need to understand regional accents, work in noisy environments, handle users switching between languages and retrieve the correct information before completing a transaction. Such systems need to be tested against these conditions before and after deployment.
The need for such testing is also increasing as AI systems move from generating responses to taking actions. Aryan Honawar, co-founder, Adzzat, which works with AI labs and enterprises on evaluating and deploying AI models, said evaluation now needs to examine whether an AI system took the right steps, recognised mistakes, knew when to stop and involved a human when required.
India presents a particularly complex evaluation environment because of its 22 scheduled languages, hundreds of dialects and widespread use of code-switching. Voice applications also have to account for regional accents and background noise, areas that are not necessarily captured by global benchmarks.
While public benchmarks such as MMLU and GPQA continue to be used to compare AI models, companies deploying these systems increasingly need to assess performance against the specific conditions in which they will operate. This includes factors such as accuracy, reliability, cost, latency and safety, rather than just a model’s benchmark score.
For Indian AI companies, this is creating an opportunity to build an independent evaluation layer between AI models and their enterprise deployment. As more companies move AI systems from pilots into production, the ability to test whether they work reliably for Indian users could become an important part of the country’s emerging AI ecosystem.