It’s an understatement that Search has come a long way – fact that people use “Google” as a verb these days, says it all. Gone are those days when Search was keyword-driven, Search results were links to other websites, and users had to sift through a number of links before really finding what they were looking for. Now, Search engines allow users to post a free-form query. It then applies Natural Language Processing (NLP) algorithms, user context – location/profile and immediate previous searches to establish query context more powerfully and return targeted content – be it an image, video, Wikipedia content (noticed that box on the right hand side of google search results?) or a fact. It at times feels magical that Search engines know, with unbelievable accuracy, exactly what you are looking for. This is the result of a heavy investment in NLP and Semantic technologies. These, along with speech-recognition, have the potential of enabling a future where search will transform into a smart machine that uses “connected knowledge” to answer significantly complex questions – a Star Trek Computer may not be too far away after all, if Amit Singhal – brain behind Google’s search engine evolution, has be to believed.
Opportunities that these technologies present isn’t limited to general-purpose search engines but is significantly more widespread because of growing demand for enhanced customer experience, increased adoption of mobile devices, leveraging big data and growth in machine to machine (M2M) technologies. Markets&Markets – a leading premium markets researcher anticipates NLP market to grow to $13.4 billion by 2020 at a CAGR of 18.4%. Clearly, this presents solid opportunity for a software developer who is looking forward to building expertise in areas that will shape the future and will continue to command premium.
More specifically, listed below are the 5 key reasons developers should build NLP, Semantic search skills:
1. Critical in realizing potential of “Big, unstructured data”
As per Reuters, global data will grow to approximately 35 zettabytes in 2020 from its current levels of 8 zetabytes i.e. approximately 35% CAGR. Exponentially increasing digitization of customer interactions across verticals like retail, e-commerce, healthcare, telecom, financial services, is giving rise to such volumes of data, and organizations realize that monetizing such data is key to staying ahead of the competition. Majority of this data is text/audio/video and NLP techniques and Search capabilities are required for analyzing/sifting through this data to understand needs and expectations of the customer, generate insights for enhancing the customer experience and optimize the business processes.
2. Commoditization of data science
Another key development has been that the tools for predictive and prescriptive analytics have become more consumable. This combined with need for monetizing unstructured data has given huge surge to text analytics as is evidenced by the focus text mining, information retrieval topics receive in major conferences these days.
3. Widening gap between enterprise search platforms and general-purpose search engines
While search engines have evolved immensely, it is quite surprising that Enterprise Search platforms have continued to lag behind. Commercial platforms still do not go beyond the basics of keyword- search, tags, faceting/filtering. The gap is so wide that one cringes because of the ‘culture shock’ one gets switching from a general-purpose Search Engine to organization’s Search platform. Organizations across verticals feel the pain from this gap and this presents huge opportunity for NLP/Search practitioners.
4. Semantic Search will force marketers rehash their SEO strategies
As Semantic search technology aims at understanding intent/context of the user queries to surface more relevant content, it will both force and provide an opportunity to marketers. Structured markups will have to be added to the sites so that crawlers understand the context and content of the site, offerings better. Such will also benefit marketers significantly as conversion rates will improve considerably.
5. It’s just cool…and cutting edge
As humans continue to push boundaries on what machines could do for them, both ability to process natural language better, and ability to sift through huge knowledge bases will be critical in creating a slingshot effect. While we have come a long way indeed, we are still able to solve only a small percentage of NLP problems through smart application of Bag of Words and POS tagging techniques. There are plenty of areas including syntactic parsing, anaphoric resolutions, text summarization where we need to evolve considerably. That’s essentially why NLP and Search continue to attract significant research dollars. Going forward, innovative platforms will be those that are able to process language better and provide friendlier interaction mechanisms beyond a keyboard. Possibilities are immense be it intelligent answering machines, machine-to-machine communications or machines that can take action on behalf of humans. Internet itself will transform from connected pages to connected knowledge if you go by the vision of Tim Berners-Lee – the father of internet. Indeed, these are exciting times for NLP and Search.
The author is Director, Technology, Sapient Global Markets (India). Views expressed are personal .