Insight drought in data deluge

The metaphor of “data as the new oil” gained traction in the 2010s, emphasising that raw data, like crude oil, requires refinement to become valuable. Wired magazine once compared data to oil in the 18th century, suggesting that those who learn to extract and use it effectively will amass wealth. In India, the launch of Reliance Jio on September 5, 2016, symbolised this transition from physical oil to digital data.

But with rise of more refined artificial intelligence (AI), it is becoming clearer that the “data is oil” analogy has its limits. First, while oil is a finite, geographically concentrated resource, data is abundant, widely distributed, and constantly generated. Unlike oil, which is depleted upon use, data can be used repeatedly without loss, shared without being exhausted, and even combined with other data to create new insights. This unique characteristic makes data a non-rivalrous and self-propagating resource. The more it is analysed, the more it fuels further discoveries, creating a cycle of continuous value generation.

Merely possessing vast amounts of data does not automatically translate into valuable insights. In the past, collecting data was expensive and labour-intensive, requiring manual record-keeping and surveys. Today, storage costs have plummeted, computational power has surged, and businesses have embraced the belief that the combination of sophisticated algorithms and massive data sets guarantees success. This assumption, however, leads to the risk of data overload, where an excess of information, rather than enabling better decision-making, results in confusion, inefficiency, and flawed conclusions. I had written about this phenomenon some years ago, when I suggested that most data was dead-on-arrival, and the best thing may be to throw non-usable data sets out immediately rather than collect them ad infinitum simply because it is cheap to store vast amounts of data.

This challenge is encapsulated in the classic “garbage in, garbage out” problem. If businesses gather vast amounts of data without a clear strategy, they may find themselves drowning in irrelevant, redundant, or even misleading information. Even the most advanced algorithms cannot compensate for poor-quality data. The key is not the quantity of data but its relevance to the problem at hand. It is akin to asking whether one is digging a pit in the ground just to dig a pit or to build something meaningful on top of the pit — the purpose behind data collection is critical.

One way to understand the importance of relevant data is through the concept of information entropy. Entropy, in an informational context, measures the level of disorder or unpredictability in data. When disorder is high, it indicates a lack of clear patterns, making analysis difficult. A data set filled with irrelevant information contributes to this disorder, increasing noise rather than clarity. For example, if a business collects thousands of customer reviews but fails to filter out spam and off-topic comments, identifying genuine patterns in customer sentiment becomes significantly harder. Similarly, in predictive modelling, an excessive number of variables — many of them irrelevant — can make it more difficult to build accurate and reliable models.

The key, then, is for businesses to collect only the data that directly contributes to solving a problem rather than hoarding everything available. Consider digital platforms dealing with fraud detection. A common tactic among fraudsters is to use newly created phone numbers and email addresses to exploit loopholes. Instead of gathering an overwhelming amount of personal data when a seemingly new user (who is actually a fraudster) signs on, a simple linkage of just three data points — phone numbers, emails, and names — can be far more effective in identifying fraudulent behaviour. Of course, while a 360-degree customer profile may be useful for marketing, it is largely irrelevant when addressing fraud. In other words, when using data begin with the end in mind.

The opposite of beginning with the end in mind is a structured approach to data analysis often begins with a null hypothesis — the simplest possible explanation that assumes no effect or no difference until proven otherwise. This method prevents premature conclusions and forces decision-makers to rigorously validate their insights. The null hypothesis is sometimes seen as “boring” because it suggests that nothing extraordinary is happening, but it is essential for ensuring that conclusions are based on actual evidence rather than assumptions. This is especially true when people mistake correlation for causation — just because two data points move together for instance, it doesn’t mean that one causes the other to move. It is also important in removing “researcher bias”, where the person doing the analysis already has a hypothesis or message s/he wants to propagate, which means that the data collected or used will tend to skew toward affirming their pet hypothesis, while ignoring data that may prove the hypothesis wrong. This approach, too, allows you to build and use data sets as you need them, rather than simply throwing data at the problem.

From an investment perspective, data is an incredibly powerful asset, but it is not a magic wand. Many organisations today experience a paradox: they are drowning in data but starved of insights. This “insights drought in a data deluge” dilemma arises because simply amassing more data does not automatically lead to better understanding. The real value lies in the ability to extract meaningful insights and translate them into action.

Ultimately, success in a data-driven world is not about blindly following trends or amassing vast quantities of information. It is about understanding the fundamental characteristics of data, recognising its limitations, and using it strategically to drive better decisions to create tangible business value.

The writer is technology consultant and venture capitalist.

Disclaimer: Views expressed are personal and do not reflect the official position or policy of FinancialExpress.com. Reproducing this content without permission is prohibited.