The amount of data in our world is exploding. From two characters, length of the first message in 1969, to two billion internet users globally in 2011, they produce data at an unprecedented rate?294 billion emails, 69,120 hours of YouTube videos, 200 million tweets, 5 billion text messages, and so on. If these statistics aren?t shocking enough, here are some more:
A four engine jumbo jet can create 640 terabytes of data in just one crossing of the Atlantic Ocean. If you multiply that by 25,000-plus flights a day around the world, you get a sense of the amount of data that is involved in ensuring flight safety. And, even at 140 characters per tweet, the high velocity (or frequency) of Twitter data ensures large volumes (over 8 TB per day). Technology research firm IDG reveals that humans created 150 exabytes of data in 2005, and that number grew eight times to 1,200 exabytes by 2010.
Businesses are no different. Enterprises are awash with data, easily amassing terabytes and even petabytes of information. Typically, a large retailer scans one million bar codes everyday. Researchers estimate that enterprise data grows nearly 60% per year?90% of that being unstructured?and the average amount of data stored per company is 200 terabytes. This growth is triggered by the increasing channels of data in today?s world. Examples include, but are not limited to, user-generated content through social media, Web and software logs, cameras and intelligent sensors embedded in physical devices that can sense, create, and communicate data.
Many business executives want more information than ever, even though they are already drowning in it. Most companies are today talking about managing humongous amounts of data. Companies in sectors like banking, financial services and insurance (BFSI), telecom and retail particularly deal with massive amounts of information that will be used to evaluate and plan business strategy.
The moot point is that in order to be able to make quick informed decisions, data needs to reside in such a manner that it can be used seamlessly by different teams?marketing, finance, product development and sales for example and not create a situation where each team has to reinvent the wheel. Industrialisation of data has become a necessity today and CIOs across the board have recognised this need and are exploring ways of making it available across the organisation.
However, several obstacles limit their ability to turn this massive amount of unstructured data, often termed as big data, into profit. The most prominent obstacle being a lack of understanding on how to add big data capabilities to the overall information architecture to build an all pervasive big data architecture.
Companies are realising that now is the time to put this data to work, says Mitesh Agarwal, CTO & director?Systems, Oracle India. ?Companies have to understand that planning big data architecture is not about understanding just what is different. It is also about understanding how to integrate what is new when compared to what is already in place.?
Soumendra Mohanty, global lead? information management services, Accenture, says, ?CIOs have increasingly realised the importance of decoupling data from applications to deliver greater flexibility, but many are finding that a further step is being demanded?freeing data to be easily moved, shared, analysed, or integrated. The demand is already there, and is growing rapidly as firms realise the value of data, but CIOs need to cater for this by industrialising the sharing of data. In turn, this will help firms extract new value from their data.?
Mohanty insists that data management needs to shift from being an IT capability buried within application support, to being a collaborative effort that enables data to be used far beyond the original application it was intended for. Doing so is now possible because of advances in the technologies used to manage, process, and store data. ?Many of these have been developed by Web pioneers such as Amazon, Facebook, Yahoo, and Netflix as solutions for their own data-management challenges. Amazon chief executive Jeff Bezos has noted that his company?s advances in data management helped provide the necessary architectures to develop its cloud storage and data management services?and to be flexible enough to respond rapidly to new ideas,? he adds.
Are there ways in which using big data can create value? Certainly, says the Oracle CTO. First, align big data initiative with specific business goals. One of the key characteristics of big data is value?value through low-density and high volumes of data. As we sort through the mountains of low-value-density big data and look for the gold nugget, do not lose sight of why we are doing this. Follow an enterprise architecture approach. Focus on the value it provides to the business. How does it support and enable the business objectives? Properly align and prioritise big data implementation with the business drivers. This is critical to ensure sponsorship and funding for the long run.
Second, ensure a centralised IT strategy for standards and governance. Some of the recent analysts? surveys indicate that one of the biggest obstacles for big data is skills shortage. A 60% skills shortfall is predicted by 2018.
Addressing such a challenge through IT governance to increase the skill level, to select and enforce standards, and to reduce the overall risks and training cost would be an ideal situation.
Third, use a centre of excellence to minimise training and risk. The Oracle CTO advocates establishing a centre of excellence to share solution knowledge, plan artifacts and ensure oversight for projects can help minimise mistakes. Fourth, enterprises should establish new capabilities constantly and leverage their prior investments in infrastructure, platform, business intelligence and data warehouse, rather than throwing them away. Investing in integration capabilities can enable the knowledge workers to correlate different types and sources of data, to make associations, and to make meaningful discoveries.
Closer to reality, retailers analyse basket data to offer the right promotions based on a particular customer?s preferences. McKinsey & Company estimates that retailers with the right processes in place to properly utilise massive amounts of data can benefit from a potential 60% increase in operating margins and a 0.5% annual productivity boost.
The bottom line is that companies would do well to harness the data deluge ahead of the rising tide.