The idea of big data goes something like this: In a world of ever-increasing digital connectivity, ever larger mountains of data are produced by our cellphones, computers, digital cameras, RFID readers, smart meters and GPS devices. The huge quantity of data becomes unwieldy and difficult for companies and governments to manage and understand.
?My smartphone produces a huge amount of data, my car produces ridiculous amounts of really valuable data, my house is throwing off data, everything is making data,? said Erik Swan, 47, co-founder of Splunk, a San Francisco-based start-up whose software indexes vast quantities of machine-generated data into searchable links. Companies search those links, as one searches Google, to analyse customer behaviour in real time.
Splunk is among a crop of enterprise software start-up companies that analyse big data and are establishing themselves in territory long controlled by giant business-technology vendors like Oracle and IBM.
Founded in 2004, before the term ?big data? had worked its way into the vocabulary of Silicon Valley, Splunk now has some 3,200 customers in more than 75 countries, including more than half the Fortune 100 companies.
Customers include the online gaming company Zynga, the maker of FarmVille and Mafia Wars, which uses the software monitor game function to determine where players get stuck or quit playing, allowing Zynga to tweak games in real time to retain players.
Macy?s uses Splunk?s software to observe its Web traffic in order to avoid costly down times, particularly during peak holiday shopping. Edmunds, an automotive research Website, started using Splunk to troubleshoot its information technology infrastructure and now uses the software to analyse all its customers? online actions. Hundreds of government agencies use Splunk to monitor suspicious activity on secure sites, and a Japanese tsunami relief organisation used it to track aid and monitor road and weather conditions. The amount of data being generated globally increases by 40% a year, according to the McKinsey Global Institute, the consulting firm?s research arm. And while Splunk has a lead in selling software to analyse machine data, big data is big enough to create new opportunities for a multitude of start-ups, many of them using the open-source software Hadoop.