A study from Domo estimates that by 2020, every person on earth will create 1.7MB of data every second.
By Deepak Visweswaraiah
We are in the midst of an explosion of data. Today, the world is creating more data than ever before in history. As per the Digital 2019 report by HootSuite, the average internet user is online for an average of 6 hours and 42 minutes each day. A study from Domo estimates that by 2020, every person on earth will create 1.7MB of data every second. As the volume of data multiplies, it also throws up the need for far more sophisticated ways of managing data.
A large part of this data is being created outside of the traditional data center. Therefore, there needs to be a data pipeline from the edge to core to cloud that would enable data scientists to analyse it, apply new algorithms, and draw more value from it. New emerging technologies such as AI can play a significant role in bringing data management up to speed for the future. Here are five top ways in which AI induces data storage to transform:
Scale and efficiency
It is imperative that the data pipelines scale tremendously to accommodate large volumes of data that could reach even exabytes in some instances. AI can play a key role in facilitating this change by enabling this scale. In addition, AI-based robotics could help take care of repairs and maintenance of the hardware.
When the system needs to process several petabytes of data, the input/output (I/O) devices need to be able to transmit large amounts of data. AI can help advance compute and I/O performance considerably and ensure that it grows constantly. AI-led automation can help put in place a self-service model to obtain more storage as required to boost performance even more. AI can enable a self-healing environment where the software could potentially write/ rewrite itself to prevent breakdowns and ensure maximum uptime.
When you are deciding on your preferred AI software framework, it is also important to ensure that your existing infrastructure is well-equipped to meet new requirements. The right architecture should not only house your new AI software, but it must have the ability to scale seamlessly from the pilot stage all the way to production.
Edge to core to cloud
Given that data resides in different locations, transporting all of it to the core or cloud for processing may no longer be feasible. This is especially true as the volume of data grows and as sensors at the end points—like driverless cars for example, allow for decisions to be made at end points. AI can help collect and process data in offline environments, which can be seamlessly moved to the on-premise data centre or the cloud.
Advanced support to data sources
As the diversity of data sources grow, the storage infrastructure needs to contend with a wide variety of workloads. For instance, one storage system may need to deal with workloads such as SAP, Oracle, Hadoop, DB2, MongoDB and unstructured data. AI offers the capability to train the system on a broad set of enterprise data sources.
While it might be technically possible for enterprises to manage their growing storage needs even without AI, speed and scale will be severely lacking. For example, as a retailer, you may still be able to get high quality insights from your sales data on which colours or styles are most popular with customers. Without AI, however, you will get the insights two weeks later; thereby severely limiting your ability to maximise your sales based on the insight. With the right tools, AI has the potential to transform data management for the future such that the enterprise is well-equipped to handle the rapidly growing demands thrown by the digital revolution.
The writer is senior VP & MD, NetApp India