Data has become one of the most valuable commodities in the world. The development of AI and Internet of Things (IoT) connected devices is set to continue to drive the trend over coming years. With so much data being produced, collected and stored, developments around the latest technology in the world of data architecture have also become both crucial and big business.
The past decade has one that has seen the increasing centralisation of data through ‘cloud’ technology. The move away from localised hardware servers to flexible cloud data storage services has both reduced the cost of data storage as well as improving security by moving data away from exposure to one concentrated point of loss. Gone are the days when a broken down or stolen laptop or server should be able to cause chaos and the irretrievable loss of important data.
While cloud computing is not going anywhere, the landscape of data architecture technology is changing again. The rise of IoT devices from home assistants such as Amazon’s Alexa and Google Home to smart fridges, health monitors and, soon, driverless cars means the amount of data being generated and processed is rising exponentially.
Cloud computing was until recently being talked of as the data storage technology that would allow AI to scale. Now experts are starting to believe that centralised cloud storage is more of a bottleneck. The first issue is that connected devices will generate so much data cloud storage simply won’t have the capacity to hold it. There is also a time lag and cost. Until now the cloud has meant economies of scale but AI and IoT are expected to tip that into diseconomies of scale.
New start-ups are now focusing on ‘edge’ storage and data processing solutions to again decentralise data storage and processing where that is the most efficient approach. Xnor.ai is one example. The early-stage start-up develops machine learning algorithmic software that runs on the simplest, low-cost tech gadgets and recently raised $12 million in investment. The ‘edge’ refers to the connected devices at the fringes of the internet generating data. The technology Xnor.ai and other companies in the space are working on looks to solve the inefficiencies in cost, time and sometimes privacy that shipping data to the cloud to be processed and then shipping it back to devices as an ‘AI’ feed involves.
Driverless cars, for example, will process a huge amount of data, some of which the software needs to be able to react to immediately. Another company, Mesophere, has now raised over $200 million in investment to work on its solution for processing driverless car data from vehicles in a particular vicinity next to telecom ‘cell towers’.
This means it won’t have to be sent to a giant datacentre, potentially hundreds of miles away. However, some computer intensive data tasks, such as retraining machine learning algorithms, are best handled by the cloud. Mesophone is working on technology that will judge which data and which data processing tasks are most efficiently handled where – moving the data back and forward as most efficient.
It’s not only start-ups recognising this. Microsoft recently unveiled image recognition software able to run on local devices, such as cameras, without needing to be sent to its datacentres. Localised speech recognition and language-based sentiment analysis are Microsoft’s next targets for being moved back to the ‘edge’.
Like many development patterns, data’s trajectory now looks to be moving from a period of centralisation back to decentralisation.