Most of global email and calendaring data is currently stored ‘in the cloud’. Other applications are following the trend, with 33% of office applications to be cloud-based by 2017.
A survey conducted at the AWS re:Invent conference shows that 69% of respondents are deploying business-critical applications into an IaaS cloud.
However, the cloud is not reliable or secure enough for such a shift. The same survey reveals that the top four concerns around cloud deployment are reliability (90%) performance (88%), security (86%), and costs (84%).
Could outages are growing in frequency. In 2013, outages affected most major cloud services, including cloud storage (Dropbox, Apple, Amazon, Microsoft, CloudFare) and email (Yahoo, Gmail).
To ensure a level of reliability, service providers usually replicate user data across multiple cloud locations (data centers). In the case of cloud failure or disconnection, requests are fulfilled through connections to mirror storage facilities. The duplication of both storage and connection are crucial for reliability. The Gmail failure of September 2013, for example, was reportedly due to “redundant network paths” failing “at the same time”.
Replication increases storage and energy costs significantly. Moreover, the existence of copies at multiple remote locations reduces data security, and further drives costs, as each copy needs to be equally secure. Excessive replication and mirroring may also have an adverse effect on reliability by causing storage and communication overloads, hence increasing outage events.
What if an operator were to distribute a large number of file copies to different storage locations, where none of the copies represents the complete original file? Oddly enough, this method has been proven to deliver data to a given location more rapidly. An experiment recently conducted at Aalborg University (Denmark) shows that storing less than 65% of a 32-packet file in five commercial clouds yields similar reconstruction delays as storing the while file in each cloud. Furthermore, storing partial copies is more secure.
But how to manage the transmission of file fragments from multiple clouds?
The increase in global data outages shows the complexity inherent to managing high levels of replication, as signaling is used to avoid the transfer of identical packets from different clouds. This underscores the need ‘smarter data or smarter storage technology’, particularly in an increasingly dynamic storage environment.
By removing state distinctions between packets of the same file, our technology replaces duplicate files with smart data. This guarantees that coded packets arriving from all clouds contribute to the reconstruction of the original file. In the 5-cloud example cited above, our algorithms yield a 35% speedup of average file reconstruction times.
Moreover, the same technology unleashes the full potential of mesh networking and peer-to-peer communications.