When the "cloud" falls to Earth, your company needs a backup plan

by Alan Zisman (c) 2011 First published in Business in Vancouver May 24-30, 2011 issue #1126 High Tech Office column

Last week’s column looked at how iPhones, iPads and Android phones were phoning in user location data. Since my deadline, Apple noted that the retention of location information in a file on its devices was a bug not a feature that would be fixed in a future update.
I suggested that despite the widespread media attention there was little cause for concern. The same week the “Locationgate” story broke, another tech story – potentially with more consequences – received far less attention.

Think of Amazon and you probably think of the company that dominates online book selling and now sells products ranging from appliances to watches. Less well-known: the company has built up its massive server pool to become one of the largest providers of data storage and virtual servers, with clients ranging from individuals through small and large businesses. In other words, when a company is storing data or running programs “in the cloud” often it's making use of Amazon Web Services (AWS).

Amazon offers a flexible range of services, where companies pay only for the storage space and bandwidth they use, both of which are scalable to instantly expand as a company’s needs grow. By using this sort of cloud, service companies can minimize the need to maintain their own network with an extensive IT staff.

Late in April, however, Amazon’s EC2 (elastic cloud compute) service crashed. A problem at a data centre in Virginia created multiple backups, which took up storage space and bandwidth. Problems cascaded across Amazon’s network, knocking cloud customers offline.
Affected businesses and web services were offline for periods ranging from several hours to several days, and Amazon reported that a small percentage of the customer data was unrecoverable.

Amazon’s network has five data centres, each with multiple “zones” providing backups and redundancies that are supposed to prevent this sort of crash. It didn’t work. (Unlike the data loss at Sony’s Playstation Network – almost simultaneous with Amazon’s EC2 crash – there’s no indication of foul play.)

Afterward, Amazon issued an apology and a detailed post mortem. It blamed a configuration error during a network upgrade. The company promised increased automation “to prevent this mistake from happening in the future” and issued credits to customers that had been affected by the crash.

Vancouver-based Bits Republic provides what company president Charles Chung refers to as “ultra-secure file collaboration in the cloud.” While an AWS customer, Bits Republic uses Amazon’s S3 service. Chung explained to me that the affected EC2 service provides clients virtual Windows or Linux servers, while S3 is a more straightforward remote storage offering. Hosted separately, the S3 service was unaffected.

Chung noted, however, that an outage such as EC2 clients experienced “can happen whether in the cloud or in self-hosted environments.” He advised, “instead of avoiding the cloud, businesses should prepare for such an outage and build disaster recovery scenarios to ensure business continuity.”

Amazon is taking steps to ensure that this problem won’t happen again, but there are other problems waiting in the wings – there’s no such thing as 100% reliability.

Local social media dashboard provider HootSuite is an EC2 customer and was knocked offline by the Amazon crash. CEO Ryan Holmes reported that his company was able to get back online more rapidly than many other affected businesses because it “had a good backup in place.” He noted that HootSuite is planning to “put even more redundancy in place to reduce downtime risk.”

Like Chung, Holmes doesn’t suggest companies move away from using cloud services, but offered these “words of wisdom”: “It’s amazing technology, but businesses need to plan for the worst case and you’ll be all right at the end of the day.”

That goes for individuals as well; if, like me, you rely on web services for email, contacts, calendar or document storage, make sure you’ve got a local backup. In fact, even if you don’t rely on the cloud, an up-to-date backup is a good thing to have.

Alan Zisman is a Vancouver educator, writer, and computer specialist. He can be contacted by e-mailing Alan