All Your Data Are Belong to Extremely Inefficient Server Farms

I have at least eight different emails, most of which I never use, but they’ve all got gigabytes of storage that’s slowly filling up all the time. For $25 a year, Flickr stores as many gigabytes of photographs I can produce. Meanwhile, for the small cost of crawling through my personal data, Facebook and Twitter manage and store all of the terrible jokes and blurry karaoke pictures I can handle. That’s not even to mention services like Dropbox, or even more mysterious ones like Gett, where I can upload massive files that disappear into the internet and are converted into a 10-character URL. With cloud-based data storage becoming cheaper by the month, soon we may not even need external hard drives for backups at home.

We all are producing more data than ever (by a very wide margin) but, increasingly, we don’t actually have to deal with managing and storing it all. Out of sight and out of mind, right? Well, no. All of the mountains of data we produce every day, and expect to have at a moment’s notice, end up in massive — massive — data centers around the country. And while plenty of people may think that the “cloud” is some sort of mysterious wireless storage or whatever — I mean, the internet comes out of the air, right? — the companies running server farms are hiding a dirty secret: Those farms are incredibly hungry for electricity, have been known to ignore what little regulations there are, and are becoming so expensive and unwieldy that the market is ripe for a major disruption.

According to a great New York Times report, experts estimate that 1.8 trillion gigabytes of data were created globally last year, three quarters of which was created by ordinary folks. And because consumers expect that their data will always be available, maintaining server operation is of utmost priority; if one company can’t keep the servers going 99.999% of the time, another one will. That’s led to a massive overbuilding of server capacity mixed with huge electrical backups, like battery arrays and diesel generators, that are wasteful on a massive scale. Part of the problem, according to the Times, is the lack of incentive to do things any other way.

In the early 1990s, Mr. Brill explained, software operating systems that would now be considered primitive crashed if they were asked to do too many things, or even if they were turned on and off. In response, computer technicians seldom ran more than one application on each server and kept the machines on around the clock, no matter how sporadically that application might be called upon. So as government energy watchdogs urged consumers to turn off computers when they were not being used, the prime directive at data centers became running computers at all cost.

A crash or a slowdown could end a career, said Michael Tresh, formerly a senior official at Viridity. A field born of cleverness and audacity is now ruled by something else: fear of failure.

"Data center operators live in fear of losing their jobs on a daily basis," Mr. Tresh said, "and that's because the business won't back them up if there's a failure." … David Cappuccio, a managing vice president and chief of research at Gartner, a technology research firm, said his own recent survey of a large sample of data centers found that typical utilizations ran from 7 percent to 12 percent. “That's how we've overprovisioned and run data centers for years," Mr. Cappuccio said. " 'Let's overbuild just in case we need it' — that level of comfort costs a lot of money. It costs a lot of energy."

Electricity and storage media are relatively cheap, and server farm managers prize minimizing down time over maximizing efficiency. But data production is increasing at a faster pace than it ever has. Server farms accounted for around two percent of U.S. electricity use in 2010, and that proportion is likely growing. (Server operators are notoriously opaque about releasing their own operational data.) At some point, companies are going to have to rethink their operations to make energy and server use more efficient.

At the same time, consumers have the greatest power to force change. How would your grandma feel if she knew all those cat pictures she uploads to Facebook means burning more coal? Some major firms like Intel are making bigger pushes to make green servers, but until the industry begins to change, I propose we start referring to the cloud as “the power-hungry smog.” Rolls off the tongue, doesn’t it?

Follow Derek Mead on Twitter: @derektmead.