By: Peter Hopton, Iceotope

Walking around SC14 in New Orleans, one can’t help but notice the density claims. Figures range upwards from 20kW with a number of impressive sounding 100kW cabinets on display too. Density is important to the future of HPC in terms of reducing energy, cost and inefficiency in interconnect, not to mention ensuring that operators can install their growing HPC systems within their real estate constraints. When it comes to cabinet density, however, things aren’t always as they appear. For instance, you would have thought that a 100kW cabinet is going to take up 1/10th of the floor space of a 10kW cabinet, but unfortunately this is often where the metric falls over.

You see, the 100kW cabinet could actually be on its side, consuming 600mm x 2400mm of floor space. Maybe it’s not on its side, instead it’s a bit bigger, not a normal footprint of 800mm x 1200mm but 1200mm x 1200mm or 1500mm x 1500mm instead. Then there’s supporting equipment to consider, coolant distribution and heat exchangers for liquid cooling, ACDC power supply units, interconnect and CRAC units for the air portion (or air side losses) of the cooling.  I’ve even seen high density solutions with three cabinets of supporting equipment for a single cabinet of server nodes – and that cabinet was oversized to start with.

Overall power per square foot or square meter (within a typical data centre design) is perhaps a better metric, and one that users can directly relate to their real estate constraints and costs. It may sound like a pretty radical departure from current practices but it’s one that makes a lot of sense. It’s not ‘out of the box thinking’ at all really, as this headline would suggest, it’s simply acknowledging how big our boxes really are and how much space they take up. There’s no cheating the system here, with extra points are awarded to technologies that physically obsolete infrastructure, thus reducing footprint too. Infrastructure is all too often ignored, which is somewhat surprising when you consider that it often has a similar capital cost to the servers themselves (a typical data centre costs $8/watt plus real estate to build, including cooling, UPS and generators). Likewise, infrastructure can consume vast amounts of power too, with the Uptime Institute putting the global average PUE at 1.7 - meaning that for every 1 watt consumed by IT, 0.7 watts is consumed by infrastructure. We wouldn’t ignore these kinds of statistics in any other scenario. Imagine if a car’s air conditioning was responsible for 70 percent of fuel used on a typical journey? Consumers would revolt and heads would roll.   

The Green 500 also currently ignores infrastructure, meaning that it theoretically could be topped by machines that have large PUE’s of 7+ but gain IT efficiency through super-cooling electronics, with an overall power consumption that is very high.

Most “web giants” have been gaining efficiencies in their operations by uniting server and infrastructure teams, in order to look at how rack level architectures reduce their overall running costs. The same approach could be taken in the HPC community too and nothing is stopping us achieving similar results. The real question is, ‘isn’t it time that we do this?’