New Supercomputer Cuts Power Bill with Water, Intel Chips

Power consumption has long been identified as a major drag on boosting computing performance. A federal agency planning a new supercomputer in Colorado is leaning on both old and new techniques to address the issue.
The U.S. Department of Energy’s National Renewable Energy Laboratory selected Hewlett-Packard
HPQ -0.97% and Intel INTC -3.61% to provide a massive system for use in renewable energy research and related topics. Its $10 million machine will join an elite class with performance ranked at a petaflop–one quadrillion scientific calculations per second–and may attract even more attention for its claimed energy efficiency.
NREL, as the lab calls itself, said its goals for reducing power consumption would not allow the most common approach to cool such systems–circulating air through the racks of processors, which requires air conditioning units that are heavy users of electricity.
The lab specified that bidders to supply the system needed to come up with a design cooled by water, said Stephen Hammond, NREL’s computational science director. That’s far from a new idea; water is better at dissipating heat than air, he said, and liquid-cooled systems were commonplace in the early days of supercomputing.
Indeed, the legendary hardware designer Seymour Cray–whose Cray 2 system in the 1980s had circuit boards bathed in an inert liquid–once famously referred to himself as an “overpaid plumber” because of his attention to such issues.
The NREL design, Hammond said, sends water at a temperature of 75 degrees Fahrenheit into the computing racks to cool the system, a process that warms the water to about 95 degrees. It plans to use that water as the primary heat source in nearby lab and office space at a facility under construction in Golden, Colo.
NREL’s approach–though it doesn’t help with keeping the office space cool–means it doesn’t have to spend so much to lower the temperature in the data center with air conditioning, an approach that can make such rooms uncomfortably cool. And there’s no need for the many cooling fans inside such system, which can also make supercomputer centers fairly deafening.
Another unusual feature is the use of a new kind of Intel chip. The NREL system, besides many conventional Xeon processors, will also have 600 of a special-purpose cousin known as the Xeon Phi, each with more than 50 simple calculating engines on it. Those Xeon Phis are expected to get answers more quickly than conventional Xeons on mathematics-heavy problems, while expending less energy.
Such accelerator chips, as they are called, are a major trend in supercomputers. But many companies so far have been turning to entirely different classes of chips, including graphics processors, rather than chips like Intel’s that share a software lineage with those used in most PCs and servers.
Asked to compare the Xeon Phi with other kinds of accelerators, Hammond said he “wasn’t going to go there.” But he added that the energy-efficiency was extremely high, and the chips were relatively easy to program.
A common measure of energy-efficiency in data centers is “power energy effectiveness”–PUE for short–which indicates how much electricity goes toward actually delivering computation versus cooling or other overhead. An ideal rating is 1.0, where the average data centers rates 1.91, according to government statistics from 2009. The NREL data center has a target of 1.06 or better, which it says would rank as the world’s most energy-efficient computer room.
Alternatives to water for cooling also are being studied. Intel last week said it completed a year-long test of technology from Green Revolution Cooling that immerses systems in mineral oil to cool them, and achieved a PUE rating of 1.02 to 1.03 for the servers alone, said Mike Patterson, a senior power and thermal architect at Intel.