Print 30 comment(s) - last by dark matter.. on Jun 21 at 8:09 AM

IBM spearheads novel heating design, develops and tests it with the LRZ -- a German supercomputing center

International Business Machines, Inc. (IBM) is in hot water with its latest supercomputer design, and that's a good thing.  The company announced this week the availability of the world's first hot water commercial supercomputer.

I. Building a Better Cooled Supercomputer

Dubbed LRZ "SuperMUC", the system is composed of IBM System x iDataPlex Direct Water Cooled dx360 M4 servers.  It can pack up to 150,000 cores (in 18,000 Intel Corp. (INTC) Xeon processors) for up to 3 petaflops of execution at a time.  Describes IBM:

[This] is equivalent to the work of more than 110,000 personal computers. Put another way, three billion people using a pocket calculator would have to perform one million operations per second each to reach equivalent SuperMUC performance.

At the same time, by ditching traditional air cooling for a liquid coolant power costs to be cut by up to 40 percent over a traditional air design.  The water is heated to a hotter than normal temperature via special microchannels in the cooling blocks, hence the "hot water" name.  The heated fluid gets up to a toasty 113 degrees Fahrenheit, or 45 degrees Celsius, cutting the power consumed by cooling to around a fifth of the levels used in traditional designs.

Of course many tech giants like Facebook, Inc. (FB) and Google Inc. (GOOGuse liquid-cooling at their data centers, but even this somewhat more efficient technology is reportedly inferior to the new hot-water cooling system in power efficiency.

The system is also more compact than traditional air or liquid cooled designs, which require bulkier blowers, piping, and/or heat transfer systems.

II. Trial Deployment in Germany Shows Superb Results

Even greater savings can be realized by repurposing the waste heat from the supercomputer to heat on-site research institutions in the winter.  This approach was tested at the Leibniz Supercomputing Centre (Leibniz-Rechenzentrum -- LRZ) in Garching near Munich, Germany.

The LRZ, who helped develop the commercial supercomputer technology, not only was able to realize 40 percent power savings, allowing it to fulfill the German government's power efficiency mandates for research institutions, it also saved $1.25M USD on heating costs at the LRZ, via the attached waste-heat recycling system.

Dr. Bruno Michel, manager, Advanced Thermal Packaging, IBM Research cheers, "As we continue to deliver on our long-term vision of a zero emission data center we may eventually achieve a million fold reduction in the size of SuperMUC, so that it can be reduced to the size of a desktop computer with a much higher efficiency than today."

LRZ indoors
The LRZ saves money by using waste heat from the hot-water cooling system to heat workspaces during the winter.  [Image Source: Steve Lionel/Flickr]

Completed in July 2012, the new LRZ supercomputer is the most powerful one in Europe, and is a member of the Partnership for Advanced Computing in Europe (PRACE).  IBM describes its work, writing:

This performance will be used to drive a wide spectrum of research -- from simulating the blood flow behind an artificial heart valve, to devise quieter airplanes to unearthing new insights in geophysics, including the understanding of earthquakes. The SuperMUC system is also connected to powerful visualization systems, including a large 4K stereoscopic power wall and a five-sided immersive artificial virtual-reality environment or CAVE for visualizing 3D data sets from fields, including Earth science, astronomy and medicine.

Munich and nearby German cities deeply rely on the Centre's computing resources for their studies.  Now they'll be able to do it more affordably, and set an example via a prototype of a design that will likely pop up elsewhere around the world before long, courtesy of IBM.

Source: IBM

Comments     Threshold

This article is over a month old, voting and posting comments is disabled

Not really hot water cooled?
By ElementZero on 6/19/2012 3:03:38 PM , Rating: 5
Seems to me like it still takes cold water, it's just that it is not dumping the hot water after that but instead using it to heat things that would otherwise need their own heating system (or am I reading that wrong?) If that's the case - calling it a "hot water cooled" system is kinda misleading. I thought they had figured out some new way to cool down the processor by using hot water by looking at the title.

RE: Not really hot water cooled?
By JasonMick (blog) on 6/19/2012 3:13:30 PM , Rating: 3
Seems to me like it still takes cold water, it's just that it is not dumping the hot water after that but instead using it to heat things that would otherwise need their own heating system (or am I reading that wrong?) If that's the case - calling it a "hot water cooled" system is kinda misleading. I thought they had figured out some new way to cool down the processor by using hot water by looking at the title.
Well that's what IBM's scientists call it.

I believe the heated liquid is actually significantly hotter than in a standard large-channel block-based design.

If you want to get down to the technical details, part of what's different/how the fluid is heated to a higher temp is the microchannel coolers built into the processors, which allows for the fluid to get closer to the processor and absorb more heat (smaller channels == more surface area).

Hence the secret sauce is likely IBM's proprietary microchannel coolers, which were likely the subject of long and laborious work to optimize path efficiency and maximize cooling.

The circulating "hot" liquid temperature -- 113 degrees Fahrenheit, or 45 degrees Celsius -- is quite hot for a system of this scale, I believe, if I'm not mistaken.

RE: Not really hot water cooled?
By drycrust3 on 6/19/2012 3:58:12 PM , Rating: 2
There were two thoughts in my mind when I read this.
The first was the running at a higher temperature will affect the Mean Time Between Failures of all the electronic components, not just the CPU, meaning you could have reduced reliability. My recollection was that most electronics assumes an average air temperature of around 20 deg C. By running at 45 deg this means higher wattage ratted components should be used than for the equivalent 20 deg C design. A worst case scenario is that every printed circuit board would have to be checked to see if it can work in this environment.
The second thought was "Do you need to employ a plumber when installing new equipment?"

RE: Not really hot water cooled?
By chmilz on 6/19/2012 6:38:33 PM , Rating: 2
One would assume that the ambient air temp would still be around 20C. The piping would definitely be fully insulated to prevent heat leak in the server room, thus transferring the heat not only off the servers, but out of the room.

RE: Not really hot water cooled?
By FITCamaro on 6/20/2012 8:30:55 AM , Rating: 2
45C is still pretty cool for a CPU though. But yeah overall its hot for a server room. Assuming no other cooling in the room.

RE: Not really hot water cooled?
By Arsynic on 6/20/2012 11:26:06 AM , Rating: 2
There comes a point where replacing equipment becomes significantly cheaper than cooling them at a lower temperature. I believe Google keeps its datacenters at a sweltering 72F and use cheap redundant hardware that's easily replaceable. It's cheaper for them to replace a switch here and there than to run the server room at 62F.

By dark matter on 6/21/2012 8:03:42 AM , Rating: 2
Water is considerably more efficient at removing heat than air. So even though the chip may be rated at a 20C air temperature, you have to factor in the efficiency of that air to remove waste heat, compared to that of water. It may well be the case that 45C water is actually comparatively similar or better than 20C air.

After all the core of the chip runs a great deal higher than 20C.

And lets face it, I'm quite sure IBM know a bit more about it than any of us here do. I don't think they just decided to "give it a go".

RE: Not really hot water cooled?
By haukionkannel on 6/19/2012 5:26:02 PM , Rating: 2
Well this is something new... Have to read my thermodynamic books again... So would colder water cause some kind of barrier between the prosessor and cooler?
Any further information about this would be nice!

RE: Not really hot water cooled?
By alexwgreen on 6/20/2012 5:30:03 AM , Rating: 2
It's been a while since I've had to fire up my scientific brain, but...

Getting heat from a processor to the ambient air in a liquid set up requires (simplified) 4 transfers of heat,

1. Processor to Block
2. Block to liquid
3. Liquid to heat exchanger
4. Heat exchanger to air

At each stage, there will be a temperature gradient, inversely proportional to the efficiency of the transfer, and the sum of those temperature gradients for the power output determines the temp difference between the air and the power source.

In these computing applications, the air side is usually chilled to obtain low core temperatures.

By allowing the water to increase to a higher temperature, the viscosity will be reduced, which in turn will tend to reduce the boundary layer between flowing water and the surface of the cooling block. This in turn will increase the efficiency with which heat is transferred both into the water at the block (2) and out of the water at the heat exchanger (3). A lower coolant viscosity will also allow for narrower channels in the cooling block, meaning the designers can get the channels closer to the processor further increasing the efficiency of the heat transfer into the block (1).

So, allowing the processor to run slightly hotter, has led to cooling blocks optimised for the higher coolant temperature, increasing the efficiency of the transfer at stages 1, 2 and 3, meaning that the incoming air doesn't need to be chilled as much, thus the huge energy savings.

Quite clever really, but calling it hot water cooling is, I think a bit misleading.

RE: Not really hot water cooled?
By Calin on 6/20/2012 4:56:56 AM , Rating: 2
They use a special design for water blocks that allows better heat transfer from processors to the cooling medium - as such, the exit water is warmer than what is would be with a conventional system dissipating the same heat (in W) from the same processor temperature (that means less cold water that gets out at a higher temperature for the same cooling effect).
Probably the cooling water is in a separate circuit, cooled either with air that is pumped inside in winter and outside in summer, or with a heat exchanger to a conventional cooling device

"We don't know how to make a $500 computer that's not a piece of junk." -- Apple CEO Steve Jobs

Copyright 2015 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki