Print 30 comment(s) - last by dark matter.. on Jun 21 at 8:09 AM

IBM spearheads novel heating design, develops and tests it with the LRZ -- a German supercomputing center

International Business Machines, Inc. (IBM) is in hot water with its latest supercomputer design, and that's a good thing.  The company announced this week the availability of the world's first hot water commercial supercomputer.

I. Building a Better Cooled Supercomputer

Dubbed LRZ "SuperMUC", the system is composed of IBM System x iDataPlex Direct Water Cooled dx360 M4 servers.  It can pack up to 150,000 cores (in 18,000 Intel Corp. (INTC) Xeon processors) for up to 3 petaflops of execution at a time.  Describes IBM:

[This] is equivalent to the work of more than 110,000 personal computers. Put another way, three billion people using a pocket calculator would have to perform one million operations per second each to reach equivalent SuperMUC performance.

At the same time, by ditching traditional air cooling for a liquid coolant power costs to be cut by up to 40 percent over a traditional air design.  The water is heated to a hotter than normal temperature via special microchannels in the cooling blocks, hence the "hot water" name.  The heated fluid gets up to a toasty 113 degrees Fahrenheit, or 45 degrees Celsius, cutting the power consumed by cooling to around a fifth of the levels used in traditional designs.

Of course many tech giants like Facebook, Inc. (FB) and Google Inc. (GOOGuse liquid-cooling at their data centers, but even this somewhat more efficient technology is reportedly inferior to the new hot-water cooling system in power efficiency.

The system is also more compact than traditional air or liquid cooled designs, which require bulkier blowers, piping, and/or heat transfer systems.

II. Trial Deployment in Germany Shows Superb Results

Even greater savings can be realized by repurposing the waste heat from the supercomputer to heat on-site research institutions in the winter.  This approach was tested at the Leibniz Supercomputing Centre (Leibniz-Rechenzentrum -- LRZ) in Garching near Munich, Germany.

The LRZ, who helped develop the commercial supercomputer technology, not only was able to realize 40 percent power savings, allowing it to fulfill the German government's power efficiency mandates for research institutions, it also saved $1.25M USD on heating costs at the LRZ, via the attached waste-heat recycling system.

Dr. Bruno Michel, manager, Advanced Thermal Packaging, IBM Research cheers, "As we continue to deliver on our long-term vision of a zero emission data center we may eventually achieve a million fold reduction in the size of SuperMUC, so that it can be reduced to the size of a desktop computer with a much higher efficiency than today."

LRZ indoors
The LRZ saves money by using waste heat from the hot-water cooling system to heat workspaces during the winter.  [Image Source: Steve Lionel/Flickr]

Completed in July 2012, the new LRZ supercomputer is the most powerful one in Europe, and is a member of the Partnership for Advanced Computing in Europe (PRACE).  IBM describes its work, writing:

This performance will be used to drive a wide spectrum of research -- from simulating the blood flow behind an artificial heart valve, to devise quieter airplanes to unearthing new insights in geophysics, including the understanding of earthquakes. The SuperMUC system is also connected to powerful visualization systems, including a large 4K stereoscopic power wall and a five-sided immersive artificial virtual-reality environment or CAVE for visualizing 3D data sets from fields, including Earth science, astronomy and medicine.

Munich and nearby German cities deeply rely on the Centre's computing resources for their studies.  Now they'll be able to do it more affordably, and set an example via a prototype of a design that will likely pop up elsewhere around the world before long, courtesy of IBM.

Source: IBM

Comments     Threshold

This article is over a month old, voting and posting comments is disabled

Not really hot water cooled?
By ElementZero on 6/19/2012 3:03:38 PM , Rating: 5
Seems to me like it still takes cold water, it's just that it is not dumping the hot water after that but instead using it to heat things that would otherwise need their own heating system (or am I reading that wrong?) If that's the case - calling it a "hot water cooled" system is kinda misleading. I thought they had figured out some new way to cool down the processor by using hot water by looking at the title.

RE: Not really hot water cooled?
By JasonMick on 6/19/2012 3:13:30 PM , Rating: 3
Seems to me like it still takes cold water, it's just that it is not dumping the hot water after that but instead using it to heat things that would otherwise need their own heating system (or am I reading that wrong?) If that's the case - calling it a "hot water cooled" system is kinda misleading. I thought they had figured out some new way to cool down the processor by using hot water by looking at the title.
Well that's what IBM's scientists call it.

I believe the heated liquid is actually significantly hotter than in a standard large-channel block-based design.

If you want to get down to the technical details, part of what's different/how the fluid is heated to a higher temp is the microchannel coolers built into the processors, which allows for the fluid to get closer to the processor and absorb more heat (smaller channels == more surface area).

Hence the secret sauce is likely IBM's proprietary microchannel coolers, which were likely the subject of long and laborious work to optimize path efficiency and maximize cooling.

The circulating "hot" liquid temperature -- 113 degrees Fahrenheit, or 45 degrees Celsius -- is quite hot for a system of this scale, I believe, if I'm not mistaken.

RE: Not really hot water cooled?
By drycrust3 on 6/19/2012 3:58:12 PM , Rating: 2
There were two thoughts in my mind when I read this.
The first was the running at a higher temperature will affect the Mean Time Between Failures of all the electronic components, not just the CPU, meaning you could have reduced reliability. My recollection was that most electronics assumes an average air temperature of around 20 deg C. By running at 45 deg this means higher wattage ratted components should be used than for the equivalent 20 deg C design. A worst case scenario is that every printed circuit board would have to be checked to see if it can work in this environment.
The second thought was "Do you need to employ a plumber when installing new equipment?"

RE: Not really hot water cooled?
By chmilz on 6/19/2012 6:38:33 PM , Rating: 2
One would assume that the ambient air temp would still be around 20C. The piping would definitely be fully insulated to prevent heat leak in the server room, thus transferring the heat not only off the servers, but out of the room.

RE: Not really hot water cooled?
By FITCamaro on 6/20/2012 8:30:55 AM , Rating: 2
45C is still pretty cool for a CPU though. But yeah overall its hot for a server room. Assuming no other cooling in the room.

RE: Not really hot water cooled?
By Arsynic on 6/20/2012 11:26:06 AM , Rating: 2
There comes a point where replacing equipment becomes significantly cheaper than cooling them at a lower temperature. I believe Google keeps its datacenters at a sweltering 72F and use cheap redundant hardware that's easily replaceable. It's cheaper for them to replace a switch here and there than to run the server room at 62F.

By dark matter on 6/21/2012 8:03:42 AM , Rating: 2
Water is considerably more efficient at removing heat than air. So even though the chip may be rated at a 20C air temperature, you have to factor in the efficiency of that air to remove waste heat, compared to that of water. It may well be the case that 45C water is actually comparatively similar or better than 20C air.

After all the core of the chip runs a great deal higher than 20C.

And lets face it, I'm quite sure IBM know a bit more about it than any of us here do. I don't think they just decided to "give it a go".

RE: Not really hot water cooled?
By haukionkannel on 6/19/2012 5:26:02 PM , Rating: 2
Well this is something new... Have to read my thermodynamic books again... So would colder water cause some kind of barrier between the prosessor and cooler?
Any further information about this would be nice!

RE: Not really hot water cooled?
By alexwgreen on 6/20/2012 5:30:03 AM , Rating: 2
It's been a while since I've had to fire up my scientific brain, but...

Getting heat from a processor to the ambient air in a liquid set up requires (simplified) 4 transfers of heat,

1. Processor to Block
2. Block to liquid
3. Liquid to heat exchanger
4. Heat exchanger to air

At each stage, there will be a temperature gradient, inversely proportional to the efficiency of the transfer, and the sum of those temperature gradients for the power output determines the temp difference between the air and the power source.

In these computing applications, the air side is usually chilled to obtain low core temperatures.

By allowing the water to increase to a higher temperature, the viscosity will be reduced, which in turn will tend to reduce the boundary layer between flowing water and the surface of the cooling block. This in turn will increase the efficiency with which heat is transferred both into the water at the block (2) and out of the water at the heat exchanger (3). A lower coolant viscosity will also allow for narrower channels in the cooling block, meaning the designers can get the channels closer to the processor further increasing the efficiency of the heat transfer into the block (1).

So, allowing the processor to run slightly hotter, has led to cooling blocks optimised for the higher coolant temperature, increasing the efficiency of the transfer at stages 1, 2 and 3, meaning that the incoming air doesn't need to be chilled as much, thus the huge energy savings.

Quite clever really, but calling it hot water cooling is, I think a bit misleading.

RE: Not really hot water cooled?
By Calin on 6/20/2012 4:56:56 AM , Rating: 2
They use a special design for water blocks that allows better heat transfer from processors to the cooling medium - as such, the exit water is warmer than what is would be with a conventional system dissipating the same heat (in W) from the same processor temperature (that means less cold water that gets out at a higher temperature for the same cooling effect).
Probably the cooling water is in a separate circuit, cooled either with air that is pumped inside in winter and outside in summer, or with a heat exchanger to a conventional cooling device

8.33 cores?
By JediJeb on 6/19/2012 6:12:46 PM , Rating: 2
150,000 cores / 18,000 processors = 8.33333 cores per processor. That is rather strange isn't it?

Or are they subtracting out some type of core that is being shared between groups as a controller?

RE: 8.33 cores?
By ShieTar on 6/20/2012 5:19:44 AM , Rating: 2
In detail it is 147,456 cores in 18,432 processors, the article just rounded the numbers in different directions.

What I find a little curious is that 150k cores are supposed to be equivalent to 110k "Personal computers". Are there really that many single-core PCs still in use that this kind of math makes sense?

RE: 8.33 cores?
By xdrol on 6/20/2012 9:12:28 AM , Rating: 2
Oh yes, there are

RE: 8.33 cores?
By dark matter on 6/21/2012 8:06:35 AM , Rating: 2
One of the billion people on the calculators introduced a rounding error.

Not a bad fault however in the grand scheme of things, given the speed they had to perform calculations.

Time warp
By jbeenemd on 6/19/2012 6:04:15 PM , Rating: 2
I am an old fart and easily confused but when I woke up today it was June 19, 2012. So Jason did you do some time travel when you wrote this?

Completed in July 2012, the new LRZ supercomputer is the most powerful one in Europe, and is a member of the Partnership for Advanced Computing in Europe (PRACE).

RE: Time warp
By JediJeb on 6/19/2012 6:13:49 PM , Rating: 2
I noticed that too.

RE: Time warp
By RedemptionAD on 6/20/2012 12:15:06 PM , Rating: 2
Super Computers are installed in stages, the "hot water" cooling is probably the last stage and it probably should have read "is scheduled for completion in July 2012.".

The heat is from inside the cpu.
By ptmmac on 6/19/2012 6:47:34 PM , Rating: 2
The phrase "hot water" is a marketing gimmick. IBM has been researching this for years. I remember an article about the patent several years ago. There are actual channels in the processor so the heat is being pulled out of the silicon directly under the electronics. I would love to see a schematic showing how they are getting the liquid into and out of those channels. Running a full load on a processor will produce this kind of heat. The processor will not burn out if the max temp on its surface is 45°C. This is well within design tolerances. These temps are from benchmarking the system before getting the whole thing up and running. Usually testing done like this includes stress testing each component so they can correct any weak parts before trying to run the whole thing. Exactly like running Pi to test your memory or burn in your CPU.

RE: The heat is from inside the cpu.
By Calin on 6/20/2012 5:04:49 AM , Rating: 2
The temperature of the cooling agent is lower than the temperature of the cooled object. Even so, with water at 45 Celsius, the processor probably doesn't break 60-65 Celsius, which is a perfectly workable temperature assuming the processor is built for that.
As an aside, some of NVidia's video card chips were rated for 95 Celsius (or just a bit under water's boiling point)

By mattclary on 6/20/2012 8:43:52 AM , Rating: 2
Finally, reading comprehension success. Not sure why the marketing speak was lost on so many.

heating water to cool computers; more efficient?
By Integral9 on 6/19/2012 5:01:44 PM , Rating: 2
Wouldn't it be more efficient to NOT heat the water before using it to cool the computers?

Kinda like driving w/ the brakes on isn't it?

By JediJeb on 6/19/2012 6:10:36 PM , Rating: 2
Looks like the water is coming in cold but being heated to 45c by the processors themselves. Maybe it has to do with the flow rate used, since most home water cooled rigs run with water near room temp both coming in and leaving the processor block because the flow rate is high enough that the water is removing the heat but not getting to such a hot temperature itself. If you are using a lower flow, then the water leaving the block will be hotter.

Hot Water = no A/C in cooling circuit
By ptmmac on 6/19/2012 11:09:13 PM , Rating: 2
The video explains a lot of information left out of the article. The water is merely cooled by a heat exchanger. There is no compressor so the temperature varies from what ever ambient temperature is to 45°C. The heat is pulled off of the whole motherboard as well as the CPU. The roadmap for this technology will push it into 10 fold smaller systems every 5 years. In 30 years it should be used in a standard PC sized system. The implication is that a petaflop box will be common by that time.

By dark matter on 6/21/2012 8:09:04 AM , Rating: 2
Probably need all that processing power to keep state sponsored hacking at bay! Probably run at same the speed as your machine does now.

One small step for mankind,
By leviathan05 on 6/19/2012 3:15:52 PM , Rating: 2
One small step towards a holo-deck, that is.

"Completed in July 2012"
By foolsgambit11 on 6/19/2012 6:23:02 PM , Rating: 2
What was that line supposed to read as? "Set to be completed" or is the date wrong?

By StormyKnight on 6/19/2012 9:02:51 PM , Rating: 2
They could use this as a hot water heater with a heat exchanger adaptor.

By jarvis on 6/20/2012 2:16:10 AM , Rating: 2
I've been in the building that is in the second picture. That big gray tube coming downstairs is actually a huge slide that you can go down, complete with pieces of carpet at the top. When you are exhausted from sliding you may go over to the little store at the left and get ice cream! Uber ;)

Very Interesting...
By Captain Orgazmo on 6/19/2012 8:36:16 PM , Rating: 1
But can it run Crysis?

*runs from the flying rotten vegetables*

By joex444 on 6/19/12, Rating: -1
"Can anyone tell me what MobileMe is supposed to do?... So why the f*** doesn't it do that?" -- Steve Jobs

Most Popular ArticlesAre you ready for this ? HyperDrive Aircraft
September 24, 2016, 9:29 AM
Leaked – Samsung S8 is a Dream and a Dream 2
September 25, 2016, 8:00 AM
Yahoo Hacked - Change Your Passwords and Security Info ASAP!
September 23, 2016, 5:45 AM
A is for Apples
September 23, 2016, 5:32 AM
Walmart may get "Robot Shopping Carts?"
September 17, 2016, 6:01 AM

Copyright 2016 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki