In 1993 when IBM introduced CMOS technology to enterprise computing, computer power consumption dropped dramatically. This enabled air to supplant liquid as the preferred cooling medium. But times have changed. Today’s CMOS processors run at gigaflops and generate just as much heat as the old bipolar technology, outrunning the capability of air cooling. Air has three problems.

1)      As the amount of heat increases, the power required to move air increases exponentially to a point where it can absorb over 20% of the power budget, adding to the downstream cooling load at the chiller or cooling tower.

2)      Inefficiencies in air distribution cause uneven cooling. Cooling air at the top of a rack may be 20OC hotter than at the bottom. This must be compensated for by over-cooling the air, reducing chiller efficiency and increasing its power consumption.

3)      The overall thermal resistance of the path between the heat generators in the electronics enclosure to the heat dispersal system is high. Additional energy is burned pumping heat across this resistive thermal barrier. This can amount to 40% of the power budget There are ongoing attempts to mitigate these problems but all add cost and complexity in return for dubious benefits.

Liquid cooling as implemented today solves these issues but introduces a different set of problems. All deployments to date require liquid within the enclosure. Some are contained totally within the enclosure and others require piped connections to an outside cooling system. While these may be efficient heat removers, their cost for volume deployment is prohibitive. Other issues include serviceability, requiring specialist technicians for even the simplest equipment swap, and reliability of pumps and liquid connectors.

Clustered Systems has solved the problems associated with liquid cooling by keeping liquid out of the electronics enclosure while still providing all of its benefits. Industry standard motherboards, enclosures and racks are used throughout. Expensive internal fans and heatsinks are replaced with simple heat risers conductively moving the heat to the enclosure lid. A proprietary thermal interface on the lid transfers the heat to a cold plate.

A standard rack is outfitted with cold plates above each 1U slot through which coolant flows. Using standard rails, standard 1U servers are inserted into the rack below each cold plate. The cold plates are then moved into contact the enclosure.

The energy savings from the conversion can exceed 50% of the total data center load. Other advantages include a very high power density, over 1KW per square foot, and noiseless operation. These latter make the technology ideal for use in container based computing as well as in more traditional data centers.

We chose LBNL’s NY Data Center #2 Energy Benchmarking and Case Study as typical to illustrate potential savings.

Click for Graph View

This DC has a PUE (power use efficiency) of 1.96[1] and energy consumption of almost 5MW in the unmodified state. A first level conversion consists of substituting chilled water to coolant heat exchangers for CRACs (computer room air conditioners) and adding racks with cold plates. This step alone reduces energy consumption to about 3.5MW, achieving a 27% energy saving.

Because of the higher system efficiencies, over-cooling is not required. By increasing the chiller’s output water temperature, energy can be reduced to about 3.2MW, saving 33%. At 20KW per rack, payback with $0.10 energy cost would be approximately 30 months. Of course, with a 33% power reduction and higher power density, 50% more servers could be added to an existing facility. Payback also comes in the form of indefinitely postponed new build.

As the wet bulb temperature in the US rarely exceeds 65F (17C) it becomes practical to eliminate the chiller totally while maintaining enclosure internal temperatures well below manufacturers’ specifications. In this case, power usage decreases to 2.6MW, saving 45%.  

 We estimate that new construction costs will drop by about 20%[2]. Major savings by applying CSys technology are achieved by the reduction in floor space, thanks to 4 times higher density and in the electrical subsystem as only 55% (2.2MW vs. 4MW) of the power is required. This offsets the moderate increase in the cooling system.

Liquid cooled systems have major power and capital cost advantages over air based systems. By eliminating the drawbacks of such systems CSys has prepared the way for liquid cooling to re-enter the mainstream.


 
[1] Note that the PUE is misleading. The servers’ internal fans should be considered part of the cooling infrastructure, not as part of the compute load. Using this metric, the air cooled DC PUE becomes 2.18. In contrast, the no chiller PUE is 1.23.

[2] Based on a chart published by Liebert, Inc. comparing air cooled 4MW data centers with a power densities of 50 to 400W per square foot