Power consumption is becoming increasingly important for information and communication technology (ICT), especially for data centres. While operators are under increasing cost pressure due to rising energy prices, the enormous consumption of resources for the required energy is also worrying. This calls politicians to prepare concrete regulations for data centre operators. The EU Directive 2012/27/EU on energy efficiency already called for measures to save energy consumption. Germany formulates this target in its 20-20-20 goals. Accordingly, primary energy consumption is to be reduced by 20 % by 2020.
In 2014, data centres in Germany were already responsible for the consumption of around 10 billion kWhor 1.8 % of total German electricity consumption. The sheer size is only made clear by a comparison: This corresponds to the electricity consumption of around 3 million 3-person households. An incredible amount. On a global scale, the figures are becoming even more impressive. At 416 TWh, the electricity consumption of the data centres installed worldwide significantly exceeded the UK's electricity requirement of around 300 TWh. And the demand for electricity is growing visibly, because the number of data centres is also growing continuously due to higher demand for computing power. In 2014, the number of physical servers in Germany had already risen to 1.7 million. The following diagram illustrates that this development is far from stagnating.
Figure 1: Power requirements for servers and data centres in Germany (Bitkom, 2014)
The increase in electricity consumption is not necessarily due to a lack of willingness to innovate. In 2014 alone, 800 million euros were invested in the modernization and new construction of data centres in Germany alone to limit this development.
You can't improve what you don't know
An important milestone in the optimization of data centres was, in addition to technological developments in energy supply, cooling systems and waste heat utilization, in particular the systematic recording of power consumption. Only the analysis of the actual situation made it possible to transparently evaluate measures to increase energy efficiency on the basis of energy performance indicators.
What is PUE and how can it be improved?
The development of an industry standard for determining the energy efficiency of data centres was promoted by The Green Grid. In 2007, the companies involved developed the key performance index of power usage effectiveness (PUE). It indicates the ratio of total energy consumption and the energy requirements of the IT components and is currently the only parameter that is used by different data centres to determine and compare the relative energy efficiency.
Ideally, all the energy is used for the IT infrastructure. In this case the PUE is 1.0, but since additional energy is always required for losses, UPS, lighting, control technology and the cooling system including the pumps, recirculation units and dry coolers, the value of 1.0 is only a theoretical target value. According to Bitkom, with a physical utilization of the available rack space of approx. 50%, a modern state-of-the-art data centre should achieve a maximum PUE of 1.4 or lower.
The calculation of the PUE depends on the definition of the system limits within a data centre, see Figure 2. The energy flows recorded at this point serve as the basis for determining the data centre.
Figure 2 System boundaries of data centres for the calculation of the PUE (own representation)
However, there is scope for interpretation when it comes to defining the IT components to be included in the calculation. For this reason, a distinction is made between PUE maturity levels, which differ in the measuring methods, measuring intervals and measuring points used, see Figure 3.
Figure 3: PUE Maturities (Bitkom, 2015)
An important criterion for the comparability of the PUE of different data centres is the consideration of the same period. The power requirement for cooling the IT infrastructure is considerably higher in summer than in winter. This should be taken into account in the calculation, which is why the collection of twelve-month averages in particular is a guarantee for meaningful PUE measured values. If a PUE is specified for an observation period of less than one year, it is referred to as interim PUE (iPUE). This is of particular advantage when determining a real-time PUE, which can be used for live monitoring.
The PUE has become the industry standard for calculating energy efficiency in the course of standardization of data centres, in particular the DIN EN 50600 series of standards. This European standard not only addresses topics such as construction, electricity and climate, but also focuses holistically on all issues relevant to data centres, including management and operation.
Air or water cooling, who is ahead in PUE?
Conventional air-cooled data centres make up a large part of the installed server systems to date. However, their biggest flaw is that cooling is often responsible for around 40 % of energy consumption due to air conditioning technology. Modern air-cooled data centres therefore achieve an average PUE of around 1.5. As an alternative concept, the use of water as a cooling medium has established itself. Water has significant physical advantages when absorbing heat compared to air, because not only is the heat capacity 3300 times higher, but the thermal conductivity is also 20 times higher than that of air. But water cooling is not just water cooling. A distinction is made between direct hot water cooling and indirect water cooling via sidecoolers. With sidecoolers, the heated air inside the servers is blown outwards by fans, analogous to air cooling. Air-water heat exchangers are then located here to cool the air. In direct hot water cooling, on the other hand, the water is conducted directly along the heat-emitting components in order to absorb the heat energy emitted. The efficiency of this cooling technology means that additional fans and air conditioning technology can be dispensed with and significant efficiency gains can be achieved.
A direct comparison of both cooling systems was carried out in a study by the U.S. Department of Energy. In an existing high-performance computing (HPC) data centre, the air cooling system was replaced by one with direct water cooling. Based on the Linpac benchmark, the PUE could thus be compared for the same performance. Figure 4 clearly shows that the PUE of the system with water cooling is not only significantly lower than that of the air-cooled alternative, it is also associated with significantly less fluctuations.
Figure 4: PUE comparison of HPC cooling systems with direct water and air cooling for Linpac benchmark test (based on US Department of Energy, 2014)
The world's lowest PUE is currently 1.014. This record value was achieved by Cloud&Heat Technologies and was possible primarily due to direct hot water cooling as well as demand control and preheating of the required cooling air through an underground car park. By comparison, Google's PUE reaches 1.14 across data centres, with Facebook reporting a PUE of 1.09 for its largest data centre in Prineville, Oregon.
Innovations "outside the box
On the way to ever lower PUEs, some companies are also trying out completely new concepts. The Internet service provider IGN from Munich has implemented a cooling system in cooperation with Rittal that uses groundwater as coolant. The cool groundwater is pumped from a well into a water circuit cooled by heat exchangers, which supplies eight redundant recirculation air conditioning systems. In combination with an optimised cold air duct, a PUE of 1.2 could be achieved. Microsoft's research experiment "Project Natick" goes even further. In order to minimize the energy required for cooling, the low temperature of the sea is taken advantage of. A container data centre was sunk in the Pacific Ocean. The cooling requirement was thus covered by the low ambient temperature. However, whether this is a model for the future remains to be seen.
A good PUE is not everything
However, an isolated view of PUE can lead to false conclusions about the actual efficiency of a data centre. This is illustrated by the following example. Let's assume that in a data centre with 1,000 servers, 700 servers run at idle due to a lower overall utilisation rate. To save energy, these 700 servers are switched off. Although this would result in a reduction in energy consumption, it would lead to an increase in PUE, since the denominator of the calculation formula, the power consumption of the remaining 300 servers, decreases much more than the meter, the total energy requirement. But there is also room for improvement in the remaining servers. The workload can be further increased by combining the workload on some servers through virtualization, for example with OpenStack, in order to shut down others. This saves energy and ultimately also costs.
In order to increase the informative value of energy performance indicators, the inclusion of further performance indicators in DIN EN 50600 is currently being discussed. A combination of these values should provide a more comprehensive picture of the efficiency of a data centre. Energy recovery, for example, offers great potential for increasing the energy efficiency of data centres. If, for example, the heat energy emitted is fed into a heating system, energy consumption can be reduced here. In this case, however, the waste heat reduces the energy requirement at points outside the system boundaries of the data centre. and is therefore not included in the PUE analysis. One indicator that is more suitable for assessing a data centre with regard to its energy recovery is "Energy Reuse Effectiveness" (ERE).