high-performance computing


NOW WE ARE LOOKING AT THE NEXT GENERATION OF PROCESSORS AND WE ARE LOOKING AT 30 KILOWATTS A RACK AND THIS IS ONLY GOING UPWARD


➤ OCF’s Dean noted the


importance of understanding requirements but also stressed the need to properly prepare for potential upgrades in the future, as many users are employing much denser computing solutions: ‘Since I started, we were delivering solutions that were 10 to 15 kilowatts a rack, and this has quickly become 20 to 25 kilowatts per rack. Now we are looking at the next generation of processors, and we are looking at 30 kilowatts a rack, and this is only going upward.’ Dean commented that the


possibility for very high-density systems such as GPU based clusters could push this as far as 70 kilowatts a rack in extreme examples. It is therefore important to understand a user’s requirements to determine which technology is right for a particular installation. ‘From the other side, it is down


to TCCO and the total available amount of electricity available to the datacentre. If you have an amount of energy or a total number of amps coming into your building, for example if you can drive the PUE as close as possible to one,


Cool tech cuts university cooling bill


The recent installation of BlueBEAR3, the University of Birmingham’s HPC cluster and part of the Birmingham Environment for Academic Research (BEAR), has demonstrated the benefits of upgrading the cooling infrastructure with significant improvements in the electricity used to cool the system. The new system has improved cooling energy usage by as much as 83 per cent by switching from air cooling to Lenovo’s NeXtScale direct on-chip warm water cooling technology.


The system takes water at up to 45°C into the rear of the server via heat sinks attached to the CPUs, dual in-line memory modules, I/O and other components. Water returning from the components withdraws heat from the system, rising in temperature by about 10°C in the process.


Simon Thompson, research computing infrastructure architect at the University of Birmingham, explained that three key features helped to achieve this energy saving.


Thompson said that there is an


energy saving from moving to water over air: ‘There are no chassis fans in the Lenovo system (except power supplies). This is important as other after-market cooling solutions still require fans to recover heat from (e.g.) voltage regulators, memory, IB card, etc.’


‘The water can be up to 45°C inlet temperature. This means that we do not need chilled water and therefore we can achieve cooling via dry-air coolers. There is no compressor load required to cool the system. We also find that we rarely need to run all of the air-blast fan units we have. Each of which is ~500W with a cooling capacity of 25kW. Therefore, operational cooling costs are significantly lower, compared to requiring chilled water systems such as rear door heat exchanger (RDHx).


‘Lenovo was among the first to be doing x86 cooling with direct cool solutions. Also importantly, it is not a rack-scale only design. Part of the big attraction for us is that we can add compute nodes in a modular manner, which is


10 SCIENTIFIC COMPUTING WORLD


then you have more energy leſt over to use for the HPC system,’ added Dean. ‘Te upfront investment in


going for one of these more novel approaches is higher, and that is one of the challenges to adoption, as datacentres are oſten procured and funded separately from the actual bit that goes in them. Novel designs cost more money, but this is offset by the savings in efficiency over the life cycle of the cooling and data centre infrastructure,’ Dean explained. ‘Te best way to maximise


efficiency savings and to ensure the lowest PUE requires a comprehensive approach from procuring the datacentre to setup, infrastructure and the choice of cooling technology.’ ‘As you get more integrated and start to look at warm water cooling


and eventually looking at things you can do with this water coming from the nodes, then it needs a much closer integration across the entire business,’ Dean concluded. Iceotope’s Hopton explained


the choices that users face when deciding to use direct liquid cooling technologies. He argues that the main choice between direct liquid cooling technologies is Partial or Total-Liquid-Cooling, with both having their ideal use cases. ‘Partial-Liquid-Cooling is a good


fix for cabinets that are under-filled in existing data facilities, or can be installed into facilities that have a low density – say 5kW/cab – provision for air cooling and spare power. ‘Te addition of liquid cooled


infrastructure adds cost and complexity, but can enable the use of spare power in the existing


The School of Computer Science in the University of Birmingham


maybe not as easy with some of the large-scale manufacturers who are working at rack-scale design. ‘I can get a single tray with two compute nodes for a research group and add it to my facility. Lenovo has a TCO calculator, which would be appropriate for other sites to gauge how well it might fit their solution. For a few nodes, it will never be cost effective, but as a modular, scalable system from even just a rack’s worth of nodes, it is a highly competitive solution when looking at TCO,’ stated Thompson. When asked about the ability


to recreate this saving at similarly sized facilities, Thompson explained that this would be possible but also recommends that for maximum efficiency saving, users ‘would want to look at how they chill their water’. ‘There is also the potential that they may be able to use the low- grade heat, e.g. Central Heating pre-heat and therefore reduce total energy load more. Alternatively, with a large enough solution, you could look at adsorption chillers to generate chilled water for your RDHx from the heat produced from the system,’ concluded Thompson.


www.scientific-computing.com


Page 1  |  Page 2  |  Page 3  |  Page 4  |  Page 5  |  Page 6  |  Page 7  |  Page 8  |  Page 9  |  Page 10  |  Page 11  |  Page 12  |  Page 13  |  Page 14  |  Page 15  |  Page 16  |  Page 17  |  Page 18  |  Page 19  |  Page 20  |  Page 21  |  Page 22  |  Page 23  |  Page 24  |  Page 25  |  Page 26  |  Page 27  |  Page 28  |  Page 29  |  Page 30  |  Page 31  |  Page 32