Archived Content
The following content is from an older version of this website, and may not display correctly.The towering sails of the roof of the Sydney Opera House – one of the world’s most iconic buildings – may not have existed if it wasn’t for supercomputing in the UK. Their irregular shape brought about huge structural challenges for the design team at Arup. The only way they could overcome problems with structural load was to use supercomputing for simulations provided by a Sirius computer at Monash University in Australia and a Ferranti Pegasus computer at the University of Southampton in the UK. This was 1959 and at the time the research facilities provided enough compute power to run a 1999 hand-held calculator, according to Leonard R Bachman, the author of Two Spheres – a book that documents the opera house design challenge.
The Ferranti was the Southampton university’s first supercomputer and it marked the start of a journey that now underpins much of its researchers’ work – most of which focuss on science and engineering. Behind the scenes, it also marked the start of some interesting times for the university’s data center and technology teams.
Until last year, the university’s high performance computing (HPC) operations were “shoehorned” into an unused basement in an on-campus building – cooling, power and all. Design challenges – from moving one-tonne pre-built racks through small access points to maintaining operations to achieving efficiency levels – were governing many of their decisions. But the latest HPC set up – Iridis4 (an IBM DataPlex supplied by UK HPC company OCF) – has finally seen the HPC facility move into its own home. And this is where the benefit of bringing architecture and infrastructure together from point of design really speaks for itself. The new build allows for better energy efficiency and future proofing of operations in the data center for the university’s growing array of research challenges, from aerodynamics to medical devices, and paves a path for future innovation.
From nice to have to need to have
The difference between the Ferranti days of the Sydney Opera House design and today’s highly technical simulations is that HPC is now critical to many of the disciplines the university supports. University of Southampton director of research computing Dr Oz Parchment says when the first version of its Iridis supercomputer (Iridis1) was installed in 2001, the university had no more than 50 users for its capacity. Iridis marks the start of the university moving to commodity hardware for supercomputing. It previously used proprietary processors from a number of specialist suppliers.
“Today we have more than 800 projects on the new machine (Iridis4),” he says. “Computational science is seen as the third pillar of scientific discovery alongside theory and experiment today. It is something that is ubiquitous across almost any discipline you can imagine.”
In January, when I visited Iridis4 – off campus and in a shell in an industrial estate – it was running at 85% capacity 24x7. Just six months prior to this Iridis3 had been holding the load at the on-campus site, running at 99% of its horsepower. “The thing is, once capacity is up at the university it flat lines,” Parchment says. “We can’t burst into the Cloud to increase capacity. Demand rises and never falls.”
Professor Richard Sandberg works in Engineering and Environment carrying out experiments on turbulence that he says could lead to new aircraft designs minimizing noise. While a certain amount of time is required in the lab, he says simulation is vital for studying the team’s large data sets. “What we are looking at becomes important for noise restrictions at airports. If they want to keep increasing traffic then you have to do something to minimize the noise generated by fan blades or the trailing end of wings on aircraft,” Sandberg says. “The simulations we do are carried out over a couple of weeks to measure the flow field. If we look at a simulation that runs over 800 cores in one day – which we do quite often – it would be equivalent to using a single desktop computer for 22 years. And we can’t wait 22 years until we have an answer.”
Iridis4 is four times more powerful than Iridis3, running 12,200 Intel Xeon E5-2670 processor cores, a petabyte of disk space and 50 terabytes of memory.
Putting baby in the corner
The move to a new building off campus marks a start of a new supercomputing era for the university – one that recognizes its true importance. The basement building its HPC previously ran out of was brought into use in 1975, when data center teams decided to install the first Iridis supercomputer. “We were promised a new building in 1975 but as it turned out we had to refurbish a room that was underneath an existing building and below the water table which was completely inappropriate at the time. Thirty-five years later, after numerous floods, we were still there in a basement,” Parchment says.
Data center manager Mike Powell says the team pushed forward with its goal of adding incrementally more compute. “Every installation of a supercomputer inside the old data center was challenging. We had to think very hard how we were going to do it,” Powell says.
The floor had to be steel plated because of the weight of the racks – delivered fully loaded with Iridis. Diamond drilling had to be done to adjust doorways for Iridis3 to allow the one-tonne racks that could not be tilted to be craned in. This was not building a data center to fit a space – as they had done in the past pre-commodity hardware days. It was fitting a pre-built, ordered facility into what room they had. “We had three doorways to drill through to get Iridis3 in and the first two diamond cuts were simple,” Parchment says. “But when they came to the third it took them four times as long as anticipated because the heat in the data center where previous Iridis installations lived had cured the concrete, making it much harder – which is really kind of scary.”
To make matters worse, alongside the HPC facility in an adjoining space was the university’s enterprise data center. For the Iridis3 installation a polythene tent was established to separate the room during drilling to maintain the data center temperature and keep critical university services online. “We didn’t even have separate plant spaces. Mechanical equipment was shoehorned into a space beside the data center that used to be an office. We had two UPSs and batteries stacked right up to the ceiling,” Powell says.
Parchment says working in such a confined space meant layers of complexity were constantly being built into its facility. “Over the course of the last decade it had really become an unstable set up. We did a lot of restorative work but we still had a number of incidences. We kept finding work completed five, six or seven years ago that nobody knew about – done by someone who had left – would suddenly cause things to go ‘bang’ and the data center would be taken out. We would find that back in 1995 somebody had put this switch inside this socket and that had caused another problem. And when that happens the university may as well just go home.”
This basement facility is still running with Iridis3 inside but now acts as backup for a new data center, which has been designed with each painful blow the basement delivered in mind.
Coming into its own
Before taking the construction route with Iridis4 the university did consider colocation – back in 2008. “But how would you get a colo data center that is used to dealing with 2 to 4kW per rack to start worrying about 20 to 30? We could not find anything financially viable,” Parchment says.
The HPC teams then embarked on a two-year project to find space on campus to build. “But the way universities are going we would have had to build another compact site. HPC is not a front-facing activity and the architecture would have had to fit in with the campus. This means you spend a huge amount of money making the facility look appropriate,” Parchment says.
In February 2012 the University of Southampton purchased a building for £1.9m (all that remains of it is the steel structure and floor) in an industrial park that houses two other data centers. This provided diverse communications links and was on the borderline of two national grids. With partners Arup, PDCM project managers and FES Limited (the main contractor) they came up with a wish-list that would overcome all the challenges they had worked with in the past. Construction began in April 2012 and the site was handed over in March the following year.
The entire project cost £19m but it wasn’t only HPC users that benefited. The uni had also embarked on a major virtualization project for enterprise systems and now these are housed alongside Iridis4 – which came online six months ago. Energy efficiencies gained from this project now offer more power capacity that can be used for HPC. “We have saved about 6MW hours per annum by delivering the virtualization project, which has cost us about £3m to £4m so far. We had about 1.3 to 1.4kW of IT load when started, and estimated we needed about 120 racks based on our future requirements.” Parchment says. “We decided this could equal a 3MW data center and with all the space and racking you need for that, it all escalates quite fast. So we accelerated our virtualization project, bringing our needs for business computing down from about 700kW to 250kW. Now we have growing room both for that estate and for our HPC environment.”
Nothing lasts forever
The facility at the Southampton industrial estate has a 10 to 15-year lifespan for chillers and UPSs, and 20 to 25 for the building cladding itself. And both Parchment and Powell believe they will be working in a very different era of compute by then. In the meantime, they have a facility built to enhance how its computing equipment can run.
The Ferranti was the Southampton university’s first supercomputer and it marked the start of a journey that now underpins much of its researchers’ work – most of which focuss on science and engineering. Behind the scenes, it also marked the start of some interesting times for the university’s data center and technology teams.
Until last year, the university’s high performance computing (HPC) operations were “shoehorned” into an unused basement in an on-campus building – cooling, power and all. Design challenges – from moving one-tonne pre-built racks through small access points to maintaining operations to achieving efficiency levels – were governing many of their decisions. But the latest HPC set up – Iridis4 (an IBM DataPlex supplied by UK HPC company OCF) – has finally seen the HPC facility move into its own home. And this is where the benefit of bringing architecture and infrastructure together from point of design really speaks for itself. The new build allows for better energy efficiency and future proofing of operations in the data center for the university’s growing array of research challenges, from aerodynamics to medical devices, and paves a path for future innovation.
From nice to have to need to have
The difference between the Ferranti days of the Sydney Opera House design and today’s highly technical simulations is that HPC is now critical to many of the disciplines the university supports. University of Southampton director of research computing Dr Oz Parchment says when the first version of its Iridis supercomputer (Iridis1) was installed in 2001, the university had no more than 50 users for its capacity. Iridis marks the start of the university moving to commodity hardware for supercomputing. It previously used proprietary processors from a number of specialist suppliers.
“Today we have more than 800 projects on the new machine (Iridis4),” he says. “Computational science is seen as the third pillar of scientific discovery alongside theory and experiment today. It is something that is ubiquitous across almost any discipline you can imagine.”
In January, when I visited Iridis4 – off campus and in a shell in an industrial estate – it was running at 85% capacity 24x7. Just six months prior to this Iridis3 had been holding the load at the on-campus site, running at 99% of its horsepower. “The thing is, once capacity is up at the university it flat lines,” Parchment says. “We can’t burst into the Cloud to increase capacity. Demand rises and never falls.”
Professor Richard Sandberg works in Engineering and Environment carrying out experiments on turbulence that he says could lead to new aircraft designs minimizing noise. While a certain amount of time is required in the lab, he says simulation is vital for studying the team’s large data sets. “What we are looking at becomes important for noise restrictions at airports. If they want to keep increasing traffic then you have to do something to minimize the noise generated by fan blades or the trailing end of wings on aircraft,” Sandberg says. “The simulations we do are carried out over a couple of weeks to measure the flow field. If we look at a simulation that runs over 800 cores in one day – which we do quite often – it would be equivalent to using a single desktop computer for 22 years. And we can’t wait 22 years until we have an answer.”
Iridis4 is four times more powerful than Iridis3, running 12,200 Intel Xeon E5-2670 processor cores, a petabyte of disk space and 50 terabytes of memory.
Putting baby in the corner
The move to a new building off campus marks a start of a new supercomputing era for the university – one that recognizes its true importance. The basement building its HPC previously ran out of was brought into use in 1975, when data center teams decided to install the first Iridis supercomputer. “We were promised a new building in 1975 but as it turned out we had to refurbish a room that was underneath an existing building and below the water table which was completely inappropriate at the time. Thirty-five years later, after numerous floods, we were still there in a basement,” Parchment says.
Data center manager Mike Powell says the team pushed forward with its goal of adding incrementally more compute. “Every installation of a supercomputer inside the old data center was challenging. We had to think very hard how we were going to do it,” Powell says.
The floor had to be steel plated because of the weight of the racks – delivered fully loaded with Iridis. Diamond drilling had to be done to adjust doorways for Iridis3 to allow the one-tonne racks that could not be tilted to be craned in. This was not building a data center to fit a space – as they had done in the past pre-commodity hardware days. It was fitting a pre-built, ordered facility into what room they had. “We had three doorways to drill through to get Iridis3 in and the first two diamond cuts were simple,” Parchment says. “But when they came to the third it took them four times as long as anticipated because the heat in the data center where previous Iridis installations lived had cured the concrete, making it much harder – which is really kind of scary.”
To make matters worse, alongside the HPC facility in an adjoining space was the university’s enterprise data center. For the Iridis3 installation a polythene tent was established to separate the room during drilling to maintain the data center temperature and keep critical university services online. “We didn’t even have separate plant spaces. Mechanical equipment was shoehorned into a space beside the data center that used to be an office. We had two UPSs and batteries stacked right up to the ceiling,” Powell says.
Parchment says working in such a confined space meant layers of complexity were constantly being built into its facility. “Over the course of the last decade it had really become an unstable set up. We did a lot of restorative work but we still had a number of incidences. We kept finding work completed five, six or seven years ago that nobody knew about – done by someone who had left – would suddenly cause things to go ‘bang’ and the data center would be taken out. We would find that back in 1995 somebody had put this switch inside this socket and that had caused another problem. And when that happens the university may as well just go home.”
This basement facility is still running with Iridis3 inside but now acts as backup for a new data center, which has been designed with each painful blow the basement delivered in mind.
Coming into its own
Before taking the construction route with Iridis4 the university did consider colocation – back in 2008. “But how would you get a colo data center that is used to dealing with 2 to 4kW per rack to start worrying about 20 to 30? We could not find anything financially viable,” Parchment says.
The HPC teams then embarked on a two-year project to find space on campus to build. “But the way universities are going we would have had to build another compact site. HPC is not a front-facing activity and the architecture would have had to fit in with the campus. This means you spend a huge amount of money making the facility look appropriate,” Parchment says.
In February 2012 the University of Southampton purchased a building for £1.9m (all that remains of it is the steel structure and floor) in an industrial park that houses two other data centers. This provided diverse communications links and was on the borderline of two national grids. With partners Arup, PDCM project managers and FES Limited (the main contractor) they came up with a wish-list that would overcome all the challenges they had worked with in the past. Construction began in April 2012 and the site was handed over in March the following year.
The entire project cost £19m but it wasn’t only HPC users that benefited. The uni had also embarked on a major virtualization project for enterprise systems and now these are housed alongside Iridis4 – which came online six months ago. Energy efficiencies gained from this project now offer more power capacity that can be used for HPC. “We have saved about 6MW hours per annum by delivering the virtualization project, which has cost us about £3m to £4m so far. We had about 1.3 to 1.4kW of IT load when started, and estimated we needed about 120 racks based on our future requirements.” Parchment says. “We decided this could equal a 3MW data center and with all the space and racking you need for that, it all escalates quite fast. So we accelerated our virtualization project, bringing our needs for business computing down from about 700kW to 250kW. Now we have growing room both for that estate and for our HPC environment.”
Nothing lasts forever
The facility at the Southampton industrial estate has a 10 to 15-year lifespan for chillers and UPSs, and 20 to 25 for the building cladding itself. And both Parchment and Powell believe they will be working in a very different era of compute by then. In the meantime, they have a facility built to enhance how its computing equipment can run.
“The key to this building really was simplicity. We had to be able to slide everything in and not worry about the fact that these things (racks) could weigh in at tonne and a half. We have seen four people trying to push a 1.5 tonne rack, probably worth commercially £500,000 pounds. I didn’t watch it. I saw the cranes coming and went in for a coffee,” Parchment says.
“We now have very tall doors, free cooling, among other things. We had our fingers burnt so we have invoked as much good practice as we can into this modern data center,” Powell adds.
About 78% of the budget has gone into mechanical and electrical. The building itself is Tier II and all M&E is N+1. Two 1.1MW FG Wilson diesel generators offer 250,000 litres of back up each. Power feeds are done by two HV feeds of 550V. It has 2.5MW reserved on the grid – at present it uses 270kW of this. For both power and cooling the feeds have been split into two – one for the HPC environment, one for the enterprise data center which sits not more than a few feet away from Iridis4. Two Climaveneta chillers and a Jeaggi hybrid cooler sit outside, offering free cooling most days of the year – and helping to contribute to a power usage effectiveness (PUE) of 1.4 ( recorded in first three months of operation during the UK winter).
“We now have very tall doors, free cooling, among other things. We had our fingers burnt so we have invoked as much good practice as we can into this modern data center,” Powell adds.
About 78% of the budget has gone into mechanical and electrical. The building itself is Tier II and all M&E is N+1. Two 1.1MW FG Wilson diesel generators offer 250,000 litres of back up each. Power feeds are done by two HV feeds of 550V. It has 2.5MW reserved on the grid – at present it uses 270kW of this. For both power and cooling the feeds have been split into two – one for the HPC environment, one for the enterprise data center which sits not more than a few feet away from Iridis4. Two Climaveneta chillers and a Jeaggi hybrid cooler sit outside, offering free cooling most days of the year – and helping to contribute to a power usage effectiveness (PUE) of 1.4 ( recorded in first three months of operation during the UK winter).
Freecooling for the project is provided by Jaeggi
Schneider Electric’s Struxureware software is used for data center infrastructure management (DCIM). The team went with Schneider’s Software-as-a-Service version of the solution. “We have transmitters in the racks and ceiling feeding information back,” Powell says.
The computing room supports the enterprise systems, a petabyte of storage for both enterprise and HPC and Iridis4 itself, which has a peak performance of 270 teraflops. Iridis4 has 50 terabytes of memory and 24 nodes with Xeon Phi co-processors. Mellanox FDR Infiniband offers low latency 40Gbps node connectivity.The university chose to use IBM’s iDataplex with M4 nodes which are built for supercomputing. “They are not complicated, they don’t have additional functions which don’t add any value to a compute node,” Parchment says. “One of the key things around supercomputing is you have two sets of components – the work engines (compute nodes) and the management nodes around these to keep it working.”
The cooling requirements are dealt with at the rack level for Iridis4, which has water cooling in the back of the rack. This is what allows it to sit so happily alongside the enterprise data center, which uses water-cooled Rittal racks.
The next step
Iridis4 has a lifespan of about three to four years and the university is already planning its next facility. This could have been difficult, however, if it was still operating out of its old building. “Our current racks are 1.43 tonnes and the floor in this new facility is designed to accommodate 2 tonnes per rack. We needed a cost efficient way to take this up to 5 tonnes, and our design team said there are ways to reinforce these if needed. We have been warned that this may need to be supported in future HPC installations, and if that is not enough to worry about, we may also need to be thinking about 100kW racks.”
The shift towards microservers could even, according to Parchment, render this facility useless in future – or at least in need of a new floor as the weight of each rack increases. But a floor, in his mind, is nothing compared to the power he believes such a shift could bring to the university’s HPC operations.
“Commodity servers now dominate the HPC market but these servers are not, in terms of performance, keeping up with our needs. We are seeing a lot of accelerators or co-processors with, for example, GPU’s or Xeon Phi’s being pushed out. This points to the fact that commodity x86 is just not keeping up with demand,” he says.
This means one day we could see the uni alter its path for HPC yet again. “Whether or not we become proprietary, I honestly don’t know,” Parchment says. But at least this time the uni will have room to make changes. Mechanical and electrical gear may require upgrades, the building a strengthened floor for higher density but a future-proofed build means this doesn’t all need to happen in a space that can’t be altered.
Iridis4 has a lifespan of about three to four years and the university is already planning its next facility. This could have been difficult, however, if it was still operating out of its old building. “Our current racks are 1.43 tonnes and the floor in this new facility is designed to accommodate 2 tonnes per rack. We needed a cost efficient way to take this up to 5 tonnes, and our design team said there are ways to reinforce these if needed. We have been warned that this may need to be supported in future HPC installations, and if that is not enough to worry about, we may also need to be thinking about 100kW racks.”
The shift towards microservers could even, according to Parchment, render this facility useless in future – or at least in need of a new floor as the weight of each rack increases. But a floor, in his mind, is nothing compared to the power he believes such a shift could bring to the university’s HPC operations.
“Commodity servers now dominate the HPC market but these servers are not, in terms of performance, keeping up with our needs. We are seeing a lot of accelerators or co-processors with, for example, GPU’s or Xeon Phi’s being pushed out. This points to the fact that commodity x86 is just not keeping up with demand,” he says.
This means one day we could see the uni alter its path for HPC yet again. “Whether or not we become proprietary, I honestly don’t know,” Parchment says. But at least this time the uni will have room to make changes. Mechanical and electrical gear may require upgrades, the building a strengthened floor for higher density but a future-proofed build means this doesn’t all need to happen in a space that can’t be altered.
This article first appeared in FOCUS issue 34. To read the full digital edition, click here. Or download a copy for the iPad from DCDFocus.