The Prime Minister of Ireland believes that, in future, companies will no longer invest in expensive infrastructure which they will run themselves, but will instead buy services through the cloud, which in turn will be delivered from large-scale data centers such as Microsoft’s newly opened mega-facility in Dublin, Ireland.
The man who officially opened Microsoft’s data center, the Irish Taoiseach Brian Cowen, had just flown in from New York. When thanked for his attendance, he said that as the company had just pumped $500m into the Irish economy, there was nowhere else he should be.
As well receiving government support, the Dublin site was chosen for its climate (-5 degrees centigrade to 27 degrees centigrade over one year), being geologically stable and having affordable energy.
DATA CENTER STATS
The site’s statistics are impressive. It is a two-storey configuration, with 303,000 sq ft of white space. It is planned for cooling to be based on free air for almost 100 per cent of the time.
There are no water chillers and no CRAC units in the five halls (colos is how Microsoft refers to them). The absence of chillers means the site will use just 1% of the water used at a traditional facility. There is no raised floor, nor is there a need for one as the air is pulled in from the roof-mounted airhandling units, sucked into Pods through the servers into a double-sided contained hot aisle, and pushed back out through the roof. The air in the halls is changed 60 times per hour.
There are DX units on the roof to augment the cooling if needed on those rare days that Dublin has a heat wave.
And as for those cold winter days and long winter nights, Microsoft said it has designed a system to mix the server-heated air with the incoming cold air to maintain a stable temperature. Inside the rack, according to Kevin Timmons, Microsoft’s general manager for data center operations and infrastructure services, the site will aggressively push inlet temperatures to the maximum.
“We are going to be pushing inlet temperature boundaries as much as we possibly can. We use both industry-standard servers, and we use servers of our own design.
“The power usage effectiveness (PUE) we’re targeting is less than 1.3, but we’re not really going to know how it’s going to perform until we load it up. I’m extremely hopeful that this will be our best-performing center,” says Timmons.
| THE KIT |
The 900mm, 96u APC racks in colo one are currently loaded with Dell 1 U servers and 1u HP SAS storage units. The hot aisle inside each double-sided Pod is running at 22-26 degrees celsius, but this is expected to rise. Each rack in colo one consists of 12 cabinets. Cable management is provided by Panduit.
Power plant comes from ABB and delivers 10,500 volts. This is transferred down to 400 volts. Power to the rack is 16-amp and 32-amp 3-phase. Battery UPS was supplied by Liebert. The battery UPS will give 20 minutes of power and the 2-CAT backup diesel generators are primed to come on within 20 seconds should the site lose the grid. There are six days’ worth of diesel on the site.
Each of the five colos have 2-CAT backup generators (the whole site is N+1). The air-handling units on the roof were supplied by York. |
“Until recently, we were told that server temperatures had to be at 68 degrees farenheit, and with a very narrow humidity band,” explains Timmons. “Now that ASHRAE standards are expanded, we’re taking full advantage of it, and server manufacturers have really started following suit as well.” The site is being seen as a testbed for efficiency. It has been recognised as a model of best practice by the European Commission and Microsoft says it will continue to push for efficiencies as it loads out the remaining so-called colos.
RAISED FLOORS“Look at the concept of taking out the raised floor,” says Timmons. “It was said for many years that it couldn’t – or shouldn’t – be done. Even using outside filtered air was a big thing at one point. What you see will be developed going forward.
“As the second, third and up to fifth colo is commissioned, change will happen. It will develop over time, and will get better and more efficient in the space of one or two years.” According to Timmons, the important thing is “continual innovation across several phases”.
The first racks are in colo one, and Microsoft said that it is already ramping up the second colo space.
“The load-out is ongoing. It is going to be very steady. I am quite happy with the capacity plan we are delivering,” says Timmons.
“You will see constant install and shipments coming, with load coming up quite steadily. We’ve already begun construction on the next phase. Within the containment Pods, we are already learning to use less energy, and we are cutting costs.
“Colo two is the same layout, but it has a different structure. We’re already using different material to kit out the colo two racks. In colo three, the Pods will be twice the length to drive more efficiency.
“For colos four and five, there is another plan to do it differently again to make the Pods much longer.
“It took the industry 15-20 years to get to Pods, and from us taking on the first concept of Pods we’re onto the fourth generation. We are planning now for the fourth generation [which is expected to be ready in 2011, according to Arne Josefberg Microsoft’s infrastructure services general manager] and colo five will probably be a different design. We will continue to see greater efficiencies,” says Timmons.
At the peak of its construction, Mercury Engineering had more than 130 electrical and mechanical engineers on site.
Colm Burke, general manager at the company, said it was on site from “as the building came out of the ground”. Mercury did the earthing grid and was responsible for installing all the copper, fibre and electrical plant.
POWER SETUPAccording to Declan Comerford, data center architect at the site: “In power terms, the colos are designed to take 5.4mW. At the moment, they are set up for five colos at 2.7mW each.
“The standard connections to the older-style racks that we have in the room are 3-phase 16-amp connections and 32-amp connections. Some of those racks will pull 13kW per rack, but some of them might only pull 4kW,” says Comerford.
“We’re flexible for power. That’s a data center manager’s expertise – to get the most from the bus bar that is running at 90kW and to use the majority of that in a combination of 13s, 4s, 6s and 8s.
 KEVIN TIMMONS |
|
Though responsible for the ongoing operation of the Dublin site Timmons joined Microsoft in June this year just as the site was becoming operational. Described as a hands on boss, he did not spend his whole
life in data centers. He joined Microsoft from being a vice president at Yahoo!, leading their data center operations team. Immediately before that Timmons was the fourth employee hired at GeoCities, where he served as a director of Operations. His background includes over 10 years experience in real-time embedded systems software development with aerospace firms such as Lockheed and Marconi Dynamics. He was instrumental in the development of the avionics display systems for the YF-22 fighter program, which was later selected by the Air Force as their next-generation fighter platform as the F-22 Raptor. |
“You don’t want to put all fours in and use 48kW on a 98kW bus bar because it is not efficient, so you have to design and work within your collocation.
“A lot of people think that is a complicated action, but it shouldn’t be,” says Comerford.
ON SCHEDULE
“It came in on schedule and on budget, with no major surprises as far we are concerned. What we will do is take the lessons found here and apply them to future phases,” Timmons adds.
“As we are expanding into further phases, you never quite know how you’re going to do from an efficiency standpoint until you get into operation and really put significant load on it. So we will continue to learn. So far, the lessons learned have been positive – how to construct a better, faster and cheaper data center.
“The important fact is that this is the most efficient and environmentally sustainable facility that Microsoft has designed, built and is operating today, and we expect to learn a lot from this facility,” he says.
This article first appeared in DatacenterDynamicsFOCUS Magazine.
