CERN, the European particle physics laboratory, has placed an order for ten data center modules to handle a major flood of data that will be produced by upgrades to the Large Hadron Collider (LHC) - the largest particle accelerator in the world.
The High Luminosity (HL) upgrade to the LHC will look beyond the success of finding the Higgs boson, and gather much more data, to answer more fundamental questions about matter and the universe. While the upgrade is in progress, individual teams are reviewing their equipment, and the increased data rates have forced two of them to adopt micro data centers close to the experiment sites on the LHC’s 27km ring. Ten pre-fabricated modules will be installed at ground level, immediately above the scientific equipment of the ALICE and LHCb experiments.
Many more collisions
The experimental data at the LHC provides serious challenges. High energy particles are accelerated towards each other to study the effects of their collisions. To detect and study these collisions means capturing a vast amount of data from microscopic events. The more collisions that can be created and captured, the more scientific discoveries are possible.
The HL LHC upgrade will generate many more collisions. In the last three years, the LHC has generated 1.5 x 1016 collisions, referred to as 150 “inverse femtobarns.”
After the upgrade it will produce some 60 percent more - a figure which could eventually increase to 4,000 inverse femtobarns.
More simply, the current LHC setup produced around three million Higgs bosons in 2017, and the high-luminosity version will churn out 15 million each year - along with other potential discoveries.
Between 2019 and 2021, the LHC will have a Long Shutdown. During this time two experiments - ALICE and LHCb - will upgrade their equipment, and add new data center hardware to handle more data. Other experiments will be upgraded during the next Long Shutdown between 2024 and 2026.
ALICE and beauty
ALICE (a large ion collider experiment) and LHCb (large hadron collider beauty) will both receive upgrades, and add a total of ten prefabricated “SAFE” modules from Automation Datacenter Facilities of Belgium, all of which will be installed as close to the experimental equipment as humanly possible, simply because of the huge amounts of data that needs to be gathered.
ALICE, near the village of St Genis-Pouilly in France, recreates conditions shortly after the Big Bang, colliding lead ions at temperatures more than 100,000 times hotter than the center of the Sun, so quarks and gluons can be observed.
LHCb is looking at the b or “beauty” quark to try and solve a puzzle in physics: what happened in the Big Bang to skew the universe so that it contains mostly regular matter, with very little anti-matter?
The LHCb experiment will be updated to capture images at 40MHz (40 million times per second) instead of 1MHz (a million times per second). Instead of making a selection of collision data at the sensor, all data will be processed. This demands a 40Tbps link to the modular facility above ground, using new 300m cables with 15,000 optical fibers.
The modules simply had to be located on top of the experiments, to avoid sending the data to other data centers around CERN, explained Niko Neufeld of the LHCb’s online team: ”It is not cost-efficient to transport more than 40Tbps over 3km!”
Two modules will handle the I/O, interfacing to high-speed communications protocols used by CERN and transferring it to four modules which will filter the LHCb data to find interesting events, using general-purpose GPUs for number crunching. Meanwhile, the ALICE experiment will have four other modules for event filtering.
The modules will be cooled using customized Stulz systems, which operate with indirect heat exchangers supplemented by evaporative cooling. The whole system will have a power usage effectiveness (PUE) rating of below 1.075, thanks to the near-Alpine climate of CERN.
The modules have a reliable power supply, although Juergen Caeyman of Automation points out there is no need for fully redundant power: “We are providing a power module with A and B feeds. There is a small amount of UPS on the ALICE system, but it’s built with no redundancy. If the electricity drops there are no particles.”
The first two CERN modules will be delivered in September 2018, with the rest installed before the 2019 Long Shutdown. During the shutdown, the fiber connections will be installed, and the IT systems built and tested before the LHC is switched on again in 2021.