Facebook has created a unique mobile device testing lab in its data center in Prineville, Oregon. The company used 60 custom-built server racks, infrastructure automation tools from Chef and data center cabling techniques to wire up almost 2,000 tablets and smartphones, aiming to improve compatibility between third-party hardware and its own software.
Facebook made several attempts at creating the lab before its engineers realized that data center equipment was the best choice for the job. Now, the company is working to open-source the designs of the racks and related Chef ‘recipes’.
Trial and error
Facebook’s original data center in Prineville was the first server farm the company designed and built for its own use. In 2013, the facility won awards from DatacenterDynamics in two categories: Future Thinking and Design Concepts and Innovation in the Mega Data Center.
The company has since added another building to the campus, and is currently building a third, expected to be completed by December.
Turns out that besides thousands of OCP-compliant servers, Prineville also hosts thousands of mobile devices.
In 2015 the company introduced CT-Scan, a platform that can monitor the effects of a code change within Facebook’s software and help developers avoid accidental regressions that lead to more data, memory, or battery usage.
According to Antoine Reversat, production engineer at Facebook, CT-Scan was initially used to test code on individual devices plugged into individual PCs. The team quickly realized that it needed a lab that could run tests across hundreds, eventually thousands of different mobile devices, from best-sellers to those produced by small, regional brands.
However building such a lab was a challenge. The first iteration featured moving sleds inside a metal rack, sliding into a metal case. “When you slide a bunch of metal sleds into a metal rack, you end up with too much metal, and we lost all Wi-Fi,” Reversat explained.
The second version was nicknamed ‘the gondola’ and consisted of plastic racks that housed 100 devices each. The plastic didn’t interfere with Wi-Fi, but the short length of the USB cables caused a lot of issues. Reversat described it as a “tangled mess.”
Next, the engineers built the ‘slatwall’, an entire room filled with shelves made combination of metal and wood, of the kind typically found in a mobile operator store. A single room of this type could accommodate 240 devices – which meant Facebook would have to build nine of them to satisfy its requirements.
It was at this point that the engineers decided that a data center was the way to go, with familiar server racks housing 32 devices per shelf.
“We custom-built our own racks, designing them to function as an electromagnetic isolation (EMI) chamber. Each rack holds eight Mac Minis (or four OCP Leopard servers for Android testing) that drive the phones to install, test, and uninstall the application we’re testing. Each Mac Mini is connected to four iPhones, and each OCP Leopard server is connected to eight Android devices, for a total of 32 phones per rack,” Reversat explained.
“The phones connect to Wi-Fi via a wireless access point in each rack. These phones lie on a slightly tilted pegboard so mounted cameras can record their screens. Engineers can access these cameras remotely to learn more about how each phone reacts to a code change.”
Engineers have also deployed Chef – a software tool used by Facebook to manage its massive server estates - to automate management of mobile devices. The company now plans to open-source both the racks and its Chef recipes. It is also working on a new rack design that would fit 64 devices, in order to double the density of hardware in its lab.
“We hope that through our open-sourcing both the hardware design of our mobile device testing rack and the Chef recipes written to control the phones, others can benefit from our learnings and contribute their own ideas to our designs,” Reversat said.