A new version of Facebook’s 100Gb top-of-rack data center switch, the Wedge 100, has been accepted by the Open Compute Project (OCP) and delivered to the wider world as a product from Edgecore Networks.
The Wedge 100 is a 32 port 100GBE switch, built by Accton Technology to Facebook’s design, for use within Facebook’s data centers. The design has been shared, and accepted by the Open Compute Project, and meanwhile Accton is making a commercial version of the switch, the Wedge 100-32X, which is being sold through Accton’s subsidiary Edgecore.
Moving to 100G
“Our strategy is to build 100G data centers, and Wedge 100, our second generation top-of-rack network switch, is one of the key components helping us achieve that goal,” said a blog post by Facebook software engineers. “At Facebook, we use Wedge 100 in our production environments, and we continue to deploy it at scale across our data centers. It is a component of our 100G data center network strategy, while also allowing us to maintain backward compatibility with existing 40G devices.”
The switch joins a growing range of Open Compute compliant switches, designed to simplify data center networking and allow large data centers to specify efficient low-cost open source hardware. Since OCP was set up by Facebook in 2011, it has been joined by large web-scale cloud companies including Apple in 2015 and Google in 2016, while delivering specifications for servers and switches used in data centers.
The Wedge 100 uses much of the same hardware as its predecessor, the Wedge 40, and runs the same FBOSS software, extended to support the the 100G ASIC chips and optics in the new model. It includes a compact microserver built to the industry standard in the standard 95mm x 95mm computer-on-module COM-Express Type 6 form factor, but also supports larger standard 95mm x 125mm size modules for more advanced CPUs.
To make life easier for Facebook’s busy technicians, the switch is more serviceable: its top can be popped off without tools, and hot-pluggable fan trays can likewise be removed without tools, using a clip instead of a thumb screw. There’s now a status LED for those fan trays.
Compatible with 19in racks
Facebook uses the OCP Open Rack standard in its data centers, and powers devices through an efficient 12V bus bar distribution system instead of using separate power supply units (PSUs). However, acknowledging the wider world, Facebook and Accton built the Wedge 100 in a standard-sized 19in box, and then added a 21in tray (shown above) to install it in the Open Racks. The system can optionally be built with its own standard PSU, instead of using the OCP bus bar.
The biggest physical hardware upgrade was to provide the airflow needed by 100G optics, which run hotter, the post says.
Software hasn’t changed much with Facebook’s open source FBOSS (available on GitHub) running on the new switch, so Facebook put it into production use very quickly, and then made updates to handle the fact that this switch is expected to be more versatile than its predecessor, the blog post says.
“We wanted to drop a Wedge 100 at any location in our network that had a Wedge 40,” the post explains. “Downlink speeds and cable types can vary. Uplink speeds can vary between old and new clusters. Correct configuration is especially important for 100G links as they have a much smaller optical power margin and can present more challenges in establishing link and optimizing performance, requiring much more careful configuration than 40G did.”
Alongside the Edgecore version from Accton, other companies want to get involved, Facebook says. Transceiver vendors will be qualifying their modules on the platform, and testing them at the University of New Hampshire InterOperability Laboratory (UNH-IOL), which has been leading OCP testing for more than a year.
Software companies will be putting modules on the Wedge 100 platform, Facebook promises, with Big Switch Networks’ Open Network Layer (ONL) already available on the operating system layer, on SnapRoute, FRINX and Apstra in the upper parts of the stack. Canonical will put its Ubuntu Core operating system on the switch.
SnapRoute announced the availability of FlexSwitch software on top of the Wedge 100 platform. FlexSwitch is an open source L2/L3 networking stack that can be run across multiple hardware platforms, offering a customizable control plane. FRINX integrates the OpenDaylight SDN platform on the switch.
The commercial Wedge 100-32X hardware switch from Edgecore includes a three-year warranty and is shipped with diagnostics, OpenBMC firmware, and the Open Network Install Environment universal NOS loader.
Did Apple lose a network team over OCP?
Reporting on this launch, Business Insider reports a story which circulates in OCP circles, which illustrates the power of OCP. Apparently a couple of years back, Apple’s networking team wanted to join OCP and use the shared expertise of the Open Compute community to build a reliable network. Apple refused, and its network team left en masse, to form one of those companies now putting software on the Wedge - SnapRoute.
Shortly thereafter, the story goes, Apple caved in and joined OCP itself.