With Covid-19-related shelter-in-place orders in effect in nearly every corner of the world, the high-tech industry is coming to grips with new ways of working in the shadow of a pandemic. Will this be the norm going forward? We certainly hope not, but there is no doubt that Business Continuity Plans (BCP) need to expand to include the pandemic scenario and this will likely mean that test lab automation will become an essential part of these plans.

For many of us, the shelter in place means working from home using email, PC- or SaaS-based productivity applications and collaboration tools like Zoom or Webex. However, for those that run test labs for functional, regression and performance testing, the situation is not as straightforward. The typical network, security, or mobile test lab environment will have multiple devices under test (DUT), along with multiple testing tools and different test cases that require reconfiguration of the layer 1 topology that interconnects all of these elements.

In the majority of labs today, this is done manually, requiring one or more test engineers to be physically present to recable and reconfigure the lab and, often, to execute some of the tests. When events conspire to prevent these test engineers from physically entering the lab, new software releases, products and services might be delayed, resulting in revenue impacts and missed customer expectations.

Approaches to test lab automation

There are various approaches to test lab automation. Going back a decade or more the traditional approach has been to leverage customer built layer 1 matrix switches. While the approach is functional it does have some drawbacks. These custom switches can be quite expensive, limited in scale to a single chassis, cannot provide media conversion and also cannot extend layer 1 links across multiple labs.

A more modern approach leverages white box ethernet switches running a Linux-based operating system with the right set of features required to implement a layer 1 switch matrix. With this approach a lab engineer can deploy such a solution as a single switch only or assembled into a layer 1 fabric consisting of multiple interconnected switches. This approach is obviously extremely cost effective because it is based on mass produced and commoditized white box switches. In addition, this approach can provide media conversion, e.g. from optical to copper, N:1 forwarding, e.g. 10G into 10 x 1G and offers a scale out architecture that can grow nearly infinitely in port count. Furthermore, depending on the implementation leveraged by the operating system software vendor, this sort of white box solution might offer extension of layer 1 links across multiple labs, dynamic load balancing and rapid reconvergence upon a link failure along with a number of other benefits. Finally, again depending upon implementation, the solution can provide fabric-wide flow, packet, link-layer visibility to accelerate troubleshooting of physical layer issues as well as applications.

In terms of the user interface for managing these switches, vendors will typically provide a command line interface (CLI) option and often a GUI based tool as well. Additionally, these solutions can be integrated into third party lab orchestration systems from vendors like Quali.

Once this system is deployed and the initial cabling is done, test lab engineers can now access the lab remotely and reconfigure the lab topology from anywhere in the world 24 x 7. This ensures that regression, performance and security testing can continue so that hardware and software product releases can maintain their schedule, providing a key pillar of business continuity.

Beyond business continuity

The value doesn’t stop at business continuity, however. There are a number of other important benefits that come from deploying a lab automation solution.

• Engineers conserve time, maximize productivity, reduce costs and eliminate errors associated with manual patching, thus accelerating the release of new products and services.
• Depending on the implementation, the solution can be deployed inside a single lab or across multiple, geographically dispersed locations, unifying disparate lab sites and processes.
• Expensive test generator ports can be easily shared between teams and tests, even across multiple sites, improving utilization and increasing ROI.
• Users can reduce testing time by storing well-known physical configurations and reproducing the configurations at will for subsequent testing.

Enabling test lab automation provides an organization with the opportunity to test devices, applications and services faster, and to establish 24/7 operations from anywhere in the world. Test lab automation can ensure business continuity and also greatly accelerate test procedures such as regression, security and performance tests.