Elsewhere, I have argued that despite all the buzz about micro-data centers at the base of cell towers, the first place “The Edge” will most likely materialize is inside traditional data centers located within secondary “NFL cities”. The incremental improvements in latency that come from deploying content and workloads locally in these large population centers (as opposed to distant cloud data centers), coupled with the ability of full-sized data centers to better scale and maintain infrastructure, actually make this approach far superior to alternatives like micro-data centers.

But while this thesis addresses the latency considerations and benefits of data center location, we all know that the effective delivery of bits depends on another critical element in the infrastructure chain: the network behind the data center. The connectivity available in a building, and the paths from which it enters and exits the property, are just as important as the redundancy and configuration of the mechanical, electrical and plumbing (MEP) systems in the facility. A weak link in this part of the chain can negate all the benefits of the data center’s proximity and redundancy to users in that metropolitan region.

Compare inside to outside

Let me propose an analogy. Everyone knows that best practices dictate that a properly designed data center must have a minimum of two Points of Entry (POEs) for fiber, and a minimum of two Meet Me Rooms (MMRs) to terminate the fiber and distribute it to IT workloads. This ensures the proper degree of physical diversity both within and outside the data center.

And what is true for a data center needs to be true for a geographic market. If we imagine that interconnection data centers are effectively the “meet me rooms” of a metro area, and the long-haul fiber paths into and out of those data centers are the “Points of Entry” for that market, it stands to reason that on this macro scale we need the same level of diversity. Within a given metropolitan market, redundant interconnection locations and diverse long-haul fiber access must be available in order to provide the market level diversity our industry is accustomed to at the micro-level. As we will see, that’s not always the case, and it has the potential to slow the development of the edge for all providers and users.

In this article, we will explore the current problem with network interconnection and fiber redundancy in secondary markets. I will argue that multiple interconnection points and physically diverse long-haul fiber connectivity options must emerge for mission-critical deployments to flourish at the Edge.

Tier 1 cities have peering points

Let’s start by looking at how most large Tier 1 markets are configured for data center interconnection and long-haul fiber. Figure 1 depicts Zayo's fiber map for metro NYC. While New York is unique in many ways the broad configuration is the same in all Tier 1 markets. The three orange dots are the three main “carrier hotels” in the New York area – 60 Hudson St., 111 8th Ave., and 165 Halsey St. in Newark, NJ. In a market such as NYC, there are other significant interconnection points (32 Ave of Americas, 325 Hudson, 75 Broad, etc.) but most people would agree that these three locations form the “core” of peering and interconnection for the New York Metro market.

The reason there are multiple locations is obvious: redundancy. With three locations, users have the ability to deploy in multiple locations to ensure uptime. A failure in one facility or portion of a facility will not impact traffic in another location. Similarly, each of these locations benefits from dozens of metro and long-haul fiber options that exit the New York area heading north, west, east, northwest and southwest from Manhattan. With multiple interconnection points and multiple routes coming into/out of the area, a provider can carefully construct a resilient architecture for connectivity and availability.

When you survey the landscape, all major Tier 1 markets also have 2-3 major interconnection points and diverse metro and long-haul fiber connectivity. Dallas (my new home) has three main locations – 1950 Stemmons (the Infomart bulding), 2323 Bryan and 400 South Akard (DataBank’s DFW1 location). Long-haul fiber enters the market from multiple, physically separate routes.

Chicago has 350 E. Cermak and 600 S. Federal. From each of these facilities or by tethering to a sub-tending data center in the same market, users can avail themselves of diverse data center locations and redundant metro fiber and long-haul fiber options to ensure that no one location can become a “single point of failure”.

Large cloud, content and technology providers understand this. DE-CIX is one of the world’s leading interconnection platforms. Until this year, its only location in Dallas was at the Infomart Building at 1950 Stemmons. As traffic grew, DE-CIX decided to create a secondary location that offers its customers an alternative peering point for diversity. In August, DE-CIX deployed its fully redundant Apollon platform at DataBank's 400 South Akard location.

In talking to the DE-CIX folks, the two main criteria for their selection were: (a) a robust data center location physically diverse from Stemmons with good fiber connectivity; (b) available long-haul fiber routes fully independent from Stemmons. South Akard checked both boxes.

Tier 2 cities don't

The situation is not the same in most secondary markets. In many of these markets multiple, diverse interconnection locations and physically diverse long-haul fiber routes have not yet developed. Figure 2 shows the situation in Minneapolis. In that market, the Cologix facility located at 511 Main Street is the predominant carrier interconnection point. To exit the market, the main long-haul fiber routes go either south to Chicago or west to Seattle (we show the Zayo routes in this figure). A third route heads due south towards Omaha.

Most of the peering in the Minneapolis market happens within the 511 building. Today, there does not exist another interconnection point within the market that has both direct access to all the diverse long-haul fiber routes and enough critical mass of local metro fiber providers to be a viable alternative location. For Minneapolis, the 511 building is the proverbial single point of failure and a bottleneck that must be addressed for the region to support fully diverse edge workloads.

New peering points

Figure 3 depicts proposed new routes for long-haul dark fiber connectivity out of the Minneapolis market. Zayo will extend its Omaha long-haul route to terminate at DataBank’s MSP1 facility and its Chicago route to terminate at DataBank’s MSP2, locations which are already linked by Zayo dark fiber [Zayo is the largest US fiber operator, and was recently bought by investors who also back DataBank - Editor].

The key element of this route is that it bypasses the 511 Main location heading West, while also providing long-haul access to Chicago and Omaha via metro fiber splices. This provides an alternative interconnection facility with diverse long-haul dark fiber options, giving Minneapolis a resilient internet infrastructure, making it a viable Edge market.

For Edge markets like Minneapolis to fully develop, redundant interconnection and long-haul fiber options must emerge, so that providers of scaled, mission-critical applications can build the redundant architectures they have come to rely on in Tier 1 markets. With the exponential growth in internet traffic, large content providers and ISPs are increasingly looking to “regionalize” their traffic flows and drain traffic from Tier 1 markets into adjacent metropolitan areas. However such users are reluctant to expand their footprint without the availability of diverse options for colocation, peering and dark fiber transit.

Diversity of options also provides another non-technical benefit: competition. As the importance of internet infrastructure grows, so does the importance of truly neutral interconnection locations. Yet we continue to hear stories of aggressive incumbent behavior, incessant price increases and anti-competitive terms of supposedly “neutral” operators of interconnection facilities. We believe that owning/controlling the dominant interconnection point in a market is not an excuse to abuse your customers, but an opportunity to foster and build a better internet, which will ultimately benefit all users and consumers of internet infrastructure.