As demand skyrockets, architectures within the data center are becoming much more complex – particularly at training-cluster scale.

This is according to global manufacturer AFL. The company has been considering what actually breaks when you scale from 16K to 128K – the point where fiber count, not just power or silicon, becomes a design-limiting variable. Senior technical advisor at AFL, Alan Keizer, explains why single-building thinking needs to be replaced.

“Simply organizing the design, sourcing, supply, and installation of that many components is a challenge,” he says. “We have to ask: how do you fan out from one device to 128,000 other devices? How do you design the networks physically? How do you deal with the density? How do you deal with the bandwidth?”

He adds: “All of those things have to be pulled together, and they’re all being pushed forward at the same time. It becomes a large, interesting technical challenge.”

Growing pains

AFL’s published reference designs confirm that the 128K architecture is not simply a scaled-up version of the 16K architecture, but rather, its fabric and deployment scope expands dramatically. In the 16K cluster, the node was a chassis, with each rack containing a stack of chassis. In the 128K cluster, the rack itself is the node.

What breaks is the building. Keizer explains how current 512-lane switch silicon can build a two-tier fabric reaching about 131,000 endpoints per plane, so a 129,024-accelerator cluster would fit. In a two-tier fabric every first-tier switch must reach every second-tier switch, so a building wall cuts straight through the densest part of the mesh. A third tier confines that mesh to each building and leaves only the sparser upper segment to cross between them.

The fiber count follows. A 16K hall carries roughly a quarter of a million fibers in its scale-out fabric. The 128K design carries about two million at the accelerator face and 6.2 million across three tiers.

“That is roughly 24 times the fiber for about eight times the accelerators: 8 times the endpoints, 2 times the fibers per accelerator, 1.5 times the tiers,” Keizer adds.

As accelerator counts scale from thousands to hundreds of thousands, AI architectures inevitably evolve too.

“The first thing about model training is that it’s a synchronous process. Over the last decade of training, it’s been consistently found that the bigger the model, the more data and training you have, meaning the better the answers become,” Keizer says.

Over time, it’s been found that faster processors with more local memory, more processors connected together, longer training, and more data all lead to better results.

However, stacking these things together is no small feat.

“In model training, all of the processors work together – each one has a task and shares what it’s learnt with everything else,” Keizer explains. “This collectivism requires all the accelerators to be in a single, coherent, time-synchronous domain.

“The bigger the fabric, the harder it is to make that happen. That means you need complex fabrics – multi-tier stacks of switches – to enable connection in a synchronized manner without blocking and failures. It’s a big networking challenge.”

This leads to the larger challenge of how you tie everything together and, with large and complex fabrics, scale introduces another layer of complexity. This is particularly important when it comes to AI server racks, in terms of power delivery and cooling challenges, as they need significant cooling.

Fiber and its limitations

Scaling accelerators in fiber architectures comes with a range of challenges, particularly as they become more critical for AI training environments. This is being driven simultaneously by larger demand on the network and the fact that training clusters are increasingly interconnected – meaning that, if any piece stops, the whole cluster stalls and has to restart from its last checkpoint.

“The bigger the cluster, the more critical it is that everything works together,” Keizer says. “Any marginal connection that leads to ‘flapping’ (a link that repeatedly drops and re-establishes) can cause issues. It can occur because of hardware difficulties in the transceiver, but it can also be because of marginal links, loss due to contamination or damaged fiber, or tight bends.”

Density makes the equation even more complicated. It means everything needs to be balanced logically and with tight loss budgets.

“That’s why we try to approach things in a structured and systematic way when we undertake the designs,” Keizer adds.

There are several ways in which to scale from 16K to 128K, many of which are very technical. Hundreds of thousands to millions of fibers require dense yet manageable organization, which can lead to very high fiber-count cabling in the whitespace. With connectors shrinking into smaller and denser components, management has become a key challenge.

At scale, installations can become harder to plan and execute, particularly if skilled labor is hard to come by.

“Fiber is extremely robust as long as it’s not abused, but it’s very sensitive to dirt contamination,” Keizer explains. “It’s common that project planners put walls up and, literally before the dust settles, want to install the cabling. The building needs to be secured properly before fit-outs start.”

To prevent this, managing installation must be carried out skillfully and carefully, but it’s a huge challenge. Ongoing maintenance, troubleshooting, moves, adds, and changes all come into play with high-fiber connectivity requirements.

For Keizer, these are systems engineering problems:

“System engineering implies that there are multiple piece parts, and multiple engineering considerations that have to be managed simultaneously,” he says. “This is true for data centers broadly, but for connectivity the systems have to be holistically designed – not just ‘how many fibers can I get in this panel’ – if you can’t build and maintain it, it’s not useful.”

The arithmetic behind the solution

AFL feels strongly about its responsibility to expose the arithmetic behind fiber architecture solutions. Given that there aren’t many super-large AI data center clusters, each one is its own beast – built by particular organizations with their particular history, methodology, sites, and vendors.

“We’ve found that many people in the field have solutions, but there’s no discussion over how the designs were derived,” Keizer says. “There is no single answer.”

In response, AFL has sought to show how it develops its own designs. Its recent whitepaper, Architecting AI Infrastructure: Building AI Training Clusters at 128K Accelerators, which extends the 16K reference design in its Building AI Training Clusters at 16K Accelerators whitepaper, seeks to help its client base improve their understanding to support the industry as a whole in moving forward.

“We’re all better off when the people we work with have better knowledge,” Keizer adds. “We’ve tried to show how we derived reference designs and what the physical and numerical considerations are that led us to it.”

One of its solutions is its method for cluster-scale fiber design, which holds across the step from a two-tier to a three-tier fabric – a particularly important consideration as cluster scale continues to grow. First, the company looks at the connectivity in each situation and asks: what is it really doing? The method splits the plant into three layers: permanent, application-agnostic backbone trunks; zone cabling that adapts to the topology and changes with the compute generation; and equipment cords that make the final connection to each device.

Keizer explains that AFL will find functional ways of expressing what needs to be done in a way that is as generic as possible:

“It’s beneficial to have a structured way of thinking about this, so there is less disruption with change,” he says. “Breaking the cabling into layers helps us think more clearly and, if you do it right, the trunk cabling probably survives a couple of generations.”

Given the infancy of the AI boom, there haven’t been many forklift upgrades – where everything has to be torn out and started from scratch.

“While connectivity is a relatively small portion of the capital cost of a facility, trunk cabling has a much bigger impact on how much work is necessary and how practical generational upgrades are without disruption – so it’s very important to get it right,” Keizer notes.

“Connectivity will be upgraded over time, but if you plan well, lay down more fibers than you need, and make sure your pathways can be added to, you’ll be able to live through several generations in the data center with minimal disruption.”

The reusable method that AFL adopts comes as the AI conversation continues to shift from training to inference. Having a proven method can make upgrades more manageable, Keizer notes, as it offers greater flexibility.

“Ultimately, the world will not end if you have to take out that trunk, but it’ll be much harder to do that as a generational update in a running facility,” he says. “We all need to think about what’s sensible, what allows things to work better, which will lead suppliers to more successful business.”

Scale across, and the limits of site size

Making clusters bigger is always possible but it can be tricky and inefficient, leading some developers to create ways to break larger training tasks into smaller chunks instead of having one monolithic network integration.

This is driving ‘scale-across’ – linking two or more coherent training domains into one distributed workload. A large amount of data is exchanged periodically and the clusters update each other on how they’re doing.

“If one cluster has a hiccup, that doesn’t stop the others, and it’s scalable: you can add another one without having to redo the whole fabric,” Keizer notes.

There are also practical considerations like how much power can actually be delivered to a site through multiple backup connections, which is why behind-the-meter (BTM) power generation has become a popular topic this year. Given the limits on where you can put a site, with long power procurement lead times, permitting challenges, and public opposition to data centers, this could be a more practical way to move forward.

“Sites simply are not going to be infinitely big – scale‑across is the answer to that,” Keizer adds. “We’re in a gold‑rush mode where people are racing out to capture territory, get market share, and get the best model.”

Giving connectivity a seat at the table

Keizer suggests inference data centers should be the bulk of what gets built moving forward.

“Training is investment, but inference is earning money,” he says.

Inference is not completed one query at a time, but rather by batching several queries with similar characteristics that require the same model. What requires consideration is how much concurrency a company’s business model needs, how many users, and how much infrastructure costs to buy and run. These decisions are based on the nature of the individual business and frame how much computing is needed.

Keizer argues that connectivity needs to be more of a priority in the data center conversation moving forward, alongside compute, networking, power and cooling, to make processes more seamless.

“There’s a huge amount of change within our individual silos, all pushing the limits and continually advancing, meaning everything is getting more highly engineered,” he says. “Let’s all talk from day one – don’t call us at the end, it’s too late.”

Download the whitepaper, Architecting AI Infrastructure: Building AI Training Clusters at 128K Accelerators, here.

New to the series? Start with Building AI Training Clusters at 16K Accelerators here.