AI data centers: Many architectures, one need for fit-for-purpose connectivity

AI infrastructure is evolving quickly. Model types, accelerator platforms, network fabrics, cooling methods, and site strategies now vary widely. That matters because operators are not building the same thing.

Each has different objectives and constraints: training versus inference mix, latency targets, resilience requirements, deployment timelines, capital structure, power availability, and geographic footprint. In this environment, optical connectivity becomes one of the strategic pillars of AI infrastructure. Like power and cooling, it has to be designed to the requirement. One size does not fit all.

This opinion piece explores why fit-for-purpose, adaptable connectivity is critical to efficient data center design, and builds on the themes outlined in AFL’s recent blog, Hyperscale Market Shifts: AI, Neoclouds, and the New Limits of Data Centers.

Why the AI stack is becoming more varied

AI use has moved from experiment to habit. Usage keeps growing in both consumer and enterprise settings. Model design has also diversified. Some workloads are dominated by large training runs. Others are dominated by inference at scale. Agentic systems add a different pattern again (e.g., long-lived sessions, many tool calls). From an infrastructure standpoint, that tends to increase sustained utilisation of accelerators and networks.

Hardware has diversified alongside software. GPUs remain central, but TPUs and other accelerators are meaningful in some environments, and the balance between compute, memory capacity, memory bandwidth, and interconnect varies by workload. The practical consequence is that data center design choices that were once standard are now more conditional.

Rack scale and the changing shape of risk

Rack power density has increased sharply, especially in rack-scale systems designed as coherent compute domains. A rack can represent a significant fraction of a pod’s capacity. When that happens, the operational impact of faults changes. Losing a rack is not a nuisance; it is a major event. The same is true of any failure that is hard to isolate or slow to repair.

Connectivity plays directly into this risk profile:

• More links means more potential fault points
• Higher density raises the importance of routing discipline, panel layout, and access for service
• Shorter deployment windows reduce tolerance for rework and ambiguous documentation
• Bigger failure domains make maintenance paths and redundancy planning more consequential

This is an observation that the cost of ‘messy’ systems is rising, because the systems they support are larger and more economically critical.

Scale-up, scale-out, and why topology choices diverge

AI clusters now span both scale-up and scale-out networks, and the balance between them varies by design philosophy and workload.

Scale-up inside the rack or pod continues to push high bandwidth and low latency within a compute domain. As speeds rise, electrical reach shrinks, and optics tends to move closer to the compute elements.

Scale-out across racks is dominated by high-speed fabrics (commonly Ethernet or InfiniBand) at 400G and 800G per port, with faster generations in development. Every inter-rack hop is an optical link, and the number of links per rack and per pod scales quickly with cluster size.

Different operators make different calls on oversubscription, pod sizing, the unit of deployment, and how to partition failure domains. Those choices drive very different connectivity designs: fiber counts, connector density, patch-field architecture, and the balance between structured cabling and more direct point-to-point runs.

This is a key reason ‘standard product lists’ are not enough. The same connector and cable families can be used in many ways, but the system design has to reflect the operator’s priorities.

DCI: No longer someone else’s problem

AI also increases the importance of connectivity between data centers. Training data must move. Checkpoints and replicas must be protected. Inference often runs across regions for latency, resilience, and capacity balancing. As a result, data center interconnect (DCI) is scaling, with operators planning for multi-terabit campus capacity and wide-area links that support both throughput and operational resilience.

This reinforces a simple point: the AI infrastructure story is not confined to a single room or building. The ‘shape’ of the network increasingly includes campus, metro, and regional connectivity.

Power and siting: Constraints that force divergence

Power availability has become a gating factor in many markets. Grid connection timelines can be long. Community pressure around water use, noise, land use, and emissions can delay or reshape projects.

Operators are responding in different ways:

• Building in power-rich regions rather than traditional hubs
• Pursuing behind-the-meter generation in some cases
• Using modular deployment approaches to reduce onsite complexity and compress schedules

These moves shift campus geography and make reliable long-haul and metro connectivity more valuable. They also change operational assumptions. A remote campus cannot depend on the same density of specialized staff or rapid vendor interventions as a core metro site. Systems need to be simpler to operate and faster to troubleshoot.

Neoclouds raise the premium on speed and repeatability

Neoclouds and other GPU-as-a-Service providers add another design vector. Many are highly focused on utilisation of expensive assets and on rapid deployment of capacity.

For connectivity teams, the implications tend to be consistent:

• Sites may be remote from major peering hubs, increasing dependence on well-designed DCI and long-haul fiber routes
• Operators prefer repeatable GPU pods and consistent structured cabling patterns that scale
• Downtime and messy change work are commercially painful, which pushes toward cleaner maintenance paths, clearer failure domains, and better documentation

Again, this does not produce one architecture. It produces a family of architectures that share a bias toward repeatability and speed.

The practical direction: More factory-built, more proven, less rework

Across hyperscalers, neoclouds, and specialized AI operators, one trend is hard to miss: more work is being moved into controlled environments.

Racks arrive integrated. Pods arrive as known units with defined power and network envelopes. Electrical skids and power rooms are prefabricated. ‘Time to first token’ is becoming a competitive metric.

Connectivity has to match that reality. The winners will be connectivity systems that are:

• Dense but serviceable – designed for access, not just packing factor
• Repeatable – standard blocks that can be deployed many times
• Proven – inspection discipline, and documentation that survives handoffs
• Compatible with factory workflows – pre-terminated assemblies and predictable integration steps
• Designed for change – expansion paths that do not degrade order and legibility

This is where optical connectivity becomes clearly strategic. It is not a passive commodity. It is a system that determines how quickly a site can be commissioned, how reliably it can run, and how safely it can be expanded.

Conclusion: Fit-for-purpose beats one size fits all

AI is multiplying the number of valid infrastructure choices. Operators will continue to diverge because their objectives and constraints differ. That makes the old assumption (one template, repeated everywhere) less and less realistic.

In this environment, optical connectivity should be treated as a first-order design pillar alongside power and cooling. It must be tailored to the workload, the topology, the operational model, and the deployment cadence. One size does not fit all.

From AFL’s perspective, our goal is to help operators make those choices deliberately: selecting the right connectivity architecture for their constraints and delivering it in a way that supports fast deployment, predictable performance, and clean expansion.