Choosing Bandwidth, Latency & Topology for Multi-DC GPU Clusters

This guide walks you through selecting bandwidth, latency targets, and network topology for GPU clusters spread across multiple datacenters. You’ll learn how to match networking choices to your workload’s communication patterns, budget, and fault tolerance needs, and end with a concrete decision framework.

What You’ll Need

Workload profile: model size, parallelism strategy (data, tensor, pipeline), and communication frequency
Number of GPUs per datacenter and total cluster size
Distance between datacenters (or expected round-trip time)
Budget for networking hardware and ongoing WAN costs
Existing network infrastructure and vendor options

Step-by-Step

Quantify your workload’s communication needs. Determine the dominant collective operations (AllReduce, All-to-All) and their frequency. For training, measure the per-step data exchanged per GPU. For inference, note the request size and response latency requirements.
Set a latency budget. For synchronous training, the added latency between datacenters must be a small fraction of a training step. For example, if a step takes 100 ms, a 10 ms inter-DC round-trip adds 10% overhead. For inference, the total response time must meet your SLA. Write down the maximum acceptable inter-DC latency.
Estimate required bandwidth. For each datacenter, calculate the aggregate bandwidth needed for your largest collective. If you have N GPUs per DC and each sends M bytes per step, the DC’s egress bandwidth should be at least N × M / step_time. Add headroom for retransmissions and control traffic (typically 20–30%).
Choose the inter-DC WAN technology. Options include dedicated fiber (dark fiber), leased wavelengths, or VPN over internet. Dark fiber offers the lowest latency and highest bandwidth but is expensive. Leased lines provide predictable performance. Internet VPN is cheap but variable and unsuitable for latency-sensitive training.
Design the intra-DC topology. Within each datacenter, use a high-radix, non-blocking topology like fat-tree or Dragonfly. For clusters under 1,000 GPUs, a two-tier fat-tree with 400Gbps or 800Gbps links is common. For larger clusters, consider Dragonfly to reduce cable count and cost.
Select inter-DC topology. For two datacenters, a simple point-to-point link suffices. For three or more, consider a full mesh (each DC connects to every other) for resilience, or a hub-and-spoke (all traffic goes through a central DC) to reduce cost. Full mesh offers lower latency but more links.
Plan for congestion control and QoS. Implement priority queues for GPU traffic over other data. Use congestion-aware routing protocols (e.g., RoCEv2 with DCQCN) to avoid packet loss. On WAN, consider using TCP with tuned buffers or RDMA over Converged Ethernet (RoCE) if the WAN supports it.
Simulate or prototype. Use network simulators (like NS-3) to test your topology under your workload’s traffic pattern. If possible, run a small-scale test with a few GPUs across two DCs to measure actual latency and bandwidth.
Validate against your latency budget. Measure the round-trip time between DCs using ping and iperf. If it exceeds your budget, consider moving to a closer DC or reducing synchronization frequency.
Document and monitor. Create a network diagram with all links, bandwidths, and latency expectations. Set up monitoring for throughput, packet loss, and latency to detect degradation.

Common Mistakes

Ignoring the impact of inter-DC latency on training convergence. Even 5 ms extra can slow down synchronous training significantly.
Underestimating bandwidth for All-to-All patterns. These can saturate links quickly, especially with large models.
Choosing a topology that is not fault-tolerant. A single WAN link failure can halt the entire cluster.
Forgetting to account for protocol overhead. TCP and RoCE have headers that reduce effective throughput.
Assuming intra-DC and inter-DC bandwidth needs are the same. Inter-DC is often lower but must be sized for peak collective traffic.

Tips

Use a hierarchical approach: high-bandwidth, low-latency intra-DC; moderate-bandwidth, higher-latency inter-DC.
For training, consider asynchronous or semi-asynchronous methods to tolerate latency, but be aware of convergence issues.
Use network telemetry to adjust congestion control parameters dynamically.
Consider using a dedicated WAN for GPU traffic, separate from general office traffic.
When in doubt, over-provision bandwidth by at least 20% to handle spikes.

Troubleshooting

Symptom: Training slows down when using multiple DCs. Fix: Check inter-DC latency; if it’s high, reduce synchronization frequency or use gradient compression.
Symptom: Packet loss on WAN. Fix: Increase TCP buffer sizes, enable QoS, or switch to a provider with better SLAs.
Symptom: Bandwidth is lower than expected. Fix: Test with iperf, check for bottlenecks in the topology, and verify that all links are negotiated at full speed.
Symptom: A DC goes down and the cluster cannot continue. Fix: Design for failover with redundant links and automatic rerouting.
Symptom: Latency spikes during peak hours. Fix: Use dedicated WAN or prioritize GPU traffic with QoS.

Frequently Asked Questions

Q: What is the maximum distance for a multi-DC GPU cluster?
A: There’s no hard limit, but latency increases with distance. For synchronous training, keep round-trip time under 10 ms if possible. For inference, it depends on your SLA.

Q: Can I use InfiniBand over WAN?
A: InfiniBand is designed for short distances. For WAN, use RoCEv2 or TCP. Some providers offer InfiniBand over dedicated fiber, but it’s rare and costly.

Q: How do I decide between a full mesh and hub-and-spoke?
A: If you have 3-4 DCs and need low latency between all, full mesh. If you have many DCs or cost is a concern, hub-and-spoke with a central DC that has high-bandwidth links.

Q: What is the typical bandwidth needed for training a 100B parameter model across 2 DCs?
A: It depends on the parallelism. For data parallelism, you need to exchange gradients each step. A rough estimate: if each GPU sends 1 GB per step and you have 1000 GPUs, that’s 1 TB per step. If your step time is 10 seconds, you need 100 GB/s aggregate.

Q: Should I use a dedicated WAN or the public internet?
A: For production, dedicated WAN is strongly recommended. Public internet has variable latency and packet loss, which can severely degrade performance.

Q: How does topology affect fault tolerance?
A: A full mesh provides multiple paths, so a single link failure doesn’t isolate a DC. Hub-and-spoke has a single point of failure at the hub. Consider redundant links and automatic rerouting.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →