NTT Docomo Business

NTT Docomo Business turned eight separate data centers scattered across 2,000 kilometers into a single Kubernetes cluster—and hit full 100 Gbps wire speed on GPU-to-GPU transfers between cities. 👇

Single urban data centers are hitting hard physical limits on power and land. You cannot just keep packing AI racks into one building. Spreading GPUs across distant physical sites usually destroys the uniform operational model, tenant isolation, and ultra-low latency required for large-scale AI workloads.

NTT Docomo Business solved this by building a distributed GPU fabric that operates eight data centers from Sapporo to Fukuoka as one multi-tenant Kubernetes cluster.

They started by stitching all eight remote locations into a single Layer 2 network domain using an IOWN All-Photonics Network (APN), which provided a 100 Gbps, low-latency optical connection across Japan.

Instead of leasing bare-metal servers, they deployed Petasus AI Cloud—a KubeVirt-based platform—on top of the fabric. KubeVirt abstracts heterogeneous GPU hardware across all eight sites, packaging them into securely isolated virtual machines that tenants can provision on demand.

To achieve near-local interconnect performance over long distances, they tuned NCCL all-reduce collectives directly inside the KubeVirt GPU VMs running over the optical network.

Moving from bare-metal provisioning to KubeVirt VM delivery slashed environment setup and cleanup times from 5 business days down to tens of minutes.

In performance testing between Tokyo and Osaka, cross-site GPU-to-GPU communication inside the KubeVirt VMs hit the 100 Gbps wire-rate limit, reaching a bus bandwidth of roughly 12.9 GB/s.