TPUv4 Interconnect Cuts Cost, Power

Google employs optical interconnect when building a 4,096-node AI supercomputer featuring its TPUv4 accelerator chip. The result is much higher potential performance coupled with a sharp drop in carbon emissions.

Bryon Moyer

Google’s latest AI tensor-processing chip, TPUv4, quadruples the size of the computing block for both AI training and inference as compared with the prior generation. Optical interconnect enables the larger size as well as higher performance and performance efficiency.

The giant hyperscaler built a supercomputer of 4,096 interconnected TPUv4 nodes. Arranged in a 3D torus, the topology balances the performance needs of older and newer algorithms. Running signals in both directions along a fiber reduces cost, halving the number of fibers in the system. Overall, the interconnect reduces latency and power relative to prior TPU versions.

Although Google deployed the technology in 2020, it’s only now disclosing details in a paper at the 2023 International Symposium on Computer Architecture (ISCA). The paper describes a rack-size 4x4x4 cube of TPUv4 nodes as a building block. Electrical connections serve within the rack; optical links connect nodes from different racks. Optical circuit switches create end-to-end optical links with no need for packet manipulations along the way.

Overall performance of the resulting supercomputer is at least 10x that of TPUv3’s smaller supercomputer. Energy consumption drops by as much as 83%, with an estimated drop in carbon emissions (CO2e) of 95% per node.