They let me take apart a SUPER COMPUTER

Quick Overview

The Simon Fraser University (SFU) 'Fir' supercomputer, built for science and AI research, is a massive liquid-cooled system featuring 640 Nvidia H100 GPUs, 165,000 CPU cores, and 1 Petabyte of RAM, costing an estimated $82 million, with its cooling infrastructure requiring significant upgrades to handle the 70,000 watts per rack and 33,000 gallons per minute of cooling capacity.

Key Points: The 'Fir' supercomputer utilizes 640 Nvidia H100 SXM5 GPUs, each providing 3.36 TB/s of bandwidth, totaling immense computational power. The system includes 165,000 CPU cores and 1 Petabyte of DDR5 RAM, supported by 12-channel memory per node. The total estimated budget for the supercomputer was over $82 million, with the GPUs alone costing around $20 million. Cooling is handled by a direct-to-chip liquid cooling system utilizing copper cold plates for CPUs and GPUs, capable of removing up to 70,000 watts per rack. The facility employs three large evaporative cooling towers outside to manage the heat load, which can exceed 33 degrees Celsius in the summer. The liquid cooling system uses specialized plumbing with a dual-loop setup, featuring Grundfos pumps capable of moving 1,500 gallons per minute of coolant. Security measures include biometric locks on server racks and extensive camera coverage throughout the data center.

Context: This video provides an in-depth look at the 'Fir' supercomputer located at Simon Fraser University (SFU) in British Columbia, Canada. The presenter explores the physical infrastructure, focusing heavily on the dense computational hardware, specifically the Nvidia H100 GPUs, and the complex, high-capacity direct-to-chip liquid cooling system required to manage the immense heat generated by the system.

Detailed Analysis

The video tours the SFU 'Fir' supercomputer, which houses 640 Nvidia H100 SXM5 GPUs and 165,000 CPU cores, supported by 1 Petabyte of RAM, all interconnected via 200Gb connections. The immense power density, requiring 70,000 watts per rack, necessitates a sophisticated direct-to-chip liquid cooling system. This system uses copper cold plates on the CPUs and GPUs, with coolant flowing through complex manifolds. The presenter shows a single node containing 8 Zen 4-based CPUs and 1.5 TB of RAM per node. The cooling infrastructure outside consists of three large evaporative cooling towers capable of handling the heat load, even when ambient temperatures reach 33 degrees Celsius. The internal cooling distribution relies on a complex piping system with two loops, managed by Vertiv Liebert XDU units and augmented by mechanical chillers when external temperatures are too high. The entire installation is secured with biometric scanners on the racks.