# They let me take apart a SUPER COMPUTER

Source: https://www.youtube.com/watch?v=U4Sqh-uJ2NA
Recap page: https://rapidrecap.app/video/U4Sqh-uJ2NA
Generated: 2025-10-16T17:32:18.806+00:00

---
## Quick Overview

The Simon Fraser University (SFU) 'Fir' supercomputer, built for science and AI research, is a massive liquid-cooled system featuring 640 Nvidia H100 GPUs, 165,000 CPU cores, and 1 Petabyte of RAM, costing an estimated $82 million, with its cooling infrastructure requiring significant upgrades to handle the 70,000 watts per rack and 33,000 gallons per minute of cooling capacity.

**Key Points:**
- The 'Fir' supercomputer utilizes 640 Nvidia H100 SXM5 GPUs, each providing 3.36 TB/s of bandwidth, totaling immense computational power.
- The system includes 165,000 CPU cores and 1 Petabyte of DDR5 RAM, supported by 12-channel memory per node.
- The total estimated budget for the supercomputer was over $82 million, with the GPUs alone costing around $20 million.
- Cooling is handled by a direct-to-chip liquid cooling system utilizing copper cold plates for CPUs and GPUs, capable of removing up to 70,000 watts per rack.
- The facility employs three large evaporative cooling towers outside to manage the heat load, which can exceed 33 degrees Celsius in the summer.
- The liquid cooling system uses specialized plumbing with a dual-loop setup, featuring Grundfos pumps capable of moving 1,500 gallons per minute of coolant.
- Security measures include biometric locks on server racks and extensive camera coverage throughout the data center.

![Screenshot at 00:05: A close-up shot reveals the interior of a server node, highlighting the complex copper piping of the direct-to-chip liquid cooling solution covering the processors and memory modules, emphasizing the high-density cooling architecture.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-00-05.png)

**Context:** This video provides an in-depth look at the 'Fir' supercomputer located at Simon Fraser University (SFU) in British Columbia, Canada. The presenter explores the physical infrastructure, focusing heavily on the dense computational hardware, specifically the Nvidia H100 GPUs, and the complex, high-capacity direct-to-chip liquid cooling system required to manage the immense heat generated by the system.

## Detailed Analysis

The video tours the SFU 'Fir' supercomputer, which houses 640 Nvidia H100 SXM5 GPUs and 165,000 CPU cores, supported by 1 Petabyte of RAM, all interconnected via 200Gb connections. The immense power density, requiring 70,000 watts per rack, necessitates a sophisticated direct-to-chip liquid cooling system. This system uses copper cold plates on the CPUs and GPUs, with coolant flowing through complex manifolds. The presenter shows a single node containing 8 Zen 4-based CPUs and 1.5 TB of RAM per node. The cooling infrastructure outside consists of three large evaporative cooling towers capable of handling the heat load, even when ambient temperatures reach 33 degrees Celsius. The internal cooling distribution relies on a complex piping system with two loops, managed by Vertiv Liebert XDU units and augmented by mechanical chillers when external temperatures are too high. The entire installation is secured with biometric scanners on the racks.

### Supercomputer Specifications

- 640 Nvidia H100 GPUs
- 165,000 CPU cores
- 1 Petabyte of DDR5 RAM
- 384 cores per one U node
- 3.36 TB/s bandwidth per GPU

### Cooling System Overview

- Direct-to-chip liquid cooling using copper cold plates
- Dual-loop coolant system
- 70,000 watts per rack handled
- Evaporative cooling towers outside with 1,500 GPM capacity

### Component Deep Dive

- Server nodes feature dual Zen 4 CPUs and 192GB RAM per node
- GPU cooling plates handle 700W each
- SSDs use NVMe for high throughput

### Power and Infrastructure

- Utilizes 48V DC-to-DC power supply for GPUs
- Cooling infrastructure includes two mechanical chillers for ambient temperatures above 33C

### Security Measures

- Biometric locks on server racks for access control
- Extensive camera coverage throughout the facility

### Sponsors and Acknowledgements

- Vessi (waterproof footwear giveaway)
- Lenovo, DDN, and Vertiv acknowledged for contributions

![Screenshot at 00:01: The initial reveal of the supercomputer's scale: 165 thousand CPU cores and $20 million worth of GPU power.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-00-01.png)
![Screenshot at 00:06: Close-up on a server module showing the dense array of RAM sticks surrounding the copper cooling plates for the processors.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-00-06.png)
![Screenshot at 00:23: A technician carefully removing the top panel of a Lenovo SD665-N V3 server node, exposing the complex internal liquid cooling system.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-00-23.png)
![Screenshot at 01:41: Graphic overlay showing the scale of the GPU count: 640 Nvidia H100 80GB GPUs.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-01-41.png)
![Screenshot at 03:30: Detailed view of one of the CPU cold plates, showing the copper construction and associated plumbing.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-03-30.png)
![Screenshot at 04:41: The presenter points out the lack of traditional fans inside the liquid-cooled server unit, highlighting efficiency.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-04-41.png)
![Screenshot at 05:55: The presenter manually touching the copper pipes running through the RAM slots, demonstrating the direct cooling of memory modules.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-05-55.png)
![Screenshot at 07:01: Close-up on a soldered wick piece used in the cooling system to prevent stray current corrosion.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-07-01.png)
![Screenshot at 10:09: Overhead view of the extensive blue and green insulated piping system used for the facility's cooling distribution.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-10-09.png)
![Screenshot at 11:29: Inside the Vertiv Liebert XDU cooling unit, showing thick hoses and brass valves managing coolant flow to the racks.](https://ss.rapidrecap.app/screens/U4Sqh-uJ2NA/00-11-29.png)