Highlights - June 2024

This is the 63rd edition of the TOP500.

The 63rd edition of the TOP500 reveals that Frontier has once again claimed the top spot, despite no longer being the only exascale machine on the list. Additionally, a new system has found its way into the Top 10.

The Frontier system at Oak Ridge National Laboratory in Tennessee, USA remains the most powerful system on the list with an HPL score of 1.206 EFlop/s. The system has a total of 8,699,904 combined CPU and GPU cores, an HPE Cray EX architecture that combines 3rd Gen AMD EPYC CPUs optimized for HPC and AI with AMD Instinct MI250X accelerators, and it relies on Cray’s Slingshot 11 network for data transfer. On top of that, this machine has an impressive power efficiency rating of 52.59 GFlops/Watt – putting Frontier at the No. 11 spot on the GREEN500.

Also like the last list, the Aurora system at the Argonne Leadership Computing Facility in Illinois, USA, has claimed the No. 2 spot on the TOP500. Despite currently being commissioned and not fully complete, Aurora is now the second machine to officially break the exascale barrier with an HPL score of 1.012 EFlop/s – an improvement over the 585.34 PFlop/s score from the last list. This system is based on HPE Cray EX- Intel Exascale Computer Blade and uses Intel Xeon CPU Max series processors, Intel Data Center GPU Max Series accelerators, and a Slingshot-11 interconnect.

The Eagle system installed on the Microsoft Azure Cloud in the USA reclaimed the No. 3 spot that it achieved after its debut appearance on the previous list, and it remains the highest-ranking cloud system on the TOP500. This Microsoft NDv5 system has an HPL score of 561.2 PFlop/s and is based on Intel Xeon Platinum 8480C processors and NVIDIA H100 accelerators.

Fugaku also retained its No. 4 spot from the previous list, despite holding the No.1 spot from June 2020 until November 2021. Based in Kobe, Japan, Fugaku has an HPL score of 442 PFlop/s and it remains the highest-ranked system outside the USA. The LUMI system at EuroHPC/CSC in Finland also remained in its spot at No. 5 with an HPL score of 380 PFlop/s. This machine is the largest system in Europe.

The only new system to find its way onto the Top 10 is the Alps machine at No. 6 from the Swiss National Supercomputing Centre (CSCS) in Switzerland. This system achieved an HPL score of 270 PFlop/s.

Here is a summary of the systems in the Top 10:

  • Frontier remains the No. 1 system in the TOP500. This HPE Cray EX system is the first US system with a performance exceeding one Exaflop/s. It is installed at the Oak Ridge National Laboratory (ORNL) in Tennessee, USA, where it is operated for the Department of Energy (DOE). It currently has achieved 1.206 Exaflop/s using 8,699,904 cores. The HPE Cray EX architecture combines 3rd Gen AMD EPYC™ CPUs optimized for HPC and AI, with AMD Instinct™ 250X accelerators, and a Slingshot-11 interconnect.

  • Aurora is currently the No. 2 with a preliminary HPL score of 1.012 Exaflop/s. It is installed at the Argonne Leadership Computing Facility, Illinois, USA, where it is also operated for the Department of Energy (DOE). This new Intel system is based on HPE Cray EX - Intel Exascale Compute Blades. It uses Intel Xeon CPU Max Series processors, Intel Data Center GPU Max Series accelerators, and a Slingshot-11 interconnect.

  • Eagle the No. 3 system is installed by Microsoft in its Azure cloud. This Microsoft NDv5 system is based on Xeon Platinum 8480C processors and NVIDIA H100 accelerators and achieved an HPL score of 561 Pflop/s.

  • Fugaku, the No. 4 system, is installed at the RIKEN Center for Computational Science (R-CCS) in Kobe, Japan. It has 7,630,848 cores which allowed it to achieve an HPL benchmark score of 442 Pflop/s.

  • The LUMI system, another HPE Cray EX system installed at EuroHPC center at CSC in Finland is at the No. 5 with a performance of 380 Pflop/s. The European High- Performance Computing Joint Undertaking (EuroHPC JU) is pooling European resources to develop top-of-the-range Exascale supercomputers for processing big data. One of the pan-European pre-Exascale supercomputers, LUMI, is located in CSC’s data center in Kajaani, Finland.

Rank Site System Cores Rmax (TFlop/s) Rpeak (TFlop/s) Power (kW)
1 DOE/SC/Oak Ridge National Laboratory
United States
Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11
HPE
8,699,904 1,206.00 1,714.81 22,786
2 DOE/SC/Argonne National Laboratory
United States
Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max, Slingshot-11
Intel
9,264,128 1,012.00 1,980.01 38,698
3 Microsoft Azure
United States
Eagle - Microsoft NDv5, Xeon Platinum 8480C 48C 2GHz, NVIDIA H100, NVIDIA Infiniband NDR
Microsoft Azure
2,073,600 561.20 846.84
4 RIKEN Center for Computational Science
Japan
Supercomputer Fugaku - Supercomputer Fugaku, A64FX 48C 2.2GHz, Tofu interconnect D
Fujitsu
7,630,848 442.01 537.21 29,899
5 EuroHPC/CSC
Finland
LUMI - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11
HPE
2,752,704 379.70 531.51 7,107
6 Swiss National Supercomputing Centre (CSCS)
Switzerland
Alps - HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11
HPE
1,305,600 270.00 353.75 5,194
7 EuroHPC/CINECA
Italy
Leonardo - BullSequana XH2000, Xeon Platinum 8358 32C 2.6GHz, NVIDIA A100 SXM4 64 GB, Quad-rail NVIDIA HDR100 Infiniband
EVIDEN
1,824,768 241.20 306.31 7,494
8 EuroHPC/BSC
Spain
MareNostrum 5 ACC - BullSequana XH3000, Xeon Platinum 8460Y+ 32C 2.3GHz, NVIDIA H100 64GB, Infiniband NDR
EVIDEN
663,040 175.30 249.44 4,159
9 DOE/SC/Oak Ridge National Laboratory
United States
Summit - IBM Power System AC922, IBM POWER9 22C 3.07GHz, NVIDIA Volta GV100, Dual-rail Mellanox EDR Infiniband
IBM
2,414,592 148.60 200.79 10,096
10 NVIDIA Corporation
United States
Eos NVIDIA DGX SuperPOD - NVIDIA DGX H100, Xeon Platinum 8480C 56C 3.8GHz, NVIDIA H100, Infiniband NDR400
Nvidia
485,888 121.40 188.65

The only new system in the TOP10 is Alps at No. 6 installed at the Swiss National Supercomputing Centre (CSCS) in Switzerland. It is an HPE Cray EX254n system with NVIDIA Grace 72C and NVIDIA GH200 Superchip and a Slingshot-11 interconnect. It achieved 270 Pflops/s.

The No. 7 system Leonardo is installed at another EuroHPC site in CINECA, Italy. It is an Atos BullSequana XH2000 system with Xeon Platinum 8358 32C 2.6GHz as main processors, NVIDIA A100 SXM4 40 GB as accelerators, and Quad-rail NVIDIA HDR100 Infiniband as interconnect. It achieved a Linpack performance of 241.2 Pflop/s.

The MareNostrum 5 ACC system was remeasured and jumped in the ranking over the Summit system. It is now at No. 8 and installed at the EuroHPC/Barcelona Supercomputing Center in Spain. This BullSequana XH3000 system uses Xeon Platinum 8460Y processors with NVIDIA H100 and Infiniband NDR200. It achieved 175.3 Pflop/s HPL performance.

Summit, an IBM-built system at the Oak Ridge National Laboratory (ORNL) in Tennessee, USA, is now listed at the No. 9 spot worldwide with a performance of 148.6 Pflop/s on the HPL benchmark, which is used to rank the TOP500 list. Summit has 4,356 nodes, each one housing two POWER9 CPUs with 22 cores each and six NVIDIA Tesla V100 GPUs each with 80 streaming multiprocessors (SM). The nodes are linked together with a Mellanox dual-rail EDR InfiniBand network.

The Eos system listed at No. 10 is a NVIDIA DGX SuperPOD based system at NVIDIA, USA. It is based on the NVIDIA DGX H100 with Xeon Platinum 8480C processors,N VIDIA H100 accelerators, and Infiniband NDR400 and it achieves 121.4 Pflop/s.

Highlights from the List

  • A total of 194 systems on the list are using accelerator/co-processor technology, up from 185 six months ago. 83 of these use NVIDIA Ampere chips, 48 use NVIDIA Volta, and 22 systems with 18.

  • Intel continues to provide the processors for the largest share (63.00 percent) of TOP500 systems, down from 67.80 % six months ago. 156 (31.20 %) of the systems in the current list used AMD processors, up from 28.00 % six months ago.

  • The entry level to the list moved up to the 2.13 Pflop/s mark on the Linpack benchmark.

  • The last system on the newest list was listed at position 458 in the previous TOP500.

  • Total combined performance of all 500 exceeded the Exaflop barrier with now 8.21 exaflop/s (Eflop/s) up from 7.01 exaflop/s (Eflop/s) 6 months ago.

  • The entry point for the TOP100 increased to 9.44 Pflop/s.

  • The average concurrency level in the TOP500 is 229,426 cores per system up from 212,027 six months ago.

General Trends

Installations by countries/regions:

HPC manufacturer:

Interconnect Technologies:

Processor Technologies:

Green500

  • The No. 1 spot on the GREEN500 was claimed by JEDI, a new system from EuroHPC/FZJ in Germany. Taking the No. 190 spot on the TOP500, JEDI achieved an energy efficiency rating of 72.73 GFlops/Watt while producing an HPL score of 4.5 PFlop/s. JEDI is a BullSequana XH3000 machine with a Grace Hopper Superchip 72C. It has 19,584 total cores.

  • The Isambard-AI machine out of the University of Bristol in the U.K. claimed the No. 2 spot with an energy efficiency rating of 68.83 GFlops/Watt and an HPL score of 7.42 PFLop/s. Isambard-AI achieved the No. 129 spot on the TOP500 and has 34,272 total cores.

  • The No. 3 spot was claimed by the Helios system from Cyfronet out of Poland. The machine achieved an energy efficiency score of 66.95 GFlops/Watt and an HPL score of 19.14 PFlop/s.

Rank TOP500 Rank System Cores Rmax (PFlop/s) Power (kW) Energy Efficiency (GFlops/watts)
1 189 JEDI - BullSequana XH3000, Grace Hopper Superchip 72C 3GHz, NVIDIA GH200 Superchip, Quad-Rail NVIDIA InfiniBand NDR200 , ParTec/EVIDEN
EuroHPC/FZJ
Germany
19,584 4.50 67 72.733
2 128 Isambard-AI phase 1 - HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11 , HPE
University of Bristol
United Kingdom
34,272 7.42 117 68.835
3 55 Helios GPU - HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11 , HPE
Cyfronet
Poland
89,760 19.14 317 66.948
4 328 Henri - ThinkSystem SR670 V2, Intel Xeon Platinum 8362 32C 2.8GHz, NVIDIA H100 80GB PCIe, Infiniband HDR , Lenovo
Flatiron Institute
United States
8,288 2.88 44 65.396
5 71 preAlps - HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11 , HPE
Swiss National Supercomputing Centre (CSCS)
Switzerland
81,600 15.47 240 64.381
6 299 HoreKa-Teal - ThinkSystem SD665-N V3, AMD EPYC 9354 32C 3.25GHz, Nvidia H100 94Gb SXM5, Infiniband NDR200 , Lenovo
Karlsruher Institut für Technologie (KIT)
Germany
13,616 3.12 50 62.964
7 54 Frontier TDS - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 , HPE
DOE/SC/Oak Ridge National Laboratory
United States
120,832 19.20 309 62.684
8 11 Venado - HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11 , HPE
DOE/NNSA/LANL
United States
481,440 98.51 1,662 59.287
9 20 Adastra - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 , HPE
Grand Equipement National de Calcul Intensif - Centre Informatique National de l'Enseignement Suprieur (GENCI-CINES)
France
319,072 46.10 921 58.021
10 28 Setonix – GPU - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 , HPE
Pawsey Supercomputing Centre, Kensington, Western Australia
Australia
181,248 27.16 477 56.983

HPCG Results

The TOP500 list has incorporated the High-Performance Conjugate Gradient (HPCG) benchmark results, which provide an alternative metric for assessing supercomputer performance. This score is meant to complement the HPL measurement to give a fuller understanding of the machine.

Supercomputer Fugaku is the leader on the HPCG benchmark with an impressive score of 16 HPCG-PFlop/s. Frontier came in second with a score of 14.1 HPCG-PFlop/s.

Aurora came in third with a score of 5.6 HPCG-PFlop/s. However, it is important to note that Aurora only used about 40% of its nodes for the HPCG run. The Aurora team plans to get a full system run for the next TOP500 list.

Rank TOP500 Rank System Cores Rmax (PFlop/s) HPCG (PFlop/s)
1 4 Supercomputer Fugaku - Supercomputer Fugaku, A64FX 48C 2.2GHz, Tofu interconnect D ,
RIKEN Center for Computational Science
Japan
7,630,848 442.01 16.00
2 1 Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 ,
DOE/SC/Oak Ridge National Laboratory
United States
8,699,904 1,206.00 14.05
3 2 Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max, Slingshot-11 ,
DOE/SC/Argonne National Laboratory
United States
9,264,128 1,012.00 5.61
4 5 LUMI - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X, Slingshot-11 ,
EuroHPC/CSC
Finland
2,752,704 379.70 4.59
5 6 Alps - HPE Cray EX254n, NVIDIA Grace 72C 3.1GHz, NVIDIA GH200 Superchip, Slingshot-11 ,
Swiss National Supercomputing Centre (CSCS)
Switzerland
1,305,600 270.00 3.67
6 7 Leonardo - BullSequana XH2000, Xeon Platinum 8358 32C 2.6GHz, NVIDIA A100 SXM4 64 GB, Quad-rail NVIDIA HDR100 Infiniband ,
EuroHPC/CINECA
Italy
1,824,768 241.20 3.11
7 9 Summit - IBM Power System AC922, IBM POWER9 22C 3.07GHz, NVIDIA Volta GV100, Dual-rail Mellanox EDR Infiniband ,
DOE/SC/Oak Ridge National Laboratory
United States
2,414,592 148.60 2.93
8 14 Perlmutter - HPE Cray EX 235n, AMD EPYC 7763 64C 2.45GHz, NVIDIA A100 SXM4 40 GB, Slingshot-11 ,
DOE/SC/LBNL/NERSC
United States
888,832 79.23 1.91
9 12 Sierra - IBM Power System AC922, IBM POWER9 22C 3.1GHz, NVIDIA Volta GV100, Dual-rail Mellanox EDR Infiniband ,
DOE/NNSA/LLNL
United States
1,572,480 94.64 1.80
10 15 Selene - NVIDIA DGX A100, AMD EPYC 7742 64C 2.25GHz, NVIDIA A100, Mellanox HDR Infiniband ,
NVIDIA Corporation
United States
555,520 63.46 1.62

HPL-MxP Results

On the HPL-MxP benchmark, which measures performance for mixed-precision calculation, the new Aurora system achieved 10.6 Exaflops beating out the previous leader Frontier at 10.2 Exaflops!

The HPL-MxP benchmark seeks to highlight the use of mixed precision computations. Traditional HPC uses 64-bit floating point computations. Today we see hardware with various levels of floating point precisions, 32-bit, 16-bit, and even 8-bit. The HPL-MxP benchmark demonstrates that by using mixed precision during the computation much higher performance is possible (see the Top 5 from the HPL-MxP benchmark), and using mathematical techniques, the same accuracy can be computed with the mixed precision technique when compared with straight 64-bit precision.

Rank (HPL-MxP) Site Computer Cores HPL-MxP (Eflop/s) TOP500 Rank HPL Rmax (Eflop/s) Speedup of HPL-MxP over HPL

1

DOE/SC/ANL, USA

Aurora, intel/HPE Cray EX

8,159,232

10.6

2

1.012

10

2

DOE/SC/ORNL, USA

Frontier, HPE Cray EX235a

8,699,904

10.2

1

1.206

8.3

3

EuroHPC/CSC, Finland

LUMI, HPE Cray EX235a

2,752,704

2.350

5

0.3797

6.2

4

RIKEN, Japan

Fugaku, Fujitsu A64FX

7,630,848

2.000

4

0.4420

4.5

5

EuroHPC/CINECA, Italy

Leonardo, Bull Sequana XH2000

1,824,768

1.842

7

0.2387

7.5

About the TOP500 List

The first version of what became today’s TOP500 list started as an exercise for a small conference in Germany in June 1993. Out of curiosity, the authors decided to revisit the list in November 1993 to see how things had changed. About that time they realized they might be onto something and decided to continue compiling the list, which is now a much-anticipated, much-watched and much-debated twice-yearly event.