One of Linpack’s advantages is also its scalability in the sense that it has allowed us in the past 20 years to benchmark systems that cover a performance range of 12 orders of magnitude. It is true that Linpack delivers performance figures that occupy the upper end of any other application performance. In fact, no other realistic application delivers a better efficiency (Rmax/Rpeak) of a system. But using the peak performance instead of Linpack, which “experts” have often recommended to us, does not make any sense. We have seen a lot of new systems that were not able to run the Linpack test because they were not stable enough. Therefore, running Linpack to measure the performance is kind of a first reliability test for new HPC systems.
The misinterpretation of the TOP500 results has surely led to a negative attitude towards Linpack. Politicians, for example, tend to see a system’s TOP500 rank as a general rank that is valid for all applications, which of course is not true. The TOP500 rank only reflects a system’s ability to solve a linear set of equations, and it does not tell anything about its performance with respect to other applications. Therefore, the TOP500 list is not a tool for selecting a supercomputer system for an organization – centers need to run their own benchmark tests that are relevant to their applications. In this context, an approach such as the “HPC Challenge Benchmark” consisting of seven different benchmarks, which test different parts of a supercomputer, is critical.
The TOP500 lists’ success lies in compiling and analyzing data over time. Despite relying solely
on Linpack, we have been able to correctly identify and track ALL developments and trends over the past 20 years, covering manufacturers and users of HPC systems, architectures, interconnects, processors, operating systems, etc. And above all, TOP500’s strength is that it has proved an exceptionally reliable tool for forecasting developments in performance.
It is very unlikely that another benchmark will replace Linpack as basis for the TOP500 lists in the near future. And in any case we would stick to the concept of a single benchmark because this is the easiest way to trigger competition between manufacturers, countries and sites, which is extremely important for the overall acceptance of the TOP500 lists.