If you’re building a high-performance data center from scratch on a budget, you might not be able to topple China’s Tianhe-2, which reigns supreme in the June 2014 Top 500 Supercomputers list once again. However, you could look to the Top 500 for some guidance as to the most effective infrastructures and architectures you can invest in.

This year’s list offers insight by way of a megaflops-per-watt category — a measure of millions of floating point instructions per second achieved with one watt of power — in tests conducted twice annually by the University of Mannheim for the International Supercomputer Conference.  When judged by this measure of efficiency, the #436 system on the list, dubbed Tsubame-KFC, built in 2013 by NEC for the Tokyo Institute of Technology, yielded a commanding 3,418.18 Mflops/watt, over 300 points over its closest competitor.

If that name sounds familiar, you may be a regular reader of ours.  It’s the same system DatacenterDynamics covered last November, that topped the Green500 list last November, and that was profiled in The New York Times.  It uses the revolutionary immersion cooling process developed by Austin, Texas-based Green Revolution Cooling (GRC).

In a recent white paper (.PDF), GRC makes this point about the most common class of cooling in datacenters: “Free air cooling that uses air directly from the outside requires massive airflow and substantial filtering.  Facilities that use air exchange mechanisms can reduce or eliminate filtration needs, but must house large mechanical components.  And for sites using free air cooling at night or during cold seasons, the datacenter must have the staffing and monitoring equipment to gracefully transition between free air and traditional chillers.”

It’s also difficult to retrofit existing datacenters with all-new free air facilities, the white paper says — and that was the very problem Tokyo faced when it made the investment in GRC.

But immersion cooling isn’t the only choice the Institute made that’s paying off.  Of the 15 top performing supercomputers on the Top 500 list’s Mflops/sec metric, 11 have a lot in common. These 11 are clusters based on Intel Xeon E5-2650v2 eight-core, hyperthreaded processors, from the Ivy Bridge generation. They’re rated at a thermal design point of 95 W (a measure of the amount of heat the CPU generates requiring dissipation).  And they utilize Intel’s AVX architecture, which is the culmination of the company’s experiments with using graphics processors for core and math functions.

All 11 of these clusters, including Tsubame-KFC, use Nvidia Tesla K20x GPU accelerators.  None of the systems on the list including the E5-2650v2 / K20x combination fared any worse than 13th in megaflops-per-watt.

Intel’s less efficient power players also score highly
Twice each year, the rankings of 500 of the world’s supercomputers are assessed by the University of Mannheim in association with California’s Berkeley National Laboratory and the University of Tennessee, Knoxville. These assessments use a high-performance implementation of the industry standard Linpack benchmark.  The key score that determines the rankings of HPC clusters in each list is maximal observed peak performance, in gigaflops (GFlops, or billions of floating-point operations per second). This performance is called the R_max rating.

One important derivative measurement from the Top 500 list is yield — comparing the theoretical top performance level R_peak against the measured maximum sustained performance level R_max. Here again, Intel’s Ivy Bridge generation can claim victory:  Higher order Xeon models — including the E5-2670v2 at 115 W TDP, and moving up to the 2680v2, 2690v2, and 2695v2 — clinched 13 of the 15 top performers in the derivative yield category.

The system with the best yield overall (a fantastic 99.8%) was the #122 performer overall: Ada, an IBM NeXtScale system built for Texas A&M University.

In the all-important heavyweight category, though, Tienhe-2 remains the champion.  Its R_max score of 33,862,700 gigaflops is beyond belief — nearly 34 petaflops!  It was only a few years ago that the petaflop barrier was broken.

Wherefore art thou, HMC?
The top 9 systems on the list remain as they were since last November, with a new Cray system with Intel E5-2697v2 processors (built for an undisclosed government) taking over the #10 spot.

There isn’t much of a contest at present in the area of processor manufacturer.  IBM Power processors were at the heart of only 38 of the clusters on the June 2014 list, while 31 clusters were powered by AMD Opterons, and 3 still there with Sun SPARC processors.

While we anticipated the possibility that a Cray-built cluster for the US Department of Energy, based on Micron’s new hybrid memory cube (HMC) technology and Intel’s new Knights Landing architecture, could land on the Top 500 in time for the ISC conference, it apparently was not to be.

Intel’s existing Knights Corner architecture, which utilizes many-core Xeon Phi coprocessor packages, powered some 17 clusters in the latest list, including the #1 Tianhe-2, the #7 Stampede (built by Dell for the University of Texas), and the #15 built by Atipa Technology for the Pacific Northwest National Laboratory.

A 26,496-core cluster in Amazon’s EC2 cloud, which premiered at #42 on the Top 500 list in 2011, currently ranks #76.