Methodology

Introduction

For years, the LINPACK Benchmarks have been widely used as a tool for measuring a computer system’s floating point performance. One prominent example is the use of the High-Performance LINPACK Benchmark as a performance indicator for ranking the world's 500 fastest computers. Despite the limelight status of the Top500 list, using the LINPACK as a tool for predicting a computer’s real-world performance is a limited approach because it only stresses the CPU portion of the system.

The release of the High Performance Computing Challenge (HPCC) suite of Benchmarks by the High Productivity Computing Systems (HPCS) Program at the Defense Advanced Research Projects Agency (DARPA) in 2003 was a response to a challenge for the HPC community to have a system performance evaluation tool across a broader spectrum of application areas.

Consisting of 28 benchmarks in seven performance tests, the HPCC Benchmark suite was designed to stress the processor, memory and interconnect subsystems of an HPC system based on computational kernels that are widely used in real-world applications. These tests and the relevant performance areas they measure are briefly described below:

HPL – High Performance LINPACK – is a benchmark that measures the floating point performance of a system by solving a double-precision dense linear system of equations using MPI. The result is expressed in TeraFLOP/sec. HPL performance depends on the processor and the interconnect bandwidth.
PTRANS – Parallel matrix TRANSpose – tests the performance of inter-processor communication patterns that are characteristic of simultaneous message-passing between pairs of processors. Expressed in GB/sec, PTRANS performance depends on the interconnect and the memory bandwidth.
STREAM is a memory I/O benchmark that measures sustainable memory bandwidth and the computation rates of four vector operations, i.e., COPY, SACLE, ADD and TRIAD. The unit of measure is GB/sec for all of these operations. STREAM performance is bounded by the memory bandwidth.
RandomAccess measures the capability of a system for random global memory updates. Expressed in GUPS (Giga Updates per Second), RandomAccess performance has a strong dependency on the memory latency and the interconnect latency.
PingPong and RandomRing Latency/Bandwidth measures communication latency and bandwidth. The measurement units are micro-second and GB/sec for the latency and the bandwidth, respectively. [Note: The GCPI excludes these tests because latency/kwatt is not a meaningful measure.]
DGEMM measures, in GFLOP/sec, the floating point performance of double precision real matrix-matrix multiplication. This test is most dependent on processor arithmetic performance.
FFT measures the floating point performance of a parallel double-precision complex Discrete Fourier Transform. The test measures the interconnect latency and bandwidth.

While HPCC has provided insight into system performance, a different challenge has rapidly evolved in the past few years. Many commercial and technical computing data centers began to face a common set of issues related to provisioning, power, cooling, electricity costs and floor space. As the problem worsened, the green IT movement quickly gained momentum. It was ranked as the number one priority in 2007 by market research firm Gartner. In addition, the area of power, cooling and floor space has consistently emerged as one of the most important issues facing CIOs and datacenter managers in many market research studies. Indeed, there was good reason for this sense of urgency.

In a report to Congress dated August 2, 2007, the U.S. Environmental Protection Agency estimated that the U.S. server and data center sector consumed about 61 billion kilowatt-hours in 2006 which accounted for 1.5 percent of total U.S. electricity consumption or a total electricity cost of about $4.5 billion. Of equal importance is the fact that IT accounts for 2% of CO₂ emissions globally. Companies need to understand that achieving energy efficiency is not only an effective cost saving measure, it also serves to fulfill one’s environmental responsibility.

One significant approach to reduce energy costs within IT is through server energy efficiency. As the energy and the economic crises take holds, more users are demanding server efficiency from vendors and specifying it as an important purchasing criterion. Two efforts in the HPC community also helped to raise the level of green computing awareness. In November 2007, the Green500 list was launched with the purpose of ranking the most energy-efficient supercomputers in the world. In June 2008, the Top500 list began to track system power data. While these efforts were positive steps in the right direction, they were inherently limited because of their dependence on the LINPACK benchmark for performance measurements.

In order to expand on these initial efforts by Top500 and Green500, SiCortex has proposed a new way of measuring system-level server energy efficiency based on the HPCC suite of benchmarks.

Details

The GCPI is now in its second revision. Due to industry feedback, the b_eff latency/bandwidth test and its 10 associated benchmarks are not included.

We first gathered peak and LINPACK system power data for 11 systems from various sources including the Top500 and the Green500 lists, vendors and research literature. When a system’s power consumption while running LINPACK was not available, we used 90% of the peak power as an estimate.

Using the power data obtained above and the results listed on the HPCC Benchmarks Web site, the Green Computing Performance Index of a specific system is calculated by following Steps (I) through (VI) below.

(I) Obtain performance-per-kilowatt information by dividing each result by the corresponding power consumption in kilowatt.

(III) Evaluate the Green Computing Performance Index component of a system for a specific benchmark by dividing the associated performance-per-kilowatt value from above by that of the reference system to arrive at a dimensionless number. This component index can be explained as the green computing efficiency relative to the reference system for that specific benchmark.

(IV) Repeat (III) for all benchmarks.

(V) For each benchmark, assign a GCPI component (or a weighting factor) to the reference system so that the sum of all GCPI components of the reference system is equal to 1. For example, the results of our calculations were based on a weighting factor of 1/6 for HPL and PTRANS, respectively.

(VI) For each remaining system, using a straightforward linear mathematical model, calculate the corresponding GCPI component for each benchmark, then sum across all benchmarks to get the overall GCPI for that system.