SPEC CPU95 Press Release

SPEC Announces SPEC95 Benchmark Suites As New Standard for Measuring Performance

Need to accommodate new technology forces retirement of SPEC92

FAIRFAX, VA, August 21, 1995 -- The Standard Performance Evaluation Corp. (SPEC) announces the release of the SPEC95 benchmark suites, the latest version of the worldwide standard for measuring and comparing computer performance across different hardware platforms. SPEC95 was developed by SPEC's Open Systems Group (OSG), which includes more than 30 leading computer vendors, systems integrators, publishers and consultants throughout the world.

"Computer systems technology evolves so rapidly that we must provide new benchmark suites every two to three years to ensure a level playing field," says Kaivalya M. Dixit, SPEC president. "SPEC92 was a great success, but it is time to make the transition to standardized benchmarks that reflect the advances in chip technologies, compilers and applications that have taken place over the last three years; those benchmarks constitute SPEC95."

SPEC95 comprises two sets (or suites) of benchmarks: CINT95 for compute-intensive integer performance and CFP95 for compute-intensive floating point performance. The two suites provide component-level benchmarks that measure the performance of the computer's processor, memory architecture and compiler. SPEC benchmarks are selected from existing application and benchmark source code running across multiple platforms. Each benchmark is tested on different platforms to obtain fair performance results across competing hardware and software systems.

SPEC95 is the third major version of the SPEC benchmark suites, which in 1989 became the first widely accepted standard for comparing computeintensive performance across various architectures. The new release replaces SPEC92, which will be gradually phased out between now and June 1996, when SPEC will stop publishing SPEC92 results and stop selling the benchmark suite. Performance results from SPEC95 cannot be compared to those from SPEC92, since new benchmarks have been added and existing ones changed.

"Compiler writers have learned to optimize performance for individual SPEC92 benchmarks," says Jeff Reilly, SPEC95 release manager. "The best way to avoid these benchmark specific optimizations is to develop new benchmark suites." SPEC95 builds on the lessons learned from the SPEC89 and SPEC92 suites, according to Reilly. The new benchmarks were analyzed to ensure that they are as resistant as possible to compiler optimizations that might not translate into realworld performance gains. Improvements to the suites include longer run times and larger problems for benchmarks, more application diversity, greater ease of use, and standard development platforms that will allow SPEC to produce additional releases for other operating systems.

A Sun SPARCstation 10/40 with 128 MB of memory was selected as the SPEC95 reference machine and Sun SC3.0.1 compilers were used to obtain reference timings on the new benchmarks. By definition, the SPECint95 and SPECfp95 numbers for the Sun SPARCstation 10/40 are both "1."

The Metrics for Measurement

SPEC95 rules permit both baseline and optimized results for CINT95 and CFP95 suites. The baseline rules restrict the number of optimizations that can be used for performance testing. In general, SPEC95 rules are more restrictive in regard to optimizations than the SPEC92 rules. Baseline metrics are mandatory for all reported results; reporting of optimized results is optional.

SPEC95 also allows performance to be measured for both speed and throughput (rate). Speed metrics such as SPECint95 measure how fast a computer completes a single task. Rate metrics such as SPECint_rate95 measure how many tasks a computer can accomplish in a certain amount of time. SPEC95 measures rate performance for single processors, symmetric multi processor systems and cluster systems.

The CINT95 suite, written in C language, contains eight CPUintensive integer benchmarks. It is used to measure and calculate the following metrics:

SPECint95: The geometric mean of eight normalized ratios (one for each integer benchmark) when compiled with aggressive optimization for each benchmark.
SPECint_base95: The geometric mean of eight normalized ratios when compiled with conservative optimization for each benchmark.
SPECint_rate95: The geometric mean of eight normalized throughput ratios when compiled with aggressive optimization for each benchmark.
SPECint_rate_base95: The geometric mean of eight normalized throughput ratios when compiled with conservative optimization for each benchmark.

The CFP95 suite, written in FORTRAN language, contains 10 CPU-intensive floating point benchmarks. It is used to measure and calculate the following metrics:

SPECfp95: The geometric mean of 10 normalized ratios (one for each floating point benchmark) when compiled with aggressive optimization for each benchmark.
SPECfp_base95: The geometric mean of 10 normalized ratios when compiled with conservative optimization for each benchmark.
SPECfp_rate95: The geometric mean of 10 normalized throughput ratios when compiled with aggressive optimization for each benchmark.
SPECfp_rate_base95: The geometric mean of 10 normalized throughput ratios when compiled with conservative optimization for each benchmark.

Vendor Reporting

Initial results for systems from six vendor companies are included with this release. Additional results will be reported in the next issue of the SPEC Newsletter, scheduled for publication at the end of September. SPEC members are being encouraged to report SPEC95 results on older platforms to provide an historical perspective for the new results.

Availability

SPEC95 (CINT95 and CFP95) is available on CD-ROM from SPEC's administrator, the National Computer Graphics Association (NCGA). The cost is $600 for new customers, $300 for new university customers, $300 for current SPEC92 licensees and $150 for current university licensees.

SPEC is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. Included in its membership are the Open Systems Group (OSG); OSG Associates, consisting of leading universities and research facilities; the HighPerformance Group (HPG); and HPG Associates.