Standard Performance Evaluation Corporation

Answers to Frequently Asked Questions
About the SPEChpc96 Benchmark Suite

November 10, 1999

Q1: What is SPEChpc96?

A1: SPEChpc96 is a benchmark suite that measures the performance of high-end computing systems running industrial-style applications. The SPEChpc line of benchmarks is especially suited for evaluating the performance of parallel and distributed computer architectures.

SPEChpc96 represents a commitment to providing benchmarks that measure sustained performance, instead of the peak performance numbers still used widely today. The benchmarks within the SPEChpc96 suite represent real-world industrial applications that run on today's high-performance systems.

Q2: What organizations were involved in developing SPEChpc96?

A2: SPEChpc96 was developed by the Standard Performance Evaluation Corp.'s High-Performance Group (SPEC/HPG), formed in January 1994. Founding partners of SPEC/HPG include SPEC members, former members of the Perfect Benchmarks effort, and other groups working in the benchmarking arena. SPEC/HPG's mission has remained the same: to maintain and endorse a suite of benchmarks that represent real-world, high-performance computing applications. Current members include: Compaq, Fujitsu America, IBM, Sun Microsystems, Tsukuba Advanced Computing Center, and current Associates include: Purdue University, Real World Computing, University of Illinois, University of Minnesota and University of Tennessee.

Q3: What is the difference between SPEChpc96 and other benchmarks for high-performance systems?

A3: The most important distinction is that SPEChpc96 includes applications used in industry and research to do real work. These applications normally run on multiprocessing systems and require the larger computing resources offered by high-end systems. Only minimal modifications were made to the applications used in the SPEChpc96 suite. By leaving even "uninteresting" functions of the application code intact, SPEChpc96 provides a realistic measure of real-world application performance. SPEC/HPG's methodology differs from previous benchmarking efforts, which concentrated only on more numerically intensive algorithms.

A second distinction is that SPEChpc96 targets all high-performance computer architectures. The applications in the suite are currently in use on a wide variety of systems, including workstations, clusters, SMPs, vector systems and MPPs. The programming models used in the SPEChpc96 application codes -- currently message-passing and serial models -- can be run on all of today's high-performance systems. Other benchmarks tend to be biased towards either distributed-memory, shared-memory or vector architectures.

Finally, SPEChpc96 provides more than just peak performance numbers. To ensure that SPEChpc96 reflects performance for real applications, only a limited number of optimizations are allowed. This contrasts with benchmarks that allow a large number of optimizations requiring unrealistic development efforts to reproduce benchmark results. It also contrasts with benchmarks that restrict optimizations altogether.

Q4: What applications are included with SPEChpc96?

A4: The SPEChpc96 suite includes three application areas: seismic processing (SPECseis96), computational chemistry (SPECchem96), and climate modeling (SPECclimate):

SPECseis96 contains Seismic, an application developed at Atlantic Richfield Corp. (ARCO). Seismic is an industrial application that performs time and depth migrations used to locate gas and oil deposits.

SPECchem96 contains GAMESS (General Atomic and Molecular Electronic Structure System), an improved version of programs that originated in the Department of Energy's National Resource for Computations in Chemistry. Many of the functions found in GAMESS are duplicated in commercial packages used in the pharmaceutical and chemical industries for drug design and bonding analysis.

SPECclimate is MM5, the PSU/NCAR limited-area, hydrostatic or non-hydrostatic, sigma-coordinate model designed to simulate or predict mesoscale and regional-scale atmospheric circulation. MM5 was developed by the Pennsylvania State University (Penn State) and the University Corporation for Atmospheric Research (UCAR).

Q5: Are different-sized datasets available for the two SPEChpc96 applications?

A5: Yes. Vendors may report SPEChpc96 results based on any of four predefined problem sizes: small, medium, large or extra large. Problem sizes depend on the application and the type of analysis the application performs. For SPECseis96, problem sizes relate directly to the number and size of seismic traces being processed. For SPECchem96, problem sizes relate to the complexity of the molecule under analysis. For SPECclimate the problems represent different weather patterns and more detailed forecasts thereof. The different problem sizes within SPEChpc96 give the application users more information about machine performance as it relates to the type of computational work they do.

Q6: What are SPEC/HPG's plans for adding applications to SPEChpc96?

A6: SPEC/HPG is examining additional applications used in other areas of computational analysis running on high-performance computers. Applications under consideration include computational fluid dynamics (CFD), molecular dynamics, climate, ocean and weather codes.

Q7: Will SPEC/HPG replace applications in conjunction with changes in industrial software code?

A7: Yes. Applications in the SPEChpc96 suite will be reviewed on a regular basis, and when newer versions are available, they will be incorporated into the benchmark suite. If an application falls out of use within its industrial area, a new, more relevant application will be adopted to replace it.

Q8: Will SPEChpc96 include more applications written in C or C++ in the future?

A8: If a suitable application representing relevant computational work in industry is written in C or C++, it will certainly be considered. In fact, both applications in SPEChpc96 V1.0 contain components written in C.

Q9: How do SPEChpc96 benchmarks address different parallel architectures, such as clusters, vector systems, SMPs and NUMA?

A9: SPEChpc96 benchmarks can be executed in serial or parallel mode. Due to the agreed-upon software standards for parallel systems, the parallel implementations have been based on the message-passing programming models PVM and MPI, and on the directive-based OpenMP API. Since high-performance computing systems use different architectures, the SPEChpc96 run rules allow for some flexibility in adapting the benchmark application to run in parallel mode. To ensure that results are relevant to end-users, SPEC/HPG requires that systems running SPEChpc96 benchmarks adhere to the following rules: - they must provide a suitable environment for running typical C and Fortran programs:

• the system vendor must offer its implementation for general use
• the implementation must be generally available, documented, and supported by the vendor

Q10: Are SPEChpc96 results comparable for these different parallel architectures?

A10: Yes. Most consumers of high-performance systems are interested in running a single important application, or perhaps a small set of critical applications, on these high-priced machines. The amount of time it takes to solve a particular computational analysis is often critical to a high-performance systems user's business. For these consumers, being able to compare different machines' abilities to complete a relevant problem of a specific size for their application is valuable information, regardless of the architectural features of the system itself.

Q11: Are SPEChpc96 results comparable across workload size? Can you compare serial results to parallel results?

A11: Varying the problem size, but not the system or parallelization, demonstrates how the application performs under a greater workload. The definition of "workload" will be application-specific and meaningful to users doing that sort of work. With SPECseis96, for example, larger trace files require more I/O, larger FFTs, and longer running times. A seismic analyst will be able to use the benchmark results to understand the ability of a machine to accomplish mission-critical tasks. Different datasets may also exercise different functionality of the codes, which must be considered when interpreting scalability with respect to data size. Comparing serial to parallel results yields significant information as well: It shows the scalability of the test system for a specific benchmark code.

Q12: How will SPEC/HPG address the evolution of parallel programming models?

A12: As standards emerge for parallel programming models, they will be reflected in the SPEChpc96 benchmarks. In response to the growing acceptance of SMP architectures, for example, SPEC/HPG is developing SAS (shared address space) parallel versions of its current benchmarks.

Q13: Can SPEChpc96 benchmarks be run on a high-end workstation?

A13: Yes, they can be run on single-processor machines. The smaller problem sizes are likely to be the most suitable for these systems.

Q14: Traditionally, SPEC has not allowed any code changes in its benchmarks. Why are code changes allowed in SPEChpc96 and how did SPEC/HPG decide what should be allowed?

A14: SPEC/HPG recognizes that customers who will spend many thousands to millions of dollars on a high-performance computer are willing to invest additional money to optimize their production codes. In addition to delivering more return on investment, code changes are required because there are so many different high-performance architectures; moving an application from one architecture to another is far more involved than porting a single CPU code from one workstation to another.

SPEC/HPG realized that since all customers optimize their programs, vendors should be allowed to perform the same level of optimization as a typical customer. There are specific rules that vendors must follow in optimizing codes. These rules were chosen to allow each vendor to show what their systems are capable of without allowing large application rewrites that would compromise performance comparisons.

Each vendor's code changes must be fully disclosed to the entire SPEC/HPG membership and approved before results are published. These changes must also be included in published reports, so customers know what changes they would have to make to duplicate results.

Q15: Do SPEChpc96 benchmarks measure speed or throughput?

A15: Both. SPEChpc96 benchmarks measure the time it takes to run an application on the system being tested -- that's a test of speed. The SPEChpc96 metric also normalizes the benchmark's elapsed time to the number of seconds in a day. So, the benchmarks also measure throughput, since the metric reports how many benchmarks could be run, back to back, in a given 24-hour period.

Q16: Does SPEChpc96 make SPECfp95 obsolete? What does it measure that SPECfp95 does not?

A16: SPEChpc96 results provide information that supplements SPECfp95 results. Consumers of high-performance computing systems usually run a particular application or set of applications. It is important for these consumers to know how applications in their area of analysis will perform on the systems under consideration. This is the kind of specific information that SPEChpc96 provides.

Q17: How does SPEChpc96 compare to the NAS parallel benchmarks or to Parkbench?

A17: The NPB (NAS Parallel Benchmarks) and Parkbench are kernels or subsets of applications; they are used to compare architectural implementations of machines. SPEChpc96 benchmarks are complete, real-world applications used by numerous organizations to solve real problems. These new benchmarks allow users to determine how well a given system performs for the entire spectrum of factors needed to solve real-world problems, including numerical computation, I/O, memory access, and many others.

Q18: Why doesn't SPEC/HPG define a metric such as M/FLOPS or price/performance?

A18: SPEC/HPG chose to focus on total application performance for large, industrially relevant applications. Within this benchmarking environment, a simple metric such as M/FLOPS is inadequate and misleading. Customers need to understand the expected performance of systems under consideration for purchase. Real-world performance includes all of the set-up, computation and post-processing work. Since the pre- and post-processing phases of applications can be significant factors in total system performance, SPEC/HPG chose to concentrate on total system performance.

Q19: Why didn't SPEC/HPG define a baseline metric?

A19: Since high-performance computer customers are willing to invest programming time to tune the applications that run on their systems, a baseline result has little meaning to them. Also, the architectures employed in the high-performance computing market are far more diverse than those found in single-CPU workstations. The baseline or "out-of-the-box" performance of any given application has no correlation to the actual performance a customer could expect to achieve on a particular architecture.

Q20: Why is there no reference machine for performance comparisons?

A20: Reference machines give benchmark users a framework for judging metrics that would otherwise just be meaningless sets of numbers. SPEChpc96 uses time as its reference, not the speed of a particular machine. The metric itself tells how many successive benchmark runs can be completed in a 24-hour period on the system being tested.

Q21: Why doesn't SPEChpc96 provide a composite or aggregate performance metric?

A21: Providing a composite or aggregate performance metric would undermine the purpose of SPEChpc96. SPEChpc96 is designed to inform users about how industrial-strength applications in their fields of analysis will perform. These users are particularly interested in how well their applications will scale as parallelism increases. This is why SPEChpc96 reporting pages provide metrics for systems with different numbers of processors running the same application and problem size.

Q22: How can I obtain SPEChpc96?

A22: SPEChpc96 is available on CD-ROM for $1,200; discounts are available for university and research organizations. For more information, contact SPEC.

Press contacts: