SPEC CPU®2017 Known Problems

Contents

Introduction

Open Problem Reports

3. Headings may appear to be off-center in ASCII plain text reports.

5. Why can't I override fields from sysinfo?

8. When measuring power, runcpu may hang.

9. Reportable Check -> FAILED may sometimes be incorrect.

10. Setting MATHLIBOPT may cause violations of the base consistency rules. Use some other make variable.

11. On certain Windows systems, updates are claimed to be corrupt.

12. On certain Linux / Unix systems, updates are not available.

Closed Problem Reports

1. Some lines may be tiny in PDF reports.

2. Some lines may be too wide in HTML reports.

4. Plain text columns should have better alignment.

6. The runcpu --comment option must not contain any blanks.

7. Can't use an undefined value as an ARRAY reference while formatting results.

Introduction

This document describes known problems in SPEC CPU®2017, a product of the SPEC® non-profit corporation (about SPEC).

SPEC CPU 2017 was tested on a variety of systems prior to release, including Unix, Linux, and Windows systems. Nevertheless, as with any software product, problems may be found too late in the testing process to be fixed in the release, or may be discovered after release.

If you discover problems that are not listed here or in the FAQ, please report them to SPEC.

About the format of this document:

• Problems may be added in the Open section from time to time. In each case, a workaround is given, if one is known.
• When problems are fixed, they are moved to the Closed section.

Notice to maintainers: When updating this document, please do not ever re-use problem numbers.

Open Problem Reports

Please see the latest on-line version of this document for information on open problems and suggested workarounds.

3. Headings may appear to be off-center in ASCII plain text reports.

• Problem summary: For text reports, when comparing report titles vs. data tables, the headings appear to be too far to the right. (#960)
• Details: The headings are actually centered relative to the widest line in the report. Therefore, if the report has many lines that are 80 columns wide, but one line that is 250 columns wide, then the headings will be printed near to column 125.
• Status: As of SPEC CPU 2017 v1.1, this problem is less severe, because the common cause for very wide lines has been fixed. Nevertheless, it is possible that you might notice that headings appear further right than desired for plain text reports.

5. Why can't I override fields from sysinfo?

• Problem summary: Unwanted lines are sometimes included for fields that are supplied by both sysinfo and by the config file. (#591)

• Details: The documentation says that when fields (such as hw_memory) are supplied by sysinfo, a config file can supply a different value by using a higher-priority section, for example:

  intrate,fprate,intspeed,fpspeed:
     hw_memory001 = 256 GB (8 x 32 GB 2Rx4 PC4-2133P-R,
     hw_memory002 = running at 1600)

• Limitation: If you want your higher-priority section to "win", then you need to make sure you supply exactly the same field names, including any line numbers:

  If sysinfo supplies    and you supply     Then the report will have
  -------------------    --------------     -------------------------
  hw_memory001           hw_memory001       2 lines for memory description
  hw_memory002           hw_memory001
  
  hw_memory001           hw_memory100       4 lines for memory description
  hw_memory002           hw_memory200
  
  sw_os                  sw_os100           3 lines for the operating system
                         sw_os110

• Workaround: Check what sysinfo will supply, and match it. If that is not possible, then you will need to edit the fields prior to using rawformat to prepare final reports.

8. When measuring power, runcpu may hang.

• Problem summary: Certain power measurements may hang. (#963)

• Details: The problem occurs only if all the following conditions are present:

  1. You have selected runcpu --power or the corresponding config file option.
  2. You are running benchmarks drawn from multiple suites.
  3. One of the suites uses preenv and another does not.

• Workaround: Split your runcpu command into one for each suite.

  Instead of:
     runcpu --config=myconfig intrate fprate
  Try:
     runcpu --config=myconfig intrate 
     runcpu --config=myconfig fprate
              

9. Reportable Check -> FAILED may sometimes be incorrect.

• Problem summary: During the transition period to a new version of SPEC CPU 2017, the local copy of the submission checker may incorrectly say that a result FAILED to meet the submission requirements. (#980)

• Details: When you use rawformat to prepare your results for publication, it automatically runs a local copy of the submission checker. The local copy can help you meet SPEC's requirements for publication at spec.org.

  Your local copy of the submission checker may differ from the master copy at spec.org.

  In particular, your copy does not know about transition periods to new releases, during which SPEC may allow submissions with both older and newer versions of the SPEC CPU suite. Your local copy of the submission checker will always attempt to encourage you to use the latest version.

  For example, when using rawfomat from v1.1.0 to format results that were generated by v1.0.5, this message is printed:

  format: Reportable Check -> FAILED.  Found the following errors:
     - Reportable runs must use a release version of CPU2017 that is approved by SPEC.

  The transition period for v1.1.0 is defined to be 10-Sep-2019 through 25-Nov-2019. During that period, only, you may ignore the above message if you are formatting v1.0.5 results with the v1.1.0 formatter.

  Of course, you should not ignore any other subcheck messages. For example, as usual, it will complain if you try to set hw_avail to 12/19 and will insist that you should spell that as Dec-2019.

• Workaround: Use the latest version both for your runs and when formatting results. Make sure you are current by using runcpu --update.

10. Setting MATHLIBOPT may cause violations of the base consistency rules. Use some other make variable.

• Problem summary: Setting MATHLIBOPT can cause inconsistent base flags for intrate or intspeed.

• Details: The Make Variables document says that users are allowed to set MATHLIBOPT in their config files. Unfortunately, doing so may cause benchmarks to be compiled inconsistently in base, therefore violating the same-for-all rule.

  The problem arises because in the integer suites, most benchmarks automatically include whatever is specified for MATHLIBOPT (by default, -lm). One benchmark does not, namely 557.xz_r (rate) / 657.xz_s (speed). If you explicitly set MATHLIBOPT to something other than the default, then when you run intrate or intspeed, your report will show that base optimization flags are not consistent.

  The report is correct - the flags really will be inconsistent, which is a violation of the same-for-all rule. However, the performance effect may be minor or nil, and the fault is arguably due to the SPEC tools rather than due to the user.

• Workaround: If you want to change the math libraries, force MATHLIBOPT to be empty, and set your desired libraries in some other Make Variable. For example, this method would apply mymath to all benchmarks:

  default=base:     
     OPTIMIZE       = -g -O3 -march=native 
     MATHLIBOPT     = 
     LIBS           = -lmymath

  and this method would apply mymath only to the integer C benchmarks:

  default=base: 
     OPTIMIZE           = -g -O3 -march=native 
  intrate,intspeed:
     MATHLIBOPT         = 
     OPTIMIZATION_CLIBS = -lmymath

11. On certain Windows systems, updates are claimed to be corrupt.

• Problem summary: On Windows systems running SPEC CPU 2017 v1.1.8 and older, the command runcpu --update fails with a message similar to:

  Update: Downloading update: 16 MB/16 MB (100%; 6983 KB/s))
  
  ERROR: The downloaded update was corrupted
  
  Suite update failed.

• Details: The download is not actually corrupt. Nevertheless, runcpu stubbornly refuses to apply it, on all Windows systems running SPEC CPU 2017 v1.1.8 or earlier.

• Workaround: Request a new ISO image by contacting [email protected]. In your email, please include the exact error message that was printed, and please mention that you were sent there by "Known Problems 11".

12. On certain Linux / Unix systems, updates are not available.

• Problem summary: On Linux / Unix systems running SPEC CPU 2017 v1.0.1, the command runcpu --update fails with message:

  Update: No updates available for current version v1.0.1

• Details: The first version of the updater, which shipped with v1.0.1, consistently fails. If you are running this very old version, see the Workaround just below. If you are running any other version of SPEC CPU 2017 on Linux / Unix systems, your problem is something else (probably proxies: see the proxy discussion in the documentation for runcpu --update .)

• Workarounds: For anything other than v1.0.1, investigate proxies. Only for v1.0.1: request a new ISO image by contacting [email protected]. In your email, please include the exact error message that was printed, and please mention that you were sent there by "Known Problems 12".

Closed Problem Reports

SPEC CPU®2017 Known Problems: Copyright © 2017-2021 Standard Performance Evaluation Corporation (SPEC®)