What is this thing called SPECviewperf®?

What is This Thing Called "SPECviewperf®"?

SPECviewperf® is a portable OpenGL performance benchmark program written in C. It was developed by IBM. Later updates and significant contributions were made by SGI, Digital (Compaq, HP), 3Dlabs (Creative Labs) and other SPECopc^SM project group members. SPECviewperf provides a vast amount of flexibility in benchmarking OpenGL performance. Currently, the program runs on most implementations of UNIX, Windows XP, Windows 2000, and Linux.

The OpenGL Performance Characterization (SPECopc) project group endorsed SPECviewperf as its first OpenGL benchmark. Performance numbers based on SPECviewperf were first published in the Q4 1994 issue of The GPC Quarterly.

SPECopc group member companies have ported the SPECviewperf code to their operating systems and window environments. The SPECopc project group maintains a single source code version of the SPECviewperf code that is available to the public.

Benchmarking with SPECviewperf

SPECviewperf parses command lines and data files, sets the rendering state, and converts data sets to a format that can be traversed using OpenGL rendering calls. It renders the data set for a pre-specified amount of time or number of frames with animation between frames. Finally, it outputs the results.

SPECviewperf reports performance in frames per second. Other information about the system under test -- all the rendering states, the time to build display lists (if applicable), and the data set used -- are also output in a standardized report.

A "benchmark" using SPECviewperf is really a single invocation of SPECviewperf with command-line options telling the SPECviewperf program which data set to read in, which texture file to use, what OpenGL primitive to use to render the data set, which attributes to apply and how frequently, whether or not to use display lists, and so on. One quickly realizes that there are an infinite number of SPECviewperf "benchmarks" (an infinite number of data sets multiplied by an almost infinite number of command-line states).

Real-World Benchmarking

SPECopc project group members recognize the importance of real-world benchmarks. From the beginning, the group has sought benchmarks representative of the OpenGL rendering portion of independent software vendor (ISV) applications. Along these lines, the project group has come up with what it calls a viewset. A viewset is a group of individual runs of SPECviewperf that attempt to characterize the graphics rendering portion of an ISV's application.

Viewsets differ from SPECapc benchmarks in that they exercise only the graphics functionality of the application. This enables direct performance comparisons of graphics hardware. Since SPECviewperf does not require an application software license to run, it is accessible to a wider range of users than SPECapc benchmarks. It is also easier to use and runs faster than SPECapc benchmarks.

Viewsets are generally not developed by the SPECopc project group; they come from the ISVs themselves. Members of the SPECopc project group often "sponsor" the ISV. Sponsorship entails helping the ISV in several areas, including how to obtain the SPECviewperf code, how to convert data sets to a SPECviewperf format, how to use SPECviewperf, how to create SPECviewperf tests to characterize the application, how to determine weights for each of the individual SPECviewperf tests based on application usage, and finally to help offer the viewset to the SPECopc project group for consideration as a standard SPECopc viewset. Any ISV wishing to develop a viewset should contact [email protected].

Currently, there are six standard SPECopc application viewsets:

3dsmax-02, based on SPECapc for 3ds max 3.1 configured with the Open GL driver, includes three models containing an average of 1.5 million vertices each, and tests performance of scenes with different levels of lighting.
dx-08, based on IBM's Data Explorer application, has 10 different tests.
drv-09, based on Intergraph's DesignReview model review package, has five different tests.
light-06, based on Discreet's Lightscape radiosity application, has four tests.
proe-02, based on SPECapc for Pro/ENGINEER 2001, measures two models in three modes - shaded, wireframe and hidden-line removal (HLR).
ugs-03, based on SPECapc for Unigraphics V17, tests performance based on an engine model containing 2.1 million vertices.

These type of applications typically render large data sets. All six viewsets represent relatively high-end applications.

The input to SPECviewperf is the command line. Here are the command line options for SPECviewperf 7.1:

Program options:

SPECviewperf v7.1

Program options:

-polygon -pg   <file>                 : Viewpoint object to be used in the tests

-triangle -tr  <file>                 : Viewpoint object to be used in the tests

-quad -qd      <file>                 : Viewpoint object to be used in the tests

-mesh -mh      <file>                 : Mesh object to be used in the tests

-frameattr -fa <file>                 : Mesh3 attributes for each frame

-rendermode -rm <mode>                : POINT, VECTOR, LINE, POLYGON, TMESH, TFAN,

                                        TRIANGLE, or QUAD - default LINE

-vcriteria -vcrit                     : AUX Visual selection criteria - EXACT, MIN

                                        - default MIN

-vid <id>                             : Ask AUX for visual with ID = <id>

-vaccum -vac                          : Ask AUX for an accumulation buffer visual

-valpha -val                          : Ask AUX for an alpha buffer visual

-vdepthbuffer -vz                     : Ask AUX for a depth buffer visual

-vstencil -vst                        : Ask AUX for a stencil buffer visual

-indirectrender -ir                   : Render indirect - default direct

-nodither -ndi                        : Disable dithering

-ortho -or                            : Parallel/Orthographic projection - default Perspective

-displaylist -dl                      : Render with display list mode

-vertexarray -va                      : Render with vertex array mode (OpenGL1.1)

-interleaved -il                      : Packing vertex data interleaved in memory

-colorper -cp  <mode>                 : FRAME = Color per Frame,

                                        PRIMITIVE = Color per Primitive,

                                        VERTEX = Color per Vertex - default FRAME

-texture -tx   <file>                 : Image for texturing

-texgen -txg <file> <mode>            : <file> is image for enviroment mapping

                                        <mode> is SPHERE_MAP, OBJECT_LINEAR, EYE_LINEAR

                                        - default EYE_LINEAR

-texobj -txo <file>                    : Texture object mode <file> is list of images

-magfilter -magf <flt>                 : NEAREST, LINEAR - default NEAREST

-minfilter -minf <flt>                : NEAREST, LINEAR, NEAREST_MIPMAP_NEAREST,

                                        LINEAR_MIPMAP_NEAREST, NEAREST_MIPMAP_LINEAR,

                                        LINEAR_MIPMAP_LINEAR - default NEAREST

-texenv -te <env>                     : Texture enviroment, MODULATE, DECAL, BLEND

                                        - default DECAL

-texcomp -tc <num>                    : Texture components where <num> is 1,2,3, or 4

                                        -default 3

-blend -bl                            : Enable Blending

-srcblendfunc -sbf                    : ZERO, ONE, DST_COLOR, ONE_MINUS_DST_COLOR, SRC_ALPHA,

                                        ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,

                                        SRC_ALPHA_SATURATE - default SRC_ALPHA

-dstblendfunc -dbf                    : ZERO, ONE, SRC_COLOR, ONE_MINUS_SRC_COLOR, SRC_ALPHA,

                                        ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,

                                        - default ONE_MINUS_SRC_ALPHA

-linewidth -lw <width>                : Linewidth for wire/vector tests - default 1.0

-xwinsize -xws <side>                 : Size of test windows X dimension - default 700

-ywinsize -yws <side>                 : Size of test windows Y dimension - default 700

-numframes -nf <num>                  : Number of frames to be rendered during measurement

                                        Takes priority over -mp

-numilights -nil <num>                : Turns on <num> infinite lights - default 0

-numllights -nll <num>                : Turns on <num> local lights - default 0

-colormaterial -cm <side> <mode>      : <side> is FRONT, BACK, FRONT_AND_BACK

                                        - default FRONT

                                        <mode> is AMBIENT, DIFFUSE, EMISSION, SPECULAR,

                                        AMBIENT_AND_DIFFUSE

                                        - default AMBIENT_AND_DIFFUSE

-backface -bf                         : Cull Backfacing primitives - default off

-frontface -ff                        : Cull Frontfacing primitives - default off

-singlebuffer -sb                     : Single buffer mode

-fog -fg                              : Enable fog

-linesmooth -ls                       : Enable line antialiasing

-polysmooth -ps                       : Enable polygon antialiasing

-facetnormal -fn                      : Use facet normals when lighting

-linestipple -lp                      : Enable line stipple

-polystipple -pp                       : Enable polygon stipple

-batch -bt <num>                      : Batch <num> primitives together per glBegin/glEnd

                                       Valid with POINT, VECTOR, TRIANGLE, and QUADS

-polymodefront -pmf                   : POINT, LINE, or FILL - default FILL

-polymodeback  -pmb                   : POINT, LINE, or FILL - default FILL

-flat -f                              : Set shademodel to FLAT - default GOURAUD

-zbuffer -zb                          : Enable zbuffer for tests - default off

-clip -c                              : Align object on 3D clip boundary

-lighttwoside -l2s                    : Light both sides of model

-localview -lv                        : Define local viewer for lit tests

-normalize -nm                        : Enable normal normalization

-minperiod -mp <num>                  : Set minimum testing period in seconds

-mblur <num>                          : Use motion blur with num being amount of decay

-aa_multi <x> <r>                     : Full scene antialiasing rendered x times at an

                                        offset of r.  r should be tuned to the viewset

-walkthru -wt                          : Walkthru mode

-geomrepeat -gr                        : Geometry repeat

-grab <script>                        : Screen capture

-grabonly -go                         : Don't run benchmark, just do screen capture

-swap <true/false>                    : Force byte swapping

-notitle -nt                          : Disable title screen

-startframe -sf <num>                 : Set starting frame of a walkthru

-framepause -fp                       : Pause after each frame waiting for enter key

-newview -nv                          : Enable new view frustum calculation

-fov <angle>                          : Set field of view in degrees

-spin x y z dx dy dz                  : Set rotation initial values and speeds (degrees)

-clipplane                            : Enables a user clip plane

-nomaterial                           : Disables material changes if present

-nolight                               : Disables viewperf lighting

-forcecalibrate                       : Forces viewperf calibration

-tracelights -tl                      : Use lights generated in scene

-norejection                          : Turns off trivial rejection for walk through.

-noframecounter                       : Turn off display of frame counter

-staticFrameCount -sfc <num>          : Count of frames between data edits (default = 10)

-staticBlockCount -sbc <num>          : Number of memory blocks changed per edit (default = 3)

-staticEditSize   -ses <num>          : Amount of frame data changed per edit (default = 0.001)

-staticEditsVisible -sev              : Attempt to make edits visible in the frame during execution.

-staticEditsDisabled                  : Allow all data to be static, no frame edits for entire test.

The -threads option is applicable only if SPECviewperf has been compiled with the C Preprocessor variable MP defined. SPECviewperf should only be compiled this way if the vendor supplied makefile has defined the MP flag (-DMP).

Here is an example of the command line from test #1 of the proe-02 viewset:
viewperf -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12 -forcecalibrate -xws 960 -yws 960 -grab grab.scr

This command line is providing the following directions:

invoke the SPECviewperf program
-mh specifies the format of the data set is mesh format (binary)
the data set file name is WorldCar6.mh3
-fa specifies some attributes to be processed per-frame (binary)
the attributes are stored in gradient.mha
-tracelights specifies that the lights contained in the command stream should be used
-va directs SPECviewperf to render using vertex arrays
-zb enables z-buffering
-mp 12 will set the minimum period to 12 seconds; SPECviewperf will run this test for at least 12 seconds
-forcecalibrate forces SPECviewperf to calibrate before running the test. This allows the frame rate to be determined so that sufficient frames are processed during the test for the defined test period.
-xws 960 -yws 960 this specifies the X and Y dimensions of the SPECviewperf window to be 960x960 pixels.
-grab grab.scr directs SPECviewperf to capture a section of the screen after completing the performance test.

Upon entering the command line, the system under test will run the benchmark for at least 12 seconds. This is to insure a valid test result. During the test, the SPECviewperf code renders the data set, WorldCar6.mh3, which is the WorldCar model used in the SPECapc for Pro/ENGINEER 2001™, rotating it about the center of the window with the above specified characteristics.

What follows is the output of SPECviewperf 7.1 running the first test from proe-02:

--------------------

Viewset: ProEngineer

Test Number 01: WorldCar6 : Weight 13.50

--------------------

    Viewperf Version                                   7.1

    Viewperf Arguments                                 -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12 -forcecalibrate -xws 960 -yws 960 -grab grab.scr

    Month                                              3

    Day                                                4

    Year                                               2003

    Host                                               system HOSTNAME

    Operating System                                   Microsoft Windows NT

    Operating System Version                           Version 5.1 (Build 2600: Service Pack 1)

    Host Vendor                                        unknown

    Host Model                                         AT/AT COMPATIBLE

    Host CPU                                           x86 Family 15 Model 2 Stepping 7

    Host CPU Count                                     unknown

    Host Memory Size (MB)                              1023

    Host Primary Cache Size (KB)                       unknown

    Host Secondary Cache Size (KB)                     unknown

    Window System                                      Win32

    Driver Version                                     Version of GFX driver

    OpenGL Vendor                                      Name of company making GFX card

    OpenGL Version                                     Version of OpenGL supported by GFX card

    OpenGL Extensions                                  List of all supported OpenGL Client extensions

    OpenGL Renderer                                    Name of GFX card

    OpenGL Client Vendor                               GFX Card OpenGL client vendor string

    OpenGL Client Version                              Version of OpenGL client supported by GFX card

    OpenGL Client Extensions                           List of all supported OpenGLClient extensions

    GLU Version                                        1.2.2.0 Microsoft Corporation

    GLU Extensions                                     List of all supported GLU extensions

    Direct Rendering                                   False

    Double Buffer                                      True

    Stereo                                             False

    RGBA                                               True

    Color Index Size                                   -1

    Red Size                                           8

    Green Size                                         8

    Blue Size                                          8

    Alpha Size                                         0

    Accum Red Size                                     16

    Accum Green Size                                   16

    Accum Blue Size                                    16

    Accum Alpha Size                                   16

    Depth Size                                         24

    Stencil Size                                       0

    Auxiliary Buffer Count                             0

    Frame BufferLevel                                  0

    Window Width (pixels)                              960

    Window Height (pixels)                             960

    Screen Width (pixels)                              1280

    Screen Height (pixels)                             1024

    Pixel Format ID                                    4

    Visual Selection Criteria                          MINIMUM

    Number of Execution Threads                        1

    Geometry File                                      WorldCar6

    Input Mode                                         -mh

    Minimum Test Period                                12.000000

    Number of Frames                                   0

    Number of Primitives                               319413

    Number of Vertices per Frame                       3917495

    Number of Vertices per Primitive                   12.264670

    Estimated bytes per frame                          0.000000

    Static frame count                                 0.000000

    Model edited every                                 10 frames

    Amount model changed                               0.001000

    Toggle Mode                                        NONE

    Batching Count                                     0

    Render Mode                                        LINE

    Color per                                          COLOR_PER_FRAME

    Orthographic Projection                            FALSE

    Display List                                       FALSE

    Vertex Array (OpenGL1.1)                           TRUE

    Interleaved Data                                   FALSE

    Clip Geometry                                      FALSE

    Walkthrough Mode                                   FALSE

    Geom Repeat Mode                                   FALSE

    Back Face Cull                                     FALSE

    Front Face Cull                                    FALSE

    Front Polygon Mode                                 FILL

    Back Polygon Mode                                  FILL

    Polygon Stipple Enable                             FALSE

    Polygon Antialiasing Enable                        FALSE

    Line Width                                         1.000000

    Line Stipple Enable                                FALSE

    Line Antialiasing Enable                           FALSE

    Number of Infinite Lights                          0

    Number of Local Lights                             0

    Color Material Enable                              FALSE

    Color Material Face                                FRONT

    Color Material Mode                                AMBIENT_AND_DIFFUSE

    Facet Normals                                      FALSE

    Two Sided Lighting Enable                          FALSE

    Local Viewer Enable                                FALSE

    Flat Shading                                       FALSE

    Fog Enable                                         FALSE

    Normal Normalization                               FALSE

    User Clip Plane                                    FALSE

    Material Changes Disabled                          FALSE

    Attribue Changes Disabled                          FALSE

    Trace Lights Enabled                               TRUE

    Texture Enable                                     FALSE

    TexObj Enable                                      FALSE

    Texture Generation Mode                            NO_TEXTURE_GENERATION

    Texture File                                       NONE

    Texture Minification Filter                        NEAREST

    Texture Magnification Filter                       NEAREST

    Texture Environment Mode                           DECAL

    Texture Components                                 3

    Depth Test Enable                                  TRUE

    Blend Enable                                       FALSE

    Source Blend Function                              SRC_ALPHA

    Destination Blend Function                         ONE_MINUS_SRC_ALPHA

    Dithering Enable                                   TRUE

    Motion Blur Amount                                 0.000000

    Full Scene Antialiasing Redraws                    0

    Full Scene Antialiasing Jitter Amount              0.000000

               Number of frames run: 280, Test period: 12.187000 (sec)

    23.0       frames/sec     --      -mh WorldCar6 -rm LINE -mp 12 -cp FRAME -ir -va -zb -tracelights -vz

During the calibration period SPECviewperf tests to see how many frames need to be rendered to meet the minimum time period and then runs the test.

The last line is the number of frames per second, which is the fundamental metric for each test. This is the result that is printed in the summary and individual report pages within the GPC News SPECopc section. Also printed in the summary and individual report pages is a single composite number for each viewset. This composite number is derived using a weighted geometric mean methodology (essentially the product of the results of each test raised to the power of the test weightings).

SPECviewperf 7.1

SPECviewperf 7.1 introduces new data changes to discourage optimizations that benefit the benchmark, but might not improve real-world application performance. SPECviewperf 7.1 inserts a small amount of variation at regular intervals within its viewsets. This ensures that the test system examines and processes each frame individually and is more aligned with the behavior of real-world applications. A second frame counter that moves around within the frame has also been added to the benchmark. This provides visual verification of whether frame changes are being rendered. For more information, please read the documentation provided with the SPECviewperf source code.

Rob Putney / Simon Green / Chris Watts / Bill Licea-Kane / Allen Jensen / Ian Williams

Standard Performance Evaluation Corporation