ORNL Sun Excalibur UltraSPARC-III tests

Oak Ridge National Laboratory (ORNL) is currently performing an in-depth evaluation of the Compaq AlphaServer SC parallel architecture as part of its evaluation of early systems project. The primary tasks of the evaluation are to

• determine the most effective approaches to using the AlphaServer SC;
• evaluate benchmark and application performance, and compare with similar systems from other vendors;
• evaluate system and system administration software reliability and performance;
• predict scalability, both in terms of problem size and in number of processors.

• architecture -- configuration summaries
• benchmarks -- benchmark descriptions
• low-level results -- base CPU and memory results
• shared-memory results -- single node/thread performance
• message-passing (MPI) results -- latency/bandwith, collectives

ARCHITECTURE

The Sparc III unit we tested had two processors sharing memory. Both the Alpha and SP consist of four processors sharing memory on a single node. The following table summarizes the main characteristics of the SP and Alpha.

BENCHMARKS

We have used widely available benchmarks in combination with our own custom benchmarks to characterize the performance of the Alpha SC cluster. Some of the older benchmarks may need to be modified for these newer faster machines -- increasing repetitions to avoid 0 elapsed times, increasing problem sizes to test out of cache performance. Unless otherwise noted, the following compiler switches were used on the Alpha and SP.

• EuroBen 3.9 -- provides serial benchmarks for low-level performance and applicaton kernels (linear algebra, eigen value, FFT, QR). Here is a summary of the benchmark modules. euroben-dm provides some communication and parallel (MPI) benchmarks. The web site includes results from other systems.
• lmbench -- provides insight into OS (UNIX) performance and memory latencies. The web site includes results from other systems.
• stream -- measures memory bandwidth for both serial and parallel configurations. The web site includes results from other systems.
• Custom low-level benchmarks that we have used over the years in evaluating memory and communication performance.

LOW LEVEL BENCHMARKS The following table compares the performance of the Alpha and SP for basic CPU operations. These numbers (peak average Mflops) are from the first 14 kernels of EuroBen's mod1a. The 14th kernel is a rough estimate of peak FORTRAN performance since it has a high re-use of operands.

The following table compares the performance of various intrinsics (EuroBen mod1f).

The following table compares optimized FORTRAN performance for Euroben mod2a, matrix-vector dot product and product.

The following table compares the single processor performance (Mflops) of the Alpha and SP for the Euroben mod2g, a 2-D Haar wavelet transform test.

The following plots the performance (Mflops) of Euroben mod2d, a dense eigenvalue test, for both optimized FORTRAN and using the BLAS from the vendor library. For the Alpha, -O4 optimization failed, so this data uses -O3. For the Sun, -fast failed for the Forte update 1 compiler, so this plot is without -fast (older compiler version was faster with -fast).

The following plots the performance (iterations/second) of Euroben mod2e, a sparse eigenvalue test.

Memory performance The, Sparc, SP and the Alpha have 64 KB L1 caches and 8 MB L2 caches. The following figure shows the data rates for a simple FORTRAN loop to load ( y = y+x(i)) for different vector sizes. For the Sparc we compare with and without the -fast compiler option.

The stream benchmark is a program that measures main memory throughput for several simple operations. The following table shows the memory data rates for a single processor.

The following figure shows the Mflops for one processor for various problem sizes for the EuroBen mod2f, a 1-D FFT. Data access is irregular, but cache boundaries are still apparent.

The hint benchmark measures computation and memory efficiency as the problem size increases. The following graph shows the performance of a single processor for the Alpha (66.9 MQUIPS), Sun (42.7 MQUIPS), and SP (27.3 MQUIPS). The L1 and L2 cache boundaries are visible.

The runtime for a FORTRAN molecular dynamics code, mdbnch, on the Sparc3 was 5.3 seconds (4.2 seconds on the SP, 3.6 seconds on the Alpha).

The lmbench benchmark measures various UNIX and system characeristics. Here are some preliminary numbers for runs on the sparc3, alpha, and SP. The cache/memory latencies reported by lmbench are

SHARED-MEMORY BENCHMARKS

Both the Alpha and SP consist of a cluster of shared-memory nodes, each node with four processors sharing a common memory. We tested the performance of a shared-memory with various C programs with explicit thread calls and with FORTRAN Open MP codes.

The following table shows the performance of thread/join in C for two processors. The test repeatedly creates and joins threads. Often, it is more efficient to create the threads once, and then provide them work as needed. I suspect this is what FORTRAN Open MP is doing for "parallel do". The table also includes time for iterative test of a parallel do in FORTRAN on two processors.

The following table shows the time required to lock-unlock using pthread_mutex_lock with various number of threads.

MESSAGE-PASSING BENCHMARKS

Internode communication can be accomplished with IP, PVM, or MPI. We report MPI performance using shared-memory on the Sparc3 (MPICH) and within a node on the SP and Alpha. The following table summarizes the measured communication characteristics of the three machines, using an 8 byte message for latency and a 1 MByte message for throughput. We also include node-to-node times for the SP and Alpha and network (IP) performance.

The following graph shows the MPI bandwidth for communications between two processrs measured with EuroBen's mod1h.

Since we have only 2 Sparc3 processors we can't say much about scaling of MPI communications, but the following table shows the performance of aggregate communication operations (barrier, broadcast, sum-reduction) using two processors for the three architectures.

Links/References

Worley's Sparc3 results PSTSWM

ORNL's Dunigan's Alpha/SP parallel performance

ORNL's Pat Worley's alpha evaluation

ORNL CCS pages for Alpha SC and SP3

UT student, Jay Patel's results July, 2000

sparc3 announcement

sun sparc 3 and solaris

sun performance tuning and sun hpc docs

Sun starfire enterprise 10000 and performance and sun hpc servers

Compaq alplha es40 cluster info and EV6 chip paper and alpha 21624 hardware ref. and compiler writer's guide

Compaq's alpha server performance info

Alpha's quadrix switch or older Meiko fat-tree network

IBM papers on POWER3 and RS6000 switch performance and SP2 architecture paper and other sp2 articles

power3 tutorial and IBM SP scientific redbook

IBM's essl scientific library and mass intrinsics and other optimization libraries mass, mpi, lapi, essl

ParkBench or euroben or NAS parallell benchmarks or hint

stream benchmark and splash and lmbench

PDS: The Performance Database Server linpack and such

benchmarks papers

atlas

SPEC

UT's papi performance counter API

Heller's rabbit

Monitoring Application Performance Using Hardware Counters

cpu timers japanese

ANL's MPICH performance