ORNL Opteron Evaluation

Revised for 1.8 GHz Opteron 4/9/04. Results for older opteron 1.6GHz

The results presented here are from standard benchmarks and some custom benchmarks and, as such, represent only one part of the evaluation. An IBM SP4, IBM Winterhawk II (noted as SP3 in the following tables and graphs),Cray X1, SGI Altix (Itanium 2), and Compaq Alpha ES40 at ORNL were used for comparison with the Opteron. The results below are in the following categories:

• architecture -- configuration summaries
• benchmarks -- benchmark descriptions
• memory performance -- memory performance
• low-level results -- base CPU and intrinsics results
• shared-memory results -- single node/thread performance
• message passing results -- latency/bandwith, collectives
• kernels -- application kernel results

ARCHITECTURE

The Opteron has 4 cpu's and 16 GB of memory (1.8 GHz). The X1 at ORNL has 8 nodes. Each node has 4 MSPs, each MSP has 4 SSPs, and each SSP has two vector units. The Cray "processor/CPU" in the results below is one MSP. All 4 MSP's on a node share memory. The Power4 consists of one node with 32 processors sharing memory. Both the Alpha and SP3 consist of four processors sharing memory on a single node. All memory is accessible on the Altix. The following table summarizes the main characteristics of the machines

BENCHMARKS

We have used widely available benchmarks in combination with our own custom benchmarks to characterize the performance of the X1. Some of the older benchmarks may need to be modified for these newer faster machines -- increasing repetitions to avoid 0 elapsed times, increasing problem sizes to test out of cache performance. Unless otherwise noted, the following compiler switches were used on the Alpha and SP.

• ParkBench 2.1 -- provides low-level sequential and communication benchmarks, parallel linear algebra benchmarks, NAS parallel benchmarks, and compact application codes. Here is a summary of the benchmark modules. Codes are in FORTRAN. Results are often reported as least-squares fit of data. We report actual performance numbers.
• EuroBen 3.9 -- provides serial benchmarks for low-level performance and applicaton kernels (linear algebra, eigen value, FFT, QR). Here is a summary of the benchmark modules. euroben-dm provides some communication and parallel (MPI) benchmarks. The web site includes results from other systems.
• lmbench -- provides insight into OS (UNIX) performance and memory latencies. The web site includes results from other systems.
• stream -- measures memory bandwidth for both serial and parallel configurations. Also we use the MAPS memory benchmark. The web sites include results from other systems.
• Custom low-level benchmarks that we have used over the years in evaluating memory and communication performance.

MEMORY PERFORMANCE

The stream benchmark is a program that measures main memory throughput for several simple operations. The aggregate data rate for multiple threads is reported in the following table. Recall, that the "peak" memory data rate for the X1 is 200 GBs, Alpha is 5.2 GBs, and for the SP3 is 1.6 GBs. Data for the 16-way SP3 (375 Mhz, Nighthawk II) is included too. Data for the Alpha ES45 (1 GHz) is obtained from the streams data base. Data for p690/sp4 is with affinity enabled (6/1/02). The X1 uses (aprun -A). The Opteron is supposed to have greater than 5.3 GB/s/cpu memory bandwidth, we don't see that yet?

The MAPS benchmark also characterizes memory access performance. Plotted are load/store bandwidth for sequential (stride 1) and random access.

The tabletoy benchmark (C) makes random writes of 64-bit integers in a shared memory, parallelization is permitted with possibly non-coherent updates. The X1 number is for vectorizing the inner loop (multistreaming was an order of magnitude slower 88 MBs). Data rate in the following table is for a 268MB table. We include multi-threaded altix, opteron, sp3 (NERSC), and sp4 data as well. Revised 2/9/04

The hint benchmark measures computation and memory efficiency as the problem size increases. (This is C hint version 1, 1994.) The following graph shows the performance of a single processor for the Opteron (176 MQUIPS), X1 (12.2 MQUIPS), Alpha (66.9 MQUIPS), Altix (88.2 MQUIPS), and SP4 (74.9 MQUIPS). The L1 and L2 cache boundaries are visible, as well as the Altix and SP4's L3.

Here results from LMbench.

LOW LEVEL BENCHMARKS (single processor)

The following table compares the performance of the X1, Alpha, and SP for basic CPU operations. These numbers are from the first few kernels of EuroBen's mod1ac. The 14th kernel (9th degree poly) is a rough estimate of peak FORTRAN performance since it has a high re-use of operands. (Revised 2/9/04)

The following table compares the performance of various intrinsics (EuroBen mod1f). For the SP, it also shows the effect of -O4 optimization versus -O3. (Revised 2/9/04)

The following plot compares the DAXPY performance of the Opteron and Itanium (Altix).

The following table compares the single processor performance (Mflops) of the Alpha and IBMs for the Euroben mod2g, a 2-D Haar wavelet transform test. (Revised 2/9/04)

The following plots the performance (Mflops) of Euroben mod2d, a dense eigenvalue test, for both optimized FORTRAN and using the BLAS from the vendor library. For the Alpha, -O4 optimization failed, so this data uses -O3.

The following plots the performance (iterations/second) of Euroben mod2e, a sparse eigenvalue test. (Revised 2/9/04)

The following figure shows the FORTRAN Mflops for one processor for various problem sizes for the EuroBen mod2f, a 1-D FFT. Data access is irregular, but cache effects are still apparent. (Revised 2/9/04).

The following compares a 1-D FFT using the FFTW benchmark.

The following graph plots 1-D FFT performance using the vendor library (-lacml, -lscs, -lsci or -lessl), initialization time is not included. Revised 2/9/04

SHARED-MEMORY BENCHMARKS

Both the Alpha and IBMs consist of a cluster of shared-memory nodes, each node with four processors sharing a common memory (16 for X1 and 32 for sp4). The X1 is cache-coherent within a node, but the memory space is global across all nodes. The Opteron shares memory among 4 processors. We tested the performance of a shared-memory node with various C programs with explicit thread calls and with FORTRAN Open MP codes.

The X1 pthreads model permits up to 16 SSP threads (-h ssp) or 4 MSP threads where each thread can also be multithreaded on each MSP. The following table shows the performance of thread/join in C as the master thread creates two, three, and four threads. The test repeatedly creates and joins threads. Revised 2/24/04.

The following table shows the time required to lock-unlock using pthread_mutex_lock with various number of threads. For the IBMs we use setenv SPINLOOPTIME 5000.

The following table compares the performance of simple C barrier program using a single lock and spinning on a shared variable along with pthread_yield.