ORNL IBM Power4 (p690) evaluation

• 4/21/05 2 rails federation latency 6 us bandwidth 1635 MBs
• 6/15/04 --federation tests after microcode upgrade (4 paths/ 2 rails for multiple processors only) Federation colony today colony '02 1-rail '02 latency (us) 7 19 17 bandwidth (MBs) 1486 306 345 175 exchange (1x1)(MBs)1037 273 367 215 exchange (32x32) 6938 394 Following graph compares colony and federation for Euroben and ParkBench communication benchmarks. Revised 6/17/04
  
  The following plot compares the variations in round-trip time for an 8-byte message between nodes (1000 tests).
  
• 10/14/03 -- federation swtich testing on 2 32-cpu nodes:
  Federation specs: 2 GBs per port ( 2 ports), latency: 5 to 9 us
• updated message-passing results, Colony/PCI, dual rail, 6/1/02
• updated stream results with affinity on p690 5/15/02
• LPAR testing 2/15/02

  We compared the performance of 8-cpu/8 GB LPAR nodes to 8-thread tests on 16-cpu/16GB node and 32-cpu/32GB node. Arguably, when running just an 8 cpu test, the other cpu's on the non-LPAR nodes are wasted/unavailable.

  QR factorization, MPI, 8 cpu's Machine Gflops LPAR 7.1 16x16 7.9 32x32 8.2 The following reveals slight performance loss for competing LPARs on the same node. NAS OpenMP SP.C 8 threads 32x32 3.7 16x16 3.5 1 LPAR 2.83 2 LPARs 2.79 (fastest of the 2) 3 LPARs 2.72 4 LPARs 2.68

  In the following we look at the STREAM memory benchmark aggregate triad memory bandwidth far various number of streams on an LPAR and nodes.
  
  The LPAR threads are restricted to one L3 and 8 GB of memory. Notice that for one thread, the LPAR does better (probably due to lower memory latency, not having to access other L3's). This is consistent with Worley's results (next bullet).

• Worley's lpar tests 2/15/02
• Results from a few tests on 32 processor (1/14/2002)
• also view Pat Worley's power4 test results

Oak Ridge National Laboratory (ORNL) is currently performing an in-depth evaluation of the IBM Power 4 (p690) system as part of its evaluation of early systems project. The primary tasks of the evaluation are to

• determine the most effective approaches to using the IBM p690
• evaluate benchmark and application performance, and compare with similar systems from other vendors;
• evaluate system and system administration software reliability and performance;
• predict scalability, both in terms of problem size and in number of processors.

• architecture -- configuration summaries
• benchmarks -- benchmark descriptions
• low-level results -- base CPU and memory results
• shared-memory results -- single node/thread performance
• message passing results -- latency/bandwith, collectives
• kernels -- application kernel results

ARCHITECTURE

The present Power4 consists of one node with 16 processors (2 MCM's) sharing memory. Both the Alpha and SP3 consist of four processors sharing memory on a single node. The following table summarizes the main characteristics of the machines

BENCHMARKS

We have used widely available benchmarks in combination with our own custom benchmarks to characterize the performance of the SP4. Some of the older benchmarks may need to be modified for these newer faster machines -- increasing repetitions to avoid 0 elapsed times, increasing problem sizes to test out of cache performance. Unless otherwise noted, the following compiler switches were used on the Alpha and SP.

• ParkBench 2.1 -- provides low-level sequential and communication benchmarks, parallel linear algebra benchmarks, NAS parallel benchmarks, and compact application codes. Here is a summary of the benchmark modules. Codes are in FORTRAN. Results are often reported as least-squares fit of data. We report actual performance numbers.
• EuroBen 3.9 -- provides serial benchmarks for low-level performance and applicaton kernels (linear algebra, eigen value, FFT, QR). Here is a summary of the benchmark modules. euroben-dm provides some communication and parallel (MPI) benchmarks. The web site includes results from other systems.
• lmbench -- provides insight into OS (UNIX) performance and memory latencies. The web site includes results from other systems.
• stream -- measures memory bandwidth for both serial and parallel configurations. The web site includes results from other systems.
• Custom low-level benchmarks that we have used over the years in evaluating memory and communication performance.

We recently received an essl tuned for the power4. The following graph plots the power3 essl vs the new power4 essl (3.3) for the Euroben mod2b benchmark. The newer essl provides signifcant improvement. Also plotted is the performance of mod2b without essl, comparing arch=pwr3 versus arch=pwr4 for xlf (both run on the power4). The large difference between the "compiled from source" version and the ESSL version is typical -- even the best compilers are quite sensitive to the details of the source code organization for this class of algorithms.

Memory performance

Both the SP3 and the Alpha have 64 KB L1 caches and 8 MB L2 caches. The SP4 has a 32KB L1 (FIFO), a 1.4 L2 (shared between 2 processors), and a 128 MB L3. The following figure shows the data rates for a simple FORTRAN loop to load ( y = y+x(i)), store (y(i)=1), and copy (y(i)=x(i)), for different vector sizes. Data is also included for four threads. (Beware of the linear interpolation between data points, and note we need to extend the test beyond 128 MB to get out of the SP4 L3 cache. It has been suggested the the "dcbz" SP4 instruction that allocates the target cache line in the L2 without loading it from memory first could further improve SP4 performance. Also see McCalpin's stream2 benchmark.)

At the tail end of the graph above, the program starts fetching data from main memory. For load, a single Alpha thread is reading data at 1.7 GBs, the SP3 at 787 MBs. For four threads, the load per-cpu rate drops to 811 MBs for the Alpha and 322 MBs for the SP. The aggregate rate for 4 CPUs from the test is then 3.2 GBs for the Alpha compared to 1.3 GBs for the SP.

The stream benchmark is a program that measures main memory throughput for several simple operations. The aggregate data rate for multiple threads is reported in the following table. Recall, that the "peak" data rate for the Alpha is 5.2 GBs and for the SP3 is 1.6 GBs. Data for the 16-way SP3 (375 Mhz, Nighthawk II) is included too. Data for the Alpha ES45 (1 GHz) is obtained from the streams data base. Data for p690/sp4 is with affinity enabled (6/1/02).

The 9690/sp4 results are from June, 2002 with affinity supported by AIX. McCalpin reports the following improvement of aggregate STREAM with affinity on 32 processor p690 (4/15/02).

The following figure shows the Mflops for one processor for various problem sizes for the EuroBen mod2f, a 1-D FFT. Data access is irregular, but cache boundaries are still apparent.

The hint benchmark measures computation and memory efficiency as the problem size increases. (This is hint Version 1, 1994.) The following graph shows the performance of a single processor for the Alpha (66.9 MQUIPS), SP3 (27.3 MQUIPS), and SP4 (74.9 MQUIPS). The L1 and L2 cache boundaries are visible, as well as the SP4's L3 (128 MB).

The lmbench benchmark measures various UNIX and system characeristics. Here are some preliminary numbers for runs on a service and compute node of alpha and SP3/4 (version 2). (Results from previous lmbench version can be found here.) Open/close times are much slower for the Alpha, though file create/delete are faster on the Alpha. The cache/memory latencies reported by lmbench are

LOW LEVEL BENCHMARKS The following table compares the performance of the Alpha and SP for basic CPU operations. These numbers are from the first 14 kernels of EuroBen's mod1ac. The 14th kernel is a rough estimate of peak FORTRAN performance since it has a high re-use of operands.

The following table compares the performance of various intrinsics (EuroBen mod1f). For the SP, it also shows the effect of -O4 optimization versus -O3.

The following table compares optimized FORTRAN performance (no essl/cxml) for Euroben mod2a, matrix-vector dot product and product.

The following table compares the single processor performance (Mflops) of the Alpha and IBMs for the Euroben mod2g, a 2-D Haar wavelet transform test.

The following plots the performance (Mflops) of Euroben mod2d, a dense eigenvalue test, for both optimized FORTRAN and using the BLAS from the vendor library. For the Alpha, -O4 optimization failed, so this data uses -O3.

The following plots the performance (iterations/second) of Euroben mod2e, a sparse eigenvalue test.

We ran a number of tests of NCAR CCM Column Radiation Model (CRM), using different compiler options, libraries, and problem sizes (300+ test cases). We used the executables from the SP3 on the SP4. The SP4 provided speedups from 1.1 to 2.6 faster than the SP3. The FORTRAN code is dominated by exponentials and square roots. Also see Worley PSTSW climate code test results.

EuroBen's mod3a tests matrix computation with file I/O (out of core). The following tables compare the Alpha with the IBM. The run was made using /tmp, and no attempt was made to optimize I/O performance.

SHARED-MEMORY BENCHMARKS

Both the Alpha and IBMs consist of a cluster of shared-memory nodes, each node with four processors sharing a common memory (16 for sp4). We tested the performance of a shared-memory node with various C programs with explicit thread calls and with FORTRAN Open MP codes.

The following table shows the performance of thread/join in C as the master thread creates two, three, and four threads. The test repeatedly creates and joins threads.

The following table shows the time required to lock-unlock using pthread_mutex_lock with various number of threads. For the IBMs we use setenv SPINLOOPTIME 5000.

MESSAGE-PASSING BENCHMARKS

Internode communication can be accomplished with IP, PVM, or MPI. We report MPI performance over the Alpha Quadrics network and the IBM SP. Each node (4 CPUs) share a single network interface. However, each CPU is a unique MPI end point, so one can measure both inter-node and intra-node communication. The following table summarizes the measured communication characteristics of the Alpha, SP3, and the SP4. SP4 is currently based on Colony switch via PCI.

The following graph shows bandwidth for communication between two processors on the same node using MPI from both EuroBen's mod1h and ParkBench comms1. The SP3 performs better for smaller messages.

The sp4 is presently equiped with a Colony switch for inter-node communcation but is limited by the PCI interface at this time (May, 2002). The following graph shows bandwidth for communication between two nodes.

The p690 also supports dual rail Colony connections that roughly doubles the bandwidth for large messages as illustrated in the following graph.

The following table shows the performance of aggregate communication operations (barrier, broadcast, sum-reduction) using one processor per node (N) and all processors on each node(n). Recall that the sp4 has 16 processors per node (the other 4 per node). Times are in microseconds.

PARALLEL KERNEL BENCHMARKS

Both ParkBench and EuroBen (euroben-dm) had MPI-based parallel kernels. However, the euroben-dm communication model was to have the processes do all of their send's before issuing receive's. On the SP, this model resulted in deadlock for the larger problem sizes. The EAGER_LIMIT can be adjusted to make some progress on the SP3 but the deadlocks could not be completely eliminated. MPI buffering on the Alpha was adequate. The maximum MPI buffering on an SP3 node was 64 MBytes, on the Alpha, 191 MBytes.

The following table show MPI parallel performance of the LU benchmark (64x64x64) for the Alpha and SP. The first column pair is one processor per node, the second pair is using all processors per node. These tests used standard FORTRAN (no vendor libraries).

The following graph shows the aggregate Mflops for a multi-grid (MG) kernel from ParkBench/NAS Parallel Benchmark. This for a 256x256x256 doubleword grid with MPI and Wallcraft's co-array version and also OpenMP on the IBM. Revised 9/3/03.

The following graph shows the aggregate Mflops for a conjugate gradient (CG) kernel (CLASS=A) from NAS Parallel Benchmarks 2.3 using MPI and OpenMP. Revised 9/22/03

We also ran the OpenMP version of the NAS Parallel Benchmarks (PBN-O-3.0b4). The following table compares the performance of three of those benchmarks on the power4 to the NERSC Power3 (seaborg, 16-way shared memory, 375 MHz).

Related Links

• Pat Worley's power4 test results (includes Compaq ES45)
• IBM power4 microacrhitecture white paper or here
• IBM's POWER4 Processor Introduction and Tuning Guide
• IBM power3 architecture
• ORNL's power4 cheetah
• IBM's power4 announcement regatta, p690 and performance
• McCalpin configuring the p690 for performance
• Lefelar's power 4 tests worley's pstwm
• our recent tests of the Cray X1 and SGI Altix
• our earlier power3/alpha tests
• NERSC blue planet
• AIX clocks/timers

Research Sponsors

Mathematical, Information, and Computational Sciences Division, within the Office of Advanced Scientific Computing Research of the Office of Science, Department of Energy. The application-specific evaluations are also supported by the sponsors of the individual applications research areas.

---

Last Modified thd@ornl.gov (touches: )
back to Tom Dunigan's page or the ORNL Evaluation of Early Systems page or
ornl | ccs | csm: research | people | sitemap | disclaimer | search