We have measured MPI performance on the NOW clusters. This document provides the performance data in both table and graph format.
Point-to-point benchmark -
comm.shar
written by Jack Dongarra and Tom Dunigan, May 1995.
Performance of MPI on top of GAM. Performance of MPI on top of AM2. Performance of MPI on top of AM2 with Shared Memory. Peroformance of MPI on top of Shared Memory SMP. Performance of MPI on top of P4 (10 Mbit Ethernet). Performance of MPI on top of P4 (Myrinet Switch ATM Network). NAS Parallel Benchmarks -
NPB 2 Parallel Benchmarks
Performance of MPI on top of GAM. Performance of MPI on top of AM2.
ScaLAPACK - A Scalable Library For Numerical Linear Algebra
Performance of MPI on top of GAM. Performance of MPI on top of AM2. System and Software Configuration
Ultra Sparc Cluster. CLUMP Cluster. Software configuration details. Point to Point One-Way Latency on UltraSparc Cluster
Point to Point One-Way Bandwidth on UltraSparc Cluster (Logscale)
Point to Point One-Way Bandwidth on UltraSparc Cluster (Non-Logscale)
Point to Point One-Way Performance on UltraSparc Cluster (Table)| Message Size (bytes) | Latency (µsec) | Bandwidth MBytes per sec | ||
|---|---|---|---|---|
| Minimum | Average | Maximum | ||
| 0 | 15.497208 | 16.676962 | 86.486343 | ------ |
| 4 | 23.007393 | 25.120676 | 141.024597 | 0.173857 |
| 8 | 24.974346 | 26.907980 | 127.494331 | 0.320329 |
| 16 | 25.510788 | 27.518570 | 135.540955 | 0.627186 |
| 32 | 28.014183 | 29.931068 | 171.542175 | 1.142279 |
| 64 | 26.524067 | 29.885174 | 189.006332 | 2.412903 |
| 128 | 30.994415 | 32.524406 | 145.494934 | 4.129776 |
| 256 | 40.471554 | 42.158783 | 132.024292 | 6.325430 |
| 512 | 52.988529 | 54.424046 | 132.024292 | 9.662469 |
| 1000 | 83.446503 | 85.931541 | 144.004822 | 11.983726 |
| 1024 | 82.015991 | 84.350761 | 165.998932 | 12.485370 |
| 2048 | 139.534470 | 145.732101 | 294.983398 | 14.677377 |
| 4096 | 254.511826 | 260.932037 | 391.483307 | 16.093555 |
| 8192 | 361.502167 | 372.565033 | 518.560425 | 22.660997 |
| 10000 | 400.960449 | 412.976318 | 590.980042 | 24.940115 |
| 16384 | 574.469543 | 587.943237 | 720.977783 | 28.520224 |
| 32768 | 994.026672 | 1028.798462 | 1207.530518 | 32.964909 |
| 65536 | 1854.002441 | 1916.803833 | 2646.505859 | 35.348389 |
| 100000 | 2767.503174 | 2827.186035 | 3497.481445 | 36.133652 |
| 131072 | 3561.019775 | 3631.266602 | 5395.531738 | 36.807434 |
| 262144 | 6938.517090 | 7063.841797 | 8363.485352 | 37.780983 |
| 524288 | 13690.531250 | 13905.816406 | 15866.995117 | 38.295666 |
| 1000000 | 26130.021484 | 26279.783203 | 30657.052734 | 38.270157 |
| 1048576 | 27150.511719 | 27541.296875 | 34483.015625 | 38.620857 |
Point to Point One-Way Latency on SUN Enterprise 5000 SMP
Point to Point One-Way Bandwidth on SUN Enterprise 5000 SMP (Logscale)
Point to Point One-Way Bandwidth on SUN Enterprise 5000 SMP (Non-LogScale)
Point to Point One-Way Performance on SUN Enterprise 5000 SMP (Table)| Message Size (bytes) | Latency (µsec) | Bandwidth MBytes per sec | ||
|---|---|---|---|---|
| Minimum | Average | Maximum | ||
| 0 | 10.550022 | 15.794218 | 46.491623 | ------ |
| 4 | 13.470650 | 17.901659 | 156.044952 | 0.296942 |
| 8 | 13.470650 | 17.836212 | 96.976753 | 0.593884 |
| 16 | 14.007092 | 18.175543 | 82.015991 | 1.142279 |
| 32 | 14.007092 | 19.110262 | 115.990639 | 2.284557 |
| 64 | 14.483929 | 20.076096 | 64.551826 | 4.418691 |
| 128 | 15.020370 | 23.255825 | 114.500526 | 8.521761 |
| 256 | 16.987324 | 26.602327 | 86.009499 | 15.070061 |
| 512 | 21.994114 | 31.268894 | 271.975983 | 23.278955 |
| 1000 | 23.543835 | 34.584286 | 100.493431 | 42.473965 |
| 1024 | 23.961067 | 35.827755 | 160.992142 | 42.735992 |
| 2048 | 41.484833 | 47.422230 | 176.012512 | 49.367439 |
| 4096 | 58.472157 | 61.613739 | 131.011002 | 70.050438 |
| 8192 | 92.983246 | 102.856216 | 234.007828 | 88.101891 |
| 10000 | 109.493729 | 131.855255 | 230.491165 | 91.329430 |
| 16384 | 163.495544 | 185.370026 | 288.009644 | 100.210686 |
| 32768 | 286.459930 | 317.540466 | 418.007385 | 114.389473 |
| 65536 | 580.489624 | 649.420776 | 759.005554 | 112.897797 |
| 100000 | 855.028625 | 944.446777 | 1042.962036 | 116.955147 |
| 131072 | 1123.547607 | 1205.272339 | 1280.009766 | 116.659058 |
| 262144 | 2472.996826 | 2545.465088 | 2708.017822 | 106.002563 |
| 524288 | 4651.486816 | 5040.685547 | 5219.519043 | 112.714066 |
| 1000000 | 9351.492188 | 9475.933594 | 9725.034180 | 106.934807 |
| 1048576 | 9813.964844 | 9906.967773 | 10177.016602 | 106.845306 |
Point to Point One-Way Latency on SUN Ultra Sparc Cluster
Point to Point One-Way Bandwidth on SUN Ultra Sparc Cluster (Logscale)
Point to Point One-Way Bandwidth on SUN Ultra Sparc Cluster (Non-Logscale)
Point to Point One-Way Performance on SUN Ultra Sparc Cluster (Table)| Message Size (bytes) | Latency (µsec) | Bandwidth MBytes per sec | ||
|---|---|---|---|---|
| Minimum | Average | Maximum | ||
| 0 | 36.478043 | 42.518852 | 370.979309 | ------ |
| 4 | 43.034554 | 49.059868 | 313.997284 | 0.092949 |
| 8 | 43.988228 | 49.665333 | 294.506561 | 0.181867 |
| 16 | 43.988228 | 50.563633 | 299.513336 | 0.363734 |
| 32 | 45.955181 | 52.802799 | 354.468811 | 0.696331 |
| 64 | 47.028065 | 56.672215 | 1570.522827 | 1.360890 |
| 128 | 49.531460 | 57.104469 | 493.049622 | 2.584216 |
| 256 | 56.564808 | 64.474403 | 220.000748 | 4.525782 |
| 512 | 73.015694 | 81.403496 | 280.559052 | 7.012192 |
| 1000 | 102.519989 | 111.058708 | 474.989410 | 9.754195 |
| 1024 | 104.486938 | 112.790825 | 512.003906 | 9.800268 |
| 2048 | 168.025497 | 175.781555 | 366.032135 | 12.188627 |
| 4096 | 291.466705 | 303.225708 | 506.997101 | 14.053063 |
| 8192 | 501.513489 | 521.024597 | 822.484497 | 16.334557 |
| 10000 | 566.482544 | 589.441467 | 1506.030518 | 17.652796 |
| 16384 | 831.007935 | 853.118469 | 1159.489136 | 19.715816 |
| 32768 | 1498.520386 | 1524.230713 | 2873.062988 | 21.866903 |
| 65536 | 2821.981934 | 2860.113037 | 4706.024902 | 23.223394 |
| 100000 | 4220.962402 | 4247.918945 | 5887.448730 | 23.691278 |
| 131072 | 5482.971680 | 5525.638184 | 13988.971680 | 23.905285 |
| 262144 | 10798.513672 | 10851.769531 | 16377.508789 | 24.275932 |
| 524288 | 21411.060547 | 21490.605469 | 31306.029297 | 24.486782 |
| 1000000 | 40659.964844 | 40784.503906 | 50556.003906 | 24.594217 |
| 1048576 | 42636.992188 | 42787.332031 | 55975.019531 | 24.593105 |
Point to Point One-Way Latency on SUN Ultra Sparc Cluster
Point to Point One-Way Bandwidth on SUN Ultra Sparc Cluster (Logscale)
Point to Point One-Way Bandwidth on SUN Ultra Sparc Cluster (Non-Logscale)
Point to Point One-Way Performance on SUN Ultra Sparc Cluster (Table)| Message Size (bytes) | Latency (µsec) | Bandwidth MBytes per sec | ||
|---|---|---|---|---|
| Minimum | Average | Maximum | ||
| 0 | 545.501709 | 553.534302 | 705.063354 | ------ |
| 4 | 551.044922 | 558.939209 | 826.001160 | 0.007259 |
| 8 | 542.521484 | 554.247925 | 645.458679 | 0.014746 |
| 16 | 545.501709 | 559.029419 | 757.515442 | 0.029331 |
| 32 | 546.991821 | 573.555603 | 1587.510132 | 0.058502 |
| 64 | 590.026367 | 614.009094 | 7968.485352 | 0.108470 |
| 128 | 693.023193 | 719.083191 | 8452.952148 | 0.184698 |
| 256 | 903.010376 | 921.634583 | 1034.498169 | 0.283496 |
| 512 | 1322.984741 | 1343.409790 | 2080.977051 | 0.387004 |
| 1000 | 2119.004639 | 2140.452637 | 3068.983643 | 0.471920 |
| 1024 | 2160.012695 | 2179.276611 | 2351.999268 | 0.474071 |
| 2048 | 3343.999268 | 3371.083008 | 8600.056641 | 0.612440 |
| 4096 | 5071.043945 | 5095.178223 | 5226.552246 | 0.807723 |
| 8192 | 8551.478516 | 8590.094727 | 9406.983398 | 0.957963 |
| 10000 | 11028.528320 | 11646.493164 | 18938.482422 | 0.906739 |
| 16384 | 17952.501953 | 18616.744141 | 24228.453125 | 0.912630 |
| 32768 | 33790.468750 | 35298.167969 | 39346.992188 | 0.969741 |
| 65536 | 66245.976562 | 68212.250000 | 71555.968750 | 0.989283 |
| 100000 | 102179.046875 | 103997.710938 | 109624.507812 | 0.978674 |
| 131072 | 133758.062500 | 136277.015625 | 142538.078125 | 0.979918 |
| 262144 | 268945.500000 | 271739.218750 | 304918.000000 | 0.974710 |
| 524288 | 531645.500000 | 540430.625000 | 750545.500000 | 0.986161 |
| 1000000 | 1011339.500000 | 1187585.625000 | 2942529.500000 | 0.988788 |
| 1048576 | 1059901.000000 | 1132243.000000 | 2919701.000000 | 0.989315 |
| System Model | Ultra 1 |
|---|---|
| Processor Name | 167 MHz UltraSPARC |
| Number of Processor | 1 |
| L1 Cache Per Processor | 32 kBytes |
| L2 Cache Per Processor | 512 kBytes |
| Main Memory | 128 MBytes |
| OS | SUN Solaris 2.5.1 |
| Network Interface | Myrinet Lanai 4.1 (37.5 MHz) SBus NI card |
| Number of NI | 1 |
| Network Switch | Myrinet 8-port Switch |
| System Model | Ultra Enterprise |
|---|---|
| Processor Name | 167 MHz UltraSPARC |
| Number of Processor | 8 |
| L1 Cache Per Processor | I-cache: 16 kBytes; D-cache: 16 kBytes |
| L2 Cache Per Processor | Unified cache 1 MBytes |
| Main Memory | 512 MBytes |
| OS | SUN Solaris 2.5.1 |
| Network Interface | Myrinet Lanai 4.1 (37.5 MHz) SBus NI card |
| Number of NI | 1 |
| Network Switch | Myrinet 8-port Switch |
