Possible performance regression?
As part of HP's Open Source and Linux Organization's Performance and Scalability Group, I've noticed what looks to be a regression in U320 SCSI performance coming down the pike. Background: These measurements were performed on Red Hat RHEL4 update 2 (2.6.9-based) and a "generic" 2.6.14-based kernel; I used an HP RX4640 with 4 dual-U320 LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08) adapters, each with two buses containing 5 72GB/15k U320 drives (for a total of 40 drives). The runs were done in single-user mode (to minimize effects from other tasks), and report results in megabytes per second. I ran tests for various block sized IOs (1KB, 2KB, 4, ...256KB per IO).
While trying to gage other things, I ran across this set of strange phenomena on single-disk IOs (meaning: tests run one at a time on an individual disk):
Read averages:
BS(KB) RHEL4U2 2.6.14 %-diff
------ -------- -------- ------
1 23.8 12.3 -48.4%
2 46.6 21.3 -54.2%
4 87.5 44.0 -49.7%
8 89.9 51.8 -42.4%
16 89.9 74.2 -17.4%
32 89.9 89.9 0.0%
64 89.9 89.9 0.0%
128 89.9 89.9 0.0%
256 89.9 89.8 -0.1%For reads, we see a huge average drop for block sizes up to 32KB.
Write averages:
BS(KB) RHEL4U2 2.6.14 %-diff
------ -------- -------- ------
1 3.6 3.6 -0.2%
2 8.2 8.1 -1.3%
4 19.7 19.0 -3.2%
8 52.6 49.7 -5.5%
16 84.8 78.3 -7.6%
32 84.7 78.8 -7.0%
64 84.8 79.9 -5.8%
128 84.6 81.8 -3.3%
256 84.7 82.0 -3.1%For writes, an across the board smaller decrease in average performance. The real kicker though came out when I show individual disk results:
Going back to the read cases: Here are a couple of (representative) drives - all the data is added as an attachment - I see some drives with monstrously large performance reductions like:
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5 2.6.14 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6
Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.0%
(0,1) means target 1 on bus 0, and then the columns represent different transfer sizes (1K up to 256K). I also see some that show little if any difference:
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9 2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9 Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
(Which tends to explain the average being "only" about 50% as listed in the first table.) And there are other drives which show some unchanged, and some changed results:
(3,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.1 86.6 91.2 91.2 91.2 91.2 91.2 91.2 2.6.14.2 23.9 46.3 66.9 31.2 91.2 91.2 91.2 91.2 91.2
Differ 0.4% 0.4% -22.7% -65.8% 0.0% 0.0% 0.0% 0.0% 0.0%
(3,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.1 89.8 89.7 89.8 89.8 89.8 89.8 2.6.14.2 23.8 46.2 15.6 89.8 62.5 89.8 89.8 89.8 89.3 Differ 0.8% 0.4% -81.9% 0.0% -30.3% 0.0% 0.0% 0.0% -0.6%
One thing I will point out: while I haven't looked to see if the same drivers consistently show similar issues, I have done multiple runs with the same characteristics noted in the averages -- meaning, it's not single run peculiarities in the general sense. [Next I am going to see if I can match drives from run-to-run to see if the same ones exhibit "bad" read regressions, or whether that moves around... I am also going to check out runs on straight 2.6.9 (instead of using Red Hat's kernel...)]
For the write case, the drives exhibited similar differences, but on a much smaller scale - here are some examples:
(0,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 54.7 86.2 85.9 86.0 85.9 85.9 2.6.14.2 3.5 8.1 19.0 52.0 79.5 80.0 80.9 83.1 83.4 Differ -2.8% -1.2% -3.1% -4.9% -7.8% -6.9% -5.9% -3.3% -2.9%
(0,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.5 86.8 86.7 86.7 86.3 86.7 2.6.14.2 3.6 8.1 19.0 51.1 79.8 80.5 81.3 83.0 83.9 Differ 0.0% 0.0% -2.6% -4.5% -8.1% -7.2% -6.2% -3.8% -3.2%
(1,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 55.0 86.7 87.3 87.1 87.0 86.8 2.6.14.2 3.5 8.1 19.0 52.0 80.2 80.9 82.7 84.0 84.0 Differ -2.8% -1.2% -3.1% -5.5% -7.5% -7.3% -5.1% -3.4% -3.2%
(1,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 86.2 86.1 85.9 86.1 86.5 2.6.14.2 3.6 8.1 19.0 51.1 78.9 80.0 82.0 83.1 83.3 Differ 0.0% 0.0% -2.6% -4.7% -8.5% -7.1% -4.5% -3.5% -3.7%
So, the natural question - has anybody else noted such things? Any ideas as to why?
Thanks,
Alan D. Brunelle
Read averages:
BS(KB) RHEL4U2 2.6.14.2 %-diff
------ -------- -------- ------
1 23.8 12.3 -48.4%
2 46.6 21.3 -54.2%
4 87.5 44.0 -49.7%
8 89.9 51.8 -42.4%
16 89.9 74.2 -17.4%
32 89.9 89.9 0.0%
64 89.9 89.9 0.0%
128 89.9 89.9 0.0%
256 89.9 89.8 -0.1%Write averages:
BS(KB) RHEL4U2 2.6.14.2 %-diff
------ -------- -------- ------
1 3.6 3.6 -0.2%
2 8.2 8.1 -1.3%
4 19.7 19.0 -3.2%
8 52.6 49.7 -5.5%
16 84.8 78.3 -7.6%
32 84.7 78.8 -7.0%
64 84.8 79.9 -5.8%
128 84.6 81.8 -3.3%
256 84.7 82.0 -3.1%-------------------------------
Read data:
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6
Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.1%
(0,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.9 87.6 89.7 89.7 89.7 89.7 89.7 89.7 2.6.14.2 3.9 7.8 15.6 31.3 62.5 89.7 89.7 89.7 89.7
Differ -83.6% -83.4% -82.2% -65.1% -30.3% 0.0% 0.0% 0.0% 0.0%
(0,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.5 89.6 89.7 89.6 89.6 89.7 89.6 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.7 89.6 89.7 89.3 Differ -83.6% -83.3% -82.4% -65.2% -30.3% 0.1% 0.0% 0.0% -0.3%
(0,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.6 89.6 89.6 89.5 89.6 89.6 89.6 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.6 89.6 89.6
Differ -83.6% -83.3% -82.4% -65.2% -30.2% 0.1% 0.0% 0.0% 0.0%
(0,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.3 89.7 89.6 89.7 89.6 89.6 89.6 2.6.14.2 3.9 7.8 15.6 31.2 61.7 89.7 89.6 89.6 89.5 Differ -83.6% -83.3% -82.3% -65.2% -31.1% 0.0% 0.0% 0.0% -0.1%
(1,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.3 89.6 89.6 89.7 89.6 89.6 89.6 2.6.14.2 3.9 7.8 16.5 31.2 66.8 89.7 89.6 89.6 89.7
Differ -83.6% -83.3% -81.3% -65.2% -25.4% 0.0% 0.0% 0.0% 0.1%
(1,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.8 88.3 90.0 89.9 90.0 89.9 90.0 89.9 2.6.14.2 24.0 7.8 15.6 31.2 62.5 89.6 90.0 90.0 89.9 Differ 0.4% -83.3% -82.3% -65.3% -30.5% -0.4% 0.1% 0.0% 0.0%
(1,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.8 87.9 90.4 90.4 90.3 90.3 90.3 90.4 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 90.3 90.3 90.2 Differ -83.7% -83.3% -82.3% -65.5% -30.9% -0.4% 0.0% 0.0% -0.2%
(1,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.5 89.9 89.8 89.8 89.8 89.9 89.8 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 89.8 89.8 89.7 Differ -83.6% -83.3% -82.4% -65.3% -30.4% 0.1% 0.0% -0.1% -0.1%
(1,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.7 89.9 89.8 89.8 89.9 89.8 89.8 2.6.14.2 3.9 7.8 15.6 31.3 61.4 89.8 89.8 89.8 89.8 Differ -83.6% -83.3% -82.2% -65.2% -31.6% 0.0% -0.1% 0.0% 0.0%
(2,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 88.6 89.7 89.7 89.8 89.8 89.8 89.8 2.6.14.2 3.9 16.7 88.8 31.2 89.7 89.7 89.7 89.7 89.8 Differ -83.6% -64.3% 0.2% -65.2% 0.0% -0.1% -0.1% -0.1% 0.0%
(2,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.4 90.0 89.9 89.9 89.9 89.9 89.9 2.6.14.2 3.9 10.6 88.5 89.9 89.9 90.0 90.0 90.0 90.0 Differ -83.7% -77.4% 0.1% -0.1% 0.0% 0.1% 0.1% 0.1% 0.1%
(2,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.9 88.1 90.2 90.3 90.2 90.2 90.2 90.2 2.6.14.2 3.9 29.0 88.1 90.3 90.3 90.3 90.2 90.2 90.2
Differ -83.6% -38.2% 0.0% 0.1% 0.0% 0.1% 0.0% 0.0% 0.0%
(2,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.8 87.8 89.9 89.8 89.9 89.8 89.8 89.8 2.6.14.2 3.9 47.0 88.0 89.9 89.8 89.9 89.9 89.8 89.7 Differ -83.7% 0.4% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1%
(2,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.9 89.9 90.0 90.0 89.9 90.0 90.0 2.6.14.2 24.0 11.5 87.9 90.0 89.9 90.0 89.9 89.9 89.9 Differ 0.8% -75.4% 0.0% 0.1% -0.1% 0.0% 0.0% -0.1% -0.1%
(3,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.1 86.6 91.2 91.2 91.2 91.2 91.2 91.2 2.6.14.2 23.9 46.3 66.9 31.2 91.2 91.2 91.2 91.2 91.2
Differ 0.4% 0.4% -22.7% -65.8% 0.0% 0.0% 0.0% 0.0% 0.0%
(3,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.1 89.8 89.7 89.8 89.8 89.8 89.8 2.6.14.2 23.8 46.2 15.6 89.8 62.5 89.8 89.8 89.8 89.3 Differ 0.8% 0.4% -81.9% 0.0% -30.3% 0.0% 0.0% 0.0% -0.6%
(3,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.0 89.7 89.7 89.7 89.7 89.7 89.7 2.6.14.2 23.7 46.2 86.2 31.2 89.7 89.7 89.7 89.7 89.3 Differ 0.4% 0.4% 0.2% -65.2% 0.0% 0.0% 0.0% 0.0% -0.4%
(3,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.5 89.9 89.9 89.9 90.0 90.0 89.9 2.6.14.2 23.8 46.2 15.6 90.0 62.5 90.0 89.9 89.9 90.0 Differ 0.8% 0.4% -82.0% 0.1% -30.5% 0.1% -0.1% -0.1% 0.1%
(3,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 85.9 89.9 89.9 89.9 89.9 89.9 89.9 2.6.14.2 23.7 46.1 86.2 89.9 89.9 89.6 90.0 89.9 89.8 Differ 0.4% 0.2% 0.3% 0.0% 0.0% -0.3% 0.1% 0.0% -0.1%
(4,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 87.7 89.6 89.5 89.6 89.6 89.5 89.6 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.2 89.6 89.5 89.6 Differ -83.7% -83.4% -82.2% -65.2% -30.2% -0.4% 0.0% 0.0% 0.0%
(4,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.2 89.5 89.5 89.5 89.5 89.5 89.6 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.6 89.5 Differ -83.7% -83.4% -82.3% -65.1% -30.2% 0.1% 0.0% 0.1% -0.1%
(4,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.8 90.9 90.8 90.9 90.8 90.9 90.9 2.6.14.2 3.9 7.8 15.6 31.2 62.5 90.9 90.9 90.9 90.6 Differ -83.7% -83.4% -82.4% -65.7% -31.2% 0.0% 0.1% 0.0% -0.3%
(4,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.6 89.7 89.6 89.7 89.6 89.6 89.7 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.7 89.6 89.6 89.6 Differ -83.7% -83.4% -82.4% -65.2% -30.2% 0.0% 0.0% 0.0% -0.1%
(4,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 24.0 47.0 88.5 89.9 89.9 89.9 89.9 89.9 89.9 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 89.9 89.9 89.9
Differ -83.8% -83.4% -82.4% -65.3% -30.5% 0.0% 0.0% 0.0% 0.0%
(5,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.2 86.0 89.7 89.8 89.7 89.7 89.7 89.7 2.6.14.2 3.9 7.8 15.6 31.3 62.5 89.7 89.7 89.7 89.6 Differ -83.6% -83.1% -81.9% -65.1% -30.4% 0.0% 0.0% 0.0% -0.1%
(5,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.2 86.1 89.9 89.9 89.9 89.9 89.9 89.9 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.9 89.9 89.9 89.6 Differ -83.6% -83.1% -81.9% -65.3% -30.5% 0.0% 0.0% 0.0% -0.3%
(5,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.7 46.2 86.7 89.6 89.6 89.6 89.6 89.6 89.6 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.6 89.6 89.6 89.5 Differ -83.5% -83.1% -82.0% -65.2% -30.2% 0.0% 0.0% 0.0% -0.1%
(5,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.7 46.1 86.5 89.7 89.6 89.7 89.6 89.7 89.6 2.6.14.2 3.9 7.8 15.6 31.2 62.5 89.7 89.7 89.7 89.7
Differ -83.5% -83.1% -82.0% -65.2% -30.2% 0.0% 0.1% 0.0% 0.1%
(5,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.1 86.6 90.6 90.5 90.5 90.6 90.5 90.5 2.6.14.2 3.9 7.8 15.6 31.3 62.5 90.6 90.5 90.5 90.6 Differ -83.6% -83.1% -82.0% -65.5% -30.9% 0.1% -0.1% 0.0% 0.1%
(6,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 89.0 89.6 89.6 89.6 89.6 89.7 89.7 2.6.14.2 24.1 47.1 89.3 89.6 89.7 89.6 89.6 89.7 89.6 Differ 0.8% 0.4% 0.3% 0.0% 0.1% 0.0% 0.0% 0.0% -0.1%
(6,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 89.8 89.8 89.9 89.9 89.9 89.9 2.6.14.2 24.1 47.1 88.8 89.8 89.9 89.9 89.9 89.8 89.9 Differ 0.8% 0.4% 0.3% 0.0% 0.1% 0.0% 0.0% -0.1% 0.0%
(6,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.9 87.7 89.8 89.9 89.8 89.8 89.9 89.9 2.6.14.2 24.0 7.9 88.1 89.9 89.8 89.9 89.8 89.9 89.7 Differ 0.8% -83.2% 0.5% 0.1% -0.1% 0.1% 0.0% 0.0% -0.2%
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9 2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9 Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
(6,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 89.0 90.8 90.7 90.8 90.7 90.8 90.7 2.6.14.2 24.0 9.9 89.2 31.2 90.7 90.7 90.7 90.8 90.8 Differ 0.8% -78.8% 0.2% -65.6% 0.0% -0.1% 0.0% 0.0% 0.1%
(7,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.1 86.5 89.7 89.7 89.7 89.7 89.7 89.7 2.6.14.2 23.8 46.2 86.8 89.7 89.7 89.4 89.7 89.7 89.7 Differ 0.8% 0.2% 0.3% 0.0% 0.0% -0.3% 0.0% 0.0% 0.0%
(7,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.5 90.7 90.8 90.7 90.8 90.8 90.7 2.6.14.2 23.8 7.8 86.4 90.8 90.7 90.7 90.7 90.8 90.6 Differ 0.8% -83.0% -0.1% 0.1% -0.1% 0.0% -0.1% 0.0% -0.1%
(7,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.1 86.0 89.8 89.7 89.7 89.8 89.8 89.8 2.6.14.2 23.8 46.2 86.3 89.7 61.4 89.7 89.7 89.7 89.5 Differ 0.8% 0.2% 0.3% -0.1% -31.5% 0.0% -0.1% -0.1% -0.3%
(7,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.1 86.6 89.7 89.7 89.6 89.6 89.7 89.6 2.6.14.2 23.8 46.2 15.6 31.2 89.6 89.3 89.6 89.7 89.6 Differ 0.8% 0.2% -82.0% -65.2% -0.1% -0.3% 0.0% 0.0% 0.0%
(7,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.6 46.0 86.5 89.6 89.6 89.6 89.6 89.6 89.6 2.6.14.2 18.9 46.2 15.6 31.2 89.6 89.6 89.6 89.6 89.6
Differ -19.9% 0.4% -82.0% -65.2% 0.0% 0.0% 0.0% 0.0% 0.0%
-------------------------------
Write data:
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.1 19.8 52.5 81.1 81.0 81.6 81.2 82.2 2.6.14.2 3.5 8.1 19.2 50.2 75.7 75.9 77.0 79.0 78.8 Differ 0.0% 0.0% -3.0% -4.4% -6.7% -6.3% -5.6% -2.7% -4.1%
(0,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.1 53.4 85.9 86.2 86.4 86.2 86.1 2.6.14.2 3.5 8.1 19.3 51.2 79.8 80.6 81.5 83.4 83.3 Differ -2.8% -1.2% -4.0% -4.1% -7.1% -6.5% -5.7% -3.2% -3.3%
(0,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 54.7 85.5 85.7 85.9 85.9 85.2 2.6.14.2 3.5 8.1 18.9 52.3 78.9 79.3 80.5 82.7 82.7 Differ -2.8% -1.2% -3.1% -4.4% -7.7% -7.5% -6.3% -3.7% -2.9%
(0,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 54.7 86.2 85.9 86.0 85.9 85.9 2.6.14.2 3.5 8.1 19.0 52.0 79.5 80.0 80.9 83.1 83.4 Differ -2.8% -1.2% -3.1% -4.9% -7.8% -6.9% -5.9% -3.3% -2.9%
(0,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.5 86.8 86.7 86.7 86.3 86.7 2.6.14.2 3.6 8.1 19.0 51.1 79.8 80.5 81.3 83.0 83.9 Differ 0.0% 0.0% -2.6% -4.5% -8.1% -7.2% -6.2% -3.8% -3.2%
(1,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 55.0 86.7 87.3 87.1 87.0 86.8 2.6.14.2 3.5 8.1 19.0 52.0 80.2 80.9 82.7 84.0 84.0 Differ -2.8% -1.2% -3.1% -5.5% -7.5% -7.3% -5.1% -3.4% -3.2%
(1,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 86.2 86.1 85.9 86.1 86.5 2.6.14.2 3.6 8.1 19.0 51.1 78.9 80.0 82.0 83.1 83.3 Differ 0.0% 0.0% -2.6% -4.7% -8.5% -7.1% -4.5% -3.5% -3.7%
(1,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.0 52.8 83.3 83.5 83.6 82.6 83.2 2.6.14.2 3.6 8.1 19.4 50.2 75.0 78.0 79.5 81.0 80.9 Differ 0.0% -1.2% -3.0% -4.9% -10.0% -6.6% -4.9% -1.9% -2.8%
(1,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 54.8 86.0 85.8 85.5 86.1 85.4 2.6.14.2 3.6 8.1 19.0 51.9 78.6 79.8 81.4 82.6 82.5 Differ 0.0% -1.2% -3.1% -5.3% -8.6% -7.0% -4.8% -4.1% -3.4%
(1,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.1 53.9 87.3 87.1 87.3 86.9 87.0 2.6.14.2 3.6 7.7 19.3 51.5 77.4 80.8 83.1 84.1 84.3 Differ 0.0% -6.1% -4.0% -4.5% -11.3% -7.2% -4.8% -3.2% -3.1%
(2,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 52.6 86.5 86.4 86.8 85.9 86.0 2.6.14.2 3.6 8.1 19.0 49.0 80.5 80.0 80.9 83.4 83.4 Differ 0.0% -1.2% -3.1% -6.8% -6.9% -7.4% -6.8% -2.9% -3.0%
(2,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.3 53.1 84.9 84.6 84.8 85.0 84.4 2.6.14.2 3.6 8.1 19.0 50.7 78.0 79.6 79.3 81.5 82.4 Differ 0.0% 0.0% -1.6% -4.5% -8.1% -5.9% -6.5% -4.1% -2.4%
(2,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 50.3 85.4 85.6 85.1 85.7 85.3 2.6.14.2 3.6 8.1 19.2 47.9 79.0 79.4 80.5 82.3 82.3 Differ 0.0% -1.2% -3.5% -4.8% -7.5% -7.2% -5.4% -4.0% -3.5%
(2,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 51.0 86.9 86.7 86.6 86.7 86.2 2.6.14.2 3.6 8.1 19.1 48.5 80.3 80.3 81.4 83.8 83.9 Differ 0.0% -1.2% -4.0% -4.9% -7.6% -7.4% -6.0% -3.3% -2.7%
(2,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.8 50.2 84.1 84.1 84.5 83.5 83.4 2.6.14.2 3.5 8.1 19.0 47.6 77.8 78.2 79.3 81.7 81.4 Differ -2.8% 0.0% -4.0% -5.2% -7.5% -7.0% -6.2% -2.2% -2.4%
(3,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 49.7 86.7 87.0 87.0 86.2 87.1 2.6.14.2 3.6 8.1 19.1 46.4 80.7 81.1 81.0 83.5 83.7 Differ 0.0% -1.2% -4.0% -6.6% -6.9% -6.8% -6.9% -3.1% -3.9%
(3,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 51.3 86.6 86.2 86.6 87.1 86.6 2.6.14.2 3.5 8.1 19.1 48.3 80.7 80.6 80.7 83.4 83.8 Differ -2.8% -1.2% -4.0% -5.8% -6.8% -6.5% -6.8% -4.2% -3.2%
(3,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.9 51.2 85.1 85.4 85.4 84.9 85.0 2.6.14.2 3.5 8.1 19.1 48.2 79.0 79.4 79.4 82.2 82.5 Differ -2.8% 0.0% -4.0% -5.9% -7.2% -7.0% -7.0% -3.2% -2.9%
(3,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 51.1 87.8 87.4 87.3 87.6 87.5 2.6.14.2 3.6 8.1 19.0 47.9 81.3 81.3 81.6 84.4 84.7 Differ 0.0% -1.2% -3.1% -6.3% -7.4% -7.0% -6.5% -3.7% -3.2%
(3,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 50.6 85.8 85.9 86.0 86.1 85.6 2.6.14.2 3.6 8.1 19.2 48.1 79.8 79.8 80.4 82.7 83.4 Differ 0.0% -1.2% -3.5% -4.9% -7.0% -7.1% -6.5% -3.9% -2.6%
(4,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.9 52.5 87.1 86.4 87.1 86.9 86.8 2.6.14.2 3.5 8.1 19.2 49.2 80.7 80.4 82.1 84.0 83.9 Differ -2.8% -1.2% -3.5% -6.3% -7.3% -6.9% -5.7% -3.3% -3.3%
(4,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.3 53.5 86.9 87.1 86.6 86.7 86.8 2.6.14.2 3.6 8.1 18.9 51.0 80.2 80.6 81.9 84.0 83.9 Differ 0.0% 0.0% -2.1% -4.7% -7.7% -7.5% -5.4% -3.1% -3.3%
(4,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 50.2 83.7 83.9 83.7 83.7 83.8 2.6.14.2 3.5 8.0 18.9 48.1 78.1 77.9 79.6 81.4 81.5 Differ -2.8% -1.2% -3.1% -4.2% -6.7% -7.2% -4.9% -2.7% -2.7%
(4,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.4 52.9 84.9 84.6 85.4 84.6 84.9 2.6.14.2 3.5 8.1 18.8 48.6 78.7 79.0 79.6 81.8 82.0 Differ -2.8% -1.2% -3.1% -8.1% -7.3% -6.6% -6.8% -3.3% -3.4%
(4,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.4 52.8 84.4 84.3 84.8 84.3 84.1 2.6.14.2 3.6 8.1 18.9 50.3 78.5 78.4 79.3 80.9 82.2 Differ 0.0% 0.0% -2.6% -4.7% -7.0% -7.0% -6.5% -4.0% -2.3%
(5,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.1 19.7 50.9 81.4 81.8 81.6 81.3 81.1 2.6.14.2 3.5 7.9 18.8 47.6 74.9 74.8 76.7 77.9 77.2 Differ 0.0% -2.5% -4.6% -6.5% -8.0% -8.6% -6.0% -4.2% -4.8%
(5,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.8 51.2 82.5 83.1 82.6 82.8 82.4 2.6.14.2 3.5 8.0 18.9 47.7 76.8 77.4 79.1 80.4 80.7 Differ -2.8% -1.2% -4.5% -6.8% -6.9% -6.9% -4.2% -2.9% -2.1%
(5,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 53.8 86.0 86.0 86.5 85.8 86.2 2.6.14.2 3.5 7.8 19.0 49.7 78.8 80.2 81.5 83.1 83.4 Differ -2.8% -4.9% -2.6% -7.6% -8.4% -6.7% -5.8% -3.1% -3.2%
(5,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.1 19.3 51.8 79.1 79.0 78.6 78.7 79.2 2.6.14.2 3.5 7.9 18.6 47.1 72.9 74.1 76.2 77.0 77.3 Differ 0.0% -2.5% -3.6% -9.1% -7.8% -6.2% -3.1% -2.2% -2.4%
(5,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 51.2 82.5 82.8 82.7 82.1 82.3 2.6.14.2 3.6 8.0 18.9 46.2 76.0 77.0 79.0 79.9 80.2 Differ 0.0% -2.4% -3.6% -9.8% -7.9% -7.0% -4.5% -2.7% -2.6%
(6,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.5 8.2 19.4 55.5 80.0 80.2 80.4 79.8 80.3 2.6.14.2 3.5 8.1 18.8 52.3 75.1 74.7 75.7 77.5 77.6 Differ 0.0% -1.2% -3.1% -5.8% -6.1% -6.9% -5.8% -2.9% -3.4%
(6,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.3 53.4 80.7 80.1 80.3 80.2 80.0 2.6.14.2 3.6 8.1 18.9 50.5 74.6 75.1 76.3 77.9 77.9 Differ 0.0% -1.2% -2.1% -5.4% -7.6% -6.2% -5.0% -2.9% -2.6%
(6,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.1 53.5 86.0 85.6 86.5 86.1 86.3 2.6.14.2 3.6 8.1 19.4 51.1 79.4 80.3 80.7 82.9 83.3 Differ 0.0% -1.2% -3.5% -4.5% -7.7% -6.2% -6.7% -3.7% -3.5%
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 85.8 85.7 86.2 84.3 84.9 2.6.14.2 3.6 8.1 19.0 51.1 78.8 78.5 79.7 80.0 81.7 Differ 0.0% 0.0% -2.6% -4.7% -8.2% -8.4% -7.5% -5.1% -3.8%
(6,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.6 50.5 78.1 78.4 78.0 78.1 78.3 2.6.14.2 3.5 8.0 18.9 48.3 72.5 72.9 74.5 75.7 77.3 Differ -2.8% -2.4% -3.6% -4.4% -7.2% -7.0% -4.5% -3.1% -1.3%
(7,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.5 53.6 86.6 86.6 86.4 86.5 86.7 2.6.14.2 3.6 8.1 18.9 50.7 80.2 80.2 81.0 83.7 83.8 Differ 0.0% 0.0% -3.1% -5.4% -7.4% -7.4% -6.3% -3.2% -3.3%
(7,2) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 20.0 51.9 86.6 86.4 87.0 86.9 86.9 2.6.14.2 3.6 8.1 19.2 49.7 80.4 80.9 81.3 84.2 83.8 Differ 0.0% -1.2% -4.0% -4.2% -7.2% -6.4% -6.6% -3.1% -3.6%
(7,3) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.1 19.9 52.7 82.3 82.7 82.5 82.5 82.3 2.6.14.2 3.5 8.1 19.3 50.3 76.8 76.4 78.1 79.7 80.2 Differ -2.8% 0.0% -3.0% -4.6% -6.7% -7.6% -5.3% -3.4% -2.6%
(7,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 54.6 86.1 86.4 86.1 86.1 86.3 2.6.14.2 3.5 8.1 18.9 52.1 79.8 80.1 80.6 83.5 83.1 Differ -2.8% -1.2% -3.1% -4.6% -7.3% -7.3% -6.4% -3.0% -3.7%
(7,5) 1 2 4 8 16 32 64 128 256
RHEL4U2 3.6 8.2 19.5 54.6 84.6 84.4 84.9 84.2 84.2 2.6.14.2 3.6 8.1 18.9 51.9 78.3 78.3 78.9 81.6 81.7 Differ 0.0% -1.2% -3.1% -4.9% -7.4% -7.2% -7.1% -3.1% -3.0%
Quoted 10 lines · Alan D. Brunelle
As part of HP's Open Source and Linux Organization's Performance and Scalability Group, I've noticed what looks to be a regression in U320 SCSI performance coming down the pike. Background: These measurements were performed on Red Hat RHEL4 update 2 (2.6.9-based) and a "generic" 2.6.14-based kernel; I used an HP RX4640 with 4 dual-U320 LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08) adapters, each with two buses containing 5 72GB/15k U320 drives (for a total of 40 drives). The runs were done in single-user mode (to minimize effects from other tasks), and report results in megabytes per second. I ran tests for various block sized IOs (1KB, 2KB, 4, ...256KB per IO).
This data doesn't seem to be mirrored by tiobench tests on other drivers, so it's probably fusion specific.
When a similar problem was found in the adaptec driver a while back, it was due to a bug which set the TCQ depth to 1, which looks very consistent with your results, so you might try starting there.
James
Quoted 7 lines · Alan D. Brunelle
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5 2.6.14 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.0%
Quoted 9 lines
(0,1) means target 1 on bus 0, and then the columns represent different transfer sizes (1K up to 256K). I also see some that show little if any difference:
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9 2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9 Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
Did you apply [PATCH] mptfusion : dv performance fix by
Eric Dean Moore? (Dec. 1st, 2005) W/o that patch the first four disks of an 1030 are performing 3.5 MB/s instead of 70 :-)
It was funny how slow a RAID 5-0 can be when four of fourteen disks are slow...
GS
Signature · 5 lines
Gerhard Schneider Institute of Lightweight Design and e-Mail:[email protected] Structural Biomechanics (E317) Tel.: +43 1 58801 31716 Vienna University of Technology / Austria Fax: +43 1 58801 31799 A-1040 Wien, Gusshausstrasse 27-29 http://www.ilsb.tuwien.ac.at/~gs/
Quoted 20 lines · Gerhard Schneider
Quoted 5 lines · Alan D. Brunelle
(0,1) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.8 46.8 87.8 89.6 89.5 89.6 89.5 89.5 89.5 2.6.14 3.9 7.8 15.6 31.2 62.5 89.6 89.5 89.5 89.6Differ -83.6% -83.3% -82.2% -65.2% -30.2% 0.0% 0.0% 0.0% 0.0%
Quoted 11 lines
(0,1) means target 1 on bus 0, and then the columns represent different transfer sizes (1K up to 256K). I also see some that show little if any difference:
(6,4) 1 2 4 8 16 32 64 128 256
RHEL4U2 23.9 46.9 88.5 90.0 89.9 89.9 89.9 90.0 89.9 2.6.14.2 24.1 47.0 88.5 90.0 89.9 90.0 89.9 89.9 89.9 Differ 0.8% 0.2% 0.0% 0.0% 0.0% 0.1% 0.0% -0.1% 0.0%
Quoted 10 lines
Did you apply [PATCH] mptfusion : dv performance fix by
Eric Dean Moore? (Dec. 1st, 2005) W/o that patch the first four disks of an 1030 are performing 3.5 MB/s instead of 70 :-)
It was funny how slow a RAID 5-0 can be when four of fourteen disks are slow...GS
I do not know how to grab that specific patch, but I am just trying a 2.6.15-rc6 kernel, and things seem much better. I'm hoping that 2.6.15-rc6 contains the patch you mention?
Thanks,
Alan
Quoted 4 lines · Alan D. Brunelle
I do not know how to grab that specific patch, but I am just trying a 2.6.15-rc6 kernel, and things seem much better. I'm hoping that 2.6.15-rc6 contains the patch you mention?
I do not know -rc6, but if mptbase.c contains the functions mpt_alt_ioc_wait and mpt_bringup_adapter the patch has been applied. Worked for me :-)
GS
Signature · 5 lines
Gerhard Schneider Institute of Lightweight Design and e-Mail: [email protected] Structural Biomechanics (E317) Tel.: +43 1 58801 31716 Vienna University of Technology / Austria Fax: +43 1 58801 31799 A-1040 Wien, Gusshausstrasse 27-29 http://www.ilsb.tuwien.ac.at/~gs/