SCSI Disk layer performance
My FC array seems to be faster than Linux can handle.
Even though I'm using FC-2 cards, Linux seems to put an artificial limit at FC-1 speeds. This is true in two cases:
1) dd to the raw device (actually, the limit is at about 120MB/s).
2) dd over the file system, even reading/writing multiple files simultaneously (about 90MB/s).
Using sgp_dd, I can easily get >190MB/s, but neither sg_dd or dd do as well (so, it's not the qla2300 module... it's something above).
Is there any method that might assure more parallel I/O in the SCSI layer? Is 2.6 the answer?
I've also seen some differences between 2.4 kernels. 2.4.21 gets 120MB/s via "dd" to the raw device, until the sg module is loaded, then it drops down to less than 100MB/s, and there are frequent memory warnings. 2.4.23 and .24 don't get the warnings, but irrespective of the sg module being loaded, get the slower speed on the raw device. I'm guessing that'll translate to even slower speeds on the file system, but I've yet to test that.
Furthermore, my reads seem to be especially slow. The SAN maker thinks that this might be due to the I/O scheduling in the SCSI Disk layer. Specifically, it's easy to see from the SAN perspective (using a statistics utility that shows read and write scsi command lengths) that reads are broken into small chunks by the SCSI Disk layer. In watching read lengths from the SAN, using "sg_dd", you can exactly specify read length, and no I/O's get broken up into chunks smaller than the bpt*512 (what is requested on the sg_dd command line is what the SAN sees). In "dd", you can specify a read block size on the command line, but the SAN shows that Linux is breaking that up into much smaller chunks.
Is there any way to force Linux to use large chunks?
Thanks,
Chris
Signature · 2 lines
Chris Worley <[email protected]> Linux Networx
Quoted 4 lines · Chris Worley
My FC array seems to be faster than Linux can handle.
Even though I'm using FC-2 cards, Linux seems to put an artificial limit at FC-1 speeds. This is true in two cases:
What kernel tree do you use? If you're looking for decent scsi performance use linux 2.6 or the SuSE/RH vendor trees. Stock Linux 2.4 is basically unusable for higher scsi loads in SMP enviroments.
Christoph Hellwig a e'crit:
Quoted 11 lines · Chris Worley
Quoted 5 lines
My FC array seems to be faster than Linux can handle.
Even though I'm using FC-2 cards, Linux seems to put an artificial limit at FC-1 speeds. This is true in two cases:
What kernel tree do you use? If you're looking for decent scsi performance use linux 2.6 or the SuSE/RH vendor trees. Stock Linux 2.4 is basically unusable for higher scsi loads in SMP enviroments.
Can you give more details about this ?
Instead of using vendor trees, is it possible to use specific and validated patchs about this ?
vendors trees are awfully patched and difficult to maintain/upgrade for customers...
Signature · 1 line
Fabien S
Quoted 8 lines · Fabien Salvi
Quoted 3 lines
What kernel tree do you use? If you're looking for decent scsi performance use linux 2.6 or the SuSE/RH vendor trees. Stock Linux 2.4 is basically unusable for higher scsi loads in SMP enviroments.
Can you give more details about this ?
Instead of using vendor trees, is it possible to use specific and validated patchs about this ?
It's rather hard as there's a lot of them and they depend on each other. Doug Ledford sayd he's going to sort put it into the 2.4 BK repos at linux-scsi.bkbits.net one day, you could monitor it.
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Fabien Salvi
Quoted 3 lines
What kernel tree do you use? If you're looking for decent scsi performance use linux 2.6 or the SuSE/RH vendor trees. Stock Linux 2.4 is basically unusable for higher scsi loads in SMP enviroments.
Quoted 8 lines
Quoted 4 lines
Can you give more details about this ?
Instead of using vendor trees, is it possible to use specific and validated patchs about this ?
It's rather hard as there's a lot of them and they depend on each other. Doug Ledford sayd he's going to sort put it into the 2.4 BK repos at linux-scsi.bkbits.net one day, you could monitor it.
Yes, I'm still working on it. The sheer number of changes we make to the SCSI stack internally makes it difficult to sort the bug fixes from all the other stuff. And, of course, if you call totally piss poor SCSI performance a bug, then a lot of the patches are in fact bug fixes since they drastically impact the 2.4 SCSI stack speed ;-)
In any case, I've sorted out the bare bugfixes (which is relatively small actually), and am now putting together the 2.4-everything tree that has all the scsi stuff in it. Keep in mind that it isn't just a "throw our patches on the current 2.4 bk tree from Marcelo". I have to merge the patches and check for areas where new code between 2.4.21 and current makes the things like the iorl patch have bugs.
Anyway, it's coming.
Signature · 4 lines
Doug Ledford <[email protected]> 919-754-3700 x44233 Red Hat, Inc. 1801 Varsity Dr. Raleigh, NC 27606