Promise SX8 driver performance
Hello all. I've got a Promise SX8 SATA controller and the performance seems a little off. I'm using a debian-built 2.6.9 kernel and the read/write I/O seems to level off around 80-90 MB/s. When I use the same hardware in Windows, sequential I/O goes up to 218 MB/s. (That's about the limit of my motherboard's buses.) If I move some disks to the on-board SATA controller, I can get the I/O to go higher but never as high as in Windows.
Is this a problem with my expectations, my testing, or the driver?
Some Bonnie numbers for a raid 0 array of Seagate 7200.7's (2 GB file, sequential, MB/s):
Disks/cntl Write Read ----- ----- ---- 1 (sx8) 57 63 2 (sx8) 69 75
2 (ESB) 111 121
3 (sx8) 75 79
3 (1 sx8, 2 ESB) 141 125
4 (sx8) 89 80
4 (2 sx8, 2 ESB) 135 150
5 (sx8) 88 79 6 (sx8) 91 75
Some numbers from Sandra (ugh) in W2K Pro with the latest Promise drivers and using software RAID 0 (sequential, MB/s):
Disks/cntl Write Read ----- ----- ---- 1 (sx8) 56 62 2 (sx8) 106 122 3 (sx8) 151 176 4 (sx8) 178 209 5 (sx8) 180 218 6 (sx8) 179 218 6 (4 sx8, 2 ESB) 169 207
My full hardware:
Supermicro P4SCi motherboard (7210 chipset, 6300ESB SATA on-board) 3.0 GHz P4 Northwood w/ HT enabled
(2) 512MB DDR400 dual channel (non-ECC)
Promise SX8 SATA controller
(6) 200 GB Seagate 7200.7
(1) 40 GB Seagate Baraccuda IV (boot)
Adaptec 2940U
Promise SX8 performance issues and CARM_MAX_Q
Hi,
Playing with a recently acquired Promise SX8 card, we've found similar performance results to Matt's post to lkml a few months back at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2
It appears that the driver is only submitting one command at a time per port, which is at least one cause of the slowdowns. By raising CARM_MAX_Q from 1 to 3 in drivers/block/sx8.c (it was 3 in an earlier pre-merge incarnation of carmel.c), we're getting very notable speed improvements, with no side effects just yet.
Knowing very little about what this change has actually done, I've a few questions:
- Should this be considered dangerous?
- Why was it taken from 3 to 1?
- Is CARM_MAX_Q a number defined (or limited) by the hardware?
Thanks in advance,
Bernard.
Signature · 1 line
Bernard Blackham <bernard at blackham dot com dot au>
Re: Promise SX8 performance issues and CARM_MAX_Q
Quoted 12 lines · Bernard Blackham
Hi,
Playing with a recently acquired Promise SX8 card, we've found similar performance results to Matt's post to lkml a few months back at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2
It appears that the driver is only submitting one command at a time per port, which is at least one cause of the slowdowns. By raising
CARM_MAX_Qfrom 1 to 3 indrivers/block/sx8.c(it was 3 in an earlier pre-merge incarnation ofcarmel.c), we're getting very notable speed improvements, with no side effects just yet.
Quoted 6 lines
Knowing very little about what this change has actually done, I've a few questions:
- Should this be considered dangerous?
- Why was it taken from 3 to 1?
- IsCARM_MAX_Qa number defined (or limited) by the hardware?
In multi-port stress tests, we couldn't get SX8 to function reliably without locking up or corrupting data, with more than one outstanding command.
Maybe a new firmware has solved this by now.
Jeff
Re: Promise SX8 performance issues and CARM_MAX_Q
Quoted 21 lines · Jeff Garzik
Quoted 12 lines · Bernard Blackham
Hi,
Playing with a recently acquired Promise SX8 card, we've found similar performance results to Matt's post to lkml a few months back at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2
It appears that the driver is only submitting one command at a time per port, which is at least one cause of the slowdowns. By raising
CARM_MAX_Qfrom 1 to 3 indrivers/block/sx8.c(it was 3 in an earlier pre-merge incarnation ofcarmel.c), we're getting very notable speed improvements, with no side effects just yet.Quoted 5 lines
Knowing very little about what this change has actually done, I've a few questions:
- Should this be considered dangerous?
- Why was it taken from 3 to 1?
- IsCARM_MAX_Qa number defined (or limited) by the hardware?
Quoted 8 lines
In multi-port stress tests, we couldn't get SX8 to function reliably without locking up or corrupting data, with more than one outstanding command.
Maybe a new firmware has solved this by now.
Jeff
Indeed there does seem to be new firmware out as of 2/23/05. I ran my tests with the 9/10/04 firmware but I did not adjust the CARM_MAX_Q value. Do either of you happen to know what firmware revisions you tested under?
I've put the machine with the SX8 controller into production despite the performance issues so I'm not going to be of any use for testing revisions to the driver :(
Matt
Re: Promise SX8 performance issues and CARM_MAX_Q
Quoted 28 lines · Matthew Collins
Quoted 26 lines · Jeff Garzik
Quoted 12 lines · Bernard Blackham
Hi,
Playing with a recently acquired Promise SX8 card, we've found similar performance results to Matt's post to lkml a few months back at http://marc.theaimsgroup.com/?l=linux-kernel&m=110175890323356&w=2
It appears that the driver is only submitting one command at a time per port, which is at least one cause of the slowdowns. By raising
CARM_MAX_Qfrom 1 to 3 indrivers/block/sx8.c(it was 3 in an earlier pre-merge incarnation ofcarmel.c), we're getting very notable speed improvements, with no side effects just yet.Quoted 5 lines
Knowing very little about what this change has actually done, I've a few questions:
- Should this be considered dangerous?
- Why was it taken from 3 to 1?
- IsCARM_MAX_Qa number defined (or limited) by the hardware?In multi-port stress tests, we couldn't get SX8 to function reliably without locking up or corrupting data, with more than one outstanding command.
Maybe a new firmware has solved this by now.
Quoted 7 lines
Quoted 4 lines
Jeff
Indeed there does seem to be new firmware out as of 2/23/05. I ran my tests with the 9/10/04 firmware but I did not adjust the
CARM_MAX_Qvalue. Do either of you happen to know what firmware revisions you tested under?I've put the machine with the SX8 controller into production despite the performance issues so I'm not going to be of any use for testing revisions to the driver :(
The driver was developed on a pre-production board, so its entirely possible Promise fixed this issue.
The driver should be solid for _at least_ CARM_MAX_Q==31, presuming that the firmware doesn't choke.
Anybody willing to
(a) change CARM_MAX_Q
(b) use the latest firmware
(c) stress the card with fsx, badblocks, iozone, and other tools on -multiple ports- simultaneously.
?
Jeff
Re: Promise SX8 performance issues and CARM_MAX_Q
Quoted 4 lines · Jeff Garzik
Quoted 4 lines
Indeed there does seem to be new firmware out as of 2/23/05. I ran my tests with the 9/10/04 firmware but I did not adjust the
CARM_MAX_Qvalue. Do either of you happen to know what firmware revisions you tested under?
BIOS boot messages didn't seem to indicate a firmware version, but this card was only available in stock in early March, so its fairly likely to have the latest firmware (2/23/05).
Quoted 2 lines
The driver should be solid for
_atleast_CARM_MAX_Q==31, presuming that the firmware doesn't choke.
We've run it here with CARM_MAX_Q = 32 with a custom stress testing script and haven't had any incidents so far. CARM_MAX_Q = 33 goes bad instantly.
Quoted 2 lines
(c) stress the card with fsx, badblocks, iozone, and other tools on -multiple ports- simultaneously.
I'll give those a workout over the weekend and let you know the results.
Thanks,
Bernard.
Signature · 1 line
Bernard Blackham <bernard at blackham dot com dot au>
Re: Promise SX8 performance issues and CARM_MAX_Q
In reply to Jeff GarzikQuoted 5 lines · Jeff Garzik
The driver was developed on a pre-production board, so its entirely possible Promise fixed this issue.
The driver should be solid for
_atleast_CARM_MAX_Q==31, presuming that the firmware doesn't choke.
Stressing this SX8 card broke with CARM_MAX_Q anything higher than
16. Tests included simultaneous iozones, bonnie++'s, fsx, badblocks, and mkfs'ing, on 4 ports. Setting CARM_MAX_Q to 17 or above resulted in errors such as:
Buffer I/O error on device sx80_0p1, logical block 108828 lost page write due to I/O error on sx80_0p1end_request: I/O error, dev sx80_2, sector 389167
It's been thrashing away with various tests most of today with CARM_MAX_Q = 16, and it hasn't missed a beat.
Thanks must go to Matt Johnston who actually did all the work of giving the card and drives a hard time.
Bernard.
Signature · 1 line
Bernard Blackham <bernard at blackham dot com dot au>