Random IO with md raid
Hello,
I'm reviewing one new server with a 3ware 9690 controller where I've got a couple of sata drives connected.
In order to see how software raid behave in comparaison with a the raid controller I setup 3 volume:
* 1 raid 1 volume of 2 1TB hard drive
* 2 single volume of 1 TB hard drive each
I assemble the 2 single volume in a raid 1 volume
Both are formatted in XFS mounted noatime and nobarrier.
Sequentials reads and writes are quite similar between software raid and 3ware raid.
But when it comes to random writes and random reads I find something strange.
The memory is 1G so I made iozone use 2G file with 4k block
On 3ware (in operation by second)
Random read: 193 Random write: 1232
On md
Random read: 178 Random write: 361
First I am bit worried that I can manage to get such good random result with the 3ware card I guess I am seeing some cache effect (there is 256 MB on the card) Also I'm surprised that the software raid is not able to take advantage of this cache effect in the same proportion (1 to 4).
I've been searching internet for tips on raid tunning but it turns out it's mostly for sequential access (ie. like readahead).
Does anyone has an any idea ?
Matthieu.
Quoted 1 line · Matthieu Patou
* 1 raid 1 volume of 2 1TB hard drive
Are these drives also on the 3ware? Are they exported as jbod?
Quoted 3 lines
* 2 single volume of 1 TB hard drive each
...
Does anyone has an any idea ?
Try anticipatory, deadline and noop schedulers on disks (not CFQ)
Try setting readahead to exactly 4K, or set it to lowest possible value (I'm not sure what is best), since this is random access... Increase stripe_cache_size as high as you can, p
Let us/me know the results afterwards...
Thank you
Asdo
Quoted 3 lines · Asdo
Quoted 1 line · Matthieu Patou
* 1 raid 1 volume of 2 1TB hard drive
Are these drives also on the 3ware?
I suppose that you are speaking of the 2 single drives. If so yes they are also connected to the 3ware controller, they are not exported as jbod but as seperate hard drives. Are they exported as jbod?
Quoted 6 lines
Quoted 3 lines
* 2 single volume of 1 TB hard drive each
...
Does anyone has an any idea ?Try anticipatory, deadline and noop schedulers on disks (not CFQ) Try setting readahead to exactly 4K, or set it to lowest possible value
I tried all the different scheduler both with 4K and with the default value and I still have raid1 software that do 1/2 or 1/3 of the raid1 hardware for random write I/O.
Quoted 2 lines
(I'm not sure what is best), since this is random access... Increase
stripe_cache_sizeas high as you can, p
So for raid1 there is no stripe_cache_size so I didn't set it ...
Quoted 1 line
Let us/me know the results afterwards...
I also made a simple tests with the CFQ scheduler without any other tunning but turning of the controller cache and at this moment the results of the RAID1 hardware and the RAID1 software are very close. Looks like linux is not able to use the onboard cache of controller as efficiently as the 3ware controller it self at least for random IO.
Any other ideas ?
Matthieu
Quoted 28 lines · Matthieu Patou
Quoted 3 lines · Asdo
Quoted 1 line · Matthieu Patou
* 1 raid 1 volume of 2 1TB hard drive
Are these drives also on the 3ware?
I suppose that you are speaking of the 2 single drives. If so yes they are also connected to the 3ware controller, they are not exported as jbod but as seperate hard drives.
Are they exported as jbod?Quoted 6 lines
Quoted 3 lines
* 2 single volume of 1 TB hard drive each
...
Does anyone has an any idea ?Try anticipatory, deadline and noop schedulers on disks (not CFQ) Try setting readahead to exactly 4K, or set it to lowest possible value
I tried all the different scheduler both with 4K and with the default value and I still have raid1 software that do 1/2 or 1/3 of the raid1 hardware for random write I/O.
Quoted 2 lines
(I'm not sure what is best), since this is random access... Increase
stripe_cache_sizeas high as you can, pSo for raid1 there is no
stripe_cache_sizeso I didn't set it ...Quoted 1 line
Let us/me know the results afterwards...
I also made a simple tests with the CFQ scheduler without any other tunning but turning of the controller cache and at this moment the results of the RAID1 hardware and the RAID1 software are very close. Looks like linux is not able to use the onboard cache of controller as efficiently as the 3ware controller it self at least for random IO.
Quoted 3 lines
Any other ideas ?
Matthieu
Software raid1 will write twice as much data. That means twice as much data goes over the system bus and into the controler cache. Effectively you have halve the cache size. Maybe that is all you see.
MfG
Goswin
Hello Goswin,
Quoted 7 lines
Software raid1 will write twice as much data. That means twice as much data goes over the system bus and into the controler cache. Effectively you have halve the cache size. Maybe that is all you see.
MfG
Goswin
Your idea seems logical, and I took a few hours today to verify it and it's the case as accessing the disk without software raid leads to almost the same result as with hardware raid.
Matthieu.
Quoted 12 lines · Matthieu Patou
Hello Goswin,
Quoted 7 lines
Software raid1 will write twice as much data. That means twice as much data goes over the system bus and into the controler cache. Effectively you have halve the cache size. Maybe that is all you see.
MfG
GoswinYour idea seems logical, and I took a few hours today to verify it and it's the case as accessing the disk without software raid leads to almost the same result as with hardware raid.
Quoted 6 lines
Matthieu.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Thinking about it, the doubling of data going over the system bus part is correct. You must still push the operation out to each device's buffer. However for -most- modern systems the system-bus will not be the bottleneck for a reasonable number of drives.
Further the later half which I'd skimmed over the first time is utterly incorrect. Each drive would still only see the commands targeting that drive. That should be virtually the same if not identical to the single-drive case.
The two most likely bottlenecks are single-drive write speed, and any IO barriers that are used to ensure file system consistency in the event of sudden interruption.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 25 lines · Michael Evans
Quoted 12 lines · Matthieu Patou
Hello Goswin,
Quoted 7 lines
Software raid1 will write twice as much data. That means twice as much data goes over the system bus and into the controler cache. Effectively you have halve the cache size. Maybe that is all you see.
MfG
GoswinYour idea seems logical, and I took a few hours today to verify it and it's the case as accessing the disk without software raid leads to almost the same result as with hardware raid.
Quoted 6 lines
Matthieu.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlThinking about it, the doubling of data going over the system bus part is correct. You must still push the operation out to each device's buffer. However for -most- modern systems the system-bus will not be the bottleneck for a reasonable number of drives.
Quoted 4 lines
Further the later half which I'd skimmed over the first time is utterly incorrect. Each drive would still only see the commands targeting that drive. That should be virtually the same if not identical to the single-drive case.
The controler itself has cache. And that is shared between all drives.
Quoted 3 lines
The two most likely bottlenecks are single-drive write speed, and any IO barriers that are used to ensure file system consistency in the event of sudden interruption.
MfG
Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html