RAID controller safety
Hello,
I am trying to determine which drivers and what hardware can be used to give reliable fsync
behavior. I see many drivers with FIXME or TODO comments which make me nervous.
I see the sd driver supports the direct SYNCHRONIZE_CACHE, but I'me having a hard time tracing
through how this translates to raid controllers and their drivers.
Specificly, I am looking at the Adaptec RAID controllers and their i2o drivers. I am told the
kernel's i2o driver lacks a strong guarantee on fsync, and so far am unable to determine if the dpt_i2o driver also falls short in this reguard.
I don't mean to start a 'drives lie - disable all caching, buy FC drives' flame war. I know
drives can lie, but I'd like to know if the drivers are lying too.
thanks,
-Kenny
__________________________________
Yahoo! for Good - Make a difference this year.
http://brand.yahoo.com/cybergivingweek2005/
Quoted 3 lines · Kenny Simpson
Specificly, I am looking at the Adaptec RAID controllers and their i2o drivers. I am told the
kernel's i2o driver lacks a strong guarantee on fsync, and so far am unable to determine if thedpt_i2odriver also falls short in this reguard.
Only dpt can tell you what their firmware actually does.
The i2o core drivers use the following rules
i2o_scsi issues SCSI commands and assumes they are pass through and that the firmware does not fake completions early (or if it does that it battery backs them). For the known hardware the i2o SCSI class interface is a pass through interface with the card cpu just doing protocol gunk and supervision
i2o_block by default assumes the card is caching. It adopts write through mode if the controller has no battery, write back if it shows battery. This can be configured differently via ioctls including the ability to tune write through of large I/O's (to avoid cache thrashing), and to do write back with no battery backup for performance in cases where losing the data on a crash doesn't matter (eg swap)
Alan
Quoted 7 lines
Quoted 1 line
Specificly, I am looking at the Adaptec RAID controllers and their i2o drivers. I am told
the
Quoted 1 line
kernel's i2o driver lacks a strong guarantee on fsync, and so far am unable to determine if
the
Quoted 1 line
dpt_i2odriver also falls short in this reguard.Only dpt can tell you what their firmware actually does.
Yeah, I wasn't so much interrested in the firmware just yet, just interrested if the device driver (dpt_i2o) gave it a fighting chance of doing the right thing.
Quoted 8 lines
The i2o core drivers use the following rules
i2o_blockby default assumes the card is caching. It adopts write through mode if the controller has no battery, write back if it shows battery. This can be configured differently via ioctls including the ability to tune write through of large I/O's (to avoid cache thrashing), and to do write back with no battery backup for performance in cases where losing the data on a crash doesn't matter (eg swap)
That's what I read in the comments too, but looking at the code I only ever see it set to write-back. I verified this with blktool - our controllers have no battery, and blktool showed the i2o-wcache state as write-back.
However, I was also told that the i2o_block driver lacks barrier support, so even in the write-back case, the controller won't be told to flush/sync.
I was sent a patch against 2.6.10 that implements barrier support in i2o_block, but the code base has shifted too much for me to make it apply.
-Kenny
__________________________________
Yahoo! for Good - Make a difference this year.
http://brand.yahoo.com/cybergivingweek2005/
Quoted 3 lines · Kenny Simpson
That's what I read in the comments too, but looking at the code I only ever see it set to write-back. I verified this with blktool - our controllers have no battery, and blktool showed the i2o-wcache state as write-back.
blktool doesn't support i2o control as far as I am aware. The blk level generic ioctls are just too crude to control it properly.
Quoted 2 lines
However, I was also told that the
i2o_blockdriver lacks barrier support, so even in the write-back case, the controller won't be told to flush/sync.
Correct, but it should only ever enable this in the battery backed case. Otherwise it uses the per command control bits to decide what mode it wishes to use for each I/O
Quoted 10 lines · --- Alan Cox
Quoted 2 lines · Kenny Simpson
That's what I read in the comments too, but looking at the code I only ever see it set to write-back. I verified this with blktool - our controllers have no battery, and blktool
showed
Quoted 1 line
the i2o-wcache state as write-back.
blktool doesn't support i2o control as far as I am aware. The blk level generic ioctls are just too crude to control it properly.
From man blktool dated August 2004:
i2o-wcache
Query or set an I2O block device's write cache.
Quoted 7 lines
Quoted 3 lines
However, I was also told that the
i2o_blockdriver lacks barrier support, so even in the write-back case, the controller won't be told to flush/sync.Correct, but it should only ever enable this in the battery backed case. Otherwise it uses the per command control bits to decide what mode it wishes to use for each I/O
So all writes would be treated as syncronous in the write-through case (no battery), making fsync a no-op?
-Kenny
__________________________________
Yahoo! for Good - Make a difference this year.
http://brand.yahoo.com/cybergivingweek2005/
Quoted 2 lines · In article <[email protected]> you wrote:
So all writes would be treated as syncronous in the write-through case (no battery), making fsync a no-op?
The device cache is IMHO not related to the higher level buffer cache. So fsync flushes the buffers and the write-through ensures the controller may not delay it in controller ram.
Gruss
Bernd
Quoted 2 lines · Kenny Simpson
So all writes would be treated as syncronous in the write-through case (no battery), making fsync a no-op?
fsync is never a no-op. fsync ensures material the OS is caching hits disk drivers/disks. Barriers or write through on the disk driver ensure that it hits the media.
The two are independant
Quoted 11 lines · --- Alan Cox
Quoted 1 line · Kenny Simpson
So all writes would be treated as syncronous in the write-through case (no battery), making
fsync
Quoted 1 line
a no-op?
fsync is never a no-op. fsync ensures material the OS is caching hits disk
drivers/disks. Barriers or write through on the disk driver ensure that it hits the media.The two are independant
Ok, the light is slowly coming on for me...
Lets see if I get it:
fsync, according to POSIX, will flush all pending writes
http://www.opengroup.org/onlinepubs/009695399/functions/fsync.html
Linux, according to the man page, takes this a little further an says that data is on stable storage.
Stable storage for a battery-backed RAID controller means its battery-backed cache. Stable storage for a RAID controller w/o battery means that the data is on disk. I2O controllers are told the caching requirement for each write in the command.
To tell a disk to force data to the platter, it needs to be sent a specific command (which some drives ignore), or the write cache must be disabled (write-through mode). The specific command varies depending on SCSI vs. SATA vs. TCQ, etc..
Ignoring O_DIRECT, Linux writes out data from the page cache. Data gets written out when the OS decides (high memory pressure, timers expire, etc..), or when a program requests it (fsync).
For Linux, to make the fsync command have stability (Durability), it must not only send the data to the controller, but must inform the controller to force the data to stable storage, and then wait for the controller to report the writes as completed.
An battery-backed I2O controller only needs to be sent the writes as write-back cache. A non-battery-backed I2O controller needs to be sent these writes as write-through. In both cases, the controller should set the drives themselves to be write-through.
To match the POSIX behavior, the onus on the OS is just the push out the data to the driver. To get the further reliability, the driver, contoller (and firmware), and drives (and firmware) must all function as advertised. If any one of these fail, the reliability is lost. Linux can at most hope to control the driver.
Ok, with all that out of the way...
Are there any known drivers that do not correctly pass on barriers/flush/sync to their controllers?
In my observations, and what others have told me in private emails, the I2O driver is such a
driver at least for my non-battery-backed controller (Adaptec 2015S). I read in the comments of the I2O driver that it should set the write-cache flag for writes as write-through for non-battery-backed controllers, but I don't observe that setting via blktool, and basic write/fsync benchmarks run too fast for the drives I have (4x 10kRPM in RAID-10).
Also from reading the source, I only see the write-cache flag being set to write-back. I see no
test for controller properties, or anything else that would modify this setting (except for the ioctl).
Of course, I could be mis-reading the I2O spec and all this is up to the controller to know if
it has a battery, so the controller is responsible for doing the right thing, and the flag in the I2O driver is irrelavant for this.
Thanks for your patience,
-Kenny
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com
Just for the record, I am looking at/using 2.6.15-rc7.
-Kenny
__________________________________
Yahoo! for Good - Make a difference this year.
http://brand.yahoo.com/cybergivingweek2005/
Ok, I finally tracked through the i2o code, and found that i2o_block_device_flush is ultimately called for fsync. Sorry for being so dense.
However, it does look like barriers are not directly supported. So, are they safe to use in ext3, or is ext3 all fine without them? Would barriers benefit i2o devices, or is there some reason to not have them?
As for the controller defaulting to write-back, I still cannot find anything that would set the cache mode to write-through in the non-battery-backed case.
-Kenny
__________________________________
Yahoo! for Good - Make a difference this year.
http://brand.yahoo.com/cybergivingweek2005/
To follow-up to myself again, I see that patches to implement barriers further have been working their way through the gauntlet, with the most recent trial being on Nov 24. I guess this will be 2.6.15+ stuff. I do not see an i2o update in that set of patches.
-Kenny
__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com