Accessing the same disk via multiple channels
Quoted 4 lines · Lars Marowsky-Bree
SuSE (Jens Axboe and myself) have also done work on the md multipathing, supporting failover and load balancing and in general giving the code a rinse; as well as extensions to mdadm to make them work.
Yay! We noticed that if a controller fails in such a way that
no interrupts are generated then md driver doesn't notice anything is wrong. Commands don't fail, but don't complete either. I played around with feature in the low level driver to periodically send a no-op command down to the controller and fail all outstanding commands and disable the controller if that command didn't come back pretty quickly, that seemed to work pretty well in a failover type situation.
(Better than putting a timeout on every command.) Also, md multipath doesn't notice if the backup path has failed, to warn the user that redundancy is no longer in effect. (Though if you set up things so i/o is going down both paths, not such a big deal, as md will notice. Probably you know all this already.
Quoted 8 lines
The patches currently live at http://lars.marowsky-bree.de/dl/md-mp/
(And are included in SuSE's kernel release, of course ;-)
Currently, for 2.5 / 2.6, I think I really like the SCSI midlayer stuff. In the past, I didn't, because it constrains everything to SCSI. But then, everything so far _has_ been SCSI, except for weird arch stuff like s390(x) DASDs ;-)
Well, the cciss driver is not a SCSI driver (except for purpsoes of tape drives & tape changers) and HP/Compaq has sold more than one million of those controllers (does popularity mean they aren't "weird"? :-), and we have mulitpath capable storage boxes they can connect to.
-- steve
Quoted 3 lines · steve cameron
Yay! We noticed that if a controller fails in such a way that no interrupts are generated then md driver doesn't notice anything is wrong. Commands don't fail, but don't complete either.
Uhhhhh. They should timeout though and then be counted as errors; hard to catch this differently.
You are saying it doesn't work?
Quoted 5 lines
(Better than putting a timeout on every command.) Also, md multipath doesn't notice if the backup path has failed, to warn the user that redundancy is no longer in effect. (Though if you set up things so i/o is going down both paths, not such a big deal, as md will notice. Probably you know all this already.
Yes. We intentionally don't do active _monitoring_ on non-active paths, same as we don't do reprobing of failed paths to see whether they are alive again.
(The LVM m-p patch does periodically send down live requests to failed paths to check this; I consider this intentional data corruption, but I'm paranoid.)
This is something which can easily be implemented safely in user-space though; the md approach still exposes the lower level devices and you can check them periodically if you care, and be it with a
while sleep 1 ; do
if ! dd if=/dev/sdaX of=/dev/null ; then
mdadm /dev/md0 --fail /dev/sdaX logger -p kern.alert "Path /dev/sdaX failed!"
fi
done
;-) Obviously, not something we need to run in kernel space.
Quoted 5 lines
Well, the cciss driver is not a SCSI driver (except for purpsoes of tape drives & tape changers) and HP/Compaq has sold more than one million of those controllers (does popularity mean they aren't "weird"? :-), and we have mulitpath capable storage boxes they can connect to.
Indeed. Yes, we'll need to figure out how to do this for 2.5/2.6; maybe porting forward the md m-p patch to 2.5 is indeed the best choice. It should be way easier, as md has been greatly cleaned up...
However, past discussions on LKML regarding "How to do m-p cleanly in 2.5" have never reached a conclusion ;-) We'll see. The good thing about the SCSI m-p is that it can also handle multipathed tape drives...
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 5 lines
Principal Squirrel SuSE Labs - Research & Development, SuSE Linux AG "If anything can go wrong, it will." "Chance favors the prepared (mind)." -- Capt. Edward A. Murphy -- Louis Pasteur
Quoted 7 lines · Lars Marowsky-Bree
Indeed. Yes, we'll need to figure out how to do this for 2.5/2.6; maybe porting forward the md m-p patch to 2.5 is indeed the best choice. It should be way easier, as md has been greatly cleaned up...
However, past discussions on LKML regarding "How to do m-p cleanly in 2.5" have never reached a conclusion ;-) We'll see. The good thing about the SCSI m-p is that it can also handle multipathed tape drives...
I thought the general consensus was it is OK for now (as a first go) to have scsi only multi-path, I have not heard anyone say don't do scsi multi-path. And then later (maybe after we have more than one subsystem supporting multi-path IO) we can add general multi-path support into the layers above scsi.
In any case, md or volume manager based multi-path solutions are good alternatives.
I have recently ported the scsi multi-path patch to 2.5.59, but haven't posted patches.
The current multi-path patch still needs at least two major changes in
scsi: error recovery (scsi_error.c) that allows other paths to be used without long delays, and a per-device queue_lock versus the current per-host queue_lock.
Hopefully we can get underlying changes for those last two into 2.5 (and maybe someday the multi-path patch), as they are improvements to scsi with or without multi-path.
-- Patrick Mansfield