Accessing the same disk via multiple channels
Hi everybody,
here's something I've been wondering about. On my work, we have an EMC2 Symmetrix in a SAN environment, with (until now) only AIX boxes attached to the SAN.
Each server is equipped with 2 FibreChannel cards. The SAN is configured to present the same disk (which is in fact a virtual Symmetrix device) over two channels. This means the host sees two physical devices (as far as that host's concerned) which is in fact really only one device. In linux terms: /dev/sda and /dev/sdc are exactly the same disks, but the (standard) OS doesn't know this.
EMC2 provide a piece of software called PowerPath, which takes advantage of this situation. It provides yet another device (let's say /dev/powersda), which uses the (identical) native devices /dev/sda and /dev/sdc. If one of those two would disappear, access to powersda would still be possible.
How does linux as it is now handle the situation of one physical device presented via multiple paths (without extra software)?
Signature · 3 lines
Jurjen Oskam PGP Key available at http://www.stupendous.org/
Quoted 6 lines · In article <[email protected]> you wrote:
Each server is equipped with 2 FibreChannel cards. The SAN is configured to present the same disk (which is in fact a virtual Symmetrix device) over two channels. This means the host sees two physical devices (as far as that host's concerned) which is in fact really only one device. In linux terms:
/dev/sdaand/dev/sdcare exactly the same disks, but the (standard) OS doesn't know this.
...
Quoted 2 lines
How does linux as it is now handle the situation of one physical device presented via multiple paths (without extra software)?
You can use the multipath option to md which can do that.
Basically there are two options, a failover and a load balancing option. The problem with failover is, to detect the actual failure reliable, toe problem with load balancing is, that not all san configurations allow this.
http://www-124.ibm.com/storageio/multipath/md-multipath/index.php
this is at least in 2.4.20-xfs
Greetings
Bernd
Signature · 2 lines
eckes privat - http://www.eckes.org/ Project Freefire - http://www.freefire.org/
Quoted 9 lines · Bernd Eckenfels
You can use the multipath option to md which can do that.
Basically there are two options, a failover and a load balancing option. The problem with failover is, to detect the actual failure reliable, toe problem with load balancing is, that not all san configurations allow this.
http://www-124.ibm.com/storageio/multipath/md-multipath/index.php
this is at least in 2.4.20-xfs
That one? Ouch, it is a bit dated according to the webpage ;-) I don't recall that it was discussed on LKML, either.
SuSE (Jens Axboe and myself) have also done work on the md multipathing, supporting failover and load balancing and in general giving the code a rinse; as well as extensions to mdadm to make them work.
The patches currently live at http://lars.marowsky-bree.de/dl/md-mp/
(And are included in SuSE's kernel release, of course ;-)
Currently, for 2.5 / 2.6, I think I really like the SCSI midlayer stuff. In the past, I didn't, because it constrains everything to SCSI. But then, everything so far _has_ been SCSI, except for weird arch stuff like s390(x) DASDs ;-)
Doing it in the SCSI layer has the advantage of not being constrained to block devices, but also working with tapes. Oh well, we'll see ;-)
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 5 lines
Principal Squirrel SuSE Labs - Research & Development, SuSE Linux AG "If anything can go wrong, it will." "Chance favors the prepared (mind)." -- Capt. Edward A. Murphy -- Louis Pasteur
Quoted 3 lines · On Fri 14 Feb at 11:03:16 +0100 [email protected] done said:
Doing it in the SCSI layer has the advantage of not being constrained to block devices, but also working with tapes. Oh well, we'll see ;-)
Tape needs special multipathing logic. Don't you think moving multipathing to the mid-layer requires the mid-layer to know much more about the upper layer and muddles up the scsi stack's layering? To keep multipathing high and generic we need better error reporting than the one bit that hits the md layer in 2.4...
t.
Signature · 4 lines
********************************************************* * tpepper@vato dot org * Venimus, Vidimus, * * http://www.vato.org/~tpepper * Dolavimus * *********************************************************
Quoted 5 lines · Tim Pepper
Quoted 3 lines · On Fri 14 Feb at 11:03:16 +0100 [email protected] done said:
Doing it in the SCSI layer has the advantage of not being constrained to block devices, but also working with tapes. Oh well, we'll see ;-)
Quoted 7 lines
Tape needs special multipathing logic. Don't you think moving multipathing to the mid-layer requires the mid-layer to know much more about the upper layer and muddles up the scsi stack's layering? To keep multipathing high and generic we need better error reporting than the one bit that hits the md layer in 2.4...
t.
If we want to be able to retry a tape read/write that actually might have made it to the media it requires special handling (as compared to a random access device) no matter where we put multi-path.
The scsi mid-layer is a bit muddled up already (but getting better). Adding multi-path there does not make it worse. Much of the same code that cleans up the scsi mid-layer also makes scsi multi-path easier (that is, recent changes cleaning up the scsi mid-layer make it easier to implement scsi multi-path).
Generic multi-path without the lower levels knowing anything can waste a lot of resources. For example, for each extra path to a block device (disk) we end up with an extra sd plus associated data structures, and an extra scsi_device including multiple request queues.
The main thing scsi multi-path needs to know is that multiple nexuses (i.e. host/channel/target/lun) correspond to the same unit. This has no interactions with any upper layer drivers, and limits what the upper layer drivers (and users) have to work with, and more closely matches the layout of the actual hardware.
-- Patrick Mansfield
Quoted 4 lines · Patrick Mansfield
Generic multi-path without the lower levels knowing anything can waste a lot of resources. For example, for each extra path to a block device (disk) we end up with an extra sd plus associated data structures, and an extra
scsi_deviceincluding multiple request queues.
I think these somehow still need to be exposed so that userspace can do per-path diagnostics; unless you also want to move this into the kernel space, which I'm not sure about.
What we really need is better error handling and escalation so that a higher layer actually has a chance at really well done recovery and retrying.
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 5 lines
Principal Squirrel SuSE Labs - Research & Development, SuSE Linux AG "If anything can go wrong, it will." "Chance favors the prepared (mind)." -- Capt. Edward A. Murphy -- Louis Pasteur