Skip to contentSkip to conversation

Accessing the same disk via multiple channels

DISCUSSION Jurjen Oskam 5 participants 6 messages Feb 13, 2003 – Feb 17, 2003

lore
Share & tools
lore
#0 lore raw

Hi everybody,

here's something I've been wondering about. On my work, we have an EMC2 Symmetrix in a SAN environment, with (until now) only AIX boxes attached to the SAN.

Each server is equipped with 2 FibreChannel cards. The SAN is configured to present the same disk (which is in fact a virtual Symmetrix device) over two channels. This means the host sees two physical devices (as far as that host's concerned) which is in fact really only one device. In linux terms: /dev/sda and /dev/sdc are exactly the same disks, but the (standard) OS doesn't know this.

EMC2 provide a piece of software called PowerPath, which takes advantage of this situation. It provides yet another device (let's say /dev/powersda), which uses the (identical) native devices /dev/sda and /dev/sdc. If one of those two would disappear, access to powersda would still be possible.

How does linux as it is now handle the situation of one physical device presented via multiple paths (without extra software)?

Signature · 3 lines
Jurjen Oskam

PGP Key available at http://www.stupendous.org/
#1 lore raw
Quoted 6 lines · In article <[email protected]> you wrote:

Each server is equipped with 2 FibreChannel cards. The SAN is configured to present the same disk (which is in fact a virtual Symmetrix device) over two channels. This means the host sees two physical devices (as far as that host's concerned) which is in fact really only one device. In linux terms: /dev/sda and /dev/sdc are exactly the same disks, but the (standard) OS doesn't know this.

...

Quoted 2 lines

How does linux as it is now handle the situation of one physical device presented via multiple paths (without extra software)?

You can use the multipath option to md which can do that.

Basically there are two options, a failover and a load balancing option. The problem with failover is, to detect the actual failure reliable, toe problem with load balancing is, that not all san configurations allow this.

http://www-124.ibm.com/storageio/multipath/md-multipath/index.php

this is at least in 2.4.20-xfs

Greetings
Bernd

Signature · 2 lines
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/
#2 lore raw
Quoted 9 lines · Bernd Eckenfels

You can use the multipath option to md which can do that.

Basically there are two options, a failover and a load balancing option. The problem with failover is, to detect the actual failure reliable, toe problem with load balancing is, that not all san configurations allow this.

http://www-124.ibm.com/storageio/multipath/md-multipath/index.php

this is at least in 2.4.20-xfs

That one? Ouch, it is a bit dated according to the webpage ;-) I don't recall that it was discussed on LKML, either.

SuSE (Jens Axboe and myself) have also done work on the md multipathing, supporting failover and load balancing and in general giving the code a rinse; as well as extensions to mdadm to make them work.

The patches currently live at http://lars.marowsky-bree.de/dl/md-mp/

(And are included in SuSE's kernel release, of course ;-)

Currently, for 2.5 / 2.6, I think I really like the SCSI midlayer stuff. In the past, I didn't, because it constrains everything to SCSI. But then, everything so far _has_ been SCSI, except for weird arch stuff like s390(x) DASDs ;-)

Doing it in the SCSI layer has the advantage of not being constrained to block devices, but also working with tapes. Oh well, we'll see ;-)

Sincerely,
Lars Marowsky-Brée <[email protected]>

Signature · 5 lines
Principal Squirrel 
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur
#3 lore raw
Quoted 3 lines · On Fri 14 Feb at 11:03:16 +0100 [email protected] done said:

Doing it in the SCSI layer has the advantage of not being constrained to block devices, but also working with tapes. Oh well, we'll see ;-)

Tape needs special multipathing logic. Don't you think moving multipathing to the mid-layer requires the mid-layer to know much more about the upper layer and muddles up the scsi stack's layering? To keep multipathing high and generic we need better error reporting than the one bit that hits the md layer in 2.4...

t.

Signature · 4 lines
*********************************************************
*  tpepper@vato dot org             * Venimus, Vidimus, *
*  http://www.vato.org/~tpepper     * Dolavimus         *
*********************************************************
#4 lore raw
Quoted 5 lines · Tim Pepper
Quoted 3 lines · On Fri 14 Feb at 11:03:16 +0100 [email protected] done said:

Doing it in the SCSI layer has the advantage of not being constrained to block devices, but also working with tapes. Oh well, we'll see ;-)

Quoted 7 lines

Tape needs special multipathing logic. Don't you think moving multipathing to the mid-layer requires the mid-layer to know much more about the upper layer and muddles up the scsi stack's layering? To keep multipathing high and generic we need better error reporting than the one bit that hits the md layer in 2.4...

t.

If we want to be able to retry a tape read/write that actually might have made it to the media it requires special handling (as compared to a random access device) no matter where we put multi-path.

The scsi mid-layer is a bit muddled up already (but getting better). Adding multi-path there does not make it worse. Much of the same code that cleans up the scsi mid-layer also makes scsi multi-path easier (that is, recent changes cleaning up the scsi mid-layer make it easier to implement scsi multi-path).

Generic multi-path without the lower levels knowing anything can waste a lot of resources. For example, for each extra path to a block device (disk) we end up with an extra sd plus associated data structures, and an extra scsi_device including multiple request queues.

The main thing scsi multi-path needs to know is that multiple nexuses (i.e. host/channel/target/lun) correspond to the same unit. This has no interactions with any upper layer drivers, and limits what the upper layer drivers (and users) have to work with, and more closely matches the layout of the actual hardware.

-- Patrick Mansfield

#5 lore raw
Quoted 4 lines · Patrick Mansfield

Generic multi-path without the lower levels knowing anything can waste a lot of resources. For example, for each extra path to a block device (disk) we end up with an extra sd plus associated data structures, and an extra scsi_device including multiple request queues.

I think these somehow still need to be exposed so that userspace can do per-path diagnostics; unless you also want to move this into the kernel space, which I'm not sure about.

What we really need is better error handling and escalation so that a higher layer actually has a chance at really well done recovery and retrying.

Sincerely,
Lars Marowsky-Brée <[email protected]>

Signature · 5 lines
Principal Squirrel 
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur