LSI SAS changes SCSI address and by-path on hot-swap

we need to buy new controllers for new storages we are building.

LSI SAS HBAs are very attractive for our purposes but I identified a problem with our existing mainboard-integrated LSI SAS 1068E . The problem is that it is apparently not possble to use the /dev/disk/by-path feature of Linux with it. At least not with the kernel 2.6.24 we are using (excuse me if it has already been fixed on latest
kernels: the server is in production now and it's not easy for us to check).

We need the /dev/disk/by-path feature because we commonly do hot-swaps with drives and we need to know for sure which HDD slot corresponds to a certain linux block device. With other controllers like 3ware 9650SE there is no such problem, ok but that's a SATA controller... I don't know if the problem is by design with SAS controllers.

Actually the problem is even more complicated because for the new storages we have planned to assemble there would be SAS expanders in the middle.

Look, here is an hot-swap seen from the dmesg:

        Feb 22 14:27:30 myserver kernel: [655437.601971] mptbase: ioc0: 
LogInfo(0x31110d00): Originator={PL}, Code={Reset}, SubCode(0x0d00)
        Feb 22 14:27:35 myserver kernel: [655442.781061] mptsas: ioc0: 
removing sata device, channel 0, id 0, phy 0
        Feb 22 14:27:35 myserver kernel: [655442.781453] sd 5:0:10:0: 
[sdu] Synchronizing SCSI cache
        Feb 22 14:27:35 myserver kernel: [655442.781495] sd 5:0:10:0: 
[sdu] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
        Feb 22 14:28:22 myserver kernel: [655489.237562] mptsas: ioc0: 
attaching sata device, channel 0, id 0, phy 0
        Feb 22 14:28:22 myserver kernel: [655489.241959] scsi 5:0:11:0: 
Direct-Access     ATA      WDC WD10EADS-00P 0A01 PQ: 0 ANSI: 5
        Feb 22 14:28:22 myserver kernel: [655489.242506] sd 5:0:11:0: 
[sdu] 1953525168 512-byte hardware sectors (1000205 MB)
        Feb 22 14:28:22 myserver kernel: [655489.248104] sd 5:0:11:0: 
[sdu] Write Protect is off
        Feb 22 14:28:22 myserver kernel: [655489.251847] sd 5:0:11:0: 
[sdu] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
        Feb 22 14:28:22 myserver kernel: [655489.252161] sd 5:0:11:0: 
[sdu] 1953525168 512-byte hardware sectors (1000205 MB)
        Feb 22 14:28:22 myserver kernel: [655489.257758] sd 5:0:11:0: 
[sdu] Write Protect is off
        Feb 22 14:28:22 myserver kernel: [655489.261518] sd 5:0:11:0: 
[sdu] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
        Feb 22 14:28:22 myserver kernel: [655489.261525]  sdu: unknown 
partition table
        Feb 22 14:28:22 myserver kernel: [655489.287152] sd 5:0:11:0: 
[sdu] Attached SCSI disk
        Feb 22 14:28:22 myserver kernel: [655489.287204] sd 5:0:11:0: 

Attached scsi generic sg21 type 0

You see, when I remove the disk it takes away device sd 5:0:10:0 and when I reinsert a new drive it becomes device sd 5:0:11:0.

the /dev/disk/by-path the file to the disk also changes, from:

/dev/disk/by-path/pci-0000:0b:00.0-sas-0x500e08101003c820:1:0-0x1221000000000000:0

to:

/dev/disk/by-path/pci-0000:0b:00.0-sas-0x500e08101003c824:1:4-0x1221000000000000:0 (note: I'm not 100% sure that these two entries come from the same hot-swap as the dmesg above)

in rare cases I noticed that after an hot swap the file in /dev/disk/by-path for the device is not even recreated.

I also cannot trust drive letters because they can change across reboot, and they also change if I remove drive A, remove drive B, insert drive B, insert drive A... the letters would be swapped. So it's not reliable enough for our use.

So is this a real bug and is maybe fixed on newer kernels, or it is by design?

How can people reliably use hot-swap hardware in this situation...? Are there other ways to determine the physical connections from within linux (possibly through SAS expanders also), which I am not aware of?

[CUT]
The thing you want is the port-3.0. If you look in sysfs at this:

ls /sys/class/sas_port/port-3\:0/device

Mine contains phy-3:4 Showing this disk is actually connected to phy 4 of the output device (as the HBA counts).

we need to buy new controllers for new storages we are building.

LSI SAS HBAs are very attractive for our purposes but I identified a problem with our existing mainboard-integrated LSI SAS 1068E . The problem is that it is apparently not possble to use the /dev/disk/by-path feature of Linux with it. At least not with the kernel 2.6.24 we are using (excuse me if it has already been fixed on latest
kernels: the server is in production now and it's not easy for us to check).

We need the /dev/disk/by-path feature because we commonly do hot-swaps with drives and we need to know for sure which HDD slot corresponds to a certain linux block device. With other controllers like 3ware 9650SE there is no such problem, ok but that's a SATA controller... I don't know if the problem is by design with SAS controllers.

Actually the problem is even more complicated because for the new storages we have planned to assemble there would be SAS expanders in the middle.

Look, here is an hot-swap seen from the dmesg:

        Feb 22 14:27:30 myserver kernel: [655437.601971] mptbase: ioc0: 
LogInfo(0x31110d00): Originator={PL}, Code={Reset}, SubCode(0x0d00)
        Feb 22 14:27:35 myserver kernel: [655442.781061] mptsas: ioc0: 
removing sata device, channel 0, id 0, phy 0
        Feb 22 14:27:35 myserver kernel: [655442.781453] sd 5:0:10:0: 
[sdu] Synchronizing SCSI cache
        Feb 22 14:27:35 myserver kernel: [655442.781495] sd 5:0:10:0: 
[sdu] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
        Feb 22 14:28:22 myserver kernel: [655489.237562] mptsas: ioc0: 
attaching sata device, channel 0, id 0, phy 0
        Feb 22 14:28:22 myserver kernel: [655489.241959] scsi 5:0:11:0: 
Direct-Access     ATA      WDC WD10EADS-00P 0A01 PQ: 0 ANSI: 5
        Feb 22 14:28:22 myserver kernel: [655489.242506] sd 5:0:11:0: 
[sdu] 1953525168 512-byte hardware sectors (1000205 MB)
        Feb 22 14:28:22 myserver kernel: [655489.248104] sd 5:0:11:0: 
[sdu] Write Protect is off
        Feb 22 14:28:22 myserver kernel: [655489.251847] sd 5:0:11:0: 
[sdu] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
        Feb 22 14:28:22 myserver kernel: [655489.252161] sd 5:0:11:0: 
[sdu] 1953525168 512-byte hardware sectors (1000205 MB)
        Feb 22 14:28:22 myserver kernel: [655489.257758] sd 5:0:11:0: 
[sdu] Write Protect is off
        Feb 22 14:28:22 myserver kernel: [655489.261518] sd 5:0:11:0: 
[sdu] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
        Feb 22 14:28:22 myserver kernel: [655489.261525]  sdu: unknown 
partition table
        Feb 22 14:28:22 myserver kernel: [655489.287152] sd 5:0:11:0: 
[sdu] Attached SCSI disk
        Feb 22 14:28:22 myserver kernel: [655489.287204] sd 5:0:11:0: 

Attached scsi generic sg21 type 0

You see, when I remove the disk it takes away device sd 5:0:10:0 and when I reinsert a new drive it becomes device sd 5:0:11:0.

the /dev/disk/by-path the file to the disk also changes, from:

/dev/disk/by-path/pci-0000:0b:00.0-sas-0x500e08101003c820:1:0-0x1221000000000000:0

to:

/dev/disk/by-path/pci-0000:0b:00.0-sas-0x500e08101003c824:1:4-0x1221000000000000:0 (note: I'm not 100% sure that these two entries come from the same hot-swap as the dmesg above)

in rare cases I noticed that after an hot swap the file in /dev/disk/by-path for the device is not even recreated.

I also cannot trust drive letters because they can change across reboot, and they also change if I remove drive A, remove drive B, insert drive B, insert drive A... the letters would be swapped. So it's not reliable enough for our use.

So is this a real bug and is maybe fixed on newer kernels, or it is by design?

How can people reliably use hot-swap hardware in this situation...? Are there other ways to determine the physical connections from within linux (possibly through SAS expanders also), which I am not aware of?

I did this with a LSI-1068E HBA and 2 x 4 drive hot swap SATA bays. I was able to create udev rules to map the drive slots to consistent /dev entries. However, the bigger problem I had was that if I had a drive inserted and mounted (say in slot A) and then I added or swapped another drive on the same port ( 4 SAS channels per port on the external HBAs) it would cause some sort of reset on the bus that would end up unmounting the drive in slot A even though I never did anything to the drive in slot A. Now, this was with SATA drives connected directly to the 1068 which should work, but since I needed this to work, I had to revert to the older setup that used Silicon Image 3124 eSATA cards.

I can try to dig up the udev rules I used if this would be helpful.

- Mike

I did this with a LSI-1068E HBA and 2 x 4 drive hot swap SATA bays. I was able to create udev rules to map the drive slots to consistent /dev entries. However, the bigger problem I had was that if I had a drive inserted and mounted (say in slot A) and then I added or swapped another drive on the same port ( 4 SAS channels per port on the external HBAs) it would cause some sort of reset on the bus that would end up unmounting the drive in slot A even though I never did anything to the drive in slot A. Now, this was with SATA drives connected directly to the 1068 which should work, but since I needed this to work, I had to revert to the older setup that used Silicon Image 3124 eSATA cards.

I can try to dig up the udev rules I used if this would be helpful.

- Mike

Sorry for top posting, but Outlook just screws it all up.

Quoted 2 lines · Michael

Sorry for top posting, but Outlook just screws it all up.

Outlook is the root of all computers evil and should be avoided as the plague. Exchange servers work just as well with ThunderBird or any other none evil email client through the IMAP protocol.

Sorry for top posting, but Outlook just screws it all up.

The cards I've used are a LSI Logic SAS 3800X (8 port External PCI-X card w/ 2 x SFF-8470 SAS connectors) and LSI SAS 3801E ( 8 Port External PCI-e card with 2 x SFF-8088 SAS connectors). Each connector has 4 SAS links. The SAS protocol is downwardly compatible with SATA, so you can run SATA drives right on a SAS cable.

So, in my setup, I basically have 1 drive per SAS link. No expanders, or anything fancy. The issues I mentioned happens to the 4 drives on the same connector. When the driver is detecting the new drive, it looks like it redetects all of the drives on the connector (or it at least reports one new drive and the other existing drives). If you were in a directory from one of the mounted drives, you get IO Errors as it appears that the drive was removed, and then remounted, but in a way that was not clean.

This has happened with Default CentOS 5 kernels (2.6.18-*.el5), 2.6.26 vanilla, 2.6.30 vanilla, Fedora latest. The issue appeared no matter what.

The udev rules used the ENV{ID_PATH} option to tie to the sysfs value that indicated which PCI ID + SAS phy on the SAS HBA used by the drives to the device detected by the kernel, and then create a symlink from the /dev/sd<X> entry to /dev/slot<Y>, where Y is the label on the slot of the hot swap bays (a-h). Here is an example of the rule:

KERNEL=="sd*", ENV{ID_PATH}=="pci-0000:04:00.0-sas-phy0:1*", SYMLINK+="slota%n"

I did this because the device ID number that the kernel reports increments every time a drive is swapped. So, even though you are using the same SAS channel, you do not have a consistent drive numbering. So I had to go down to the SAS phy to get something consistent. The SiI-3124/libata setup had consistent device ID's (the ID was tied to the SATA channel, and I used the device ID to do the mapping. Perhaps udev is the reason for the issues, but I tend to think it is the way the SAS/SCSI subsystem works as I have never seen the SATA/libata subsystem have this "rescan/remount" behavior.

Quoted 12 lines · Michael

Sorry for top posting, but Outlook just screws it all up.

The cards I've used are a LSI Logic SAS 3800X (8 port External PCI-X card w/ 2 x SFF-8470 SAS connectors) and LSI SAS 3801E ( 8 Port External PCI-e card with 2 x SFF-8088 SAS connectors). Each connector has 4 SAS links. The SAS protocol is downwardly compatible with SATA, so you can run SATA drives right on a SAS cable.

So, in my setup, I basically have 1 drive per SAS link. No expanders, or anything fancy. The issues I mentioned happens to the 4 drives on the same connector. When the driver is detecting the new drive, it looks like it redetects all of the drives on the connector (or it at least reports one new drive and the other existing drives). If you were in a directory from one of the mounted drives, you get IO Errors as it appears that the drive was removed, and then remounted, but in a way that was not clean.

This has happened with Default CentOS 5 kernels (2.6.18-*.el5), 2.6.26 vanilla, 2.6.30 vanilla, Fedora latest. The issue appeared no matter what.

The udev rules used the ENV{ID_PATH} option to tie to the sysfs value that indicated which PCI ID + SAS phy on the SAS HBA used by the drives to the device detected by the kernel, and then create a symlink from the /dev/sd<X> entry to /dev/slot<Y>, where Y is the label on the slot of the hot swap bays (a-h). Here is an example of the rule:

Quoted 4 lines

KERNEL=="sd*", ENV{ID_PATH}=="pci-0000:04:00.0-sas-phy0:1*", SYMLINK+="slota%n"

I did this because the device ID number that the kernel reports increments every time a drive is swapped. So, even though you are using the same SAS channel, you do not have a consistent drive numbering. So I had to go down to the SAS phy to get something consistent. The SiI-3124/libata setup had consistent device ID's (the ID was tied to the SATA channel, and I used the device ID to do the mapping. Perhaps udev is the reason for the issues, but I tend to think it is the way the SAS/SCSI subsystem works as I have never seen the SATA/libata subsystem have this "rescan/remount" behavior.

This looks like a horrible bug for people having software RAID on the disks (or maybe even hardware RAID)

I seem not to have this bug on ubuntu kernel 2.6.24, I mean my situation was similar with the mainboard-integrated LSISAS 1068E and it didn't happen to me, but that doesn't mean much...

Also, LSI controllers are very much used by linuxers.
Have you tried reporting it here and try to get it fixed?
Or reporting it to the LSI tech support? They are pretty responsive even if their web interface is a bit strange.

I'm thinking about buying a few of LSI HBA controllers for linux software RAID use, probably external ones like the one you have. Maybe attached to expanders. I'll keep my fingers crossed!

Quoted 16 lines · Asdo

Quoted 14 lines · Michael

Sorry for top posting, but Outlook just screws it all up.

The cards I've used are a LSI Logic SAS 3800X (8 port External PCI-X card w/ 2 x SFF-8470 SAS connectors) and LSI SAS 3801E ( 8 Port External PCI-e card with 2 x SFF-8088 SAS connectors). Each connector has 4 SAS links. The SAS protocol is downwardly compatible with SATA, so you can run SATA drives right on a SAS cable.

So, in my setup, I basically have 1 drive per SAS link. No expanders, or anything fancy. The issues I mentioned happens to the 4 drives on the same connector. When the driver is detecting the new drive, it looks like it redetects all of the drives on the connector (or it at least reports one new drive and the other existing drives). If you were in a directory from one of the mounted drives, you get IO Errors as it appears that the drive was removed, and then remounted, but in a way that was not clean.

This has happened with Default CentOS 5 kernels (2.6.18-*.el5), 2.6.26 vanilla, 2.6.30 vanilla, Fedora latest. The issue appeared no matter what.

The udev rules used the ENV{ID_PATH} option to tie to the sysfs value that indicated which PCI ID + SAS phy on the SAS HBA used by the drives to the device detected by the kernel, and then create a symlink from the /dev/sd<X> entry to /dev/slot<Y>, where Y is the label on the slot of the hot swap bays (a-h). Here is an example of the rule:

KERNEL=="sd*", ENV{ID_PATH}=="pci-0000:04:00.0-sas-phy0:1*", SYMLINK+="slota%n"

Quoted 5 lines

Quoted 3 lines

I did this because the device ID number that the kernel reports increments every time a drive is swapped. So, even though you are using the same SAS channel, you do not have a consistent drive numbering. So I had to go down to the SAS phy to get something consistent. The SiI-3124/libata setup had consistent device ID's (the ID was tied to the SATA channel, and I used the device ID to do the mapping. Perhaps udev is the reason for the issues, but I tend to think it is the way the SAS/SCSI subsystem works as I have never seen the SATA/libata subsystem have this "rescan/remount" behavior.

This looks like a horrible bug for people having software RAID on the disks (or maybe even hardware RAID)

Not really, most people want to identify the disk permanently, not the slot, so that's what /dev/disk/by-id and /dev/disk/by-uuid is for.

So, in my setup, I basically have 1 drive per SAS link. No expanders, or anything fancy. The issues I mentioned happens to the 4 drives on the same connector. When the driver is detecting the new drive, it looks like it redetects all of the drives on the connector (or it at least reports one new drive and the other existing drives). If you were in a directory from one of the mounted drives, you get IO Errors as it appears that the drive was removed, and then remounted, but in a way that was not clean.