qla2xxx: automatically rescan removed luns.

DISCUSSION Benjamin ESTRABAUD 4 participants 6 messages Nov 25, 2013 – Dec 3, 2013 SCSI

Share & tools

Benjamin ESTRABAUD

author

2013-11-25 15:44 UTC #0 lore raw

Hi,

Using the qla2xxx driver from Linux 3.10.1 (release), if a LUN from the target side (multiple drives exported over a LIO IBlock qla2xxx export) is removed at the LIO level, the initiator side will not automatically detect that this LUN was removed.

All commands to this LUN will fail (sg_inq, read, writes) but the qla2xxx will never kick the drive out.

If another drive was to be exported on the target side using the same lun as the previously removed drive, commands sent to the same block id would now work again but be sent to the new LUN.

I tried to echo "- - -" in the scsi_host sysfs "scan" file corresponding to the FC port. This worked to detect new LUNs but not to "clear" removed ones.

I tried issuing a lip_reset on both sides, but no change.

Looking through Google, I realized that, on RedHat, a /proc/scsi/qla2xxx virtual FS exists which allows to send a "scsi-qlascan" command to the driver that apparently can detect removed LUNs. Unfortunately I don't have this FS visible on my Linux kernel, and I made sure /proc/scsi was enabled.

The only way I found was to "echo 1 > /sys/block/sdX/device/delete" to the stale device (that corresponds to the removed LUN from the target side), then replace the removed LUN on the target side before finally issuing a "scan" command on the initiator side.

Am I missing something here? Is there a way to rescan a Qlogic FC port using the "qla2xxx" driver from the official, stock kernel?

Thank you very much in advance for your help!

Regards,

Ben - MPSTOR

Ewan Milne

2013-11-25 17:17 UTC #1 lore raw

Quoted 26 lines · Benjamin ESTRABAUD

Hi,
Using the qla2xxx driver from Linux 3.10.1 (release), if a LUN from the target side (multiple drives exported over a LIO IBlock qla2xxx export) is removed at the LIO level, the initiator side will not automatically detect that this LUN was removed.
All commands to this LUN will fail (sg_inq, read, writes) but the qla2xxx will never kick the drive out.
If another drive was to be exported on the target side using the same lun as the previously removed drive, commands sent to the same block id would now work again but be sent to the new LUN.
I tried to echo "- - -" in the scsi_host sysfs "scan" file corresponding to the FC port. This worked to detect new LUNs but not to "clear" removed ones.
I tried issuing a lip_reset on both sides, but no change.
Looking through Google, I realized that, on RedHat, a /proc/scsi/qla2xxx virtual FS exists which allows to send a "scsi-qlascan" command to the driver that apparently can detect removed LUNs. Unfortunately I don't have this FS visible on my Linux kernel, and I made sure /proc/scsi was enabled.

Quoted 7 lines

The only way I found was to "echo 1 > /sys/block/sdX/device/delete" to the stale device (that corresponds to the removed LUN from the target side), then replace the removed LUN on the target side before finally issuing a "scan" command on the initiator side.
Am I missing something here? Is there a way to rescan a Qlogic FC port using the "qla2xxx" driver from the official, stock kernel?

The kernel does not contain support for removing devices automatically during a subsequent rescan. You have to delete them individually. You can, however, update the properties detected when scanning a device (e.g. disk capacity) by scanning the individual device in sysfs. For SCSI devices, there is a "rescan" capability, e.g.

echo 1> /sys/devices/<pci device path>/host0/target0:0:0/0:0:0:0/rescan

*Very* recent kernels contain a udev notification mechanism when a SCSI device reports a Unit Attention, so you can have udev rules like:

ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="INQUIRY_DATA_HAS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"

ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="CAPACITY_DATA_HAS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"

ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="THIN_PROVISIONING_SOFT_THRESHOLD_REACHED", \
TEST=="rescan", ATTR{rescan}="x"

ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="MODE_PARAMETERS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"

ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="REPORTED_LUNS_DATA_HAS_CHANGED", \
RUN+="<script to scan devices> $env{DEVPATH}"

...

A script to scan devices can use the devpath of the SCSI device reporting the Unit Attention to perform the desired action. If you just want to scan the target port, something like this might work, although it is a little clumsy...

#!/bin/sh -e

HOST_PATH1=echo $1 | awk '{print substr($1,0,index($1,"/target")-1)}' HOST_PATH2="scsi_host/host"echo $1 | awk '{split(substr($1,index($1,"target")),a,"/"); split(a[2],b,":"); print b[1]}'"/scan"

SCAN_STRING=echo $1 | awk '{split(substr($1,index($1,"target")),a,"/"); split(a[2],b,":"); print b[2],b[3],"-"}'

echo $SCAN_STRING > "/sys/"$HOST_PATH1"/"$HOST_PATH2

Or, "rescan-scsi-bus.sh" from (I think) sg3_utils might do what you want. Beware of running shell scripts from udev rules, however, as that doesn't scale well in large configurations.

-Ewan

Quoted 10 lines

Thank you very much in advance for your help!
Regards,
Ben - MPSTOR
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Benjamin ESTRABAUD

author

2013-11-26 16:06 UTC #2 lore raw

Quoted 27 lines · Ewan Milne

Quoted 26 lines · Benjamin ESTRABAUD
Hi,
Using the qla2xxx driver from Linux 3.10.1 (release), if a LUN from the target side (multiple drives exported over a LIO IBlock qla2xxx export) is removed at the LIO level, the initiator side will not automatically detect that this LUN was removed.
All commands to this LUN will fail (sg_inq, read, writes) but the qla2xxx will never kick the drive out.
If another drive was to be exported on the target side using the same lun as the previously removed drive, commands sent to the same block id would now work again but be sent to the new LUN.
I tried to echo "- - -" in the scsi_host sysfs "scan" file corresponding to the FC port. This worked to detect new LUNs but not to "clear" removed ones.
I tried issuing a lip_reset on both sides, but no change.
Looking through Google, I realized that, on RedHat, a /proc/scsi/qla2xxx virtual FS exists which allows to send a "scsi-qlascan" command to the driver that apparently can detect removed LUNs. Unfortunately I don't have this FS visible on my Linux kernel, and I made sure /proc/scsi was enabled.

Quoted 8 lines

Quoted 7 lines
The only way I found was to "echo 1 > /sys/block/sdX/device/delete" to the stale device (that corresponds to the removed LUN from the target side), then replace the removed LUN on the target side before finally issuing a "scan" command on the initiator side.
Am I missing something here? Is there a way to rescan a Qlogic FC port using the "qla2xxx" driver from the official, stock kernel?

Hi,

Thank you very much for your reply, this is much appreciated.

Quoted 2 lines

The kernel does not contain support for removing devices automatically during a subsequent rescan.

I thought that it used to with older kernels, is that possible (was this a recent change of behaviour)? It does, however, remove devices which port or connection has gone offline (pulled SAS disk, FC target inaccessible because of a disconnected cable).

Looking into it, the 'echo "- - -" > /sys/class/scsi_host/hostX/scan' scan command does issue a request_lun command (at least in SAS, used a target on which I could trace incoming commands).

When ran and a new LUN is detected, Linux does the right thing and adds the device.

When ran and a LUN is gone, it does nothing, as you explained above.

You have to delete them individually.
Noted.

Quoted 6 lines

You can, however, update the properties detected when scanning a device (e.g. disk capacity) by scanning the individual device in sysfs. For SCSI devices, there is a "rescan" capability, e.g.
echo 1> /sys/devices/<pci device path>/host0/target0:0:0/0:0:0:0/rescan

I used that rescan feature that I didn't know about and it's *very* useful, thank you!!

Quoted 19 lines

*Very* recent kernels contain a udev notification mechanism when a SCSI device reports a Unit Attention, so you can have udev rules like:
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="INQUIRY_DATA_HAS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="CAPACITY_DATA_HAS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="THIN_PROVISIONING_SOFT_THRESHOLD_REACHED", \
TEST=="rescan", ATTR{rescan}="x"
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="MODE_PARAMETERS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"

Quoted 6 lines

ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="REPORTED_LUNS_DATA_HAS_CHANGED", \
RUN+="<script to scan devices> $env{DEVPATH}"
...

That sounded promising, unfortunately the version we use (3.10.y branch) doesn't have this yet.

Quoted 5 lines

A script to scan devices can use the devpath of the SCSI device reporting the Unit Attention to perform the desired action. If you just want to scan the target port, something like this might work, although it is a little clumsy...

Thank you very much for that script, that gives me a pretty good idea of what needs to be done here.

Quoted 7 lines

#!/bin/sh -e
HOST_PATH1=echo $1 | awk '{print substr($1,0,index($1,"/target")-1)}' HOST_PATH2="scsi_host/host"echo $1 | awk '{split(substr($1,index($1,"target")),a,"/"); split(a[2],b,":"); print b[1]}'"/scan"

Quoted 9 lines

SCAN_STRING=echo $1 | awk '{split(substr($1,index($1,"target")),a,"/"); split(a[2],b,":"); print b[2],b[3],"-"}'
echo $SCAN_STRING > "/sys/"$HOST_PATH1"/"$HOST_PATH2
Or, "rescan-scsi-bus.sh" from (I think) sg3_utils might do what you want. Beware of running shell scripts from udev rules, however, as that doesn't scale well in large configurations.

"rescan-scsi-bus.sh" did detect new LUN, but apparently not removed ones. However I need to test it on a system with a compatible bash shell as I wasn't able to run the script without errors.

Thanks a lot again for all your help.

Quoted 3 lines

-Ewan

Regards,
Ben.

Quoted 13 lines

Quoted 10 lines
Thank you very much in advance for your help!
Regards,
Ben - MPSTOR
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Douglas Gilbert

2013-11-26 16:27 UTC #3 lore raw

Quoted 117 lines · Benjamin ESTRABAUD

Quoted 27 lines · Ewan Milne
Quoted 26 lines · Benjamin ESTRABAUD
Hi,
Using the qla2xxx driver from Linux 3.10.1 (release), if a LUN from the target side (multiple drives exported over a LIO IBlock qla2xxx export) is removed at the LIO level, the initiator side will not automatically detect that this LUN was removed.
All commands to this LUN will fail (sg_inq, read, writes) but the qla2xxx will never kick the drive out.
If another drive was to be exported on the target side using the same lun as the previously removed drive, commands sent to the same block id would now work again but be sent to the new LUN.
I tried to echo "- - -" in the scsi_host sysfs "scan" file corresponding to the FC port. This worked to detect new LUNs but not to "clear" removed ones.
I tried issuing a lip_reset on both sides, but no change.
Looking through Google, I realized that, on RedHat, a /proc/scsi/qla2xxx virtual FS exists which allows to send a "scsi-qlascan" command to the driver that apparently can detect removed LUNs. Unfortunately I don't have this FS visible on my Linux kernel, and I made sure /proc/scsi was enabled.
Quoted 8 lines
Quoted 7 lines
The only way I found was to "echo 1 > /sys/block/sdX/device/delete" to the stale device (that corresponds to the removed LUN from the target side), then replace the removed LUN on the target side before finally issuing a "scan" command on the initiator side.
Am I missing something here? Is there a way to rescan a Qlogic FC port using the "qla2xxx" driver from the official, stock kernel?
Hi,
Thank you very much for your reply, this is much appreciated.
Quoted 2 lines
The kernel does not contain support for removing devices automatically during a subsequent rescan.
I thought that it used to with older kernels, is that possible (was this a recent change of behaviour)? It does, however, remove devices which port or connection has gone offline (pulled SAS disk, FC target inaccessible because of a disconnected cable).
Looking into it, the 'echo "- - -" > /sys/class/scsi_host/hostX/scan' scan command does issue a request_lun command (at least in SAS, used a target on which I could trace incoming commands).
When ran and a new LUN is detected, Linux does the right thing and adds the device.
When ran and a LUN is gone, it does nothing, as you explained above.
You have to delete them individually.
Noted.
Quoted 6 lines
You can, however, update the properties detected when scanning a device (e.g. disk capacity) by scanning the individual device in sysfs. For SCSI devices, there is a "rescan" capability, e.g.
echo 1> /sys/devices/<pci device path>/host0/target0:0:0/0:0:0:0/rescan
I used that rescan feature that I didn't know about and it's *very* useful, thank you!!
Quoted 19 lines
*Very* recent kernels contain a udev notification mechanism when a SCSI device reports a Unit Attention, so you can have udev rules like:
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="INQUIRY_DATA_HAS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="CAPACITY_DATA_HAS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="THIN_PROVISIONING_SOFT_THRESHOLD_REACHED", \
TEST=="rescan", ATTR{rescan}="x"
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="MODE_PARAMETERS_CHANGED", TEST=="rescan", \
ATTR{rescan}="x"
Quoted 6 lines
ACTION=="change", SUBSYSTEM=="scsi", \
ENV{SDEV_UA}=="REPORTED_LUNS_DATA_HAS_CHANGED", \
RUN+="<script to scan devices> $env{DEVPATH}"
...
That sounded promising, unfortunately the version we use (3.10.y branch) doesn't have this yet.
Quoted 5 lines
A script to scan devices can use the devpath of the SCSI device reporting the Unit Attention to perform the desired action. If you just want to scan the target port, something like this might work, although it is a little clumsy...
Thank you very much for that script, that gives me a pretty good idea of what needs to be done here.
Quoted 12 lines
#!/bin/sh -e
HOST_PATH1=echo $1 | awk '{print substr($1,0,index($1,"/target")-1)}' HOST_PATH2="scsi_host/host"echo $1 | awk '{split(substr($1,index($1,"target")),a,"/"); split(a[2],b,":"); print b[1]}'"/scan"
SCAN_STRING=echo $1 | awk '{split(substr($1,index($1,"target")),a,"/"); split(a[2],b,":"); print b[2],b[3],"-"}'
echo $SCAN_STRING > "/sys/"$HOST_PATH1"/"$HOST_PATH2

Quoted 7 lines

Quoted 4 lines
Or, "rescan-scsi-bus.sh" from (I think) sg3_utils might do what you want. Beware of running shell scripts from udev rules, however, as that doesn't scale well in large configurations.
"rescan-scsi-bus.sh" did detect new LUN, but apparently not removed ones. However I need to test it on a system with a compatible bash shell as I wasn't able to run the script without errors.

Did you try the rescan-scsi-bus.sh from sg3_utils v 1.37 or earlier? The reason I ask is that a fair amount of work was done on the rescan-scsi-bus.sh found in version 1.37 including syncing with Kurt Garloff's version 1.57 plus patches from Hannes Reinecke and Sean Stewart.

Doug Gilbert

Quoted 24 lines

Quoted 3 lines
-Ewan
Regards,
Ben.
Quoted 13 lines
Quoted 10 lines
Thank you very much in advance for your help!
Regards,
Ben - MPSTOR
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hannes Reinecke

2013-11-28 13:29 UTC #4 lore raw

Quoted 1 line · Douglas Gilbert

On 13-11-26 11:06 AM, Benjamin ESTRABAUD wrote:

[ .. ]

Quoted 6 lines

Quoted 5 lines
"rescan-scsi-bus.sh" did detect new LUN, but apparently not removed ones.
However I need to test it on a system with a compatible bash shell as I wasn't
able to run the script without errors.

Quoted 6 lines

Did you try the rescan-scsi-bus.sh from sg3_utils v 1.37 or earlier? The reason I ask is that a fair amount of work was done on the rescan-scsi-bus.sh found in version 1.37 including syncing with Kurt Garloff's version 1.57 plus patches from Hannes Reinecke and Sean Stewart.

Plus you need to call it with '-r', otherwise it won't remove any stale LUNs. I'm sure it's documented somewhere ...

Cheers,

Hannes

Signature · 8 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Benjamin ESTRABAUD

author

2013-12-03 17:35 UTC #5 lore raw

Quoted 9 lines · Hannes Reinecke

Quoted 1 line · Douglas Gilbert
On 13-11-26 11:06 AM, Benjamin ESTRABAUD wrote:
[ .. ]
Quoted 6 lines
Quoted 5 lines
"rescan-scsi-bus.sh" did detect new LUN, but apparently not removed ones.
However I need to test it on a system with a compatible bash shell as I wasn't
able to run the script without errors.

Quoted 9 lines

Quoted 6 lines
Did you try the rescan-scsi-bus.sh from sg3_utils v 1.37 or earlier? The reason I ask is that a fair amount of work was done on the rescan-scsi-bus.sh found in version 1.37 including syncing with Kurt Garloff's version 1.57 plus patches from Hannes Reinecke and Sean Stewart.
Plus you need to call it with '-r', otherwise it won't remove any stale LUNs. I'm sure it's documented somewhere ...

Hi Doug and Hannes,

I did indeed try but I was using an older version and without specifying "-r" so it didn't remove anything.

I looked at the latest version's "remove" code (at around line 445 from rescan-scsi-bus.sh) and it seems to delete disks using the scsi_device sysfs "delete" function (which is the function I'm using right now).

I was unable to tell how the script detects whether a drive is gone or has been replaced with another backend storage at the same LUN, which in fact turns out to be the thing I'm more interested in (since I can now delete stale luns).

Thanks in advance for your help!

Quoted 4 lines

Cheers,
Hannes

Regards,
Ben.