SCSI device rescan, detection of disconnected device, or switched devices.

I know that when adding new device there is no problem;
by echo "- - -" >/sys/class/scsi_host/hostX/scan the SCSI subsystem will recognize the new device and new /dev/sgX device will be created, but if someone remove a device, scan will only recognize that the report luns has changed, but it will not remove the /dev/sgX device.
If now someone add new device it will mapped to the /dev/sgX that previously mapped to the device that just removed.

1. Why the SCSI subsystem doesn't release devices that removed ?

2. In the situation that I described above someone can switched devices without notifying the application that use those devices. The notification will come only when the next SCSI command will return with unit attention saying "Power on, reset, or bus device reset occured", or

if device just removed without adding new device it will return "Reported luns data has changed".
If I have an application that control SCSI devices using sg driver and I would like to have the ability to change configuration online, what is the preferred way to rescan the bus and update the application that sgX that previously controls device Y is now controlling device Z ? In other words, what is the best way for the application to identify that device has been removed or changed ?

3. I am using Qlogic firmware ability to create virtual ports, and I notice that on disconnect and then reconnect the FC cable, the sg mapping can changed. If on module load the physical port got sg0, and on creating vport it got sg1, now the SCSI subsystem scans the vport first and mapped it to sg0 and the physical port gets sg1.
Is there a way to control the mapping (scanning) ?

Quoted 9 lines · Gal Rosen

I know that when adding new device there is no problem;
by echo "- - -" >/sys/class/scsi_host/hostX/scan the SCSI subsystem will recognize the new device and new /dev/sgX device will be created, but if someone remove a device, scan will only recognize that the report luns has changed, but it will not remove the /dev/sgX device.
If now someone add new device it will mapped to the /dev/sgX that previously mapped to the device that just removed.

1. Why the SCSI subsystem doesn't release devices that removed ?

Because somewhere someone still holds a reference on the device (for a good reason, or due to a bug).

Quoted 4 lines

2. In the situation that I described above someone can switched devices without notifying the application that use those devices. The notification will come only when the next SCSI command will return with unit attention saying "Power on, reset, or bus device reset occured", or

Quoted 8 lines

if device just removed without adding new device it will return "Reported luns data has changed".
If I have an application that control SCSI devices using sg driver and I would like to have the ability to change configuration online, what is the preferred way to rescan the bus and update the application that sgX that previously controls device Y is now controlling device Z ? In other words, what is the best way for the application to identify that device has been removed or changed ?

Hmm, I too am curious what the answer to this would be.

Quoted 6 lines

3. I am using Qlogic firmware ability to create virtual ports, and I notice that on disconnect and then reconnect the FC cable, the sg mapping can changed. If on module load the physical port got sg0, and on creating vport it got sg1, now the SCSI subsystem scans the vport first and mapped it to sg0 and the physical port gets sg1.
Is there a way to control the mapping (scanning) ?

There is currently one canonical way: The low-level userspace (udev, hal, or even your application) looks at the device representation in sysfs and fetches persistent and unique properties of the target and logical unit from there. The target identifier and logical unit identifier come first to mind as suitable properties. Then the low-level userspace can for example create symlinks. Most distributions contain udev scripts which do this for sd devices; see /dev/disk/by-id/.

Alas there is no standard place where to look for target identifier and logical unit identifier. At least at the last time when I looked for them, the paths to respective sysfs attributes are so far implementation details of the various transport-layer drivers or so-called transport classes (FC, SAS, iSCSI, SBP-2...).

I will try to make my question more clear.
I would like to put a box between the client and the storage, and to be fully transparent against the host. All changes on the storage (add/remove/extend devices) should be present automatically to the client. In a direct connection between client and storage the administrator will coordinate between the client and storage before making any changes, in my configuration I do not want to add

administration work on my box that stands between the client and the storage.
Addition is simple, the application on my box can once in a while scan the SCSI BUS, and then the Linux system will create new sg devices. In removal of a device the Linux system will not remove sg devices after scan. In that case again I can do it from my application, such when I get attention on SCSI command, I can decide to remove the device by echoing to the /sys file system.

Quoted 7 lines · Gal Rosen

I will try to make my question more clear.
I would like to put a box between the client and the storage, and to be fully transparent against the host. All changes on the storage (add/remove/extend devices) should be present automatically to the client. In a direct connection between client and storage the administrator will coordinate between the client and storage before making any changes, in my configuration I do not want to add

Quoted 8 lines

administration work on my box that stands between the client and the storage.
Addition is simple, the application on my box can once in a while scan the SCSI BUS, and then the Linux system will create new sg devices. In removal of a device the Linux system will not remove sg devices after scan. In that case again I can do it from my application, such when I get attention on SCSI command, I can decide to remove the device by echoing to the /sys file system.

I really don't know how the FibreChannel drivers handle device removal. But from what I gather from the documentation of fc_remote_port_delete(), which the qla2xxx drivers use, the /dev/sg devices should automatically disappear
- after a connection loss timeout,
- provided that userspace doesn't have the device file opened (among else this is manifest in the kernel in a reference count of the logical unit device representation),
- provided that you manage your device files with udev.

http://lxr.linux.no/linux+v2.6.26/drivers/scsi/scsi_transport_fc.c#L2756

That would be basically just like SCSI device files are created and destroyed on the fly for USB or FireWire devices for example. Except that FC has the additional configurable connection loss timeout which hides temporary connection losses from applications.

(However, better wait for an answer of somebody who actually uses FC or knows the drivers. I only have experience with FireWire.)

Quoted 27 lines · Stefan Richter

Quoted 7 lines · Gal Rosen

I will try to make my question more clear.
I would like to put a box between the client and the storage, and to be fully transparent against the host. All changes on the storage (add/remove/extend devices) should be present automatically to the client. In a direct connection between client and storage the administrator will coordinate between the client and storage before making any changes, in my configuration I do not want to add

Quoted 8 lines

administration work on my box that stands between the client and the storage.
Addition is simple, the application on my box can once in a while scan the SCSI BUS, and then the Linux system will create new sg devices. In removal of a device the Linux system will not remove sg devices after scan. In that case again I can do it from my application, such when I get attention on SCSI command, I can decide to remove the device by echoing to the /sys file system.

I really don't know how the FibreChannel drivers handle device removal. But from what I gather from the documentation of fc_remote_port_delete(), which the qla2xxx drivers use, the /dev/sg devices should automatically disappear
- after a connection loss timeout,
- provided that userspace doesn't have the device file opened (among else this is manifest in the kernel in a reference count of the logical unit device representation),
- provided that you manage your device files with udev.

Quoted 2 lines

http://lxr.linux.no/linux+v2.6.26/drivers/scsi/scsi_transport_fc.c#L2756

This is true; when the remote port is no longer exists, after timeout all the sg devices will disappear, but what if I am changing the configuration in the disk array, and remove only one logical device. There will not be any event for that. The hotplug implementation in the SCSI today creates events of add/remove when adding/removing HBA, or when the FC link go down and up again, but not for additional and removal of devices. In the linux documentation in scsi_mid_low_api.txt it is written that "The hotplug concept may be extended to SCSI devices". I guess that nobody implemented it because what you said previously in this thread that someone could hold reference to the device.

Quoted 29 lines · Gal Rosen

Quoted 26 lines · Stefan Richter

Quoted 7 lines · Gal Rosen

I will try to make my question more clear.
I would like to put a box between the client and the storage, and to be fully transparent against the host. All changes on the storage (add/remove/extend devices) should be present automatically to the client. In a direct connection between client and storage the administrator will coordinate between the client and storage before making any changes, in my configuration I do not want to add

Quoted 8 lines

administration work on my box that stands between the client and the storage.
Addition is simple, the application on my box can once in a while scan the SCSI BUS, and then the Linux system will create new sg devices. In removal of a device the Linux system will not remove sg devices after scan. In that case again I can do it from my application, such when I get attention on SCSI command, I can decide to remove the device by echoing to the /sys file system.

I really don't know how the FibreChannel drivers handle device removal. But from what I gather from the documentation of fc_remote_port_delete(), which the qla2xxx drivers use, the /dev/sg devices should automatically disappear
- after a connection loss timeout,
- provided that userspace doesn't have the device file opened (among else this is manifest in the kernel in a reference count of the logical unit device representation),
- provided that you manage your device files with udev.

Quoted 2 lines

http://lxr.linux.no/linux+v2.6.26/drivers/scsi/scsi_transport_fc.c#L2756

Quoted 11 lines

This is true; when the remote port is no longer exists, after timeout all the sg devices will disappear, but what if I am changing the configuration in the disk array, and remove only one logical device. There will not be any event for that. The hotplug implementation in the SCSI today creates events of add/remove when adding/removing HBA, or when the FC link go down and up again, but not for additional and removal of devices. In the linux documentation in scsi_mid_low_api.txt it is written that "The hotplug concept may be extended to SCSI devices". I guess that nobody implemented it because what you said previously in this thread that someone could hold reference to the device.

Oh ... you're not really talking about hotplug, which is why everyone is confused. Hotplug is when you add or remove something from the bus. What you've done is reconfigure the array.

Most of the hotplug we do depends on the transport model (because what's on the transport is changing). Array reconfiguration has no hotplug event because SAM-3 has no real way of passing the information asynchronously. The best it can do is Unit Attention/reported luns data has changed (asc=0x3f/ascq=0xe) on the next command.

The problem is that there's no way to process the event correctly even when we get it. All we can do is issue another report LUNS command and compare. However, just because it looks like a single LUN disappeared doesn't mean the others weren't permuted or altered in some way (which data we cannot get).

Quoted 11 lines · James Bottomley

Most of the hotplug we do depends on the transport model (because what's on the transport is changing). Array reconfiguration has no hotplug event because SAM-3 has no real way of passing the information asynchronously. The best it can do is Unit Attention/reported luns data has changed (asc=0x3f/ascq=0xe) on the next command.

The problem is that there's no way to process the event correctly even when we get it. All we can do is issue another report LUNS command and compare. However, just because it looks like a single LUN disappeared doesn't mean the others weren't permuted or altered in some way (which data we cannot get).

But why don't do what's possible to do and for what there is complete
information: add new LUN for new LUN and delete deleted LUN for reported luns data has changed UA? As well as get new capacity for a LU on capacity data has changed UA after, e.g., new space added to it?

Quoted 7 lines · Vladislav Bolkhovitin

Quoted 6 lines · James Bottomley

Most of the hotplug we do depends on the transport model (because what's on the transport is changing). Array reconfiguration has no hotplug event because SAM-3 has no real way of passing the information asynchronously. The best it can do is Unit Attention/reported luns data has changed (asc=0x3f/ascq=0xe) on the next command.

Quoted 6 lines

Quoted 5 lines

The problem is that there's no way to process the event correctly even when we get it. All we can do is issue another report LUNS command and compare. However, just because it looks like a single LUN disappeared doesn't mean the others weren't permuted or altered in some way (which data we cannot get).

But why don't do what's possible to do and for what there is complete information: add new LUN for new LUN and delete deleted LUN for reported luns data has changed UA? As well as get new capacity for a LU on capacity data has changed UA after, e.g., new space added to it?

Well, feel free to submit a patch, but it's an incredible mine field. We certainly can't do something that builds a trap for users (as in a policy which gets it right 10% of the time and wrong 90% of the time) so I'm strongly inclined to feel array reconfiguration is a manual process. If you think you can do it online, we don't panic or throw a spanner in the works, but we wait until you tell us what you did, but you'd be well advised to do it offline.

The point is that we can't distinguish LUN added from LUN split or space reapportioned or something ... even LUN renumbered would be seen as a remove followed by an add.

Oh ... you're not really talking about hotplug, which is why everyone is confused. Hotplug is when you add or remove something from the bus. What you've done is reconfigure the array.

Most of the hotplug we do depends on the transport model (because what's on the transport is changing). Array reconfiguration has no hotplug event because SAM-3 has no real way of passing the information asynchronously. The best it can do is Unit Attention/reported luns data has changed (asc=0x3f/ascq=0xe) on the next command.

The problem is that there's no way to process the event correctly even when we get it. All we can do is issue another report LUNS command and compare. However, just because it looks like a single LUN disappeared doesn't mean the others weren't permuted or altered in some way (which data we cannot get).

So ... this event is manual intervention required because you have to tell SCSI what you did to the array, it can't work it out for itself.

James

Quoted 10 lines · James Bottomley

Oh ... you're not really talking about hotplug, which is why everyone is confused. Hotplug is when you add or remove something from the bus. What you've done is reconfigure the array.

Most of the hotplug we do depends on the transport model (because what's on the transport is changing). Array reconfiguration has no hotplug event because SAM-3 has no real way of passing the information asynchronously. The best it can do is Unit Attention/reported luns data has changed (asc=0x3f/ascq=0xe) on the next command.

Quoted 6 lines

The problem is that there's no way to process the event correctly even when we get it. All we can do is issue another report LUNS command and compare. However, just because it looks like a single LUN disappeared doesn't mean the others weren't permuted or altered in some way (which data we cannot get).

Why? if I keep a mapping between sg devices and device id (Inquiry page 0x83) then I have all the data.

It won't change the device identification of the existing 10 LUNs, but it will report UA of "Power on, reset, or bus device reset occured", then I can send read capacity command.

-----Original Message-----
From: James Bottomley [mailto:[email protected]]
Sent: Thu 7/31/2008 7:20 PM
To: Gal Rosen
Cc: Stefan Richter; [email protected]
Subject: Re: SCSI device rescan, detection of disconnected device,or switched devices.

Quoted 2 lines · Gal Rosen

Quoted 1 line · James Bottomley

Oh ... you're not really talking about hotplug, which is why

everyone is

Quoted 1 line

Quoted 1 line

confused. Hotplug is when you add or remove something from the

bus.

Quoted 3 lines

Quoted 3 lines

What you've done is reconfigure the array.

Most of the hotplug we do depends on the transport model (because

what's

Quoted 1 line

Quoted 1 line

on the transport is changing). Array reconfiguration has no

hotplug

Quoted 2 lines

Quoted 2 lines

event because SAM-3 has no real way of passing the information asynchronously. The best it can do is Unit Attention/reported

luns data

Quoted 3 lines

Quoted 3 lines

has changed (asc=0x3f/ascq=0xe) on the next command.

The problem is that there's no way to process the event correctly

even

Quoted 1 line

Quoted 1 line

when we get it. All we can do is issue another report LUNS

command and

Quoted 1 line

Quoted 1 line

compare. However, just because it looks like a single LUN

disappeared

Quoted 1 line

Quoted 1 line

doesn't mean the others weren't permuted or altered in some way

(which

Quoted 4 lines

Quoted 2 lines

data we cannot get).

?Why? if I keep a mapping between sg devices and device id (Inquiry

page

Quoted 1 line

0x83) then I have all the data.

No we don't, it depends on array policy. Most arrays when told to take
10% of space from 10 luns and create a new one won't change the WWN of the existing 10 LUNS, so we have no idea what just happened or what we should do about it (incidentally, this is an example of a reconfiguration you probably wanted to do offline anyway).

James

-----Original Message-----
This is true; when the remote port is no longer exists, after timeout all the sg devices will disappear, but what if I am changing the configuration in the disk array, and remove only one logical device. There will not be any event for that. The hotplug implementation in the
SCSI today creates events of add/remove when adding/removing HBA, or when the FC link go down and up again, but not for additional and removal of devices. In the linux documentation in scsi_mid_low_api.txt

it is written that "The hotplug concept may be extended to SCSI devices". I guess that nobody implemented it because what you said previously in this thread that someone could hold reference to the device.
I think that I understand the concept. When the FC port is going down/up
you get interrupt, so you can remove all devices or initiate scsi scan and add devices.

But when device is added you can't get any event,
because this device is not known to the client yet (no body is working with it), all you can do is to initiate scan every once a while. For removing a device there is a possibility to do it automatically. The LLD
can identify that the device is not ready, and then call to the mid layer to remove this device. It will succeed only if nobody is holding a
reference to that device.

Gal.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Stefan Richter
Sent: Thursday, July 31, 2008 4:35 AM
To: Gal Rosen
Cc: [email protected]
Subject: Re: SCSI device rescan, detection of disconnected device, or switched devices.

Quoted 2 lines · Gal Rosen

I will try to make my question more clear.
I would like to put a box between the client and the

storage, and to be

Quoted 8 lines

fully transparent against the host. All changes on the storage (add/remove/extend devices) should be present automatically to the client. In a direct connection between client and storage the administrator will coordinate between the client and storage before making any changes, in my configuration I do not want to add administration work on my box that stands between the client and the storage.
Addition is simple, the application on my box can once in a

while scan

Quoted 2 lines

the SCSI BUS, and then the Linux system will create new sg devices. In removal of a device the Linux system will not remove sg

devices after

Quoted 1 line

scan. In that case again I can do it from my application,

such when I

Quoted 2 lines

get attention on SCSI command, I can decide to remove the device by echoing to the /sys file system.

I really don't know how the FibreChannel drivers handle device removal.
But from what I gather from the documentation of fc_remote_port_delete(), which the qla2xxx drivers use, the /dev/sg devices should automatically disappear
- after a connection loss timeout,
- provided that userspace doesn't have the device file opened (among else this is manifest in the kernel in a reference count of the logical unit device representation),
- provided that you manage your device files with udev.

http://lxr.linux.no/linux+v2.6.26/drivers/scsi/scsi_transport_
fc.c#L2756

That would be basically just like SCSI device files are created and destroyed on the fly for USB or FireWire devices for example. Except that FC has the additional configurable connection loss timeout which hides temporary connection losses from applications.

(However, better wait for an answer of somebody who actually uses FC or
knows the drivers. I only have experience with FireWire.)

Signature · 8 lines

Stefan Richter
-=====-==--- -=== =====
http://arcgraph.de/sr/
--
To unsubscribe from this list: send the line "unsubscribe 
linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html