SRP automatic target discovery

Hi;

I'm having a problem with SRP automatic target discovery. I am using srp_daemon with OFED 1.5.2 on the initiator. My one target machine with several LUNs also uses OFED 1.5.2, and I am using scst 2.0.0.1 and srpt 2.0.0 on top of that.

On the initiator, I have SRP_LOAD=yes and SRP_DAEMON_ENABLE=yes. This seems to work fine, and after rebooting, I can see the SRP targets on the initiator, and srp_daemon running in the background:

[root@hpcoss2 ~]# ps auxww | grep srp

root      6635  0.0  0.0  63840  1176 ?        S    12:04   0:00 /bin/bash /usr/sbin/srp_daemon.sh
root      6638  0.0  0.0  63836  1164 ?        S    12:04   0:00 /bin/bash /usr/sbin/run_srp_daemon -e -c -n -i mlx4_1 -p 1 -R 60 -V
root      6647  0.0  0.0  31012   996 ?        SLl  12:04   0:00 srp_daemon -e -c -n -i mlx4_1 -p 1 -R 60 -V

But if I add an additional target on the same target machine with, say:

[root@hpcoss1 ~]# echo "add 7:0:1:0 3" >/proc/scsi_tgt/groups/Default/devices

the srp_daemon on the initiator never sees the new storage and restarting srp_daemon doesn't seem to help.

If I remove the SRP scsi devices, unload the ib_srp module, and reload it, I can see the new LUN (and I don't need to restart srp_daemon, either). But I don't think that is how it is supposed to work.

Am I doing something wrong?

Thanks,
Craig Prescott
UF HPC Center

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Sounds about right for the existing code. There's no notification to the initiator that you added a new LUN on an existing connection, so the LUN will not show up on the client. There is a mechanism in the SCSI standard for such things, but the Linux SCSI mid-layer does not currently implement it.

Restarting the daemon without removing the ib_srp module doesn't help you, because when you restart the daemon, it notices you already have a connection to that particular SRP target. Unloading/reloading the ib_srp module tears down the existing connections, so srp_daemon will cause a reconnection when it notices.

If you want to avoid having to unload/reload right now, you can manually tell the client's SCSI layer to scan that LUN to get it to show up.

Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

I'm sorry, but I need to correct you. There is notification to the initiator that you added a new LUN on an existing connection. It is REPORT LUNS DATA HAVE CHANGED Unit Attention. SCST sends it. But it isn't processed anyhow on the initiator.

Vlad
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello Craig,

In addition to what Dave and Vlad already wrote:
* In the SCSI Architectural Model (SAM) it is specified that a logical unit shall generate a unit attention condition whenever e.g. the INQUIRY data has changed. The mechanism for reporting unit attention conditions to an initiator is transport specific.
* As Vlad explained, the SCST core supports this mechanism. The ib_srpt driver however ignores unit attention conditions and does not propagate these to the SRP initiator.
* The mechanism defined in the SRP draft for reporting unit attention conditions by a target to an initiator is the SRP_AER_REQ information unit.
* Neither the Linux SRP initiator nor the VMware ESX initiator nor the Windows SRP initiator currently support the SRP_AER_REQ information unit. Even worse, some versions of these initiators cause a kernel crash upon reception of a SRP_AER_REQ information unit.
* Support in the Linux SCSI layer (at the initiator side) for unit attention conditions is very limited. Currently only media change events are supported (SDEV_EVT_MEDIA_CHANGE).
* As Dave already wrote, you can tell the initiator SCSI layer to rescan LUNs manually. This script can be convenient to rescan LUNs:
http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh-1.48.
* The goal of srp_daemon is to discover targets. It is out of scope for srp_daemon to detect LUN changes of a target.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html