Skip to contentSkip to conversation

: Add proper ALL_TG_PT=1 allocation support

PATCH New Nicholas A. Bellinger 1 participant 1 message Mar 23, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 : Add proper ALL_TG_PT=1 allocation support

From: Nicholas Bellinger <[email protected]>

This patch adds proper per target port allocation of t10_pr_registration_t to handle the SPC4 defined ALL_TG_PT=1 feature bit using __core_scsi3_alloc_registration() The ALL_TG_PT=1 logic had only working as expected across multiple target ports the same I_T Nexus, so this patch adds proper support for doing ALL_TG_PT=1 across multiple I_T Nexus for all target ports within the same underlying fabric module.

The main logic in __core_scsi3_alloc_registration() walks the se_device_t port list ( dev->dev_sep_list) and grabs a local reference via se_port_t->sep_tg_pt_ref_cnt, then drops the port lock and walks se_port_t->sep_alua_list for the list of the SCSI Initiator Node ACL MappedLUNs entries for the matching target port. From there this patch uses the existing reference logic in se_dev_entry_t->pr_ref_count and core_scsi3_lunacl_depend_item() in order to make sure that the underlying configfs group for the destination MappedLUN (and everything below) does not disappear before a successful PR registration with ALL_TG_PT=1 can be completed.

At this point the allocated ALL_TG_PT=1 registrations for I_T Nexus'es other than the one that the original PROUT REGISTER was received on will be added to t10_pr_registration_t->pr_reg_atp_list, and these PR registrations are added to se_device_t PR REG list by the existing calls to __core_scsi3_add_registration(). From there __core_scsi3_add_registration() will walk to pr_reg_atp_list and add each registration and drop any configfs group references with core_scsi3_lunacl_undepend_item() that where created in __core_scsi3_alloc_registration()

This patch also adds proper exception path handling for ALL_TG_PT=1 failures with the SPEC_I_PT=1 feature bit in core_scsi3_decode_spec_i_port(). Here is how a multi I_T Nexus ALL_TG_PT=1 test case looks now between two SCSI Target Ports (LIO-Target iSCSI) across two SCSI I_T Nexuses (iSCSI Sessions) using APTPL=1.

root@ubuntu:~# multipath -ll
<SNIP>
mpath4 (36001405a97e4ce21c0711de829b000c2) dm-2 LIO-ORG,IBLOCK
[size=1.5G][features=0][hwhandler=1 alua]
\_ round-robin 0 [prio=50][enabled]
 \_ 3:0:0:0 sde 8:64  [active][ready]
\_ round-robin 0 [prio=10][active]
 \_ 4:0:0:0 sdf 8:80  [active][ready]
root@ubuntu:~# lsscsi --transport
<SNIP>
[3:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1  /dev/sde
[4:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdf

*) Call PROUT REGISTER with ALL_TG_PT=1 and APTPL=1 on SCSI RTPI=2 at /dev/sde

root@ubuntu:~# sg_persist -vvv --out -Y -Z --device /dev/sde --register --param-sark=0x1234ffff
open /dev/sde with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
      duration=0 ms
  LIO-ORG  IBLOCK  3.1
  Peripheral device type: disk
open /dev/sde with flags=0x802
    Persistent Reservation Out cmd: 5f 00 00 00 00 00 00 00 18 00
    Persistent Reservation Out parameters:
 00     00 00 00 00 00 00 00 00  00 00 00 00 12 34 ff ff    .............4..
 10     00 00 00 00 05 00 00 00                             ........
      duration=8 ms
PR out: command (Register) successful

*) Call PROUT RESERVE on SCSI RPTI=1 at /dev/sdf

root@ubuntu:~# sg_persist -vvv --out --device /dev/sdf --reserve --param-rk=0x1234ffff -T 5
open /dev/sdf with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
      duration=120 ms
  LIO-ORG  IBLOCK  3.1
  Peripheral device type: disk
open /dev/sdf with flags=0x802
    Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
    Persistent Reservation Out parameters:
 00     00 00 00 00 12 34 ff ff  00 00 00 00 00 00 00 00    .....4..........
 10     00 00 00 00 00 00 00 00                             ........
      duration=112 ms
PR out: command (Reserve) successful

*) PRIN READ_FULL_STATUS

root@ubuntu:~# sg_persist --in -s -d /dev/sde
  LIO-ORG  IBLOCK  3.1
  Peripheral device type: disk
  PR generation=0x2
    Key=0x1234ffff
      All target ports bit set
      not reservation holder
      Transport Id of initiator:
        iSCSI world wide unique port id: iqn.1993-08.org.debian:01:2dadf92d0ef
    Key=0x1234ffff
      All target ports bit set
      << Reservation holder >>
      scope: LU_SCOPE,  type: Write Exclusive, registrants only
      Transport Id of initiator:
        iSCSI world wide unique port id: iqn.1993-08.org.debian:01:2dadf92d0ef

*) TCM side kernel ring buffer output

SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] registration on Target Port: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,0x0001 SPC-3 PR [iSCSI] for ALL TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000001234ffff PRgeneration: 0x00000000 APTPL: 1
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] registration on Target Port: iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,0x0001 SPC-3 PR [iSCSI] for ALL TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000001234ffff PRgeneration: 0x00000001 APTPL: 1
SPC-3 PR: Set APTPL Bit Activated for REGISTER
SPC-3 PR [iSCSI] Service Action: RESERVE created new reservation holder TYPE: Write Exclusive Access, Registrants Only ALL_TG_PT: 1 SPC-3 PR [iSCSI] RESERVE Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR: Updated APTPL metadata for RESERVE

*) TCM side PR APTPL metadata for storage object backstore

target:~/lio-core-2.6.git# cat /var/target/pr/aptpl_a97e4ce21c0711de829b000c2943d57b
PR_REG_START: 0
initiator_fabric=iSCSI
initiator_node=iqn.1993-08.org.debian:01:2dadf92d0ef
sa_res_key=305463295
res_holder=0
res_all_tg_pt=4
mapped_lun=0
target_fabric=iSCSI
target_node=iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd
tpgt=1
port_rtpi=2
target_lun=0
PR_REG_END: 0
PR_REG_START: 1
initiator_fabric=iSCSI
initiator_node=iqn.1993-08.org.debian:01:2dadf92d0ef
sa_res_key=305463295
res_holder=1
res_type=05
res_scope=00
res_all_tg_pt=4
mapped_lun=0
target_fabric=iSCSI
target_node=iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0
tpgt=1
port_rtpi=1
target_lun=0
PR_REG_END: 1
---
 drivers/target/target_core_device.c |    9 ++
 drivers/target/target_core_pr.c     |  237 +++++++++++++++++++++++++++++++----
 include/target/target_core_base.h   |    4 +
 3 files changed, 224 insertions(+), 26 deletions(-)
drivers/target/target_core_device.c +9 -0
index c6a8eda..b0cbf8d 100644
@@ -752,6 +752,15 @@ void core_export_port(
  */
 void core_release_port(se_device_t *dev, se_port_t *port)
 {
+	/*
+	 * Wait for any port reference for PR ALL_TG_PT=1 operation
+	 * to complete in __core_scsi3_alloc_registration()
+	 */
+	spin_unlock(&dev->se_port_lock);
+	if (atomic_read(&port->sep_tg_pt_ref_cnt))
+		msleep(100);
+	spin_lock(&dev->se_port_lock);
+
 	core_alua_free_tg_pt_gp_mem(port);
 
 	list_del(&port->sep_list);
drivers/target/target_core_pr.c +211 -26

299 lines not on this page · raw message

include/target/target_core_base.h +4 -0
index b6e4f0a..04741ae 100644
@@ -361,6 +361,8 @@ typedef struct t10_pr_registration_s {
 	struct list_head pr_reg_list;
 	struct list_head pr_reg_abort_list;
 	struct list_head pr_reg_aptpl_list;
+	struct list_head pr_reg_atp_list;
+	struct list_head pr_reg_atp_mem_list;
 } ____cacheline_aligned t10_pr_registration_t;
 
 typedef struct t10_reservation_template_s {
@@ -929,6 +931,8 @@ typedef struct se_port_s {
 	/* Used for ALUA Target Port Groups membership */
 	atomic_t	sep_tg_pt_gp_active;
 	atomic_t	sep_tg_pt_secondary_offline;
+	/* Used for PR ALL_TG_PT=1 */
+	atomic_t	sep_tg_pt_ref_cnt;
 	spinlock_t	sep_alua_lock;
 	struct mutex	sep_tg_pt_md_mutex;
 	struct t10_alua_tg_pt_gp_member_s *sep_alua_tg_pt_gp_mem;
Signature · 1 line
1.5.6.5