: Add proper ALL_TG_PT=1 allocation support
Subsystems (2)
PATCH 1/1 : Add proper ALL_TG_PT=1 allocation support
From: Nicholas Bellinger <[email protected]>
This patch adds proper per target port allocation of t10_pr_registration_t to handle the SPC4 defined ALL_TG_PT=1 feature bit using __core_scsi3_alloc_registration() The ALL_TG_PT=1 logic had only working as expected across multiple target ports the same I_T Nexus, so this patch adds proper support for doing ALL_TG_PT=1 across multiple I_T Nexus for all target ports within the same underlying fabric module.
The main logic in __core_scsi3_alloc_registration() walks the se_device_t port list ( dev->dev_sep_list) and grabs a local reference via se_port_t->sep_tg_pt_ref_cnt, then drops the port lock and walks se_port_t->sep_alua_list for the list of the SCSI Initiator Node ACL MappedLUNs entries for the matching target port. From there this patch uses the existing reference logic in se_dev_entry_t->pr_ref_count and core_scsi3_lunacl_depend_item() in order to make sure that the underlying configfs group for the destination MappedLUN (and everything below) does not disappear before a successful PR registration with ALL_TG_PT=1 can be completed.
At this point the allocated ALL_TG_PT=1 registrations for I_T Nexus'es other than the one that the original PROUT REGISTER was received on will be added to t10_pr_registration_t->pr_reg_atp_list, and these PR registrations are added to se_device_t PR REG list by the existing calls to __core_scsi3_add_registration(). From there __core_scsi3_add_registration() will walk to pr_reg_atp_list and add each registration and drop any configfs group references with core_scsi3_lunacl_undepend_item() that where created in __core_scsi3_alloc_registration()
This patch also adds proper exception path handling for ALL_TG_PT=1 failures with the SPEC_I_PT=1 feature bit in core_scsi3_decode_spec_i_port(). Here is how a multi I_T Nexus ALL_TG_PT=1 test case looks now between two SCSI Target Ports (LIO-Target iSCSI) across two SCSI I_T Nexuses (iSCSI Sessions) using APTPL=1.
root@ubuntu:~# multipath -ll <SNIP> mpath4 (36001405a97e4ce21c0711de829b000c2) dm-2 LIO-ORG,IBLOCK [size=1.5G][features=0][hwhandler=1 alua] \_ round-robin 0 [prio=50][enabled] \_ 3:0:0:0 sde 8:64 [active][ready] \_ round-robin 0 [prio=10][active] \_ 4:0:0:0 sdf 8:80 [active][ready]
root@ubuntu:~# lsscsi --transport <SNIP> [3:0:0:0] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1 /dev/sde [4:0:0:0] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1 /dev/sdf
*) Call PROUT REGISTER with ALL_TG_PT=1 and APTPL=1 on SCSI RTPI=2 at /dev/sde
root@ubuntu:~# sg_persist -vvv --out -Y -Z --device /dev/sde --register --param-sark=0x1234ffff
open /dev/sde with flags=0x800
inquiry cdb: 12 00 00 00 24 00
duration=0 ms
LIO-ORG IBLOCK 3.1
Peripheral device type: disk
open /dev/sde with flags=0x802
Persistent Reservation Out cmd: 5f 00 00 00 00 00 00 00 18 00
Persistent Reservation Out parameters:
00 00 00 00 00 00 00 00 00 00 00 00 00 12 34 ff ff .............4..
10 00 00 00 00 05 00 00 00 ........
duration=8 ms
PR out: command (Register) successful*) Call PROUT RESERVE on SCSI RPTI=1 at /dev/sdf
root@ubuntu:~# sg_persist -vvv --out --device /dev/sdf --reserve --param-rk=0x1234ffff -T 5
open /dev/sdf with flags=0x800
inquiry cdb: 12 00 00 00 24 00
duration=120 ms
LIO-ORG IBLOCK 3.1
Peripheral device type: disk
open /dev/sdf with flags=0x802
Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
Persistent Reservation Out parameters:
00 00 00 00 00 12 34 ff ff 00 00 00 00 00 00 00 00 .....4..........
10 00 00 00 00 00 00 00 00 ........
duration=112 ms
PR out: command (Reserve) successful*) PRIN READ_FULL_STATUS
root@ubuntu:~# sg_persist --in -s -d /dev/sde
LIO-ORG IBLOCK 3.1
Peripheral device type: disk
PR generation=0x2
Key=0x1234ffff
All target ports bit set
not reservation holder
Transport Id of initiator:
iSCSI world wide unique port id: iqn.1993-08.org.debian:01:2dadf92d0ef
Key=0x1234ffff
All target ports bit set
<< Reservation holder >>
scope: LU_SCOPE, type: Write Exclusive, registrants only
Transport Id of initiator:
iSCSI world wide unique port id: iqn.1993-08.org.debian:01:2dadf92d0ef*) TCM side kernel ring buffer output
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] registration on Target Port: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,0x0001 SPC-3 PR [iSCSI] for ALL TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000001234ffff PRgeneration: 0x00000000 APTPL: 1
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] registration on Target Port: iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,0x0001 SPC-3 PR [iSCSI] for ALL TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000001234ffff PRgeneration: 0x00000001 APTPL: 1
SPC-3 PR: Set APTPL Bit Activated for REGISTER
SPC-3 PR [iSCSI] Service Action: RESERVE created new reservation holder TYPE: Write Exclusive Access, Registrants Only ALL_TG_PT: 1 SPC-3 PR [iSCSI] RESERVE Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR: Updated APTPL metadata for RESERVE
*) TCM side PR APTPL metadata for storage object backstore
target:~/lio-core-2.6.git# cat /var/target/pr/aptpl_a97e4ce21c0711de829b000c2943d57b PR_REG_START: 0 initiator_fabric=iSCSI initiator_node=iqn.1993-08.org.debian:01:2dadf92d0ef sa_res_key=305463295 res_holder=0 res_all_tg_pt=4 mapped_lun=0 target_fabric=iSCSI target_node=iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd tpgt=1 port_rtpi=2 target_lun=0 PR_REG_END: 0 PR_REG_START: 1 initiator_fabric=iSCSI initiator_node=iqn.1993-08.org.debian:01:2dadf92d0ef sa_res_key=305463295 res_holder=1 res_type=05 res_scope=00 res_all_tg_pt=4 mapped_lun=0 target_fabric=iSCSI target_node=iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0 tpgt=1 port_rtpi=1 target_lun=0 PR_REG_END: 1
--- drivers/target/target_core_device.c | 9 ++ drivers/target/target_core_pr.c | 237 +++++++++++++++++++++++++++++++---- include/target/target_core_base.h | 4 + 3 files changed, 224 insertions(+), 26 deletions(-)
drivers/target/target_core_device.c +9 -0
@@ -752,6 +752,15 @@ void core_export_port(*/ void core_release_port (se_device_t *dev,se_port_t *port) { +/* +* Wait for any port reference for PR ALL_TG_PT=1 operation +* to complete in __core_scsi3_alloc_registration() +*/ +spin_unlock (&dev->se_port_lock); +if (atomic_read (&port->sep_tg_pt_ref_cnt)) +msleep (100 ); +spin_lock (&dev->se_port_lock); +core_alua_free_tg_pt_gp_mem (port);list_del (&port->sep_list);
drivers/target/target_core_pr.c +211 -26
299 lines not on this page · raw message
include/target/target_core_base.h +4 -0
@@ -361,6 +361,8 @@ typedef struct t10_pr_registration_s {struct list_head pr_reg_list;struct list_head pr_reg_abort_list;struct list_head pr_reg_aptpl_list; +struct list_head pr_reg_atp_list; +struct list_head pr_reg_atp_mem_list; }____cacheline_aligned t10_pr_registration_t; typedef struct t10_reservation_template_s {
@@ -929,6 +931,8 @@ typedef struct se_port_s {/* Used for ALUA Target Port Groups membership */ atomic_t sep_tg_pt_gp_active;atomic_t sep_tg_pt_secondary_offline; +/* Used for PR ALL_TG_PT=1 */ +atomic_t sep_tg_pt_ref_cnt;spinlock_t sep_alua_lock;struct mutex sep_tg_pt_md_mutex;struct t10_alua_tg_pt_gp_member_s *sep_alua_tg_pt_gp_mem;
Signature · 1 line
1.5.6.5