Skip to contentSkip to conversation

Target cluster implementation over DLM

RFC PATCH In review Dmitry Bogdanov 2 participants 51 messages Aug 3, 2022 – Aug 4, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Target cluster implementation over DLM

Hi linux target comminity.

Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.

The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.

Patches 1-9:
Make RTPI be part of se_tpg instead of se_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).

Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undepend pr_reg from se_nacl and se_tpg to be just a registration holder. Make APTPL registrations (not linked to se_dev_entry) be full-fledged registrations.

Patches 30-34:
DLM_CKV module that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.

Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.

Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attrib cluster_impl regulates which implementation to use for that device. 'single' is for default (no cluster) implementation.

Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* Use DLM_CKV Lock service to serialize order of PR OUT commands
* Use DLM_CKV Key-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* Use DLM_CKV Notification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.

It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster

How to test

1. Setup DLM over worked corosync & pacemaker cluster

 $ zypper install libdlm libdlm3
 $ crm configure primitive dlm ocf:pacemaker:controld args="-q0 -f0" allow_stonith_disabled=true  op monitor interval="60" timeout="60"
 $ crm configure clone clone-dlm dlm meta interleave=true target-role=Started
2. setup TCM cluster configuration
You can use my script [2] that I used to use while developing the patchset.
3. Test what you want.
For example, my test report [3].

[1] https://lore.kernel.org/all/[email protected]/
[2] https://pastebin.com/HgCfjywh
[3] https://pastebin.com/AgLSgnWn

Dmitry Bogdanov (39):
target: core: check RTPI uniquity for enabled TPG
target: core: fix preempt and abort for allreg res
target: core: fix memory leak in preempt_and_abort
target: core: abort all preempted regs if requested
target: core: new key must be used for moved PR
target: core: remove unused variable in se_dev_entry
target: core: undepend PR registrant of nacl
target: core: make some functions public
target: core: proper clear reservation on LUN RESET
target: core: remove superfluous checks
target: core: proper check of SCSI-2 reservation
target: core: checks against peer node SCSI2 reservation
target: core: UA on all luns after reset
target: core: refactor LUN_RESET code
target: core: pr: use RTPI in APTPL
target: core: pr: have Transport ID stored
target: core: pr: remove se_tpg from pr_reg
target: core: fix parsing PR OUT TID
target: core: add function to compare TransportID
target: core: store proto_id in APTPL
target: core: rethink APTPL registrations
dlm_ckv: introduce DLM cluster key-value storage
dlm_ckv: add notification service
dlm_ckv: add key-value storage service
dlm_ckv: add KV get/set async API
target: add virtual remote target
target: cluster: introduce cluster ops
target: cluster: introduce dlm cluster
target: cluster: store PR data in DLM cluster
target: cluster: read PR data from cluster
target: cluster: sync PR for dynamic acls
target: cluster: sync-up PR data on cluster join
target: cluster: sync SPC-2 reservations
target: cluster: allocate UAs on PR sync
target: cluster: support PR OUT preempt and abort
target: cluster: add reset cluster function
target: cluster: implement LUN reset in DLM cluster
target: cluster: split cluster sync function
target: cluster: request data on initial sync

Konstantin Shelekhin (1):
scsi: target/core: Unlock PR generation bump

Roman Bolshakov (8):
scsi: target/core: Add a way to hide a port group
scsi: target/core: Set MULTIP bit for se_device with multiple ports
scsi: target/core: Add cleanup sequence in core_tpg_register()
scsi: target/core: Add RTPI field to target port
scsi: target/core: Use RTPI from target port
scsi: target/core: Drop device-based RTPI
scsi: target/core: Add common port attributes
scsi: target/core: Add RTPI attribute for target port

 drivers/target/Kconfig                       |    7 +
 drivers/target/Makefile                      |    4 +
 drivers/target/dlm_ckv.c                     |  757 +++++++++++++
 drivers/target/dlm_ckv.h                     |   44 +
 drivers/target/target_cluster_dlm.c          | 1012 ++++++++++++++++++
 drivers/target/target_core_alua.c            |   12 +-
 drivers/target/target_core_configfs.c        |  191 +++-
 drivers/target/target_core_device.c          |  244 ++++-
 drivers/target/target_core_fabric_configfs.c |   68 +-
 drivers/target/target_core_fabric_lib.c      |  262 +++--
 drivers/target/target_core_internal.h        |   29 +-
 drivers/target/target_core_pr.c              |  613 +++++------
 drivers/target/target_core_pr.h              |   33 +-
 drivers/target/target_core_sbc.c             |   12 +-
 drivers/target/target_core_spc.c             |   19 +-
 drivers/target/target_core_stat.c            |    6 +-
 drivers/target/target_core_tmr.c             |   38 +-
 drivers/target/target_core_tpg.c             |  193 +++-
 drivers/target/target_core_transport.c       |    9 +-
 drivers/target/target_core_ua.c              |    1 +
 drivers/target/tcm_remote/Kconfig            |    8 +
 drivers/target/tcm_remote/Makefile           |    2 +
 drivers/target/tcm_remote/tcm_remote.c       |  405 +++++++
 drivers/target/tcm_remote/tcm_remote.h       |   29 +
 include/target/target_core_base.h            |   56 +-
 25 files changed, 3436 insertions(+), 618 deletions(-)
 create mode 100644 drivers/target/dlm_ckv.c
 create mode 100644 drivers/target/dlm_ckv.h
 create mode 100644 drivers/target/target_cluster_dlm.c
 create mode 100644 drivers/target/tcm_remote/Kconfig
 create mode 100644 drivers/target/tcm_remote/Makefile
 create mode 100644 drivers/target/tcm_remote/tcm_remote.c
 create mode 100644 drivers/target/tcm_remote/tcm_remote.h
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH 1/48 scsi: target/core: Add a way to hide a port group

From: Roman Bolshakov <[email protected]>

Default target port group is always returned in the list of port groups, even if the behaviour is unwanted, i.e. it has no members and non-default port groups are primary port groups.

A new port group attribute - "hidden" can be used to hide empty port groups with no ports in REPORT TARGET PORT GROUPS, including default target port group:

echo 1 > $DEVICE/alua/default_tg_pt_gp/hidden

---
 drivers/target/target_core_alua.c     |  8 ++++++++
 drivers/target/target_core_configfs.c | 29 +++++++++++++++++++++++++++
 include/target/target_core_base.h     |  1 +
 3 files changed, 38 insertions(+)
drivers/target/target_core_alua.c +8 -0
index 58df0145e8d0..9d1a8238a4c6 100644
@@ -164,6 +164,8 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)
 	spin_lock(&dev->t10_alua.tg_pt_gps_lock);
 	list_for_each_entry(tg_pt_gp, &dev->t10_alua.tg_pt_gps_list,
 			tg_pt_gp_list) {
+		if (tg_pt_gp->tg_pt_gp_hidden && !tg_pt_gp->tg_pt_gp_members)
+			continue;
 		/*
 		 * Check if the Target port group and Target port descriptor list
 		 * based on tg_pt_gp_members count will fit into the response payload.
@@ -308,6 +310,12 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 		rc = TCM_UNSUPPORTED_SCSI_OPCODE;
 		goto out;
 	}
+	if (l_tg_pt_gp->tg_pt_gp_hidden && !tg_pt_gp->tg_pt_gp_members) {
+		spin_unlock(&l_lun->lun_tg_pt_gp_lock);
+		pr_err("Unable to change state for hidden port group with no members\n");
+		rc = TCM_INVALID_CDB_FIELD;
+		goto out;
+	}
 	valid_states = l_tg_pt_gp->tg_pt_gp_alua_supported_states;
 	rcu_read_unlock();
 
drivers/target/target_core_configfs.c +29 -0
index 416514c5c7ac..7c0e42e782de 100644
@@ -3029,6 +3029,33 @@ static ssize_t target_tg_pt_gp_preferred_store(struct config_item *item,
 	return core_alua_store_preferred_bit(to_tg_pt_gp(item), page, count);
 }
 
+static ssize_t target_tg_pt_gp_hidden_show(struct config_item *item,
+					   char *page)
+{
+	return sprintf(page, "%d\n", to_tg_pt_gp(item)->tg_pt_gp_hidden);
+}
+
+static ssize_t target_tg_pt_gp_hidden_store(struct config_item *item,
+					    const char *page, size_t count)
+{
+	struct t10_alua_tg_pt_gp *tg_pt_gp = to_tg_pt_gp(item);
+	int tmp, ret;
+
+	ret = kstrtoint(page, 0, &tmp);
+	if (ret < 0) {
+		pr_err("Unable to extract hidden flag: %d\n", ret);
+		return ret;
+	}
+
+	if (tmp != 0 && tmp != 1) {
+		pr_err("Illegal value for hidden flag: %d\n", tmp);
+		return -EINVAL;
+	}
+	tg_pt_gp->tg_pt_gp_hidden = tmp;
+
+	return count;
+}
+
 static ssize_t target_tg_pt_gp_tg_pt_gp_id_show(struct config_item *item,
 		char *page)
 {
@@ -3115,6 +3142,7 @@ CONFIGFS_ATTR(target_tg_pt_gp_, alua_support_standby);
 CONFIGFS_ATTR(target_tg_pt_gp_, alua_support_active_optimized);
 CONFIGFS_ATTR(target_tg_pt_gp_, alua_support_active_nonoptimized);
 CONFIGFS_ATTR(target_tg_pt_gp_, alua_write_metadata);
+CONFIGFS_ATTR(target_tg_pt_gp_, hidden);
 CONFIGFS_ATTR(target_tg_pt_gp_, nonop_delay_msecs);
 CONFIGFS_ATTR(target_tg_pt_gp_, trans_delay_msecs);
 CONFIGFS_ATTR(target_tg_pt_gp_, implicit_trans_secs);
@@ -3138,6 +3166,7 @@ static struct configfs_attribute *target_core_alua_tg_pt_gp_attrs[] = {
 	&target_tg_pt_gp_attr_trans_delay_msecs,
 	&target_tg_pt_gp_attr_implicit_trans_secs,
 	&target_tg_pt_gp_attr_preferred,
+	&target_tg_pt_gp_attr_hidden,
 	&target_tg_pt_gp_attr_tg_pt_gp_id,
 	&target_tg_pt_gp_attr_members,
 	NULL,
include/target/target_core_base.h +1 -0
index c2b36f7d917d..f607ff74c368 100644
@@ -297,6 +297,7 @@ struct t10_alua_tg_pt_gp {
 	int	tg_pt_gp_trans_delay_msecs;
 	int	tg_pt_gp_implicit_trans_secs;
 	int	tg_pt_gp_pref;
+	int	tg_pt_gp_hidden;
 	int	tg_pt_gp_write_metadata;
 	u32	tg_pt_gp_members;
 	int	tg_pt_gp_alua_access_state;
Signature · 1 line
2.25.1
#2 lore raw

RFC PATCH 2/48 scsi: target/core: Set MULTIP bit for se_device with multiple ports

In reply to Dmitry Bogdanov

From: Roman Bolshakov <[email protected]>

SAM-5 4.8.3 (SCSI target device with multiple SCSI ports structure) obligates to set MULTIP bit when there's multiple SCSI target ports:

Quoted 3 lines

Each device server shall indicate the presence of multiple SCSI target ports by setting the MULTIP bit to one in its standard INQUIRY data (see SPC-4).

The change sets MULTIP bit automatically to indicate the presence of multiple SCSI target ports within standard inquiry response data if there are multiple target ports in all target port groups of the se_device.

---
 drivers/target/target_core_spc.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
drivers/target/target_core_spc.c +17 -0
index c14441c89bed..252d704264ed 100644
@@ -75,6 +75,8 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	struct se_portal_group *tpg = lun->lun_tpg;
 	struct se_device *dev = cmd->se_dev;
 	struct se_session *sess = cmd->se_sess;
+	struct t10_alua_tg_pt_gp *tg_pt_gp;
+	u64 dev_ports = 0;
 
 	/* Set RMB (removable media) for tape devices */
 	if (dev->transport->get_device_type(dev) == TYPE_TAPE)
@@ -115,6 +117,21 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 			buf[5] |= 0x1;
 	}
 
+	spin_lock(&dev->t10_alua.tg_pt_gps_lock);
+	list_for_each_entry(tg_pt_gp, &dev->t10_alua.tg_pt_gps_list,
+			    tg_pt_gp_list) {
+		dev_ports += tg_pt_gp->tg_pt_gp_members;
+		/* Exact number of ports does not matter for MULTIP bit */
+		if (dev_ports > 1)
+			break;
+	}
+	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
+	/*
+	 * Set MULTIP bit to indicate presence of multiple SCSI target ports
+	 */
+	if (dev_ports > 1)
+		buf[6] |= 0x10;
+
 	buf[7] = 0x2; /* CmdQue=1 */
 
 	/*
Signature · 1 line
2.25.1
#3 lore raw

RFC PATCH 3/48 scsi: target/core: Add cleanup sequence in core_tpg_register()

In reply to Dmitry Bogdanov

From: Roman Bolshakov <[email protected]>

It does not change any functionality but allows to introduce more steps in the cleanup sequence without code duplication later.

---
 drivers/target/target_core_tpg.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/target/target_core_tpg.c +6 -4
index 736847c933e5..f0d38d77edcc 100644
@@ -484,10 +484,8 @@ int core_tpg_register(
 
 		ret = core_tpg_add_lun(se_tpg, se_tpg->tpg_virt_lun0,
 				true, g_lun0_dev);
-		if (ret < 0) {
-			kfree(se_tpg->tpg_virt_lun0);
-			return ret;
-		}
+		if (ret < 0)
+			goto out_free_lun0;
 	}
 
 	pr_debug("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, "
@@ -497,6 +495,10 @@ int core_tpg_register(
 		se_tpg->proto_id, se_tpg->se_tpg_tfo->tpg_get_tag(se_tpg));
 
 	return 0;
+
+out_free_lun0:
+	kfree(se_tpg->tpg_virt_lun0);
+	return ret;
 }
 EXPORT_SYMBOL(core_tpg_register);
 
Signature · 1 line
2.25.1
#4 lore raw

RFC PATCH 4/48 scsi: target/core: Add RTPI field to target port

In reply to Dmitry Bogdanov

From: Roman Bolshakov <[email protected]>

SAM-5 4.6.5.2 (Relative Port Identifier attribute) defines the attribute as unique across SCSI target ports.

The change introduces RTPI attribute to se_portal group. The value is auto-incremented and unique across all SCSI target ports. It also limits number of SCSI target ports to 65535.

---
 drivers/target/target_core_tpg.c  | 78 +++++++++++++++++++++++++++++--
 include/target/target_core_base.h |  4 ++
 2 files changed, 77 insertions(+), 5 deletions(-)
drivers/target/target_core_tpg.c +73 -5
index f0d38d77edcc..1b7ab98bbd04 100644
@@ -31,6 +31,10 @@
 #include "target_core_ua.h"
 
 extern struct se_device *g_lun0_dev;
+static u16 g_tpg_count;
+static u16 g_tpg_rtpi_counter = 1;
+static LIST_HEAD(g_tpg_list);
+static DEFINE_SPINLOCK(g_tpg_lock);
 
 /*	__core_tpg_get_initiator_node_acl():
  *
@@ -439,6 +443,57 @@ static void core_tpg_lun_ref_release(struct percpu_ref *ref)
 	complete(&lun->lun_shutdown_comp);
 }
 
+static int core_tpg_register_rtpi(struct se_portal_group *se_tpg)
+{
+	struct se_portal_group *tpg;
+
+	/*
+	 * Allocate the next RELATIVE TARGET PORT IDENTIFIER.
+	 * Here is the table from SPC-4 4.3.4:
+	 *
+	 *    Table 34 -- Relative target port identifier values
+	 *
+	 * Value		Description
+	 * 0h			Reserved
+	 * 1h			Relative port 1, historically known as port A
+	 * 2h			Relative port 2, historically known as port B
+	 * 3h to FFFFh		Relative port 3 through 65 535
+	 */
+	spin_lock(&g_tpg_lock);
+
+	if (g_tpg_count == 0xffff) {
+		spin_unlock(&g_tpg_lock);
+		pr_warn("Reached g_tpg_count == 0xffff\n");
+		return -ENOSPC;
+	}
+again:
+	se_tpg->tpg_rtpi = g_tpg_rtpi_counter++;
+	if (!se_tpg->tpg_rtpi)
+		goto again;
+
+	list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
+		/*
+		 * Make sure RELATIVE TARGET PORT IDENTIFIER is unique
+		 * for 16-bit wrap..
+		 */
+		if (se_tpg->tpg_rtpi == tpg->tpg_rtpi)
+			goto again;
+	}
+	list_add(&se_tpg->tpg_list, &g_tpg_list);
+	g_tpg_count++;
+	spin_unlock(&g_tpg_lock);
+
+	return 0;
+}
+
+static void core_tpg_deregister_rtpi(struct se_portal_group *se_tpg)
+{
+	spin_lock(&g_tpg_lock);
+	list_del(&se_tpg->tpg_list);
+	g_tpg_count--;
+	spin_unlock(&g_tpg_lock);
+}
+
 /* Does not change se_wwn->priv. */
 int core_tpg_register(
 	struct se_wwn *se_wwn,
@@ -471,6 +526,7 @@ int core_tpg_register(
 	se_tpg->proto_id = proto_id;
 	se_tpg->se_tpg_wwn = se_wwn;
 	atomic_set(&se_tpg->tpg_pr_ref_count, 0);
+	INIT_LIST_HEAD(&se_tpg->tpg_list);
 	INIT_LIST_HEAD(&se_tpg->acl_node_list);
 	INIT_LIST_HEAD(&se_tpg->tpg_sess_list);
 	spin_lock_init(&se_tpg->session_lock);
@@ -478,9 +534,15 @@ int core_tpg_register(
 	mutex_init(&se_tpg->acl_node_mutex);
 
 	if (se_tpg->proto_id >= 0) {
+		ret = core_tpg_register_rtpi(se_tpg);
+		if (ret < 0)
+			return ret;
+
 		se_tpg->tpg_virt_lun0 = core_tpg_alloc_lun(se_tpg, 0);
-		if (IS_ERR(se_tpg->tpg_virt_lun0))
-			return PTR_ERR(se_tpg->tpg_virt_lun0);
+		if (IS_ERR(se_tpg->tpg_virt_lun0)) {
+			ret = PTR_ERR(se_tpg->tpg_virt_lun0);
+			goto out_deregister_rtpi;
+		}
 
 		ret = core_tpg_add_lun(se_tpg, se_tpg->tpg_virt_lun0,
 				true, g_lun0_dev);
@@ -488,16 +550,20 @@ int core_tpg_register(
 			goto out_free_lun0;
 	}
 
-	pr_debug("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, "
-		 "Proto: %d, Portal Tag: %u\n", se_tpg->se_tpg_tfo->fabric_name,
+	pr_debug("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, Proto: %d, Portal Tag: %u, RTPI: %#2x\n",
+		se_tpg->se_tpg_tfo->fabric_name,
 		se_tpg->se_tpg_tfo->tpg_get_wwn(se_tpg) ?
 		se_tpg->se_tpg_tfo->tpg_get_wwn(se_tpg) : NULL,
-		se_tpg->proto_id, se_tpg->se_tpg_tfo->tpg_get_tag(se_tpg));
+		se_tpg->proto_id,
+		se_tpg->se_tpg_tfo->tpg_get_tag(se_tpg),
+		se_tpg->tpg_rtpi);
 
 	return 0;
 
 out_free_lun0:
 	kfree(se_tpg->tpg_virt_lun0);
+out_deregister_rtpi:
+	core_tpg_deregister_rtpi(se_tpg);
 	return ret;
 }
 EXPORT_SYMBOL(core_tpg_register);
@@ -537,6 +603,8 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 		kfree_rcu(se_tpg->tpg_virt_lun0, rcu_head);
 	}
 
+	core_tpg_deregister_rtpi(se_tpg);
+
 	return 0;
 }
 EXPORT_SYMBOL(core_tpg_deregister);
include/target/target_core_base.h +4 -0
index f607ff74c368..72fe94afc7a5 100644
@@ -904,6 +904,8 @@ struct se_portal_group {
 	 */
 	int			proto_id;
 	bool			enabled;
+	/* RELATIVE TARGET PORT IDENTIFIER */
+	u16			tpg_rtpi;
 	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
 	atomic_t		tpg_pr_ref_count;
 	/* Spinlock for adding/removing ACLed Nodes */
@@ -911,6 +913,8 @@ struct se_portal_group {
 	/* Spinlock for adding/removing sessions */
 	spinlock_t		session_lock;
 	struct mutex		tpg_lun_mutex;
+	/* List of all SCSI target ports */
+	struct list_head	tpg_list;
 	/* linked list for initiator ACL list */
 	struct list_head	acl_node_list;
 	struct hlist_head	tpg_lun_hlist;
Signature · 1 line
2.25.1
#5 lore raw

RFC PATCH 5/48 scsi: target/core: Use RTPI from target port

In reply to Dmitry Bogdanov

From: Roman Bolshakov <[email protected]>

Replace all references to RTPI from LUN field to se_portal_group field. It introduces consitent reporting of RTPI for all LUNs and all target ports.

---
 drivers/target/target_core_alua.c   | 4 ++--
 drivers/target/target_core_device.c | 2 +-
 drivers/target/target_core_pr.c     | 8 ++++----
 drivers/target/target_core_spc.c    | 2 +-
 drivers/target/target_core_stat.c   | 6 +++---
 5 files changed, 11 insertions(+), 11 deletions(-)
drivers/target/target_core_alua.c +2 -2
index 9d1a8238a4c6..774750715ba8 100644
@@ -224,7 +224,7 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)
 			/*
 			 * Set RELATIVE TARGET PORT IDENTIFIER
 			 */
-			put_unaligned_be16(lun->lun_rtpi, &buf[off]);
+			put_unaligned_be16(lun->lun_tpg->tpg_rtpi, &buf[off]);
 			off += 2;
 			rd_len += 4;
 		}
@@ -404,7 +404,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 			spin_lock(&dev->se_port_lock);
 			list_for_each_entry(lun, &dev->dev_sep_list,
 							lun_dev_link) {
-				if (lun->lun_rtpi != rtpi)
+				if (lun->lun_tpg->tpg_rtpi != rtpi)
 					continue;
 
 				// XXX: racy unlock
drivers/target/target_core_device.c +1 -1
index 086ac9c9343c..bbef6b869ee9 100644
@@ -223,7 +223,7 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
 				tpg->se_tpg_tfo->fabric_name);
 			continue;
 		}
-		if (lun->lun_rtpi != rtpi)
+		if (lun->lun_tpg->tpg_rtpi != rtpi)
 			continue;
 
 		kref_get(&deve->pr_kref);
drivers/target/target_core_pr.c +4 -4
index 3829b61b56c1..668670b1414a 100644
@@ -663,7 +663,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 	}
 	pr_reg->pr_res_mapped_lun = mapped_lun;
 	pr_reg->pr_aptpl_target_lun = lun->unpacked_lun;
-	pr_reg->tg_pt_sep_rtpi = lun->lun_rtpi;
+	pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;
 	pr_reg->pr_res_key = sa_res_key;
 	pr_reg->pr_reg_all_tg_pt = all_tg_pt;
 	pr_reg->pr_reg_aptpl = aptpl;
@@ -969,7 +969,7 @@ static int __core_scsi3_check_aptpl_registration(
 			rcu_read_unlock();
 
 			pr_reg->pr_reg_nacl = nacl;
-			pr_reg->tg_pt_sep_rtpi = lun->lun_rtpi;
+			pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;
 			list_del(&pr_reg->pr_reg_aptpl_list);
 			spin_unlock(&pr_tmpl->aptpl_reg_lock);
 			/*
@@ -1577,7 +1577,7 @@ core_scsi3_decode_spec_i_port(
 			 */
 			if (tmp_tpg->proto_id != proto_ident)
 				continue;
-			dest_rtpi = tmp_lun->lun_rtpi;
+			dest_rtpi = tmp_lun->lun_tpg->tpg_rtpi;
 
 			iport_ptr = NULL;
 			i_str = target_parse_pr_out_transport_id(tmp_tpg,
@@ -3221,7 +3221,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 
 	spin_lock(&dev->se_port_lock);
 	list_for_each_entry(tmp_lun, &dev->dev_sep_list, lun_dev_link) {
-		if (tmp_lun->lun_rtpi != rtpi)
+		if (tmp_lun->lun_tpg->tpg_rtpi != rtpi)
 			continue;
 		dest_se_tpg = tmp_lun->lun_tpg;
 		dest_tf_ops = dest_se_tpg->se_tpg_tfo;
drivers/target/target_core_spc.c +1 -1
index 252d704264ed..be18a72d9f8a 100644
@@ -332,7 +332,7 @@ spc_emulate_evpd_83(struct se_cmd *cmd, unsigned char *buf)
 		/* Skip over Obsolete field in RTPI payload
 		 * in Table 472 */
 		off += 2;
-		put_unaligned_be16(lun->lun_rtpi, &buf[off]);
+		put_unaligned_be16(lun->lun_tpg->tpg_rtpi, &buf[off]);
 		off += 2;
 		len += 8; /* Header size + Designation descriptor */
 		/*
drivers/target/target_core_stat.c +3 -3
index 62d15bcc3d93..8fd6efa2b978 100644
@@ -455,7 +455,7 @@ static ssize_t target_stat_port_indx_show(struct config_item *item, char *page)
 	rcu_read_lock();
 	dev = rcu_dereference(lun->lun_se_dev);
 	if (dev)
-		ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_rtpi);
+		ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_tpg->tpg_rtpi);
 	rcu_read_unlock();
 	return ret;
 }
@@ -561,7 +561,7 @@ static ssize_t target_stat_tgt_port_indx_show(struct config_item *item,
 	rcu_read_lock();
 	dev = rcu_dereference(lun->lun_se_dev);
 	if (dev)
-		ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_rtpi);
+		ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_tpg->tpg_rtpi);
 	rcu_read_unlock();
 	return ret;
 }
@@ -579,7 +579,7 @@ static ssize_t target_stat_tgt_port_name_show(struct config_item *item,
 	if (dev)
 		ret = snprintf(page, PAGE_SIZE, "%sPort#%u\n",
 			tpg->se_tpg_tfo->fabric_name,
-			lun->lun_rtpi);
+			lun->lun_tpg->tpg_rtpi);
 	rcu_read_unlock();
 	return ret;
 }
Signature · 1 line
2.25.1
#6 lore raw

RFC PATCH 6/48 scsi: target/core: Drop device-based RTPI

In reply to Dmitry Bogdanov

From: Roman Bolshakov <[email protected]>

The code is not needed since target port-based RTPI allocation replaced it.

---
 drivers/target/target_core_device.c   | 41 ---------------------------
 drivers/target/target_core_internal.h |  1 -
 drivers/target/target_core_tpg.c      |  6 ----
 include/target/target_core_base.h     |  4 ---
 4 files changed, 52 deletions(-)
drivers/target/target_core_device.c +0 -41
index bbef6b869ee9..40a582b084d7 100644
@@ -470,47 +470,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 	mutex_unlock(&tpg->acl_node_mutex);
 }
 
-int core_alloc_rtpi(struct se_lun *lun, struct se_device *dev)
-{
-	struct se_lun *tmp;
-
-	spin_lock(&dev->se_port_lock);
-	if (dev->export_count == 0x0000ffff) {
-		pr_warn("Reached dev->dev_port_count =="
-				" 0x0000ffff\n");
-		spin_unlock(&dev->se_port_lock);
-		return -ENOSPC;
-	}
-again:
-	/*
-	 * Allocate the next RELATIVE TARGET PORT IDENTIFIER for this struct se_device
-	 * Here is the table from spc4r17 section 7.7.3.8.
-	 *
-	 *    Table 473 -- RELATIVE TARGET PORT IDENTIFIER field
-	 *
-	 * Code      Description
-	 * 0h        Reserved
-	 * 1h        Relative port 1, historically known as port A
-	 * 2h        Relative port 2, historically known as port B
-	 * 3h to FFFFh    Relative port 3 through 65 535
-	 */
-	lun->lun_rtpi = dev->dev_rpti_counter++;
-	if (!lun->lun_rtpi)
-		goto again;
-
-	list_for_each_entry(tmp, &dev->dev_sep_list, lun_dev_link) {
-		/*
-		 * Make sure RELATIVE TARGET PORT IDENTIFIER is unique
-		 * for 16-bit wrap..
-		 */
-		if (lun->lun_rtpi == tmp->lun_rtpi)
-			goto again;
-	}
-	spin_unlock(&dev->se_port_lock);
-
-	return 0;
-}
-
 static void se_release_vpd_for_dev(struct se_device *dev)
 {
 	struct t10_vpd *vpd, *vpd_tmp;
drivers/target/target_core_internal.h +0 -1
index a889a6237d9c..7dcc62749edd 100644
@@ -59,7 +59,6 @@ struct target_fabric_configfs {
 extern struct t10_alua_lu_gp *default_lu_gp;
 
 /* target_core_device.c */
-int	core_alloc_rtpi(struct se_lun *lun, struct se_device *dev);
 struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
 void	target_pr_kref_release(struct kref *);
 void	core_free_device_list_for_node(struct se_node_acl *,
drivers/target/target_core_tpg.c +0 -6
index 1b7ab98bbd04..502bd85bf75c 100644
@@ -648,10 +648,6 @@ int core_tpg_add_lun(
 	if (ret < 0)
 		goto out;
 
-	ret = core_alloc_rtpi(lun, dev);
-	if (ret)
-		goto out_kill_ref;
-
 	if (!(dev->transport_flags & TRANSPORT_FLAG_PASSTHROUGH_ALUA) &&
 	    !(dev->se_hba->hba_flags & HBA_FLAGS_INTERNAL_USE))
 		target_attach_tg_pt_gp(lun, dev->t10_alua.default_tg_pt_gp);
@@ -675,8 +671,6 @@ int core_tpg_add_lun(
 
 	return 0;
 
-out_kill_ref:
-	percpu_ref_exit(&lun->lun_ref);
 out:
 	return ret;
 }
include/target/target_core_base.h +0 -4
index 72fe94afc7a5..6ed484683ab6 100644
@@ -734,8 +734,6 @@ struct se_lun {
 	bool			lun_access_ro;
 	u32			lun_index;
 
-	/* RELATIVE TARGET PORT IDENTIFER */
-	u16			lun_rtpi;
 	atomic_t		lun_acl_count;
 	struct se_device __rcu	*lun_se_dev;
 
@@ -787,8 +785,6 @@ struct se_device_queue {
 };
 
 struct se_device {
-	/* RELATIVE TARGET PORT IDENTIFER Counter */
-	u16			dev_rpti_counter;
 	/* Used for SAM Task Attribute ordering */
 	u32			dev_cur_ordered_id;
 	u32			dev_flags;
Signature · 1 line
2.25.1
#7 lore raw

RFC PATCH 7/48 scsi: target/core: Add common port attributes

In reply to Dmitry Bogdanov

From: Roman Bolshakov <[email protected]>

All SCSI target port attributes (tpgN/attribs/attrname) are defined and implemented in fabric modules in existing implementation.

The change introduces a way to have common tpg attribs in configfs for all fabrics without code duplication.

---
 drivers/target/target_core_configfs.c        |  9 ++++-
 drivers/target/target_core_fabric_configfs.c | 39 +++++++++++++++++++-
 drivers/target/target_core_internal.h        |  1 +
 drivers/target/target_core_tpg.c             |  4 ++
 4 files changed, 50 insertions(+), 3 deletions(-)
drivers/target/target_core_configfs.c +8 -1
index 7c0e42e782de..40b9808738d2 100644
@@ -464,13 +464,19 @@ int target_register_template(const struct target_core_fabric_ops *fo)
 	INIT_LIST_HEAD(&tf->tf_list);
 	atomic_set(&tf->tf_access_cnt, 0);
 	tf->tf_ops = fo;
-	target_fabric_setup_cits(tf);
+	ret = target_fabric_setup_cits(tf);
+	if (ret)
+		goto out;
 
 	mutex_lock(&g_tf_lock);
 	list_add_tail(&tf->tf_list, &g_tf_list);
 	mutex_unlock(&g_tf_lock);
 
 	return 0;
+
+out:
+	kfree(tf);
+	return ret;
 }
 EXPORT_SYMBOL(target_register_template);
 
@@ -491,6 +497,7 @@ void target_unregister_template(const struct target_core_fabric_ops *fo)
 			 */
 			rcu_barrier();
 			kfree(t->tf_tpg_base_cit.ct_attrs);
+			kfree(t->tf_tpg_attrib_cit.ct_attrs);
 			kfree(t);
 			return;
 		}
drivers/target/target_core_fabric_configfs.c +37 -2
index 95a88f6224cd..a34b5db4eec5 100644
@@ -795,7 +795,38 @@ TF_CIT_SETUP(tpg_lun, NULL, &target_fabric_lun_group_ops, NULL);
 
 /* End of tfc_tpg_lun_cit */
 
-TF_CIT_SETUP_DRV(tpg_attrib, NULL, NULL);
+static int
+target_fabric_setup_tpg_attrib_cit(struct target_fabric_configfs *tf)
+{
+	int i, k, len = 0;
+	struct config_item_type *cit = &tf->tf_tpg_attrib_cit;
+	struct configfs_attribute **attrs;
+
+	for (i = 0; core_tpg_attrib_attrs[i]; i++)
+		len += sizeof(struct configfs_attribute *);
+	if (tf->tf_ops->tfc_tpg_attrib_attrs)
+		for (i = 0; tf->tf_ops->tfc_tpg_attrib_attrs[i]; i++)
+			len += sizeof(struct configfs_attribute *);
+	len += sizeof(struct configfs_attribute *);
+
+	attrs = kzalloc(len, GFP_KERNEL);
+	if (!attrs)
+		return -ENOMEM;
+
+	for (i = 0; core_tpg_attrib_attrs[i]; i++)
+		attrs[i] = core_tpg_attrib_attrs[i];
+	if (tf->tf_ops->tfc_tpg_attrib_attrs)
+		for (k = 0; tf->tf_ops->tfc_tpg_attrib_attrs[k]; k++, i++)
+			attrs[i] = tf->tf_ops->tfc_tpg_attrib_attrs[k];
+	attrs[i] = NULL;
+
+	cit->ct_attrs = attrs;
+	cit->ct_owner = tf->tf_ops->module;
+	pr_debug("Setup generic tpg_attrib\n");
+
+	return 0;
+}
+
 TF_CIT_SETUP_DRV(tpg_auth, NULL, NULL);
 TF_CIT_SETUP_DRV(tpg_param, NULL, NULL);
 
@@ -1113,7 +1144,11 @@ int target_fabric_setup_cits(struct target_fabric_configfs *tf)
 	target_fabric_setup_tpg_lun_cit(tf);
 	target_fabric_setup_tpg_np_cit(tf);
 	target_fabric_setup_tpg_np_base_cit(tf);
-	target_fabric_setup_tpg_attrib_cit(tf);
+
+	ret = target_fabric_setup_tpg_attrib_cit(tf);
+	if (ret)
+		return ret;
+
 	target_fabric_setup_tpg_auth_cit(tf);
 	target_fabric_setup_tpg_param_cit(tf);
 	target_fabric_setup_tpg_nacl_cit(tf);
drivers/target/target_core_internal.h +1 -0
index 7dcc62749edd..d662cdc9a04c 100644
@@ -117,6 +117,7 @@ int	core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
 
 /* target_core_tpg.c */
 extern struct se_device *g_lun0_dev;
+extern struct configfs_attribute *core_tpg_attrib_attrs[];
 
 struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tpg,
 		const char *);
drivers/target/target_core_tpg.c +4 -0
index 502bd85bf75c..61fad44cf205 100644
@@ -714,3 +714,7 @@ void core_tpg_remove_lun(
 
 	percpu_ref_exit(&lun->lun_ref);
 }
+
+struct configfs_attribute *core_tpg_attrib_attrs[] = {
+	NULL,
+};
Signature · 1 line
2.25.1
#8 lore raw

RFC PATCH 8/48 scsi: target/core: Add RTPI attribute for target port

In reply to Dmitry Bogdanov

From: Roman Bolshakov <[email protected]>

RELATIVE TARGET PORT IDENTIFIER can be read and configured via configfs:

$ echo 0x10 > $TARGET/tpgt_N/attrib/rtpi

RTPI has to be unique among all target ports, otherwise the write fails.

Configuration of a new RTPI value triggers INQUIRY DATA HAS CHANGED unit attention as defined in SPC-4 "6.6.1 INQUIRY command introduction":

Quoted 4 lines

If INQUIRY data changes for any reason, the device server shall establish a unit attention condition for the initiator port associated with every I_T nexus (see SAM-5), with the additional sense code set to INQUIRY DATA HAS CHANGED.

---
 drivers/target/target_core_tpg.c | 70 ++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
drivers/target/target_core_tpg.c +70 -0
index 61fad44cf205..a409cf1c4ca0 100644
@@ -715,6 +715,76 @@ void core_tpg_remove_lun(
 	percpu_ref_exit(&lun->lun_ref);
 }
 
+/* Set a Unit Attention on all I_T nexuses related to the port */
+static void core_tpg_ua(struct se_portal_group *se_tpg, u8 asc, u8 ascq)
+{
+	struct se_lun *lun;
+	struct se_dev_entry *se_deve;
+
+	mutex_lock(&se_tpg->tpg_lun_mutex);
+	hlist_for_each_entry_rcu(lun, &se_tpg->tpg_lun_hlist, link) {
+		spin_lock(&lun->lun_deve_lock);
+		list_for_each_entry(se_deve, &lun->lun_deve_list, lun_link)
+			core_scsi3_ua_allocate(se_deve, asc, ascq);
+		spin_unlock(&lun->lun_deve_lock);
+	}
+	mutex_unlock(&se_tpg->tpg_lun_mutex);
+}
+
+static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)
+{
+	struct se_portal_group *se_tpg = attrib_to_tpg(item);
+
+	return sprintf(page, "%#x\n", se_tpg->tpg_rtpi);
+}
+
+static ssize_t core_tpg_rtpi_store(struct config_item *item,
+				   const char *page, size_t count)
+{
+	struct se_portal_group *se_tpg = attrib_to_tpg(item);
+	struct se_portal_group *tpg;
+	bool rtpi_changed = false;
+	u16 val;
+	int ret;
+
+	ret = kstrtou16(page, 0, &val);
+	if (ret < 0)
+		return ret;
+	if (val == 0)
+		return -EINVAL;
+
+	/* RTPI shouldn't conflict with values of existing ports */
+	spin_lock(&g_tpg_lock);
+
+	list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
+		if (se_tpg != tpg && val == tpg->tpg_rtpi) {
+			spin_unlock(&g_tpg_lock);
+			pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
+			       se_tpg->se_tpg_tfo->fabric_name,
+			       se_tpg->se_tpg_tfo->tpg_get_tag(tpg),
+			       val,
+			       tpg->se_tpg_tfo->fabric_name,
+			       tpg->se_tpg_tfo->tpg_get_tag(tpg));
+			return -EINVAL;
+		}
+	}
+
+	if (se_tpg->tpg_rtpi != val) {
+		se_tpg->tpg_rtpi = val;
+		rtpi_changed = true;
+	}
+	spin_unlock(&g_tpg_lock);
+
+	if (rtpi_changed)
+		core_tpg_ua(se_tpg, 0x3f, ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED);
+	ret = count;
+
+	return ret;
+}
+
+CONFIGFS_ATTR(core_tpg_, rtpi);
+
 struct configfs_attribute *core_tpg_attrib_attrs[] = {
+	&core_tpg_attr_rtpi,
 	NULL,
 };
Signature · 1 line
2.25.1
#9 lore raw

RFC PATCH 9/48 target: core: check RTPI uniquity for enabled TPG

In reply to Dmitry Bogdanov

Garantee uniquity of RTPI only for enabled target port groups. Allow any RPTI for disabled tpg until it is enabled.

---
 drivers/target/target_core_fabric_configfs.c | 29 +++++++++++++-
 drivers/target/target_core_internal.h        |  2 +
 drivers/target/target_core_tpg.c             | 40 +++++++++++++-------
 3 files changed, 56 insertions(+), 15 deletions(-)
drivers/target/target_core_fabric_configfs.c +28 -1
index a34b5db4eec5..fc1b8f54fb54 100644
@@ -857,6 +857,7 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,
 						   size_t count)
 {
 	struct se_portal_group *se_tpg = to_tpg(item);
+	struct se_portal_group *tpg;
 	int ret;
 	bool op;
 
@@ -867,11 +868,37 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,
 	if (se_tpg->enabled == op)
 		return count;
 
+	spin_lock(&g_tpg_lock);
+
+	if (op) {
+		tpg = core_get_tpg_by_rtpi(se_tpg->tpg_rtpi);
+		if (tpg) {
+			spin_unlock(&g_tpg_lock);
+
+			pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
+			       se_tpg->se_tpg_tfo->fabric_name,
+			       se_tpg->se_tpg_tfo->tpg_get_tag(tpg),
+			       se_tpg->tpg_rtpi,
+			       tpg->se_tpg_tfo->fabric_name,
+			       tpg->se_tpg_tfo->tpg_get_tag(tpg));
+			return -EINVAL;
+		}
+	}
+
+	se_tpg->enabled |= 0x10; /* transient state */
+	spin_unlock(&g_tpg_lock);
+
 	ret = se_tpg->se_tpg_tfo->fabric_enable_tpg(se_tpg, op);
-	if (ret)
+
+	spin_lock(&g_tpg_lock);
+	if (ret < 0) {
+		se_tpg->enabled	&= ~0x10; /* clear transient state */
+		spin_unlock(&g_tpg_lock);
 		return ret;
+	}
 
 	se_tpg->enabled = op;
+	spin_unlock(&g_tpg_lock);
 
 	return count;
 }
drivers/target/target_core_internal.h +2 -0
index d662cdc9a04c..d4ace697edb0 100644
@@ -116,6 +116,7 @@ int	core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
 		struct list_head *, struct se_cmd *);
 
 /* target_core_tpg.c */
+extern struct spinlock g_tpg_lock;
 extern struct se_device *g_lun0_dev;
 extern struct configfs_attribute *core_tpg_attrib_attrs[];
 
@@ -131,6 +132,7 @@ void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
 struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *tpg,
 		const char *initiatorname);
 void core_tpg_del_initiator_node_acl(struct se_node_acl *acl);
+struct se_portal_group *core_get_tpg_by_rtpi(u16 rtpi);
 
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;
drivers/target/target_core_tpg.c +26 -14
index a409cf1c4ca0..5a81d592530f 100644
@@ -34,7 +34,7 @@ extern struct se_device *g_lun0_dev;
 static u16 g_tpg_count;
 static u16 g_tpg_rtpi_counter = 1;
 static LIST_HEAD(g_tpg_list);
-static DEFINE_SPINLOCK(g_tpg_lock);
+DEFINE_SPINLOCK(g_tpg_lock);
 
 /*	__core_tpg_get_initiator_node_acl():
  *
@@ -476,7 +476,7 @@ static int core_tpg_register_rtpi(struct se_portal_group *se_tpg)
 		 * Make sure RELATIVE TARGET PORT IDENTIFIER is unique
 		 * for 16-bit wrap..
 		 */
-		if (se_tpg->tpg_rtpi == tpg->tpg_rtpi)
+		if (tpg->enabled && se_tpg->tpg_rtpi == tpg->tpg_rtpi)
 			goto again;
 	}
 	list_add(&se_tpg->tpg_list, &g_tpg_list);
@@ -738,12 +738,26 @@ static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)
 	return sprintf(page, "%#x\n", se_tpg->tpg_rtpi);
 }
 
+struct se_portal_group *
+core_get_tpg_by_rtpi(u16 rtpi)
+{
+	struct se_portal_group *tpg;
+
+	lockdep_assert_held(&g_tpg_lock);
+
+	list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
+		if (tpg->enabled && rtpi == tpg->tpg_rtpi)
+			return tpg;
+	}
+
+	return NULL;
+}
+
 static ssize_t core_tpg_rtpi_store(struct config_item *item,
 				   const char *page, size_t count)
 {
 	struct se_portal_group *se_tpg = attrib_to_tpg(item);
 	struct se_portal_group *tpg;
-	bool rtpi_changed = false;
 	u16 val;
 	int ret;
 
@@ -753,11 +767,14 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,
 	if (val == 0)
 		return -EINVAL;
 
-	/* RTPI shouldn't conflict with values of existing ports */
+	if (se_tpg->tpg_rtpi == val)
+		return count;
+
 	spin_lock(&g_tpg_lock);
 
-	list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
-		if (se_tpg != tpg && val == tpg->tpg_rtpi) {
+	if (se_tpg->enabled) {
+		tpg = core_get_tpg_by_rtpi(val);
+		if (tpg) {
 			spin_unlock(&g_tpg_lock);
 			pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
 			       se_tpg->se_tpg_tfo->fabric_name,
@@ -769,17 +786,12 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,
 		}
 	}
 
-	if (se_tpg->tpg_rtpi != val) {
-		se_tpg->tpg_rtpi = val;
-		rtpi_changed = true;
-	}
+	se_tpg->tpg_rtpi = val;
 	spin_unlock(&g_tpg_lock);
 
-	if (rtpi_changed)
-		core_tpg_ua(se_tpg, 0x3f, ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED);
-	ret = count;
+	core_tpg_ua(se_tpg, 0x3f, ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED);
 
-	return ret;
+	return count;
 }
 
 CONFIGFS_ATTR(core_tpg_, rtpi);
Signature · 1 line
2.25.1
#10 lore raw

RFC PATCH 10/48 scsi: target/core: Unlock PR generation bump

In reply to Dmitry Bogdanov

From: Konstantin Shelekhin <[email protected]>

Get rid of spinlock and bump PR generation atomically. This is required to prevent deadlock later on.

---
 drivers/target/target_core_configfs.c |  3 ++-
 drivers/target/target_core_pr.c       | 25 ++++++-------------------
 include/target/target_core_base.h     |  2 +-
 3 files changed, 9 insertions(+), 21 deletions(-)
drivers/target/target_core_configfs.c +2 -1
index 40b9808738d2..610de251994e 100644
@@ -1814,7 +1814,8 @@ static ssize_t target_pr_res_pr_all_tgt_pts_show(struct config_item *item,
 static ssize_t target_pr_res_pr_generation_show(struct config_item *item,
 		char *page)
 {
-	return sprintf(page, "0x%08x\n", pr_to_dev(item)->t10_pr.pr_generation);
+	return sprintf(page, "0x%08x\n",
+			atomic_read(&pr_to_dev(item)->t10_pr.pr_generation));
 }
 
 
drivers/target/target_core_pr.c +6 -19
index 668670b1414a..8f3d920d2590 100644
@@ -594,8 +594,6 @@ target_scsi3_pr_reservation_check(struct se_cmd *cmd)
 
 static u32 core_scsi3_pr_generation(struct se_device *dev)
 {
-	u32 prg;
-
 	/*
 	 * PRGeneration field shall contain the value of a 32-bit wrapping
 	 * counter mainted by the device server.
@@ -605,11 +603,7 @@ static u32 core_scsi3_pr_generation(struct se_device *dev)
 	 *
 	 * See spc4r17 section 6.3.12 READ_KEYS service action
 	 */
-	spin_lock(&dev->dev_reservation_lock);
-	prg = dev->t10_pr.pr_generation++;
-	spin_unlock(&dev->dev_reservation_lock);
-
-	return prg;
+	return atomic_inc_return(&dev->t10_pr.pr_generation);
 }
 
 static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
@@ -1057,15 +1051,8 @@ static void __core_scsi3_add_registration(
 	/*
 	 * Increment PRgeneration counter for struct se_device upon a successful
 	 * REGISTER, see spc4r17 section 6.3.2 READ_KEYS service action
-	 *
-	 * Also, when register_move = 1 for PROUT REGISTER_AND_MOVE service
-	 * action, the struct se_device->dev_reservation_lock will already be held,
-	 * so we do not call core_scsi3_pr_generation() which grabs the lock
-	 * for the REGISTER.
 	 */
-	pr_reg->pr_res_generation = (register_move) ?
-			dev->t10_pr.pr_generation++ :
-			core_scsi3_pr_generation(dev);
+	pr_reg->pr_res_generation = core_scsi3_pr_generation(dev);
 
 	spin_lock(&pr_tmpl->registration_lock);
 	list_add_tail(&pr_reg->pr_reg_list, &pr_tmpl->registration_list);
@@ -3481,7 +3468,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 	 * Increment PRGeneration for existing registrations..
 	 */
 	if (!new_reg)
-		dest_pr_reg->pr_res_generation = pr_tmpl->pr_generation++;
+		dest_pr_reg->pr_res_generation = core_scsi3_pr_generation(dev);
 	spin_unlock(&dev->dev_reservation_lock);
 
 	pr_debug("SPC-3 PR [%s] Service Action: REGISTER_AND_MOVE"
@@ -3703,7 +3690,7 @@ core_scsi3_pri_read_keys(struct se_cmd *cmd)
 	if (!buf)
 		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 
-	put_unaligned_be32(dev->t10_pr.pr_generation, buf);
+	put_unaligned_be32(atomic_read(&dev->t10_pr.pr_generation), buf);
 
 	spin_lock(&dev->t10_pr.registration_lock);
 	list_for_each_entry(pr_reg, &dev->t10_pr.registration_list,
@@ -3758,7 +3745,7 @@ core_scsi3_pri_read_reservation(struct se_cmd *cmd)
 	if (!buf)
 		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 
-	put_unaligned_be32(dev->t10_pr.pr_generation, &buf[0]);
+	put_unaligned_be32(atomic_read(&dev->t10_pr.pr_generation), &buf[0]);
 
 	spin_lock(&dev->dev_reservation_lock);
 	pr_reg = dev->dev_pr_res_holder;
@@ -3900,7 +3887,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 	if (!buf)
 		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 
-	put_unaligned_be32(dev->t10_pr.pr_generation, &buf[0]);
+	put_unaligned_be32(atomic_read(&dev->t10_pr.pr_generation), &buf[0]);
 
 	spin_lock(&dev->dev_reservation_lock);
 	if (dev->dev_pr_res_holder) {
include/target/target_core_base.h +1 -1
index 6ed484683ab6..7e13db647faa 100644
@@ -385,7 +385,7 @@ struct t10_reservation {
 	 * for SCSI device */
 	int pr_aptpl_active;
 #define PR_APTPL_BUF_LEN			262144
-	u32 pr_generation;
+	atomic_t pr_generation;
 	spinlock_t registration_lock;
 	spinlock_t aptpl_reg_lock;
 	/*
Signature · 1 line
2.25.1
#11 lore raw

RFC PATCH 11/48 target: core: fix preempt and abort for allreg res

In reply to Dmitry Bogdanov

Match a key only if SARK is not zero according to SPC-4 and the comment above the code:
If an all registrants persistent reservation is present and the SERVICE ACTION RESERVATION KEY field is set to zero, then all registrations shall be removed except for that of the I_T nexus that is being used for the PERSISTENT RESERVE OUT command;

---
 drivers/target/target_core_pr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/target/target_core_pr.c +1 -1
index 8f3d920d2590..c278ff721da3 100644
@@ -3020,7 +3020,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		if (calling_it_nexus)
 			continue;
 
-		if (pr_reg->pr_res_key != sa_res_key)
+		if ((sa_res_key) && (pr_reg->pr_res_key != sa_res_key))
 			continue;
 
 		pr_reg_nacl = pr_reg->pr_reg_nacl;
Signature · 1 line
2.25.1
#12 lore raw

RFC PATCH 12/48 target: core: fix memory leak in preempt_and_abort

In reply to Dmitry Bogdanov

Always release preempt_and_abort_list to avoid memory leak of t10_pr_registration objects in it.

---
 drivers/target/target_core_pr.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/target/target_core_pr.c +5 -4
index c278ff721da3..0b3a22f51c5e 100644
@@ -2954,11 +2954,12 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 			__core_scsi3_complete_pro_preempt(dev, pr_reg_n,
 				(preempt_type == PREEMPT_AND_ABORT) ? &preempt_and_abort_list : NULL,
 				type, scope, preempt_type);
-
-			if (preempt_type == PREEMPT_AND_ABORT)
-				core_scsi3_release_preempt_and_abort(
-					&preempt_and_abort_list, pr_reg_n);
 		}
+
+		if (preempt_type == PREEMPT_AND_ABORT)
+			core_scsi3_release_preempt_and_abort(
+				&preempt_and_abort_list, pr_reg_n);
+
 		spin_unlock(&dev->dev_reservation_lock);
 
 		if (pr_tmpl->pr_aptpl_active)
Signature · 1 line
2.25.1
#13 lore raw

RFC PATCH 13/48 target: core: abort all preempted regs if requested

In reply to Dmitry Bogdanov

According to SPC the aborting shall always happened.

SPC-4: 5.12.11.2.6 Preempting and aborting
If the SERVICE ACTION RESERVATION KEY field is set to zero and an all registrants persistent reservation is present, the device server shall abort all commands for all registered I_T nexuses.

---
 drivers/target/target_core_pr.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/target/target_core_pr.c +15 -1
index 0b3a22f51c5e..b943f8f379cf 100644
@@ -2956,9 +2956,23 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 				type, scope, preempt_type);
 		}
 
-		if (preempt_type == PREEMPT_AND_ABORT)
+		/*
+		 * SPC-4 5.12.11.2.6 Preempting and aborting
+		 * The actions described in this subclause shall be performed
+		 * for all I_T nexuses that are registered with the non-zero
+		 * SERVICE ACTION RESERVATION KEY value, without regard for
+		 * whether the preempted I_T nexuses hold the persistent
+		 * reservation. If the SERVICE ACTION RESERVATION KEY field is
+		 * set to zero and an all registrants persistent reservation is
+		 * present, the device server shall abort all commands for all
+		 * registered I_T nexuses.
+		 */
+		if (preempt_type == PREEMPT_AND_ABORT) {
+			core_tmr_lun_reset(dev, NULL, &preempt_and_abort_list,
+					   cmd);
 			core_scsi3_release_preempt_and_abort(
 				&preempt_and_abort_list, pr_reg_n);
+		}
 
 		spin_unlock(&dev->dev_reservation_lock);
 
Signature · 1 line
2.25.1
#14 lore raw

RFC PATCH 14/48 target: core: new key must be used for moved PR

In reply to Dmitry Bogdanov

According to SPC4 5.12.8:
e) Retain the reservation key specified in the SERVICE ACTION RESERVATION KEY field and associated information;

But currently sa_res_key is only used for the not existing I_T nexus. The patch adds a changing of the key for the existing I_T nexus the PR moved to.

---
 drivers/target/target_core_pr.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/target/target_core_pr.c +6 -2
index b943f8f379cf..844b971827e6 100644
@@ -3438,8 +3438,6 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 	 *       transport protocols where port names are not required;
 	 * d) Register the reservation key specified in the SERVICE ACTION
 	 *    RESERVATION KEY field;
-	 * e) Retain the reservation key specified in the SERVICE ACTION
-	 *    RESERVATION KEY field and associated information;
 	 *
 	 * Also, It is not an error for a REGISTER AND MOVE service action to
 	 * register an I_T nexus that is already registered with the same
@@ -3462,6 +3460,12 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 		dest_pr_reg = __core_scsi3_locate_pr_reg(dev, dest_node_acl,
 						iport_ptr);
 		new_reg = 1;
+	} else {
+	/*
+	 * e) Retain the reservation key specified in the SERVICE ACTION
+	 *    RESERVATION KEY field and associated information;
+	 */
+		dest_pr_reg->pr_res_key = sa_res_key;
 	}
 	/*
 	 * f) Release the persistent reservation for the persistent reservation
Signature · 1 line
2.25.1
#15 lore raw

RFC PATCH 15/48 target: core: remove unused variable in se_dev_entry

In reply to Dmitry Bogdanov

deve->pr_res_key is never set, and se_cmd->pr_res_key is set in the checks inside __target_execute_cmd

---
 drivers/target/target_core_device.c | 2 --
 include/target/target_core_base.h   | 4 ----
 2 files changed, 6 deletions(-)
drivers/target/target_core_device.c +0 -2
index 40a582b084d7..c0e4fb7b95d6 100644
@@ -83,7 +83,6 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd)
 		}
 
 		se_cmd->se_lun = se_lun;
-		se_cmd->pr_res_key = deve->pr_res_key;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 		se_cmd->lun_ref_active = true;
 	}
@@ -160,7 +159,6 @@ int transport_lookup_tmr_lun(struct se_cmd *se_cmd)
 		}
 
 		se_cmd->se_lun = se_lun;
-		se_cmd->pr_res_key = deve->pr_res_key;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 		se_cmd->lun_ref_active = true;
 	}
include/target/target_core_base.h +0 -4
index 7e13db647faa..91c068525c02 100644
@@ -379,8 +379,6 @@ struct t10_pr_registration {
 };
 
 struct t10_reservation {
-	/* Reservation effects all target ports */
-	int pr_all_tg_pt;
 	/* Activate Persistence across Target Power Loss enabled
 	 * for SCSI device */
 	int pr_aptpl_active;
@@ -398,7 +396,6 @@ struct t10_reservation {
 	 * a single *pr_res_holder of the reservation, but all
 	 * registrations are considered reservation holders.
 	 */
-	struct se_node_acl *pr_res_holder;
 	struct list_head registration_list;
 	struct list_head aptpl_reg_list;
 };
@@ -656,7 +653,6 @@ struct se_lun_acl {
 
 struct se_dev_entry {
 	u64			mapped_lun;
-	u64			pr_res_key;
 	u64			creation_time;
 	bool			lun_access_ro;
 	u32			attach_count;
Signature · 1 line
2.25.1
#16 lore raw

RFC PATCH 16/48 target: core: undepend PR registrant of nacl

In reply to Dmitry Bogdanov

Make registration info be independent of nacl. Store and use an information about initiatror name and target port inside registration object.

---
 drivers/target/target_core_configfs.c   |  14 +-
 drivers/target/target_core_device.c     |   7 -
 drivers/target/target_core_fabric_lib.c |  50 ++++----
 drivers/target/target_core_internal.h   |   9 +-
 drivers/target/target_core_pr.c         | 164 ++++++++----------------
 drivers/target/target_core_pr.h         |   2 -
 include/target/target_core_base.h       |   1 +
 7 files changed, 88 insertions(+), 159 deletions(-)
drivers/target/target_core_configfs.c +5 -9
index 610de251994e..6e654edc4399 100644
@@ -1735,7 +1735,6 @@ static struct se_device *pr_to_dev(struct config_item *item)
 static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
 		char *page)
 {
-	struct se_node_acl *se_nacl;
 	struct t10_pr_registration *pr_reg;
 	char i_buf[PR_REG_ISID_ID_LEN] = { };
 
@@ -1743,12 +1742,11 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
 	if (!pr_reg)
 		return sprintf(page, "No SPC-3 Reservation holder\n");
 
-	se_nacl = pr_reg->pr_reg_nacl;
 	core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
 
 	return sprintf(page, "SPC-3 Reservation: %s Initiator: %s%s\n",
-		se_nacl->se_tpg->se_tpg_tfo->fabric_name,
-		se_nacl->initiatorname, i_buf);
+		pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+		pr_reg->pr_iport, i_buf);
 }
 
 static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
@@ -1823,7 +1821,6 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item,
 		char *page)
 {
 	struct se_device *dev = pr_to_dev(item);
-	struct se_node_acl *se_nacl;
 	struct se_portal_group *se_tpg;
 	struct t10_pr_registration *pr_reg;
 	const struct target_core_fabric_ops *tfo;
@@ -1836,8 +1833,7 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item,
 		goto out_unlock;
 	}
 
-	se_nacl = pr_reg->pr_reg_nacl;
-	se_tpg = se_nacl->se_tpg;
+	se_tpg = pr_reg->se_tpg;
 	tfo = se_tpg->se_tpg_tfo;
 
 	len += sprintf(page+len, "SPC-3 Reservation: %s"
@@ -1874,12 +1870,12 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,
 
 		memset(buf, 0, 384);
 		memset(i_buf, 0, PR_REG_ISID_ID_LEN);
-		tfo = pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo;
+		tfo = pr_reg->se_tpg->se_tpg_tfo;
 		core_pr_dump_initiator_port(pr_reg, i_buf,
 					PR_REG_ISID_ID_LEN);
 		sprintf(buf, "%s Node: %s%s Key: 0x%016Lx PRgen: 0x%08x\n",
 			tfo->fabric_name,
-			pr_reg->pr_reg_nacl->initiatorname, i_buf, pr_reg->pr_res_key,
+			pr_reg->pr_iport, i_buf, pr_reg->pr_res_key,
 			pr_reg->pr_res_generation);
 
 		if (len + strlen(buf) >= PAGE_SIZE)
drivers/target/target_core_device.c +0 -7
index c0e4fb7b95d6..a4c31f3112ce 100644
@@ -391,12 +391,6 @@ void core_disable_device_list_for_node(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
-	/*
-	 * rcu_dereference_raw protected by se_lun->lun_group symlink
-	 * reference to se_device->dev_group.
-	 */
-	struct se_device *dev = rcu_dereference_raw(lun->lun_se_dev);
-
 	lockdep_assert_held(&nacl->lun_entry_mutex);
 
 	/*
@@ -437,7 +431,6 @@ void core_disable_device_list_for_node(
 
 	kfree_rcu(orig, rcu_head);
 
-	core_scsi3_free_pr_reg_from_nacl(dev, nacl);
 	target_luns_data_has_changed(nacl, NULL, false);
 }
 
drivers/target/target_core_fabric_lib.c +22 -28

169 lines not on this page · raw message

drivers/target/target_core_internal.h +4 -5
index d4ace697edb0..8bd48eb1cf9b 100644
@@ -97,11 +97,10 @@ void	target_setup_backend_cits(struct target_backend *);
 int	target_fabric_setup_cits(struct target_fabric_configfs *);
 
 /* target_core_fabric_lib.c */
-int	target_get_pr_transport_id_len(struct se_node_acl *nacl,
-		struct t10_pr_registration *pr_reg, int *format_code);
-int	target_get_pr_transport_id(struct se_node_acl *nacl,
-		struct t10_pr_registration *pr_reg, int *format_code,
-		unsigned char *buf);
+int	target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg,
+				       int *format_code);
+int	target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
+				   int *format_code, unsigned char *buf);
 const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
 		char *buf, u32 *out_tid_len, char **port_nexus_ptr);
 
drivers/target/target_core_pr.c +56 -108

458 lines not on this page · raw message

drivers/target/target_core_pr.h +0 -2
index b793c99637ab..202f9aec963a 100644
@@ -68,8 +68,6 @@ extern int core_scsi3_alloc_aptpl_registration(
 extern int core_scsi3_check_aptpl_registration(struct se_device *,
 			struct se_portal_group *, struct se_lun *,
 			struct se_node_acl *, u64);
-extern void core_scsi3_free_pr_reg_from_nacl(struct se_device *,
-					     struct se_node_acl *);
 extern void core_scsi3_free_all_registrations(struct se_device *);
 extern unsigned char *core_scsi3_pr_dump_type(int);
 
include/target/target_core_base.h +1 -0
index 91c068525c02..1b673b25c6cd 100644
@@ -369,6 +369,7 @@ struct t10_pr_registration {
 	u64 pr_res_key;
 	atomic_t pr_res_holders;
 	struct se_node_acl *pr_reg_nacl;
+	struct se_portal_group *se_tpg;
 	/* Used by ALL_TG_PT=1 registration with deve->pr_ref taken */
 	struct se_dev_entry *pr_reg_deve;
 	struct list_head pr_reg_list;
Signature · 1 line
2.25.1
#17 lore raw

RFC PATCH 17/48 target: core: make some functions public

In reply to Dmitry Bogdanov

Make functions of alloc/free and reserve/release reservation be public to use them from cluster code.

---
 drivers/target/target_core_pr.c | 51 +++++++++++++++++++--------------
 drivers/target/target_core_pr.h | 24 ++++++++++++++++
 2 files changed, 54 insertions(+), 21 deletions(-)
drivers/target/target_core_pr.c +30 -21
index aa967ea74e5b..aaa383e1ee58 100644
@@ -64,9 +64,6 @@ enum preempt_type {
 	PREEMPT_AND_ABORT,
 };
 
-static void __core_scsi3_complete_pro_release(struct se_device *,
-					      struct t10_pr_registration *, int, int);
-
 static int is_reservation_holder(
 	struct t10_pr_registration *pr_res_holder,
 	struct t10_pr_registration *pr_reg)
@@ -606,7 +603,7 @@ static u32 core_scsi3_pr_generation(struct se_device *dev)
 	return atomic_inc_return(&dev->t10_pr.pr_generation);
 }
 
-static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
+struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 	struct se_device *dev,
 	struct se_node_acl *nacl,
 	const char *initiatorname,
@@ -1261,7 +1258,7 @@ static int core_scsi3_check_implicit_release(
 	return ret;
 }
 
-static void __core_scsi3_free_registration(
+void __core_scsi3_free_registration(
 	struct se_device *dev,
 	struct t10_pr_registration *pr_reg,
 	struct list_head *preempt_and_abort_list,
@@ -2219,6 +2216,32 @@ unsigned char *core_scsi3_pr_dump_type(int type)
 	return "Unknown SPC-3 PR Type";
 }
 
+void __core_scsi3_set_reservation(struct se_device *dev,
+				  struct t10_pr_registration *pr_reg,
+				  int scope, int type)
+{
+	char i_buf[PR_REG_ISID_ID_LEN];
+
+	memset(i_buf, 0, PR_REG_ISID_ID_LEN);
+
+	lockdep_assert_held(&dev->dev_reservation_lock);
+
+	pr_reg->pr_res_scope = scope;
+	pr_reg->pr_res_type = type;
+	pr_reg->pr_res_holder = 1;
+	dev->dev_pr_res_holder = pr_reg;
+	core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
+
+	pr_debug("SPC-3 PR [%s] Service Action: RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n",
+		pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+		core_scsi3_pr_dump_type(type),
+		(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
+	pr_debug("SPC-3 PR [%s] RESERVE Node: %s%s\n",
+			pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+			pr_reg->pr_iport,
+			i_buf);
+}
+
 static sense_reason_t
 core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
 {
@@ -2227,7 +2250,6 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
 	struct se_lun *se_lun = cmd->se_lun;
 	struct t10_pr_registration *pr_reg, *pr_res_holder;
 	struct t10_reservation *pr_tmpl = &dev->t10_pr;
-	char i_buf[PR_REG_ISID_ID_LEN] = { };
 	sense_reason_t ret;
 
 	if (!se_sess || !se_lun) {
@@ -2346,20 +2368,7 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
 	 * Otherwise, our *pr_reg becomes the PR reservation holder for said
 	 * TYPE/SCOPE.  Also set the received scope and type in *pr_reg.
 	 */
-	pr_reg->pr_res_scope = scope;
-	pr_reg->pr_res_type = type;
-	pr_reg->pr_res_holder = 1;
-	dev->dev_pr_res_holder = pr_reg;
-	core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
-
-	pr_debug("SPC-3 PR [%s] Service Action: RESERVE created new"
-		" reservation holder TYPE: %s ALL_TG_PT: %d\n",
-		cmd->se_tfo->fabric_name, core_scsi3_pr_dump_type(type),
-		(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
-	pr_debug("SPC-3 PR [%s] RESERVE Node: %s%s\n",
-			cmd->se_tfo->fabric_name,
-			se_sess->se_node_acl->initiatorname,
-			i_buf);
+	__core_scsi3_set_reservation(dev, pr_reg, scope, type);
 	spin_unlock(&dev->dev_reservation_lock);
 
 	if (pr_tmpl->pr_aptpl_active)
@@ -2390,7 +2399,7 @@ core_scsi3_emulate_pro_reserve(struct se_cmd *cmd, int type, int scope,
 	}
 }
 
-static void __core_scsi3_complete_pro_release(
+void __core_scsi3_complete_pro_release(
 	struct se_device *dev,
 	struct t10_pr_registration *pr_reg,
 	int explicit,
drivers/target/target_core_pr.h +24 -0
index 202f9aec963a..9c4710f34d94 100644
@@ -75,4 +75,28 @@ extern sense_reason_t target_scsi3_emulate_pr_in(struct se_cmd *);
 extern sense_reason_t target_scsi3_emulate_pr_out(struct se_cmd *);
 extern sense_reason_t target_check_reservation(struct se_cmd *);
 
+void __core_scsi3_set_reservation(struct se_device *dev,
+				  struct t10_pr_registration *pr_reg,
+				  int scope, int type);
+void __core_scsi3_complete_pro_release(struct se_device *dev,
+				       struct t10_pr_registration *pr_reg,
+				       int explicit, int unreg);
+struct t10_pr_registration *__core_scsi3_do_alloc_registration(
+					struct se_device *dev,
+					struct se_node_acl *nacl,
+					const char *initiatorname,
+					u64 unpacked_lun,
+					struct se_portal_group *se_tpg,
+					struct se_dev_entry *dest_deve,
+					u64 mapped_lun,
+					unsigned char *isid,
+					u64 sa_res_key,
+					int all_tg_pt,
+					int aptpl);
+void __core_scsi3_free_registration(struct se_device *dev,
+				    struct t10_pr_registration *pr_reg,
+				    struct list_head *preempt_and_abort_list,
+				    int dec_holders);
+
+
 #endif /* TARGET_CORE_PR_H */
Signature · 1 line
2.25.1
#18 lore raw

RFC PATCH 18/48 target: core: proper clear reservation on LUN RESET

In reply to Dmitry Bogdanov

Use an existing helper to clear SCSI-2 reservation on LUN RESET.

---
 drivers/target/target_core_tmr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/target/target_core_tmr.c +1 -2
index bac111456fa1..51d95e0b8885 100644
@@ -394,8 +394,7 @@ int core_tmr_lun_reset(
 	if (!preempt_and_abort_list &&
 	     (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)) {
 		spin_lock(&dev->dev_reservation_lock);
-		dev->reservation_holder = NULL;
-		dev->dev_reservation_flags &= ~DRF_SPC2_RESERVATIONS;
+		target_release_reservation(dev);
 		spin_unlock(&dev->dev_reservation_lock);
 		pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
 	}
Signature · 1 line
2.25.1
#19 lore raw

RFC PATCH 19/48 target: core: remove superfluous checks

In reply to Dmitry Bogdanov

remove double checks on path of SCSI-2 reservations

---
 drivers/target/target_core_pr.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/target/target_core_pr.c +1 -4
index aaa383e1ee58..4bb16de29a1e 100644
@@ -95,9 +95,6 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
 		break;
 	}
 
-	if (!dev->reservation_holder || !sess)
-		return 0;
-
 	if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
 		return TCM_RESERVATION_CONFLICT;
 
@@ -211,7 +208,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
 		return TCM_RESERVATION_CONFLICT;
 
 	spin_lock(&dev->dev_reservation_lock);
-	if (!dev->reservation_holder || !sess)
+	if (!dev->reservation_holder)
 		goto out_unlock;
 
 	if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
Signature · 1 line
2.25.1
#20 lore raw

RFC PATCH 20/48 target: core: proper check of SCSI-2 reservation

In reply to Dmitry Bogdanov

Use a check of the flag instead of session pointer like in other places.

---
 drivers/target/target_core_configfs.c | 17 +++++++++++------
 drivers/target/target_core_pr.c       |  6 +++---
 2 files changed, 14 insertions(+), 9 deletions(-)
drivers/target/target_core_configfs.c +11 -6
index 6e654edc4399..34901f0b2350 100644
@@ -1756,12 +1756,17 @@ static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
 	struct se_node_acl *se_nacl;
 	ssize_t len;
 
-	if (sess) {
-		se_nacl = sess->se_node_acl;
-		len = sprintf(page,
-			      "SPC-2 Reservation: %s Initiator: %s\n",
-			      se_nacl->se_tpg->se_tpg_tfo->fabric_name,
-			      se_nacl->initiatorname);
+	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
+		if (sess) {
+			se_nacl = sess->se_node_acl;
+			len = sprintf(page,
+				"SPC-2 Reservation: %s Initiator: %s\n",
+				se_nacl->se_tpg->se_tpg_tfo->fabric_name,
+				se_nacl->initiatorname);
+		} else {
+			len = sprintf(page,
+				"SPC-2 Reservation: peer node\n");
+		}
 	} else {
 		len = sprintf(page, "No SPC-2 Reservation holder\n");
 	}
drivers/target/target_core_pr.c +3 -3
index 4bb16de29a1e..2f4c65d5d766 100644
@@ -208,7 +208,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
 		return TCM_RESERVATION_CONFLICT;
 
 	spin_lock(&dev->dev_reservation_lock);
-	if (!dev->reservation_holder)
+	if (!(dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS))
 		goto out_unlock;
 
 	if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
@@ -261,8 +261,8 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd)
 
 	tpg = sess->se_tpg;
 	spin_lock(&dev->dev_reservation_lock);
-	if (dev->reservation_holder &&
-	    dev->reservation_holder->se_node_acl != sess->se_node_acl) {
+	if ((dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) &&
+	     dev->reservation_holder->se_node_acl != sess->se_node_acl) {
 		pr_err("SCSI-2 RESERVATION CONFLICT for %s fabric\n",
 			tpg->se_tpg_tfo->fabric_name);
 		pr_err("Original reserver LUN: %llu %s\n",
Signature · 1 line
2.25.1
#21 lore raw

RFC PATCH 21/48 target: core: checks against peer node SCSI2 reservation

In reply to Dmitry Bogdanov

For the case of SCSI-2 reservation by peer node dev->reservation_holder session is NULL. So use this information as reservation existence.

---
 drivers/target/target_core_pr.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/target/target_core_pr.c +10 -3
index 2f4c65d5d766..c4329f639323 100644
@@ -95,7 +95,8 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
 		break;
 	}
 
-	if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
+	if (!dev->reservation_holder ||
+	    dev->reservation_holder->se_node_acl != sess->se_node_acl)
 		return TCM_RESERVATION_CONFLICT;
 
 	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS_WITH_ISID) {
@@ -211,6 +212,9 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
 	if (!(dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS))
 		goto out_unlock;
 
+	if (!dev->reservation_holder)
+		goto out_unlock;
+
 	if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
 		goto out_unlock;
 
@@ -262,12 +266,15 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd)
 	tpg = sess->se_tpg;
 	spin_lock(&dev->dev_reservation_lock);
 	if ((dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) &&
-	     dev->reservation_holder->se_node_acl != sess->se_node_acl) {
+	    (!dev->reservation_holder ||
+	     dev->reservation_holder->se_node_acl != sess->se_node_acl)) {
 		pr_err("SCSI-2 RESERVATION CONFLICT for %s fabric\n",
 			tpg->se_tpg_tfo->fabric_name);
 		pr_err("Original reserver LUN: %llu %s\n",
 			cmd->se_lun->unpacked_lun,
-			dev->reservation_holder->se_node_acl->initiatorname);
+			(dev->reservation_holder) ?
+			 dev->reservation_holder->se_node_acl->initiatorname :
+			 "peer");
 		pr_err("Current attempt - LUN: %llu -> MAPPED LUN: %llu"
 			" from %s \n", cmd->se_lun->unpacked_lun,
 			cmd->orig_fe_lun,
Signature · 1 line
2.25.1
#22 lore raw

RFC PATCH 22/48 target: core: UA on all luns after reset

In reply to Dmitry Bogdanov

Allocate UA "BUS DEVICE RESET OCCURRED" on all LUNs on the device upon reception of TMF LUN RESET.

That patch passes a libisci test SCSI.MultipathIO.Reset.

---
 drivers/target/target_core_device.c    | 19 +++++++++++++++++++
 drivers/target/target_core_internal.h  |  1 +
 drivers/target/target_core_transport.c |  3 +--
 3 files changed, 21 insertions(+), 2 deletions(-)
drivers/target/target_core_device.c +19 -0
index a4c31f3112ce..7c887792bb88 100644
@@ -285,6 +285,25 @@ void target_pr_kref_release(struct kref *kref)
 	complete(&deve->pr_comp);
 }
 
+/*
+ * Establish UA condition on SCSI device - all LUNs
+ */
+void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq)
+{
+	struct se_dev_entry *se_deve;
+	struct se_lun *lun;
+
+	spin_lock(&dev->se_port_lock);
+	list_for_each_entry(lun, &dev->dev_sep_list, lun_dev_link) {
+
+		spin_lock(&lun->lun_deve_lock);
+		list_for_each_entry(se_deve, &lun->lun_deve_list, lun_link)
+			core_scsi3_ua_allocate(se_deve, asc, ascq);
+		spin_unlock(&lun->lun_deve_lock);
+	}
+	spin_unlock(&dev->se_port_lock);
+}
+
 static void
 target_luns_data_has_changed(struct se_node_acl *nacl, struct se_dev_entry *new,
 			     bool skip_new)
drivers/target/target_core_internal.h +1 -0
index 8bd48eb1cf9b..eeb405908987 100644
@@ -88,6 +88,7 @@ int	target_configure_device(struct se_device *dev);
 void	target_free_device(struct se_device *);
 int	target_for_each_device(int (*fn)(struct se_device *dev, void *data),
 			       void *data);
+void	target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
 
 /* target_core_configfs.c */
 extern struct configfs_item_operations target_core_dev_item_ops;
drivers/target/target_core_transport.c +1 -2
index 7838dc20f713..5926316252eb 100644
@@ -3531,8 +3531,7 @@ static void target_tmr_work(struct work_struct *work)
 		tmr->response = (!ret) ? TMR_FUNCTION_COMPLETE :
 					 TMR_FUNCTION_REJECTED;
 		if (tmr->response == TMR_FUNCTION_COMPLETE) {
-			target_ua_allocate_lun(cmd->se_sess->se_node_acl,
-					       cmd->orig_fe_lun, 0x29,
+			target_dev_ua_allocate(dev, 0x29,
 					       ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
 		}
 		break;
Signature · 1 line
2.25.1
#23 lore raw

RFC PATCH 23/48 target: core: refactor LUN_RESET code

In reply to Dmitry Bogdanov

No function change, just extract a code of aborting all active commands on the device. To make it possible to call TMF LUN_RESET code without cluster related things - SCSI2 reservation or ACA condition.

---
 drivers/target/target_core_internal.h  |  5 +++--
 drivers/target/target_core_pr.c        |  4 ++--
 drivers/target/target_core_tmr.c       | 27 +++++++++++++++++++-------
 drivers/target/target_core_transport.c |  6 +-----
 4 files changed, 26 insertions(+), 16 deletions(-)
drivers/target/target_core_internal.h +3 -2
index eeb405908987..b32ac78a26b0 100644
@@ -112,8 +112,9 @@ int	core_delete_hba(struct se_hba *);
 /* target_core_tmr.c */
 void	core_tmr_abort_task(struct se_device *, struct se_tmr_req *,
 			struct se_session *);
-int	core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
-		struct list_head *, struct se_cmd *);
+int	core_tmr_lun_reset(struct se_device *dev, struct se_tmr_req *tmr);
+void	core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
+		struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
 
 /* target_core_tpg.c */
 extern struct spinlock g_tpg_lock;
drivers/target/target_core_pr.c +2 -2
index c4329f639323..dfecb155bf88 100644
@@ -2936,7 +2936,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		 * registered I_T nexuses.
 		 */
 		if (preempt_type == PREEMPT_AND_ABORT) {
-			core_tmr_lun_reset(dev, NULL, &preempt_and_abort_list,
+			core_local_lun_reset(dev, NULL, &preempt_and_abort_list,
 					   cmd);
 			core_scsi3_release_preempt_and_abort(
 				&preempt_and_abort_list, pr_reg_n);
@@ -3067,7 +3067,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 	 * new persistent reservation holder, the calling Initiator Port.
 	 */
 	if (preempt_type == PREEMPT_AND_ABORT) {
-		core_tmr_lun_reset(dev, NULL, &preempt_and_abort_list, cmd);
+		core_local_lun_reset(dev, NULL, &preempt_and_abort_list, cmd);
 		core_scsi3_release_preempt_and_abort(&preempt_and_abort_list,
 						pr_reg_n);
 	}
drivers/target/target_core_tmr.c +20 -7
index 51d95e0b8885..2db45efedd97 100644
@@ -22,6 +22,7 @@
 #include "target_core_internal.h"
 #include "target_core_alua.h"
 #include "target_core_pr.h"
+#include "target_core_ua.h"
 
 int core_tmr_alloc_req(
 	struct se_cmd *se_cmd,
@@ -342,7 +343,8 @@ static void core_tmr_drain_state_list(
 	}
 }
 
-int core_tmr_lun_reset(
+
+void core_local_lun_reset(
         struct se_device *dev,
         struct se_tmr_req *tmr,
         struct list_head *preempt_and_abort_list,
@@ -352,7 +354,8 @@ int core_tmr_lun_reset(
 	struct se_portal_group *tmr_tpg = NULL;
 	struct se_session *tmr_sess = NULL;
 	bool tas;
-        /*
+
+	/*
 	 * TASK_ABORTED status bit, this is configurable via ConfigFS
 	 * struct se_device attributes.  spc4r17 section 7.4.6 Control mode page
 	 *
@@ -387,23 +390,33 @@ int core_tmr_lun_reset(
 	core_tmr_drain_state_list(dev, prout_cmd, tmr_sess, tas,
 				preempt_and_abort_list);
 
+	pr_debug("LUN_RESET: %s for [%s] Complete\n",
+			(preempt_and_abort_list) ? "Preempt" : "TMR",
+			dev->transport->name);
+}
+
+int core_tmr_lun_reset(
+	struct se_device *dev,
+	struct se_tmr_req *tmr)
+{
+	core_local_lun_reset(dev, tmr, NULL, NULL);
+
 	/*
 	 * Clear any legacy SPC-2 reservation when called during
 	 * LOGICAL UNIT RESET
 	 */
-	if (!preempt_and_abort_list &&
-	     (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)) {
+	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
 		spin_lock(&dev->dev_reservation_lock);
 		target_release_reservation(dev);
 		spin_unlock(&dev->dev_reservation_lock);
 		pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
 	}
 
+	target_dev_ua_allocate(dev, 0x29,
+			ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
+
 	atomic_long_inc(&dev->num_resets);
 
-	pr_debug("LUN_RESET: %s for [%s] Complete\n",
-			(preempt_and_abort_list) ? "Preempt" : "TMR",
-			dev->transport->name);
 	return 0;
 }
 
drivers/target/target_core_transport.c +1 -5
index 5926316252eb..d3cbba294510 100644
@@ -3527,13 +3527,9 @@ static void target_tmr_work(struct work_struct *work)
 		tmr->response = TMR_TASK_MGMT_FUNCTION_NOT_SUPPORTED;
 		break;
 	case TMR_LUN_RESET:
-		ret = core_tmr_lun_reset(dev, tmr, NULL, NULL);
+		ret = core_tmr_lun_reset(dev, tmr);
 		tmr->response = (!ret) ? TMR_FUNCTION_COMPLETE :
 					 TMR_FUNCTION_REJECTED;
-		if (tmr->response == TMR_FUNCTION_COMPLETE) {
-			target_dev_ua_allocate(dev, 0x29,
-					       ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
-		}
 		break;
 	case TMR_TARGET_WARM_RESET:
 		tmr->response = TMR_FUNCTION_REJECTED;
Signature · 1 line
2.25.1
#24 lore raw

RFC PATCH 24/48 target: core: pr: use RTPI in APTPL

In reply to Dmitry Bogdanov

Since RTPI is an unique identifier of target port there is no need to check fabric name and port tag.
RTPI is already saved in APTPL data, so it is backward compatible change.

---
 drivers/target/target_core_configfs.c |  4 ++-
 drivers/target/target_core_pr.c       | 41 +++++++--------------------
 drivers/target/target_core_pr.h       |  2 +-
 include/target/target_core_base.h     |  5 ----
 4 files changed, 15 insertions(+), 37 deletions(-)
drivers/target/target_core_configfs.c +3 -1
index 34901f0b2350..f469cfde3f1e 100644
@@ -1993,6 +1993,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
 	u64 mapped_lun = 0, target_lun = 0;
 	int ret = -1, res_holder = 0, all_tg_pt = 0, arg, token;
 	u16 tpgt = 0;
+	u16 rtpi = 0;
 	u8 type = 0;
 
 	if (!dev->dev_attrib.emulate_pr ||
@@ -2127,6 +2128,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
 			ret = match_int(args, &arg);
 			if (ret)
 				goto out;
+			rtpi = (u16)arg;
 			break;
 		case Opt_target_lun:
 			ret = match_u64(args, &tmp_ll);
@@ -2153,7 +2155,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
 	}
 
 	ret = core_scsi3_alloc_aptpl_registration(&dev->t10_pr, sa_res_key,
-			i_port, isid, mapped_lun, t_port, tpgt, target_lun,
+			i_port, isid, mapped_lun, rtpi, target_lun,
 			res_holder, all_tg_pt, type);
 out:
 	kfree(i_fabric);
drivers/target/target_core_pr.c +11 -30

105 lines not on this page · raw message

drivers/target/target_core_pr.h +1 -1
index 9c4710f34d94..e4bdabfc4b31 100644
@@ -64,7 +64,7 @@ extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);
 extern int core_scsi3_alloc_aptpl_registration(
 			struct t10_reservation *, u64,
 			unsigned char *, unsigned char *, u64,
-			unsigned char *, u16, u64, int, int, u8);
+			u16, u64, int, int, u8);
 extern int core_scsi3_check_aptpl_registration(struct se_device *,
 			struct se_portal_group *, struct se_lun *,
 			struct se_node_acl *, u64);
include/target/target_core_base.h +0 -5
index 1b673b25c6cd..095b96cb3557 100644
@@ -347,11 +347,6 @@ struct t10_pr_registration {
 	/* Used during APTPL metadata reading */
 #define PR_APTPL_MAX_IPORT_LEN			256
 	unsigned char pr_iport[PR_APTPL_MAX_IPORT_LEN];
-	/* Used during APTPL metadata reading */
-#define PR_APTPL_MAX_TPORT_LEN			256
-	unsigned char pr_tport[PR_APTPL_MAX_TPORT_LEN];
-	u16 pr_aptpl_rpti;
-	u16 pr_reg_tpgt;
 	/* Reservation effects all target ports */
 	int pr_reg_all_tg_pt;
 	/* Activate Persistence across Target Power Loss */
Signature · 1 line
2.25.1
#25 lore raw

RFC PATCH 25/48 target: core: pr: have Transport ID stored

In reply to Dmitry Bogdanov

Store TransportID in pr_reg at its creation.

---
 drivers/target/target_core_fabric_lib.c | 130 +++++++++---------------
 drivers/target/target_core_internal.h   |   9 +-
 drivers/target/target_core_pr.c         |  19 ++--
 include/target/target_core_base.h       |   2 +
 4 files changed, 68 insertions(+), 92 deletions(-)
drivers/target/target_core_fabric_lib.c +50 -80
index cd1360095036..32c2afbb3033 100644
@@ -33,14 +33,14 @@
 
 
 static int sas_get_pr_transport_id(
-	struct t10_pr_registration *pr_reg,
-	int *format_code,
+	const char *initiatorname,
 	unsigned char *buf)
 {
 	int ret;
 
+	buf[0] = SCSI_PROTOCOL_SAS;
 	/* Skip over 'naa. prefix */
-	ret = hex2bin(&buf[4], &pr_reg->pr_iport[4], 8);
+	ret = hex2bin(&buf[4], &initiatorname[4], 8);
 	if (ret) {
 		pr_debug("%s: invalid hex string\n", __func__);
 		return ret;
@@ -50,19 +50,19 @@ static int sas_get_pr_transport_id(
 }
 
 static int fc_get_pr_transport_id(
-	struct t10_pr_registration *pr_reg,
-	int *format_code,
+	const char *initiatorname,
 	unsigned char *buf)
 {
-	unsigned char *ptr;
+	const char *ptr;
 	int i, ret;
 	u32 off = 8;
 
+	buf[0] = SCSI_PROTOCOL_FCP;
 	/*
 	 * We convert the ASCII formatted N Port name into a binary
 	 * encoded TransportID.
 	 */
-	ptr = &pr_reg->pr_iport[0];
+	ptr = &initiatorname[0];
 	for (i = 0; i < 23; ) {
 		if (!strncmp(&ptr[i], ":", 1)) {
 			i++;
@@ -82,13 +82,14 @@ static int fc_get_pr_transport_id(
 }
 
 static int sbp_get_pr_transport_id(
-	struct t10_pr_registration *pr_reg,
-	int *format_code,
+	const char *initiatorname,
 	unsigned char *buf)
 {
 	int ret;
 
-	ret = hex2bin(&buf[8], pr_reg->pr_iport, 8);
+	buf[0] = SCSI_PROTOCOL_SBP;
+
+	ret = hex2bin(&buf[8], initiatorname, 8);
 	if (ret) {
 		pr_debug("%s: invalid hex string\n", __func__);
 		return ret;
@@ -98,15 +99,16 @@ static int sbp_get_pr_transport_id(
 }
 
 static int srp_get_pr_transport_id(
-	struct t10_pr_registration *pr_reg,
-	int *format_code,
+	const char *initiatorname,
 	unsigned char *buf)
 {
 	const char *p;
 	unsigned len, count, leading_zero_bytes;
 	int rc;
 
-	p = pr_reg->pr_iport;
+	buf[0] = SCSI_PROTOCOL_SRP;
+
+	p = initiatorname;
 	if (strncasecmp(p, "0x", 2) == 0)
 		p += 2;
 	len = strlen(p);
@@ -126,14 +128,16 @@ static int srp_get_pr_transport_id(
 }
 
 static int iscsi_get_pr_transport_id(
-	struct t10_pr_registration *pr_reg,
-	int *format_code,
+	const char *initiatorname,
+	unsigned char *isid,
 	unsigned char *buf)
 {
 	u32 off = 4, padding = 0;
 	int isid_len;
 	u16 len = 0;
 
+	buf[0] = SCSI_PROTOCOL_ISCSI;
+
 	/*
 	 * Only null terminate the last field.
 	 *
@@ -149,9 +153,9 @@ static int iscsi_get_pr_transport_id(
 	 * length of the iSCSI TransportID or the contents of the ADDITIONAL
 	 * LENGTH field.
 	 */
-	len = sprintf(&buf[off], "%s", pr_reg->pr_iport);
+	len = sprintf(&buf[off], "%s", initiatorname);
 	off += len;
-	if ((*format_code == 1) && (pr_reg->isid_present_at_reg)) {
+	if (isid) {
 		/*
 		 * Set FORMAT CODE 01b for iSCSI Initiator port TransportID
 		 * format.
@@ -185,7 +189,7 @@ static int iscsi_get_pr_transport_id(
 		buf[off++] = 0x78; /* ASCII Character: "x" */
 		len += 5;
 
-		isid_len = sprintf(buf + off, "%s", pr_reg->pr_reg_isid);
+		isid_len = sprintf(buf + off, "%s", isid);
 		off += isid_len;
 		len += isid_len;
 	}
@@ -210,47 +214,6 @@ static int iscsi_get_pr_transport_id(
 	return len;
 }
 
-static int iscsi_get_pr_transport_id_len(
-	struct t10_pr_registration *pr_reg,
-	int *format_code)
-{
-	u32 len = 0, padding = 0;
-
-	len = strlen(pr_reg->pr_iport);
-	/*
-	 * Add extra byte for NULL terminator
-	 */
-	len++;
-	/*
-	 * If there is ISID present with the registration, use format code:
-	 * 01b: iSCSI Initiator port TransportID format
-	 *
-	 * If there is not an active iSCSI session, use format code:
-	 * 00b: iSCSI Initiator device TransportID format
-	 */
-	if (pr_reg->isid_present_at_reg) {
-		len += 5; /* For ",i,0x" ASCII separator */
-		len += strlen(pr_reg->pr_reg_isid);
-		*format_code = 1;
-	} else
-		*format_code = 0;
-	/*
-	 * The ADDITIONAL LENGTH field specifies the number of bytes that follow
-	 * in the TransportID. The additional length shall be at least 20 and
-	 * shall be a multiple of four.
-	 */
-	padding = ((-len) & 3);
-	if (padding != 0)
-		len += padding;
-	/*
-	 * Increment value for total payload + header length for
-	 * full status descriptor
-	 */
-	len += 4;
-
-	return len;
-}
-
 static char *iscsi_parse_pr_out_transport_id(
 	struct se_portal_group *se_tpg,
 	char *buf,
@@ -336,47 +299,54 @@ static char *iscsi_parse_pr_out_transport_id(
 	return &buf[4];
 }
 
-int target_get_pr_transport_id_len(
-		struct t10_pr_registration *pr_reg, int *format_code)
+int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
 {
-	switch (pr_reg->se_tpg->proto_id) {
+	switch (pr_reg->pr_tid[0] & 0xF) {
 	case SCSI_PROTOCOL_FCP:
 	case SCSI_PROTOCOL_SBP:
 	case SCSI_PROTOCOL_SRP:
 	case SCSI_PROTOCOL_SAS:
-		break;
+		return 24;
 	case SCSI_PROTOCOL_ISCSI:
-		return iscsi_get_pr_transport_id_len(pr_reg, format_code);
+		return get_unaligned_be16(&pr_reg->pr_tid[2]) + 4;
 	default:
-		pr_err("Unknown proto_id: 0x%02x\n", pr_reg->se_tpg->proto_id);
+		WARN(1, "Unknown proto_id: %#x\n", pr_reg->pr_tid[0] & 0xF);
 		return -EINVAL;
 	}
-
-	/*
-	 * Most transports use a fixed length 24 byte identifier.
-	 */
-	*format_code = 0;
-	return 24;
 }
 
 int target_get_pr_transport_id(
-		struct t10_pr_registration *pr_reg, int *format_code,
+		struct t10_pr_registration *pr_reg,
 		unsigned char *buf)
 {
-	switch (pr_reg->se_tpg->proto_id) {
+	int len =  target_get_pr_transport_id_len(pr_reg);
+
+	if (len > 0)
+		memcpy(buf, pr_reg->pr_tid, len);
+
+	return  len;
+}
+
+int target_gen_pr_transport_id(
+	struct t10_pr_registration *pr_reg,
+	int proto_id,
+	const char *initiatorname,
+	unsigned char *isid)
+{
+
+	switch (proto_id) {
 	case SCSI_PROTOCOL_SAS:
-		return sas_get_pr_transport_id(pr_reg, format_code, buf);
+		return sas_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
 	case SCSI_PROTOCOL_SBP:
-		return sbp_get_pr_transport_id(pr_reg, format_code, buf);
+		return sbp_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
 	case SCSI_PROTOCOL_SRP:
-		return srp_get_pr_transport_id(pr_reg, format_code, buf);
+		return srp_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
 	case SCSI_PROTOCOL_FCP:
-		return fc_get_pr_transport_id(pr_reg, format_code, buf);
+		return fc_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
 	case SCSI_PROTOCOL_ISCSI:
-		return iscsi_get_pr_transport_id(pr_reg, format_code,
-				buf);
+		return iscsi_get_pr_transport_id(initiatorname, isid, pr_reg->pr_tid);
 	default:
-		pr_err("Unknown proto_id: 0x%02x\n", pr_reg->se_tpg->proto_id);
+		pr_err("Unknown proto_id: 0x%02x\n", proto_id);
 		return -EINVAL;
 	}
 }
drivers/target/target_core_internal.h +6 -3
index b32ac78a26b0..95d5add73578 100644
@@ -98,10 +98,13 @@ void	target_setup_backend_cits(struct target_backend *);
 int	target_fabric_setup_cits(struct target_fabric_configfs *);
 
 /* target_core_fabric_lib.c */
-int	target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg,
-				       int *format_code);
+int	target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
 int	target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
-				   int *format_code, unsigned char *buf);
+				   unsigned char *buf);
+int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
+			       int proto_id,
+			       const char *initiatorname,
+			       unsigned char *isid);
 const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
 		char *buf, u32 *out_tid_len, char **port_nexus_ptr);
 
drivers/target/target_core_pr.c +10 -9
index 1f19bfd0fa00..d9e7d177b65a 100644
@@ -716,6 +716,10 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 						    aptpl);
 	if (!pr_reg)
 		return NULL;
+
+	target_gen_pr_transport_id(pr_reg, nacl->se_tpg->proto_id,
+				   nacl->initiatorname, isid);
+
 	/*
 	 * Return pointer to pr_reg for ALL_TG_PT=0
 	 */
@@ -802,6 +806,9 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 				core_scsi3_lunacl_undepend_item(deve_tmp);
 				goto out;
 			}
+			target_gen_pr_transport_id(pr_reg_atp,
+						   nacl_tmp->se_tpg->proto_id,
+						   nacl_tmp->initiatorname, isid);
 
 			list_add_tail(&pr_reg_atp->pr_reg_atp_mem_list,
 				      &pr_reg->pr_reg_atp_list);
@@ -3829,13 +3836,12 @@ static sense_reason_t
 core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 {
 	struct se_device *dev = cmd->se_dev;
-	struct se_portal_group *se_tpg;
 	struct t10_pr_registration *pr_reg, *pr_reg_tmp;
 	struct t10_reservation *pr_tmpl = &dev->t10_pr;
 	unsigned char *buf;
 	u32 add_desc_len = 0, add_len = 0;
 	u32 off = 8; /* off into first Full Status descriptor */
-	int format_code = 0, pr_res_type = 0, pr_res_scope = 0;
+	int pr_res_type = 0, pr_res_scope = 0;
 	int exp_desc_len, desc_len;
 	bool all_reg = false;
 
@@ -3868,7 +3874,6 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 	list_for_each_entry_safe(pr_reg, pr_reg_tmp,
 			&pr_tmpl->registration_list, pr_reg_list) {
 
-		se_tpg = pr_reg->se_tpg;
 		add_desc_len = 0;
 
 		atomic_inc_mb(&pr_reg->pr_res_holders);
@@ -3877,8 +3882,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 		 * Determine expected length of $FABRIC_MOD specific
 		 * TransportID full status descriptor..
 		 */
-		exp_desc_len = target_get_pr_transport_id_len(pr_reg,
-					&format_code);
+		exp_desc_len = target_get_pr_transport_id_len(pr_reg);
 		if (exp_desc_len < 0 ||
 		    exp_desc_len + add_len > cmd->data_length) {
 			pr_warn("SPC-3 PRIN READ_FULL_STATUS ran"
@@ -3938,13 +3942,10 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 		} else
 			off += 2; /* Skip over RELATIVE TARGET PORT IDENTIFIER */
 
-		buf[off+4] = se_tpg->proto_id;
-
 		/*
 		 * Now, have the $FABRIC_MOD fill in the transport ID.
 		 */
-		desc_len = target_get_pr_transport_id(pr_reg,
-				&format_code, &buf[off+4]);
+		desc_len = target_get_pr_transport_id(pr_reg, &buf[off+4]);
 
 		spin_lock(&pr_tmpl->registration_lock);
 		atomic_dec_mb(&pr_reg->pr_res_holders);
include/target/target_core_base.h +2 -0
index 095b96cb3557..64cb943b5d3a 100644
@@ -347,6 +347,8 @@ struct t10_pr_registration {
 	/* Used during APTPL metadata reading */
 #define PR_APTPL_MAX_IPORT_LEN			256
 	unsigned char pr_iport[PR_APTPL_MAX_IPORT_LEN];
+#define PR_REG_TID_LEN				228
+	char pr_tid[PR_REG_TID_LEN];
 	/* Reservation effects all target ports */
 	int pr_reg_all_tg_pt;
 	/* Activate Persistence across Target Power Loss */
Signature · 1 line
2.25.1
#26 lore raw

RFC PATCH 26/48 target: core: pr: remove se_tpg from pr_reg

In reply to Dmitry Bogdanov

Undepend pr_reg from se_tpg.
Use RTPI at PR creation.

---
 drivers/target/target_core_configfs.c        |  47 ++++---
 drivers/target/target_core_fabric_configfs.c |   2 +-
 drivers/target/target_core_internal.h        |   1 +
 drivers/target/target_core_pr.c              | 126 ++++++++-----------
 drivers/target/target_core_pr.h              |   2 +-
 drivers/target/target_core_tpg.c             |  17 ++-
 include/target/target_core_base.h            |   1 -
 7 files changed, 104 insertions(+), 92 deletions(-)
drivers/target/target_core_configfs.c +31 -16
index f469cfde3f1e..0978279bd5e3 100644
@@ -1736,6 +1736,8 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
 		char *page)
 {
 	struct t10_pr_registration *pr_reg;
+	struct se_portal_group *se_tpg;
+	const char *fabric_name = "Unknown";
 	char i_buf[PR_REG_ISID_ID_LEN] = { };
 
 	pr_reg = dev->dev_pr_res_holder;
@@ -1744,9 +1746,12 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
 
 	core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
 
+	se_tpg = core_get_tpg_by_rtpi(pr_reg->tg_pt_sep_rtpi);
+	if (se_tpg)
+		fabric_name = se_tpg->se_tpg_tfo->fabric_name;
+
 	return sprintf(page, "SPC-3 Reservation: %s Initiator: %s%s\n",
-		pr_reg->se_tpg->se_tpg_tfo->fabric_name,
-		pr_reg->pr_iport, i_buf);
+		fabric_name, pr_reg->pr_iport, i_buf);
 }
 
 static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
@@ -1838,18 +1843,24 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item,
 		goto out_unlock;
 	}
 
-	se_tpg = pr_reg->se_tpg;
-	tfo = se_tpg->se_tpg_tfo;
-
-	len += sprintf(page+len, "SPC-3 Reservation: %s"
-		" Target Node Endpoint: %s\n", tfo->fabric_name,
-		tfo->tpg_get_wwn(se_tpg));
-	len += sprintf(page+len, "SPC-3 Reservation: Relative Port"
-		" Identifier Tag: %hu %s Portal Group Tag: %hu"
-		" %s Logical Unit: %llu\n", pr_reg->tg_pt_sep_rtpi,
-		tfo->fabric_name, tfo->tpg_get_tag(se_tpg),
-		tfo->fabric_name, pr_reg->pr_aptpl_target_lun);
+	se_tpg = core_get_tpg_by_rtpi(pr_reg->tg_pt_sep_rtpi);
+	if (se_tpg) {
+		tfo = se_tpg->se_tpg_tfo;
 
+		len += sprintf(page+len, "SPC-3 Reservation: %s Target Node Endpoint: %s\n",
+			tfo->fabric_name, tfo->tpg_get_wwn(se_tpg));
+		len += sprintf(page+len, "SPC-3 Reservation: Relative Port Identifier Tag: %hu %s Portal Group Tag: %hu %s Logical Unit: %llu\n",
+			pr_reg->tg_pt_sep_rtpi,
+			tfo->fabric_name, tfo->tpg_get_tag(se_tpg),
+			tfo->fabric_name, pr_reg->pr_aptpl_target_lun);
+	} else {
+		len += sprintf(page+len, "SPC-3 Reservation: %s Target Node Endpoint: %s\n",
+			"Unknown", "Unknown");
+		len += sprintf(page+len, "SPC-3 Reservation: Relative Port Identifier Tag: %hu %s Portal Group Tag: %hu %s Logical Unit: %llu\n",
+			pr_reg->tg_pt_sep_rtpi,
+			"Unknown", 0,
+			"Unknown", pr_reg->pr_aptpl_target_lun);
+	}
 out_unlock:
 	spin_unlock(&dev->dev_reservation_lock);
 	return len;
@@ -1860,8 +1871,9 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,
 		char *page)
 {
 	struct se_device *dev = pr_to_dev(item);
-	const struct target_core_fabric_ops *tfo;
 	struct t10_pr_registration *pr_reg;
+	const char *fabric_name = "Unknown";
+	struct se_portal_group *se_tpg;
 	unsigned char buf[384];
 	char i_buf[PR_REG_ISID_ID_LEN];
 	ssize_t len = 0;
@@ -1875,11 +1887,14 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,
 
 		memset(buf, 0, 384);
 		memset(i_buf, 0, PR_REG_ISID_ID_LEN);
-		tfo = pr_reg->se_tpg->se_tpg_tfo;
+		se_tpg = core_get_tpg_by_rtpi(pr_reg->tg_pt_sep_rtpi);
+		if (se_tpg)
+			fabric_name = se_tpg->se_tpg_tfo->fabric_name;
+
 		core_pr_dump_initiator_port(pr_reg, i_buf,
 					PR_REG_ISID_ID_LEN);
 		sprintf(buf, "%s Node: %s%s Key: 0x%016Lx PRgen: 0x%08x\n",
-			tfo->fabric_name,
+			fabric_name,
 			pr_reg->pr_iport, i_buf, pr_reg->pr_res_key,
 			pr_reg->pr_res_generation);
 
drivers/target/target_core_fabric_configfs.c +1 -1
index fc1b8f54fb54..fd8f3b4c4bf8 100644
@@ -871,7 +871,7 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,
 	spin_lock(&g_tpg_lock);
 
 	if (op) {
-		tpg = core_get_tpg_by_rtpi(se_tpg->tpg_rtpi);
+		tpg = _core_get_tpg_by_rtpi(se_tpg->tpg_rtpi);
 		if (tpg) {
 			spin_unlock(&g_tpg_lock);
 
drivers/target/target_core_internal.h +1 -0
index 95d5add73578..51c0151c05c2 100644
@@ -136,6 +136,7 @@ void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
 struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *tpg,
 		const char *initiatorname);
 void core_tpg_del_initiator_node_acl(struct se_node_acl *acl);
+struct se_portal_group *_core_get_tpg_by_rtpi(u16 rtpi);
 struct se_portal_group *core_get_tpg_by_rtpi(u16 rtpi);
 
 /* target_core_transport.c */
drivers/target/target_core_pr.c +55 -71

279 lines not on this page · raw message

drivers/target/target_core_pr.h +1 -1
index e4bdabfc4b31..954b0dff32bf 100644
@@ -86,7 +86,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 					struct se_node_acl *nacl,
 					const char *initiatorname,
 					u64 unpacked_lun,
-					struct se_portal_group *se_tpg,
+					u16 rtpi,
 					struct se_dev_entry *dest_deve,
 					u64 mapped_lun,
 					unsigned char *isid,
drivers/target/target_core_tpg.c +15 -2
index 5a81d592530f..bdcd65ce0f62 100644
@@ -738,8 +738,9 @@ static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)
 	return sprintf(page, "%#x\n", se_tpg->tpg_rtpi);
 }
 
+
 struct se_portal_group *
-core_get_tpg_by_rtpi(u16 rtpi)
+_core_get_tpg_by_rtpi(u16 rtpi)
 {
 	struct se_portal_group *tpg;
 
@@ -753,6 +754,18 @@ core_get_tpg_by_rtpi(u16 rtpi)
 	return NULL;
 }
 
+struct se_portal_group *
+core_get_tpg_by_rtpi(u16 rtpi)
+{
+	struct se_portal_group *tpg;
+
+	spin_lock(&g_tpg_lock);
+	tpg = _core_get_tpg_by_rtpi(rtpi);
+	spin_unlock(&g_tpg_lock);
+
+	return tpg;
+}
+
 static ssize_t core_tpg_rtpi_store(struct config_item *item,
 				   const char *page, size_t count)
 {
@@ -773,7 +786,7 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,
 	spin_lock(&g_tpg_lock);
 
 	if (se_tpg->enabled) {
-		tpg = core_get_tpg_by_rtpi(val);
+		tpg = _core_get_tpg_by_rtpi(val);
 		if (tpg) {
 			spin_unlock(&g_tpg_lock);
 			pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
include/target/target_core_base.h +0 -1
index 64cb943b5d3a..5378e2d3825f 100644
@@ -366,7 +366,6 @@ struct t10_pr_registration {
 	u64 pr_res_key;
 	atomic_t pr_res_holders;
 	struct se_node_acl *pr_reg_nacl;
-	struct se_portal_group *se_tpg;
 	/* Used by ALL_TG_PT=1 registration with deve->pr_ref taken */
 	struct se_dev_entry *pr_reg_deve;
 	struct list_head pr_reg_list;
Signature · 1 line
2.25.1
#27 lore raw

RFC PATCH 27/48 target: core: fix parsing PR OUT TID

In reply to Dmitry Bogdanov

Now only iSCSI TransportID is parsed. From other protocols the binary representation is passed to tcm core. That is wrong as that TID is compared against nacl->initiatorname which is in human readable format. This patch adds a parsing of TID for other protocols.

---
 drivers/target/target_core_fabric_lib.c | 119 ++++++++++++++++--------
 drivers/target/target_core_internal.h   |   4 +-
 drivers/target/target_core_pr.c         |  14 ++-
 3 files changed, 89 insertions(+), 48 deletions(-)
drivers/target/target_core_fabric_lib.c +81 -38
index 32c2afbb3033..34fdab558e24 100644
@@ -214,15 +214,67 @@ static int iscsi_get_pr_transport_id(
 	return len;
 }
 
-static char *iscsi_parse_pr_out_transport_id(
-	struct se_portal_group *se_tpg,
-	char *buf,
-	u32 *out_tid_len,
+static u32 sas_parse_pr_out_transport_id(
+	const char *buf,
+	char *initiatorname,
+	char **port_nexus_ptr)
+{
+	char port_name[17];
+
+	bin2hex(port_name, buf + 4, 8);
+	snprintf(initiatorname, 36, "naa.%s", port_name);
+
+	return 24;
+}
+
+
+static u32 sbp_parse_pr_out_transport_id(
+	const char *buf,
+	char *initiatorname,
+	char **port_nexus_ptr)
+{
+	bin2hex(initiatorname, buf + 8, 8);
+
+	return 24;
+}
+
+static u32 srp_parse_pr_out_transport_id(
+	const char *buf,
+	char *initiatorname,
+	char **port_nexus_ptr)
+{
+	char port_name[33];
+
+	bin2hex(port_name, buf + 8, 16);
+	snprintf(initiatorname, 36, "0x%s", port_name);
+
+	return 24;
+}
+
+static u32 fc_parse_pr_out_transport_id(
+	const char *buf,
+	char *initiatorname,
+	char **port_nexus_ptr)
+{
+	/*
+	 * Format the FCP Initiator port_name into colon separated values to
+	 * match the format by tcm_qla2xxx explicit ConfigFS NodeACLs.
+	 */
+	snprintf(initiatorname, 36, "%8phC", buf + 8);
+
+	return 24;
+}
+
+u32 iscsi_parse_pr_out_transport_id(
+	const char *buf,
+	char *initiatorname,
 	char **port_nexus_ptr)
 {
 	char *p;
 	int i;
 	u8 format_code = (buf[0] & 0xc0);
+	u32 out_tid_len;
+
 	/*
 	 * Check for FORMAT CODE 00b or 01b from spc4r17, section 7.5.4.6:
 	 *
@@ -239,18 +291,10 @@ static char *iscsi_parse_pr_out_transport_id(
 	if ((format_code != 0x00) && (format_code != 0x40)) {
 		pr_err("Illegal format code: 0x%02x for iSCSI"
 			" Initiator Transport ID\n", format_code);
-		return NULL;
-	}
-	/*
-	 * If the caller wants the TransportID Length, we set that value for the
-	 * entire iSCSI Tarnsport ID now.
-	 */
-	if (out_tid_len) {
-		/* The shift works thanks to integer promotion rules */
-		*out_tid_len = get_unaligned_be16(&buf[2]);
-		/* Add four bytes for iSCSI Transport ID header */
-		*out_tid_len += 4;
+		return 0;
 	}
+	/* The shift works thanks to integer promotion rules */
+	out_tid_len = get_unaligned_be16(&buf[2]);
 
 	/*
 	 * Check for ',i,0x' separator between iSCSI Name and iSCSI Initiator
@@ -263,9 +307,10 @@ static char *iscsi_parse_pr_out_transport_id(
 			pr_err("Unable to locate \",i,0x\" separator"
 				" for Initiator port identifier: %s\n",
 				&buf[4]);
-			return NULL;
+			return 0;
 		}
-		*p = '\0'; /* Terminate iSCSI Name */
+		memcpy(initiatorname, &buf[4], p - buf - 4);
+		initiatorname[p - buf - 4] = '\0'; /* Terminate iSCSI Name */
 		p += 5; /* Skip over ",i,0x" separator */
 
 		*port_nexus_ptr = p;
@@ -293,10 +338,16 @@ static char *iscsi_parse_pr_out_transport_id(
 			*p = tolower(*p);
 			p++;
 		}
-	} else
+	} else {
+		strscpy(initiatorname, &buf[4], out_tid_len);
+		initiatorname[p - buf] = '\0'; /* Terminate iSCSI Name */
 		*port_nexus_ptr = NULL;
+	}
+
+	/* Add four bytes for iSCSI Transport ID header */
+	out_tid_len += 4;
 
-	return &buf[4];
+	return out_tid_len;
 }
 
 int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
@@ -351,33 +402,25 @@ int target_gen_pr_transport_id(
 	}
 }
 
-const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
-		char *buf, u32 *out_tid_len, char **port_nexus_ptr)
+u32 target_parse_pr_out_transport_id(
+	const char *buf, char *initiatorname, char **port_nexus_ptr)
 {
-	u32 offset;
-
-	switch (tpg->proto_id) {
+	switch (buf[0] & 0xF) {
 	case SCSI_PROTOCOL_SAS:
-		/*
-		 * Assume the FORMAT CODE 00b from spc4r17, 7.5.4.7 TransportID
-		 * for initiator ports using SCSI over SAS Serial SCSI Protocol.
-		 */
-		offset = 4;
-		break;
+		return sas_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
 	case SCSI_PROTOCOL_SBP:
+		return sbp_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
 	case SCSI_PROTOCOL_SRP:
+		return srp_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
 	case SCSI_PROTOCOL_FCP:
-		offset = 8;
-		break;
+		return fc_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
 	case SCSI_PROTOCOL_ISCSI:
-		return iscsi_parse_pr_out_transport_id(tpg, buf, out_tid_len,
-					port_nexus_ptr);
+		return iscsi_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
 	default:
-		pr_err("Unknown proto_id: 0x%02x\n", tpg->proto_id);
-		return NULL;
+		pr_err("Unknown proto_id: 0x%02x\n", buf[0] & 0xF);
+		return 0;
 	}
 
 	*port_nexus_ptr = NULL;
-	*out_tid_len = 24;
-	return buf + offset;
+	return 24;
 }
drivers/target/target_core_internal.h +2 -2
index 51c0151c05c2..bbdddcd83fa4 100644
@@ -105,8 +105,8 @@ int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
 			       int proto_id,
 			       const char *initiatorname,
 			       unsigned char *isid);
-const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
-		char *buf, u32 *out_tid_len, char **port_nexus_ptr);
+u32 target_parse_pr_out_transport_id(
+	const char *buf, char *initiatorname, char **port_nexus_ptr);
 
 /* target_core_hba.c */
 struct se_hba *core_alloc_hba(const char *, u32, u32);
drivers/target/target_core_pr.c +6 -8
index c4ee6999cf96..f5796d60ed76 100644
@@ -1419,7 +1419,7 @@ core_scsi3_decode_spec_i_port(
 	LIST_HEAD(tid_dest_list);
 	struct pr_transport_id_holder *tidh_new, *tidh, *tidh_tmp;
 	unsigned char *buf, *ptr, proto_ident;
-	const unsigned char *i_str = NULL;
+	unsigned char i_str[TRANSPORT_IQN_LEN] = {0};
 	char *iport_ptr = NULL, i_buf[PR_REG_ISID_ID_LEN];
 	sense_reason_t ret;
 	u32 tpdl, tid_len = 0;
@@ -1512,9 +1512,8 @@ core_scsi3_decode_spec_i_port(
 			dest_rtpi = tmp_lun->lun_tpg->tpg_rtpi;
 
 			iport_ptr = NULL;
-			i_str = target_parse_pr_out_transport_id(tmp_tpg,
-					ptr, &tid_len, &iport_ptr);
-			if (!i_str)
+			tid_len = target_parse_pr_out_transport_id(ptr, i_str, &iport_ptr);
+			if (!tid_len)
 				continue;
 			/*
 			 * Determine if this SCSI device server requires that
@@ -3087,7 +3086,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 	struct t10_pr_registration *pr_reg, *pr_res_holder, *dest_pr_reg;
 	struct t10_reservation *pr_tmpl = &dev->t10_pr;
 	unsigned char *buf;
-	const unsigned char *initiator_str;
+	unsigned char initiator_str[TRANSPORT_IQN_LEN] = { };
 	char *iport_ptr = NULL, i_buf[PR_REG_ISID_ID_LEN] = { };
 	u32 tid_len, tmp_tid_len;
 	int new_reg = 0, type, scope, matching_iname;
@@ -3212,9 +3211,8 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 		ret = TCM_INVALID_PARAMETER_LIST;
 		goto out;
 	}
-	initiator_str = target_parse_pr_out_transport_id(dest_se_tpg,
-			&buf[24], &tmp_tid_len, &iport_ptr);
-	if (!initiator_str) {
+	tmp_tid_len = target_parse_pr_out_transport_id(&buf[24], initiator_str, &iport_ptr);
+	if (!tmp_tid_len) {
 		pr_err("SPC-3 PR REGISTER_AND_MOVE: Unable to locate"
 			" initiator_str from Transport ID\n");
 		ret = TCM_INVALID_PARAMETER_LIST;
Signature · 1 line
2.25.1
#28 lore raw

RFC PATCH 28/48 target: core: add function to compare TransportID

In reply to Dmitry Bogdanov
---
 drivers/target/target_core_fabric_lib.c | 27 +++++++++++++++++++++----
 drivers/target/target_core_internal.h   |  2 ++
 2 files changed, 25 insertions(+), 4 deletions(-)
drivers/target/target_core_fabric_lib.c +23 -4
index 34fdab558e24..0eb03d7b9e78 100644
@@ -350,22 +350,27 @@ u32 iscsi_parse_pr_out_transport_id(
 	return out_tid_len;
 }
 
-int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
+static int __target_get_pr_transport_id_len(const char *tid)
 {
-	switch (pr_reg->pr_tid[0] & 0xF) {
+	switch (tid[0] & 0xF) {
 	case SCSI_PROTOCOL_FCP:
 	case SCSI_PROTOCOL_SBP:
 	case SCSI_PROTOCOL_SRP:
 	case SCSI_PROTOCOL_SAS:
 		return 24;
 	case SCSI_PROTOCOL_ISCSI:
-		return get_unaligned_be16(&pr_reg->pr_tid[2]) + 4;
+		return get_unaligned_be16(&tid[2]) + 4;
 	default:
-		WARN(1, "Unknown proto_id: %#x\n", pr_reg->pr_tid[0] & 0xF);
+		WARN(1, "Unknown proto_id: %#x\n", tid[0] & 0xF);
 		return -EINVAL;
 	}
 }
 
+int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
+{
+	return __target_get_pr_transport_id_len(pr_reg->pr_tid);
+}
+
 int target_get_pr_transport_id(
 		struct t10_pr_registration *pr_reg,
 		unsigned char *buf)
@@ -378,6 +383,20 @@ int target_get_pr_transport_id(
 	return  len;
 }
 
+int target_cmp_pr_transport_id(
+		struct t10_pr_registration *pr_reg,
+		unsigned char *tid)
+{
+	int len1 =  __target_get_pr_transport_id_len(pr_reg->pr_tid);
+	int len2 =  __target_get_pr_transport_id_len(tid);
+
+	if (len1 != len2)
+		return 1;
+
+	return memcmp(pr_reg->pr_tid, tid, len1);
+}
+
+
 int target_gen_pr_transport_id(
 	struct t10_pr_registration *pr_reg,
 	int proto_id,
drivers/target/target_core_internal.h +2 -0
index bbdddcd83fa4..b0d29c30ab1e 100644
@@ -101,6 +101,8 @@ int	target_fabric_setup_cits(struct target_fabric_configfs *);
 int	target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
 int	target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
 				   unsigned char *buf);
+int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
+			       unsigned char *tid);
 int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
 			       int proto_id,
 			       const char *initiatorname,
Signature · 1 line
2.25.1
#29 lore raw

RFC PATCH 29/48 target: core: store proto_id in APTPL

In reply to Dmitry Bogdanov

Store and read proto_id in APTPL to create TransportID.
That will allow to report registrations for not yet created TPG/ACL/LUNs

---
 drivers/target/target_core_configfs.c | 13 +++++++++++--
 drivers/target/target_core_pr.c       | 15 ++++++++++-----
 drivers/target/target_core_pr.h       |  2 +-
 3 files changed, 22 insertions(+), 8 deletions(-)
drivers/target/target_core_configfs.c +11 -2
index 0978279bd5e3..f63c37344104 100644
@@ -1974,7 +1974,8 @@ enum {
 	Opt_initiator_fabric, Opt_initiator_node, Opt_initiator_sid,
 	Opt_sa_res_key, Opt_res_holder, Opt_res_type, Opt_res_scope,
 	Opt_res_all_tg_pt, Opt_mapped_lun, Opt_target_fabric,
-	Opt_target_node, Opt_tpgt, Opt_port_rtpi, Opt_target_lun, Opt_err
+	Opt_target_node, Opt_tpgt, Opt_port_rtpi, Opt_target_lun,
+	Opt_proto_id, Opt_err
 };
 
 static match_table_t tokens = {
@@ -1992,6 +1993,7 @@ static match_table_t tokens = {
 	{Opt_tpgt, "tpgt=%d"},
 	{Opt_port_rtpi, "port_rtpi=%d"},
 	{Opt_target_lun, "target_lun=%u"},
+	{Opt_proto_id, "proto_id=%u"},
 	{Opt_err, NULL}
 };
 
@@ -2010,6 +2012,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
 	u16 tpgt = 0;
 	u16 rtpi = 0;
 	u8 type = 0;
+	u8 proto_id = 0;
 
 	if (!dev->dev_attrib.emulate_pr ||
 	    (dev->transport_flags & TRANSPORT_FLAG_PASSTHROUGH_PGR))
@@ -2151,6 +2154,12 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
 				goto out;
 			target_lun = (u64)tmp_ll;
 			break;
+		case Opt_proto_id:
+			ret = match_int(args, &arg);
+			if (ret)
+				goto out;
+			proto_id = (u16)arg;
+			break;
 		default:
 			break;
 		}
@@ -2171,7 +2180,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
 
 	ret = core_scsi3_alloc_aptpl_registration(&dev->t10_pr, sa_res_key,
 			i_port, isid, mapped_lun, rtpi, target_lun,
-			res_holder, all_tg_pt, type);
+			res_holder, all_tg_pt, type, proto_id);
 out:
 	kfree(i_fabric);
 	kfree(i_port);
drivers/target/target_core_pr.c +10 -5
index f5796d60ed76..d177db3024c9 100644
@@ -842,7 +842,8 @@ int core_scsi3_alloc_aptpl_registration(
 	u64 target_lun,
 	int res_holder,
 	int all_tg_pt,
-	u8 type)
+	u8 type,
+	u8 proto_id)
 {
 	struct t10_pr_registration *pr_reg;
 
@@ -886,6 +887,7 @@ int core_scsi3_alloc_aptpl_registration(
 	 * Copy the i_port information from caller.
 	 */
 	snprintf(pr_reg->pr_iport, PR_APTPL_MAX_IPORT_LEN, "%s", i_port);
+	target_gen_pr_transport_id(pr_reg, proto_id, i_port, isid);
 	/*
 	 * Set pr_res_holder from caller, the pr_reg who is the reservation
 	 * holder will get it's pointer set in core_scsi3_aptpl_reserve() once
@@ -1836,19 +1838,22 @@ static int core_scsi3_update_aptpl_buf(
 				"sa_res_key=%llu\n"
 				"res_holder=1\nres_type=%02x\n"
 				"res_scope=%02x\nres_all_tg_pt=%d\n"
-				"mapped_lun=%llu\n", reg_count,
+				"mapped_lun=%llu\nproto_id=%u\n", reg_count,
 				pr_reg->pr_iport, isid_buf,
 				pr_reg->pr_res_key, pr_reg->pr_res_type,
 				pr_reg->pr_res_scope, pr_reg->pr_reg_all_tg_pt,
-				pr_reg->pr_res_mapped_lun);
+				pr_reg->pr_res_mapped_lun,
+				pr_reg->pr_tid[0] & 0xF);
 		} else {
 			snprintf(tmp, 512, "PR_REG_START: %d\n"
 				"initiator_node=%s\n%s"
 				"sa_res_key=%llu\nres_holder=0\n"
-				"res_all_tg_pt=%d\nmapped_lun=%llu\n",
+				"res_all_tg_pt=%d\nmapped_lun=%llu\n"
+				"proto_id=%u\n",
 				reg_count, pr_reg->pr_iport, isid_buf,
 				pr_reg->pr_res_key, pr_reg->pr_reg_all_tg_pt,
-				pr_reg->pr_res_mapped_lun);
+				pr_reg->pr_res_mapped_lun,
+				pr_reg->pr_tid[0] & 0xF);
 		}
 
 		if ((len + strlen(tmp) >= pr_aptpl_buf_len)) {
drivers/target/target_core_pr.h +1 -1
index 954b0dff32bf..589fdac5470f 100644
@@ -64,7 +64,7 @@ extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);
 extern int core_scsi3_alloc_aptpl_registration(
 			struct t10_reservation *, u64,
 			unsigned char *, unsigned char *, u64,
-			u16, u64, int, int, u8);
+			u16, u64, int, int, u8, u8);
 extern int core_scsi3_check_aptpl_registration(struct se_device *,
 			struct se_portal_group *, struct se_lun *,
 			struct se_node_acl *, u64);
Signature · 1 line
2.25.1
#30 lore raw

RFC PATCH 30/48 target: core: rethink APTPL registrations

In reply to Dmitry Bogdanov

Make APTPL registrations be real registrations from the very beginning. Just mark at deve creation that it has a PR registration if it exists.

---
 drivers/target/target_core_pr.c | 104 +++++++++++++-------------------
 1 file changed, 43 insertions(+), 61 deletions(-)
drivers/target/target_core_pr.c +43 -61
index d177db3024c9..a984987f0a82 100644
@@ -832,6 +832,28 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 	return NULL;
 }
 
+static void __core_scsi3_add_registration(struct se_device *, struct se_node_acl *,
+				struct t10_pr_registration *, enum register_type, int);
+
+static void core_scsi3_aptpl_reserve(
+	struct se_device *dev,
+	const char *initiatorname,
+	struct t10_pr_registration *pr_reg)
+{
+	char i_buf[PR_REG_ISID_ID_LEN] = { };
+
+	core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
+
+	spin_lock(&dev->dev_reservation_lock);
+	dev->dev_pr_res_holder = pr_reg;
+	spin_unlock(&dev->dev_reservation_lock);
+
+	pr_debug("SPC-3 PR Service Action: APTPL RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n",
+		core_scsi3_pr_dump_type(pr_reg->pr_res_type),
+		(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
+	pr_debug("SPC-3 PR RESERVE Node: %s%s\n", initiatorname, i_buf);
+}
+
 int core_scsi3_alloc_aptpl_registration(
 	struct t10_reservation *pr_tmpl,
 	u64 sa_res_key,
@@ -845,6 +867,7 @@ int core_scsi3_alloc_aptpl_registration(
 	u8 type,
 	u8 proto_id)
 {
+	struct se_device *dev = container_of(pr_tmpl, struct se_device, t10_pr);
 	struct t10_pr_registration *pr_reg;
 
 	if (!i_port || !sa_res_key) {
@@ -888,47 +911,24 @@ int core_scsi3_alloc_aptpl_registration(
 	 */
 	snprintf(pr_reg->pr_iport, PR_APTPL_MAX_IPORT_LEN, "%s", i_port);
 	target_gen_pr_transport_id(pr_reg, proto_id, i_port, isid);
+
+	pr_reg->pr_res_holder = res_holder;
+	__core_scsi3_add_registration(dev, NULL, pr_reg, REGISTER, 0);
+
+	if (pr_reg->pr_res_holder)
+		core_scsi3_aptpl_reserve(dev, i_port, pr_reg);
+
 	/*
-	 * Set pr_res_holder from caller, the pr_reg who is the reservation
-	 * holder will get it's pointer set in core_scsi3_aptpl_reserve() once
-	 * the Initiator Node LUN ACL from the fabric module is created for
-	 * this registration.
+	 * Reenable pr_aptpl_active to accept new metadata
+	 * updates once the SCSI device is active again..
 	 */
-	pr_reg->pr_res_holder = res_holder;
+	pr_tmpl->pr_aptpl_active = 1;
 
-	list_add_tail(&pr_reg->pr_reg_aptpl_list, &pr_tmpl->aptpl_reg_list);
 	pr_debug("SPC-3 PR APTPL Successfully added registration%s from"
 			" metadata\n", (res_holder) ? "+reservation" : "");
 	return 0;
 }
 
-static void core_scsi3_aptpl_reserve(
-	struct se_device *dev,
-	struct se_portal_group *tpg,
-	struct se_node_acl *node_acl,
-	struct t10_pr_registration *pr_reg)
-{
-	char i_buf[PR_REG_ISID_ID_LEN] = { };
-
-	core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
-
-	spin_lock(&dev->dev_reservation_lock);
-	dev->dev_pr_res_holder = pr_reg;
-	spin_unlock(&dev->dev_reservation_lock);
-
-	pr_debug("SPC-3 PR [%s] Service Action: APTPL RESERVE created"
-		" new reservation holder TYPE: %s ALL_TG_PT: %d\n",
-		tpg->se_tpg_tfo->fabric_name,
-		core_scsi3_pr_dump_type(pr_reg->pr_res_type),
-		(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
-	pr_debug("SPC-3 PR [%s] RESERVE Node: %s%s\n",
-		tpg->se_tpg_tfo->fabric_name, node_acl->initiatorname,
-		i_buf);
-}
-
-static void __core_scsi3_add_registration(struct se_device *, struct se_node_acl *,
-				struct t10_pr_registration *, enum register_type, int);
-
 static int __core_scsi3_check_aptpl_registration(
 	struct se_device *dev,
 	struct se_portal_group *tpg,
@@ -946,12 +946,13 @@ static int __core_scsi3_check_aptpl_registration(
 	 * may exist for fabrics that use ISIDs in their SCSI Initiator Port
 	 * TransportIDs.
 	 */
-	spin_lock(&pr_tmpl->aptpl_reg_lock);
-	list_for_each_entry_safe(pr_reg, pr_reg_tmp, &pr_tmpl->aptpl_reg_list,
-				pr_reg_aptpl_list) {
+	spin_lock(&pr_tmpl->registration_lock);
+	list_for_each_entry_safe(pr_reg, pr_reg_tmp,
+			&pr_tmpl->registration_list, pr_reg_list) {
 
-		if (!strcmp(pr_reg->pr_iport, nacl->initiatorname) &&
-		     (pr_reg->tg_pt_sep_rtpi == tpg->tpg_rtpi)) {
+		if (pr_reg->pr_reg_nacl == NULL &&
+		    !strcmp(pr_reg->pr_iport, nacl->initiatorname) &&
+		    (pr_reg->tg_pt_sep_rtpi == tpg->tpg_rtpi)) {
 			/*
 			 * Obtain the ->pr_reg_deve pointer + reference, that
 			 * is released by __core_scsi3_add_registration() below.
@@ -964,34 +965,15 @@ static int __core_scsi3_check_aptpl_registration(
 				rcu_read_unlock();
 				continue;
 			}
-			kref_get(&pr_reg->pr_reg_deve->pr_kref);
+			set_bit(DEF_PR_REG_ACTIVE, &pr_reg->pr_reg_deve->deve_flags);
+			pr_reg->pr_reg_deve = NULL;
 			rcu_read_unlock();
 
 			pr_reg->pr_reg_nacl = nacl;
-			pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;
-			list_del(&pr_reg->pr_reg_aptpl_list);
-			spin_unlock(&pr_tmpl->aptpl_reg_lock);
-			/*
-			 * At this point all of the pointers in *pr_reg will
-			 * be setup, so go ahead and add the registration.
-			 */
-			__core_scsi3_add_registration(dev, nacl, pr_reg, 0, 0);
-			/*
-			 * If this registration is the reservation holder,
-			 * make that happen now..
-			 */
-			if (pr_reg->pr_res_holder)
-				core_scsi3_aptpl_reserve(dev, tpg,
-						nacl, pr_reg);
-			/*
-			 * Reenable pr_aptpl_active to accept new metadata
-			 * updates once the SCSI device is active again..
-			 */
-			spin_lock(&pr_tmpl->aptpl_reg_lock);
-			pr_tmpl->pr_aptpl_active = 1;
+			pr_reg->pr_res_mapped_lun = mapped_lun;
 		}
 	}
-	spin_unlock(&pr_tmpl->aptpl_reg_lock);
+	spin_unlock(&pr_tmpl->registration_lock);
 
 	return 0;
 }
Signature · 1 line
2.25.1
#31 lore raw

RFC PATCH 31/48 dlm_ckv: introduce DLM cluster key-value storage

In reply to Dmitry Bogdanov

Introduce the first version of DLM CKV module that could be used by different kernel subsystems to share some information in a cluster.

This commit has just cluster level locks.

---
 drivers/target/Kconfig   |   6 +
 drivers/target/Makefile  |   2 +
 drivers/target/dlm_ckv.c | 323 +++++++++++++++++++++++++++++++++++++++
 drivers/target/dlm_ckv.h |  19 +++
 4 files changed, 350 insertions(+)
 create mode 100644 drivers/target/dlm_ckv.c
 create mode 100644 drivers/target/dlm_ckv.h
drivers/target/Kconfig +6 -0
index 72171ea3dd53..75d5e1d23a1c 100644
@@ -35,6 +35,12 @@ config TCM_PSCSI
 	Say Y here to enable the TCM/pSCSI subsystem plugin for non-buffered
 	passthrough access to Linux/SCSI device
 
+config DLM_CKV
+	tristate "Cluster key value storage over DLM"
+	depends on DLM
+	help
+	Say Y here to enable the cluster key value storage over DLM
+
 config TCM_USER2
 	tristate "TCM/USER Subsystem Plugin for Linux"
 	depends on UIO && NET
drivers/target/Makefile +2 -0
index 45634747377e..8bc9ac2bd629 100644
@@ -30,3 +30,5 @@ obj-$(CONFIG_LOOPBACK_TARGET)	+= loopback/
 obj-$(CONFIG_TCM_FC)		+= tcm_fc/
 obj-$(CONFIG_ISCSI_TARGET)	+= iscsi/
 obj-$(CONFIG_SBP_TARGET)	+= sbp/
+
+obj-$(CONFIG_DLM_CKV)			+= dlm_ckv.o
drivers/target/dlm_ckv.c new file +323 -0

323 lines not on this page · raw message

drivers/target/dlm_ckv.h new file +19 -0
new file mode 100644
index 000000000000..1a3f79e42bf6
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef DLM_CKV_H
+#define DLM_CKV_H
+
+struct dlm_ckv_bucket;
+struct dlm_ckv_lock;
+
+struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
+					   const char *cluster_name,
+					   void *userarg);
+int dlm_ckv_close_bucket(struct dlm_ckv_bucket *bucket);
+
+struct dlm_ckv_lock *
+dlm_ckv_create_lock(struct dlm_ckv_bucket *bucket, const char *name);
+void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);
+int dlm_ckv_lock_get(struct dlm_ckv_lock *ckv_lock);
+int dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock);
+
+#endif
Signature · 1 line
2.25.1
#32 lore raw

RFC PATCH 32/48 dlm_ckv: add notification service

In reply to Dmitry Bogdanov

Notification broadcasts over DLM cluster.
They do not have any payload. Used to be used to notify other nodes to read an updated data.

---
 drivers/target/dlm_ckv.c | 186 +++++++++++++++++++++++++++++++++++++++
 drivers/target/dlm_ckv.h |   8 ++
 2 files changed, 194 insertions(+)
drivers/target/dlm_ckv.c +186 -0
index a2e1a191c433..cffe4f2dcb82 100644
@@ -10,6 +10,7 @@
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/sched.h>
+#include <linux/workqueue.h>
 #include <target/target_core_base.h>
 #include "dlm_ckv.h"
 
@@ -33,8 +34,17 @@ struct dlm_ckv_bucket {
 	int nodeid[64];
 	void *userarg;
 	struct completion sync_compl;
+	struct workqueue_struct *notify_wq;
 };
 
+struct dlm_ckv_notify {
+	dlm_ckv_notify_cb notify_cb;
+	char name[DLM_RESNAME_MAXLEN];
+	struct dlm_ckv_lock pre_n;
+	struct dlm_ckv_lock post_n;
+	struct work_struct pre_n_work;
+	struct work_struct post_n_work;
+};
 
 #define DLM_CKV_LVB_SIZE	256
 
@@ -230,6 +240,179 @@ dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock)
 }
 EXPORT_SYMBOL(dlm_ckv_lock_release);
 
+static void dlm_cvk_pre_n_bast(void *astarg, int mode)
+{
+	struct dlm_ckv_lksb *lksb = astarg;
+	struct dlm_ckv_lock *ckv_lock;
+	struct dlm_ckv_bucket *bucket;
+	struct dlm_ckv_notify *notify;
+
+	ckv_lock = container_of(lksb, struct dlm_ckv_lock, lksb);
+	notify = container_of(ckv_lock, struct dlm_ckv_notify, pre_n);
+	bucket = ckv_lock->bucket;
+
+	queue_work(bucket->notify_wq, &notify->pre_n_work);
+}
+
+static void dlm_cvk_post_n_bast(void *astarg, int mode)
+{
+	struct dlm_ckv_lksb *lksb = astarg;
+	struct dlm_ckv_lock *ckv_lock;
+	struct dlm_ckv_bucket *bucket;
+	struct dlm_ckv_notify *notify;
+
+	ckv_lock = container_of(lksb, struct dlm_ckv_lock, lksb);
+	notify = container_of(ckv_lock, struct dlm_ckv_notify, post_n);
+	bucket = ckv_lock->bucket;
+
+	queue_work(bucket->notify_wq, &notify->post_n_work);
+}
+
+static void
+dlm_cvk_pre_n_work(struct work_struct *work)
+{
+	struct dlm_ckv_notify *notify = container_of(work,
+				struct dlm_ckv_notify, pre_n_work);
+	struct dlm_ckv_bucket *bucket = notify->pre_n.bucket;
+
+	dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_EX,
+			  &notify->post_n.lksb,
+			  DLM_LKF_CONVERT,
+			  notify->post_n.name, dlm_cvk_post_n_bast);
+	dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL,
+			  &notify->pre_n.lksb,
+			  DLM_LKF_CONVERT,
+			  notify->pre_n.name, dlm_cvk_pre_n_bast);
+}
+
+static void
+dlm_cvk_post_n_work(struct work_struct *work)
+{
+	struct dlm_ckv_notify *notify = container_of(work,
+				struct dlm_ckv_notify, post_n_work);
+	struct dlm_ckv_bucket *bucket = notify->post_n.bucket;
+
+	dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_EX,
+			  &notify->pre_n.lksb,
+			  DLM_LKF_CONVERT,
+			  notify->pre_n.name, dlm_cvk_pre_n_bast);
+
+	notify->notify_cb(bucket->userarg);
+
+	dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL,
+			  &notify->post_n.lksb,
+			  DLM_LKF_CONVERT,
+			  notify->post_n.name, dlm_cvk_post_n_bast);
+}
+
+struct dlm_ckv_notify *
+dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
+			      dlm_ckv_notify_cb cb)
+{
+	char lockname[DLM_RESNAME_MAXLEN];
+	struct dlm_ckv_notify *notify;
+	int res;
+
+	notify = kzalloc(sizeof(struct dlm_ckv_bucket), GFP_KERNEL);
+	if (!notify)
+		return NULL;
+
+	INIT_WORK(&notify->post_n_work, dlm_cvk_post_n_work);
+	INIT_WORK(&notify->pre_n_work, dlm_cvk_pre_n_work);
+
+	strscpy(notify->name, name, DLM_RESNAME_MAXLEN);
+	notify->notify_cb = cb;
+
+	kref_get(&bucket->refcount);
+
+	sprintf(lockname, "pre.%s.%d", name, bucket->local_nodeid);
+	dlm_ckv_lock_init(&notify->pre_n, bucket, lockname);
+
+	sprintf(lockname, "post.%s.%d", name, bucket->local_nodeid);
+	dlm_ckv_lock_init(&notify->post_n, bucket, lockname);
+
+	res = dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_EX,
+				&notify->pre_n.lksb, 0,
+				notify->pre_n.name, dlm_cvk_pre_n_bast);
+	if (res)
+		goto fail_locks;
+
+	res = dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL,
+				&notify->post_n.lksb, 0,
+				notify->post_n.name, dlm_cvk_post_n_bast);
+	if (res)
+		goto fail_locks;
+
+	return notify;
+
+fail_locks:
+	kref_put(&bucket->refcount, bucket_release);
+	kfree(notify);
+
+	return NULL;
+}
+EXPORT_SYMBOL(dlm_ckv_create_notification);
+
+void
+dlm_ckv_free_notification(struct dlm_ckv_notify *notify)
+{
+	struct dlm_ckv_bucket *bucket = notify->pre_n.bucket;
+
+	cancel_work_sync(&notify->pre_n_work);
+	cancel_work_sync(&notify->post_n_work);
+
+	kfree(notify);
+	kref_put(&bucket->refcount, bucket_release);
+}
+EXPORT_SYMBOL(dlm_ckv_free_notification);
+
+static void
+dlm_ckv_toggle_lock(struct dlm_ckv_bucket *bucket, const char *name_prefix)
+{
+	char lockname[DLM_RESNAME_MAXLEN];
+	struct dlm_ckv_lksb lksb;
+	int res;
+	int i;
+
+	init_completion(&lksb.compl);
+
+	for (i = 0; i < bucket->num_nodes; ++i) {
+		if (bucket->nodeid[i] == bucket->local_nodeid)
+			continue;
+		snprintf(lockname, sizeof(lockname), "%s.%d", name_prefix,
+			 bucket->nodeid[i]);
+
+		lksb.lksb.sb_lkid = 0;
+		res = dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_PR,
+				&lksb, 0, lockname, NULL);
+		if (res < 0)
+			pr_warn("Locking %s failed (%d)", lockname, res);
+		if (!lksb.lksb.sb_lkid)
+			continue;
+
+		dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL, &lksb,
+					DLM_LKF_CONVERT, lockname, NULL);
+		dlm_ckv_unlock_wait(bucket->ls, &lksb);
+	}
+}
+
+int dlm_ckv_notify(struct dlm_ckv_notify *notify)
+{
+	struct dlm_ckv_bucket *bucket = notify->pre_n.bucket;
+	char lockname[DLM_RESNAME_MAXLEN];
+
+	snprintf(lockname, sizeof(lockname), "post.%s", notify->name);
+	dlm_ckv_toggle_lock(bucket, lockname);
+
+	snprintf(lockname, sizeof(lockname), "pre.%s", notify->name);
+	dlm_ckv_toggle_lock(bucket, lockname);
+
+	snprintf(lockname, sizeof(lockname), "post.%s", notify->name);
+	dlm_ckv_toggle_lock(bucket, lockname);
+
+	return 0;
+}
+EXPORT_SYMBOL(dlm_ckv_notify);
 
 static void bucket_release(struct kref *ref)
 {
@@ -237,6 +420,8 @@ static void bucket_release(struct kref *ref)
 						     refcount);
 	int res;
 
+	destroy_workqueue(bucket->notify_wq);
+
 	res = dlm_release_lockspace(bucket->ls, 2);
 	if (res)
 		pr_err("forcibly releasing lockspace failed: %d\n",
@@ -264,6 +449,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
 
 	bucket->userarg = userarg;
 	init_completion(&bucket->sync_compl);
+	bucket->notify_wq = alloc_ordered_workqueue("notify_wq-%s", 0, name);
 
 	err = dlm_new_lockspace(name, cluster_name,
 				DLM_LSFL_FS | DLM_LSFL_NEWEXCL, DLM_CKV_LVB_SIZE,
drivers/target/dlm_ckv.h +8 -0
index 1a3f79e42bf6..080d9498f5f9 100644
@@ -5,6 +5,8 @@
 struct dlm_ckv_bucket;
 struct dlm_ckv_lock;
 
+typedef void (*dlm_ckv_notify_cb)(void *userarg);
+
 struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
 					   const char *cluster_name,
 					   void *userarg);
@@ -16,4 +18,10 @@ void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);
 int dlm_ckv_lock_get(struct dlm_ckv_lock *ckv_lock);
 int dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock);
 
+struct dlm_ckv_notify *
+dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
+			    dlm_ckv_notify_cb cb);
+void dlm_ckv_free_notification(struct dlm_ckv_notify *notify);
+int dlm_ckv_notify(struct dlm_ckv_notify *notify);
+
 #endif
Signature · 1 line
2.25.1
#33 lore raw

RFC PATCH 33/48 dlm_ckv: add key-value storage service

In reply to Dmitry Bogdanov

Simple key-value cluster storage.

---
 drivers/target/dlm_ckv.c | 128 +++++++++++++++++++++++++++++++++++++++
 drivers/target/dlm_ckv.h |   9 +++
 2 files changed, 137 insertions(+)
drivers/target/dlm_ckv.c +128 -0
index cffe4f2dcb82..22c5f0827595 100644
@@ -25,6 +25,10 @@ struct dlm_ckv_lock {
 	char name[DLM_RESNAME_MAXLEN];
 };
 
+struct dlm_ckv_kv {
+	struct dlm_ckv_lock lock;
+};
+
 struct dlm_ckv_bucket {
 	dlm_lockspace_t *ls;
 	struct kref refcount;
@@ -240,6 +244,130 @@ dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock)
 }
 EXPORT_SYMBOL(dlm_ckv_lock_release);
 
+struct dlm_ckv_kv *
+dlm_ckv_create_kv(struct dlm_ckv_bucket *bucket, const char *key)
+{
+	struct dlm_ckv_lock *ckv_lock;
+	void *ptr;
+	int res;
+
+	ptr = kzalloc(DLM_CKV_LVB_SIZE, GFP_KERNEL);
+	if (!ptr)
+		return NULL;
+
+	ckv_lock = dlm_ckv_create_lock(bucket, key);
+	if (!ckv_lock)
+		goto create_fail;
+
+	ckv_lock->lksb.lksb.sb_lvbptr = ptr;
+
+	res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_NL,
+				&ckv_lock->lksb, 0, ckv_lock->name, NULL);
+	if (res)
+		goto lock_failed;
+
+	return (struct dlm_ckv_kv *)ckv_lock;
+
+lock_failed:
+	dlm_ckv_free_lock(ckv_lock);
+create_fail:
+	kfree(ptr);
+	return NULL;
+}
+EXPORT_SYMBOL(dlm_ckv_create_kv);
+
+void
+dlm_ckv_free_kv(struct dlm_ckv_kv *kv)
+{
+	struct dlm_ckv_bucket *bucket = kv->lock.bucket;
+	struct dlm_ckv_lock *ckv_lock = &kv->lock;
+
+	dlm_ckv_unlock_wait(bucket->ls, &ckv_lock->lksb);
+
+	kfree(ckv_lock->lksb.lksb.sb_lvbptr);
+	kfree(ckv_lock);
+
+	kref_put(&bucket->refcount, bucket_release);
+
+}
+EXPORT_SYMBOL(dlm_ckv_free_kv);
+
+int
+dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len)
+{
+	struct dlm_ckv_lock *ckv_lock = &kv->lock;
+	struct dlm_ckv_bucket *bucket;
+	int res;
+
+	BUG_ON(!ckv_lock);
+	bucket = ckv_lock->bucket;
+
+	res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_CR,
+				&ckv_lock->lksb,
+				DLM_LKF_VALBLK | DLM_LKF_CONVERT,
+				ckv_lock->name, NULL);
+	if (res) {
+		pr_info("Can not get lock %s, rc=%d\n",
+			ckv_lock->name, res);
+		goto fail;
+	}
+
+	if (ckv_lock->lksb.lksb.sb_flags & DLM_SBF_VALNOTVALID) {
+		pr_info(" %s LVB was invalid\n", ckv_lock->name);
+		memset(value, 0, len);
+	} else
+		memcpy(value, ckv_lock->lksb.lksb.sb_lvbptr, len);
+
+	res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_NL,
+				&ckv_lock->lksb,
+				DLM_LKF_CONVERT,
+				ckv_lock->name, NULL);
+	if (res) {
+		pr_info("Can not release lock %s\n", ckv_lock->name);
+		goto fail;
+	}
+
+	return res;
+fail:
+	return res;
+}
+EXPORT_SYMBOL(dlm_ckv_get);
+
+int
+dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len)
+{
+	struct dlm_ckv_lock *ckv_lock = &kv->lock;
+	struct dlm_ckv_bucket *bucket;
+	int res;
+
+	BUG_ON(!ckv_lock);
+	bucket = ckv_lock->bucket;
+
+	res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_EX,
+				&ckv_lock->lksb,
+				DLM_LKF_CONVERT,
+				ckv_lock->name, NULL);
+	if (res) {
+		pr_info("Can not get lock %s\n", ckv_lock->name);
+		goto fail;
+	}
+
+	memcpy(ckv_lock->lksb.lksb.sb_lvbptr, value, len);
+	res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_NL,
+				&ckv_lock->lksb,
+				DLM_LKF_VALBLK | DLM_LKF_CONVERT,
+				ckv_lock->name, NULL);
+	if (res) {
+		pr_info("Can not release lock %s\n", ckv_lock->name);
+		goto fail;
+	}
+
+	return res;
+fail:
+	return res;
+}
+EXPORT_SYMBOL(dlm_ckv_set);
+
 static void dlm_cvk_pre_n_bast(void *astarg, int mode)
 {
 	struct dlm_ckv_lksb *lksb = astarg;
drivers/target/dlm_ckv.h +9 -0
index 080d9498f5f9..c01904313f1e 100644
@@ -4,6 +4,9 @@
 
 struct dlm_ckv_bucket;
 struct dlm_ckv_lock;
+struct dlm_ckv_kv;
+
+#define DLM_CKV_VALUE_MAX_SIZE	255
 
 typedef void (*dlm_ckv_notify_cb)(void *userarg);
 
@@ -18,6 +21,12 @@ void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);
 int dlm_ckv_lock_get(struct dlm_ckv_lock *ckv_lock);
 int dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock);
 
+struct dlm_ckv_kv *
+dlm_ckv_create_kv(struct dlm_ckv_bucket *bucket, const char *key);
+void dlm_ckv_free_kv(struct dlm_ckv_kv *kv);
+int dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len);
+int dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len);
+
 struct dlm_ckv_notify *
 dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
 			    dlm_ckv_notify_cb cb);
Signature · 1 line
2.25.1
#34 lore raw

RFC PATCH 34/48 dlm_ckv: add KV get/set async API

In reply to Dmitry Bogdanov

The sync versions of KV set/get functions may take too long time if user want to store/read a number of keys.
Add async versions of KV get/set functions to allow the user to group the requests and to wait for completion of all requests together.

---
 drivers/target/dlm_ckv.c | 110 ++++++++++++++++++++++++++++++++++++++-
 drivers/target/dlm_ckv.h |   6 +++
 2 files changed, 115 insertions(+), 1 deletion(-)
drivers/target/dlm_ckv.c +109 -1
index 22c5f0827595..417159f18fc6 100644
@@ -14,15 +14,27 @@
 #include <target/target_core_base.h>
 #include "dlm_ckv.h"
 
+enum dlm_lksb_state {
+	ONE_STAGE_SYNC,
+	READ_FIRST_STAGE,
+	READ_SECOND_STAGE,
+	WRITE_FIRST_STAGE,
+	WRITE_SECOND_STAGE,
+};
 struct dlm_ckv_lksb {
 	struct dlm_lksb lksb;
 	struct completion compl;
+	enum dlm_lksb_state state;
 };
 
 struct dlm_ckv_lock {
 	struct dlm_ckv_bucket *bucket;
 	struct dlm_ckv_lksb lksb;
 	char name[DLM_RESNAME_MAXLEN];
+	dlm_ckv_async_cb async_cb;
+	void *cb_arg;
+	void *value;
+	size_t len;
 };
 
 struct dlm_ckv_kv {
@@ -97,8 +109,48 @@ static const struct dlm_lockspace_ops dlm_ckv_lockspace_ops = {
 static void dlm_ast(void *astarg)
 {
 	struct dlm_ckv_lksb *dlm_ckv_lksb = astarg;
+	struct dlm_ckv_lock *ckv_lock;
+	struct dlm_ckv_bucket *bucket;
+	int res;
+
+	ckv_lock = container_of(dlm_ckv_lksb, struct dlm_ckv_lock, lksb);
+	bucket = ckv_lock->bucket;
 
-	complete(&dlm_ckv_lksb->compl);
+	switch (dlm_ckv_lksb->state) {
+	case ONE_STAGE_SYNC:
+		complete(&dlm_ckv_lksb->compl);
+		break;
+	case READ_FIRST_STAGE:
+		if (dlm_ckv_lksb->lksb.sb_flags & DLM_SBF_VALNOTVALID) {
+			pr_debug("%s LVB was invalid\n", ckv_lock->name);
+			memset(ckv_lock->value, 0, ckv_lock->len);
+		} else
+			memcpy(ckv_lock->value, dlm_ckv_lksb->lksb.sb_lvbptr,
+			       ckv_lock->len);
+
+		dlm_ckv_lksb->state = READ_SECOND_STAGE;
+		res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_NL,
+			       &dlm_ckv_lksb->lksb, DLM_LKF_CONVERT, 0, 0, 0,
+			       dlm_ast, dlm_ckv_lksb, NULL);
+		if (res)
+			ckv_lock->async_cb(ckv_lock->cb_arg, res);
+		break;
+	case WRITE_FIRST_STAGE:
+
+		dlm_ckv_lksb->state = WRITE_SECOND_STAGE;
+		res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_NL,
+			       &dlm_ckv_lksb->lksb,
+			       DLM_LKF_VALBLK | DLM_LKF_CONVERT, 0, 0, 0,
+			       dlm_ast, dlm_ckv_lksb, NULL);
+		if (res)
+			ckv_lock->async_cb(ckv_lock->cb_arg, res);
+		break;
+	case READ_SECOND_STAGE:
+		fallthrough;
+	case WRITE_SECOND_STAGE:
+		ckv_lock->async_cb(ckv_lock->cb_arg, dlm_ckv_lksb->lksb.sb_status);
+		break;
+	}
 }
 
 /*
@@ -135,6 +187,7 @@ static int dlm_ckv_lock_wait(dlm_lockspace_t *ls, int mode,
 {
 	int res;
 
+	lksb->state = ONE_STAGE_SYNC;
 	res = dlm_lock(ls, mode, &lksb->lksb, flags,
 		       (void *)name, name ? strlen(name) : 0, 0,
 		       dlm_ast, lksb, bast);
@@ -164,6 +217,7 @@ static int dlm_ckv_unlock_wait(dlm_lockspace_t *ls, struct dlm_ckv_lksb *lksb)
 {
 	int res;
 
+	lksb->state = ONE_STAGE_SYNC;
 	res = dlm_unlock(ls, lksb->lksb.sb_lkid, 0, &lksb->lksb, lksb);
 	if (res < 0)
 		goto out;
@@ -333,6 +387,32 @@ dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len)
 }
 EXPORT_SYMBOL(dlm_ckv_get);
 
+int
+dlm_ckv_get_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+		  dlm_ckv_async_cb cb, void *cb_arg)
+{
+	struct dlm_ckv_lock *ckv_lock = &kv->lock;
+	struct dlm_ckv_bucket *bucket;
+	int res;
+
+	BUG_ON(!ckv_lock);
+	bucket = ckv_lock->bucket;
+
+	ckv_lock->lksb.state = READ_FIRST_STAGE;
+	ckv_lock->len = len;
+	ckv_lock->value = value;
+	ckv_lock->async_cb = cb;
+	ckv_lock->cb_arg = cb_arg;
+	res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_CR, &ckv_lock->lksb.lksb,
+		       DLM_LKF_VALBLK | DLM_LKF_CONVERT, NULL, 0, 0,
+		       dlm_ast, &ckv_lock->lksb, NULL);
+	if (res)
+		pr_info("Can not get lock %s, rc=%d\n", ckv_lock->name, res);
+
+	return res;
+}
+EXPORT_SYMBOL(dlm_ckv_get_async);
+
 int
 dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len)
 {
@@ -368,6 +448,34 @@ dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len)
 }
 EXPORT_SYMBOL(dlm_ckv_set);
 
+int
+dlm_ckv_set_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+		  dlm_ckv_async_cb cb, void *cb_arg)
+{
+	struct dlm_ckv_lock *ckv_lock = &kv->lock;
+	struct dlm_ckv_bucket *bucket;
+	int res;
+
+	BUG_ON(!ckv_lock);
+	bucket = ckv_lock->bucket;
+
+	ckv_lock->lksb.state = WRITE_FIRST_STAGE;
+	ckv_lock->len = len;
+	ckv_lock->value = value;
+	ckv_lock->async_cb = cb;
+	ckv_lock->cb_arg = cb_arg;
+	memcpy(ckv_lock->lksb.lksb.sb_lvbptr, ckv_lock->value, ckv_lock->len);
+
+	res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_EX, &ckv_lock->lksb.lksb,
+		       DLM_LKF_CONVERT, NULL, 0, 0,
+		       dlm_ast, &ckv_lock->lksb, NULL);
+	if (res)
+		pr_info("Can not get lock %s\n", ckv_lock->name);
+
+	return res;
+}
+EXPORT_SYMBOL(dlm_ckv_set_async);
+
 static void dlm_cvk_pre_n_bast(void *astarg, int mode)
 {
 	struct dlm_ckv_lksb *lksb = astarg;
drivers/target/dlm_ckv.h +6 -0
index c01904313f1e..e8045917067e 100644
@@ -9,6 +9,8 @@ struct dlm_ckv_kv;
 #define DLM_CKV_VALUE_MAX_SIZE	255
 
 typedef void (*dlm_ckv_notify_cb)(void *userarg);
+typedef void (*dlm_ckv_async_cb)(void *userarg, int res);
+typedef void (*dlm_ckv_nodeleft_cb)(void *arg, int nodeid);
 
 struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
 					   const char *cluster_name,
@@ -26,6 +28,10 @@ dlm_ckv_create_kv(struct dlm_ckv_bucket *bucket, const char *key);
 void dlm_ckv_free_kv(struct dlm_ckv_kv *kv);
 int dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len);
 int dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len);
+int dlm_ckv_get_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+		      dlm_ckv_async_cb cb, void *cb_arg);
+int dlm_ckv_set_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+		      dlm_ckv_async_cb cb, void *cb_arg);
 
 struct dlm_ckv_notify *
 dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
Signature · 1 line
2.25.1
#35 lore raw

RFC PATCH 35/48 target: add virtual remote target

In reply to Dmitry Bogdanov

Create virtual remote target module.
It can be used to see a whole acl/lun/tpg configuration from all nodes in storage cluster.

---
 drivers/target/Kconfig                 |   1 +
 drivers/target/Makefile                |   1 +
 drivers/target/tcm_remote/Kconfig      |   8 +
 drivers/target/tcm_remote/Makefile     |   2 +
 drivers/target/tcm_remote/tcm_remote.c | 405 +++++++++++++++++++++++++
 drivers/target/tcm_remote/tcm_remote.h |  29 ++
 6 files changed, 446 insertions(+)
 create mode 100644 drivers/target/tcm_remote/Kconfig
 create mode 100644 drivers/target/tcm_remote/Makefile
 create mode 100644 drivers/target/tcm_remote/tcm_remote.c
 create mode 100644 drivers/target/tcm_remote/tcm_remote.h
drivers/target/Kconfig +1 -0
index 75d5e1d23a1c..5440c0e93e1e 100644
@@ -53,5 +53,6 @@ source "drivers/target/loopback/Kconfig"
 source "drivers/target/tcm_fc/Kconfig"
 source "drivers/target/iscsi/Kconfig"
 source "drivers/target/sbp/Kconfig"
+source "drivers/target/tcm_remote/Kconfig"
 
 endif
drivers/target/Makefile +1 -0
index 8bc9ac2bd629..be4d1bfcf79a 100644
@@ -30,5 +30,6 @@ obj-$(CONFIG_LOOPBACK_TARGET)	+= loopback/
 obj-$(CONFIG_TCM_FC)		+= tcm_fc/
 obj-$(CONFIG_ISCSI_TARGET)	+= iscsi/
 obj-$(CONFIG_SBP_TARGET)	+= sbp/
+obj-$(CONFIG_REMOTE_TARGET)	+= tcm_remote/
 
 obj-$(CONFIG_DLM_CKV)			+= dlm_ckv.o
drivers/target/tcm_remote/Kconfig new file +8 -0
new file mode 100644
index 000000000000..e6bebb5fe6f1
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config REMOTE_TARGET
+	tristate "TCM Virtual Remote target"
+	depends on SCSI
+	help
+	  Say Y here to enable the TCM Virtual Remote fabric
+	  That fabric is a dummy fabric to tell TCM about configuration
+	  of TPG/ACL/LUN on peer nodes in a cluster.
drivers/target/tcm_remote/Makefile new file +2 -0
new file mode 100644
index 000000000000..5818ffd0b0fa
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_REMOTE_TARGET)	+= tcm_remote.o
drivers/target/tcm_remote/tcm_remote.c new file +405 -0

405 lines not on this page · raw message

drivers/target/tcm_remote/tcm_remote.h new file +29 -0
new file mode 100644
index 000000000000..8d674c87989e
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/types.h>
+#include <linux/device.h>
+
+#define TCM_REMOTE_VERSION		"v0.1"
+#define TL_WWN_ADDR_LEN			256
+#define TL_TPGS_PER_HBA			32
+
+
+#define TCM_TRANSPORT_ONLINE 0
+#define TCM_TRANSPORT_OFFLINE 1
+
+struct tcm_remote_tpg {
+	unsigned short remote_tpgt;
+	enum target_prot_type remote_fabric_prot_type;
+	atomic_t remote_tpg_port_count;
+	struct se_portal_group remote_se_tpg;
+	struct tcm_remote_hba *remote_hba;
+};
+
+struct tcm_remote_hba {
+	u8 remote_proto_id;
+	unsigned char remote_wwn_address[TL_WWN_ADDR_LEN];
+	struct se_hba_s *se_hba;
+	struct se_lun *remote_hba_lun;
+	struct se_port *remote_hba_lun_sep;
+	struct tcm_remote_tpg remote_hba_tpgs[TL_TPGS_PER_HBA];
+	struct se_wwn remote_hba_wwn;
+};
Signature · 1 line
2.25.1
#36 lore raw

RFC PATCH 36/48 target: cluster: introduce cluster ops

In reply to Dmitry Bogdanov

Introduce cluster ops common for all backend devices. Those ops will implement cluster level locks, store clusterwide data (like persistent reservations) and provide clusterwide notifications. That ops could be registered dynamically. Ops have own configfs folder in .../target/cluster/<cluster_impl_name>

Create default non-cluster ops implementation.

Use cluster ops for CAW lock.

---
 drivers/target/target_core_configfs.c |  63 +++++++++++-
 drivers/target/target_core_device.c   | 139 +++++++++++++++++++++++++-
 drivers/target/target_core_internal.h |   8 ++
 drivers/target/target_core_sbc.c      |  12 +--
 include/target/target_core_base.h     |  18 ++++
 5 files changed, 232 insertions(+), 8 deletions(-)
drivers/target/target_core_configfs.c +62 -1
index f63c37344104..6c36b1f4bd92 100644
@@ -69,6 +69,7 @@ static LIST_HEAD(g_tf_list);
 static DEFINE_MUTEX(g_tf_lock);
 
 static struct config_group target_core_hbagroup;
+static struct config_group *target_core_cluster_group;
 static struct config_group alua_group;
 static struct config_group alua_lu_gps_group;
 
@@ -1193,6 +1194,32 @@ static ssize_t pgr_support_store(struct config_item *item,
 	return count;
 }
 
+static ssize_t cluster_impl_show(struct config_item *item, char *page)
+{
+	return snprintf(page, PAGE_SIZE, "%s\n",
+			to_attrib(item)->da_dev->cl_ops->name);
+}
+
+static ssize_t cluster_impl_store(struct config_item *item,
+				  const char *page, size_t count)
+{
+	struct se_dev_attrib *da = to_attrib(item);
+	struct se_device *dev = da->da_dev;
+	char cluster_impl[16];
+	int res;
+
+	strscpy(cluster_impl, page, 16);
+	if (cluster_impl[strlen(cluster_impl) - 1] == '\n')
+		cluster_impl[strlen(cluster_impl) - 1] = 0;
+
+	res = core_dev_set_cluster_mode(dev, cluster_impl);
+	if (res)
+		return res;
+
+	return count;
+}
+CONFIGFS_ATTR(, cluster_impl);
+
 CONFIGFS_ATTR(, emulate_model_alias);
 CONFIGFS_ATTR(, emulate_dpo);
 CONFIGFS_ATTR(, emulate_fua_write);
@@ -1268,6 +1295,7 @@ struct configfs_attribute *sbc_attrib_attrs[] = {
 	&attr_max_write_same_len,
 	&attr_alua_support,
 	&attr_pgr_support,
+	&attr_cluster_impl,
 	NULL,
 };
 EXPORT_SYMBOL(sbc_attrib_attrs);
@@ -3615,6 +3643,38 @@ static const struct config_item_type target_core_cit = {
 
 /* Stop functions for struct config_item_type target_core_hba_cit */
 
+static const struct config_item_type target_core_cluster_cit = {
+	.ct_owner	= THIS_MODULE,
+};
+
+int target_core_register_cluster_configfs(struct target_cluster_impl *tci)
+{
+	int ret = 0;
+
+	if (!target_core_cluster_group) {
+		target_core_cluster_group = configfs_register_default_group(
+					&target_core_fabrics.su_group,
+					"cluster",
+					&target_core_cluster_cit);
+		if (IS_ERR(target_core_cluster_group)) {
+			ret = PTR_ERR(target_core_cluster_group);
+			pr_err("Error %d while registering cluster group\n", ret);
+			goto err;
+		}
+	}
+
+	ret = configfs_register_group(target_core_cluster_group,
+				      tci->ops->cg_group);
+
+err:
+	return ret;
+}
+
+void target_core_unregister_cluster_configfs(struct target_cluster_impl *tci)
+{
+	configfs_unregister_group(tci->ops->cg_group);
+}
+
 void target_setup_backend_cits(struct target_backend *tb)
 {
 	target_core_setup_dev_cit(tb);
@@ -3749,7 +3809,8 @@ static void __exit target_core_exit_configfs(void)
 	configfs_remove_default_groups(&alua_lu_gps_group);
 	configfs_remove_default_groups(&alua_group);
 	configfs_remove_default_groups(&target_core_hbagroup);
-
+	if (target_core_cluster_group)
+		configfs_remove_default_groups(target_core_cluster_group);
 	/*
 	 * We expect subsys->su_group.default_groups to be released
 	 * by configfs subsystem provider logic..
drivers/target/target_core_device.c +138 -1

175 lines not on this page · raw message

drivers/target/target_core_internal.h +8 -0
index b0d29c30ab1e..58261c716776 100644
@@ -55,6 +55,11 @@ struct target_fabric_configfs {
 	struct config_item_type tf_tpg_mappedlun_stat_cit;
 };
 
+struct target_cluster_impl {
+	const struct target_cluster_ops *ops;
+	struct list_head list;
+};
+
 /* target_core_alua.c */
 extern struct t10_alua_lu_gp *default_lu_gp;
 
@@ -83,6 +88,7 @@ void	core_dev_free_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_lun_acl *lacl);
 int	core_dev_setup_virtual_lun0(void);
 void	core_dev_release_virtual_lun0(void);
+int	core_dev_set_cluster_mode(struct se_device *dev, const char *cluster_impl);
 struct se_device *target_alloc_device(struct se_hba *hba, const char *name);
 int	target_configure_device(struct se_device *dev);
 void	target_free_device(struct se_device *);
@@ -93,6 +99,8 @@ void	target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
 /* target_core_configfs.c */
 extern struct configfs_item_operations target_core_dev_item_ops;
 void	target_setup_backend_cits(struct target_backend *);
+int	target_core_register_cluster_configfs(struct target_cluster_impl *tci);
+void	target_core_unregister_cluster_configfs(struct target_cluster_impl *tci);
 
 /* target_core_fabric_configfs.c */
 int	target_fabric_setup_cits(struct target_fabric_configfs *);
drivers/target/target_core_sbc.c +6 -6
index 786ca58008b0..ab2038a510c0 100644
@@ -367,7 +367,7 @@ static sense_reason_t compare_and_write_post(struct se_cmd *cmd, bool success,
 	 * Unlock ->caw_sem originally obtained during sbc_compare_and_write()
 	 * before the original READ I/O submission.
 	 */
-	up(&dev->caw_sem);
+	dev->cl_ops->caw_unlock(dev, cmd->cawlock);
 
 	return ret;
 }
@@ -550,7 +550,7 @@ static sense_reason_t compare_and_write_callback(struct se_cmd *cmd, bool succes
 	 * In the MISCOMPARE or failure case, unlock ->caw_sem obtained in
 	 * sbc_compare_and_write() before the original READ I/O submission.
 	 */
-	up(&dev->caw_sem);
+	dev->cl_ops->caw_unlock(dev, cmd->cawlock);
 	sg_free_table(&write_tbl);
 	return ret;
 }
@@ -561,13 +561,13 @@ sbc_compare_and_write(struct se_cmd *cmd)
 	struct sbc_ops *ops = cmd->protocol_data;
 	struct se_device *dev = cmd->se_dev;
 	sense_reason_t ret;
-	int rc;
+
 	/*
 	 * Submit the READ first for COMPARE_AND_WRITE to perform the
 	 * comparision using SGLs at cmd->t_bidi_data_sg..
 	 */
-	rc = down_interruptible(&dev->caw_sem);
-	if (rc != 0) {
+	cmd->cawlock = dev->cl_ops->caw_lock(dev, cmd->t_task_lba);
+	if (!cmd->cawlock) {
 		cmd->transport_complete_callback = NULL;
 		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 	}
@@ -582,7 +582,7 @@ sbc_compare_and_write(struct se_cmd *cmd)
 			      DMA_FROM_DEVICE);
 	if (ret) {
 		cmd->transport_complete_callback = NULL;
-		up(&dev->caw_sem);
+		dev->cl_ops->caw_unlock(dev, cmd->cawlock);
 		return ret;
 	}
 	/*
include/target/target_core_base.h +18 -0
index 5378e2d3825f..fbeb4777654b 100644
@@ -546,6 +546,7 @@ struct se_cmd {
 	 * initialized on. Drivers can override.
 	 */
 	int			cpuid;
+	void			*cawlock;
 };
 
 struct se_ua {
@@ -777,6 +778,17 @@ struct se_device_queue {
 	struct se_cmd_queue	sq;
 };
 
+struct target_cluster_ops {
+	char name[16];
+	struct module *owner;
+	struct config_group *cg_group;
+
+	int (*init)(struct se_device *dev);
+	int (*cleanup)(struct se_device *dev);
+	void *(*caw_lock)(struct se_device *dev, u64 lba);
+	int (*caw_unlock)(struct se_device *dev, void *lock);
+};
+
 struct se_device {
 	/* Used for SAM Task Attribute ordering */
 	u32			dev_cur_ordered_id;
@@ -847,6 +859,9 @@ struct se_device {
 	unsigned char		udev_path[SE_UDEV_PATH_LEN];
 	/* Pointer to template of function pointers for transport */
 	const struct target_backend_ops *transport;
+	/* Pointer to template of function pointers for cluster mode */
+	const struct target_cluster_ops *cl_ops;
+	void			*cluster_data;
 	struct se_lun		xcopy_lun;
 	/* Protection Information */
 	int			prot_length;
@@ -981,4 +996,7 @@ static inline void target_free_tag(struct se_session *sess, struct se_cmd *cmd)
 	sbitmap_queue_clear(&sess->sess_tag_pool, cmd->map_tag, cmd->map_cpu);
 }
 
+int target_cluster_impl_register(const struct target_cluster_ops *ops);
+void target_cluster_impl_unregister(const struct target_cluster_ops *ops);
+
 #endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
#37 lore raw

RFC PATCH 37/48 target: cluster: introduce dlm cluster

In reply to Dmitry Bogdanov

Introduce DLM Cluster implementation of cluster ops of backstore devices.

Use one DLM lockspace per device. The device is idenitfied by concatenation of T10 model and T10 unit_serial.

Use DLM CKV lock for Compare And Write in cluster mode. One lock for one LBA. That allows a parallell execution of CAW commands.

User should set configfs/target/cluster/dlm/cluster_name identical to configfs/dlm/cluster/cluster_name.

---
 drivers/target/Makefile             |   1 +
 drivers/target/target_cluster_dlm.c | 175 ++++++++++++++++++++++++++++
 2 files changed, 176 insertions(+)
 create mode 100644 drivers/target/target_cluster_dlm.c
drivers/target/Makefile +1 -0
index be4d1bfcf79a..16b625108eec 100644
@@ -33,3 +33,4 @@ obj-$(CONFIG_SBP_TARGET)	+= sbp/
 obj-$(CONFIG_REMOTE_TARGET)	+= tcm_remote/
 
 obj-$(CONFIG_DLM_CKV)			+= dlm_ckv.o
+obj-$(CONFIG_DLM_CKV)			+= target_cluster_dlm.o
drivers/target/target_cluster_dlm.c new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.25.1
#38 lore raw

RFC PATCH 38/48 target: cluster: store PR data in DLM cluster

In reply to Dmitry Bogdanov

Follow the sequence bellow to sync all PR data throughout nodes in cluster:

	pr_lock()
	Handle PR OUT locally
	if (success)
		pr_sync()
	pr_unlock()

where pr_sync() is a synchronical function.
That will garantee an atomic update of PR data in a cluster.

---
 drivers/target/target_cluster_dlm.c     | 247 +++++++++++++++++++++++-
 drivers/target/target_core_device.c     |   8 +
 drivers/target/target_core_fabric_lib.c |   1 +
 drivers/target/target_core_internal.h   |   2 -
 drivers/target/target_core_pr.c         |   7 +
 include/target/target_core_base.h       |   6 +
 6 files changed, 268 insertions(+), 3 deletions(-)
drivers/target/target_cluster_dlm.c +246 -1
index 2edae188811d..14465764eaaf 100644
@@ -5,12 +5,21 @@
 #include <linux/module.h>
 #include <linux/configfs.h>
 #include <target/target_core_base.h>
+#include <target/target_core_fabric.h>
+#include <target/target_core_backend.h>
+#include "target_core_pr.h"
 
 #include "dlm_ckv.h"
 
 struct target_cluster_data {
 	struct dlm_ckv_bucket *bucket;
 	u32 local_nodeid;
+	struct mutex pr_lock_mutex;
+	struct dlm_ckv_lock *pr_lock;
+	struct dlm_ckv_kv *pr_data;
+	int reserved_node_id;
+	struct dlm_ckv_kv **pr_reg_kv;
+	size_t pr_reg_kv_len;
 };
 
 struct target_lksb {
@@ -60,6 +69,44 @@ static const struct config_item_type ci_cluster_config = {
 };
 
 static struct target_cluster_config cluster_cfg;
+struct pr_lvb {
+	u32	nr_registrants;
+	u32	pr_generation;
+	u8	version;
+	u8	pr_is_set;
+	u8	pr_type;
+	u8	pr_scope;
+	u8	pr_aptpl;
+	u8	reserved[3];
+	u32	reserved_by_nodeid;
+};
+
+struct pr_reg_lvb {
+	u64	key;
+	u16	rtpi;
+	u8	version;
+	u8	is_holder;
+	u8	is_all_tg_pt;
+	u8	tid[PR_REG_TID_LEN];
+} __packed ; /* to fit in 256 bytes */
+
+struct async_group {
+	refcount_t pending;
+	atomic_t status;
+	struct completion compl;
+};
+
+static int pr_reg_realloc(struct target_cluster_data *cluster_data,
+			  size_t nr_registrants);
+
+static void group_compl_cb(void *arg, int res)
+{
+	struct async_group *grp = arg;
+
+	atomic_cmpxchg(&grp->status, 0, res);
+	if (refcount_dec_and_test(&grp->pending))
+		complete(&grp->compl);
+}
 
 static int target_init_dlm(struct se_device *dev)
 {
@@ -68,6 +115,7 @@ static int target_init_dlm(struct se_device *dev)
 	int err = 0;
 
 	BUG_ON(dev->cluster_data);
+	BUILD_BUG_ON(sizeof(struct pr_reg_lvb) > DLM_CKV_VALUE_MAX_SIZE);
 
 	snprintf(ls_name, sizeof(ls_name), "%s%s",
 			       dev->t10_wwn.model, dev->t10_wwn.unit_serial);
@@ -85,16 +133,42 @@ static int target_init_dlm(struct se_device *dev)
 		dev->cluster_data = NULL;
 		return err;
 	}
+
+	mutex_init(&cluster_data->pr_lock_mutex);
+	cluster_data->pr_lock = dlm_ckv_create_lock(cluster_data->bucket, "pr_lock");
+	if (!cluster_data->pr_lock)
+		goto fail;
+
+	cluster_data->pr_data = dlm_ckv_create_kv(cluster_data->bucket, "pr_data");
+	if (!cluster_data->pr_data)
+		goto fail;
+
 	dev->cluster_data = cluster_data;
 
 	return err;
+fail:
+	if (cluster_data->pr_lock)
+		dlm_ckv_free_lock(cluster_data->pr_lock);
+	if (cluster_data->pr_data)
+		dlm_ckv_free_kv(cluster_data->pr_data);
+	dlm_ckv_close_bucket(cluster_data->bucket);
+	kfree(cluster_data);
+	return -EIO;
 }
 
 static int target_cleanup_dlm(struct se_device *dev)
 {
 	struct target_cluster_data *cluster_data = dev->cluster_data;
 	int res;
+	int i;
+
+	for (i = 0; i < cluster_data->pr_reg_kv_len; i++)
+		if (cluster_data->pr_reg_kv[i])
+			dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
+	kfree(cluster_data->pr_reg_kv);
 
+	dlm_ckv_free_lock(cluster_data->pr_lock);
+	dlm_ckv_free_kv(cluster_data->pr_data);
 	res = dlm_ckv_close_bucket(cluster_data->bucket);
 	if (res)
 		pr_err("TARGET_CORE[%d]:  closing bucket failed: %d\n",
@@ -142,7 +216,175 @@ static int target_caw_unlock_dlm(struct se_device *dev, void *lock)
 	return res;
 }
 
-struct target_cluster_ops dlm_cluster_ops = {
+static int target_pr_lock_dlm(struct se_device *dev)
+{
+	struct target_cluster_data *cluster_data = dev->cluster_data;
+	int res = -ENOMEM;
+
+	mutex_lock(&cluster_data->pr_lock_mutex);
+	res = dlm_ckv_lock_get(cluster_data->pr_lock);
+	if (res) {
+		pr_warn("target_dlm_lock_wait failed with %d\n", res);
+		mutex_unlock(&cluster_data->pr_lock_mutex);
+	}
+
+	return res;
+}
+
+static int target_pr_unlock_dlm(struct se_device *dev)
+{
+	struct target_cluster_data *cluster_data = dev->cluster_data;
+	int res;
+
+	res = dlm_ckv_lock_release(cluster_data->pr_lock);
+	mutex_unlock(&cluster_data->pr_lock_mutex);
+
+	return res;
+}
+
+static int target_get_nr_registrants(struct se_device *dev)
+{
+	struct t10_pr_registration *pr_reg;
+	int count = 0;
+
+	list_for_each_entry(pr_reg,
+			&dev->t10_pr.registration_list, pr_reg_list)
+		count++;
+
+	return count;
+}
+
+static int pr_reg_realloc(struct target_cluster_data *cluster_data,
+			  size_t nr_registrants)
+{
+	char key_name[16];
+	void *p;
+	int res = 0;
+	int i;
+
+	if (nr_registrants > cluster_data->pr_reg_kv_len) {
+		p = krealloc(cluster_data->pr_reg_kv,
+			     nr_registrants * sizeof(struct dlm_ckv_kv *),
+			     GFP_KERNEL | __GFP_ZERO);
+		if (!p) {
+			res = -ENOMEM;
+			goto done;
+		}
+		cluster_data->pr_reg_kv = p;
+		for (i = cluster_data->pr_reg_kv_len;
+		     i < nr_registrants; i++) {
+			snprintf(key_name, sizeof(key_name), "pr_reg.%d", i);
+			if (!cluster_data->pr_reg_kv[i])
+				cluster_data->pr_reg_kv[i] = dlm_ckv_create_kv(
+								cluster_data->bucket,
+								key_name);
+			if (!cluster_data->pr_reg_kv[i]) {
+				res = -ENOMEM;
+				goto fail_create;
+			}
+		}
+		cluster_data->pr_reg_kv_len = nr_registrants;
+	}
+done:
+	return res;
+fail_create:
+	/* free newly created objects */
+	for (i = cluster_data->pr_reg_kv_len;
+	     i < nr_registrants; i++) {
+		if (!cluster_data->pr_reg_kv[i])
+			dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
+	}
+	return res;
+}
+
+static int target_pr_sync_dlm(struct se_device *dev)
+{
+	struct target_cluster_data *cluster_data = dev->cluster_data;
+	struct t10_pr_registration *pr_reg;
+	struct pr_reg_lvb pr_reg_data;
+	struct async_group grp;
+	struct pr_lvb pr_data;
+	int i = 0;
+	int res;
+
+	spin_lock(&dev->t10_pr.registration_lock);
+
+	pr_data.version = 1;
+	pr_data.pr_generation = atomic_read(&dev->t10_pr.pr_generation);
+	pr_data.nr_registrants = target_get_nr_registrants(dev);
+	pr_data.pr_is_set = !!dev->dev_pr_res_holder;
+	pr_data.pr_aptpl = dev->t10_pr.pr_aptpl_active;
+	if (dev->dev_pr_res_holder) {
+		pr_data.pr_type = dev->dev_pr_res_holder->pr_res_type;
+		pr_data.pr_scope = dev->dev_pr_res_holder->pr_res_scope;
+	}
+	pr_data.reserved_by_nodeid = cluster_data->reserved_node_id;
+
+	spin_unlock(&dev->t10_pr.registration_lock);
+
+	res = dlm_ckv_set(cluster_data->pr_data, (char *)&pr_data,
+			  sizeof(pr_data));
+	if (res)
+		goto done;
+
+
+	res = pr_reg_realloc(cluster_data, pr_data.nr_registrants);
+	if (res)
+		goto done;
+
+	if (pr_data.nr_registrants == 0)
+		goto skip_pr_reg;
+
+	refcount_set(&grp.pending, 1); /* 1 for a loop */
+	atomic_set(&grp.status, 0);
+	init_completion(&grp.compl);
+
+	list_for_each_entry(pr_reg,
+			&dev->t10_pr.registration_list, pr_reg_list) {
+
+		pr_reg_data.key = pr_reg->pr_res_key;
+		pr_reg_data.rtpi = pr_reg->tg_pt_sep_rtpi;
+		pr_reg_data.version = 1;
+		pr_reg_data.is_holder = pr_reg->pr_res_holder;
+		pr_reg_data.is_all_tg_pt = !!pr_reg->pr_reg_all_tg_pt;
+		target_get_pr_transport_id(pr_reg, pr_reg_data.tid);
+
+		refcount_inc(&grp.pending);
+		res = dlm_ckv_set_async(cluster_data->pr_reg_kv[i],
+					(char *)&pr_reg_data,
+					sizeof(pr_reg_data),
+					group_compl_cb,
+					&grp);
+		if (res) {
+			pr_err("TARGET_CORE[%d]: set pr_reg_%d failed: %d\n",
+			       dev->dev_index, i, res);
+			refcount_dec(&grp.pending);
+			break;
+		}
+		i++;
+	}
+	group_compl_cb(&grp, 0);
+	res = wait_for_completion_timeout(&grp.compl, 60 * HZ);
+	if (!res) {
+		pr_err("TARGET_CORE[%d]: timeout of waiting for dlm_ckv_set_async\n",
+			dev->dev_index);
+		goto done;
+	}
+	res = atomic_read(&grp.status);
+	if (res) {
+		pr_err("TARGET_CORE[%d]: fail of group for dlm_ckv_set_async %d\n",
+			dev->dev_index, res);
+		goto done;
+	}
+
+skip_pr_reg:
+
+done:
+
+	return res;
+}
+
+const struct target_cluster_ops dlm_cluster_ops = {
 	.name = "dlm",
 	.owner = THIS_MODULE,
 	.cg_group = &cluster_cfg.cg_group,
@@ -151,6 +393,9 @@ struct target_cluster_ops dlm_cluster_ops = {
 	.cleanup = target_cleanup_dlm,
 	.caw_lock = target_caw_lock_dlm,
 	.caw_unlock = target_caw_unlock_dlm,
+	.pr_lock = target_pr_lock_dlm,
+	.pr_unlock = target_pr_unlock_dlm,
+	.pr_sync = target_pr_sync_dlm,
 };
 
 static int __init target_cluster_dlm_module_init(void)
drivers/target/target_core_device.c +8 -0
index 9c7f4dc1b85d..f5da3fc17ad1 100644
@@ -782,6 +782,11 @@ static int target_caw_unlock_nodlm(struct se_device *dev, void *lock)
 	return 0;
 }
 
+int target_dummy_nodlm(struct se_device *dev)
+{
+	return 0;
+}
+
 const struct target_cluster_ops nodlm_cluster_ops = {
 	.name = "single",
 	.owner = NULL,
@@ -789,6 +794,9 @@ const struct target_cluster_ops nodlm_cluster_ops = {
 	.cleanup = target_cleanup_nodlm,
 	.caw_lock = target_caw_lock_nodlm,
 	.caw_unlock = target_caw_unlock_nodlm,
+	.pr_lock = target_dummy_nodlm,
+	.pr_unlock = target_dummy_nodlm,
+	.pr_sync = target_dummy_nodlm,
 };
 
 struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
drivers/target/target_core_fabric_lib.c +1 -0
index 0eb03d7b9e78..8232d872db35 100644
@@ -382,6 +382,7 @@ int target_get_pr_transport_id(
 
 	return  len;
 }
+EXPORT_SYMBOL(target_get_pr_transport_id);
 
 int target_cmp_pr_transport_id(
 		struct t10_pr_registration *pr_reg,
drivers/target/target_core_internal.h +0 -2
index 58261c716776..8669ff6f48e9 100644
@@ -107,8 +107,6 @@ int	target_fabric_setup_cits(struct target_fabric_configfs *);
 
 /* target_core_fabric_lib.c */
 int	target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
-int	target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
-				   unsigned char *buf);
 int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
 			       unsigned char *tid);
 int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
drivers/target/target_core_pr.c +7 -0
index a984987f0a82..a9fe192f5fe4 100644
@@ -3560,6 +3560,8 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)
 		return TCM_PARAMETER_LIST_LENGTH_ERROR;
 	}
 
+	if (dev->cl_ops->pr_lock(dev))
+		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 	/*
 	 * (core_scsi3_emulate_pro_* function parameters
 	 * are defined by spc4r17 Table 174:
@@ -3598,9 +3600,14 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)
 	default:
 		pr_err("Unknown PERSISTENT_RESERVE_OUT service"
 			" action: 0x%02x\n", sa);
+		dev->cl_ops->pr_unlock(dev);
 		return TCM_INVALID_CDB_FIELD;
 	}
 
+	if (!ret)
+		dev->cl_ops->pr_sync(dev);
+	dev->cl_ops->pr_unlock(dev);
+
 	if (!ret)
 		target_complete_cmd(cmd, SAM_STAT_GOOD);
 	return ret;
include/target/target_core_base.h +6 -0
index fbeb4777654b..9edf3c1ac204 100644
@@ -787,6 +787,9 @@ struct target_cluster_ops {
 	int (*cleanup)(struct se_device *dev);
 	void *(*caw_lock)(struct se_device *dev, u64 lba);
 	int (*caw_unlock)(struct se_device *dev, void *lock);
+	int (*pr_lock)(struct se_device *dev);
+	int (*pr_unlock)(struct se_device *dev);
+	int (*pr_sync)(struct se_device *dev);
 };
 
 struct se_device {
@@ -999,4 +1002,7 @@ static inline void target_free_tag(struct se_session *sess, struct se_cmd *cmd)
 int target_cluster_impl_register(const struct target_cluster_ops *ops);
 void target_cluster_impl_unregister(const struct target_cluster_ops *ops);
 
+int	target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
+				   unsigned char *buf);
+
 #endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
#39 lore raw

RFC PATCH 39/48 target: cluster: read PR data from cluster

In reply to Dmitry Bogdanov

Notify other nodes in cluster that PR data updated.
On notification do the following under PR cluster lock to read PR data:
Update an existing registrants.
Remove an outdated registrants (absent in cluster data). Create new registrants.
Update Reservation status.

---
 drivers/target/target_cluster_dlm.c     | 304 ++++++++++++++++++++++++
 drivers/target/target_core_device.c     |   1 +
 drivers/target/target_core_fabric_lib.c |   3 +-
 drivers/target/target_core_internal.h   |   5 -
 drivers/target/target_core_pr.c         |   8 +-
 drivers/target/target_core_pr.h         |   1 +
 include/target/target_core_base.h       |   5 +
 7 files changed, 320 insertions(+), 7 deletions(-)
drivers/target/target_cluster_dlm.c +304 -0
index 14465764eaaf..6b7cb3175e3a 100644
@@ -17,6 +17,7 @@ struct target_cluster_data {
 	struct mutex pr_lock_mutex;
 	struct dlm_ckv_lock *pr_lock;
 	struct dlm_ckv_kv *pr_data;
+	struct dlm_ckv_notify *pr_sync_notify;
 	int reserved_node_id;
 	struct dlm_ckv_kv **pr_reg_kv;
 	size_t pr_reg_kv_len;
@@ -96,6 +97,7 @@ struct async_group {
 	struct completion compl;
 };
 
+static void target_pr_sync_cb(void *arg);
 static int pr_reg_realloc(struct target_cluster_data *cluster_data,
 			  size_t nr_registrants);
 
@@ -143,10 +145,17 @@ static int target_init_dlm(struct se_device *dev)
 	if (!cluster_data->pr_data)
 		goto fail;
 
+	cluster_data->pr_sync_notify = dlm_ckv_create_notification(
+		cluster_data->bucket, "pr_sync", target_pr_sync_cb);
+	if (!cluster_data->pr_sync_notify)
+		goto fail;
+
 	dev->cluster_data = cluster_data;
 
 	return err;
 fail:
+	if (cluster_data->pr_sync_notify)
+		dlm_ckv_free_notification(cluster_data->pr_sync_notify);
 	if (cluster_data->pr_lock)
 		dlm_ckv_free_lock(cluster_data->pr_lock);
 	if (cluster_data->pr_data)
@@ -167,6 +176,7 @@ static int target_cleanup_dlm(struct se_device *dev)
 			dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
 	kfree(cluster_data->pr_reg_kv);
 
+	dlm_ckv_free_notification(cluster_data->pr_sync_notify);
 	dlm_ckv_free_lock(cluster_data->pr_lock);
 	dlm_ckv_free_kv(cluster_data->pr_data);
 	res = dlm_ckv_close_bucket(cluster_data->bucket);
@@ -379,11 +389,305 @@ static int target_pr_sync_dlm(struct se_device *dev)
 
 skip_pr_reg:
 
+	/* request to update PR data on other nodes in cluster */
+	dlm_ckv_notify(cluster_data->pr_sync_notify);
+
 done:
 
 	return res;
 }
 
+static int target_update_pr_reg(struct se_device *dev,
+				struct t10_pr_registration *pr_reg,
+				const struct pr_lvb *pr_data,
+				const struct pr_reg_lvb *pr_reg_data,
+				struct t10_pr_registration **pr_reg_res_holder)
+{
+	/* update existing registrant */
+	pr_reg->pr_res_key = pr_reg_data->key;
+	pr_reg->pr_reg_all_tg_pt = pr_reg_data->is_all_tg_pt;
+
+	if (pr_reg_data->is_holder)
+		*pr_reg_res_holder = pr_reg;
+
+	return 0;
+}
+
+static struct t10_pr_registration *
+target_create_pr_reg(struct se_device *dev,
+		     struct pr_lvb *pr_data,
+		     struct pr_reg_lvb *pr_reg_data)
+{
+	struct t10_pr_registration *pr_reg = NULL;
+	struct se_lun *lun, *lun_tmp;
+	struct se_lun_acl *lacl_tmp;
+	struct se_node_acl *nacl_tmp;
+	struct se_dev_entry *deve;
+	char i_buf[PR_REG_ISID_ID_LEN];
+	u8 initiatorname[PR_REG_ISID_LEN];
+	char *isid = NULL;
+
+	target_parse_pr_out_transport_id(pr_reg_data->tid, initiatorname, &isid);
+
+	spin_lock(&dev->se_port_lock);
+	list_for_each_entry_safe(lun, lun_tmp, &dev->dev_sep_list, lun_dev_link) {
+		if (!percpu_ref_tryget_live(&lun->lun_ref))
+			continue;
+
+		if (lun->lun_tpg->tpg_rtpi != pr_reg_data->rtpi) {
+			percpu_ref_put(&lun->lun_ref);
+			continue;
+		}
+		spin_unlock(&dev->se_port_lock);
+
+		spin_lock(&lun->lun_deve_lock);
+		list_for_each_entry(deve, &lun->lun_deve_list, lun_link) {
+			if (!deve->se_lun_acl)
+				continue;
+
+			lacl_tmp = deve->se_lun_acl;
+			nacl_tmp = lacl_tmp->se_lun_nacl;
+
+			if (strcmp(nacl_tmp->initiatorname, initiatorname))
+				continue;
+
+			kref_get(&deve->pr_kref);
+			pr_reg = __core_scsi3_do_alloc_registration(
+					dev, nacl_tmp,
+					nacl_tmp->initiatorname, lun->unpacked_lun,
+					pr_reg_data->rtpi,
+					deve, lacl_tmp->mapped_lun,
+					isid,
+					pr_reg_data->key,
+					pr_reg_data->is_all_tg_pt,
+					pr_data->pr_aptpl);
+
+			percpu_ref_put(&lun->lun_ref);
+			spin_unlock(&lun->lun_deve_lock);
+			goto out;
+		}
+		percpu_ref_put(&lun->lun_ref);
+		spin_unlock(&lun->lun_deve_lock);
+		spin_lock(&dev->se_port_lock);
+	}
+	spin_unlock(&dev->se_port_lock);
+out:
+
+	if (!pr_reg) {
+		/* target not yet configured,
+		 * allocate registration like for APTPL case
+		 */
+		pr_reg = __core_scsi3_do_alloc_registration(
+				dev, NULL, initiatorname, 0,
+				pr_reg_data->rtpi,
+				NULL, 0,
+				isid,
+				pr_reg_data->key,
+				pr_reg_data->is_all_tg_pt,
+				pr_data->pr_aptpl);
+	}
+
+	if (pr_reg) {
+		spin_lock(&dev->t10_pr.registration_lock);
+		memcpy(pr_reg->pr_tid, pr_reg_data->tid, PR_REG_TID_LEN);
+		list_add_tail(&pr_reg->pr_reg_list,
+				&dev->t10_pr.registration_list);
+
+		/*
+		 * Drop deve->pr_kref obtained in __core_scsi3_do_alloc_registration()
+		 */
+		rcu_read_lock();
+		deve = pr_reg->pr_reg_deve;
+		if (deve) {
+			set_bit(DEF_PR_REG_ACTIVE, &deve->deve_flags);
+			kref_put(&deve->pr_kref, target_pr_kref_release);
+			pr_reg->pr_reg_deve = NULL;
+		}
+		rcu_read_unlock();
+
+		memset(&i_buf[0], 0, PR_REG_ISID_ID_LEN);
+		core_pr_dump_initiator_port(pr_reg, i_buf,
+						PR_REG_ISID_ID_LEN);
+		pr_debug("SPC-3 PR Service Action: REGISTER Initiator Node: %s%s\n",
+			initiatorname, i_buf);
+		pr_debug("SPC-3 PR registration on Target Port: %d\n",
+			pr_reg->tg_pt_sep_rtpi);
+		pr_debug("SPC-3 PR for %s TCM Subsystem %s Object Target Port(s)\n",
+			(pr_reg->pr_reg_all_tg_pt) ? "ALL" : "SINGLE",
+			dev->transport->name);
+		pr_debug("SPC-3 PR SA Res Key: 0x%016llx PRgeneration: 0x%08x  APTPL: %d\n",
+			pr_reg->pr_res_key,
+			pr_reg->pr_res_generation,
+			pr_reg->pr_reg_aptpl);
+		spin_unlock(&dev->t10_pr.registration_lock);
+		return pr_reg;
+	}
+
+	pr_warn("TARGET_CORE[%d]: PR data from cluster is invalid: could not find local nacl/deve for Initiator %s - RTPI %d\n",
+		dev->dev_index, initiatorname, pr_reg_data->rtpi);
+
+	return NULL;
+}
+
+static void target_pr_sync_cb(void *arg)
+{
+	struct se_device *dev = arg;
+	struct target_cluster_data *cluster_data = dev->cluster_data;
+	struct t10_pr_registration *pr_reg, *pr_reg_tmp;
+	struct t10_pr_registration *pr_reg_res_holder = NULL;
+	struct t10_pr_registration *pr_prev_res_holder = NULL;
+	struct pr_reg_lvb *pr_reg_data = NULL;
+	LIST_HEAD(to_be_deleted_list);
+	struct async_group grp;
+	struct pr_lvb pr_data;
+	bool res_to_delete = false;
+	bool was_held;
+	u8 was_type;
+	u8 was_scope;
+	bool found;
+	int i = 0;
+	int res;
+
+	res = dlm_ckv_get(cluster_data->pr_data, (char *)&pr_data, sizeof(pr_data));
+	if (res)
+		goto done;
+
+	if (!pr_data.version) {
+		pr_info("TARGET_CORE[%d]: PR data from cluster is invalid\n",
+			dev->dev_index);
+		goto done;
+	}
+
+	pr_reg_data = kzalloc(sizeof(struct pr_reg_lvb) * pr_data.nr_registrants,
+			    GFP_KERNEL);
+	if (!pr_reg_data) {
+		res = -ENOMEM;
+		goto done;
+	}
+
+	res = pr_reg_realloc(cluster_data, pr_data.nr_registrants);
+	if (res)
+		goto done;
+
+	if (pr_data.nr_registrants == 0)
+		goto skip_pr_reg;
+
+	refcount_set(&grp.pending, 1); /* 1 for a loop */
+	atomic_set(&grp.status, 0);
+	init_completion(&grp.compl);
+
+	for (i = 0; i < pr_data.nr_registrants; ++i) {
+		refcount_inc(&grp.pending);
+		res = dlm_ckv_get_async(cluster_data->pr_reg_kv[i], (char *)(pr_reg_data + i),
+				sizeof(struct pr_reg_lvb), group_compl_cb, &grp);
+		if (res) {
+			refcount_dec(&grp.pending);
+			break;
+		}
+	}
+	group_compl_cb(&grp, 0);
+	res = wait_for_completion_timeout(&grp.compl, 60 * HZ);
+	if (!res) {
+		pr_err("TARGET_CORE[%d]: timeout of waiting for dlm_ckv_get_async\n",
+			dev->dev_index);
+		goto done;
+	}
+	res = atomic_read(&grp.status);
+	if (res) {
+		pr_err("TARGET_CORE[%d]: fail of group for dlm_ckv_get_async %d\n",
+			dev->dev_index, res);
+		goto done;
+	}
+
+skip_pr_reg:
+	/*
+	 * Update existing registrations
+	 */
+	spin_lock(&dev->t10_pr.registration_lock);
+
+	was_held = !!dev->dev_pr_res_holder;
+	pr_prev_res_holder = dev->dev_pr_res_holder;
+	if (was_held) {
+		was_scope = pr_prev_res_holder->pr_res_scope;
+		was_type = pr_prev_res_holder->pr_res_type;
+	}
+
+	list_for_each_entry_safe(pr_reg, pr_reg_tmp, &dev->t10_pr.registration_list,
+				 pr_reg_list) {
+		found = false;
+
+		for (i = 0; i < pr_data.nr_registrants; ++i) {
+			if (!pr_reg_data[i].version)
+				continue;
+
+			if (pr_reg->tg_pt_sep_rtpi == pr_reg_data[i].rtpi &&
+			   !target_cmp_pr_transport_id(pr_reg, pr_reg_data[i].tid)) {
+				found = true;
+				/* mark existing registrants */
+				pr_reg_data[i].version = 0;
+				target_update_pr_reg(dev, pr_reg, &pr_data,
+						     &pr_reg_data[i],
+						     &pr_reg_res_holder);
+				break;
+			}
+		}
+
+		if (!found)
+			list_move_tail(&pr_reg->pr_reg_list, &to_be_deleted_list);
+	}
+
+	/* deregister obsolete entries */
+	list_for_each_entry_safe(pr_reg, pr_reg_tmp, &to_be_deleted_list,
+			pr_reg_list) {
+		if (dev->dev_pr_res_holder != pr_reg)
+			__core_scsi3_free_registration(dev, pr_reg, NULL, 0);
+		else
+			res_to_delete = true;
+	}
+	spin_unlock(&dev->t10_pr.registration_lock);
+
+	/* register new entries */
+	for (i = 0; i < pr_data.nr_registrants; ++i) {
+		/* skip existing registrants */
+		if (!pr_reg_data[i].version)
+			continue;
+
+		pr_reg = target_create_pr_reg(dev, &pr_data, &pr_reg_data[i]);
+		if (!pr_reg)
+			pr_err("TARGET_CORE[%d]: can not create new registration\n",
+				dev->dev_index);
+		else if (pr_reg_data[i].is_holder)
+			pr_reg_res_holder = pr_reg;
+	}
+
+	/* update general data */
+	atomic_set(&dev->t10_pr.pr_generation, pr_data.pr_generation);
+	dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
+
+	/* update reservation */
+	spin_lock(&dev->dev_reservation_lock);
+	if (pr_prev_res_holder &&
+	    ((pr_reg_res_holder != pr_prev_res_holder) ||
+	     (was_type != pr_reg_res_holder->pr_res_type)))
+		__core_scsi3_complete_pro_release(dev, pr_prev_res_holder, 0, 0);
+
+	if (pr_reg_res_holder)
+		__core_scsi3_set_reservation(dev, pr_reg_res_holder,
+					pr_data.pr_scope, pr_data.pr_type);
+	spin_unlock(&dev->dev_reservation_lock);
+
+	if (res_to_delete) {
+		spin_lock(&dev->t10_pr.registration_lock);
+		__core_scsi3_free_registration(dev, pr_prev_res_holder, NULL, 0);
+		spin_unlock(&dev->t10_pr.registration_lock);
+	}
+
+	core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
+
+done:
+	kfree(pr_reg_data);
+}
+
 const struct target_cluster_ops dlm_cluster_ops = {
 	.name = "dlm",
 	.owner = THIS_MODULE,
drivers/target/target_core_device.c +1 -0
index f5da3fc17ad1..297bf5985f48 100644
@@ -355,6 +355,7 @@ void target_pr_kref_release(struct kref *kref)
 						 pr_kref);
 	complete(&deve->pr_comp);
 }
+EXPORT_SYMBOL(target_pr_kref_release);
 
 /*
  * Establish UA condition on SCSI device - all LUNs
drivers/target/target_core_fabric_lib.c +2 -1
index 8232d872db35..2937bf4f6865 100644
@@ -396,7 +396,7 @@ int target_cmp_pr_transport_id(
 
 	return memcmp(pr_reg->pr_tid, tid, len1);
 }
-
+EXPORT_SYMBOL(target_cmp_pr_transport_id);
 
 int target_gen_pr_transport_id(
 	struct t10_pr_registration *pr_reg,
@@ -444,3 +444,4 @@ u32 target_parse_pr_out_transport_id(
 	*port_nexus_ptr = NULL;
 	return 24;
 }
+EXPORT_SYMBOL(target_parse_pr_out_transport_id);
drivers/target/target_core_internal.h +0 -5
index 8669ff6f48e9..89f0ddca35ed 100644
@@ -65,7 +65,6 @@ extern struct t10_alua_lu_gp *default_lu_gp;
 
 /* target_core_device.c */
 struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
-void	target_pr_kref_release(struct kref *);
 void	core_free_device_list_for_node(struct se_node_acl *,
 		struct se_portal_group *);
 void	core_update_device_list_access(u64, bool, struct se_node_acl *);
@@ -107,14 +106,10 @@ int	target_fabric_setup_cits(struct target_fabric_configfs *);
 
 /* target_core_fabric_lib.c */
 int	target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
-int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
-			       unsigned char *tid);
 int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
 			       int proto_id,
 			       const char *initiatorname,
 			       unsigned char *isid);
-u32 target_parse_pr_out_transport_id(
-	const char *buf, char *initiatorname, char **port_nexus_ptr);
 
 /* target_core_hba.c */
 struct se_hba *core_alloc_hba(const char *, u32, u32);
drivers/target/target_core_pr.c +7 -1
index a9fe192f5fe4..fe2f64826e9e 100644
@@ -52,6 +52,7 @@ void core_pr_dump_initiator_port(
 
 	snprintf(buf, size, ",i,0x%s", pr_reg->pr_reg_isid);
 }
+EXPORT_SYMBOL(core_pr_dump_initiator_port);
 
 enum register_type {
 	REGISTER,
@@ -677,6 +678,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 
 	return pr_reg;
 }
+EXPORT_SYMBOL(__core_scsi3_do_alloc_registration);
 
 static int core_scsi3_lunacl_depend_item(struct se_dev_entry *);
 static void core_scsi3_lunacl_undepend_item(struct se_dev_entry *);
@@ -1296,6 +1298,7 @@ void __core_scsi3_free_registration(
 	 */
 	list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
 }
+EXPORT_SYMBOL(__core_scsi3_free_registration);
 
 void core_scsi3_free_all_registrations(
 	struct se_device *dev)
@@ -1925,7 +1928,7 @@ static int __core_scsi3_write_aptpl_to_file(
  * Clear the APTPL metadata if APTPL has been disabled, otherwise
  * write out the updated metadata to struct file for this SCSI device.
  */
-static sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, bool aptpl)
+sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, bool aptpl)
 {
 	unsigned char *buf;
 	int rc, len = PR_APTPL_BUF_LEN;
@@ -1965,6 +1968,7 @@ static sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, b
 	pr_debug("SPC-3 PR: Set APTPL Bit Activated\n");
 	return 0;
 }
+EXPORT_SYMBOL(core_scsi3_update_and_write_aptpl);
 
 static sense_reason_t
 core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
@@ -2214,6 +2218,7 @@ void __core_scsi3_set_reservation(struct se_device *dev,
 			pr_reg->pr_iport,
 			i_buf);
 }
+EXPORT_SYMBOL(__core_scsi3_set_reservation);
 
 static sense_reason_t
 core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
@@ -2430,6 +2435,7 @@ void __core_scsi3_complete_pro_release(
 	 */
 	pr_reg->pr_res_holder = pr_reg->pr_res_type = pr_reg->pr_res_scope = 0;
 }
+EXPORT_SYMBOL(__core_scsi3_complete_pro_release);
 
 static sense_reason_t
 core_scsi3_emulate_pro_release(struct se_cmd *cmd, int type, int scope,
drivers/target/target_core_pr.h +1 -0
index 589fdac5470f..e7eff50a01f2 100644
@@ -68,6 +68,7 @@ extern int core_scsi3_alloc_aptpl_registration(
 extern int core_scsi3_check_aptpl_registration(struct se_device *,
 			struct se_portal_group *, struct se_lun *,
 			struct se_node_acl *, u64);
+sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, bool aptpl);
 extern void core_scsi3_free_all_registrations(struct se_device *);
 extern unsigned char *core_scsi3_pr_dump_type(int);
 
include/target/target_core_base.h +5 -0
index 9edf3c1ac204..b0f0a6b93f6e 100644
@@ -1004,5 +1004,10 @@ void target_cluster_impl_unregister(const struct target_cluster_ops *ops);
 
 int	target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
 				   unsigned char *buf);
+u32 target_parse_pr_out_transport_id(
+	const char *buf, char *initiatorname, char **port_nexus_ptr);
+int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
+			       unsigned char *tid);
+void	target_pr_kref_release(struct kref *kref);
 
 #endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
#40 lore raw

RFC PATCH 40/48 target: cluster: sync PR for dynamic acls

In reply to Dmitry Bogdanov

Dynamic acls are not present in remote target, so use NULL for nacl pointer for such registrants.

---
 drivers/target/target_cluster_dlm.c | 10 ++++++++++
 drivers/target/target_core_pr.c     | 17 +++++++++++------
 2 files changed, 21 insertions(+), 6 deletions(-)
drivers/target/target_cluster_dlm.c +10 -0
index 6b7cb3175e3a..6dbe1a0b21e2 100644
@@ -466,6 +466,16 @@ target_create_pr_reg(struct se_device *dev,
 			spin_unlock(&lun->lun_deve_lock);
 			goto out;
 		}
+		/* deve not found - that is a dynamic acl */
+		pr_reg = __core_scsi3_do_alloc_registration(
+				dev, NULL, initiatorname, lun->unpacked_lun,
+				pr_reg_data->rtpi,
+				NULL, lun->unpacked_lun,
+				isid,
+				pr_reg_data->key,
+				pr_reg_data->is_all_tg_pt,
+				pr_data->pr_aptpl);
+
 		percpu_ref_put(&lun->lun_ref);
 		spin_unlock(&lun->lun_deve_lock);
 		spin_lock(&dev->se_port_lock);
drivers/target/target_core_pr.c +11 -6
index fe2f64826e9e..22f2aa91cf2e 100644
@@ -644,10 +644,13 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 	 *
 	 * Otherwise, locate se_dev_entry now and obtain a reference until
 	 * registration completes in __core_scsi3_add_registration().
+	 *
+	 * NULL nacl means that this registration is for dynamic acl on a remote
+	 * target - no need to find a deve.
 	 */
 	if (dest_deve) {
 		pr_reg->pr_reg_deve = dest_deve;
-	} else {
+	} else if (nacl) {
 		rcu_read_lock();
 		pr_reg->pr_reg_deve = target_nacl_find_deve(nacl, mapped_lun);
 		if (!pr_reg->pr_reg_deve) {
@@ -1270,11 +1273,13 @@ void __core_scsi3_free_registration(
 		cpu_relax();
 	}
 
-	rcu_read_lock();
-	deve = target_nacl_find_deve(nacl, pr_reg->pr_res_mapped_lun);
-	if (deve)
-		clear_bit(DEF_PR_REG_ACTIVE, &deve->deve_flags);
-	rcu_read_unlock();
+	if (nacl) {
+		rcu_read_lock();
+		deve = target_nacl_find_deve(nacl, pr_reg->pr_res_mapped_lun);
+		if (deve)
+			clear_bit(DEF_PR_REG_ACTIVE, &deve->deve_flags);
+		rcu_read_unlock();
+	}
 
 	spin_lock(&pr_tmpl->registration_lock);
 	pr_debug("SPC-3 PR Service Action: UNREGISTER Initiator Node: %s%s\n",
Signature · 1 line
2.25.1
#41 lore raw

RFC PATCH 41/48 target: cluster: sync-up PR data on cluster join

In reply to Dmitry Bogdanov

Read PR data from cluster on joining to the cluster. acl/lun/tpg are supposed to be configured already.

---
 drivers/target/target_cluster_dlm.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/target/target_cluster_dlm.c +5 -0
index 6dbe1a0b21e2..16391a8661fa 100644
@@ -152,6 +152,11 @@ static int target_init_dlm(struct se_device *dev)
 
 	dev->cluster_data = cluster_data;
 
+	/* initial sync-up on joining the cluster */
+	dlm_ckv_lock_get(cluster_data->pr_lock);
+	target_pr_sync_cb(dev);
+	dlm_ckv_lock_release(cluster_data->pr_lock);
+
 	return err;
 fail:
 	if (cluster_data->pr_sync_notify)
Signature · 1 line
2.25.1
#42 lore raw

RFC PATCH 42/48 target: cluster: sync SPC-2 reservations

In reply to Dmitry Bogdanov

Reservation status is stored in DLM_CKV as part of PR data. If a reservation holder node is left the cluster other nodes will clear reservation status locally.

---
 drivers/target/dlm_ckv.c               | 16 ++++++-
 drivers/target/dlm_ckv.h               |  4 +-
 drivers/target/target_cluster_dlm.c    | 59 +++++++++++++++++++++++++-
 drivers/target/target_core_device.c    | 10 +++++
 drivers/target/target_core_pr.c        | 22 +++++++---
 drivers/target/target_core_pr.h        |  1 +
 drivers/target/target_core_tmr.c       |  4 +-
 drivers/target/target_core_transport.c |  2 +-
 include/target/target_core_base.h      |  1 +
 9 files changed, 104 insertions(+), 15 deletions(-)
drivers/target/dlm_ckv.c +14 -2
index 417159f18fc6..f4e7bc410a08 100644
@@ -49,6 +49,7 @@ struct dlm_ckv_bucket {
 	size_t num_nodes;
 	int nodeid[64];
 	void *userarg;
+	dlm_ckv_nodeleft_cb nodeleft_cb;
 	struct completion sync_compl;
 	struct workqueue_struct *notify_wq;
 };
@@ -79,7 +80,12 @@ static void dlm_ckv_recover_prep(void *arg)
 
 static void dlm_ckv_recover_slot(void *arg, struct dlm_slot *slot)
 {
-	pr_info("nodeid %d left the cluster\n", slot->nodeid);
+	struct dlm_ckv_bucket *bucket = arg;
+
+	pr_debug("nodeid %d left the cluster\n", slot->nodeid);
+
+	if (bucket->nodeleft_cb)
+		bucket->nodeleft_cb(bucket->userarg, slot->nodeid);
 }
 
 /* dlm calls after recover_slot and after it completes lock recovery */
@@ -667,7 +673,8 @@ static void bucket_release(struct kref *ref)
 }
 
 struct dlm_ckv_bucket *
-dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
+dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg,
+		    dlm_ckv_nodeleft_cb nodeleft_cb, int *local_nodeid)
 {
 	struct dlm_ckv_bucket *bucket;
 	int name_len = strlen(name);
@@ -677,6 +684,9 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
 	if (!name)
 		return ERR_PTR(-EINVAL);
 
+	if (!local_nodeid)
+		return ERR_PTR(-EINVAL);
+
 	if (name_len > DLM_LOCKSPACE_LEN)
 		return ERR_PTR(-EINVAL);
 
@@ -684,6 +694,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
 	kref_init(&bucket->refcount);
 
 	bucket->userarg = userarg;
+	bucket->nodeleft_cb = nodeleft_cb;
 	init_completion(&bucket->sync_compl);
 	bucket->notify_wq = alloc_ordered_workqueue("notify_wq-%s", 0, name);
 
@@ -708,6 +719,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
 		goto fail_init;
 	}
 
+	*local_nodeid = bucket->local_nodeid;
 	return bucket;
 
 fail_init:
drivers/target/dlm_ckv.h +3 -1
index e8045917067e..c5a69486a3bf 100644
@@ -14,7 +14,9 @@ typedef void (*dlm_ckv_nodeleft_cb)(void *arg, int nodeid);
 
 struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
 					   const char *cluster_name,
-					   void *userarg);
+					   void *userarg,
+					   dlm_ckv_nodeleft_cb nodeleft_cb,
+					   int *local_nodeid);
 int dlm_ckv_close_bucket(struct dlm_ckv_bucket *bucket);
 
 struct dlm_ckv_lock *
drivers/target/target_cluster_dlm.c +58 -1

89 lines not on this page · raw message

drivers/target/target_core_device.c +10 -0
index 297bf5985f48..9f01a28ef72f 100644
@@ -788,6 +788,15 @@ int target_dummy_nodlm(struct se_device *dev)
 	return 0;
 }
 
+static void target_reserve2_nodlm(struct se_device *dev, struct se_session *sess)
+{
+	if (sess) {
+		target_get_reservation(dev, sess);
+	} else {
+		target_release_reservation(dev);
+	}
+}
+
 const struct target_cluster_ops nodlm_cluster_ops = {
 	.name = "single",
 	.owner = NULL,
@@ -798,6 +807,7 @@ const struct target_cluster_ops nodlm_cluster_ops = {
 	.pr_lock = target_dummy_nodlm,
 	.pr_unlock = target_dummy_nodlm,
 	.pr_sync = target_dummy_nodlm,
+	.reserve = target_reserve2_nodlm,
 };
 
 struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
drivers/target/target_core_pr.c +15 -7
index 22f2aa91cf2e..930f0bebd75d 100644
@@ -192,6 +192,7 @@ void target_release_reservation(struct se_device *dev)
 		dev->dev_reservation_flags &= ~DRF_SPC2_RESERVATIONS_WITH_ISID;
 	}
 }
+EXPORT_SYMBOL(target_release_reservation);
 
 sense_reason_t
 target_scsi2_reservation_release(struct se_cmd *cmd)
@@ -222,7 +223,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
 	if (dev->dev_res_bin_isid != sess->sess_bin_isid)
 		goto out_unlock;
 
-	target_release_reservation(dev);
+	dev->cl_ops->reserve(dev, NULL);
 	tpg = sess->se_tpg;
 	pr_debug("SCSI-2 Released reservation for %s LUN: %llu ->"
 		" MAPPED LUN: %llu for %s\n",
@@ -237,6 +238,17 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
 	return 0;
 }
 
+void target_get_reservation(struct se_device *dev, struct se_session *sess)
+{
+	dev->reservation_holder = sess;
+	dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS;
+	if (sess->sess_bin_isid != 0) {
+		dev->dev_res_bin_isid = sess->sess_bin_isid;
+		dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS_WITH_ISID;
+	}
+}
+EXPORT_SYMBOL(target_get_reservation);
+
 sense_reason_t
 target_scsi2_reservation_reserve(struct se_cmd *cmd)
 {
@@ -284,12 +296,8 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd)
 		goto out_unlock;
 	}
 
-	dev->reservation_holder = sess;
-	dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS;
-	if (sess->sess_bin_isid != 0) {
-		dev->dev_res_bin_isid = sess->sess_bin_isid;
-		dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS_WITH_ISID;
-	}
+	dev->cl_ops->reserve(dev, sess);
+
 	pr_debug("SCSI-2 Reserved %s LUN: %llu -> MAPPED LUN: %llu"
 		" for %s\n", tpg->se_tpg_tfo->fabric_name,
 		cmd->se_lun->unpacked_lun, cmd->orig_fe_lun,
drivers/target/target_core_pr.h +1 -0
index e7eff50a01f2..a3a6013dc530 100644
@@ -59,6 +59,7 @@ extern struct kmem_cache *t10_pr_reg_cache;
 extern void core_pr_dump_initiator_port(struct t10_pr_registration *,
 			char *, u32);
 extern void target_release_reservation(struct se_device *dev);
+extern void target_get_reservation(struct se_device *dev, struct se_session *sess);
 extern sense_reason_t target_scsi2_reservation_release(struct se_cmd *);
 extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);
 extern int core_scsi3_alloc_aptpl_registration(
drivers/target/target_core_tmr.c +1 -3
index 2db45efedd97..26df0f685499 100644
@@ -406,9 +406,7 @@ int core_tmr_lun_reset(
 	 * LOGICAL UNIT RESET
 	 */
 	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
-		spin_lock(&dev->dev_reservation_lock);
-		target_release_reservation(dev);
-		spin_unlock(&dev->dev_reservation_lock);
+		dev->cl_ops->reserve(dev, NULL);
 		pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
 	}
 
drivers/target/target_core_transport.c +1 -1
index d3cbba294510..47b44a1e6925 100644
@@ -612,7 +612,7 @@ static int target_release_res(struct se_device *dev, void *data)
 	struct se_session *sess = data;
 
 	if (dev->reservation_holder == sess)
-		target_release_reservation(dev);
+		dev->cl_ops->reserve(dev, NULL);
 	return 0;
 }
 
include/target/target_core_base.h +1 -0
index b0f0a6b93f6e..c49ddf0828f6 100644
@@ -790,6 +790,7 @@ struct target_cluster_ops {
 	int (*pr_lock)(struct se_device *dev);
 	int (*pr_unlock)(struct se_device *dev);
 	int (*pr_sync)(struct se_device *dev);
+	void (*reserve)(struct se_device *dev, struct se_session *sess);
 };
 
 struct se_device {
Signature · 1 line
2.25.1
#43 lore raw

RFC PATCH 43/48 target: cluster: allocate UAs on PR sync

In reply to Dmitry Bogdanov

Establish Unit Attention condition on the peer nodes in the cluster on PR OUT commands according to SPC-4.
Pass PR OUT Service Action to pr_sync to allow peer nodes to know the reason of the update of PR state.

---
 drivers/target/target_cluster_dlm.c | 115 +++++++++++++++++++++++++++-
 drivers/target/target_core_device.c |   7 +-
 drivers/target/target_core_pr.c     |   2 +-
 drivers/target/target_core_ua.c     |   1 +
 include/target/target_core_base.h   |   2 +-
 5 files changed, 121 insertions(+), 6 deletions(-)
drivers/target/target_cluster_dlm.c +112 -3
index 4c4d337e6a1a..cb44f6c0f0ac 100644
@@ -7,6 +7,7 @@
 #include <target/target_core_base.h>
 #include <target/target_core_fabric.h>
 #include <target/target_core_backend.h>
+#include "target_core_ua.h"
 #include "target_core_pr.h"
 
 #include "dlm_ckv.h"
@@ -78,7 +79,8 @@ struct pr_lvb {
 	u8	pr_type;
 	u8	pr_scope;
 	u8	pr_aptpl;
-	u8	reserved[3];
+	u8	pro_sa;
+	u8	reserved[2];
 	u32	reserved_by_nodeid;
 };
 
@@ -314,7 +316,7 @@ static int pr_reg_realloc(struct target_cluster_data *cluster_data,
 	return res;
 }
 
-static int target_pr_sync_dlm(struct se_device *dev)
+static int target_pr_sync_dlm(struct se_device *dev, u8 pro_sa)
 {
 	struct target_cluster_data *cluster_data = dev->cluster_data;
 	struct t10_pr_registration *pr_reg;
@@ -328,6 +330,7 @@ static int target_pr_sync_dlm(struct se_device *dev)
 
 	pr_data.version = 1;
 	pr_data.pr_generation = atomic_read(&dev->t10_pr.pr_generation);
+	pr_data.pro_sa = pro_sa;
 	pr_data.nr_registrants = target_get_nr_registrants(dev);
 	pr_data.pr_is_set = !!dev->dev_pr_res_holder;
 	pr_data.pr_aptpl = dev->t10_pr.pr_aptpl_active;
@@ -546,6 +549,19 @@ target_create_pr_reg(struct se_device *dev,
 	return NULL;
 }
 
+static void target_allocate_pr_ua(struct se_device *dev, u8 asc)
+{
+	struct t10_pr_registration *pr_reg;
+
+	list_for_each_entry(pr_reg, &dev->t10_pr.registration_list, pr_reg_list) {
+		target_ua_allocate_lun(
+			pr_reg->pr_reg_nacl,
+			pr_reg->pr_res_mapped_lun,
+			0x2A,
+			asc);
+	}
+}
+
 static void target_pr_sync_cb(void *arg)
 {
 	struct se_device *dev = arg;
@@ -558,6 +574,9 @@ static void target_pr_sync_cb(void *arg)
 	struct async_group grp;
 	struct pr_lvb pr_data;
 	bool res_to_delete = false;
+	struct se_node_acl *pr_reg_nacl;
+	u64 pr_res_mapped_lun;
+	bool reg_deleted = false;
 	bool was_held;
 	u8 was_type;
 	u8 was_scope;
@@ -656,10 +675,41 @@ static void target_pr_sync_cb(void *arg)
 	/* deregister obsolete entries */
 	list_for_each_entry_safe(pr_reg, pr_reg_tmp, &to_be_deleted_list,
 			pr_reg_list) {
+		reg_deleted = true;
+		pr_reg_nacl = pr_reg->pr_reg_nacl;
+		pr_res_mapped_lun = pr_reg->pr_res_mapped_lun;
+
 		if (dev->dev_pr_res_holder != pr_reg)
 			__core_scsi3_free_registration(dev, pr_reg, NULL, 0);
 		else
 			res_to_delete = true;
+
+		switch (pr_data.pro_sa) {
+		case PRO_CLEAR:
+			/*
+			 * establish a unit attention condition for the initiator
+			 * port associated with every registered I_T nexus other
+			 * than the I_T nexus on which the PERSISTENT RESERVE OUT
+			 * command with CLEAR service action was received, with
+			 * the additional sense code set to RESERVATIONS PREEMPTED
+			 */
+			target_ua_allocate_lun(pr_reg_nacl,
+					pr_res_mapped_lun, 0x2A,
+					ASCQ_2AH_RESERVATIONS_PREEMPTED);
+			break;
+		case PRO_PREEMPT_AND_ABORT:
+		case PRO_REGISTER_AND_IGNORE_EXISTING_KEY:
+			/*
+			 * establish a unit attention condition for the initiator
+			 * port associated with every I_T nexus that lost its
+			 * persistent reservation and/or registration, with the
+			 * additional sense code set to REGISTRATIONS PREEMPTED;
+			 */
+			target_ua_allocate_lun(pr_reg_nacl,
+					pr_res_mapped_lun, 0x2A,
+					ASCQ_2AH_REGISTRATIONS_PREEMPTED);
+			break;
+		}
 	}
 	spin_unlock(&dev->t10_pr.registration_lock);
 
@@ -677,6 +727,65 @@ static void target_pr_sync_cb(void *arg)
 			pr_reg_res_holder = pr_reg;
 	}
 
+	switch (pr_data.pro_sa) {
+	case PRO_REGISTER_AND_IGNORE_EXISTING_KEY:
+	case PRO_REGISTER:
+		/*
+		 * If the I_T nexus is the reservation holder and the persistent
+		 * reservation is of a type other than all registrants, then
+		 * the device server shall also release the persistent reservation
+		 *
+		 * If the persistent reservation is a registrants only
+		 * type, the device server shall establish a unit
+		 * attention condition for the initiator port associated
+		 * with every registered I_T nexus except for the I_T
+		 * nexus on which the PERSISTENT RESERVE OUT command was
+		 * received, with the additional sense code set to
+		 * RESERVATIONS RELEASED.
+		 */
+		if (reg_deleted && was_held &&
+		    (was_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY ||
+		     was_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY)) {
+			target_allocate_pr_ua(dev,
+				ASCQ_2AH_RESERVATIONS_RELEASED);
+		}
+		break;
+	case PRO_RELEASE:
+		/*
+		 * if the released persistent reservation is either a registrants
+		 * only type or an all registrants type persistent reservation,
+		 * then the device server shall establish a unit attention
+		 * condition for the initiator port associated with every
+		 * registered I_T nexus other than I_T nexus on which the
+		 * PERSISTENT RESERVE OUT command with RELEASE service action was
+		 * received, with the additional sense code set to RESERVATIONS RELEASED
+		 */
+		if ((was_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY) ||
+		    (was_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY) ||
+		    (was_type == PR_TYPE_WRITE_EXCLUSIVE_ALLREG) ||
+		    (was_type == PR_TYPE_EXCLUSIVE_ACCESS_ALLREG)) {
+			target_allocate_pr_ua(dev,
+				ASCQ_2AH_RESERVATIONS_RELEASED);
+		}
+		break;
+	case PRO_PREEMPT:
+	case PRO_PREEMPT_AND_ABORT:
+		/*
+		 * if the type or scope has changed, then for every I_T nexus
+		 * whose reservation key was not removed, except for the I_T nexus
+		 * on which the PERSISTENT RESERVE OUT command was received, the
+		 * device server shall establish a unit attention condition for
+		 * the initiator port associated with that I_T nexus, with the
+		 * additional sense code set to RESERVATIONS RELEASED. I
+		 */
+		if (pr_prev_res_holder && pr_reg_res_holder &&
+		    pr_prev_res_holder != pr_reg_res_holder &&
+		    (was_type != pr_reg_res_holder->pr_res_type ||
+		     was_scope != pr_reg_res_holder->pr_res_scope))
+			target_allocate_pr_ua(dev, ASCQ_2AH_RESERVATIONS_RELEASED);
+		break;
+	}
+
 	/* update general data */
 	atomic_set(&dev->t10_pr.pr_generation, pr_data.pr_generation);
 	dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
@@ -741,7 +850,7 @@ target_spc2_reserve(struct se_device *dev, struct se_session *sess)
 		cluster_data->reserved_node_id = 0;
 	}
 
-	target_pr_sync_dlm(dev);
+	target_pr_sync_dlm(dev, -1);
 	target_pr_unlock_dlm(dev);
 }
 
drivers/target/target_core_device.c +6 -1
index 9f01a28ef72f..4f36d12d2213 100644
@@ -788,6 +788,11 @@ int target_dummy_nodlm(struct se_device *dev)
 	return 0;
 }
 
+int target_prsync_nodlm(struct se_device *dev, u8 pro_sa)
+{
+	return 0;
+}
+
 static void target_reserve2_nodlm(struct se_device *dev, struct se_session *sess)
 {
 	if (sess) {
@@ -806,7 +811,7 @@ const struct target_cluster_ops nodlm_cluster_ops = {
 	.caw_unlock = target_caw_unlock_nodlm,
 	.pr_lock = target_dummy_nodlm,
 	.pr_unlock = target_dummy_nodlm,
-	.pr_sync = target_dummy_nodlm,
+	.pr_sync = target_prsync_nodlm,
 	.reserve = target_reserve2_nodlm,
 };
 
drivers/target/target_core_pr.c +1 -1
index 930f0bebd75d..fd89647c5ac7 100644
@@ -3624,7 +3624,7 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)
 	}
 
 	if (!ret)
-		dev->cl_ops->pr_sync(dev);
+		dev->cl_ops->pr_sync(dev, sa);
 	dev->cl_ops->pr_unlock(dev);
 
 	if (!ret)
drivers/target/target_core_ua.c +1 -0
index 4276690fb6cb..68a7339611f3 100644
@@ -171,6 +171,7 @@ void target_ua_allocate_lun(struct se_node_acl *nacl,
 	core_scsi3_ua_allocate(deve, asc, ascq);
 	rcu_read_unlock();
 }
+EXPORT_SYMBOL(target_ua_allocate_lun);
 
 void core_scsi3_ua_release_all(
 	struct se_dev_entry *deve)
include/target/target_core_base.h +1 -1
index c49ddf0828f6..6c99fb4aa151 100644
@@ -789,7 +789,7 @@ struct target_cluster_ops {
 	int (*caw_unlock)(struct se_device *dev, void *lock);
 	int (*pr_lock)(struct se_device *dev);
 	int (*pr_unlock)(struct se_device *dev);
-	int (*pr_sync)(struct se_device *dev);
+	int (*pr_sync)(struct se_device *dev, u8 pro_sa);
 	void (*reserve)(struct se_device *dev, struct se_session *sess);
 };
 
Signature · 1 line
2.25.1
#44 lore raw

RFC PATCH 44/48 target: cluster: support PR OUT preempt and abort

In reply to Dmitry Bogdanov

Abort commands in the preempted I_T nexuses on all nodes in cluster by PR OUT (PREEMPT AND ABORT) command.

---
 drivers/target/target_cluster_dlm.c   | 17 +++++++++++++++--
 drivers/target/target_core_internal.h |  2 --
 drivers/target/target_core_pr.c       |  3 ++-
 drivers/target/target_core_pr.h       |  3 +++
 drivers/target/target_core_tmr.c      |  1 +
 include/target/target_core_base.h     |  3 +++
 6 files changed, 24 insertions(+), 5 deletions(-)
drivers/target/target_cluster_dlm.c +15 -2
index cb44f6c0f0ac..91f8ffc7ad2f 100644
@@ -571,6 +571,7 @@ static void target_pr_sync_cb(void *arg)
 	struct t10_pr_registration *pr_prev_res_holder = NULL;
 	struct pr_reg_lvb *pr_reg_data = NULL;
 	LIST_HEAD(to_be_deleted_list);
+	LIST_HEAD(preempt_and_abort_list);
 	struct async_group grp;
 	struct pr_lvb pr_data;
 	bool res_to_delete = false;
@@ -680,7 +681,9 @@ static void target_pr_sync_cb(void *arg)
 		pr_res_mapped_lun = pr_reg->pr_res_mapped_lun;
 
 		if (dev->dev_pr_res_holder != pr_reg)
-			__core_scsi3_free_registration(dev, pr_reg, NULL, 0);
+			__core_scsi3_free_registration(dev, pr_reg,
+				(pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+				&preempt_and_abort_list : NULL, 0);
 		else
 			res_to_delete = true;
 
@@ -825,10 +828,20 @@ static void target_pr_sync_cb(void *arg)
 
 	if (res_to_delete) {
 		spin_lock(&dev->t10_pr.registration_lock);
-		__core_scsi3_free_registration(dev, pr_prev_res_holder, NULL, 0);
+		__core_scsi3_free_registration(dev, pr_prev_res_holder,
+				(pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+				&preempt_and_abort_list : NULL, 0);
 		spin_unlock(&dev->t10_pr.registration_lock);
 	}
 
+	if (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) {
+		core_local_lun_reset(dev, NULL, &preempt_and_abort_list, NULL);
+
+		core_scsi3_release_preempt_and_abort(
+					&preempt_and_abort_list,
+					pr_reg_res_holder);
+	}
+
 	core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
 
 done:
drivers/target/target_core_internal.h +0 -2
index 89f0ddca35ed..fc7c2a9d9507 100644
@@ -119,8 +119,6 @@ int	core_delete_hba(struct se_hba *);
 void	core_tmr_abort_task(struct se_device *, struct se_tmr_req *,
 			struct se_session *);
 int	core_tmr_lun_reset(struct se_device *dev, struct se_tmr_req *tmr);
-void	core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
-		struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
 
 /* target_core_tpg.c */
 extern struct spinlock g_tpg_lock;
drivers/target/target_core_pr.c +2 -1
index fd89647c5ac7..a476dc3aa03f 100644
@@ -2725,7 +2725,7 @@ static void __core_scsi3_complete_pro_preempt(
 				preempt_and_abort_list);
 }
 
-static void core_scsi3_release_preempt_and_abort(
+void core_scsi3_release_preempt_and_abort(
 	struct list_head *preempt_and_abort_list,
 	struct t10_pr_registration *pr_reg_holder)
 {
@@ -2747,6 +2747,7 @@ static void core_scsi3_release_preempt_and_abort(
 		kmem_cache_free(t10_pr_reg_cache, pr_reg);
 	}
 }
+EXPORT_SYMBOL(core_scsi3_release_preempt_and_abort);
 
 static sense_reason_t
 core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
drivers/target/target_core_pr.h +3 -0
index a3a6013dc530..82b4a270f711 100644
@@ -99,6 +99,9 @@ void __core_scsi3_free_registration(struct se_device *dev,
 				    struct t10_pr_registration *pr_reg,
 				    struct list_head *preempt_and_abort_list,
 				    int dec_holders);
+void core_scsi3_release_preempt_and_abort(
+				struct list_head *preempt_and_abort_list,
+				struct t10_pr_registration *pr_reg_holder);
 
 
 #endif /* TARGET_CORE_PR_H */
drivers/target/target_core_tmr.c +1 -0
index 26df0f685499..6984bfa976a4 100644
@@ -394,6 +394,7 @@ void core_local_lun_reset(
 			(preempt_and_abort_list) ? "Preempt" : "TMR",
 			dev->transport->name);
 }
+EXPORT_SYMBOL(core_local_lun_reset);
 
 int core_tmr_lun_reset(
 	struct se_device *dev,
include/target/target_core_base.h +3 -0
index 6c99fb4aa151..af3c972c3376 100644
@@ -1010,5 +1010,8 @@ u32 target_parse_pr_out_transport_id(
 int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
 			       unsigned char *tid);
 void	target_pr_kref_release(struct kref *kref);
+void	core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
+		struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
+
 
 #endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
#45 lore raw

RFC PATCH 45/48 target: cluster: add reset cluster function

In reply to Dmitry Bogdanov

Introduce `reset' function pointer in target cluster ops. For no cluster case do just clear SCSI2 reservation status.

---
 drivers/target/target_core_device.c | 9 +++++++++
 drivers/target/target_core_tmr.c    | 9 +--------
 include/target/target_core_base.h   | 1 +
 3 files changed, 11 insertions(+), 8 deletions(-)
drivers/target/target_core_device.c +9 -0
index 4f36d12d2213..7d0d889961c2 100644
@@ -802,6 +802,14 @@ static void target_reserve2_nodlm(struct se_device *dev, struct se_session *sess
 	}
 }
 
+static void target_reset_nodlm(struct se_device *dev)
+{
+	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
+		target_release_reservation(dev);
+		pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
+	}
+}
+
 const struct target_cluster_ops nodlm_cluster_ops = {
 	.name = "single",
 	.owner = NULL,
@@ -813,6 +821,7 @@ const struct target_cluster_ops nodlm_cluster_ops = {
 	.pr_unlock = target_dummy_nodlm,
 	.pr_sync = target_prsync_nodlm,
 	.reserve = target_reserve2_nodlm,
+	.reset = target_reset_nodlm,
 };
 
 struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
drivers/target/target_core_tmr.c +1 -8
index 6984bfa976a4..878401ab8145 100644
@@ -402,14 +402,7 @@ int core_tmr_lun_reset(
 {
 	core_local_lun_reset(dev, tmr, NULL, NULL);
 
-	/*
-	 * Clear any legacy SPC-2 reservation when called during
-	 * LOGICAL UNIT RESET
-	 */
-	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
-		dev->cl_ops->reserve(dev, NULL);
-		pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
-	}
+	dev->cl_ops->reset(dev);
 
 	target_dev_ua_allocate(dev, 0x29,
 			ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
include/target/target_core_base.h +1 -0
index af3c972c3376..c8900da705b6 100644
@@ -791,6 +791,7 @@ struct target_cluster_ops {
 	int (*pr_unlock)(struct se_device *dev);
 	int (*pr_sync)(struct se_device *dev, u8 pro_sa);
 	void (*reserve)(struct se_device *dev, struct se_session *sess);
+	void (*reset)(struct se_device *dev);
 };
 
 struct se_device {
Signature · 1 line
2.25.1
#46 lore raw

RFC PATCH 46/48 target: cluster: implement LUN reset in DLM cluster

In reply to Dmitry Bogdanov

Since LUN RESET clears SCSI2 Reservation (cluster wide data of persistence reservation status) make LUN RESET just one of reasons to update the Persistence Reservation status as well as Service Actions of PR OUT.

---
 drivers/target/target_cluster_dlm.c   | 31 ++++++++++++++++++++++++++-
 drivers/target/target_core_device.c   |  1 +
 drivers/target/target_core_internal.h |  1 -
 include/target/target_core_base.h     |  2 +-
 4 files changed, 32 insertions(+), 3 deletions(-)
drivers/target/target_cluster_dlm.c +30 -1
index 91f8ffc7ad2f..546e28062b8f 100644
@@ -99,6 +99,9 @@ struct async_group {
 	struct completion compl;
 };
 
+#define PR_SYNC_REASON_RESERVE	10
+#define PR_SYNC_REASON_RESET	11
+
 static void target_pr_sync_cb(void *arg);
 static void target_nodeleft_cb(void *arg, int nodeid);
 static int pr_reg_realloc(struct target_cluster_data *cluster_data,
@@ -842,6 +845,15 @@ static void target_pr_sync_cb(void *arg)
 					pr_reg_res_holder);
 	}
 
+	if (pr_data.pro_sa == PR_SYNC_REASON_RESET) {
+		core_local_lun_reset(dev, NULL, NULL, NULL);
+
+		target_dev_ua_allocate(dev, 0x29,
+				ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
+
+		atomic_long_inc(&dev->num_resets);
+	}
+
 	core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
 
 done:
@@ -863,7 +875,23 @@ target_spc2_reserve(struct se_device *dev, struct se_session *sess)
 		cluster_data->reserved_node_id = 0;
 	}
 
-	target_pr_sync_dlm(dev, -1);
+	target_pr_sync_dlm(dev, PR_SYNC_REASON_RESERVE);
+	target_pr_unlock_dlm(dev);
+}
+
+static void
+target_dlm_lun_reset(struct se_device *dev)
+{
+	struct target_cluster_data *cluster_data = dev->cluster_data;
+
+	target_pr_lock_dlm(dev);
+	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
+		target_release_reservation(dev);
+		cluster_data->reserved_node_id = 0;
+		pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
+	}
+
+	target_pr_sync_dlm(dev, PR_SYNC_REASON_RESET);
 	target_pr_unlock_dlm(dev);
 }
 
@@ -894,6 +922,7 @@ const struct target_cluster_ops dlm_cluster_ops = {
 	.pr_unlock = target_pr_unlock_dlm,
 	.pr_sync = target_pr_sync_dlm,
 	.reserve = target_spc2_reserve,
+	.reset = target_dlm_lun_reset,
 };
 
 static int __init target_cluster_dlm_module_init(void)
drivers/target/target_core_device.c +1 -0
index 7d0d889961c2..6fe9f40b86f5 100644
@@ -375,6 +375,7 @@ void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq)
 	}
 	spin_unlock(&dev->se_port_lock);
 }
+EXPORT_SYMBOL(target_dev_ua_allocate);
 
 static void
 target_luns_data_has_changed(struct se_node_acl *nacl, struct se_dev_entry *new,
drivers/target/target_core_internal.h +0 -1
index fc7c2a9d9507..7c495dddcb4b 100644
@@ -93,7 +93,6 @@ int	target_configure_device(struct se_device *dev);
 void	target_free_device(struct se_device *);
 int	target_for_each_device(int (*fn)(struct se_device *dev, void *data),
 			       void *data);
-void	target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
 
 /* target_core_configfs.c */
 extern struct configfs_item_operations target_core_dev_item_ops;
include/target/target_core_base.h +1 -1
index c8900da705b6..4dad7837f603 100644
@@ -1013,6 +1013,6 @@ int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
 void	target_pr_kref_release(struct kref *kref);
 void	core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
 		struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
-
+void	target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
 
 #endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
#47 lore raw

RFC PATCH 47/48 target: cluster: split cluster sync function

In reply to Dmitry Bogdanov

Split cluster sync function to two parts:
1. Read cluster data.
2. Apply cluster data.

---
 drivers/target/target_cluster_dlm.c | 117 ++++++++++++++++++----------
 1 file changed, 74 insertions(+), 43 deletions(-)
drivers/target/target_cluster_dlm.c +74 -43
index 546e28062b8f..ff0c207be141 100644
@@ -565,61 +565,53 @@ static void target_allocate_pr_ua(struct se_device *dev, u8 asc)
 	}
 }
 
-static void target_pr_sync_cb(void *arg)
+static int target_dlm_read_cluster_data(
+	struct se_device *dev,
+	struct pr_lvb *pr_data,
+	struct pr_reg_lvb **pr_reg_datap)
 {
-	struct se_device *dev = arg;
 	struct target_cluster_data *cluster_data = dev->cluster_data;
-	struct t10_pr_registration *pr_reg, *pr_reg_tmp;
-	struct t10_pr_registration *pr_reg_res_holder = NULL;
-	struct t10_pr_registration *pr_prev_res_holder = NULL;
 	struct pr_reg_lvb *pr_reg_data = NULL;
-	LIST_HEAD(to_be_deleted_list);
-	LIST_HEAD(preempt_and_abort_list);
 	struct async_group grp;
-	struct pr_lvb pr_data;
-	bool res_to_delete = false;
-	struct se_node_acl *pr_reg_nacl;
-	u64 pr_res_mapped_lun;
-	bool reg_deleted = false;
-	bool was_held;
-	u8 was_type;
-	u8 was_scope;
-	bool found;
+	int res = 0;
 	int i = 0;
-	int res;
 
-	res = dlm_ckv_get(cluster_data->pr_data, (char *)&pr_data, sizeof(pr_data));
+	res = dlm_ckv_get(cluster_data->pr_data, (char *)pr_data,
+			  sizeof(struct pr_lvb));
 	if (res)
 		goto done;
 
-	if (!pr_data.version) {
+	if (!pr_data->version) {
 		pr_info("TARGET_CORE[%d]: PR data from cluster is invalid\n",
 			dev->dev_index);
 		goto done;
 	}
 
-	pr_reg_data = kzalloc(sizeof(struct pr_reg_lvb) * pr_data.nr_registrants,
+	if (pr_data->nr_registrants == 0)
+		goto done;
+
+	*pr_reg_datap = kzalloc(sizeof(struct pr_reg_lvb) * pr_data->nr_registrants,
 			    GFP_KERNEL);
+	pr_reg_data = *pr_reg_datap;
 	if (!pr_reg_data) {
 		res = -ENOMEM;
 		goto done;
 	}
 
-	res = pr_reg_realloc(cluster_data, pr_data.nr_registrants);
+	res = pr_reg_realloc(cluster_data, pr_data->nr_registrants);
 	if (res)
 		goto done;
 
-	if (pr_data.nr_registrants == 0)
-		goto skip_pr_reg;
-
 	refcount_set(&grp.pending, 1); /* 1 for a loop */
 	atomic_set(&grp.status, 0);
 	init_completion(&grp.compl);
 
-	for (i = 0; i < pr_data.nr_registrants; ++i) {
+	for (i = 0; i < pr_data->nr_registrants; ++i) {
 		refcount_inc(&grp.pending);
-		res = dlm_ckv_get_async(cluster_data->pr_reg_kv[i], (char *)(pr_reg_data + i),
-				sizeof(struct pr_reg_lvb), group_compl_cb, &grp);
+		res = dlm_ckv_get_async(cluster_data->pr_reg_kv[i],
+					(char *)(pr_reg_data + i),
+					sizeof(struct pr_reg_lvb),
+					group_compl_cb, &grp);
 		if (res) {
 			refcount_dec(&grp.pending);
 			break;
@@ -639,7 +631,33 @@ static void target_pr_sync_cb(void *arg)
 		goto done;
 	}
 
-skip_pr_reg:
+done:
+	if (res)
+		kfree(pr_reg_data);
+	return res;
+}
+
+static void target_dlm_apply_cluster_data(
+	struct se_device *dev,
+	struct pr_lvb *pr_data,
+	struct pr_reg_lvb *pr_reg_data)
+{
+	struct target_cluster_data *cluster_data = dev->cluster_data;
+	struct t10_pr_registration *pr_reg, *pr_reg_tmp;
+	struct t10_pr_registration *pr_reg_res_holder = NULL;
+	struct t10_pr_registration *pr_prev_res_holder = NULL;
+	LIST_HEAD(to_be_deleted_list);
+	LIST_HEAD(preempt_and_abort_list);
+	bool res_to_delete = false;
+	struct se_node_acl *pr_reg_nacl;
+	u64 pr_res_mapped_lun;
+	bool reg_deleted = false;
+	bool was_held;
+	u8 was_type;
+	u8 was_scope;
+	bool found;
+	int i = 0;
+
 	/*
 	 * Update existing registrations
 	 */
@@ -656,7 +674,7 @@ static void target_pr_sync_cb(void *arg)
 				 pr_reg_list) {
 		found = false;
 
-		for (i = 0; i < pr_data.nr_registrants; ++i) {
+		for (i = 0; i < pr_data->nr_registrants; ++i) {
 			if (!pr_reg_data[i].version)
 				continue;
 
@@ -665,7 +683,7 @@ static void target_pr_sync_cb(void *arg)
 				found = true;
 				/* mark existing registrants */
 				pr_reg_data[i].version = 0;
-				target_update_pr_reg(dev, pr_reg, &pr_data,
+				target_update_pr_reg(dev, pr_reg, pr_data,
 						     &pr_reg_data[i],
 						     &pr_reg_res_holder);
 				break;
@@ -685,12 +703,12 @@ static void target_pr_sync_cb(void *arg)
 
 		if (dev->dev_pr_res_holder != pr_reg)
 			__core_scsi3_free_registration(dev, pr_reg,
-				(pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+				(pr_data->pro_sa == PRO_PREEMPT_AND_ABORT) ?
 				&preempt_and_abort_list : NULL, 0);
 		else
 			res_to_delete = true;
 
-		switch (pr_data.pro_sa) {
+		switch (pr_data->pro_sa) {
 		case PRO_CLEAR:
 			/*
 			 * establish a unit attention condition for the initiator
@@ -720,12 +738,12 @@ static void target_pr_sync_cb(void *arg)
 	spin_unlock(&dev->t10_pr.registration_lock);
 
 	/* register new entries */
-	for (i = 0; i < pr_data.nr_registrants; ++i) {
+	for (i = 0; i < pr_data->nr_registrants; ++i) {
 		/* skip existing registrants */
 		if (!pr_reg_data[i].version)
 			continue;
 
-		pr_reg = target_create_pr_reg(dev, &pr_data, &pr_reg_data[i]);
+		pr_reg = target_create_pr_reg(dev, pr_data, &pr_reg_data[i]);
 		if (!pr_reg)
 			pr_err("TARGET_CORE[%d]: can not create new registration\n",
 				dev->dev_index);
@@ -733,7 +751,7 @@ static void target_pr_sync_cb(void *arg)
 			pr_reg_res_holder = pr_reg;
 	}
 
-	switch (pr_data.pro_sa) {
+	switch (pr_data->pro_sa) {
 	case PRO_REGISTER_AND_IGNORE_EXISTING_KEY:
 	case PRO_REGISTER:
 		/*
@@ -793,11 +811,11 @@ static void target_pr_sync_cb(void *arg)
 	}
 
 	/* update general data */
-	atomic_set(&dev->t10_pr.pr_generation, pr_data.pr_generation);
-	dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
+	atomic_set(&dev->t10_pr.pr_generation, pr_data->pr_generation);
+	dev->t10_pr.pr_aptpl_active = pr_data->pr_aptpl;
 
 	/* update SPC-2 reservation */
-	cluster_data->reserved_node_id = pr_data.reserved_by_nodeid;
+	cluster_data->reserved_node_id = pr_data->reserved_by_nodeid;
 	spin_lock(&dev->dev_reservation_lock);
 	if (cluster_data->reserved_node_id == cluster_data->local_nodeid &&
 	    dev->reservation_holder == NULL) {
@@ -826,18 +844,18 @@ static void target_pr_sync_cb(void *arg)
 
 	if (pr_reg_res_holder)
 		__core_scsi3_set_reservation(dev, pr_reg_res_holder,
-					pr_data.pr_scope, pr_data.pr_type);
+					pr_data->pr_scope, pr_data->pr_type);
 	spin_unlock(&dev->dev_reservation_lock);
 
 	if (res_to_delete) {
 		spin_lock(&dev->t10_pr.registration_lock);
 		__core_scsi3_free_registration(dev, pr_prev_res_holder,
-				(pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+				(pr_data->pro_sa == PRO_PREEMPT_AND_ABORT) ?
 				&preempt_and_abort_list : NULL, 0);
 		spin_unlock(&dev->t10_pr.registration_lock);
 	}
 
-	if (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) {
+	if (pr_data->pro_sa == PRO_PREEMPT_AND_ABORT) {
 		core_local_lun_reset(dev, NULL, &preempt_and_abort_list, NULL);
 
 		core_scsi3_release_preempt_and_abort(
@@ -845,7 +863,7 @@ static void target_pr_sync_cb(void *arg)
 					pr_reg_res_holder);
 	}
 
-	if (pr_data.pro_sa == PR_SYNC_REASON_RESET) {
+	if (pr_data->pro_sa == PR_SYNC_REASON_RESET) {
 		core_local_lun_reset(dev, NULL, NULL, NULL);
 
 		target_dev_ua_allocate(dev, 0x29,
@@ -856,10 +874,23 @@ static void target_pr_sync_cb(void *arg)
 
 	core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
 
-done:
 	kfree(pr_reg_data);
 }
 
+static void target_pr_sync_cb(void *arg)
+{
+	struct pr_reg_lvb *pr_reg_data = NULL;
+	struct se_device *dev = arg;
+	struct pr_lvb pr_data;
+	int res;
+
+	res = target_dlm_read_cluster_data(dev, &pr_data, &pr_reg_data);
+	if (res)
+		return;
+
+	target_dlm_apply_cluster_data(dev, &pr_data, pr_reg_data);
+}
+
 static void
 target_spc2_reserve(struct se_device *dev, struct se_session *sess)
 {
Signature · 1 line
2.25.1
#48 lore raw

RFC PATCH 48/48 target: cluster: request data on initial sync

In reply to Dmitry Bogdanov

If there is no cluster data in cluster (master node died), request it from alive nodes.

---
 drivers/target/target_cluster_dlm.c | 36 ++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)
drivers/target/target_cluster_dlm.c +35 -1
index ff0c207be141..39754d3671b7 100644
@@ -19,6 +19,7 @@ struct target_cluster_data {
 	struct dlm_ckv_lock *pr_lock;
 	struct dlm_ckv_kv *pr_data;
 	struct dlm_ckv_notify *pr_sync_notify;
+	struct dlm_ckv_notify *init_sync_notify;
 	int reserved_node_id;
 	struct dlm_ckv_kv **pr_reg_kv;
 	size_t pr_reg_kv_len;
@@ -101,7 +102,10 @@ struct async_group {
 
 #define PR_SYNC_REASON_RESERVE	10
 #define PR_SYNC_REASON_RESET	11
+#define PR_SYNC_REASON_INIT	13
 
+static void target_initial_sync(struct se_device *dev);
+static void target_initial_sync_cb(void *arg);
 static void target_pr_sync_cb(void *arg);
 static void target_nodeleft_cb(void *arg, int nodeid);
 static int pr_reg_realloc(struct target_cluster_data *cluster_data,
@@ -157,15 +161,22 @@ static int target_init_dlm(struct se_device *dev)
 	if (!cluster_data->pr_sync_notify)
 		goto fail;
 
+	cluster_data->init_sync_notify = dlm_ckv_create_notification(
+		cluster_data->bucket, "init_sync", target_initial_sync_cb);
+	if (!cluster_data->init_sync_notify)
+		goto fail;
+
 	dev->cluster_data = cluster_data;
 
 	/* initial sync-up on joining the cluster */
 	dlm_ckv_lock_get(cluster_data->pr_lock);
-	target_pr_sync_cb(dev);
+	target_initial_sync(dev);
 	dlm_ckv_lock_release(cluster_data->pr_lock);
 
 	return err;
 fail:
+	if (cluster_data->init_sync_notify)
+		dlm_ckv_free_notification(cluster_data->init_sync_notify);
 	if (cluster_data->pr_sync_notify)
 		dlm_ckv_free_notification(cluster_data->pr_sync_notify);
 	if (cluster_data->pr_lock)
@@ -188,6 +199,7 @@ static int target_cleanup_dlm(struct se_device *dev)
 			dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
 	kfree(cluster_data->pr_reg_kv);
 
+	dlm_ckv_free_notification(cluster_data->init_sync_notify);
 	dlm_ckv_free_notification(cluster_data->pr_sync_notify);
 	dlm_ckv_free_lock(cluster_data->pr_lock);
 	dlm_ckv_free_kv(cluster_data->pr_data);
@@ -410,6 +422,13 @@ static int target_pr_sync_dlm(struct se_device *dev, u8 pro_sa)
 	return res;
 }
 
+static void target_initial_sync_cb(void *arg)
+{
+	struct se_device *dev = arg;
+
+	target_pr_sync_dlm(dev, PR_SYNC_REASON_INIT);
+}
+
 static int target_update_pr_reg(struct se_device *dev,
 				struct t10_pr_registration *pr_reg,
 				const struct pr_lvb *pr_data,
@@ -584,6 +603,7 @@ static int target_dlm_read_cluster_data(
 	if (!pr_data->version) {
 		pr_info("TARGET_CORE[%d]: PR data from cluster is invalid\n",
 			dev->dev_index);
+		res = -ENOLCK;
 		goto done;
 	}
 
@@ -877,6 +897,20 @@ static void target_dlm_apply_cluster_data(
 	kfree(pr_reg_data);
 }
 
+static void target_initial_sync(struct se_device *dev)
+{
+	struct target_cluster_data *cluster_data = dev->cluster_data;
+	struct pr_reg_lvb *pr_reg_data = NULL;
+	struct pr_lvb pr_data;
+	int res;
+
+	res = target_dlm_read_cluster_data(dev, &pr_data, &pr_reg_data);
+	if (!res)
+		target_dlm_apply_cluster_data(dev, &pr_data, pr_reg_data);
+	if (res == -ENOLCK)
+		dlm_ckv_notify(cluster_data->init_sync_notify);
+}
+
 static void target_pr_sync_cb(void *arg)
 {
 	struct pr_reg_lvb *pr_reg_data = NULL;
Signature · 1 line
2.25.1
#49 lore raw
In reply to Dmitry Bogdanov
Quoted 21 lines · Dmitry Bogdanov

Hi linux target comminity.

Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.

The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.

Patches 1-9:
Make RTPI be part of se_tpg instead of se_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).

Quoted 5 lines

Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undepend pr_reg from se_nacl and se_tpg to be just a registration holder. Make APTPL registrations (not linked to se_dev_entry) be full-fledged registrations.

What are the arguable parts? Do you think it will be the DLM part and coordinating it with nvmet developers? Or was it patches 1-9 and the multi-node support? Or both :)

Is it possible and would it be valuable to at least kind of break this up a little?

I would break this up and post the fixes in one set. I'll help you get them in as soon as possible.

For patches 1-9, I think I remember you posting them before, but I was in the middle of starting a new job so I didn't review them. I really needed something like that at my last 2 jobs so I think it's a valuable feature and I'll review that as well.

If we could at least get those 2 chunks separated then it would make the DLM parts below easier to get eyeballs on. I'm ok with the idea in general. I think every nvmet developer will see the massive patchset and not even look at this first 0/48 email :)

Quoted 38 lines

Patches 30-34:
DLM_CKV module that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.

Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.

Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attrib cluster_impl regulates which implementation to use for that device. 'single' is for default (no cluster) implementation.

Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* Use DLM_CKV Lock service to serialize order of PR OUT commands
* Use DLM_CKV Key-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* Use DLM_CKV Notification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.

It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster

#50 lore raw
Quoted 30 lines · Mike Christie
Quoted 21 lines · Dmitry Bogdanov

Hi linux target comminity.

Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.

The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.

Patches 1-9:
Make RTPI be part of se_tpg instead of se_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).

Quoted 5 lines

Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undepend pr_reg from se_nacl and se_tpg to be just a registration holder. Make APTPL registrations (not linked to se_dev_entry) be full-fledged registrations.

Quoted 3 lines

What are the arguable parts? Do you think it will be the DLM part and coordinating it with nvmet developers? Or was it patches 1-9 and the multi-node support? Or both :)

In fact every subset can be a subject to argue :)
* RTPI patchset - changing data model from RTPI-set on backstore device to RTPI-set on a whole node.
* PR refactoring - to much changes, may be APTPL changes are not backward compatible
* remote/dummy fabric - name
* DLM_CKV - name, place and even a meaning of the module
* tcm_cluster - too much new exported symbols, not resistant to node death in between of storing PR data in DLM_CKV and other error cases.

Quoted 5 lines

Is it possible and would it be valuable to at least kind of break this up a little?

I would break this up and post the fixes in one set. I'll help you get them in as soon as possible.

After approve of the idea I can break the patch set to several ones and start to post it without RFC prefix. The only problem is that they all depend on previous ones. So I have to post each after the previous gets merged.

Quoted 10 lines

For patches 1-9, I think I remember you posting them before, but I was in the middle of starting a new job so I didn't review them. I really needed something like that at my last 2 jobs so I think it's a valuable feature and I'll review that as well.

If we could at least get those 2 chunks separated then it would make the DLM parts below easier to get eyeballs on. I'm ok with the idea in general. I think every nvmet developer will see the massive patchset and not even look at this first 0/48 email :)

I am not going to share this patchset to nvmet dev list :) nvmet does not yet have a local version of CompareAndWrite and Reservations features, so it is too early for them.

Quoted 40 lines
Quoted 40 lines

Patches 30-34:
DLM_CKV module that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.

Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.

Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attrib cluster_impl regulates which implementation to use for that device. 'single' is for default (no cluster) implementation.

Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* Use DLM_CKV Lock service to serialize order of PR OUT commands
* Use DLM_CKV Key-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* Use DLM_CKV Notification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.

It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster