Target cluster implementation over DLM
Subsystems (2)
cover Target cluster implementation over DLM
Hi linux target comminity.
Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.
The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.
Patches 1-9:
Make RTPI be part of se_tpg instead of se_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).
Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undepend pr_reg from se_nacl and se_tpg to be just a registration holder. Make APTPL registrations (not linked to se_dev_entry) be full-fledged registrations.
Patches 30-34:DLM_CKV module that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.
Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.
Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attrib cluster_impl regulates which implementation to use for that device. 'single' is for default (no cluster) implementation.
Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* Use DLM_CKV Lock service to serialize order of PR OUT commands
* Use DLM_CKV Key-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* Use DLM_CKV Notification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.
It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster
How to test
1. Setup DLM over worked corosync & pacemaker cluster
$ zypper install libdlm libdlm3 $ crm configure primitive dlm ocf:pacemaker:controld args="-q0 -f0" allow_stonith_disabled=true op monitor interval="60" timeout="60" $ crm configure clone clone-dlm dlm meta interleave=true target-role=Started 2. setup TCM cluster configuration You can use my script [2] that I used to use while developing the patchset. 3. Test what you want. For example, my test report [3].
[1] https://lore.kernel.org/all/[email protected]/
[2] https://pastebin.com/HgCfjywh
[3] https://pastebin.com/AgLSgnWn
Dmitry Bogdanov (39):
target: core: check RTPI uniquity for enabled TPG
target: core: fix preempt and abort for allreg res
target: core: fix memory leak in preempt_and_abort
target: core: abort all preempted regs if requested
target: core: new key must be used for moved PR
target: core: remove unused variable in se_dev_entry
target: core: undepend PR registrant of nacl
target: core: make some functions public
target: core: proper clear reservation on LUN RESET
target: core: remove superfluous checks
target: core: proper check of SCSI-2 reservation
target: core: checks against peer node SCSI2 reservation
target: core: UA on all luns after reset
target: core: refactor LUN_RESET code
target: core: pr: use RTPI in APTPL
target: core: pr: have Transport ID stored
target: core: pr: remove se_tpg from pr_reg
target: core: fix parsing PR OUT TID
target: core: add function to compare TransportID
target: core: store proto_id in APTPL
target: core: rethink APTPL registrations
dlm_ckv: introduce DLM cluster key-value storage
dlm_ckv: add notification service
dlm_ckv: add key-value storage service
dlm_ckv: add KV get/set async API
target: add virtual remote target
target: cluster: introduce cluster ops
target: cluster: introduce dlm cluster
target: cluster: store PR data in DLM cluster
target: cluster: read PR data from cluster
target: cluster: sync PR for dynamic acls
target: cluster: sync-up PR data on cluster join
target: cluster: sync SPC-2 reservations
target: cluster: allocate UAs on PR sync
target: cluster: support PR OUT preempt and abort
target: cluster: add reset cluster function
target: cluster: implement LUN reset in DLM cluster
target: cluster: split cluster sync function
target: cluster: request data on initial sync
Konstantin Shelekhin (1):
scsi: target/core: Unlock PR generation bump
Roman Bolshakov (8):
scsi: target/core: Add a way to hide a port group
scsi: target/core: Set MULTIP bit for se_device with multiple ports
scsi: target/core: Add cleanup sequence in core_tpg_register()
scsi: target/core: Add RTPI field to target port
scsi: target/core: Use RTPI from target port
scsi: target/core: Drop device-based RTPI
scsi: target/core: Add common port attributes
scsi: target/core: Add RTPI attribute for target port
drivers/target/Kconfig | 7 + drivers/target/Makefile | 4 + drivers/target/dlm_ckv.c | 757 +++++++++++++ drivers/target/dlm_ckv.h | 44 + drivers/target/target_cluster_dlm.c | 1012 ++++++++++++++++++ drivers/target/target_core_alua.c | 12 +- drivers/target/target_core_configfs.c | 191 +++- drivers/target/target_core_device.c | 244 ++++- drivers/target/target_core_fabric_configfs.c | 68 +- drivers/target/target_core_fabric_lib.c | 262 +++-- drivers/target/target_core_internal.h | 29 +- drivers/target/target_core_pr.c | 613 +++++------ drivers/target/target_core_pr.h | 33 +- drivers/target/target_core_sbc.c | 12 +- drivers/target/target_core_spc.c | 19 +- drivers/target/target_core_stat.c | 6 +- drivers/target/target_core_tmr.c | 38 +- drivers/target/target_core_tpg.c | 193 +++- drivers/target/target_core_transport.c | 9 +- drivers/target/target_core_ua.c | 1 + drivers/target/tcm_remote/Kconfig | 8 + drivers/target/tcm_remote/Makefile | 2 + drivers/target/tcm_remote/tcm_remote.c | 405 +++++++ drivers/target/tcm_remote/tcm_remote.h | 29 + include/target/target_core_base.h | 56 +- 25 files changed, 3436 insertions(+), 618 deletions(-) create mode 100644 drivers/target/dlm_ckv.c create mode 100644 drivers/target/dlm_ckv.h create mode 100644 drivers/target/target_cluster_dlm.c create mode 100644 drivers/target/tcm_remote/Kconfig create mode 100644 drivers/target/tcm_remote/Makefile create mode 100644 drivers/target/tcm_remote/tcm_remote.c create mode 100644 drivers/target/tcm_remote/tcm_remote.h
Signature · 1 line
2.25.1
RFC PATCH 1/48 scsi: target/core: Add a way to hide a port group
From: Roman Bolshakov <[email protected]>
Default target port group is always returned in the list of port groups, even if the behaviour is unwanted, i.e. it has no members and non-default port groups are primary port groups.
A new port group attribute - "hidden" can be used to hide empty port groups with no ports in REPORT TARGET PORT GROUPS, including default target port group:
echo 1 > $DEVICE/alua/default_tg_pt_gp/hidden
--- drivers/target/target_core_alua.c | 8 ++++++++ drivers/target/target_core_configfs.c | 29 +++++++++++++++++++++++++++ include/target/target_core_base.h | 1 + 3 files changed, 38 insertions(+)
drivers/target/target_core_alua.c +8 -0
@@ -164,6 +164,8 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)spin_lock (&dev->t10_alua.tg_pt_gps_lock);list_for_each_entry (tg_pt_gp, &dev->t10_alua.tg_pt_gps_list, tg_pt_gp_list) { +if (tg_pt_gp->tg_pt_gp_hidden && !tg_pt_gp->tg_pt_gp_members) +continue ;/* * Check if the Target port group and Target port descriptor list * based on tg_pt_gp_members count will fit into the response payload.
@@ -308,6 +310,12 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd) rc =TCM_UNSUPPORTED_SCSI_OPCODE ; goto out; } +if (l_tg_pt_gp->tg_pt_gp_hidden && !tg_pt_gp->tg_pt_gp_members) { +spin_unlock (&l_lun->lun_tg_pt_gp_lock); +pr_err ("Unable to change state for hidden port group with no members\n" ); + rc =TCM_INVALID_CDB_FIELD ; + goto out; + } valid_states = l_tg_pt_gp->tg_pt_gp_alua_supported_states;rcu_read_unlock ();
drivers/target/target_core_configfs.c +29 -0
@@ -3029,6 +3029,33 @@ static ssize_t target_tg_pt_gp_preferred_store(struct config_item *item,return core_alua_store_preferred_bit (to_tg_pt_gp (item), page, count); } +static ssize_t target_tg_pt_gp_hidden_show (struct config_item *item, +char *page) +{ +return sprintf (page,"%d\n" ,to_tg_pt_gp (item)->tg_pt_gp_hidden); +} + +static ssize_t target_tg_pt_gp_hidden_store (struct config_item *item, +const char *page,size_t count) +{ +struct t10_alua_tg_pt_gp *tg_pt_gp =to_tg_pt_gp (item); +int tmp, ret; + + ret =kstrtoint (page,0 , &tmp); +if (ret <0 ) { +pr_err ("Unable to extract hidden flag: %d\n" , ret); +return ret; + } + +if (tmp !=0 && tmp !=1 ) { +pr_err ("Illegal value for hidden flag: %d\n" , tmp); +return -EINVAL ; + } + tg_pt_gp->tg_pt_gp_hidden = tmp; + +return count; +} +static ssize_t target_tg_pt_gp_tg_pt_gp_id_show (struct config_item *item,char *page) {
@@ -3115,6 +3142,7 @@ CONFIGFS_ATTR(target_tg_pt_gp_, alua_support_standby);CONFIGFS_ATTR (target_tg_pt_gp_, alua_support_active_optimized);CONFIGFS_ATTR (target_tg_pt_gp_, alua_support_active_nonoptimized);CONFIGFS_ATTR (target_tg_pt_gp_, alua_write_metadata); +CONFIGFS_ATTR (target_tg_pt_gp_, hidden);CONFIGFS_ATTR (target_tg_pt_gp_, nonop_delay_msecs);CONFIGFS_ATTR (target_tg_pt_gp_, trans_delay_msecs);CONFIGFS_ATTR (target_tg_pt_gp_, implicit_trans_secs);
@@ -3138,6 +3166,7 @@ static struct configfs_attribute *target_core_alua_tg_pt_gp_attrs[] = { &target_tg_pt_gp_attr_trans_delay_msecs, &target_tg_pt_gp_attr_implicit_trans_secs, &target_tg_pt_gp_attr_preferred, + &target_tg_pt_gp_attr_hidden, &target_tg_pt_gp_attr_tg_pt_gp_id, &target_tg_pt_gp_attr_members, NULL,
include/target/target_core_base.h +1 -0
@@ -297,6 +297,7 @@ struct t10_alua_tg_pt_gp {int tg_pt_gp_trans_delay_msecs;int tg_pt_gp_implicit_trans_secs;int tg_pt_gp_pref; +int tg_pt_gp_hidden;int tg_pt_gp_write_metadata;u32 tg_pt_gp_members;int tg_pt_gp_alua_access_state;
Signature · 1 line
2.25.1
RFC PATCH 2/48 scsi: target/core: Set MULTIP bit for se_device with multiple ports
In reply to Dmitry BogdanovFrom: Roman Bolshakov <[email protected]>
SAM-5 4.8.3 (SCSI target device with multiple SCSI ports structure) obligates to set MULTIP bit when there's multiple SCSI target ports:
Quoted 3 lines
Each device server shall indicate the presence of multiple SCSI target ports by setting the MULTIP bit to one in its standard INQUIRY data (see SPC-4).
The change sets MULTIP bit automatically to indicate the presence of multiple SCSI target ports within standard inquiry response data if there are multiple target ports in all target port groups of the se_device.
--- drivers/target/target_core_spc.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
drivers/target/target_core_spc.c +17 -0
@@ -75,6 +75,8 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)struct se_portal_group *tpg = lun->lun_tpg;struct se_device *dev = cmd->se_dev;struct se_session *sess = cmd->se_sess; +struct t10_alua_tg_pt_gp *tg_pt_gp; +u64 dev_ports =0 ;/* Set RMB (removable media) for tape devices */ if (dev->transport->get_device_type (dev) ==TYPE_TAPE )
@@ -115,6 +117,21 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf) buf[5 ] |=0x1 ; } +spin_lock (&dev->t10_alua.tg_pt_gps_lock); +list_for_each_entry (tg_pt_gp, &dev->t10_alua.tg_pt_gps_list, + tg_pt_gp_list) { + dev_ports += tg_pt_gp->tg_pt_gp_members; +/* Exact number of ports does not matter for MULTIP bit */ +if (dev_ports >1 ) +break ; + } +spin_unlock (&dev->t10_alua.tg_pt_gps_lock); +/* +* Set MULTIP bit to indicate presence of multiple SCSI target ports +*/ +if (dev_ports >1 ) + buf[6 ] |=0x10 ; + buf[7 ] =0x2 ;/* CmdQue=1 */ /*
Signature · 1 line
2.25.1
RFC PATCH 3/48 scsi: target/core: Add cleanup sequence in core_tpg_register()
In reply to Dmitry BogdanovFrom: Roman Bolshakov <[email protected]>
It does not change any functionality but allows to introduce more steps in the cleanup sequence without code duplication later.
--- drivers/target/target_core_tpg.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/target/target_core_tpg.c +6 -4
@@ -484,10 +484,8 @@ int core_tpg_register( ret =core_tpg_add_lun (se_tpg, se_tpg->tpg_virt_lun0, true, g_lun0_dev); -if (ret <0 ) { -kfree (se_tpg->tpg_virt_lun0); -return ret; - } +if (ret <0 ) + goto out_free_lun0; } pr_debug("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, "
@@ -497,6 +495,10 @@ int core_tpg_register( se_tpg->proto_id, se_tpg->se_tpg_tfo->tpg_get_tag(se_tpg));return 0 ; + +out_free_lun0: +kfree (se_tpg->tpg_virt_lun0); +return ret; }EXPORT_SYMBOL (core_tpg_register);
Signature · 1 line
2.25.1
RFC PATCH 4/48 scsi: target/core: Add RTPI field to target port
In reply to Dmitry BogdanovFrom: Roman Bolshakov <[email protected]>
SAM-5 4.6.5.2 (Relative Port Identifier attribute) defines the attribute as unique across SCSI target ports.
The change introduces RTPI attribute to se_portal group. The value is auto-incremented and unique across all SCSI target ports. It also limits number of SCSI target ports to 65535.
--- drivers/target/target_core_tpg.c | 78 +++++++++++++++++++++++++++++-- include/target/target_core_base.h | 4 ++ 2 files changed, 77 insertions(+), 5 deletions(-)
drivers/target/target_core_tpg.c +73 -5
@@ -31,6 +31,10 @@#include "target_core_ua.h" extern struct se_device *g_lun0_dev; +static u16 g_tpg_count; +static u16 g_tpg_rtpi_counter =1 ; +static LIST_HEAD (g_tpg_list); +static DEFINE_SPINLOCK (g_tpg_lock);/* __core_tpg_get_initiator_node_acl(): *
@@ -439,6 +443,57 @@ static void core_tpg_lun_ref_release(struct percpu_ref *ref)complete (&lun->lun_shutdown_comp); } +static int core_tpg_register_rtpi (struct se_portal_group *se_tpg) +{ +struct se_portal_group *tpg; + +/* +* Allocate the next RELATIVE TARGET PORT IDENTIFIER. +* Here is the table from SPC-4 4.3.4: +* +* Table 34 -- Relative target port identifier values +* +* Value Description +* 0h Reserved +* 1h Relative port 1, historically known as port A +* 2h Relative port 2, historically known as port B +* 3h to FFFFh Relative port 3 through 65 535 +*/ +spin_lock (&g_tpg_lock); + +if (g_tpg_count ==0xffff ) { +spin_unlock (&g_tpg_lock); +pr_warn ("Reached g_tpg_count == 0xffff\n" ); +return -ENOSPC ; + } +again: + se_tpg->tpg_rtpi = g_tpg_rtpi_counter++; +if (!se_tpg->tpg_rtpi) + goto again; + +list_for_each_entry (tpg, &g_tpg_list, tpg_list) { +/* +* Make sure RELATIVE TARGET PORT IDENTIFIER is unique +* for 16-bit wrap.. +*/ +if (se_tpg->tpg_rtpi == tpg->tpg_rtpi) + goto again; + } +list_add (&se_tpg->tpg_list, &g_tpg_list); + g_tpg_count++; +spin_unlock (&g_tpg_lock); + +return 0 ; +} + +static void core_tpg_deregister_rtpi (struct se_portal_group *se_tpg) +{ +spin_lock (&g_tpg_lock); +list_del (&se_tpg->tpg_list); + g_tpg_count--; +spin_unlock (&g_tpg_lock); +} +/* Does not change se_wwn->priv. */ int core_tpg_register( struct se_wwn *se_wwn,
@@ -471,6 +526,7 @@ int core_tpg_register( se_tpg->proto_id = proto_id; se_tpg->se_tpg_wwn = se_wwn;atomic_set (&se_tpg->tpg_pr_ref_count,0 ); +INIT_LIST_HEAD (&se_tpg->tpg_list);INIT_LIST_HEAD (&se_tpg->acl_node_list);INIT_LIST_HEAD (&se_tpg->tpg_sess_list);spin_lock_init (&se_tpg->session_lock);
@@ -478,9 +534,15 @@ int core_tpg_register(mutex_init (&se_tpg->acl_node_mutex);if (se_tpg->proto_id >=0 ) { + ret =core_tpg_register_rtpi (se_tpg); +if (ret <0 ) +return ret; + se_tpg->tpg_virt_lun0 =core_tpg_alloc_lun (se_tpg,0 ); -if (IS_ERR (se_tpg->tpg_virt_lun0)) -return PTR_ERR (se_tpg->tpg_virt_lun0); +if (IS_ERR (se_tpg->tpg_virt_lun0)) { + ret =PTR_ERR (se_tpg->tpg_virt_lun0); + goto out_deregister_rtpi; + } ret =core_tpg_add_lun (se_tpg, se_tpg->tpg_virt_lun0, true, g_lun0_dev);
@@ -488,16 +550,20 @@ int core_tpg_register( goto out_free_lun0; } -pr_debug ("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, " -"Proto: %d, Portal Tag: %u\n" , se_tpg->se_tpg_tfo->fabric_name, +pr_debug ("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, Proto: %d, Portal Tag: %u, RTPI: %#2x\n" , + se_tpg->se_tpg_tfo->fabric_name, se_tpg->se_tpg_tfo->tpg_get_wwn (se_tpg) ? se_tpg->se_tpg_tfo->tpg_get_wwn (se_tpg) :NULL , - se_tpg->proto_id, se_tpg->se_tpg_tfo->tpg_get_tag (se_tpg)); + se_tpg->proto_id, + se_tpg->se_tpg_tfo->tpg_get_tag (se_tpg), + se_tpg->tpg_rtpi);return 0 ; out_free_lun0:kfree (se_tpg->tpg_virt_lun0); +out_deregister_rtpi: +core_tpg_deregister_rtpi (se_tpg);return ret; }EXPORT_SYMBOL (core_tpg_register);
@@ -537,6 +603,8 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)kfree_rcu (se_tpg->tpg_virt_lun0, rcu_head); } +core_tpg_deregister_rtpi (se_tpg); +return 0 ; }EXPORT_SYMBOL (core_tpg_deregister);
include/target/target_core_base.h +4 -0
@@ -904,6 +904,8 @@ struct se_portal_group {*/ int proto_id;bool enabled; +/* RELATIVE TARGET PORT IDENTIFIER */ +u16 tpg_rtpi;/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */ atomic_t tpg_pr_ref_count;/* Spinlock for adding/removing ACLed Nodes */
@@ -911,6 +913,8 @@ struct se_portal_group {/* Spinlock for adding/removing sessions */ spinlock_t session_lock;struct mutex tpg_lun_mutex; +/* List of all SCSI target ports */ +struct list_head tpg_list;/* linked list for initiator ACL list */ struct list_head acl_node_list;struct hlist_head tpg_lun_hlist;
Signature · 1 line
2.25.1
RFC PATCH 5/48 scsi: target/core: Use RTPI from target port
In reply to Dmitry BogdanovFrom: Roman Bolshakov <[email protected]>
Replace all references to RTPI from LUN field to se_portal_group field. It introduces consitent reporting of RTPI for all LUNs and all target ports.
--- drivers/target/target_core_alua.c | 4 ++-- drivers/target/target_core_device.c | 2 +- drivers/target/target_core_pr.c | 8 ++++---- drivers/target/target_core_spc.c | 2 +- drivers/target/target_core_stat.c | 6 +++--- 5 files changed, 11 insertions(+), 11 deletions(-)
drivers/target/target_core_alua.c +2 -2
@@ -224,7 +224,7 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)/* * Set RELATIVE TARGET PORT IDENTIFIER */ -put_unaligned_be16 (lun->lun_rtpi, &buf[off]); +put_unaligned_be16 (lun->lun_tpg->tpg_rtpi, &buf[off]); off +=2 ; rd_len +=4 ; }
@@ -404,7 +404,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)spin_lock (&dev->se_port_lock);list_for_each_entry (lun, &dev->dev_sep_list, lun_dev_link) { -if (lun->lun_rtpi != rtpi) +if (lun->lun_tpg->tpg_rtpi != rtpi)continue ;// XXX: racy unlock
drivers/target/target_core_device.c +1 -1
@@ -223,7 +223,7 @@ struct se_dev_entry *core_get_se_deve_from_rtpi( tpg->se_tpg_tfo->fabric_name);continue ; } -if (lun->lun_rtpi != rtpi) +if (lun->lun_tpg->tpg_rtpi != rtpi)continue ;kref_get (&deve->pr_kref);
drivers/target/target_core_pr.c +4 -4
@@ -663,7 +663,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration( } pr_reg->pr_res_mapped_lun = mapped_lun; pr_reg->pr_aptpl_target_lun = lun->unpacked_lun; - pr_reg->tg_pt_sep_rtpi = lun->lun_rtpi; + pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi; pr_reg->pr_res_key = sa_res_key; pr_reg->pr_reg_all_tg_pt = all_tg_pt; pr_reg->pr_reg_aptpl = aptpl;
@@ -969,7 +969,7 @@ static int __core_scsi3_check_aptpl_registration(rcu_read_unlock (); pr_reg->pr_reg_nacl = nacl; - pr_reg->tg_pt_sep_rtpi = lun->lun_rtpi; + pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;list_del (&pr_reg->pr_reg_aptpl_list);spin_unlock (&pr_tmpl->aptpl_reg_lock);/*
@@ -1577,7 +1577,7 @@ core_scsi3_decode_spec_i_port(*/ if (tmp_tpg->proto_id != proto_ident)continue ; - dest_rtpi = tmp_lun->lun_rtpi; + dest_rtpi = tmp_lun->lun_tpg->tpg_rtpi; iport_ptr =NULL ; i_str = target_parse_pr_out_transport_id(tmp_tpg,
@@ -3221,7 +3221,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,spin_lock (&dev->se_port_lock);list_for_each_entry (tmp_lun, &dev->dev_sep_list, lun_dev_link) { -if (tmp_lun->lun_rtpi != rtpi) +if (tmp_lun->lun_tpg->tpg_rtpi != rtpi)continue ; dest_se_tpg = tmp_lun->lun_tpg; dest_tf_ops = dest_se_tpg->se_tpg_tfo;
drivers/target/target_core_spc.c +1 -1
@@ -332,7 +332,7 @@ spc_emulate_evpd_83(struct se_cmd *cmd, unsigned char *buf)/* Skip over Obsolete field in RTPI payload * in Table 472 */ off +=2 ; -put_unaligned_be16 (lun->lun_rtpi, &buf[off]); +put_unaligned_be16 (lun->lun_tpg->tpg_rtpi, &buf[off]); off +=2 ; len +=8 ;/* Header size + Designation descriptor */ /*
drivers/target/target_core_stat.c +3 -3
@@ -455,7 +455,7 @@ static ssize_t target_stat_port_indx_show(struct config_item *item, char *page)rcu_read_lock (); dev =rcu_dereference (lun->lun_se_dev);if (dev) - ret =snprintf (page,PAGE_SIZE ,"%u\n" , lun->lun_rtpi); + ret =snprintf (page,PAGE_SIZE ,"%u\n" , lun->lun_tpg->tpg_rtpi);rcu_read_unlock ();return ret; }
@@ -561,7 +561,7 @@ static ssize_t target_stat_tgt_port_indx_show(struct config_item *item,rcu_read_lock (); dev =rcu_dereference (lun->lun_se_dev);if (dev) - ret =snprintf (page,PAGE_SIZE ,"%u\n" , lun->lun_rtpi); + ret =snprintf (page,PAGE_SIZE ,"%u\n" , lun->lun_tpg->tpg_rtpi);rcu_read_unlock ();return ret; }
@@ -579,7 +579,7 @@ static ssize_t target_stat_tgt_port_name_show(struct config_item *item,if (dev) ret =snprintf (page,PAGE_SIZE ,"%sPort#%u\n" , tpg->se_tpg_tfo->fabric_name, - lun->lun_rtpi); + lun->lun_tpg->tpg_rtpi);rcu_read_unlock ();return ret; }
Signature · 1 line
2.25.1
RFC PATCH 6/48 scsi: target/core: Drop device-based RTPI
In reply to Dmitry BogdanovFrom: Roman Bolshakov <[email protected]>
The code is not needed since target port-based RTPI allocation replaced it.
--- drivers/target/target_core_device.c | 41 --------------------------- drivers/target/target_core_internal.h | 1 - drivers/target/target_core_tpg.c | 6 ---- include/target/target_core_base.h | 4 --- 4 files changed, 52 deletions(-)
drivers/target/target_core_device.c +0 -41
@@ -470,47 +470,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)mutex_unlock (&tpg->acl_node_mutex); } -int core_alloc_rtpi (struct se_lun *lun,struct se_device *dev) -{ -struct se_lun *tmp; - -spin_lock (&dev->se_port_lock); -if (dev->export_count ==0x0000ffff ) { -pr_warn ("Reached dev->dev_port_count ==" -" 0x0000ffff\n" ); -spin_unlock (&dev->se_port_lock); -return -ENOSPC ; - } -again: -/* -* Allocate the next RELATIVE TARGET PORT IDENTIFIER for this struct se_device -* Here is the table from spc4r17 section 7.7.3.8. -* -* Table 473 -- RELATIVE TARGET PORT IDENTIFIER field -* -* Code Description -* 0h Reserved -* 1h Relative port 1, historically known as port A -* 2h Relative port 2, historically known as port B -* 3h to FFFFh Relative port 3 through 65 535 -*/ - lun->lun_rtpi = dev->dev_rpti_counter++; -if (!lun->lun_rtpi) - goto again; - -list_for_each_entry (tmp, &dev->dev_sep_list, lun_dev_link) { -/* -* Make sure RELATIVE TARGET PORT IDENTIFIER is unique -* for 16-bit wrap.. -*/ -if (lun->lun_rtpi == tmp->lun_rtpi) - goto again; - } -spin_unlock (&dev->se_port_lock); - -return 0 ; -} -static void se_release_vpd_for_dev (struct se_device *dev) {struct t10_vpd *vpd, *vpd_tmp;
drivers/target/target_core_internal.h +0 -1
@@ -59,7 +59,6 @@ struct target_fabric_configfs {extern struct t10_alua_lu_gp *default_lu_gp;/* target_core_device.c */ -int core_alloc_rtpi (struct se_lun *lun,struct se_device *dev);struct se_dev_entry *core_get_se_deve_from_rtpi (struct se_node_acl *,u16 );void target_pr_kref_release (struct kref *); void core_free_device_list_for_node(struct se_node_acl *,
drivers/target/target_core_tpg.c +0 -6
@@ -648,10 +648,6 @@ int core_tpg_add_lun(if (ret <0 ) goto out; - ret =core_alloc_rtpi (lun, dev); -if (ret) - goto out_kill_ref; -if (!(dev->transport_flags &TRANSPORT_FLAG_PASSTHROUGH_ALUA ) && !(dev->se_hba->hba_flags &HBA_FLAGS_INTERNAL_USE ))target_attach_tg_pt_gp (lun, dev->t10_alua.default_tg_pt_gp);
@@ -675,8 +671,6 @@ int core_tpg_add_lun(return 0 ; -out_kill_ref: -percpu_ref_exit (&lun->lun_ref); out:return ret; }
include/target/target_core_base.h +0 -4
@@ -734,8 +734,6 @@ struct se_lun {bool lun_access_ro;u32 lun_index; -/* RELATIVE TARGET PORT IDENTIFER */ -u16 lun_rtpi;atomic_t lun_acl_count;struct se_device __rcu *lun_se_dev;
@@ -787,8 +785,6 @@ struct se_device_queue { };struct se_device { -/* RELATIVE TARGET PORT IDENTIFER Counter */ -u16 dev_rpti_counter;/* Used for SAM Task Attribute ordering */ u32 dev_cur_ordered_id;u32 dev_flags;
Signature · 1 line
2.25.1
RFC PATCH 7/48 scsi: target/core: Add common port attributes
In reply to Dmitry BogdanovFrom: Roman Bolshakov <[email protected]>
All SCSI target port attributes (tpgN/attribs/attrname) are defined and implemented in fabric modules in existing implementation.
The change introduces a way to have common tpg attribs in configfs for all fabrics without code duplication.
--- drivers/target/target_core_configfs.c | 9 ++++- drivers/target/target_core_fabric_configfs.c | 39 +++++++++++++++++++- drivers/target/target_core_internal.h | 1 + drivers/target/target_core_tpg.c | 4 ++ 4 files changed, 50 insertions(+), 3 deletions(-)
drivers/target/target_core_configfs.c +8 -1
@@ -464,13 +464,19 @@ int target_register_template(const struct target_core_fabric_ops *fo)INIT_LIST_HEAD (&tf->tf_list);atomic_set (&tf->tf_access_cnt,0 ); tf->tf_ops = fo; -target_fabric_setup_cits (tf); + ret =target_fabric_setup_cits (tf); +if (ret) + goto out;mutex_lock (&g_tf_lock);list_add_tail (&tf->tf_list, &g_tf_list);mutex_unlock (&g_tf_lock);return 0 ; + +out: +kfree (tf); +return ret; }EXPORT_SYMBOL (target_register_template);
@@ -491,6 +497,7 @@ void target_unregister_template(const struct target_core_fabric_ops *fo)*/ rcu_barrier ();kfree (t->tf_tpg_base_cit.ct_attrs); +kfree (t->tf_tpg_attrib_cit.ct_attrs);kfree (t);return ; }
drivers/target/target_core_fabric_configfs.c +37 -2
@@ -795,7 +795,38 @@ TF_CIT_SETUP(tpg_lun, NULL, &target_fabric_lun_group_ops, NULL);/* End of tfc_tpg_lun_cit */ -TF_CIT_SETUP_DRV (tpg_attrib,NULL ,NULL ); +static int +target_fabric_setup_tpg_attrib_cit (struct target_fabric_configfs *tf) +{ +int i, k, len =0 ; +struct config_item_type *cit = &tf->tf_tpg_attrib_cit; +struct configfs_attribute **attrs; + +for (i =0 ; core_tpg_attrib_attrs[i]; i++) + len +=sizeof (struct configfs_attribute *); +if (tf->tf_ops->tfc_tpg_attrib_attrs) +for (i =0 ; tf->tf_ops->tfc_tpg_attrib_attrs[i]; i++) + len +=sizeof (struct configfs_attribute *); + len +=sizeof (struct configfs_attribute *); + + attrs =kzalloc (len,GFP_KERNEL ); +if (!attrs) +return -ENOMEM ; + +for (i =0 ; core_tpg_attrib_attrs[i]; i++) + attrs[i] = core_tpg_attrib_attrs[i]; +if (tf->tf_ops->tfc_tpg_attrib_attrs) +for (k =0 ; tf->tf_ops->tfc_tpg_attrib_attrs[k]; k++, i++) + attrs[i] = tf->tf_ops->tfc_tpg_attrib_attrs[k]; + attrs[i] =NULL ; + + cit->ct_attrs = attrs; + cit->ct_owner = tf->tf_ops->module; +pr_debug ("Setup generic tpg_attrib\n" ); + +return 0 ; +} +TF_CIT_SETUP_DRV (tpg_auth,NULL ,NULL );TF_CIT_SETUP_DRV (tpg_param,NULL ,NULL );
@@ -1113,7 +1144,11 @@ int target_fabric_setup_cits(struct target_fabric_configfs *tf)target_fabric_setup_tpg_lun_cit (tf);target_fabric_setup_tpg_np_cit (tf);target_fabric_setup_tpg_np_base_cit (tf); -target_fabric_setup_tpg_attrib_cit (tf); + + ret =target_fabric_setup_tpg_attrib_cit (tf); +if (ret) +return ret; +target_fabric_setup_tpg_auth_cit (tf);target_fabric_setup_tpg_param_cit (tf);target_fabric_setup_tpg_nacl_cit (tf);
drivers/target/target_core_internal.h +1 -0
@@ -117,6 +117,7 @@ int core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,/* target_core_tpg.c */ extern struct se_device *g_lun0_dev; +extern struct configfs_attribute *core_tpg_attrib_attrs[];struct se_node_acl *__core_tpg_get_initiator_node_acl (struct se_portal_group *tpg,const char *);
drivers/target/target_core_tpg.c +4 -0
@@ -714,3 +714,7 @@ void core_tpg_remove_lun(percpu_ref_exit (&lun->lun_ref); } + +struct configfs_attribute *core_tpg_attrib_attrs[] = { +NULL , +};
Signature · 1 line
2.25.1
RFC PATCH 8/48 scsi: target/core: Add RTPI attribute for target port
In reply to Dmitry BogdanovFrom: Roman Bolshakov <[email protected]>
RELATIVE TARGET PORT IDENTIFIER can be read and configured via configfs:
$ echo 0x10 > $TARGET/tpgt_N/attrib/rtpi
RTPI has to be unique among all target ports, otherwise the write fails.
Configuration of a new RTPI value triggers INQUIRY DATA HAS CHANGED unit attention as defined in SPC-4 "6.6.1 INQUIRY command introduction":
Quoted 4 lines
If INQUIRY data changes for any reason, the device server shall establish a unit attention condition for the initiator port associated with every
I_Tnexus (see SAM-5), with the additional sense code set to INQUIRY DATA HAS CHANGED.
--- drivers/target/target_core_tpg.c | 70 ++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
drivers/target/target_core_tpg.c +70 -0
@@ -715,6 +715,76 @@ void core_tpg_remove_lun(percpu_ref_exit (&lun->lun_ref); } +/* Set a Unit Attention on all I_T nexuses related to the port */ +static void core_tpg_ua (struct se_portal_group *se_tpg,u8 asc,u8 ascq) +{ +struct se_lun *lun; +struct se_dev_entry *se_deve; + +mutex_lock (&se_tpg->tpg_lun_mutex); +hlist_for_each_entry_rcu (lun, &se_tpg->tpg_lun_hlist, link) { +spin_lock (&lun->lun_deve_lock); +list_for_each_entry (se_deve, &lun->lun_deve_list, lun_link) +core_scsi3_ua_allocate (se_deve, asc, ascq); +spin_unlock (&lun->lun_deve_lock); + } +mutex_unlock (&se_tpg->tpg_lun_mutex); +} + +static ssize_t core_tpg_rtpi_show (struct config_item *item,char *page) +{ +struct se_portal_group *se_tpg =attrib_to_tpg (item); + +return sprintf (page,"%#x\n" , se_tpg->tpg_rtpi); +} + +static ssize_t core_tpg_rtpi_store (struct config_item *item, +const char *page,size_t count) +{ +struct se_portal_group *se_tpg =attrib_to_tpg (item); +struct se_portal_group *tpg; +bool rtpi_changed = false; +u16 val; +int ret; + + ret =kstrtou16 (page,0 , &val); +if (ret <0 ) +return ret; +if (val ==0 ) +return -EINVAL ; + +/* RTPI shouldn't conflict with values of existing ports */ +spin_lock (&g_tpg_lock); + +list_for_each_entry (tpg, &g_tpg_list, tpg_list) { +if (se_tpg != tpg && val == tpg->tpg_rtpi) { +spin_unlock (&g_tpg_lock); +pr_err ("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n" , + se_tpg->se_tpg_tfo->fabric_name, + se_tpg->se_tpg_tfo->tpg_get_tag (tpg), + val, + tpg->se_tpg_tfo->fabric_name, + tpg->se_tpg_tfo->tpg_get_tag (tpg)); +return -EINVAL ; + } + } + +if (se_tpg->tpg_rtpi != val) { + se_tpg->tpg_rtpi = val; + rtpi_changed = true; + } +spin_unlock (&g_tpg_lock); + +if (rtpi_changed) +core_tpg_ua (se_tpg,0x3f ,ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED ); + ret = count; + +return ret; +} + +CONFIGFS_ATTR (core_tpg_, rtpi); +struct configfs_attribute *core_tpg_attrib_attrs[] = { + &core_tpg_attr_rtpi,NULL , };
Signature · 1 line
2.25.1
RFC PATCH 9/48 target: core: check RTPI uniquity for enabled TPG
In reply to Dmitry BogdanovGarantee uniquity of RTPI only for enabled target port groups. Allow any RPTI for disabled tpg until it is enabled.
--- drivers/target/target_core_fabric_configfs.c | 29 +++++++++++++- drivers/target/target_core_internal.h | 2 + drivers/target/target_core_tpg.c | 40 +++++++++++++------- 3 files changed, 56 insertions(+), 15 deletions(-)
drivers/target/target_core_fabric_configfs.c +28 -1
@@ -857,6 +857,7 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item, size_t count) {struct se_portal_group *se_tpg =to_tpg (item); +struct se_portal_group *tpg;int ret;bool op;
@@ -867,11 +868,37 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,if (se_tpg->enabled == op)return count; +spin_lock (&g_tpg_lock); + +if (op) { + tpg =core_get_tpg_by_rtpi (se_tpg->tpg_rtpi); +if (tpg) { +spin_unlock (&g_tpg_lock); + +pr_err ("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n" , + se_tpg->se_tpg_tfo->fabric_name, + se_tpg->se_tpg_tfo->tpg_get_tag (tpg), + se_tpg->tpg_rtpi, + tpg->se_tpg_tfo->fabric_name, + tpg->se_tpg_tfo->tpg_get_tag (tpg)); +return -EINVAL ; + } + } + + se_tpg->enabled |=0x10 ;/* transient state */ +spin_unlock (&g_tpg_lock); + ret = se_tpg->se_tpg_tfo->fabric_enable_tpg (se_tpg, op); -if (ret) + +spin_lock (&g_tpg_lock); +if (ret <0 ) { + se_tpg->enabled &= ~0x10 ;/* clear transient state */ +spin_unlock (&g_tpg_lock);return ret; + } se_tpg->enabled = op; +spin_unlock (&g_tpg_lock);return count; }
drivers/target/target_core_internal.h +2 -0
@@ -116,6 +116,7 @@ int core_tmr_lun_reset(struct se_device *, struct se_tmr_req *, struct list_head *, struct se_cmd *);/* target_core_tpg.c */ +extern struct spinlock g_tpg_lock;extern struct se_device *g_lun0_dev;extern struct configfs_attribute *core_tpg_attrib_attrs[];
@@ -131,6 +132,7 @@ void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);struct se_node_acl *core_tpg_add_initiator_node_acl (struct se_portal_group *tpg,const char *initiatorname);void core_tpg_del_initiator_node_acl (struct se_node_acl *acl); +struct se_portal_group *core_get_tpg_by_rtpi (u16 rtpi);/* target_core_transport.c */ extern struct kmem_cache *se_tmr_req_cache;
drivers/target/target_core_tpg.c +26 -14
@@ -34,7 +34,7 @@ extern struct se_device *g_lun0_dev;static u16 g_tpg_count;static u16 g_tpg_rtpi_counter =1 ;static LIST_HEAD (g_tpg_list); -static DEFINE_SPINLOCK (g_tpg_lock); +DEFINE_SPINLOCK (g_tpg_lock);/* __core_tpg_get_initiator_node_acl(): *
@@ -476,7 +476,7 @@ static int core_tpg_register_rtpi(struct se_portal_group *se_tpg)* Make sure RELATIVE TARGET PORT IDENTIFIER is unique * for 16-bit wrap.. */ -if (se_tpg->tpg_rtpi == tpg->tpg_rtpi) +if (tpg->enabled && se_tpg->tpg_rtpi == tpg->tpg_rtpi) goto again; }list_add (&se_tpg->tpg_list, &g_tpg_list);
@@ -738,12 +738,26 @@ static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)return sprintf (page,"%#x\n" , se_tpg->tpg_rtpi); } +struct se_portal_group * +core_get_tpg_by_rtpi (u16 rtpi) +{ +struct se_portal_group *tpg; + +lockdep_assert_held (&g_tpg_lock); + +list_for_each_entry (tpg, &g_tpg_list, tpg_list) { +if (tpg->enabled && rtpi == tpg->tpg_rtpi) +return tpg; + } + +return NULL ; +} +static ssize_t core_tpg_rtpi_store (struct config_item *item,const char *page,size_t count) {struct se_portal_group *se_tpg =attrib_to_tpg (item);struct se_portal_group *tpg; -bool rtpi_changed = false;u16 val;int ret;
@@ -753,11 +767,14 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,if (val ==0 )return -EINVAL ; -/* RTPI shouldn't conflict with values of existing ports */ +if (se_tpg->tpg_rtpi == val) +return count; +spin_lock (&g_tpg_lock); -list_for_each_entry (tpg, &g_tpg_list, tpg_list) { -if (se_tpg != tpg && val == tpg->tpg_rtpi) { +if (se_tpg->enabled) { + tpg =core_get_tpg_by_rtpi (val); +if (tpg) {spin_unlock (&g_tpg_lock); pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n", se_tpg->se_tpg_tfo->fabric_name,
@@ -769,17 +786,12 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item, } } -if (se_tpg->tpg_rtpi != val) { - se_tpg->tpg_rtpi = val; - rtpi_changed = true; - } + se_tpg->tpg_rtpi = val;spin_unlock (&g_tpg_lock); -if (rtpi_changed) -core_tpg_ua (se_tpg,0x3f ,ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED ); - ret = count; +core_tpg_ua (se_tpg,0x3f ,ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED ); -return ret; +return count; }CONFIGFS_ATTR (core_tpg_, rtpi);
Signature · 1 line
2.25.1
RFC PATCH 10/48 scsi: target/core: Unlock PR generation bump
In reply to Dmitry BogdanovFrom: Konstantin Shelekhin <[email protected]>
Get rid of spinlock and bump PR generation atomically. This is required to prevent deadlock later on.
--- drivers/target/target_core_configfs.c | 3 ++- drivers/target/target_core_pr.c | 25 ++++++------------------- include/target/target_core_base.h | 2 +- 3 files changed, 9 insertions(+), 21 deletions(-)
drivers/target/target_core_configfs.c +2 -1
@@ -1814,7 +1814,8 @@ static ssize_t target_pr_res_pr_all_tgt_pts_show(struct config_item *item,static ssize_t target_pr_res_pr_generation_show (struct config_item *item,char *page) { -return sprintf (page,"0x%08x\n" ,pr_to_dev (item)->t10_pr.pr_generation); +return sprintf (page,"0x%08x\n" , +atomic_read (&pr_to_dev (item)->t10_pr.pr_generation)); }
drivers/target/target_core_pr.c +6 -19
@@ -594,8 +594,6 @@ target_scsi3_pr_reservation_check(struct se_cmd *cmd)static u32 core_scsi3_pr_generation (struct se_device *dev) { -u32 prg; -/* * PRGeneration field shall contain the value of a 32-bit wrapping * counter mainted by the device server.
@@ -605,11 +603,7 @@ static u32 core_scsi3_pr_generation(struct se_device *dev)* * See spc4r17 section 6.3.12 READ_KEYS service action */ -spin_lock (&dev->dev_reservation_lock); - prg = dev->t10_pr.pr_generation++; -spin_unlock (&dev->dev_reservation_lock); - -return prg; +return atomic_inc_return (&dev->t10_pr.pr_generation); } static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
@@ -1057,15 +1051,8 @@ static void __core_scsi3_add_registration(/* * Increment PRgeneration counter for struct se_device upon a successful * REGISTER, see spc4r17 section 6.3.2 READ_KEYS service action -* -* Also, when register_move = 1 for PROUT REGISTER_AND_MOVE service -* action, the struct se_device->dev_reservation_lock will already be held, -* so we do not call core_scsi3_pr_generation() which grabs the lock -* for the REGISTER. */ - pr_reg->pr_res_generation = (register_move) ? - dev->t10_pr.pr_generation++ : -core_scsi3_pr_generation (dev); + pr_reg->pr_res_generation =core_scsi3_pr_generation (dev);spin_lock (&pr_tmpl->registration_lock);list_add_tail (&pr_reg->pr_reg_list, &pr_tmpl->registration_list);
@@ -3481,7 +3468,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,* Increment PRGeneration for existing registrations.. */ if (!new_reg) - dest_pr_reg->pr_res_generation = pr_tmpl->pr_generation++; + dest_pr_reg->pr_res_generation =core_scsi3_pr_generation (dev);spin_unlock (&dev->dev_reservation_lock); pr_debug("SPC-3 PR [%s] Service Action: REGISTER_AND_MOVE"
@@ -3703,7 +3690,7 @@ core_scsi3_pri_read_keys(struct se_cmd *cmd)if (!buf)return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; -put_unaligned_be32 (dev->t10_pr.pr_generation, buf); +put_unaligned_be32 (atomic_read (&dev->t10_pr.pr_generation), buf);spin_lock (&dev->t10_pr.registration_lock); list_for_each_entry(pr_reg, &dev->t10_pr.registration_list,
@@ -3758,7 +3745,7 @@ core_scsi3_pri_read_reservation(struct se_cmd *cmd)if (!buf)return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; -put_unaligned_be32 (dev->t10_pr.pr_generation, &buf[0 ]); +put_unaligned_be32 (atomic_read (&dev->t10_pr.pr_generation), &buf[0 ]);spin_lock (&dev->dev_reservation_lock); pr_reg = dev->dev_pr_res_holder;
@@ -3900,7 +3887,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)if (!buf)return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; -put_unaligned_be32 (dev->t10_pr.pr_generation, &buf[0 ]); +put_unaligned_be32 (atomic_read (&dev->t10_pr.pr_generation), &buf[0 ]);spin_lock (&dev->dev_reservation_lock);if (dev->dev_pr_res_holder) {
include/target/target_core_base.h +1 -1
@@ -385,7 +385,7 @@ struct t10_reservation {* for SCSI device */ int pr_aptpl_active;#define PR_APTPL_BUF_LEN 262144 -u32 pr_generation; +atomic_t pr_generation;spinlock_t registration_lock;spinlock_t aptpl_reg_lock;/*
Signature · 1 line
2.25.1
RFC PATCH 11/48 target: core: fix preempt and abort for allreg res
In reply to Dmitry BogdanovMatch a key only if SARK is not zero according to SPC-4 and the comment above the code:
If an all registrants persistent reservation is present and the SERVICE ACTION RESERVATION KEY field is set to zero, then all registrations shall be removed except for that of the I_T nexus that is being used for the PERSISTENT RESERVE OUT command;
--- drivers/target/target_core_pr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/target/target_core_pr.c +1 -1
@@ -3020,7 +3020,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,if (calling_it_nexus)continue ; -if (pr_reg->pr_res_key != sa_res_key) +if ((sa_res_key) && (pr_reg->pr_res_key != sa_res_key))continue ; pr_reg_nacl = pr_reg->pr_reg_nacl;
Signature · 1 line
2.25.1
RFC PATCH 12/48 target: core: fix memory leak in preempt_and_abort
In reply to Dmitry BogdanovAlways release preempt_and_abort_list to avoid memory leak of t10_pr_registration objects in it.
--- drivers/target/target_core_pr.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/target/target_core_pr.c +5 -4
@@ -2954,11 +2954,12 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,__core_scsi3_complete_pro_preempt (dev, pr_reg_n, (preempt_type ==PREEMPT_AND_ABORT ) ? &preempt_and_abort_list :NULL , type, scope, preempt_type); - -if (preempt_type ==PREEMPT_AND_ABORT ) -core_scsi3_release_preempt_and_abort ( - &preempt_and_abort_list, pr_reg_n); } + +if (preempt_type ==PREEMPT_AND_ABORT ) +core_scsi3_release_preempt_and_abort ( + &preempt_and_abort_list, pr_reg_n); +spin_unlock (&dev->dev_reservation_lock);if (pr_tmpl->pr_aptpl_active)
Signature · 1 line
2.25.1
RFC PATCH 13/48 target: core: abort all preempted regs if requested
In reply to Dmitry BogdanovAccording to SPC the aborting shall always happened.
SPC-4: 5.12.11.2.6 Preempting and aborting
If the SERVICE ACTION RESERVATION KEY field is set to zero and an all registrants persistent reservation is present, the device server shall abort all commands for all registered I_T nexuses.
--- drivers/target/target_core_pr.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/target/target_core_pr.c +15 -1
@@ -2956,9 +2956,23 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key, type, scope, preempt_type); } -if (preempt_type ==PREEMPT_AND_ABORT ) +/* +* SPC-4 5.12.11.2.6 Preempting and aborting +* The actions described in this subclause shall be performed +* for all I_T nexuses that are registered with the non-zero +* SERVICE ACTION RESERVATION KEY value, without regard for +* whether the preempted I_T nexuses hold the persistent +* reservation. If the SERVICE ACTION RESERVATION KEY field is +* set to zero and an all registrants persistent reservation is +* present, the device server shall abort all commands for all +* registered I_T nexuses. +*/ +if (preempt_type ==PREEMPT_AND_ABORT ) { +core_tmr_lun_reset (dev,NULL , &preempt_and_abort_list, + cmd);core_scsi3_release_preempt_and_abort ( &preempt_and_abort_list, pr_reg_n); + }spin_unlock (&dev->dev_reservation_lock);
Signature · 1 line
2.25.1
RFC PATCH 14/48 target: core: new key must be used for moved PR
In reply to Dmitry BogdanovAccording to SPC4 5.12.8:
e) Retain the reservation key specified in the SERVICE ACTION RESERVATION KEY field and associated information;
But currently sa_res_key is only used for the not existing I_T nexus. The patch adds a changing of the key for the existing I_T nexus the PR moved to.
--- drivers/target/target_core_pr.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/target/target_core_pr.c +6 -2
@@ -3438,8 +3438,6 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,* transport protocols where port names are not required; * d) Register the reservation key specified in the SERVICE ACTION * RESERVATION KEY field; -* e) Retain the reservation key specified in the SERVICE ACTION -* RESERVATION KEY field and associated information; * * Also, It is not an error for a REGISTER AND MOVE service action to * register an I_T nexus that is already registered with the same
@@ -3462,6 +3460,12 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key, dest_pr_reg =__core_scsi3_locate_pr_reg (dev, dest_node_acl, iport_ptr); new_reg =1 ; + }else { +/* +* e) Retain the reservation key specified in the SERVICE ACTION +* RESERVATION KEY field and associated information; +*/ + dest_pr_reg->pr_res_key = sa_res_key; }/* * f) Release the persistent reservation for the persistent reservation
Signature · 1 line
2.25.1
RFC PATCH 15/48 target: core: remove unused variable in se_dev_entry
In reply to Dmitry Bogdanovdeve->pr_res_key is never set, and se_cmd->pr_res_key is set in the checks inside __target_execute_cmd
--- drivers/target/target_core_device.c | 2 -- include/target/target_core_base.h | 4 ---- 2 files changed, 6 deletions(-)
drivers/target/target_core_device.c +0 -2
@@ -83,7 +83,6 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd) } se_cmd->se_lun = se_lun; - se_cmd->pr_res_key = deve->pr_res_key; se_cmd->se_cmd_flags |=SCF_SE_LUN_CMD ; se_cmd->lun_ref_active = true; }
@@ -160,7 +159,6 @@ int transport_lookup_tmr_lun(struct se_cmd *se_cmd) } se_cmd->se_lun = se_lun; - se_cmd->pr_res_key = deve->pr_res_key; se_cmd->se_cmd_flags |=SCF_SE_LUN_CMD ; se_cmd->lun_ref_active = true; }
include/target/target_core_base.h +0 -4
@@ -379,8 +379,6 @@ struct t10_pr_registration { };struct t10_reservation { -/* Reservation effects all target ports */ -int pr_all_tg_pt;/* Activate Persistence across Target Power Loss enabled * for SCSI device */ int pr_aptpl_active;
@@ -398,7 +396,6 @@ struct t10_reservation {* a single *pr_res_holder of the reservation, but all * registrations are considered reservation holders. */ -struct se_node_acl *pr_res_holder;struct list_head registration_list;struct list_head aptpl_reg_list; };
@@ -656,7 +653,6 @@ struct se_lun_acl {struct se_dev_entry {u64 mapped_lun; -u64 pr_res_key;u64 creation_time;bool lun_access_ro;u32 attach_count;
Signature · 1 line
2.25.1
RFC PATCH 16/48 target: core: undepend PR registrant of nacl
In reply to Dmitry BogdanovMake registration info be independent of nacl. Store and use an information about initiatror name and target port inside registration object.
--- drivers/target/target_core_configfs.c | 14 +- drivers/target/target_core_device.c | 7 - drivers/target/target_core_fabric_lib.c | 50 ++++---- drivers/target/target_core_internal.h | 9 +- drivers/target/target_core_pr.c | 164 ++++++++---------------- drivers/target/target_core_pr.h | 2 - include/target/target_core_base.h | 1 + 7 files changed, 88 insertions(+), 159 deletions(-)
drivers/target/target_core_configfs.c +5 -9
@@ -1735,7 +1735,6 @@ static struct se_device *pr_to_dev(struct config_item *item)static ssize_t target_core_dev_pr_show_spc3_res (struct se_device *dev,char *page) { -struct se_node_acl *se_nacl;struct t10_pr_registration *pr_reg;char i_buf[PR_REG_ISID_ID_LEN ] = { };
@@ -1743,12 +1742,11 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,if (!pr_reg)return sprintf (page,"No SPC-3 Reservation holder\n" ); - se_nacl = pr_reg->pr_reg_nacl;core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN );return sprintf (page,"SPC-3 Reservation: %s Initiator: %s%s\n" , - se_nacl->se_tpg->se_tpg_tfo->fabric_name, - se_nacl->initiatorname, i_buf); + pr_reg->se_tpg->se_tpg_tfo->fabric_name, + pr_reg->pr_iport, i_buf); } static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
@@ -1823,7 +1821,6 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item, char *page) {struct se_device *dev =pr_to_dev (item); -struct se_node_acl *se_nacl;struct se_portal_group *se_tpg;struct t10_pr_registration *pr_reg;const struct target_core_fabric_ops *tfo;
@@ -1836,8 +1833,7 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item, goto out_unlock; } - se_nacl = pr_reg->pr_reg_nacl; - se_tpg = se_nacl->se_tpg; + se_tpg = pr_reg->se_tpg; tfo = se_tpg->se_tpg_tfo; len += sprintf(page+len, "SPC-3 Reservation: %s"
@@ -1874,12 +1870,12 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,memset (buf,0 ,384 );memset (i_buf,0 ,PR_REG_ISID_ID_LEN ); - tfo = pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo; + tfo = pr_reg->se_tpg->se_tpg_tfo;core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN );sprintf (buf,"%s Node: %s%s Key: 0x%016Lx PRgen: 0x%08x\n" , tfo->fabric_name, - pr_reg->pr_reg_nacl->initiatorname, i_buf, pr_reg->pr_res_key, + pr_reg->pr_iport, i_buf, pr_reg->pr_res_key, pr_reg->pr_res_generation);if (len +strlen (buf) >=PAGE_SIZE )
drivers/target/target_core_device.c +0 -7
@@ -391,12 +391,6 @@ void core_disable_device_list_for_node(struct se_node_acl *nacl, struct se_portal_group *tpg) { -/* -* rcu_dereference_raw protected by se_lun->lun_group symlink -* reference to se_device->dev_group. -*/ -struct se_device *dev =rcu_dereference_raw (lun->lun_se_dev); - lockdep_assert_held(&nacl->lun_entry_mutex);/*
@@ -437,7 +431,6 @@ void core_disable_device_list_for_node(kfree_rcu (orig, rcu_head); -core_scsi3_free_pr_reg_from_nacl (dev, nacl);target_luns_data_has_changed (nacl,NULL , false); }
drivers/target/target_core_fabric_lib.c +22 -28
169 lines not on this page · raw message
drivers/target/target_core_internal.h +4 -5
@@ -97,11 +97,10 @@ void target_setup_backend_cits(struct target_backend *);int target_fabric_setup_cits (struct target_fabric_configfs *);/* target_core_fabric_lib.c */ -int target_get_pr_transport_id_len (struct se_node_acl *nacl, -struct t10_pr_registration *pr_reg,int *format_code); -int target_get_pr_transport_id (struct se_node_acl *nacl, -struct t10_pr_registration *pr_reg,int *format_code, -unsigned char *buf); +int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg, +int *format_code); +int target_get_pr_transport_id (struct t10_pr_registration *pr_reg, +int *format_code,unsigned char *buf);const char *target_parse_pr_out_transport_id (struct se_portal_group *tpg,char *buf,u32 *out_tid_len,char **port_nexus_ptr);
drivers/target/target_core_pr.c +56 -108
458 lines not on this page · raw message
drivers/target/target_core_pr.h +0 -2
@@ -68,8 +68,6 @@ extern int core_scsi3_alloc_aptpl_registration(extern int core_scsi3_check_aptpl_registration (struct se_device *,struct se_portal_group *,struct se_lun *,struct se_node_acl *,u64 ); -extern void core_scsi3_free_pr_reg_from_nacl (struct se_device *, -struct se_node_acl *);extern void core_scsi3_free_all_registrations (struct se_device *);extern unsigned char *core_scsi3_pr_dump_type (int );
include/target/target_core_base.h +1 -0
@@ -369,6 +369,7 @@ struct t10_pr_registration {u64 pr_res_key;atomic_t pr_res_holders;struct se_node_acl *pr_reg_nacl; +struct se_portal_group *se_tpg;/* Used by ALL_TG_PT=1 registration with deve->pr_ref taken */ struct se_dev_entry *pr_reg_deve;struct list_head pr_reg_list;
Signature · 1 line
2.25.1
RFC PATCH 17/48 target: core: make some functions public
In reply to Dmitry BogdanovMake functions of alloc/free and reserve/release reservation be public to use them from cluster code.
--- drivers/target/target_core_pr.c | 51 +++++++++++++++++++-------------- drivers/target/target_core_pr.h | 24 ++++++++++++++++ 2 files changed, 54 insertions(+), 21 deletions(-)
drivers/target/target_core_pr.c +30 -21
@@ -64,9 +64,6 @@ enum preempt_type { PREEMPT_AND_ABORT, }; -static void __core_scsi3_complete_pro_release (struct se_device *, -struct t10_pr_registration *,int ,int ); -static int is_reservation_holder (struct t10_pr_registration *pr_res_holder,struct t10_pr_registration *pr_reg)
@@ -606,7 +603,7 @@ static u32 core_scsi3_pr_generation(struct se_device *dev)return atomic_inc_return (&dev->t10_pr.pr_generation); } -static struct t10_pr_registration *__core_scsi3_do_alloc_registration( +struct t10_pr_registration *__core_scsi3_do_alloc_registration( struct se_device *dev, struct se_node_acl *nacl, const char *initiatorname,
@@ -1261,7 +1258,7 @@ static int core_scsi3_check_implicit_release(return ret; } -static void __core_scsi3_free_registration( +void __core_scsi3_free_registration( struct se_device *dev, struct t10_pr_registration *pr_reg, struct list_head *preempt_and_abort_list,
@@ -2219,6 +2216,32 @@ unsigned char *core_scsi3_pr_dump_type(int type)return "Unknown SPC-3 PR Type" ; } +void __core_scsi3_set_reservation (struct se_device *dev, +struct t10_pr_registration *pr_reg, +int scope,int type) +{ +char i_buf[PR_REG_ISID_ID_LEN ]; + +memset (i_buf,0 ,PR_REG_ISID_ID_LEN ); + +lockdep_assert_held (&dev->dev_reservation_lock); + + pr_reg->pr_res_scope = scope; + pr_reg->pr_res_type = type; + pr_reg->pr_res_holder =1 ; + dev->dev_pr_res_holder = pr_reg; +core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN ); + +pr_debug ("SPC-3 PR [%s] Service Action: RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n" , + pr_reg->se_tpg->se_tpg_tfo->fabric_name, +core_scsi3_pr_dump_type (type), + (pr_reg->pr_reg_all_tg_pt) ?1 :0 ); +pr_debug ("SPC-3 PR [%s] RESERVE Node: %s%s\n" , + pr_reg->se_tpg->se_tpg_tfo->fabric_name, + pr_reg->pr_iport, + i_buf); +} +static sense_reason_t core_scsi3_pro_reserve (struct se_cmd *cmd,int type,int scope,u64 res_key) {
@@ -2227,7 +2250,6 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)struct se_lun *se_lun = cmd->se_lun;struct t10_pr_registration *pr_reg, *pr_res_holder;struct t10_reservation *pr_tmpl = &dev->t10_pr; -char i_buf[PR_REG_ISID_ID_LEN ] = { };sense_reason_t ret;if (!se_sess || !se_lun) {
@@ -2346,20 +2368,7 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)* Otherwise, our *pr_reg becomes the PR reservation holder for said * TYPE/SCOPE. Also set the received scope and type in *pr_reg. */ - pr_reg->pr_res_scope = scope; - pr_reg->pr_res_type = type; - pr_reg->pr_res_holder =1 ; - dev->dev_pr_res_holder = pr_reg; -core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN ); - -pr_debug ("SPC-3 PR [%s] Service Action: RESERVE created new" -" reservation holder TYPE: %s ALL_TG_PT: %d\n" , - cmd->se_tfo->fabric_name,core_scsi3_pr_dump_type (type), - (pr_reg->pr_reg_all_tg_pt) ?1 :0 ); -pr_debug ("SPC-3 PR [%s] RESERVE Node: %s%s\n" , - cmd->se_tfo->fabric_name, - se_sess->se_node_acl->initiatorname, - i_buf); +__core_scsi3_set_reservation (dev, pr_reg, scope, type);spin_unlock (&dev->dev_reservation_lock);if (pr_tmpl->pr_aptpl_active)
@@ -2390,7 +2399,7 @@ core_scsi3_emulate_pro_reserve(struct se_cmd *cmd, int type, int scope, } } -static void __core_scsi3_complete_pro_release( +void __core_scsi3_complete_pro_release( struct se_device *dev, struct t10_pr_registration *pr_reg, int explicit,
drivers/target/target_core_pr.h +24 -0
@@ -75,4 +75,28 @@ extern sense_reason_t target_scsi3_emulate_pr_in(struct se_cmd *);extern sense_reason_t target_scsi3_emulate_pr_out (struct se_cmd *);extern sense_reason_t target_check_reservation (struct se_cmd *); +void __core_scsi3_set_reservation (struct se_device *dev, +struct t10_pr_registration *pr_reg, +int scope,int type); +void __core_scsi3_complete_pro_release (struct se_device *dev, +struct t10_pr_registration *pr_reg, +int explicit,int unreg); +struct t10_pr_registration *__core_scsi3_do_alloc_registration ( +struct se_device *dev, +struct se_node_acl *nacl, +const char *initiatorname, +u64 unpacked_lun, +struct se_portal_group *se_tpg, +struct se_dev_entry *dest_deve, +u64 mapped_lun, +unsigned char *isid, +u64 sa_res_key, +int all_tg_pt, +int aptpl); +void __core_scsi3_free_registration (struct se_device *dev, +struct t10_pr_registration *pr_reg, +struct list_head *preempt_and_abort_list, +int dec_holders); + +#endif /* TARGET_CORE_PR_H */
Signature · 1 line
2.25.1
RFC PATCH 18/48 target: core: proper clear reservation on LUN RESET
In reply to Dmitry BogdanovUse an existing helper to clear SCSI-2 reservation on LUN RESET.
--- drivers/target/target_core_tmr.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/target/target_core_tmr.c +1 -2
@@ -394,8 +394,7 @@ int core_tmr_lun_reset(if (!preempt_and_abort_list && (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS )) {spin_lock (&dev->dev_reservation_lock); - dev->reservation_holder =NULL ; - dev->dev_reservation_flags &= ~DRF_SPC2_RESERVATIONS ; +target_release_reservation (dev);spin_unlock (&dev->dev_reservation_lock);pr_debug ("LUN_RESET: SCSI-2 Released reservation\n" ); }
Signature · 1 line
2.25.1
RFC PATCH 19/48 target: core: remove superfluous checks
In reply to Dmitry Bogdanovremove double checks on path of SCSI-2 reservations
--- drivers/target/target_core_pr.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/target/target_core_pr.c +1 -4
@@ -95,9 +95,6 @@ target_scsi2_reservation_check(struct se_cmd *cmd)break ; } -if (!dev->reservation_holder || !sess) -return 0 ; -if (dev->reservation_holder->se_node_acl != sess->se_node_acl)return TCM_RESERVATION_CONFLICT ;
@@ -211,7 +208,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)return TCM_RESERVATION_CONFLICT ;spin_lock (&dev->dev_reservation_lock); -if (!dev->reservation_holder || !sess) +if (!dev->reservation_holder) goto out_unlock;if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
Signature · 1 line
2.25.1
RFC PATCH 20/48 target: core: proper check of SCSI-2 reservation
In reply to Dmitry BogdanovUse a check of the flag instead of session pointer like in other places.
--- drivers/target/target_core_configfs.c | 17 +++++++++++------ drivers/target/target_core_pr.c | 6 +++--- 2 files changed, 14 insertions(+), 9 deletions(-)
drivers/target/target_core_configfs.c +11 -6
@@ -1756,12 +1756,17 @@ static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,struct se_node_acl *se_nacl;ssize_t len; -if (sess) { - se_nacl = sess->se_node_acl; - len =sprintf (page, -"SPC-2 Reservation: %s Initiator: %s\n" , - se_nacl->se_tpg->se_tpg_tfo->fabric_name, - se_nacl->initiatorname); +if (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) { +if (sess) { + se_nacl = sess->se_node_acl; + len =sprintf (page, +"SPC-2 Reservation: %s Initiator: %s\n" , + se_nacl->se_tpg->se_tpg_tfo->fabric_name, + se_nacl->initiatorname); + }else { + len =sprintf (page, +"SPC-2 Reservation: peer node\n" ); + } }else { len =sprintf (page,"No SPC-2 Reservation holder\n" ); }
drivers/target/target_core_pr.c +3 -3
@@ -208,7 +208,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)return TCM_RESERVATION_CONFLICT ;spin_lock (&dev->dev_reservation_lock); -if (!dev->reservation_holder) +if (!(dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS )) goto out_unlock;if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
@@ -261,8 +261,8 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd) tpg = sess->se_tpg;spin_lock (&dev->dev_reservation_lock); -if (dev->reservation_holder && - dev->reservation_holder->se_node_acl != sess->se_node_acl) { +if ((dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) && + dev->reservation_holder->se_node_acl != sess->se_node_acl) {pr_err ("SCSI-2 RESERVATION CONFLICT for %s fabric\n" , tpg->se_tpg_tfo->fabric_name); pr_err("Original reserver LUN: %llu %s\n",
Signature · 1 line
2.25.1
RFC PATCH 21/48 target: core: checks against peer node SCSI2 reservation
In reply to Dmitry BogdanovFor the case of SCSI-2 reservation by peer node dev->reservation_holder session is NULL. So use this information as reservation existence.
--- drivers/target/target_core_pr.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/target/target_core_pr.c +10 -3
@@ -95,7 +95,8 @@ target_scsi2_reservation_check(struct se_cmd *cmd)break ; } -if (dev->reservation_holder->se_node_acl != sess->se_node_acl) +if (!dev->reservation_holder || + dev->reservation_holder->se_node_acl != sess->se_node_acl)return TCM_RESERVATION_CONFLICT ;if (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS_WITH_ISID ) {
@@ -211,6 +212,9 @@ target_scsi2_reservation_release(struct se_cmd *cmd)if (!(dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS )) goto out_unlock; +if (!dev->reservation_holder) + goto out_unlock; +if (dev->reservation_holder->se_node_acl != sess->se_node_acl) goto out_unlock;
@@ -262,12 +266,15 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd) tpg = sess->se_tpg;spin_lock (&dev->dev_reservation_lock);if ((dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) && - dev->reservation_holder->se_node_acl != sess->se_node_acl) { + (!dev->reservation_holder || + dev->reservation_holder->se_node_acl != sess->se_node_acl)) {pr_err ("SCSI-2 RESERVATION CONFLICT for %s fabric\n" , tpg->se_tpg_tfo->fabric_name);pr_err ("Original reserver LUN: %llu %s\n" , cmd->se_lun->unpacked_lun, - dev->reservation_holder->se_node_acl->initiatorname); + (dev->reservation_holder) ? + dev->reservation_holder->se_node_acl->initiatorname : +"peer" ); pr_err("Current attempt - LUN: %llu -> MAPPED LUN: %llu" " from %s \n", cmd->se_lun->unpacked_lun, cmd->orig_fe_lun,
Signature · 1 line
2.25.1
RFC PATCH 22/48 target: core: UA on all luns after reset
In reply to Dmitry BogdanovAllocate UA "BUS DEVICE RESET OCCURRED" on all LUNs on the device upon reception of TMF LUN RESET.
That patch passes a libisci test SCSI.MultipathIO.Reset.
--- drivers/target/target_core_device.c | 19 +++++++++++++++++++ drivers/target/target_core_internal.h | 1 + drivers/target/target_core_transport.c | 3 +-- 3 files changed, 21 insertions(+), 2 deletions(-)
drivers/target/target_core_device.c +19 -0
@@ -285,6 +285,25 @@ void target_pr_kref_release(struct kref *kref)complete (&deve->pr_comp); } +/* +* Establish UA condition on SCSI device - all LUNs +*/ +void target_dev_ua_allocate (struct se_device *dev,u8 asc,u8 ascq) +{ +struct se_dev_entry *se_deve; +struct se_lun *lun; + +spin_lock (&dev->se_port_lock); +list_for_each_entry (lun, &dev->dev_sep_list, lun_dev_link) { + +spin_lock (&lun->lun_deve_lock); +list_for_each_entry (se_deve, &lun->lun_deve_list, lun_link) +core_scsi3_ua_allocate (se_deve, asc, ascq); +spin_unlock (&lun->lun_deve_lock); + } +spin_unlock (&dev->se_port_lock); +} +static void target_luns_data_has_changed (struct se_node_acl *nacl,struct se_dev_entry *new,bool skip_new)
drivers/target/target_core_internal.h +1 -0
@@ -88,6 +88,7 @@ int target_configure_device(struct se_device *dev);void target_free_device (struct se_device *);int target_for_each_device (int (*fn)(struct se_device *dev,void *data),void *data); +void target_dev_ua_allocate (struct se_device *dev,u8 asc,u8 ascq);/* target_core_configfs.c */ extern struct configfs_item_operations target_core_dev_item_ops;
drivers/target/target_core_transport.c +1 -2
@@ -3531,8 +3531,7 @@ static void target_tmr_work(struct work_struct *work) tmr->response = (!ret) ?TMR_FUNCTION_COMPLETE :TMR_FUNCTION_REJECTED ;if (tmr->response ==TMR_FUNCTION_COMPLETE ) { -target_ua_allocate_lun (cmd->se_sess->se_node_acl, - cmd->orig_fe_lun,0x29 , +target_dev_ua_allocate (dev,0x29 ,ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED ); }break ;
Signature · 1 line
2.25.1
RFC PATCH 23/48 target: core: refactor LUN_RESET code
In reply to Dmitry BogdanovNo function change, just extract a code of aborting all active commands on the device. To make it possible to call TMF LUN_RESET code without cluster related things - SCSI2 reservation or ACA condition.
--- drivers/target/target_core_internal.h | 5 +++-- drivers/target/target_core_pr.c | 4 ++-- drivers/target/target_core_tmr.c | 27 +++++++++++++++++++------- drivers/target/target_core_transport.c | 6 +----- 4 files changed, 26 insertions(+), 16 deletions(-)
drivers/target/target_core_internal.h +3 -2
@@ -112,8 +112,9 @@ int core_delete_hba(struct se_hba *);/* target_core_tmr.c */ void core_tmr_abort_task (struct se_device *,struct se_tmr_req *,struct se_session *); -int core_tmr_lun_reset (struct se_device *,struct se_tmr_req *, -struct list_head *,struct se_cmd *); +int core_tmr_lun_reset (struct se_device *dev,struct se_tmr_req *tmr); +void core_local_lun_reset (struct se_device *swc,struct se_tmr_req *tmr, +struct list_head *preempt_and_abort_list,struct se_cmd *prout_cmd);/* target_core_tpg.c */ extern struct spinlock g_tpg_lock;
drivers/target/target_core_pr.c +2 -2
@@ -2936,7 +2936,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,* registered I_T nexuses. */ if (preempt_type ==PREEMPT_AND_ABORT ) { -core_tmr_lun_reset (dev,NULL , &preempt_and_abort_list, +core_local_lun_reset (dev,NULL , &preempt_and_abort_list, cmd);core_scsi3_release_preempt_and_abort ( &preempt_and_abort_list, pr_reg_n);
@@ -3067,7 +3067,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,* new persistent reservation holder, the calling Initiator Port. */ if (preempt_type ==PREEMPT_AND_ABORT ) { -core_tmr_lun_reset (dev,NULL , &preempt_and_abort_list, cmd); +core_local_lun_reset (dev,NULL , &preempt_and_abort_list, cmd);core_scsi3_release_preempt_and_abort (&preempt_and_abort_list, pr_reg_n); }
drivers/target/target_core_tmr.c +20 -7
@@ -22,6 +22,7 @@#include "target_core_internal.h" #include "target_core_alua.h" #include "target_core_pr.h" +#include "target_core_ua.h" int core_tmr_alloc_req( struct se_cmd *se_cmd,
@@ -342,7 +343,8 @@ static void core_tmr_drain_state_list( } } -int core_tmr_lun_reset( + +void core_local_lun_reset( struct se_device *dev, struct se_tmr_req *tmr, struct list_head *preempt_and_abort_list,
@@ -352,7 +354,8 @@ int core_tmr_lun_reset(struct se_portal_group *tmr_tpg =NULL ;struct se_session *tmr_sess =NULL ;bool tas; -/* + +/* * TASK_ABORTED status bit, this is configurable via ConfigFS * struct se_device attributes. spc4r17 section 7.4.6 Control mode page *
@@ -387,23 +390,33 @@ int core_tmr_lun_reset(core_tmr_drain_state_list (dev, prout_cmd, tmr_sess, tas, preempt_and_abort_list); +pr_debug ("LUN_RESET: %s for [%s] Complete\n" , + (preempt_and_abort_list) ?"Preempt" :"TMR" , + dev->transport->name); +} + +int core_tmr_lun_reset ( +struct se_device *dev, +struct se_tmr_req *tmr) +{ +core_local_lun_reset (dev, tmr,NULL ,NULL ); +/* * Clear any legacy SPC-2 reservation when called during * LOGICAL UNIT RESET */ -if (!preempt_and_abort_list && - (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS )) { +if (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) {spin_lock (&dev->dev_reservation_lock);target_release_reservation (dev);spin_unlock (&dev->dev_reservation_lock);pr_debug ("LUN_RESET: SCSI-2 Released reservation\n" ); } +target_dev_ua_allocate (dev,0x29 , +ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED ); +atomic_long_inc (&dev->num_resets); -pr_debug ("LUN_RESET: %s for [%s] Complete\n" , - (preempt_and_abort_list) ?"Preempt" :"TMR" , - dev->transport->name);return 0 ; }
drivers/target/target_core_transport.c +1 -5
@@ -3527,13 +3527,9 @@ static void target_tmr_work(struct work_struct *work) tmr->response =TMR_TASK_MGMT_FUNCTION_NOT_SUPPORTED ;break ;case TMR_LUN_RESET : - ret =core_tmr_lun_reset (dev, tmr,NULL ,NULL ); + ret =core_tmr_lun_reset (dev, tmr); tmr->response = (!ret) ?TMR_FUNCTION_COMPLETE :TMR_FUNCTION_REJECTED ; -if (tmr->response ==TMR_FUNCTION_COMPLETE ) { -target_dev_ua_allocate (dev,0x29 , -ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED ); - }break ;case TMR_TARGET_WARM_RESET : tmr->response =TMR_FUNCTION_REJECTED ;
Signature · 1 line
2.25.1
RFC PATCH 24/48 target: core: pr: use RTPI in APTPL
In reply to Dmitry BogdanovSince RTPI is an unique identifier of target port there is no need to check fabric name and port tag.
RTPI is already saved in APTPL data, so it is backward compatible change.
--- drivers/target/target_core_configfs.c | 4 ++- drivers/target/target_core_pr.c | 41 +++++++-------------------- drivers/target/target_core_pr.h | 2 +- include/target/target_core_base.h | 5 ---- 4 files changed, 15 insertions(+), 37 deletions(-)
drivers/target/target_core_configfs.c +3 -1
@@ -1993,6 +1993,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,u64 mapped_lun =0 , target_lun =0 ;int ret =-1 , res_holder =0 , all_tg_pt =0 , arg, token;u16 tpgt =0 ; +u16 rtpi =0 ;u8 type =0 ; if (!dev->dev_attrib.emulate_pr ||
@@ -2127,6 +2128,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item, ret =match_int (args, &arg);if (ret) goto out; + rtpi = (u16 )arg;break ;case Opt_target_lun: ret =match_u64 (args, &tmp_ll);
@@ -2153,7 +2155,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item, } ret =core_scsi3_alloc_aptpl_registration (&dev->t10_pr, sa_res_key, - i_port, isid, mapped_lun, t_port, tpgt, target_lun, + i_port, isid, mapped_lun, rtpi, target_lun, res_holder, all_tg_pt, type); out:kfree (i_fabric);
drivers/target/target_core_pr.c +11 -30
105 lines not on this page · raw message
drivers/target/target_core_pr.h +1 -1
@@ -64,7 +64,7 @@ extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);extern int core_scsi3_alloc_aptpl_registration (struct t10_reservation *,u64 ,unsigned char *,unsigned char *,u64 , -unsigned char *,u16 ,u64 ,int ,int ,u8 ); +u16 ,u64 ,int ,int ,u8 );extern int core_scsi3_check_aptpl_registration (struct se_device *,struct se_portal_group *,struct se_lun *,struct se_node_acl *,u64 );
include/target/target_core_base.h +0 -5
@@ -347,11 +347,6 @@ struct t10_pr_registration {/* Used during APTPL metadata reading */ #define PR_APTPL_MAX_IPORT_LEN 256unsigned char pr_iport[PR_APTPL_MAX_IPORT_LEN ]; -/* Used during APTPL metadata reading */ -#define PR_APTPL_MAX_TPORT_LEN 256 -unsigned char pr_tport[PR_APTPL_MAX_TPORT_LEN ]; -u16 pr_aptpl_rpti; -u16 pr_reg_tpgt;/* Reservation effects all target ports */ int pr_reg_all_tg_pt;/* Activate Persistence across Target Power Loss */
Signature · 1 line
2.25.1
RFC PATCH 25/48 target: core: pr: have Transport ID stored
In reply to Dmitry BogdanovStore TransportID in pr_reg at its creation.
--- drivers/target/target_core_fabric_lib.c | 130 +++++++++--------------- drivers/target/target_core_internal.h | 9 +- drivers/target/target_core_pr.c | 19 ++-- include/target/target_core_base.h | 2 + 4 files changed, 68 insertions(+), 92 deletions(-)
drivers/target/target_core_fabric_lib.c +50 -80
@@ -33,14 +33,14 @@static int sas_get_pr_transport_id ( -struct t10_pr_registration *pr_reg, -int *format_code, +const char *initiatorname,unsigned char *buf) {int ret; + buf[0 ] =SCSI_PROTOCOL_SAS ;/* Skip over 'naa. prefix */ - ret =hex2bin (&buf[4 ], &pr_reg->pr_iport[4 ],8 ); + ret =hex2bin (&buf[4 ], &initiatorname[4 ],8 );if (ret) {pr_debug ("%s: invalid hex string\n" , __func__);return ret;
@@ -50,19 +50,19 @@ static int sas_get_pr_transport_id( }static int fc_get_pr_transport_id ( -struct t10_pr_registration *pr_reg, -int *format_code, +const char *initiatorname,unsigned char *buf) { -unsigned char *ptr; +const char *ptr;int i, ret;u32 off =8 ; + buf[0 ] =SCSI_PROTOCOL_FCP ;/* * We convert the ASCII formatted N Port name into a binary * encoded TransportID. */ - ptr = &pr_reg->pr_iport[0 ]; + ptr = &initiatorname[0 ];for (i =0 ; i <23 ; ) {if (!strncmp (&ptr[i],":" ,1 )) { i++;
@@ -82,13 +82,14 @@ static int fc_get_pr_transport_id( }static int sbp_get_pr_transport_id ( -struct t10_pr_registration *pr_reg, -int *format_code, +const char *initiatorname,unsigned char *buf) {int ret; - ret =hex2bin (&buf[8 ], pr_reg->pr_iport,8 ); + buf[0 ] =SCSI_PROTOCOL_SBP ; + + ret =hex2bin (&buf[8 ], initiatorname,8 );if (ret) {pr_debug ("%s: invalid hex string\n" , __func__);return ret;
@@ -98,15 +99,16 @@ static int sbp_get_pr_transport_id( }static int srp_get_pr_transport_id ( -struct t10_pr_registration *pr_reg, -int *format_code, +const char *initiatorname,unsigned char *buf) {const char *p;unsigned len, count, leading_zero_bytes;int rc; - p = pr_reg->pr_iport; + buf[0 ] =SCSI_PROTOCOL_SRP ; + + p = initiatorname;if (strncasecmp (p,"0x" ,2 ) ==0 ) p +=2 ; len =strlen (p);
@@ -126,14 +128,16 @@ static int srp_get_pr_transport_id( }static int iscsi_get_pr_transport_id ( -struct t10_pr_registration *pr_reg, -int *format_code, +const char *initiatorname, +unsigned char *isid,unsigned char *buf) {u32 off =4 , padding =0 ;int isid_len;u16 len =0 ; + buf[0 ] =SCSI_PROTOCOL_ISCSI ; +/* * Only null terminate the last field. *
@@ -149,9 +153,9 @@ static int iscsi_get_pr_transport_id(* length of the iSCSI TransportID or the contents of the ADDITIONAL * LENGTH field. */ - len =sprintf (&buf[off],"%s" , pr_reg->pr_iport); + len =sprintf (&buf[off],"%s" , initiatorname); off += len; -if ((*format_code ==1 ) && (pr_reg->isid_present_at_reg)) { +if (isid) {/* * Set FORMAT CODE 01b for iSCSI Initiator port TransportID * format.
@@ -185,7 +189,7 @@ static int iscsi_get_pr_transport_id( buf[off++] =0x78 ;/* ASCII Character: "x" */ len +=5 ; - isid_len =sprintf (buf + off,"%s" , pr_reg->pr_reg_isid); + isid_len =sprintf (buf + off,"%s" , isid); off += isid_len; len += isid_len; }
@@ -210,47 +214,6 @@ static int iscsi_get_pr_transport_id(return len; } -static int iscsi_get_pr_transport_id_len ( -struct t10_pr_registration *pr_reg, -int *format_code) -{ -u32 len =0 , padding =0 ; - - len =strlen (pr_reg->pr_iport); -/* -* Add extra byte for NULL terminator -*/ - len++; -/* -* If there is ISID present with the registration, use format code: -* 01b: iSCSI Initiator port TransportID format -* -* If there is not an active iSCSI session, use format code: -* 00b: iSCSI Initiator device TransportID format -*/ -if (pr_reg->isid_present_at_reg) { - len +=5 ;/* For ",i,0x" ASCII separator */ - len +=strlen (pr_reg->pr_reg_isid); - *format_code =1 ; - }else - *format_code =0 ; -/* -* The ADDITIONAL LENGTH field specifies the number of bytes that follow -* in the TransportID. The additional length shall be at least 20 and -* shall be a multiple of four. -*/ - padding = ((-len) &3 ); -if (padding !=0 ) - len += padding; -/* -* Increment value for total payload + header length for -* full status descriptor -*/ - len +=4 ; - -return len; -} - static char *iscsi_parse_pr_out_transport_id( struct se_portal_group *se_tpg, char *buf,
@@ -336,47 +299,54 @@ static char *iscsi_parse_pr_out_transport_id(return &buf[4 ]; } -int target_get_pr_transport_id_len ( -struct t10_pr_registration *pr_reg,int *format_code) +int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg) { -switch (pr_reg->se_tpg->proto_id) { +switch (pr_reg->pr_tid[0 ] &0xF ) {case SCSI_PROTOCOL_FCP :case SCSI_PROTOCOL_SBP :case SCSI_PROTOCOL_SRP :case SCSI_PROTOCOL_SAS : -break ; +return 24 ;case SCSI_PROTOCOL_ISCSI : -return iscsi_get_pr_transport_id_len (pr_reg, format_code); +return get_unaligned_be16 (&pr_reg->pr_tid[2 ]) +4 ;default : -pr_err ("Unknown proto_id: 0x%02x\n" , pr_reg->se_tpg->proto_id); +WARN (1 ,"Unknown proto_id: %#x\n" , pr_reg->pr_tid[0 ] &0xF );return -EINVAL ; } - -/* -* Most transports use a fixed length 24 byte identifier. -*/ - *format_code =0 ; -return 24 ; }int target_get_pr_transport_id ( -struct t10_pr_registration *pr_reg,int *format_code, +struct t10_pr_registration *pr_reg,unsigned char *buf) { -switch (pr_reg->se_tpg->proto_id) { +int len =target_get_pr_transport_id_len (pr_reg); + +if (len >0 ) +memcpy (buf, pr_reg->pr_tid, len); + +return len; +} + +int target_gen_pr_transport_id ( +struct t10_pr_registration *pr_reg, +int proto_id, +const char *initiatorname, +unsigned char *isid) +{ + +switch (proto_id) {case SCSI_PROTOCOL_SAS : -return sas_get_pr_transport_id (pr_reg, format_code, buf); +return sas_get_pr_transport_id (initiatorname, pr_reg->pr_tid);case SCSI_PROTOCOL_SBP : -return sbp_get_pr_transport_id (pr_reg, format_code, buf); +return sbp_get_pr_transport_id (initiatorname, pr_reg->pr_tid);case SCSI_PROTOCOL_SRP : -return srp_get_pr_transport_id (pr_reg, format_code, buf); +return srp_get_pr_transport_id (initiatorname, pr_reg->pr_tid);case SCSI_PROTOCOL_FCP : -return fc_get_pr_transport_id (pr_reg, format_code, buf); +return fc_get_pr_transport_id (initiatorname, pr_reg->pr_tid);case SCSI_PROTOCOL_ISCSI : -return iscsi_get_pr_transport_id (pr_reg, format_code, - buf); +return iscsi_get_pr_transport_id (initiatorname, isid, pr_reg->pr_tid);default : -pr_err ("Unknown proto_id: 0x%02x\n" , pr_reg->se_tpg->proto_id); +pr_err ("Unknown proto_id: 0x%02x\n" , proto_id);return -EINVAL ; } }
drivers/target/target_core_internal.h +6 -3
@@ -98,10 +98,13 @@ void target_setup_backend_cits(struct target_backend *);int target_fabric_setup_cits (struct target_fabric_configfs *);/* target_core_fabric_lib.c */ -int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg, -int *format_code); +int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg);int target_get_pr_transport_id (struct t10_pr_registration *pr_reg, -int *format_code,unsigned char *buf); +unsigned char *buf); +int target_gen_pr_transport_id (struct t10_pr_registration *pr_reg, +int proto_id, +const char *initiatorname, +unsigned char *isid);const char *target_parse_pr_out_transport_id (struct se_portal_group *tpg,char *buf,u32 *out_tid_len,char **port_nexus_ptr);
drivers/target/target_core_pr.c +10 -9
@@ -716,6 +716,10 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration( aptpl);if (!pr_reg)return NULL ; + +target_gen_pr_transport_id (pr_reg, nacl->se_tpg->proto_id, + nacl->initiatorname, isid); +/* * Return pointer to pr_reg for ALL_TG_PT=0 */
@@ -802,6 +806,9 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(core_scsi3_lunacl_undepend_item (deve_tmp); goto out; } +target_gen_pr_transport_id (pr_reg_atp, + nacl_tmp->se_tpg->proto_id, + nacl_tmp->initiatorname, isid);list_add_tail (&pr_reg_atp->pr_reg_atp_mem_list, &pr_reg->pr_reg_atp_list);
@@ -3829,13 +3836,12 @@ static sense_reason_t core_scsi3_pri_read_full_status(struct se_cmd *cmd) {struct se_device *dev = cmd->se_dev; -struct se_portal_group *se_tpg;struct t10_pr_registration *pr_reg, *pr_reg_tmp;struct t10_reservation *pr_tmpl = &dev->t10_pr;unsigned char *buf;u32 add_desc_len =0 , add_len =0 ;u32 off =8 ;/* off into first Full Status descriptor */ -int format_code =0 , pr_res_type =0 , pr_res_scope =0 ; +int pr_res_type =0 , pr_res_scope =0 ;int exp_desc_len, desc_len;bool all_reg = false;
@@ -3868,7 +3874,6 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)list_for_each_entry_safe (pr_reg, pr_reg_tmp, &pr_tmpl->registration_list, pr_reg_list) { - se_tpg = pr_reg->se_tpg; add_desc_len =0 ;atomic_inc_mb (&pr_reg->pr_res_holders);
@@ -3877,8 +3882,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)* Determine expected length of $FABRIC_MOD specific * TransportID full status descriptor.. */ - exp_desc_len =target_get_pr_transport_id_len (pr_reg, - &format_code); + exp_desc_len =target_get_pr_transport_id_len (pr_reg);if (exp_desc_len <0 || exp_desc_len + add_len > cmd->data_length) { pr_warn("SPC-3 PRIN READ_FULL_STATUS ran"
@@ -3938,13 +3942,10 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd) }else off +=2 ;/* Skip over RELATIVE TARGET PORT IDENTIFIER */ - buf[off+4 ] = se_tpg->proto_id; -/* * Now, have the $FABRIC_MOD fill in the transport ID. */ - desc_len =target_get_pr_transport_id (pr_reg, - &format_code, &buf[off+4 ]); + desc_len =target_get_pr_transport_id (pr_reg, &buf[off+4 ]);spin_lock (&pr_tmpl->registration_lock);atomic_dec_mb (&pr_reg->pr_res_holders);
include/target/target_core_base.h +2 -0
@@ -347,6 +347,8 @@ struct t10_pr_registration {/* Used during APTPL metadata reading */ #define PR_APTPL_MAX_IPORT_LEN 256unsigned char pr_iport[PR_APTPL_MAX_IPORT_LEN ]; +#define PR_REG_TID_LEN 228 +char pr_tid[PR_REG_TID_LEN ];/* Reservation effects all target ports */ int pr_reg_all_tg_pt;/* Activate Persistence across Target Power Loss */
Signature · 1 line
2.25.1
RFC PATCH 26/48 target: core: pr: remove se_tpg from pr_reg
In reply to Dmitry BogdanovUndepend pr_reg from se_tpg.
Use RTPI at PR creation.
--- drivers/target/target_core_configfs.c | 47 ++++--- drivers/target/target_core_fabric_configfs.c | 2 +- drivers/target/target_core_internal.h | 1 + drivers/target/target_core_pr.c | 126 ++++++++----------- drivers/target/target_core_pr.h | 2 +- drivers/target/target_core_tpg.c | 17 ++- include/target/target_core_base.h | 1 - 7 files changed, 104 insertions(+), 92 deletions(-)
drivers/target/target_core_configfs.c +31 -16
@@ -1736,6 +1736,8 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev, char *page) {struct t10_pr_registration *pr_reg; +struct se_portal_group *se_tpg; +const char *fabric_name ="Unknown" ;char i_buf[PR_REG_ISID_ID_LEN ] = { }; pr_reg = dev->dev_pr_res_holder;
@@ -1744,9 +1746,12 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN ); + se_tpg =core_get_tpg_by_rtpi (pr_reg->tg_pt_sep_rtpi); +if (se_tpg) + fabric_name = se_tpg->se_tpg_tfo->fabric_name; +return sprintf (page,"SPC-3 Reservation: %s Initiator: %s%s\n" , - pr_reg->se_tpg->se_tpg_tfo->fabric_name, - pr_reg->pr_iport, i_buf); + fabric_name, pr_reg->pr_iport, i_buf); } static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
@@ -1838,18 +1843,24 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item, goto out_unlock; } - se_tpg = pr_reg->se_tpg; - tfo = se_tpg->se_tpg_tfo; - - len +=sprintf (page+len,"SPC-3 Reservation: %s" -" Target Node Endpoint: %s\n" , tfo->fabric_name, - tfo->tpg_get_wwn (se_tpg)); - len +=sprintf (page+len,"SPC-3 Reservation: Relative Port" -" Identifier Tag: %hu %s Portal Group Tag: %hu" -" %s Logical Unit: %llu\n" , pr_reg->tg_pt_sep_rtpi, - tfo->fabric_name, tfo->tpg_get_tag (se_tpg), - tfo->fabric_name, pr_reg->pr_aptpl_target_lun); + se_tpg =core_get_tpg_by_rtpi (pr_reg->tg_pt_sep_rtpi); +if (se_tpg) { + tfo = se_tpg->se_tpg_tfo; + len +=sprintf (page+len,"SPC-3 Reservation: %s Target Node Endpoint: %s\n" , + tfo->fabric_name, tfo->tpg_get_wwn (se_tpg)); + len +=sprintf (page+len,"SPC-3 Reservation: Relative Port Identifier Tag: %hu %s Portal Group Tag: %hu %s Logical Unit: %llu\n" , + pr_reg->tg_pt_sep_rtpi, + tfo->fabric_name, tfo->tpg_get_tag (se_tpg), + tfo->fabric_name, pr_reg->pr_aptpl_target_lun); + }else { + len +=sprintf (page+len,"SPC-3 Reservation: %s Target Node Endpoint: %s\n" , +"Unknown" ,"Unknown" ); + len +=sprintf (page+len,"SPC-3 Reservation: Relative Port Identifier Tag: %hu %s Portal Group Tag: %hu %s Logical Unit: %llu\n" , + pr_reg->tg_pt_sep_rtpi, +"Unknown" ,0 , +"Unknown" , pr_reg->pr_aptpl_target_lun); + } out_unlock:spin_unlock (&dev->dev_reservation_lock);return len;
@@ -1860,8 +1871,9 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item, char *page) {struct se_device *dev =pr_to_dev (item); -const struct target_core_fabric_ops *tfo;struct t10_pr_registration *pr_reg; +const char *fabric_name ="Unknown" ; +struct se_portal_group *se_tpg;unsigned char buf[384 ];char i_buf[PR_REG_ISID_ID_LEN ];ssize_t len =0 ;
@@ -1875,11 +1887,14 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,memset (buf,0 ,384 );memset (i_buf,0 ,PR_REG_ISID_ID_LEN ); - tfo = pr_reg->se_tpg->se_tpg_tfo; + se_tpg =core_get_tpg_by_rtpi (pr_reg->tg_pt_sep_rtpi); +if (se_tpg) + fabric_name = se_tpg->se_tpg_tfo->fabric_name; +core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN );sprintf (buf,"%s Node: %s%s Key: 0x%016Lx PRgen: 0x%08x\n" , - tfo->fabric_name, + fabric_name, pr_reg->pr_iport, i_buf, pr_reg->pr_res_key, pr_reg->pr_res_generation);
drivers/target/target_core_fabric_configfs.c +1 -1
@@ -871,7 +871,7 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,spin_lock (&g_tpg_lock);if (op) { - tpg =core_get_tpg_by_rtpi (se_tpg->tpg_rtpi); + tpg =_core_get_tpg_by_rtpi (se_tpg->tpg_rtpi);if (tpg) {spin_unlock (&g_tpg_lock);
drivers/target/target_core_internal.h +1 -0
@@ -136,6 +136,7 @@ void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);struct se_node_acl *core_tpg_add_initiator_node_acl (struct se_portal_group *tpg,const char *initiatorname);void core_tpg_del_initiator_node_acl (struct se_node_acl *acl); +struct se_portal_group *_core_get_tpg_by_rtpi (u16 rtpi);struct se_portal_group *core_get_tpg_by_rtpi (u16 rtpi);/* target_core_transport.c */
drivers/target/target_core_pr.c +55 -71
279 lines not on this page · raw message
drivers/target/target_core_pr.h +1 -1
@@ -86,7 +86,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(struct se_node_acl *nacl, const char *initiatorname, u64 unpacked_lun, - struct se_portal_group *se_tpg, + u16 rtpi, struct se_dev_entry *dest_deve, u64 mapped_lun, unsigned char *isid,
drivers/target/target_core_tpg.c +15 -2
@@ -738,8 +738,9 @@ static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)return sprintf (page,"%#x\n" , se_tpg->tpg_rtpi); } +struct se_portal_group * -core_get_tpg_by_rtpi (u16 rtpi) +_core_get_tpg_by_rtpi (u16 rtpi) {struct se_portal_group *tpg;
@@ -753,6 +754,18 @@ core_get_tpg_by_rtpi(u16 rtpi)return NULL ; } +struct se_portal_group * +core_get_tpg_by_rtpi (u16 rtpi) +{ +struct se_portal_group *tpg; + +spin_lock (&g_tpg_lock); + tpg =_core_get_tpg_by_rtpi (rtpi); +spin_unlock (&g_tpg_lock); + +return tpg; +} +static ssize_t core_tpg_rtpi_store (struct config_item *item,const char *page,size_t count) {
@@ -773,7 +786,7 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,spin_lock (&g_tpg_lock);if (se_tpg->enabled) { - tpg =core_get_tpg_by_rtpi (val); + tpg =_core_get_tpg_by_rtpi (val);if (tpg) {spin_unlock (&g_tpg_lock); pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
include/target/target_core_base.h +0 -1
@@ -366,7 +366,6 @@ struct t10_pr_registration {u64 pr_res_key;atomic_t pr_res_holders;struct se_node_acl *pr_reg_nacl; -struct se_portal_group *se_tpg;/* Used by ALL_TG_PT=1 registration with deve->pr_ref taken */ struct se_dev_entry *pr_reg_deve;struct list_head pr_reg_list;
Signature · 1 line
2.25.1
RFC PATCH 27/48 target: core: fix parsing PR OUT TID
In reply to Dmitry BogdanovNow only iSCSI TransportID is parsed. From other protocols the binary representation is passed to tcm core. That is wrong as that TID is compared against nacl->initiatorname which is in human readable format. This patch adds a parsing of TID for other protocols.
--- drivers/target/target_core_fabric_lib.c | 119 ++++++++++++++++-------- drivers/target/target_core_internal.h | 4 +- drivers/target/target_core_pr.c | 14 ++- 3 files changed, 89 insertions(+), 48 deletions(-)
drivers/target/target_core_fabric_lib.c +81 -38
@@ -214,15 +214,67 @@ static int iscsi_get_pr_transport_id(return len; } -static char *iscsi_parse_pr_out_transport_id ( -struct se_portal_group *se_tpg, -char *buf, -u32 *out_tid_len, +static u32 sas_parse_pr_out_transport_id ( +const char *buf, +char *initiatorname, +char **port_nexus_ptr) +{ +char port_name[17 ]; + +bin2hex (port_name, buf +4 ,8 ); +snprintf (initiatorname,36 ,"naa.%s" , port_name); + +return 24 ; +} + + +static u32 sbp_parse_pr_out_transport_id ( +const char *buf, +char *initiatorname, +char **port_nexus_ptr) +{ +bin2hex (initiatorname, buf +8 ,8 ); + +return 24 ; +} + +static u32 srp_parse_pr_out_transport_id ( +const char *buf, +char *initiatorname, +char **port_nexus_ptr) +{ +char port_name[33 ]; + +bin2hex (port_name, buf +8 ,16 ); +snprintf (initiatorname,36 ,"0x%s" , port_name); + +return 24 ; +} + +static u32 fc_parse_pr_out_transport_id ( +const char *buf, +char *initiatorname, +char **port_nexus_ptr) +{ +/* +* Format the FCP Initiator port_name into colon separated values to +* match the format by tcm_qla2xxx explicit ConfigFS NodeACLs. +*/ +snprintf (initiatorname,36 ,"%8phC" , buf +8 ); + +return 24 ; +} + +u32 iscsi_parse_pr_out_transport_id ( +const char *buf, +char *initiatorname,char **port_nexus_ptr) {char *p;int i;u8 format_code = (buf[0 ] &0xc0 ); +u32 out_tid_len; +/* * Check for FORMAT CODE 00b or 01b from spc4r17, section 7.5.4.6: *
@@ -239,18 +291,10 @@ static char *iscsi_parse_pr_out_transport_id(if ((format_code !=0x00 ) && (format_code !=0x40 )) {pr_err ("Illegal format code: 0x%02x for iSCSI" " Initiator Transport ID\n" , format_code); -return NULL ; - } -/* -* If the caller wants the TransportID Length, we set that value for the -* entire iSCSI Tarnsport ID now. -*/ -if (out_tid_len) { -/* The shift works thanks to integer promotion rules */ - *out_tid_len =get_unaligned_be16 (&buf[2 ]); -/* Add four bytes for iSCSI Transport ID header */ - *out_tid_len +=4 ; +return 0 ; } +/* The shift works thanks to integer promotion rules */ + out_tid_len =get_unaligned_be16 (&buf[2 ]);/* * Check for ',i,0x' separator between iSCSI Name and iSCSI Initiator
@@ -263,9 +307,10 @@ static char *iscsi_parse_pr_out_transport_id(pr_err ("Unable to locate \",i,0x\" separator" " for Initiator port identifier: %s\n" , &buf[4 ]); -return NULL ; +return 0 ; } - *p ='\0' ;/* Terminate iSCSI Name */ +memcpy (initiatorname, &buf[4 ], p - buf -4 ); + initiatorname[p - buf -4 ] ='\0' ;/* Terminate iSCSI Name */ p +=5 ;/* Skip over ",i,0x" separator */ *port_nexus_ptr = p;
@@ -293,10 +338,16 @@ static char *iscsi_parse_pr_out_transport_id( *p =tolower (*p); p++; } - }else + }else { +strscpy (initiatorname, &buf[4 ], out_tid_len); + initiatorname[p - buf] ='\0' ;/* Terminate iSCSI Name */ *port_nexus_ptr =NULL ; + } + +/* Add four bytes for iSCSI Transport ID header */ + out_tid_len +=4 ; -return &buf[4 ]; +return out_tid_len; }int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg)
@@ -351,33 +402,25 @@ int target_gen_pr_transport_id( } } -const char *target_parse_pr_out_transport_id (struct se_portal_group *tpg, -char *buf,u32 *out_tid_len,char **port_nexus_ptr) +u32 target_parse_pr_out_transport_id ( +const char *buf,char *initiatorname,char **port_nexus_ptr) { -u32 offset; - -switch (tpg->proto_id) { +switch (buf[0 ] &0xF ) {case SCSI_PROTOCOL_SAS : -/* -* Assume the FORMAT CODE 00b from spc4r17, 7.5.4.7 TransportID -* for initiator ports using SCSI over SAS Serial SCSI Protocol. -*/ - offset =4 ; -break ; +return sas_parse_pr_out_transport_id (buf, initiatorname, port_nexus_ptr);case SCSI_PROTOCOL_SBP : +return sbp_parse_pr_out_transport_id (buf, initiatorname, port_nexus_ptr);case SCSI_PROTOCOL_SRP : +return srp_parse_pr_out_transport_id (buf, initiatorname, port_nexus_ptr);case SCSI_PROTOCOL_FCP : - offset =8 ; -break ; +return fc_parse_pr_out_transport_id (buf, initiatorname, port_nexus_ptr);case SCSI_PROTOCOL_ISCSI : -return iscsi_parse_pr_out_transport_id (tpg, buf, out_tid_len, - port_nexus_ptr); +return iscsi_parse_pr_out_transport_id (buf, initiatorname, port_nexus_ptr);default : -pr_err ("Unknown proto_id: 0x%02x\n" , tpg->proto_id); -return NULL ; +pr_err ("Unknown proto_id: 0x%02x\n" , buf[0 ] &0xF ); +return 0 ; } *port_nexus_ptr =NULL ; - *out_tid_len =24 ; -return buf + offset; +return 24 ; }
drivers/target/target_core_internal.h +2 -2
@@ -105,8 +105,8 @@ int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,int proto_id, const char *initiatorname, unsigned char *isid); -const char *target_parse_pr_out_transport_id (struct se_portal_group *tpg, -char *buf,u32 *out_tid_len,char **port_nexus_ptr); +u32 target_parse_pr_out_transport_id ( +const char *buf,char *initiatorname,char **port_nexus_ptr);/* target_core_hba.c */ struct se_hba *core_alloc_hba (const char *,u32 ,u32 );
drivers/target/target_core_pr.c +6 -8
@@ -1419,7 +1419,7 @@ core_scsi3_decode_spec_i_port(LIST_HEAD (tid_dest_list);struct pr_transport_id_holder *tidh_new, *tidh, *tidh_tmp;unsigned char *buf, *ptr, proto_ident; -const unsigned char *i_str =NULL ; +unsigned char i_str[TRANSPORT_IQN_LEN ] = {0 };char *iport_ptr =NULL , i_buf[PR_REG_ISID_ID_LEN ];sense_reason_t ret;u32 tpdl, tid_len =0 ;
@@ -1512,9 +1512,8 @@ core_scsi3_decode_spec_i_port( dest_rtpi = tmp_lun->lun_tpg->tpg_rtpi; iport_ptr =NULL ; - i_str =target_parse_pr_out_transport_id (tmp_tpg, - ptr, &tid_len, &iport_ptr); -if (!i_str) + tid_len =target_parse_pr_out_transport_id (ptr, i_str, &iport_ptr); +if (!tid_len)continue ;/* * Determine if this SCSI device server requires that
@@ -3087,7 +3086,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,struct t10_pr_registration *pr_reg, *pr_res_holder, *dest_pr_reg;struct t10_reservation *pr_tmpl = &dev->t10_pr;unsigned char *buf; -const unsigned char *initiator_str; +unsigned char initiator_str[TRANSPORT_IQN_LEN ] = { };char *iport_ptr =NULL , i_buf[PR_REG_ISID_ID_LEN ] = { };u32 tid_len, tmp_tid_len;int new_reg =0 , type, scope, matching_iname;
@@ -3212,9 +3211,8 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key, ret =TCM_INVALID_PARAMETER_LIST ; goto out; } - initiator_str =target_parse_pr_out_transport_id (dest_se_tpg, - &buf[24 ], &tmp_tid_len, &iport_ptr); -if (!initiator_str) { + tmp_tid_len =target_parse_pr_out_transport_id (&buf[24 ], initiator_str, &iport_ptr); +if (!tmp_tid_len) {pr_err ("SPC-3 PR REGISTER_AND_MOVE: Unable to locate" " initiator_str from Transport ID\n" ); ret =TCM_INVALID_PARAMETER_LIST ;
Signature · 1 line
2.25.1
RFC PATCH 28/48 target: core: add function to compare TransportID
In reply to Dmitry Bogdanov--- drivers/target/target_core_fabric_lib.c | 27 +++++++++++++++++++++---- drivers/target/target_core_internal.h | 2 ++ 2 files changed, 25 insertions(+), 4 deletions(-)
drivers/target/target_core_fabric_lib.c +23 -4
@@ -350,22 +350,27 @@ u32 iscsi_parse_pr_out_transport_id(return out_tid_len; } -int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg) +static int __target_get_pr_transport_id_len (const char *tid) { -switch (pr_reg->pr_tid[0 ] &0xF ) { +switch (tid[0 ] &0xF ) {case SCSI_PROTOCOL_FCP :case SCSI_PROTOCOL_SBP :case SCSI_PROTOCOL_SRP :case SCSI_PROTOCOL_SAS :return 24 ;case SCSI_PROTOCOL_ISCSI : -return get_unaligned_be16 (&pr_reg->pr_tid[2 ]) +4 ; +return get_unaligned_be16 (&tid[2 ]) +4 ;default : -WARN (1 ,"Unknown proto_id: %#x\n" , pr_reg->pr_tid[0 ] &0xF ); +WARN (1 ,"Unknown proto_id: %#x\n" , tid[0 ] &0xF );return -EINVAL ; } } +int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg) +{ +return __target_get_pr_transport_id_len (pr_reg->pr_tid); +} +int target_get_pr_transport_id (struct t10_pr_registration *pr_reg,unsigned char *buf)
@@ -378,6 +383,20 @@ int target_get_pr_transport_id(return len; } +int target_cmp_pr_transport_id ( +struct t10_pr_registration *pr_reg, +unsigned char *tid) +{ +int len1 =__target_get_pr_transport_id_len (pr_reg->pr_tid); +int len2 =__target_get_pr_transport_id_len (tid); + +if (len1 != len2) +return 1 ; + +return memcmp (pr_reg->pr_tid, tid, len1); +} + + int target_gen_pr_transport_id( struct t10_pr_registration *pr_reg, int proto_id,
drivers/target/target_core_internal.h +2 -0
@@ -101,6 +101,8 @@ int target_fabric_setup_cits(struct target_fabric_configfs *);int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg);int target_get_pr_transport_id (struct t10_pr_registration *pr_reg,unsigned char *buf); +int target_cmp_pr_transport_id (struct t10_pr_registration *pr_reg, +unsigned char *tid); int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg, int proto_id, const char *initiatorname,
Signature · 1 line
2.25.1
RFC PATCH 29/48 target: core: store proto_id in APTPL
In reply to Dmitry BogdanovStore and read proto_id in APTPL to create TransportID.
That will allow to report registrations for not yet created TPG/ACL/LUNs
--- drivers/target/target_core_configfs.c | 13 +++++++++++-- drivers/target/target_core_pr.c | 15 ++++++++++----- drivers/target/target_core_pr.h | 2 +- 3 files changed, 22 insertions(+), 8 deletions(-)
drivers/target/target_core_configfs.c +11 -2
@@ -1974,7 +1974,8 @@ enum { Opt_initiator_fabric, Opt_initiator_node, Opt_initiator_sid, Opt_sa_res_key, Opt_res_holder, Opt_res_type, Opt_res_scope, Opt_res_all_tg_pt, Opt_mapped_lun, Opt_target_fabric, - Opt_target_node, Opt_tpgt, Opt_port_rtpi, Opt_target_lun, Opt_err + Opt_target_node, Opt_tpgt, Opt_port_rtpi, Opt_target_lun, + Opt_proto_id, Opt_err }; static match_table_t tokens = {
@@ -1992,6 +1993,7 @@ static match_table_t tokens = { {Opt_tpgt, "tpgt=%d"}, {Opt_port_rtpi, "port_rtpi=%d"}, {Opt_target_lun, "target_lun=%u"}, + {Opt_proto_id, "proto_id=%u"}, {Opt_err, NULL} };
@@ -2010,6 +2012,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,u16 tpgt =0 ;u16 rtpi =0 ;u8 type =0 ; +u8 proto_id =0 ;if (!dev->dev_attrib.emulate_pr || (dev->transport_flags &TRANSPORT_FLAG_PASSTHROUGH_PGR ))
@@ -2151,6 +2154,12 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item, goto out; target_lun = (u64 )tmp_ll;break ; +case Opt_proto_id: + ret =match_int (args, &arg); +if (ret) + goto out; + proto_id = (u16 )arg; +break ;default :break ; }
@@ -2171,7 +2180,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item, ret =core_scsi3_alloc_aptpl_registration (&dev->t10_pr, sa_res_key, i_port, isid, mapped_lun, rtpi, target_lun, - res_holder, all_tg_pt, type); + res_holder, all_tg_pt, type, proto_id); out:kfree (i_fabric);kfree (i_port);
drivers/target/target_core_pr.c +10 -5
@@ -842,7 +842,8 @@ int core_scsi3_alloc_aptpl_registration(u64 target_lun, int res_holder, int all_tg_pt, - u8 type) + u8 type, + u8 proto_id) { struct t10_pr_registration *pr_reg;
@@ -886,6 +887,7 @@ int core_scsi3_alloc_aptpl_registration(* Copy the i_port information from caller. */ snprintf (pr_reg->pr_iport,PR_APTPL_MAX_IPORT_LEN ,"%s" , i_port); +target_gen_pr_transport_id (pr_reg, proto_id, i_port, isid);/* * Set pr_res_holder from caller, the pr_reg who is the reservation * holder will get it's pointer set in core_scsi3_aptpl_reserve() once
@@ -1836,19 +1838,22 @@ static int core_scsi3_update_aptpl_buf("sa_res_key=%llu\n" "res_holder=1\nres_type=%02x\n" "res_scope=%02x\nres_all_tg_pt=%d\n" -"mapped_lun=%llu\n" , reg_count, +"mapped_lun=%llu\nproto_id=%u\n" , reg_count, pr_reg->pr_iport, isid_buf, pr_reg->pr_res_key, pr_reg->pr_res_type, pr_reg->pr_res_scope, pr_reg->pr_reg_all_tg_pt, - pr_reg->pr_res_mapped_lun); + pr_reg->pr_res_mapped_lun, + pr_reg->pr_tid[0] & 0xF); }else {snprintf (tmp,512 ,"PR_REG_START: %d\n" "initiator_node=%s\n%s" "sa_res_key=%llu\nres_holder=0\n" -"res_all_tg_pt=%d\nmapped_lun=%llu\n" , +"res_all_tg_pt=%d\nmapped_lun=%llu\n" +"proto_id=%u\n" , reg_count, pr_reg->pr_iport, isid_buf, pr_reg->pr_res_key, pr_reg->pr_reg_all_tg_pt, - pr_reg->pr_res_mapped_lun); + pr_reg->pr_res_mapped_lun, + pr_reg->pr_tid[0 ] &0xF ); }if ((len +strlen (tmp) >= pr_aptpl_buf_len)) {
drivers/target/target_core_pr.h +1 -1
@@ -64,7 +64,7 @@ extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);extern int core_scsi3_alloc_aptpl_registration (struct t10_reservation *,u64 ,unsigned char *,unsigned char *,u64 , -u16 ,u64 ,int ,int ,u8 ); +u16 ,u64 ,int ,int ,u8 ,u8 );extern int core_scsi3_check_aptpl_registration (struct se_device *,struct se_portal_group *,struct se_lun *,struct se_node_acl *,u64 );
Signature · 1 line
2.25.1
RFC PATCH 30/48 target: core: rethink APTPL registrations
In reply to Dmitry BogdanovMake APTPL registrations be real registrations from the very beginning. Just mark at deve creation that it has a PR registration if it exists.
--- drivers/target/target_core_pr.c | 104 +++++++++++++------------------- 1 file changed, 43 insertions(+), 61 deletions(-)
drivers/target/target_core_pr.c +43 -61
@@ -832,6 +832,28 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(return NULL ; } +static void __core_scsi3_add_registration (struct se_device *,struct se_node_acl *, +struct t10_pr_registration *,enum register_type ,int ); + +static void core_scsi3_aptpl_reserve ( +struct se_device *dev, +const char *initiatorname, +struct t10_pr_registration *pr_reg) +{ +char i_buf[PR_REG_ISID_ID_LEN ] = { }; + +core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN ); + +spin_lock (&dev->dev_reservation_lock); + dev->dev_pr_res_holder = pr_reg; +spin_unlock (&dev->dev_reservation_lock); + +pr_debug ("SPC-3 PR Service Action: APTPL RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n" , +core_scsi3_pr_dump_type (pr_reg->pr_res_type), + (pr_reg->pr_reg_all_tg_pt) ?1 :0 ); +pr_debug ("SPC-3 PR RESERVE Node: %s%s\n" , initiatorname, i_buf); +} + int core_scsi3_alloc_aptpl_registration( struct t10_reservation *pr_tmpl, u64 sa_res_key,
@@ -845,6 +867,7 @@ int core_scsi3_alloc_aptpl_registration( u8 type, u8 proto_id) { + struct se_device *dev = container_of(pr_tmpl, struct se_device, t10_pr);struct t10_pr_registration *pr_reg;if (!i_port || !sa_res_key) {
@@ -888,47 +911,24 @@ int core_scsi3_alloc_aptpl_registration(*/ snprintf (pr_reg->pr_iport,PR_APTPL_MAX_IPORT_LEN ,"%s" , i_port);target_gen_pr_transport_id (pr_reg, proto_id, i_port, isid); + + pr_reg->pr_res_holder = res_holder; +__core_scsi3_add_registration (dev,NULL , pr_reg,REGISTER ,0 ); + +if (pr_reg->pr_res_holder) +core_scsi3_aptpl_reserve (dev, i_port, pr_reg); +/* -* Set pr_res_holder from caller, the pr_reg who is the reservation -* holder will get it's pointer set in core_scsi3_aptpl_reserve() once -* the Initiator Node LUN ACL from the fabric module is created for -* this registration. +* Reenable pr_aptpl_active to accept new metadata +* updates once the SCSI device is active again.. */ - pr_reg->pr_res_holder = res_holder; + pr_tmpl->pr_aptpl_active =1 ; -list_add_tail (&pr_reg->pr_reg_aptpl_list, &pr_tmpl->aptpl_reg_list);pr_debug ("SPC-3 PR APTPL Successfully added registration%s from" " metadata\n" , (res_holder) ?"+reservation" :"" );return 0 ; } -static void core_scsi3_aptpl_reserve ( -struct se_device *dev, -struct se_portal_group *tpg, -struct se_node_acl *node_acl, -struct t10_pr_registration *pr_reg) -{ -char i_buf[PR_REG_ISID_ID_LEN ] = { }; - -core_pr_dump_initiator_port (pr_reg, i_buf,PR_REG_ISID_ID_LEN ); - -spin_lock (&dev->dev_reservation_lock); - dev->dev_pr_res_holder = pr_reg; -spin_unlock (&dev->dev_reservation_lock); - -pr_debug ("SPC-3 PR [%s] Service Action: APTPL RESERVE created" -" new reservation holder TYPE: %s ALL_TG_PT: %d\n" , - tpg->se_tpg_tfo->fabric_name, -core_scsi3_pr_dump_type (pr_reg->pr_res_type), - (pr_reg->pr_reg_all_tg_pt) ?1 :0 ); -pr_debug ("SPC-3 PR [%s] RESERVE Node: %s%s\n" , - tpg->se_tpg_tfo->fabric_name, node_acl->initiatorname, - i_buf); -} - -static void __core_scsi3_add_registration (struct se_device *,struct se_node_acl *, -struct t10_pr_registration *,enum register_type ,int ); - static int __core_scsi3_check_aptpl_registration( struct se_device *dev, struct se_portal_group *tpg,
@@ -946,12 +946,13 @@ static int __core_scsi3_check_aptpl_registration(* may exist for fabrics that use ISIDs in their SCSI Initiator Port * TransportIDs. */ -spin_lock (&pr_tmpl->aptpl_reg_lock); -list_for_each_entry_safe (pr_reg, pr_reg_tmp, &pr_tmpl->aptpl_reg_list, - pr_reg_aptpl_list) { +spin_lock (&pr_tmpl->registration_lock); +list_for_each_entry_safe (pr_reg, pr_reg_tmp, + &pr_tmpl->registration_list, pr_reg_list) { -if (!strcmp (pr_reg->pr_iport, nacl->initiatorname) && - (pr_reg->tg_pt_sep_rtpi == tpg->tpg_rtpi)) { +if (pr_reg->pr_reg_nacl ==NULL && + !strcmp (pr_reg->pr_iport, nacl->initiatorname) && + (pr_reg->tg_pt_sep_rtpi == tpg->tpg_rtpi)) {/* * Obtain the ->pr_reg_deve pointer + reference, that * is released by __core_scsi3_add_registration() below.
@@ -964,34 +965,15 @@ static int __core_scsi3_check_aptpl_registration(rcu_read_unlock ();continue ; } -kref_get (&pr_reg->pr_reg_deve->pr_kref); +set_bit (DEF_PR_REG_ACTIVE , &pr_reg->pr_reg_deve->deve_flags); + pr_reg->pr_reg_deve =NULL ;rcu_read_unlock (); pr_reg->pr_reg_nacl = nacl; - pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi; -list_del (&pr_reg->pr_reg_aptpl_list); -spin_unlock (&pr_tmpl->aptpl_reg_lock); -/* -* At this point all of the pointers in *pr_reg will -* be setup, so go ahead and add the registration. -*/ -__core_scsi3_add_registration (dev, nacl, pr_reg,0 ,0 ); -/* -* If this registration is the reservation holder, -* make that happen now.. -*/ -if (pr_reg->pr_res_holder) -core_scsi3_aptpl_reserve (dev, tpg, - nacl, pr_reg); -/* -* Reenable pr_aptpl_active to accept new metadata -* updates once the SCSI device is active again.. -*/ -spin_lock (&pr_tmpl->aptpl_reg_lock); - pr_tmpl->pr_aptpl_active =1 ; + pr_reg->pr_res_mapped_lun = mapped_lun; } } -spin_unlock (&pr_tmpl->aptpl_reg_lock); +spin_unlock (&pr_tmpl->registration_lock);return 0 ; }
Signature · 1 line
2.25.1
RFC PATCH 31/48 dlm_ckv: introduce DLM cluster key-value storage
In reply to Dmitry BogdanovIntroduce the first version of DLM CKV module that could be used by different kernel subsystems to share some information in a cluster.
This commit has just cluster level locks.
--- drivers/target/Kconfig | 6 + drivers/target/Makefile | 2 + drivers/target/dlm_ckv.c | 323 +++++++++++++++++++++++++++++++++++++++ drivers/target/dlm_ckv.h | 19 +++ 4 files changed, 350 insertions(+) create mode 100644 drivers/target/dlm_ckv.c create mode 100644 drivers/target/dlm_ckv.h
drivers/target/Kconfig +6 -0
@@ -35,6 +35,12 @@ config TCM_PSCSI Say Y here to enable the TCM/pSCSI subsystem plugin for non-buffered passthrough access to Linux/SCSI device +config DLM_CKV + tristate "Cluster key value storage over DLM" + depends on DLM + help + Say Y here to enable the cluster key value storage over DLM + config TCM_USER2 tristate "TCM/USER Subsystem Plugin for Linux" depends on UIO && NET
drivers/target/Makefile +2 -0
@@ -30,3 +30,5 @@ obj-$(CONFIG_LOOPBACK_TARGET) += loopback/ obj-$(CONFIG_TCM_FC) += tcm_fc/ obj-$(CONFIG_ISCSI_TARGET) += iscsi/ obj-$(CONFIG_SBP_TARGET) += sbp/ + +obj-$(CONFIG_DLM_CKV) += dlm_ckv.o
drivers/target/dlm_ckv.c new file +323 -0
323 lines not on this page · raw message
drivers/target/dlm_ckv.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef DLM_CKV_H +#define DLM_CKV_H + +struct dlm_ckv_bucket ; +struct dlm_ckv_lock ; + +struct dlm_ckv_bucket *dlm_ckv_open_bucket (const char *name, +const char *cluster_name, +void *userarg); +int dlm_ckv_close_bucket (struct dlm_ckv_bucket *bucket); + +struct dlm_ckv_lock * +dlm_ckv_create_lock (struct dlm_ckv_bucket *bucket,const char *name); +void dlm_ckv_free_lock (struct dlm_ckv_lock *ckv_lock); +int dlm_ckv_lock_get (struct dlm_ckv_lock *ckv_lock); +int dlm_ckv_lock_release (struct dlm_ckv_lock *ckv_lock); + +#endif
Signature · 1 line
2.25.1
RFC PATCH 32/48 dlm_ckv: add notification service
In reply to Dmitry BogdanovNotification broadcasts over DLM cluster.
They do not have any payload. Used to be used to notify other nodes to read an updated data.
--- drivers/target/dlm_ckv.c | 186 +++++++++++++++++++++++++++++++++++++++ drivers/target/dlm_ckv.h | 8 ++ 2 files changed, 194 insertions(+)
drivers/target/dlm_ckv.c +186 -0
@@ -10,6 +10,7 @@#include <linux/module.h> #include <linux/kthread.h> #include <linux/sched.h> +#include <linux/workqueue.h> #include <target/target_core_base.h> #include "dlm_ckv.h"
@@ -33,8 +34,17 @@ struct dlm_ckv_bucket {int nodeid[64 ];void *userarg;struct completion sync_compl; +struct workqueue_struct *notify_wq; }; +struct dlm_ckv_notify { +dlm_ckv_notify_cb notify_cb; +char name[DLM_RESNAME_MAXLEN ]; +struct dlm_ckv_lock pre_n; +struct dlm_ckv_lock post_n; +struct work_struct pre_n_work; +struct work_struct post_n_work; +};#define DLM_CKV_LVB_SIZE 256
@@ -230,6 +240,179 @@ dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock) }EXPORT_SYMBOL (dlm_ckv_lock_release); +static void dlm_cvk_pre_n_bast (void *astarg,int mode) +{ +struct dlm_ckv_lksb *lksb = astarg; +struct dlm_ckv_lock *ckv_lock; +struct dlm_ckv_bucket *bucket; +struct dlm_ckv_notify *notify; + + ckv_lock = container_of(lksb, struct dlm_ckv_lock, lksb); + notify = container_of(ckv_lock, struct dlm_ckv_notify, pre_n); + bucket = ckv_lock->bucket; + +queue_work (bucket->notify_wq, ¬ify->pre_n_work); +} + +static void dlm_cvk_post_n_bast (void *astarg,int mode) +{ +struct dlm_ckv_lksb *lksb = astarg; +struct dlm_ckv_lock *ckv_lock; +struct dlm_ckv_bucket *bucket; +struct dlm_ckv_notify *notify; + + ckv_lock = container_of(lksb, struct dlm_ckv_lock, lksb); + notify = container_of(ckv_lock, struct dlm_ckv_notify, post_n); + bucket = ckv_lock->bucket; + +queue_work (bucket->notify_wq, ¬ify->post_n_work); +} + +static void +dlm_cvk_pre_n_work (struct work_struct *work) +{ +struct dlm_ckv_notify *notify =container_of (work, + struct dlm_ckv_notify, pre_n_work); +struct dlm_ckv_bucket *bucket = notify->pre_n.bucket; + +dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_EX , + ¬ify->post_n.lksb, +DLM_LKF_CONVERT , + notify->post_n.name, dlm_cvk_post_n_bast); +dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_NL , + ¬ify->pre_n.lksb, +DLM_LKF_CONVERT , + notify->pre_n.name, dlm_cvk_pre_n_bast); +} + +static void +dlm_cvk_post_n_work (struct work_struct *work) +{ +struct dlm_ckv_notify *notify =container_of (work, + struct dlm_ckv_notify, post_n_work); +struct dlm_ckv_bucket *bucket = notify->post_n.bucket; + +dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_EX , + ¬ify->pre_n.lksb, +DLM_LKF_CONVERT , + notify->pre_n.name, dlm_cvk_pre_n_bast); + + notify->notify_cb (bucket->userarg); + +dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_NL , + ¬ify->post_n.lksb, +DLM_LKF_CONVERT , + notify->post_n.name, dlm_cvk_post_n_bast); +} + +struct dlm_ckv_notify * +dlm_ckv_create_notification (struct dlm_ckv_bucket *bucket,const char *name, +dlm_ckv_notify_cb cb) +{ +char lockname[DLM_RESNAME_MAXLEN ]; +struct dlm_ckv_notify *notify; +int res; + + notify =kzalloc (sizeof (struct dlm_ckv_bucket ),GFP_KERNEL ); +if (!notify) +return NULL ; + +INIT_WORK (¬ify->post_n_work, dlm_cvk_post_n_work); +INIT_WORK (¬ify->pre_n_work, dlm_cvk_pre_n_work); + +strscpy (notify->name, name,DLM_RESNAME_MAXLEN ); + notify->notify_cb = cb; + +kref_get (&bucket->refcount); + +sprintf (lockname,"pre.%s.%d" , name, bucket->local_nodeid); +dlm_ckv_lock_init (¬ify->pre_n, bucket, lockname); + +sprintf (lockname,"post.%s.%d" , name, bucket->local_nodeid); +dlm_ckv_lock_init (¬ify->post_n, bucket, lockname); + + res =dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_EX , + ¬ify->pre_n.lksb,0 , + notify->pre_n.name, dlm_cvk_pre_n_bast); +if (res) + goto fail_locks; + + res =dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_NL , + ¬ify->post_n.lksb,0 , + notify->post_n.name, dlm_cvk_post_n_bast); +if (res) + goto fail_locks; + +return notify; + +fail_locks: +kref_put (&bucket->refcount, bucket_release); +kfree (notify); + +return NULL ; +} +EXPORT_SYMBOL (dlm_ckv_create_notification); + +void +dlm_ckv_free_notification (struct dlm_ckv_notify *notify) +{ +struct dlm_ckv_bucket *bucket = notify->pre_n.bucket; + +cancel_work_sync (¬ify->pre_n_work); +cancel_work_sync (¬ify->post_n_work); + +kfree (notify); +kref_put (&bucket->refcount, bucket_release); +} +EXPORT_SYMBOL (dlm_ckv_free_notification); + +static void +dlm_ckv_toggle_lock (struct dlm_ckv_bucket *bucket,const char *name_prefix) +{ +char lockname[DLM_RESNAME_MAXLEN ]; +struct dlm_ckv_lksb lksb; +int res; +int i; + +init_completion (&lksb.compl); + +for (i =0 ; i < bucket->num_nodes; ++i) { +if (bucket->nodeid[i] == bucket->local_nodeid) +continue ; +snprintf (lockname,sizeof (lockname),"%s.%d" , name_prefix, + bucket->nodeid[i]); + + lksb.lksb.sb_lkid =0 ; + res =dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_PR , + &lksb,0 , lockname,NULL ); +if (res <0 ) +pr_warn ("Locking %s failed (%d)" , lockname, res); +if (!lksb.lksb.sb_lkid) +continue ; + +dlm_ckv_lock_wait (bucket->ls,DLM_LOCK_NL , &lksb, +DLM_LKF_CONVERT , lockname,NULL ); +dlm_ckv_unlock_wait (bucket->ls, &lksb); + } +} + +int dlm_ckv_notify (struct dlm_ckv_notify *notify) +{ +struct dlm_ckv_bucket *bucket = notify->pre_n.bucket; +char lockname[DLM_RESNAME_MAXLEN ]; + +snprintf (lockname,sizeof (lockname),"post.%s" , notify->name); +dlm_ckv_toggle_lock (bucket, lockname); + +snprintf (lockname,sizeof (lockname),"pre.%s" , notify->name); +dlm_ckv_toggle_lock (bucket, lockname); + +snprintf (lockname,sizeof (lockname),"post.%s" , notify->name); +dlm_ckv_toggle_lock (bucket, lockname); + +return 0 ; +} +EXPORT_SYMBOL (dlm_ckv_notify);static void bucket_release (struct kref *ref) {
@@ -237,6 +420,8 @@ static void bucket_release(struct kref *ref) refcount);int res; +destroy_workqueue (bucket->notify_wq); + res =dlm_release_lockspace (bucket->ls,2 );if (res) pr_err("forcibly releasing lockspace failed: %d\n",
@@ -264,6 +449,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg) bucket->userarg = userarg;init_completion (&bucket->sync_compl); + bucket->notify_wq =alloc_ordered_workqueue ("notify_wq-%s" ,0 , name); err = dlm_new_lockspace(name, cluster_name, DLM_LSFL_FS | DLM_LSFL_NEWEXCL, DLM_CKV_LVB_SIZE,
drivers/target/dlm_ckv.h +8 -0
@@ -5,6 +5,8 @@struct dlm_ckv_bucket ;struct dlm_ckv_lock ; +typedef void (*dlm_ckv_notify_cb )(void *userarg); +struct dlm_ckv_bucket *dlm_ckv_open_bucket (const char *name,const char *cluster_name,void *userarg);
@@ -16,4 +18,10 @@ void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);int dlm_ckv_lock_get (struct dlm_ckv_lock *ckv_lock);int dlm_ckv_lock_release (struct dlm_ckv_lock *ckv_lock); +struct dlm_ckv_notify * +dlm_ckv_create_notification (struct dlm_ckv_bucket *bucket,const char *name, +dlm_ckv_notify_cb cb); +void dlm_ckv_free_notification (struct dlm_ckv_notify *notify); +int dlm_ckv_notify (struct dlm_ckv_notify *notify); +#endif
Signature · 1 line
2.25.1
RFC PATCH 33/48 dlm_ckv: add key-value storage service
In reply to Dmitry BogdanovSimple key-value cluster storage.
--- drivers/target/dlm_ckv.c | 128 +++++++++++++++++++++++++++++++++++++++ drivers/target/dlm_ckv.h | 9 +++ 2 files changed, 137 insertions(+)
drivers/target/dlm_ckv.c +128 -0
@@ -25,6 +25,10 @@ struct dlm_ckv_lock {char name[DLM_RESNAME_MAXLEN ]; }; +struct dlm_ckv_kv { +struct dlm_ckv_lock lock; +}; +struct dlm_ckv_bucket {dlm_lockspace_t *ls;struct kref refcount;
@@ -240,6 +244,130 @@ dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock) }EXPORT_SYMBOL (dlm_ckv_lock_release); +struct dlm_ckv_kv * +dlm_ckv_create_kv (struct dlm_ckv_bucket *bucket,const char *key) +{ +struct dlm_ckv_lock *ckv_lock; +void *ptr; +int res; + + ptr =kzalloc (DLM_CKV_LVB_SIZE ,GFP_KERNEL ); +if (!ptr) +return NULL ; + + ckv_lock =dlm_ckv_create_lock (bucket, key); +if (!ckv_lock) + goto create_fail; + + ckv_lock->lksb.lksb.sb_lvbptr = ptr; + + res =dlm_ckv_lock_wait (ckv_lock->bucket->ls,DLM_LOCK_NL , + &ckv_lock->lksb,0 , ckv_lock->name,NULL ); +if (res) + goto lock_failed; + +return (struct dlm_ckv_kv *)ckv_lock; + +lock_failed: +dlm_ckv_free_lock (ckv_lock); +create_fail: +kfree (ptr); +return NULL ; +} +EXPORT_SYMBOL (dlm_ckv_create_kv); + +void +dlm_ckv_free_kv (struct dlm_ckv_kv *kv) +{ +struct dlm_ckv_bucket *bucket = kv->lock.bucket; +struct dlm_ckv_lock *ckv_lock = &kv->lock; + +dlm_ckv_unlock_wait (bucket->ls, &ckv_lock->lksb); + +kfree (ckv_lock->lksb.lksb.sb_lvbptr); +kfree (ckv_lock); + +kref_put (&bucket->refcount, bucket_release); + +} +EXPORT_SYMBOL (dlm_ckv_free_kv); + +int +dlm_ckv_get (struct dlm_ckv_kv *kv,char *value,size_t len) +{ +struct dlm_ckv_lock *ckv_lock = &kv->lock; +struct dlm_ckv_bucket *bucket; +int res; + +BUG_ON (!ckv_lock); + bucket = ckv_lock->bucket; + + res =dlm_ckv_lock_wait (ckv_lock->bucket->ls,DLM_LOCK_CR , + &ckv_lock->lksb, +DLM_LKF_VALBLK |DLM_LKF_CONVERT , + ckv_lock->name,NULL ); +if (res) { +pr_info ("Can not get lock %s, rc=%d\n" , + ckv_lock->name, res); + goto fail; + } + +if (ckv_lock->lksb.lksb.sb_flags &DLM_SBF_VALNOTVALID ) { +pr_info (" %s LVB was invalid\n" , ckv_lock->name); +memset (value,0 , len); + }else +memcpy (value, ckv_lock->lksb.lksb.sb_lvbptr, len); + + res =dlm_ckv_lock_wait (ckv_lock->bucket->ls,DLM_LOCK_NL , + &ckv_lock->lksb, +DLM_LKF_CONVERT , + ckv_lock->name,NULL ); +if (res) { +pr_info ("Can not release lock %s\n" , ckv_lock->name); + goto fail; + } + +return res; +fail: +return res; +} +EXPORT_SYMBOL (dlm_ckv_get); + +int +dlm_ckv_set (struct dlm_ckv_kv *kv,const char *value,size_t len) +{ +struct dlm_ckv_lock *ckv_lock = &kv->lock; +struct dlm_ckv_bucket *bucket; +int res; + +BUG_ON (!ckv_lock); + bucket = ckv_lock->bucket; + + res =dlm_ckv_lock_wait (ckv_lock->bucket->ls,DLM_LOCK_EX , + &ckv_lock->lksb, +DLM_LKF_CONVERT , + ckv_lock->name,NULL ); +if (res) { +pr_info ("Can not get lock %s\n" , ckv_lock->name); + goto fail; + } + +memcpy (ckv_lock->lksb.lksb.sb_lvbptr, value, len); + res =dlm_ckv_lock_wait (ckv_lock->bucket->ls,DLM_LOCK_NL , + &ckv_lock->lksb, +DLM_LKF_VALBLK |DLM_LKF_CONVERT , + ckv_lock->name,NULL ); +if (res) { +pr_info ("Can not release lock %s\n" , ckv_lock->name); + goto fail; + } + +return res; +fail: +return res; +} +EXPORT_SYMBOL (dlm_ckv_set); +static void dlm_cvk_pre_n_bast (void *astarg,int mode) {struct dlm_ckv_lksb *lksb = astarg;
drivers/target/dlm_ckv.h +9 -0
@@ -4,6 +4,9 @@struct dlm_ckv_bucket ;struct dlm_ckv_lock ; +struct dlm_ckv_kv ; + +#define DLM_CKV_VALUE_MAX_SIZE 255typedef void (*dlm_ckv_notify_cb )(void *userarg);
@@ -18,6 +21,12 @@ void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);int dlm_ckv_lock_get (struct dlm_ckv_lock *ckv_lock);int dlm_ckv_lock_release (struct dlm_ckv_lock *ckv_lock); +struct dlm_ckv_kv * +dlm_ckv_create_kv (struct dlm_ckv_bucket *bucket,const char *key); +void dlm_ckv_free_kv (struct dlm_ckv_kv *kv); +int dlm_ckv_get (struct dlm_ckv_kv *kv,char *value,size_t len); +int dlm_ckv_set (struct dlm_ckv_kv *kv,const char *value,size_t len); +struct dlm_ckv_notify *dlm_ckv_create_notification (struct dlm_ckv_bucket *bucket,const char *name,dlm_ckv_notify_cb cb);
Signature · 1 line
2.25.1
RFC PATCH 34/48 dlm_ckv: add KV get/set async API
In reply to Dmitry BogdanovThe sync versions of KV set/get functions may take too long time if user want to store/read a number of keys.
Add async versions of KV get/set functions to allow the user to group the requests and to wait for completion of all requests together.
--- drivers/target/dlm_ckv.c | 110 ++++++++++++++++++++++++++++++++++++++- drivers/target/dlm_ckv.h | 6 +++ 2 files changed, 115 insertions(+), 1 deletion(-)
drivers/target/dlm_ckv.c +109 -1
@@ -14,15 +14,27 @@#include <target/target_core_base.h> #include "dlm_ckv.h" +enum dlm_lksb_state { +ONE_STAGE_SYNC , +READ_FIRST_STAGE , +READ_SECOND_STAGE , +WRITE_FIRST_STAGE , +WRITE_SECOND_STAGE , +};struct dlm_ckv_lksb {struct dlm_lksb lksb;struct completion compl; +enum dlm_lksb_state state; };struct dlm_ckv_lock {struct dlm_ckv_bucket *bucket;struct dlm_ckv_lksb lksb;char name[DLM_RESNAME_MAXLEN ]; +dlm_ckv_async_cb async_cb; +void *cb_arg; +void *value; +size_t len; };struct dlm_ckv_kv {
@@ -97,8 +109,48 @@ static const struct dlm_lockspace_ops dlm_ckv_lockspace_ops = {static void dlm_ast (void *astarg) {struct dlm_ckv_lksb *dlm_ckv_lksb = astarg; +struct dlm_ckv_lock *ckv_lock; +struct dlm_ckv_bucket *bucket; +int res; + + ckv_lock = container_of(dlm_ckv_lksb, struct dlm_ckv_lock, lksb); + bucket = ckv_lock->bucket; -complete (&dlm_ckv_lksb->compl); +switch (dlm_ckv_lksb->state) { +case ONE_STAGE_SYNC : +complete (&dlm_ckv_lksb->compl); +break ; +case READ_FIRST_STAGE : +if (dlm_ckv_lksb->lksb.sb_flags &DLM_SBF_VALNOTVALID ) { +pr_debug ("%s LVB was invalid\n" , ckv_lock->name); +memset (ckv_lock->value,0 , ckv_lock->len); + }else +memcpy (ckv_lock->value, dlm_ckv_lksb->lksb.sb_lvbptr, + ckv_lock->len); + + dlm_ckv_lksb->state =READ_SECOND_STAGE ; + res =dlm_lock (ckv_lock->bucket->ls,DLM_LOCK_NL , + &dlm_ckv_lksb->lksb,DLM_LKF_CONVERT ,0 ,0 ,0 , + dlm_ast, dlm_ckv_lksb,NULL ); +if (res) + ckv_lock->async_cb (ckv_lock->cb_arg, res); +break ; +case WRITE_FIRST_STAGE : + + dlm_ckv_lksb->state =WRITE_SECOND_STAGE ; + res =dlm_lock (ckv_lock->bucket->ls,DLM_LOCK_NL , + &dlm_ckv_lksb->lksb, +DLM_LKF_VALBLK |DLM_LKF_CONVERT ,0 ,0 ,0 , + dlm_ast, dlm_ckv_lksb,NULL ); +if (res) + ckv_lock->async_cb (ckv_lock->cb_arg, res); +break ; +case READ_SECOND_STAGE : + fallthrough; +case WRITE_SECOND_STAGE : + ckv_lock->async_cb (ckv_lock->cb_arg, dlm_ckv_lksb->lksb.sb_status); +break ; + } }/*
@@ -135,6 +187,7 @@ static int dlm_ckv_lock_wait(dlm_lockspace_t *ls, int mode, {int res; + lksb->state =ONE_STAGE_SYNC ; res =dlm_lock (ls, mode, &lksb->lksb, flags, (void *)name, name ?strlen (name) :0 ,0 , dlm_ast, lksb, bast);
@@ -164,6 +217,7 @@ static int dlm_ckv_unlock_wait(dlm_lockspace_t *ls, struct dlm_ckv_lksb *lksb) {int res; + lksb->state =ONE_STAGE_SYNC ; res =dlm_unlock (ls, lksb->lksb.sb_lkid,0 , &lksb->lksb, lksb);if (res <0 ) goto out;
@@ -333,6 +387,32 @@ dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len) }EXPORT_SYMBOL (dlm_ckv_get); +int +dlm_ckv_get_async (struct dlm_ckv_kv *kv,char *value,size_t len, +dlm_ckv_async_cb cb,void *cb_arg) +{ +struct dlm_ckv_lock *ckv_lock = &kv->lock; +struct dlm_ckv_bucket *bucket; +int res; + +BUG_ON (!ckv_lock); + bucket = ckv_lock->bucket; + + ckv_lock->lksb.state =READ_FIRST_STAGE ; + ckv_lock->len = len; + ckv_lock->value = value; + ckv_lock->async_cb = cb; + ckv_lock->cb_arg = cb_arg; + res =dlm_lock (ckv_lock->bucket->ls,DLM_LOCK_CR , &ckv_lock->lksb.lksb, +DLM_LKF_VALBLK |DLM_LKF_CONVERT ,NULL ,0 ,0 , + dlm_ast, &ckv_lock->lksb,NULL ); +if (res) +pr_info ("Can not get lock %s, rc=%d\n" , ckv_lock->name, res); + +return res; +} +EXPORT_SYMBOL (dlm_ckv_get_async); +int dlm_ckv_set (struct dlm_ckv_kv *kv,const char *value,size_t len) {
@@ -368,6 +448,34 @@ dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len) }EXPORT_SYMBOL (dlm_ckv_set); +int +dlm_ckv_set_async (struct dlm_ckv_kv *kv,char *value,size_t len, +dlm_ckv_async_cb cb,void *cb_arg) +{ +struct dlm_ckv_lock *ckv_lock = &kv->lock; +struct dlm_ckv_bucket *bucket; +int res; + +BUG_ON (!ckv_lock); + bucket = ckv_lock->bucket; + + ckv_lock->lksb.state =WRITE_FIRST_STAGE ; + ckv_lock->len = len; + ckv_lock->value = value; + ckv_lock->async_cb = cb; + ckv_lock->cb_arg = cb_arg; +memcpy (ckv_lock->lksb.lksb.sb_lvbptr, ckv_lock->value, ckv_lock->len); + + res =dlm_lock (ckv_lock->bucket->ls,DLM_LOCK_EX , &ckv_lock->lksb.lksb, +DLM_LKF_CONVERT ,NULL ,0 ,0 , + dlm_ast, &ckv_lock->lksb,NULL ); +if (res) +pr_info ("Can not get lock %s\n" , ckv_lock->name); + +return res; +} +EXPORT_SYMBOL (dlm_ckv_set_async); +static void dlm_cvk_pre_n_bast (void *astarg,int mode) {struct dlm_ckv_lksb *lksb = astarg;
drivers/target/dlm_ckv.h +6 -0
@@ -9,6 +9,8 @@ struct dlm_ckv_kv;#define DLM_CKV_VALUE_MAX_SIZE 255typedef void (*dlm_ckv_notify_cb )(void *userarg); +typedef void (*dlm_ckv_async_cb )(void *userarg,int res); +typedef void (*dlm_ckv_nodeleft_cb )(void *arg,int nodeid); struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name, const char *cluster_name,
@@ -26,6 +28,10 @@ dlm_ckv_create_kv(struct dlm_ckv_bucket *bucket, const char *key);void dlm_ckv_free_kv (struct dlm_ckv_kv *kv);int dlm_ckv_get (struct dlm_ckv_kv *kv,char *value,size_t len);int dlm_ckv_set (struct dlm_ckv_kv *kv,const char *value,size_t len); +int dlm_ckv_get_async (struct dlm_ckv_kv *kv,char *value,size_t len, +dlm_ckv_async_cb cb,void *cb_arg); +int dlm_ckv_set_async (struct dlm_ckv_kv *kv,char *value,size_t len, +dlm_ckv_async_cb cb,void *cb_arg);struct dlm_ckv_notify * dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
Signature · 1 line
2.25.1
RFC PATCH 35/48 target: add virtual remote target
In reply to Dmitry BogdanovCreate virtual remote target module.
It can be used to see a whole acl/lun/tpg configuration from all nodes in storage cluster.
--- drivers/target/Kconfig | 1 + drivers/target/Makefile | 1 + drivers/target/tcm_remote/Kconfig | 8 + drivers/target/tcm_remote/Makefile | 2 + drivers/target/tcm_remote/tcm_remote.c | 405 +++++++++++++++++++++++++ drivers/target/tcm_remote/tcm_remote.h | 29 ++ 6 files changed, 446 insertions(+) create mode 100644 drivers/target/tcm_remote/Kconfig create mode 100644 drivers/target/tcm_remote/Makefile create mode 100644 drivers/target/tcm_remote/tcm_remote.c create mode 100644 drivers/target/tcm_remote/tcm_remote.h
drivers/target/Kconfig +1 -0
@@ -53,5 +53,6 @@ source "drivers/target/loopback/Kconfig" source "drivers/target/tcm_fc/Kconfig" source "drivers/target/iscsi/Kconfig" source "drivers/target/sbp/Kconfig" +source "drivers/target/tcm_remote/Kconfig" endif
drivers/target/Makefile +1 -0
@@ -30,5 +30,6 @@ obj-$(CONFIG_LOOPBACK_TARGET) += loopback/ obj-$(CONFIG_TCM_FC) += tcm_fc/ obj-$(CONFIG_ISCSI_TARGET) += iscsi/ obj-$(CONFIG_SBP_TARGET) += sbp/ +obj-$(CONFIG_REMOTE_TARGET) += tcm_remote/ obj-$(CONFIG_DLM_CKV) += dlm_ckv.o
drivers/target/tcm_remote/Kconfig new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0-only +config REMOTE_TARGET + tristate "TCM Virtual Remote target" + depends on SCSI + help + Say Y here to enable the TCM Virtual Remote fabric + That fabric is a dummy fabric to tell TCM about configuration + of TPG/ACL/LUN on peer nodes in a cluster.
drivers/target/tcm_remote/Makefile new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_REMOTE_TARGET) += tcm_remote.o
drivers/target/tcm_remote/tcm_remote.c new file +405 -0
405 lines not on this page · raw message
drivers/target/tcm_remote/tcm_remote.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/types.h> +#include <linux/device.h> + +#define TCM_REMOTE_VERSION "v0.1" +#define TL_WWN_ADDR_LEN 256 +#define TL_TPGS_PER_HBA 32 + + +#define TCM_TRANSPORT_ONLINE 0 +#define TCM_TRANSPORT_OFFLINE 1 + +struct tcm_remote_tpg { +unsigned short remote_tpgt; +enum target_prot_type remote_fabric_prot_type; +atomic_t remote_tpg_port_count; +struct se_portal_group remote_se_tpg; +struct tcm_remote_hba *remote_hba; +}; + +struct tcm_remote_hba { +u8 remote_proto_id; +unsigned char remote_wwn_address[TL_WWN_ADDR_LEN ]; +struct se_hba_s *se_hba; +struct se_lun *remote_hba_lun; +struct se_port *remote_hba_lun_sep; +struct tcm_remote_tpg remote_hba_tpgs[TL_TPGS_PER_HBA ]; +struct se_wwn remote_hba_wwn; +};
Signature · 1 line
2.25.1
RFC PATCH 36/48 target: cluster: introduce cluster ops
In reply to Dmitry BogdanovIntroduce cluster ops common for all backend devices. Those ops will implement cluster level locks, store clusterwide data (like persistent reservations) and provide clusterwide notifications. That ops could be registered dynamically. Ops have own configfs folder in .../target/cluster/<cluster_impl_name>
Create default non-cluster ops implementation.
Use cluster ops for CAW lock.
--- drivers/target/target_core_configfs.c | 63 +++++++++++- drivers/target/target_core_device.c | 139 +++++++++++++++++++++++++- drivers/target/target_core_internal.h | 8 ++ drivers/target/target_core_sbc.c | 12 +-- include/target/target_core_base.h | 18 ++++ 5 files changed, 232 insertions(+), 8 deletions(-)
drivers/target/target_core_configfs.c +62 -1
@@ -69,6 +69,7 @@ static LIST_HEAD(g_tf_list);static DEFINE_MUTEX (g_tf_lock);static struct config_group target_core_hbagroup; +static struct config_group *target_core_cluster_group;static struct config_group alua_group;static struct config_group alua_lu_gps_group;
@@ -1193,6 +1194,32 @@ static ssize_t pgr_support_store(struct config_item *item,return count; } +static ssize_t cluster_impl_show (struct config_item *item,char *page) +{ +return snprintf (page,PAGE_SIZE ,"%s\n" , +to_attrib (item)->da_dev->cl_ops->name); +} + +static ssize_t cluster_impl_store (struct config_item *item, +const char *page,size_t count) +{ +struct se_dev_attrib *da =to_attrib (item); +struct se_device *dev = da->da_dev; +char cluster_impl[16 ]; +int res; + +strscpy (cluster_impl, page,16 ); +if (cluster_impl[strlen (cluster_impl) -1 ] =='\n' ) + cluster_impl[strlen (cluster_impl) -1 ] =0 ; + + res =core_dev_set_cluster_mode (dev, cluster_impl); +if (res) +return res; + +return count; +} +CONFIGFS_ATTR(, cluster_impl); + CONFIGFS_ATTR(, emulate_model_alias); CONFIGFS_ATTR(, emulate_dpo); CONFIGFS_ATTR(, emulate_fua_write);
@@ -1268,6 +1295,7 @@ struct configfs_attribute *sbc_attrib_attrs[] = { &attr_max_write_same_len, &attr_alua_support, &attr_pgr_support, + &attr_cluster_impl,NULL , };EXPORT_SYMBOL (sbc_attrib_attrs);
@@ -3615,6 +3643,38 @@ static const struct config_item_type target_core_cit = {/* Stop functions for struct config_item_type target_core_hba_cit */ +static const struct config_item_type target_core_cluster_cit = { + .ct_owner =THIS_MODULE , +}; + +int target_core_register_cluster_configfs (struct target_cluster_impl *tci) +{ +int ret =0 ; + +if (!target_core_cluster_group) { + target_core_cluster_group =configfs_register_default_group ( + &target_core_fabrics.su_group, +"cluster" , + &target_core_cluster_cit); +if (IS_ERR (target_core_cluster_group)) { + ret =PTR_ERR (target_core_cluster_group); +pr_err ("Error %d while registering cluster group\n" , ret); + goto err; + } + } + + ret =configfs_register_group (target_core_cluster_group, + tci->ops->cg_group); + +err: +return ret; +} + +void target_core_unregister_cluster_configfs (struct target_cluster_impl *tci) +{ +configfs_unregister_group (tci->ops->cg_group); +} +void target_setup_backend_cits (struct target_backend *tb) {target_core_setup_dev_cit (tb);
@@ -3749,7 +3809,8 @@ static void __exit target_core_exit_configfs(void)configfs_remove_default_groups (&alua_lu_gps_group);configfs_remove_default_groups (&alua_group);configfs_remove_default_groups (&target_core_hbagroup); - +if (target_core_cluster_group) +configfs_remove_default_groups (target_core_cluster_group);/* * We expect subsys->su_group.default_groups to be released * by configfs subsystem provider logic..
drivers/target/target_core_device.c +138 -1
175 lines not on this page · raw message
drivers/target/target_core_internal.h +8 -0
@@ -55,6 +55,11 @@ struct target_fabric_configfs {struct config_item_type tf_tpg_mappedlun_stat_cit; }; +struct target_cluster_impl { +const struct target_cluster_ops *ops; +struct list_head list; +}; +/* target_core_alua.c */ extern struct t10_alua_lu_gp *default_lu_gp;
@@ -83,6 +88,7 @@ void core_dev_free_initiator_node_lun_acl(struct se_portal_group *, struct se_lun_acl *lacl);int core_dev_setup_virtual_lun0 (void );void core_dev_release_virtual_lun0 (void ); +int core_dev_set_cluster_mode (struct se_device *dev,const char *cluster_impl);struct se_device *target_alloc_device (struct se_hba *hba,const char *name);int target_configure_device (struct se_device *dev);void target_free_device (struct se_device *);
@@ -93,6 +99,8 @@ void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);/* target_core_configfs.c */ extern struct configfs_item_operations target_core_dev_item_ops;void target_setup_backend_cits (struct target_backend *); +int target_core_register_cluster_configfs (struct target_cluster_impl *tci); +void target_core_unregister_cluster_configfs (struct target_cluster_impl *tci);/* target_core_fabric_configfs.c */ int target_fabric_setup_cits (struct target_fabric_configfs *);
drivers/target/target_core_sbc.c +6 -6
@@ -367,7 +367,7 @@ static sense_reason_t compare_and_write_post(struct se_cmd *cmd, bool success,* Unlock ->caw_sem originally obtained during sbc_compare_and_write() * before the original READ I/O submission. */ -up (&dev->caw_sem); + dev->cl_ops->caw_unlock (dev, cmd->cawlock);return ret; }
@@ -550,7 +550,7 @@ static sense_reason_t compare_and_write_callback(struct se_cmd *cmd, bool succes* In the MISCOMPARE or failure case, unlock ->caw_sem obtained in * sbc_compare_and_write() before the original READ I/O submission. */ -up (&dev->caw_sem); + dev->cl_ops->caw_unlock (dev, cmd->cawlock);sg_free_table (&write_tbl);return ret; }
@@ -561,13 +561,13 @@ sbc_compare_and_write(struct se_cmd *cmd)struct sbc_ops *ops = cmd->protocol_data;struct se_device *dev = cmd->se_dev;sense_reason_t ret; -int rc; +/* * Submit the READ first for COMPARE_AND_WRITE to perform the * comparision using SGLs at cmd->t_bidi_data_sg.. */ - rc =down_interruptible (&dev->caw_sem); -if (rc !=0 ) { + cmd->cawlock = dev->cl_ops->caw_lock (dev, cmd->t_task_lba); +if (!cmd->cawlock) { cmd->transport_complete_callback =NULL ;return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; }
@@ -582,7 +582,7 @@ sbc_compare_and_write(struct se_cmd *cmd) DMA_FROM_DEVICE);if (ret) { cmd->transport_complete_callback =NULL ; -up (&dev->caw_sem); + dev->cl_ops->caw_unlock (dev, cmd->cawlock);return ret; }/*
include/target/target_core_base.h +18 -0
@@ -546,6 +546,7 @@ struct se_cmd {* initialized on. Drivers can override. */ int cpuid; +void *cawlock; };struct se_ua {
@@ -777,6 +778,17 @@ struct se_device_queue {struct se_cmd_queue sq; }; +struct target_cluster_ops { +char name[16 ]; +struct module *owner; +struct config_group *cg_group; + +int (*init)(struct se_device *dev); +int (*cleanup)(struct se_device *dev); +void *(*caw_lock)(struct se_device *dev,u64 lba); +int (*caw_unlock)(struct se_device *dev,void *lock); +}; +struct se_device {/* Used for SAM Task Attribute ordering */ u32 dev_cur_ordered_id;
@@ -847,6 +859,9 @@ struct se_device {unsigned char udev_path[SE_UDEV_PATH_LEN ];/* Pointer to template of function pointers for transport */ const struct target_backend_ops *transport; +/* Pointer to template of function pointers for cluster mode */ +const struct target_cluster_ops *cl_ops; +void *cluster_data;struct se_lun xcopy_lun;/* Protection Information */ int prot_length;
@@ -981,4 +996,7 @@ static inline void target_free_tag(struct se_session *sess, struct se_cmd *cmd) sbitmap_queue_clear(&sess->sess_tag_pool, cmd->map_tag, cmd->map_cpu); } +int target_cluster_impl_register (const struct target_cluster_ops *ops); +void target_cluster_impl_unregister (const struct target_cluster_ops *ops); +#endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
RFC PATCH 37/48 target: cluster: introduce dlm cluster
In reply to Dmitry BogdanovIntroduce DLM Cluster implementation of cluster ops of backstore devices.
Use one DLM lockspace per device. The device is idenitfied by concatenation of T10 model and T10 unit_serial.
Use DLM CKV lock for Compare And Write in cluster mode. One lock for one LBA. That allows a parallell execution of CAW commands.
User should set configfs/target/cluster/dlm/cluster_name identical to configfs/dlm/cluster/cluster_name.
--- drivers/target/Makefile | 1 + drivers/target/target_cluster_dlm.c | 175 ++++++++++++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 drivers/target/target_cluster_dlm.c
drivers/target/Makefile +1 -0
@@ -33,3 +33,4 @@ obj-$(CONFIG_SBP_TARGET) += sbp/ obj-$(CONFIG_REMOTE_TARGET) += tcm_remote/ obj-$(CONFIG_DLM_CKV) += dlm_ckv.o +obj-$(CONFIG_DLM_CKV) += target_cluster_dlm.o
drivers/target/target_cluster_dlm.c new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH 38/48 target: cluster: store PR data in DLM cluster
In reply to Dmitry BogdanovFollow the sequence bellow to sync all PR data throughout nodes in cluster:
pr_lock() Handle PR OUT locally if (success) pr_sync() pr_unlock()
where pr_sync() is a synchronical function.
That will garantee an atomic update of PR data in a cluster.
--- drivers/target/target_cluster_dlm.c | 247 +++++++++++++++++++++++- drivers/target/target_core_device.c | 8 + drivers/target/target_core_fabric_lib.c | 1 + drivers/target/target_core_internal.h | 2 - drivers/target/target_core_pr.c | 7 + include/target/target_core_base.h | 6 + 6 files changed, 268 insertions(+), 3 deletions(-)
drivers/target/target_cluster_dlm.c +246 -1
@@ -5,12 +5,21 @@#include <linux/module.h> #include <linux/configfs.h> #include <target/target_core_base.h> +#include <target/target_core_fabric.h> +#include <target/target_core_backend.h> +#include "target_core_pr.h" #include "dlm_ckv.h" struct target_cluster_data {struct dlm_ckv_bucket *bucket;u32 local_nodeid; +struct mutex pr_lock_mutex; +struct dlm_ckv_lock *pr_lock; +struct dlm_ckv_kv *pr_data; +int reserved_node_id; +struct dlm_ckv_kv **pr_reg_kv; +size_t pr_reg_kv_len; };struct target_lksb {
@@ -60,6 +69,44 @@ static const struct config_item_type ci_cluster_config = { };static struct target_cluster_config cluster_cfg; +struct pr_lvb { +u32 nr_registrants; +u32 pr_generation; +u8 version; +u8 pr_is_set; +u8 pr_type; +u8 pr_scope; +u8 pr_aptpl; +u8 reserved[3 ]; +u32 reserved_by_nodeid; +}; + +struct pr_reg_lvb { +u64 key; +u16 rtpi; +u8 version; +u8 is_holder; +u8 is_all_tg_pt; +u8 tid[PR_REG_TID_LEN ]; +} __packed ;/* to fit in 256 bytes */ + +struct async_group { +refcount_t pending; +atomic_t status; +struct completion compl; +}; + +static int pr_reg_realloc (struct target_cluster_data *cluster_data, +size_t nr_registrants); + +static void group_compl_cb (void *arg,int res) +{ +struct async_group *grp = arg; + +atomic_cmpxchg (&grp->status,0 , res); +if (refcount_dec_and_test (&grp->pending)) +complete (&grp->compl); +}static int target_init_dlm (struct se_device *dev) {
@@ -68,6 +115,7 @@ static int target_init_dlm(struct se_device *dev)int err =0 ;BUG_ON (dev->cluster_data); +BUILD_BUG_ON (sizeof (struct pr_reg_lvb ) >DLM_CKV_VALUE_MAX_SIZE );snprintf (ls_name,sizeof (ls_name),"%s%s" , dev->t10_wwn.model, dev->t10_wwn.unit_serial);
@@ -85,16 +133,42 @@ static int target_init_dlm(struct se_device *dev) dev->cluster_data =NULL ;return err; } + +mutex_init (&cluster_data->pr_lock_mutex); + cluster_data->pr_lock =dlm_ckv_create_lock (cluster_data->bucket,"pr_lock" ); +if (!cluster_data->pr_lock) + goto fail; + + cluster_data->pr_data =dlm_ckv_create_kv (cluster_data->bucket,"pr_data" ); +if (!cluster_data->pr_data) + goto fail; + dev->cluster_data = cluster_data;return err; +fail: +if (cluster_data->pr_lock) +dlm_ckv_free_lock (cluster_data->pr_lock); +if (cluster_data->pr_data) +dlm_ckv_free_kv (cluster_data->pr_data); +dlm_ckv_close_bucket (cluster_data->bucket); +kfree (cluster_data); +return -EIO ; }static int target_cleanup_dlm (struct se_device *dev) {struct target_cluster_data *cluster_data = dev->cluster_data;int res; +int i; + +for (i =0 ; i < cluster_data->pr_reg_kv_len; i++) +if (cluster_data->pr_reg_kv[i]) +dlm_ckv_free_kv (cluster_data->pr_reg_kv[i]); +kfree (cluster_data->pr_reg_kv); +dlm_ckv_free_lock (cluster_data->pr_lock); +dlm_ckv_free_kv (cluster_data->pr_data); res =dlm_ckv_close_bucket (cluster_data->bucket);if (res) pr_err("TARGET_CORE[%d]: closing bucket failed: %d\n",
@@ -142,7 +216,175 @@ static int target_caw_unlock_dlm(struct se_device *dev, void *lock)return res; } -struct target_cluster_ops dlm_cluster_ops = { +static int target_pr_lock_dlm (struct se_device *dev) +{ +struct target_cluster_data *cluster_data = dev->cluster_data; +int res = -ENOMEM ; + +mutex_lock (&cluster_data->pr_lock_mutex); + res =dlm_ckv_lock_get (cluster_data->pr_lock); +if (res) { +pr_warn ("target_dlm_lock_wait failed with %d\n" , res); +mutex_unlock (&cluster_data->pr_lock_mutex); + } + +return res; +} + +static int target_pr_unlock_dlm (struct se_device *dev) +{ +struct target_cluster_data *cluster_data = dev->cluster_data; +int res; + + res =dlm_ckv_lock_release (cluster_data->pr_lock); +mutex_unlock (&cluster_data->pr_lock_mutex); + +return res; +} + +static int target_get_nr_registrants (struct se_device *dev) +{ +struct t10_pr_registration *pr_reg; +int count =0 ; + +list_for_each_entry (pr_reg, + &dev->t10_pr.registration_list, pr_reg_list) + count++; + +return count; +} + +static int pr_reg_realloc (struct target_cluster_data *cluster_data, +size_t nr_registrants) +{ +char key_name[16 ]; +void *p; +int res =0 ; +int i; + +if (nr_registrants > cluster_data->pr_reg_kv_len) { + p =krealloc (cluster_data->pr_reg_kv, + nr_registrants *sizeof (struct dlm_ckv_kv *), +GFP_KERNEL | __GFP_ZERO); +if (!p) { + res = -ENOMEM ; + goto done; + } + cluster_data->pr_reg_kv = p; +for (i = cluster_data->pr_reg_kv_len; + i < nr_registrants; i++) { +snprintf (key_name,sizeof (key_name),"pr_reg.%d" , i); +if (!cluster_data->pr_reg_kv[i]) + cluster_data->pr_reg_kv[i] =dlm_ckv_create_kv ( + cluster_data->bucket, + key_name); +if (!cluster_data->pr_reg_kv[i]) { + res = -ENOMEM ; + goto fail_create; + } + } + cluster_data->pr_reg_kv_len = nr_registrants; + } +done: +return res; +fail_create: +/* free newly created objects */ +for (i = cluster_data->pr_reg_kv_len; + i < nr_registrants; i++) { +if (!cluster_data->pr_reg_kv[i]) +dlm_ckv_free_kv (cluster_data->pr_reg_kv[i]); + } +return res; +} + +static int target_pr_sync_dlm (struct se_device *dev) +{ +struct target_cluster_data *cluster_data = dev->cluster_data; +struct t10_pr_registration *pr_reg; +struct pr_reg_lvb pr_reg_data; +struct async_group grp; +struct pr_lvb pr_data; +int i =0 ; +int res; + +spin_lock (&dev->t10_pr.registration_lock); + + pr_data.version =1 ; + pr_data.pr_generation =atomic_read (&dev->t10_pr.pr_generation); + pr_data.nr_registrants =target_get_nr_registrants (dev); + pr_data.pr_is_set = !!dev->dev_pr_res_holder; + pr_data.pr_aptpl = dev->t10_pr.pr_aptpl_active; +if (dev->dev_pr_res_holder) { + pr_data.pr_type = dev->dev_pr_res_holder->pr_res_type; + pr_data.pr_scope = dev->dev_pr_res_holder->pr_res_scope; + } + pr_data.reserved_by_nodeid = cluster_data->reserved_node_id; + +spin_unlock (&dev->t10_pr.registration_lock); + + res =dlm_ckv_set (cluster_data->pr_data, (char *)&pr_data, +sizeof (pr_data)); +if (res) + goto done; + + + res =pr_reg_realloc (cluster_data, pr_data.nr_registrants); +if (res) + goto done; + +if (pr_data.nr_registrants ==0 ) + goto skip_pr_reg; + +refcount_set (&grp.pending,1 );/* 1 for a loop */ +atomic_set (&grp.status,0 ); +init_completion (&grp.compl); + +list_for_each_entry (pr_reg, + &dev->t10_pr.registration_list, pr_reg_list) { + + pr_reg_data.key = pr_reg->pr_res_key; + pr_reg_data.rtpi = pr_reg->tg_pt_sep_rtpi; + pr_reg_data.version =1 ; + pr_reg_data.is_holder = pr_reg->pr_res_holder; + pr_reg_data.is_all_tg_pt = !!pr_reg->pr_reg_all_tg_pt; +target_get_pr_transport_id (pr_reg, pr_reg_data.tid); + +refcount_inc (&grp.pending); + res =dlm_ckv_set_async (cluster_data->pr_reg_kv[i], + (char *)&pr_reg_data, +sizeof (pr_reg_data), + group_compl_cb, + &grp); +if (res) { +pr_err ("TARGET_CORE[%d]: set pr_reg_%d failed: %d\n" , + dev->dev_index, i, res); +refcount_dec (&grp.pending); +break ; + } + i++; + } +group_compl_cb (&grp,0 ); + res =wait_for_completion_timeout (&grp.compl,60 *HZ ); +if (!res) { +pr_err ("TARGET_CORE[%d]: timeout of waiting for dlm_ckv_set_async\n" , + dev->dev_index); + goto done; + } + res =atomic_read (&grp.status); +if (res) { +pr_err ("TARGET_CORE[%d]: fail of group for dlm_ckv_set_async %d\n" , + dev->dev_index, res); + goto done; + } + +skip_pr_reg: + +done: + +return res; +} + +const struct target_cluster_ops dlm_cluster_ops = { .name ="dlm" , .owner =THIS_MODULE , .cg_group = &cluster_cfg.cg_group,
@@ -151,6 +393,9 @@ struct target_cluster_ops dlm_cluster_ops = { .cleanup = target_cleanup_dlm, .caw_lock = target_caw_lock_dlm, .caw_unlock = target_caw_unlock_dlm, + .pr_lock = target_pr_lock_dlm, + .pr_unlock = target_pr_unlock_dlm, + .pr_sync = target_pr_sync_dlm, };static int __init target_cluster_dlm_module_init (void )
drivers/target/target_core_device.c +8 -0
@@ -782,6 +782,11 @@ static int target_caw_unlock_nodlm(struct se_device *dev, void *lock)return 0 ; } +int target_dummy_nodlm (struct se_device *dev) +{ +return 0 ; +} +const struct target_cluster_ops nodlm_cluster_ops = { .name ="single" , .owner =NULL ,
@@ -789,6 +794,9 @@ const struct target_cluster_ops nodlm_cluster_ops = { .cleanup = target_cleanup_nodlm, .caw_lock = target_caw_lock_nodlm, .caw_unlock = target_caw_unlock_nodlm, + .pr_lock = target_dummy_nodlm, + .pr_unlock = target_dummy_nodlm, + .pr_sync = target_dummy_nodlm, };struct se_device *target_alloc_device (struct se_hba *hba,const char *name)
drivers/target/target_core_fabric_lib.c +1 -0
@@ -382,6 +382,7 @@ int target_get_pr_transport_id(return len; } +EXPORT_SYMBOL (target_get_pr_transport_id); int target_cmp_pr_transport_id( struct t10_pr_registration *pr_reg,
drivers/target/target_core_internal.h +0 -2
@@ -107,8 +107,6 @@ int target_fabric_setup_cits(struct target_fabric_configfs *);/* target_core_fabric_lib.c */ int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg); -int target_get_pr_transport_id (struct t10_pr_registration *pr_reg, -unsigned char *buf);int target_cmp_pr_transport_id (struct t10_pr_registration *pr_reg,unsigned char *tid); int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
drivers/target/target_core_pr.c +7 -0
@@ -3560,6 +3560,8 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)return TCM_PARAMETER_LIST_LENGTH_ERROR ; } +if (dev->cl_ops->pr_lock (dev)) +return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ;/* * (core_scsi3_emulate_pro_* function parameters * are defined by spc4r17 Table 174:
@@ -3598,9 +3600,14 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)default :pr_err ("Unknown PERSISTENT_RESERVE_OUT service" " action: 0x%02x\n" , sa); + dev->cl_ops->pr_unlock (dev);return TCM_INVALID_CDB_FIELD ; } +if (!ret) + dev->cl_ops->pr_sync (dev); + dev->cl_ops->pr_unlock (dev); +if (!ret)target_complete_cmd (cmd,SAM_STAT_GOOD );return ret;
include/target/target_core_base.h +6 -0
@@ -787,6 +787,9 @@ struct target_cluster_ops {int (*cleanup)(struct se_device *dev);void *(*caw_lock)(struct se_device *dev,u64 lba);int (*caw_unlock)(struct se_device *dev,void *lock); +int (*pr_lock)(struct se_device *dev); +int (*pr_unlock)(struct se_device *dev); +int (*pr_sync)(struct se_device *dev); };struct se_device {
@@ -999,4 +1002,7 @@ static inline void target_free_tag(struct se_session *sess, struct se_cmd *cmd)int target_cluster_impl_register (const struct target_cluster_ops *ops);void target_cluster_impl_unregister (const struct target_cluster_ops *ops); +int target_get_pr_transport_id (struct t10_pr_registration *pr_reg, +unsigned char *buf); +#endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
RFC PATCH 39/48 target: cluster: read PR data from cluster
In reply to Dmitry BogdanovNotify other nodes in cluster that PR data updated.
On notification do the following under PR cluster lock to read PR data:
Update an existing registrants.
Remove an outdated registrants (absent in cluster data). Create new registrants.
Update Reservation status.
--- drivers/target/target_cluster_dlm.c | 304 ++++++++++++++++++++++++ drivers/target/target_core_device.c | 1 + drivers/target/target_core_fabric_lib.c | 3 +- drivers/target/target_core_internal.h | 5 - drivers/target/target_core_pr.c | 8 +- drivers/target/target_core_pr.h | 1 + include/target/target_core_base.h | 5 + 7 files changed, 320 insertions(+), 7 deletions(-)
drivers/target/target_cluster_dlm.c +304 -0
@@ -17,6 +17,7 @@ struct target_cluster_data {struct mutex pr_lock_mutex;struct dlm_ckv_lock *pr_lock;struct dlm_ckv_kv *pr_data; +struct dlm_ckv_notify *pr_sync_notify;int reserved_node_id;struct dlm_ckv_kv **pr_reg_kv;size_t pr_reg_kv_len;
@@ -96,6 +97,7 @@ struct async_group {struct completion compl; }; +static void target_pr_sync_cb (void *arg);static int pr_reg_realloc (struct target_cluster_data *cluster_data,size_t nr_registrants);
@@ -143,10 +145,17 @@ static int target_init_dlm(struct se_device *dev)if (!cluster_data->pr_data) goto fail; + cluster_data->pr_sync_notify =dlm_ckv_create_notification ( + cluster_data->bucket,"pr_sync" , target_pr_sync_cb); +if (!cluster_data->pr_sync_notify) + goto fail; + dev->cluster_data = cluster_data;return err; fail: +if (cluster_data->pr_sync_notify) +dlm_ckv_free_notification (cluster_data->pr_sync_notify);if (cluster_data->pr_lock)dlm_ckv_free_lock (cluster_data->pr_lock);if (cluster_data->pr_data)
@@ -167,6 +176,7 @@ static int target_cleanup_dlm(struct se_device *dev)dlm_ckv_free_kv (cluster_data->pr_reg_kv[i]);kfree (cluster_data->pr_reg_kv); +dlm_ckv_free_notification (cluster_data->pr_sync_notify);dlm_ckv_free_lock (cluster_data->pr_lock);dlm_ckv_free_kv (cluster_data->pr_data); res =dlm_ckv_close_bucket (cluster_data->bucket);
@@ -379,11 +389,305 @@ static int target_pr_sync_dlm(struct se_device *dev) skip_pr_reg: +/* request to update PR data on other nodes in cluster */ +dlm_ckv_notify (cluster_data->pr_sync_notify); + done:return res; } +static int target_update_pr_reg (struct se_device *dev, +struct t10_pr_registration *pr_reg, +const struct pr_lvb *pr_data, +const struct pr_reg_lvb *pr_reg_data, +struct t10_pr_registration **pr_reg_res_holder) +{ +/* update existing registrant */ + pr_reg->pr_res_key = pr_reg_data->key; + pr_reg->pr_reg_all_tg_pt = pr_reg_data->is_all_tg_pt; + +if (pr_reg_data->is_holder) + *pr_reg_res_holder = pr_reg; + +return 0 ; +} + +static struct t10_pr_registration * +target_create_pr_reg (struct se_device *dev, +struct pr_lvb *pr_data, +struct pr_reg_lvb *pr_reg_data) +{ +struct t10_pr_registration *pr_reg =NULL ; +struct se_lun *lun, *lun_tmp; +struct se_lun_acl *lacl_tmp; +struct se_node_acl *nacl_tmp; +struct se_dev_entry *deve; +char i_buf[PR_REG_ISID_ID_LEN ]; +u8 initiatorname[PR_REG_ISID_LEN ]; +char *isid =NULL ; + +target_parse_pr_out_transport_id (pr_reg_data->tid, initiatorname, &isid); + +spin_lock (&dev->se_port_lock); +list_for_each_entry_safe (lun, lun_tmp, &dev->dev_sep_list, lun_dev_link) { +if (!percpu_ref_tryget_live (&lun->lun_ref)) +continue ; + +if (lun->lun_tpg->tpg_rtpi != pr_reg_data->rtpi) { +percpu_ref_put (&lun->lun_ref); +continue ; + } +spin_unlock (&dev->se_port_lock); + +spin_lock (&lun->lun_deve_lock); +list_for_each_entry (deve, &lun->lun_deve_list, lun_link) { +if (!deve->se_lun_acl) +continue ; + + lacl_tmp = deve->se_lun_acl; + nacl_tmp = lacl_tmp->se_lun_nacl; + +if (strcmp (nacl_tmp->initiatorname, initiatorname)) +continue ; + +kref_get (&deve->pr_kref); + pr_reg =__core_scsi3_do_alloc_registration ( + dev, nacl_tmp, + nacl_tmp->initiatorname, lun->unpacked_lun, + pr_reg_data->rtpi, + deve, lacl_tmp->mapped_lun, + isid, + pr_reg_data->key, + pr_reg_data->is_all_tg_pt, + pr_data->pr_aptpl); + +percpu_ref_put (&lun->lun_ref); +spin_unlock (&lun->lun_deve_lock); + goto out; + } +percpu_ref_put (&lun->lun_ref); +spin_unlock (&lun->lun_deve_lock); +spin_lock (&dev->se_port_lock); + } +spin_unlock (&dev->se_port_lock); +out: + +if (!pr_reg) { +/* target not yet configured, +* allocate registration like for APTPL case +*/ + pr_reg =__core_scsi3_do_alloc_registration ( + dev,NULL , initiatorname,0 , + pr_reg_data->rtpi, +NULL ,0 , + isid, + pr_reg_data->key, + pr_reg_data->is_all_tg_pt, + pr_data->pr_aptpl); + } + +if (pr_reg) { +spin_lock (&dev->t10_pr.registration_lock); +memcpy (pr_reg->pr_tid, pr_reg_data->tid,PR_REG_TID_LEN ); +list_add_tail (&pr_reg->pr_reg_list, + &dev->t10_pr.registration_list); + +/* +* Drop deve->pr_kref obtained in __core_scsi3_do_alloc_registration() +*/ +rcu_read_lock (); + deve = pr_reg->pr_reg_deve; +if (deve) { +set_bit (DEF_PR_REG_ACTIVE , &deve->deve_flags); +kref_put (&deve->pr_kref, target_pr_kref_release); + pr_reg->pr_reg_deve =NULL ; + } +rcu_read_unlock (); + +memset (&i_buf[0 ],0 ,PR_REG_ISID_ID_LEN ); +core_pr_dump_initiator_port (pr_reg, i_buf, +PR_REG_ISID_ID_LEN ); +pr_debug ("SPC-3 PR Service Action: REGISTER Initiator Node: %s%s\n" , + initiatorname, i_buf); +pr_debug ("SPC-3 PR registration on Target Port: %d\n" , + pr_reg->tg_pt_sep_rtpi); +pr_debug ("SPC-3 PR for %s TCM Subsystem %s Object Target Port(s)\n" , + (pr_reg->pr_reg_all_tg_pt) ?"ALL" :"SINGLE" , + dev->transport->name); +pr_debug ("SPC-3 PR SA Res Key: 0x%016llx PRgeneration: 0x%08x APTPL: %d\n" , + pr_reg->pr_res_key, + pr_reg->pr_res_generation, + pr_reg->pr_reg_aptpl); +spin_unlock (&dev->t10_pr.registration_lock); +return pr_reg; + } + +pr_warn ("TARGET_CORE[%d]: PR data from cluster is invalid: could not find local nacl/deve for Initiator %s - RTPI %d\n" , + dev->dev_index, initiatorname, pr_reg_data->rtpi); + +return NULL ; +} + +static void target_pr_sync_cb (void *arg) +{ +struct se_device *dev = arg; +struct target_cluster_data *cluster_data = dev->cluster_data; +struct t10_pr_registration *pr_reg, *pr_reg_tmp; +struct t10_pr_registration *pr_reg_res_holder =NULL ; +struct t10_pr_registration *pr_prev_res_holder =NULL ; +struct pr_reg_lvb *pr_reg_data =NULL ; +LIST_HEAD (to_be_deleted_list); +struct async_group grp; +struct pr_lvb pr_data; +bool res_to_delete = false; +bool was_held; +u8 was_type; +u8 was_scope; +bool found; +int i =0 ; +int res; + + res =dlm_ckv_get (cluster_data->pr_data, (char *)&pr_data,sizeof (pr_data)); +if (res) + goto done; + +if (!pr_data.version) { +pr_info ("TARGET_CORE[%d]: PR data from cluster is invalid\n" , + dev->dev_index); + goto done; + } + + pr_reg_data =kzalloc (sizeof (struct pr_reg_lvb ) * pr_data.nr_registrants, +GFP_KERNEL ); +if (!pr_reg_data) { + res = -ENOMEM ; + goto done; + } + + res =pr_reg_realloc (cluster_data, pr_data.nr_registrants); +if (res) + goto done; + +if (pr_data.nr_registrants ==0 ) + goto skip_pr_reg; + +refcount_set (&grp.pending,1 );/* 1 for a loop */ +atomic_set (&grp.status,0 ); +init_completion (&grp.compl); + +for (i =0 ; i < pr_data.nr_registrants; ++i) { +refcount_inc (&grp.pending); + res =dlm_ckv_get_async (cluster_data->pr_reg_kv[i], (char *)(pr_reg_data + i), +sizeof (struct pr_reg_lvb ), group_compl_cb, &grp); +if (res) { +refcount_dec (&grp.pending); +break ; + } + } +group_compl_cb (&grp,0 ); + res =wait_for_completion_timeout (&grp.compl,60 *HZ ); +if (!res) { +pr_err ("TARGET_CORE[%d]: timeout of waiting for dlm_ckv_get_async\n" , + dev->dev_index); + goto done; + } + res =atomic_read (&grp.status); +if (res) { +pr_err ("TARGET_CORE[%d]: fail of group for dlm_ckv_get_async %d\n" , + dev->dev_index, res); + goto done; + } + +skip_pr_reg: +/* +* Update existing registrations +*/ +spin_lock (&dev->t10_pr.registration_lock); + + was_held = !!dev->dev_pr_res_holder; + pr_prev_res_holder = dev->dev_pr_res_holder; +if (was_held) { + was_scope = pr_prev_res_holder->pr_res_scope; + was_type = pr_prev_res_holder->pr_res_type; + } + +list_for_each_entry_safe (pr_reg, pr_reg_tmp, &dev->t10_pr.registration_list, + pr_reg_list) { + found = false; + +for (i =0 ; i < pr_data.nr_registrants; ++i) { +if (!pr_reg_data[i].version) +continue ; + +if (pr_reg->tg_pt_sep_rtpi == pr_reg_data[i].rtpi && + !target_cmp_pr_transport_id (pr_reg, pr_reg_data[i].tid)) { + found = true; +/* mark existing registrants */ + pr_reg_data[i].version =0 ; +target_update_pr_reg (dev, pr_reg, &pr_data, + &pr_reg_data[i], + &pr_reg_res_holder); +break ; + } + } + +if (!found) +list_move_tail (&pr_reg->pr_reg_list, &to_be_deleted_list); + } + +/* deregister obsolete entries */ +list_for_each_entry_safe (pr_reg, pr_reg_tmp, &to_be_deleted_list, + pr_reg_list) { +if (dev->dev_pr_res_holder != pr_reg) +__core_scsi3_free_registration (dev, pr_reg,NULL ,0 ); +else + res_to_delete = true; + } +spin_unlock (&dev->t10_pr.registration_lock); + +/* register new entries */ +for (i =0 ; i < pr_data.nr_registrants; ++i) { +/* skip existing registrants */ +if (!pr_reg_data[i].version) +continue ; + + pr_reg =target_create_pr_reg (dev, &pr_data, &pr_reg_data[i]); +if (!pr_reg) +pr_err ("TARGET_CORE[%d]: can not create new registration\n" , + dev->dev_index); +else if (pr_reg_data[i].is_holder) + pr_reg_res_holder = pr_reg; + } + +/* update general data */ +atomic_set (&dev->t10_pr.pr_generation, pr_data.pr_generation); + dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl; + +/* update reservation */ +spin_lock (&dev->dev_reservation_lock); +if (pr_prev_res_holder && + ((pr_reg_res_holder != pr_prev_res_holder) || + (was_type != pr_reg_res_holder->pr_res_type))) +__core_scsi3_complete_pro_release (dev, pr_prev_res_holder,0 ,0 ); + +if (pr_reg_res_holder) +__core_scsi3_set_reservation (dev, pr_reg_res_holder, + pr_data.pr_scope, pr_data.pr_type); +spin_unlock (&dev->dev_reservation_lock); + +if (res_to_delete) { +spin_lock (&dev->t10_pr.registration_lock); +__core_scsi3_free_registration (dev, pr_prev_res_holder,NULL ,0 ); +spin_unlock (&dev->t10_pr.registration_lock); + } + +core_scsi3_update_and_write_aptpl (dev, dev->t10_pr.pr_aptpl_active); + +done: +kfree (pr_reg_data); +} +const struct target_cluster_ops dlm_cluster_ops = { .name ="dlm" , .owner =THIS_MODULE ,
drivers/target/target_core_device.c +1 -0
@@ -355,6 +355,7 @@ void target_pr_kref_release(struct kref *kref) pr_kref);complete (&deve->pr_comp); } +EXPORT_SYMBOL (target_pr_kref_release);/* * Establish UA condition on SCSI device - all LUNs
drivers/target/target_core_fabric_lib.c +2 -1
@@ -396,7 +396,7 @@ int target_cmp_pr_transport_id(return memcmp (pr_reg->pr_tid, tid, len1); } - +EXPORT_SYMBOL (target_cmp_pr_transport_id); int target_gen_pr_transport_id( struct t10_pr_registration *pr_reg,
@@ -444,3 +444,4 @@ u32 target_parse_pr_out_transport_id( *port_nexus_ptr =NULL ;return 24 ; } +EXPORT_SYMBOL (target_parse_pr_out_transport_id);
drivers/target/target_core_internal.h +0 -5
@@ -65,7 +65,6 @@ extern struct t10_alua_lu_gp *default_lu_gp;/* target_core_device.c */ struct se_dev_entry *core_get_se_deve_from_rtpi (struct se_node_acl *,u16 ); -void target_pr_kref_release (struct kref *);void core_free_device_list_for_node (struct se_node_acl *,struct se_portal_group *);void core_update_device_list_access (u64 ,bool ,struct se_node_acl *);
@@ -107,14 +106,10 @@ int target_fabric_setup_cits(struct target_fabric_configfs *);/* target_core_fabric_lib.c */ int target_get_pr_transport_id_len (struct t10_pr_registration *pr_reg); -int target_cmp_pr_transport_id (struct t10_pr_registration *pr_reg, -unsigned char *tid);int target_gen_pr_transport_id (struct t10_pr_registration *pr_reg,int proto_id,const char *initiatorname,unsigned char *isid); -u32 target_parse_pr_out_transport_id ( -const char *buf,char *initiatorname,char **port_nexus_ptr);/* target_core_hba.c */ struct se_hba *core_alloc_hba (const char *,u32 ,u32 );
drivers/target/target_core_pr.c +7 -1
@@ -52,6 +52,7 @@ void core_pr_dump_initiator_port(snprintf (buf, size,",i,0x%s" , pr_reg->pr_reg_isid); } +EXPORT_SYMBOL (core_pr_dump_initiator_port);enum register_type {REGISTER ,
@@ -677,6 +678,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(return pr_reg; } +EXPORT_SYMBOL (__core_scsi3_do_alloc_registration);static int core_scsi3_lunacl_depend_item (struct se_dev_entry *);static void core_scsi3_lunacl_undepend_item (struct se_dev_entry *);
@@ -1296,6 +1298,7 @@ void __core_scsi3_free_registration(*/ list_add_tail (&pr_reg->pr_reg_abort_list, preempt_and_abort_list); } +EXPORT_SYMBOL (__core_scsi3_free_registration);void core_scsi3_free_all_registrations (struct se_device *dev)
@@ -1925,7 +1928,7 @@ static int __core_scsi3_write_aptpl_to_file(* Clear the APTPL metadata if APTPL has been disabled, otherwise * write out the updated metadata to struct file for this SCSI device. */ -static sense_reason_t core_scsi3_update_and_write_aptpl (struct se_device *dev,bool aptpl) +sense_reason_t core_scsi3_update_and_write_aptpl (struct se_device *dev,bool aptpl) {unsigned char *buf;int rc, len =PR_APTPL_BUF_LEN ;
@@ -1965,6 +1968,7 @@ static sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, bpr_debug ("SPC-3 PR: Set APTPL Bit Activated\n" );return 0 ; } +EXPORT_SYMBOL (core_scsi3_update_and_write_aptpl); static sense_reason_t core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
@@ -2214,6 +2218,7 @@ void __core_scsi3_set_reservation(struct se_device *dev, pr_reg->pr_iport, i_buf); } +EXPORT_SYMBOL (__core_scsi3_set_reservation);static sense_reason_t core_scsi3_pro_reserve (struct se_cmd *cmd,int type,int scope,u64 res_key)
@@ -2430,6 +2435,7 @@ void __core_scsi3_complete_pro_release(*/ pr_reg->pr_res_holder = pr_reg->pr_res_type = pr_reg->pr_res_scope =0 ; } +EXPORT_SYMBOL (__core_scsi3_complete_pro_release); static sense_reason_t core_scsi3_emulate_pro_release(struct se_cmd *cmd, int type, int scope,
drivers/target/target_core_pr.h +1 -0
@@ -68,6 +68,7 @@ extern int core_scsi3_alloc_aptpl_registration(extern int core_scsi3_check_aptpl_registration (struct se_device *,struct se_portal_group *,struct se_lun *,struct se_node_acl *,u64 ); +sense_reason_t core_scsi3_update_and_write_aptpl (struct se_device *dev,bool aptpl);extern void core_scsi3_free_all_registrations (struct se_device *);extern unsigned char *core_scsi3_pr_dump_type (int );
include/target/target_core_base.h +5 -0
@@ -1004,5 +1004,10 @@ void target_cluster_impl_unregister(const struct target_cluster_ops *ops);int target_get_pr_transport_id (struct t10_pr_registration *pr_reg,unsigned char *buf); +u32 target_parse_pr_out_transport_id ( +const char *buf,char *initiatorname,char **port_nexus_ptr); +int target_cmp_pr_transport_id (struct t10_pr_registration *pr_reg, +unsigned char *tid); +void target_pr_kref_release (struct kref *kref);#endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
RFC PATCH 40/48 target: cluster: sync PR for dynamic acls
In reply to Dmitry BogdanovDynamic acls are not present in remote target, so use NULL for nacl pointer for such registrants.
--- drivers/target/target_cluster_dlm.c | 10 ++++++++++ drivers/target/target_core_pr.c | 17 +++++++++++------ 2 files changed, 21 insertions(+), 6 deletions(-)
drivers/target/target_cluster_dlm.c +10 -0
@@ -466,6 +466,16 @@ target_create_pr_reg(struct se_device *dev,spin_unlock (&lun->lun_deve_lock); goto out; } +/* deve not found - that is a dynamic acl */ + pr_reg =__core_scsi3_do_alloc_registration ( + dev,NULL , initiatorname, lun->unpacked_lun, + pr_reg_data->rtpi, +NULL , lun->unpacked_lun, + isid, + pr_reg_data->key, + pr_reg_data->is_all_tg_pt, + pr_data->pr_aptpl); +percpu_ref_put (&lun->lun_ref);spin_unlock (&lun->lun_deve_lock);spin_lock (&dev->se_port_lock);
drivers/target/target_core_pr.c +11 -6
@@ -644,10 +644,13 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(* * Otherwise, locate se_dev_entry now and obtain a reference until * registration completes in __core_scsi3_add_registration(). +* +* NULL nacl means that this registration is for dynamic acl on a remote +* target - no need to find a deve. */ if (dest_deve) { pr_reg->pr_reg_deve = dest_deve; - }else { + }else if (nacl) {rcu_read_lock (); pr_reg->pr_reg_deve =target_nacl_find_deve (nacl, mapped_lun);if (!pr_reg->pr_reg_deve) {
@@ -1270,11 +1273,13 @@ void __core_scsi3_free_registration(cpu_relax (); } -rcu_read_lock (); - deve =target_nacl_find_deve (nacl, pr_reg->pr_res_mapped_lun); -if (deve) -clear_bit (DEF_PR_REG_ACTIVE , &deve->deve_flags); -rcu_read_unlock (); +if (nacl) { +rcu_read_lock (); + deve =target_nacl_find_deve (nacl, pr_reg->pr_res_mapped_lun); +if (deve) +clear_bit (DEF_PR_REG_ACTIVE , &deve->deve_flags); +rcu_read_unlock (); + }spin_lock (&pr_tmpl->registration_lock); pr_debug("SPC-3 PR Service Action: UNREGISTER Initiator Node: %s%s\n",
Signature · 1 line
2.25.1
RFC PATCH 41/48 target: cluster: sync-up PR data on cluster join
In reply to Dmitry BogdanovRead PR data from cluster on joining to the cluster. acl/lun/tpg are supposed to be configured already.
--- drivers/target/target_cluster_dlm.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/target/target_cluster_dlm.c +5 -0
@@ -152,6 +152,11 @@ static int target_init_dlm(struct se_device *dev) dev->cluster_data = cluster_data; +/* initial sync-up on joining the cluster */ +dlm_ckv_lock_get (cluster_data->pr_lock); +target_pr_sync_cb (dev); +dlm_ckv_lock_release (cluster_data->pr_lock); +return err; fail:if (cluster_data->pr_sync_notify)
Signature · 1 line
2.25.1
RFC PATCH 42/48 target: cluster: sync SPC-2 reservations
In reply to Dmitry BogdanovReservation status is stored in DLM_CKV as part of PR data. If a reservation holder node is left the cluster other nodes will clear reservation status locally.
--- drivers/target/dlm_ckv.c | 16 ++++++- drivers/target/dlm_ckv.h | 4 +- drivers/target/target_cluster_dlm.c | 59 +++++++++++++++++++++++++- drivers/target/target_core_device.c | 10 +++++ drivers/target/target_core_pr.c | 22 +++++++--- drivers/target/target_core_pr.h | 1 + drivers/target/target_core_tmr.c | 4 +- drivers/target/target_core_transport.c | 2 +- include/target/target_core_base.h | 1 + 9 files changed, 104 insertions(+), 15 deletions(-)
drivers/target/dlm_ckv.c +14 -2
@@ -49,6 +49,7 @@ struct dlm_ckv_bucket {size_t num_nodes;int nodeid[64 ];void *userarg; +dlm_ckv_nodeleft_cb nodeleft_cb;struct completion sync_compl;struct workqueue_struct *notify_wq; };
@@ -79,7 +80,12 @@ static void dlm_ckv_recover_prep(void *arg)static void dlm_ckv_recover_slot (void *arg,struct dlm_slot *slot) { -pr_info ("nodeid %d left the cluster\n" , slot->nodeid); +struct dlm_ckv_bucket *bucket = arg; + +pr_debug ("nodeid %d left the cluster\n" , slot->nodeid); + +if (bucket->nodeleft_cb) + bucket->nodeleft_cb (bucket->userarg, slot->nodeid); }/* dlm calls after recover_slot and after it completes lock recovery */
@@ -667,7 +673,8 @@ static void bucket_release(struct kref *ref) }struct dlm_ckv_bucket * -dlm_ckv_open_bucket (const char *name,const char *cluster_name,void *userarg) +dlm_ckv_open_bucket (const char *name,const char *cluster_name,void *userarg, +dlm_ckv_nodeleft_cb nodeleft_cb,int *local_nodeid) {struct dlm_ckv_bucket *bucket;int name_len =strlen (name);
@@ -677,6 +684,9 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)if (!name)return ERR_PTR (-EINVAL ); +if (!local_nodeid) +return ERR_PTR (-EINVAL ); +if (name_len >DLM_LOCKSPACE_LEN )return ERR_PTR (-EINVAL );
@@ -684,6 +694,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)kref_init (&bucket->refcount); bucket->userarg = userarg; + bucket->nodeleft_cb = nodeleft_cb;init_completion (&bucket->sync_compl); bucket->notify_wq =alloc_ordered_workqueue ("notify_wq-%s" ,0 , name);
@@ -708,6 +719,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg) goto fail_init; } + *local_nodeid = bucket->local_nodeid;return bucket; fail_init:
drivers/target/dlm_ckv.h +3 -1
@@ -14,7 +14,9 @@ typedef void (*dlm_ckv_nodeleft_cb)(void *arg, int nodeid);struct dlm_ckv_bucket *dlm_ckv_open_bucket (const char *name,const char *cluster_name, -void *userarg); +void *userarg, +dlm_ckv_nodeleft_cb nodeleft_cb, +int *local_nodeid);int dlm_ckv_close_bucket (struct dlm_ckv_bucket *bucket); struct dlm_ckv_lock *
drivers/target/target_cluster_dlm.c +58 -1
89 lines not on this page · raw message
drivers/target/target_core_device.c +10 -0
@@ -788,6 +788,15 @@ int target_dummy_nodlm(struct se_device *dev)return 0 ; } +static void target_reserve2_nodlm (struct se_device *dev,struct se_session *sess) +{ +if (sess) { +target_get_reservation (dev, sess); + }else { +target_release_reservation (dev); + } +} +const struct target_cluster_ops nodlm_cluster_ops = { .name ="single" , .owner =NULL ,
@@ -798,6 +807,7 @@ const struct target_cluster_ops nodlm_cluster_ops = { .pr_lock = target_dummy_nodlm, .pr_unlock = target_dummy_nodlm, .pr_sync = target_dummy_nodlm, + .reserve = target_reserve2_nodlm, };struct se_device *target_alloc_device (struct se_hba *hba,const char *name)
drivers/target/target_core_pr.c +15 -7
@@ -192,6 +192,7 @@ void target_release_reservation(struct se_device *dev) dev->dev_reservation_flags &= ~DRF_SPC2_RESERVATIONS_WITH_ISID ; } } +EXPORT_SYMBOL (target_release_reservation);sense_reason_t target_scsi2_reservation_release (struct se_cmd *cmd)
@@ -222,7 +223,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)if (dev->dev_res_bin_isid != sess->sess_bin_isid) goto out_unlock; -target_release_reservation (dev); + dev->cl_ops->reserve (dev,NULL ); tpg = sess->se_tpg; pr_debug("SCSI-2 Released reservation for %s LUN: %llu ->" " MAPPED LUN: %llu for %s\n",
@@ -237,6 +238,17 @@ target_scsi2_reservation_release(struct se_cmd *cmd)return 0 ; } +void target_get_reservation (struct se_device *dev,struct se_session *sess) +{ + dev->reservation_holder = sess; + dev->dev_reservation_flags |=DRF_SPC2_RESERVATIONS ; +if (sess->sess_bin_isid !=0 ) { + dev->dev_res_bin_isid = sess->sess_bin_isid; + dev->dev_reservation_flags |=DRF_SPC2_RESERVATIONS_WITH_ISID ; + } +} +EXPORT_SYMBOL (target_get_reservation); +sense_reason_t target_scsi2_reservation_reserve (struct se_cmd *cmd) {
@@ -284,12 +296,8 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd) goto out_unlock; } - dev->reservation_holder = sess; - dev->dev_reservation_flags |=DRF_SPC2_RESERVATIONS ; -if (sess->sess_bin_isid !=0 ) { - dev->dev_res_bin_isid = sess->sess_bin_isid; - dev->dev_reservation_flags |=DRF_SPC2_RESERVATIONS_WITH_ISID ; - } + dev->cl_ops->reserve (dev, sess); + pr_debug("SCSI-2 Reserved %s LUN: %llu -> MAPPED LUN: %llu" " for %s\n", tpg->se_tpg_tfo->fabric_name, cmd->se_lun->unpacked_lun, cmd->orig_fe_lun,
drivers/target/target_core_pr.h +1 -0
@@ -59,6 +59,7 @@ extern struct kmem_cache *t10_pr_reg_cache;extern void core_pr_dump_initiator_port (struct t10_pr_registration *,char *,u32 );extern void target_release_reservation (struct se_device *dev); +extern void target_get_reservation (struct se_device *dev,struct se_session *sess);extern sense_reason_t target_scsi2_reservation_release (struct se_cmd *);extern sense_reason_t target_scsi2_reservation_reserve (struct se_cmd *); extern int core_scsi3_alloc_aptpl_registration(
drivers/target/target_core_tmr.c +1 -3
@@ -406,9 +406,7 @@ int core_tmr_lun_reset(* LOGICAL UNIT RESET */ if (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) { -spin_lock (&dev->dev_reservation_lock); -target_release_reservation (dev); -spin_unlock (&dev->dev_reservation_lock); + dev->cl_ops->reserve (dev,NULL );pr_debug ("LUN_RESET: SCSI-2 Released reservation\n" ); }
drivers/target/target_core_transport.c +1 -1
@@ -612,7 +612,7 @@ static int target_release_res(struct se_device *dev, void *data)struct se_session *sess = data;if (dev->reservation_holder == sess) -target_release_reservation (dev); + dev->cl_ops->reserve (dev,NULL );return 0 ; }
include/target/target_core_base.h +1 -0
@@ -790,6 +790,7 @@ struct target_cluster_ops {int (*pr_lock)(struct se_device *dev);int (*pr_unlock)(struct se_device *dev);int (*pr_sync)(struct se_device *dev); +void (*reserve)(struct se_device *dev,struct se_session *sess); };struct se_device {
Signature · 1 line
2.25.1
RFC PATCH 43/48 target: cluster: allocate UAs on PR sync
In reply to Dmitry BogdanovEstablish Unit Attention condition on the peer nodes in the cluster on PR OUT commands according to SPC-4.
Pass PR OUT Service Action to pr_sync to allow peer nodes to know the reason of the update of PR state.
--- drivers/target/target_cluster_dlm.c | 115 +++++++++++++++++++++++++++- drivers/target/target_core_device.c | 7 +- drivers/target/target_core_pr.c | 2 +- drivers/target/target_core_ua.c | 1 + include/target/target_core_base.h | 2 +- 5 files changed, 121 insertions(+), 6 deletions(-)
drivers/target/target_cluster_dlm.c +112 -3
@@ -7,6 +7,7 @@#include <target/target_core_base.h> #include <target/target_core_fabric.h> #include <target/target_core_backend.h> +#include "target_core_ua.h" #include "target_core_pr.h" #include "dlm_ckv.h"
@@ -78,7 +79,8 @@ struct pr_lvb {u8 pr_type;u8 pr_scope;u8 pr_aptpl; -u8 reserved[3 ]; +u8 pro_sa; +u8 reserved[2 ];u32 reserved_by_nodeid; };
@@ -314,7 +316,7 @@ static int pr_reg_realloc(struct target_cluster_data *cluster_data,return res; } -static int target_pr_sync_dlm (struct se_device *dev) +static int target_pr_sync_dlm (struct se_device *dev,u8 pro_sa) {struct target_cluster_data *cluster_data = dev->cluster_data;struct t10_pr_registration *pr_reg;
@@ -328,6 +330,7 @@ static int target_pr_sync_dlm(struct se_device *dev) pr_data.version =1 ; pr_data.pr_generation =atomic_read (&dev->t10_pr.pr_generation); + pr_data.pro_sa = pro_sa; pr_data.nr_registrants =target_get_nr_registrants (dev); pr_data.pr_is_set = !!dev->dev_pr_res_holder; pr_data.pr_aptpl = dev->t10_pr.pr_aptpl_active;
@@ -546,6 +549,19 @@ target_create_pr_reg(struct se_device *dev,return NULL ; } +static void target_allocate_pr_ua (struct se_device *dev,u8 asc) +{ +struct t10_pr_registration *pr_reg; + +list_for_each_entry (pr_reg, &dev->t10_pr.registration_list, pr_reg_list) { +target_ua_allocate_lun ( + pr_reg->pr_reg_nacl, + pr_reg->pr_res_mapped_lun, +0x2A , + asc); + } +} +static void target_pr_sync_cb (void *arg) {struct se_device *dev = arg;
@@ -558,6 +574,9 @@ static void target_pr_sync_cb(void *arg)struct async_group grp;struct pr_lvb pr_data;bool res_to_delete = false; +struct se_node_acl *pr_reg_nacl; +u64 pr_res_mapped_lun; +bool reg_deleted = false;bool was_held;u8 was_type;u8 was_scope;
@@ -656,10 +675,41 @@ static void target_pr_sync_cb(void *arg)/* deregister obsolete entries */ list_for_each_entry_safe (pr_reg, pr_reg_tmp, &to_be_deleted_list, pr_reg_list) { + reg_deleted = true; + pr_reg_nacl = pr_reg->pr_reg_nacl; + pr_res_mapped_lun = pr_reg->pr_res_mapped_lun; +if (dev->dev_pr_res_holder != pr_reg)__core_scsi3_free_registration (dev, pr_reg,NULL ,0 );else res_to_delete = true; + +switch (pr_data.pro_sa) { +case PRO_CLEAR : +/* +* establish a unit attention condition for the initiator +* port associated with every registered I_T nexus other +* than the I_T nexus on which the PERSISTENT RESERVE OUT +* command with CLEAR service action was received, with +* the additional sense code set to RESERVATIONS PREEMPTED +*/ +target_ua_allocate_lun (pr_reg_nacl, + pr_res_mapped_lun,0x2A , +ASCQ_2AH_RESERVATIONS_PREEMPTED ); +break ; +case PRO_PREEMPT_AND_ABORT : +case PRO_REGISTER_AND_IGNORE_EXISTING_KEY : +/* +* establish a unit attention condition for the initiator +* port associated with every I_T nexus that lost its +* persistent reservation and/or registration, with the +* additional sense code set to REGISTRATIONS PREEMPTED; +*/ +target_ua_allocate_lun (pr_reg_nacl, + pr_res_mapped_lun,0x2A , +ASCQ_2AH_REGISTRATIONS_PREEMPTED ); +break ; + } }spin_unlock (&dev->t10_pr.registration_lock);
@@ -677,6 +727,65 @@ static void target_pr_sync_cb(void *arg) pr_reg_res_holder = pr_reg; } +switch (pr_data.pro_sa) { +case PRO_REGISTER_AND_IGNORE_EXISTING_KEY : +case PRO_REGISTER : +/* +* If the I_T nexus is the reservation holder and the persistent +* reservation is of a type other than all registrants, then +* the device server shall also release the persistent reservation +* +* If the persistent reservation is a registrants only +* type, the device server shall establish a unit +* attention condition for the initiator port associated +* with every registered I_T nexus except for the I_T +* nexus on which the PERSISTENT RESERVE OUT command was +* received, with the additional sense code set to +* RESERVATIONS RELEASED. +*/ +if (reg_deleted && was_held && + (was_type ==PR_TYPE_WRITE_EXCLUSIVE_REGONLY || + was_type ==PR_TYPE_EXCLUSIVE_ACCESS_REGONLY )) { +target_allocate_pr_ua (dev, +ASCQ_2AH_RESERVATIONS_RELEASED ); + } +break ; +case PRO_RELEASE : +/* +* if the released persistent reservation is either a registrants +* only type or an all registrants type persistent reservation, +* then the device server shall establish a unit attention +* condition for the initiator port associated with every +* registered I_T nexus other than I_T nexus on which the +* PERSISTENT RESERVE OUT command with RELEASE service action was +* received, with the additional sense code set to RESERVATIONS RELEASED +*/ +if ((was_type ==PR_TYPE_WRITE_EXCLUSIVE_REGONLY ) || + (was_type ==PR_TYPE_EXCLUSIVE_ACCESS_REGONLY ) || + (was_type ==PR_TYPE_WRITE_EXCLUSIVE_ALLREG ) || + (was_type ==PR_TYPE_EXCLUSIVE_ACCESS_ALLREG )) { +target_allocate_pr_ua (dev, +ASCQ_2AH_RESERVATIONS_RELEASED ); + } +break ; +case PRO_PREEMPT : +case PRO_PREEMPT_AND_ABORT : +/* +* if the type or scope has changed, then for every I_T nexus +* whose reservation key was not removed, except for the I_T nexus +* on which the PERSISTENT RESERVE OUT command was received, the +* device server shall establish a unit attention condition for +* the initiator port associated with that I_T nexus, with the +* additional sense code set to RESERVATIONS RELEASED. I +*/ +if (pr_prev_res_holder && pr_reg_res_holder && + pr_prev_res_holder != pr_reg_res_holder && + (was_type != pr_reg_res_holder->pr_res_type || + was_scope != pr_reg_res_holder->pr_res_scope)) +target_allocate_pr_ua (dev,ASCQ_2AH_RESERVATIONS_RELEASED ); +break ; + } +/* update general data */ atomic_set (&dev->t10_pr.pr_generation, pr_data.pr_generation); dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
@@ -741,7 +850,7 @@ target_spc2_reserve(struct se_device *dev, struct se_session *sess) cluster_data->reserved_node_id =0 ; } -target_pr_sync_dlm (dev); +target_pr_sync_dlm (dev,-1 );target_pr_unlock_dlm (dev); }
drivers/target/target_core_device.c +6 -1
@@ -788,6 +788,11 @@ int target_dummy_nodlm(struct se_device *dev)return 0 ; } +int target_prsync_nodlm (struct se_device *dev,u8 pro_sa) +{ +return 0 ; +} +static void target_reserve2_nodlm (struct se_device *dev,struct se_session *sess) {if (sess) {
@@ -806,7 +811,7 @@ const struct target_cluster_ops nodlm_cluster_ops = { .caw_unlock = target_caw_unlock_nodlm, .pr_lock = target_dummy_nodlm, .pr_unlock = target_dummy_nodlm, - .pr_sync = target_dummy_nodlm, + .pr_sync = target_prsync_nodlm, .reserve = target_reserve2_nodlm, };
drivers/target/target_core_pr.c +1 -1
@@ -3624,7 +3624,7 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd) }if (!ret) - dev->cl_ops->pr_sync (dev); + dev->cl_ops->pr_sync (dev, sa); dev->cl_ops->pr_unlock (dev);if (!ret)
drivers/target/target_core_ua.c +1 -0
@@ -171,6 +171,7 @@ void target_ua_allocate_lun(struct se_node_acl *nacl,core_scsi3_ua_allocate (deve, asc, ascq);rcu_read_unlock (); } +EXPORT_SYMBOL (target_ua_allocate_lun);void core_scsi3_ua_release_all (struct se_dev_entry *deve)
include/target/target_core_base.h +1 -1
@@ -789,7 +789,7 @@ struct target_cluster_ops {int (*caw_unlock)(struct se_device *dev,void *lock);int (*pr_lock)(struct se_device *dev);int (*pr_unlock)(struct se_device *dev); -int (*pr_sync)(struct se_device *dev); +int (*pr_sync)(struct se_device *dev,u8 pro_sa);void (*reserve)(struct se_device *dev,struct se_session *sess); };
Signature · 1 line
2.25.1
RFC PATCH 44/48 target: cluster: support PR OUT preempt and abort
In reply to Dmitry BogdanovAbort commands in the preempted I_T nexuses on all nodes in cluster by PR OUT (PREEMPT AND ABORT) command.
--- drivers/target/target_cluster_dlm.c | 17 +++++++++++++++-- drivers/target/target_core_internal.h | 2 -- drivers/target/target_core_pr.c | 3 ++- drivers/target/target_core_pr.h | 3 +++ drivers/target/target_core_tmr.c | 1 + include/target/target_core_base.h | 3 +++ 6 files changed, 24 insertions(+), 5 deletions(-)
drivers/target/target_cluster_dlm.c +15 -2
@@ -571,6 +571,7 @@ static void target_pr_sync_cb(void *arg)struct t10_pr_registration *pr_prev_res_holder =NULL ;struct pr_reg_lvb *pr_reg_data =NULL ;LIST_HEAD (to_be_deleted_list); +LIST_HEAD (preempt_and_abort_list);struct async_group grp;struct pr_lvb pr_data;bool res_to_delete = false;
@@ -680,7 +681,9 @@ static void target_pr_sync_cb(void *arg) pr_res_mapped_lun = pr_reg->pr_res_mapped_lun;if (dev->dev_pr_res_holder != pr_reg) -__core_scsi3_free_registration (dev, pr_reg,NULL ,0 ); +__core_scsi3_free_registration (dev, pr_reg, + (pr_data.pro_sa ==PRO_PREEMPT_AND_ABORT ) ? + &preempt_and_abort_list :NULL ,0 );else res_to_delete = true;
@@ -825,10 +828,20 @@ static void target_pr_sync_cb(void *arg)if (res_to_delete) {spin_lock (&dev->t10_pr.registration_lock); -__core_scsi3_free_registration (dev, pr_prev_res_holder,NULL ,0 ); +__core_scsi3_free_registration (dev, pr_prev_res_holder, + (pr_data.pro_sa ==PRO_PREEMPT_AND_ABORT ) ? + &preempt_and_abort_list :NULL ,0 );spin_unlock (&dev->t10_pr.registration_lock); } +if (pr_data.pro_sa ==PRO_PREEMPT_AND_ABORT ) { +core_local_lun_reset (dev,NULL , &preempt_and_abort_list,NULL ); + +core_scsi3_release_preempt_and_abort ( + &preempt_and_abort_list, + pr_reg_res_holder); + } +core_scsi3_update_and_write_aptpl (dev, dev->t10_pr.pr_aptpl_active); done:
drivers/target/target_core_internal.h +0 -2
@@ -119,8 +119,6 @@ int core_delete_hba(struct se_hba *);void core_tmr_abort_task (struct se_device *,struct se_tmr_req *,struct se_session *);int core_tmr_lun_reset (struct se_device *dev,struct se_tmr_req *tmr); -void core_local_lun_reset (struct se_device *swc,struct se_tmr_req *tmr, -struct list_head *preempt_and_abort_list,struct se_cmd *prout_cmd);/* target_core_tpg.c */ extern struct spinlock g_tpg_lock;
drivers/target/target_core_pr.c +2 -1
@@ -2725,7 +2725,7 @@ static void __core_scsi3_complete_pro_preempt( preempt_and_abort_list); } -static void core_scsi3_release_preempt_and_abort ( +void core_scsi3_release_preempt_and_abort (struct list_head *preempt_and_abort_list,struct t10_pr_registration *pr_reg_holder) {
@@ -2747,6 +2747,7 @@ static void core_scsi3_release_preempt_and_abort(kmem_cache_free (t10_pr_reg_cache, pr_reg); } } +EXPORT_SYMBOL (core_scsi3_release_preempt_and_abort); static sense_reason_t core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
drivers/target/target_core_pr.h +3 -0
@@ -99,6 +99,9 @@ void __core_scsi3_free_registration(struct se_device *dev,struct t10_pr_registration *pr_reg, struct list_head *preempt_and_abort_list, int dec_holders); +void core_scsi3_release_preempt_and_abort ( +struct list_head *preempt_and_abort_list, +struct t10_pr_registration *pr_reg_holder);#endif /* TARGET_CORE_PR_H */
drivers/target/target_core_tmr.c +1 -0
@@ -394,6 +394,7 @@ void core_local_lun_reset( (preempt_and_abort_list) ?"Preempt" :"TMR" , dev->transport->name); } +EXPORT_SYMBOL (core_local_lun_reset); int core_tmr_lun_reset( struct se_device *dev,
include/target/target_core_base.h +3 -0
@@ -1010,5 +1010,8 @@ u32 target_parse_pr_out_transport_id(int target_cmp_pr_transport_id (struct t10_pr_registration *pr_reg,unsigned char *tid);void target_pr_kref_release (struct kref *kref); +void core_local_lun_reset (struct se_device *swc,struct se_tmr_req *tmr, +struct list_head *preempt_and_abort_list,struct se_cmd *prout_cmd); +#endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
RFC PATCH 45/48 target: cluster: add reset cluster function
In reply to Dmitry BogdanovIntroduce `reset' function pointer in target cluster ops. For no cluster case do just clear SCSI2 reservation status.
--- drivers/target/target_core_device.c | 9 +++++++++ drivers/target/target_core_tmr.c | 9 +-------- include/target/target_core_base.h | 1 + 3 files changed, 11 insertions(+), 8 deletions(-)
drivers/target/target_core_device.c +9 -0
@@ -802,6 +802,14 @@ static void target_reserve2_nodlm(struct se_device *dev, struct se_session *sess } } +static void target_reset_nodlm (struct se_device *dev) +{ +if (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) { +target_release_reservation (dev); +pr_debug ("LUN_RESET: SCSI-2 Released reservation\n" ); + } +} +const struct target_cluster_ops nodlm_cluster_ops = { .name ="single" , .owner =NULL ,
@@ -813,6 +821,7 @@ const struct target_cluster_ops nodlm_cluster_ops = { .pr_unlock = target_dummy_nodlm, .pr_sync = target_prsync_nodlm, .reserve = target_reserve2_nodlm, + .reset = target_reset_nodlm, };struct se_device *target_alloc_device (struct se_hba *hba,const char *name)
drivers/target/target_core_tmr.c +1 -8
@@ -402,14 +402,7 @@ int core_tmr_lun_reset( {core_local_lun_reset (dev, tmr,NULL ,NULL ); -/* -* Clear any legacy SPC-2 reservation when called during -* LOGICAL UNIT RESET -*/ -if (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) { - dev->cl_ops->reserve (dev,NULL ); -pr_debug ("LUN_RESET: SCSI-2 Released reservation\n" ); - } + dev->cl_ops->reset (dev);target_dev_ua_allocate (dev,0x29 ,ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED );
include/target/target_core_base.h +1 -0
@@ -791,6 +791,7 @@ struct target_cluster_ops {int (*pr_unlock)(struct se_device *dev);int (*pr_sync)(struct se_device *dev,u8 pro_sa);void (*reserve)(struct se_device *dev,struct se_session *sess); +void (*reset)(struct se_device *dev); };struct se_device {
Signature · 1 line
2.25.1
RFC PATCH 46/48 target: cluster: implement LUN reset in DLM cluster
In reply to Dmitry BogdanovSince LUN RESET clears SCSI2 Reservation (cluster wide data of persistence reservation status) make LUN RESET just one of reasons to update the Persistence Reservation status as well as Service Actions of PR OUT.
--- drivers/target/target_cluster_dlm.c | 31 ++++++++++++++++++++++++++- drivers/target/target_core_device.c | 1 + drivers/target/target_core_internal.h | 1 - include/target/target_core_base.h | 2 +- 4 files changed, 32 insertions(+), 3 deletions(-)
drivers/target/target_cluster_dlm.c +30 -1
@@ -99,6 +99,9 @@ struct async_group {struct completion compl; }; +#define PR_SYNC_REASON_RESERVE 10 +#define PR_SYNC_REASON_RESET 11 +static void target_pr_sync_cb (void *arg);static void target_nodeleft_cb (void *arg,int nodeid); static int pr_reg_realloc(struct target_cluster_data *cluster_data,
@@ -842,6 +845,15 @@ static void target_pr_sync_cb(void *arg) pr_reg_res_holder); } +if (pr_data.pro_sa ==PR_SYNC_REASON_RESET ) { +core_local_lun_reset (dev,NULL ,NULL ,NULL ); + +target_dev_ua_allocate (dev,0x29 , +ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED ); + +atomic_long_inc (&dev->num_resets); + } +core_scsi3_update_and_write_aptpl (dev, dev->t10_pr.pr_aptpl_active); done:
@@ -863,7 +875,23 @@ target_spc2_reserve(struct se_device *dev, struct se_session *sess) cluster_data->reserved_node_id =0 ; } -target_pr_sync_dlm (dev,-1 ); +target_pr_sync_dlm (dev,PR_SYNC_REASON_RESERVE ); +target_pr_unlock_dlm (dev); +} + +static void +target_dlm_lun_reset (struct se_device *dev) +{ +struct target_cluster_data *cluster_data = dev->cluster_data; + +target_pr_lock_dlm (dev); +if (dev->dev_reservation_flags &DRF_SPC2_RESERVATIONS ) { +target_release_reservation (dev); + cluster_data->reserved_node_id =0 ; +pr_debug ("LUN_RESET: SCSI-2 Released reservation\n" ); + } + +target_pr_sync_dlm (dev,PR_SYNC_REASON_RESET );target_pr_unlock_dlm (dev); }
@@ -894,6 +922,7 @@ const struct target_cluster_ops dlm_cluster_ops = { .pr_unlock = target_pr_unlock_dlm, .pr_sync = target_pr_sync_dlm, .reserve = target_spc2_reserve, + .reset = target_dlm_lun_reset, };static int __init target_cluster_dlm_module_init (void )
drivers/target/target_core_device.c +1 -0
@@ -375,6 +375,7 @@ void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq) }spin_unlock (&dev->se_port_lock); } +EXPORT_SYMBOL (target_dev_ua_allocate); static void target_luns_data_has_changed(struct se_node_acl *nacl, struct se_dev_entry *new,
drivers/target/target_core_internal.h +0 -1
@@ -93,7 +93,6 @@ int target_configure_device(struct se_device *dev);void target_free_device (struct se_device *);int target_for_each_device (int (*fn)(struct se_device *dev,void *data),void *data); -void target_dev_ua_allocate (struct se_device *dev,u8 asc,u8 ascq);/* target_core_configfs.c */ extern struct configfs_item_operations target_core_dev_item_ops;
include/target/target_core_base.h +1 -1
@@ -1013,6 +1013,6 @@ int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,void target_pr_kref_release (struct kref *kref);void core_local_lun_reset (struct se_device *swc,struct se_tmr_req *tmr,struct list_head *preempt_and_abort_list,struct se_cmd *prout_cmd); - +void target_dev_ua_allocate (struct se_device *dev,u8 asc,u8 ascq);#endif /* TARGET_CORE_BASE_H */
Signature · 1 line
2.25.1
RFC PATCH 47/48 target: cluster: split cluster sync function
In reply to Dmitry BogdanovSplit cluster sync function to two parts:
1. Read cluster data.
2. Apply cluster data.
--- drivers/target/target_cluster_dlm.c | 117 ++++++++++++++++++---------- 1 file changed, 74 insertions(+), 43 deletions(-)
drivers/target/target_cluster_dlm.c +74 -43
@@ -565,61 +565,53 @@ static void target_allocate_pr_ua(struct se_device *dev, u8 asc) } } -static void target_pr_sync_cb (void *arg) +static int target_dlm_read_cluster_data ( +struct se_device *dev, +struct pr_lvb *pr_data, +struct pr_reg_lvb **pr_reg_datap) { -struct se_device *dev = arg;struct target_cluster_data *cluster_data = dev->cluster_data; -struct t10_pr_registration *pr_reg, *pr_reg_tmp; -struct t10_pr_registration *pr_reg_res_holder =NULL ; -struct t10_pr_registration *pr_prev_res_holder =NULL ;struct pr_reg_lvb *pr_reg_data =NULL ; -LIST_HEAD (to_be_deleted_list); -LIST_HEAD (preempt_and_abort_list);struct async_group grp; -struct pr_lvb pr_data; -bool res_to_delete = false; -struct se_node_acl *pr_reg_nacl; -u64 pr_res_mapped_lun; -bool reg_deleted = false; -bool was_held; -u8 was_type; -u8 was_scope; -bool found; +int res =0 ;int i =0 ; -int res; - res =dlm_ckv_get (cluster_data->pr_data, (char *)&pr_data,sizeof (pr_data)); + res =dlm_ckv_get (cluster_data->pr_data, (char *)pr_data, +sizeof (struct pr_lvb ));if (res) goto done; -if (!pr_data.version) { +if (!pr_data->version) {pr_info ("TARGET_CORE[%d]: PR data from cluster is invalid\n" , dev->dev_index); goto done; } - pr_reg_data =kzalloc (sizeof (struct pr_reg_lvb ) * pr_data.nr_registrants, +if (pr_data->nr_registrants ==0 ) + goto done; + + *pr_reg_datap =kzalloc (sizeof (struct pr_reg_lvb ) * pr_data->nr_registrants,GFP_KERNEL ); + pr_reg_data = *pr_reg_datap;if (!pr_reg_data) { res = -ENOMEM ; goto done; } - res =pr_reg_realloc (cluster_data, pr_data.nr_registrants); + res =pr_reg_realloc (cluster_data, pr_data->nr_registrants);if (res) goto done; -if (pr_data.nr_registrants ==0 ) - goto skip_pr_reg; -refcount_set (&grp.pending,1 );/* 1 for a loop */ atomic_set (&grp.status,0 );init_completion (&grp.compl); -for (i =0 ; i < pr_data.nr_registrants; ++i) { +for (i =0 ; i < pr_data->nr_registrants; ++i) {refcount_inc (&grp.pending); - res =dlm_ckv_get_async (cluster_data->pr_reg_kv[i], (char *)(pr_reg_data + i), -sizeof (struct pr_reg_lvb ), group_compl_cb, &grp); + res =dlm_ckv_get_async (cluster_data->pr_reg_kv[i], + (char *)(pr_reg_data + i), +sizeof (struct pr_reg_lvb ), + group_compl_cb, &grp);if (res) {refcount_dec (&grp.pending);break ;
@@ -639,7 +631,33 @@ static void target_pr_sync_cb(void *arg) goto done; } -skip_pr_reg: +done: +if (res) +kfree (pr_reg_data); +return res; +} + +static void target_dlm_apply_cluster_data ( +struct se_device *dev, +struct pr_lvb *pr_data, +struct pr_reg_lvb *pr_reg_data) +{ +struct target_cluster_data *cluster_data = dev->cluster_data; +struct t10_pr_registration *pr_reg, *pr_reg_tmp; +struct t10_pr_registration *pr_reg_res_holder =NULL ; +struct t10_pr_registration *pr_prev_res_holder =NULL ; +LIST_HEAD (to_be_deleted_list); +LIST_HEAD (preempt_and_abort_list); +bool res_to_delete = false; +struct se_node_acl *pr_reg_nacl; +u64 pr_res_mapped_lun; +bool reg_deleted = false; +bool was_held; +u8 was_type; +u8 was_scope; +bool found; +int i =0 ; +/* * Update existing registrations */
@@ -656,7 +674,7 @@ static void target_pr_sync_cb(void *arg) pr_reg_list) { found = false; -for (i =0 ; i < pr_data.nr_registrants; ++i) { +for (i =0 ; i < pr_data->nr_registrants; ++i) {if (!pr_reg_data[i].version)continue ;
@@ -665,7 +683,7 @@ static void target_pr_sync_cb(void *arg) found = true;/* mark existing registrants */ pr_reg_data[i].version =0 ; -target_update_pr_reg (dev, pr_reg, &pr_data, +target_update_pr_reg (dev, pr_reg, pr_data, &pr_reg_data[i], &pr_reg_res_holder);break ;
@@ -685,12 +703,12 @@ static void target_pr_sync_cb(void *arg)if (dev->dev_pr_res_holder != pr_reg)__core_scsi3_free_registration (dev, pr_reg, - (pr_data.pro_sa ==PRO_PREEMPT_AND_ABORT ) ? + (pr_data->pro_sa ==PRO_PREEMPT_AND_ABORT ) ? &preempt_and_abort_list :NULL ,0 );else res_to_delete = true; -switch (pr_data.pro_sa) { +switch (pr_data->pro_sa) {case PRO_CLEAR :/* * establish a unit attention condition for the initiator
@@ -720,12 +738,12 @@ static void target_pr_sync_cb(void *arg)spin_unlock (&dev->t10_pr.registration_lock);/* register new entries */ -for (i =0 ; i < pr_data.nr_registrants; ++i) { +for (i =0 ; i < pr_data->nr_registrants; ++i) {/* skip existing registrants */ if (!pr_reg_data[i].version)continue ; - pr_reg =target_create_pr_reg (dev, &pr_data, &pr_reg_data[i]); + pr_reg =target_create_pr_reg (dev, pr_data, &pr_reg_data[i]);if (!pr_reg)pr_err ("TARGET_CORE[%d]: can not create new registration\n" , dev->dev_index);
@@ -733,7 +751,7 @@ static void target_pr_sync_cb(void *arg) pr_reg_res_holder = pr_reg; } -switch (pr_data.pro_sa) { +switch (pr_data->pro_sa) {case PRO_REGISTER_AND_IGNORE_EXISTING_KEY :case PRO_REGISTER :/*
@@ -793,11 +811,11 @@ static void target_pr_sync_cb(void *arg) }/* update general data */ -atomic_set (&dev->t10_pr.pr_generation, pr_data.pr_generation); - dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl; +atomic_set (&dev->t10_pr.pr_generation, pr_data->pr_generation); + dev->t10_pr.pr_aptpl_active = pr_data->pr_aptpl;/* update SPC-2 reservation */ - cluster_data->reserved_node_id = pr_data.reserved_by_nodeid; + cluster_data->reserved_node_id = pr_data->reserved_by_nodeid;spin_lock (&dev->dev_reservation_lock);if (cluster_data->reserved_node_id == cluster_data->local_nodeid && dev->reservation_holder ==NULL ) {
@@ -826,18 +844,18 @@ static void target_pr_sync_cb(void *arg)if (pr_reg_res_holder)__core_scsi3_set_reservation (dev, pr_reg_res_holder, - pr_data.pr_scope, pr_data.pr_type); + pr_data->pr_scope, pr_data->pr_type);spin_unlock (&dev->dev_reservation_lock);if (res_to_delete) {spin_lock (&dev->t10_pr.registration_lock);__core_scsi3_free_registration (dev, pr_prev_res_holder, - (pr_data.pro_sa ==PRO_PREEMPT_AND_ABORT ) ? + (pr_data->pro_sa ==PRO_PREEMPT_AND_ABORT ) ? &preempt_and_abort_list :NULL ,0 );spin_unlock (&dev->t10_pr.registration_lock); } -if (pr_data.pro_sa ==PRO_PREEMPT_AND_ABORT ) { +if (pr_data->pro_sa ==PRO_PREEMPT_AND_ABORT ) {core_local_lun_reset (dev,NULL , &preempt_and_abort_list,NULL ); core_scsi3_release_preempt_and_abort(
@@ -845,7 +863,7 @@ static void target_pr_sync_cb(void *arg) pr_reg_res_holder); } -if (pr_data.pro_sa ==PR_SYNC_REASON_RESET ) { +if (pr_data->pro_sa ==PR_SYNC_REASON_RESET ) {core_local_lun_reset (dev,NULL ,NULL ,NULL ); target_dev_ua_allocate(dev, 0x29,
@@ -856,10 +874,23 @@ static void target_pr_sync_cb(void *arg)core_scsi3_update_and_write_aptpl (dev, dev->t10_pr.pr_aptpl_active); -done:kfree (pr_reg_data); } +static void target_pr_sync_cb (void *arg) +{ +struct pr_reg_lvb *pr_reg_data =NULL ; +struct se_device *dev = arg; +struct pr_lvb pr_data; +int res; + + res =target_dlm_read_cluster_data (dev, &pr_data, &pr_reg_data); +if (res) +return ; + +target_dlm_apply_cluster_data (dev, &pr_data, pr_reg_data); +} +static void target_spc2_reserve (struct se_device *dev,struct se_session *sess) {
Signature · 1 line
2.25.1
RFC PATCH 48/48 target: cluster: request data on initial sync
In reply to Dmitry BogdanovIf there is no cluster data in cluster (master node died), request it from alive nodes.
--- drivers/target/target_cluster_dlm.c | 36 ++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-)
drivers/target/target_cluster_dlm.c +35 -1
@@ -19,6 +19,7 @@ struct target_cluster_data {struct dlm_ckv_lock *pr_lock;struct dlm_ckv_kv *pr_data;struct dlm_ckv_notify *pr_sync_notify; +struct dlm_ckv_notify *init_sync_notify;int reserved_node_id;struct dlm_ckv_kv **pr_reg_kv;size_t pr_reg_kv_len;
@@ -101,7 +102,10 @@ struct async_group {#define PR_SYNC_REASON_RESERVE 10#define PR_SYNC_REASON_RESET 11 +#define PR_SYNC_REASON_INIT 13 +static void target_initial_sync (struct se_device *dev); +static void target_initial_sync_cb (void *arg);static void target_pr_sync_cb (void *arg);static void target_nodeleft_cb (void *arg,int nodeid); static int pr_reg_realloc(struct target_cluster_data *cluster_data,
@@ -157,15 +161,22 @@ static int target_init_dlm(struct se_device *dev)if (!cluster_data->pr_sync_notify) goto fail; + cluster_data->init_sync_notify =dlm_ckv_create_notification ( + cluster_data->bucket,"init_sync" , target_initial_sync_cb); +if (!cluster_data->init_sync_notify) + goto fail; + dev->cluster_data = cluster_data;/* initial sync-up on joining the cluster */ dlm_ckv_lock_get (cluster_data->pr_lock); -target_pr_sync_cb (dev); +target_initial_sync (dev);dlm_ckv_lock_release (cluster_data->pr_lock);return err; fail: +if (cluster_data->init_sync_notify) +dlm_ckv_free_notification (cluster_data->init_sync_notify);if (cluster_data->pr_sync_notify)dlm_ckv_free_notification (cluster_data->pr_sync_notify);if (cluster_data->pr_lock)
@@ -188,6 +199,7 @@ static int target_cleanup_dlm(struct se_device *dev)dlm_ckv_free_kv (cluster_data->pr_reg_kv[i]);kfree (cluster_data->pr_reg_kv); +dlm_ckv_free_notification (cluster_data->init_sync_notify);dlm_ckv_free_notification (cluster_data->pr_sync_notify);dlm_ckv_free_lock (cluster_data->pr_lock);dlm_ckv_free_kv (cluster_data->pr_data);
@@ -410,6 +422,13 @@ static int target_pr_sync_dlm(struct se_device *dev, u8 pro_sa)return res; } +static void target_initial_sync_cb (void *arg) +{ +struct se_device *dev = arg; + +target_pr_sync_dlm (dev,PR_SYNC_REASON_INIT ); +} + static int target_update_pr_reg(struct se_device *dev, struct t10_pr_registration *pr_reg, const struct pr_lvb *pr_data,
@@ -584,6 +603,7 @@ static int target_dlm_read_cluster_data(if (!pr_data->version) {pr_info ("TARGET_CORE[%d]: PR data from cluster is invalid\n" , dev->dev_index); + res = -ENOLCK ; goto done; }
@@ -877,6 +897,20 @@ static void target_dlm_apply_cluster_data(kfree (pr_reg_data); } +static void target_initial_sync (struct se_device *dev) +{ +struct target_cluster_data *cluster_data = dev->cluster_data; +struct pr_reg_lvb *pr_reg_data =NULL ; +struct pr_lvb pr_data; +int res; + + res =target_dlm_read_cluster_data (dev, &pr_data, &pr_reg_data); +if (!res) +target_dlm_apply_cluster_data (dev, &pr_data, pr_reg_data); +if (res == -ENOLCK ) +dlm_ckv_notify (cluster_data->init_sync_notify); +} +static void target_pr_sync_cb (void *arg) {struct pr_reg_lvb *pr_reg_data =NULL ;
Signature · 1 line
2.25.1
Quoted 21 lines · Dmitry Bogdanov
Hi linux target comminity.
Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.
The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.
Patches 1-9:
Make RTPI be part ofse_tpginstead ofse_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).
Quoted 5 lines
Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undependpr_regfromse_naclandse_tpgto be just a registration holder. Make APTPL registrations (not linked tose_dev_entry) be full-fledged registrations.
What are the arguable parts? Do you think it will be the DLM part and coordinating it with nvmet developers? Or was it patches 1-9 and the multi-node support? Or both :)
Is it possible and would it be valuable to at least kind of break this up a little?
I would break this up and post the fixes in one set. I'll help you get them in as soon as possible.
For patches 1-9, I think I remember you posting them before, but I was in the middle of starting a new job so I didn't review them. I really needed something like that at my last 2 jobs so I think it's a valuable feature and I'll review that as well.
If we could at least get those 2 chunks separated then it would make the DLM parts below easier to get eyeballs on. I'm ok with the idea in general. I think every nvmet developer will see the massive patchset and not even look at this first 0/48 email :)
Quoted 38 lines
Patches 30-34:
DLM_CKVmodule that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attribcluster_implregulates which implementation to use for that device. 'single' is for default (no cluster) implementation.Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* UseDLM_CKVLock service to serialize order of PR OUT commands
* UseDLM_CKVKey-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* UseDLM_CKVNotification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster
Quoted 30 lines · Mike Christie
Quoted 21 lines · Dmitry Bogdanov
Hi linux target comminity.
Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.
The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.
Patches 1-9:
Make RTPI be part ofse_tpginstead ofse_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).Quoted 5 lines
Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undependpr_regfromse_naclandse_tpgto be just a registration holder. Make APTPL registrations (not linked tose_dev_entry) be full-fledged registrations.
Quoted 3 lines
What are the arguable parts? Do you think it will be the DLM part and coordinating it with nvmet developers? Or was it patches 1-9 and the multi-node support? Or both :)
In fact every subset can be a subject to argue :)
* RTPI patchset - changing data model from RTPI-set on backstore device to RTPI-set on a whole node.
* PR refactoring - to much changes, may be APTPL changes are not backward compatible
* remote/dummy fabric - name
* DLM_CKV - name, place and even a meaning of the module
* tcm_cluster - too much new exported symbols, not resistant to node death in between of storing PR data in DLM_CKV and other error cases.
Quoted 5 lines
Is it possible and would it be valuable to at least kind of break this up a little?
I would break this up and post the fixes in one set. I'll help you get them in as soon as possible.
After approve of the idea I can break the patch set to several ones and start to post it without RFC prefix. The only problem is that they all depend on previous ones. So I have to post each after the previous gets merged.
Quoted 10 lines
For patches 1-9, I think I remember you posting them before, but I was in the middle of starting a new job so I didn't review them. I really needed something like that at my last 2 jobs so I think it's a valuable feature and I'll review that as well.
If we could at least get those 2 chunks separated then it would make the DLM parts below easier to get eyeballs on. I'm ok with the idea in general. I think every nvmet developer will see the massive patchset and not even look at this first 0/48 email :)
I am not going to share this patchset to nvmet dev list :) nvmet does not yet have a local version of CompareAndWrite and Reservations features, so it is too early for them.
Quoted 40 lines
Quoted 40 lines
Patches 30-34:
DLM_CKVmodule that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attribcluster_implregulates which implementation to use for that device. 'single' is for default (no cluster) implementation.Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* UseDLM_CKVLock service to serialize order of PR OUT commands
* UseDLM_CKVKey-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* UseDLM_CKVNotification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster