Refcounts and rbtrees to increase luns above 255

PATCH Applied Andy Grover 3 participants 81 messages Dec 13, 2013 – Feb 7, 2014

Subsystems (10)

Documentation SCSI USB InfiniBand +6 more subsystems

lore

Share & tools

lore

Andy Grover

author

2013-12-13 23:58 UTC #0 lore raw

cover Refcounts and rbtrees to increase luns above 255

Hi Nicholas,

This patchset uses krefs to refcount structures shared across threads. LIO is full of these because configfs-based configuration actions can be removing an object, even while that object is being used by a SCSI command.

Using kref to free the struct on whichever thread drops the last reference allows us to avoid busy-waiting in configfs removal functions. Next, this set removes the statically-sized tpg lun and deve arrays in favor of dynamically adding entries into rbtrees. This reduces memory consumption and allows more than 255 luns per tpg and initiator mapping.

Except for some rbtree lookups, these changes are entirely in the configuration paths of Lio. I have tested these as extensively as I can, and it's ready for wider testing.

Note: patch 22 converts a percpu refcount to a normal kref. I'd argue the benefit is really in the "refcount" part rather than the "percpu", so a simpler kref does the job, but we might want to discuss this some more.

Please review.

Thanks -- Regards -- Andy

p.s. https://github.com/agrover/linux.git for_nab

Andy Grover

author

2013-12-13 23:59 UTC #1 lore raw

PATCH 1/32 target: Remove unused ua_dev_list member in struct se_ua

Initialized but not used.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_ua.c   |    1 -
 include/target/target_core_base.h |    1 -
 2 files changed, 0 insertions(+), 2 deletions(-)

drivers/target/target_core_ua.c +0 -1

index b04467e..505519b 100644
@@ -98,7 +98,6 @@ int core_scsi3_ua_allocate(
 		pr_err("Unable to allocate struct se_ua\n");
 		return -ENOMEM;
 	}
-	INIT_LIST_HEAD(&ua->ua_dev_list);
 	INIT_LIST_HEAD(&ua->ua_nacl_list);
 
 	ua->ua_nacl = nacl;

include/target/target_core_base.h +0 -1

index 45412a6..d6f96c7 100644
@@ -504,7 +504,6 @@ struct se_ua {
 	u8			ua_asc;
 	u8			ua_ascq;
 	struct se_node_acl	*ua_nacl;
-	struct list_head	ua_dev_list;
 	struct list_head	ua_nacl_list;
 };

Signature · 1 line

1.7.1

Nicholas A. Bellinger

Applied

2013-12-16 20:39 UTC #2 lore raw

Re: [PATCH 01/32] target: Remove unused ua_dev_list member in struct se_ua

Quoted 8 lines · Andy Grover

Initialized but not used.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_ua.c   |    1 -
 include/target/target_core_base.h |    1 -
 2 files changed, 0 insertions(+), 2 deletions(-)

Applied to for-next.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #3 lore raw

PATCH 2/32 target: Don't keep looping in report_luns if too big

In reply to Andy Grover

All further loops will still fail the conditional so just bail right away.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_spc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

drivers/target/target_core_spc.c +1 -1

index 021c3f4..8f52974 100644
@@ -1162,7 +1162,7 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 		 */
 		lun_count++;
 		if ((offset + 8) > cmd->data_length)
-			continue;
+			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
 		offset += 8;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 20:40 UTC #4 lore raw

Re: [PATCH 02/32] target: Don't keep looping in report_luns if too big

Quoted 13 lines · Andy Grover

All further loops will still fail the conditional so just bail right away.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_spc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/target/target_core_spc.c +0 -0
index 021c3f4..8f52974 100644

drivers/target/target_core_spc.c sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd) · line 1168

Quoted 8 lines

drivers/target/target_core_spc.c +1 -1

@@ -1162,7 +1162,7 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 		 */
 		lun_count++;
 		if ((offset + 8) > cmd->data_length)
-			continue;
+			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
 		offset += 8;

NAK.

REPORT_LUNS is supposed to return the total lun_count, even if there is no more buffer space to fill..

This is how the initiator knows it should send another REPORT_LUNS with a larger payload to receive the complete layout.

--nab

Andy Grover

author

2013-12-13 23:58 UTC #5 lore raw

PATCH 3/32 target: Allocate more room for port default groups

In reply to Andy Grover

See target_stat_setup_port_default_groups, we need a 4 element array.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_fabric_configfs.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

drivers/target/target_core_fabric_configfs.c +1 -1

index dae2ad6..fdadc4d 100644
@@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
 	lun_cg->default_groups[1] = NULL;
 
 	port_stat_grp = &lun->port_stat_grps.stat_group;
-	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 3,
+	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");

Signature · 1 line

1.7.1

Nicholas A. Bellinger

Applied

2013-12-16 20:42 UTC #6 lore raw

Re: [PATCH 03/32] target: Allocate more room for port default groups

Quoted 12 lines · Andy Grover

See target_stat_setup_port_default_groups, we need a 4 element array.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_fabric_configfs.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/target/target_core_fabric_configfs.c +0 -0
index dae2ad6..fdadc4d 100644

drivers/target/target_core_fabric_configfs.c static struct config_group *target_fabric_make_lun( · line 912

Quoted 8 lines

drivers/target/target_core_fabric_configfs.c +1 -1

@@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
 	lun_cg->default_groups[1] = NULL;
 
 	port_stat_grp = &lun->port_stat_grps.stat_group;
-	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 3,
+	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");

Applied to for-next.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #7 lore raw

PATCH 4/32 target: Fix sizeof in kmalloc for some default_groups arrays

In reply to Andy Grover

Allocating an array of pointers, not the objects themselves. These two sites now match all the other sites.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_configfs.c        |    2 +-
 drivers/target/target_core_fabric_configfs.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

drivers/target/target_core_configfs.c +1 -1

index 272755d..a1c23d1 100644
@@ -2937,7 +2937,7 @@ static int __init target_core_init_configfs(void)
 	 * and ALUA Logical Unit Group and Target Port Group infrastructure.
 	 */
 	target_cg = &subsys->su_group;
-	target_cg->default_groups = kmalloc(sizeof(struct config_group) * 2,
+	target_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
 				GFP_KERNEL);
 	if (!target_cg->default_groups) {
 		pr_err("Unable to allocate target_cg->default_groups\n");

drivers/target/target_core_fabric_configfs.c +1 -1

index fdadc4d..7de9f04 100644
@@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
 	lun_cg->default_groups[1] = NULL;
 
 	port_stat_grp = &lun->port_stat_grps.stat_group;
-	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
+	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group *) * 4,
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");

Signature · 1 line

1.7.1

Nicholas A. Bellinger

Applied

2013-12-16 20:43 UTC #8 lore raw

Re: [PATCH 04/32] target: Fix sizeof in kmalloc for some default_groups arrays

Quoted 9 lines · Andy Grover

Allocating an array of pointers, not the objects themselves. These two sites now match all the other sites.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_configfs.c        |    2 +-
 drivers/target/target_core_fabric_configfs.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Applied to for-next.

--nab

Quoted 26 lines

drivers/target/target_core_configfs.c +1 -1

index 272755d..a1c23d1 100644
@@ -2937,7 +2937,7 @@ static int __init target_core_init_configfs(void)
 	 * and ALUA Logical Unit Group and Target Port Group infrastructure.
 	 */
 	target_cg = &subsys->su_group;
-	target_cg->default_groups = kmalloc(sizeof(struct config_group) * 2,
+	target_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
 				GFP_KERNEL);
 	if (!target_cg->default_groups) {
 		pr_err("Unable to allocate target_cg->default_groups\n");

drivers/target/target_core_fabric_configfs.c +1 -1

index fdadc4d..7de9f04 100644
@@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
 	lun_cg->default_groups[1] = NULL;
 
 	port_stat_grp = &lun->port_stat_grps.stat_group;
-	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
+	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group *) * 4,
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");

Andy Grover

author

2013-12-13 23:58 UTC #9 lore raw

PATCH 5/32 target: Rename some list heads used as nodes

In reply to Andy Grover

Since everything's a list_head, naming it _node makes it clearer it's to put the struct on a list, not a list head itself.

Rename ua_nacl_list to ua_nacl_node
Rename alua_port_list to alua_port_node
Rename lu_gp_mem_list to lu_gp_mem_node
Rename tg_pt_gp_mem_list to tg_pt_gp_mem_node
Rename sep_list to sep_node
Rename tg_pt_gp_list to tg_pt_gp_node
Rename se_session sess_list to sess_node
Rename se_node_acl acl_list to acl_node

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/iscsi/iscsi_target.c       |    2 +-
 drivers/target/iscsi/iscsi_target_login.c |    4 +-
 drivers/target/sbp/sbp_target.c           |    6 ++--
 drivers/target/target_core_alua.c         |   52 ++++++++++++++--------------
 drivers/target/target_core_configfs.c     |    4 +-
 drivers/target/target_core_device.c       |   18 +++++-----
 drivers/target/target_core_pr.c           |    8 ++--
 drivers/target/target_core_tpg.c          |   20 +++++-----
 drivers/target/target_core_transport.c    |    8 ++--
 drivers/target/target_core_ua.c           |   28 ++++++++--------
 drivers/target/target_core_xcopy.c        |    6 ++--
 drivers/target/tcm_fc/tfc_conf.c          |    2 +-
 include/target/target_core_base.h         |   16 ++++----
 13 files changed, 87 insertions(+), 87 deletions(-)

drivers/target/iscsi/iscsi_target.c +1 -1

index d70e911..2844aff 100644
@@ -4703,7 +4703,7 @@ int iscsit_release_sessions_for_tpg(struct iscsi_portal_group *tpg, int force)
 	}
 
 	list_for_each_entry_safe(se_sess, se_sess_tmp, &se_tpg->tpg_sess_list,
-			sess_list) {
+			sess_node) {
 		sess = (struct iscsi_session *)se_sess->fabric_sess_ptr;
 
 		spin_lock(&sess->conn_lock);

drivers/target/iscsi/iscsi_target_login.c +2 -2

index 4eb93b2..99cd298 100644
@@ -180,7 +180,7 @@ int iscsi_check_for_session_reinstatement(struct iscsi_conn *conn)
 
 	spin_lock_bh(&se_tpg->session_lock);
 	list_for_each_entry_safe(se_sess, se_sess_tmp, &se_tpg->tpg_sess_list,
-			sess_list) {
+			sess_node) {
 
 		sess_p = se_sess->fabric_sess_ptr;
 		spin_lock(&sess_p->conn_lock);

@@ -534,7 +534,7 @@ static int iscsi_login_non_zero_tsih_s2(
 
 	spin_lock_bh(&se_tpg->session_lock);
 	list_for_each_entry_safe(se_sess, se_sess_tmp, &se_tpg->tpg_sess_list,
-			sess_list) {
+			sess_node) {
 
 		sess_p = (struct iscsi_session *)se_sess->fabric_sess_ptr;
 		if (atomic_read(&sess_p->session_fall_back_to_erl0) ||

drivers/target/sbp/sbp_target.c +3 -3

index 24884ca..e1ceae5 100644
@@ -99,7 +99,7 @@ static struct sbp_session *sbp_session_find_by_guid(
 	struct sbp_session *sess, *found = NULL;
 
 	spin_lock_bh(&tpg->se_tpg.session_lock);
-	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_list) {
+	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_node) {
 		sess = se_sess->fabric_sess_ptr;
 		if (sess->guid == guid)
 			found = sess;

@@ -135,7 +135,7 @@ static int sbp_login_count_all_by_lun(
 	int count = 0;
 
 	spin_lock_bh(&tpg->se_tpg.session_lock);
-	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_list) {
+	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_node) {
 		sess = se_sess->fabric_sess_ptr;
 
 		spin_lock_bh(&sess->lock);

@@ -161,7 +161,7 @@ static struct sbp_login_descriptor *sbp_login_find_by_id(
 	struct sbp_login_descriptor *login, *found = NULL;
 
 	spin_lock_bh(&tpg->se_tpg.session_lock);
-	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_list) {
+	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_node) {
 		sess = se_sess->fabric_sess_ptr;
 
 		spin_lock_bh(&sess->lock);

drivers/target/target_core_alua.c +26 ~~-26~~

198 lines not on this page · raw message

drivers/target/target_core_configfs.c +2 -2

index a1c23d1..bdf4299 100644
@@ -1887,7 +1887,7 @@ static ssize_t target_core_alua_lu_gp_show_attr_members(
 	memset(buf, 0, LU_GROUP_NAME_BUF);
 
 	spin_lock(&lu_gp->lu_gp_lock);
-	list_for_each_entry(lu_gp_mem, &lu_gp->lu_gp_mem_list, lu_gp_mem_list) {
+	list_for_each_entry(lu_gp_mem, &lu_gp->lu_gp_mem_list, lu_gp_mem_node) {
 		dev = lu_gp_mem->lu_gp_mem_dev;
 		hba = dev->se_hba;

@@ -2400,7 +2400,7 @@ static ssize_t target_core_alua_tg_pt_gp_show_attr_members(
 
 	spin_lock(&tg_pt_gp->tg_pt_gp_lock);
 	list_for_each_entry(tg_pt_gp_mem, &tg_pt_gp->tg_pt_gp_mem_list,
-			tg_pt_gp_mem_list) {
+			tg_pt_gp_mem_node) {
 		port = tg_pt_gp_mem->tg_pt;
 		tpg = port->sep_tpg;
 		lun = port->sep_lun;

drivers/target/target_core_device.c +9 -9

index 207b340..3350467 100644
@@ -364,7 +364,7 @@ int core_enable_device_list_for_node(
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	spin_lock_bh(&port->sep_alua_lock);
-	list_add_tail(&deve->alua_port_list, &port->sep_alua_list);
+	list_add_tail(&deve->alua_port_node, &port->sep_alua_list);
 	spin_unlock_bh(&port->sep_alua_lock);
 
 	return 0;

@@ -393,13 +393,13 @@ int core_disable_device_list_for_node(
 	 *
 	 * deve->se_lun_acl will be NULL for demo-mode created LUNs
 	 * that have not been explicitly converted to MappedLUNs ->
-	 * struct se_lun_acl, but we remove deve->alua_port_list from
+	 * struct se_lun_acl, but we remove deve->alua_port_node from
 	 * port->sep_alua_list. This also means that active UAs and
 	 * NodeACL context specific PR metadata for demo-mode
 	 * MappedLUN *deve will be released below..
 	 */
 	spin_lock_bh(&port->sep_alua_lock);
-	list_del(&deve->alua_port_list);
+	list_del(&deve->alua_port_node);
 	spin_unlock_bh(&port->sep_alua_lock);
 	/*
 	 * Wait for any in process SPEC_I_PT=1 or REGISTER_AND_MOVE

@@ -435,7 +435,7 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 	u32 i;
 
 	spin_lock_irq(&tpg->acl_node_lock);
-	list_for_each_entry(nacl, &tpg->acl_node_list, acl_list) {
+	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
 		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);

@@ -468,7 +468,7 @@ static struct se_port *core_alloc_port(struct se_device *dev)
 		return ERR_PTR(-ENOMEM);
 	}
 	INIT_LIST_HEAD(&port->sep_alua_list);
-	INIT_LIST_HEAD(&port->sep_list);
+	INIT_LIST_HEAD(&port->sep_node);
 	atomic_set(&port->sep_tg_pt_secondary_offline, 0);
 	spin_lock_init(&port->sep_alua_lock);
 	mutex_init(&port->sep_tg_pt_md_mutex);

@@ -497,7 +497,7 @@ again:
 	if (!port->sep_rtpi)
 		goto again;
 
-	list_for_each_entry(port_tmp, &dev->dev_sep_list, sep_list) {
+	list_for_each_entry(port_tmp, &dev->dev_sep_list, sep_node) {
 		/*
 		 * Make sure RELATIVE TARGET PORT IDENTIFIER is unique
 		 * for 16-bit wrap..

@@ -525,7 +525,7 @@ static void core_export_port(
 	lun->lun_sep = port;
 	spin_unlock(&lun->lun_sep_lock);
 
-	list_add_tail(&port->sep_list, &dev->dev_sep_list);
+	list_add_tail(&port->sep_node, &dev->dev_sep_list);
 	spin_unlock(&dev->se_port_lock);
 
 	if (dev->transport->transport_type != TRANSPORT_PLUGIN_PHBA_PDEV &&

@@ -566,7 +566,7 @@ static void core_release_port(struct se_device *dev, struct se_port *port)
 
 	core_alua_free_tg_pt_gp_mem(port);
 
-	list_del(&port->sep_list);
+	list_del(&port->sep_node);
 	dev->dev_port_count--;
 	kfree(port);
 }

@@ -1137,7 +1137,7 @@ struct se_lun *core_dev_add_lun(
 	if (tpg->se_tpg_tfo->tpg_check_demo_mode(tpg)) {
 		struct se_node_acl *acl;
 		spin_lock_irq(&tpg->acl_node_lock);
-		list_for_each_entry(acl, &tpg->acl_node_list, acl_list) {
+		list_for_each_entry(acl, &tpg->acl_node_list, acl_node) {
 			if (acl->dynamic_node_acl &&
 			    (!tpg->se_tpg_tfo->tpg_check_demo_mode_login_only ||
 			     !tpg->se_tpg_tfo->tpg_check_demo_mode_login_only(tpg))) {

drivers/target/target_core_pr.c +4 -4

index 2f5d779..d141c7f 100644
@@ -673,14 +673,14 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 	 * for ALL_TG_PT=1
 	 */
 	spin_lock(&dev->se_port_lock);
-	list_for_each_entry_safe(port, port_tmp, &dev->dev_sep_list, sep_list) {
+	list_for_each_entry_safe(port, port_tmp, &dev->dev_sep_list, sep_node) {
 		atomic_inc(&port->sep_tg_pt_ref_cnt);
 		smp_mb__after_atomic_inc();
 		spin_unlock(&dev->se_port_lock);
 
 		spin_lock_bh(&port->sep_alua_lock);
 		list_for_each_entry(deve_tmp, &port->sep_alua_list,
-					alua_port_list) {
+					alua_port_node) {
 			/*
 			 * This pointer will be NULL for demo mode MappedLUNs
 			 * that have not been make explicit via a ConfigFS

@@ -1526,7 +1526,7 @@ core_scsi3_decode_spec_i_port(
 		dest_tpg = NULL;
 
 		spin_lock(&dev->se_port_lock);
-		list_for_each_entry(tmp_port, &dev->dev_sep_list, sep_list) {
+		list_for_each_entry(tmp_port, &dev->dev_sep_list, sep_node) {
 			tmp_tpg = tmp_port->sep_tpg;
 			if (!tmp_tpg)
 				continue;

@@ -3154,7 +3154,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 	}
 
 	spin_lock(&dev->se_port_lock);
-	list_for_each_entry(se_port, &dev->dev_sep_list, sep_list) {
+	list_for_each_entry(se_port, &dev->dev_sep_list, sep_node) {
 		if (se_port->sep_rtpi != rtpi)
 			continue;
 		dest_se_tpg = se_port->sep_tpg;

drivers/target/target_core_tpg.c +10 ~~-10~~

index f697f8b..1fd90fc 100644
@@ -92,7 +92,7 @@ struct se_node_acl *__core_tpg_get_initiator_node_acl(
 {
 	struct se_node_acl *acl;
 
-	list_for_each_entry(acl, &tpg->acl_node_list, acl_list) {
+	list_for_each_entry(acl, &tpg->acl_node_list, acl_node) {
 		if (!strcmp(acl->initiatorname, initiatorname))
 			return acl;
 	}

@@ -239,7 +239,7 @@ static int core_create_device_list_for_node(struct se_node_acl *nacl)
 		atomic_set(&deve->ua_count, 0);
 		atomic_set(&deve->pr_ref_count, 0);
 		spin_lock_init(&deve->ua_lock);
-		INIT_LIST_HEAD(&deve->alua_port_list);
+		INIT_LIST_HEAD(&deve->alua_port_node);
 		INIT_LIST_HEAD(&deve->ua_list);
 	}

@@ -267,7 +267,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	if (!acl)
 		return NULL;
 
-	INIT_LIST_HEAD(&acl->acl_list);
+	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
 	kref_init(&acl->acl_kref);
 	init_completion(&acl->acl_free_comp);

@@ -303,7 +303,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 		core_tpg_add_node_to_devs(acl, tpg);
 
 	spin_lock_irq(&tpg->acl_node_lock);
-	list_add_tail(&acl->acl_list, &tpg->acl_node_list);
+	list_add_tail(&acl->acl_node, &tpg->acl_node_list);
 	tpg->num_node_acls++;
 	spin_unlock_irq(&tpg->acl_node_lock);

@@ -395,7 +395,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	 */
 	acl = se_nacl;
 
-	INIT_LIST_HEAD(&acl->acl_list);
+	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
 	kref_init(&acl->acl_kref);
 	init_completion(&acl->acl_free_comp);

@@ -422,7 +422,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	}
 
 	spin_lock_irq(&tpg->acl_node_lock);
-	list_add_tail(&acl->acl_list, &tpg->acl_node_list);
+	list_add_tail(&acl->acl_node, &tpg->acl_node_list);
 	tpg->num_node_acls++;
 	spin_unlock_irq(&tpg->acl_node_lock);

@@ -454,7 +454,7 @@ int core_tpg_del_initiator_node_acl(
 	if (acl->dynamic_node_acl) {
 		acl->dynamic_node_acl = 0;
 	}
-	list_del(&acl->acl_list);
+	list_del(&acl->acl_node);
 	tpg->num_node_acls--;
 	spin_unlock_irq(&tpg->acl_node_lock);

@@ -532,7 +532,7 @@ int core_tpg_set_initiator_node_queue_depth(
 	spin_unlock_irq(&tpg->acl_node_lock);
 
 	spin_lock_irqsave(&tpg->session_lock, flags);
-	list_for_each_entry(sess, &tpg->tpg_sess_list, sess_list) {
+	list_for_each_entry(sess, &tpg->tpg_sess_list, sess_node) {
 		if (sess->se_node_acl != acl)
 			continue;

@@ -767,8 +767,8 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 	 */
 	spin_lock_irq(&se_tpg->acl_node_lock);
 	list_for_each_entry_safe(nacl, nacl_tmp, &se_tpg->acl_node_list,
-			acl_list) {
-		list_del(&nacl->acl_list);
+			acl_node) {
+		list_del(&nacl->acl_node);
 		se_tpg->num_node_acls--;
 		spin_unlock_irq(&se_tpg->acl_node_lock);

drivers/target/target_core_transport.c +4 -4

index 91953da..accaca5 100644
@@ -219,7 +219,7 @@ struct se_session *transport_init_session(void)
 				" se_sess_cache\n");
 		return ERR_PTR(-ENOMEM);
 	}
-	INIT_LIST_HEAD(&se_sess->sess_list);
+	INIT_LIST_HEAD(&se_sess->sess_node);
 	INIT_LIST_HEAD(&se_sess->sess_acl_list);
 	INIT_LIST_HEAD(&se_sess->sess_cmd_list);
 	INIT_LIST_HEAD(&se_sess->sess_wait_list);

@@ -324,7 +324,7 @@ void __transport_register_session(
 			      &se_nacl->acl_sess_list);
 		spin_unlock_irq(&se_nacl->nacl_sess_lock);
 	}
-	list_add_tail(&se_sess->sess_list, &se_tpg->tpg_sess_list);
+	list_add_tail(&se_sess->sess_node, &se_tpg->tpg_sess_list);
 
 	pr_debug("TARGET_CORE[%s]: Registered fabric_sess_ptr: %p\n",
 		se_tpg->se_tpg_tfo->get_fabric_name(), se_sess->fabric_sess_ptr);

@@ -442,7 +442,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	se_tfo = se_tpg->se_tpg_tfo;
 
 	spin_lock_irqsave(&se_tpg->session_lock, flags);
-	list_del(&se_sess->sess_list);
+	list_del(&se_sess->sess_node);
 	se_sess->se_tpg = NULL;
 	se_sess->fabric_sess_ptr = NULL;
 	spin_unlock_irqrestore(&se_tpg->session_lock, flags);

@@ -456,7 +456,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	spin_lock_irqsave(&se_tpg->acl_node_lock, flags);
 	if (se_nacl && se_nacl->dynamic_node_acl) {
 		if (!se_tfo->tpg_check_demo_mode_cache(se_tpg)) {
-			list_del(&se_nacl->acl_list);
+			list_del(&se_nacl->acl_node);
 			se_tpg->num_node_acls--;
 			spin_unlock_irqrestore(&se_tpg->acl_node_lock, flags);
 			core_tpg_wait_for_nacl_pr_ref(se_nacl);

drivers/target/target_core_ua.c +14 ~~-14~~

index 505519b..2a1cbf9 100644
@@ -98,7 +98,7 @@ int core_scsi3_ua_allocate(
 		pr_err("Unable to allocate struct se_ua\n");
 		return -ENOMEM;
 	}
-	INIT_LIST_HEAD(&ua->ua_nacl_list);
+	INIT_LIST_HEAD(&ua->ua_nacl_node);
 
 	ua->ua_nacl = nacl;
 	ua->ua_asc = asc;

@@ -108,7 +108,7 @@ int core_scsi3_ua_allocate(
 	deve = nacl->device_list[unpacked_lun];
 
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_list) {
+	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_node) {
 		/*
 		 * Do not report the same UNIT ATTENTION twice..
 		 */

@@ -139,10 +139,10 @@ int core_scsi3_ua_allocate(
 		 */
 		if (ua_p->ua_asc == 0x29) {
 			if ((asc == 0x29) && (ascq > ua_p->ua_ascq))
-				list_add(&ua->ua_nacl_list,
+				list_add(&ua->ua_nacl_node,
 						&deve->ua_list);
 			else
-				list_add_tail(&ua->ua_nacl_list,
+				list_add_tail(&ua->ua_nacl_node,
 						&deve->ua_list);
 		} else if (ua_p->ua_asc == 0x2a) {
 			/*

@@ -150,13 +150,13 @@ int core_scsi3_ua_allocate(
 			 * Family 2AHh ASCQ codes for Unit Attention condition.
 			 */
 			if ((asc == 0x29) || (ascq > ua_p->ua_asc))
-				list_add(&ua->ua_nacl_list,
+				list_add(&ua->ua_nacl_node,
 					&deve->ua_list);
 			else
-				list_add_tail(&ua->ua_nacl_list,
+				list_add_tail(&ua->ua_nacl_node,
 						&deve->ua_list);
 		} else
-			list_add_tail(&ua->ua_nacl_list,
+			list_add_tail(&ua->ua_nacl_node,
 				&deve->ua_list);
 		spin_unlock(&deve->ua_lock);
 		spin_unlock_irq(&nacl->device_list_lock);

@@ -165,7 +165,7 @@ int core_scsi3_ua_allocate(
 		smp_mb__after_atomic_inc();
 		return 0;
 	}
-	list_add_tail(&ua->ua_nacl_list, &deve->ua_list);
+	list_add_tail(&ua->ua_nacl_node, &deve->ua_list);
 	spin_unlock(&deve->ua_lock);
 	spin_unlock_irq(&nacl->device_list_lock);

@@ -185,8 +185,8 @@ void core_scsi3_ua_release_all(
 	struct se_ua *ua, *ua_p;
 
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_list) {
-		list_del(&ua->ua_nacl_list);
+	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_node) {
+		list_del(&ua->ua_nacl_node);
 		kmem_cache_free(se_ua_cache, ua);
 
 		atomic_dec(&deve->ua_count);

@@ -226,7 +226,7 @@ void core_scsi3_ua_for_check_condition(
 	 * sense data for the received CDB.
 	 */
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_list) {
+	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_node) {
 		/*
 		 * For ua_intlck_ctrl code not equal to 00b, only report the
 		 * highest priority UNIT_ATTENTION and ASC/ASCQ without

@@ -247,7 +247,7 @@ void core_scsi3_ua_for_check_condition(
 			*ascq = ua->ua_ascq;
 			head = 0;
 		}
-		list_del(&ua->ua_nacl_list);
+		list_del(&ua->ua_nacl_node);
 		kmem_cache_free(se_ua_cache, ua);
 
 		atomic_dec(&deve->ua_count);

@@ -300,13 +300,13 @@ int core_scsi3_ua_clear_for_request_sense(
 	 * struct se_lun.
 	 */
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_list) {
+	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_node) {
 		if (head) {
 			*asc = ua->ua_asc;
 			*ascq = ua->ua_ascq;
 			head = 0;
 		}
-		list_del(&ua->ua_nacl_list);
+		list_del(&ua->ua_nacl_node);
 		kmem_cache_free(se_ua_cache, ua);
 
 		atomic_dec(&deve->ua_count);

drivers/target/target_core_xcopy.c +3 -3

index 6b88a99..bf22d4b 100644
@@ -463,7 +463,7 @@ int target_xcopy_setup_pt(void)
 
 	memset(&xcopy_pt_port, 0, sizeof(struct se_port));
 	INIT_LIST_HEAD(&xcopy_pt_port.sep_alua_list);
-	INIT_LIST_HEAD(&xcopy_pt_port.sep_list);
+	INIT_LIST_HEAD(&xcopy_pt_port.sep_node);
 	mutex_init(&xcopy_pt_port.sep_tg_pt_md_mutex);
 
 	memset(&xcopy_pt_tpg, 0, sizeof(struct se_portal_group));

@@ -475,10 +475,10 @@ int target_xcopy_setup_pt(void)
 	xcopy_pt_tpg.se_tpg_tfo = &xcopy_pt_tfo;
 
 	memset(&xcopy_pt_nacl, 0, sizeof(struct se_node_acl));
-	INIT_LIST_HEAD(&xcopy_pt_nacl.acl_list);
+	INIT_LIST_HEAD(&xcopy_pt_nacl.acl_node);
 	INIT_LIST_HEAD(&xcopy_pt_nacl.acl_sess_list);
 	memset(&xcopy_pt_sess, 0, sizeof(struct se_session));
-	INIT_LIST_HEAD(&xcopy_pt_sess.sess_list);
+	INIT_LIST_HEAD(&xcopy_pt_sess.sess_node);
 	INIT_LIST_HEAD(&xcopy_pt_sess.sess_acl_list);
 
 	xcopy_pt_nacl.se_tpg = &xcopy_pt_tpg;

drivers/target/tcm_fc/tfc_conf.c +1 -1

index c6932fb..47d0038 100644
@@ -250,7 +250,7 @@ struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)
 	struct se_node_acl *se_acl;
 
 	spin_lock_irq(&se_tpg->acl_node_lock);
-	list_for_each_entry(se_acl, &se_tpg->acl_node_list, acl_list) {
+	list_for_each_entry(se_acl, &se_tpg->acl_node_list, acl_node) {
 		acl = container_of(se_acl, struct ft_node_acl, se_node_acl);
 		pr_debug("acl %p port_name %llx\n",
 			acl, (unsigned long long)acl->node_auth.port_name);

include/target/target_core_base.h +8 -8

index d6f96c7..d5e0da1 100644
@@ -277,7 +277,7 @@ struct t10_alua_lu_gp_member {
 	spinlock_t lu_gp_mem_lock;
 	struct t10_alua_lu_gp *lu_gp;
 	struct se_device *lu_gp_mem_dev;
-	struct list_head lu_gp_mem_list;
+	struct list_head lu_gp_mem_node;
 };
 
 struct t10_alua_tg_pt_gp {

@@ -301,7 +301,7 @@ struct t10_alua_tg_pt_gp {
 	struct mutex tg_pt_gp_md_mutex;
 	struct se_device *tg_pt_gp_dev;
 	struct config_group tg_pt_gp_group;
-	struct list_head tg_pt_gp_list;
+	struct list_head tg_pt_gp_node;
 	struct list_head tg_pt_gp_mem_list;
 };

@@ -311,7 +311,7 @@ struct t10_alua_tg_pt_gp_member {
 	spinlock_t tg_pt_gp_mem_lock;
 	struct t10_alua_tg_pt_gp *tg_pt_gp;
 	struct se_port *tg_pt;
-	struct list_head tg_pt_gp_mem_list;
+	struct list_head tg_pt_gp_mem_node;
 };
 
 struct t10_vpd {

@@ -504,7 +504,7 @@ struct se_ua {
 	u8			ua_asc;
 	u8			ua_ascq;
 	struct se_node_acl	*ua_nacl;
-	struct list_head	ua_nacl_list;
+	struct list_head	ua_nacl_node;
 };
 
 struct se_node_acl {

@@ -533,7 +533,7 @@ struct se_node_acl {
 	struct config_group	acl_param_group;
 	struct config_group	acl_fabric_stat_group;
 	struct config_group	*acl_default_groups[5];
-	struct list_head	acl_list;
+	struct list_head	acl_node;
 	struct list_head	acl_sess_list;
 	struct completion	acl_free_comp;
 	struct kref		acl_kref;

@@ -545,7 +545,7 @@ struct se_session {
 	struct se_node_acl	*se_node_acl;
 	struct se_portal_group *se_tpg;
 	void			*fabric_sess_ptr;
-	struct list_head	sess_list;
+	struct list_head	sess_node;
 	struct list_head	sess_acl_list;
 	struct list_head	sess_cmd_list;
 	struct list_head	sess_wait_list;

@@ -592,7 +592,7 @@ struct se_dev_entry {
 	struct se_lun_acl	*se_lun_acl;
 	spinlock_t		ua_lock;
 	struct se_lun		*se_lun;
-	struct list_head	alua_port_list;
+	struct list_head	alua_port_node;
 	struct list_head	ua_list;
 };

@@ -780,7 +780,7 @@ struct se_port {
 	struct se_lun *sep_lun;
 	struct se_portal_group *sep_tpg;
 	struct list_head sep_alua_list;
-	struct list_head sep_list;
+	struct list_head sep_node;
 };
 
 struct se_tpg_np {

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 20:45 UTC #10 lore raw

Re: [PATCH 05/32] target: Rename some list heads used as nodes

Quoted 13 lines · Andy Grover

Since everything's a list_head, naming it _node makes it clearer it's to put the struct on a list, not a list head itself.
Rename ua_nacl_list to ua_nacl_node
Rename alua_port_list to alua_port_node
Rename lu_gp_mem_list to lu_gp_mem_node
Rename tg_pt_gp_mem_list to tg_pt_gp_mem_node
Rename sep_list to sep_node
Rename tg_pt_gp_list to tg_pt_gp_node
Rename se_session sess_list to sess_node
Rename se_node_acl acl_list to acl_node
Signed-off-by Andy Grover <[email protected]>

Quoted 15 lines

---
 drivers/target/iscsi/iscsi_target.c       |    2 +-
 drivers/target/iscsi/iscsi_target_login.c |    4 +-
 drivers/target/sbp/sbp_target.c           |    6 ++--
 drivers/target/target_core_alua.c         |   52 ++++++++++++++--------------
 drivers/target/target_core_configfs.c     |    4 +-
 drivers/target/target_core_device.c       |   18 +++++-----
 drivers/target/target_core_pr.c           |    8 ++--
 drivers/target/target_core_tpg.c          |   20 +++++-----
 drivers/target/target_core_transport.c    |    8 ++--
 drivers/target/target_core_ua.c           |   28 ++++++++--------
 drivers/target/target_core_xcopy.c        |    6 ++--
 drivers/target/tcm_fc/tfc_conf.c          |    2 +-
 include/target/target_core_base.h         |   16 ++++----
 13 files changed, 87 insertions(+), 87 deletions(-)

Holding off on this one for the moment, as it conflict's with Hannes's patches for ALUA + Referrals.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #11 lore raw

PATCH 6/32 target: Convert lu_gp_ref_cnt to kref

In reply to Andy Grover

Use kref to handle reference counting

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 21 insertions(+), 18 deletions(-)

drivers/target/target_core_alua.c +20 ~~-17~~

index 2ac2f11..8c01ade 100644
@@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
 
 struct t10_alua_lu_gp *default_lu_gp;
 
+static void release_alua_lu_gp(struct kref *ref)
+{
+	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+}
+
+#define get_alua_lu_gp(x) kref_get(&x->refcount)
+#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -898,8 +908,7 @@ int core_alua_do_port_transition(
 	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
 	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = local_lu_gp_mem->lu_gp;
-	atomic_inc(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_inc();
+	get_alua_lu_gp(lu_gp);
 	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
 	/*
 	 * For storage objects that are members of the 'default_lu_gp',

@@ -913,8 +922,8 @@ int core_alua_do_port_transition(
 		 */
 		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
 					md_buf, new_state, explicit);
-		atomic_dec(&lu_gp->lu_gp_ref_cnt);
-		smp_mb__after_atomic_dec();
+
+		put_alua_lu_gp(lu_gp);
 		kfree(md_buf);
 		return 0;
 	}

@@ -985,8 +994,7 @@ int core_alua_do_port_transition(
 		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
 		core_alua_dump_state(new_state));
 
-	atomic_dec(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_dec();
+	put_alua_lu_gp(lu_gp);
 	kfree(md_buf);
 	return 0;
 }

@@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
 	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
 	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
 	spin_lock_init(&lu_gp->lu_gp_lock);
-	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
+
+	kref_init(&lu_gp->refcount);
 
 	if (def_group) {
 		lu_gp->lu_gp_id = alua_lu_gps_counter++;

@@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	list_del(&lu_gp->lu_gp_node);
 	alua_lu_gps_count--;
 	spin_unlock(&lu_gps_lock);
-	/*
-	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
-	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
-	 * released with core_alua_put_lu_gp_from_name()
-	 */
-	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
-		cpu_relax();
+
 	/*
 	 * Release reference to struct t10_alua_lu_gp * from all associated
 	 * struct se_device.

@@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);
 
-	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+	put_alua_lu_gp(lu_gp);
 }
 
 void core_alua_free_lu_gp_mem(struct se_device *dev)

@@ -1284,7 +1287,7 @@ struct t10_alua_lu_gp *core_alua_get_lu_gp_by_name(const char *name)
 			continue;
 		ci = &lu_gp->lu_gp_group.cg_item;
 		if (!strcmp(config_item_name(ci), name)) {
-			atomic_inc(&lu_gp->lu_gp_ref_cnt);
+			get_alua_lu_gp(lu_gp);
 			spin_unlock(&lu_gps_lock);
 			return lu_gp;
 		}

@@ -1297,7 +1300,7 @@ struct t10_alua_lu_gp *core_alua_get_lu_gp_by_name(const char *name)
 void core_alua_put_lu_gp_from_name(struct t10_alua_lu_gp *lu_gp)
 {
 	spin_lock(&lu_gps_lock);
-	atomic_dec(&lu_gp->lu_gp_ref_cnt);
+	put_alua_lu_gp(lu_gp);
 	spin_unlock(&lu_gps_lock);
 }

include/target/target_core_base.h +1 -1

index d5e0da1..5bdf0d5 100644
@@ -264,7 +264,7 @@ struct t10_alua_lu_gp {
 	u16	lu_gp_id;
 	int	lu_gp_valid_id;
 	u32	lu_gp_members;
-	atomic_t lu_gp_ref_cnt;
+	struct kref refcount;
 	spinlock_t lu_gp_lock;
 	struct config_group lu_gp_group;
 	struct list_head lu_gp_node;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 20:51 UTC #12 lore raw

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

Quoted 86 lines · Andy Grover

Use kref to handle reference counting

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 21 insertions(+), 18 deletions(-)

drivers/target/target_core_alua.c +17 ~~-14~~

index 2ac2f11..8c01ade 100644
@@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
 
 struct t10_alua_lu_gp *default_lu_gp;
 
+static void release_alua_lu_gp(struct kref *ref)
+{
+	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+}
+
+#define get_alua_lu_gp(x) kref_get(&x->refcount)
+#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -898,8 +908,7 @@ int core_alua_do_port_transition(
 	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
 	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = local_lu_gp_mem->lu_gp;
-	atomic_inc(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_inc();
+	get_alua_lu_gp(lu_gp);
 	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
 	/*
 	 * For storage objects that are members of the 'default_lu_gp',

@@ -913,8 +922,8 @@ int core_alua_do_port_transition(
 		 */
 		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
 					md_buf, new_state, explicit);
-		atomic_dec(&lu_gp->lu_gp_ref_cnt);
-		smp_mb__after_atomic_dec();
+
+		put_alua_lu_gp(lu_gp);
 		kfree(md_buf);
 		return 0;
 	}

@@ -985,8 +994,7 @@ int core_alua_do_port_transition(
 		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
 		core_alua_dump_state(new_state));
 
-	atomic_dec(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_dec();
+	put_alua_lu_gp(lu_gp);
 	kfree(md_buf);
 	return 0;
 }

@@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
 	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
 	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
 	spin_lock_init(&lu_gp->lu_gp_lock);
-	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
+
+	kref_init(&lu_gp->refcount);
 
 	if (def_group) {
 		lu_gp->lu_gp_id = alua_lu_gps_counter++;

@@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	list_del(&lu_gp->lu_gp_node);
 	alua_lu_gps_count--;
 	spin_unlock(&lu_gps_lock);
-	/*
-	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
-	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
-	 * released with core_alua_put_lu_gp_from_name()
-	 */
-	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
-		cpu_relax();
+
 	/*
 	 * Release reference to struct t10_alua_lu_gp * from all associated
 	 * struct se_device.

drivers/target/target_core_alua.c void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp) · line 1249

Quoted 7 lines

drivers/target/target_core_alua.c +1 -1

@@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);
 
-	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+	put_alua_lu_gp(lu_gp);
 }

The assumption that it's safe to 'Release reference to struct t10_alua_lu_gp * from all associated struct device' below the original cpu_relax(), while there are still other process contexts doing their respective put_alua_lu_gp() is totally wrong.

Furthermore, allowing a configfs_group_ops->drop_item() to return while there are still active references from other process contexts means that the parent struct config_group is no longer referenced counted (eg:
configfs child is removed), and introduces a whole host of potential bugs.

So that said, NAK on this patch.

--nab

Andy Grover

author

2014-02-05 22:02 UTC #13 lore raw

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

Hi nab, I'm getting back to looking at this patchset, but wanted to just discuss and understand this one first because all the kref ones are similar. see below.

Quoted 90 lines · Nicholas A. Bellinger

Quoted 89 lines · Andy Grover

Use kref to handle reference counting

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 21 insertions(+), 18 deletions(-)

drivers/target/target_core_alua.c +17 ~~-14~~

index 2ac2f11..8c01ade 100644
@@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
 
 struct t10_alua_lu_gp *default_lu_gp;
 
+static void release_alua_lu_gp(struct kref *ref)
+{
+	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+}
+
+#define get_alua_lu_gp(x) kref_get(&x->refcount)
+#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -898,8 +908,7 @@ int core_alua_do_port_transition(
 	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
 	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = local_lu_gp_mem->lu_gp;
-	atomic_inc(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_inc();
+	get_alua_lu_gp(lu_gp);
 	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
 	/*
 	 * For storage objects that are members of the 'default_lu_gp',

@@ -913,8 +922,8 @@ int core_alua_do_port_transition(
 		 */
 		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
 					md_buf, new_state, explicit);
-		atomic_dec(&lu_gp->lu_gp_ref_cnt);
-		smp_mb__after_atomic_dec();
+
+		put_alua_lu_gp(lu_gp);
 		kfree(md_buf);
 		return 0;
 	}

@@ -985,8 +994,7 @@ int core_alua_do_port_transition(
 		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
 		core_alua_dump_state(new_state));
 
-	atomic_dec(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_dec();
+	put_alua_lu_gp(lu_gp);
 	kfree(md_buf);
 	return 0;
 }

@@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
 	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
 	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
 	spin_lock_init(&lu_gp->lu_gp_lock);
-	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
+
+	kref_init(&lu_gp->refcount);
 
 	if (def_group) {
 		lu_gp->lu_gp_id = alua_lu_gps_counter++;

@@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	list_del(&lu_gp->lu_gp_node);
 	alua_lu_gps_count--;
 	spin_unlock(&lu_gps_lock);
-	/*
-	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
-	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
-	 * released with core_alua_put_lu_gp_from_name()
-	 */
-	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
-		cpu_relax();
+
 	/*
 	 * Release reference to struct t10_alua_lu_gp * from all associated
 	 * struct se_device.

@@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);

Quoted 9 lines

drivers/target/target_core_alua.c void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp) · line 1249
Quoted 4 lines
drivers/target/target_core_alua.c +1 -1
-	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+	put_alua_lu_gp(lu_gp);
 }
 
The assumption that it's safe to 'Release reference to struct t10_alua_lu_gp * from all associated struct device' below the original cpu_relax(), while there are still other process contexts doing their respective put_alua_lu_gp() is totally wrong.

The only other spot is core_alua_do_port_transition, afaics. I think if it races with free_lu_gp, lu_gp will either be the old lu_gp (which no longer will have anything on lu_gp_mem_list) or will be default_lu_gp. If it's the old lu_gp then it iterates over an empty list, and then the lu_gp gets finally freed by the put() at the bottom.

Quoted 7 lines

Furthermore, allowing a configfs_group_ops->drop_item() to return while there are still active references from other process contexts means that the parent struct config_group is no longer referenced counted (eg:
configfs child is removed), and introduces a whole host of potential bugs.
So that said, NAK on this patch.

I think some of the other patches used drop_item() and thus were bad, but in this one the existing code is already calling core_alua_free_lu_gp() from release().

Thoughts?

Thanks in advance -- Regards -- Andy

Nicholas A. Bellinger

2014-02-06 23:48 UTC #14 lore raw

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

Quoted 117 lines · Andy Grover

Hi nab, I'm getting back to looking at this patchset, but wanted to just discuss and understand this one first because all the kref ones are similar. see below.
Quoted 90 lines · Nicholas A. Bellinger
Quoted 89 lines · Andy Grover
Use kref to handle reference counting
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 21 insertions(+), 18 deletions(-)
drivers/target/target_core_alua.c +17 ~~-14~~
index 2ac2f11..8c01ade 100644
@@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
 
 struct t10_alua_lu_gp *default_lu_gp;
 
+static void release_alua_lu_gp(struct kref *ref)
+{
+	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+}
+
+#define get_alua_lu_gp(x) kref_get(&x->refcount)
+#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *
@@ -898,8 +908,7 @@ int core_alua_do_port_transition(
 	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
 	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = local_lu_gp_mem->lu_gp;
-	atomic_inc(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_inc();
+	get_alua_lu_gp(lu_gp);
 	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
 	/*
 	 * For storage objects that are members of the 'default_lu_gp',
@@ -913,8 +922,8 @@ int core_alua_do_port_transition(
 		 */
 		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
 					md_buf, new_state, explicit);
-		atomic_dec(&lu_gp->lu_gp_ref_cnt);
-		smp_mb__after_atomic_dec();
+
+		put_alua_lu_gp(lu_gp);
 		kfree(md_buf);
 		return 0;
 	}
@@ -985,8 +994,7 @@ int core_alua_do_port_transition(
 		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
 		core_alua_dump_state(new_state));
 
-	atomic_dec(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_dec();
+	put_alua_lu_gp(lu_gp);
 	kfree(md_buf);
 	return 0;
 }
@@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
 	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
 	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
 	spin_lock_init(&lu_gp->lu_gp_lock);
-	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
+
+	kref_init(&lu_gp->refcount);
 
 	if (def_group) {
 		lu_gp->lu_gp_id = alua_lu_gps_counter++;
@@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	list_del(&lu_gp->lu_gp_node);
 	alua_lu_gps_count--;
 	spin_unlock(&lu_gps_lock);
-	/*
-	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
-	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
-	 * released with core_alua_put_lu_gp_from_name()
-	 */
-	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
-		cpu_relax();
+
 	/*
 	 * Release reference to struct t10_alua_lu_gp * from all associated
 	 * struct se_device.
@@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);
 
Quoted 9 lines
drivers/target/target_core_alua.c void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp) · line 1249
Quoted 4 lines
drivers/target/target_core_alua.c +1 -1
-	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+	put_alua_lu_gp(lu_gp);
 }
 
The assumption that it's safe to 'Release reference to struct t10_alua_lu_gp * from all associated struct device' below the original cpu_relax(), while there are still other process contexts doing their respective put_alua_lu_gp() is totally wrong.
The only other spot is core_alua_do_port_transition, afaics. I think if it races with free_lu_gp, lu_gp will either be the old lu_gp (which no longer will have anything on lu_gp_mem_list) or will be default_lu_gp. If it's the old lu_gp then it iterates over an empty list, and then the lu_gp gets finally freed by the put() at the bottom.
Quoted 6 lines
Furthermore, allowing a configfs_group_ops->drop_item() to return while there are still active references from other process contexts means that the parent struct config_group is no longer referenced counted (eg:
configfs child is removed), and introduces a whole host of potential bugs.

Quoted 8 lines

Quoted 1 line
So that said, NAK on this patch.
I think some of the other patches used drop_item() and thus were bad, but in this one the existing code is already calling core_alua_free_lu_gp() from release().
Thoughts?

The problem with this patch and all of the other patches that follow the same logic is the false assumption that it's safe to return from configfs_group_operations->drop_item() before all references to the underlying data structure containing the associated struct config_group have been dropped.

In this particular case, target_core_alua_drop_lu_gp() -> config_item_put() -> target_core_alua_lu_gp_release() -> core_alua_free_lu_gp() is still being called from ->drop_item() via target_core_alua_lu_gp_ops->release(), so the same holds true here as well.

--nab

Andy Grover

author

2014-02-07 02:56 UTC #15 lore raw

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

Quoted 11 lines · Nicholas A. Bellinger

The problem with this patch and all of the other patches that follow the same logic is the false assumption that it's safe to return from configfs_group_operations->drop_item() before all references to the underlying data structure containing the associated struct config_group have been dropped.
In this particular case, target_core_alua_drop_lu_gp() -> config_item_put() -> target_core_alua_lu_gp_release() -> core_alua_free_lu_gp() is still being called from ->drop_item() via target_core_alua_lu_gp_ops->release(), so the same holds true here as well.

Yes exactly. That's why the configfs release() doesn't free the lu_gp, it just lowers the lu_gp refcount. release() being called doesn't mean the object is going away, it just means configfs is done with it.

If do_port_transition has incremented it in the meantime, the lu_gp won't be freed from the release() (because the underlying object's refcount will still be nonzero) but only when do_port_transition is done. In the normal case where there isn't a ref from do_port_transition, then it is safe to free the lu_gp from release -> put_alua_lu_gp -> release_alua_lu_gp -> kmem_cache_free.

-- Andy

Nicholas A. Bellinger

2014-02-07 19:17 UTC #16 lore raw

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

Quoted 17 lines · Andy Grover

Quoted 11 lines · Nicholas A. Bellinger
The problem with this patch and all of the other patches that follow the same logic is the false assumption that it's safe to return from configfs_group_operations->drop_item() before all references to the underlying data structure containing the associated struct config_group have been dropped.
In this particular case, target_core_alua_drop_lu_gp() -> config_item_put() -> target_core_alua_lu_gp_release() -> core_alua_free_lu_gp() is still being called from ->drop_item() via target_core_alua_lu_gp_ops->release(), so the same holds true here as well.
Yes exactly. That's why the configfs release() doesn't free the lu_gp, it just lowers the lu_gp refcount. release() being called doesn't mean the object is going away, it just means configfs is done with it.

Quoted 6 lines

If do_port_transition has incremented it in the meantime, the lu_gp won't be freed from the release() (because the underlying object's refcount will still be nonzero) but only when do_port_transition is done. In the normal case where there isn't a ref from do_port_transition, then it is safe to free the lu_gp from release -> put_alua_lu_gp -> release_alua_lu_gp -> kmem_cache_free.

It's still completely wrong to return from ->drop_item() before all references to the associated struct config_group have dropped because nothing prevents a parent struct config_group (and it's parent above that) from also being removed.

The whole point is that this type of parent/child reference counting comes for free with configfs, and patches that introduce changes that don't actually wait for individual references to drop, but instead lazily allow references to drop in the background at their leisure break this model.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #17 lore raw

PATCH 7/32 target: Convert struct alua_lu_gp_member to kref

In reply to Andy Grover

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   24 +++++++++++++++---------
 include/target/target_core_base.h |    2 +-
 2 files changed, 16 insertions(+), 10 deletions(-)

drivers/target/target_core_alua.c +15 -9

index 8c01ade..fe2eada 100644
@@ -64,6 +64,16 @@ static void release_alua_lu_gp(struct kref *ref)
 #define get_alua_lu_gp(x) kref_get(&x->refcount)
 #define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
 
+static void release_alua_lu_gp_mem(struct kref *ref)
+{
+	struct t10_alua_lu_gp_member *lu_gp_mem = container_of(ref, struct t10_alua_lu_gp_member, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
+}
+
+#define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
+#define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -937,8 +947,7 @@ int core_alua_do_port_transition(
 				lu_gp_mem_node) {
 
 		dev = lu_gp_mem->lu_gp_mem_dev;
-		atomic_inc(&lu_gp_mem->lu_gp_mem_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_alua_lu_gp_mem(lu_gp_mem);
 		spin_unlock(&lu_gp->lu_gp_lock);
 
 		spin_lock(&dev->t10_alua.tg_pt_gps_lock);

@@ -983,8 +992,7 @@ int core_alua_do_port_transition(
 		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
 		spin_lock(&lu_gp->lu_gp_lock);
-		atomic_dec(&lu_gp_mem->lu_gp_mem_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_alua_lu_gp_mem(lu_gp_mem);
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);

@@ -1186,7 +1194,8 @@ core_alua_allocate_lu_gp_mem(struct se_device *dev)
 	}
 	INIT_LIST_HEAD(&lu_gp_mem->lu_gp_mem_node);
 	spin_lock_init(&lu_gp_mem->lu_gp_mem_lock);
-	atomic_set(&lu_gp_mem->lu_gp_mem_ref_cnt, 0);
+
+	kref_init(&lu_gp_mem->refcount);
 
 	lu_gp_mem->lu_gp_mem_dev = dev;
 	dev->dev_alua_lu_gp_mem = lu_gp_mem;

@@ -1256,9 +1265,6 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
 	if (!lu_gp_mem)
 		return;
 
-	while (atomic_read(&lu_gp_mem->lu_gp_mem_ref_cnt))
-		cpu_relax();
-
 	spin_lock(&lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = lu_gp_mem->lu_gp;
 	if (lu_gp) {

@@ -1273,7 +1279,7 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
 	}
 	spin_unlock(&lu_gp_mem->lu_gp_mem_lock);
 
-	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
+	put_alua_lu_gp_mem(lu_gp_mem);
 }
 
 struct t10_alua_lu_gp *core_alua_get_lu_gp_by_name(const char *name)

include/target/target_core_base.h +1 -1

index 5bdf0d5..2d56eaf 100644
@@ -273,7 +273,7 @@ struct t10_alua_lu_gp {
 
 struct t10_alua_lu_gp_member {
 	bool lu_gp_assoc;
-	atomic_t lu_gp_mem_ref_cnt;
+	struct kref refcount;
 	spinlock_t lu_gp_mem_lock;
 	struct t10_alua_lu_gp *lu_gp;
 	struct se_device *lu_gp_mem_dev;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 20:55 UTC #18 lore raw

Re: [PATCH 07/32] target: Convert struct alua_lu_gp_member to kref

Quoted 68 lines · Andy Grover

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   24 +++++++++++++++---------
 include/target/target_core_base.h |    2 +-
 2 files changed, 16 insertions(+), 10 deletions(-)

drivers/target/target_core_alua.c +14 -8

index 8c01ade..fe2eada 100644
@@ -64,6 +64,16 @@ static void release_alua_lu_gp(struct kref *ref)
 #define get_alua_lu_gp(x) kref_get(&x->refcount)
 #define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
 
+static void release_alua_lu_gp_mem(struct kref *ref)
+{
+	struct t10_alua_lu_gp_member *lu_gp_mem = container_of(ref, struct t10_alua_lu_gp_member, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
+}
+
+#define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
+#define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -937,8 +947,7 @@ int core_alua_do_port_transition(
 				lu_gp_mem_node) {
 
 		dev = lu_gp_mem->lu_gp_mem_dev;
-		atomic_inc(&lu_gp_mem->lu_gp_mem_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_alua_lu_gp_mem(lu_gp_mem);
 		spin_unlock(&lu_gp->lu_gp_lock);
 
 		spin_lock(&dev->t10_alua.tg_pt_gps_lock);

@@ -983,8 +992,7 @@ int core_alua_do_port_transition(
 		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
 		spin_lock(&lu_gp->lu_gp_lock);
-		atomic_dec(&lu_gp_mem->lu_gp_mem_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_alua_lu_gp_mem(lu_gp_mem);
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);

@@ -1186,7 +1194,8 @@ core_alua_allocate_lu_gp_mem(struct se_device *dev)
 	}
 	INIT_LIST_HEAD(&lu_gp_mem->lu_gp_mem_node);
 	spin_lock_init(&lu_gp_mem->lu_gp_mem_lock);
-	atomic_set(&lu_gp_mem->lu_gp_mem_ref_cnt, 0);
+
+	kref_init(&lu_gp_mem->refcount);
 
 	lu_gp_mem->lu_gp_mem_dev = dev;
 	dev->dev_alua_lu_gp_mem = lu_gp_mem;

@@ -1256,9 +1265,6 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
 	if (!lu_gp_mem)
 		return;
 
-	while (atomic_read(&lu_gp_mem->lu_gp_mem_ref_cnt))
-		cpu_relax();
-
 	spin_lock(&lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = lu_gp_mem->lu_gp;
 	if (lu_gp) {

drivers/target/target_core_alua.c void core_alua_free_lu_gp_mem(struct se_device *dev) · line 1284

Quoted 7 lines

drivers/target/target_core_alua.c +1 -1

@@ -1273,7 +1279,7 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
 	}
 	spin_unlock(&lu_gp_mem->lu_gp_mem_lock);
 
-	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
+	put_alua_lu_gp_mem(lu_gp_mem);
 }

Same problem on this one as well. Assuming that it's safe to clear the lu_gp_mem->lu_gp association while there are still active references is wrong.

Also, considering that core_alua_free_lu_gp_mem() is called from target_free_device(), it absolutely needs to wait until the reference drops before target_free_device() completes.

So that being the case, NAK.

--nab

Andy Grover

author

2013-12-13 23:58 UTC #19 lore raw

PATCH 8/32 target: Convert tg_pt_gp_ref_cnt to kref

In reply to Andy Grover

Use kref to handle reference counting.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   39 +++++++++++++++++-------------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 19 insertions(+), 22 deletions(-)

drivers/target/target_core_alua.c +18 ~~-21~~

index fe2eada..ba7b3d6 100644
@@ -74,6 +74,16 @@ static void release_alua_lu_gp_mem(struct kref *ref)
 #define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
 #define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
 
+static void release_alua_tg_pt_gp(struct kref *ref)
+{
+	struct t10_alua_tg_pt_gp *tg_pt_gp = container_of(ref, struct t10_alua_tg_pt_gp, refcount);
+
+	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
+}
+
+#define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
+#define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -327,8 +337,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 				if (tg_pt_id != tg_pt_gp->tg_pt_gp_id)
 					continue;
 
-				atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
-				smp_mb__after_atomic_inc();
+				get_alua_tg_pt_gp(tg_pt_gp);
 
 				spin_unlock(&dev->t10_alua.tg_pt_gps_lock);

@@ -338,8 +347,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 					found = true;
 
 				spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-				atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
-				smp_mb__after_atomic_dec();
+				put_alua_tg_pt_gp(tg_pt_gp);
 				break;
 			}
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);

@@ -975,8 +983,7 @@ int core_alua_do_port_transition(
 				port = NULL;
 				nacl = NULL;
 			}
-			atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
-			smp_mb__after_atomic_inc();
+			get_alua_tg_pt_gp(tg_pt_gp);
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 			/*
 			 * core_alua_do_transition_tg_pt() will always return

@@ -986,8 +993,7 @@ int core_alua_do_port_transition(
 					nacl, md_buf, new_state, explicit);
 
 			spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-			atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
-			smp_mb__after_atomic_dec();
+			put_alua_tg_pt_gp(tg_pt_gp);
 		}
 		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);

@@ -1354,7 +1360,7 @@ struct t10_alua_tg_pt_gp *core_alua_allocate_tg_pt_gp(struct se_device *dev,
 	INIT_LIST_HEAD(&tg_pt_gp->tg_pt_gp_mem_list);
 	mutex_init(&tg_pt_gp->tg_pt_gp_md_mutex);
 	spin_lock_init(&tg_pt_gp->tg_pt_gp_lock);
-	atomic_set(&tg_pt_gp->tg_pt_gp_ref_cnt, 0);
+	kref_init(&tg_pt_gp->refcount);
 	tg_pt_gp->tg_pt_gp_dev = dev;
 	tg_pt_gp->tg_pt_gp_md_buf_len = ALUA_MD_BUF_LEN;
 	atomic_set(&tg_pt_gp->tg_pt_gp_alua_access_state,

@@ -1485,15 +1491,6 @@ void core_alua_free_tg_pt_gp(
 	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
 	/*
-	 * Allow a struct t10_alua_tg_pt_gp_member * referenced by
-	 * core_alua_get_tg_pt_gp_by_name() in
-	 * target_core_configfs.c:target_core_store_alua_tg_pt_gp()
-	 * to be released with core_alua_put_tg_pt_gp_from_name().
-	 */
-	while (atomic_read(&tg_pt_gp->tg_pt_gp_ref_cnt))
-		cpu_relax();
-
-	/*
 	 * Release reference to struct t10_alua_tg_pt_gp from all associated
 	 * struct se_port.
 	 */

@@ -1527,7 +1524,7 @@ void core_alua_free_tg_pt_gp(
 	}
 	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 
-	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
+	put_alua_tg_pt_gp(tg_pt_gp);
 }
 
 void core_alua_free_tg_pt_gp_mem(struct se_port *port)

@@ -1572,7 +1569,7 @@ static struct t10_alua_tg_pt_gp *core_alua_get_tg_pt_gp_by_name(
 			continue;
 		ci = &tg_pt_gp->tg_pt_gp_group.cg_item;
 		if (!strcmp(config_item_name(ci), name)) {
-			atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
+			get_alua_tg_pt_gp(tg_pt_gp);
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 			return tg_pt_gp;
 		}

@@ -1588,7 +1585,7 @@ static void core_alua_put_tg_pt_gp_from_name(
 	struct se_device *dev = tg_pt_gp->tg_pt_gp_dev;
 
 	spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-	atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
+	put_alua_tg_pt_gp(tg_pt_gp);
 	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 }

include/target/target_core_base.h +1 -1

index 2d56eaf..58b04f9 100644
@@ -296,7 +296,7 @@ struct t10_alua_tg_pt_gp {
 	u32	tg_pt_gp_md_buf_len;
 	u32	tg_pt_gp_members;
 	atomic_t tg_pt_gp_alua_access_state;
-	atomic_t tg_pt_gp_ref_cnt;
+	struct kref refcount;
 	spinlock_t tg_pt_gp_lock;
 	struct mutex tg_pt_gp_md_mutex;
 	struct se_device *tg_pt_gp_dev;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 20:59 UTC #20 lore raw

Re: [PATCH 08/32] target: Convert tg_pt_gp_ref_cnt to kref

Quoted 95 lines · Andy Grover

Use kref to handle reference counting.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   39 +++++++++++++++++-------------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 19 insertions(+), 22 deletions(-)

drivers/target/target_core_alua.c +15 ~~-18~~

index fe2eada..ba7b3d6 100644
@@ -74,6 +74,16 @@ static void release_alua_lu_gp_mem(struct kref *ref)
 #define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
 #define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
 
+static void release_alua_tg_pt_gp(struct kref *ref)
+{
+	struct t10_alua_tg_pt_gp *tg_pt_gp = container_of(ref, struct t10_alua_tg_pt_gp, refcount);
+
+	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
+}
+
+#define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
+#define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -327,8 +337,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 				if (tg_pt_id != tg_pt_gp->tg_pt_gp_id)
 					continue;
 
-				atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
-				smp_mb__after_atomic_inc();
+				get_alua_tg_pt_gp(tg_pt_gp);
 
 				spin_unlock(&dev->t10_alua.tg_pt_gps_lock);

@@ -338,8 +347,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 					found = true;
 
 				spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-				atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
-				smp_mb__after_atomic_dec();
+				put_alua_tg_pt_gp(tg_pt_gp);
 				break;
 			}
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);

@@ -975,8 +983,7 @@ int core_alua_do_port_transition(
 				port = NULL;
 				nacl = NULL;
 			}
-			atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
-			smp_mb__after_atomic_inc();
+			get_alua_tg_pt_gp(tg_pt_gp);
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 			/*
 			 * core_alua_do_transition_tg_pt() will always return

@@ -986,8 +993,7 @@ int core_alua_do_port_transition(
 					nacl, md_buf, new_state, explicit);
 
 			spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-			atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
-			smp_mb__after_atomic_dec();
+			put_alua_tg_pt_gp(tg_pt_gp);
 		}
 		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);

@@ -1354,7 +1360,7 @@ struct t10_alua_tg_pt_gp *core_alua_allocate_tg_pt_gp(struct se_device *dev,
 	INIT_LIST_HEAD(&tg_pt_gp->tg_pt_gp_mem_list);
 	mutex_init(&tg_pt_gp->tg_pt_gp_md_mutex);
 	spin_lock_init(&tg_pt_gp->tg_pt_gp_lock);
-	atomic_set(&tg_pt_gp->tg_pt_gp_ref_cnt, 0);
+	kref_init(&tg_pt_gp->refcount);
 	tg_pt_gp->tg_pt_gp_dev = dev;
 	tg_pt_gp->tg_pt_gp_md_buf_len = ALUA_MD_BUF_LEN;
 	atomic_set(&tg_pt_gp->tg_pt_gp_alua_access_state,

@@ -1485,15 +1491,6 @@ void core_alua_free_tg_pt_gp(
 	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
 	/*
-	 * Allow a struct t10_alua_tg_pt_gp_member * referenced by
-	 * core_alua_get_tg_pt_gp_by_name() in
-	 * target_core_configfs.c:target_core_store_alua_tg_pt_gp()
-	 * to be released with core_alua_put_tg_pt_gp_from_name().
-	 */
-	while (atomic_read(&tg_pt_gp->tg_pt_gp_ref_cnt))
-		cpu_relax();
-
-	/*
 	 * Release reference to struct t10_alua_tg_pt_gp from all associated
 	 * struct se_port.
 	 */

drivers/target/target_core_alua.c void core_alua_free_tg_pt_gp( · line 1529

Quoted 7 lines

drivers/target/target_core_alua.c +1 -1

@@ -1527,7 +1524,7 @@ void core_alua_free_tg_pt_gp(
 	}
 	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 
-	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
+	put_alua_tg_pt_gp(tg_pt_gp);
 }

Same problem here again. It's incorrect to assume that it's safe to 'Relase reference to struct t10_alua_tg_pt_gp from all associated struct se_port', while other process contexts are referencing this memory.

Also same problem with configfs reference counting, where a parent struct config_group can be removed since it's child struct config_group has been dropped, but still be referenced elsewhere.

NAK.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #21 lore raw

PATCH 9/32 target: convert tg_pt_gp_mem_ref_cnt to kref

In reply to Andy Grover

Use kref to handle reference counting.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   23 ++++++++++++++---------
 include/target/target_core_base.h |    2 +-
 2 files changed, 15 insertions(+), 10 deletions(-)

drivers/target/target_core_alua.c +14 -9

index ba7b3d6..4ee08a2 100644
@@ -84,6 +84,16 @@ static void release_alua_tg_pt_gp(struct kref *ref)
 #define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
 #define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
 
+static void release_alua_tg_pt_gp_mem(struct kref *ref)
+{
+	struct t10_alua_tg_pt_gp_member *tg_pt_gp_mem = container_of(ref, struct t10_alua_tg_pt_gp_member, refcount);
+
+	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
+}
+
+#define get_alua_tg_pt_gp_mem(x) kref_get(&x->refcount)
+#define put_alua_tg_pt_gp_mem(x) kref_put(&x->refcount, release_alua_tg_pt_gp_mem)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -834,8 +844,7 @@ static int core_alua_do_transition_tg_pt(
 		 * every I_T nexus other than the I_T nexus on which the SET
 		 * TARGET PORT GROUPS command
 		 */
-		atomic_inc(&mem->tg_pt_gp_mem_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_alua_tg_pt_gp_mem(mem);
 		spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 
 		spin_lock_bh(&port->sep_alua_lock);

@@ -861,8 +870,7 @@ static int core_alua_do_transition_tg_pt(
 		spin_unlock_bh(&port->sep_alua_lock);
 
 		spin_lock(&tg_pt_gp->tg_pt_gp_lock);
-		atomic_dec(&mem->tg_pt_gp_mem_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_alua_tg_pt_gp_mem(mem);
 	}
 	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 	/*

@@ -1463,7 +1471,7 @@ struct t10_alua_tg_pt_gp_member *core_alua_allocate_tg_pt_gp_mem(
 	}
 	INIT_LIST_HEAD(&tg_pt_gp_mem->tg_pt_gp_mem_node);
 	spin_lock_init(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
-	atomic_set(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt, 0);
+	kref_init(&tg_pt_gp_mem->refcount);
 
 	tg_pt_gp_mem->tg_pt = port;
 	port->sep_alua_tg_pt_gp_mem = tg_pt_gp_mem;

@@ -1536,9 +1544,6 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
 	if (!tg_pt_gp_mem)
 		return;
 
-	while (atomic_read(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt))
-		cpu_relax();
-
 	spin_lock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
 	tg_pt_gp = tg_pt_gp_mem->tg_pt_gp;
 	if (tg_pt_gp) {

@@ -1553,7 +1558,7 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
 	}
 	spin_unlock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
 
-	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
+	put_alua_tg_pt_gp_mem(tg_pt_gp_mem);
 }
 
 static struct t10_alua_tg_pt_gp *core_alua_get_tg_pt_gp_by_name(

include/target/target_core_base.h +1 -1

index 58b04f9..6c517cd 100644
@@ -307,7 +307,7 @@ struct t10_alua_tg_pt_gp {
 
 struct t10_alua_tg_pt_gp_member {
 	bool tg_pt_gp_assoc;
-	atomic_t tg_pt_gp_mem_ref_cnt;
+	struct kref refcount;
 	spinlock_t tg_pt_gp_mem_lock;
 	struct t10_alua_tg_pt_gp *tg_pt_gp;
 	struct se_port *tg_pt;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:08 UTC #22 lore raw

Re: [PATCH 09/32] target: convert tg_pt_gp_mem_ref_cnt to kref

Quoted 69 lines · Andy Grover

Use kref to handle reference counting.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_alua.c |   23 ++++++++++++++---------
 include/target/target_core_base.h |    2 +-
 2 files changed, 15 insertions(+), 10 deletions(-)

drivers/target/target_core_alua.c +13 -8

index ba7b3d6..4ee08a2 100644
@@ -84,6 +84,16 @@ static void release_alua_tg_pt_gp(struct kref *ref)
 #define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
 #define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
 
+static void release_alua_tg_pt_gp_mem(struct kref *ref)
+{
+	struct t10_alua_tg_pt_gp_member *tg_pt_gp_mem = container_of(ref, struct t10_alua_tg_pt_gp_member, refcount);
+
+	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
+}
+
+#define get_alua_tg_pt_gp_mem(x) kref_get(&x->refcount)
+#define put_alua_tg_pt_gp_mem(x) kref_put(&x->refcount, release_alua_tg_pt_gp_mem)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *

@@ -834,8 +844,7 @@ static int core_alua_do_transition_tg_pt(
 		 * every I_T nexus other than the I_T nexus on which the SET
 		 * TARGET PORT GROUPS command
 		 */
-		atomic_inc(&mem->tg_pt_gp_mem_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_alua_tg_pt_gp_mem(mem);
 		spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 
 		spin_lock_bh(&port->sep_alua_lock);

@@ -861,8 +870,7 @@ static int core_alua_do_transition_tg_pt(
 		spin_unlock_bh(&port->sep_alua_lock);
 
 		spin_lock(&tg_pt_gp->tg_pt_gp_lock);
-		atomic_dec(&mem->tg_pt_gp_mem_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_alua_tg_pt_gp_mem(mem);
 	}
 	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 	/*

@@ -1463,7 +1471,7 @@ struct t10_alua_tg_pt_gp_member *core_alua_allocate_tg_pt_gp_mem(
 	}
 	INIT_LIST_HEAD(&tg_pt_gp_mem->tg_pt_gp_mem_node);
 	spin_lock_init(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
-	atomic_set(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt, 0);
+	kref_init(&tg_pt_gp_mem->refcount);
 
 	tg_pt_gp_mem->tg_pt = port;
 	port->sep_alua_tg_pt_gp_mem = tg_pt_gp_mem;

@@ -1536,9 +1544,6 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
 	if (!tg_pt_gp_mem)
 		return;
 
-	while (atomic_read(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt))
-		cpu_relax();
-
 	spin_lock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
 	tg_pt_gp = tg_pt_gp_mem->tg_pt_gp;
 	if (tg_pt_gp) {

drivers/target/target_core_alua.c void core_alua_free_tg_pt_gp_mem(struct se_port *port) · line 1562

Quoted 6 lines

drivers/target/target_core_alua.c +1 -1

@@ -1553,7 +1558,7 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
 	}
 	spin_unlock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
 
-	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
+	put_alua_tg_pt_gp_mem(tg_pt_gp_mem);
 }

Same issue here between original cpu_relax() location and put_alua_tg_pt_gp_mem(). It's not safe to drop the tg_pt_gp_mem->tg_pt_gp association while there are still active references in other process contexts.

Also, the same problem exists with configfs reference counting here, where core_alua_free_tg_pt_gp_mem() is called from core_dev_unexport() -> core_release_port(), where the struct lun could be immediately be reused, or the struct device be removed before the final put_alua_tg_pt_gp() is called from the other process contexts.

NAK.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #23 lore raw

PATCH 10/32 target: Change sep_tg_pt_ref_cnt to use kref

In reply to Andy Grover

Use the kernel's std kref for refcounting.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c   |   12 ++----------
 drivers/target/target_core_internal.h |   10 ++++++++++
 drivers/target/target_core_pr.c       |   12 ++++--------
 include/target/target_core_base.h     |    2 +-
 4 files changed, 17 insertions(+), 19 deletions(-)

drivers/target/target_core_device.c +2 ~~-10~~

index 3350467..1954b0f 100644
@@ -472,6 +472,7 @@ static struct se_port *core_alloc_port(struct se_device *dev)
 	atomic_set(&port->sep_tg_pt_secondary_offline, 0);
 	spin_lock_init(&port->sep_alua_lock);
 	mutex_init(&port->sep_tg_pt_md_mutex);
+	kref_init(&port->refcount);
 
 	spin_lock(&dev->se_port_lock);
 	if (dev->dev_port_count == 0x0000ffff) {

@@ -555,20 +556,11 @@ static void core_export_port(
 static void core_release_port(struct se_device *dev, struct se_port *port)
 	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
 {
-	/*
-	 * Wait for any port reference for PR ALL_TG_PT=1 operation
-	 * to complete in __core_scsi3_alloc_registration()
-	 */
-	spin_unlock(&dev->se_port_lock);
-	if (atomic_read(&port->sep_tg_pt_ref_cnt))
-		cpu_relax();
-	spin_lock(&dev->se_port_lock);
-
 	core_alua_free_tg_pt_gp_mem(port);
 
 	list_del(&port->sep_node);
 	dev->dev_port_count--;
-	kfree(port);
+	put_port(port);
 }
 
 int core_dev_export(

drivers/target/target_core_internal.h +10 -0

index 47b63b0..b11512c 100644
@@ -60,6 +60,16 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name);
 int	target_configure_device(struct se_device *dev);
 void	target_free_device(struct se_device *);
 
+static inline void release_port(struct kref *ref)
+{
+	struct se_port *port = container_of(ref, struct se_port, refcount);
+
+	kfree(port);
+}
+
+#define get_port(x) kref_get(&x->refcount)
+#define put_port(x) kref_put(&x->refcount, release_port)
+
 /* target_core_hba.c */
 struct se_hba *core_alloc_hba(const char *, u32, u32);
 int	core_delete_hba(struct se_hba *);

drivers/target/target_core_pr.c +4 -8

index d141c7f..fc6029b 100644
@@ -674,8 +674,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 	 */
 	spin_lock(&dev->se_port_lock);
 	list_for_each_entry_safe(port, port_tmp, &dev->dev_sep_list, sep_node) {
-		atomic_inc(&port->sep_tg_pt_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_port(port);
 		spin_unlock(&dev->se_port_lock);
 
 		spin_lock_bh(&port->sep_alua_lock);

@@ -722,8 +721,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 			if (ret < 0) {
 				pr_err("core_scsi3_lunacl_depend"
 						"_item() failed\n");
-				atomic_dec(&port->sep_tg_pt_ref_cnt);
-				smp_mb__after_atomic_dec();
+				put_port(port);
 				atomic_dec(&deve_tmp->pr_ref_count);
 				smp_mb__after_atomic_dec();
 				goto out;

@@ -739,8 +737,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 						nacl_tmp, deve_tmp, NULL,
 						sa_res_key, all_tg_pt, aptpl);
 			if (!pr_reg_atp) {
-				atomic_dec(&port->sep_tg_pt_ref_cnt);
-				smp_mb__after_atomic_dec();
+				put_port(port);
 				atomic_dec(&deve_tmp->pr_ref_count);
 				smp_mb__after_atomic_dec();
 				core_scsi3_lunacl_undepend_item(deve_tmp);

@@ -754,8 +751,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 		spin_unlock_bh(&port->sep_alua_lock);
 
 		spin_lock(&dev->se_port_lock);
-		atomic_dec(&port->sep_tg_pt_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_port(port);
 	}
 	spin_unlock(&dev->se_port_lock);

include/target/target_core_base.h +1 -1

index 6c517cd..f0bc5c1 100644
@@ -773,7 +773,7 @@ struct se_port {
 	/* Used for ALUA Target Port Groups membership */
 	atomic_t	sep_tg_pt_secondary_offline;
 	/* Used for PR ALL_TG_PT=1 */
-	atomic_t	sep_tg_pt_ref_cnt;
+	struct kref refcount;
 	spinlock_t	sep_alua_lock;
 	struct mutex	sep_tg_pt_md_mutex;
 	struct t10_alua_tg_pt_gp_member *sep_alua_tg_pt_gp_mem;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:10 UTC #24 lore raw

Re: [PATCH 10/32] target: Change sep_tg_pt_ref_cnt to use kref

Quoted 33 lines · Andy Grover

Use the kernel's std kref for refcounting.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c   |   12 ++----------
 drivers/target/target_core_internal.h |   10 ++++++++++
 drivers/target/target_core_pr.c       |   12 ++++--------
 include/target/target_core_base.h     |    2 +-
 4 files changed, 17 insertions(+), 19 deletions(-)

drivers/target/target_core_device.c +1 -7

index 3350467..1954b0f 100644
@@ -472,6 +472,7 @@ static struct se_port *core_alloc_port(struct se_device *dev)
 	atomic_set(&port->sep_tg_pt_secondary_offline, 0);
 	spin_lock_init(&port->sep_alua_lock);
 	mutex_init(&port->sep_tg_pt_md_mutex);
+	kref_init(&port->refcount);
 
 	spin_lock(&dev->se_port_lock);
 	if (dev->dev_port_count == 0x0000ffff) {

@@ -555,20 +556,11 @@ static void core_export_port(
 static void core_release_port(struct se_device *dev, struct se_port *port)
 	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
 {
-	/*
-	 * Wait for any port reference for PR ALL_TG_PT=1 operation
-	 * to complete in __core_scsi3_alloc_registration()
-	 */
-	spin_unlock(&dev->se_port_lock);
-	if (atomic_read(&port->sep_tg_pt_ref_cnt))
-		cpu_relax();

drivers/target/target_core_device.c static void core_export_port( · line 565

Quoted 10 lines

drivers/target/target_core_device.c +1 -3

-	spin_lock(&dev->se_port_lock);
-
 	core_alua_free_tg_pt_gp_mem(port);
 
 	list_del(&port->sep_node);
 	dev->dev_port_count--;
-	kfree(port);
+	put_port(port);
 }

Same problem yet again. It's not safe to release the se_port from it's tg_pt_gp_mem association, while there are active references from other process contexts.

Also the same issue with the configfs child / parent reference here, where returning (instead of waiting for references to drop) introduces a whole new host of reference counting issues.

NAK.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #25 lore raw

PATCH 11/32 target: Convert se_dev_entry to kref

In reply to Andy Grover

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c   |   12 +++-------
 drivers/target/target_core_internal.h |   12 ++++++++++
 drivers/target/target_core_pr.c       |   38 ++++++++++++--------------------
 drivers/target/target_core_tpg.c      |    2 +-
 include/target/target_core_base.h     |    3 +-
 5 files changed, 32 insertions(+), 35 deletions(-)

drivers/target/target_core_device.c +4 -8

index 1954b0f..295b5e4 100644
@@ -224,8 +224,7 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
 		if (port->sep_rtpi != rtpi)
 			continue;
 
-		atomic_inc(&deve->pr_ref_count);
-		smp_mb__after_atomic_inc();
+		get_deve(deve);
 		spin_unlock_irq(&nacl->device_list_lock);
 
 		return deve;

@@ -401,12 +400,6 @@ int core_disable_device_list_for_node(
 	spin_lock_bh(&port->sep_alua_lock);
 	list_del(&deve->alua_port_node);
 	spin_unlock_bh(&port->sep_alua_lock);
-	/*
-	 * Wait for any in process SPEC_I_PT=1 or REGISTER_AND_MOVE
-	 * PR operation to complete.
-	 */
-	while (atomic_read(&deve->pr_ref_count) != 0)
-		cpu_relax();
 
 	spin_lock_irq(&nacl->device_list_lock);
 	/*

@@ -421,6 +414,9 @@ int core_disable_device_list_for_node(
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
+
+	put_deve(deve);
+
 	return 0;
 }

drivers/target/target_core_internal.h +12 -0

index b11512c..fddf44c 100644
@@ -93,6 +93,18 @@ int	core_tpg_post_addlun(struct se_portal_group *, struct se_lun *,
 struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
 int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
 
+static inline void release_deve(struct kref *kref)
+{
+	struct se_dev_entry *deve = container_of(kref,
+				struct se_dev_entry, refcount);
+
+	/* doesn't really get freed because device_list is a static array */
+	deve->lun_flags &= ~TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
+}
+
+#define get_deve(x) kref_get(&x->refcount)
+#define put_deve(x) kref_put(&x->refcount, release_deve)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;

drivers/target/target_core_pr.c +14 ~~-24~~

index fc6029b..9155df0 100644
@@ -708,8 +708,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 			if (strcmp(nacl->initiatorname, nacl_tmp->initiatorname))
 				continue;
 
-			atomic_inc(&deve_tmp->pr_ref_count);
-			smp_mb__after_atomic_inc();
+			get_deve(deve_tmp);
 			spin_unlock_bh(&port->sep_alua_lock);
 			/*
 			 * Grab a configfs group dependency that is released

@@ -722,8 +721,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 				pr_err("core_scsi3_lunacl_depend"
 						"_item() failed\n");
 				put_port(port);
-				atomic_dec(&deve_tmp->pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_deve(deve_tmp);
 				goto out;
 			}
 			/*

@@ -738,8 +736,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 						sa_res_key, all_tg_pt, aptpl);
 			if (!pr_reg_atp) {
 				put_port(port);
-				atomic_dec(&deve_tmp->pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_deve(deve_tmp);
 				core_scsi3_lunacl_undepend_item(deve_tmp);
 				goto out;
 			}

@@ -1399,22 +1396,17 @@ static void core_scsi3_lunacl_undepend_item(struct se_dev_entry *se_deve)
 	struct se_lun_acl *lun_acl = se_deve->se_lun_acl;
 	struct se_node_acl *nacl;
 	struct se_portal_group *tpg;
-	/*
-	 * For nacl->dynamic_node_acl=1
-	 */
-	if (!lun_acl) {
-		atomic_dec(&se_deve->pr_ref_count);
-		smp_mb__after_atomic_dec();
-		return;
-	}
-	nacl = lun_acl->se_lun_nacl;
-	tpg = nacl->se_tpg;
 
-	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
-			&lun_acl->se_lun_group.cg_item);
+	/* for non-demo-mode */
+	if (lun_acl) {
+		nacl = lun_acl->se_lun_nacl;
+		tpg = nacl->se_tpg;
 
-	atomic_dec(&se_deve->pr_ref_count);
-	smp_mb__after_atomic_dec();
+		configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
+				       &lun_acl->se_lun_group.cg_item);
+	}
+
+	put_deve(se_deve);
 }
 
 static sense_reason_t

@@ -1642,8 +1634,7 @@ core_scsi3_decode_spec_i_port(
 		if (core_scsi3_lunacl_depend_item(dest_se_deve)) {
 			pr_err("core_scsi3_lunacl_depend_item()"
 					" failed\n");
-			atomic_dec(&dest_se_deve->pr_ref_count);
-			smp_mb__after_atomic_dec();
+			put_deve(dest_se_deve);
 			core_scsi3_nodeacl_undepend_item(dest_node_acl);
 			core_scsi3_tpg_undepend_item(dest_tpg);
 			ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;

@@ -3306,8 +3297,7 @@ after_iport_check:
 
 	if (core_scsi3_lunacl_depend_item(dest_se_deve)) {
 		pr_err("core_scsi3_lunacl_depend_item() failed\n");
-		atomic_dec(&dest_se_deve->pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_deve(dest_se_deve);
 		dest_se_deve = NULL;
 		ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 		goto out;

drivers/target/target_core_tpg.c +1 -1

index 1fd90fc..8771b23 100644
@@ -237,7 +237,7 @@ static int core_create_device_list_for_node(struct se_node_acl *nacl)
 		deve = nacl->device_list[i];
 
 		atomic_set(&deve->ua_count, 0);
-		atomic_set(&deve->pr_ref_count, 0);
+		kref_init(&deve->refcount);
 		spin_lock_init(&deve->ua_lock);
 		INIT_LIST_HEAD(&deve->alua_port_node);
 		INIT_LIST_HEAD(&deve->ua_list);

include/target/target_core_base.h +1 -2

index f0bc5c1..6bf1da9 100644
@@ -587,8 +587,7 @@ struct se_dev_entry {
 	u64			read_bytes;
 	u64			write_bytes;
 	atomic_t		ua_count;
-	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
-	atomic_t		pr_ref_count;
+	struct kref		refcount;
 	struct se_lun_acl	*se_lun_acl;
 	spinlock_t		ua_lock;
 	struct se_lun		*se_lun;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:15 UTC #26 lore raw

Re: [PATCH 11/32] target: Convert se_dev_entry to kref

Quoted 37 lines · Andy Grover

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c   |   12 +++-------
 drivers/target/target_core_internal.h |   12 ++++++++++
 drivers/target/target_core_pr.c       |   38 ++++++++++++--------------------
 drivers/target/target_core_tpg.c      |    2 +-
 include/target/target_core_base.h     |    3 +-
 5 files changed, 32 insertions(+), 35 deletions(-)

drivers/target/target_core_device.c +1 -8

index 1954b0f..295b5e4 100644
@@ -224,8 +224,7 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
 		if (port->sep_rtpi != rtpi)
 			continue;
 
-		atomic_inc(&deve->pr_ref_count);
-		smp_mb__after_atomic_inc();
+		get_deve(deve);
 		spin_unlock_irq(&nacl->device_list_lock);
 
 		return deve;

@@ -401,12 +400,6 @@ int core_disable_device_list_for_node(
 	spin_lock_bh(&port->sep_alua_lock);
 	list_del(&deve->alua_port_node);
 	spin_unlock_bh(&port->sep_alua_lock);
-	/*
-	 * Wait for any in process SPEC_I_PT=1 or REGISTER_AND_MOVE
-	 * PR operation to complete.
-	 */
-	while (atomic_read(&deve->pr_ref_count) != 0)
-		cpu_relax();
 
 	spin_lock_irq(&nacl->device_list_lock);
 	/*

drivers/target/target_core_device.c int core_disable_device_list_for_node( · line 421

Quoted 8 lines

drivers/target/target_core_device.c +3 -0

@@ -421,6 +414,9 @@ int core_disable_device_list_for_node(
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
+
+	put_deve(deve);
+
 	return 0;
 }

Same issue yet again. It's not safe to release all of the se_dev_entry pointers, UAs, and PR associations until the other references from different process contexts have been released.

Also the issue with reuse of se_dev_entry also exists here. configfs_group_operations->drop_item() callbacks should not return until all references have completed.

NAK.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #27 lore raw

PATCH 12/32 target: Convert t10_pr_registration to kref

In reply to Andy Grover

Free struct when kref becomes 0.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_pr.c   |   99 +++++++++++++++---------------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 42 insertions(+), 59 deletions(-)

drivers/target/target_core_pr.c +41 ~~-58~~

index 9155df0..17a4c3b 100644
@@ -75,6 +75,17 @@ enum preempt_type {
 	PREEMPT_AND_ABORT,
 };
 
+static void release_pr_reg(struct kref *kref)
+{
+	struct t10_pr_registration *pr_reg = container_of(kref,
+							  struct t10_pr_registration, refcount);
+
+	kmem_cache_free(t10_pr_reg_cache, pr_reg);
+}
+
+#define get_pr_reg(x) kref_get(&x->refcount)
+#define put_pr_reg(x) kref_put(&x->refcount, release_pr_reg)
+
 static void __core_scsi3_complete_pro_release(struct se_device *, struct se_node_acl *,
 			struct t10_pr_registration *, int);

@@ -109,7 +120,6 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
 
 static struct t10_pr_registration *core_scsi3_locate_pr_reg(struct se_device *,
 					struct se_node_acl *, struct se_session *);
-static void core_scsi3_put_pr_reg(struct t10_pr_registration *);
 
 static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
 {

@@ -144,17 +154,17 @@ static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
 		 * as defined in SPC-2.
 		 */
 		if (pr_reg->pr_res_holder) {
-			core_scsi3_put_pr_reg(pr_reg);
+			put_pr_reg(pr_reg);
 			return 1;
 		}
 		if ((pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY) ||
 		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY) ||
 		    (pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_ALLREG) ||
 		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_ALLREG)) {
-			core_scsi3_put_pr_reg(pr_reg);
+			put_pr_reg(pr_reg);
 			return 1;
 		}
-		core_scsi3_put_pr_reg(pr_reg);
+		put_pr_reg(pr_reg);
 		conflict = 1;
 	} else {
 		/*

@@ -611,7 +621,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
-	atomic_set(&pr_reg->pr_res_holders, 0);
+	kref_init(&pr_reg->refcount);
 	pr_reg->pr_reg_nacl = nacl;
 	pr_reg->pr_reg_deve = deve;
 	pr_reg->pr_res_mapped_lun = deve->mapped_lun;

@@ -795,7 +805,7 @@ int core_scsi3_alloc_aptpl_registration(
 	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
-	atomic_set(&pr_reg->pr_res_holders, 0);
+	kref_init(&pr_reg->refcount);
 	pr_reg->pr_reg_nacl = NULL;
 	pr_reg->pr_reg_deve = NULL;
 	pr_reg->pr_res_mapped_lun = mapped_lun;

@@ -1102,8 +1112,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
 				if (dev->dev_attrib.enforce_pr_isids)
 					continue;
 			}
-			atomic_inc(&pr_reg->pr_res_holders);
-			smp_mb__after_atomic_inc();
+			get_pr_reg(pr_reg);
 			spin_unlock(&pr_tmpl->registration_lock);
 			return pr_reg;
 		}

@@ -1117,8 +1126,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
 		if (strcmp(isid, pr_reg->pr_reg_isid))
 			continue;
 
-		atomic_inc(&pr_reg->pr_res_holders);
-		smp_mb__after_atomic_inc();
+		get_pr_reg(pr_reg);
 		spin_unlock(&pr_tmpl->registration_lock);
 		return pr_reg;
 	}

@@ -1145,12 +1153,6 @@ static struct t10_pr_registration *core_scsi3_locate_pr_reg(
 	return __core_scsi3_locate_pr_reg(dev, nacl, isid_ptr);
 }
 
-static void core_scsi3_put_pr_reg(struct t10_pr_registration *pr_reg)
-{
-	atomic_dec(&pr_reg->pr_res_holders);
-	smp_mb__after_atomic_dec();
-}
-
 static int core_scsi3_check_implicit_release(
 	struct se_device *dev,
 	struct t10_pr_registration *pr_reg)

@@ -1213,7 +1215,6 @@ static void __core_scsi3_free_registration(
 {
 	struct target_core_fabric_ops *tfo =
 			pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo;
-	struct t10_reservation *pr_tmpl = &dev->t10_pr;
 	char i_buf[PR_REG_ISID_ID_LEN];
 
 	memset(i_buf, 0, PR_REG_ISID_ID_LEN);

@@ -1227,20 +1228,7 @@ static void __core_scsi3_free_registration(
 	 * so call core_scsi3_put_pr_reg() to decrement our reference.
 	 */
 	if (dec_holders)
-		core_scsi3_put_pr_reg(pr_reg);
-	/*
-	 * Wait until all reference from any other I_T nexuses for this
-	 * *pr_reg have been released.  Because list_del() is called above,
-	 * the last core_scsi3_put_pr_reg(pr_reg) will release this reference
-	 * count back to zero, and we release *pr_reg.
-	 */
-	while (atomic_read(&pr_reg->pr_res_holders) != 0) {
-		spin_unlock(&pr_tmpl->registration_lock);
-		pr_debug("SPC-3 PR [%s] waiting for pr_res_holders\n",
-				tfo->get_fabric_name());
-		cpu_relax();
-		spin_lock(&pr_tmpl->registration_lock);
-	}
+		put_pr_reg(pr_reg);
 
 	pr_debug("SPC-3 PR [%s] Service Action: UNREGISTER Initiator"
 		" Node: %s%s\n", tfo->get_fabric_name(),

@@ -1254,17 +1242,14 @@ static void __core_scsi3_free_registration(
 		" 0x%08x\n", tfo->get_fabric_name(), pr_reg->pr_res_key,
 		pr_reg->pr_res_generation);
 
-	if (!preempt_and_abort_list) {
-		pr_reg->pr_reg_deve = NULL;
-		pr_reg->pr_reg_nacl = NULL;
-		kmem_cache_free(t10_pr_reg_cache, pr_reg);
-		return;
-	}
 	/*
 	 * For PREEMPT_AND_ABORT, the list of *pr_reg in preempt_and_abort_list
 	 * are released once the ABORT_TASK_SET has completed..
 	 */
-	list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
+	if (preempt_and_abort_list)
+		list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
+	else
+		put_pr_reg(pr_reg);
 }
 
 void core_scsi3_free_pr_reg_from_nacl(

@@ -1653,7 +1638,7 @@ core_scsi3_decode_spec_i_port(
 		pr_reg_e = __core_scsi3_locate_pr_reg(dev, dest_node_acl,
 					iport_ptr);
 		if (pr_reg_e) {
-			core_scsi3_put_pr_reg(pr_reg_e);
+			put_pr_reg(pr_reg_e);
 			core_scsi3_lunacl_undepend_item(dest_se_deve);
 			core_scsi3_nodeacl_undepend_item(dest_node_acl);
 			core_scsi3_tpg_undepend_item(dest_tpg);

@@ -2181,7 +2166,7 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 	ret = core_scsi3_update_and_write_aptpl(dev, aptpl);
 
 out:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }

@@ -2359,7 +2344,7 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
 
 	ret = 0;
 out_put_pr_reg:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }

@@ -2573,7 +2558,7 @@ write_aptpl:
 		core_scsi3_update_and_write_aptpl(cmd->se_dev, true);
 
 out_put_pr_reg:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }

@@ -2613,7 +2598,7 @@ core_scsi3_emulate_pro_clear(struct se_cmd *cmd, u64 res_key)
 			" res_key: 0x%016Lx does not match"
 			" existing SA REGISTER res_key:"
 			" 0x%016Lx\n", res_key, pr_reg_n->pr_res_key);
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_RESERVATION_CONFLICT;
 	}
 	/*

@@ -2757,12 +2742,12 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		return TCM_RESERVATION_CONFLICT;
 	}
 	if (pr_reg_n->pr_res_key != res_key) {
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_RESERVATION_CONFLICT;
 	}
 	if (scope != PR_SCOPE_LU_SCOPE) {
 		pr_err("SPC-3 PR: Illegal SCOPE: 0x%02x\n", scope);
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_INVALID_PARAMETER_LIST;
 	}

@@ -2775,7 +2760,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 
 	if (!all_reg && !sa_res_key) {
 		spin_unlock(&dev->dev_reservation_lock);
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_INVALID_PARAMETER_LIST;
 	}
 	/*

@@ -2869,7 +2854,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		 */
 		if (!released_regs) {
 			spin_unlock(&dev->dev_reservation_lock);
-			core_scsi3_put_pr_reg(pr_reg_n);
+			put_pr_reg(pr_reg_n);
 			return TCM_RESERVATION_CONFLICT;
 		}
 		/*

@@ -2891,7 +2876,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		if (pr_tmpl->pr_aptpl_active)
 			core_scsi3_update_and_write_aptpl(cmd->se_dev, true);
 
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		core_scsi3_pr_generation(cmd->se_dev);
 		return 0;
 	}

@@ -3020,7 +3005,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 	if (pr_tmpl->pr_aptpl_active)
 		core_scsi3_update_and_write_aptpl(cmd->se_dev, true);
 
-	core_scsi3_put_pr_reg(pr_reg_n);
+	put_pr_reg(pr_reg_n);
 	core_scsi3_pr_generation(cmd->se_dev);
 	return 0;
 }

@@ -3442,13 +3427,13 @@ after_iport_check:
 		__core_scsi3_free_registration(dev, pr_reg, NULL, 1);
 		spin_unlock(&pr_tmpl->registration_lock);
 	} else
-		core_scsi3_put_pr_reg(pr_reg);
+		put_pr_reg(pr_reg);
 
 	core_scsi3_update_and_write_aptpl(cmd->se_dev, aptpl);
 
 	transport_kunmap_data_sg(cmd);
 
-	core_scsi3_put_pr_reg(dest_pr_reg);
+	put_pr_reg(dest_pr_reg);
 	return 0;
 out:
 	if (buf)

@@ -3460,7 +3445,7 @@ out:
 	core_scsi3_tpg_undepend_item(dest_se_tpg);
 
 out_put_pr_reg:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }

@@ -3862,8 +3847,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 		se_tpg = pr_reg->pr_reg_nacl->se_tpg;
 		add_desc_len = 0;
 
-		atomic_inc(&pr_reg->pr_res_holders);
-		smp_mb__after_atomic_inc();
+		get_pr_reg(pr_reg);
 		spin_unlock(&pr_tmpl->registration_lock);
 		/*
 		 * Determine expected length of $FABRIC_MOD specific

@@ -3876,8 +3860,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 			pr_warn("SPC-3 PRIN READ_FULL_STATUS ran"
 				" out of buffer: %d\n", cmd->data_length);
 			spin_lock(&pr_tmpl->registration_lock);
-			atomic_dec(&pr_reg->pr_res_holders);
-			smp_mb__after_atomic_dec();
+			put_pr_reg(pr_reg);
 			break;
 		}
 		/*

@@ -3938,8 +3921,8 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 				se_nacl, pr_reg, &format_code, &buf[off+4]);
 
 		spin_lock(&pr_tmpl->registration_lock);
-		atomic_dec(&pr_reg->pr_res_holders);
-		smp_mb__after_atomic_dec();
+		put_pr_reg(pr_reg);
+
 		/*
 		 * Set the ADDITIONAL DESCRIPTOR LENGTH
 		 */

include/target/target_core_base.h +1 -1

index 6bf1da9..513429a 100644
@@ -363,7 +363,7 @@ struct t10_pr_registration {
 	u32 pr_res_generation;
 	u64 pr_reg_bin_isid;
 	u64 pr_res_key;
-	atomic_t pr_res_holders;
+	struct kref refcount;
 	struct se_node_acl *pr_reg_nacl;
 	struct se_dev_entry *pr_reg_deve;
 	struct se_lun *pr_reg_tg_pt_lun;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:20 UTC #28 lore raw

Re: [PATCH 12/32] target: Convert t10_pr_registration to kref

Quoted 150 lines · Andy Grover

Free struct when kref becomes 0.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_pr.c   |   99 +++++++++++++++---------------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 42 insertions(+), 59 deletions(-)

drivers/target/target_core_pr.c +19 ~~-37~~

index 9155df0..17a4c3b 100644
@@ -75,6 +75,17 @@ enum preempt_type {
 	PREEMPT_AND_ABORT,
 };
 
+static void release_pr_reg(struct kref *kref)
+{
+	struct t10_pr_registration *pr_reg = container_of(kref,
+							  struct t10_pr_registration, refcount);
+
+	kmem_cache_free(t10_pr_reg_cache, pr_reg);
+}
+
+#define get_pr_reg(x) kref_get(&x->refcount)
+#define put_pr_reg(x) kref_put(&x->refcount, release_pr_reg)
+
 static void __core_scsi3_complete_pro_release(struct se_device *, struct se_node_acl *,
 			struct t10_pr_registration *, int);

@@ -109,7 +120,6 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
 
 static struct t10_pr_registration *core_scsi3_locate_pr_reg(struct se_device *,
 					struct se_node_acl *, struct se_session *);
-static void core_scsi3_put_pr_reg(struct t10_pr_registration *);
 
 static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
 {

@@ -144,17 +154,17 @@ static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
 		 * as defined in SPC-2.
 		 */
 		if (pr_reg->pr_res_holder) {
-			core_scsi3_put_pr_reg(pr_reg);
+			put_pr_reg(pr_reg);
 			return 1;
 		}
 		if ((pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY) ||
 		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY) ||
 		    (pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_ALLREG) ||
 		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_ALLREG)) {
-			core_scsi3_put_pr_reg(pr_reg);
+			put_pr_reg(pr_reg);
 			return 1;
 		}
-		core_scsi3_put_pr_reg(pr_reg);
+		put_pr_reg(pr_reg);
 		conflict = 1;
 	} else {
 		/*

@@ -611,7 +621,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
-	atomic_set(&pr_reg->pr_res_holders, 0);
+	kref_init(&pr_reg->refcount);
 	pr_reg->pr_reg_nacl = nacl;
 	pr_reg->pr_reg_deve = deve;
 	pr_reg->pr_res_mapped_lun = deve->mapped_lun;

@@ -795,7 +805,7 @@ int core_scsi3_alloc_aptpl_registration(
 	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
-	atomic_set(&pr_reg->pr_res_holders, 0);
+	kref_init(&pr_reg->refcount);
 	pr_reg->pr_reg_nacl = NULL;
 	pr_reg->pr_reg_deve = NULL;
 	pr_reg->pr_res_mapped_lun = mapped_lun;

@@ -1102,8 +1112,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
 				if (dev->dev_attrib.enforce_pr_isids)
 					continue;
 			}
-			atomic_inc(&pr_reg->pr_res_holders);
-			smp_mb__after_atomic_inc();
+			get_pr_reg(pr_reg);
 			spin_unlock(&pr_tmpl->registration_lock);
 			return pr_reg;
 		}

@@ -1117,8 +1126,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
 		if (strcmp(isid, pr_reg->pr_reg_isid))
 			continue;
 
-		atomic_inc(&pr_reg->pr_res_holders);
-		smp_mb__after_atomic_inc();
+		get_pr_reg(pr_reg);
 		spin_unlock(&pr_tmpl->registration_lock);
 		return pr_reg;
 	}

@@ -1145,12 +1153,6 @@ static struct t10_pr_registration *core_scsi3_locate_pr_reg(
 	return __core_scsi3_locate_pr_reg(dev, nacl, isid_ptr);
 }
 
-static void core_scsi3_put_pr_reg(struct t10_pr_registration *pr_reg)
-{
-	atomic_dec(&pr_reg->pr_res_holders);
-	smp_mb__after_atomic_dec();
-}
-
 static int core_scsi3_check_implicit_release(
 	struct se_device *dev,
 	struct t10_pr_registration *pr_reg)

@@ -1213,7 +1215,6 @@ static void __core_scsi3_free_registration(
 {
 	struct target_core_fabric_ops *tfo =
 			pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo;
-	struct t10_reservation *pr_tmpl = &dev->t10_pr;
 	char i_buf[PR_REG_ISID_ID_LEN];
 
 	memset(i_buf, 0, PR_REG_ISID_ID_LEN);

@@ -1227,20 +1228,7 @@ static void __core_scsi3_free_registration(
 	 * so call core_scsi3_put_pr_reg() to decrement our reference.
 	 */
 	if (dec_holders)
-		core_scsi3_put_pr_reg(pr_reg);
-	/*
-	 * Wait until all reference from any other I_T nexuses for this
-	 * *pr_reg have been released.  Because list_del() is called above,
-	 * the last core_scsi3_put_pr_reg(pr_reg) will release this reference
-	 * count back to zero, and we release *pr_reg.
-	 */
-	while (atomic_read(&pr_reg->pr_res_holders) != 0) {
-		spin_unlock(&pr_tmpl->registration_lock);
-		pr_debug("SPC-3 PR [%s] waiting for pr_res_holders\n",
-				tfo->get_fabric_name());
-		cpu_relax();
-		spin_lock(&pr_tmpl->registration_lock);
-	}
+		put_pr_reg(pr_reg);
 
 	pr_debug("SPC-3 PR [%s] Service Action: UNREGISTER Initiator"
 		" Node: %s%s\n", tfo->get_fabric_name(),

@@ -1254,17 +1242,14 @@ static void __core_scsi3_free_registration(
 		" 0x%08x\n", tfo->get_fabric_name(), pr_reg->pr_res_key,
 		pr_reg->pr_res_generation);
 
-	if (!preempt_and_abort_list) {
-		pr_reg->pr_reg_deve = NULL;
-		pr_reg->pr_reg_nacl = NULL;
-		kmem_cache_free(t10_pr_reg_cache, pr_reg);
-		return;
-	}

drivers/target/target_core_pr.c static void __core_scsi3_free_registration( · line 1253

Quoted 10 lines

drivers/target/target_core_pr.c +4 -1

 	/*
 	 * For PREEMPT_AND_ABORT, the list of *pr_reg in preempt_and_abort_list
 	 * are released once the ABORT_TASK_SET has completed..
 	 */
-	list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
+	if (preempt_and_abort_list)
+		list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
+	else
+		put_pr_reg(pr_reg);
 }

Same issue again. It's not safe to add pr_reg->pr_reg_abort_list while there are still active references from other process contexts.

Also, given that core_scsi3_free_pr_reg_from_nacl() is called during se_node_acl removal, and core_scsi3_free_all_registrations() called during se_device removal, returning here before ensuring that all references have completed is completely broken.

NAK.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #29 lore raw

PATCH 13/32 target: Move spinlock inside core_release_port

In reply to Andy Grover

Keep locks closer to things they need to lock.

sparse annotation no longer needed.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c |   10 +++-------
 1 files changed, 3 insertions(+), 7 deletions(-)

drivers/target/target_core_device.c +3 -7

index 295b5e4..7b5697a 100644
@@ -546,16 +546,14 @@ static void core_export_port(
 	port->sep_index = port->sep_rtpi; /* RELATIVE TARGET PORT IDENTIFIER */
 }
 
-/*
- *	Called with struct se_device->se_port_lock spinlock held.
- */
 static void core_release_port(struct se_device *dev, struct se_port *port)
-	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
 {
+	spin_lock(&dev->se_port_lock);
 	core_alua_free_tg_pt_gp_mem(port);
-
 	list_del(&port->sep_node);
 	dev->dev_port_count--;
+	spin_unlock(&dev->se_port_lock);
+
 	put_port(port);
 }

@@ -596,9 +594,7 @@ void core_dev_unexport(
 	}
 	spin_unlock(&lun->lun_sep_lock);
 
-	spin_lock(&dev->se_port_lock);
 	core_release_port(dev, port);
-	spin_unlock(&dev->se_port_lock);
 
 	spin_lock(&hba->device_lock);
 	dev->export_count--;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 21:21 UTC #30 lore raw

Re: [PATCH 13/32] target: Move spinlock inside core_release_port

Quoted 9 lines · Andy Grover

Keep locks closer to things they need to lock.
sparse annotation no longer needed.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_device.c |   10 +++-------
 1 files changed, 3 insertions(+), 7 deletions(-)

I'll take this one, but it needs to be re-posted without the other patches.

--nab

Quoted 34 lines

drivers/target/target_core_device.c +3 -7

index 295b5e4..7b5697a 100644
@@ -546,16 +546,14 @@ static void core_export_port(
 	port->sep_index = port->sep_rtpi; /* RELATIVE TARGET PORT IDENTIFIER */
 }
 
-/*
- *	Called with struct se_device->se_port_lock spinlock held.
- */
 static void core_release_port(struct se_device *dev, struct se_port *port)
-	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
 {
+	spin_lock(&dev->se_port_lock);
 	core_alua_free_tg_pt_gp_mem(port);
-
 	list_del(&port->sep_node);
 	dev->dev_port_count--;
+	spin_unlock(&dev->se_port_lock);
+
 	put_port(port);
 }

@@ -596,9 +594,7 @@ void core_dev_unexport(
 	}
 	spin_unlock(&lun->lun_sep_lock);
 
-	spin_lock(&dev->se_port_lock);
 	core_release_port(dev, port);
-	spin_unlock(&dev->se_port_lock);
 
 	spin_lock(&hba->device_lock);
 	dev->export_count--;

Andy Grover

author

2013-12-13 23:59 UTC #31 lore raw

PATCH 14/32 target: Remove extra percpu_ref_init

In reply to Andy Grover

lun->lun_ref is also initialized in core_tpg_post_addlun, so it doesn't need to be done in core_tpg_setup_virtual_lun0.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_tpg.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)

drivers/target/target_core_tpg.c +0 -4

index 8771b23..2f6df57 100644
@@ -658,10 +658,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	spin_lock_init(&lun->lun_sep_lock);
 	init_completion(&lun->lun_ref_comp);
 
-	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
-	if (ret < 0)
-		return ret;
-
 	ret = core_tpg_post_addlun(se_tpg, lun, lun_access, dev);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);

Signature · 1 line

1.7.1

Nicholas A. Bellinger

Applied

2013-12-16 21:23 UTC #32 lore raw

Re: [PATCH 14/32] target: Remove extra percpu_ref_init

Quoted 13 lines · Andy Grover

lun->lun_ref is also initialized in core_tpg_post_addlun, so it doesn't need to be done in core_tpg_setup_virtual_lun0.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_tpg.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)
drivers/target/target_core_tpg.c +0 -0
index 8771b23..2f6df57 100644

drivers/target/target_core_tpg.c static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg) · line 663

Quoted 10 lines

drivers/target/target_core_tpg.c +0 -4

@@ -658,10 +658,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	spin_lock_init(&lun->lun_sep_lock);
 	init_completion(&lun->lun_ref_comp);
 
-	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
-	if (ret < 0)
-		return ret;
-
 	ret = core_tpg_post_addlun(se_tpg, lun, lun_access, dev);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);

Applied to target-pending/queue, and will be including in the next -rc5 PULL request.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #33 lore raw

PATCH 15/32 target: Refer to u32 luns as unpacked_lun

In reply to Andy Grover

It's clearer to refer to pointers to the struct se_lun as "lun" and the actual number itself as "unpacked_lun".

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

drivers/target/target_core_device.c +8 -8

index 7b5697a..da290f6 100644
@@ -1096,23 +1096,23 @@ int se_dev_set_block_size(struct se_device *dev, u32 block_size)
 struct se_lun *core_dev_add_lun(
 	struct se_portal_group *tpg,
 	struct se_device *dev,
-	u32 lun)
+	u32 unpacked_lun)
 {
-	struct se_lun *lun_p;
+	struct se_lun *lun;
 	int rc;
 
-	lun_p = core_tpg_pre_addlun(tpg, lun);
-	if (IS_ERR(lun_p))
-		return lun_p;
+	lun = core_tpg_pre_addlun(tpg, unpacked_lun);
+	if (IS_ERR(lun))
+		return lun;
 
-	rc = core_tpg_post_addlun(tpg, lun_p,
+	rc = core_tpg_post_addlun(tpg, lun,
 				TRANSPORT_LUNFLAGS_READ_WRITE, dev);
 	if (rc < 0)
 		return ERR_PTR(rc);
 
 	pr_debug("%s_TPG[%u]_LUN[%u] - Activated %s Logical Unit from"
 		" CORE HBA: %u\n", tpg->se_tpg_tfo->get_fabric_name(),
-		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun_p->unpacked_lun,
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		tpg->se_tpg_tfo->get_fabric_name(), dev->se_hba->hba_id);
 	/*
 	 * Update LUN maps for dynamically added initiators when

@@ -1133,7 +1133,7 @@ struct se_lun *core_dev_add_lun(
 		spin_unlock_irq(&tpg->acl_node_lock);
 	}
 
-	return lun_p;
+	return lun;
 }
 
 /*      core_dev_del_lun():

Signature · 1 line

1.7.1

Nicholas A. Bellinger

Applied

2013-12-16 21:25 UTC #34 lore raw

Re: [PATCH 15/32] target: Refer to u32 luns as unpacked_lun

Quoted 8 lines · Andy Grover

It's clearer to refer to pointers to the struct se_lun as "lun" and the actual number itself as "unpacked_lun".
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_device.c |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

Applied to target-pending/for-next.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #35 lore raw

PATCH 16/32 target: Rename core_tpg_{pre,post}_addlun for clarity

In reply to Andy Grover

"pre" is really an allocation function. The only time it isn't called is for virtual_lun0, which is statically allocated. Renaming that to "alloc" lets the other function not need to be "post", and just be called core_tpg_add_lun.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c   |    4 ++--
 drivers/target/target_core_internal.h |    4 ++--
 drivers/target/target_core_tpg.c      |    6 +++---
 3 files changed, 7 insertions(+), 7 deletions(-)

drivers/target/target_core_device.c +2 -2

index da290f6..faa17a4 100644
@@ -1101,11 +1101,11 @@ struct se_lun *core_dev_add_lun(
 	struct se_lun *lun;
 	int rc;
 
-	lun = core_tpg_pre_addlun(tpg, unpacked_lun);
+	lun = core_tpg_alloc_lun(tpg, unpacked_lun);
 	if (IS_ERR(lun))
 		return lun;
 
-	rc = core_tpg_post_addlun(tpg, lun,
+	rc = core_tpg_add_lun(tpg, lun,
 				TRANSPORT_LUNFLAGS_READ_WRITE, dev);
 	if (rc < 0)
 		return ERR_PTR(rc);

drivers/target/target_core_internal.h +2 -2

index fddf44c..4093936 100644
@@ -87,8 +87,8 @@ struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tp
 		const char *);
 void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
 void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
-struct se_lun *core_tpg_pre_addlun(struct se_portal_group *, u32);
-int	core_tpg_post_addlun(struct se_portal_group *, struct se_lun *,
+struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
+int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, void *);
 struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
 int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);

drivers/target/target_core_tpg.c +3 -3

index 2f6df57..55a9d38 100644
@@ -658,7 +658,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	spin_lock_init(&lun->lun_sep_lock);
 	init_completion(&lun->lun_ref_comp);
 
-	ret = core_tpg_post_addlun(se_tpg, lun, lun_access, dev);
+	ret = core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);
 		return ret;

@@ -785,7 +785,7 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 }
 EXPORT_SYMBOL(core_tpg_deregister);
 
-struct se_lun *core_tpg_pre_addlun(
+struct se_lun *core_tpg_alloc_lun(
 	struct se_portal_group *tpg,
 	u32 unpacked_lun)
 {

@@ -815,7 +815,7 @@ struct se_lun *core_tpg_pre_addlun(
 	return lun;
 }
 
-int core_tpg_post_addlun(
+int core_tpg_add_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
 	u32 lun_access,

Signature · 1 line

1.7.1

Nicholas A. Bellinger

Applied

2013-12-16 21:29 UTC #36 lore raw

Re: [PATCH 16/32] target: Rename core_tpg_{pre,post}_addlun for clarity

Quoted 11 lines · Andy Grover

"pre" is really an allocation function. The only time it isn't called is for virtual_lun0, which is statically allocated. Renaming that to "alloc" lets the other function not need to be "post", and just be called core_tpg_add_lun.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_device.c   |    4 ++--
 drivers/target/target_core_internal.h |    4 ++--
 drivers/target/target_core_tpg.c      |    6 +++---
 3 files changed, 7 insertions(+), 7 deletions(-)

Applied, with minor patch fuzz in core_tpg_setup_virtual_lun0().

--nab

Andy Grover

author

2013-12-13 23:59 UTC #37 lore raw

PATCH 17/32 target: Don't use void* when passing dev in core_tpg_add_lun

In reply to Andy Grover

Especially since it's actually a device.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_internal.h |    2 +-
 drivers/target/target_core_tpg.c      |    4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

drivers/target/target_core_internal.h +1 -1

index 4093936..aabb730 100644
@@ -89,7 +89,7 @@ void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
 void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
-		u32, void *);
+		u32, struct se_device *);
 struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
 int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);

drivers/target/target_core_tpg.c +2 -2

index 55a9d38..da7febb 100644
@@ -819,7 +819,7 @@ int core_tpg_add_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
 	u32 lun_access,
-	void *lun_ptr)
+	struct se_device *dev)
 {
 	int ret;

@@ -827,7 +827,7 @@ int core_tpg_add_lun(
 	if (ret < 0)
 		return ret;
 
-	ret = core_dev_export(lun_ptr, tpg, lun);
+	ret = core_dev_export(dev, tpg, lun);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);
 		return ret;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

Applied

2013-12-16 21:29 UTC #38 lore raw

Re: [PATCH 17/32] target: Don't use void* when passing dev in core_tpg_add_lun

Quoted 8 lines · Andy Grover

Especially since it's actually a device.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_internal.h |    2 +-
 drivers/target/target_core_tpg.c      |    4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

Applied to target-pending/for-next.

--nab

Quoted 35 lines

drivers/target/target_core_internal.h +1 -1

index 4093936..aabb730 100644
@@ -89,7 +89,7 @@ void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
 void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
-		u32, void *);
+		u32, struct se_device *);
 struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
 int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);

drivers/target/target_core_tpg.c +2 -2

index 55a9d38..da7febb 100644
@@ -819,7 +819,7 @@ int core_tpg_add_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
 	u32 lun_access,
-	void *lun_ptr)
+	struct se_device *dev)
 {
 	int ret;

@@ -827,7 +827,7 @@ int core_tpg_add_lun(
 	if (ret < 0)
 		return ret;
 
-	ret = core_dev_export(lun_ptr, tpg, lun);
+	ret = core_dev_export(dev, tpg, lun);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);
 		return ret;

Andy Grover

author

2013-12-13 23:59 UTC #39 lore raw

PATCH 18/32 target: More core_dev_del cleanups

In reply to Andy Grover

core_dev_del_lun needs no return value. Also change it to take a se_lun* instead of the unpacked lun.

Rename core_tpg_pre_dellun to core_tpg_free_lun, and post_dellun to remove_lun. Swap the order they are called, and hold off on setting lun_status to STATUS_FREE until the end of free_lun.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c          |   16 ++++---------
 drivers/target/target_core_fabric_configfs.c |    2 +-
 drivers/target/target_core_internal.h        |    6 ++--
 drivers/target/target_core_tpg.c             |   32 +++++++++-----------------
 4 files changed, 20 insertions(+), 36 deletions(-)

drivers/target/target_core_device.c +5 ~~-11~~

index faa17a4..af12456 100644
@@ -1140,24 +1140,18 @@ struct se_lun *core_dev_add_lun(
  *
  *
  */
-int core_dev_del_lun(
+void core_dev_del_lun(
 	struct se_portal_group *tpg,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
-
-	lun = core_tpg_pre_dellun(tpg, unpacked_lun);
-	if (IS_ERR(lun))
-		return PTR_ERR(lun);
-
-	core_tpg_post_dellun(tpg, lun);
+	core_tpg_remove_lun(tpg, lun);
 
 	pr_debug("%s_TPG[%u]_LUN[%u] - Deactivated %s Logical Unit from"
 		" device object\n", tpg->se_tpg_tfo->get_fabric_name(),
-		tpg->se_tpg_tfo->tpg_get_tag(tpg), unpacked_lun,
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		tpg->se_tpg_tfo->get_fabric_name());
 
-	return 0;
+	core_tpg_free_lun(tpg, lun);
 }
 
 struct se_lun *core_get_lun_from_tpg(struct se_portal_group *tpg, u32 unpacked_lun)

drivers/target/target_core_fabric_configfs.c +1 -1

index 7de9f04..64cc4dc 100644
@@ -821,7 +821,7 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_dev_del_lun(se_tpg, lun->unpacked_lun);
+	core_dev_del_lun(se_tpg, lun);
 	return 0;
 }

drivers/target/target_core_internal.h +3 -3

index aabb730..5d1e4ec 100644
@@ -44,7 +44,7 @@ int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
 int	se_dev_set_optimal_sectors(struct se_device *, u32);
 int	se_dev_set_block_size(struct se_device *, u32);
 struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
-int	core_dev_del_lun(struct se_portal_group *, u32);
+void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
 struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_node_acl *, u32, int *);

@@ -90,8 +90,8 @@ void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);
-struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
-int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
+void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
+void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
 
 static inline void release_deve(struct kref *kref)
 {

drivers/target/target_core_tpg.c +11 ~~-21~~

index da7febb..e6f9dfb 100644
@@ -336,7 +336,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 			continue;
 
 		spin_unlock(&tpg->tpg_lun_lock);
-		core_dev_del_lun(tpg, lun->unpacked_lun);
+		core_dev_del_lun(tpg, lun);
 		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);

@@ -671,7 +671,7 @@ static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)
 {
 	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
 
-	core_tpg_post_dellun(se_tpg, lun);
+	core_tpg_remove_lun(se_tpg, lun);
 }
 
 int core_tpg_register(

@@ -841,48 +841,38 @@ int core_tpg_add_lun(
 	return 0;
 }
 
-struct se_lun *core_tpg_pre_dellun(
+void core_tpg_free_lun(
 	struct se_portal_group *tpg,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
 
-	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
+	if (lun->unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
 		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS_PER_TPG"
 			"-1: %u for Target Portal Group: %u\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
+			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
 			TRANSPORT_MAX_LUNS_PER_TPG-1,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		return ERR_PTR(-EOVERFLOW);
+		return;
 	}
 
 	spin_lock(&tpg->tpg_lun_lock);
-	lun = tpg->tpg_lun_list[unpacked_lun];
 	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
 		pr_err("%s Logical Unit Number: %u is not active on"
 			" Target Portal Group: %u, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
+			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
 		spin_unlock(&tpg->tpg_lun_lock);
-		return ERR_PTR(-ENODEV);
+		return;
 	}
+	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
 	spin_unlock(&tpg->tpg_lun_lock);
-
-	return lun;
 }
 
-int core_tpg_post_dellun(
+void core_tpg_remove_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
 	transport_clear_lun_ref(lun);
-
 	core_dev_unexport(lun->lun_se_dev, tpg, lun);
-
-	spin_lock(&tpg->tpg_lun_lock);
-	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
-	spin_unlock(&tpg->tpg_lun_lock);
-
-	return 0;
 }

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:34 UTC #40 lore raw

Re: [PATCH 18/32] target: More core_dev_del cleanups

Quoted 7 lines · Andy Grover

core_dev_del_lun needs no return value. Also change it to take a se_lun* instead of the unpacked lun.
Rename core_tpg_pre_dellun to core_tpg_free_lun, and post_dellun to remove_lun. Swap the order they are called, and hold off on setting lun_status to STATUS_FREE until the end of free_lun.

The swapping of the order makes no sense here..

Why should the checks now happen after everything released to the se_lun has been released, and yet still ignore the return value. !?

NAK.

--nab

Quoted 159 lines

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c          |   16 ++++---------
 drivers/target/target_core_fabric_configfs.c |    2 +-
 drivers/target/target_core_internal.h        |    6 ++--
 drivers/target/target_core_tpg.c             |   32 +++++++++-----------------
 4 files changed, 20 insertions(+), 36 deletions(-)

drivers/target/target_core_device.c +5 ~~-11~~

index faa17a4..af12456 100644
@@ -1140,24 +1140,18 @@ struct se_lun *core_dev_add_lun(
  *
  *
  */
-int core_dev_del_lun(
+void core_dev_del_lun(
 	struct se_portal_group *tpg,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
-
-	lun = core_tpg_pre_dellun(tpg, unpacked_lun);
-	if (IS_ERR(lun))
-		return PTR_ERR(lun);
-
-	core_tpg_post_dellun(tpg, lun);
+	core_tpg_remove_lun(tpg, lun);
 
 	pr_debug("%s_TPG[%u]_LUN[%u] - Deactivated %s Logical Unit from"
 		" device object\n", tpg->se_tpg_tfo->get_fabric_name(),
-		tpg->se_tpg_tfo->tpg_get_tag(tpg), unpacked_lun,
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		tpg->se_tpg_tfo->get_fabric_name());
 
-	return 0;
+	core_tpg_free_lun(tpg, lun);
 }
 
 struct se_lun *core_get_lun_from_tpg(struct se_portal_group *tpg, u32 unpacked_lun)

drivers/target/target_core_fabric_configfs.c +1 -1

index 7de9f04..64cc4dc 100644
@@ -821,7 +821,7 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_dev_del_lun(se_tpg, lun->unpacked_lun);
+	core_dev_del_lun(se_tpg, lun);
 	return 0;
 }

drivers/target/target_core_internal.h +3 -3

index aabb730..5d1e4ec 100644
@@ -44,7 +44,7 @@ int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
 int	se_dev_set_optimal_sectors(struct se_device *, u32);
 int	se_dev_set_block_size(struct se_device *, u32);
 struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
-int	core_dev_del_lun(struct se_portal_group *, u32);
+void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
 struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_node_acl *, u32, int *);

@@ -90,8 +90,8 @@ void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);
-struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
-int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
+void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
+void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
 
 static inline void release_deve(struct kref *kref)
 {

drivers/target/target_core_tpg.c +11 ~~-21~~

index da7febb..e6f9dfb 100644
@@ -336,7 +336,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 			continue;
 
 		spin_unlock(&tpg->tpg_lun_lock);
-		core_dev_del_lun(tpg, lun->unpacked_lun);
+		core_dev_del_lun(tpg, lun);
 		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);

@@ -671,7 +671,7 @@ static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)
 {
 	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
 
-	core_tpg_post_dellun(se_tpg, lun);
+	core_tpg_remove_lun(se_tpg, lun);
 }
 
 int core_tpg_register(

@@ -841,48 +841,38 @@ int core_tpg_add_lun(
 	return 0;
 }
 
-struct se_lun *core_tpg_pre_dellun(
+void core_tpg_free_lun(
 	struct se_portal_group *tpg,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
 
-	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
+	if (lun->unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
 		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS_PER_TPG"
 			"-1: %u for Target Portal Group: %u\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
+			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
 			TRANSPORT_MAX_LUNS_PER_TPG-1,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		return ERR_PTR(-EOVERFLOW);
+		return;
 	}
 
 	spin_lock(&tpg->tpg_lun_lock);
-	lun = tpg->tpg_lun_list[unpacked_lun];
 	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
 		pr_err("%s Logical Unit Number: %u is not active on"
 			" Target Portal Group: %u, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
+			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
 		spin_unlock(&tpg->tpg_lun_lock);
-		return ERR_PTR(-ENODEV);
+		return;
 	}
+	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
 	spin_unlock(&tpg->tpg_lun_lock);
-
-	return lun;
 }
 
-int core_tpg_post_dellun(
+void core_tpg_remove_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
 	transport_clear_lun_ref(lun);
-
 	core_dev_unexport(lun->lun_se_dev, tpg, lun);
-
-	spin_lock(&tpg->tpg_lun_lock);
-	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
-	spin_unlock(&tpg->tpg_lun_lock);
-
-	return 0;
 }

Andy Grover

author

2013-12-13 23:59 UTC #41 lore raw

PATCH 19/32 target: Convert to rbtree for se_dev_entry in se_node_acl

In reply to Andy Grover

Instead of an array, use a rbtree. Less memory use on average, and can allow >255 entries. We go from O(1) to O(log n) on lookups. If this shows up on profiling (it won't) then transition to other kernel lookup methods is straightforward from here.

Change core_disable_device_list_for_node to be called with device_list_lock held, and tweaked params a little.

TRANSPORT_LUNFLAGS_INITIATOR_ACCESS no longer needed, presence in the rbtree is equivalent to this being set.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c          |  213 ++++++++++++++-----------
 drivers/target/target_core_fabric_configfs.c |   24 ++--
 drivers/target/target_core_internal.h        |   15 +-
 drivers/target/target_core_pr.c              |   25 +++-
 drivers/target/target_core_spc.c             |   11 +-
 drivers/target/target_core_stat.c            |   72 +++++-----
 drivers/target/target_core_tpg.c             |   53 +------
 drivers/target/target_core_ua.c              |   18 ++-
 include/target/target_core_base.h            |    8 +-
 9 files changed, 222 insertions(+), 217 deletions(-)

drivers/target/target_core_device.c +119 ~~-94~~

index af12456..a18724b 100644
@@ -54,6 +54,51 @@ static struct se_hba *lun0_hba;
 /* not static, needed by tpg.c */
 struct se_device *g_lun0_dev;
 
+bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve)
+{
+	struct rb_root *root = &nacl->rb_device_list;
+	struct rb_node **new = &(root->rb_node), *parent = NULL;
+
+	/* Figure out where to put new node */
+	while (*new) {
+		struct se_dev_entry *this = rb_entry(*new, struct se_dev_entry, rb_node);
+
+		parent = *new;
+		if (deve->mapped_lun < this->mapped_lun)
+			new = &((*new)->rb_left);
+		else if (deve->mapped_lun > this->mapped_lun)
+			new = &((*new)->rb_right);
+		else
+			return false;
+	}
+
+	/* Add new node and rebalance tree. */
+	rb_link_node(&deve->rb_node, parent, new);
+	rb_insert_color(&deve->rb_node, root);
+
+	return true;
+}
+
+
+struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun)
+{
+	struct rb_root *root = &nacl->rb_device_list;
+	struct rb_node *node = root->rb_node;
+
+	while (node) {
+		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
+
+		if (mapped_lun < deve->mapped_lun)
+			node = node->rb_left;
+		else if (mapped_lun > deve->mapped_lun)
+			node = node->rb_right;
+		else
+			return deve;
+	}
+	return NULL;
+}
+
+
 sense_reason_t
 transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 {

@@ -66,8 +111,8 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		return TCM_NON_EXISTENT_LUN;
 
 	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
-	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
-	if (se_cmd->se_deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
+	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
+	if (se_cmd->se_deve) {
 		struct se_dev_entry *deve = se_cmd->se_deve;
 
 		deve->total_cmds++;

@@ -153,10 +198,10 @@ int transport_lookup_tmr_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		return -ENODEV;
 
 	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
-	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
-	deve = se_cmd->se_deve;
+	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
+	if (se_cmd->se_deve) {
+		deve = se_cmd->se_deve;
 
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
 		se_tmr->tmr_lun = deve->se_lun;
 		se_cmd->se_lun = deve->se_lun;
 		se_lun = deve->se_lun;

@@ -187,25 +232,22 @@ EXPORT_SYMBOL(transport_lookup_tmr_lun);
 
 /*
  * This function is called from core_scsi3_emulate_pro_register_and_move()
- * and core_scsi3_decode_spec_i_port(), and will increment &deve->pr_ref_count
+ * and core_scsi3_decode_spec_i_port(), and will increment deve->refcount
  * when a matching rtpi is found.
  */
 struct se_dev_entry *core_get_se_deve_from_rtpi(
 	struct se_node_acl *nacl,
 	u16 rtpi)
 {
-	struct se_dev_entry *deve;
 	struct se_lun *lun;
 	struct se_port *port;
 	struct se_portal_group *tpg = nacl->se_tpg;
-	u32 i;
+	struct rb_node *node;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
+	for (node = rb_first(&nacl->rb_device_list); node; node = rb_next(node))
+	{
+		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
 
 		lun = deve->se_lun;
 		if (!lun) {

@@ -234,43 +276,24 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
 	return NULL;
 }
 
-int core_free_device_list_for_node(
+void core_free_device_list_for_node(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
-	struct se_dev_entry *deve;
-	struct se_lun *lun;
-	u32 i;
-
-	if (!nacl->device_list)
-		return 0;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
-
-		if (!deve->se_lun) {
+	rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
+		if (deve->se_lun) {
+			core_disable_device_list_for_node(deve->se_lun, deve,
+				TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
+		} else {
 			pr_err("%s device entries device pointer is"
 				" NULL, but Initiator has access.\n",
 				tpg->se_tpg_tfo->get_fabric_name());
-			continue;
 		}
-		lun = deve->se_lun;
-
-		spin_unlock_irq(&nacl->device_list_lock);
-		core_disable_device_list_for_node(lun, NULL, deve->mapped_lun,
-			TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
-		spin_lock_irq(&nacl->device_list_lock);
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
-
-	array_free(nacl->device_list, TRANSPORT_MAX_LUNS_PER_TPG);
-	nacl->device_list = NULL;
-
-	return 0;
 }
 
 void core_update_device_list_access(

@@ -281,13 +304,15 @@ void core_update_device_list_access(
 	struct se_dev_entry *deve;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[mapped_lun];
-	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
-	} else {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+	deve = transport_search_deve(nacl, mapped_lun);
+	if (deve) {
+		if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
+			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
+			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
+		} else {
+			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
+			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+		}
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
 }

@@ -305,18 +330,30 @@ int core_enable_device_list_for_node(
 	struct se_portal_group *tpg)
 {
 	struct se_port *port = lun->lun_sep;
-	struct se_dev_entry *deve;
+	struct se_dev_entry *deve, *new_deve;
 
-	spin_lock_irq(&nacl->device_list_lock);
+	new_deve = kzalloc(sizeof(*deve), GFP_KERNEL);
+	if (!new_deve)
+		return -ENOMEM;
 
-	deve = nacl->device_list[mapped_lun];
+	atomic_set(&new_deve->ua_count, 0);
+	kref_init(&new_deve->refcount);
+	spin_lock_init(&new_deve->ua_lock);
+	INIT_LIST_HEAD(&new_deve->alua_port_node);
+	INIT_LIST_HEAD(&new_deve->ua_list);
+	new_deve->se_lun = lun;
+	new_deve->se_lun_acl = lun_acl;
+	new_deve->mapped_lun = mapped_lun;
 
+	spin_lock_irq(&nacl->device_list_lock);
 	/*
 	 * Check if the call is handling demo mode -> explicit LUN ACL
 	 * transition.  This transition must be for the same struct se_lun
 	 * + mapped_lun that was setup in demo mode..
 	 */
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
+	deve = transport_search_deve(nacl, mapped_lun);
+	if (deve) {
+		kfree(new_deve);
 		if (deve->se_lun_acl != NULL) {
 			pr_err("struct se_dev_entry->se_lun_acl"
 			       " already set for demo mode -> explicit"

@@ -345,25 +382,24 @@ int core_enable_device_list_for_node(
 		return 0;
 	}
 
-	deve->se_lun = lun;
-	deve->se_lun_acl = lun_acl;
-	deve->mapped_lun = mapped_lun;
-	deve->lun_flags |= TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
-
-	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
-	} else {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+	if (!transport_insert_deve(nacl, new_deve)) {
+		pr_err("Could not insert new deve for lun %d\n", mapped_lun);
+		kfree(new_deve);
+		spin_unlock_irq(&nacl->device_list_lock);
+		return -EINVAL;
 	}
 
-	deve->creation_time = get_jiffies_64();
-	deve->attach_count++;
+	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE)
+		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
+	else
+		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+
+	new_deve->creation_time = get_jiffies_64();
+	new_deve->attach_count++;
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	spin_lock_bh(&port->sep_alua_lock);
-	list_add_tail(&deve->alua_port_node, &port->sep_alua_list);
+	list_add_tail(&new_deve->alua_port_node, &port->sep_alua_list);
 	spin_unlock_bh(&port->sep_alua_lock);
 
 	return 0;

@@ -371,18 +407,16 @@ int core_enable_device_list_for_node(
 
 /*      core_disable_device_list_for_node():
  *
- *
+ *	CALLED WITH device_list_lock HELD
  */
-int core_disable_device_list_for_node(
+void core_disable_device_list_for_node(
 	struct se_lun *lun,
-	struct se_lun_acl *lun_acl,
-	u32 mapped_lun,
+	struct se_dev_entry *deve,
 	u32 lun_access,
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
 	struct se_port *port = lun->lun_sep;
-	struct se_dev_entry *deve = nacl->device_list[mapped_lun];
 
 	/*
 	 * If the MappedLUN entry is being disabled, the entry in

@@ -397,11 +431,10 @@ int core_disable_device_list_for_node(
 	 * NodeACL context specific PR metadata for demo-mode
 	 * MappedLUN *deve will be released below..
 	 */
-	spin_lock_bh(&port->sep_alua_lock);
+	spin_lock(&port->sep_alua_lock);
 	list_del(&deve->alua_port_node);
-	spin_unlock_bh(&port->sep_alua_lock);
+	spin_unlock(&port->sep_alua_lock);
 
-	spin_lock_irq(&nacl->device_list_lock);
 	/*
 	 * Disable struct se_dev_entry LUN ACL mapping
 	 */

@@ -411,13 +444,12 @@ int core_disable_device_list_for_node(
 	deve->lun_flags = 0;
 	deve->creation_time = 0;
 	deve->attach_count--;
-	spin_unlock_irq(&nacl->device_list_lock);
 
 	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
 
-	put_deve(deve);
+	rb_erase(&deve->rb_node, &nacl->rb_device_list);
 
-	return 0;
+	put_deve(deve);
 }
 
 /*      core_clear_lun_from_tpg():

@@ -427,25 +459,17 @@ int core_disable_device_list_for_node(
 void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 {
 	struct se_node_acl *nacl;
-	struct se_dev_entry *deve;
-	u32 i;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&tpg->acl_node_lock);
 	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
 		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);
-		for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-			deve = nacl->device_list[i];
-			if (lun != deve->se_lun)
-				continue;
-			spin_unlock_irq(&nacl->device_list_lock);
-
-			core_disable_device_list_for_node(lun, NULL,
-				deve->mapped_lun, TRANSPORT_LUNFLAGS_NO_ACCESS,
-				nacl, tpg);
-
-			spin_lock_irq(&nacl->device_list_lock);
+		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
+			if (lun == deve->se_lun)
+				core_disable_device_list_for_node(lun, deve,
+					TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
 		}
 		spin_unlock_irq(&nacl->device_list_lock);

@@ -1301,16 +1325,17 @@ int core_dev_add_initiator_node_lun_acl(
  *
  *
  */
-int core_dev_del_initiator_node_lun_acl(
+void core_dev_del_initiator_node_lun_acl(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
-	struct se_lun_acl *lacl)
+	struct se_lun_acl *lacl,
+	struct se_dev_entry *deve)
 {
 	struct se_node_acl *nacl;
 
 	nacl = lacl->se_lun_nacl;
 	if (!nacl)
-		return -EINVAL;
+		return;
 
 	spin_lock(&lun->lun_acl_lock);
 	list_del(&lacl->lacl_list);

@@ -1318,8 +1343,10 @@ int core_dev_del_initiator_node_lun_acl(
 	smp_mb__after_atomic_dec();
 	spin_unlock(&lun->lun_acl_lock);
 
-	core_disable_device_list_for_node(lun, NULL, lacl->mapped_lun,
+	spin_lock_irq(&nacl->device_list_lock);
+	core_disable_device_list_for_node(lun, deve,
 		TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
+	spin_unlock_irq(&nacl->device_list_lock);
 
 	lacl->se_lun = NULL;

@@ -1328,8 +1355,6 @@ int core_dev_del_initiator_node_lun_acl(
 		tpg->se_tpg_tfo->get_fabric_name(),
 		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		lacl->initiatorname, lacl->mapped_lun);
-
-	return 0;
 }
 
 void core_dev_free_initiator_node_lun_acl(

drivers/target/target_core_fabric_configfs.c +13 ~~-11~~

index 64cc4dc..eaaed43 100644
@@ -112,8 +112,8 @@ static int target_fabric_mappedlun_link(
 	 * tpg_1/attrib/demo_mode_write_protect=1
 	 */
 	spin_lock_irq(&lacl->se_lun_nacl->device_list_lock);
-	deve = lacl->se_lun_nacl->device_list[lacl->mapped_lun];
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS)
+	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
+	if (deve)
 		lun_access = deve->lun_flags;
 	else
 		lun_access =

@@ -140,19 +140,20 @@ static int target_fabric_mappedlun_unlink(
 	struct se_lun *lun;
 	struct se_lun_acl *lacl = container_of(to_config_group(lun_acl_ci),
 			struct se_lun_acl, se_lun_group);
-	struct se_node_acl *nacl = lacl->se_lun_nacl;
-	struct se_dev_entry *deve = nacl->device_list[lacl->mapped_lun];
+	struct se_dev_entry *deve;
 	struct se_portal_group *se_tpg;
+
 	/*
 	 * Determine if the underlying MappedLUN has already been released..
 	 */
-	if (!deve->se_lun)
+	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
+	if (!(deve && deve->se_lun))
 		return 0;
 
 	lun = container_of(to_config_group(lun_ci), struct se_lun, lun_group);
 	se_tpg = lun->lun_sep->sep_tpg;
 
-	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl);
+	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl, deve);
 	return 0;
 }

@@ -169,13 +170,14 @@ static ssize_t target_fabric_mappedlun_show_write_protect(
 {
 	struct se_node_acl *se_nacl = lacl->se_lun_nacl;
 	struct se_dev_entry *deve;
-	ssize_t len;
+	ssize_t len = 0;
 
 	spin_lock_irq(&se_nacl->device_list_lock);
-	deve = se_nacl->device_list[lacl->mapped_lun];
-	len = sprintf(page, "%d\n",
-			(deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
-			1 : 0);
+	deve = transport_search_deve(se_nacl, lacl->mapped_lun);
+	if (deve)
+		len = sprintf(page, "%d\n",
+			      (deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
+			      1 : 0);
 	spin_unlock_irq(&se_nacl->device_list_lock);
 
 	return len;

drivers/target/target_core_internal.h +8 -7

index 5d1e4ec..3803dd8 100644
@@ -5,14 +5,16 @@
 extern struct t10_alua_lu_gp *default_lu_gp;
 
 /* target_core_device.c */
+bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve);
+struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun);
 struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
-int	core_free_device_list_for_node(struct se_node_acl *,
+void	core_free_device_list_for_node(struct se_node_acl *,
 		struct se_portal_group *);
 void	core_update_device_list_access(u32, u32, struct se_node_acl *);
 int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
 		u32, u32, struct se_node_acl *, struct se_portal_group *);
-int	core_disable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
-		u32, u32, struct se_node_acl *, struct se_portal_group *);
+void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
+		u32, struct se_node_acl *, struct se_portal_group *);
 void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
 int	core_dev_export(struct se_device *, struct se_portal_group *,
 		struct se_lun *);

@@ -50,8 +52,8 @@ struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *
 		struct se_node_acl *, u32, int *);
 int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_lun_acl *, u32, u32);
-int	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
-		struct se_lun *, struct se_lun_acl *);
+void	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
+					    struct se_lun *, struct se_lun_acl *, struct se_dev_entry*);
 void	core_dev_free_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_lun_acl *lacl);
 int	core_dev_setup_virtual_lun0(void);

@@ -98,8 +100,7 @@ static inline void release_deve(struct kref *kref)
 	struct se_dev_entry *deve = container_of(kref,
 				struct se_dev_entry, refcount);
 
-	/* doesn't really get freed because device_list is a static array */
-	deve->lun_flags &= ~TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
+	kfree(deve);
 }
 
 #define get_deve(x) kref_get(&x->refcount)

drivers/target/target_core_pr.c +21 -4

index 17a4c3b..0da6696 100644
@@ -322,7 +322,11 @@ static int core_scsi3_pr_seq_non_holder(
 	int legacy = 0; /* Act like a legacy device and return
 			 * RESERVATION CONFLICT on some CDBs */
 
-	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
+	se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+	/* no deve means no conflict */
+	if (!se_deve)
+		return 0;
+
 	/*
 	 * Determine if the registration should be ignored due to
 	 * non-matching ISIDs in target_scsi3_pr_reservation_check().

@@ -950,7 +954,11 @@ int core_scsi3_check_aptpl_registration(
 	struct se_lun_acl *lun_acl)
 {
 	struct se_node_acl *nacl = lun_acl->se_lun_nacl;
-	struct se_dev_entry *deve = nacl->device_list[lun_acl->mapped_lun];
+	struct se_dev_entry *deve;
+
+	deve = transport_search_deve(nacl, lun_acl->mapped_lun);
+	if (!deve)
+		return 0;
 
 	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)
 		return 0;

@@ -1424,7 +1432,10 @@ core_scsi3_decode_spec_i_port(
 
 	memset(dest_iport, 0, 64);
 
-	local_se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
+	local_se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+	if (!local_se_deve)
+		return 0;
+
 	/*
 	 * Allocate a struct pr_transport_id_holder and setup the
 	 * local_node_acl and local_se_deve pointers and add to

@@ -1988,7 +1999,6 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 	}
 	se_tpg = se_sess->se_tpg;
-	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
 
 	if (se_tpg->se_tpg_tfo->sess_get_initiator_sid) {
 		memset(&isid_buf[0], 0, PR_REG_ISID_LEN);

@@ -2013,6 +2023,13 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 			return 0;
 
 		if (!spec_i_pt) {
+
+			se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+			if (!se_deve) {
+				pr_err("se_deve not found\n");
+				return TCM_INVALID_PARAMETER_LIST;
+			}
+
 			/*
 			 * Perform the Service Action REGISTER on the Initiator
 			 * Port Endpoint that the PRO was received from on the

drivers/target/target_core_spc.c +5 -6

index 8f52974..0b678fc 100644
@@ -1124,10 +1124,10 @@ static sense_reason_t spc_emulate_request_sense(struct se_cmd *cmd)
 
 sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 {
-	struct se_dev_entry *deve;
 	struct se_session *sess = cmd->se_sess;
 	unsigned char *buf;
-	u32 lun_count = 0, offset = 8, i;
+	u32 lun_count = 0, offset = 8;
+	struct rb_node *node;
 
 	if (cmd->data_length < 16) {
 		pr_warn("REPORT LUNS allocation length %u too small\n",

@@ -1151,10 +1151,9 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 	}
 
 	spin_lock_irq(&sess->se_node_acl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = sess->se_node_acl->device_list[i];
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
+	for (node = rb_first(&sess->se_node_acl->rb_device_list); node; node = rb_next(node)) {
+		struct se_dev_entry *deve = container_of(node, struct se_dev_entry, rb_node);
+
 		/*
 		 * We determine the correct LUN LIST LENGTH even once we
 		 * have reached the initial allocation length.

drivers/target/target_core_stat.c +36 ~~-36~~

180 lines not on this page · raw message

drivers/target/target_core_tpg.c +5 ~~-48~~

87 lines not on this page · raw message

drivers/target/target_core_ua.c +11 -7

index 2a1cbf9..5260442 100644
@@ -51,8 +51,8 @@ target_scsi3_ua_check(struct se_cmd *cmd)
 	if (!nacl)
 		return 0;
 
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count))
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count))
 		return 0;
 	/*
 	 * From sam4r14, section 5.14 Unit attention condition:

@@ -105,7 +105,11 @@ int core_scsi3_ua_allocate(
 	ua->ua_ascq = ascq;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[unpacked_lun];
+	deve = transport_search_deve(nacl, unpacked_lun);
+	if (!deve) {
+		spin_unlock_irq(&nacl->device_list_lock);
+		return 0;
+	}
 
 	spin_lock(&deve->ua_lock);
 	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_node) {

@@ -215,8 +219,8 @@ void core_scsi3_ua_for_check_condition(
 		return;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count)) {
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count)) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return;
 	}

@@ -284,8 +288,8 @@ int core_scsi3_ua_clear_for_request_sense(
 		return -EINVAL;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count)) {
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count)) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -EPERM;
 	}

include/target/target_core_base.h +4 -4

index 513429a..3f54fee 100644
@@ -171,9 +171,8 @@ enum se_cmd_flags_table {
 /* struct se_dev_entry->lun_flags and struct se_lun->lun_access */
 enum transport_lunflags_table {
 	TRANSPORT_LUNFLAGS_NO_ACCESS		= 0x00,
-	TRANSPORT_LUNFLAGS_INITIATOR_ACCESS	= 0x01,
-	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x02,
-	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x04,
+	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x01,
+	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x02,
 };
 
 /*

@@ -522,7 +521,7 @@ struct se_node_acl {
 	spinlock_t		stats_lock;
 	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
 	atomic_t		acl_pr_ref_count;
-	struct se_dev_entry	**device_list;
+	struct rb_root		rb_device_list;
 	struct se_session	*nacl_sess;
 	struct se_portal_group *se_tpg;
 	spinlock_t		device_list_lock;

@@ -576,6 +575,7 @@ struct se_lun_acl {
 };
 
 struct se_dev_entry {
+	struct rb_node		rb_node;
 	bool			def_pr_registered;
 	/* See transport_lunflags_table */
 	u32			lun_flags;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:40 UTC #42 lore raw

Re: [PATCH 19/32] target: Convert to rbtree for se_dev_entry in se_node_acl

Quoted 5 lines · Andy Grover

Instead of an array, use a rbtree. Less memory use on average, and can allow >255 entries. We go from O(1) to O(log n) on lookups. If this shows up on profiling (it won't) then transition to other kernel lookup methods is straightforward from here.

Ugh.

There is no reason to be using rbtrees in a performance critical path here.

The number of pointer lookups is what ends up hurting the most vs. a flat array, so given that 256 LUNs per endpoint is currently not an issue, and there is no hard limit on the number of endpoints with virtual addressing, I don't see the benefit of this patch.

NAK.

--nab

Quoted 985 lines

Change core_disable_device_list_for_node to be called with device_list_lock held, and tweaked params a little.

TRANSPORT_LUNFLAGS_INITIATOR_ACCESS no longer needed, presence in the rbtree is equivalent to this being set.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c          |  213 ++++++++++++++-----------
 drivers/target/target_core_fabric_configfs.c |   24 ++--
 drivers/target/target_core_internal.h        |   15 +-
 drivers/target/target_core_pr.c              |   25 +++-
 drivers/target/target_core_spc.c             |   11 +-
 drivers/target/target_core_stat.c            |   72 +++++-----
 drivers/target/target_core_tpg.c             |   53 +------
 drivers/target/target_core_ua.c              |   18 ++-
 include/target/target_core_base.h            |    8 +-
 9 files changed, 222 insertions(+), 217 deletions(-)

drivers/target/target_core_device.c +119 ~~-94~~

index af12456..a18724b 100644
@@ -54,6 +54,51 @@ static struct se_hba *lun0_hba;
 /* not static, needed by tpg.c */
 struct se_device *g_lun0_dev;
 
+bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve)
+{
+	struct rb_root *root = &nacl->rb_device_list;
+	struct rb_node **new = &(root->rb_node), *parent = NULL;
+
+	/* Figure out where to put new node */
+	while (*new) {
+		struct se_dev_entry *this = rb_entry(*new, struct se_dev_entry, rb_node);
+
+		parent = *new;
+		if (deve->mapped_lun < this->mapped_lun)
+			new = &((*new)->rb_left);
+		else if (deve->mapped_lun > this->mapped_lun)
+			new = &((*new)->rb_right);
+		else
+			return false;
+	}
+
+	/* Add new node and rebalance tree. */
+	rb_link_node(&deve->rb_node, parent, new);
+	rb_insert_color(&deve->rb_node, root);
+
+	return true;
+}
+
+
+struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun)
+{
+	struct rb_root *root = &nacl->rb_device_list;
+	struct rb_node *node = root->rb_node;
+
+	while (node) {
+		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
+
+		if (mapped_lun < deve->mapped_lun)
+			node = node->rb_left;
+		else if (mapped_lun > deve->mapped_lun)
+			node = node->rb_right;
+		else
+			return deve;
+	}
+	return NULL;
+}
+
+
 sense_reason_t
 transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 {

@@ -66,8 +111,8 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		return TCM_NON_EXISTENT_LUN;
 
 	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
-	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
-	if (se_cmd->se_deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
+	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
+	if (se_cmd->se_deve) {
 		struct se_dev_entry *deve = se_cmd->se_deve;
 
 		deve->total_cmds++;

@@ -153,10 +198,10 @@ int transport_lookup_tmr_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		return -ENODEV;
 
 	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
-	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
-	deve = se_cmd->se_deve;
+	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
+	if (se_cmd->se_deve) {
+		deve = se_cmd->se_deve;
 
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
 		se_tmr->tmr_lun = deve->se_lun;
 		se_cmd->se_lun = deve->se_lun;
 		se_lun = deve->se_lun;

@@ -187,25 +232,22 @@ EXPORT_SYMBOL(transport_lookup_tmr_lun);
 
 /*
  * This function is called from core_scsi3_emulate_pro_register_and_move()
- * and core_scsi3_decode_spec_i_port(), and will increment &deve->pr_ref_count
+ * and core_scsi3_decode_spec_i_port(), and will increment deve->refcount
  * when a matching rtpi is found.
  */
 struct se_dev_entry *core_get_se_deve_from_rtpi(
 	struct se_node_acl *nacl,
 	u16 rtpi)
 {
-	struct se_dev_entry *deve;
 	struct se_lun *lun;
 	struct se_port *port;
 	struct se_portal_group *tpg = nacl->se_tpg;
-	u32 i;
+	struct rb_node *node;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
+	for (node = rb_first(&nacl->rb_device_list); node; node = rb_next(node))
+	{
+		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
 
 		lun = deve->se_lun;
 		if (!lun) {

@@ -234,43 +276,24 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
 	return NULL;
 }
 
-int core_free_device_list_for_node(
+void core_free_device_list_for_node(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
-	struct se_dev_entry *deve;
-	struct se_lun *lun;
-	u32 i;
-
-	if (!nacl->device_list)
-		return 0;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
-
-		if (!deve->se_lun) {
+	rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
+		if (deve->se_lun) {
+			core_disable_device_list_for_node(deve->se_lun, deve,
+				TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
+		} else {
 			pr_err("%s device entries device pointer is"
 				" NULL, but Initiator has access.\n",
 				tpg->se_tpg_tfo->get_fabric_name());
-			continue;
 		}
-		lun = deve->se_lun;
-
-		spin_unlock_irq(&nacl->device_list_lock);
-		core_disable_device_list_for_node(lun, NULL, deve->mapped_lun,
-			TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
-		spin_lock_irq(&nacl->device_list_lock);
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
-
-	array_free(nacl->device_list, TRANSPORT_MAX_LUNS_PER_TPG);
-	nacl->device_list = NULL;
-
-	return 0;
 }
 
 void core_update_device_list_access(

@@ -281,13 +304,15 @@ void core_update_device_list_access(
 	struct se_dev_entry *deve;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[mapped_lun];
-	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
-	} else {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+	deve = transport_search_deve(nacl, mapped_lun);
+	if (deve) {
+		if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
+			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
+			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
+		} else {
+			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
+			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+		}
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
 }

@@ -305,18 +330,30 @@ int core_enable_device_list_for_node(
 	struct se_portal_group *tpg)
 {
 	struct se_port *port = lun->lun_sep;
-	struct se_dev_entry *deve;
+	struct se_dev_entry *deve, *new_deve;
 
-	spin_lock_irq(&nacl->device_list_lock);
+	new_deve = kzalloc(sizeof(*deve), GFP_KERNEL);
+	if (!new_deve)
+		return -ENOMEM;
 
-	deve = nacl->device_list[mapped_lun];
+	atomic_set(&new_deve->ua_count, 0);
+	kref_init(&new_deve->refcount);
+	spin_lock_init(&new_deve->ua_lock);
+	INIT_LIST_HEAD(&new_deve->alua_port_node);
+	INIT_LIST_HEAD(&new_deve->ua_list);
+	new_deve->se_lun = lun;
+	new_deve->se_lun_acl = lun_acl;
+	new_deve->mapped_lun = mapped_lun;
 
+	spin_lock_irq(&nacl->device_list_lock);
 	/*
 	 * Check if the call is handling demo mode -> explicit LUN ACL
 	 * transition.  This transition must be for the same struct se_lun
 	 * + mapped_lun that was setup in demo mode..
 	 */
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
+	deve = transport_search_deve(nacl, mapped_lun);
+	if (deve) {
+		kfree(new_deve);
 		if (deve->se_lun_acl != NULL) {
 			pr_err("struct se_dev_entry->se_lun_acl"
 			       " already set for demo mode -> explicit"

@@ -345,25 +382,24 @@ int core_enable_device_list_for_node(
 		return 0;
 	}
 
-	deve->se_lun = lun;
-	deve->se_lun_acl = lun_acl;
-	deve->mapped_lun = mapped_lun;
-	deve->lun_flags |= TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
-
-	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
-	} else {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+	if (!transport_insert_deve(nacl, new_deve)) {
+		pr_err("Could not insert new deve for lun %d\n", mapped_lun);
+		kfree(new_deve);
+		spin_unlock_irq(&nacl->device_list_lock);
+		return -EINVAL;
 	}
 
-	deve->creation_time = get_jiffies_64();
-	deve->attach_count++;
+	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE)
+		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
+	else
+		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+
+	new_deve->creation_time = get_jiffies_64();
+	new_deve->attach_count++;
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	spin_lock_bh(&port->sep_alua_lock);
-	list_add_tail(&deve->alua_port_node, &port->sep_alua_list);
+	list_add_tail(&new_deve->alua_port_node, &port->sep_alua_list);
 	spin_unlock_bh(&port->sep_alua_lock);
 
 	return 0;

@@ -371,18 +407,16 @@ int core_enable_device_list_for_node(
 
 /*      core_disable_device_list_for_node():
  *
- *
+ *	CALLED WITH device_list_lock HELD
  */
-int core_disable_device_list_for_node(
+void core_disable_device_list_for_node(
 	struct se_lun *lun,
-	struct se_lun_acl *lun_acl,
-	u32 mapped_lun,
+	struct se_dev_entry *deve,
 	u32 lun_access,
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
 	struct se_port *port = lun->lun_sep;
-	struct se_dev_entry *deve = nacl->device_list[mapped_lun];
 
 	/*
 	 * If the MappedLUN entry is being disabled, the entry in

@@ -397,11 +431,10 @@ int core_disable_device_list_for_node(
 	 * NodeACL context specific PR metadata for demo-mode
 	 * MappedLUN *deve will be released below..
 	 */
-	spin_lock_bh(&port->sep_alua_lock);
+	spin_lock(&port->sep_alua_lock);
 	list_del(&deve->alua_port_node);
-	spin_unlock_bh(&port->sep_alua_lock);
+	spin_unlock(&port->sep_alua_lock);
 
-	spin_lock_irq(&nacl->device_list_lock);
 	/*
 	 * Disable struct se_dev_entry LUN ACL mapping
 	 */

@@ -411,13 +444,12 @@ int core_disable_device_list_for_node(
 	deve->lun_flags = 0;
 	deve->creation_time = 0;
 	deve->attach_count--;
-	spin_unlock_irq(&nacl->device_list_lock);
 
 	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
 
-	put_deve(deve);
+	rb_erase(&deve->rb_node, &nacl->rb_device_list);
 
-	return 0;
+	put_deve(deve);
 }
 
 /*      core_clear_lun_from_tpg():

@@ -427,25 +459,17 @@ int core_disable_device_list_for_node(
 void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 {
 	struct se_node_acl *nacl;
-	struct se_dev_entry *deve;
-	u32 i;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&tpg->acl_node_lock);
 	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
 		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);
-		for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-			deve = nacl->device_list[i];
-			if (lun != deve->se_lun)
-				continue;
-			spin_unlock_irq(&nacl->device_list_lock);
-
-			core_disable_device_list_for_node(lun, NULL,
-				deve->mapped_lun, TRANSPORT_LUNFLAGS_NO_ACCESS,
-				nacl, tpg);
-
-			spin_lock_irq(&nacl->device_list_lock);
+		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
+			if (lun == deve->se_lun)
+				core_disable_device_list_for_node(lun, deve,
+					TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
 		}
 		spin_unlock_irq(&nacl->device_list_lock);

@@ -1301,16 +1325,17 @@ int core_dev_add_initiator_node_lun_acl(
  *
  *
  */
-int core_dev_del_initiator_node_lun_acl(
+void core_dev_del_initiator_node_lun_acl(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
-	struct se_lun_acl *lacl)
+	struct se_lun_acl *lacl,
+	struct se_dev_entry *deve)
 {
 	struct se_node_acl *nacl;
 
 	nacl = lacl->se_lun_nacl;
 	if (!nacl)
-		return -EINVAL;
+		return;
 
 	spin_lock(&lun->lun_acl_lock);
 	list_del(&lacl->lacl_list);

@@ -1318,8 +1343,10 @@ int core_dev_del_initiator_node_lun_acl(
 	smp_mb__after_atomic_dec();
 	spin_unlock(&lun->lun_acl_lock);
 
-	core_disable_device_list_for_node(lun, NULL, lacl->mapped_lun,
+	spin_lock_irq(&nacl->device_list_lock);
+	core_disable_device_list_for_node(lun, deve,
 		TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
+	spin_unlock_irq(&nacl->device_list_lock);
 
 	lacl->se_lun = NULL;

@@ -1328,8 +1355,6 @@ int core_dev_del_initiator_node_lun_acl(
 		tpg->se_tpg_tfo->get_fabric_name(),
 		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		lacl->initiatorname, lacl->mapped_lun);
-
-	return 0;
 }
 
 void core_dev_free_initiator_node_lun_acl(

drivers/target/target_core_fabric_configfs.c +13 ~~-11~~

index 64cc4dc..eaaed43 100644
@@ -112,8 +112,8 @@ static int target_fabric_mappedlun_link(
 	 * tpg_1/attrib/demo_mode_write_protect=1
 	 */
 	spin_lock_irq(&lacl->se_lun_nacl->device_list_lock);
-	deve = lacl->se_lun_nacl->device_list[lacl->mapped_lun];
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS)
+	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
+	if (deve)
 		lun_access = deve->lun_flags;
 	else
 		lun_access =

@@ -140,19 +140,20 @@ static int target_fabric_mappedlun_unlink(
 	struct se_lun *lun;
 	struct se_lun_acl *lacl = container_of(to_config_group(lun_acl_ci),
 			struct se_lun_acl, se_lun_group);
-	struct se_node_acl *nacl = lacl->se_lun_nacl;
-	struct se_dev_entry *deve = nacl->device_list[lacl->mapped_lun];
+	struct se_dev_entry *deve;
 	struct se_portal_group *se_tpg;
+
 	/*
 	 * Determine if the underlying MappedLUN has already been released..
 	 */
-	if (!deve->se_lun)
+	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
+	if (!(deve && deve->se_lun))
 		return 0;
 
 	lun = container_of(to_config_group(lun_ci), struct se_lun, lun_group);
 	se_tpg = lun->lun_sep->sep_tpg;
 
-	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl);
+	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl, deve);
 	return 0;
 }

@@ -169,13 +170,14 @@ static ssize_t target_fabric_mappedlun_show_write_protect(
 {
 	struct se_node_acl *se_nacl = lacl->se_lun_nacl;
 	struct se_dev_entry *deve;
-	ssize_t len;
+	ssize_t len = 0;
 
 	spin_lock_irq(&se_nacl->device_list_lock);
-	deve = se_nacl->device_list[lacl->mapped_lun];
-	len = sprintf(page, "%d\n",
-			(deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
-			1 : 0);
+	deve = transport_search_deve(se_nacl, lacl->mapped_lun);
+	if (deve)
+		len = sprintf(page, "%d\n",
+			      (deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
+			      1 : 0);
 	spin_unlock_irq(&se_nacl->device_list_lock);
 
 	return len;

drivers/target/target_core_internal.h +8 -7

index 5d1e4ec..3803dd8 100644
@@ -5,14 +5,16 @@
 extern struct t10_alua_lu_gp *default_lu_gp;
 
 /* target_core_device.c */
+bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve);
+struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun);
 struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
-int	core_free_device_list_for_node(struct se_node_acl *,
+void	core_free_device_list_for_node(struct se_node_acl *,
 		struct se_portal_group *);
 void	core_update_device_list_access(u32, u32, struct se_node_acl *);
 int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
 		u32, u32, struct se_node_acl *, struct se_portal_group *);
-int	core_disable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
-		u32, u32, struct se_node_acl *, struct se_portal_group *);
+void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
+		u32, struct se_node_acl *, struct se_portal_group *);
 void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
 int	core_dev_export(struct se_device *, struct se_portal_group *,
 		struct se_lun *);

@@ -50,8 +52,8 @@ struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *
 		struct se_node_acl *, u32, int *);
 int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_lun_acl *, u32, u32);
-int	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
-		struct se_lun *, struct se_lun_acl *);
+void	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
+					    struct se_lun *, struct se_lun_acl *, struct se_dev_entry*);
 void	core_dev_free_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_lun_acl *lacl);
 int	core_dev_setup_virtual_lun0(void);

@@ -98,8 +100,7 @@ static inline void release_deve(struct kref *kref)
 	struct se_dev_entry *deve = container_of(kref,
 				struct se_dev_entry, refcount);
 
-	/* doesn't really get freed because device_list is a static array */
-	deve->lun_flags &= ~TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
+	kfree(deve);
 }
 
 #define get_deve(x) kref_get(&x->refcount)

drivers/target/target_core_pr.c +21 -4

index 17a4c3b..0da6696 100644
@@ -322,7 +322,11 @@ static int core_scsi3_pr_seq_non_holder(
 	int legacy = 0; /* Act like a legacy device and return
 			 * RESERVATION CONFLICT on some CDBs */
 
-	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
+	se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+	/* no deve means no conflict */
+	if (!se_deve)
+		return 0;
+
 	/*
 	 * Determine if the registration should be ignored due to
 	 * non-matching ISIDs in target_scsi3_pr_reservation_check().

@@ -950,7 +954,11 @@ int core_scsi3_check_aptpl_registration(
 	struct se_lun_acl *lun_acl)
 {
 	struct se_node_acl *nacl = lun_acl->se_lun_nacl;
-	struct se_dev_entry *deve = nacl->device_list[lun_acl->mapped_lun];
+	struct se_dev_entry *deve;
+
+	deve = transport_search_deve(nacl, lun_acl->mapped_lun);
+	if (!deve)
+		return 0;
 
 	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)
 		return 0;

@@ -1424,7 +1432,10 @@ core_scsi3_decode_spec_i_port(
 
 	memset(dest_iport, 0, 64);
 
-	local_se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
+	local_se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+	if (!local_se_deve)
+		return 0;
+
 	/*
 	 * Allocate a struct pr_transport_id_holder and setup the
 	 * local_node_acl and local_se_deve pointers and add to

@@ -1988,7 +1999,6 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 	}
 	se_tpg = se_sess->se_tpg;
-	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
 
 	if (se_tpg->se_tpg_tfo->sess_get_initiator_sid) {
 		memset(&isid_buf[0], 0, PR_REG_ISID_LEN);

@@ -2013,6 +2023,13 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 			return 0;
 
 		if (!spec_i_pt) {
+
+			se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+			if (!se_deve) {
+				pr_err("se_deve not found\n");
+				return TCM_INVALID_PARAMETER_LIST;
+			}
+
 			/*
 			 * Perform the Service Action REGISTER on the Initiator
 			 * Port Endpoint that the PRO was received from on the

drivers/target/target_core_spc.c +5 -6

index 8f52974..0b678fc 100644
@@ -1124,10 +1124,10 @@ static sense_reason_t spc_emulate_request_sense(struct se_cmd *cmd)
 
 sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 {
-	struct se_dev_entry *deve;
 	struct se_session *sess = cmd->se_sess;
 	unsigned char *buf;
-	u32 lun_count = 0, offset = 8, i;
+	u32 lun_count = 0, offset = 8;
+	struct rb_node *node;
 
 	if (cmd->data_length < 16) {
 		pr_warn("REPORT LUNS allocation length %u too small\n",

@@ -1151,10 +1151,9 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 	}
 
 	spin_lock_irq(&sess->se_node_acl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = sess->se_node_acl->device_list[i];
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
+	for (node = rb_first(&sess->se_node_acl->rb_device_list); node; node = rb_next(node)) {
+		struct se_dev_entry *deve = container_of(node, struct se_dev_entry, rb_node);
+
 		/*
 		 * We determine the correct LUN LIST LENGTH even once we
 		 * have reached the initial allocation length.

drivers/target/target_core_stat.c +36 ~~-36~~

index 0353899..ef03d51 100644
@@ -1086,8 +1086,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_inst(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1111,8 +1111,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_dev(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1135,8 +1135,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_port(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1158,8 +1158,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_indx(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1180,8 +1180,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_dev_or_port(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1202,8 +1202,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_intr_name(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1224,8 +1224,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_map_indx(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1246,8 +1246,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_att_count(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1268,8 +1268,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_num_cmds(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1290,8 +1290,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_read_mbytes(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1312,8 +1312,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_write_mbytes(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1334,8 +1334,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_hs_num_cmds(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1356,8 +1356,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_creation_time(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1379,8 +1379,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_row_status(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1452,8 +1452,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_inst(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1477,8 +1477,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_dev(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1501,8 +1501,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_port(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

@@ -1550,8 +1550,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_port_auth_indx(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}

drivers/target/target_core_tpg.c +5 ~~-48~~

index e6f9dfb..3efa7c1 100644
@@ -54,17 +54,11 @@ static void core_clear_initiator_node_from_tpg(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
-	int i;
-	struct se_dev_entry *deve;
 	struct se_lun *lun;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
-
+	rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
 		if (!deve->se_lun) {
 			pr_err("%s device entries device pointer is"
 				" NULL, but Initiator has access.\n",

@@ -73,11 +67,8 @@ static void core_clear_initiator_node_from_tpg(
 		}
 
 		lun = deve->se_lun;
-		spin_unlock_irq(&nacl->device_list_lock);
-		core_disable_device_list_for_node(lun, NULL, deve->mapped_lun,
+		core_disable_device_list_for_node(lun, deve,
 			TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
-
-		spin_lock_irq(&nacl->device_list_lock);
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
 }

@@ -217,34 +208,6 @@ static void *array_zalloc(int n, size_t size, gfp_t flags)
 	return a;
 }
 
-/*      core_create_device_list_for_node():
- *
- *
- */
-static int core_create_device_list_for_node(struct se_node_acl *nacl)
-{
-	struct se_dev_entry *deve;
-	int i;
-
-	nacl->device_list = array_zalloc(TRANSPORT_MAX_LUNS_PER_TPG,
-			sizeof(struct se_dev_entry), GFP_KERNEL);
-	if (!nacl->device_list) {
-		pr_err("Unable to allocate memory for"
-			" struct se_node_acl->device_list\n");
-		return -ENOMEM;
-	}
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		atomic_set(&deve->ua_count, 0);
-		kref_init(&deve->refcount);
-		spin_lock_init(&deve->ua_lock);
-		INIT_LIST_HEAD(&deve->alua_port_node);
-		INIT_LIST_HEAD(&deve->ua_list);
-	}
-
-	return 0;
-}
 
 /*	core_tpg_check_initiator_node_acl()
  *

@@ -283,10 +246,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 
 	tpg->se_tpg_tfo->set_default_node_attributes(acl);
 
-	if (core_create_device_list_for_node(acl) < 0) {
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
-		return NULL;
-	}
+	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
 		core_free_device_list_for_node(acl, tpg);

@@ -410,10 +370,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 
 	tpg->se_tpg_tfo->set_default_node_attributes(acl);
 
-	if (core_create_device_list_for_node(acl) < 0) {
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
-		return ERR_PTR(-ENOMEM);
-	}
+	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
 		core_free_device_list_for_node(acl, tpg);

drivers/target/target_core_ua.c +11 -7

index 2a1cbf9..5260442 100644
@@ -51,8 +51,8 @@ target_scsi3_ua_check(struct se_cmd *cmd)
 	if (!nacl)
 		return 0;
 
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count))
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count))
 		return 0;
 	/*
 	 * From sam4r14, section 5.14 Unit attention condition:

@@ -105,7 +105,11 @@ int core_scsi3_ua_allocate(
 	ua->ua_ascq = ascq;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[unpacked_lun];
+	deve = transport_search_deve(nacl, unpacked_lun);
+	if (!deve) {
+		spin_unlock_irq(&nacl->device_list_lock);
+		return 0;
+	}
 
 	spin_lock(&deve->ua_lock);
 	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_node) {

@@ -215,8 +219,8 @@ void core_scsi3_ua_for_check_condition(
 		return;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count)) {
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count)) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return;
 	}

@@ -284,8 +288,8 @@ int core_scsi3_ua_clear_for_request_sense(
 		return -EINVAL;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count)) {
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count)) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -EPERM;
 	}

include/target/target_core_base.h +4 -4

index 513429a..3f54fee 100644
@@ -171,9 +171,8 @@ enum se_cmd_flags_table {
 /* struct se_dev_entry->lun_flags and struct se_lun->lun_access */
 enum transport_lunflags_table {
 	TRANSPORT_LUNFLAGS_NO_ACCESS		= 0x00,
-	TRANSPORT_LUNFLAGS_INITIATOR_ACCESS	= 0x01,
-	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x02,
-	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x04,
+	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x01,
+	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x02,
 };
 
 /*

@@ -522,7 +521,7 @@ struct se_node_acl {
 	spinlock_t		stats_lock;
 	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
 	atomic_t		acl_pr_ref_count;
-	struct se_dev_entry	**device_list;
+	struct rb_root		rb_device_list;
 	struct se_session	*nacl_sess;
 	struct se_portal_group *se_tpg;
 	spinlock_t		device_list_lock;

@@ -576,6 +575,7 @@ struct se_lun_acl {
 };
 
 struct se_dev_entry {
+	struct rb_node		rb_node;
 	bool			def_pr_registered;
 	/* See transport_lunflags_table */
 	u32			lun_flags;

Andy Grover

author

2013-12-17 01:00 UTC #43 lore raw

Re: [PATCH 19/32] target: Convert to rbtree for se_dev_entry in se_node_acl

Quoted 12 lines · Nicholas A. Bellinger

Quoted 5 lines · Andy Grover
Instead of an array, use a rbtree. Less memory use on average, and can allow >255 entries. We go from O(1) to O(log n) on lookups. If this shows up on profiling (it won't) then transition to other kernel lookup methods is straightforward from here.
Ugh.
There is no reason to be using rbtrees in a performance critical path here.

Quoted 6 lines

The number of pointer lookups is what ends up hurting the most vs. a flat array, so given that 256 LUNs per endpoint is currently not an issue, and there is no hard limit on the number of endpoints with virtual addressing, I don't see the benefit of this patch.
NAK.

What does virtual addressing mean in this context? I'm not familiar -- so instead of reporting 257+ luns per tpg, it would get split up somehow?

-- Andy

Nicholas A. Bellinger

2013-12-17 01:54 UTC #44 lore raw

Re: [PATCH 19/32] target: Convert to rbtree for se_dev_entry in se_node_acl

Quoted 18 lines · Andy Grover

Quoted 17 lines · Nicholas A. Bellinger
Quoted 5 lines · Andy Grover
Instead of an array, use a rbtree. Less memory use on average, and can allow >255 entries. We go from O(1) to O(log n) on lookups. If this shows up on profiling (it won't) then transition to other kernel lookup methods is straightforward from here.
Ugh.
There is no reason to be using rbtrees in a performance critical path here.
The number of pointer lookups is what ends up hurting the most vs. a flat array, so given that 256 LUNs per endpoint is currently not an issue, and there is no hard limit on the number of endpoints with virtual addressing, I don't see the benefit of this patch.

Quoted 4 lines

Quoted 1 line
NAK.
What does virtual addressing mean in this context? I'm not familiar -- so instead of reporting 257+ luns per tpg, it would get split up somehow?

Virtual addressing meaning /sys/kernel/config/target/$FABRIC/$WWPN/ endpoints that are not tied to a specific hardware mapping, so any arbitrary number of $WWPNs can be created.

Right now the only use-case where people are using more than a few dozen LUNs per endpoint is with non NPIV qla2xxx mode, where a single physical Fibre Channel WWPN is exporting 256-512 LUNs, with different per NodeACL LUN mappings for different initiators.

With NPIV mode for qla2xxx (currently in development), the WWPN becomes virtualized like iSCSI, and a unsigned short about of virtual port addresses can be configured per physical FC port.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #45 lore raw

PATCH 20/32 target: Convert to rbtree for se_lun list in se_portal_group

In reply to Andy Grover

As with previous commit, this results in less average memory use, and allows lun count to no longer be restrained by the array size.

Remove array_free and array_zalloc.

For some reason, sbp fabric needs core_search lun, so export it for now.

Remove core_alloc_lun, it duplicates core_tpg_alloc_lun.

Change core_dev_add_lun to take a se_lun and return int

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/sbp/sbp_target.c              |   25 ++---
 drivers/target/target_core_device.c          |   79 +-----------
 drivers/target/target_core_fabric_configfs.c |   27 ++---
 drivers/target/target_core_internal.h        |    5 +-
 drivers/target/target_core_tpg.c             |  168 ++++++++++++-------------
 include/target/target_core_base.h            |   11 +--
 include/target/target_core_fabric.h          |    3 +
 7 files changed, 116 insertions(+), 202 deletions(-)

drivers/target/sbp/sbp_target.c +9 ~~-16~~

index e1ceae5..103998c 100644
@@ -185,9 +185,8 @@ static struct se_lun *sbp_get_lun_from_tpg(struct sbp_tpg *tpg, int lun)
 		return ERR_PTR(-EINVAL);
 
 	spin_lock(&se_tpg->tpg_lun_lock);
-	se_lun = se_tpg->tpg_lun_list[lun];
-
-	if (se_lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE)
+	se_lun = core_search_lun(se_tpg, lun);
+	if (!se_lun)
 		se_lun = ERR_PTR(-ENODEV);
 
 	spin_unlock(&se_tpg->tpg_lun_lock);

@@ -1936,15 +1935,11 @@ static char *sbp_parse_pr_out_transport_id(
 
 static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
 {
-	int i, count = 0;
+	int count = 0;
+	struct rb_node *node;
 
 	spin_lock(&tpg->tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		struct se_lun *se_lun = tpg->tpg_lun_list[i];
-
-		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
-			continue;
-
+	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
 		count++;
 	}
 	spin_unlock(&tpg->tpg_lun_lock);

@@ -1954,8 +1949,9 @@ static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
 
 static int sbp_update_unit_directory(struct sbp_tport *tport)
 {
-	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret, i;
+	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret;
 	u32 *data;
+	struct rb_node *node;
 
 	if (tport->unit_directory.data) {
 		fw_core_remove_descriptor(&tport->unit_directory);

@@ -2017,14 +2013,11 @@ static int sbp_update_unit_directory(struct sbp_tport *tport)
 	data[idx++] = 0x8d000000 | (num_luns + 1);
 
 	spin_lock(&tport->tpg->se_tpg.tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		struct se_lun *se_lun = tport->tpg->se_tpg.tpg_lun_list[i];
+	for (node = rb_first(&tport->tpg->se_tpg.rb_tpg_lun_list); node; node = rb_next(node)) {
+		struct se_lun *se_lun = container_of(node, struct se_lun, rb_node);
 		struct se_device *dev;
 		int type;
 
-		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
-			continue;
-
 		spin_unlock(&tport->tpg->se_tpg.tpg_lun_lock);
 
 		dev = se_lun->lun_se_dev;

drivers/target/target_core_device.c +7 ~~-72~~

111 lines not on this page · raw message

drivers/target/target_core_fabric_configfs.c +11 ~~-16~~

index eaaed43..1a31b46 100644
@@ -756,7 +756,6 @@ static int target_fabric_port_link(
 	struct config_item *tpg_ci;
 	struct se_lun *lun = container_of(to_config_group(lun_ci),
 				struct se_lun, lun_group);
-	struct se_lun *lun_p;
 	struct se_portal_group *se_tpg;
 	struct se_device *dev =
 		container_of(to_config_group(se_dev_ci), struct se_device, dev_group);

@@ -784,11 +783,10 @@ static int target_fabric_port_link(
 		return -EEXIST;
 	}
 
-	lun_p = core_dev_add_lun(se_tpg, dev, lun->unpacked_lun);
-	if (IS_ERR(lun_p)) {
+	ret = core_dev_add_lun(se_tpg, dev, lun);
+	if (ret < 0) {
 		pr_err("core_dev_add_lun() failed\n");
-		ret = PTR_ERR(lun_p);
-		goto out;
+		return ret;
 	}
 
 	if (tf->tf_ops.fabric_post_link) {

@@ -801,8 +799,6 @@ static int target_fabric_port_link(
 	}
 
 	return 0;
-out:
-	return ret;
 }
 
 static int target_fabric_port_unlink(

@@ -888,15 +884,16 @@ static struct config_group *target_fabric_make_lun(
 	if (unpacked_lun > UINT_MAX)
 		return ERR_PTR(-EINVAL);
 
-	lun = core_get_lun_from_tpg(se_tpg, unpacked_lun);
-	if (!lun)
-		return ERR_PTR(-EINVAL);
+	lun = core_tpg_alloc_lun(se_tpg, unpacked_lun);
+	if (IS_ERR(lun))
+		return ERR_CAST(lun);
 
 	lun_cg = &lun->lun_group;
 	lun_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
 				GFP_KERNEL);
 	if (!lun_cg->default_groups) {
 		pr_err("Unable to allocate lun_cg->default_groups\n");
+		core_tpg_free_lun(se_tpg, lun);
 		return ERR_PTR(-ENOMEM);
 	}

@@ -912,16 +909,14 @@ static struct config_group *target_fabric_make_lun(
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");
-		errno = -ENOMEM;
-		goto out;
+		kfree(lun_cg->default_groups);
+		core_tpg_free_lun(se_tpg, lun);
+		return ERR_PTR(-ENOMEM);
 	}
+
 	target_stat_setup_port_default_groups(lun);
 
 	return &lun->lun_group;
-out:
-	if (lun_cg)
-		kfree(lun_cg->default_groups);
-	return ERR_PTR(errno);
 }
 
 static void target_fabric_drop_lun(

drivers/target/target_core_internal.h +3 -2

index 3803dd8..ed4d514 100644
@@ -45,9 +45,8 @@ int	se_dev_set_max_sectors(struct se_device *, u32);
 int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
 int	se_dev_set_optimal_sectors(struct se_device *, u32);
 int	se_dev_set_block_size(struct se_device *, u32);
-struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
+int	core_dev_add_lun(struct se_portal_group *, struct se_device *, struct se_lun *);
 void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
-struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_node_acl *, u32, int *);
 int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,

@@ -94,6 +93,8 @@ int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);
 void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
 void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
+bool core_insert_lun(struct se_portal_group *tpg, struct se_lun *lun);
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 mapped_lun);
 
 static inline void release_deve(struct kref *kref)
 {

drivers/target/target_core_tpg.c +81 ~~-87~~

248 lines not on this page · raw message

include/target/target_core_base.h +2 -9

index 3f54fee..06f9bea 100644
@@ -122,12 +122,6 @@ enum hba_flags_table {
 	HBA_FLAGS_PSCSI_MODE	= 0x02,
 };
 
-/* struct se_lun->lun_status */
-enum transport_lun_status_table {
-	TRANSPORT_LUN_STATUS_FREE = 0,
-	TRANSPORT_LUN_STATUS_ACTIVE = 1,
-};
-
 /* struct se_portal_group->se_tpg_type */
 enum transport_tpg_type_table {
 	TRANSPORT_TPG_TYPE_NORMAL = 0,

@@ -634,10 +628,9 @@ struct se_port_stat_grps {
 };
 
 struct se_lun {
+	struct rb_node		rb_node;
 #define SE_LUN_LINK_MAGIC			0xffff7771
 	u32			lun_link_magic;
-	/* See transport_lun_status_table */
-	enum transport_lun_status_table lun_status;
 	u32			lun_access;
 	u32			lun_flags;
 	u32			unpacked_lun;

@@ -804,7 +797,7 @@ struct se_portal_group {
 	struct list_head	se_tpg_node;
 	/* linked list for initiator ACL list */
 	struct list_head	acl_node_list;
-	struct se_lun		**tpg_lun_list;
+	struct rb_root		rb_tpg_lun_list;
 	struct se_lun		tpg_virt_lun0;
 	/* List of TCM sessions associated wth this TPG */
 	struct list_head	tpg_sess_list;

include/target/target_core_fabric.h +3 -0

index 4cf4fda..0267780 100644
@@ -154,6 +154,9 @@ int	core_tpg_register(struct target_core_fabric_ops *, struct se_wwn *,
 		struct se_portal_group *, void *, int);
 int	core_tpg_deregister(struct se_portal_group *);
 
+/* SBP needs this. TODO: fix */
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 unpacked_lun);
+
 /* SAS helpers */
 u8	sas_get_fabric_proto_ident(struct se_portal_group *);
 u32	sas_get_pr_transport_id(struct se_portal_group *, struct se_node_acl *,

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 21:41 UTC #46 lore raw

Re: [PATCH 20/32] target: Convert to rbtree for se_lun list in se_portal_group

Quoted 3 lines · Andy Grover

As with previous commit, this results in less average memory use, and allows lun count to no longer be restrained by the array size.

NAK for the same reasons as the previous patch. rbtree introduces many extra memory addresses for pointer chasing during lookup vs. a flat array, so for a performance critical path this needs to be avoided.

--nab

Quoted 617 lines

Remove array_free and array_zalloc.

For some reason, sbp fabric needs core_search lun, so export it for now.

Remove core_alloc_lun, it duplicates core_tpg_alloc_lun.

Change core_dev_add_lun to take a se_lun and return int

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/sbp/sbp_target.c              |   25 ++---
 drivers/target/target_core_device.c          |   79 +-----------
 drivers/target/target_core_fabric_configfs.c |   27 ++---
 drivers/target/target_core_internal.h        |    5 +-
 drivers/target/target_core_tpg.c             |  168 ++++++++++++-------------
 include/target/target_core_base.h            |   11 +--
 include/target/target_core_fabric.h          |    3 +
 7 files changed, 116 insertions(+), 202 deletions(-)

drivers/target/sbp/sbp_target.c +9 ~~-16~~

index e1ceae5..103998c 100644
@@ -185,9 +185,8 @@ static struct se_lun *sbp_get_lun_from_tpg(struct sbp_tpg *tpg, int lun)
 		return ERR_PTR(-EINVAL);
 
 	spin_lock(&se_tpg->tpg_lun_lock);
-	se_lun = se_tpg->tpg_lun_list[lun];
-
-	if (se_lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE)
+	se_lun = core_search_lun(se_tpg, lun);
+	if (!se_lun)
 		se_lun = ERR_PTR(-ENODEV);
 
 	spin_unlock(&se_tpg->tpg_lun_lock);

@@ -1936,15 +1935,11 @@ static char *sbp_parse_pr_out_transport_id(
 
 static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
 {
-	int i, count = 0;
+	int count = 0;
+	struct rb_node *node;
 
 	spin_lock(&tpg->tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		struct se_lun *se_lun = tpg->tpg_lun_list[i];
-
-		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
-			continue;
-
+	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
 		count++;
 	}
 	spin_unlock(&tpg->tpg_lun_lock);

@@ -1954,8 +1949,9 @@ static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
 
 static int sbp_update_unit_directory(struct sbp_tport *tport)
 {
-	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret, i;
+	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret;
 	u32 *data;
+	struct rb_node *node;
 
 	if (tport->unit_directory.data) {
 		fw_core_remove_descriptor(&tport->unit_directory);

@@ -2017,14 +2013,11 @@ static int sbp_update_unit_directory(struct sbp_tport *tport)
 	data[idx++] = 0x8d000000 | (num_luns + 1);
 
 	spin_lock(&tport->tpg->se_tpg.tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		struct se_lun *se_lun = tport->tpg->se_tpg.tpg_lun_list[i];
+	for (node = rb_first(&tport->tpg->se_tpg.rb_tpg_lun_list); node; node = rb_next(node)) {
+		struct se_lun *se_lun = container_of(node, struct se_lun, rb_node);
 		struct se_device *dev;
 		int type;
 
-		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
-			continue;
-
 		spin_unlock(&tport->tpg->se_tpg.tpg_lun_lock);
 
 		dev = se_lun->lun_se_dev;

drivers/target/target_core_device.c +7 ~~-72~~

index a18724b..f2e1415 100644
@@ -1117,22 +1117,17 @@ int se_dev_set_block_size(struct se_device *dev, u32 block_size)
 	return 0;
 }
 
-struct se_lun *core_dev_add_lun(
+int core_dev_add_lun(
 	struct se_portal_group *tpg,
 	struct se_device *dev,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
 	int rc;
 
-	lun = core_tpg_alloc_lun(tpg, unpacked_lun);
-	if (IS_ERR(lun))
-		return lun;
-
 	rc = core_tpg_add_lun(tpg, lun,
 				TRANSPORT_LUNFLAGS_READ_WRITE, dev);
 	if (rc < 0)
-		return ERR_PTR(rc);
+		return rc;
 
 	pr_debug("%s_TPG[%u]_LUN[%u] - Activated %s Logical Unit from"
 		" CORE HBA: %u\n", tpg->se_tpg_tfo->get_fabric_name(),

@@ -1157,7 +1152,7 @@ struct se_lun *core_dev_add_lun(
 		spin_unlock_irq(&tpg->acl_node_lock);
 	}
 
-	return lun;
+	return rc;
 }
 
 /*      core_dev_del_lun():

@@ -1178,68 +1173,6 @@ void core_dev_del_lun(
 	core_tpg_free_lun(tpg, lun);
 }
 
-struct se_lun *core_get_lun_from_tpg(struct se_portal_group *tpg, u32 unpacked_lun)
-{
-	struct se_lun *lun;
-
-	spin_lock(&tpg->tpg_lun_lock);
-	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
-		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS"
-			"_PER_TPG-1: %u for Target Portal Group: %hu\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			TRANSPORT_MAX_LUNS_PER_TPG-1,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	lun = tpg->tpg_lun_list[unpacked_lun];
-
-	if (lun->lun_status != TRANSPORT_LUN_STATUS_FREE) {
-		pr_err("%s Logical Unit Number: %u is not free on"
-			" Target Portal Group: %hu, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	spin_unlock(&tpg->tpg_lun_lock);
-
-	return lun;
-}
-
-/*      core_dev_get_lun():
- *
- *
- */
-static struct se_lun *core_dev_get_lun(struct se_portal_group *tpg, u32 unpacked_lun)
-{
-	struct se_lun *lun;
-
-	spin_lock(&tpg->tpg_lun_lock);
-	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
-		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS_PER"
-			"_TPG-1: %u for Target Portal Group: %hu\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			TRANSPORT_MAX_LUNS_PER_TPG-1,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	lun = tpg->tpg_lun_list[unpacked_lun];
-
-	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
-		pr_err("%s Logical Unit Number: %u is not active on"
-			" Target Portal Group: %hu, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	spin_unlock(&tpg->tpg_lun_lock);
-
-	return lun;
-}
-
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(
 	struct se_portal_group *tpg,
 	struct se_node_acl *nacl,

@@ -1279,7 +1212,9 @@ int core_dev_add_initiator_node_lun_acl(
 	struct se_lun *lun;
 	struct se_node_acl *nacl;
 
-	lun = core_dev_get_lun(tpg, unpacked_lun);
+	spin_lock(&tpg->tpg_lun_lock);
+	lun = core_search_lun(tpg, unpacked_lun);
+	spin_unlock(&tpg->tpg_lun_lock);
 	if (!lun) {
 		pr_err("%s Logical Unit Number: %u is not active on"
 			" Target Portal Group: %hu, ignoring request.\n",

drivers/target/target_core_fabric_configfs.c +11 ~~-16~~

index eaaed43..1a31b46 100644
@@ -756,7 +756,6 @@ static int target_fabric_port_link(
 	struct config_item *tpg_ci;
 	struct se_lun *lun = container_of(to_config_group(lun_ci),
 				struct se_lun, lun_group);
-	struct se_lun *lun_p;
 	struct se_portal_group *se_tpg;
 	struct se_device *dev =
 		container_of(to_config_group(se_dev_ci), struct se_device, dev_group);

@@ -784,11 +783,10 @@ static int target_fabric_port_link(
 		return -EEXIST;
 	}
 
-	lun_p = core_dev_add_lun(se_tpg, dev, lun->unpacked_lun);
-	if (IS_ERR(lun_p)) {
+	ret = core_dev_add_lun(se_tpg, dev, lun);
+	if (ret < 0) {
 		pr_err("core_dev_add_lun() failed\n");
-		ret = PTR_ERR(lun_p);
-		goto out;
+		return ret;
 	}
 
 	if (tf->tf_ops.fabric_post_link) {

@@ -801,8 +799,6 @@ static int target_fabric_port_link(
 	}
 
 	return 0;
-out:
-	return ret;
 }
 
 static int target_fabric_port_unlink(

@@ -888,15 +884,16 @@ static struct config_group *target_fabric_make_lun(
 	if (unpacked_lun > UINT_MAX)
 		return ERR_PTR(-EINVAL);
 
-	lun = core_get_lun_from_tpg(se_tpg, unpacked_lun);
-	if (!lun)
-		return ERR_PTR(-EINVAL);
+	lun = core_tpg_alloc_lun(se_tpg, unpacked_lun);
+	if (IS_ERR(lun))
+		return ERR_CAST(lun);
 
 	lun_cg = &lun->lun_group;
 	lun_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
 				GFP_KERNEL);
 	if (!lun_cg->default_groups) {
 		pr_err("Unable to allocate lun_cg->default_groups\n");
+		core_tpg_free_lun(se_tpg, lun);
 		return ERR_PTR(-ENOMEM);
 	}

@@ -912,16 +909,14 @@ static struct config_group *target_fabric_make_lun(
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");
-		errno = -ENOMEM;
-		goto out;
+		kfree(lun_cg->default_groups);
+		core_tpg_free_lun(se_tpg, lun);
+		return ERR_PTR(-ENOMEM);
 	}
+
 	target_stat_setup_port_default_groups(lun);
 
 	return &lun->lun_group;
-out:
-	if (lun_cg)
-		kfree(lun_cg->default_groups);
-	return ERR_PTR(errno);
 }
 
 static void target_fabric_drop_lun(

drivers/target/target_core_internal.h +3 -2

index 3803dd8..ed4d514 100644
@@ -45,9 +45,8 @@ int	se_dev_set_max_sectors(struct se_device *, u32);
 int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
 int	se_dev_set_optimal_sectors(struct se_device *, u32);
 int	se_dev_set_block_size(struct se_device *, u32);
-struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
+int	core_dev_add_lun(struct se_portal_group *, struct se_device *, struct se_lun *);
 void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
-struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_node_acl *, u32, int *);
 int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,

@@ -94,6 +93,8 @@ int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);
 void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
 void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
+bool core_insert_lun(struct se_portal_group *tpg, struct se_lun *lun);
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 mapped_lun);
 
 static inline void release_deve(struct kref *kref)
 {

drivers/target/target_core_tpg.c +81 ~~-87~~

index 3efa7c1..0467ad8 100644
@@ -109,6 +109,50 @@ struct se_node_acl *core_tpg_get_initiator_node_acl(
 }
 EXPORT_SYMBOL(core_tpg_get_initiator_node_acl);
 
+bool core_insert_lun(struct se_portal_group *tpg, struct se_lun *lun)
+{
+	struct rb_root *root = &tpg->rb_tpg_lun_list;
+	struct rb_node **new = &(root->rb_node), *parent = NULL;
+
+	/* Figure out where to put new node */
+	while (*new) {
+		struct se_lun *this = rb_entry(*new, struct se_lun, rb_node);
+
+		parent = *new;
+		if (lun->unpacked_lun < this->unpacked_lun)
+			new = &((*new)->rb_left);
+		else if (lun->unpacked_lun > this->unpacked_lun)
+			new = &((*new)->rb_right);
+		else
+			return false;
+	}
+
+	/* Add new node and rebalance tree. */
+	rb_link_node(&lun->rb_node, parent, new);
+	rb_insert_color(&lun->rb_node, root);
+
+	return true;
+}
+
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 unpacked_lun)
+{
+	struct rb_root *root = &tpg->rb_tpg_lun_list;
+	struct rb_node *node = root->rb_node;
+
+	while (node) {
+		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
+
+		if (unpacked_lun < lun->unpacked_lun)
+			node = node->rb_left;
+		else if (unpacked_lun > lun->unpacked_lun)
+			node = node->rb_right;
+		else
+			return lun;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(core_search_lun);
+
 /*	core_tpg_add_node_to_devs():
  *
  *

@@ -117,16 +161,13 @@ void core_tpg_add_node_to_devs(
 	struct se_node_acl *acl,
 	struct se_portal_group *tpg)
 {
-	int i = 0;
 	u32 lun_access = 0;
-	struct se_lun *lun;
 	struct se_device *dev;
+	struct rb_node *node;
 
 	spin_lock(&tpg->tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		lun = tpg->tpg_lun_list[i];
-		if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE)
-			continue;
+	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
+		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
 
 		spin_unlock(&tpg->tpg_lun_lock);

@@ -180,35 +221,6 @@ static int core_set_queue_depth_for_node(
 	return 0;
 }
 
-void array_free(void *array, int n)
-{
-	void **a = array;
-	int i;
-
-	for (i = 0; i < n; i++)
-		kfree(a[i]);
-	kfree(a);
-}
-
-static void *array_zalloc(int n, size_t size, gfp_t flags)
-{
-	void **a;
-	int i;
-
-	a = kzalloc(n * sizeof(void*), flags);
-	if (!a)
-		return NULL;
-	for (i = 0; i < n; i++) {
-		a[i] = kzalloc(size, flags);
-		if (!a[i]) {
-			array_free(a, n);
-			return NULL;
-		}
-	}
-	return a;
-}
-
-
 /*	core_tpg_check_initiator_node_acl()
  *
  *

@@ -284,15 +296,13 @@ void core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *nacl)
 
 void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 {
-	int i;
-	struct se_lun *lun;
+	struct rb_node *node;
 
 	spin_lock(&tpg->tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		lun = tpg->tpg_lun_list[i];
+	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
+		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
 
-		if ((lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) ||
-		    (lun->lun_se_dev == NULL))
+		if (!lun->lun_se_dev)
 			continue;
 
 		spin_unlock(&tpg->tpg_lun_lock);

@@ -607,7 +617,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	int ret;
 
 	lun->unpacked_lun = 0;
-	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
+	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);

@@ -638,30 +648,6 @@ int core_tpg_register(
 	void *tpg_fabric_ptr,
 	int se_tpg_type)
 {
-	struct se_lun *lun;
-	u32 i;
-
-	se_tpg->tpg_lun_list = array_zalloc(TRANSPORT_MAX_LUNS_PER_TPG,
-			sizeof(struct se_lun), GFP_KERNEL);
-	if (!se_tpg->tpg_lun_list) {
-		pr_err("Unable to allocate struct se_portal_group->"
-				"tpg_lun_list\n");
-		return -ENOMEM;
-	}
-
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		lun = se_tpg->tpg_lun_list[i];
-		lun->unpacked_lun = i;
-		lun->lun_link_magic = SE_LUN_LINK_MAGIC;
-		lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
-		atomic_set(&lun->lun_acl_count, 0);
-		init_completion(&lun->lun_shutdown_comp);
-		INIT_LIST_HEAD(&lun->lun_acl_list);
-		spin_lock_init(&lun->lun_acl_lock);
-		spin_lock_init(&lun->lun_sep_lock);
-		init_completion(&lun->lun_ref_comp);
-	}
-
 	se_tpg->se_tpg_type = se_tpg_type;
 	se_tpg->se_tpg_fabric_ptr = tpg_fabric_ptr;
 	se_tpg->se_tpg_tfo = tfo;

@@ -673,13 +659,11 @@ int core_tpg_register(
 	spin_lock_init(&se_tpg->acl_node_lock);
 	spin_lock_init(&se_tpg->session_lock);
 	spin_lock_init(&se_tpg->tpg_lun_lock);
+	se_tpg->rb_tpg_lun_list = RB_ROOT;
 
 	if (se_tpg->se_tpg_type == TRANSPORT_TPG_TYPE_NORMAL) {
-		if (core_tpg_setup_virtual_lun0(se_tpg) < 0) {
-			array_free(se_tpg->tpg_lun_list,
-				   TRANSPORT_MAX_LUNS_PER_TPG);
+		if (core_tpg_setup_virtual_lun0(se_tpg) < 0)
 			return -ENOMEM;
-		}
 	}
 
 	spin_lock_bh(&tpg_lock);

@@ -737,7 +721,10 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 		core_tpg_release_virtual_lun0(se_tpg);
 
 	se_tpg->se_tpg_fabric_ptr = NULL;
-	array_free(se_tpg->tpg_lun_list, TRANSPORT_MAX_LUNS_PER_TPG);
+
+	/* Shouldn't be able to release tpg if luns present */
+	WARN_ON_ONCE(!RB_EMPTY_ROOT(&se_tpg->rb_tpg_lun_list));
+
 	return 0;
 }
 EXPORT_SYMBOL(core_tpg_deregister);

@@ -757,15 +744,28 @@ struct se_lun *core_tpg_alloc_lun(
 		return ERR_PTR(-EOVERFLOW);
 	}
 
+	lun = kzalloc(sizeof(*lun), GFP_KERNEL);
+	if (!lun)
+		return ERR_PTR(-ENOMEM);
+
+	lun->unpacked_lun = unpacked_lun;
+	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
+	atomic_set(&lun->lun_acl_count, 0);
+	init_completion(&lun->lun_shutdown_comp);
+	INIT_LIST_HEAD(&lun->lun_acl_list);
+	spin_lock_init(&lun->lun_acl_lock);
+	spin_lock_init(&lun->lun_sep_lock);
+	init_completion(&lun->lun_ref_comp);
+
 	spin_lock(&tpg->tpg_lun_lock);
-	lun = tpg->tpg_lun_list[unpacked_lun];
-	if (lun->lun_status == TRANSPORT_LUN_STATUS_ACTIVE) {
-		pr_err("TPG Logical Unit Number: %u is already active"
-			" on %s Target Portal Group: %u, ignoring request.\n",
-			unpacked_lun, tpg->se_tpg_tfo->get_fabric_name(),
+	if (!core_insert_lun(tpg, lun)) {
+		pr_err("%s Logical Unit Number: %u is not free on"
+			" Target Portal Group: %hu, ignoring request.\n",
+			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
 		spin_unlock(&tpg->tpg_lun_lock);
-		return ERR_PTR(-EINVAL);
+		kfree(lun);
+		return ERR_PTR(-EEXIST);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);

@@ -792,7 +792,6 @@ int core_tpg_add_lun(
 
 	spin_lock(&tpg->tpg_lun_lock);
 	lun->lun_access = lun_access;
-	lun->lun_status = TRANSPORT_LUN_STATUS_ACTIVE;
 	spin_unlock(&tpg->tpg_lun_lock);
 
 	return 0;

@@ -812,16 +811,11 @@ void core_tpg_free_lun(
 		return;
 	}
 
+	WARN_ON_ONCE(!list_empty(&lun->lun_acl_list));
+
 	spin_lock(&tpg->tpg_lun_lock);
-	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
-		pr_err("%s Logical Unit Number: %u is not active on"
-			" Target Portal Group: %u, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return;
-	}
-	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
+	rb_erase(&lun->rb_node, &tpg->rb_tpg_lun_list);
+	kfree(lun);
 	spin_unlock(&tpg->tpg_lun_lock);
 }

include/target/target_core_base.h +2 -9

index 3f54fee..06f9bea 100644
@@ -122,12 +122,6 @@ enum hba_flags_table {
 	HBA_FLAGS_PSCSI_MODE	= 0x02,
 };
 
-/* struct se_lun->lun_status */
-enum transport_lun_status_table {
-	TRANSPORT_LUN_STATUS_FREE = 0,
-	TRANSPORT_LUN_STATUS_ACTIVE = 1,
-};
-
 /* struct se_portal_group->se_tpg_type */
 enum transport_tpg_type_table {
 	TRANSPORT_TPG_TYPE_NORMAL = 0,

@@ -634,10 +628,9 @@ struct se_port_stat_grps {
 };
 
 struct se_lun {
+	struct rb_node		rb_node;
 #define SE_LUN_LINK_MAGIC			0xffff7771
 	u32			lun_link_magic;
-	/* See transport_lun_status_table */
-	enum transport_lun_status_table lun_status;
 	u32			lun_access;
 	u32			lun_flags;
 	u32			unpacked_lun;

@@ -804,7 +797,7 @@ struct se_portal_group {
 	struct list_head	se_tpg_node;
 	/* linked list for initiator ACL list */
 	struct list_head	acl_node_list;
-	struct se_lun		**tpg_lun_list;
+	struct rb_root		rb_tpg_lun_list;
 	struct se_lun		tpg_virt_lun0;
 	/* List of TCM sessions associated wth this TPG */
 	struct list_head	tpg_sess_list;

include/target/target_core_fabric.h +3 -0

index 4cf4fda..0267780 100644
@@ -154,6 +154,9 @@ int	core_tpg_register(struct target_core_fabric_ops *, struct se_wwn *,
 		struct se_portal_group *, void *, int);
 int	core_tpg_deregister(struct se_portal_group *);
 
+/* SBP needs this. TODO: fix */
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 unpacked_lun);
+
 /* SAS helpers */
 u8	sas_get_fabric_proto_ident(struct se_portal_group *);
 u32	sas_get_pr_transport_id(struct se_portal_group *, struct se_node_acl *,

Andy Grover

author

2013-12-13 23:59 UTC #47 lore raw

PATCH 21/32 target: Remove lun_link and device magic

In reply to Andy Grover

Not needed.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c          |    1 -
 drivers/target/target_core_fabric_configfs.c |   11 -----------
 drivers/target/target_core_tpg.c             |    2 --
 include/target/target_core_base.h            |    4 ----
 4 files changed, 0 insertions(+), 18 deletions(-)

drivers/target/target_core_device.c +0 -1

index f2e1415..e85a647 100644
@@ -1350,7 +1350,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
 	if (!dev)
 		return NULL;
 
-	dev->dev_link_magic = SE_DEV_LINK_MAGIC;
 	dev->se_hba = hba;
 	dev->transport = hba->transport;

drivers/target/target_core_fabric_configfs.c +0 ~~-11~~

index 1a31b46..4715836 100644
@@ -71,11 +71,6 @@ static int target_fabric_mappedlun_link(
 	struct config_item *nacl_ci, *tpg_ci, *tpg_ci_s, *wwn_ci, *wwn_ci_s;
 	int ret = 0, lun_access;
 
-	if (lun->lun_link_magic != SE_LUN_LINK_MAGIC) {
-		pr_err("Bad lun->lun_link_magic, not a valid lun_ci pointer:"
-			" %p to struct lun: %p\n", lun_ci, lun);
-		return -EFAULT;
-	}
 	/*
 	 * Ensure that the source port exists
 	 */

@@ -762,12 +757,6 @@ static int target_fabric_port_link(
 	struct target_fabric_configfs *tf;
 	int ret;
 
-	if (dev->dev_link_magic != SE_DEV_LINK_MAGIC) {
-		pr_err("Bad dev->dev_link_magic, not a valid se_dev_ci pointer:"
-			" %p to struct se_device: %p\n", se_dev_ci, dev);
-		return -EFAULT;
-	}
-
 	if (!(dev->dev_flags & DF_CONFIGURED)) {
 		pr_err("se_device not configured yet, cannot port link\n");
 		return -ENODEV;

drivers/target/target_core_tpg.c +0 -2

index 0467ad8..977b05c 100644
@@ -617,7 +617,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	int ret;
 
 	lun->unpacked_lun = 0;
-	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);

@@ -749,7 +748,6 @@ struct se_lun *core_tpg_alloc_lun(
 		return ERR_PTR(-ENOMEM);
 
 	lun->unpacked_lun = unpacked_lun;
-	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);

include/target/target_core_base.h +0 -4

index 06f9bea..2c0a595 100644
@@ -629,8 +629,6 @@ struct se_port_stat_grps {
 
 struct se_lun {
 	struct rb_node		rb_node;
-#define SE_LUN_LINK_MAGIC			0xffff7771
-	u32			lun_link_magic;
 	u32			lun_access;
 	u32			lun_flags;
 	u32			unpacked_lun;

@@ -655,8 +653,6 @@ struct se_dev_stat_grps {
 };
 
 struct se_device {
-#define SE_DEV_LINK_MAGIC			0xfeeddeef
-	u32			dev_link_magic;
 	/* RELATIVE TARGET PORT IDENTIFER Counter */
 	u16			dev_rpti_counter;
 	/* Used for SAM Task Attribute ordering */

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 21:43 UTC #48 lore raw

Re: [PATCH 21/32] target: Remove lun_link and device magic

Quoted 2 lines · Andy Grover

Not needed.

Big NAK on this one. It most certainly is still required in order to prevent any random pointer from being passed into a configfs symlink sink for Port/LUN and MappedLUN creation.

--nab

Quoted 91 lines

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c          |    1 -
 drivers/target/target_core_fabric_configfs.c |   11 -----------
 drivers/target/target_core_tpg.c             |    2 --
 include/target/target_core_base.h            |    4 ----
 4 files changed, 0 insertions(+), 18 deletions(-)

drivers/target/target_core_device.c +0 -1

index f2e1415..e85a647 100644
@@ -1350,7 +1350,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
 	if (!dev)
 		return NULL;
 
-	dev->dev_link_magic = SE_DEV_LINK_MAGIC;
 	dev->se_hba = hba;
 	dev->transport = hba->transport;

drivers/target/target_core_fabric_configfs.c +0 ~~-11~~

index 1a31b46..4715836 100644
@@ -71,11 +71,6 @@ static int target_fabric_mappedlun_link(
 	struct config_item *nacl_ci, *tpg_ci, *tpg_ci_s, *wwn_ci, *wwn_ci_s;
 	int ret = 0, lun_access;
 
-	if (lun->lun_link_magic != SE_LUN_LINK_MAGIC) {
-		pr_err("Bad lun->lun_link_magic, not a valid lun_ci pointer:"
-			" %p to struct lun: %p\n", lun_ci, lun);
-		return -EFAULT;
-	}
 	/*
 	 * Ensure that the source port exists
 	 */

@@ -762,12 +757,6 @@ static int target_fabric_port_link(
 	struct target_fabric_configfs *tf;
 	int ret;
 
-	if (dev->dev_link_magic != SE_DEV_LINK_MAGIC) {
-		pr_err("Bad dev->dev_link_magic, not a valid se_dev_ci pointer:"
-			" %p to struct se_device: %p\n", se_dev_ci, dev);
-		return -EFAULT;
-	}
-
 	if (!(dev->dev_flags & DF_CONFIGURED)) {
 		pr_err("se_device not configured yet, cannot port link\n");
 		return -ENODEV;

drivers/target/target_core_tpg.c +0 -2

index 0467ad8..977b05c 100644
@@ -617,7 +617,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	int ret;
 
 	lun->unpacked_lun = 0;
-	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);

@@ -749,7 +748,6 @@ struct se_lun *core_tpg_alloc_lun(
 		return ERR_PTR(-ENOMEM);
 
 	lun->unpacked_lun = unpacked_lun;
-	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);

include/target/target_core_base.h +0 -4

index 06f9bea..2c0a595 100644
@@ -629,8 +629,6 @@ struct se_port_stat_grps {
 
 struct se_lun {
 	struct rb_node		rb_node;
-#define SE_LUN_LINK_MAGIC			0xffff7771
-	u32			lun_link_magic;
 	u32			lun_access;
 	u32			lun_flags;
 	u32			unpacked_lun;

@@ -655,8 +653,6 @@ struct se_dev_stat_grps {
 };
 
 struct se_device {
-#define SE_DEV_LINK_MAGIC			0xfeeddeef
-	u32			dev_link_magic;
 	/* RELATIVE TARGET PORT IDENTIFER Counter */
 	u16			dev_rpti_counter;
 	/* Used for SAM Task Attribute ordering */

Andy Grover

author

2013-12-13 23:59 UTC #49 lore raw

PATCH 22/32 target: Convert percpu_ref to kref

In reply to Andy Grover

Wasn't getting release called when I expected, so punted and went down to krefs. Much simpler.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c    |    5 ++---
 drivers/target/target_core_internal.h  |   10 ++++++++++
 drivers/target/target_core_tpg.c       |   29 ++++-------------------------
 drivers/target/target_core_transport.c |   29 +----------------------------
 include/target/target_core_base.h      |    3 +--
 5 files changed, 18 insertions(+), 58 deletions(-)

drivers/target/target_core_device.c +2 -3

index e85a647..753e7ca 100644
@@ -138,7 +138,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		se_cmd->orig_fe_lun = unpacked_lun;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 
-		percpu_ref_get(&se_lun->lun_ref);
+		get_lun(se_lun);
 		se_cmd->lun_ref_active = true;
 	}
 	spin_unlock_irqrestore(&se_sess->se_node_acl->device_list_lock, flags);

@@ -168,7 +168,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		se_cmd->orig_fe_lun = 0;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 
-		percpu_ref_get(&se_lun->lun_ref);
+		get_lun(se_lun);
 		se_cmd->lun_ref_active = true;
 	}

@@ -1411,7 +1411,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
 	INIT_LIST_HEAD(&xcopy_lun->lun_acl_list);
 	spin_lock_init(&xcopy_lun->lun_acl_lock);
 	spin_lock_init(&xcopy_lun->lun_sep_lock);
-	init_completion(&xcopy_lun->lun_ref_comp);
 
 	return dev;
 }

drivers/target/target_core_internal.h +10 -0

index ed4d514..4d3b559 100644
@@ -107,6 +107,16 @@ static inline void release_deve(struct kref *kref)
 #define get_deve(x) kref_get(&x->refcount)
 #define put_deve(x) kref_put(&x->refcount, release_deve)
 
+static inline void release_lun(struct kref *kref)
+{
+	struct se_lun *lun = container_of(kref, struct se_lun, refcount);
+
+	core_tpg_free_lun(lun->lun_tpg, lun);
+}
+
+#define get_lun(x) kref_get(&x->refcount)
+#define put_lun(x) kref_put(&x->refcount, release_lun)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;

drivers/target/target_core_tpg.c +4 ~~-25~~

index 977b05c..6b4b0e6 100644
@@ -601,20 +601,12 @@ int core_tpg_set_initiator_node_tag(
 }
 EXPORT_SYMBOL(core_tpg_set_initiator_node_tag);
 
-static void core_tpg_lun_ref_release(struct percpu_ref *ref)
-{
-	struct se_lun *lun = container_of(ref, struct se_lun, lun_ref);
-
-	complete(&lun->lun_ref_comp);
-}
-
 static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 {
 	/* Set in core_dev_setup_virtual_lun0() */
 	struct se_device *dev = g_lun0_dev;
 	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
 	u32 lun_access = TRANSPORT_LUNFLAGS_READ_ONLY;
-	int ret;
 
 	lun->unpacked_lun = 0;
 	atomic_set(&lun->lun_acl_count, 0);

@@ -622,15 +614,8 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
-	init_completion(&lun->lun_ref_comp);
-
-	ret = core_tpg_add_lun(se_tpg, lun, lun_access, dev);
-	if (ret < 0) {
-		percpu_ref_cancel_init(&lun->lun_ref);
-		return ret;
-	}
 
-	return 0;
+	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 }
 
 static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)

@@ -753,7 +738,7 @@ struct se_lun *core_tpg_alloc_lun(
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
-	init_completion(&lun->lun_ref_comp);
+	kref_init(&lun->refcount);
 
 	spin_lock(&tpg->tpg_lun_lock);
 	if (!core_insert_lun(tpg, lun)) {

@@ -778,15 +763,9 @@ int core_tpg_add_lun(
 {
 	int ret;
 
-	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
-	if (ret < 0)
-		return ret;
-
 	ret = core_dev_export(dev, tpg, lun);
-	if (ret < 0) {
-		percpu_ref_cancel_init(&lun->lun_ref);
+	if (ret < 0)
 		return ret;
-	}
 
 	spin_lock(&tpg->tpg_lun_lock);
 	lun->lun_access = lun_access;

@@ -822,6 +801,6 @@ void core_tpg_remove_lun(
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
-	transport_clear_lun_ref(lun);
 	core_dev_unexport(lun->lun_se_dev, tpg, lun);
+	put_lun(lun);
 }

drivers/target/target_core_transport.c +1 ~~-28~~

index accaca5..a6ef32a 100644
@@ -571,7 +571,7 @@ static void transport_lun_remove_cmd(struct se_cmd *cmd)
 	if (!lun || !cmd->lun_ref_active)
 		return;
 
-	percpu_ref_put(&lun->lun_ref);
+	put_lun(lun);
 }
 
 void transport_cmd_finish_abort(struct se_cmd *cmd, int remove)

@@ -2364,33 +2364,6 @@ void target_wait_for_sess_cmds(struct se_session *se_sess)
 }
 EXPORT_SYMBOL(target_wait_for_sess_cmds);
 
-static int transport_clear_lun_ref_thread(void *p)
-{
-	struct se_lun *lun = p;
-
-	percpu_ref_kill(&lun->lun_ref);
-
-	wait_for_completion(&lun->lun_ref_comp);
-	complete(&lun->lun_shutdown_comp);
-
-	return 0;
-}
-
-int transport_clear_lun_ref(struct se_lun *lun)
-{
-	struct task_struct *kt;
-
-	kt = kthread_run(transport_clear_lun_ref_thread, lun,
-			"tcm_cl_%u", lun->unpacked_lun);
-	if (IS_ERR(kt)) {
-		pr_err("Unable to start clear_lun thread\n");
-		return PTR_ERR(kt);
-	}
-	wait_for_completion(&lun->lun_shutdown_comp);
-
-	return 0;
-}
-
 /**
  * transport_wait_for_tasks - wait for completion to occur
  * @cmd:	command to wait

include/target/target_core_base.h +1 -2

index 2c0a595..6d67c31 100644
@@ -641,8 +641,7 @@ struct se_lun {
 	struct se_port		*lun_sep;
 	struct config_group	lun_group;
 	struct se_port_stat_grps port_stat_grps;
-	struct completion	lun_ref_comp;
-	struct percpu_ref	lun_ref;
+	struct kref		refcount;
 };
 
 struct se_dev_stat_grps {

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:44 UTC #50 lore raw

Re: [PATCH 22/32] target: Convert percpu_ref to kref

Quoted 3 lines · Andy Grover

Wasn't getting release called when I expected, so punted and went down to krefs. Much simpler.

This is not an acceptable reason.

NAK.

--nab

Quoted 197 lines

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c    |    5 ++---
 drivers/target/target_core_internal.h  |   10 ++++++++++
 drivers/target/target_core_tpg.c       |   29 ++++-------------------------
 drivers/target/target_core_transport.c |   29 +----------------------------
 include/target/target_core_base.h      |    3 +--
 5 files changed, 18 insertions(+), 58 deletions(-)

drivers/target/target_core_device.c +2 -3

index e85a647..753e7ca 100644
@@ -138,7 +138,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		se_cmd->orig_fe_lun = unpacked_lun;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 
-		percpu_ref_get(&se_lun->lun_ref);
+		get_lun(se_lun);
 		se_cmd->lun_ref_active = true;
 	}
 	spin_unlock_irqrestore(&se_sess->se_node_acl->device_list_lock, flags);

@@ -168,7 +168,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		se_cmd->orig_fe_lun = 0;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 
-		percpu_ref_get(&se_lun->lun_ref);
+		get_lun(se_lun);
 		se_cmd->lun_ref_active = true;
 	}

@@ -1411,7 +1411,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
 	INIT_LIST_HEAD(&xcopy_lun->lun_acl_list);
 	spin_lock_init(&xcopy_lun->lun_acl_lock);
 	spin_lock_init(&xcopy_lun->lun_sep_lock);
-	init_completion(&xcopy_lun->lun_ref_comp);
 
 	return dev;
 }

drivers/target/target_core_internal.h +10 -0

index ed4d514..4d3b559 100644
@@ -107,6 +107,16 @@ static inline void release_deve(struct kref *kref)
 #define get_deve(x) kref_get(&x->refcount)
 #define put_deve(x) kref_put(&x->refcount, release_deve)
 
+static inline void release_lun(struct kref *kref)
+{
+	struct se_lun *lun = container_of(kref, struct se_lun, refcount);
+
+	core_tpg_free_lun(lun->lun_tpg, lun);
+}
+
+#define get_lun(x) kref_get(&x->refcount)
+#define put_lun(x) kref_put(&x->refcount, release_lun)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;

drivers/target/target_core_tpg.c +4 ~~-25~~

index 977b05c..6b4b0e6 100644
@@ -601,20 +601,12 @@ int core_tpg_set_initiator_node_tag(
 }
 EXPORT_SYMBOL(core_tpg_set_initiator_node_tag);
 
-static void core_tpg_lun_ref_release(struct percpu_ref *ref)
-{
-	struct se_lun *lun = container_of(ref, struct se_lun, lun_ref);
-
-	complete(&lun->lun_ref_comp);
-}
-
 static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 {
 	/* Set in core_dev_setup_virtual_lun0() */
 	struct se_device *dev = g_lun0_dev;
 	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
 	u32 lun_access = TRANSPORT_LUNFLAGS_READ_ONLY;
-	int ret;
 
 	lun->unpacked_lun = 0;
 	atomic_set(&lun->lun_acl_count, 0);

@@ -622,15 +614,8 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
-	init_completion(&lun->lun_ref_comp);
-
-	ret = core_tpg_add_lun(se_tpg, lun, lun_access, dev);
-	if (ret < 0) {
-		percpu_ref_cancel_init(&lun->lun_ref);
-		return ret;
-	}
 
-	return 0;
+	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 }
 
 static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)

@@ -753,7 +738,7 @@ struct se_lun *core_tpg_alloc_lun(
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
-	init_completion(&lun->lun_ref_comp);
+	kref_init(&lun->refcount);
 
 	spin_lock(&tpg->tpg_lun_lock);
 	if (!core_insert_lun(tpg, lun)) {

@@ -778,15 +763,9 @@ int core_tpg_add_lun(
 {
 	int ret;
 
-	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
-	if (ret < 0)
-		return ret;
-
 	ret = core_dev_export(dev, tpg, lun);
-	if (ret < 0) {
-		percpu_ref_cancel_init(&lun->lun_ref);
+	if (ret < 0)
 		return ret;
-	}
 
 	spin_lock(&tpg->tpg_lun_lock);
 	lun->lun_access = lun_access;

@@ -822,6 +801,6 @@ void core_tpg_remove_lun(
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
-	transport_clear_lun_ref(lun);
 	core_dev_unexport(lun->lun_se_dev, tpg, lun);
+	put_lun(lun);
 }

drivers/target/target_core_transport.c +1 ~~-28~~

index accaca5..a6ef32a 100644
@@ -571,7 +571,7 @@ static void transport_lun_remove_cmd(struct se_cmd *cmd)
 	if (!lun || !cmd->lun_ref_active)
 		return;
 
-	percpu_ref_put(&lun->lun_ref);
+	put_lun(lun);
 }
 
 void transport_cmd_finish_abort(struct se_cmd *cmd, int remove)

@@ -2364,33 +2364,6 @@ void target_wait_for_sess_cmds(struct se_session *se_sess)
 }
 EXPORT_SYMBOL(target_wait_for_sess_cmds);
 
-static int transport_clear_lun_ref_thread(void *p)
-{
-	struct se_lun *lun = p;
-
-	percpu_ref_kill(&lun->lun_ref);
-
-	wait_for_completion(&lun->lun_ref_comp);
-	complete(&lun->lun_shutdown_comp);
-
-	return 0;
-}
-
-int transport_clear_lun_ref(struct se_lun *lun)
-{
-	struct task_struct *kt;
-
-	kt = kthread_run(transport_clear_lun_ref_thread, lun,
-			"tcm_cl_%u", lun->unpacked_lun);
-	if (IS_ERR(kt)) {
-		pr_err("Unable to start clear_lun thread\n");
-		return PTR_ERR(kt);
-	}
-	wait_for_completion(&lun->lun_shutdown_comp);
-
-	return 0;
-}
-
 /**
  * transport_wait_for_tasks - wait for completion to occur
  * @cmd:	command to wait

include/target/target_core_base.h +1 -2

index 2c0a595..6d67c31 100644
@@ -641,8 +641,7 @@ struct se_lun {
 	struct se_port		*lun_sep;
 	struct config_group	lun_group;
 	struct se_port_stat_grps port_stat_grps;
-	struct completion	lun_ref_comp;
-	struct percpu_ref	lun_ref;
+	struct kref		refcount;
 };
 
 struct se_dev_stat_grps {

Andy Grover

author

2013-12-13 23:59 UTC #51 lore raw

PATCH 23/32 target: Add lun->lun_tpg pointer

In reply to Andy Grover

Although the port also has a tpg pointer, if there's no port to link (lun->lun_sep is NULL) then we can't get to it. So we need one in se_lun.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_tpg.c  |    2 ++
 include/target/target_core_base.h |    1 +
 2 files changed, 3 insertions(+), 0 deletions(-)

drivers/target/target_core_tpg.c +2 -0

index 6b4b0e6..c63f6cd 100644
@@ -614,6 +614,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
+	lun->lun_tpg = se_tpg;
 
 	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 }

@@ -739,6 +740,7 @@ struct se_lun *core_tpg_alloc_lun(
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
 	kref_init(&lun->refcount);
+	lun->lun_tpg = tpg;
 
 	spin_lock(&tpg->tpg_lun_lock);
 	if (!core_insert_lun(tpg, lun)) {

include/target/target_core_base.h +1 -0

index 6d67c31..e4d5119 100644
@@ -639,6 +639,7 @@ struct se_lun {
 	struct list_head	lun_acl_list;
 	struct se_device	*lun_se_dev;
 	struct se_port		*lun_sep;
+	struct se_portal_group	*lun_tpg;
 	struct config_group	lun_group;
 	struct se_port_stat_grps port_stat_grps;
 	struct kref		refcount;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 21:44 UTC #52 lore raw

Re: [PATCH 23/32] target: Add lun->lun_tpg pointer

Quoted 9 lines · Andy Grover

Although the port also has a tpg pointer, if there's no port to link (lun->lun_sep is NULL) then we can't get to it. So we need one in se_lun.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_tpg.c  |    2 ++
 include/target/target_core_base.h |    1 +
 2 files changed, 3 insertions(+), 0 deletions(-)

Given the other NAKs, I don't believe this patch is necessary.

--nab

Quoted 32 lines

drivers/target/target_core_tpg.c +2 -0

index 6b4b0e6..c63f6cd 100644
@@ -614,6 +614,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
+	lun->lun_tpg = se_tpg;
 
 	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 }

@@ -739,6 +740,7 @@ struct se_lun *core_tpg_alloc_lun(
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
 	kref_init(&lun->refcount);
+	lun->lun_tpg = tpg;
 
 	spin_lock(&tpg->tpg_lun_lock);
 	if (!core_insert_lun(tpg, lun)) {

include/target/target_core_base.h +1 -0

index 6d67c31..e4d5119 100644
@@ -639,6 +639,7 @@ struct se_lun {
 	struct list_head	lun_acl_list;
 	struct se_device	*lun_se_dev;
 	struct se_port		*lun_sep;
+	struct se_portal_group	*lun_tpg;
 	struct config_group	lun_group;
 	struct se_port_stat_grps port_stat_grps;
 	struct kref		refcount;

Andy Grover

author

2013-12-13 23:59 UTC #53 lore raw

PATCH 24/32 target: Remove tpg from core_dev_export/unexport params

In reply to Andy Grover

lun->lun_tpg should always be set.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c   |   17 +++++++----------
 drivers/target/target_core_internal.h |    6 ++----
 drivers/target/target_core_tpg.c      |    4 ++--
 3 files changed, 11 insertions(+), 16 deletions(-)

drivers/target/target_core_device.c +7 ~~-10~~

index 753e7ca..a432d7b 100644
@@ -583,7 +583,6 @@ static void core_release_port(struct se_device *dev, struct se_port *port)
 
 int core_dev_export(
 	struct se_device *dev,
-	struct se_portal_group *tpg,
 	struct se_lun *lun)
 {
 	struct se_hba *hba = dev->se_hba;

@@ -599,17 +598,13 @@ int core_dev_export(
 	dev->export_count++;
 	spin_unlock(&hba->device_lock);
 
-	core_export_port(dev, tpg, port, lun);
+	core_export_port(dev, lun->lun_tpg, port, lun);
 	return 0;
 }
 
-void core_dev_unexport(
-	struct se_device *dev,
-	struct se_portal_group *tpg,
-	struct se_lun *lun)
+void core_dev_unexport(struct se_lun *lun)
 {
-	struct se_hba *hba = dev->se_hba;
-	struct se_port *port = lun->lun_sep;
+	struct se_hba *hba;
 
 	spin_lock(&lun->lun_sep_lock);
 	if (lun->lun_se_dev == NULL) {

@@ -618,10 +613,12 @@ void core_dev_unexport(
 	}
 	spin_unlock(&lun->lun_sep_lock);
 
-	core_release_port(dev, port);
+	core_release_port(lun->lun_se_dev, lun->lun_sep);
+
+	hba = lun->lun_se_dev->se_hba;
 
 	spin_lock(&hba->device_lock);
-	dev->export_count--;
+	lun->lun_se_dev->export_count--;
 	spin_unlock(&hba->device_lock);
 
 	lun->lun_se_dev = NULL;

drivers/target/target_core_internal.h +2 -4

index 4d3b559..5b232a7 100644
@@ -16,10 +16,8 @@ int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
 void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
 		u32, struct se_node_acl *, struct se_portal_group *);
 void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
-int	core_dev_export(struct se_device *, struct se_portal_group *,
-		struct se_lun *);
-void	core_dev_unexport(struct se_device *, struct se_portal_group *,
-		struct se_lun *);
+int	core_dev_export(struct se_device *, struct se_lun *);
+void	core_dev_unexport(struct se_lun *);
 int	se_dev_set_task_timeout(struct se_device *, u32);
 int	se_dev_set_max_unmap_lba_count(struct se_device *, u32);
 int	se_dev_set_max_unmap_block_desc_count(struct se_device *, u32);

drivers/target/target_core_tpg.c +2 -2

index c63f6cd..d9fbdd0 100644
@@ -765,7 +765,7 @@ int core_tpg_add_lun(
 {
 	int ret;
 
-	ret = core_dev_export(dev, tpg, lun);
+	ret = core_dev_export(dev, lun);
 	if (ret < 0)
 		return ret;

@@ -803,6 +803,6 @@ void core_tpg_remove_lun(
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
-	core_dev_unexport(lun->lun_se_dev, tpg, lun);
+	core_dev_unexport(lun);
 	put_lun(lun);
 }

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 21:46 UTC #54 lore raw

Re: [PATCH 24/32] target: Remove tpg from core_dev_export/unexport params

Quoted 9 lines · Andy Grover

lun->lun_tpg should always be set.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c   |   17 +++++++----------
 drivers/target/target_core_internal.h |    6 ++----
 drivers/target/target_core_tpg.c      |    4 ++--
 3 files changed, 11 insertions(+), 16 deletions(-)

Ditto here, skipping..

--nab

Quoted 86 lines

drivers/target/target_core_device.c +7 ~~-10~~

index 753e7ca..a432d7b 100644
@@ -583,7 +583,6 @@ static void core_release_port(struct se_device *dev, struct se_port *port)
 
 int core_dev_export(
 	struct se_device *dev,
-	struct se_portal_group *tpg,
 	struct se_lun *lun)
 {
 	struct se_hba *hba = dev->se_hba;

@@ -599,17 +598,13 @@ int core_dev_export(
 	dev->export_count++;
 	spin_unlock(&hba->device_lock);
 
-	core_export_port(dev, tpg, port, lun);
+	core_export_port(dev, lun->lun_tpg, port, lun);
 	return 0;
 }
 
-void core_dev_unexport(
-	struct se_device *dev,
-	struct se_portal_group *tpg,
-	struct se_lun *lun)
+void core_dev_unexport(struct se_lun *lun)
 {
-	struct se_hba *hba = dev->se_hba;
-	struct se_port *port = lun->lun_sep;
+	struct se_hba *hba;
 
 	spin_lock(&lun->lun_sep_lock);
 	if (lun->lun_se_dev == NULL) {

@@ -618,10 +613,12 @@ void core_dev_unexport(
 	}
 	spin_unlock(&lun->lun_sep_lock);
 
-	core_release_port(dev, port);
+	core_release_port(lun->lun_se_dev, lun->lun_sep);
+
+	hba = lun->lun_se_dev->se_hba;
 
 	spin_lock(&hba->device_lock);
-	dev->export_count--;
+	lun->lun_se_dev->export_count--;
 	spin_unlock(&hba->device_lock);
 
 	lun->lun_se_dev = NULL;

drivers/target/target_core_internal.h +2 -4

index 4d3b559..5b232a7 100644
@@ -16,10 +16,8 @@ int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
 void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
 		u32, struct se_node_acl *, struct se_portal_group *);
 void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
-int	core_dev_export(struct se_device *, struct se_portal_group *,
-		struct se_lun *);
-void	core_dev_unexport(struct se_device *, struct se_portal_group *,
-		struct se_lun *);
+int	core_dev_export(struct se_device *, struct se_lun *);
+void	core_dev_unexport(struct se_lun *);
 int	se_dev_set_task_timeout(struct se_device *, u32);
 int	se_dev_set_max_unmap_lba_count(struct se_device *, u32);
 int	se_dev_set_max_unmap_block_desc_count(struct se_device *, u32);

drivers/target/target_core_tpg.c +2 -2

index c63f6cd..d9fbdd0 100644
@@ -765,7 +765,7 @@ int core_tpg_add_lun(
 {
 	int ret;
 
-	ret = core_dev_export(dev, tpg, lun);
+	ret = core_dev_export(dev, lun);
 	if (ret < 0)
 		return ret;

@@ -803,6 +803,6 @@ void core_tpg_remove_lun(
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
-	core_dev_unexport(lun->lun_se_dev, tpg, lun);
+	core_dev_unexport(lun);
 	put_lun(lun);
 }

Andy Grover

author

2013-12-13 23:59 UTC #55 lore raw

PATCH 25/32 target: Call remove_lun instead of del_lun in fabric_port_unlink

In reply to Andy Grover

We want to be freeing the port here, not freeing the lun.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_fabric_configfs.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

drivers/target/target_core_fabric_configfs.c +1 -1

index 4715836..fe940d4 100644
@@ -808,7 +808,7 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_dev_del_lun(se_tpg, lun);
+	core_tpg_remove_lun(se_tpg, lun);
 	return 0;
 }

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 21:46 UTC #56 lore raw

Re: [PATCH 25/32] target: Call remove_lun instead of del_lun in fabric_port_unlink

Quoted 7 lines · Andy Grover

We want to be freeing the port here, not freeing the lun.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_fabric_configfs.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

Skipping this one as well.

--nab

Quoted 13 lines

drivers/target/target_core_fabric_configfs.c +1 -1

index 4715836..fe940d4 100644
@@ -808,7 +808,7 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_dev_del_lun(se_tpg, lun);
+	core_tpg_remove_lun(se_tpg, lun);
 	return 0;
 }

Andy Grover

author

2013-12-13 23:59 UTC #57 lore raw

PATCH 26/32 target: Convert tpg_pr_ref_count to kref

In reply to Andy Grover

Don't call fabric_drop_tpg from configfs release(), just lower the refcount, and call fabric_drop_tpg when refcount goes to zero.

We don't need cpu_relax because core_tpg_deregister will only be called after we know there is no PR use of this tpg (step 4 below):

1) configfs drop_item (target_fabric_drop_tpg) calls config_item_put()
2) if really removed, configfs calls release (target_fabric_tpg_release)
3) tpg_release just calls put_tpg(), so if PR has a ref, the tpg still isn't freed until PR code drops the ref
4) Last put_tpg(), release_tpg() calls tf_ops.fabric_drop_tpg() which eventually calls core_tpg_deregister
5) struct contaning tpg freed by fabric after core_tpg_deregister returns

Add a target_core_tpg.h just to stick the get/put_tpg in there.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_fabric_configfs.c |    5 ++---
 drivers/target/target_core_pr.c              |   16 ++++++----------
 drivers/target/target_core_tpg.c             |    7 ++++---
 drivers/target/target_core_tpg.h             |   13 +++++++++++++
 include/target/target_core_base.h            |    3 +--
 5 files changed, 26 insertions(+), 18 deletions(-)
 create mode 100644 drivers/target/target_core_tpg.h

drivers/target/target_core_fabric_configfs.c +2 -3

index fe940d4..45a1763 100644
@@ -42,6 +42,7 @@
 #include "target_core_internal.h"
 #include "target_core_alua.h"
 #include "target_core_pr.h"
+#include "target_core_tpg.h"
 
 #define TF_CIT_SETUP(_name, _item_ops, _group_ops, _attrs)		\
 static void target_fabric_setup_##_name##_cit(struct target_fabric_configfs *tf) \

@@ -995,10 +996,8 @@ static void target_fabric_tpg_release(struct config_item *item)
 {
 	struct se_portal_group *se_tpg = container_of(to_config_group(item),
 			struct se_portal_group, tpg_group);
-	struct se_wwn *wwn = se_tpg->se_tpg_wwn;
-	struct target_fabric_configfs *tf = wwn->wwn_tf;
 
-	tf->tf_ops.fabric_drop_tpg(se_tpg);
+	put_tpg(se_tpg);
 }
 
 static struct configfs_item_operations target_fabric_tpg_base_item_ops = {

drivers/target/target_core_pr.c +6 ~~-10~~

index 0da6696..e2b656b 100644
@@ -40,6 +40,7 @@
 #include "target_core_internal.h"
 #include "target_core_pr.h"
 #include "target_core_ua.h"
+#include "target_core_tpg.h"
 
 /*
  * Used for Specify Initiator Ports Capable Bit (SPEC_I_PT)

@@ -1334,8 +1335,7 @@ static void core_scsi3_tpg_undepend_item(struct se_portal_group *tpg)
 	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
 			&tpg->tpg_group.cg_item);
 
-	atomic_dec(&tpg->tpg_pr_ref_count);
-	smp_mb__after_atomic_dec();
+	put_tpg(tpg);
 }
 
 static int core_scsi3_nodeacl_depend_item(struct se_node_acl *nacl)

@@ -1535,15 +1535,13 @@ core_scsi3_decode_spec_i_port(
 			if (!i_str)
 				continue;
 
-			atomic_inc(&tmp_tpg->tpg_pr_ref_count);
-			smp_mb__after_atomic_inc();
+			get_tpg(tmp_tpg);
 			spin_unlock(&dev->se_port_lock);
 
 			if (core_scsi3_tpg_depend_item(tmp_tpg)) {
 				pr_err(" core_scsi3_tpg_depend_item()"
 					" for tmp_tpg\n");
-				atomic_dec(&tmp_tpg->tpg_pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_tpg(tmp_tpg);
 				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 				goto out_unmap;
 			}

@@ -3153,15 +3151,13 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 		if (!dest_tf_ops)
 			continue;
 
-		atomic_inc(&dest_se_tpg->tpg_pr_ref_count);
-		smp_mb__after_atomic_inc();
+		get_tpg(dest_se_tpg);
 		spin_unlock(&dev->se_port_lock);
 
 		if (core_scsi3_tpg_depend_item(dest_se_tpg)) {
 			pr_err("core_scsi3_tpg_depend_item() failed"
 				" for dest_se_tpg\n");
-			atomic_dec(&dest_se_tpg->tpg_pr_ref_count);
-			smp_mb__after_atomic_dec();
+			put_tpg(dest_se_tpg);
 			ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 			goto out_put_pr_reg;
 		}

drivers/target/target_core_tpg.c +4 -3

index d9fbdd0..0e30ced 100644
@@ -38,8 +38,11 @@
 #include <target/target_core_base.h>
 #include <target/target_core_backend.h>
 #include <target/target_core_fabric.h>
+#include <target/target_core_configfs.h>
+
 
 #include "target_core_internal.h"
+#include "target_core_tpg.h"
 
 extern struct se_device *g_lun0_dev;

@@ -637,7 +640,7 @@ int core_tpg_register(
 	se_tpg->se_tpg_fabric_ptr = tpg_fabric_ptr;
 	se_tpg->se_tpg_tfo = tfo;
 	se_tpg->se_tpg_wwn = se_wwn;
-	atomic_set(&se_tpg->tpg_pr_ref_count, 0);
+	kref_init(&se_tpg->refcount);
 	INIT_LIST_HEAD(&se_tpg->acl_node_list);
 	INIT_LIST_HEAD(&se_tpg->se_tpg_node);
 	INIT_LIST_HEAD(&se_tpg->tpg_sess_list);

@@ -680,8 +683,6 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 	list_del(&se_tpg->se_tpg_node);
 	spin_unlock_bh(&tpg_lock);
 
-	while (atomic_read(&se_tpg->tpg_pr_ref_count) != 0)
-		cpu_relax();
 	/*
 	 * Release any remaining demo-mode generated se_node_acl that have
 	 * not been released because of TFO->tpg_check_demo_mode_cache() == 1

drivers/target/target_core_tpg.h new file +13 -0

new file mode 100644
index 0000000..964d69b
@@ -0,0 +1,13 @@
+
+static inline void release_tpg(struct kref *ref)
+{
+	struct se_portal_group *tpg = container_of(ref,
+		struct se_portal_group, refcount);
+	struct se_wwn *wwn = tpg->se_tpg_wwn;
+	struct target_fabric_configfs *tf = wwn->wwn_tf;
+
+	tf->tf_ops.fabric_drop_tpg(tpg);
+}
+
+#define get_tpg(x) kref_get(&x->refcount)
+#define put_tpg(x) kref_put(&x->refcount, release_tpg)

include/target/target_core_base.h +1 -2

index e4d5119..dc8bf39 100644
@@ -781,8 +781,7 @@ struct se_portal_group {
 	enum transport_tpg_type_table se_tpg_type;
 	/* Number of ACLed Initiator Nodes for this TPG */
 	u32			num_node_acls;
-	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
-	atomic_t		tpg_pr_ref_count;
+	struct kref		refcount;
 	/* Spinlock for adding/removing ACLed Nodes */
 	spinlock_t		acl_node_lock;
 	/* Spinlock for adding/removing sessions */

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:49 UTC #58 lore raw

Re: [PATCH 26/32] target: Convert tpg_pr_ref_count to kref

Quoted 134 lines · Andy Grover

Don't call fabric_drop_tpg from configfs release(), just lower the refcount, and call fabric_drop_tpg when refcount goes to zero.

We don't need cpu_relax because core_tpg_deregister will only be called after we know there is no PR use of this tpg (step 4 below):

Add a target_core_tpg.h just to stick the get/put_tpg in there.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_fabric_configfs.c |    5 ++---
 drivers/target/target_core_pr.c              |   16 ++++++----------
 drivers/target/target_core_tpg.c             |    7 ++++---
 drivers/target/target_core_tpg.h             |   13 +++++++++++++
 include/target/target_core_base.h            |    3 +--
 5 files changed, 26 insertions(+), 18 deletions(-)
 create mode 100644 drivers/target/target_core_tpg.h

drivers/target/target_core_fabric_configfs.c +2 -3

index fe940d4..45a1763 100644
@@ -42,6 +42,7 @@
 #include "target_core_internal.h"
 #include "target_core_alua.h"
 #include "target_core_pr.h"
+#include "target_core_tpg.h"
 
 #define TF_CIT_SETUP(_name, _item_ops, _group_ops, _attrs)		\
 static void target_fabric_setup_##_name##_cit(struct target_fabric_configfs *tf) \

@@ -995,10 +996,8 @@ static void target_fabric_tpg_release(struct config_item *item)
 {
 	struct se_portal_group *se_tpg = container_of(to_config_group(item),
 			struct se_portal_group, tpg_group);
-	struct se_wwn *wwn = se_tpg->se_tpg_wwn;
-	struct target_fabric_configfs *tf = wwn->wwn_tf;
 
-	tf->tf_ops.fabric_drop_tpg(se_tpg);
+	put_tpg(se_tpg);
 }
 
 static struct configfs_item_operations target_fabric_tpg_base_item_ops = {

drivers/target/target_core_pr.c +6 ~~-10~~

index 0da6696..e2b656b 100644
@@ -40,6 +40,7 @@
 #include "target_core_internal.h"
 #include "target_core_pr.h"
 #include "target_core_ua.h"
+#include "target_core_tpg.h"
 
 /*
  * Used for Specify Initiator Ports Capable Bit (SPEC_I_PT)

@@ -1334,8 +1335,7 @@ static void core_scsi3_tpg_undepend_item(struct se_portal_group *tpg)
 	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
 			&tpg->tpg_group.cg_item);
 
-	atomic_dec(&tpg->tpg_pr_ref_count);
-	smp_mb__after_atomic_dec();
+	put_tpg(tpg);
 }
 
 static int core_scsi3_nodeacl_depend_item(struct se_node_acl *nacl)

@@ -1535,15 +1535,13 @@ core_scsi3_decode_spec_i_port(
 			if (!i_str)
 				continue;
 
-			atomic_inc(&tmp_tpg->tpg_pr_ref_count);
-			smp_mb__after_atomic_inc();
+			get_tpg(tmp_tpg);
 			spin_unlock(&dev->se_port_lock);
 
 			if (core_scsi3_tpg_depend_item(tmp_tpg)) {
 				pr_err(" core_scsi3_tpg_depend_item()"
 					" for tmp_tpg\n");
-				atomic_dec(&tmp_tpg->tpg_pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_tpg(tmp_tpg);
 				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 				goto out_unmap;
 			}

@@ -3153,15 +3151,13 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 		if (!dest_tf_ops)
 			continue;
 
-		atomic_inc(&dest_se_tpg->tpg_pr_ref_count);
-		smp_mb__after_atomic_inc();
+		get_tpg(dest_se_tpg);
 		spin_unlock(&dev->se_port_lock);
 
 		if (core_scsi3_tpg_depend_item(dest_se_tpg)) {
 			pr_err("core_scsi3_tpg_depend_item() failed"
 				" for dest_se_tpg\n");
-			atomic_dec(&dest_se_tpg->tpg_pr_ref_count);
-			smp_mb__after_atomic_dec();
+			put_tpg(dest_se_tpg);
 			ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 			goto out_put_pr_reg;
 		}

drivers/target/target_core_tpg.c +4 -1

index d9fbdd0..0e30ced 100644
@@ -38,8 +38,11 @@
 #include <target/target_core_base.h>
 #include <target/target_core_backend.h>
 #include <target/target_core_fabric.h>
+#include <target/target_core_configfs.h>
+
 
 #include "target_core_internal.h"
+#include "target_core_tpg.h"
 
 extern struct se_device *g_lun0_dev;

@@ -637,7 +640,7 @@ int core_tpg_register(
 	se_tpg->se_tpg_fabric_ptr = tpg_fabric_ptr;
 	se_tpg->se_tpg_tfo = tfo;
 	se_tpg->se_tpg_wwn = se_wwn;
-	atomic_set(&se_tpg->tpg_pr_ref_count, 0);
+	kref_init(&se_tpg->refcount);
 	INIT_LIST_HEAD(&se_tpg->acl_node_list);
 	INIT_LIST_HEAD(&se_tpg->se_tpg_node);
 	INIT_LIST_HEAD(&se_tpg->tpg_sess_list);

drivers/target/target_core_tpg.c int core_tpg_deregister(struct se_portal_group *se_tpg) · line 688

Quoted 8 lines

drivers/target/target_core_tpg.c +0 -2

@@ -680,8 +683,6 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 	list_del(&se_tpg->se_tpg_node);
 	spin_unlock_bh(&tpg_lock);
 
-	while (atomic_read(&se_tpg->tpg_pr_ref_count) != 0)
-		cpu_relax();
 	/*
 	 * Release any remaining demo-mode generated se_node_acl that have
 	 * not been released because of TFO->tpg_check_demo_mode_cache() == 1

Same problem here as with the other second reference count conversions.

Releasing all of the demo-mode generated node_acls + releasing virtual LUN0 while there are still references from other process contexts is not correct.

Also, the same configfs reference counting issue exists here. What prevents the underlying struct se_wwn->wwn_group from being removed before the last put_tpg() is called from the other process contexts..?

NAK.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #59 lore raw

PATCH 27/32 target: Move call to remove_lun to the release function from drop_link

In reply to Andy Grover

Configfs is still using the memory until release() is called, so it's not safe to free it in drop_link().

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_fabric_configfs.c |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)

drivers/target/target_core_fabric_configfs.c +9 -1

index 45a1763..1e584fa 100644
@@ -809,11 +809,19 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_tpg_remove_lun(se_tpg, lun);
 	return 0;
 }
 
+static void target_fabric_release_lun(struct config_item *item)
+{
+	struct se_lun *lun = container_of(to_config_group(item),
+				struct se_lun, lun_group);
+
+	core_tpg_remove_lun(lun->lun_tpg, lun);
+}
+
 static struct configfs_item_operations target_fabric_port_item_ops = {
+	.release		= target_fabric_release_lun,
 	.show_attribute		= target_fabric_port_attr_show,
 	.store_attribute	= target_fabric_port_attr_store,
 	.allow_link		= target_fabric_port_link,

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 21:51 UTC #60 lore raw

Re: [PATCH 27/32] target: Move call to remove_lun to the release function from drop_link

Quoted 8 lines · Andy Grover

Configfs is still using the memory until release() is called, so it's not safe to free it in drop_link().
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_fabric_configfs.c |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)

No longer is applicable, given the previous NAKs.

--nab

Quoted 25 lines

drivers/target/target_core_fabric_configfs.c +9 -1

index 45a1763..1e584fa 100644
@@ -809,11 +809,19 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_tpg_remove_lun(se_tpg, lun);
 	return 0;
 }
 
+static void target_fabric_release_lun(struct config_item *item)
+{
+	struct se_lun *lun = container_of(to_config_group(item),
+				struct se_lun, lun_group);
+
+	core_tpg_remove_lun(lun->lun_tpg, lun);
+}
+
 static struct configfs_item_operations target_fabric_port_item_ops = {
+	.release		= target_fabric_release_lun,
 	.show_attribute		= target_fabric_port_attr_show,
 	.store_attribute	= target_fabric_port_attr_store,
 	.allow_link		= target_fabric_port_link,

Andy Grover

author

2013-12-13 23:59 UTC #61 lore raw

PATCH 28/32 target: Convert acl_pr_ref_count to kref

In reply to Andy Grover

In fabrics' drop_nodeacl function, do not kfree the nacl. We are now calling fabrics' tpg_release_fabric_acl later when its refcount goes to zero, which will kfree it.

Signed-off-by Andy Grover <[email protected]>

---
 Documentation/target/tcm_mod_builder.py      |    1 -
 drivers/infiniband/ulp/srpt/ib_srpt.c        |    1 -
 drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    7 +----
 drivers/target/iscsi/iscsi_target_configfs.c |    2 -
 drivers/target/sbp/sbp_target.c              |    4 ---
 drivers/target/target_core_internal.h        |   22 ++++++++++++++++-
 drivers/target/target_core_pr.c              |   24 ++++++------------
 drivers/target/target_core_tpg.c             |   34 +++++--------------------
 drivers/target/target_core_transport.c       |   12 ++-------
 drivers/target/tcm_fc/tfc_conf.c             |    1 -
 drivers/usb/gadget/tcm_usb_gadget.c          |    3 --
 drivers/vhost/scsi.c                         |    3 --
 include/target/target_core_base.h            |    3 +-
 13 files changed, 41 insertions(+), 76 deletions(-)

Documentation/target/tcm_mod_builder.py +0 -1

index 230ce71..c8e0572 100755
@@ -285,7 +285,6 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
 	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
 	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
 	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
-	buf += "	kfree(nacl);\n"
 	buf += "}\n\n"
 
 	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"

drivers/infiniband/ulp/srpt/ib_srpt.c +0 -1

index 520a7e5..4995b91 100644
@@ -3645,7 +3645,6 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
 	list_del(&nacl->list);
 	spin_unlock_irq(&sport->port_acl_lock);
 	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
-	srpt_release_fabric_acl(NULL, se_nacl);
 }
 
 static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(

drivers/scsi/qla2xxx/tcm_qla2xxx.c +1 -6

index 7eb19be..4fe684a 100644
@@ -828,12 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
 
 static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct se_portal_group *se_tpg = se_acl->se_tpg;
-	struct tcm_qla2xxx_nacl *nacl = container_of(se_acl,
-				struct tcm_qla2xxx_nacl, se_node_acl);
-
-	core_tpg_del_initiator_node_acl(se_tpg, se_acl, 1);
-	kfree(nacl);
+	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
 }
 
 /* Start items for tcm_qla2xxx_tpg_attrib_cit */

drivers/target/iscsi/iscsi_target_configfs.c +0 -2

index e3318ed..bd05ab5 100644
@@ -916,7 +916,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
 		pr_err("Unable to allocate memory for"
 				" stats_cg->default_groups\n");
 		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
-		kfree(acl);
 		return ERR_PTR(-ENOMEM);
 	}

@@ -947,7 +946,6 @@ static void lio_target_drop_nodeacl(
 	kfree(stats_cg->default_groups);
 
 	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
-	kfree(acl);
 }
 
 /* End items for lio_target_acl_cit */

drivers/target/sbp/sbp_target.c +0 -4

index 103998c..6fabd9c 100644
@@ -2126,11 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
 
 static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct sbp_nacl *nacl =
-		container_of(se_acl, struct sbp_nacl, se_node_acl);
-
 	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
-	kfree(nacl);
 }
 
 static int sbp_post_link_lun(

drivers/target/target_core_internal.h +21 -1

index 5b232a7..65a4de9 100644
@@ -82,10 +82,11 @@ int	core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
 /* target_core_tpg.c */
 extern struct se_device *g_lun0_dev;
 
+void core_clear_initiator_node_from_tpg(struct se_node_acl *,
+					struct se_portal_group *);
 struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tpg,
 		const char *);
 void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
-void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);

@@ -115,6 +116,25 @@ static inline void release_lun(struct kref *kref)
 #define get_lun(x) kref_get(&x->refcount)
 #define put_lun(x) kref_put(&x->refcount, release_lun)
 
+static inline void target_release_nacl(struct kref *kref)
+{
+	struct se_node_acl *nacl = container_of(kref,
+						struct se_node_acl, refcount);
+	struct se_portal_group *tpg = nacl->se_tpg;
+
+	pr_debug("%s_TPG[%hu] - Deleted ACL with TCQ Depth: %d for %s"
+		" Initiator Node: %s\n", tpg->se_tpg_tfo->get_fabric_name(),
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), nacl->queue_depth,
+		tpg->se_tpg_tfo->get_fabric_name(), nacl->initiatorname);
+
+	core_clear_initiator_node_from_tpg(nacl, tpg);
+	core_free_device_list_for_node(nacl, tpg);
+	tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, nacl);
+}
+
+#define get_nacl(x) kref_get(&x->refcount)
+#define put_nacl(x) kref_put(&x->refcount, target_release_nacl)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;

drivers/target/target_core_pr.c +8 ~~-16~~

index e2b656b..835958b 100644
@@ -1354,16 +1354,14 @@ static void core_scsi3_nodeacl_undepend_item(struct se_node_acl *nacl)
 	struct se_portal_group *tpg = nacl->se_tpg;
 
 	if (nacl->dynamic_node_acl) {
-		atomic_dec(&nacl->acl_pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_nacl(nacl);
 		return;
 	}
 
 	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
 			&nacl->acl_group.cg_item);
 
-	atomic_dec(&nacl->acl_pr_ref_count);
-	smp_mb__after_atomic_dec();
+	put_nacl(nacl);
 }
 
 static int core_scsi3_lunacl_depend_item(struct se_dev_entry *se_deve)

@@ -1553,10 +1551,8 @@ core_scsi3_decode_spec_i_port(
 			spin_lock_irq(&tmp_tpg->acl_node_lock);
 			dest_node_acl = __core_tpg_get_initiator_node_acl(
 						tmp_tpg, i_str);
-			if (dest_node_acl) {
-				atomic_inc(&dest_node_acl->acl_pr_ref_count);
-				smp_mb__after_atomic_inc();
-			}
+			if (dest_node_acl)
+				get_nacl(dest_node_acl);
 			spin_unlock_irq(&tmp_tpg->acl_node_lock);
 
 			if (!dest_node_acl) {

@@ -1568,8 +1564,7 @@ core_scsi3_decode_spec_i_port(
 			if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
 				pr_err("configfs_depend_item() failed"
 					" for dest_node_acl->acl_group\n");
-				atomic_dec(&dest_node_acl->acl_pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_nacl(dest_node_acl);
 				core_scsi3_tpg_undepend_item(tmp_tpg);
 				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 				goto out_unmap;

@@ -3253,10 +3248,8 @@ after_iport_check:
 	spin_lock_irq(&dest_se_tpg->acl_node_lock);
 	dest_node_acl = __core_tpg_get_initiator_node_acl(dest_se_tpg,
 				initiator_str);
-	if (dest_node_acl) {
-		atomic_inc(&dest_node_acl->acl_pr_ref_count);
-		smp_mb__after_atomic_inc();
-	}
+	if (dest_node_acl)
+		get_nacl(dest_node_acl);
 	spin_unlock_irq(&dest_se_tpg->acl_node_lock);
 
 	if (!dest_node_acl) {

@@ -3270,8 +3263,7 @@ after_iport_check:
 	if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
 		pr_err("core_scsi3_nodeacl_depend_item() for"
 			" dest_node_acl\n");
-		atomic_dec(&dest_node_acl->acl_pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_nacl(dest_node_acl);
 		dest_node_acl = NULL;
 		ret = TCM_INVALID_PARAMETER_LIST;
 		goto out;

drivers/target/target_core_tpg.c +7 ~~-27~~

index 0e30ced..6aaf50f 100644
@@ -53,7 +53,7 @@ static LIST_HEAD(tpg_list);
  *
  *
  */
-static void core_clear_initiator_node_from_tpg(
+void core_clear_initiator_node_from_tpg(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {

@@ -251,7 +251,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
-	atomic_set(&acl->acl_pr_ref_count, 0);
+	kref_init(&acl->refcount);
 	acl->queue_depth = tpg->se_tpg_tfo->tpg_get_default_depth(tpg);
 	snprintf(acl->initiatorname, TRANSPORT_IQN_LEN, "%s", initiatorname);
 	acl->se_tpg = tpg;

@@ -264,8 +264,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
-		core_free_device_list_for_node(acl, tpg);
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
+		put_nacl(acl);
 		return NULL;
 	}
 	/*

@@ -291,12 +290,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 }
 EXPORT_SYMBOL(core_tpg_check_initiator_node_acl);
 
-void core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *nacl)
-{
-	while (atomic_read(&nacl->acl_pr_ref_count) != 0)
-		cpu_relax();
-}
-
 void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 {
 	struct rb_node *node;

@@ -374,7 +367,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
-	atomic_set(&acl->acl_pr_ref_count, 0);
+	kref_init(&acl->refcount);
 	acl->queue_depth = queue_depth;
 	snprintf(acl->initiatorname, TRANSPORT_IQN_LEN, "%s", initiatorname);
 	acl->se_tpg = tpg;

@@ -386,8 +379,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
-		core_free_device_list_for_node(acl, tpg);
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
+		put_nacl(acl);
 		return ERR_PTR(-EINVAL);
 	}

@@ -457,14 +449,7 @@ int core_tpg_del_initiator_node_acl(
 	 */
 	wait_for_completion(&acl->acl_free_comp);
 
-	core_tpg_wait_for_nacl_pr_ref(acl);
-	core_clear_initiator_node_from_tpg(acl, tpg);
-	core_free_device_list_for_node(acl, tpg);
-
-	pr_debug("%s_TPG[%hu] - Deleted ACL with TCQ Depth: %d for %s"
-		" Initiator Node: %s\n", tpg->se_tpg_tfo->get_fabric_name(),
-		tpg->se_tpg_tfo->tpg_get_tag(tpg), acl->queue_depth,
-		tpg->se_tpg_tfo->get_fabric_name(), acl->initiatorname);
+	put_nacl(acl);
 
 	return 0;
 }

@@ -693,13 +678,8 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 			acl_node) {
 		list_del(&nacl->acl_node);
 		se_tpg->num_node_acls--;
-		spin_unlock_irq(&se_tpg->acl_node_lock);
-
-		core_tpg_wait_for_nacl_pr_ref(nacl);
-		core_free_device_list_for_node(nacl, se_tpg);
-		se_tpg->se_tpg_tfo->tpg_release_fabric_acl(se_tpg, nacl);
 
-		spin_lock_irq(&se_tpg->acl_node_lock);
+		put_nacl(nacl);
 	}
 	spin_unlock_irq(&se_tpg->acl_node_lock);

drivers/target/target_core_transport.c +3 -9

index a6ef32a..51e294b 100644
@@ -433,7 +433,6 @@ void transport_deregister_session(struct se_session *se_sess)
 	struct target_core_fabric_ops *se_tfo;
 	struct se_node_acl *se_nacl;
 	unsigned long flags;
-	bool comp_nacl = true;
 
 	if (!se_tpg) {
 		transport_free_session(se_sess);

@@ -458,13 +457,8 @@ void transport_deregister_session(struct se_session *se_sess)
 		if (!se_tfo->tpg_check_demo_mode_cache(se_tpg)) {
 			list_del(&se_nacl->acl_node);
 			se_tpg->num_node_acls--;
-			spin_unlock_irqrestore(&se_tpg->acl_node_lock, flags);
-			core_tpg_wait_for_nacl_pr_ref(se_nacl);
-			core_free_device_list_for_node(se_nacl, se_tpg);
-			se_tfo->tpg_release_fabric_acl(se_tpg, se_nacl);
-
-			comp_nacl = false;
-			spin_lock_irqsave(&se_tpg->acl_node_lock, flags);
+			put_nacl(se_nacl);
+			se_nacl = NULL;
 		}
 	}
 	spin_unlock_irqrestore(&se_tpg->acl_node_lock, flags);

@@ -476,7 +470,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	 * ->acl_free_comp caller to wakeup configfs se_node_acl->acl_group
 	 * removal context.
 	 */
-	if (se_nacl && comp_nacl == true)
+	if (se_nacl)
 		target_put_nacl(se_nacl);
 
 	transport_free_session(se_sess);

drivers/target/tcm_fc/tfc_conf.c +0 -1

index 47d0038..2a44ad6 100644
@@ -239,7 +239,6 @@ static void ft_del_acl(struct se_node_acl *se_acl)
 		    acl, se_acl, tpg, &tpg->se_tpg);
 
 	core_tpg_del_initiator_node_acl(&tpg->se_tpg, se_acl, 1);
-	kfree(acl);
 }
 
 struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)

drivers/usb/gadget/tcm_usb_gadget.c +0 -3

index 6c3d795..2cedb4b 100644
@@ -1526,10 +1526,7 @@ static struct se_node_acl *usbg_make_nodeacl(
 
 static void usbg_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct usbg_nacl *nacl = container_of(se_acl,
-				struct usbg_nacl, se_node_acl);
 	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
-	kfree(nacl);
 }
 
 struct usbg_tpg *the_only_tpg_I_currently_have;

drivers/vhost/scsi.c +0 -3

index f175629..a93169b 100644
@@ -1687,10 +1687,7 @@ tcm_vhost_make_nodeacl(struct se_portal_group *se_tpg,
 
 static void tcm_vhost_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct tcm_vhost_nacl *nacl = container_of(se_acl,
-				struct tcm_vhost_nacl, se_node_acl);
 	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
-	kfree(nacl);
 }
 
 static void tcm_vhost_free_cmd_map_res(struct tcm_vhost_nexus *nexus,

include/target/target_core_base.h +1 -2

index dc8bf39..393bcfd 100644
@@ -513,8 +513,7 @@ struct se_node_acl {
 	u64			read_bytes;
 	u64			write_bytes;
 	spinlock_t		stats_lock;
-	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
-	atomic_t		acl_pr_ref_count;
+	struct kref		refcount;
 	struct rb_root		rb_device_list;
 	struct se_session	*nacl_sess;
 	struct se_portal_group *se_tpg;

Signature · 1 line

1.7.1

Nicholas A. Bellinger

NAK

2013-12-16 21:57 UTC #62 lore raw

Re: [PATCH 28/32] target: Convert acl_pr_ref_count to kref

Quoted 244 lines · Andy Grover

In fabrics' drop_nodeacl function, do not kfree the nacl. We are now calling fabrics' tpg_release_fabric_acl later when its refcount goes to zero, which will kfree it.

Signed-off-by Andy Grover <[email protected]>

---
 Documentation/target/tcm_mod_builder.py      |    1 -
 drivers/infiniband/ulp/srpt/ib_srpt.c        |    1 -
 drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    7 +----
 drivers/target/iscsi/iscsi_target_configfs.c |    2 -
 drivers/target/sbp/sbp_target.c              |    4 ---
 drivers/target/target_core_internal.h        |   22 ++++++++++++++++-
 drivers/target/target_core_pr.c              |   24 ++++++------------
 drivers/target/target_core_tpg.c             |   34 +++++--------------------
 drivers/target/target_core_transport.c       |   12 ++-------
 drivers/target/tcm_fc/tfc_conf.c             |    1 -
 drivers/usb/gadget/tcm_usb_gadget.c          |    3 --
 drivers/vhost/scsi.c                         |    3 --
 include/target/target_core_base.h            |    3 +-
 13 files changed, 41 insertions(+), 76 deletions(-)

Documentation/target/tcm_mod_builder.py +0 -1

index 230ce71..c8e0572 100755
@@ -285,7 +285,6 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
 	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
 	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
 	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
-	buf += "	kfree(nacl);\n"
 	buf += "}\n\n"
 
 	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"

drivers/infiniband/ulp/srpt/ib_srpt.c +0 -1

index 520a7e5..4995b91 100644
@@ -3645,7 +3645,6 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
 	list_del(&nacl->list);
 	spin_unlock_irq(&sport->port_acl_lock);
 	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
-	srpt_release_fabric_acl(NULL, se_nacl);
 }
 
 static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(

drivers/scsi/qla2xxx/tcm_qla2xxx.c +1 -6

index 7eb19be..4fe684a 100644
@@ -828,12 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
 
 static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct se_portal_group *se_tpg = se_acl->se_tpg;
-	struct tcm_qla2xxx_nacl *nacl = container_of(se_acl,
-				struct tcm_qla2xxx_nacl, se_node_acl);
-
-	core_tpg_del_initiator_node_acl(se_tpg, se_acl, 1);
-	kfree(nacl);
+	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
 }
 
 /* Start items for tcm_qla2xxx_tpg_attrib_cit */

drivers/target/iscsi/iscsi_target_configfs.c +0 -2

index e3318ed..bd05ab5 100644
@@ -916,7 +916,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
 		pr_err("Unable to allocate memory for"
 				" stats_cg->default_groups\n");
 		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
-		kfree(acl);
 		return ERR_PTR(-ENOMEM);
 	}

@@ -947,7 +946,6 @@ static void lio_target_drop_nodeacl(
 	kfree(stats_cg->default_groups);
 
 	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
-	kfree(acl);
 }
 
 /* End items for lio_target_acl_cit */

drivers/target/sbp/sbp_target.c +0 -4

index 103998c..6fabd9c 100644
@@ -2126,11 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
 
 static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct sbp_nacl *nacl =
-		container_of(se_acl, struct sbp_nacl, se_node_acl);
-
 	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
-	kfree(nacl);
 }
 
 static int sbp_post_link_lun(

drivers/target/target_core_internal.h +21 -1

index 5b232a7..65a4de9 100644
@@ -82,10 +82,11 @@ int	core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
 /* target_core_tpg.c */
 extern struct se_device *g_lun0_dev;
 
+void core_clear_initiator_node_from_tpg(struct se_node_acl *,
+					struct se_portal_group *);
 struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tpg,
 		const char *);
 void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
-void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);

@@ -115,6 +116,25 @@ static inline void release_lun(struct kref *kref)
 #define get_lun(x) kref_get(&x->refcount)
 #define put_lun(x) kref_put(&x->refcount, release_lun)
 
+static inline void target_release_nacl(struct kref *kref)
+{
+	struct se_node_acl *nacl = container_of(kref,
+						struct se_node_acl, refcount);
+	struct se_portal_group *tpg = nacl->se_tpg;
+
+	pr_debug("%s_TPG[%hu] - Deleted ACL with TCQ Depth: %d for %s"
+		" Initiator Node: %s\n", tpg->se_tpg_tfo->get_fabric_name(),
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), nacl->queue_depth,
+		tpg->se_tpg_tfo->get_fabric_name(), nacl->initiatorname);
+
+	core_clear_initiator_node_from_tpg(nacl, tpg);
+	core_free_device_list_for_node(nacl, tpg);
+	tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, nacl);
+}
+
+#define get_nacl(x) kref_get(&x->refcount)
+#define put_nacl(x) kref_put(&x->refcount, target_release_nacl)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;

drivers/target/target_core_pr.c +8 ~~-16~~

index e2b656b..835958b 100644
@@ -1354,16 +1354,14 @@ static void core_scsi3_nodeacl_undepend_item(struct se_node_acl *nacl)
 	struct se_portal_group *tpg = nacl->se_tpg;
 
 	if (nacl->dynamic_node_acl) {
-		atomic_dec(&nacl->acl_pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_nacl(nacl);
 		return;
 	}
 
 	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
 			&nacl->acl_group.cg_item);
 
-	atomic_dec(&nacl->acl_pr_ref_count);
-	smp_mb__after_atomic_dec();
+	put_nacl(nacl);
 }
 
 static int core_scsi3_lunacl_depend_item(struct se_dev_entry *se_deve)

@@ -1553,10 +1551,8 @@ core_scsi3_decode_spec_i_port(
 			spin_lock_irq(&tmp_tpg->acl_node_lock);
 			dest_node_acl = __core_tpg_get_initiator_node_acl(
 						tmp_tpg, i_str);
-			if (dest_node_acl) {
-				atomic_inc(&dest_node_acl->acl_pr_ref_count);
-				smp_mb__after_atomic_inc();
-			}
+			if (dest_node_acl)
+				get_nacl(dest_node_acl);
 			spin_unlock_irq(&tmp_tpg->acl_node_lock);
 
 			if (!dest_node_acl) {

@@ -1568,8 +1564,7 @@ core_scsi3_decode_spec_i_port(
 			if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
 				pr_err("configfs_depend_item() failed"
 					" for dest_node_acl->acl_group\n");
-				atomic_dec(&dest_node_acl->acl_pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_nacl(dest_node_acl);
 				core_scsi3_tpg_undepend_item(tmp_tpg);
 				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 				goto out_unmap;

@@ -3253,10 +3248,8 @@ after_iport_check:
 	spin_lock_irq(&dest_se_tpg->acl_node_lock);
 	dest_node_acl = __core_tpg_get_initiator_node_acl(dest_se_tpg,
 				initiator_str);
-	if (dest_node_acl) {
-		atomic_inc(&dest_node_acl->acl_pr_ref_count);
-		smp_mb__after_atomic_inc();
-	}
+	if (dest_node_acl)
+		get_nacl(dest_node_acl);
 	spin_unlock_irq(&dest_se_tpg->acl_node_lock);
 
 	if (!dest_node_acl) {

@@ -3270,8 +3263,7 @@ after_iport_check:
 	if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
 		pr_err("core_scsi3_nodeacl_depend_item() for"
 			" dest_node_acl\n");
-		atomic_dec(&dest_node_acl->acl_pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_nacl(dest_node_acl);
 		dest_node_acl = NULL;
 		ret = TCM_INVALID_PARAMETER_LIST;
 		goto out;

drivers/target/target_core_tpg.c +3 -4

index 0e30ced..6aaf50f 100644
@@ -53,7 +53,7 @@ static LIST_HEAD(tpg_list);
  *
  *
  */
-static void core_clear_initiator_node_from_tpg(
+void core_clear_initiator_node_from_tpg(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {

@@ -251,7 +251,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
-	atomic_set(&acl->acl_pr_ref_count, 0);
+	kref_init(&acl->refcount);
 	acl->queue_depth = tpg->se_tpg_tfo->tpg_get_default_depth(tpg);
 	snprintf(acl->initiatorname, TRANSPORT_IQN_LEN, "%s", initiatorname);
 	acl->se_tpg = tpg;

@@ -264,8 +264,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
-		core_free_device_list_for_node(acl, tpg);
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
+		put_nacl(acl);
 		return NULL;
 	}
 	/*

drivers/target/target_core_tpg.c struct se_node_acl *core_tpg_check_initiator_node_acl( · line 299

Quoted 9 lines

drivers/target/target_core_tpg.c +0 -6

@@ -291,12 +290,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 }
 EXPORT_SYMBOL(core_tpg_check_initiator_node_acl);
 
-void core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *nacl)
-{
-	while (atomic_read(&nacl->acl_pr_ref_count) != 0)
-		cpu_relax();
-}
-

Same issues here again, although this is the only one of the nine conversions that that actually pays attention to what should and should not be released before the final put appears.

However, still returning from configfs_group_operations->drop_item() before all references have completed is bad, bad, bad.

NAK.

--nab

Andy Grover

author

2013-12-13 23:59 UTC #63 lore raw

PATCH 29/32 target: Simplify params to core_tpg_del_initiator_node_acl

In reply to Andy Grover

'force' parameter is not used.

tpg parameter is not needed, since we have acl->se_tpg.

Signed-off-by Andy Grover <[email protected]>

---
 Documentation/target/tcm_mod_builder.py      |    2 +-
 drivers/infiniband/ulp/srpt/ib_srpt.c        |    2 +-
 drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    2 +-
 drivers/target/iscsi/iscsi_target_configfs.c |    5 ++---
 drivers/target/sbp/sbp_target.c              |    2 +-
 drivers/target/target_core_tpg.c             |    6 ++----
 drivers/target/tcm_fc/tfc_conf.c             |    2 +-
 drivers/usb/gadget/tcm_usb_gadget.c          |    2 +-
 drivers/vhost/scsi.c                         |    2 +-
 include/target/target_core_fabric.h          |    3 +--
 10 files changed, 12 insertions(+), 16 deletions(-)

Documentation/target/tcm_mod_builder.py +1 -1

index c8e0572..bfebc66 100755
@@ -284,7 +284,7 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
 	buf += "{\n"
 	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
 	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
-	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
+	buf += "	core_tpg_del_initiator_node_acl(se_acl);\n"
 	buf += "}\n\n"
 
 	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"

drivers/infiniband/ulp/srpt/ib_srpt.c +1 -1

index 4995b91..7a23e75 100644
@@ -3644,7 +3644,7 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
 	spin_lock_irq(&sport->port_acl_lock);
 	list_del(&nacl->list);
 	spin_unlock_irq(&sport->port_acl_lock);
-	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
+	core_tpg_del_initiator_node_acl(se_nacl);
 }
 
 static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(

drivers/scsi/qla2xxx/tcm_qla2xxx.c +1 -1

index 4fe684a..05967eb 100644
@@ -828,7 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
 
 static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 /* Start items for tcm_qla2xxx_tpg_attrib_cit */

drivers/target/iscsi/iscsi_target_configfs.c +2 -3

index bd05ab5..3bc1443 100644
@@ -915,7 +915,7 @@ static struct se_node_acl *lio_target_make_nodeacl(
 	if (!stats_cg->default_groups) {
 		pr_err("Unable to allocate memory for"
 				" stats_cg->default_groups\n");
-		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
+		core_tpg_del_initiator_node_acl(se_nacl);
 		return ERR_PTR(-ENOMEM);
 	}

@@ -930,7 +930,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
 static void lio_target_drop_nodeacl(
 	struct se_node_acl *se_nacl)
 {
-	struct se_portal_group *se_tpg = se_nacl->se_tpg;
 	struct iscsi_node_acl *acl = container_of(se_nacl,
 			struct iscsi_node_acl, se_node_acl);
 	struct config_item *df_item;

@@ -945,7 +944,7 @@ static void lio_target_drop_nodeacl(
 	}
 	kfree(stats_cg->default_groups);
 
-	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
+	core_tpg_del_initiator_node_acl(se_nacl);
 }
 
 /* End items for lio_target_acl_cit */

drivers/target/sbp/sbp_target.c +1 -1

index 6fabd9c..1078ee6 100644
@@ -2126,7 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
 
 static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 static int sbp_post_link_lun(

drivers/target/target_core_tpg.c +2 -4

index 6aaf50f..1233d04 100644
@@ -402,13 +402,11 @@ EXPORT_SYMBOL(core_tpg_add_initiator_node_acl);
  *
  *
  */
-int core_tpg_del_initiator_node_acl(
-	struct se_portal_group *tpg,
-	struct se_node_acl *acl,
-	int force)
+int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
 {
 	LIST_HEAD(sess_list);
 	struct se_session *sess, *sess_tmp;
+	struct se_portal_group *tpg = acl->se_tpg;
 	unsigned long flags;
 	int rc;

drivers/target/tcm_fc/tfc_conf.c +1 -1

index 2a44ad6..171950b 100644
@@ -238,7 +238,7 @@ static void ft_del_acl(struct se_node_acl *se_acl)
 	pr_debug("del acl %p se_acl %p tpg %p se_tpg %p\n",
 		    acl, se_acl, tpg, &tpg->se_tpg);
 
-	core_tpg_del_initiator_node_acl(&tpg->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)

drivers/usb/gadget/tcm_usb_gadget.c +1 -1

index 2cedb4b..ffbbed1 100644
@@ -1526,7 +1526,7 @@ static struct se_node_acl *usbg_make_nodeacl(
 
 static void usbg_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 struct usbg_tpg *the_only_tpg_I_currently_have;

drivers/vhost/scsi.c +1 -1

index a93169b..96171db 100644
@@ -1687,7 +1687,7 @@ tcm_vhost_make_nodeacl(struct se_portal_group *se_tpg,
 
 static void tcm_vhost_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 static void tcm_vhost_free_cmd_map_res(struct tcm_vhost_nexus *nexus,

include/target/target_core_fabric.h +1 -2

index 0267780..35a77af 100644
@@ -144,8 +144,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(struct se_portal_group *,
 void	core_tpg_clear_object_luns(struct se_portal_group *);
 struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *,
 		struct se_node_acl *, const char *, u32);
-int	core_tpg_del_initiator_node_acl(struct se_portal_group *,
-		struct se_node_acl *, int);
+int	core_tpg_del_initiator_node_acl(struct se_node_acl *);
 int	core_tpg_set_initiator_node_queue_depth(struct se_portal_group *,
 		unsigned char *, u32, int);
 int	core_tpg_set_initiator_node_tag(struct se_portal_group *,

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 22:00 UTC #64 lore raw

Re: [PATCH 29/32] target: Simplify params to core_tpg_del_initiator_node_acl

Quoted 5 lines · Andy Grover

'force' parameter is not used.
tpg parameter is not needed, since we have acl->se_tpg.
Signed-off-by Andy Grover <[email protected]>

Quoted 13 lines

---
 Documentation/target/tcm_mod_builder.py      |    2 +-
 drivers/infiniband/ulp/srpt/ib_srpt.c        |    2 +-
 drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    2 +-
 drivers/target/iscsi/iscsi_target_configfs.c |    5 ++---
 drivers/target/sbp/sbp_target.c              |    2 +-
 drivers/target/target_core_tpg.c             |    6 ++----
 drivers/target/tcm_fc/tfc_conf.c             |    2 +-
 drivers/usb/gadget/tcm_usb_gadget.c          |    2 +-
 drivers/vhost/scsi.c                         |    2 +-
 include/target/target_core_fabric.h          |    3 +--
 10 files changed, 12 insertions(+), 16 deletions(-)

I'm OK to take this one, but it does not apply without #28.

--nab

Quoted 155 lines

Documentation/target/tcm_mod_builder.py +1 -1

index c8e0572..bfebc66 100755
@@ -284,7 +284,7 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
 	buf += "{\n"
 	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
 	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
-	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
+	buf += "	core_tpg_del_initiator_node_acl(se_acl);\n"
 	buf += "}\n\n"
 
 	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"

drivers/infiniband/ulp/srpt/ib_srpt.c +1 -1

index 4995b91..7a23e75 100644
@@ -3644,7 +3644,7 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
 	spin_lock_irq(&sport->port_acl_lock);
 	list_del(&nacl->list);
 	spin_unlock_irq(&sport->port_acl_lock);
-	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
+	core_tpg_del_initiator_node_acl(se_nacl);
 }
 
 static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(

drivers/scsi/qla2xxx/tcm_qla2xxx.c +1 -1

index 4fe684a..05967eb 100644
@@ -828,7 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
 
 static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 /* Start items for tcm_qla2xxx_tpg_attrib_cit */

drivers/target/iscsi/iscsi_target_configfs.c +2 -3

index bd05ab5..3bc1443 100644
@@ -915,7 +915,7 @@ static struct se_node_acl *lio_target_make_nodeacl(
 	if (!stats_cg->default_groups) {
 		pr_err("Unable to allocate memory for"
 				" stats_cg->default_groups\n");
-		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
+		core_tpg_del_initiator_node_acl(se_nacl);
 		return ERR_PTR(-ENOMEM);
 	}

@@ -930,7 +930,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
 static void lio_target_drop_nodeacl(
 	struct se_node_acl *se_nacl)
 {
-	struct se_portal_group *se_tpg = se_nacl->se_tpg;
 	struct iscsi_node_acl *acl = container_of(se_nacl,
 			struct iscsi_node_acl, se_node_acl);
 	struct config_item *df_item;

@@ -945,7 +944,7 @@ static void lio_target_drop_nodeacl(
 	}
 	kfree(stats_cg->default_groups);
 
-	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
+	core_tpg_del_initiator_node_acl(se_nacl);
 }
 
 /* End items for lio_target_acl_cit */

drivers/target/sbp/sbp_target.c +1 -1

index 6fabd9c..1078ee6 100644
@@ -2126,7 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
 
 static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 static int sbp_post_link_lun(

drivers/target/target_core_tpg.c +2 -4

index 6aaf50f..1233d04 100644
@@ -402,13 +402,11 @@ EXPORT_SYMBOL(core_tpg_add_initiator_node_acl);
  *
  *
  */
-int core_tpg_del_initiator_node_acl(
-	struct se_portal_group *tpg,
-	struct se_node_acl *acl,
-	int force)
+int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
 {
 	LIST_HEAD(sess_list);
 	struct se_session *sess, *sess_tmp;
+	struct se_portal_group *tpg = acl->se_tpg;
 	unsigned long flags;
 	int rc;

drivers/target/tcm_fc/tfc_conf.c +1 -1

index 2a44ad6..171950b 100644
@@ -238,7 +238,7 @@ static void ft_del_acl(struct se_node_acl *se_acl)
 	pr_debug("del acl %p se_acl %p tpg %p se_tpg %p\n",
 		    acl, se_acl, tpg, &tpg->se_tpg);
 
-	core_tpg_del_initiator_node_acl(&tpg->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)

drivers/usb/gadget/tcm_usb_gadget.c +1 -1

index 2cedb4b..ffbbed1 100644
@@ -1526,7 +1526,7 @@ static struct se_node_acl *usbg_make_nodeacl(
 
 static void usbg_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 struct usbg_tpg *the_only_tpg_I_currently_have;

drivers/vhost/scsi.c +1 -1

index a93169b..96171db 100644
@@ -1687,7 +1687,7 @@ tcm_vhost_make_nodeacl(struct se_portal_group *se_tpg,
 
 static void tcm_vhost_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 static void tcm_vhost_free_cmd_map_res(struct tcm_vhost_nexus *nexus,

include/target/target_core_fabric.h +1 -2

index 0267780..35a77af 100644
@@ -144,8 +144,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(struct se_portal_group *,
 void	core_tpg_clear_object_luns(struct se_portal_group *);
 struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *,
 		struct se_node_acl *, const char *, u32);
-int	core_tpg_del_initiator_node_acl(struct se_portal_group *,
-		struct se_node_acl *, int);
+int	core_tpg_del_initiator_node_acl(struct se_node_acl *);
 int	core_tpg_set_initiator_node_queue_depth(struct se_portal_group *,
 		unsigned char *, u32, int);
 int	core_tpg_set_initiator_node_tag(struct se_portal_group *,

Andy Grover

author

2013-12-13 23:59 UTC #65 lore raw

PATCH 30/32 target: Change nacl's session refcount to use existing refcount

In reply to Andy Grover

core_tpg_del_initiator_node_acl is now safe to call with spinlocks, since it no longer potentially sleeps.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_tpg.c       |   10 ----------
 drivers/target/target_core_transport.c |   17 ++---------------
 include/target/target_core_base.h      |    2 --
 include/target/target_core_fabric.h    |    1 -
 4 files changed, 2 insertions(+), 28 deletions(-)

drivers/target/target_core_tpg.c +0 ~~-10~~

index 1233d04..30af019 100644
@@ -247,8 +247,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 
 	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
-	kref_init(&acl->acl_kref);
-	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
 	kref_init(&acl->refcount);

@@ -363,8 +361,6 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 
 	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
-	kref_init(&acl->acl_kref);
-	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
 	kref_init(&acl->refcount);

@@ -440,12 +436,6 @@ int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
 			continue;
 		target_put_session(sess);
 	}
-	target_put_nacl(acl);
-	/*
-	 * Wait for last target_put_nacl() to complete in target_complete_nacl()
-	 * for active fabric session transport_deregister_session() callbacks.
-	 */
-	wait_for_completion(&acl->acl_free_comp);
 
 	put_nacl(acl);

drivers/target/target_core_transport.c +2 ~~-15~~

index 51e294b..e5e054e 100644
@@ -311,7 +311,7 @@ void __transport_register_session(
 					&buf[0], PR_REG_ISID_LEN);
 			se_sess->sess_bin_isid = get_unaligned_be64(&buf[0]);
 		}
-		kref_get(&se_nacl->acl_kref);
+		get_nacl(se_nacl);
 
 		spin_lock_irq(&se_nacl->nacl_sess_lock);
 		/*

@@ -372,19 +372,6 @@ void target_put_session(struct se_session *se_sess)
 }
 EXPORT_SYMBOL(target_put_session);
 
-static void target_complete_nacl(struct kref *kref)
-{
-	struct se_node_acl *nacl = container_of(kref,
-				struct se_node_acl, acl_kref);
-
-	complete(&nacl->acl_free_comp);
-}
-
-void target_put_nacl(struct se_node_acl *nacl)
-{
-	kref_put(&nacl->acl_kref, target_complete_nacl);
-}
-
 void transport_deregister_session_configfs(struct se_session *se_sess)
 {
 	struct se_node_acl *se_nacl;

@@ -471,7 +458,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	 * removal context.
 	 */
 	if (se_nacl)
-		target_put_nacl(se_nacl);
+		put_nacl(se_nacl);
 
 	transport_free_session(se_sess);
 }

include/target/target_core_base.h +0 -2

index 393bcfd..08ecfed 100644
@@ -527,8 +527,6 @@ struct se_node_acl {
 	struct config_group	*acl_default_groups[5];
 	struct list_head	acl_node;
 	struct list_head	acl_sess_list;
-	struct completion	acl_free_comp;
-	struct kref		acl_kref;
 };
 
 struct se_session {

include/target/target_core_fabric.h +0 -1

index 35a77af..c6b9303 100644
@@ -94,7 +94,6 @@ void	transport_register_session(struct se_portal_group *,
 void	target_get_session(struct se_session *);
 void	target_put_session(struct se_session *);
 void	transport_free_session(struct se_session *);
-void	target_put_nacl(struct se_node_acl *);
 void	transport_deregister_session_configfs(struct se_session *);
 void	transport_deregister_session(struct se_session *);

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 22:01 UTC #66 lore raw

Re: [PATCH 30/32] target: Change nacl's session refcount to use existing refcount

Quoted 11 lines · Andy Grover

core_tpg_del_initiator_node_acl is now safe to call with spinlocks, since it no longer potentially sleeps.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_tpg.c       |   10 ----------
 drivers/target/target_core_transport.c |   17 ++---------------
 include/target/target_core_base.h      |    2 --
 include/target/target_core_fabric.h    |    1 -
 4 files changed, 2 insertions(+), 28 deletions(-)

Ignoring, given the other NAKs.

--nab

Quoted 102 lines

drivers/target/target_core_tpg.c +0 ~~-10~~

index 1233d04..30af019 100644
@@ -247,8 +247,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 
 	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
-	kref_init(&acl->acl_kref);
-	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
 	kref_init(&acl->refcount);

@@ -363,8 +361,6 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 
 	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
-	kref_init(&acl->acl_kref);
-	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
 	kref_init(&acl->refcount);

@@ -440,12 +436,6 @@ int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
 			continue;
 		target_put_session(sess);
 	}
-	target_put_nacl(acl);
-	/*
-	 * Wait for last target_put_nacl() to complete in target_complete_nacl()
-	 * for active fabric session transport_deregister_session() callbacks.
-	 */
-	wait_for_completion(&acl->acl_free_comp);
 
 	put_nacl(acl);

drivers/target/target_core_transport.c +2 ~~-15~~

index 51e294b..e5e054e 100644
@@ -311,7 +311,7 @@ void __transport_register_session(
 					&buf[0], PR_REG_ISID_LEN);
 			se_sess->sess_bin_isid = get_unaligned_be64(&buf[0]);
 		}
-		kref_get(&se_nacl->acl_kref);
+		get_nacl(se_nacl);
 
 		spin_lock_irq(&se_nacl->nacl_sess_lock);
 		/*

@@ -372,19 +372,6 @@ void target_put_session(struct se_session *se_sess)
 }
 EXPORT_SYMBOL(target_put_session);
 
-static void target_complete_nacl(struct kref *kref)
-{
-	struct se_node_acl *nacl = container_of(kref,
-				struct se_node_acl, acl_kref);
-
-	complete(&nacl->acl_free_comp);
-}
-
-void target_put_nacl(struct se_node_acl *nacl)
-{
-	kref_put(&nacl->acl_kref, target_complete_nacl);
-}
-
 void transport_deregister_session_configfs(struct se_session *se_sess)
 {
 	struct se_node_acl *se_nacl;

@@ -471,7 +458,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	 * removal context.
 	 */
 	if (se_nacl)
-		target_put_nacl(se_nacl);
+		put_nacl(se_nacl);
 
 	transport_free_session(se_sess);
 }

include/target/target_core_base.h +0 -2

index 393bcfd..08ecfed 100644
@@ -527,8 +527,6 @@ struct se_node_acl {
 	struct config_group	*acl_default_groups[5];
 	struct list_head	acl_node;
 	struct list_head	acl_sess_list;
-	struct completion	acl_free_comp;
-	struct kref		acl_kref;
 };
 
 struct se_session {

include/target/target_core_fabric.h +0 -1

index 35a77af..c6b9303 100644
@@ -94,7 +94,6 @@ void	transport_register_session(struct se_portal_group *,
 void	target_get_session(struct se_session *);
 void	target_put_session(struct se_session *);
 void	transport_free_session(struct se_session *);
-void	target_put_nacl(struct se_node_acl *);
 void	transport_deregister_session_configfs(struct se_session *);
 void	transport_deregister_session(struct se_session *);

Andy Grover

author

2013-12-13 23:59 UTC #67 lore raw

PATCH 31/32 target: Don't release and re-acquire some spinlocks in loops

In reply to Andy Grover

Here are some instances where we're looping, but then dropping the spinlock around the loop in the loop, because we need to be able to sleep in the calls. Since everything is refcounted now, this should no longer be needed and we can just hold the locks the whole time.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_device.c |    4 ----
 drivers/target/target_core_tpg.c    |    5 -----
 2 files changed, 0 insertions(+), 9 deletions(-)

drivers/target/target_core_device.c +0 -4

index a432d7b..3896c99 100644
@@ -463,7 +463,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 
 	spin_lock_irq(&tpg->acl_node_lock);
 	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
-		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);
 		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {

@@ -473,7 +472,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 		}
 		spin_unlock_irq(&nacl->device_list_lock);
 
-		spin_lock_irq(&tpg->acl_node_lock);
 	}
 	spin_unlock_irq(&tpg->acl_node_lock);
 }

@@ -1141,9 +1139,7 @@ int core_dev_add_lun(
 			if (acl->dynamic_node_acl &&
 			    (!tpg->se_tpg_tfo->tpg_check_demo_mode_login_only ||
 			     !tpg->se_tpg_tfo->tpg_check_demo_mode_login_only(tpg))) {
-				spin_unlock_irq(&tpg->acl_node_lock);
 				core_tpg_add_node_to_devs(acl, tpg);
-				spin_lock_irq(&tpg->acl_node_lock);
 			}
 		}
 		spin_unlock_irq(&tpg->acl_node_lock);

drivers/target/target_core_tpg.c +0 -5

index 30af019..1bcb665 100644
@@ -172,8 +172,6 @@ void core_tpg_add_node_to_devs(
 	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
 		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
 
-		spin_unlock(&tpg->tpg_lun_lock);
-
 		dev = lun->lun_se_dev;
 		/*
 		 * By default in LIO-Target $FABRIC_MOD,

@@ -201,7 +199,6 @@ void core_tpg_add_node_to_devs(
 
 		core_enable_device_list_for_node(lun, NULL, lun->unpacked_lun,
 				lun_access, acl, tpg);
-		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
 }

@@ -299,9 +296,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 		if (!lun->lun_se_dev)
 			continue;
 
-		spin_unlock(&tpg->tpg_lun_lock);
 		core_dev_del_lun(tpg, lun);
-		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
 }

Signature · 1 line

1.7.1

Nicholas A. Bellinger

2013-12-16 22:01 UTC #68 lore raw

Re: [PATCH 31/32] target: Don't release and re-acquire some spinlocks in loops

Quoted 11 lines · Andy Grover

Here are some instances where we're looping, but then dropping the spinlock around the loop in the loop, because we need to be able to sleep in the calls. Since everything is refcounted now, this should no longer be needed and we can just hold the locks the whole time.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_device.c |    4 ----
 drivers/target/target_core_tpg.c    |    5 -----
 2 files changed, 0 insertions(+), 9 deletions(-)

Ignoring, given the other NAKs.

--nab

Quoted 61 lines

drivers/target/target_core_device.c +0 -4

index a432d7b..3896c99 100644
@@ -463,7 +463,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 
 	spin_lock_irq(&tpg->acl_node_lock);
 	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
-		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);
 		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {

@@ -473,7 +472,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 		}
 		spin_unlock_irq(&nacl->device_list_lock);
 
-		spin_lock_irq(&tpg->acl_node_lock);
 	}
 	spin_unlock_irq(&tpg->acl_node_lock);
 }

@@ -1141,9 +1139,7 @@ int core_dev_add_lun(
 			if (acl->dynamic_node_acl &&
 			    (!tpg->se_tpg_tfo->tpg_check_demo_mode_login_only ||
 			     !tpg->se_tpg_tfo->tpg_check_demo_mode_login_only(tpg))) {
-				spin_unlock_irq(&tpg->acl_node_lock);
 				core_tpg_add_node_to_devs(acl, tpg);
-				spin_lock_irq(&tpg->acl_node_lock);
 			}
 		}
 		spin_unlock_irq(&tpg->acl_node_lock);

drivers/target/target_core_tpg.c +0 -5

index 30af019..1bcb665 100644
@@ -172,8 +172,6 @@ void core_tpg_add_node_to_devs(
 	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
 		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
 
-		spin_unlock(&tpg->tpg_lun_lock);
-
 		dev = lun->lun_se_dev;
 		/*
 		 * By default in LIO-Target $FABRIC_MOD,

@@ -201,7 +199,6 @@ void core_tpg_add_node_to_devs(
 
 		core_enable_device_list_for_node(lun, NULL, lun->unpacked_lun,
 				lun_access, acl, tpg);
-		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
 }

@@ -299,9 +296,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 		if (!lun->lun_se_dev)
 			continue;
 
-		spin_unlock(&tpg->tpg_lun_lock);
 		core_dev_del_lun(tpg, lun);
-		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
 }

Andy Grover

author

2013-12-13 23:59 UTC #69 lore raw

PATCH 32/32 target: Increase MAX_LUNS_PER_TPG to 16384

In reply to Andy Grover

Indicate support for hierarchical LUN addressing.

Set address method field in each LUN reported by REPORT LUNS to 1, in accordance with SCSI SAM specs.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_spc.c  |    8 ++++++--
 include/target/target_core_base.h |    4 ++--
 2 files changed, 8 insertions(+), 4 deletions(-)

drivers/target/target_core_spc.c +6 -2

index 0b678fc..04690bf 100644
@@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	buf[2] = 0x05; /* SPC-3 */
 
 	/*
-	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
+	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
 	 *
 	 * SPC4 says:
 	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the

@@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	 *   obsolete. Response data format values greater than 2h are
 	 *   reserved.
 	 */
-	buf[3] = 2;
+	buf[3] = 0x12;
 
 	/*
 	 * Enable SCCS and TPGS fields for Emulated ALUA

@@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
+
+		/* Address method 1 for hier flat-space address. see SAM-5 */
+		buf[offset] |= (1 << 6);
+
 		offset += 8;
 	}
 	spin_unlock_irq(&sess->se_node_acl->device_list_lock);

include/target/target_core_base.h +2 -2

index 08ecfed..537cc67 100644
@@ -14,8 +14,8 @@
 #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
 
 /* Maximum Number of LUNs per Target Portal Group */
-/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
-#define TRANSPORT_MAX_LUNS_PER_TPG		256
+/* Don't raise above 16384 or a different format in report_luns is needed */
+#define TRANSPORT_MAX_LUNS_PER_TPG		16384
 /*
  * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
  *

Signature · 1 line

1.7.1

Hannes Reinecke

2013-12-16 09:20 UTC #70 lore raw

Re: [PATCH 32/32] target: Increase MAX_LUNS_PER_TPG to 16384

Quoted 50 lines · Andy Grover

Indicate support for hierarchical LUN addressing.

Set address method field in each LUN reported by REPORT LUNS to 1, in accordance with SCSI SAM specs.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_spc.c  |    8 ++++++--
 include/target/target_core_base.h |    4 ++--
 2 files changed, 8 insertions(+), 4 deletions(-)

drivers/target/target_core_spc.c +6 -2

index 0b678fc..04690bf 100644
@@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	buf[2] = 0x05; /* SPC-3 */
 
 	/*
-	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
+	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
 	 *
 	 * SPC4 says:
 	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the

@@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	 *   obsolete. Response data format values greater than 2h are
 	 *   reserved.
 	 */
-	buf[3] = 2;
+	buf[3] = 0x12;
 
 	/*
 	 * Enable SCCS and TPGS fields for Emulated ALUA

@@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
+
+		/* Address method 1 for hier flat-space address. see SAM-5 */
+		buf[offset] |= (1 << 6);
+
 		offset += 8;
 	}
 	spin_unlock_irq(&sess->se_node_acl->device_list_lock);

include/target/target_core_base.h +0 -0

index 08ecfed..537cc67 100644

include/target/target_core_base.h line 21

Quoted 10 lines

include/target/target_core_base.h +2 -2

@@ -14,8 +14,8 @@
 #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
 
 /* Maximum Number of LUNs per Target Portal Group */
-/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
-#define TRANSPORT_MAX_LUNS_PER_TPG		256
+/* Don't raise above 16384 or a different format in report_luns is needed */
+#define TRANSPORT_MAX_LUNS_PER_TPG		16384
 /*
  * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
  *

NO WAY.

TCM is (currently) keeping the LUNs in a static array. Which is sort-of-okay if the number of LUNs is somewhat reasonable, as it was for 256.
But 16k is way beyond any sensible number and will increase the memory footprint by a massive amount, irrespective whether someone will actually _use_ such a large number of LUNs.

Before attempting that one would need to convert the LUN array into something scalable, like a linked list or somesuch.

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Hannes Reinecke

2013-12-16 09:24 UTC #71 lore raw

Re: [PATCH 32/32] target: Increase MAX_LUNS_PER_TPG to 16384

Quoted 70 lines · Hannes Reinecke

Quoted 50 lines · Andy Grover

Indicate support for hierarchical LUN addressing.

Set address method field in each LUN reported by REPORT LUNS to 1, in accordance with SCSI SAM specs.

Signed-off-by Andy Grover <[email protected]>

---
 drivers/target/target_core_spc.c  |    8 ++++++--
 include/target/target_core_base.h |    4 ++--
 2 files changed, 8 insertions(+), 4 deletions(-)

drivers/target/target_core_spc.c +6 -2

index 0b678fc..04690bf 100644
@@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	buf[2] = 0x05; /* SPC-3 */
 
 	/*
-	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
+	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
 	 *
 	 * SPC4 says:
 	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the

@@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	 *   obsolete. Response data format values greater than 2h are
 	 *   reserved.
 	 */
-	buf[3] = 2;
+	buf[3] = 0x12;
 
 	/*
 	 * Enable SCCS and TPGS fields for Emulated ALUA

@@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
+
+		/* Address method 1 for hier flat-space address. see SAM-5 */
+		buf[offset] |= (1 << 6);
+
 		offset += 8;
 	}
 	spin_unlock_irq(&sess->se_node_acl->device_list_lock);

include/target/target_core_base.h +0 -0

index 08ecfed..537cc67 100644

include/target/target_core_base.h line 21

Quoted 10 lines

include/target/target_core_base.h +2 -2

@@ -14,8 +14,8 @@
 #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
 
 /* Maximum Number of LUNs per Target Portal Group */
-/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
-#define TRANSPORT_MAX_LUNS_PER_TPG		256
+/* Don't raise above 16384 or a different format in report_luns is needed */
+#define TRANSPORT_MAX_LUNS_PER_TPG		16384
 /*
  * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
  *

NO WAY.

Quoted 3 lines

Before attempting that one would need to convert the LUN array into something scalable, like a linked list or somesuch.

Ouch.

Please ignore this mail.

One should _really_ read the entire patchset before jumping to conclusions.
Monday morning and all that ...

Sorry for the noise.

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Nicholas A. Bellinger

2013-12-16 22:01 UTC #72 lore raw

Re: [PATCH 32/32] target: Increase MAX_LUNS_PER_TPG to 16384

In reply to Andy Grover

Quoted 11 lines · Andy Grover

Indicate support for hierarchical LUN addressing.
Set address method field in each LUN reported by REPORT LUNS to 1, in accordance with SCSI SAM specs.
Signed-off-by Andy Grover <[email protected]>
---
 drivers/target/target_core_spc.c  |    8 ++++++--
 include/target/target_core_base.h |    4 ++--
 2 files changed, 8 insertions(+), 4 deletions(-)

And ignoring, given the other NAKs.

--nab

Quoted 48 lines

drivers/target/target_core_spc.c +6 -2

index 0b678fc..04690bf 100644
@@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	buf[2] = 0x05; /* SPC-3 */
 
 	/*
-	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
+	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
 	 *
 	 * SPC4 says:
 	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the

@@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	 *   obsolete. Response data format values greater than 2h are
 	 *   reserved.
 	 */
-	buf[3] = 2;
+	buf[3] = 0x12;
 
 	/*
 	 * Enable SCCS and TPGS fields for Emulated ALUA

@@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
+
+		/* Address method 1 for hier flat-space address. see SAM-5 */
+		buf[offset] |= (1 << 6);
+
 		offset += 8;
 	}
 	spin_unlock_irq(&sess->se_node_acl->device_list_lock);

include/target/target_core_base.h +2 -2

index 08ecfed..537cc67 100644
@@ -14,8 +14,8 @@
 #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
 
 /* Maximum Number of LUNs per Target Portal Group */
-/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
-#define TRANSPORT_MAX_LUNS_PER_TPG		256
+/* Don't raise above 16384 or a different format in report_luns is needed */
+#define TRANSPORT_MAX_LUNS_PER_TPG		16384
 /*
  * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
  *

Nicholas A. Bellinger

2013-12-16 22:02 UTC #73 lore raw

In reply to Andy Grover

Quoted 13 lines · Andy Grover

Hi Nicholas,
This patchset uses krefs to refcount structures shared across threads. LIO is full of these because configfs-based configuration actions can be removing an object, even while that object is being used by a SCSI command.
Using kref to free the struct on whichever thread drops the last reference allows us to avoid busy-waiting in configfs removal functions. Next, this set removes the statically-sized tpg lun and deve arrays in favor of dynamically adding entries into rbtrees. This reduces memory consumption and allows more than 255 luns per tpg and initiator mapping.

Quoted 9 lines

Except for some rbtree lookups, these changes are entirely in the configuration paths of Lio. I have tested these as extensively as I can, and it's ready for wider testing.
Note: patch 22 converts a percpu refcount to a normal kref. I'd argue the benefit is really in the "refcount" part rather than the "percpu", so a simpler kref does the job, but we might want to discuss this some more.

It would be helpful to breakup future patches into different series based on:

* Bugfixes
* New features
* Minor improvements

Thanks,

--nab

Andy Grover

author

2013-12-17 01:49 UTC #74 lore raw

Quoted 24 lines · Nicholas A. Bellinger

Quoted 13 lines · Andy Grover
Hi Nicholas,
This patchset uses krefs to refcount structures shared across threads. LIO is full of these because configfs-based configuration actions can be removing an object, even while that object is being used by a SCSI command.
Using kref to free the struct on whichever thread drops the last reference allows us to avoid busy-waiting in configfs removal functions. Next, this set removes the statically-sized tpg lun and deve arrays in favor of dynamically adding entries into rbtrees. This reduces memory consumption and allows more than 255 luns per tpg and initiator mapping.
Quoted 9 lines
Except for some rbtree lookups, these changes are entirely in the configuration paths of Lio. I have tested these as extensively as I can, and it's ready for wider testing.
Note: patch 22 converts a percpu refcount to a normal kref. I'd argue the benefit is really in the "refcount" part rather than the "percpu", so a simpler kref does the job, but we might want to discuss this some more.

Quoted 6 lines

It would be helpful to breakup future patches into different series based on:
* Bugfixes
* New features
* Minor improvements

Thanks for all the reviews. For the remaining changes, it'll be after the new year before a v2. I'm still hopeful we can work out the issues preventing the use of krefs.

I'm somewhat surprised over the resistance to rbtrees. I guess I'll need to hold off on resubmitting that until I have some performance data on their effect.

Regards -- Andy

Nicholas A. Bellinger

2013-12-17 01:59 UTC #75 lore raw

Quoted 28 lines · Andy Grover

Quoted 27 lines · Nicholas A. Bellinger
Quoted 13 lines · Andy Grover
Hi Nicholas,
This patchset uses krefs to refcount structures shared across threads. LIO is full of these because configfs-based configuration actions can be removing an object, even while that object is being used by a SCSI command.
Using kref to free the struct on whichever thread drops the last reference allows us to avoid busy-waiting in configfs removal functions. Next, this set removes the statically-sized tpg lun and deve arrays in favor of dynamically adding entries into rbtrees. This reduces memory consumption and allows more than 255 luns per tpg and initiator mapping.
Quoted 9 lines
Except for some rbtree lookups, these changes are entirely in the configuration paths of Lio. I have tested these as extensively as I can, and it's ready for wider testing.
Note: patch 22 converts a percpu refcount to a normal kref. I'd argue the benefit is really in the "refcount" part rather than the "percpu", so a simpler kref does the job, but we might want to discuss this some more.
It would be helpful to breakup future patches into different series based on:

Quoted 8 lines

Quoted 3 lines
* Bugfixes
* New features
* Minor improvements
Thanks for all the reviews. For the remaining changes, it'll be after the new year before a v2. I'm still hopeful we can work out the issues preventing the use of krefs.

So I'd like to see krefs implemented for these cases, but the rub is that configfs_group_operations->drop_items() cannot return before the last reference has dropped.

Quoted 4 lines

I'm somewhat surprised over the resistance to rbtrees. I guess I'll need to hold off on resubmitting that until I have some performance data on their effect.

I do appreciate the effort, but the extra pointer chasing here makes rbtrees prohibitively expensive for fast past operations.

--nab

Andy Grover

author

2013-12-17 02:03 UTC #76 lore raw

Quoted 2 lines · Nicholas A. Bellinger

I do appreciate the effort, but the extra pointer chasing here makes rbtrees prohibitively expensive for fast past operations.

Hmm, were they actually implemented and tried at some point?

Nicholas A. Bellinger

2013-12-17 03:03 UTC #77 lore raw

Quoted 7 lines · Andy Grover

Quoted 2 lines · Nicholas A. Bellinger
I do appreciate the effort, but the extra pointer chasing here makes rbtrees prohibitively expensive for fast past operations.
Hmm, were they actually implemented and tried at some point?

This is based upon experiences with PCIe IOV addressing using rbtrees.

--nab

Hannes Reinecke

2013-12-17 10:53 UTC #78 lore raw

In reply to Nicholas A. Bellinger

Quoted 24 lines · Nicholas A. Bellinger

Quoted 13 lines · Andy Grover
Hi Nicholas,
This patchset uses krefs to refcount structures shared across threads. LIO is full of these because configfs-based configuration actions can be removing an object, even while that object is being used by a SCSI command.
Using kref to free the struct on whichever thread drops the last reference allows us to avoid busy-waiting in configfs removal functions. Next, this set removes the statically-sized tpg lun and deve arrays in favor of dynamically adding entries into rbtrees. This reduces memory consumption and allows more than 255 luns per tpg and initiator mapping.
Quoted 9 lines
Except for some rbtree lookups, these changes are entirely in the configuration paths of Lio. I have tested these as extensively as I can, and it's ready for wider testing.
Note: patch 22 converts a percpu refcount to a normal kref. I'd argue the benefit is really in the "refcount" part rather than the "percpu", so a simpler kref does the job, but we might want to discuss this some more.

Quoted 7 lines

It would be helpful to breakup future patches into different series based on:
* Bugfixes
* New features
* Minor improvements

But the LUN addressing improvements is interesting.

What I found during development of the 64bit LUN patchset is that the target core stuff has a very rudimentary LUN handling:
- 256 LUNs only
- LUNs are kept in a static array
- Identity mapping between LUN numbers and array indices.

What I _really_ would like to have is to do away with the LUN array, and introduce a dynamic LUN mapping.
This will allow us to easily implement different LUN enumeration methods (Think of hierarchical LUNs ...).

Also the assumption that a static array is always faster for lookup than a linked list is wrong.
A static array is faster if the entire array fits into the processor cache. If it doesn't we basically have an immediate
cache miss _for every array access_.
Then linked lists etc really are faster.

So do not take things at face value; only real measurements count here.

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Andy Grover

author

2013-12-17 17:10 UTC #79 lore raw

Quoted 20 lines · Hannes Reinecke

But the LUN addressing improvements is interesting.
What I found during development of the 64bit LUN patchset is that the target core stuff has a very rudimentary LUN handling:
- 256 LUNs only
- LUNs are kept in a static array
- Identity mapping between LUN numbers and array indices.
What I _really_ would like to have is to do away with the LUN array, and introduce a dynamic LUN mapping.
This will allow us to easily implement different LUN enumeration methods (Think of hierarchical LUNs ...).
Also the assumption that a static array is always faster for lookup than a linked list is wrong.
A static array is faster if the entire array fits into the processor cache. If it doesn't we basically have an immediate
cache miss _for every array access_.
Then linked lists etc really are faster.

Quoted 2 lines

So do not take things at face value; only real measurements count here.

Yeah, seems true. So do either of you have any tips for how best to do measurements?

BTW I was looking a little more and am now inclined to think a radix tree is more suitable than rbtree. It seems ideally suited for our case: want to lookup by an index (i.e. LUN) that is usually a small integer, very fast lookups, yet also can grow and still be space-efficient.

Regards -- Andy

Nicholas A. Bellinger

2013-12-17 21:25 UTC #80 lore raw

Quoted 22 lines · Andy Grover

Quoted 20 lines · Hannes Reinecke
But the LUN addressing improvements is interesting.
What I found during development of the 64bit LUN patchset is that the target core stuff has a very rudimentary LUN handling:
- 256 LUNs only
- LUNs are kept in a static array
- Identity mapping between LUN numbers and array indices.
What I _really_ would like to have is to do away with the LUN array, and introduce a dynamic LUN mapping.
This will allow us to easily implement different LUN enumeration methods (Think of hierarchical LUNs ...).
Also the assumption that a static array is always faster for lookup than a linked list is wrong.
A static array is faster if the entire array fits into the processor cache. If it doesn't we basically have an immediate
cache miss _for every array access_.
Then linked lists etc really are faster.

Quoted 6 lines

Quoted 2 lines
So do not take things at face value; only real measurements count here.
Yeah, seems true. So do either of you have any tips for how best to do measurements?

perf mem -t all record -a sleep 20

perf mem report --sort=mem,sym,dso,symbol_daddr,dso_daddr,tlb,locked

Quoted 5 lines

BTW I was looking a little more and am now inclined to think a radix tree is more suitable than rbtree. It seems ideally suited for our case:
want to lookup by an index (i.e. LUN) that is usually a small integer, very fast lookups, yet also can grow and still be space-efficient.

My understanding is that radix trees would involve less memory accesses vs. rbtrees, and less misses when cache is cold.

--nab