target: fix up locking/refcounting in IO paths
Subsystems (6)
v3 cover target: fix up locking/refcounting in IO paths
The following patches made over Martin's staging branch fix some ref counting issues I hit while testing and improves the locking in the IO paths. To do the latter, the patches:
1. move the sess_cmd_lock to tcm_qla2xxx since it was the only driver using the sess_cmd_list.
2. makes the execution lock/list per cpu'ish. I just allocate nr_cpu_ids's worth of lock/lists then make sure we complete the cmd on the cpu it was started on.
With the patches I'm seeing a 25% improvement in IOPs for small IO tests like:
fio --filename=/dev/sdXYZ --direct=1 --rw=randrw --bs=4k \
--iodepth8 --numjobs
with drivers like vhost (with those other patches on the list to fix up multiple virtqueue support) and with the included loop patch when nr hw queues is increased.
v3:
- Fixed issue where qla2xxx's cpuid was overwritten.
- Fixed up email submit prefix to have "qla2xxx".
v2:
- Got access to qla2xxx setup and tested patch. Fixed various issues.
- Added fixes for issues found in the same code paths I was testing:
- target: fix lun ref count handling
- target: fix cmd_count ref leak
v1/RFC
- Initial posting.
PATCH v3 1/8 target: fix lun ref count handling
This fixes 2 bugs in the lun refcounting.
1. For the TCM_WRITE_PROTECTED case we were returning an error after taking a ref to the lun, but never dropping it (caller just send status and drops cmd ref).
2. We still need to do a percpu_ref_tryget_live for the virt lun0 like we do for other luns, because the tpg code does the refcount/wait process like it does with other luns.
--- drivers/target/target_core_device.c | 43 +++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 23 deletions(-)
drivers/target/target_core_device.c +20 -23
@@ -65,6 +65,16 @@atomic_long_add (se_cmd->data_length, &deve->read_bytes); +if ((se_cmd->data_direction =DMA_TO_DEVICE ) && + deve->lun_access_ro) { +pr_err ("TARGET_CORE[%s]: Detected WRITE_PROTECTED LUN" +" Access for 0x%08llx\n" , + se_cmd->se_tfo->fabric_name, + se_cmd->orig_fe_lun); +rcu_read_unlock (); +return TCM_WRITE_PROTECTED ; + } + se_lun =rcu_dereference (deve->se_lun);if (!percpu_ref_tryget_live (&se_lun->lun_ref)) {
@@ -76,17 +86,6 @@ se_cmd->pr_res_key = deve->pr_res_key; se_cmd->se_cmd_flags |=SCF_SE_LUN_CMD ; se_cmd->lun_ref_active = true; - -if ((se_cmd->data_direction =DMA_TO_DEVICE ) && - deve->lun_access_ro) { -pr_err ("TARGET_CORE[%s]: Detected WRITE_PROTECTED LUN" -" Access for 0x%08llx\n" , - se_cmd->se_tfo->fabric_name, - se_cmd->orig_fe_lun); -rcu_read_unlock (); - ret =TCM_WRITE_PROTECTED ; - goto ref_dev; - } } out_unlock:rcu_read_unlock ();
@@ -106,21 +105,20 @@return TCM_NON_EXISTENT_LUN ; } - se_lun = se_sess->se_tpg->tpg_virt_lun0; - se_cmd->se_lun = se_sess->se_tpg->tpg_virt_lun0; - se_cmd->se_cmd_flags |=SCF_SE_LUN_CMD ; - -percpu_ref_get (&se_lun->lun_ref); - se_cmd->lun_ref_active = true; -/* * Force WRITE PROTECT for virtual LUN 0 */ if ((se_cmd->data_direction !=DMA_FROM_DEVICE ) && - (se_cmd->data_direction !=DMA_NONE )) { - ret =TCM_WRITE_PROTECTED ; - goto ref_dev; - } + (se_cmd->data_direction !=DMA_NONE )) +return TCM_WRITE_PROTECTED ; + + se_lun = se_sess->se_tpg->tpg_virt_lun0; +if (!percpu_ref_tryget_live (&se_lun->lun_ref)) +return TCM_NON_EXISTENT_LUN ; + + se_cmd->se_lun = se_sess->se_tpg->tpg_virt_lun0; + se_cmd->se_cmd_flags |=SCF_SE_LUN_CMD ; + se_cmd->lun_ref_active = true; }/* * RCU reference protected by percpu se_lun->lun_ref taken above that
@@ -128,7 +126,6 @@* pointer can be kfree_rcu() by the final se_lun->lun_group put via * target_core_fabric_configfs.c:target_fabric_port_release */ -ref_dev: se_cmd->se_dev =rcu_dereference_raw (se_lun->lun_se_dev);atomic_long_inc (&se_cmd->se_dev->num_cmds);
Signature · 1 line
1.8.3.1
Martin K. Petersen
AppliedRe: [PATCH 1/8] target: fix lun ref count handling
Quoted 9 lines · Mike Christie
This fixes 2 bugs in the lun refcounting.
1. For the
TCM_WRITE_PROTECTEDcase we were returning an error after taking a ref to the lun, but never dropping it (caller just send status and drops cmd ref).2. We still need to do a
percpu_ref_tryget_livefor the virt lun0 like we do for other luns, because the tpg code does the refcount/wait process like it does with other luns.
Applied to 5.11/scsi-queue, thanks!
[1/8] scsi: target: Fix LUN ref count handling
https://git.kernel.org/mkp/scsi/c/a2b5d6f975a4 [2/8] scsi: target: Fix cmd_count ref leak https://git.kernel.org/mkp/scsi/c/02dd4914b0bc [3/8] scsi: qla2xxx: Drop TARGET_SCF_LOOKUP_LUN_FROM_TAG https://git.kernel.org/mkp/scsi/c/8f394da36a36 [4/8] scsi: target: Remove TARGET_SCF_LOOKUP_LUN_FROM_TAG https://git.kernel.org/mkp/scsi/c/27b0efd15d52 [5/8] scsi: qla2xxx: Move sess cmd list/lock to driver https://git.kernel.org/mkp/scsi/c/605e74025f95 [6/8] scsi: target: Drop sess_cmd_lock from I/O path https://git.kernel.org/mkp/scsi/c/6f55b06f9b07 [7/8] scsi: target: Make state_list per CPU https://git.kernel.org/mkp/scsi/c/1526d9f10c61 [8/8] scsi: tcm_loop: Allow queues, can_queue and cmd_per_lun to be settable https://git.kernel.org/mkp/scsi/c/94a0dfcf7d33
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
PATCH v3 2/8 target: fix cmd_count ref leak
In reply to Mike Christiepercpu_ref_init sets the refcount to 1 and percpu_ref_kill drops it. Drivers like iscsi and loop do not call target_sess_cmd_list_set_waiting during session shutdown though, so they have been calling percpu_ref_exit
with a refcount still taken and leaking the cmd_counts memory.
--- drivers/target/target_core_transport.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/target/target_core_transport.c +8 -0
@@ -238,6 +238,14 @@ int transport_init_session(struct se_session *se_sess)void transport_uninit_session (struct se_session *se_sess) { +/* +* Drivers like iscsi and loop do not call +* target_sess_cmd_list_set_waiting during session shutdown so we +* have to drop the ref taken at init time here. +*/ +if (!se_sess->sess_tearing_down) +percpu_ref_put (&se_sess->cmd_count); +percpu_ref_exit (&se_sess->cmd_count); }
Signature · 1 line
1.8.3.1
PATCH v3 3/8 qla2xxx: drop TARGET_SCF_LOOKUP_LUN_FROM_TAG
In reply to Mike ChristieIt looks like only the __qlt_24xx_handle_abts code path does not know the lun for an abort and it uses the TARGET_SCF_LOOKUP_LUN_FROM_TAG flag to have lio core look it up. LIO uses target_lookup_lun_from_tag to go from cmd tag to lun for the driver. However, qla2xxx has a tcm_qla2xxx_find_cmd_by_tag which does almost the same thing as the LIO helper (it finds the cmd but does not return the lun). This patch has qla2xxx use its internal helper.
This is more of a transition patch and that is why I'm having qla2xxx use its internal function instead of killing it. The tcm qla2xxx driver is the only that needs the sess_cmd_list, so the first couple of patches move that list from LIO core to the driver. The final patches then remove the sess_cmd_lock from the main IO path.
--- drivers/scsi/qla2xxx/qla_target.c | 21 +++++++++++---------- drivers/scsi/qla2xxx/tcm_qla2xxx.c | 4 +--- 2 files changed, 12 insertions(+), 13 deletions(-)
drivers/scsi/qla2xxx/qla_target.c +11 -10
@@ -2083,6 +2083,7 @@ static int __qlt_24xx_handle_abts(struct scsi_qla_host *vha,struct qla_hw_data *ha = vha->hw;struct qla_tgt_mgmt_cmd *mcmd;struct qla_qpair_hint *h = &vha->vha_tgt.qla_tgt->qphints[0 ]; +struct qla_tgt_cmd *abort_cmd; ql_dbg(ql_dbg_tgt_mgt, vha, 0xf00f, "qla_target(%d): task abort (tag=%d)\n",
@@ -2110,17 +2111,17 @@ static int __qlt_24xx_handle_abts(struct scsi_qla_host *vha,*/ mcmd->se_cmd.cpuid = h->cpuid; -if (ha->tgt.tgt_ops->find_cmd_by_tag) { -struct qla_tgt_cmd *abort_cmd; - - abort_cmd = ha->tgt.tgt_ops->find_cmd_by_tag (sess, + abort_cmd = ha->tgt.tgt_ops->find_cmd_by_tag (sess,le32_to_cpu (abts->exchange_addr_to_abort)); -if (abort_cmd && abort_cmd->qpair) { - mcmd->qpair = abort_cmd->qpair; - mcmd->se_cmd.cpuid = abort_cmd->se_cmd.cpuid; - mcmd->abort_io_attr = abort_cmd->atio.u.isp24.attr; - mcmd->flags =QLA24XX_MGMT_ABORT_IO_ATTR_VALID ; - } +if (!abort_cmd) +return -EIO ; + mcmd->unpacked_lun = abort_cmd->se_cmd.orig_fe_lun; + +if (abort_cmd->qpair) { + mcmd->qpair = abort_cmd->qpair; + mcmd->se_cmd.cpuid = abort_cmd->se_cmd.cpuid; + mcmd->abort_io_attr = abort_cmd->atio.u.isp24.attr; + mcmd->flags =QLA24XX_MGMT_ABORT_IO_ATTR_VALID ; }INIT_WORK (&mcmd->work, qlt_do_tmr_work);
drivers/scsi/qla2xxx/tcm_qla2xxx.c +1 -3
@@ -574,13 +574,11 @@ static int tcm_qla2xxx_handle_tmr(struct qla_tgt_mgmt_cmd *mcmd, u64 lun,struct fc_port *sess = mcmd->sess;struct se_cmd *se_cmd = &mcmd->se_cmd;int transl_tmr_func =0 ; -int flags =TARGET_SCF_ACK_KREF ;switch (tmr_func) {case QLA_TGT_ABTS :pr_debug ("%ld: ABTS received\n" , sess->vha->host_no); transl_tmr_func =TMR_ABORT_TASK ; - flags |=TARGET_SCF_LOOKUP_LUN_FROM_TAG ;break ;case QLA_TGT_2G_ABORT_TASK :pr_debug ("%ld: 2G Abort Task received\n" , sess->vha->host_no);
@@ -613,7 +611,7 @@ static int tcm_qla2xxx_handle_tmr(struct qla_tgt_mgmt_cmd *mcmd, u64 lun, }return target_submit_tmr (se_cmd, sess->se_sess,NULL , lun, mcmd, - transl_tmr_func,GFP_ATOMIC , tag, flags); + transl_tmr_func,GFP_ATOMIC , tag,TARGET_SCF_ACK_KREF ); } static struct qla_tgt_cmd *tcm_qla2xxx_find_cmd_by_tag(struct fc_port *sess,
Signature · 1 line
1.8.3.1
PATCH v3 4/8 target: remove TARGET_SCF_LOOKUP_LUN_FROM_TAG
In reply to Mike ChristieTARGET_SCF_LOOKUP_LUN_FROM_TAG is no longer used so remove it.
--- drivers/target/target_core_transport.c | 33 --------------------------------- include/target/target_core_base.h | 1 - 2 files changed, 34 deletions(-)
drivers/target/target_core_transport.c +0 -33
@@ -1772,29 +1772,6 @@ static void target_complete_tmr_failure(struct work_struct *work)transport_cmd_check_stop_to_fabric (se_cmd); } -static bool target_lookup_lun_from_tag (struct se_session *se_sess,u64 tag, -u64 *unpacked_lun) -{ -struct se_cmd *se_cmd; -unsigned long flags; -bool ret = false; - -spin_lock_irqsave (&se_sess->sess_cmd_lock, flags); -list_for_each_entry (se_cmd, &se_sess->sess_cmd_list, se_cmd_list) { -if (se_cmd->se_cmd_flags &SCF_SCSI_TMR_CDB ) -continue ; - -if (se_cmd->tag = tag) { - *unpacked_lun = se_cmd->orig_fe_lun; - ret = true; -break ; - } - } -spin_unlock_irqrestore (&se_sess->sess_cmd_lock, flags); - -return ret; -} -/** * target_submit_tmr - lookup unpacked lun and submit uninitialized se_cmd * for TMR CDBs
@@ -1842,16 +1819,6 @@ int target_submit_tmr(struct se_cmd *se_cmd, struct se_session *se_sess,core_tmr_release_req (se_cmd->se_tmr_req);return ret; } -/* -* If this is ABORT_TASK with no explicit fabric provided LUN, -* go ahead and search active session tags for a match to figure -* out unpacked_lun for the original se_cmd. -*/ -if (tm_type =TMR_ABORT_TASK && (flags &TARGET_SCF_LOOKUP_LUN_FROM_TAG )) { -if (!target_lookup_lun_from_tag (se_sess, tag, - &se_cmd->orig_fe_lun)) - goto failure; - } ret =transport_lookup_tmr_lun (se_cmd);if (ret)
include/target/target_core_base.h +0 -1
@@ -195,7 +195,6 @@ enum target_sc_flags_table {TARGET_SCF_ACK_KREF =0x02 ,TARGET_SCF_UNKNOWN_SIZE =0x04 , TARGET_SCF_USE_CPUID = 0x08, - TARGET_SCF_LOOKUP_LUN_FROM_TAG = 0x10, };/* fabric independent task management function values */
Signature · 1 line
1.8.3.1
PATCH v3 5/8 qla2xxx: move sess cmd list/lock to driver
In reply to Mike ChristieExcept for debug output in the shutdown path, tcm qla2xxx is the only driver using the se_session sess_cmd_list. This moves the list to that driver so in the next patches we can remove the sess_cmd_lock from the main IO path for the rest of the drivers.
--- drivers/scsi/qla2xxx/qla_def.h | 2 ++ drivers/scsi/qla2xxx/qla_init.c | 3 ++ drivers/scsi/qla2xxx/qla_target.c | 1 + drivers/scsi/qla2xxx/qla_target.h | 1 + drivers/scsi/qla2xxx/tcm_qla2xxx.c | 57 +++++++++++++++++++++++--------------- 5 files changed, 42 insertions(+), 22 deletions(-)
drivers/scsi/qla2xxx/qla_def.h +2 -0
@@ -2493,6 +2493,8 @@ enum rscn_addr_format {int generation;struct se_session *se_sess; +struct list_head sess_cmd_list; +spinlock_t sess_cmd_lock;struct kref sess_kref;struct qla_tgt *tgt;unsigned long expires;
drivers/scsi/qla2xxx/qla_init.c +3 -0
@@ -4974,6 +4974,9 @@ void qla2x00_set_fcport_state(fc_port_t *fcport, int state)INIT_LIST_HEAD (&fcport->gnl_entry);INIT_LIST_HEAD (&fcport->list); +INIT_LIST_HEAD (&fcport->sess_cmd_list); +spin_lock_init (&fcport->sess_cmd_lock); +return fcport; }
drivers/scsi/qla2xxx/qla_target.c +1 -0
@@ -4292,6 +4292,7 @@ static struct qla_tgt_cmd *qlt_get_tag(scsi_qla_host_t *vha, cmd->cmd_type =TYPE_TGT_CMD ;memcpy (&cmd->atio, atio,sizeof (*atio)); +INIT_LIST_HEAD (&cmd->sess_cmd_list); cmd->state =QLA_TGT_STATE_NEW ; cmd->tgt = vha->vha_tgt.qla_tgt;qlt_incr_num_pend_cmds (vha);
drivers/scsi/qla2xxx/qla_target.h +1 -0
@@ -856,6 +856,7 @@ struct qla_tgt_cmd {uint8_t cmd_type;uint8_t pad[7 ];struct se_cmd se_cmd; +struct list_head sess_cmd_list;struct fc_port *sess;struct qla_qpair *qpair;uint32_t reset_count;
drivers/scsi/qla2xxx/tcm_qla2xxx.c +35 -22
107 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v3 6/8 target: Drop sess_cmd_lock from IO path
In reply to Mike ChristieTo remove the sess_cmd_lock this patch does:
- Removes the sess_cmd_list use from lio core, because it's been moved to qla2xxx.
- Removes sess_tearing_down check in the IO path. Instead of using that bit and the sess_cmd_lock, we rely on the cmd_count percpu ref. To do this we switch to percpu_ref_kill_and_confirm/percpu_ref_tryget_live.
--- drivers/infiniband/ulp/isert/ib_isert.c | 2 +- drivers/infiniband/ulp/srpt/ib_srpt.c | 2 +- drivers/scsi/qla2xxx/tcm_qla2xxx.c | 9 +--- drivers/target/target_core_tpg.c | 2 +- drivers/target/target_core_transport.c | 77 ++++++++++++++------------------- drivers/target/tcm_fc/tfc_sess.c | 2 +- include/target/target_core_base.h | 6 +-- include/target/target_core_fabric.h | 2 +- 8 files changed, 43 insertions(+), 59 deletions(-)
drivers/infiniband/ulp/isert/ib_isert.c +1 -1
@@ -2471,7 +2471,7 @@ static void isert_release_work(struct work_struct *work)isert_info ("iscsi_conn %p\n" , conn);if (conn->sess) { -target_sess_cmd_list_set_waiting (conn->sess->se_sess); +target_stop_session (conn->sess->se_sess);target_wait_for_sess_cmds (conn->sess->se_sess); } }
drivers/infiniband/ulp/srpt/ib_srpt.c +1 -1
@@ -2084,7 +2084,7 @@ static void srpt_release_channel_work(struct work_struct *w) se_sess = ch->sess;BUG_ON (!se_sess); -target_sess_cmd_list_set_waiting (se_sess); +target_stop_session (se_sess);target_wait_for_sess_cmds (se_sess);target_remove_session (se_sess);
drivers/scsi/qla2xxx/tcm_qla2xxx.c +2 -7
@@ -368,15 +368,10 @@ static void tcm_qla2xxx_put_sess(struct fc_port *sess)static void tcm_qla2xxx_close_session (struct se_session *se_sess) {struct fc_port *sess = se_sess->fabric_sess_ptr; -struct scsi_qla_host *vha; -unsigned long flags;BUG_ON (!sess); - vha = sess->vha; -spin_lock_irqsave (&vha->hw->tgt.sess_lock, flags); -target_sess_cmd_list_set_waiting (se_sess); -spin_unlock_irqrestore (&vha->hw->tgt.sess_lock, flags); +target_stop_session (se_sess); sess->explicit_logout =1 ;tcm_qla2xxx_put_sess (sess);
@@ -831,7 +826,7 @@ static void tcm_qla2xxx_clear_nacl_from_fcport_map(struct fc_port *sess)static void tcm_qla2xxx_shutdown_sess (struct fc_port *sess) { -target_sess_cmd_list_set_waiting (sess->se_sess); +target_stop_session (sess->se_sess); } static int tcm_qla2xxx_init_nodeacl(struct se_node_acl *se_nacl,
drivers/target/target_core_tpg.c +1 -1
@@ -328,7 +328,7 @@ static void target_shutdown_sessions(struct se_node_acl *acl) restart:spin_lock_irqsave (&acl->nacl_sess_lock, flags);list_for_each_entry (sess, &acl->acl_sess_list, sess_acl_list) { -if (sess->sess_tearing_down) +if (atomic_read (&sess->stopped))continue ;list_del_init (&sess->sess_acl_list);
drivers/target/target_core_transport.c +33 -44
151 lines not on this page · raw message
drivers/target/tcm_fc/tfc_sess.c +1 -1
@@ -275,7 +275,7 @@ static struct ft_sess *ft_sess_delete(struct ft_tport *tport, u32 port_id)static void ft_close_sess (struct ft_sess *sess) { -target_sess_cmd_list_set_waiting (sess->se_sess); +target_stop_session (sess->se_sess);target_wait_for_sess_cmds (sess->se_sess);ft_sess_put (sess); }
include/target/target_core_base.h +3 -3
@@ -608,7 +608,7 @@ static inline struct se_node_acl *fabric_stat_to_nacl(struct config_item *item) }struct se_session { -unsigned sess_tearing_down:1 ; +atomic_t stopped;u64 sess_bin_isid;enum target_prot_op sup_prot_ops;enum target_prot_type sess_prot_type;
@@ -618,9 +618,9 @@ struct se_session {struct percpu_ref cmd_count;struct list_head sess_list;struct list_head sess_acl_list; -struct list_head sess_cmd_list;spinlock_t sess_cmd_lock; -wait_queue_head_t cmd_list_wq; +wait_queue_head_t cmd_count_wq; +struct completion stop_done;void *sess_cmd_map;struct sbitmap_queue sess_tag_pool; };
include/target/target_core_fabric.h +1 -1
@@ -178,7 +178,7 @@ int transport_send_check_condition_and_sense(struct se_cmd *,int target_send_busy (struct se_cmd *cmd);int target_get_sess_cmd (struct se_cmd *,bool );int target_put_sess_cmd (struct se_cmd *); -void target_sess_cmd_list_set_waiting (struct se_session *); +void target_stop_session (struct se_session *se_sess);void target_wait_for_sess_cmds (struct se_session *);void target_show_cmd (const char *pfx,struct se_cmd *cmd);
Signature · 1 line
1.8.3.1
PATCH v3 7/8 target: make state_list per cpu
In reply to Mike ChristieDo a state_list/execute_task_lock per cpu, so we can do submissions from different CPUs without contention with each other.
Note: tcm_fc was passing TARGET_SCF_USE_CPUID, but never set cpuid. I think it wanted to set the cpuid to the CPU it was submitting from so it will get this behavior with this patch.
--- drivers/target/target_core_device.c | 16 +++- drivers/target/target_core_tmr.c | 166 +++++++++++++++++---------------- drivers/target/target_core_transport.c | 27 +++--- drivers/target/tcm_fc/tfc_cmd.c | 2 +- include/target/target_core_base.h | 13 ++- 5 files changed, 126 insertions(+), 98 deletions(-)
drivers/target/target_core_device.c +14 -2
@@ -721,11 +721,24 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) {struct se_device *dev;struct se_lun *xcopy_lun; +int i; dev = hba->backend->ops->alloc_device (hba, name);if (!dev)return NULL ; + dev->queues =kcalloc (nr_cpu_ids,sizeof (*dev->queues),GFP_KERNEL ); +if (!dev->queues) { + dev->transport->free_device (dev); +return NULL ; + } + + dev->queue_cnt = nr_cpu_ids; +for (i =0 ; i < dev->queue_cnt; i++) { +INIT_LIST_HEAD (&dev->queues[i].state_list); +spin_lock_init (&dev->queues[i].lock); + } + dev->se_hba = hba; dev->transport = hba->backend->ops; dev->transport_flags = dev->transport->transport_flags_default;
@@ -735,9 +748,7 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)INIT_LIST_HEAD (&dev->dev_sep_list);INIT_LIST_HEAD (&dev->dev_tmr_list);INIT_LIST_HEAD (&dev->delayed_cmd_list); -INIT_LIST_HEAD (&dev->state_list);INIT_LIST_HEAD (&dev->qf_cmd_list); -spin_lock_init (&dev->execute_task_lock);spin_lock_init (&dev->delayed_cmd_lock);spin_lock_init (&dev->dev_reservation_lock);spin_lock_init (&dev->se_port_lock);
@@ -1010,6 +1021,7 @@ void target_free_device(struct se_device *dev)if (dev->transport->free_prot) dev->transport->free_prot (dev); +kfree (dev->queues); dev->transport->free_device (dev); }
drivers/target/target_core_tmr.c +87 -79
186 lines not on this page · raw message
drivers/target/target_core_transport.c +13 -14
@@ -659,12 +659,12 @@ static void target_remove_from_state_list(struct se_cmd *cmd)if (!dev)return ; -spin_lock_irqsave (&dev->execute_task_lock, flags); +spin_lock_irqsave (&dev->queues[cmd->cpuid].lock, flags);if (cmd->state_active) {list_del (&cmd->state_list); cmd->state_active = false; } -spin_unlock_irqrestore (&dev->execute_task_lock, flags); +spin_unlock_irqrestore (&dev->queues[cmd->cpuid].lock, flags); }/*
@@ -875,10 +875,7 @@ void target_complete_cmd(struct se_cmd *cmd, u8 scsi_status)INIT_WORK (&cmd->work, success ? target_complete_ok_work : target_complete_failure_work); -if (cmd->se_cmd_flags &SCF_USE_CPUID ) -queue_work_on (cmd->cpuid, target_completion_wq, &cmd->work); -else -queue_work (target_completion_wq, &cmd->work); +queue_work_on (cmd->cpuid, target_completion_wq, &cmd->work); }EXPORT_SYMBOL (target_complete_cmd);
@@ -906,12 +903,13 @@ static void target_add_to_state_list(struct se_cmd *cmd)struct se_device *dev = cmd->se_dev;unsigned long flags; -spin_lock_irqsave (&dev->execute_task_lock, flags); +spin_lock_irqsave (&dev->queues[cmd->cpuid].lock, flags);if (!cmd->state_active) { -list_add_tail (&cmd->state_list, &dev->state_list); +list_add_tail (&cmd->state_list, + &dev->queues[cmd->cpuid].state_list); cmd->state_active = true; } -spin_unlock_irqrestore (&dev->execute_task_lock, flags); +spin_unlock_irqrestore (&dev->queues[cmd->cpuid].lock, flags); }/*
@@ -1399,6 +1397,9 @@ void transport_init_se_cmd( cmd->sense_buffer = sense_buffer; cmd->orig_fe_lun = unpacked_lun; +if (!(cmd->se_cmd_flags &SCF_USE_CPUID )) + cmd->cpuid =smp_processor_id (); + cmd->state_active = false; }EXPORT_SYMBOL (transport_init_se_cmd);
@@ -1616,6 +1617,9 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sessBUG_ON (!se_tpg);BUG_ON (se_cmd->se_tfo || se_cmd->se_sess);BUG_ON (in_interrupt ()); + +if (flags &TARGET_SCF_USE_CPUID ) + se_cmd->se_cmd_flags |=SCF_USE_CPUID ;/* * Initialize se_cmd for target operation. From this point * exceptions are handled by sending exception status via
@@ -1625,11 +1629,6 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sess data_length, data_dir, task_attr, sense, unpacked_lun); -if (flags &TARGET_SCF_USE_CPUID ) - se_cmd->se_cmd_flags |=SCF_USE_CPUID ; -else - se_cmd->cpuid =WORK_CPU_UNBOUND ; -if (flags &TARGET_SCF_UNKNOWN_SIZE ) se_cmd->unknown_data_length =1 ;/*
drivers/target/tcm_fc/tfc_cmd.c +1 -1
@@ -551,7 +551,7 @@ static void ft_send_work(struct work_struct *work)if (target_submit_cmd (&cmd->se_cmd, cmd->sess->se_sess, fcp->fc_cdb, &cmd->ft_sense_buffer[0 ],scsilun_to_int (&fcp->fc_lun),ntohl (fcp->fc_dl), task_attr, data_dir, -TARGET_SCF_ACK_KREF |TARGET_SCF_USE_CPUID )) +TARGET_SCF_ACK_KREF )) goto err;pr_debug ("r_ctl %x target_submit_cmd %p\n" , fh->fh_r_ctl, cmd);
include/target/target_core_base.h +11 -2
@@ -540,6 +540,10 @@ struct se_cmd {unsigned int t_prot_nents;sense_reason_t pi_err;sector_t bad_sector; +/* +* CPU LIO will execute the cmd on. Defaults to the CPU the cmd is +* initialized on. Drivers can override. +*/ int cpuid; };
@@ -760,6 +764,11 @@ struct se_dev_stat_grps {struct config_group scsi_lu_group; }; +struct se_device_queue { +struct list_head state_list; +spinlock_t lock; +}; +struct se_device {/* RELATIVE TARGET PORT IDENTIFER Counter */ u16 dev_rpti_counter;
@@ -792,7 +801,6 @@ struct se_device {atomic_t dev_qf_count;u32 export_count;spinlock_t delayed_cmd_lock; -spinlock_t execute_task_lock;spinlock_t dev_reservation_lock;unsigned int dev_reservation_flags;#define DRF_SPC2_RESERVATIONS 0x00000001
@@ -811,7 +819,6 @@ struct se_device {struct list_head dev_tmr_list;struct work_struct qf_work_queue;struct list_head delayed_cmd_list; -struct list_head state_list;struct list_head qf_cmd_list;/* Pointer to associated SE HBA */ struct se_hba *se_hba;
@@ -838,6 +845,8 @@ struct se_device {/* For se_lun->lun_se_dev RCU read-side critical access */ u32 hba_index;struct rcu_head rcu_head; +int queue_cnt; +struct se_device_queue *queues; };struct se_hba {
Signature · 1 line
1.8.3.1
PATCH v3 8/8 tcm loop: allow queues, can queue and cmd per lun to be settable
In reply to Mike ChristieMake can_queue, nr_hw_queues and cmd_per_lun settable by the user instead of hard coding them.
--- drivers/target/loopback/tcm_loop.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/target/loopback/tcm_loop.c +12 -2
@@ -46,6 +46,15 @@static int tcm_loop_queue_status (struct se_cmd *se_cmd); +static unsigned int tcm_loop_nr_hw_queues =1 ; +module_param_named (nr_hw_queues, tcm_loop_nr_hw_queues, uint,0644 ); + +static unsigned int tcm_loop_can_queue =1024 ; +module_param_named (can_queue, tcm_loop_can_queue, uint,0644 ); + +static unsigned int tcm_loop_cmd_per_lun =1024 ; +module_param_named (cmd_per_lun, tcm_loop_cmd_per_lun, uint,0644 ); +/* * Called from struct target_core_fabric_ops->check_stop_free() */
@@ -305,10 +314,8 @@ static int tcm_loop_target_reset(struct scsi_cmnd *sc) .eh_abort_handler = tcm_loop_abort_task, .eh_device_reset_handler = tcm_loop_device_reset, .eh_target_reset_handler = tcm_loop_target_reset, - .can_queue = 1024, .this_id = -1, .sg_tablesize = 256, - .cmd_per_lun = 1024, .max_sectors = 0xFFFF, .dma_boundary = PAGE_SIZE - 1, .module = THIS_MODULE,
@@ -342,6 +349,9 @@ static int tcm_loop_driver_probe(struct device *dev) sh->max_lun =0 ; sh->max_channel =0 ; sh->max_cmd_len =SCSI_MAX_VARLEN_CDB_SIZE ; + sh->nr_hw_queues = tcm_loop_nr_hw_queues; + sh->can_queue = tcm_loop_can_queue; + sh->cmd_per_lun = tcm_loop_cmd_per_lun; host_prot =SHOST_DIF_TYPE1_PROTECTION |SHOST_DIF_TYPE2_PROTECTION | SHOST_DIF_TYPE3_PROTECTION | SHOST_DIX_TYPE1_PROTECTION |
Signature · 1 line
1.8.3.1
Martin K. Petersen
AppliedMike,
Quoted 3 lines
The following patches made over Martin's staging branch fix some ref counting issues I hit while testing and improves the locking in the IO paths. To do the latter, the patches:
Applied to 5.11/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering