Fix a deadlock in the UFS error handler
cover Fix a deadlock in the UFS error handler
This patch series fixes a deadlock in the UFS error handler. I will repost this patch series after the merge window has closed.
Bart Van Assche (2):
scsi: core: Add support for reserved tags
scsi: ufs: Fix a deadlock in the error handler
drivers/scsi/scsi_lib.c | 4 +++- drivers/scsi/ufs/ufshcd.c | 20 ++++++-------------- include/scsi/scsi_host.h | 9 ++++++++- 3 files changed, 17 insertions(+), 16 deletions(-)
PATCH 1/2 scsi: core: Add support for reserved tags
Allow SCSI LLDs to allocate reserved tags by passing the BLK_MQ_REQ_RESERVED flag to blk_get_request().
--- drivers/scsi/scsi_lib.c | 4 +++- include/scsi/scsi_host.h | 9 ++++++++- 2 files changed, 11 insertions(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +3 -1
@@ -1917,6 +1917,7 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) {unsigned int cmd_size, sgl_size;struct blk_mq_tag_set *tag_set = &shost->tag_set; +unsigned int reserved_tags = shost->hostt->reserved_tags; sgl_size = max_t(unsigned int, sizeof(struct scatterlist),scsi_mq_inline_sgl_size (shost));
@@ -1932,7 +1933,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->ops = &scsi_mq_ops_no_commit; tag_set->nr_hw_queues = shost->nr_hw_queues ? :1 ; tag_set->nr_maps = shost->nr_maps ? :1 ; - tag_set->queue_depth = shost->can_queue; + tag_set->queue_depth = shost->can_queue + reserved_tags; + tag_set->reserved_tags = reserved_tags; tag_set->cmd_size = cmd_size; tag_set->numa_node =NUMA_NO_NODE ; tag_set->flags =BLK_MQ_F_SHOULD_MERGE ;
include/scsi/scsi_host.h +8 -1
@@ -367,10 +367,17 @@ struct scsi_host_template {/* * This determines if we will use a non-interrupt driven * or an interrupt driven scheme. It is set to the maximum number -* of simultaneous commands a single hw queue in HBA will accept. +* of simultaneous commands a single hw queue in HBA will accept. Does +* not include @reserved_tags. */ int can_queue; +/* +* Number of tags to reserve. A reserved tag can be allocated by passing +* the BLK_MQ_REQ_RESERVED flag to blk_get_request(). +*/ +unsigned reserved_tags; +/* * In many instances, especially where disconnect / reconnect are * supported, our host also has an ID on the SCSI bus. If this is
Re: [PATCH 1/2] scsi: core: Add support for reserved tags
Quoted 2 lines · Bart Van Assche
Allow SCSI LLDs to allocate reserved tags by passing the
BLK_MQ_REQ_RESERVEDflag toblk_get_request().
(replying to my own email)
Hannes and John, please help with reviewing this patch.
Thanks,
Bart.
Re: [PATCH 1/2] scsi: core: Add support for reserved tags
In reply to Bart Van AsscheQuoted 47 lines · Bart Van Assche
Allow SCSI LLDs to allocate reserved tags by passing the
BLK_MQ_REQ_RESERVEDflag toblk_get_request().Cc Hannes Reinecke <[email protected]>Cc John Garry <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- drivers/scsi/scsi_lib.c | 4 +++- include/scsi/scsi_host.h | 9 ++++++++- 2 files changed, 11 insertions(+), 2 deletions(-)drivers/scsi/scsi_lib.c +3
-1@@ -1917,6 +1917,7 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) {unsigned int cmd_size, sgl_size;struct blk_mq_tag_set *tag_set = &shost->tag_set; +unsigned int reserved_tags = shost->hostt->reserved_tags; sgl_size = max_t(unsigned int, sizeof(struct scatterlist),scsi_mq_inline_sgl_size (shost));@@ -1932,7 +1933,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->ops = &scsi_mq_ops_no_commit; tag_set->nr_hw_queues = shost->nr_hw_queues ? :1 ; tag_set->nr_maps = shost->nr_maps ? :1 ; - tag_set->queue_depth = shost->can_queue; + tag_set->queue_depth = shost->can_queue + reserved_tags; + tag_set->reserved_tags = reserved_tags; tag_set->cmd_size = cmd_size; tag_set->numa_node =NUMA_NO_NODE ; tag_set->flags =BLK_MQ_F_SHOULD_MERGE ;include/scsi/scsi_host.h +2
-1@@ -367,10 +367,17 @@ struct scsi_host_template {/* * This determines if we will use a non-interrupt driven * or an interrupt driven scheme. It is set to the maximum number -* of simultaneous commands a single hw queue in HBA will accept. +* of simultaneous commands a single hw queue in HBA will accept. Does +* not include @reserved_tags. */ int can_queue;
include/scsi/scsi_host.h struct scsi_host_template { · line 383
Quoted 10 lines
include/scsi/scsi_host.h +6
-0+/* +* Number of tags to reserve. A reserved tag can be allocated by passing +* the BLK_MQ_REQ_RESERVED flag to blk_get_request(). +*/ +unsigned reserved_tags; +/* * In many instances, especially where disconnect / reconnect are * supported, our host also has an ID on the SCSI bus. If this is
This is essentially the same patch as I posted a while ago (cf https://lore.kernel.org/linux-scsi/20210222132405.91369-1- [email protected]/).
But there had been push-back on that series as it would also try to use a scsi host device for driver-internal commands.
Maybe we should combine our efforts; patch 2 is equivalent to your patch, and 3-5 are a conversion of the fnic driver to use those commands. So we should merge our efforts to get this thing off the ground.
For the remainder of the patchset I'm currently working on a solution to address the upstream concerns.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
Re: [PATCH 1/2] scsi: core: Add support for reserved tags
Quoted 7 lines · Hannes Reinecke
This is essentially the same patch as I posted a while ago (cf https://lore.kernel.org/linux-scsi/20210222132405.91369-1- [email protected]/).
But there had been push-back on that series as it would also try to use a scsi host device for driver-internal commands.
Maybe we should combine our efforts; patch 2 is equivalent to your patch, and 3-5 are a conversion of the fnic driver to use those commands. So we should merge our efforts to get this thing off the ground.
For the remainder of the patchset I'm currently working on a solution to address the upstream concerns.
Hi Hannes,
In the UFS driver we are using a request queue that is not associated with any SCSI device for allocating driver-internal tags from the same tags space as SCSI commands. Is this a solution that is generic enough to be re-used by other SCSI drivers? See also the output of git grep -nH 'hba->cmd_queue'.
Thanks,
Bart.
Re: [PATCH 1/2] scsi: core: Add support for reserved tags
Quoted 5 lines · Bart Van Assche
Quoted 4 lines · Hannes Reinecke
This is essentially the same patch as I posted a while ago (cf https://lore.kernel.org/linux-scsi/20210222132405.91369-1- [email protected]/).
But there had been push-back on that series as it would also try to use a scsi host device for driver-internal commands.
Quoted 8 lines
Quoted 3 lines
Maybe we should combine our efforts; patch 2 is equivalent to your patch, and 3-5 are a conversion of the fnic driver to use those commands. So we should merge our efforts to get this thing off the ground.
For the remainder of the patchset I'm currently working on a solution to address the upstream concerns.
Hi Hannes,
In the UFS driver we are using a request queue that is not associated with any SCSI device for allocating driver-internal tags from the same tags space as SCSI commands. Is this a solution that is generic enough to be re-used by other SCSI drivers? See also the output of git grep -nH '
hba->cmd_queue'.
Ah. Even easier.
I've made a prototype for this kind of operation in git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git branch scsi-internal.v1
That introduces a function 'scsi_host_get_internal_tag()' to retrieve a tag from the reserved pool of the host tagset. Would that work for you?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
Re: [PATCH 1/2] scsi: core: Add support for reserved tags
Quoted 4 lines · Hannes Reinecke
Quoted 2 lines · Bart Van Assche
In the UFS driver we are using a request queue that is not associated with any SCSI device for allocating driver-internal tags from the same tags space as SCSI commands. Is this a solution that is generic enough to be re-used by other SCSI drivers? See also the output of git grep -nH '
hba->cmd_queue'.
Quoted 8 lines
Ah. Even easier.
I've made a prototype for this kind of operation in git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git branch scsi-internal.v1That introduces a function '
scsi_host_get_internal_tag()' to retrieve a tag from the reserved pool of the host tagset. Would that work for you?
That works for me. Thanks for the feedback!
Bart.
Re: [PATCH 1/2] scsi: core: Add support for reserved tags
In reply to Hannes ReineckeQuoted 8 lines · Hannes Reinecke
Ah. Even easier.
I've made a prototype for this kind of operation in git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git branch scsi-internal.v1That introduces a function '
scsi_host_get_internal_tag()' to retrieve a tag from the reserved pool of the host tagset. Would that work for you?
Hi Hannes,
Thanks for the pointer, that patch looks very interesting. However, in the UFS driver we need both a request pointer and a tag. Since converting a unique tag into a request pointer is much more work than converting a request or SCSI command pointer into a tag, I prefer functions that return a request pointer or SCSI command pointer.
Thanks,
Bart.
PATCH 2/2 scsi: ufs: Fix a deadlock in the error handler
In reply to Bart Van AsscheThe following deadlock has been observed on a test setup:
* All tags allocated.
* The SCSI error handler calls ufshcd_eh_host_reset_handler()
* ufshcd_eh_host_reset_handler() queues work that calls ufshcd_err_handler()
* ufshcd_err_handler() locks up as follows:
Workqueue: ufs_eh_wq_0 ufshcd_err_handler.cfi_jt Call trace: __switch_to+0x298/0x5d8 __schedule+0x6cc/0xa94 schedule+0x12c/0x298 blk_mq_get_tag+0x210/0x480 __blk_mq_alloc_request+0x1c8/0x284 blk_get_request+0x74/0x134 ufshcd_exec_dev_cmd+0x68/0x640 ufshcd_verify_dev_init+0x68/0x35c ufshcd_probe_hba+0x12c/0x1cb8 ufshcd_host_reset_and_restore+0x88/0x254 ufshcd_reset_and_restore+0xd0/0x354 ufshcd_err_handler+0x408/0xc58 process_one_work+0x24c/0x66c worker_thread+0x3e8/0xa4c kthread+0x150/0x1b4 ret_from_fork+0x10/0x30
Fix this lockup by making ufshcd_exec_dev_cmd() allocate a reserved request.
--- drivers/scsi/ufs/ufshcd.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-)
drivers/scsi/ufs/ufshcd.c +6 -14
@@ -2904,12 +2904,7 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,down_read (&hba->clk_scaling_lock); -/* -* Get free slot, sleep if slots are unavailable. -* Even though we use wait_event() which sleeps indefinitely, -* the maximum wait time is bounded by SCSI request timeout. -*/ - req =blk_get_request (q,REQ_OP_DRV_OUT ,0 ); + req =blk_mq_alloc_request (q,REQ_OP_DRV_OUT ,BLK_MQ_REQ_RESERVED );if (IS_ERR (req)) { err =PTR_ERR (req); goto out_unlock;
@@ -4919,11 +4914,7 @@ static int ufshcd_slave_alloc(struct scsi_device *sdev)*/ static int ufshcd_change_queue_depth (struct scsi_device *sdev,int depth) { -struct ufs_hba *hba =shost_priv (sdev->host); - -if (depth > hba->nutrs) - depth = hba->nutrs; -return scsi_change_queue_depth (sdev, depth); +return scsi_change_queue_depth (sdev,min (depth, sdev->host->can_queue)); }static void ufshcd_hpb_destroy (struct ufs_hba *hba,struct scsi_device *sdev)
@@ -8124,7 +8115,8 @@ static struct scsi_host_template ufshcd_driver_template = { .this_id = -1, .sg_tablesize = SG_ALL, .cmd_per_lun = UFSHCD_CMD_PER_LUN, - .can_queue = UFSHCD_CAN_QUEUE, + .can_queue = UFSHCD_CAN_QUEUE - 1, + .reserved_tags = 1, .max_segment_size = PRDT_DATA_BYTE_COUNT_MAX, .max_host_blocked = 1, .track_queue_depth = 1,
@@ -9485,8 +9477,8 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)/* Configure LRB */ ufshcd_host_memory_configure (hba); - host->can_queue = hba->nutrs; - host->cmd_per_lun = hba->nutrs; + host->can_queue = hba->nutrs -1 ; + host->cmd_per_lun = hba->nutrs -1 ; host->max_id =UFSHCD_MAX_ID ; host->max_lun =UFS_MAX_LUNS ; host->max_channel =UFSHCD_MAX_CHANNEL ;
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 25 lines · Bart Van Assche
The following deadlock has been observed on a test setup:
* All tags allocated.
* The SCSI error handler callsufshcd_eh_host_reset_handler()
*ufshcd_eh_host_reset_handler()queues work that callsufshcd_err_handler()
*ufshcd_err_handler()locks up as follows:Workqueue: ufs_eh_wq_0 ufshcd_err_handler.cfi_jt Call trace: __switch_to+0x298/0x5d8 __schedule+0x6cc/0xa94 schedule+0x12c/0x298 blk_mq_get_tag+0x210/0x480 __blk_mq_alloc_request+0x1c8/0x284 blk_get_request+0x74/0x134 ufshcd_exec_dev_cmd+0x68/0x640 ufshcd_verify_dev_init+0x68/0x35c ufshcd_probe_hba+0x12c/0x1cb8 ufshcd_host_reset_and_restore+0x88/0x254 ufshcd_reset_and_restore+0xd0/0x354 ufshcd_err_handler+0x408/0xc58 process_one_work+0x24c/0x66c worker_thread+0x3e8/0xa4c kthread+0x150/0x1b4 ret_from_fork+0x10/0x30
Quoted 2 lines
Fix this lockup by making
ufshcd_exec_dev_cmd()allocate a reserved request.
It is worth noting that the error handler itself could always find a free slot, either by waiting for one, or by taking the reset path which clears all slots.
However, the problem would then be places that cause the error handler to wait, like sysfs (due to hba->host_sem), exception event handler (due to cancel_work_sync(&hba->eeh_work)), or potentially any other dev cmd user (due to hba->dev_cmd.lock).
Once the layering and locking is sorted out, it might be possible to get rid of the reserved tag, if there was a performance benefit.
More to the point though, for the reasons above, you need to change the other dev cmd path also
i.e. ufshcd_issue_devman_upiu_cmd()
For UFS-specific patch sets please always cc me on all patches in a series including the cover letter.
Quoted 11 lines
Signed-off-by Bart Van Assche <[email protected]>--- drivers/scsi/ufs/ufshcd.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-)drivers/scsi/ufs/ufshcd.c +0
-0
drivers/scsi/ufs/ufshcd.c static int ufshcd_exec_dev_cmd(struct ufs_hba *hba, · line 2907
Quoted 10 lines
drivers/scsi/ufs/ufshcd.c +1
-6@@ -2904,12 +2904,7 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,down_read (&hba->clk_scaling_lock); -/* -* Get free slot, sleep if slots are unavailable. -* Even though we use wait_event() which sleeps indefinitely, -* the maximum wait time is bounded by SCSI request timeout. -*/ - req =blk_get_request (q,REQ_OP_DRV_OUT ,0 ); + req =blk_mq_alloc_request (q,REQ_OP_DRV_OUT ,BLK_MQ_REQ_RESERVED );
ufshcd_issue_devman_upiu_cmd() needs this also.
Quoted 17 lines
drivers/scsi/ufs/ufshcd.c +1
-5if (IS_ERR (req)) { err =PTR_ERR (req); goto out_unlock;@@ -4919,11 +4914,7 @@ static int ufshcd_slave_alloc(struct scsi_device *sdev)*/ static int ufshcd_change_queue_depth (struct scsi_device *sdev,int depth) { -struct ufs_hba *hba =shost_priv (sdev->host); - -if (depth > hba->nutrs) - depth = hba->nutrs; -return scsi_change_queue_depth (sdev, depth); +return scsi_change_queue_depth (sdev,min (depth, sdev->host->can_queue)); }static void ufshcd_hpb_destroy (struct ufs_hba *hba,struct scsi_device *sdev)
drivers/scsi/ufs/ufshcd.c static struct scsi_host_template ufshcd_driver_template = { · line 8119
Quoted 6 lines
drivers/scsi/ufs/ufshcd.c +2
-1@@ -8124,7 +8115,8 @@ static struct scsi_host_template ufshcd_driver_template = { .this_id = -1, .sg_tablesize = SG_ALL, .cmd_per_lun = UFSHCD_CMD_PER_LUN, - .can_queue = UFSHCD_CAN_QUEUE, + .can_queue = UFSHCD_CAN_QUEUE - 1, + .reserved_tags = 1,
Number of reserved tags needs to be #define'd
drivers/scsi/ufs/ufshcd.c int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq) · line 9481
Quoted 11 lines
drivers/scsi/ufs/ufshcd.c +2
-2.max_segment_size = PRDT_DATA_BYTE_COUNT_MAX, .max_host_blocked = 1, .track_queue_depth = 1,@@ -9485,8 +9477,8 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)/* Configure LRB */ ufshcd_host_memory_configure (hba); - host->can_queue = hba->nutrs; - host->cmd_per_lun = hba->nutrs; + host->can_queue = hba->nutrs -1 ; + host->cmd_per_lun = hba->nutrs -1 ;
Number of reserved tags needs to be #define'd
Quoted 4 lines
drivers/scsi/ufs/ufshcd.c +0
-0host->max_id =UFSHCD_MAX_ID ; host->max_lun =UFS_MAX_LUNS ; host->max_channel =UFSHCD_MAX_CHANNEL ;
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 26 lines · Adrian Hunter
Quoted 25 lines · Bart Van Assche
The following deadlock has been observed on a test setup:
* All tags allocated.
* The SCSI error handler callsufshcd_eh_host_reset_handler()
*ufshcd_eh_host_reset_handler()queues work that callsufshcd_err_handler()
*ufshcd_err_handler()locks up as follows:Workqueue: ufs_eh_wq_0 ufshcd_err_handler.cfi_jt Call trace: __switch_to+0x298/0x5d8 __schedule+0x6cc/0xa94 schedule+0x12c/0x298 blk_mq_get_tag+0x210/0x480 __blk_mq_alloc_request+0x1c8/0x284 blk_get_request+0x74/0x134 ufshcd_exec_dev_cmd+0x68/0x640 ufshcd_verify_dev_init+0x68/0x35c ufshcd_probe_hba+0x12c/0x1cb8 ufshcd_host_reset_and_restore+0x88/0x254 ufshcd_reset_and_restore+0xd0/0x354 ufshcd_err_handler+0x408/0xc58 process_one_work+0x24c/0x66c worker_thread+0x3e8/0xa4c kthread+0x150/0x1b4 ret_from_fork+0x10/0x30
Quoted 6 lines
Quoted 2 lines
Fix this lockup by making
ufshcd_exec_dev_cmd()allocate a reserved request.It is worth noting that the error handler itself could always find a free slot, either by waiting for one, or by taking the reset path which clears all slots.
I do not agree. As mentioned in the patch description, this patch is a fix for a scenario in which ufshcd_eh_host_reset_handler() waits until ufshcd_err_handler() finishes. ufshcd_err_handler() does not finish since there are no tags and no tags will be freed since that is the responsibility of ufshcd_eh_host_reset_handler() but it is blocked ...
Quoted 2 lines
For UFS-specific patch sets please always cc me on all patches in a series including the cover letter.
I will do that.
Thanks,
Bart.
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 27 lines · Bart Van Assche
Quoted 26 lines · Adrian Hunter
Quoted 25 lines · Bart Van Assche
The following deadlock has been observed on a test setup:
* All tags allocated.
* The SCSI error handler callsufshcd_eh_host_reset_handler()
*ufshcd_eh_host_reset_handler()queues work that callsufshcd_err_handler()
*ufshcd_err_handler()locks up as follows:Workqueue: ufs_eh_wq_0 ufshcd_err_handler.cfi_jt Call trace: __switch_to+0x298/0x5d8 __schedule+0x6cc/0xa94 schedule+0x12c/0x298 blk_mq_get_tag+0x210/0x480 __blk_mq_alloc_request+0x1c8/0x284 blk_get_request+0x74/0x134 ufshcd_exec_dev_cmd+0x68/0x640 ufshcd_verify_dev_init+0x68/0x35c ufshcd_probe_hba+0x12c/0x1cb8 ufshcd_host_reset_and_restore+0x88/0x254 ufshcd_reset_and_restore+0xd0/0x354 ufshcd_err_handler+0x408/0xc58 process_one_work+0x24c/0x66c worker_thread+0x3e8/0xa4c kthread+0x150/0x1b4 ret_from_fork+0x10/0x30
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Fix this lockup by making
ufshcd_exec_dev_cmd()allocate a reserved request.It is worth noting that the error handler itself could always find a free slot, either by waiting for one, or by taking the reset path which clears all slots.
I do not agree. As mentioned in the patch description, this patch is a fix for a scenario in which
ufshcd_eh_host_reset_handler()waits untilufshcd_err_handler()finishes.ufshcd_err_handler()does not finish since there are no tags and no tags will be freed since that is the responsibility ofufshcd_eh_host_reset_handler()but it is blocked ...
I am referring to the host controller slots, not block layer tags. The error handler does not need a free tag, it only needs a free slot.
RE: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
In reply to Adrian HunterQuoted 11 lines
Quoted 2 lines
Fix this lockup by making
ufshcd_exec_dev_cmd()allocate a reserved request.It is worth noting that the error handler itself could always find a free slot, either by waiting for one, or by taking the reset path which clears all slots.
However, the problem would then be places that cause the error handler to wait, like sysfs (due to
hba->host_sem), exception event handler (due tocancel_work_sync(&hba->eeh_work)), or potentially any other dev cmd user (due tohba->dev_cmd.lock).
Instead of the reserved tag mechanism, since we are in reset-and-restore path, how about forcing one slot out if the doorbell is full?
Quoted 8 lines
Once the layering and locking is sorted out, it might be possible to get rid of the reserved tag, if there was a performance benefit.
More to the point though, for the reasons above, you need to change the other dev cmd path also
i.e.ufshcd_issue_devman_upiu_cmd()
Maybe we can just return ebusy if ufshcd_eh_in_progress()?
Thanks,
Avri
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 2 lines · Avri Altman
Instead of the reserved tag mechanism, since we are in reset-and-restore path, how about forcing one slot out if the doorbell is full?
That would result in significantly more complex code. I prefer the reserved tag approach since it results in elegant code and since this approach does not affect performance significantly.
Thanks,
Bart.
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
In reply to Bart Van Asschedrivers/scsi/ufs/ufshcd.c static int ufshcd_exec_dev_cmd(struct ufs_hba *hba, · line 2907
Quoted 2 lines · Bart Van Assche
drivers/scsi/ufs/ufshcd.c +1
-1- req =blk_get_request (q,REQ_OP_DRV_OUT ,0 ); + req =blk_mq_alloc_request (q,REQ_OP_DRV_OUT ,BLK_MQ_REQ_RESERVED );
blk_get_request will be gone in 5.16-rc, so this won't apply.
But more importantly: SCSI LLDDs have absolutel no business calling blk_get_request or blk_mq_alloc_request directly, but as usual UFS is completely fucked up here.
Please add a SCSI midlayer helper to allocate the reserved tags, and switch _all_ of this UFS we're sending our own commands crap to it so it doesn't mix with the actual SCSI requests. We might or might not want a separate request_queue for them as well as non-SCSI requests really should not show up in ->queuecommand. Hannes and John have been looking into this for a while and we need to sort this out properly.
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 9 lines · Christoph Hellwig
drivers/scsi/ufs/ufshcd.c static int ufshcd_exec_dev_cmd(struct ufs_hba *hba, · line 2907
Quoted 2 lines · Bart Van Assche
drivers/scsi/ufs/ufshcd.c +1
-1- req =blk_get_request (q,REQ_OP_DRV_OUT ,0 ); + req =blk_mq_alloc_request (q,REQ_OP_DRV_OUT ,BLK_MQ_REQ_RESERVED );
blk_get_requestwill be gone in 5.16-rc, so this won't apply.But more importantly: SCSI LLDDs have absolutel no business calling
blk_get_requestorblk_mq_alloc_requestdirectly, but as usual UFS is completely fucked up here.
The UFS driver does not issue device commands to the block layer. blk_get_request() is used only to get a free slot.
Quoted 3 lines
Please add a SCSI midlayer helper to allocate the reserved tags, and switch _all_ of this UFS we're sending our own commands crap to it so it doesn't mix with the actual SCSI requests.
It doesn't mix them.
Quoted 3 lines
We might or might not
want a separaterequest_queuefor them as well as non-SCSI requests really should not show up in ->queuecommand.
Already has one (hba->cmd_queue), but as noted above, device commands aren't issued to it at the moment - they are sent directly.
Quoted 2 lines
Hannes and John have been
looking into this for a while and we need to sort this out properly.
To avoid involving SCSI, since device commands aren't issued to the block layer, the UFS driver could instead keep 1 slot for itself (i.e. reduce the number of tags by one), and drop blk_get_request(); noting that currently only 1 device command is sent at a time due to hba->dev_cmd.lock mutex.
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 2 lines · Adrian Hunter
The UFS driver does not issue device commands to the block layer.
blk_get_request()is used only to get a free slot.
That is indeed the case for the code touched here, but not in general:
ch@brick:~/work/linux$ git-grep blk_execute_rq drivers/scsi/ufs/ drivers/scsi/ufs/ufshcd.c: blk_execute_rq_nowait(/*bd_disk=*/NULL, req, /*at_head=*/true, drivers/scsi/ufs/ufshpb.c: blk_execute_rq_nowait(NULL, req, 1, ufshpb_umap_req_compl_fn); drivers/scsi/ufs/ufshpb.c: blk_execute_rq_nowait(NULL, req, 1, ufshpb_map_req_compl_fn);
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 10 lines · Christoph Hellwig
Quoted 2 lines · Adrian Hunter
The UFS driver does not issue device commands to the block layer.
blk_get_request()is used only to get a free slot.That is indeed the case for the code touched here, but not in general:
ch@brick:~/work/linux$ git-grep blk_execute_rq drivers/scsi/ufs/ drivers/scsi/ufs/ufshcd.c: blk_execute_rq_nowait(/*bd_disk=*/NULL, req, /*at_head=*/true, drivers/scsi/ufs/ufshpb.c: blk_execute_rq_nowait(NULL, req, 1, ufshpb_umap_req_compl_fn); drivers/scsi/ufs/ufshpb.c: blk_execute_rq_nowait(NULL, req, 1, ufshpb_map_req_compl_fn);
Hi Christoph,
The blk_execute_rq_nowait() call in drivers/scsi/ufs/ufshcd.c will disappear from Linus' master branch once Linus merges the v5.16 scsi core pull request. See also the following patch authored by myself: "[PATCH 2/2] scsi: ufs: Stop clearing unit attentions"
(https://lore.kernel.org/linux-scsi/[email protected]/).
Thanks,
Bart.
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
In reply to Christoph HellwigQuoted 5 lines · Christoph Hellwig
drivers/scsi/ufs/ufshcd.c static int ufshcd_exec_dev_cmd(struct ufs_hba *hba, · line 2907
Quoted 2 lines · Bart Van Assche
drivers/scsi/ufs/ufshcd.c +1
-1- req =blk_get_request (q,REQ_OP_DRV_OUT ,0 ); + req =blk_mq_alloc_request (q,REQ_OP_DRV_OUT ,BLK_MQ_REQ_RESERVED );
blk_get_requestwill be gone in 5.16-rc, so this won't apply.
Thanks for the reminder Christoph. This is something I am aware of. Hence the promise in the cover letter to rebase and repost this patch series after the merge window has closed.
Quoted 3 lines
But more importantly: SCSI LLDDs have absolutel no business calling
blk_get_requestorblk_mq_alloc_requestdirectly, but as usual UFS is completely fucked up here.
As explained by Adrian, the UFS protocol uses a single tag space for SCSI commands and UFS device commands. blk_mq_alloc_request() is used in this context to allocate a tag only from the shared tag space only. I think using blk_mq_alloc_request() for that purpose is fine.
Thanks,
Bart.
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 8 lines · Bart Van Assche
Quoted 3 lines
But more importantly: SCSI LLDDs have absolutel no business calling
blk_get_requestorblk_mq_alloc_requestdirectly, but as usual UFS is completely fucked up here.As explained by Adrian, the UFS protocol uses a single tag space for SCSI commands and UFS device commands.
blk_mq_alloc_request()is used in this context to allocate a tag only from the shared tag space only. I think usingblk_mq_alloc_request()for that purpose is fine.
The problem is not the shared tag space, the problem is that it is poking through layers. IFF we have good use cases for just allocating a tag (and we don't want to reserve it, which does make sense for many things like task management) we need a SCSI layer API that returns just the tag (and preferably also poisons the request in some way).
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Bart Van Assche
Quoted 3 lines
But more importantly: SCSI LLDDs have absolutel no business calling
blk_get_requestorblk_mq_alloc_requestdirectly, but as usual UFS is completely fucked up here.As explained by Adrian, the UFS protocol uses a single tag space for SCSI commands and UFS device commands.
blk_mq_alloc_request()is used in this context to allocate a tag only from the shared tag space only. I think usingblk_mq_alloc_request()for that purpose is fine.
Quoted 5 lines
The problem is not the shared tag space, the problem is that it is poking through layers. IFF we have good use cases for just allocating a tag (and we don't want to reserve it, which does make sense for many things like task management) we need a SCSI layer API that returns just the tag (and preferably also poisons the request in some way).
How about changing the blk_mq_alloc_request() / blk_put_request() calls in ufshcd_exec_dev_cmd() and __ufshcd_issue_tm_cmd() into scsi_{get,put}_internal_command() calls once Hannes' patch series that introduces these functions is upstream? See also "[PATCH 03/18] scsi: add scsi_{get,put}_internal_cmd() helper"
(https://lore.kernel.org/linux-scsi/[email protected]/).
See also patch "[PATCH 10/18] scsi: implement reserved command handling"
(https://lore.kernel.org/linux-scsi/[email protected]/).
Thanks,
Bart.
Re: [PATCH 2/2] scsi: ufs: Fix a deadlock in the error handler
Quoted 8 lines · Bart Van Assche
How about changing the
blk_mq_alloc_request()/blk_put_request()calls inufshcd_exec_dev_cmd()and__ufshcd_issue_tm_cmd()into scsi_{get,put}_internal_command() calls once Hannes' patch series that introduces these functions is upstream? See also "[PATCH 03/18] scsi: add scsi_{get,put}_internal_cmd() helper"
(https://lore.kernel.org/linux-scsi/[email protected]/).
See also patch "[PATCH 10/18] scsi: implement reserved command handling"
(https://lore.kernel.org/linux-scsi/[email protected]/).
Well, if they don't even need the command at all an API that just returns the tag might be even better. But yes, we need some SCSI layer abstraction and Hannes and John are currently looking into that.