UFS patches for kernel v5.17
Subsystems (2)
v4 cover UFS patches for kernel v5.17
Hi Martin,
This patch series includes the following changes:
- Fix a deadlock in the UFS error handler.
- Add polling support in the UFS driver.
- Several smaller fixes for the UFS driver.
Please consider these UFS driver kernel patches for kernel v5.17.
Thanks,
Bart.
Changes compared to v3:
- Dropped patch "scsi: core: Fix a race between scsi_done() and scsi_times_out()" since the conversation around that patch is still ongoing.
- Added patch "scsi: ufs: Remove hba->cmd_queue".
- Modified patch "scsi: ufs: Optimize the command queueing code".
Changes compared to v2:
- Dropped SCSI core patches that add support for internal commands.
- Reworked patch "Fix a deadlock in the error handler" such that it uses a reserved tag as proposed by Adrian.
- Split patch "ufs: Introduce ufshcd_release_scsi_cmd()" into two patches.
Changes compared to v1:
- Add internal command support to the SCSI core.
- Reworked patch "ufs: Optimize the command queueing code".
Bart Van Assche (17):
scsi: core: Fix scsi_device_max_queue_depth()
scsi: ufs: Rename a function argument
scsi: ufs: Remove is_rpmb_wlun()
scsi: ufs: Remove the sdev_rpmb member
scsi: ufs: Remove dead code
scsi: ufs: Fix race conditions related to driver data
scsi: ufs: Remove ufshcd_any_tag_in_use()
scsi: ufs: Rework ufshcd_change_queue_depth()
scsi: ufs: Fix a deadlock in the error handler
scsi: ufs: Remove hba->cmd_queue
scsi: ufs: Remove the 'update_scaling' local variable
scsi: ufs: Introduce ufshcd_release_scsi_cmd()
scsi: ufs: Improve SCSI abort handling further
scsi: ufs: Fix a kernel crash during shutdown
scsi: ufs: Stop using the clock scaling lock in the error handler
scsi: ufs: Optimize the command queueing code
scsi: ufs: Implement polling support
drivers/scsi/scsi.c | 4 +- drivers/scsi/ufs/tc-dwc-g210-pci.c | 1 - drivers/scsi/ufs/ufs-exynos.c | 4 +- drivers/scsi/ufs/ufshcd-pci.c | 2 - drivers/scsi/ufs/ufshcd-pltfrm.c | 2 - drivers/scsi/ufs/ufshcd.c | 300 ++++++++++++++++------------- drivers/scsi/ufs/ufshcd.h | 9 +- 7 files changed, 174 insertions(+), 148 deletions(-)
PATCH v4 1/17 scsi: core: Fix scsi_device_max_queue_depth()
The comment above scsi_device_max_queue_depth() and also the description of commit ca4453213951 ("scsi: core: Make sure sdev->queue_depth is <= max(shost->can_queue, 1024)") contradict the implementation of the function scsi_device_max_queue_depth(). Additionally, the maximum queue depth of a SCSI LUN never exceeds host->can_queue. Fix scsi_device_max_queue_depth() by changing max_t() into min_t().
--- drivers/scsi/scsi.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/scsi/scsi.c +2 -2
@@ -200,11 +200,11 @@ void scsi_finish_command(struct scsi_cmnd *cmd)/* -* 1024 is big enough for saturating the fast scsi LUN now +* 1024 is big enough for saturating fast SCSI LUNs. */ int scsi_device_max_queue_depth (struct scsi_device *sdev) { -return max_t (int, sdev->host->can_queue,1024 ); +return min_t (int, sdev->host->can_queue,1024 ); }/**
PATCH v4 2/17 scsi: ufs: Rename a function argument
In reply to Bart Van AsscheThe new name makes it clear what the meaning of the function argument is.
--- drivers/scsi/ufs/ufs-exynos.c | 4 ++-- drivers/scsi/ufs/ufshcd.h | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-)
drivers/scsi/ufs/ufs-exynos.c +2 -2
@@ -853,14 +853,14 @@ static int exynos_ufs_post_pwr_mode(struct ufs_hba *hba, }static void exynos_ufs_specify_nexus_t_xfer_req (struct ufs_hba *hba, -int tag,bool op) +int tag,bool is_scsi_cmd) {struct exynos_ufs *ufs =ufshcd_get_variant (hba);u32 type; type =hci_readl (ufs,HCI_UTRL_NEXUS_TYPE ); -if (op) +if (is_scsi_cmd)hci_writel (ufs, type | (1 << tag),HCI_UTRL_NEXUS_TYPE );else hci_writel (ufs, type & ~(1 << tag),HCI_UTRL_NEXUS_TYPE );
drivers/scsi/ufs/ufshcd.h +2 -1
@@ -338,7 +338,8 @@ struct ufs_hba_variant_ops {enum ufs_notify_change_status status, struct ufs_pa_layer_attr *, struct ufs_pa_layer_attr *); -void (*setup_xfer_req)(struct ufs_hba *,int ,bool ); +void (*setup_xfer_req)(struct ufs_hba *hba,int tag, +bool is_scsi_cmd);void (*setup_task_mgmt)(struct ufs_hba *,int ,u8 );void (*hibern8_notify)(struct ufs_hba *,enum uic_cmd_dme ,enum ufs_notify_change_status );
PATCH v4 3/17 scsi: ufs: Remove is_rpmb_wlun()
In reply to Bart Van AsscheCommit edc0596cc04b ("scsi: ufs: core: Stop clearing UNIT ATTENTIONS") removed all callers of is_rpmb_wlun(). Hence also remove the function itself.
--- drivers/scsi/ufs/ufshcd.c | 5 ----- 1 file changed, 5 deletions(-)
drivers/scsi/ufs/ufshcd.c +0 -5
@@ -2650,11 +2650,6 @@ static inline u16 ufshcd_upiu_wlun_to_scsi_wlun(u8 upiu_wlun_id)return (upiu_wlun_id & ~UFS_UPIU_WLUN_ID ) |SCSI_W_LUN_BASE ; } -static inline bool is_rpmb_wlun (struct scsi_device *sdev) -{ -return sdev->lun ==ufshcd_upiu_wlun_to_scsi_wlun (UFS_UPIU_RPMB_WLUN ); -} -static inline bool is_device_wlun (struct scsi_device *sdev) { return sdev->lun ==
PATCH v4 4/17 scsi: ufs: Remove the sdev_rpmb member
In reply to Bart Van AsscheSince the sdev_rpmb member of struct ufs_hba is only used inside ufshcd_scsi_add_wlus(), convert it into a local variable.
--- drivers/scsi/ufs/ufshcd.c | 12 ++++++------ drivers/scsi/ufs/ufshcd.h | 1 - 2 files changed, 6 insertions(+), 7 deletions(-)
drivers/scsi/ufs/ufshcd.c +6 -6
@@ -7412,7 +7412,7 @@ static inline void ufshcd_blk_pm_runtime_init(struct scsi_device *sdev)static int ufshcd_scsi_add_wlus (struct ufs_hba *hba) {int ret =0 ; -struct scsi_device *sdev_boot; +struct scsi_device *sdev_boot, *sdev_rpmb; hba->sdev_ufs_device =__scsi_add_device (hba->host,0 ,0 ,ufshcd_upiu_wlun_to_scsi_wlun (UFS_UPIU_UFS_DEVICE_WLUN ),NULL );
@@ -7423,14 +7423,14 @@ static int ufshcd_scsi_add_wlus(struct ufs_hba *hba) }scsi_device_put (hba->sdev_ufs_device); - hba->sdev_rpmb =__scsi_add_device (hba->host,0 ,0 , + sdev_rpmb =__scsi_add_device (hba->host,0 ,0 ,ufshcd_upiu_wlun_to_scsi_wlun (UFS_UPIU_RPMB_WLUN ),NULL ); -if (IS_ERR (hba->sdev_rpmb)) { - ret =PTR_ERR (hba->sdev_rpmb); +if (IS_ERR (sdev_rpmb)) { + ret =PTR_ERR (sdev_rpmb); goto remove_sdev_ufs_device; } -ufshcd_blk_pm_runtime_init (hba->sdev_rpmb); -scsi_device_put (hba->sdev_rpmb); +ufshcd_blk_pm_runtime_init (sdev_rpmb); +scsi_device_put (sdev_rpmb); sdev_boot =__scsi_add_device (hba->host,0 ,0 ,ufshcd_upiu_wlun_to_scsi_wlun (UFS_UPIU_BOOT_WLUN ),NULL );
drivers/scsi/ufs/ufshcd.h +0 -1
@@ -809,7 +809,6 @@ struct ufs_hba {* "UFS device" W-LU. */ struct scsi_device *sdev_ufs_device; -struct scsi_device *sdev_rpmb;#ifdef CONFIG_SCSI_UFS_HWMON struct device *hwmon_device;
PATCH v4 5/17 scsi: ufs: Remove dead code
In reply to Bart Van AsscheCommit 7252a3603015 ("scsi: ufs: Avoid busy-waiting by eliminating tag conflicts") guarantees that 'tag' is not in use by any SCSI command. Remove the check that returns early if a conflict occurs.
--- drivers/scsi/ufs/ufshcd.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
drivers/scsi/ufs/ufshcd.c +1 -6
@@ -6730,11 +6730,6 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba, tag = req->tag;WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag); -if (unlikely (test_bit (tag, &hba->outstanding_reqs))) { - err = -EBUSY ; - goto out; - } - lrbp = &hba->lrb[tag];WARN_ON (lrbp->cmd); lrbp->cmd =NULL ;
@@ -6802,8 +6797,8 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,ufshcd_add_query_upiu_trace (hba, err ?UFS_QUERY_ERR :UFS_QUERY_COMP , (struct utp_upiu_req *)lrbp->ucd_rsp_ptr); -out:blk_mq_free_request (req); + out_unlock:up_read (&hba->clk_scaling_lock);return err;
PATCH v4 6/17 scsi: ufs: Fix race conditions related to driver data
In reply to Bart Van AsscheThe driver data pointer must be set before any callbacks are registered that use that pointer. Hence move the initialization of that pointer from after the ufshcd_init() call to inside ufshcd_init().
--- drivers/scsi/ufs/tc-dwc-g210-pci.c | 1 - drivers/scsi/ufs/ufshcd-pci.c | 2 -- drivers/scsi/ufs/ufshcd-pltfrm.c | 2 -- drivers/scsi/ufs/ufshcd.c | 7 +++++++ 4 files changed, 7 insertions(+), 5 deletions(-)
drivers/scsi/ufs/tc-dwc-g210-pci.c +0 -1
@@ -110,7 +110,6 @@ tc_dwc_g210_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)return err; } -pci_set_drvdata (pdev, hba);pm_runtime_put_noidle (&pdev->dev);pm_runtime_allow (&pdev->dev);
drivers/scsi/ufs/ufshcd-pci.c +0 -2
@@ -522,8 +522,6 @@ ufshcd_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)return err; } -pci_set_drvdata (pdev, hba); - hba->vops = (struct ufs_hba_variant_ops *)id->driver_data; err =ufshcd_init (hba, mmio_base, pdev->irq);
drivers/scsi/ufs/ufshcd-pltfrm.c +0 -2
@@ -361,8 +361,6 @@ int ufshcd_pltfrm_init(struct platform_device *pdev, goto dealloc_host; } -platform_set_drvdata (pdev, hba); -pm_runtime_set_active (&pdev->dev);pm_runtime_enable (&pdev->dev);
drivers/scsi/ufs/ufshcd.c +7 -0
@@ -9481,6 +9481,13 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)struct device *dev = hba->dev;char eh_wq_name[sizeof ("ufs_eh_wq_00" )]; +/* +* dev_set_drvdata() must be called before any callbacks are registered +* that use dev_get_drvdata() (frequency scaling, clock scaling, hwmon, +* sysfs). +*/ +dev_set_drvdata (dev, hba); +if (!mmio_base) {dev_err (hba->dev,"Invalid memory reference for mmio_base is NULL\n" );
PATCH v4 7/17 scsi: ufs: Remove ufshcd_any_tag_in_use()
In reply to Bart Van AsscheUse hba->outstanding_reqs instead of ufshcd_any_tag_in_use(). This patch prepares for removal of the blk_mq_start_request() call from ufshcd_wait_for_dev_cmd(). blk_mq_tagset_busy_iter() only iterates over started requests.
--- drivers/scsi/ufs/ufshcd.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)
drivers/scsi/ufs/ufshcd.c +1 -20
@@ -1352,25 +1352,6 @@ static int ufshcd_devfreq_target(struct device *dev,return ret; } -static bool ufshcd_is_busy (struct request *req,void *priv,bool reserved) -{ -int *busy = priv; - -WARN_ON_ONCE (reserved); - (*busy)++; -return false; -} - -/* Whether or not any tag is in use by a request that is in progress. */ -static bool ufshcd_any_tag_in_use (struct ufs_hba *hba) -{ -struct request_queue *q = hba->cmd_queue; -int busy =0 ; - -blk_mq_tagset_busy_iter (q->tag_set, ufshcd_is_busy, &busy); -return busy; -} -static int ufshcd_devfreq_get_dev_status (struct device *dev,struct devfreq_dev_status *stat) {
@@ -1769,7 +1750,7 @@ static void ufshcd_gate_work(struct work_struct *work)if (hba->clk_gating.active_reqs || hba->ufshcd_state !=UFSHCD_STATE_OPERATIONAL - ||ufshcd_any_tag_in_use (hba) || hba->outstanding_tasks + || hba->outstanding_reqs || hba->outstanding_tasks || hba->active_uic_cmd || hba->uic_async_done) goto rel_lock;
PATCH v4 8/17 scsi: ufs: Rework ufshcd_change_queue_depth()
In reply to Bart Van AsschePrepare for making sdev->host->can_queue less than hba->nutrs. This patch does not change any functionality.
--- drivers/scsi/ufs/ufshcd.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
drivers/scsi/ufs/ufshcd.c +1 -5
@@ -4936,11 +4936,7 @@ static int ufshcd_slave_alloc(struct scsi_device *sdev)*/ static int ufshcd_change_queue_depth (struct scsi_device *sdev,int depth) { -struct ufs_hba *hba =shost_priv (sdev->host); - -if (depth > hba->nutrs) - depth = hba->nutrs; -return scsi_change_queue_depth (sdev, depth); +return scsi_change_queue_depth (sdev,min (depth, sdev->host->can_queue)); }static void ufshcd_hpb_destroy (struct ufs_hba *hba,struct scsi_device *sdev)
PATCH v4 9/17 scsi: ufs: Fix a deadlock in the error handler
In reply to Bart Van AsscheThe following deadlock has been observed on a test setup:
* All tags allocated.
* The SCSI error handler calls ufshcd_eh_host_reset_handler()
* ufshcd_eh_host_reset_handler() queues work that calls ufshcd_err_handler()
* ufshcd_err_handler() locks up as follows:
Workqueue: ufs_eh_wq_0 ufshcd_err_handler.cfi_jt Call trace: __switch_to+0x298/0x5d8 __schedule+0x6cc/0xa94 schedule+0x12c/0x298 blk_mq_get_tag+0x210/0x480 __blk_mq_alloc_request+0x1c8/0x284 blk_get_request+0x74/0x134 ufshcd_exec_dev_cmd+0x68/0x640 ufshcd_verify_dev_init+0x68/0x35c ufshcd_probe_hba+0x12c/0x1cb8 ufshcd_host_reset_and_restore+0x88/0x254 ufshcd_reset_and_restore+0xd0/0x354 ufshcd_err_handler+0x408/0xc58 process_one_work+0x24c/0x66c worker_thread+0x3e8/0xa4c kthread+0x150/0x1b4 ret_from_fork+0x10/0x30
Fix this lockup by making ufshcd_exec_dev_cmd() allocate a reserved request.
--- drivers/scsi/ufs/ufshcd.c | 53 +++++++++++---------------------------- drivers/scsi/ufs/ufshcd.h | 2 ++ 2 files changed, 16 insertions(+), 39 deletions(-)
drivers/scsi/ufs/ufshcd.c +14 -39
@@ -128,8 +128,9 @@ EXPORT_SYMBOL_GPL(ufshcd_dump_regs);enum {UFSHCD_MAX_CHANNEL =0 ,UFSHCD_MAX_ID =1 , -UFSHCD_CMD_PER_LUN =32 , -UFSHCD_CAN_QUEUE =32 , +UFSHCD_NUM_RESERVED =1 , +UFSHCD_CMD_PER_LUN =32 -UFSHCD_NUM_RESERVED , +UFSHCD_CAN_QUEUE =32 -UFSHCD_NUM_RESERVED , };static const char *const ufshcd_state_name[] = {
@@ -2170,6 +2171,7 @@ static inline int ufshcd_hba_capabilities(struct ufs_hba *hba) hba->nutrs = (hba->capabilities &MASK_TRANSFER_REQUESTS_SLOTS ) +1 ; hba->nutmrs = ((hba->capabilities &MASK_TASK_MANAGEMENT_REQUEST_SLOTS ) >>16 ) +1 ; + hba->reserved_slot = hba->nutrs -1 ;/* Read crypto capabilities */ err =ufshcd_hba_init_crypto_capabilities (hba);
@@ -2912,30 +2914,15 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,static int ufshcd_exec_dev_cmd (struct ufs_hba *hba,enum dev_cmd_type cmd_type,int timeout) { -struct request_queue *q = hba->cmd_queue;DECLARE_COMPLETION_ONSTACK (wait); -struct request *req; +const u32 tag = hba->reserved_slot;struct ufshcd_lrb *lrbp;int err; -int tag; -down_read (&hba->clk_scaling_lock); +/* Protects use of hba->reserved_slot. */ +lockdep_assert_held (&hba->dev_cmd.lock); -/* -* Get free slot, sleep if slots are unavailable. -* Even though we use wait_event() which sleeps indefinitely, -* the maximum wait time is bounded by SCSI request timeout. -*/ - req =blk_mq_alloc_request (q,REQ_OP_DRV_OUT ,0 ); -if (IS_ERR (req)) { - err =PTR_ERR (req); - goto out_unlock; - } - tag = req->tag; -WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag); -/* Set the timeout such that the SCSI error handler is not activated. */ - req->timeout =msecs_to_jiffies (2 * timeout); -blk_mq_start_request (req); +down_read (&hba->clk_scaling_lock); lrbp = &hba->lrb[tag];WARN_ON (lrbp->cmd);
@@ -2953,8 +2940,6 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba, (struct utp_upiu_req *)lrbp->ucd_rsp_ptr); out: -blk_mq_free_request (req); -out_unlock:up_read (&hba->clk_scaling_lock);return err; }
@@ -6689,23 +6674,16 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,enum dev_cmd_type cmd_type, enum query_opcode desc_op) { - struct request_queue *q = hba->cmd_queue;DECLARE_COMPLETION_ONSTACK (wait ); -struct request *req; +const u32 tag = hba->reserved_slot;struct ufshcd_lrb *lrbp;int err =0 ; -int tag;u8 upiu_flags; -down_read (&hba->clk_scaling_lock); +/* Protects use of hba->reserved_slot. */ +lockdep_assert_held (&hba->dev_cmd.lock); - req =blk_mq_alloc_request (q,REQ_OP_DRV_OUT ,0 ); -if (IS_ERR (req)) { - err =PTR_ERR (req); - goto out_unlock; - } - tag = req->tag; -WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag); +down_read (&hba->clk_scaling_lock); lrbp = &hba->lrb[tag];WARN_ON (lrbp->cmd);
@@ -6774,9 +6752,6 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,ufshcd_add_query_upiu_trace (hba, err ?UFS_QUERY_ERR :UFS_QUERY_COMP , (struct utp_upiu_req *)lrbp->ucd_rsp_ptr); -blk_mq_free_request (req); - -out_unlock:up_read (&hba->clk_scaling_lock);return err; }
@@ -9507,8 +9482,8 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)/* Configure LRB */ ufshcd_host_memory_configure (hba); - host->can_queue = hba->nutrs; - host->cmd_per_lun = hba->nutrs; + host->can_queue = hba->nutrs -UFSHCD_NUM_RESERVED ; + host->cmd_per_lun = hba->nutrs -UFSHCD_NUM_RESERVED ; host->max_id =UFSHCD_MAX_ID ; host->max_lun =UFS_MAX_LUNS ; host->max_channel =UFSHCD_MAX_CHANNEL ;
drivers/scsi/ufs/ufshcd.h +2 -0
@@ -745,6 +745,7 @@ struct ufs_hba_monitor {* @capabilities: UFS Controller Capabilities * @nutrs: Transfer Request Queue depth supported by controller * @nutmrs: Task Management Queue depth supported by controller +* @reserved_slot: Used to submit device commands. Protected by @dev_cmd.lock. * @ufs_version: UFS Version to which controller complies * @vops: pointer to variant specific operations * @priv: pointer to variant specific private data
@@ -836,6 +837,7 @@ struct ufs_hba {u32 capabilities;int nutrs;int nutmrs; +u32 reserved_slot;u32 ufs_version;const struct ufs_hba_variant_ops *vops;struct ufs_hba_variant_params *vps;
PATCH v4 10/17 scsi: ufs: Remove hba->cmd_queue
In reply to Bart Van AsscheThe previous patch removed all code that uses hba->cmd_queue. Hence also remove hba->cmd_queue itself.
--- drivers/scsi/ufs/ufshcd.c | 11 +---------- drivers/scsi/ufs/ufshcd.h | 2 -- 2 files changed, 1 insertion(+), 12 deletions(-)
drivers/scsi/ufs/ufshcd.c +1 -10
@@ -9331,7 +9331,6 @@ void ufshcd_remove(struct ufs_hba *hba)ufs_sysfs_remove_nodes (hba->dev);blk_cleanup_queue (hba->tmf_queue);blk_mq_free_tag_set (&hba->tmf_tag_set); -blk_cleanup_queue (hba->cmd_queue);scsi_remove_host (hba->host);/* disable interrupts */ ufshcd_disable_intr (hba, hba->intr_mask);
@@ -9551,12 +9550,6 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq) goto out_disable; } - hba->cmd_queue =blk_mq_init_queue (&hba->host->tag_set); -if (IS_ERR (hba->cmd_queue)) { - err =PTR_ERR (hba->cmd_queue); - goto out_remove_scsi_host; - } - hba->tmf_tag_set = (struct blk_mq_tag_set ) { .nr_hw_queues =1 , .queue_depth = hba->nutmrs,
@@ -9565,7 +9558,7 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq) }; err =blk_mq_alloc_tag_set (&hba->tmf_tag_set);if (err <0 ) - goto free_cmd_queue; + goto out_remove_scsi_host; hba->tmf_queue =blk_mq_init_queue (&hba->tmf_tag_set);if (IS_ERR (hba->tmf_queue)) { err =PTR_ERR (hba->tmf_queue);
@@ -9634,8 +9627,6 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)blk_cleanup_queue (hba->tmf_queue); free_tmf_tag_set:blk_mq_free_tag_set (&hba->tmf_tag_set); -free_cmd_queue: -blk_cleanup_queue (hba->cmd_queue); out_remove_scsi_host:scsi_remove_host (hba->host); out_disable:
drivers/scsi/ufs/ufshcd.h +0 -2
@@ -738,7 +738,6 @@ struct ufs_hba_monitor {* @host: Scsi_Host instance of the driver * @dev: device handle * @lrb: local reference block -* @cmd_queue: Used to allocate command tags from hba->host->tag_set. * @outstanding_tasks: Bits representing outstanding task requests * @outstanding_lock: Protects @outstanding_reqs. * @outstanding_reqs: Bits representing outstanding transfer requests
@@ -804,7 +803,6 @@ struct ufs_hba {struct Scsi_Host *host;struct device *dev; -struct request_queue *cmd_queue;/* * This field is to keep a reference to "scsi_device" corresponding to * "UFS device" W-LU.
PATCH v4 11/17 scsi: ufs: Remove the 'update_scaling' local variable
In reply to Bart Van AsscheThis patch does not change any functionality but makes the next patch in this series easier to read.
--- drivers/scsi/ufs/ufshcd.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
drivers/scsi/ufs/ufshcd.c +2 -5
@@ -5225,7 +5225,6 @@ static void __ufshcd_transfer_req_compl(struct ufs_hba *hba,struct scsi_cmnd *cmd;int result;int index; -bool update_scaling = false;for_each_set_bit (index, &completed_reqs, hba->nutrs) { lrbp = &hba->lrb[index];
@@ -5243,18 +5242,16 @@ static void __ufshcd_transfer_req_compl(struct ufs_hba *hba,/* Do not touch lrbp after scsi done */ scsi_done (cmd);ufshcd_release (hba); - update_scaling = true; +ufshcd_clk_scaling_update_busy (hba); }else if (lrbp->command_type ==UTP_CMD_TYPE_DEV_MANAGE || lrbp->command_type ==UTP_CMD_TYPE_UFS_STORAGE ) {if (hba->dev_cmd.complete) {ufshcd_add_command_trace (hba, index,UFS_DEV_COMP );complete (hba->dev_cmd.complete); - update_scaling = true; +ufshcd_clk_scaling_update_busy (hba); } } -if (update_scaling) -ufshcd_clk_scaling_update_busy (hba); } }
PATCH v4 12/17 scsi: ufs: Introduce ufshcd_release_scsi_cmd()
In reply to Bart Van AsscheThe only functional change in this patch is that scsi_done() is now called after ufshcd_release() and ufshcd_clk_scaling_update_busy() instead of before.
The next patch in this series will introduce a call to ufshcd_release_scsi_cmd() in the abort handler.
--- drivers/scsi/ufs/ufshcd.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/scsi/ufs/ufshcd.c +14 -8
@@ -5213,6 +5213,18 @@ static irqreturn_t ufshcd_uic_cmd_compl(struct ufs_hba *hba, u32 intr_status)return retval; } +/* Release the resources allocated for processing a SCSI command. */ +static void ufshcd_release_scsi_cmd (struct ufs_hba *hba, +struct ufshcd_lrb *lrbp) +{ +struct scsi_cmnd *cmd = lrbp->cmd; + +scsi_dma_unmap (cmd); + lrbp->cmd =NULL ;/* Mark the command as completed. */ +ufshcd_release (hba); +ufshcd_clk_scaling_update_busy (hba); +} +/** * __ufshcd_transfer_req_compl - handle SCSI and query command completion * @hba: per adapter instance
@@ -5223,7 +5235,6 @@ static void __ufshcd_transfer_req_compl(struct ufs_hba *hba, {struct ufshcd_lrb *lrbp;struct scsi_cmnd *cmd; -int result;int index;for_each_set_bit (index, &completed_reqs, hba->nutrs) {
@@ -5234,15 +5245,10 @@ static void __ufshcd_transfer_req_compl(struct ufs_hba *hba,if (unlikely (ufshcd_should_inform_monitor (hba, lrbp)))ufshcd_update_monitor (hba, lrbp);ufshcd_add_command_trace (hba, index,UFS_CMD_COMP ); - result =ufshcd_transfer_rsp_status (hba, lrbp); -scsi_dma_unmap (cmd); - cmd->result = result; -/* Mark completed command as NULL in LRB */ - lrbp->cmd =NULL ; + cmd->result =ufshcd_transfer_rsp_status (hba, lrbp); +ufshcd_release_scsi_cmd (hba, lrbp);/* Do not touch lrbp after scsi done */ scsi_done (cmd); -ufshcd_release (hba); -ufshcd_clk_scaling_update_busy (hba); }else if (lrbp->command_type ==UTP_CMD_TYPE_DEV_MANAGE || lrbp->command_type ==UTP_CMD_TYPE_UFS_STORAGE ) {if (hba->dev_cmd.complete) {
PATCH v4 13/17 scsi: ufs: Improve SCSI abort handling further
In reply to Bart Van AsscheRelease resources when aborting a command. Make sure that aborted commands are completed once by clearing the corresponding tag bit from hba->outstanding_reqs. This patch is an improved version of commit 3ff1f6b6ba6f ("scsi: ufs: core: Improve SCSI abort handling").
--- drivers/scsi/ufs/ufshcd.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/scsi/ufs/ufshcd.c +12 -0
@@ -6984,6 +6984,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)struct ufshcd_lrb *lrbp = &hba->lrb[tag];unsigned long flags;int err =FAILED ; +bool outstanding;u32 reg;WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag);
@@ -7061,6 +7062,17 @@ static int ufshcd_abort(struct scsi_cmnd *cmd) goto release; } +/* +* Clear the corresponding bit from outstanding_reqs since the command +* has been aborted successfully. +*/ +spin_lock_irqsave (&hba->outstanding_lock, flags); + outstanding =__test_and_clear_bit (tag, &hba->outstanding_reqs); +spin_unlock_irqrestore (&hba->outstanding_lock, flags); + +if (outstanding) +ufshcd_release_scsi_cmd (hba, lrbp); + err =SUCCESS ; release:
PATCH v4 14/17 scsi: ufs: Fix a kernel crash during shutdown
In reply to Bart Van AsscheFix the following kernel crash:
Unable to handle kernel paging request at virtual address ffffffc91e735000 Call trace: __queue_work+0x26c/0x624 queue_work_on+0x6c/0xf0 ufshcd_hold+0x12c/0x210 __ufshcd_wl_suspend+0xc0/0x400 ufshcd_wl_shutdown+0xb8/0xcc device_shutdown+0x184/0x224 kernel_restart+0x4c/0x124 __arm64_sys_reboot+0x194/0x264 el0_svc_common+0xc8/0x1d4 do_el0_svc+0x30/0x8c el0_svc+0x20/0x30 el0_sync_handler+0x84/0xe4 el0_sync+0x1bc/0x1c0
Fix this crash by ungating the clock before destroying the work queue on which clock gating work is queued.
--- drivers/scsi/ufs/ufshcd.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
drivers/scsi/ufs/ufshcd.c +10 -5
@@ -1648,7 +1648,8 @@ int ufshcd_hold(struct ufs_hba *hba, bool async)bool flush_result;unsigned long flags; -if (!ufshcd_is_clkgating_allowed (hba)) +if (!ufshcd_is_clkgating_allowed (hba) || + !hba->clk_gating.is_initialized) goto out;spin_lock_irqsave (hba->host->host_lock, flags); hba->clk_gating.active_reqs++;
@@ -1808,7 +1809,7 @@ static void __ufshcd_release(struct ufs_hba *hba)if (hba->clk_gating.active_reqs || hba->clk_gating.is_suspended || hba->ufshcd_state !=UFSHCD_STATE_OPERATIONAL || - hba->outstanding_tasks || + hba->outstanding_tasks || !hba->clk_gating.is_initialized || hba->active_uic_cmd || hba->uic_async_done || hba->clk_gating.state ==CLKS_OFF )return ;
@@ -1943,11 +1944,15 @@ static void ufshcd_exit_clk_gating(struct ufs_hba *hba) {if (!hba->clk_gating.is_initialized)return ; +ufshcd_remove_clk_gating_sysfs (hba); -cancel_work_sync (&hba->clk_gating.ungate_work); -cancel_delayed_work_sync (&hba->clk_gating.gate_work); -destroy_workqueue (hba->clk_gating.clk_gating_workq); + +/* Ungate the clock if necessary. */ +ufshcd_hold (hba, false); hba->clk_gating.is_initialized = false; +ufshcd_release (hba); + +destroy_workqueue (hba->clk_gating.clk_gating_workq); }/* Must be called with host lock acquired */
PATCH v4 15/17 scsi: ufs: Stop using the clock scaling lock in the error handler
In reply to Bart Van AsscheInstead of locking and unlocking the clock scaling lock, surround the command queueing code with an RCU reader lock and call synchronize_rcu(). This patch prepares for removal of the clock scaling lock.
--- drivers/scsi/ufs/ufshcd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +10 -2
@@ -2684,6 +2684,12 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)if (!down_read_trylock (&hba->clk_scaling_lock))return SCSI_MLQUEUE_HOST_BUSY ; +/* +* Allows the UFS error handler to wait for prior ufshcd_queuecommand() +* calls. +*/ +rcu_read_lock (); +switch (hba->ufshcd_state) {case UFSHCD_STATE_OPERATIONAL :break ;
@@ -2762,7 +2768,10 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) }ufshcd_send_command (hba, tag); + out: +rcu_read_unlock (); +up_read (&hba->clk_scaling_lock);if (ufs_trigger_eh ()) {
@@ -5951,8 +5960,7 @@ static void ufshcd_err_handling_prepare(struct ufs_hba *hba) }ufshcd_scsi_block_requests (hba);/* Drain ufshcd_queuecommand() */ -down_write (&hba->clk_scaling_lock); -up_write (&hba->clk_scaling_lock); +synchronize_rcu ();cancel_work_sync (&hba->eeh_work); }
PATCH v4 16/17 scsi: ufs: Optimize the command queueing code
In reply to Bart Van AsscheRemove the clock scaling lock from ufshcd_queuecommand() since it is a performance bottleneck. Instead check the SCSI device budget bitmaps in the code that waits for ongoing ufshcd_queuecommand() calls. A bit is set in sdev->budget_map just before scsi_queue_rq() is called and a bit is cleared from that bitmap if scsi_queue_rq() does not submit the request or after the request has finished. See also the blk_mq_{get,put}_dispatch_budget() calls in the block layer.
There is no risk for a livelock since the block layer delays queue reruns if queueing a request fails because the SCSI host has been blocked.
--- drivers/scsi/ufs/ufshcd.c | 33 +++++++++++++++++++++++---------- drivers/scsi/ufs/ufshcd.h | 1 + 2 files changed, 24 insertions(+), 10 deletions(-)
drivers/scsi/ufs/ufshcd.c +23 -10
@@ -1070,13 +1070,31 @@ static bool ufshcd_is_devfreq_scaling_required(struct ufs_hba *hba,return false; } +/* +* Determine the number of pending commands by counting the bits in the SCSI +* device budget maps. This approach has been selected because a bit is set in +* the budget map before scsi_host_queue_ready() checks the host_self_blocked +* flag. The host_self_blocked flag can be modified by calling +* scsi_block_requests() or scsi_unblock_requests(). +*/ +static u32 ufshcd_pending_cmds (struct ufs_hba *hba) +{ +struct scsi_device *sdev; +u32 pending =0 ; + +shost_for_each_device (sdev, hba->host) + pending +=sbitmap_weight (&sdev->budget_map); + +return pending; +} +static int ufshcd_wait_for_doorbell_clr (struct ufs_hba *hba,u64 wait_timeout_us) {unsigned long flags;int ret =0 ;u32 tm_doorbell; -u32 tr_doorbell; +u32 tr_pending;bool timeout = false, do_last_check = false;ktime_t start;
@@ -1094,8 +1112,8 @@ static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba, } tm_doorbell =ufshcd_readl (hba,REG_UTP_TASK_REQ_DOOR_BELL ); - tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); -if (!tm_doorbell && !tr_doorbell) { + tr_pending =ufshcd_pending_cmds (hba); +if (!tm_doorbell && !tr_pending) { timeout = false;break ; }else if (do_last_check) {
@@ -1115,12 +1133,12 @@ static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba, do_last_check = true; }spin_lock_irqsave (hba->host->host_lock, flags); - }while (tm_doorbell || tr_doorbell); + }while (tm_doorbell || tr_pending);if (timeout) {dev_err (hba->dev,"%s: timedout waiting for doorbell to clear (tm=0x%x, tr=0x%x)\n" , - __func__, tm_doorbell, tr_doorbell); + __func__, tm_doorbell, tr_pending); ret = -EBUSY ; } out:
@@ -2681,9 +2699,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag); -if (!down_read_trylock (&hba->clk_scaling_lock)) -return SCSI_MLQUEUE_HOST_BUSY ; -/* * Allows the UFS error handler to wait for prior ufshcd_queuecommand() * calls.
@@ -2772,8 +2787,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) out:rcu_read_unlock (); -up_read (&hba->clk_scaling_lock); -if (ufs_trigger_eh ()) {unsigned long flags;
drivers/scsi/ufs/ufshcd.h +1 -0
@@ -778,6 +778,7 @@ struct ufs_hba_monitor {* @clk_list_head: UFS host controller clocks list node head * @pwr_info: holds current power mode * @max_pwr_info: keeps the device max valid pwm +* @clk_scaling_lock: used to serialize device commands and clock scaling * @desc_size: descriptor sizes reported by device * @urgent_bkops_lvl: keeps track of urgent bkops level for device * @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
Re: [PATCH v4 16/17] scsi: ufs: Optimize the command queueing code
Quoted 8 lines · Bart Van Assche
Remove the clock scaling lock from
ufshcd_queuecommand()since it is a performance bottleneck. Instead check the SCSI device budget bitmaps in the code that waits for ongoingufshcd_queuecommand()calls. A bit is set insdev->budget_mapjust beforescsi_queue_rq()is called and a bit is cleared from that bitmap ifscsi_queue_rq()does not submit the request or after the request has finished. See also the blk_mq_{get,put}_dispatch_budget() calls in the block layer.
Quoted 7 lines
There is no risk for a livelock since the block layer delays queue reruns if queueing a request fails because the SCSI host has been blocked.
Cc Asutosh Das (asd) <[email protected]>Signed-off-by Bart Van Assche <[email protected]>---
Quoted 99 lines
drivers/scsi/ufs/ufshcd.c | 33 +++++++++++++++++++++++---------- drivers/scsi/ufs/ufshcd.h | 1 + 2 files changed, 24 insertions(+), 10 deletions(-)drivers/scsi/ufs/ufshcd.c +23
-10@@ -1070,13 +1070,31 @@ static bool ufshcd_is_devfreq_scaling_required(struct ufs_hba *hba,return false; } +/* +* Determine the number of pending commands by counting the bits in the SCSI +* device budget maps. This approach has been selected because a bit is set in +* the budget map before scsi_host_queue_ready() checks the host_self_blocked +* flag. The host_self_blocked flag can be modified by calling +* scsi_block_requests() or scsi_unblock_requests(). +*/ +static u32 ufshcd_pending_cmds (struct ufs_hba *hba) +{ +struct scsi_device *sdev; +u32 pending =0 ; + +shost_for_each_device (sdev, hba->host) + pending +=sbitmap_weight (&sdev->budget_map); + +return pending; +} +static int ufshcd_wait_for_doorbell_clr (struct ufs_hba *hba,u64 wait_timeout_us) {unsigned long flags;int ret =0 ;u32 tm_doorbell; -u32 tr_doorbell; +u32 tr_pending;bool timeout = false, do_last_check = false;ktime_t start;@@ -1094,8 +1112,8 @@ static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba, } tm_doorbell =ufshcd_readl (hba,REG_UTP_TASK_REQ_DOOR_BELL ); - tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); -if (!tm_doorbell && !tr_doorbell) { + tr_pending =ufshcd_pending_cmds (hba); +if (!tm_doorbell && !tr_pending) { timeout = false;break ; }else if (do_last_check) {@@ -1115,12 +1133,12 @@ static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba, do_last_check = true; }spin_lock_irqsave (hba->host->host_lock, flags); - }while (tm_doorbell || tr_doorbell); + }while (tm_doorbell || tr_pending);if (timeout) {dev_err (hba->dev,"%s: timedout waiting for doorbell to clear (tm=0x%x, tr=0x%x)\n" , - __func__, tm_doorbell, tr_doorbell); + __func__, tm_doorbell, tr_pending); ret = -EBUSY ; } out:@@ -2681,9 +2699,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag); -if (!down_read_trylock (&hba->clk_scaling_lock)) -return SCSI_MLQUEUE_HOST_BUSY ; -/* * Allows the UFS error handler to wait for prior ufshcd_queuecommand() * calls. @@ -2772,8 +2787,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) out:rcu_read_unlock (); -up_read (&hba->clk_scaling_lock); -if (ufs_trigger_eh ()) {unsigned long flags;drivers/scsi/ufs/ufshcd.h +1
-0@@ -778,6 +778,7 @@ struct ufs_hba_monitor {* @clk_list_head: UFS host controller clocks list node head * @pwr_info: holds current power mode * @max_pwr_info: keeps the device max valid pwm +* @clk_scaling_lock: used to serialize device commands and clock scaling * @desc_size: descriptor sizes reported by device * @urgent_bkops_lvl: keeps track of urgent bkops level for device * @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, Linux Foundation Collaborative Project
Re: [PATCH v4 16/17] scsi: ufs: Optimize the command queueing code
Quoted 13 lines · Asutosh Das (asd)
Quoted 12 lines · Bart Van Assche
Remove the clock scaling lock from
ufshcd_queuecommand()since it is a performance bottleneck. Instead check the SCSI device budget bitmaps in the code that waits for ongoingufshcd_queuecommand()calls. A bit is set insdev->budget_mapjust beforescsi_queue_rq()is called and a bit is cleared from that bitmap ifscsi_queue_rq()does not submit the request or after the request has finished. See also the blk_mq_{get,put}_dispatch_budget() calls in the block layer.There is no risk for a livelock since the block layer delays queue reruns if queueing a request fails because the SCSI host has been blocked.
Quoted 6 lines
Quoted 3 lines
Cc Asutosh Das (asd) <[email protected]>Signed-off-by Bart Van Assche <[email protected]>---
Reviewed-by Asutosh Das <[email protected]>
Replying to my own mail.
Hi Bart,
Quoted 22 lines
Quoted 22 lines
drivers/scsi/ufs/ufshcd.c| 33 +++++++++++++++++++++++----------drivers/scsi/ufs/ufshcd.h| 1 +2 files changed, 24 insertions(+), 10 deletions(-)drivers/scsi/ufs/ufshcd.c +2
-0@@ -1070,13 +1070,31 @@ static bool ufshcd_is_devfreq_scaling_required(struct ufs_hba *hba, return false; } +/* +* Determine the number of pending commands by counting the bits in the SCSI
+ * device budget maps. This approach has been selected because a bit is set in
+ * the budget map beforescsi_host_queue_ready()checks thehost_self_blocked
+ * flag. Thehost_self_blockedflag can be modified by calling
+ *scsi_block_requests()orscsi_unblock_requests().
+ */
Quoted 8 lines
drivers/scsi/ufs/ufshcd.c static bool ufshcd_is_devfreq_scaling_required(struct ufs_hba *hba, · line 1087
Quoted 8 lines
drivers/scsi/ufs/ufshcd.c +8
-0+static u32 ufshcd_pending_cmds (struct ufs_hba *hba) +{ + struct scsi_device *sdev; + u32 pending = 0; + + shost_for_each_device(sdev, hba->host) + pending += sbitmap_weight(&sdev->budget_map); +
I was porting this change to my downstream code and it occurred to me that in a high IO rate scenario it's possible that bits in the budget_map may be set even when that particular IO may not be issued to driver. So there would unnecessary waiting for that to be cleared. Do you think it's possible?
I think we should wait only for requests which are already started.
e.g. blk_mq_tagset_busy_iter() ?
PLMK your thoughts on this.
Quoted 72 lines
Quoted 70 lines
+ return pending;
drivers/scsi/ufs/ufshcd.c +2
-0+} +static int
ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba, u64 wait_timeout_us)
{
unsigned long flags;
int ret = 0;
u32tm_doorbell;
- u32tr_doorbell;
+ u32tr_pending;
bool timeout = false,do_last_check= false;ktime_tstart;drivers/scsi/ufs/ufshcd.c +4
-4@@ -1094,8 +1112,8 @@ static int ufshcd_wait_for_doorbell_clr(structufs_hba *hba, } tm_doorbell = ufshcd_readl(hba, REG_UTP_TASK_REQ_DOOR_BELL); - tr_doorbell = ufshcd_readl(hba, REG_UTP_TRANSFER_REQ_DOOR_BELL); - if (!tm_doorbell && !tr_doorbell) { + tr_pending = ufshcd_pending_cmds(hba); + if (!tm_doorbell && !tr_pending) { timeout = false; break; } else if (do_last_check) {@@ -1115,12 +1133,12 @@ static int ufshcd_wait_for_doorbell_clr(structufs_hba *hba, do_last_check = true; } spin_lock_irqsave(hba->host->host_lock, flags); - } while (tm_doorbell || tr_doorbell); + } while (tm_doorbell || tr_pending); if (timeout) { dev_err(hba->dev, "%s: timedout waiting for doorbell to clear (tm=0x%x, tr=0x%x)\n", - __func__, tm_doorbell, tr_doorbell); + __func__, tm_doorbell, tr_pending); ret = -EBUSY; }out:
drivers/scsi/ufs/ufshcd.c +0
-3@@ -2681,9 +2699,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) WARN_ONCE(tag < 0, "Invalid tag %d\n", tag); - if (!down_read_trylock(&hba->clk_scaling_lock)) - return SCSI_MLQUEUE_HOST_BUSY; - /** Allows the UFS error handler to wait for prior ufshcd_queuecommand() * calls. @@ -2772,8 +2787,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)out:
rcu_read_unlock();
-up_read(&hba->clk_scaling_lock);
-
if (ufs_trigger_eh()) {
unsigned long flags;drivers/scsi/ufs/ufshcd.h +1
-0@@ -778,6 +778,7 @@ struct ufs_hba_monitor { * @clk_list_head: UFS host controller clocks list node head * @pwr_info: holds current power mode * @max_pwr_info: keeps the device max valid pwm + * @clk_scaling_lock: used to serialize device commands and clockscaling
* @desc_size: descriptor sizes reported by device
* @urgent_bkops_lvl: keeps track of urgent bkops level for device
* @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, Linux Foundation Collaborative Project
Re: [PATCH v4 16/17] scsi: ufs: Optimize the command queueing code
Quoted 13 lines · Asutosh Das (asd)
Quoted 8 lines · Asutosh Das (asd)
drivers/scsi/ufs/ufshcd.c static bool ufshcd_is_devfreq_scaling_required(struct ufs_hba *hba, · line 1087
Quoted 8 lines
drivers/scsi/ufs/ufshcd.c +8
-0+static u32 ufshcd_pending_cmds (struct ufs_hba *hba) +{ + struct scsi_device *sdev; + u32 pending = 0; + + shost_for_each_device(sdev, hba->host) + pending += sbitmap_weight(&sdev->budget_map); +I was porting this change to my downstream code and it occurred to me that in a high IO rate scenario it's possible that bits in the
budget_mapmay be set even when that particular IO may not be issued to driver. So there would unnecessary waiting for that to be cleared. Do you think it's possible?
I think we should wait only for requests which are already started.
e.g.blk_mq_tagset_busy_iter()?
Hi Asutosh,
Using blk_mq_tagset_busy_iter() would be racy since the "busy" state is set after host_self_blocked has been checked.
Checking the budget_map should work fine since a bit in that bitmap is set just before scsi_queue_rq() is called and since the corresponding bit is cleared from that bitmap if scsi_queue_rq() fails or if a command is completed.
See also the output of the following command:
git grep -nHE 'blk_mq_(get|put)_dispatch_budget\('
See also the blk_mq_release_budgets() call in blk_mq_dispatch_rq_list().
Thanks,
Bart.
Re: [PATCH v4 16/17] scsi: ufs: Optimize the command queueing code
In reply to Bart Van AsscheQuoted 8 lines · Bart Van Assche
Remove the clock scaling lock from
ufshcd_queuecommand()since it is a performance bottleneck. Instead check the SCSI device budget bitmaps in the code that waits for ongoingufshcd_queuecommand()calls. A bit is set insdev->budget_mapjust beforescsi_queue_rq()is called and a bit is cleared from that bitmap ifscsi_queue_rq()does not submit the request or after the request has finished. See also the blk_mq_{get,put}_dispatch_budget() calls in the block layer.
Quoted 4 lines
There is no risk for a livelock since the block layer delays queue reruns if queueing a request fails because the SCSI host has been blocked.
This patch landed between next-20211203 and today's (20211210) linux-next/master and prevents all Qualcomm boards I've tested to boot.
Sometimes it locks up right around probe, sometimes I actually get some partitions, but attempts to then access the storage media (e.g. fdisk -l) results in one or more of my CPUs to be unresponsive.
Quoted 18 lines
Cc Asutosh Das (asd) <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- drivers/scsi/ufs/ufshcd.c | 33 +++++++++++++++++++++++---------- drivers/scsi/ufs/ufshcd.h | 1 + 2 files changed, 24 insertions(+), 10 deletions(-)drivers/scsi/ufs/ufshcd.c +3
-0@@ -1070,13 +1070,31 @@ static bool ufshcd_is_devfreq_scaling_required(struct ufs_hba *hba,return false; } +/* +* Determine the number of pending commands by counting the bits in the SCSI +* device budget maps. This approach has been selected because a bit is set in
drivers/scsi/ufs/ufshcd.c static bool ufshcd_is_devfreq_scaling_required(struct ufs_hba *hba, · line 1085
Quoted 10 lines
drivers/scsi/ufs/ufshcd.c +10
-0+* the budget map before scsi_host_queue_ready() checks the host_self_blocked +* flag. The host_self_blocked flag can be modified by calling +* scsi_block_requests() or scsi_unblock_requests(). +*/ +static u32 ufshcd_pending_cmds (struct ufs_hba *hba) +{ +struct scsi_device *sdev; +u32 pending =0 ; + +shost_for_each_device (sdev, hba->host)
As far as I can tell, this will crab walk across hba->host->__devices, while grabbing:
spin_lock_irqsave(shost->host_lock, flags);
which afaict is:
spin_lock_irqsave(hba->host->host_lock, flags);
drivers/scsi/ufs/ufshcd.c static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba,
Quoted 17 lines
drivers/scsi/ufs/ufshcd.c +6
-1+ pending +=sbitmap_weight (&sdev->budget_map); + +return pending; +} +static int ufshcd_wait_for_doorbell_clr (struct ufs_hba *hba,u64 wait_timeout_us) {unsigned long flags;int ret =0 ;u32 tm_doorbell; -u32 tr_doorbell; +u32 tr_pending;bool timeout = false, do_last_check = false;ktime_t start;@@ -1094,8 +1112,8 @@ static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba,
But just before entering this loop you do:
spin_lock_irqsave(hba->host->host_lock, flags);
In other words, you're taking the same spinlock twice, while being in ufshcd_scsi_block_requests(). To me this seems that if we ever enter this code path (i.e. try to do clock scaling) we will deadlock one CPU.
Can you please help me understand what I'm missing? Or how you tested this?
Thanks,
Bjorn
Quoted 57 lines
drivers/scsi/ufs/ufshcd.c +4
-9} tm_doorbell =ufshcd_readl (hba,REG_UTP_TASK_REQ_DOOR_BELL ); - tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); -if (!tm_doorbell && !tr_doorbell) { + tr_pending =ufshcd_pending_cmds (hba); +if (!tm_doorbell && !tr_pending) { timeout = false;break ; }else if (do_last_check) {@@ -1115,12 +1133,12 @@ static int ufshcd_wait_for_doorbell_clr(struct ufs_hba *hba, do_last_check = true; }spin_lock_irqsave (hba->host->host_lock, flags); - }while (tm_doorbell || tr_doorbell); + }while (tm_doorbell || tr_pending);if (timeout) {dev_err (hba->dev,"%s: timedout waiting for doorbell to clear (tm=0x%x, tr=0x%x)\n" , - __func__, tm_doorbell, tr_doorbell); + __func__, tm_doorbell, tr_pending); ret = -EBUSY ; } out:@@ -2681,9 +2699,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag); -if (!down_read_trylock (&hba->clk_scaling_lock)) -return SCSI_MLQUEUE_HOST_BUSY ; -/* * Allows the UFS error handler to wait for prior ufshcd_queuecommand() * calls. @@ -2772,8 +2787,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) out:rcu_read_unlock (); -up_read (&hba->clk_scaling_lock); -if (ufs_trigger_eh ()) {unsigned long flags;drivers/scsi/ufs/ufshcd.h +1
-0@@ -778,6 +778,7 @@ struct ufs_hba_monitor {* @clk_list_head: UFS host controller clocks list node head * @pwr_info: holds current power mode * @max_pwr_info: keeps the device max valid pwm +* @clk_scaling_lock: used to serialize device commands and clock scaling * @desc_size: descriptor sizes reported by device * @urgent_bkops_lvl: keeps track of urgent bkops level for device * @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
Re: [PATCH v4 16/17] scsi: ufs: Optimize the command queueing code
Quoted 2 lines · Bjorn Andersson
Can you please help me understand what I'm missing? Or how you tested this?
Hi Bjorn,
Unfortunately I don't have access to a test setup with a Qualcomm chipset. Please help verifying whether this patch is sufficient as a fix (see also https://lore.kernel.org/linux-scsi/[email protected]/):
drivers/scsi/ufs/ufshcd.c +3 -1
@@ -1084,7 +1084,9 @@ static u32 ufshcd_pending_cmds(struct ufs_hba *hba)struct scsi_device *sdev;u32 pending =0 ; -shost_for_each_device (sdev, hba->host) +lockdep_assert_held (hba->host->host_lock); + +__shost_for_each_device (sdev, hba->host) pending +=sbitmap_weight (&sdev->budget_map);return pending;
Thanks,
Bart.
Re: [PATCH v4 16/17] scsi: ufs: Optimize the command queueing code
Quoted 8 lines · Bart Van Assche
Quoted 2 lines · Bjorn Andersson
Can you please help me understand what I'm missing? Or how you tested this?
Hi Bjorn,
Unfortunately I don't have access to a test setup with a Qualcomm chipset.
Please help verifying whether this patch is sufficient as a fix (see also https://lore.kernel.org/linux-scsi/[email protected]/):
Thanks for the proposed fix!
drivers/scsi/ufs/ufshcd.c static u32 ufshcd_pending_cmds(struct ufs_hba *hba) · line 1089
Quoted 13 lines
drivers/scsi/ufs/ufshcd.c +3
-1@@ -1084,7 +1084,9 @@ static u32 ufshcd_pending_cmds(struct ufs_hba *hba)struct scsi_device *sdev;u32 pending =0 ; -shost_for_each_device (sdev, hba->host) +lockdep_assert_held (hba->host->host_lock); + +__shost_for_each_device (sdev, hba->host)
I can confirm that this resolves the issue I saw and allow me to boot my boards again. Can you please spin this in a patch?
Feel free to add my:
Thanks,
Bjorn
Quoted 7 lines
drivers/scsi/ufs/ufshcd.c +0
-0pending +=sbitmap_weight (&sdev->budget_map);return pending;Thanks,
Bart.
Re: [PATCH v4 16/17] scsi: ufs: Optimize the command queueing code
Quoted 2 lines · Bjorn Andersson
I can confirm that this resolves the issue I saw and allow me to boot my boards again. Can you please spin this in a patch?
Thanks for having tested this patch. Since Bean already posted this patch in the proper form, please take a look at
https://lore.kernel.org/linux-scsi/[email protected]/
Thanks,
Bart.
PATCH v4 17/17 scsi: ufs: Implement polling support
In reply to Bart Van AsscheThe time spent in io_schedule() and also the interrupt latency are significant when submitting direct I/O to a UFS device. Hence this patch that implements polling support. User space software can enable polling by passing the RWF_HIPRI flag to the preadv2() system call or the IORING_SETUP_IOPOLL flag to the io_uring interface.
Although the block layer supports to partition the tag space for interrupt-based completions (HCTX_TYPE_DEFAULT) purposes and polling (HCTX_TYPE_POLL), the choice has been made to use the same hardware queue for both hctx types because partitioning the tag space would negatively affect performance.
On my test setup this patch increases IOPS from 2736 to 22000 (8x) for the following test:
for hipri in 0 1; do
fio --ioengine=io_uring --iodepth=1 --rw=randread \
--runtime=60 --time_based=1 --direct=1 --name=qd1 \
--filename=/dev/block/sda --ioscheduler=none --gtod_reduce=1 \
--norandommap --hipri=$hipridone
--- drivers/scsi/ufs/ufshcd.c | 85 ++++++++++++++++++++++++++++++--------- 1 file changed, 67 insertions(+), 18 deletions(-)
drivers/scsi/ufs/ufshcd.c +67 -18
@@ -2662,6 +2662,36 @@ static inline bool is_device_wlun(struct scsi_device *sdev)ufshcd_upiu_wlun_to_scsi_wlun (UFS_UPIU_UFS_DEVICE_WLUN ); } +/* +* Associate the UFS controller queue with the default and poll HCTX types. +* Initialize the mq_map[] arrays. +*/ +static int ufshcd_map_queues (struct Scsi_Host *shost) +{ +int i, ret; + +for (i =0 ; i < shost->nr_maps; i++) { +struct blk_mq_queue_map *map = &shost->tag_set.map[i]; + +switch (i) { +case HCTX_TYPE_DEFAULT : +case HCTX_TYPE_POLL : + map->nr_queues =1 ; +break ; +case HCTX_TYPE_READ : + map->nr_queues =0 ; +break ; +default : +WARN_ON_ONCE (true); + } + map->queue_offset =0 ; + ret =blk_mq_map_queues (map); +WARN_ON_ONCE (ret); + } + +return 0 ; +} +static void ufshcd_init_lrb (struct ufs_hba *hba,struct ufshcd_lrb *lrb,int i) {struct utp_transfer_cmd_desc *cmd_descp = hba->ucdl_base_addr;
@@ -2697,7 +2727,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)struct ufshcd_lrb *lrbp;int err =0 ; -WARN_ONCE (tag <0 ,"Invalid tag %d\n" , tag); +WARN_ONCE (tag <0 || tag >= hba->nutrs,"Invalid tag %d\n" , tag);/* * Allows the UFS error handler to wait for prior ufshcd_queuecommand()
@@ -5288,6 +5318,31 @@ static void __ufshcd_transfer_req_compl(struct ufs_hba *hba, } } +/* +* Returns > 0 if one or more commands have been completed or 0 if no +* requests have been completed. +*/ +static int ufshcd_poll (struct Scsi_Host *shost,unsigned int queue_num) +{ +struct ufs_hba *hba =shost_priv (shost); +unsigned long completed_reqs, flags; +u32 tr_doorbell; + +spin_lock_irqsave (&hba->outstanding_lock, flags); + tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); + completed_reqs = ~tr_doorbell & hba->outstanding_reqs; +WARN_ONCE (completed_reqs & ~hba->outstanding_reqs, +"completed: %#lx; outstanding: %#lx\n" , completed_reqs, + hba->outstanding_reqs); + hba->outstanding_reqs &= ~completed_reqs; +spin_unlock_irqrestore (&hba->outstanding_lock, flags); + +if (completed_reqs) +__ufshcd_transfer_req_compl (hba, completed_reqs); + +return completed_reqs; +} +/** * ufshcd_transfer_req_compl - handle SCSI and query command completion * @hba: per adapter instance
@@ -5298,9 +5353,6 @@ static void __ufshcd_transfer_req_compl(struct ufs_hba *hba,*/ static irqreturn_t ufshcd_transfer_req_compl (struct ufs_hba *hba) { -unsigned long completed_reqs, flags; -u32 tr_doorbell; -/* Resetting interrupt aggregation counters first and reading the * DOOR_BELL afterward allows us to handle all the completed requests. * In order to prevent other interrupts starvation the DB is read once
@@ -5315,21 +5367,13 @@ static irqreturn_t ufshcd_transfer_req_compl(struct ufs_hba *hba)if (ufs_fail_completion ())return IRQ_HANDLED ; -spin_lock_irqsave (&hba->outstanding_lock, flags); - tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); - completed_reqs = ~tr_doorbell & hba->outstanding_reqs; -WARN_ONCE (completed_reqs & ~hba->outstanding_reqs, -"completed: %#lx; outstanding: %#lx\n" , completed_reqs, - hba->outstanding_reqs); - hba->outstanding_reqs &= ~completed_reqs; -spin_unlock_irqrestore (&hba->outstanding_lock, flags); +/* +* Ignore the ufshcd_poll() return value and return IRQ_HANDLED since we +* do not want polling to trigger spurious interrupt complaints. +*/ +ufshcd_poll (hba->host,0 ); -if (completed_reqs) { -__ufshcd_transfer_req_compl (hba, completed_reqs); -return IRQ_HANDLED ; - }else { -return IRQ_NONE ; - } +return IRQ_HANDLED ; }int __ufshcd_write_ee_control (struct ufs_hba *hba,u32 ee_ctrl_mask)
@@ -6581,6 +6625,8 @@ static int __ufshcd_issue_tm_cmd(struct ufs_hba *hba,spin_lock_irqsave (host->host_lock, flags); task_tag = req->tag; +WARN_ONCE (task_tag <0 || task_tag >= hba->nutmrs,"Invalid tag %d\n" , + task_tag); hba->tmf_rqs[req->tag] = req; treq->upiu_req.req_header.dword_0 |=cpu_to_be32 (task_tag);
@@ -8144,7 +8190,9 @@ static struct scsi_host_template ufshcd_driver_template = { .module = THIS_MODULE, .name = UFSHCD, .proc_name = UFSHCD, + .map_queues = ufshcd_map_queues, .queuecommand = ufshcd_queuecommand, + .mq_poll = ufshcd_poll, .slave_alloc = ufshcd_slave_alloc, .slave_configure = ufshcd_slave_configure, .slave_destroy = ufshcd_slave_destroy,
@@ -9432,6 +9480,7 @@ int ufshcd_alloc_host(struct device *dev, struct ufs_hba **hba_handle) err = -ENOMEM ; goto out_error; } + host->nr_maps =HCTX_TYPE_POLL +1 ; hba =shost_priv (host); hba->host = host; hba->dev = dev;
Martin K. Petersen
AppliedBart,
Quoted 4 lines
This patch series includes the following changes:
- Fix a deadlock in the UFS error handler.
- Add polling support in the UFS driver.
- Several smaller fixes for the UFS driver.
Applied to 5.17/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin K. Petersen
AppliedQuoted 8 lines · Bart Van Assche
This patch series includes the following changes:
- Fix a deadlock in the UFS error handler.
- Add polling support in the UFS driver.
- Several smaller fixes for the UFS driver.Please consider these UFS driver kernel patches for kernel v5.17.
[...]
Applied to 5.17/scsi-queue, thanks!
[01/17] scsi: core: Fix scsi_device_max_queue_depth()
https://git.kernel.org/mkp/scsi/c/4bc3bffc1a88 [02/17] scsi: ufs: Rename a function argument https://git.kernel.org/mkp/scsi/c/b427609e11ee [03/17] scsi: ufs: Remove is_rpmb_wlun() https://git.kernel.org/mkp/scsi/c/d656dc9b0b79 [04/17] scsi: ufs: Remove the sdev_rpmb member https://git.kernel.org/mkp/scsi/c/59830c095cf0 [05/17] scsi: ufs: Remove dead code https://git.kernel.org/mkp/scsi/c/d77ea8226b3b [06/17] scsi: ufs: Fix race conditions related to driver data https://git.kernel.org/mkp/scsi/c/21ad0e49085d [07/17] scsi: ufs: Remove ufshcd_any_tag_in_use() https://git.kernel.org/mkp/scsi/c/bd0b35383193 [08/17] scsi: ufs: Rework ufshcd_change_queue_depth() https://git.kernel.org/mkp/scsi/c/fc21da8a840a [09/17] scsi: ufs: Fix a deadlock in the error handler https://git.kernel.org/mkp/scsi/c/945c3cca05d7 [10/17] scsi: ufs: Remove hba->cmd_queue https://git.kernel.org/mkp/scsi/c/511a083b8b6b [11/17] scsi: ufs: Remove the 'update_scaling' local variable https://git.kernel.org/mkp/scsi/c/3eb9dcc027e2 [12/17] scsi: ufs: Introduce ufshcd_release_scsi_cmd() https://git.kernel.org/mkp/scsi/c/6f8dafdee6ae [13/17] scsi: ufs: Improve SCSI abort handling further https://git.kernel.org/mkp/scsi/c/1fbaa02dfd05 [14/17] scsi: ufs: Fix a kernel crash during shutdown https://git.kernel.org/mkp/scsi/c/3489c34bd02b [15/17] scsi: ufs: Stop using the clock scaling lock in the error handler https://git.kernel.org/mkp/scsi/c/5675c381ea51 [16/17] scsi: ufs: Optimize the command queueing code https://git.kernel.org/mkp/scsi/c/8d077ede48c1 [17/17] scsi: ufs: Implement polling support https://git.kernel.org/mkp/scsi/c/eaab9b573054
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin hi,
Quoted 7 lines · Bart Van Assche
Quoted 6 lines
This patch series includes the following changes:
- Fix a deadlock in the UFS error handler.
- Add polling support in the UFS driver.
- Several smaller fixes for the UFS driver.
Quoted 5 lines
Quoted 3 lines
Please consider these UFS driver kernel patches for kernel v5.17.
[...]
Applied to 5.17/scsi-queue, thanks!
16/17 is causing a deadlock - maybe you can wait for v5 that will fix it?
Thanks,
Avri
Avri,
Quoted 2 lines
16/17 is causing a deadlock - maybe you can wait for v5 that will fix it?
I'll just amend it.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering