Skip to contentSkip to conversation

Optimize the hot path in the UFS driver

PATCH Superseded Bart Van Assche 3 participants 59 messages Aug 11, 2025 – Aug 21, 2025

Subsystems (2)

v2 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Bart Van Assche

author reviewer
#0 lore raw

v2 cover Optimize the hot path in the UFS driver

Hi Martin,

This patch series optimizes the hot path of the UFS driver by making struct scsi_cmnd and struct ufshcd_lrb adjacent. Making these two data structures adjacent is realized as follows:

@@ -9040,6 +9046,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
.name = UFSHCD,
.proc_name = UFSHCD,
.map_queues = ufshcd_map_queues,
+ .cmd_size = sizeof(struct ufshcd_lrb),
.init_cmd_priv = ufshcd_init_cmd_priv,
.queuecommand = ufshcd_queuecommand,
.mq_poll = ufshcd_poll,

The following changes had to be made prior to making these two data structures adjacent:
* Add support for driver-internal and reserved commands in the SCSI core.
* Instead of making the reserved command slot (hba->reserved_slot) invisible to the SCSI core, let the SCSI core allocate a reserved command.
* Remove all UFS data structure members that are no longer needed because struct scsi_cmnd and struct ufshcd_lrb are now adjacent
* Call ufshcd_init_lrb() from inside ufshcd_queuecommand() instead of calling this function before I/O starts. This is necessary because ufshcd_memory_alloc() allocates fewer instances than the block layer allocates requests. See also the following code in the block layer core:

    if (blk_mq_init_request(set, hctx->fq->flush_rq, hctx_idx,
                hctx->numa_node))

Although the UFS driver could be modified such that ufshcd_init_lrb() is called from ufshcd_init_cmd_priv(), realizing this would require moving the memory allocations that happen from inside ufshcd_memory_alloc() into ufshcd_init_cmd_priv(). That would make this patch series even larger. Although ufshcd_init_lrb() is called for each command, the benefits of reduced indirection and better cache efficiency outweigh the small overhead of per-command lrb initialization.
* ufshcd_add_scsi_host() happens now before any device management commands are submitted. This change is necessary because this patch makes device management command allocation happen when the SCSI host is allocated.
* Start with allocating 32 command slots and increase this number later after it is clear whether or not the UFS device supports more than 32 command slots. Introduce scsi_host_update_can_queue() to support this approach.

Changes compared to v1:
- Left out the kernel patches related to support for const SCSI command arguments.
- Added SCSI core patches for allocating a pseudo SCSI device and reserved command support.
- Added several kernel patches to switch the UFS driver from a hardcoded reserved slot to calling scsi_get_internal_cmd().
- Enable .alloc_pseudo_sdev in the scsi_debug driver.

Bart Van Assche (27):
scsi: core: Do not allocate a budget token for reserved commands
scsi: core: Introduce scsi_host_update_can_queue()
scsi_debug: Set .alloc_pseudo_sdev
ufs: core: Move an assignment
ufs: core: Change the type of one ufshcd_add_cmd_upiu_trace() argument
ufs: core: Only call ufshcd_add_command_trace() for SCSI commands
ufs: core: Change the type of one ufshcd_add_command_trace() argument
ufs: core: Change the type of one ufshcd_send_command() argument
ufs: core: Only call ufshcd_should_inform_monitor() for SCSI commands
ufs: core: Change the monitor function argument types
ufs: core: Rework ufshcd_mcq_compl_pending_transfer()
ufs: core: Rework ufshcd_eh_device_reset_handler()
ufs: core: Cache the DMA buffer sizes
ufs: core: Add an argument to ufshcd_mcq_decide_queue_depth()
ufs: core: Add an argument to ufshcd_alloc_mcq()
ufs: core: Call ufshcd_mcq_init() once
ufs: core: Allocate the SCSI host earlier
ufs: core: Make ufshcd_mcq_init() independent of hba->nutrs
ufs: core: Call ufshcd_init_lrb() later
ufs: core: Use hba->reserved_slot
ufs: core: Make the reserved slot a reserved request
ufs: core: Do not clear driver-private command data
ufs: core: Optimize the hot path
ufs: core: Remove the ufshcd_lrb task_tag member
ufs: core: Initialize the 'hwq' variable earlier
ufs: core: Make blk_mq_tagset_busy_iter() skip reserved requests
ufs: core: Switch to scsi_get_internal_cmd()

Hannes Reinecke (3):
scsi: core: Support allocating reserved commands
scsi: core: Support allocating a pseudo SCSI device
scsi: core: Add scsi_{get,put}_internal_cmd() helpers

 drivers/scsi/hosts.c             |  11 +
 drivers/scsi/scsi.c              |  52 +-
 drivers/scsi/scsi_debug.c        |   1 +
 drivers/scsi/scsi_lib.c          |  97 +++-
 drivers/scsi/scsi_priv.h         |   2 +
 drivers/scsi/scsi_scan.c         |  70 ++-
 drivers/scsi/scsi_sysfs.c        |   4 +-
 drivers/ufs/core/ufs-mcq.c       |  50 +-
 drivers/ufs/core/ufshcd-crypto.h |  18 +-
 drivers/ufs/core/ufshcd-priv.h   |  20 +-
 drivers/ufs/core/ufshcd.c        | 845 ++++++++++++++++---------------
 include/scsi/scsi_cmnd.h         |   2 +
 include/scsi/scsi_device.h       |  23 +
 include/scsi/scsi_host.h         |  33 +-
 include/ufs/ufshcd.h             |  17 +-
 15 files changed, 783 insertions(+), 462 deletions(-)
author reviewer
#1 lore raw

PATCH v2 1/30 scsi: core: Support allocating reserved commands

From: Hannes Reinecke <[email protected]>

Quite some drivers are using management commands internally. These commands typically use the same tag pool as regular SCSI commands. Tags for these management commands are set aside before allocating the block-mq tag bitmap for regular SCSI commands. The block layer already supports this via the reserved tag mechanism. Add a new field 'nr_reserved_cmds' to the SCSI host template to instruct the block layer to set aside a tag space for these management commands by using reserved tags. Exclude reserved commands from .can_queue because .can_queue is visible in sysfs.

[ bvanassche: modified patch title and patch description. Left out the
following statement: "if (sdev->host->nr_reserved_cmds)
flags |= BLK_MQ_REQ_RESERVED;". See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]

---
 drivers/scsi/hosts.c     |  3 +++
 drivers/scsi/scsi_lib.c  |  3 ++-
 include/scsi/scsi_host.h | 22 +++++++++++++++++++++-
 3 files changed, 26 insertions(+), 2 deletions(-)
drivers/scsi/hosts.c +3 -0
index e021f1106bea..c2c6c96781e3 100644
@@ -495,6 +495,9 @@ struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int priv
 	if (sht->virt_boundary_mask)
 		shost->virt_boundary_mask = sht->virt_boundary_mask;
 
+	if (sht->nr_reserved_cmds)
+		shost->nr_reserved_cmds = sht->nr_reserved_cmds;
+
 	device_initialize(&shost->shost_gendev);
 	dev_set_name(&shost->shost_gendev, "host%d", shost->host_no);
 	shost->shost_gendev.bus = &scsi_bus_type;
drivers/scsi/scsi_lib.c +2 -1
index 0c65ecfedfbd..9c67e04265ce 100644
@@ -2083,7 +2083,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
 		tag_set->ops = &scsi_mq_ops_no_commit;
 	tag_set->nr_hw_queues = shost->nr_hw_queues ? : 1;
 	tag_set->nr_maps = shost->nr_maps ? : 1;
-	tag_set->queue_depth = shost->can_queue;
+	tag_set->queue_depth = shost->can_queue + shost->nr_reserved_cmds;
+	tag_set->reserved_tags = shost->nr_reserved_cmds;
 	tag_set->cmd_size = cmd_size;
 	tag_set->numa_node = dev_to_node(shost->dma_dev);
 	if (shost->hostt->tag_alloc_policy_rr)
include/scsi/scsi_host.h +21 -1
index c53812b9026f..722ecbee938e 100644
@@ -375,10 +375,19 @@ struct scsi_host_template {
 	/*
 	 * This determines if we will use a non-interrupt driven
 	 * or an interrupt driven scheme.  It is set to the maximum number
-	 * of simultaneous commands a single hw queue in HBA will accept.
+	 * of simultaneous commands a single hw queue in HBA will accept
+	 * excluding internal commands.
 	 */
 	int can_queue;
 
+	/*
+	 * This determines how many commands the HBA will set aside
+	 * for internal commands. This number will be added to
+	 * @can_queue to calcumate the maximum number of simultaneous
+	 * commands sent to the host.
+	 */
+	int nr_reserved_cmds;
+
 	/*
 	 * In many instances, especially where disconnect / reconnect are
 	 * supported, our host also has an ID on the SCSI bus.  If this is
@@ -611,6 +620,11 @@ struct Scsi_Host {
 	unsigned short max_cmd_len;
 
 	int this_id;
+
+	/*
+	 * Number of commands this host can handle at the same time.
+	 * This excludes reserved commands as specified by nr_reserved_cmds.
+	 */
 	int can_queue;
 	short cmd_per_lun;
 	short unsigned int sg_tablesize;
@@ -631,6 +645,12 @@ struct Scsi_Host {
 	 */
 	unsigned nr_hw_queues;
 	unsigned nr_maps;
+
+	/*
+	 * Number of reserved commands to allocate, if any.
+	 */
+	unsigned int nr_reserved_cmds;
+
 	unsigned active_mode:2;
 
 	/*
author reviewer
#2 lore raw

PATCH v2 2/30 scsi: core: Support allocating a pseudo SCSI device

In reply to Bart Van Assche

From: Hannes Reinecke <[email protected]>

Add the flag 'alloc_pseudo_sdev' to the SCSI host template and allocate a pseudo SCSI device if the flag is set. This device has the SCSI ID <max_id>:U64_MAX so it won't clash with any devices the LLD might create. It's excluded from scanning and will not show up in sysfs. Additionally, pseudo SCSI devices are skipped by shost_for_each_device(). This prevents that the SCSI error handler tries to submit a reset to a non-existent logical unit.

The intention is to use this device to send internal commands to the storage device.

[ bvanassche: edited patch description / renamed host_sdev into
pseudo_sdev / unexported scsi_get_host_dev() / modified error path in scsi_get_pseudo_dev() / skip pseudo devices in __scsi_iterate_devices() and also when calling sdev_init(), sdev_configure() and sdev_destroy(). See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]

---
 drivers/scsi/hosts.c       |  8 +++++
 drivers/scsi/scsi.c        |  7 ++--
 drivers/scsi/scsi_priv.h   |  2 ++
 drivers/scsi/scsi_scan.c   | 70 ++++++++++++++++++++++++++++++++++++--
 drivers/scsi/scsi_sysfs.c  |  4 ++-
 include/scsi/scsi_device.h | 16 +++++++++
 include/scsi/scsi_host.h   |  9 +++++
 7 files changed, 110 insertions(+), 6 deletions(-)
drivers/scsi/hosts.c +8 -0
index c2c6c96781e3..8632bf61160b 100644
@@ -307,6 +307,14 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
 	if (error)
 		goto out_del_dev;
 
+	if (sht->alloc_pseudo_sdev) {
+		shost->pseudo_sdev = scsi_get_pseudo_dev(shost);
+		if (!shost->pseudo_sdev) {
+			error = -ENOMEM;
+			goto out_del_dev;
+		}
+	}
+
 	scsi_proc_host_add(shost);
 	scsi_autopm_put_host(shost);
 	return error;
drivers/scsi/scsi.c +5 -2
index 9a0f467264b3..a290c3aa7b88 100644
@@ -826,8 +826,11 @@ struct scsi_device *__scsi_iterate_devices(struct Scsi_Host *shost,
 	spin_lock_irqsave(shost->host_lock, flags);
 	while (list->next != &shost->__devices) {
 		next = list_entry(list->next, struct scsi_device, siblings);
-		/* skip devices that we can't get a reference to */
-		if (!scsi_device_get(next))
+		/*
+		 * Skip pseudo devices and also devices for which
+		 * scsi_device_get() fails.
+		 */
+		if (!scsi_device_is_pseudo_dev(next) && !scsi_device_get(next))
 			break;
 		next = NULL;
 		list = list->next;
drivers/scsi/scsi_priv.h +2 -0
index 5b2b19f5e8ec..da3bc87ac5a6 100644
@@ -135,6 +135,8 @@ extern int scsi_complete_async_scans(void);
 extern int scsi_scan_host_selected(struct Scsi_Host *, unsigned int,
 				   unsigned int, u64, enum scsi_scan_mode);
 extern void scsi_forget_host(struct Scsi_Host *);
+struct scsi_device *scsi_get_pseudo_dev(struct Scsi_Host *);
+bool scsi_device_is_pseudo_dev(struct scsi_device *sdev);
 
 /* scsi_sysctl.c */
 #ifdef CONFIG_SYSCTL
drivers/scsi/scsi_scan.c +67 -3

104 lines not on this page · raw message

drivers/scsi/scsi_sysfs.c +3 -1
index e6464b998960..fa510455886e 100644
@@ -1406,6 +1406,8 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev)
 	int error;
 	struct scsi_target *starget = sdev->sdev_target;
 
+	WARN_ON_ONCE(scsi_device_is_pseudo_dev(sdev));
+
 	error = scsi_target_add(starget);
 	if (error)
 		return error;
@@ -1513,7 +1515,7 @@ void __scsi_remove_device(struct scsi_device *sdev)
 	kref_put(&sdev->host->tagset_refcnt, scsi_mq_free_tags);
 	cancel_work_sync(&sdev->requeue_work);
 
-	if (sdev->host->hostt->sdev_destroy)
+	if (!scsi_device_is_pseudo_dev(sdev) && sdev->host->hostt->sdev_destroy)
 		sdev->host->hostt->sdev_destroy(sdev);
 	transport_destroy_device(dev);
 
include/scsi/scsi_device.h +16 -0
index 6d6500148c4b..3846f5dfc51c 100644
@@ -589,6 +589,22 @@ static inline unsigned int sdev_id(struct scsi_device *sdev)
 #define scmd_id(scmd) sdev_id((scmd)->device)
 #define scmd_channel(scmd) sdev_channel((scmd)->device)
 
+/**
+ * scsi_device_is_pseudo_dev() - Whether a device is a pseudo SCSI device.
+ * @sdev: SCSI device to examine
+ *
+ * A pseudo SCSI device can be used to allocate SCSI commands but does not show
+ * up in sysfs. Additionally, the logical unit information in *@sdev is made up.
+ *
+ * This function tests the LUN number instead of comparing @sdev with
+ * @sdev->host->pseudo_sdev because this function may be called before
+ * @sdev->host->pseudo_sdev has been initialized.
+ */
+static inline bool scsi_device_is_pseudo_dev(struct scsi_device *sdev)
+{
+	return sdev->lun == U64_MAX;
+}
+
 /*
  * checks for positions of the SCSI state machine
  */
include/scsi/scsi_host.h +9 -0
index 722ecbee938e..3b5150759c44 100644
@@ -463,6 +463,9 @@ struct scsi_host_template {
 	 */
 	unsigned emulated:1;
 
+	/* True if a pseudo sdev should be allocated */
+	unsigned alloc_pseudo_sdev:1;
+
 	/*
 	 * True if the low-level driver performs its own reset-settle delays.
 	 */
@@ -722,6 +725,12 @@ struct Scsi_Host {
 	/* ldm bits */
 	struct device		shost_gendev, shost_dev;
 
+	/*
+	 * A SCSI device structure used for sending internal commands to the
+	 * HBA. There is no corresponding logical unit inside the SCSI device.
+	 */
+	struct scsi_device *pseudo_sdev;
+
 	/*
 	 * Points to the transport data (if any) which is allocated
 	 * separately
author reviewer
#3 lore raw

PATCH v2 3/30 scsi: core: Do not allocate a budget token for reserved commands

In reply to Bart Van Assche

The SCSI budget mechanism is used to implement the host->cmd_per_lun limit. This limit does not apply to reserved commands. Hence, do not allocate a budget token for reserved commands.

Cc Hannes Reinecke <[email protected]>
Cc John Garry <[email protected]>
Cc Ming Lei <[email protected]>
---
 drivers/scsi/scsi_lib.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +12 -2
index 9c67e04265ce..0112ad3859ff 100644
@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)
 	if (starget->can_queue > 0)
 		atomic_dec(&starget->target_busy);
 
-	sbitmap_put(&sdev->budget_map, cmd->budget_token);
+	if (cmd->budget_token < INT_MAX)
+		sbitmap_put(&sdev->budget_map, cmd->budget_token);
 	cmd->budget_token = -1;
 }
 
@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct request_queue *q,
 {
 	int token;
 
+	/*
+	 * Do not allocate a budget token for reserved SCSI commands. Budget
+	 * tokens are used to enforce the cmd_per_lun limit. That limit does not
+	 * apply to reserved commands.
+	 */
+	if (scsi_device_is_pseudo_dev(sdev))
+		return INT_MAX;
+
 	token = sbitmap_get(&sdev->budget_map);
 	if (token < 0)
 		return -1;
@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token)
 {
 	struct scsi_device *sdev = q->queuedata;
 
-	sbitmap_put(&sdev->budget_map, budget_token);
+	if (budget_token < INT_MAX)
+		sbitmap_put(&sdev->budget_map, budget_token);
 }
 
 /*
#4 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 42 lines · Bart Van Assche

The SCSI budget mechanism is used to implement the host->cmd_per_lun limit. This limit does not apply to reserved commands. Hence, do not allocate a budget token for reserved commands.

Cc Hannes Reinecke <[email protected]>
Cc John Garry <[email protected]>
Cc Ming Lei <[email protected]>
---
  drivers/scsi/scsi_lib.c | 14 ++++++++++++--
  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +10 -1
index 9c67e04265ce..0112ad3859ff 100644
@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)
  	if (starget->can_queue > 0)
  		atomic_dec(&starget->target_busy);
 
-	sbitmap_put(&sdev->budget_map, cmd->budget_token);
+	if (cmd->budget_token < INT_MAX)
+		sbitmap_put(&sdev->budget_map, cmd->budget_token);
  	cmd->budget_token = -1;
  }
 
@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct request_queue *q,
  {
  	int token;
 
+	/*
+	 * Do not allocate a budget token for reserved SCSI commands. Budget
+	 * tokens are used to enforce the cmd_per_lun limit. That limit does not
+	 * apply to reserved commands.
+	 */
+	if (scsi_device_is_pseudo_dev(sdev))
+		return INT_MAX;
+
  	token = sbitmap_get(&sdev->budget_map);
  	if (token < 0)
  		return -1;

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q, int budget_token) · line 1765

Quoted 9 lines
drivers/scsi/scsi_lib.c +2 -1
@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token)
  {
  	struct scsi_device *sdev = q->queuedata;
 
-	sbitmap_put(&sdev->budget_map, budget_token);
+	if (budget_token < INT_MAX)
+		sbitmap_put(&sdev->budget_map, budget_token);
  }
 
  /*

Good idea, but we should document that somewhere that 'INT_MAX' now has a distinct meaning.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#5 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 46 lines · Hannes Reinecke
Quoted 45 lines · Bart Van Assche

The SCSI budget mechanism is used to implement the host->cmd_per_lun limit. This limit does not apply to reserved commands. Hence, do not allocate a budget token for reserved commands.

Cc Hannes Reinecke <[email protected]>
Cc John Garry <[email protected]>
Cc Ming Lei <[email protected]>

---
  drivers/scsi/scsi_lib.c | 14 ++++++++++++--

  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +5 -1
index 9c67e04265ce..0112ad3859ff 100644
@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, 
 struct scsi_cmnd *cmd)
       if (starget->can_queue > 0)
           atomic_dec(&starget->target_busy);
-    sbitmap_put(&sdev->budget_map, cmd->budget_token);
+    if (cmd->budget_token < INT_MAX)
+        sbitmap_put(&sdev->budget_map, cmd->budget_token);
       cmd->budget_token = -1;
   }
@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct 
 request_queue *q,
   {
       int token;
+    /*
+     * Do not allocate a budget token for reserved SCSI commands. Budget
+     * tokens are used to enforce the cmd_per_lun limit. That limit 

does not
+     * apply to reserved commands.
+     */
+    if (scsi_device_is_pseudo_dev(sdev))
+        return INT_MAX;
+
      token = sbitmap_get(&sdev->budget_map);
      if (token < 0)
          return -1;

drivers/scsi/scsi_lib.c +0 -0
@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct 
 request_queue *q, int budget_token)
   {
       struct scsi_device *sdev = q->queuedata;
Quoted 8 lines

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q, int budget_token) · line 1765

Quoted 5 lines
drivers/scsi/scsi_lib.c +2 -1
-    sbitmap_put(&sdev->budget_map, budget_token);
+    if (budget_token < INT_MAX)
+        sbitmap_put(&sdev->budget_map, budget_token);
   }
   /*

Good idea, but we should document that somewhere that 'INT_MAX' now has a distinct meaning.

JFYI, when I attempted this previously, I put the check in the blk-mq code, like here https://lore.kernel.org/linux-scsi/[email protected]/

Not needing budget for reserved tags seems a common blk-mq feature. But then only SCSI implements the budget CBs ...

Thanks,
John

#6 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 57 lines · John Garry
Quoted 47 lines · Hannes Reinecke
Quoted 46 lines · Bart Van Assche

The SCSI budget mechanism is used to implement the host->cmd_per_lun limit. This limit does not apply to reserved commands. Hence, do not allocate a budget token for reserved commands.

Cc Hannes Reinecke <[email protected]>
Cc John Garry <[email protected]>
Cc Ming Lei <[email protected]>

---
  drivers/scsi/scsi_lib.c | 14 ++++++++++++--

  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +4 -1
index 9c67e04265ce..0112ad3859ff 100644
@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, 
 struct scsi_cmnd *cmd)
       if (starget->can_queue > 0)
           atomic_dec(&starget->target_busy);
-    sbitmap_put(&sdev->budget_map, cmd->budget_token);
+    if (cmd->budget_token < INT_MAX)
+        sbitmap_put(&sdev->budget_map, cmd->budget_token);
       cmd->budget_token = -1;
   }
@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct 
 request_queue *q,
   {
       int token;
+    /*
+     * Do not allocate a budget token for reserved SCSI commands. 

Budget
+     * tokens are used to enforce the cmd_per_lun limit. That limit does not
+     * apply to reserved commands.
+     */
+    if (scsi_device_is_pseudo_dev(sdev))
+        return INT_MAX;
+
      token = sbitmap_get(&sdev->budget_map);
      if (token < 0)
          return -1;

drivers/scsi/scsi_lib.c +0 -0
@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct 
 request_queue *q, int budget_token)
   {
       struct scsi_device *sdev = q->queuedata;
Quoted 8 lines

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q, int budget_token) · line 1765

Quoted 5 lines
drivers/scsi/scsi_lib.c +2 -1
-    sbitmap_put(&sdev->budget_map, budget_token);
+    if (budget_token < INT_MAX)
+        sbitmap_put(&sdev->budget_map, budget_token);
   }
   /*

Good idea, but we should document that somewhere that 'INT_MAX' now has a distinct meaning.

Quoted 4 lines

JFYI, when I attempted this previously, I put the check in the blk-mq code, like here https://lore.kernel.org/linux-scsi/1666693096-180008-2- [email protected]/

Not needing budget for reserved tags seems a common blk-mq feature. But then only SCSI implements the budget CBs ...

Yeah, that is a far better approach.
Bart? Care to include it?

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author reviewer
#7 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 6 lines · Hannes Reinecke
Quoted 4 lines · John Garry

JFYI, when I attempted this previously, I put the check in the blk-mq code, like here https://lore.kernel.org/linux- scsi/1666693096-180008-2- [email protected]/

Not needing budget for reserved tags seems a common blk-mq feature. But then only SCSI implements the budget CBs ...

Yeah, that is a far better approach.

I do not agree. Only the SCSI core can know whether or not a budget should be allocated for reserved requests. The block layer core can't know whether or not it should allocate a budget for reserved requests. The block layer core can't know whether the SCSI core is allocating a reserved request to guarantee forward progress or whether it is allocating a request that should not be counted against the cmd_per_lun limit. Hence, the decision whether or not to allocate a budget for a reserved request should be taken by the SCSI core.

Bart.

#8 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 7 lines · Bart Van Assche
Quoted 6 lines · Hannes Reinecke
Quoted 4 lines · John Garry

JFYI, when I attempted this previously, I put the check in the blk-mq code, like here https://lore.kernel.org/linux- scsi/1666693096-180008-2- [email protected]/

Not needing budget for reserved tags seems a common blk-mq feature. But then only SCSI implements the budget CBs ...

Yeah, that is a far better approach.

Quoted 9 lines

I do not agree. Only the SCSI core can know whether or not a budget should be allocated for reserved requests. The block layer core can't know whether or not it should allocate a budget for reserved requests. The block layer core can't know whether the SCSI core is allocating a reserved request to guarantee forward progress or whether it is allocating a request that should not be counted against the cmd_per_lun limit. Hence, the decision whether or not to allocate a budget for a reserved request should be taken by the SCSI core.

To my understanding reserved commands are _never_ included in cmd_per_lun; and if we move TMFs and all non-I/O commands to reserved commands cmd_per_lun will only be applicable to I/O commands.
So yes, once we move to reserved commands the budget applies only to normal requests, and never to reserved commands.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author reviewer
#9 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 2 lines · Hannes Reinecke

and if we move TMFs and all non-I/O commands to reserved commands cmd_per_lun will only be applicable to I/O commands.

Whether or not reserved commands should be used for allocating TMFs depends on the SCSI transport. As an example, the approach mentioned above is not appropriate for the UFS driver. The UFS driver assigns integers in the range 0..31 to TMFs. These integers are passed directly to the UFS host controller. Hence, allocating TMFs as reserved commands from the same tag set as SCSI commands is not appropriate for the UFS host controller driver.

Thanks,

Bart.

#10 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 11 lines · Bart Van Assche
Quoted 2 lines · Hannes Reinecke

and if we move TMFs and all non-I/O commands to reserved commands cmd_per_lun will only be applicable to I/O commands.

Whether or not reserved commands should be used for allocating TMFs depends on the SCSI transport. As an example, the approach mentioned above is not appropriate for the UFS driver. The UFS driver assigns integers in the range 0..31 to TMFs. These integers are passed directly to the UFS host controller. Hence, allocating TMFs as reserved commands from the same tag set as SCSI commands is not appropriate for the UFS host controller driver.

So TMFs use a separate tagset than normal commands? IE can I submit TMF tag '3' when I/O tag '3' is currently in flight?

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author reviewer
#11 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 13 lines · Hannes Reinecke
Quoted 4 lines · Bart Van Assche
Quoted 2 lines · Hannes Reinecke

and if we move TMFs and all non-I/O commands to reserved commands cmd_per_lun will only be applicable to I/O commands.

Quoted 8 lines

Whether or not reserved commands should be used for allocating TMFs depends on the SCSI transport. As an example, the approach mentioned above is not appropriate for the UFS driver. The UFS driver assigns integers in the range 0..31 to TMFs. These integers are passed directly to the UFS host controller. Hence, allocating TMFs as reserved commands from the same tag set as SCSI commands is not appropriate for the UFS host controller driver.

Quoted 3 lines

So TMFs use a separate tagset than normal commands? IE can I submit TMF tag '3' when I/O tag '3' is currently in flight?

Yes, that's correct. This follows directly from the UFSHCI specification. In legacy mode (which is easier to explain than MCQ mode), there is one bitwise doorbell register for regular commands (UTRLDBR = UTP Transfer Request List DoorBell Register) and another bitwise doorbell register for task management functions (UTMRLDBR = UTP Task Management Request List DoorBell Register). A command is submitted by setting the appropriate bit in the UTRLDBR register. A TMF is submitted by setting the appropriate bit in the UTMRLDBR register. This is why the UFS driver allocates two tag sets - one for regular commands and another for TMFs.

These registers are called REG_UTP_TRANSFER_REQ_DOOR_BELL and REG_UTP_TASK_REQ_DOOR_BELL in the UFS driver source code.

Thanks,

Bart.

#12 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 29 lines · Bart Van Assche
Quoted 13 lines · Hannes Reinecke
Quoted 4 lines · Bart Van Assche
Quoted 2 lines · Hannes Reinecke

and if we move TMFs and all non-I/O commands to reserved commands cmd_per_lun will only be applicable to I/O commands.

Quoted 8 lines

Whether or not reserved commands should be used for allocating TMFs depends on the SCSI transport. As an example, the approach mentioned above is not appropriate for the UFS driver. The UFS driver assigns integers in the range 0..31 to TMFs. These integers are passed directly to the UFS host controller. Hence, allocating TMFs as reserved commands from the same tag set as SCSI commands is not appropriate for the UFS host controller driver.

Quoted 3 lines

So TMFs use a separate tagset than normal commands? IE can I submit TMF tag '3' when I/O tag '3' is currently in flight?

Yes, that's correct. This follows directly from the UFSHCI specification. In legacy mode (which is easier to explain than MCQ mode), there is one bitwise doorbell register for regular commands (UTRLDBR = UTP Transfer Request List DoorBell Register) and another bitwise doorbell register for task management functions (UTMRLDBR = UTP Task Management Request List DoorBell Register). A command is submitted by setting the appropriate bit in the UTRLDBR register. A TMF is submitted by setting the appropriate bit in the UTMRLDBR register. This is why the UFS driver allocates two tag sets - one for regular commands and another for TMFs.

Quoted 3 lines

These registers are called REG_UTP_TRANSFER_REQ_DOOR_BELL and REG_UTP_TASK_REQ_DOOR_BELL in the UFS driver source code.

Ah. But then you can allocate two separate queues & tagsets, one for TMFs and one for 'normal' commands (like NVMe does it). Then you wouldn't need reserved commands at all; they only make sense if TMFs and commands share the same tagspace.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author reviewer
#13 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

Quoted 2 lines · Hannes Reinecke

Ah. But then you can allocate two separate queues & tagsets, one for TMFs and one for 'normal' commands (like NVMe does it).

Agreed.

Quoted 2 lines

Then you wouldn't need reserved commands at all; they only make sense if TMFs and commands share the same tagspace.

That's wrong. There are other use cases for reserved commands, e.g. a SCSI LLD driver allocating and submitting a SCSI or non-SCSI command to the storage controller.

Here is an overview of the types of commands submitted by the UFS driver to a UFS host controller:
* Task management functions. As explained in the previous email, TMFs have their own tag space that is separate from SCSI and non-SCSI commands.
* SCSI commands that are the result of a block layer request having been translated into a SCSI command, e.g. to implement filesystem I/O.
* SCSI passthrough commands submitted through the SG I/O interface.
* SCSI commands allocated and submitted by the SCSI core, e.g. the INQUIRY and MODE SELECT commands.
* SCSI commands allocated by the UFS driver, e.g. START STOP UNIT to change the power mode of the UFS device during runtime and system-wide power transitions.
* non-SCSI UPIU commands allocated by the UFS driver to perform tasks supported by the UFS standard that fall outside the scope of the SCSI standard, e.g. to query the supported queue depth or to interact with the real-time clock in the UFS device.
* non-SCSI UPIU commands submitted through the BSG interface.

The upstream UFS driver implements its own mechanism for reserving a tag for non-SCSI UPIU commands (hba->reserved_slot). This patch series converts that mechanism into allocating a reserved tag (BLK_MQ_REQ_RESERVED).

Bart.

author reviewer
#14 lore raw

Re: [PATCH v2 03/30] scsi: core: Do not allocate a budget token for reserved commands

In reply to Hannes Reinecke
Quoted 2 lines · Hannes Reinecke

Good idea, but we should document that somewhere that 'INT_MAX' now has a distinct meaning.

I think the comment added by this patch is sufficient. Any comment that would be added in or above the struct scsi_cmnd definition is so far away from the above code that it probably wouldn't be updated the next time the SCSI budget management functions are updated.

Thanks,

Bart.

author reviewer
#15 lore raw

PATCH v2 4/30 scsi: core: Add scsi_{get,put}_internal_cmd() helpers

In reply to Bart Van Assche

From: Hannes Reinecke <[email protected]>

Add helper functions to allow LLDDs to allocate and free internal commands.

[ bvanassche: changed the 'nowait' argument into a 'flags' argument /
added scsi_get_internal_cmd_hctx(). See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]

---
 drivers/scsi/scsi_lib.c    | 80 ++++++++++++++++++++++++++++++++++++++
 include/scsi/scsi_cmnd.h   |  2 +
 include/scsi/scsi_device.h |  7 ++++
 3 files changed, 89 insertions(+)
drivers/scsi/scsi_lib.c +80 -0
index 0112ad3859ff..b0a223dc86f3 100644
@@ -1247,6 +1247,18 @@ struct request *scsi_alloc_request(struct request_queue *q, blk_opf_t opf,
 }
 EXPORT_SYMBOL_GPL(scsi_alloc_request);
 
+struct request *scsi_alloc_request_hctx(struct request_queue *q, blk_opf_t opf,
+			blk_mq_req_flags_t flags, unsigned int hctx_idx)
+{
+	struct request *rq;
+
+	rq = blk_mq_alloc_request_hctx(q, opf, flags, hctx_idx);
+	if (!IS_ERR(rq))
+		scsi_initialize_rq(rq);
+	return rq;
+}
+EXPORT_SYMBOL_GPL(scsi_alloc_request_hctx);
+
 /*
  * Only called when the request isn't completed by SCSI, and not freed by
  * SCSI
@@ -2117,6 +2129,74 @@ void scsi_mq_free_tags(struct kref *kref)
 	complete(&shost->tagset_freed);
 }
 
+/**
+ * scsi_get_internal_cmd() - Allocate an internal SCSI command.
+ * @sdev: SCSI device from which to allocate the command
+ * @data_direction: Data direction for the allocated command
+ * @flags: request allocation flags, e.g. BLK_MQ_REQ_RESERVED or
+ *	BLK_MQ_REQ_NOWAIT.
+ *
+ * Allocates a SCSI command for internal LLDD use.
+ */
+struct scsi_cmnd *scsi_get_internal_cmd(struct scsi_device *sdev,
+					enum dma_data_direction data_direction,
+					blk_mq_req_flags_t flags)
+{
+	enum req_op op = data_direction == DMA_TO_DEVICE ? REQ_OP_DRV_OUT :
+							   REQ_OP_DRV_IN;
+	struct scsi_cmnd *scmd;
+	struct request *rq;
+
+	rq = scsi_alloc_request(sdev->request_queue, op, flags);
+	if (IS_ERR(rq))
+		return NULL;
+	scmd = blk_mq_rq_to_pdu(rq);
+	scmd->device = sdev;
+
+	return scmd;
+}
+EXPORT_SYMBOL_GPL(scsi_get_internal_cmd);
+
+/**
+ * scsi_get_internal_cmd_hctx() - Allocate an internal SCSI command from a given
+ *	hardware queue.
+ * @sdev: SCSI device from which to allocate the command
+ * @data_direction: Data direction for the allocated command
+ * @flags: request allocation flags, e.g. BLK_MQ_REQ_RESERVED or
+ *	BLK_MQ_REQ_NOWAIT.
+ * @hctx_idx: Hardware queue index.
+ *
+ * Allocates a SCSI command for internal LLDD use.
+ */
+struct scsi_cmnd *scsi_get_internal_cmd_hctx(struct scsi_device *sdev,
+			enum dma_data_direction data_direction,
+			blk_mq_req_flags_t flags, unsigned int hctx_idx)
+{
+	enum req_op op = data_direction == DMA_TO_DEVICE ? REQ_OP_DRV_OUT :
+							   REQ_OP_DRV_IN;
+	struct scsi_cmnd *scmd;
+	struct request *rq;
+
+	rq = scsi_alloc_request_hctx(sdev->request_queue, op, flags, hctx_idx);
+	if (IS_ERR(rq))
+		return NULL;
+	scmd = blk_mq_rq_to_pdu(rq);
+	scmd->device = sdev;
+
+	return scmd;
+}
+EXPORT_SYMBOL_GPL(scsi_get_internal_cmd_hctx);
+
+/**
+ * scsi_put_internal_cmd() - Free an internal SCSI command.
+ * @scmd: SCSI command to be freed
+ */
+void scsi_put_internal_cmd(struct scsi_cmnd *scmd)
+{
+	blk_mq_free_request(blk_mq_rq_from_pdu(scmd));
+}
+EXPORT_SYMBOL_GPL(scsi_put_internal_cmd);
+
 /**
  * scsi_device_from_queue - return sdev associated with a request_queue
  * @q: The request queue to return the sdev from
include/scsi/scsi_cmnd.h +2 -0
index 8ecfb94049db..a4cb836809df 100644
@@ -396,5 +396,7 @@ extern void scsi_build_sense(struct scsi_cmnd *scmd, int desc,
 
 struct request *scsi_alloc_request(struct request_queue *q, blk_opf_t opf,
 				   blk_mq_req_flags_t flags);
+struct request *scsi_alloc_request_hctx(struct request_queue *q, blk_opf_t opf,
+			blk_mq_req_flags_t flags, unsigned int hctx_idx);
 
 #endif /* _SCSI_SCSI_CMND_H */
include/scsi/scsi_device.h +7 -0
index 3846f5dfc51c..1d83b17d95e6 100644
@@ -558,6 +558,13 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
 		     const struct scsi_exec_args *args);
 void scsi_failures_reset_retries(struct scsi_failures *failures);
 
+struct scsi_cmnd *scsi_get_internal_cmd(struct scsi_device *sdev,
+					enum dma_data_direction data_direction,
+					blk_mq_req_flags_t flags);
+struct scsi_cmnd *scsi_get_internal_cmd_hctx(struct scsi_device *sdev,
+			enum dma_data_direction data_direction,
+			blk_mq_req_flags_t flags, unsigned int hctx_idx);
+void scsi_put_internal_cmd(struct scsi_cmnd *scmd);
 extern void sdev_disable_disk_events(struct scsi_device *sdev);
 extern void sdev_enable_disk_events(struct scsi_device *sdev);
 extern int scsi_vpd_lun_id(struct scsi_device *, char *, size_t);
author reviewer
#16 lore raw

PATCH v2 5/30 scsi: core: Introduce scsi_host_update_can_queue()

In reply to Bart Van Assche

If a SCSI driver must use reserved commands to discover the supported queue depth then the queue depth must be initialized to a small value and must be changed after the queue depth has been queried. Support such SCSI drivers by introducing the function scsi_host_update_can_queue().

Cc Hannes Reinecke <[email protected]>
Cc John Garry <[email protected]>
---
 drivers/scsi/scsi.c      | 45 ++++++++++++++++++++++++++++++++++++++++
 include/scsi/scsi_host.h |  2 ++
 2 files changed, 47 insertions(+)
drivers/scsi/scsi.c +45 -0
index a290c3aa7b88..3d3603b74d9f 100644
@@ -198,6 +198,51 @@ void scsi_finish_command(struct scsi_cmnd *cmd)
 	scsi_io_completion(cmd, good_bytes);
 }
 
+/**
+ * scsi_host_update_can_queue - Modify @host->can_queue
+ * @host:	SCSI host pointer
+ * @can_queue:	New value for @host->can_queue.
+ *
+ * @host->__devices must be empty except for the pseudo SCSI device and I/O
+ * must have been quiesced before this function is called.
+ */
+int scsi_host_update_can_queue(struct Scsi_Host *host, int can_queue)
+{
+	struct blk_mq_tag_set prev_set;
+	bool realloc_pseudo_sdev = false;
+	struct scsi_device *sdev;
+	int prev_can_queue, ret;
+
+	scoped_guard(spinlock_irq, host->host_lock)
+		list_for_each_entry(sdev, &host->__devices, siblings)
+			if (WARN_ON_ONCE(sdev != host->pseudo_sdev))
+				return -EINVAL;
+
+	if (host->pseudo_sdev) {
+		realloc_pseudo_sdev = true;
+		__scsi_remove_device(host->pseudo_sdev);
+		host->pseudo_sdev = NULL;
+	}
+
+	prev_can_queue = host->can_queue;
+	prev_set = host->tag_set;
+	host->can_queue = can_queue;
+	ret = scsi_mq_setup_tags(host);
+	if (ret) {
+		host->can_queue = prev_can_queue;
+		return ret;
+	}
+	blk_mq_free_tag_set(&prev_set);
+
+	if (realloc_pseudo_sdev) {
+		host->pseudo_sdev = scsi_get_pseudo_dev(host);
+		if (!host->pseudo_sdev)
+			return -ENOMEM;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(scsi_host_update_can_queue);
 
 /*
  * 4096 is big enough for saturating fast SCSI LUNs.
include/scsi/scsi_host.h +2 -0
index 3b5150759c44..dd967f35df33 100644
@@ -837,6 +837,8 @@ extern void scsi_block_requests(struct Scsi_Host *);
 extern int scsi_host_block(struct Scsi_Host *shost);
 extern int scsi_host_unblock(struct Scsi_Host *shost, int new_state);
 
+int scsi_host_update_can_queue(struct Scsi_Host *host, int can_queue);
+
 void scsi_host_busy_iter(struct Scsi_Host *,
 			 bool (*fn)(struct scsi_cmnd *, void *), void *priv);
 
#17 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

Quoted 5 lines · Bart Van Assche

If a SCSI driver must use reserved commands to discover the supported queue depth then the queue depth must be initialized to a small value and must be changed after the queue depth has been queried. Support such SCSI drivers by introducing the function scsi_host_update_can_queue().

why can you not just query this before allocating the shost via a direct HW access?

Quoted 3 lines
Cc Hannes Reinecke<[email protected]>
Cc John Garry<[email protected]>
author reviewer
#18 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

Quoted 7 lines · John Garry
Quoted 4 lines · Bart Van Assche

If a SCSI driver must use reserved commands to discover the supported queue depth then the queue depth must be initialized to a small value and must be changed after the queue depth has been queried. Support such SCSI drivers by introducing the function scsi_host_update_can_queue().

why can you not just query this before allocating the shost via a direct HW access?

Hi John,

Hasn't that already been explained in the above patch description?

The only way to discover the maximum queue depth supported by the UFS device (bQueueDepth parameter) is by sending a QUERY REQUEST / STANDARD READ REQUEST to the UFS device. The current approach in the UFS driver is to reserve one command slot for device management commands, not to make that command slot visible to the SCSI core and also to have code for sending and completing device management commands that duplicates some block layer and SCSI core code.

In this patch series the choice has been made not to duplicate any SCSI core / block layer core code. Hence, the SCSI host is allocated first and the QUERY REQUEST is allocated as a reserved command. Existing SCSI core functionality is used for submitting that reserved command and also for handling timeouts. Hence the need for the scsi_host_update_can_queue() function.

As one can see this approach allows to remove a significant amount of code in patch 30/30: the entire ufshcd_wait_for_dev_cmd() function is removed.

Thanks,

Bart.

#19 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

Quoted 7 lines · Bart Van Assche
Quoted 6 lines · John Garry
Quoted 4 lines · Bart Van Assche

If a SCSI driver must use reserved commands to discover the supported queue depth then the queue depth must be initialized to a small value and must be changed after the queue depth has been queried. Support such SCSI drivers by introducing the function scsi_host_update_can_queue().

Quoted 7 lines
Quoted 1 line

why can you not just query this before allocating the shost via a direct HW access?

Hi John,

Hasn't that already been explained in the above patch description?

The only way to discover the maximum queue depth

To be exact, do you mean the device or the host? I assume device.

Quoted 11 lines

supported by the UFS
device (bQueueDepth parameter) is by sending a QUERY REQUEST / STANDARD READ REQUEST to the UFS device. The current approach in the UFS driver is to reserve one command slot for device management commands, not to make that command slot visible to the SCSI core and also to have code for sending and completing device management commands that duplicates some block layer and SCSI core code.

In this patch series the choice has been made not to duplicate any SCSI core / block layer core code. Hence, the SCSI host is allocated first and the QUERY REQUEST is allocated as a reserved command.

From my limited understanding of ufs architecture, we have a ufs host and a single ufs device.

The host max commands is read in function ufshcd_mcq_decide_queue_depth() and only MMIO-based register accesses are required for this.

The ufs device max commands is read by sending some query command to the device. This is held in hba->dev_info.bqueuedepth.

In the driver, we seem to combine the host queue depth and device queue depth into a single value, hba->nutrs - see ufshcd_async_scan() -> ufshcd_alloc_mcq(hba, bqueuedepth) -> ufshcd_mcq_devide_queue_depth(ufs_dev_qd)

ufshcd_mcq_devide_queue_depth(hba, ufs_dev_qd)

{
	...
	return min(ufs_dev_qd, mac); //mac is host max commands

}

ufshcd_alloc_mcq(bqueuedepth)

{
	...
	hba->nutrs = ufshcd_mcq_devide_queue_depth(hba, bqueuedepth)
}

And hba->nutrs is used for the shost->can_queue. And that is why you need to re-add the shost, as the hba->nutrs may change after querying the ufs device, which can only be done after adding the shost (in this series).

Am I correct so far?

If so, I just don't know why the ufs host and device queue depths are not handled separately like other SCSI drivers. That would be the shost->can_queue = ufs host queue depth and the scsi slave device qd would be the ufs device queue depth.

Thanks,
John

Quoted 12 lines

Existing SCSI
core functionality is used for submitting that reserved command and also for handling timeouts. Hence the need for the scsi_host_update_can_queue() function.

As one can see this approach allows to remove a significant amount of code in patch 30/30: the entire ufshcd_wait_for_dev_cmd() function is removed.

Thanks,

Bart.

author reviewer
#20 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

Quoted 1 line · John Garry

From my limited understanding of ufs architecture, we have a ufs host and a single ufs device.

That's correct. JEDEC publishes a standard for UFS host controllers (the UFSHCI specification) and another standard for UFS devices (the UFS specification). UFSHCI stands for UFS host controller interface. Each UFS host controller can support only one UFS device.

Quoted 19 lines

The host max commands is read in function ufshcd_mcq_decide_queue_depth() and only MMIO-based register accesses are required for this.

The ufs device max commands is read by sending some query command to the device. This is held in hba->dev_info.bqueuedepth.

Quoted 1 line

In the driver, we seem to combine the host queue depth and device queue

depth into a single value, hba->nutrs - see ufshcd_async_scan() -> ufshcd_alloc_mcq(hba, bqueuedepth) -> ufshcd_mcq_devide_queue_depth(ufs_dev_qd)

ufshcd_mcq_devide_queue_depth(hba, ufs_dev_qd)
{
    ...
    return min(ufs_dev_qd, mac); //mac is host max commands

}

ufshcd_alloc_mcq(bqueuedepth)
{
    ...
    hba->nutrs = ufshcd_mcq_devide_queue_depth(hba, bqueuedepth)
}

Quoted 3 lines

And hba->nutrs is used for the shost->can_queue. And that is why you need to re-add the shost, as the hba->nutrs may change after querying the ufs device, which can only be done after adding the shost (in this series).

Am I correct so far?

The above matches my understanding.

Quoted 3 lines

If so, I just don't know why the ufs host and device queue depths are not handled separately like other SCSI drivers. That would be the shost-

Quoted 1 line

can_queue = ufs host queue depth and the scsi slave device qd would be

the ufs device queue depth.

It is probably possible to convert the UFS kernel driver to this approach. However, that will lead to some wasted memory. UFSHCI 4.0 controllers may support up to 256 outstanding commands. The UFS devices that I'm familiar with support 32, 64 or 128 outstanding commands. pahole reports the following for a local aarch64 kernel build:
* sizeof(struct scsi_cmnd) = 344.
* sizeof(ufshcd_lrb) = 152.
* sizeof(utp_transfer_cmd_desc) = 1024
* SG_ALL = 128
* ufshcd_sg_entry_size(hba) = sizeof(struct ufshcd_sg_entry) = 16
* ufshcd_get_ucd_size(hba) = 1024 + 128 * 16 = 3072
* sizeof(struct utp_transfer_req_desc) = 32

For a controller that supports 256 outstanding commands and a UFS device
that supports 32 outstanding commands this will result in (256 - 32) *
(344 + 152 + 3072 + 32) = 224 * 3600 = 806,400 bytes of memory that is never used. I'm not sure all UFS users will agree with this.

BTW, reserving space for SG_ALL (128) scatter/gather entries in each command seems excessive to me but even if that would be addressed, there still would be a significant amount of memory that is not used without scsi_host_update_can_queue().

Thanks,

Bart.

#21 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

Quoted 15 lines · Bart Van Assche
Quoted 1 line

If so, I just don't know why the ufs host and device queue depths are not handled separately like other SCSI drivers. That would be the shost-  >can_queue = ufs host queue depth and the scsi slave device qd would be the ufs device queue depth.

It is probably possible to convert the UFS kernel driver to this approach. However, that will lead to some wasted memory. UFSHCI 4.0 controllers may support up to 256 outstanding commands. The UFS devices that I'm familiar with support 32, 64 or 128 outstanding commands. pahole reports the following for a local aarch64 kernel build:
* sizeof(struct scsi_cmnd) = 344.
* sizeof(ufshcd_lrb) = 152.
* sizeof(utp_transfer_cmd_desc) = 1024
* SG_ALL = 128
* ufshcd_sg_entry_size(hba) = sizeof(struct ufshcd_sg_entry) = 16
* ufshcd_get_ucd_size(hba) = 1024 + 128 * 16 = 3072
* sizeof(struct utp_transfer_req_desc) = 32

Quoted 8 lines

For a controller that supports 256 outstanding commands and a UFS device
that supports 32 outstanding commands this will result in (256 - 32) *
(344 + 152 + 3072 + 32) = 224 * 3600 = 806,400 bytes of memory that is never used. I'm not sure all UFS users will agree with this

BTW, reserving space for SG_ALL (128) scatter/gather entries in each command seems excessive to me but even if that would be addressed, there still would be a significant amount of memory that is not used without scsi_host_update_can_queue().

Well that is how life is for other SCSI drivers. Changing the shost q depth with scsi_host_update_can_queue() is a bit ghastly, IMHO.

Earlier you wrote "The current approach in the UFS driver is to reserve one command slot for device management commands, not to make that command slot visible to the SCSI core and also to have code"

I don't know why a raw QEURY command cannot be sent before scsi_host_alloc() to get this ufs device queue depth. An arbitrary tag can be used. That tag could then be re-used as part of the shost tagset. There prob would be some duplicated code for sending this raw QEURY command, but that seems to be worth it if you really want to save memory.

Thanks,
John

author reviewer
#22 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

Quoted 1 line · John Garry

Well that is how life is for other SCSI drivers. Changing the shost q depth with scsi_host_update_can_queue() is a bit ghastly, IMHO.

I will look into setting host->can_queue before the UFS device queue depth is queried and also into setting host->cmd_per_lun after the UFS device queue depth has been queried.

Thanks,

Bart.

#23 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

In reply to Bart Van Assche
Quoted 57 lines · Bart Van Assche

If a SCSI driver must use reserved commands to discover the supported queue depth then the queue depth must be initialized to a small value and must be changed after the queue depth has been queried. Support such SCSI drivers by introducing the function scsi_host_update_can_queue().

Cc Hannes Reinecke <[email protected]>
Cc John Garry <[email protected]>
---
  drivers/scsi/scsi.c      | 45 ++++++++++++++++++++++++++++++++++++++++
  include/scsi/scsi_host.h |  2 ++
  2 files changed, 47 insertions(+)
drivers/scsi/scsi.c +36 -0
index a290c3aa7b88..3d3603b74d9f 100644
@@ -198,6 +198,51 @@ void scsi_finish_command(struct scsi_cmnd *cmd)
  	scsi_io_completion(cmd, good_bytes);
  }
 
+/**
+ * scsi_host_update_can_queue - Modify @host->can_queue
+ * @host:	SCSI host pointer
+ * @can_queue:	New value for @host->can_queue.
+ *
+ * @host->__devices must be empty except for the pseudo SCSI device and I/O
+ * must have been quiesced before this function is called.
+ */
+int scsi_host_update_can_queue(struct Scsi_Host *host, int can_queue)
+{
+	struct blk_mq_tag_set prev_set;
+	bool realloc_pseudo_sdev = false;
+	struct scsi_device *sdev;
+	int prev_can_queue, ret;
+
+	scoped_guard(spinlock_irq, host->host_lock)
+		list_for_each_entry(sdev, &host->__devices, siblings)
+			if (WARN_ON_ONCE(sdev != host->pseudo_sdev))
+				return -EINVAL;
+
+	if (host->pseudo_sdev) {
+		realloc_pseudo_sdev = true;
+		__scsi_remove_device(host->pseudo_sdev);
+		host->pseudo_sdev = NULL;
+	}
+
+	prev_can_queue = host->can_queue;
+	prev_set = host->tag_set;
+	host->can_queue = can_queue;
+	ret = scsi_mq_setup_tags(host);
+	if (ret) {
+		host->can_queue = prev_can_queue;
+		return ret;
+	}
+	blk_mq_free_tag_set(&prev_set);
+

drivers/scsi/scsi.c void scsi_finish_command(struct scsi_cmnd *cmd) · line 246

Quoted 10 lines
drivers/scsi/scsi.c +9 -0
+	if (realloc_pseudo_sdev) {
+		host->pseudo_sdev = scsi_get_pseudo_dev(host);
+		if (!host->pseudo_sdev)
+			return -ENOMEM;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(scsi_host_update_can_queue);
 

Reallocate the SCSI host device? Really?

We do have blk_mq_update_nr_requests(), why can't you use that?
I would really go with the idea from John; the minimal number of
commands which can be allocated is trivially '1'.
And the number of reserved tags typically should be known a priori,
too. So you can start off with a tagset having just one 'normal'
tag (and a pre-defined number of reserved tags).
And then after querying you should be able to call 'blk_mq_update_nr_requests()', and we should be good here...

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author reviewer
#24 lore raw

Re: [PATCH v2 05/30] scsi: core: Introduce scsi_host_update_can_queue()

Quoted 1 line · Hannes Reinecke

Reallocate the SCSI host device? Really?

The SCSI host device is not reallocated - only the tag set is reallocated.

Quoted 7 lines

We do have  blk_mq_update_nr_requests(), why can't you use that?
I would really go with the idea from John; the minimal number of
commands which can be allocated is trivially '1'.
And the number of reserved tags typically should be known a priori,
too. So you can start off with a tagset having just one 'normal'
tag (and a pre-defined number of reserved tags).
And then after querying you should be able to call 'blk_mq_update_nr_requests()', and we should be good here...

blk_mq_update_nr_requests() is not exported by the block layer. Anyway, I will look into this approach.

Thanks,

Bart.

author reviewer
#25 lore raw

PATCH v2 6/30 scsi_debug: Set .alloc_pseudo_sdev

In reply to Bart Van Assche

Make sure that the code for allocating a pseudo SCSI device gets triggered while running blktests.

---
 drivers/scsi/scsi_debug.c | 1 +
 1 file changed, 1 insertion(+)
drivers/scsi/scsi_debug.c +1 -0
index 353cb60e1abe..620560c2676c 100644
@@ -9463,6 +9463,7 @@ static const struct scsi_host_template sdebug_driver_template = {
 	.module =		THIS_MODULE,
 	.skip_settle_delay =	1,
 	.track_queue_depth =	1,
+	.alloc_pseudo_sdev =	1,
 	.cmd_size = sizeof(struct sdebug_scsi_cmd),
 	.init_cmd_priv = sdebug_init_cmd_priv,
 	.target_alloc =		sdebug_target_alloc,
author reviewer
#26 lore raw

PATCH v2 7/30 ufs: core: Move an assignment

In reply to Bart Van Assche

Prepare for introducing a WARN_ON_ONCE() call in ufshcd_mcq_get_tag() if the tag lookup fails.

---
 drivers/ufs/core/ufs-mcq.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/ufs/core/ufs-mcq.c +2 -1
index 1e50675772fe..4eb4f1af7800 100644
@@ -303,9 +303,10 @@ static void ufshcd_mcq_process_cqe(struct ufs_hba *hba,
 				   struct ufs_hw_queue *hwq)
 {
 	struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq);
-	int tag = ufshcd_mcq_get_tag(hba, cqe);
 
 	if (cqe->command_desc_base_addr) {
+		int tag = ufshcd_mcq_get_tag(hba, cqe);
+
 		ufshcd_compl_one_cqe(hba, tag, cqe);
 		/* After processed the cqe, mark it empty (invalid) entry */
 		cqe->command_desc_base_addr = 0;
author reviewer
#27 lore raw

PATCH v2 8/30 ufs: core: Change the type of one ufshcd_add_cmd_upiu_trace() argument

In reply to Bart Van Assche

Change the 'tag' argument into an LRB pointer. This patch prepares for the removal of the hba->lrb[] array.

---
 drivers/ufs/core/ufshcd.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +5 -4
index b5057ce27aa9..fee144f36c4a 100644
@@ -405,10 +405,11 @@ static void ufshcd_configure_wb(struct ufs_hba *hba)
 		ufshcd_wb_toggle_buf_flush(hba, true);
 }
 
-static void ufshcd_add_cmd_upiu_trace(struct ufs_hba *hba, unsigned int tag,
+static void ufshcd_add_cmd_upiu_trace(struct ufs_hba *hba,
+				      struct ufshcd_lrb *lrb,
 				      enum ufs_trace_str_t str_t)
 {
-	struct utp_upiu_req *rq = hba->lrb[tag].ucd_req_ptr;
+	struct utp_upiu_req *rq = lrb->ucd_req_ptr;
 	struct utp_upiu_header *header;
 
 	if (!trace_ufshcd_upiu_enabled())
@@ -417,7 +418,7 @@ static void ufshcd_add_cmd_upiu_trace(struct ufs_hba *hba, unsigned int tag,
 	if (str_t == UFS_CMD_SEND)
 		header = &rq->header;
 	else
-		header = &hba->lrb[tag].ucd_rsp_ptr->header;
+		header = &lrb->ucd_rsp_ptr->header;
 
 	trace_ufshcd_upiu(hba, str_t, header, &rq->sc.cdb,
 			  UFS_TSF_CDB);
@@ -491,7 +492,7 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 		return;
 
 	/* trace UPIU also */
-	ufshcd_add_cmd_upiu_trace(hba, tag, str_t);
+	ufshcd_add_cmd_upiu_trace(hba, lrbp, str_t);
 	if (!trace_ufshcd_command_enabled())
 		return;
 
author reviewer
#28 lore raw

PATCH v2 9/30 ufs: core: Only call ufshcd_add_command_trace() for SCSI commands

In reply to Bart Van Assche

Instead of checking inside ufshcd_add_command_trace() whether 'cmd' points at a SCSI command, let the caller perform that check. This patch prepares for removing the lrbp->cmd pointer.

---
 drivers/ufs/core/ufshcd.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +3 -5
index fee144f36c4a..2ee735872e4d 100644
@@ -488,9 +488,6 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 	struct request *rq = scsi_cmd_to_rq(cmd);
 	int transfer_len = -1;
 
-	if (!cmd)
-		return;
-
 	/* trace UPIU also */
 	ufshcd_add_cmd_upiu_trace(hba, lrbp, str_t);
 	if (!trace_ufshcd_command_enabled())
@@ -2359,9 +2356,10 @@ void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag,
 	lrbp->issue_time_stamp_local_clock = local_clock();
 	lrbp->compl_time_stamp = ktime_set(0, 0);
 	lrbp->compl_time_stamp_local_clock = 0;
-	ufshcd_add_command_trace(hba, task_tag, UFS_CMD_SEND);
-	if (lrbp->cmd)
+	if (lrbp->cmd) {
+		ufshcd_add_command_trace(hba, task_tag, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
+	}
 	if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
 		ufshcd_start_monitor(hba, lrbp);
 
author reviewer
#29 lore raw

PATCH v2 10/30 ufs: core: Change the type of one ufshcd_add_command_trace() argument

In reply to Bart Van Assche

Change the 'tag' argument into a SCSI command pointer. This patch prepares for the removal of the hba->lrb[] array.

---
 drivers/ufs/core/ufshcd.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/ufs/core/ufshcd.c +6 -6
index 2ee735872e4d..ba9367f748a5 100644
@@ -475,7 +475,7 @@ static void ufshcd_add_uic_command_trace(struct ufs_hba *hba,
 				 ufshcd_readl(hba, REG_UIC_COMMAND_ARG_3));
 }
 
-static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
+static void ufshcd_add_command_trace(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 				     enum ufs_trace_str_t str_t)
 {
 	u64 lba = 0;
@@ -483,9 +483,9 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 	u32 doorbell = 0;
 	u32 intr;
 	u32 hwq_id = 0;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
-	struct scsi_cmnd *cmd = lrbp->cmd;
 	struct request *rq = scsi_cmd_to_rq(cmd);
+	unsigned int tag = rq->tag;
+	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
 	int transfer_len = -1;
 
 	/* trace UPIU also */
@@ -503,7 +503,7 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 		       be32_to_cpu(lrbp->ucd_req_ptr->sc.exp_data_transfer_len);
 		lba = scsi_get_lba(cmd);
 		if (opcode == WRITE_10)
-			group_id = lrbp->cmd->cmnd[6];
+			group_id = cmd->cmnd[6];
 	} else if (opcode == UNMAP) {
 		/*
 		 * The number of Bytes to be unmapped beginning with the lba.
@@ -2357,7 +2357,7 @@ void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag,
 	lrbp->compl_time_stamp = ktime_set(0, 0);
 	lrbp->compl_time_stamp_local_clock = 0;
 	if (lrbp->cmd) {
-		ufshcd_add_command_trace(hba, task_tag, UFS_CMD_SEND);
+		ufshcd_add_command_trace(hba, lrbp->cmd, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
 	}
 	if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
@@ -5627,7 +5627,7 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 	if (cmd) {
 		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
 			ufshcd_update_monitor(hba, lrbp);
-		ufshcd_add_command_trace(hba, task_tag, UFS_CMD_COMP);
+		ufshcd_add_command_trace(hba, cmd, UFS_CMD_COMP);
 		cmd->result = ufshcd_transfer_rsp_status(hba, lrbp, cqe);
 		ufshcd_release_scsi_cmd(hba, lrbp);
 		/* Do not touch lrbp after scsi done */
author reviewer
#30 lore raw

PATCH v2 11/30 ufs: core: Change the type of one ufshcd_send_command() argument

In reply to Bart Van Assche

Change the 'task_tag' argument into an LRB pointer. This patch prepares for the removal of the hba->lrb[] array.

---
 drivers/ufs/core/ufshcd.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/ufs/core/ufshcd.c +6 -7
index ba9367f748a5..c188f17a50b2 100644
@@ -2342,14 +2342,13 @@ static void ufshcd_update_monitor(struct ufs_hba *hba, const struct ufshcd_lrb *
 /**
  * ufshcd_send_command - Send SCSI or device management commands
  * @hba: per adapter instance
- * @task_tag: Task tag of the command
+ * @lrbp: Local reference block of SCSI command
  * @hwq: pointer to hardware queue instance
  */
-static inline
-void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag,
-			 struct ufs_hw_queue *hwq)
+static inline void ufshcd_send_command(struct ufs_hba *hba,
+				       struct ufshcd_lrb *lrbp,
+				       struct ufs_hw_queue *hwq)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
 	unsigned long flags;
 
 	lrbp->issue_time_stamp = ktime_get();
@@ -3056,7 +3055,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 	if (hba->mcq_enabled)
 		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
 
-	ufshcd_send_command(hba, tag, hwq);
+	ufshcd_send_command(hba, lrbp, hwq);
 
 out:
 	if (ufs_trigger_eh(hba)) {
@@ -3302,7 +3301,7 @@ static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 	int err;
 
 	ufshcd_add_query_upiu_trace(hba, UFS_QUERY_SEND, lrbp->ucd_req_ptr);
-	ufshcd_send_command(hba, tag, hba->dev_cmd_queue);
+	ufshcd_send_command(hba, lrbp, hba->dev_cmd_queue);
 	err = ufshcd_wait_for_dev_cmd(hba, lrbp, timeout);
 
 	ufshcd_add_query_upiu_trace(hba, err ? UFS_QUERY_ERR : UFS_QUERY_COMP,
author reviewer
#31 lore raw

PATCH v2 12/30 ufs: core: Only call ufshcd_should_inform_monitor() for SCSI commands

In reply to Bart Van Assche

ufshcd_should_inform_monitor() only returns 'true' for SCSI commands. Instead of checking inside ufshcd_should_inform_monitor() whether its second argument represents a SCSI command, only call this function for SCSI commands. This patch prepares for removing the lrbp->cmd member.

---
 drivers/ufs/core/ufshcd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c +4 -3
index c188f17a50b2..46c322c66d9e 100644
@@ -2285,12 +2285,13 @@ static inline int ufshcd_monitor_opcode2dir(u8 opcode)
 		return -EINVAL;
 }
 
+/* Must only be called for SCSI commands. */
 static inline bool ufshcd_should_inform_monitor(struct ufs_hba *hba,
 						struct ufshcd_lrb *lrbp)
 {
 	const struct ufs_hba_monitor *m = &hba->monitor;
 
-	return (m->enabled && lrbp && lrbp->cmd &&
+	return (m->enabled &&
 		(!m->chunk_size || m->chunk_size == lrbp->cmd->sdb.length) &&
 		ktime_before(hba->monitor.enabled_ts, lrbp->issue_time_stamp));
 }
@@ -2358,9 +2359,9 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 	if (lrbp->cmd) {
 		ufshcd_add_command_trace(hba, lrbp->cmd, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
+		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
+			ufshcd_start_monitor(hba, lrbp);
 	}
-	if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
-		ufshcd_start_monitor(hba, lrbp);
 
 	if (hba->mcq_enabled) {
 		int utrd_size = sizeof(struct utp_transfer_req_desc);
author reviewer
#32 lore raw

PATCH v2 13/30 ufs: core: Change the monitor function argument types

In reply to Bart Van Assche

Pass a SCSI command pointer instead of a struct ufshcd_lrb pointer. This patch prepares for combining the SCSI command and ufshcd_lrb data structures into a single data structure.

---
 drivers/ufs/core/ufshcd.c | 44 +++++++++++++++++++--------------------
 1 file changed, 22 insertions(+), 22 deletions(-)
drivers/ufs/core/ufshcd.c +22 -22
index 46c322c66d9e..923a19ff668a 100644
@@ -2287,19 +2287,20 @@ static inline int ufshcd_monitor_opcode2dir(u8 opcode)
 
 /* Must only be called for SCSI commands. */
 static inline bool ufshcd_should_inform_monitor(struct ufs_hba *hba,
-						struct ufshcd_lrb *lrbp)
+						struct scsi_cmnd *cmd)
 {
 	const struct ufs_hba_monitor *m = &hba->monitor;
+	struct request *rq = scsi_cmd_to_rq(cmd);
+	struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag];
 
-	return (m->enabled &&
-		(!m->chunk_size || m->chunk_size == lrbp->cmd->sdb.length) &&
-		ktime_before(hba->monitor.enabled_ts, lrbp->issue_time_stamp));
+	return m->enabled &&
+	       (!m->chunk_size || m->chunk_size == cmd->sdb.length) &&
+	       ktime_before(hba->monitor.enabled_ts, lrbp->issue_time_stamp);
 }
 
-static void ufshcd_start_monitor(struct ufs_hba *hba,
-				 const struct ufshcd_lrb *lrbp)
+static void ufshcd_start_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
-	int dir = ufshcd_monitor_opcode2dir(*lrbp->cmd->cmnd);
+	int dir = ufshcd_monitor_opcode2dir(cmd->cmnd[0]);
 	unsigned long flags;
 
 	spin_lock_irqsave(hba->host->host_lock, flags);
@@ -2308,14 +2309,15 @@ static void ufshcd_start_monitor(struct ufs_hba *hba,
 	spin_unlock_irqrestore(hba->host->host_lock, flags);
 }
 
-static void ufshcd_update_monitor(struct ufs_hba *hba, const struct ufshcd_lrb *lrbp)
+static void ufshcd_update_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
-	int dir = ufshcd_monitor_opcode2dir(*lrbp->cmd->cmnd);
+	struct request *req = scsi_cmd_to_rq(cmd);
+	struct ufshcd_lrb *lrbp = &hba->lrb[req->tag];
+	int dir = ufshcd_monitor_opcode2dir(cmd->cmnd[0]);
 	unsigned long flags;
 
 	spin_lock_irqsave(hba->host->host_lock, flags);
 	if (dir >= 0 && hba->monitor.nr_queued[dir] > 0) {
-		const struct request *req = scsi_cmd_to_rq(lrbp->cmd);
 		struct ufs_hba_monitor *m = &hba->monitor;
 		ktime_t now, inc, lat;
 
@@ -2350,17 +2352,18 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 				       struct ufshcd_lrb *lrbp,
 				       struct ufs_hw_queue *hwq)
 {
+	struct scsi_cmnd *cmd = lrbp->cmd;
 	unsigned long flags;
 
 	lrbp->issue_time_stamp = ktime_get();
 	lrbp->issue_time_stamp_local_clock = local_clock();
 	lrbp->compl_time_stamp = ktime_set(0, 0);
 	lrbp->compl_time_stamp_local_clock = 0;
-	if (lrbp->cmd) {
-		ufshcd_add_command_trace(hba, lrbp->cmd, UFS_CMD_SEND);
+	if (cmd) {
+		ufshcd_add_command_trace(hba, cmd, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
-		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
-			ufshcd_start_monitor(hba, lrbp);
+		if (unlikely(ufshcd_should_inform_monitor(hba, cmd)))
+			ufshcd_start_monitor(hba, cmd);
 	}
 
 	if (hba->mcq_enabled) {
@@ -2376,8 +2379,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 	} else {
 		spin_lock_irqsave(&hba->outstanding_lock, flags);
 		if (hba->vops && hba->vops->setup_xfer_req)
-			hba->vops->setup_xfer_req(hba, lrbp->task_tag,
-						  !!lrbp->cmd);
+			hba->vops->setup_xfer_req(hba, lrbp->task_tag, !!cmd);
 		__set_bit(lrbp->task_tag, &hba->outstanding_reqs);
 		ufshcd_writel(hba, 1 << lrbp->task_tag,
 			      REG_UTP_TRANSFER_REQ_DOOR_BELL);
@@ -5616,17 +5618,15 @@ void ufshcd_release_scsi_cmd(struct ufs_hba *hba,
 void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 			  struct cq_entry *cqe)
 {
-	struct ufshcd_lrb *lrbp;
-	struct scsi_cmnd *cmd;
+	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
+	struct scsi_cmnd *cmd = lrbp->cmd;
 	enum utp_ocs ocs;
 
-	lrbp = &hba->lrb[task_tag];
 	lrbp->compl_time_stamp = ktime_get();
 	lrbp->compl_time_stamp_local_clock = local_clock();
-	cmd = lrbp->cmd;
 	if (cmd) {
-		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
-			ufshcd_update_monitor(hba, lrbp);
+		if (unlikely(ufshcd_should_inform_monitor(hba, cmd)))
+			ufshcd_update_monitor(hba, cmd);
 		ufshcd_add_command_trace(hba, cmd, UFS_CMD_COMP);
 		cmd->result = ufshcd_transfer_rsp_status(hba, lrbp, cqe);
 		ufshcd_release_scsi_cmd(hba, lrbp);
author reviewer
#33 lore raw

PATCH v2 14/30 ufs: core: Rework ufshcd_mcq_compl_pending_transfer()

In reply to Bart Van Assche

Replace a tag loop with blk_mq_tagset_busy_iter(). This patch prepares for removing the hba->lrb[] array.

---
 drivers/ufs/core/ufshcd.c | 80 ++++++++++++++++++++++-----------------
 1 file changed, 46 insertions(+), 34 deletions(-)
drivers/ufs/core/ufshcd.c +46 -34
index 923a19ff668a..e506843d4f33 100644
@@ -5711,6 +5711,48 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)
 	return completed_reqs != 0;
 }
 
+static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
+{
+	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
+	struct scsi_device *sdev = rq->q->queuedata;
+	struct Scsi_Host *shost = sdev->host;
+	struct ufs_hba *hba = shost_priv(shost);
+	struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag];
+	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
+
+	if (!hwq)
+		return true;
+
+	ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
+
+	/*
+	 * For those cmds of which the cqes are not present in the cq, complete
+	 * them explicitly.
+	 */
+	scoped_guard(spinlock_irqsave, &hwq->cq_lock) {
+		if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) {
+			set_host_byte(cmd, DID_REQUEUE);
+			ufshcd_release_scsi_cmd(hba, lrbp);
+			scsi_done(cmd);
+		}
+	}
+
+	return true;
+}
+
+static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)
+{
+	struct scsi_device *sdev = rq->q->queuedata;
+	struct Scsi_Host *shost = sdev->host;
+	struct ufs_hba *hba = shost_priv(shost);
+	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
+
+	if (hwq)
+		ufshcd_mcq_poll_cqe_lock(hba, hwq);
+
+	return true;
+}
+
 /**
  * ufshcd_mcq_compl_pending_transfer - MCQ mode function. It is
  * invoked from the error handler context or ufshcd_host_reset_and_restore()
@@ -5725,40 +5767,10 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)
 static void ufshcd_mcq_compl_pending_transfer(struct ufs_hba *hba,
 					      bool force_compl)
 {
-	struct ufs_hw_queue *hwq;
-	struct ufshcd_lrb *lrbp;
-	struct scsi_cmnd *cmd;
-	unsigned long flags;
-	int tag;
-
-	for (tag = 0; tag < hba->nutrs; tag++) {
-		lrbp = &hba->lrb[tag];
-		cmd = lrbp->cmd;
-		if (!ufshcd_cmd_inflight(cmd) ||
-		    test_bit(SCMD_STATE_COMPLETE, &cmd->state))
-			continue;
-
-		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
-		if (!hwq)
-			continue;
-
-		if (force_compl) {
-			ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
-			/*
-			 * For those cmds of which the cqes are not present
-			 * in the cq, complete them explicitly.
-			 */
-			spin_lock_irqsave(&hwq->cq_lock, flags);
-			if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) {
-				set_host_byte(cmd, DID_REQUEUE);
-				ufshcd_release_scsi_cmd(hba, lrbp);
-				scsi_done(cmd);
-			}
-			spin_unlock_irqrestore(&hwq->cq_lock, flags);
-		} else {
-			ufshcd_mcq_poll_cqe_lock(hba, hwq);
-		}
-	}
+	blk_mq_tagset_busy_iter(&hba->host->tag_set,
+				force_compl ? ufshcd_mcq_force_compl_one :
+					      ufshcd_mcq_compl_one,
+				NULL);
 }
 
 /**
author reviewer
#34 lore raw

PATCH v2 15/30 ufs: core: Rework ufshcd_eh_device_reset_handler()

In reply to Bart Van Assche

Merge the MCQ mode and legacy mode loops into a single loop. This patch prepares for optimizing the hot path by removing the direct hba->lrb[] accesses from ufshcd_eh_device_reset_handler().

---
 drivers/ufs/core/ufshcd.c | 84 ++++++++++++++++++---------------------
 1 file changed, 38 insertions(+), 46 deletions(-)
drivers/ufs/core/ufshcd.c +38 -46
index e506843d4f33..ac4c54a34fd8 100644
@@ -7538,6 +7538,36 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 	return err ? : result;
 }
 
+static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)
+{
+	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(req);
+	struct scsi_device *sdev = cmd->device;
+	struct Scsi_Host *shost = sdev->host;
+	struct ufs_hba *hba = shost_priv(shost);
+	const u64 lun = *(u64 *)priv;
+	const u32 tag = req->tag;
+
+	if (sdev->lun != lun)
+		return true;
+
+	if (ufshcd_clear_cmd(hba, tag) < 0) {
+		dev_err(hba->dev, "%s: failed to clear request %d\n", __func__,
+			tag);
+		return true;
+	}
+
+	if (hba->mcq_enabled) {
+		struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, req);
+
+		if (hwq)
+			ufshcd_mcq_poll_cqe_lock(hba, hwq);
+		return true;
+	}
+
+	ufshcd_compl_one_cqe(hba, tag, NULL);
+	return true;
+}
+
 /**
  * ufshcd_eh_device_reset_handler() - Reset a single logical unit.
  * @cmd: SCSI command pointer
@@ -7546,12 +7576,8 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
  */
 static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd)
 {
-	unsigned long flags, pending_reqs = 0, not_cleared = 0;
 	struct Scsi_Host *host;
 	struct ufs_hba *hba;
-	struct ufs_hw_queue *hwq;
-	struct ufshcd_lrb *lrbp;
-	u32 pos, not_cleared_mask = 0;
 	int err;
 	u8 resp = 0xF, lun;
 
@@ -7560,50 +7586,16 @@ static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd)
 
 	lun = ufshcd_scsi_to_upiu_lun(cmd->device->lun);
 	err = ufshcd_issue_tm_cmd(hba, lun, 0, UFS_LOGICAL_RESET, &resp);
-	if (err || resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) {
-		if (!err)
-			err = resp;
-		goto out;
-	}
-
-	if (hba->mcq_enabled) {
-		for (pos = 0; pos < hba->nutrs; pos++) {
-			lrbp = &hba->lrb[pos];
-			if (ufshcd_cmd_inflight(lrbp->cmd) &&
-			    lrbp->lun == lun) {
-				ufshcd_clear_cmd(hba, pos);
-				hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(lrbp->cmd));
-				ufshcd_mcq_poll_cqe_lock(hba, hwq);
-			}
-		}
-		err = 0;
-		goto out;
-	}
-
-	/* clear the commands that were pending for corresponding LUN */
-	spin_lock_irqsave(&hba->outstanding_lock, flags);
-	for_each_set_bit(pos, &hba->outstanding_reqs, hba->nutrs)
-		if (hba->lrb[pos].lun == lun)
-			__set_bit(pos, &pending_reqs);
-	hba->outstanding_reqs &= ~pending_reqs;
-	spin_unlock_irqrestore(&hba->outstanding_lock, flags);
-
-	for_each_set_bit(pos, &pending_reqs, hba->nutrs) {
-		if (ufshcd_clear_cmd(hba, pos) < 0) {
-			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			not_cleared = 1U << pos &
-				ufshcd_readl(hba, REG_UTP_TRANSFER_REQ_DOOR_BELL);
-			hba->outstanding_reqs |= not_cleared;
-			not_cleared_mask |= not_cleared;
-			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
-
-			dev_err(hba->dev, "%s: failed to clear request %d\n",
-				__func__, pos);
-		}
+	if (err) {
+	} else if (resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) {
+		err = resp;
+	} else {
+		/* clear the commands that were pending for corresponding LUN */
+		blk_mq_tagset_busy_iter(&hba->host->tag_set,
+					ufshcd_clear_lu_cmds,
+					&cmd->device->lun);
 	}
-	__ufshcd_transfer_req_compl(hba, pending_reqs & ~not_cleared_mask);
 
-out:
 	hba->req_abort_count = 0;
 	ufshcd_update_evt_hist(hba, UFS_EVT_DEV_RESET, (u32)err);
 	if (!err) {
author reviewer
#35 lore raw

PATCH v2 16/30 ufs: core: Cache the DMA buffer sizes

In reply to Bart Van Assche

Prepare for supporting DMA buffer reallocation. Caching the DMA buffer sizes is essential because a later patch will modify hba->nutrs between the ufshcd_memory_alloc() and the ufshcd_memory_free() calls.

---
 drivers/ufs/core/ufshcd.c | 62 ++++++++++++++++++++-------------------
 include/ufs/ufshcd.h      |  5 ++++
 2 files changed, 37 insertions(+), 30 deletions(-)
drivers/ufs/core/ufshcd.c +32 -30
index ac4c54a34fd8..09727a94595c 100644
@@ -3909,14 +3909,10 @@ static int ufshcd_get_ref_clk_gating_wait(struct ufs_hba *hba)
  */
 static int ufshcd_memory_alloc(struct ufs_hba *hba)
 {
-	size_t utmrdl_size, utrdl_size, ucdl_size;
-
 	/* Allocate memory for UTP command descriptors */
-	ucdl_size = ufshcd_get_ucd_size(hba) * hba->nutrs;
-	hba->ucdl_base_addr = dmam_alloc_coherent(hba->dev,
-						  ucdl_size,
-						  &hba->ucdl_dma_addr,
-						  GFP_KERNEL);
+	hba->ucdl_size = ufshcd_get_ucd_size(hba) * hba->nutrs;
+	hba->ucdl_base_addr = dmam_alloc_coherent(
+		hba->dev, hba->ucdl_size, &hba->ucdl_dma_addr, GFP_KERNEL);
 
 	/*
 	 * UFSHCI requires UTP command descriptor to be 128 byte aligned.
@@ -3932,11 +3928,9 @@ static int ufshcd_memory_alloc(struct ufs_hba *hba)
 	 * Allocate memory for UTP Transfer descriptors
 	 * UFSHCI requires 1KB alignment of UTRD
 	 */
-	utrdl_size = (sizeof(struct utp_transfer_req_desc) * hba->nutrs);
-	hba->utrdl_base_addr = dmam_alloc_coherent(hba->dev,
-						   utrdl_size,
-						   &hba->utrdl_dma_addr,
-						   GFP_KERNEL);
+	hba->utrdl_size = sizeof(struct utp_transfer_req_desc) * hba->nutrs;
+	hba->utrdl_base_addr = dmam_alloc_coherent(
+		hba->dev, hba->utrdl_size, &hba->utrdl_dma_addr, GFP_KERNEL);
 	if (!hba->utrdl_base_addr ||
 	    WARN_ON(hba->utrdl_dma_addr & (SZ_1K - 1))) {
 		dev_err(hba->dev,
@@ -3948,7 +3942,7 @@ static int ufshcd_memory_alloc(struct ufs_hba *hba)
 	 * Skip utmrdl allocation; it may have been
 	 * allocated during first pass and not released during
 	 * MCQ memory allocation.
-	 * See ufshcd_release_sdb_queue() and ufshcd_config_mcq()
+	 * See ufshcd_memory_free() and ufshcd_config_mcq()
 	 */
 	if (hba->utmrdl_base_addr)
 		goto skip_utmrdl;
@@ -3956,11 +3950,9 @@ static int ufshcd_memory_alloc(struct ufs_hba *hba)
 	 * Allocate memory for UTP Task Management descriptors
 	 * UFSHCI requires 1KB alignment of UTMRD
 	 */
-	utmrdl_size = sizeof(struct utp_task_req_desc) * hba->nutmrs;
-	hba->utmrdl_base_addr = dmam_alloc_coherent(hba->dev,
-						    utmrdl_size,
-						    &hba->utmrdl_dma_addr,
-						    GFP_KERNEL);
+	hba->utmrdl_size = sizeof(struct utp_task_req_desc) * hba->nutmrs;
+	hba->utmrdl_base_addr = dmam_alloc_coherent(
+		hba->dev, hba->utmrdl_size, &hba->utmrdl_dma_addr, GFP_KERNEL);
 	if (!hba->utmrdl_base_addr ||
 	    WARN_ON(hba->utmrdl_dma_addr & (SZ_1K - 1))) {
 		dev_err(hba->dev,
@@ -8852,20 +8844,30 @@ static int ufshcd_add_lus(struct ufs_hba *hba)
 	return ret;
 }
 
-/* SDB - Single Doorbell */
-static void ufshcd_release_sdb_queue(struct ufs_hba *hba, int nutrs)
+/*
+ * Free the memory allocated by ufshcd_memory_alloc() except the utmrdl DMA
+ * memory.
+ */
+static void ufshcd_memory_free(struct ufs_hba *hba)
 {
-	size_t ucdl_size, utrdl_size;
-
-	ucdl_size = ufshcd_get_ucd_size(hba) * nutrs;
-	dmam_free_coherent(hba->dev, ucdl_size, hba->ucdl_base_addr,
-			   hba->ucdl_dma_addr);
+	if (hba->ucdl_base_addr) {
+		dmam_free_coherent(hba->dev, hba->ucdl_size,
+				   hba->ucdl_base_addr, hba->ucdl_dma_addr);
+		hba->ucdl_base_addr = NULL;
+		hba->ucdl_dma_addr = 0;
+	}
 
-	utrdl_size = sizeof(struct utp_transfer_req_desc) * nutrs;
-	dmam_free_coherent(hba->dev, utrdl_size, hba->utrdl_base_addr,
-			   hba->utrdl_dma_addr);
+	if (hba->utrdl_base_addr) {
+		dmam_free_coherent(hba->dev, hba->utrdl_size,
+				   hba->utrdl_base_addr, hba->utrdl_dma_addr);
+		hba->utrdl_base_addr = NULL;
+		hba->utrdl_dma_addr = 0;
+	}
 
-	devm_kfree(hba->dev, hba->lrb);
+	if (hba->lrb) {
+		devm_kfree(hba->dev, hba->lrb);
+		hba->lrb = NULL;
+	}
 }
 
 static int ufshcd_alloc_mcq(struct ufs_hba *hba)
@@ -8887,7 +8889,7 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba)
 	 * Number of supported tags in MCQ mode may be larger than SDB mode.
 	 */
 	if (hba->nutrs != old_nutrs) {
-		ufshcd_release_sdb_queue(hba, old_nutrs);
+		ufshcd_memory_free(hba);
 		ret = ufshcd_memory_alloc(hba);
 		if (ret)
 			goto err;
include/ufs/ufshcd.h +5 -0
index 1d3943777584..3e71d691d731 100644
@@ -842,6 +842,9 @@ enum ufshcd_mcq_opr {
 /**
  * struct ufs_hba - per adapter private structure
  * @mmio_base: UFSHCI base register address
+ * @ucdl_size: Size of UFS Command Descriptor buffer
+ * @utrdl_size: Size of UTP Transfer Request Descriptor buffer
+ * @utmrdl_size: Size of UTP Task Management Descriptor buffer
  * @ucdl_base_addr: UFS Command Descriptor base address
  * @utrdl_base_addr: UTP Transfer Request Descriptor base address
  * @utmrdl_base_addr: UTP Task Management Descriptor base address
@@ -971,6 +974,8 @@ enum ufshcd_mcq_opr {
 struct ufs_hba {
 	void __iomem *mmio_base;
 
+	u32 ucdl_size, utrdl_size, utmrdl_size;
+
 	/* Virtual memory reference */
 	struct utp_transfer_cmd_desc *ucdl_base_addr;
 	struct utp_transfer_req_desc *utrdl_base_addr;
author reviewer
#36 lore raw

PATCH v2 17/30 ufs: core: Add an argument to ufshcd_mcq_decide_queue_depth()

In reply to Bart Van Assche

Prepare for allocating SCSI commands in two steps by making the UFS device queue depth an argument of ufshcd_mcq_decide_queue_depth().

---
 drivers/ufs/core/ufs-mcq.c     | 14 ++++++++------
 drivers/ufs/core/ufshcd-priv.h |  2 +-
 drivers/ufs/core/ufshcd.c      |  2 +-
 3 files changed, 10 insertions(+), 8 deletions(-)
drivers/ufs/core/ufs-mcq.c +8 -6
index 4eb4f1af7800..a44ab6c010d8 100644
@@ -132,6 +132,7 @@ EXPORT_SYMBOL_GPL(ufshcd_mcq_queue_cfg_addr);
 /**
  * ufshcd_mcq_decide_queue_depth - decide the queue depth
  * @hba: per adapter instance
+ * @ufs_dev_qd: maximum queue depth supported by the UFS device
  *
  * Return: queue-depth on success, non-zero on error
  *
@@ -140,7 +141,7 @@ EXPORT_SYMBOL_GPL(ufshcd_mcq_queue_cfg_addr);
  * Calculates and adjusts the queue depth based on the depth
  * supported by the HC and ufs device.
  */
-int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba)
+int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba, int ufs_dev_qd)
 {
 	int mac;
 
@@ -160,13 +161,14 @@ int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba)
 	if (mac < 0)
 		goto err;
 
-	WARN_ON_ONCE(!hba->dev_info.bqueuedepth);
 	/*
-	 * max. value of bqueuedepth = 256, mac is host dependent.
-	 * It is mandatory for UFS device to define bQueueDepth if
-	 * shared queuing architecture is enabled.
+	 * According to the UFS standard, the UFS device queue depth
+	 * (bQueueDepth) must be in the range 1..255 if the shared queueing
+	 * architecture is supported. bQueueDepth is zero if the shared queueing
+	 * architecture is not supported.
 	 */
-	return min_t(int, mac, hba->dev_info.bqueuedepth);
+	WARN_ON_ONCE(!ufs_dev_qd);
+	return min(mac, ufs_dev_qd);
 
 err:
 	dev_err(hba->dev, "Failed to get mac, err=%d\n", mac);
drivers/ufs/core/ufshcd-priv.h +1 -1
index d0a2c963a27d..c02f9a6af137 100644
@@ -65,7 +65,7 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 			  struct cq_entry *cqe);
 int ufshcd_mcq_init(struct ufs_hba *hba);
 void ufshcd_mcq_disable(struct ufs_hba *hba);
-int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba);
+int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba, int ufs_dev_qd);
 int ufshcd_mcq_memory_alloc(struct ufs_hba *hba);
 struct ufs_hw_queue *ufshcd_mcq_req_to_hwq(struct ufs_hba *hba,
 					   struct request *req);
drivers/ufs/core/ufshcd.c +1 -1
index 09727a94595c..bed88caea319 100644
@@ -8875,7 +8875,7 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba)
 	int ret;
 	int old_nutrs = hba->nutrs;
 
-	ret = ufshcd_mcq_decide_queue_depth(hba);
+	ret = ufshcd_mcq_decide_queue_depth(hba, hba->dev_info.bqueuedepth);
 	if (ret < 0)
 		return ret;
 
author reviewer
#37 lore raw

PATCH v2 18/30 ufs: core: Add an argument to ufshcd_alloc_mcq()

In reply to Bart Van Assche

Prepare for allocating SCSI commands in two steps by making the UFS device queue depth an argument of ufshcd_alloc_mcq().

---
 drivers/ufs/core/ufshcd.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c +3 -3
index bed88caea319..d63bf5b3a414 100644
@@ -8870,12 +8870,12 @@ static void ufshcd_memory_free(struct ufs_hba *hba)
 	}
 }
 
-static int ufshcd_alloc_mcq(struct ufs_hba *hba)
+static int ufshcd_alloc_mcq(struct ufs_hba *hba, int ufs_dev_qd)
 {
 	int ret;
 	int old_nutrs = hba->nutrs;
 
-	ret = ufshcd_mcq_decide_queue_depth(hba, hba->dev_info.bqueuedepth);
+	ret = ufshcd_mcq_decide_queue_depth(hba, ufs_dev_qd);
 	if (ret < 0)
 		return ret;
 
@@ -10569,7 +10569,7 @@ static int ufshcd_add_scsi_host(struct ufs_hba *hba)
 
 	if (is_mcq_supported(hba)) {
 		ufshcd_mcq_enable(hba);
-		err = ufshcd_alloc_mcq(hba);
+		err = ufshcd_alloc_mcq(hba, hba->dev_info.bqueuedepth);
 		if (!err) {
 			ufshcd_config_mcq(hba);
 		} else {
author reviewer
#38 lore raw

PATCH v2 19/30 ufs: core: Call ufshcd_mcq_init() once

In reply to Bart Van Assche

In a later patch scsi_host_template.cmd_size will be set to make struct scsi_cmnd and struct ufshcd_lrb adjacent. Since scsi_add_host() allocates memory for SCSI commands, the SCSI host must be allocated before any device management commands are sent to the UFS device and before the UFS device queue depth has been queried. Hence, the queue depth must be modified after the UFS device queue depth has been queried. Modifiying the queue depth involves calling ufshcd_alloc_mcq() a second time. This patch prepares for calling ufshcd_alloc_mcq() twice and makes sure that ufshcd_mcq_init() is called once even if ufshcd_alloc_mcq() is called twice.

---
 drivers/ufs/core/ufshcd.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c +10 -3
index d63bf5b3a414..23f7734d3cc1 100644
@@ -8880,9 +8880,16 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba, int ufs_dev_qd)
 		return ret;
 
 	hba->nutrs = ret;
-	ret = ufshcd_mcq_init(hba);
-	if (ret)
-		goto err;
+	if (hba->host->nr_hw_queues == 0) {
+		/*
+		 * ufshcd_mcq_init() is independent of hba->nutrs. Hence, only
+		 * call ufshcd_mcq_init() the first time ufshcd_alloc_mcq() is
+		 * called.
+		 */
+		ret = ufshcd_mcq_init(hba);
+		if (ret)
+			goto err;
+	}
 
 	/*
 	 * Previously allocated memory for nutrs may not be enough in MCQ mode.
author reviewer
#39 lore raw

PATCH v2 20/30 ufs: core: Allocate the SCSI host earlier

In reply to Bart Van Assche

Call ufshcd_add_scsi_host() before any UPIU commands are sent to the UFS device. This patch prepares for letting ufshcd_add_scsi_host() allocate memory for both SCSI and UPIU commands.

Initially, allocate 32 commands (SCSI + reserved). In ufshcd_alloc_mcq(), increase the SCSI host queue depth if necessary.

---
 drivers/ufs/core/ufshcd.c | 38 +++++++++++++++++++++++++++++++++-----
 1 file changed, 33 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +33 -5
index 23f7734d3cc1..82d804dd0c85 100644
@@ -9089,6 +9089,31 @@ static void ufshcd_async_scan(void *data, async_cookie_t cookie)
 	ktime_t probe_start;
 	int ret;
 
+	if (is_mcq_supported(hba)) {
+		ufshcd_mcq_enable(hba);
+		ret = ufshcd_alloc_mcq(hba, hba->dev_info.bqueuedepth);
+		if (ret == 0) {
+			ufshcd_config_mcq(hba);
+			ret = scsi_host_update_can_queue(
+				hba->host, hba->nutrs - UFSHCD_NUM_RESERVED);
+			if (ret)
+				goto out;
+		} else {
+			/* Continue with SDB mode */
+			ufshcd_mcq_disable(hba);
+			use_mcq_mode = false;
+			dev_err(hba->dev, "MCQ mode is disabled, err=%d\n",
+				ret);
+		}
+	}
+	if (!is_mcq_supported(hba) && !hba->lsdb_sup) {
+		dev_err(hba->dev,
+			"%s: failed to initialize (legacy doorbell mode not supported)\n",
+			__func__);
+		ret = -EINVAL;
+		goto out;
+	}
+
 	down(&hba->host_sem);
 	/* Initialize hba, detect and initialize UFS device */
 	probe_start = ktime_get();
@@ -10574,9 +10599,12 @@ static int ufshcd_add_scsi_host(struct ufs_hba *hba)
 {
 	int err;
 
+	WARN_ON_ONCE(!hba->host->can_queue);
+	WARN_ON_ONCE(!hba->host->cmd_per_lun);
+
 	if (is_mcq_supported(hba)) {
 		ufshcd_mcq_enable(hba);
-		err = ufshcd_alloc_mcq(hba, hba->dev_info.bqueuedepth);
+		err = ufshcd_alloc_mcq(hba, 32);
 		if (!err) {
 			ufshcd_config_mcq(hba);
 		} else {
@@ -10815,6 +10843,10 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 			    FIELD_PREP(UFSHCI_AHIBERN8_SCALE_MASK, 3);
 	}
 
+	err = ufshcd_add_scsi_host(hba);
+	if (err)
+		goto out_disable;
+
 	/* Hold auto suspend until async scan completes */
 	pm_runtime_get_sync(dev);
 
@@ -10865,10 +10897,6 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 	if (err)
 		goto out_disable;
 
-	err = ufshcd_add_scsi_host(hba);
-	if (err)
-		goto out_disable;
-
 	async_schedule(ufshcd_async_scan, hba);
 	ufs_sysfs_add_nodes(hba->dev);
 
author reviewer
#40 lore raw

PATCH v2 21/30 ufs: core: Make ufshcd_mcq_init() independent of hba->nutrs

In reply to Bart Van Assche

Prepare for modifying hba->nutrs after ufshcd_mcq_init() has been called. No functionality has been changed.

---
 drivers/ufs/core/ufs-mcq.c | 1 -
 drivers/ufs/core/ufshcd.c  | 5 ++++-
 2 files changed, 4 insertions(+), 2 deletions(-)
drivers/ufs/core/ufs-mcq.c +0 -1
index a44ab6c010d8..6a1838ced02e 100644
@@ -477,7 +477,6 @@ int ufshcd_mcq_init(struct ufs_hba *hba)
 
 	for (i = 0; i < hba->nr_hw_queues; i++) {
 		hwq = &hba->uhq[i];
-		hwq->max_entries = hba->nutrs + 1;
 		spin_lock_init(&hwq->sq_lock);
 		spin_lock_init(&hwq->cq_lock);
 		mutex_init(&hwq->sq_mutex);
drivers/ufs/core/ufshcd.c +4 -1
index 82d804dd0c85..27140432d737 100644
@@ -8872,8 +8872,8 @@ static void ufshcd_memory_free(struct ufs_hba *hba)
 
 static int ufshcd_alloc_mcq(struct ufs_hba *hba, int ufs_dev_qd)
 {
-	int ret;
 	int old_nutrs = hba->nutrs;
+	int i, ret;
 
 	ret = ufshcd_mcq_decide_queue_depth(hba, ufs_dev_qd);
 	if (ret < 0)
@@ -8891,6 +8891,9 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba, int ufs_dev_qd)
 			goto err;
 	}
 
+	for (i = 0; i < hba->nr_hw_queues; i++)
+		hba->uhq[i].max_entries = hba->nutrs + 1;
+
 	/*
 	 * Previously allocated memory for nutrs may not be enough in MCQ mode.
 	 * Number of supported tags in MCQ mode may be larger than SDB mode.
author reviewer
#41 lore raw

PATCH v2 22/30 ufs: core: Call ufshcd_init_lrb() later

In reply to Bart Van Assche

Call ufshcd_init_lrb() from inside ufshcd_setup_dev_cmd() instead of ufshcd_host_memory_configure(). This patch prepares for calling ufshcd_host_memory_configure() before the information is available that is required to call ufshcd_setup_dev_cmd().

---
 drivers/ufs/core/ufshcd.c | 53 ++++++++++++++++++++-------------------
 1 file changed, 27 insertions(+), 26 deletions(-)
drivers/ufs/core/ufshcd.c +27 -26
index 27140432d737..a932f62c5c2c 100644
@@ -2894,8 +2894,32 @@ static void ufshcd_comp_scsi_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 	ufshcd_prepare_utp_scsi_cmd_upiu(lrbp, upiu_flags);
 }
 
-static void __ufshcd_setup_cmd(struct ufshcd_lrb *lrbp, struct scsi_cmnd *cmd, u8 lun, int tag)
+static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i)
+{
+	struct utp_transfer_cmd_desc *cmd_descp =
+		(void *)hba->ucdl_base_addr + i * ufshcd_get_ucd_size(hba);
+	struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr;
+	dma_addr_t cmd_desc_element_addr =
+		hba->ucdl_dma_addr + i * ufshcd_get_ucd_size(hba);
+	u16 response_offset = le16_to_cpu(utrdlp[i].response_upiu_offset);
+	u16 prdt_offset = le16_to_cpu(utrdlp[i].prd_table_offset);
+
+	lrb->utr_descriptor_ptr = utrdlp + i;
+	lrb->utrd_dma_addr =
+		hba->utrdl_dma_addr + i * sizeof(struct utp_transfer_req_desc);
+	lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp->command_upiu;
+	lrb->ucd_req_dma_addr = cmd_desc_element_addr;
+	lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu;
+	lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset;
+	lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table;
+	lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset;
+}
+
+static void __ufshcd_setup_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
+			       struct scsi_cmnd *cmd, u8 lun, int tag)
 {
+	ufshcd_init_lrb(hba, lrbp, tag);
+
 	memset(lrbp->ucd_req_ptr, 0, sizeof(*lrbp->ucd_req_ptr));
 
 	lrbp->cmd = cmd;
@@ -2907,7 +2931,7 @@ static void __ufshcd_setup_cmd(struct ufshcd_lrb *lrbp, struct scsi_cmnd *cmd, u
 static void ufshcd_setup_scsi_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 				  struct scsi_cmnd *cmd, u8 lun, int tag)
 {
-	__ufshcd_setup_cmd(lrbp, cmd, lun, tag);
+	__ufshcd_setup_cmd(hba, lrbp, cmd, lun, tag);
 	lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed(hba);
 	lrbp->req_abort_skip = false;
 
@@ -2962,27 +2986,6 @@ static void ufshcd_map_queues(struct Scsi_Host *shost)
 	}
 }
 
-static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i)
-{
-	struct utp_transfer_cmd_desc *cmd_descp = (void *)hba->ucdl_base_addr +
-		i * ufshcd_get_ucd_size(hba);
-	struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr;
-	dma_addr_t cmd_desc_element_addr = hba->ucdl_dma_addr +
-		i * ufshcd_get_ucd_size(hba);
-	u16 response_offset = le16_to_cpu(utrdlp[i].response_upiu_offset);
-	u16 prdt_offset = le16_to_cpu(utrdlp[i].prd_table_offset);
-
-	lrb->utr_descriptor_ptr = utrdlp + i;
-	lrb->utrd_dma_addr = hba->utrdl_dma_addr +
-		i * sizeof(struct utp_transfer_req_desc);
-	lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp->command_upiu;
-	lrb->ucd_req_dma_addr = cmd_desc_element_addr;
-	lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu;
-	lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset;
-	lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table;
-	lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset;
-}
-
 /**
  * ufshcd_queuecommand - main entry point for SCSI requests
  * @host: SCSI host pointer
@@ -3075,7 +3078,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 			     enum dev_cmd_type cmd_type, u8 lun, int tag)
 {
-	__ufshcd_setup_cmd(lrbp, NULL, lun, tag);
+	__ufshcd_setup_cmd(hba, lrbp, NULL, lun, tag);
 	lrbp->intr_cmd = true; /* No interrupt aggregation */
 	hba->dev_cmd.type = cmd_type;
 }
@@ -4030,8 +4033,6 @@ static void ufshcd_host_memory_configure(struct ufs_hba *hba)
 			utrdlp[i].response_upiu_length =
 				cpu_to_le16(ALIGNED_UPIU_SIZE >> 2);
 		}
-
-		ufshcd_init_lrb(hba, &hba->lrb[i], i);
 	}
 }
 
author reviewer
#42 lore raw

PATCH v2 23/30 ufs: core: Use hba->reserved_slot

In reply to Bart Van Assche

Use hba->reserved_slot instead of open-coding it. This patch prepares for changing the value of hba->reserved_slot.

---
 drivers/ufs/core/ufs-mcq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ufs/core/ufs-mcq.c +1 -1
index 6a1838ced02e..da2da57130c8 100644
@@ -547,7 +547,7 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 	if (hba->quirks & UFSHCD_QUIRK_MCQ_BROKEN_RTC)
 		return -ETIMEDOUT;
 
-	if (task_tag != hba->nutrs - UFSHCD_NUM_RESERVED) {
+	if (task_tag != hba->reserved_slot) {
 		if (!cmd)
 			return -EINVAL;
 		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
author reviewer
#43 lore raw

PATCH v2 24/30 ufs: core: Make the reserved slot a reserved request

In reply to Bart Van Assche

Instead of letting the SCSI core allocate hba->nutrs - 1 commands, let the SCSI core allocate hba->nutrs commands, set the number of reserved tags to 1 and use the reserved tag for device management commands. This patch changes the 'reserved slot' from hba->nutrs - 1 into 0 because the block layer reserves the smallest tags for reserved commands.

---
 drivers/ufs/core/ufshcd.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/ufs/core/ufshcd.c +4 -2
index a932f62c5c2c..8014b6ec0e42 100644
@@ -2466,7 +2466,7 @@ static inline int ufshcd_hba_capabilities(struct ufs_hba *hba)
 	hba->nutrs = (hba->capabilities & MASK_TRANSFER_REQUESTS_SLOTS_SDB) + 1;
 	hba->nutmrs =
 	((hba->capabilities & MASK_TASK_MANAGEMENT_REQUEST_SLOTS) >> 16) + 1;
-	hba->reserved_slot = hba->nutrs - 1;
+	hba->reserved_slot = 0;
 
 	hba->nortt = FIELD_GET(MASK_NUMBER_OUTSTANDING_RTT, hba->capabilities) + 1;
 
@@ -8912,7 +8912,8 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba, int ufs_dev_qd)
 		goto err;
 
 	hba->host->can_queue = hba->nutrs - UFSHCD_NUM_RESERVED;
-	hba->reserved_slot = hba->nutrs - UFSHCD_NUM_RESERVED;
+	hba->host->nr_reserved_cmds = UFSHCD_NUM_RESERVED;
+	hba->reserved_slot = 0;
 
 	return 0;
 err:
@@ -10757,6 +10758,7 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 
 	host->can_queue = hba->nutrs - UFSHCD_NUM_RESERVED;
 	host->cmd_per_lun = hba->nutrs - UFSHCD_NUM_RESERVED;
+	host->nr_reserved_cmds = UFSHCD_NUM_RESERVED;
 	host->max_id = UFSHCD_MAX_ID;
 	host->max_lun = UFS_MAX_LUNS;
 	host->max_channel = UFSHCD_MAX_CHANNEL;
author reviewer
#44 lore raw

PATCH v2 25/30 ufs: core: Do not clear driver-private command data

In reply to Bart Van Assche

Tell the SCSI core to skip the memset() call that clears driver-private data because __ufshcd_setup_cmd() performs all necessary initialization.

---
 drivers/ufs/core/ufshcd.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/ufs/core/ufshcd.c +10 -0
index 8014b6ec0e42..b887041aafcd 100644
@@ -2986,6 +2986,15 @@ static void ufshcd_map_queues(struct Scsi_Host *shost)
 	}
 }
 
+/*
+ * The only purpose of this function is to make the SCSI core skip the memset()
+ * call for the private command data.
+ */
+static int ufshcd_init_cmd_priv(struct Scsi_Host *host, struct scsi_cmnd *cmd)
+{
+	return 0;
+}
+
 /**
  * ufshcd_queuecommand - main entry point for SCSI requests
  * @host: SCSI host pointer
@@ -9181,6 +9190,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
 	.name			= UFSHCD,
 	.proc_name		= UFSHCD,
 	.map_queues		= ufshcd_map_queues,
+	.init_cmd_priv		= ufshcd_init_cmd_priv,
 	.queuecommand		= ufshcd_queuecommand,
 	.mq_poll		= ufshcd_poll,
 	.sdev_init		= ufshcd_sdev_init,
author reviewer
#45 lore raw

PATCH v2 26/30 ufs: core: Optimize the hot path

In reply to Bart Van Assche

Set .cmd_size in the SCSI host template such that the SCSI core makes struct scsi_cmnd and struct ufshcd_lrb adjacent. Convert the cmd->lrbp and lrbp->cmd memory loads into pointer offset calculations. Remove the data structure members that became superfluous, namely ufshcd_lrb.cmd and ufs_hba.lrb.

---
 drivers/ufs/core/ufs-mcq.c       |   9 +-
 drivers/ufs/core/ufshcd-crypto.h |  18 ++-
 drivers/ufs/core/ufshcd-priv.h   |  41 +++++-
 drivers/ufs/core/ufshcd.c        | 235 ++++++++++++++++---------------
 include/ufs/ufshcd.h             |   5 -
 5 files changed, 177 insertions(+), 131 deletions(-)
drivers/ufs/core/ufs-mcq.c +5 -4
index da2da57130c8..1f9ff9fc44c0 100644
@@ -537,8 +537,8 @@ static int ufshcd_mcq_sq_start(struct ufs_hba *hba, struct ufs_hw_queue *hwq)
  */
 int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
-	struct scsi_cmnd *cmd = lrbp->cmd;
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, task_tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct ufs_hw_queue *hwq;
 	void __iomem *reg, *opr_sqd_base;
 	u32 nexus, id, val;
@@ -623,7 +623,8 @@ static void ufshcd_mcq_nullify_sqe(struct utp_transfer_req_desc *utrd)
 static bool ufshcd_mcq_sqe_search(struct ufs_hba *hba,
 				  struct ufs_hw_queue *hwq, int task_tag)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, task_tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct utp_transfer_req_desc *utrd;
 	__le64  cmd_desc_base_addr;
 	bool ret = false;
@@ -674,7 +675,7 @@ int ufshcd_mcq_abort(struct scsi_cmnd *cmd)
 	struct Scsi_Host *host = cmd->device->host;
 	struct ufs_hba *hba = shost_priv(host);
 	int tag = scsi_cmd_to_rq(cmd)->tag;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct ufs_hw_queue *hwq;
 	int err;
 
drivers/ufs/core/ufshcd-crypto.h +11 -7
index 89bb97c14c15..c148a5194378 100644
@@ -38,10 +38,10 @@ ufshcd_prepare_req_desc_hdr_crypto(struct ufshcd_lrb *lrbp,
 }
 
 static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba,
-					  struct ufshcd_lrb *lrbp)
+					  struct scsi_cmnd *cmd)
 {
-	struct scsi_cmnd *cmd = lrbp->cmd;
 	const struct bio_crypt_ctx *crypt_ctx = scsi_cmd_to_rq(cmd)->crypt_ctx;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 
 	if (crypt_ctx && hba->vops && hba->vops->fill_crypto_prdt)
 		return hba->vops->fill_crypto_prdt(hba, crypt_ctx,
@@ -51,17 +51,19 @@ static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba,
 }
 
 static inline void ufshcd_crypto_clear_prdt(struct ufs_hba *hba,
-					    struct ufshcd_lrb *lrbp)
+					    struct scsi_cmnd *cmd)
 {
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
 	if (!(hba->quirks & UFSHCD_QUIRK_KEYS_IN_PRDT))
 		return;
 
-	if (!(scsi_cmd_to_rq(lrbp->cmd)->crypt_ctx))
+	if (!(scsi_cmd_to_rq(cmd)->crypt_ctx))
 		return;
 
 	/* Zeroize the PRDT because it can contain cryptographic keys. */
 	memzero_explicit(lrbp->ucd_prdt_ptr,
-			 ufshcd_sg_entry_size(hba) * scsi_sg_count(lrbp->cmd));
+			 ufshcd_sg_entry_size(hba) * scsi_sg_count(cmd));
 }
 
 bool ufshcd_crypto_enable(struct ufs_hba *hba);
@@ -82,13 +84,15 @@ ufshcd_prepare_req_desc_hdr_crypto(struct ufshcd_lrb *lrbp,
 				   struct request_desc_header *h) { }
 
 static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba,
-					  struct ufshcd_lrb *lrbp)
+					  struct scsi_cmnd *cmd)
 {
 	return 0;
 }
 
 static inline void ufshcd_crypto_clear_prdt(struct ufs_hba *hba,
-					    struct ufshcd_lrb *lrbp) { }
+					    struct scsi_cmnd *cmd)
+{
+}
 
 static inline bool ufshcd_crypto_enable(struct ufs_hba *hba)
 {
drivers/ufs/core/ufshcd-priv.h +39 -2

53 lines not on this page · raw message

drivers/ufs/core/ufshcd.c +122 -113

632 lines not on this page · raw message

include/ufs/ufshcd.h +0 -5
index 3e71d691d731..07d3255e02af 100644
@@ -161,7 +161,6 @@ struct ufs_pm_lvl_states {
  * @ucd_prdt_dma_addr: PRDT dma address for debug
  * @ucd_rsp_dma_addr: UPIU response dma address for debug
  * @ucd_req_dma_addr: UPIU request dma address for debug
- * @cmd: pointer to SCSI command
  * @scsi_status: SCSI status of the command
  * @command_type: SCSI, UFS, Query.
  * @task_tag: Task tag of the command
@@ -186,7 +185,6 @@ struct ufshcd_lrb {
 	dma_addr_t ucd_rsp_dma_addr;
 	dma_addr_t ucd_prdt_dma_addr;
 
-	struct scsi_cmnd *cmd;
 	int scsi_status;
 
 	int command_type;
@@ -861,7 +859,6 @@ enum ufshcd_mcq_opr {
  * @spm_lvl: desired UFS power management level during system PM.
  * @pm_op_in_progress: whether or not a PM operation is in progress.
  * @ahit: value of Auto-Hibernate Idle Timer register.
- * @lrb: local reference block
  * @outstanding_tasks: Bits representing outstanding task requests
  * @outstanding_lock: Protects @outstanding_reqs.
  * @outstanding_reqs: Bits representing outstanding transfer requests
@@ -1005,8 +1002,6 @@ struct ufs_hba {
 	/* Auto-Hibernate Idle Timer register value */
 	u32 ahit;
 
-	struct ufshcd_lrb *lrb;
-
 	unsigned long outstanding_tasks;
 	spinlock_t outstanding_lock;
 	unsigned long outstanding_reqs;
author reviewer
#46 lore raw

PATCH v2 27/30 ufs: core: Remove the ufshcd_lrb task_tag member

In reply to Bart Van Assche

Remove the ufshcd_lrb task_tag member and use scsi_cmd_to_rq(cmd)->tag instead. Use rq->tag instead of lrbp->task_tag.

---
 drivers/ufs/core/ufshcd.c | 64 +++++++++++++++++++--------------------
 include/ufs/ufshcd.h      |  1 -
 2 files changed, 32 insertions(+), 33 deletions(-)
drivers/ufs/core/ufshcd.c +32 -32
index a0ff91205d68..9b021567dbdb 100644
@@ -601,7 +601,7 @@ static void ufshcd_print_tr(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 			    bool pr_prdt)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
-	const int tag = lrbp->task_tag;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	int prdt_length;
 
 	dev_err(hba->dev, "UPIU[%d] - issue time %lld us\n",
@@ -2361,6 +2361,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 				       struct ufs_hw_queue *hwq)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	unsigned long flags;
 
 	lrbp->issue_time_stamp = ktime_get();
@@ -2387,11 +2388,10 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 	} else {
 		spin_lock_irqsave(&hba->outstanding_lock, flags);
 		if (hba->vops && hba->vops->setup_xfer_req)
-			hba->vops->setup_xfer_req(hba, lrbp->task_tag,
+			hba->vops->setup_xfer_req(hba, tag,
 						  ufshcd_is_scsi_cmd(cmd));
-		__set_bit(lrbp->task_tag, &hba->outstanding_reqs);
-		ufshcd_writel(hba, 1 << lrbp->task_tag,
-			      REG_UTP_TRANSFER_REQ_DOOR_BELL);
+		__set_bit(tag, &hba->outstanding_reqs);
+		ufshcd_writel(hba, 1 << tag, REG_UTP_TRANSFER_REQ_DOOR_BELL);
 		spin_unlock_irqrestore(&hba->outstanding_lock, flags);
 	}
 }
@@ -2788,6 +2788,7 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd,
 					     u8 upiu_flags)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
 	unsigned short cdb_len;
 
@@ -2795,11 +2796,11 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd,
 		.transaction_code = UPIU_TRANSACTION_COMMAND,
 		.flags = upiu_flags,
 		.lun = lrbp->lun,
-		.task_tag = lrbp->task_tag,
+		.task_tag = tag,
 		.command_set_type = UPIU_COMMAND_SET_TYPE_SCSI,
 	};
 
-	WARN_ON_ONCE(ucd_req_ptr->header.task_tag != lrbp->task_tag);
+	WARN_ON_ONCE(ucd_req_ptr->header.task_tag != tag);
 
 	ucd_req_ptr->sc.exp_data_transfer_len = cpu_to_be32(cmd->sdb.length);
 
@@ -2819,6 +2820,8 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,
 				struct ufshcd_lrb *lrbp, u8 upiu_flags)
 {
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
+	struct scsi_cmnd *cmd = (struct scsi_cmnd *)lrbp - 1;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	struct ufs_query *query = &hba->dev_cmd.query;
 	u16 len = be16_to_cpu(query->request.upiu_req.length);
 
@@ -2827,7 +2830,7 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,
 		.transaction_code = UPIU_TRANSACTION_QUERY_REQ,
 		.flags = upiu_flags,
 		.lun = lrbp->lun,
-		.task_tag = lrbp->task_tag,
+		.task_tag = tag,
 		.query_function = query->request.query_func,
 		/* Data segment length only need for WRITE_DESC */
 		.data_segment_length =
@@ -2849,12 +2852,14 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,
 static inline void ufshcd_prepare_utp_nop_upiu(struct ufshcd_lrb *lrbp)
 {
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
+	struct scsi_cmnd *cmd = (struct scsi_cmnd *)lrbp - 1;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 
 	memset(ucd_req_ptr, 0, sizeof(struct utp_upiu_req));
 
 	ucd_req_ptr->header = (struct utp_upiu_header){
 		.transaction_code = UPIU_TRANSACTION_NOP_OUT,
-		.task_tag = lrbp->task_tag,
+		.task_tag = tag,
 	};
 }
 
@@ -2938,7 +2943,6 @@ static void __ufshcd_setup_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 
 	memset(lrbp->ucd_req_ptr, 0, sizeof(*lrbp->ucd_req_ptr));
 
-	lrbp->task_tag = tag;
 	lrbp->lun = lun;
 	ufshcd_prepare_lrbp_crypto(cmd ? scsi_cmd_to_rq(cmd) : NULL, lrbp);
 }
@@ -3237,6 +3241,8 @@ ufshcd_dev_cmd_completion(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 		struct ufshcd_lrb *lrbp, int max_timeout)
 {
+	struct scsi_cmnd *cmd = (struct scsi_cmnd *)lrbp - 1;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	unsigned long time_left = msecs_to_jiffies(max_timeout);
 	unsigned long flags;
 	bool pending;
@@ -3253,18 +3259,18 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 	} else {
 		err = -ETIMEDOUT;
 		dev_dbg(hba->dev, "%s: dev_cmd request timedout, tag %d\n",
-			__func__, lrbp->task_tag);
+			__func__, tag);
 
 		/* MCQ mode */
 		if (hba->mcq_enabled) {
 			/* successfully cleared the command, retry if needed */
-			if (ufshcd_clear_cmd(hba, lrbp->task_tag) == 0)
+			if (ufshcd_clear_cmd(hba, tag) == 0)
 				err = -EAGAIN;
 			return err;
 		}
 
 		/* SDB mode */
-		if (ufshcd_clear_cmd(hba, lrbp->task_tag) == 0) {
+		if (ufshcd_clear_cmd(hba, tag) == 0) {
 			/* successfully cleared the command, retry if needed */
 			err = -EAGAIN;
 			/*
@@ -3273,11 +3279,9 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 			 * variable.
 			 */
 			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			pending = test_bit(lrbp->task_tag,
-					   &hba->outstanding_reqs);
+			pending = test_bit(tag, &hba->outstanding_reqs);
 			if (pending)
-				__clear_bit(lrbp->task_tag,
-					    &hba->outstanding_reqs);
+				__clear_bit(tag, &hba->outstanding_reqs);
 			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
 
 			if (!pending) {
@@ -3290,11 +3294,10 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 			}
 		} else {
 			dev_err(hba->dev, "%s: failed to clear tag %d\n",
-				__func__, lrbp->task_tag);
+				__func__, tag);
 
 			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			pending = test_bit(lrbp->task_tag,
-					   &hba->outstanding_reqs);
+			pending = test_bit(tag, &hba->outstanding_reqs);
 			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
 
 			if (!pending) {
@@ -5448,6 +5451,7 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,
 					     struct cq_entry *cqe)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	int result = 0;
 	int scsi_status;
 	enum utp_ocs ocs;
@@ -5519,10 +5523,8 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,
 	case OCS_ABORTED:
 	case OCS_INVALID_COMMAND_STATUS:
 		result |= DID_REQUEUE << 16;
-		dev_warn(hba->dev,
-				"OCS %s from controller for tag %d\n",
-				(ocs == OCS_ABORTED ? "aborted" : "invalid"),
-				lrbp->task_tag);
+		dev_warn(hba->dev, "OCS %s from controller for tag %d\n",
+			 ocs == OCS_ABORTED ? "aborted" : "invalid", tag);
 		break;
 	case OCS_INVALID_CMD_TABLE_ATTR:
 	case OCS_INVALID_PRDT_ATTR:
@@ -5535,9 +5537,8 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,
 	case OCS_GENERAL_CRYPTO_ERROR:
 	default:
 		result |= DID_ERROR << 16;
-		dev_err(hba->dev,
-				"OCS error from controller = %x for tag %d\n",
-				ocs, lrbp->task_tag);
+		dev_err(hba->dev, "OCS error from controller = %x for tag %d\n",
+			ocs, tag);
 		ufshcd_print_evt_hist(hba);
 		ufshcd_print_host_state(hba);
 		break;
@@ -7654,8 +7655,8 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)
 	u8 resp = 0xF;
 
 	for (poll_cnt = 100; poll_cnt; poll_cnt--) {
-		err = ufshcd_issue_tm_cmd(hba, lrbp->lun, lrbp->task_tag,
-				UFS_QUERY_TASK, &resp);
+		err = ufshcd_issue_tm_cmd(hba, lrbp->lun, tag, UFS_QUERY_TASK,
+					  &resp);
 		if (!err && resp == UPIU_TASK_MANAGEMENT_FUNC_SUCCEEDED) {
 			/* cmd pending in the device */
 			dev_err(hba->dev, "%s: cmd pending in the device. tag = %d\n",
@@ -7688,8 +7689,7 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)
 	if (!poll_cnt)
 		return -EBUSY;
 
-	err = ufshcd_issue_tm_cmd(hba, lrbp->lun, lrbp->task_tag,
-			UFS_ABORT_TASK, &resp);
+	err = ufshcd_issue_tm_cmd(hba, lrbp->lun, tag, UFS_ABORT_TASK, &resp);
 	if (err || resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) {
 		if (!err) {
 			err = resp; /* service response error */
@@ -7799,7 +7799,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 		goto release;
 	}
 
-	err = ufshcd_try_to_abort_task(hba, lrbp->task_tag);
+	err = ufshcd_try_to_abort_task(hba, tag);
 	if (err) {
 		dev_err(hba->dev, "%s: failed with err %d\n", __func__, err);
 		ufshcd_set_req_abort_skip(hba, hba->outstanding_reqs);
include/ufs/ufshcd.h +0 -1
index 07d3255e02af..972cab87244a 100644
@@ -188,7 +188,6 @@ struct ufshcd_lrb {
 	int scsi_status;
 
 	int command_type;
-	int task_tag;
 	u8 lun; /* UPIU LUN id field is only 8-bit wide */
 	bool intr_cmd;
 	ktime_t issue_time_stamp;
author reviewer
#47 lore raw

PATCH v2 28/30 ufs: core: Initialize the 'hwq' variable earlier

In reply to Bart Van Assche

In ufshcd_queuecommand(), initialize the 'hwq' variable earlier. This patch prepares for adding more code that uses this variable. No functionality has been changed.

---
 drivers/ufs/core/ufshcd.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +4 -5
index 9b021567dbdb..56af37706155 100644
@@ -3025,10 +3025,12 @@ static int ufshcd_init_cmd_priv(struct Scsi_Host *host, struct scsi_cmnd *cmd)
  */
 static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 {
+	struct request *rq = scsi_cmd_to_rq(cmd);
 	struct ufs_hba *hba = shost_priv(host);
-	int tag = scsi_cmd_to_rq(cmd)->tag;
+	struct ufs_hw_queue *hwq =
+		hba->mcq_enabled ? ufshcd_mcq_req_to_hwq(hba, rq) : NULL;
+	int tag = rq->tag;
 	int err = 0;
-	struct ufs_hw_queue *hwq = NULL;
 
 	switch (hba->ufshcd_state) {
 	case UFSHCD_STATE_OPERATIONAL:
@@ -3086,9 +3088,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 		goto out;
 	}
 
-	if (hba->mcq_enabled)
-		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
-
 	ufshcd_send_command(hba, cmd, hwq);
 
 out:
author reviewer
#48 lore raw

PATCH v2 29/30 ufs: core: Make blk_mq_tagset_busy_iter() skip reserved requests

In reply to Bart Van Assche

A later patch will convert hba->reserved_slot into a reserved tag. Make blk_mq_tagset_busy_iter() skip reserved requests such that device management commands are skipped.

---
 drivers/ufs/core/ufshcd.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +8 -4
index 56af37706155..84875a6aee4e 100644
@@ -644,7 +644,8 @@ static bool ufshcd_print_tr_iter(struct request *req, void *priv)
 	struct Scsi_Host *shost = sdev->host;
 	struct ufs_hba *hba = shost_priv(shost);
 
-	ufshcd_print_tr(hba, blk_mq_rq_to_pdu(req), *(bool *)priv);
+	if (!blk_mq_is_reserved_rq(req))
+		ufshcd_print_tr(hba, blk_mq_rq_to_pdu(req), *(bool *)priv);
 
 	return true;
 }
@@ -5730,7 +5731,7 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
 	struct ufs_hba *hba = shost_priv(shost);
 	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
 
-	if (!hwq)
+	if (blk_mq_is_reserved_rq(rq) || !hwq)
 		return true;
 
 	ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
@@ -5757,7 +5758,7 @@ static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)
 	struct ufs_hba *hba = shost_priv(shost);
 	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
 
-	if (hwq)
+	if (!blk_mq_is_reserved_rq(rq) && hwq)
 		ufshcd_mcq_poll_cqe_lock(hba, hwq);
 
 	return true;
@@ -6624,6 +6625,9 @@ static bool ufshcd_abort_one(struct request *rq, void *priv)
 	struct Scsi_Host *shost = sdev->host;
 	struct ufs_hba *hba = shost_priv(shost);
 
+	if (blk_mq_is_reserved_rq(rq))
+		return true;
+
 	*ret = ufshcd_try_to_abort_task(hba, tag);
 	dev_err(hba->dev, "Aborting tag %d / CDB %#02x %s\n", tag,
 		ufshcd_is_scsi_cmd(cmd) ? cmd->cmnd[0] : -1,
@@ -7560,7 +7564,7 @@ static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)
 	const u64 lun = *(u64 *)priv;
 	const u32 tag = req->tag;
 
-	if (sdev->lun != lun)
+	if (blk_mq_is_reserved_rq(req) || sdev->lun != lun)
 		return true;
 
 	if (ufshcd_clear_cmd(hba, tag) < 0) {
author reviewer
#49 lore raw

PATCH v2 30/30 ufs: core: Switch to scsi_get_internal_cmd()

In reply to Bart Van Assche

Instead of storing the tag of the reserved command in hba->reserved_slot, use scsi_get_internal_cmd() and scsi_put_internal_cmd() to allocate the tag for the reserved command dynamically. Add support in ufshcd_queuecommand() for submitting reserved commands. Add support in ufshcd_abort() for device management commands. Use blk_execute_rq() for submitting reserved commands. Remove the code and data structures that became superfluous. This includes ufshcd_wait_for_dev_cmd(), hba->reserved_slot and ufs_dev_cmd.complete.

---
 drivers/ufs/core/ufs-mcq.c     |  23 ++--
 drivers/ufs/core/ufshcd-priv.h |  25 +----
 drivers/ufs/core/ufshcd.c      | 195 ++++++++++++++-------------------
 include/ufs/ufshcd.h           |   6 -
 4 files changed, 92 insertions(+), 157 deletions(-)
drivers/ufs/core/ufs-mcq.c +7 -16
index 1f9ff9fc44c0..ee732a35e0c3 100644
@@ -192,10 +192,6 @@ static int ufshcd_mcq_config_nr_queues(struct ufs_hba *hba)
 		return -EOPNOTSUPP;
 	}
 
-	/*
-	 * Device should support at least one I/O queue to handle device
-	 * commands via hba->dev_cmd_queue.
-	 */
 	if (hba_maxq == poll_queues) {
 		dev_err(hba->dev, "At least one non-poll queue required\n");
 		return -EOPNOTSUPP;
@@ -482,9 +478,6 @@ int ufshcd_mcq_init(struct ufs_hba *hba)
 		mutex_init(&hwq->sq_mutex);
 	}
 
-	/* The very first HW queue serves device commands */
-	hba->dev_cmd_queue = &hba->uhq[0];
-
 	host->host_tagset = 1;
 	return 0;
 }
@@ -539,6 +532,7 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 {
 	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, task_tag);
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	struct request *rq = scsi_cmd_to_rq(cmd);
 	struct ufs_hw_queue *hwq;
 	void __iomem *reg, *opr_sqd_base;
 	u32 nexus, id, val;
@@ -547,15 +541,12 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 	if (hba->quirks & UFSHCD_QUIRK_MCQ_BROKEN_RTC)
 		return -ETIMEDOUT;
 
-	if (task_tag != hba->reserved_slot) {
-		if (!cmd)
-			return -EINVAL;
-		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
-		if (!hwq)
-			return 0;
-	} else {
-		hwq = hba->dev_cmd_queue;
-	}
+	if (!cmd)
+		return -EINVAL;
+
+	hwq = ufshcd_mcq_req_to_hwq(hba, rq);
+	if (!hwq)
+		return 0;
 
 	id = hwq->id;
 
drivers/ufs/core/ufshcd-priv.h +1 -24
index e56111d8e7ec..92462204a227 100644
@@ -367,30 +367,7 @@ static inline bool ufs_is_valid_unit_desc_lun(struct ufs_dev_info *dev_info, u8
 static inline struct scsi_cmnd *ufshcd_tag_to_cmd(struct ufs_hba *hba, u32 tag)
 {
 	struct blk_mq_tags *tags = hba->host->tag_set.shared_tags;
-	struct request *rq;
-
-	/*
-	 * Handle reserved tags differently because the UFS driver does not
-	 * call blk_mq_alloc_request() for allocating reserved requests.
-	 * Allocating reserved tags with blk_mq_alloc_request() would require
-	 * the following:
-	 * - Allocate an additional request queue from &hba->host->tag_set for
-	 *   allocating reserved requests from.
-	 * - For that request queue, allocate a SCSI device.
-	 * - Calling blk_mq_alloc_request(hba->dev_mgmt_queue, REQ_OP_DRV_OUT,
-	 *   BLK_MQ_REQ_RESERVED) for allocating a reserved request and
-	 *   blk_mq_free_request() for freeing reserved requests.
-	 * - Set the .device pointer for these reserved requests.
-	 * - Submit reserved requests with blk_execute_rq().
-	 * - Modify ufshcd_queuecommand() such that it handles reserved requests
-	 *   in another way than SCSI requests.
-	 * - Modify ufshcd_compl_one_cqe() such that it calls scsi_done() for
-	 *   device management commands.
-	 * - Modify all callback functions called by blk_mq_tagset_busy_iter()
-	 *   calls in the UFS driver and skip device management commands.
-	 */
-	rq = tag < UFSHCD_NUM_RESERVED ? tags->static_rqs[tag] :
-					 blk_mq_tag_to_rq(tags, tag);
+	struct request *rq = blk_mq_tag_to_rq(tags, tag);
 
 	if (WARN_ON_ONCE(!rq))
 		return NULL;
drivers/ufs/core/ufshcd.c +84 -111

334 lines not on this page · raw message

include/ufs/ufshcd.h +0 -6
index 972cab87244a..cfc7226cab3c 100644
@@ -237,13 +237,11 @@ struct ufs_query {
  * struct ufs_dev_cmd - all assosiated fields with device management commands
  * @type: device management command type - Query, NOP OUT
  * @lock: lock to allow one command at a time
- * @complete: internal commands completion
  * @query: Device management query information
  */
 struct ufs_dev_cmd {
 	enum dev_cmd_type type;
 	struct mutex lock;
-	struct completion complete;
 	struct ufs_query query;
 };
 
@@ -866,7 +864,6 @@ enum ufshcd_mcq_opr {
  * @nutrs: Transfer Request Queue depth supported by controller
  * @nortt - Max outstanding RTTs supported by controller
  * @nutmrs: Task Management Queue depth supported by controller
- * @reserved_slot: Used to submit device commands. Protected by @dev_cmd.lock.
  * @ufs_version: UFS Version to which controller complies
  * @vops: pointer to variant specific operations
  * @vps: pointer to variant specific parameters
@@ -957,7 +954,6 @@ enum ufshcd_mcq_opr {
  * @res: array of resource info of MCQ registers
  * @mcq_base: Multi circular queue registers base address
  * @uhq: array of supported hardware queues
- * @dev_cmd_queue: Queue for issuing device management commands
  * @mcq_opr: MCQ operation and runtime registers
  * @ufs_rtc_update_work: A work for UFS RTC periodic update
  * @pm_qos_req: PM QoS request handle
@@ -1010,7 +1006,6 @@ struct ufs_hba {
 	int nortt;
 	u32 mcq_capabilities;
 	int nutmrs;
-	u32 reserved_slot;
 	u32 ufs_version;
 	const struct ufs_hba_variant_ops *vops;
 	struct ufs_hba_variant_params *vps;
@@ -1129,7 +1124,6 @@ struct ufs_hba {
 	struct ufshcd_res_info res[RES_MAX];
 	void __iomem *mcq_base;
 	struct ufs_hw_queue *uhq;
-	struct ufs_hw_queue *dev_cmd_queue;
 	struct ufshcd_mcq_opr_info_t mcq_opr[OPR_MAX];
 
 	struct delayed_work ufs_rtc_update_work;
#50 lore raw
In reply to Bart Van Assche
Quoted 5 lines · Bart Van Assche

Hi Martin,

This patch series optimizes the hot path of the UFS driver by making struct scsi_cmnd and struct ufshcd_lrb adjacent. Making these two data structures adjacent is realized as follows:

what is the baseline for this series? It does not apply to v6.17-rc1 or mkp-scsi 6.18 queue. Sometimes it's better to apply the changes when checking them.

Quoted 13 lines
(quoted diff) +1 -0
@@ -9040,6 +9046,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
      .name            = UFSHCD,
      .proc_name        = UFSHCD,
      .map_queues        = ufshcd_map_queues,
+    .cmd_size        = sizeof(struct ufshcd_lrb),
      .init_cmd_priv        = ufshcd_init_cmd_priv,
      .queuecommand        = ufshcd_queuecommand,
      .mq_poll        = ufshcd_poll,

The following changes had to be made prior to making these two data structures adjacent:
* Add support for driver-internal and reserved commands in the SCSI core.

#52 lore raw

On 13/08/2025 16:06, Bart Van Assche wrote:

+

Quoted 11 lines · John Garry
Quoted 1 line

what is the baseline for this series? It does not apply to v6.17-rc1 or mkp-scsi 6.18 queue.

Hi John,

The baseline for this series is Martin's mkp-scsi/for-next branch with this patch series applied on top: "[PATCH 0/4] UFS driver bug fixes"
(https://lore.kernel.org/linux-scsi/20250811154711.394297-1- [email protected]/).

See also https://github.com/bvanassche/linux/tree/ufs-lrbp-as-priv-data.

thanks

Some further initial points on this series:
- the driver still has the separate tmf tag set. Why cannot the tmf be allocated as a reserved command in the shost tagset?

- I like that you are using blk_execute_rq(), but why do we need the pseudo sdev (and not the ufs sdev)? The idea of the psuedo sdev was originally for sending reserved commands for the host.

- IIRC, I was advised to have a check in the scsi core dispatch command patch to check for a reserved command, and have a separate handler for that, i.e. don't use sht->queuecommand for reserved commands. I can try to find the exact discussion if you like.

Thanks,
John

author reviewer
#53 lore raw
Quoted 3 lines · John Garry

Some further initial points on this series:
- the driver still has the separate tmf tag set. Why cannot the tmf be allocated as a reserved command in the shost tagset?

The UFSHCI standard defines different tag spaces for TMFs and for UFS
commands: [0..hba->nutmrs) for TMFs and [0..hba->nutrs) for UFS commands. These ranges have to be kept separate because of how UFS host controllers use these numbers. The TMF tag is passed to UFS host controllers by writing it into a bitwise register. From __ufshcd_issue_tm_cmd():

ufshcd_writel(hba, 1 << task_tag, REG_UTP_TASK_REQ_DOOR_BELL);

The UFS command tag ends up in a protocol header. From ufshcd_prepare_utp_scsi_cmd_upiu():

	ucd_req_ptr->header = (struct utp_upiu_header){
		[ ... ]
		.task_tag = lrbp->task_tag,
		[ ... ]
	};

Mixing up TMFs and the reserved UFS command and allocating all these from a single tag space would require a additional code and data structures to translate reserved tags from the [0..hba->nutmrs] range to the [0..hba->nutmrs) range. In other words, code would become more complex and harder to maintain. Hence, I prefer to keep the TMF tag set.

Quoted 2 lines

- I like that you are using blk_execute_rq(), but why do we need the
pseudo sdev (and not the ufs sdev)? The idea of the psuedo sdev was originally for sending reserved commands for the host.

In the UFS driver several reserved commands are sent before
ufshcd_scsi_add_wlus() and scsi_scan_host() are called. Hence, the
pseudo sdev is the only sdev that is available when sending reserved
commands like the initial NOP OUT. Allocating the well-known LUNs before
sending the initial NOP OUT is not possible because ufshcd_sdev_init() gets called while adding WLUNs and there is code in that function that is based on the assumption that UFS device initialization has completed (ufshcd_lu_init() calls ufshcd_read_unit_desc_param()).

Quoted 2 lines

- IIRC, I was advised to have a check in the scsi core dispatch command
patch to check for a reserved command, and have a separate handler for that, i.e. don't use sht->queuecommand for reserved commands. I can try to find the exact discussion if you like.

It would be appreciated if a link to that conversation could be shared.

Thanks,

Bart.

#54 lore raw
Quoted 22 lines · Bart Van Assche
Quoted 3 lines · John Garry

Some further initial points on this series:
- the driver still has the separate tmf tag set. Why cannot the tmf be allocated as a reserved command in the shost tagset?

The UFSHCI standard defines different tag spaces for TMFs and for UFS
commands: [0..hba->nutmrs) for TMFs and [0..hba->nutrs) for UFS commands. These ranges have to be kept separate because of how UFS host controllers use these numbers. The TMF tag is passed to UFS host controllers by writing it into a bitwise register. From __ufshcd_issue_tm_cmd():

     ufshcd_writel(hba, 1 << task_tag, REG_UTP_TASK_REQ_DOOR_BELL);

The UFS command tag ends up in a protocol header. From ufshcd_prepare_utp_scsi_cmd_upiu():

    ucd_req_ptr->header = (struct utp_upiu_header){ [ ... ]
        .task_tag = lrbp->task_tag,
        [ ... ]
    };

Quoted 6 lines

Mixing up TMFs and the reserved UFS command and allocating all these from a single tag space would require a additional code and data structures to translate reserved tags from the [0..hba->nutmrs] range to the [0..hba->nutmrs) range. In other words, code would become more complex and harder to maintain. Hence, I prefer to keep the TMF tag set.

Maybe so. But it is still less than ideal how the TMF tags are managed in the UFS driver, specifically having a stub in ufshcd_tmf_ops. Have you considered modelling on how the NVMe driver manages the admin queue?

Quoted 5 lines
Quoted 2 lines

- I like that you are using blk_execute_rq(), but why do we need the
pseudo sdev (and not the ufs sdev)? The idea of the psuedo sdev was originally for sending reserved commands for the host.

In the UFS driver several reserved commands are sent before ufshcd_scsi_add_wlus() and scsi_scan_host() are called.

If you must send "host" reserved commands (which are not for a specific SCSI target), then it would be ok.

But do you need to send any "reserved" commands to a specific SCSI target (which are not TMFs)?

Quoted 10 lines

Hence, the
pseudo sdev is the only sdev that is available when sending reserved
commands like the initial NOP OUT. Allocating the well-known LUNs before
sending the initial NOP OUT is not possible because ufshcd_sdev_init() gets called while adding WLUNs and there is code in that function that is based on the assumption that UFS device initialization has completed (ufshcd_lu_init() calls ufshcd_read_unit_desc_param()).

Quoted 2 lines

- IIRC, I was advised to have a check in the scsi core dispatch
command patch to check for a reserved command, and have a separate handler for that, i.e. don't use sht->queuecommand for reserved commands. I can try to find the exact discussion if you like.

It would be appreciated if a link to that conversation could be shared.

I looked, but I could not find it. It may have been a private conversation while at my old employer (so now lost).

Anyway, here is a reference implementation:
https://lore.kernel.org/linux-scsi/[email protected]/

Thanks,
John

author reviewer
#55 lore raw
Quoted 1 line · John Garry

Maybe so. But it is still less than ideal how the TMF tags are managed in the UFS driver, specifically having a stub in ufshcd_tmf_ops. Have you considered modelling on how the NVMe driver manages the admin queue?

The PCIe NVMe driver has different tag sets for the admin queue and I/O queues. Additionally, it allocates separate request queues for admin commands and for I/O commands, just like the UFS driver. Can you be more specific about why you are referring to the NVMe driver?

Regarding the stub in ufshcd_tmf_ops: moving the code for queueing a TMF into ufshcd_queue_tmf() and switching to blk_execute_rq() for submitting TMFs shouldn't be that hard. However, I consider that change as out-of-scope for this patch series.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

- I like that you are using blk_execute_rq(), but why do we need the
pseudo sdev (and not the ufs sdev)? The idea of the psuedo sdev was originally for sending reserved commands for the host.

In the UFS driver several reserved commands are sent before ufshcd_scsi_add_wlus() and scsi_scan_host() are called.

If you must send "host" reserved commands (which are not for a specific SCSI target), then it would be ok.

But do you need to send any "reserved" commands to a specific SCSI target (which are not TMFs)?

In this patch series, reserved commands are used for communicating with the UFS device. It doesn't matter which struct scsi_device instance is used for sending reserved commands since UFS host controllers only support a single UFS device.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

- IIRC, I was advised to have a check in the scsi core dispatch
command patch to check for a reserved command, and have a separate handler for that, i.e. don't use sht->queuecommand for reserved commands. I can try to find the exact discussion if you like.

It would be appreciated if a link to that conversation could be shared.

I looked, but I could not find it. It may have been a private conversation while at my old employer (so now lost).

Anyway, here is a reference implementation:
https://lore.kernel.org/linux-scsi/1666693096-180008-5-git-send-email- [email protected]/

The description of that patch says what the patch does but not why the .reserved_queuecommand() function pointer is introduced. Integrating that patch into this UFS kernel driver series wouldn't simplify any code but would lead to some code duplication. Hence, I will leave it to someone else to integrate that patch in the upstream kernel.

Thanks,

Bart.

#56 lore raw
Quoted 8 lines · Bart Van Assche
Quoted 1 line · John Garry

Maybe so. But it is still less than ideal how the TMF tags are managed in the UFS driver, specifically having a stub in ufshcd_tmf_ops. Have you considered modelling on how the NVMe driver manages the admin queue?

The PCIe NVMe driver has different tag sets for the admin queue and I/O queues. Additionally, it allocates separate request queues for admin commands and for I/O commands, just like the UFS driver. Can you be more specific about why you are referring to the NVMe driver?

Quoted 4 lines

Regarding the stub in ufshcd_tmf_ops: moving the code for queueing a TMF into ufshcd_queue_tmf() and switching to blk_execute_rq() for submitting TMFs shouldn't be that hard. However, I consider that change as out-of-scope for this patch series.

That is really the change which I am hinting at here.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

- I like that you are using blk_execute_rq(), but why do we need the
pseudo sdev (and not the ufs sdev)? The idea of the psuedo sdev was originally for sending reserved commands for the host.

In the UFS driver several reserved commands are sent before ufshcd_scsi_add_wlus() and scsi_scan_host() are called.

Quoted 8 lines
Quoted 3 lines

If you must send "host" reserved commands (which are not for a specific SCSI target), then it would be ok.

But do you need to send any "reserved" commands to a specific SCSI target (which are not TMFs)?

In this patch series, reserved commands are used for communicating with the UFS device. It doesn't matter which struct scsi_device instance is used for sending reserved commands since UFS host controllers only support a single UFS device.

For more general scsi reserved commands support, we have to consider possibility of more than one target sdev in future.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

- IIRC, I was advised to have a check in the scsi core dispatch
command patch to check for a reserved command, and have a separate handler for that, i.e. don't use sht->queuecommand for reserved commands. I can try to find the exact discussion if you like.

It would be appreciated if a link to that conversation could be shared.

Quoted 7 lines
Quoted 4 lines

I looked, but I could not find it. It may have been a private conversation while at my old employer (so now lost).

Anyway, here is a reference implementation:
https://lore.kernel.org/linux-scsi/1666693096-180008-5-git-send-email- [email protected]/

The description of that patch says what the patch does but not why the .reserved_queuecommand() function pointer is introduced.

Motivation:

Any driver which supports reserved commands will need to have a check for reserved command in the .queuecommand callback (for special handling). In addition, some more general reserved command handling will prob need to be added to scsi_dispatch_cmd().

As such, having common handling for reserved commands in scsi_dispatch_cmd() makes sense (so that each LLD does not have to duplicate handling).

Quoted 5 lines

Integrating
that patch into this UFS kernel driver series wouldn't simplify any code but would lead to some code duplication. Hence, I will leave it to someone else to integrate that patch in the upstream kernel.

Thanks,
John

author reviewer
#57 lore raw
Quoted 5 lines · John Garry
Quoted 4 lines · Bart Van Assche
Quoted 2 lines · John Garry

Anyway, here is a reference implementation:
https://lore.kernel.org/linux-scsi/1666693096-180008-5-git-send- email- [email protected]/

Quoted 8 lines
Quoted 2 lines

The description of that patch says what the patch does but not why the .reserved_queuecommand() function pointer is introduced.

Motivation:

Any driver which supports reserved commands will need to have a check for reserved command in the .queuecommand callback (for special handling). In addition, some more general reserved command handling will prob need to be added to scsi_dispatch_cmd().

As such, having common handling for reserved commands in scsi_dispatch_cmd() makes sense (so that each LLD does not have to duplicate handling).

Hi John,

Thanks for having explained the purpose of that patch. Although that patch will make it harder to integrate .queue_rqs() support in the SCSI core, I will look into integrating that patch in this patch series.

Thanks,

Bart.

author reviewer
#58 lore raw
In reply to John Garry
Quoted 2 lines · John Garry

Anyway, here is a reference implementation:
https://lore.kernel.org/linux-scsi/1666693096-180008-5-git-send-email- [email protected]/

How about using the (lightly tested) patch below instead of that patch? The patch below has the advantage that it doesn't duplicate any SCSI core code. Additionally, it preserves the logic for initializing driver- private data for reserved commands.

Thanks,

Bart.

[PATCH] scsi: core: Bypass the queue limit checks for reserved commands

Reserved commands can be TMFs or commands allocated and submitted by the LLD. These commands can be SCSI commands or non-SCSI commands. For all these cases, bypass the SCSI host, target and device queue limit checks for reserved commands. Additionally, do not activate the SCSI error handler if a reserved command fails.

[ bvanassche: modified patch title and patch description. Renamed
.reserved_queuecommand() into .queue_reserved_command(). Changed the second argument of __blk_mq_end_request() from 0 into error code in the completion path if cmd->result != 0. Rewrote the scsi_queue_rq() changes. See also

https://lore.kernel.org/linux-scsi/[email protected]/ ]

---
 drivers/scsi/hosts.c     |  6 +++++
 drivers/scsi/scsi_lib.c  | 54 ++++++++++++++++++++++++++++------------
 include/scsi/scsi_host.h |  7 ++++++
 3 files changed, 51 insertions(+), 16 deletions(-)
drivers/scsi/hosts.c +6 -0
index e860ac93499d..75fe624366c3 100644
@@ -231,6 +231,12 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
 		goto fail;
 	}
 
+	if (shost->nr_reserved_cmds && !sht->queue_reserved_command) {
+		shost_printk(KERN_ERR, shost,
+			     "nr_reserved_cmds set but no method to queue\n");
+		goto fail;
+	}
+
 	/* Use min_t(int, ...) in case shost->can_queue exceeds SHRT_MAX */
 	shost->cmd_per_lun = min_t(int, shost->cmd_per_lun,
 				   shost->can_queue);
drivers/scsi/scsi_lib.c +38 -16

75 lines not on this page · raw message

include/scsi/scsi_host.h +7 -0
index 3b5150759c44..a615dcaa0ae8 100644
@@ -86,6 +86,13 @@ struct scsi_host_template {
 	 */
 	int (* queuecommand)(struct Scsi_Host *, struct scsi_cmnd *);
 
+	/*
+	 * Queue a reserved command (BLK_MQ_REQ_RESERVED). The .queuecommand()
+	 * documentation also applies to the .queue_reserved_command() callback.
+	 *
+	 */
+	int (*queue_reserved_command)(struct Scsi_Host *, struct scsi_cmnd *);
+
 	/*
 	 * The commit_rqs function is used to trigger a hardware
 	 * doorbell after some requests have been queued with