Optimize the hot path in the UFS driver
Subsystems (2)
Bart Van Assche
reviewerv3 cover Optimize the hot path in the UFS driver
Hi Martin,
This patch series optimizes the hot path of the UFS driver by making struct scsi_cmnd and struct ufshcd_lrb adjacent. Making these two data structures adjacent is realized as follows:
@@ -9040,6 +9046,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
.name = UFSHCD,
.proc_name = UFSHCD,
.map_queues = ufshcd_map_queues,
+ .cmd_size = sizeof(struct ufshcd_lrb),
.init_cmd_priv = ufshcd_init_cmd_priv,
.queuecommand = ufshcd_queuecommand,
.mq_poll = ufshcd_poll,
The following changes had to be made prior to making these two data structures adjacent:
* Add support for driver-internal and reserved commands in the SCSI core.
* Instead of making the reserved command slot (hba->reserved_slot) invisible to the SCSI core, let the SCSI core allocate a reserved command.
* Remove all UFS data structure members that are no longer needed because struct scsi_cmnd and struct ufshcd_lrb are now adjacent
* Call ufshcd_init_lrb() from inside ufshcd_queuecommand() instead of calling this function before I/O starts. This is necessary because ufshcd_memory_alloc() allocates fewer instances than the block layer allocates requests. See also the following code in the block layer core:
if (blk_mq_init_request(set, hctx->fq->flush_rq, hctx_idx,
hctx->numa_node)) Although the UFS driver could be modified such that ufshcd_init_lrb() is called from ufshcd_init_cmd_priv(), realizing this would require moving the memory allocations that happen from inside ufshcd_memory_alloc() into ufshcd_init_cmd_priv(). That would make this patch series even larger. Although ufshcd_init_lrb() is called for each command, the benefits of reduced indirection and better cache efficiency outweigh the small overhead of per-command lrb initialization.
* ufshcd_add_scsi_host() happens now before any device management commands are submitted. This change is necessary because this patch makes device management command allocation happen when the SCSI host is allocated.
* Allocate as many command slots as the host controller supports. Decrease host->cmds_per_lun if necessary once it is clear whether or not the UFS device supports less command slots than the host controller.
Changes compared to v2:
- Removed scsi_host_update_can_queue() and also the UFS driver refactoring patches that were introduced to support this call.
-- Added .queue_reserved_command(). Added ufshcd_queue_reserved_command(). -- Removed a BUG_ON() statement from ufshcd_get_dev_mgmt_cmd(). -- Modified and renamed ufshcd_mcq_decide_queue_depth().
Changes compared to v1:
- Left out the kernel patches related to support for const SCSI command arguments.
- Added SCSI core patches for allocating a pseudo SCSI device and reserved command support.
- Added several kernel patches to switch the UFS driver from a hardcoded reserved slot to calling scsi_get_internal_cmd().
- Enable .alloc_pseudo_sdev in the scsi_debug driver.
*** BLURB HERE ***
Bart Van Assche (22):
scsi: core: Do not allocate a budget token for reserved commands
scsi_debug: Set .alloc_pseudo_sdev
ufs: core: Move an assignment
ufs: core: Change the type of one ufshcd_add_cmd_upiu_trace() argument
ufs: core: Only call ufshcd_add_command_trace() for SCSI commands
ufs: core: Change the type of one ufshcd_add_command_trace() argument
ufs: core: Change the type of one ufshcd_send_command() argument
ufs: core: Only call ufshcd_should_inform_monitor() for SCSI commands
ufs: core: Change the monitor function argument types
ufs: core: Rework ufshcd_mcq_compl_pending_transfer()
ufs: core: Rework ufshcd_eh_device_reset_handler()
ufs: core: Allocate more commands for the SCSI host
ufs: core: Allocate the SCSI host earlier
ufs: core: Call ufshcd_init_lrb() later
ufs: core: Use hba->reserved_slot
ufs: core: Make the reserved slot a reserved request
ufs: core: Do not clear driver-private command data
ufs: core: Optimize the hot path
ufs: core: Pass a SCSI pointer instead of an LRB pointer
ufs: core: Remove the ufshcd_lrb task_tag member
ufs: core: Make blk_mq_tagset_busy_iter() skip reserved requests
ufs: core: Switch to scsi_get_internal_cmd()
Hannes Reinecke (3):
scsi: core: Support allocating reserved commands
scsi: core: Support allocating a pseudo SCSI device
scsi: core: Add scsi_{get,put}_internal_cmd() helpers
John Garry (1):
scsi: core: Bypass the queue limit checks for reserved commands
drivers/scsi/hosts.c | 17 + drivers/scsi/scsi.c | 7 +- drivers/scsi/scsi_debug.c | 1 + drivers/scsi/scsi_lib.c | 151 +++++- drivers/scsi/scsi_priv.h | 2 + drivers/scsi/scsi_scan.c | 70 ++- drivers/scsi/scsi_sysfs.c | 4 +- drivers/ufs/core/ufs-mcq.c | 51 +- drivers/ufs/core/ufshcd-crypto.h | 18 +- drivers/ufs/core/ufshcd-priv.h | 20 +- drivers/ufs/core/ufshcd.c | 802 +++++++++++++++++-------------- include/scsi/scsi_cmnd.h | 2 + include/scsi/scsi_device.h | 23 + include/scsi/scsi_host.h | 37 +- include/ufs/ufshcd.h | 12 - 15 files changed, 763 insertions(+), 454 deletions(-)
Bart Van Assche
reviewerPATCH v3 1/26 scsi: core: Support allocating reserved commands
From: Hannes Reinecke <[email protected]>
Quite some drivers are using management commands internally. These commands typically use the same tag pool as regular SCSI commands. Tags for these management commands are set aside before allocating the block-mq tag bitmap for regular SCSI commands. The block layer already supports this via the reserved tag mechanism. Add a new field 'nr_reserved_cmds' to the SCSI host template to instruct the block layer to set aside a tag space for these management commands by using reserved tags. Exclude reserved commands from .can_queue because .can_queue is visible in sysfs.
[ bvanassche: modified patch title and patch description. Left out the
following statement: "if (sdev->host->nr_reserved_cmds)
flags |= BLK_MQ_REQ_RESERVED;". See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]
--- drivers/scsi/hosts.c | 3 +++ drivers/scsi/scsi_lib.c | 3 ++- include/scsi/scsi_host.h | 22 +++++++++++++++++++++- 3 files changed, 26 insertions(+), 2 deletions(-)
drivers/scsi/hosts.c +3 -0
@@ -499,6 +499,9 @@ struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int privelse shost->dma_boundary =0xffffffff ; +if (sht->nr_reserved_cmds) + shost->nr_reserved_cmds = sht->nr_reserved_cmds; +device_initialize (&shost->shost_gendev);dev_set_name (&shost->shost_gendev,"host%d" , shost->host_no); shost->shost_gendev.bus = &scsi_bus_type;
drivers/scsi/scsi_lib.c +2 -1
@@ -2083,7 +2083,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->ops = &scsi_mq_ops_no_commit; tag_set->nr_hw_queues = shost->nr_hw_queues ? :1 ; tag_set->nr_maps = shost->nr_maps ? :1 ; - tag_set->queue_depth = shost->can_queue; + tag_set->queue_depth = shost->can_queue + shost->nr_reserved_cmds; + tag_set->reserved_tags = shost->nr_reserved_cmds; tag_set->cmd_size = cmd_size; tag_set->numa_node =dev_to_node (shost->dma_dev);if (shost->hostt->tag_alloc_policy_rr)
include/scsi/scsi_host.h +21 -1
@@ -375,10 +375,19 @@ struct scsi_host_template {/* * This determines if we will use a non-interrupt driven * or an interrupt driven scheme. It is set to the maximum number -* of simultaneous commands a single hw queue in HBA will accept. +* of simultaneous commands a single hw queue in HBA will accept +* excluding internal commands. */ int can_queue; +/* +* This determines how many commands the HBA will set aside +* for internal commands. This number will be added to +* @can_queue to calcumate the maximum number of simultaneous +* commands sent to the host. +*/ +int nr_reserved_cmds; +/* * In many instances, especially where disconnect / reconnect are * supported, our host also has an ID on the SCSI bus. If this is
@@ -611,6 +620,11 @@ struct Scsi_Host {unsigned short max_cmd_len;int this_id; + +/* +* Number of commands this host can handle at the same time. +* This excludes reserved commands as specified by nr_reserved_cmds. +*/ int can_queue;short cmd_per_lun;short unsigned int sg_tablesize;
@@ -631,6 +645,12 @@ struct Scsi_Host {*/ unsigned nr_hw_queues;unsigned nr_maps; + +/* +* Number of reserved commands to allocate, if any. +*/ +unsigned int nr_reserved_cmds; + unsigned active_mode:2;/*
Re: [PATCH v3 01/26] scsi: core: Support allocating reserved commands
Quoted 12 lines · Bart Van Assche
From: Hannes Reinecke <[email protected]>
Quite some drivers are using management commands internally. These commands typically use the same tag pool as regular SCSI commands. Tags for these management commands are set aside before allocating the block-mq tag bitmap for regular SCSI commands. The block layer already supports this via the reserved tag mechanism. Add a new field '
nr_reserved_cmds' to the SCSI host template to instruct the block layer to set aside a tag space for these management commands by using reserved tags. Exclude reserved commands from .can_queue because .can_queue is visible in sysfs.
Quoted 7 lines
Signed-off-by Hannes Reinecke <[email protected]>[ bvanassche: modified patch title and patch description. Left out the
following statement: "if (sdev->host->nr_reserved_cmds)
flags |=BLK_MQ_REQ_RESERVED;". See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]Cc John Garry <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
Apart from some small comments, below:
Quoted 11 lines
--- drivers/scsi/hosts.c | 3 +++ drivers/scsi/scsi_lib.c | 3 ++- include/scsi/scsi_host.h | 22 +++++++++++++++++++++- 3 files changed, 26 insertions(+), 2 deletions(-)drivers/scsi/hosts.c +0
-0
drivers/scsi/hosts.c struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int priv · line 503
Quoted 5 lines
drivers/scsi/hosts.c +2
-0@@ -499,6 +499,9 @@ struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int privelse shost->dma_boundary =0xffffffff ; +if (sht->nr_reserved_cmds) + shost->nr_reserved_cmds = sht->nr_reserved_cmds;
small comment: shost->nr_reserved_cmds is always zero-init'ed, so this could be simply:
shost->nr_reserved_cmds = sht->nr_reserved_cmds;
Quoted 9 lines
drivers/scsi/hosts.c +1
-0+device_initialize (&shost->shost_gendev);dev_set_name (&shost->shost_gendev,"host%d" , shost->host_no); shost->shost_gendev.bus = &scsi_bus_type;drivers/scsi/scsi_lib.c +0
-0
drivers/scsi/scsi_lib.c int scsi_mq_setup_tags(struct Scsi_Host *shost) · line 2086
Quoted 5 lines
drivers/scsi/scsi_lib.c +1
-1@@ -2083,7 +2083,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->ops = &scsi_mq_ops_no_commit; tag_set->nr_hw_queues = shost->nr_hw_queues ? :1 ; tag_set->nr_maps = shost->nr_maps ? :1 ; - tag_set->queue_depth = shost->can_queue; + tag_set->queue_depth = shost->can_queue + shost->nr_reserved_cmds;
this seems the proper thing to do, as tag_set->queue_depth matches blk_mq_init_tags() @nr_tags, which is the total
Quoted 12 lines
drivers/scsi/scsi_lib.c +1
-0+ tag_set->reserved_tags = shost->nr_reserved_cmds; tag_set->cmd_size = cmd_size; tag_set->numa_node =dev_to_node (shost->dma_dev);if (shost->hostt->tag_alloc_policy_rr)include/scsi/scsi_host.h +0
-0@@ -375,10 +375,19 @@ struct scsi_host_template {/* * This determines if we will use a non-interrupt driven * or an interrupt driven scheme. It is set to the maximum number
include/scsi/scsi_host.h struct scsi_host_template { · line 386
Quoted 10 lines
include/scsi/scsi_host.h +6
-1-* of simultaneous commands a single hw queue in HBA will accept. +* of simultaneous commands a single hw queue in HBA will accept +* excluding internal commands. */ int can_queue; +/* +* This determines how many commands the HBA will set aside +* for internal commands. This number will be added to +* @can_queue to calcumate the maximum number of simultaneous
calcumate -> calculate
Quoted 20 lines
include/scsi/scsi_host.h +9
-0+* commands sent to the host. +*/ +int nr_reserved_cmds; +/* * In many instances, especially where disconnect / reconnect are * supported, our host also has an ID on the SCSI bus. If this is @@ -611,6 +620,11 @@ struct Scsi_Host {unsigned short max_cmd_len;int this_id; + +/* +* Number of commands this host can handle at the same time. +* This excludes reserved commands as specified by nr_reserved_cmds. +*/ int can_queue;short cmd_per_lun;short unsigned int sg_tablesize;
include/scsi/scsi_host.h struct Scsi_Host { · line 652
Quoted 8 lines
include/scsi/scsi_host.h +5
-0@@ -631,6 +645,12 @@ struct Scsi_Host {*/ unsigned nr_hw_queues;unsigned nr_maps; + +/* +* Number of reserved commands to allocate, if any. +*/ +unsigned int nr_reserved_cmds;
nit: maybe co-locate with can_queue
Quoted 4 lines
include/scsi/scsi_host.h +1
-0+ unsigned active_mode:2;/*
Bart Van Assche
reviewerRe: [PATCH v3 01/26] scsi: core: Support allocating reserved commands
Quoted 14 lines · John Garry
Quoted 10 lines · Bart Van Assche
drivers/scsi/hosts.c +0
-0@@ -499,6 +499,9 @@ struct Scsi_Host *scsi_host_alloc(const struct
scsi_host_template*sht, int priv
else
shost->dma_boundary=0xffffffff;
+ if (sht->nr_reserved_cmds)
+shost->nr_reserved_cmds=sht->nr_reserved_cmds;small comment:
shost->nr_reserved_cmdsis always zero-init'ed, so this could be simply:
Quoted 1 line
shost->nr_reserved_cmds =
sht->nr_reserved_cmds;
Agreed. I will include this change when I repost this patch series.
Quoted 6 lines
Quoted 4 lines
+ /*
+ * Number of reserved commands to allocate, if any. + */
+ unsigned intnr_reserved_cmds;nit: maybe co-locate with
can_queue
That sounds good to me. I will include this change and also the spelling fix for "calculate".
Thanks,
Bart.
Bart Van Assche
reviewerPATCH v3 2/26 scsi: core: Support allocating a pseudo SCSI device
In reply to Bart Van AsscheFrom: Hannes Reinecke <[email protected]>
Add the flag 'alloc_pseudo_sdev' to the SCSI host template and allocate a pseudo SCSI device if the flag is set. This device has the SCSI ID <max_id>:U64_MAX so it won't clash with any devices the LLD might create. It's excluded from scanning and will not show up in sysfs. Additionally, pseudo SCSI devices are skipped by shost_for_each_device(). This prevents that the SCSI error handler tries to submit a reset to a non-existent logical unit.
The intention is to use this device to send internal commands to the storage device.
[ bvanassche: edited patch description / renamed host_sdev into
pseudo_sdev / unexported scsi_get_host_dev() / modified error path in scsi_get_pseudo_dev() / skip pseudo devices in __scsi_iterate_devices() and also when calling sdev_init(), sdev_configure() and sdev_destroy(). See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]
--- drivers/scsi/hosts.c | 8 +++++ drivers/scsi/scsi.c | 7 ++-- drivers/scsi/scsi_priv.h | 2 ++ drivers/scsi/scsi_scan.c | 70 ++++++++++++++++++++++++++++++++++++-- drivers/scsi/scsi_sysfs.c | 4 ++- include/scsi/scsi_device.h | 16 +++++++++ include/scsi/scsi_host.h | 9 +++++ 7 files changed, 110 insertions(+), 6 deletions(-)
drivers/scsi/hosts.c +8 -0
@@ -307,6 +307,14 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,if (error) goto out_del_dev; +if (sht->alloc_pseudo_sdev) { + shost->pseudo_sdev =scsi_get_pseudo_dev (shost); +if (!shost->pseudo_sdev) { + error = -ENOMEM ; + goto out_del_dev; + } + } +scsi_proc_host_add (shost);scsi_autopm_put_host (shost);return error;
drivers/scsi/scsi.c +5 -2
@@ -826,8 +826,11 @@ struct scsi_device *__scsi_iterate_devices(struct Scsi_Host *shost,spin_lock_irqsave (shost->host_lock, flags);while (list->next != &shost->__devices) { next = list_entry(list->next, struct scsi_device, siblings); -/* skip devices that we can't get a reference to */ -if (!scsi_device_get (next)) +/* +* Skip pseudo devices and also devices for which +* scsi_device_get() fails. +*/ +if (!scsi_device_is_pseudo_dev (next) && !scsi_device_get (next))break ; next =NULL ; list = list->next;
drivers/scsi/scsi_priv.h +2 -0
@@ -135,6 +135,8 @@ extern int scsi_complete_async_scans(void);extern int scsi_scan_host_selected (struct Scsi_Host *,unsigned int ,unsigned int ,u64 ,enum scsi_scan_mode );extern void scsi_forget_host (struct Scsi_Host *); +struct scsi_device *scsi_get_pseudo_dev (struct Scsi_Host *); +bool scsi_device_is_pseudo_dev (struct scsi_device *sdev);/* scsi_sysctl.c */ #ifdef CONFIG_SYSCTL
drivers/scsi/scsi_scan.c +67 -3
104 lines not on this page · raw message
drivers/scsi/scsi_sysfs.c +3 -1
@@ -1406,6 +1406,8 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev)int error;struct scsi_target *starget = sdev->sdev_target; +WARN_ON_ONCE (scsi_device_is_pseudo_dev (sdev)); + error =scsi_target_add (starget);if (error)return error;
@@ -1513,7 +1515,7 @@ void __scsi_remove_device(struct scsi_device *sdev)kref_put (&sdev->host->tagset_refcnt, scsi_mq_free_tags);cancel_work_sync (&sdev->requeue_work); -if (sdev->host->hostt->sdev_destroy) +if (!scsi_device_is_pseudo_dev (sdev) && sdev->host->hostt->sdev_destroy) sdev->host->hostt->sdev_destroy (sdev);transport_destroy_device (dev);
include/scsi/scsi_device.h +16 -0
@@ -589,6 +589,22 @@ static inline unsigned int sdev_id(struct scsi_device *sdev)#define scmd_id (scmd) sdev_id((scmd)->device)#define scmd_channel (scmd) sdev_channel((scmd)->device) +/** +* scsi_device_is_pseudo_dev() - Whether a device is a pseudo SCSI device. +* @sdev: SCSI device to examine +* +* A pseudo SCSI device can be used to allocate SCSI commands but does not show +* up in sysfs. Additionally, the logical unit information in *@sdev is made up. +* +* This function tests the LUN number instead of comparing @sdev with +* @sdev->host->pseudo_sdev because this function may be called before +* @sdev->host->pseudo_sdev has been initialized. +*/ +static inline bool scsi_device_is_pseudo_dev (struct scsi_device *sdev) +{ +return sdev->lun ==U64_MAX ; +} +/* * checks for positions of the SCSI state machine */
include/scsi/scsi_host.h +9 -0
@@ -463,6 +463,9 @@ struct scsi_host_template {*/ unsigned emulated:1; +/* True if a pseudo sdev should be allocated */ + unsigned alloc_pseudo_sdev:1; +/* * True if the low-level driver performs its own reset-settle delays. */
@@ -722,6 +725,12 @@ struct Scsi_Host {/* ldm bits */ struct device shost_gendev, shost_dev; +/* +* A SCSI device structure used for sending internal commands to the +* HBA. There is no corresponding logical unit inside the SCSI device. +*/ +struct scsi_device *pseudo_sdev; +/* * Points to the transport data (if any) which is allocated * separately
Re: [PATCH v3 02/26] scsi: core: Support allocating a pseudo SCSI device
Quoted 9 lines
/*
* checks for positions of the SCSI state machine */include/scsi/scsi_host.h +0
-0
include/scsi/scsi_host.h struct scsi_host_template { · line 467
Quoted 5 lines
include/scsi/scsi_host.h +2
-0@@ -463,6 +463,9 @@ struct scsi_host_template {*/ unsigned emulated:1; +/* True if a pseudo sdev should be allocated */ + unsigned alloc_pseudo_sdev:1;
should this always be set on some conditions, like nr_reserved_tags > 1 or we prove a method to queue reserved commands in the sht?
Quoted 3 lines
include/scsi/scsi_host.h +1
-0+/* * True if the low-level driver performs its own reset-settle delays.
Re: [PATCH v3 02/26] scsi: core: Support allocating a pseudo SCSI device
In reply to Bart Van AsscheQuoted 1 line · Bart Van Assche
From: Hannes Reinecke <[email protected]>
some more comments.
Quoted 9 lines
Add the flag '
alloc_pseudo_sdev' to the SCSI host template and allocate a pseudo SCSI device if the flag is set. This device has the SCSI ID <max_id>:U64_MAX so it won't clash with any devices the LLD might create. It's excluded from scanning and will not show up in sysfs. Additionally, pseudo SCSI devices are skipped byshost_for_each_device(). This prevents that the SCSI error handler tries to submit a reset to a non-existent logical unit.
Quoted 3 lines
The intention is to use this device to send internal commands to the storage device.
about naming, I think that "shost_sdev" is nicer than "pseudo_dev", but that is just my personal taste.
Quoted 74 lines
Cc John Garry <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>[ bvanassche: edited patch description / renamed
host_sdevinto
pseudo_sdev/ unexportedscsi_get_host_dev()/ modified error path inscsi_get_pseudo_dev()/ skip pseudo devices in__scsi_iterate_devices()and also when callingsdev_init(),sdev_configure()andsdev_destroy(). See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]Signed-off-by Bart Van Assche <[email protected]>--- drivers/scsi/hosts.c | 8 +++++ drivers/scsi/scsi.c | 7 ++-- drivers/scsi/scsi_priv.h | 2 ++ drivers/scsi/scsi_scan.c | 70 ++++++++++++++++++++++++++++++++++++-- drivers/scsi/scsi_sysfs.c | 4 ++- include/scsi/scsi_device.h | 16 +++++++++ include/scsi/scsi_host.h | 9 +++++ 7 files changed, 110 insertions(+), 6 deletions(-)drivers/scsi/hosts.c +8
-0@@ -307,6 +307,14 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,if (error) goto out_del_dev; +if (sht->alloc_pseudo_sdev) { + shost->pseudo_sdev =scsi_get_pseudo_dev (shost); +if (!shost->pseudo_sdev) { + error = -ENOMEM ; + goto out_del_dev; + } + } +scsi_proc_host_add (shost);scsi_autopm_put_host (shost);return error;drivers/scsi/scsi.c +5
-2@@ -826,8 +826,11 @@ struct scsi_device *__scsi_iterate_devices(struct Scsi_Host *shost,spin_lock_irqsave (shost->host_lock, flags);while (list->next != &shost->__devices) { next = list_entry(list->next, struct scsi_device, siblings); -/* skip devices that we can't get a reference to */ -if (!scsi_device_get (next)) +/* +* Skip pseudo devices and also devices for which +* scsi_device_get() fails. +*/ +if (!scsi_device_is_pseudo_dev (next) && !scsi_device_get (next))break ; next =NULL ; list = list->next;drivers/scsi/scsi_priv.h +2
-0@@ -135,6 +135,8 @@ extern int scsi_complete_async_scans(void);extern int scsi_scan_host_selected (struct Scsi_Host *,unsigned int ,unsigned int ,u64 ,enum scsi_scan_mode );extern void scsi_forget_host (struct Scsi_Host *); +struct scsi_device *scsi_get_pseudo_dev (struct Scsi_Host *); +bool scsi_device_is_pseudo_dev (struct scsi_device *sdev);/* scsi_sysctl.c */ #ifdef CONFIG_SYSCTL drivers/scsi/scsi_scan.c +0
-0
drivers/scsi/scsi_scan.c static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget, · line 366
Quoted 2 lines
drivers/scsi/scsi_scan.c +0
-0@@ -365,7 +365,7 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,scsi_sysfs_device_initialize (sdev);
is the pseudo sdev visible in sysfs?
drivers/scsi/scsi_scan.c static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result, · line 1080
Quoted 12 lines
drivers/scsi/scsi_scan.c +2
-2-if (shost->hostt->sdev_init) { +if (!scsi_device_is_pseudo_dev (sdev) && shost->hostt->sdev_init) { ret = shost->hostt->sdev_init (sdev);if (ret) {/* @@ -1077,7 +1077,7 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,else if (*bflags &BLIST_MAX_1024 ) lim.max_hw_sectors =1024 ; -if (hostt->sdev_configure) +if (!scsi_device_is_pseudo_dev (sdev) && hostt->sdev_configure)
we also have an sdev_configure check later for calling scsi_realloc_sdev_budget_map() (not shown) - should we have that call (to scsi_realloc_sdev_budget_map())?
drivers/scsi/scsi_scan.c static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result, · line 1083
Quoted 3 lines
drivers/scsi/scsi_scan.c +0
-0ret = hostt->sdev_configure (sdev, &lim);if (ret) {queue_limits_cancel_update (sdev->request_queue);
should we really be updating the queue limits (not shown) for the pseudo sdev?
Quoted 16 lines
drivers/scsi/scsi_scan.c +6
-0@@ -1212,6 +1212,12 @@ static int scsi_probe_and_add_lun(struct scsi_target *starget,if (!sdev) goto out; +if (scsi_device_is_pseudo_dev (sdev)) { +if (bflagsp) + *bflagsp =BLIST_NOLUN ; +return SCSI_SCAN_LUN_PRESENT ; + } + result =kmalloc (result_len,GFP_KERNEL );if (!result) goto out_free_sdev;@@ -2077,12 +2083,16 @@ EXPORT_SYMBOL(scsi_scan_host);void scsi_forget_host (struct Scsi_Host *shost)
drivers/scsi/scsi_scan.c EXPORT_SYMBOL(scsi_scan_host); · line 2093
Quoted 10 lines
drivers/scsi/scsi_scan.c +3
-1{ -struct scsi_device *sdev; +struct scsi_device *sdev, *pseudo_sdev =NULL ;unsigned long flags; restart:spin_lock_irqsave (shost->host_lock, flags);list_for_each_entry (sdev, &shost->__devices, siblings) { +if (scsi_device_is_pseudo_dev (sdev)) { + pseudo_sdev = sdev;
it's not ideal that we can be re-assigning this to the same value since this loop may be restarted
Quoted 6 lines
drivers/scsi/scsi_scan.c +2
-0+continue ; + }if (sdev->sdev_state ==SDEV_DEL )continue ;spin_unlock_irqrestore (shost->host_lock, flags);
drivers/scsi/scsi_scan.c void scsi_forget_host(struct Scsi_Host *shost) · line 2107
Quoted 8 lines
drivers/scsi/scsi_scan.c +5
-0@@ -2090,5 +2100,59 @@ void scsi_forget_host(struct Scsi_Host *shost) goto restart; }spin_unlock_irqrestore (shost->host_lock, flags); + +/* +* Remove the pseudo device last since it may be needed during removal +* of other SCSI devices. +*/
don't we already have a pointer to pseudo_sdev in shost->pseudo_sdev?
Quoted 40 lines
drivers/scsi/scsi_scan.c +38
-0+if (pseudo_sdev) +__scsi_remove_device (pseudo_sdev); } +/** +* scsi_get_pseudo_dev() - Attach a pseudo SCSI device to a SCSI host +* @shost: Host that needs a pseudo SCSI device +* +* Lock status: None assumed. +* +* Returns: The scsi_device or NULL +* +* Notes: +* Attach a single scsi_device to the Scsi_Host. The primary aim for this +* device is to serve as a container from which SCSI commands can be +* allocated. Each SCSI command will carry a command tag allocated by the +* block layer. These SCSI commands can be used by the LLDD to send +* internal or passthrough commands without having to manage tag allocation +* inside the LLDD. +*/ +struct scsi_device *scsi_get_pseudo_dev (struct Scsi_Host *shost) +{ +struct scsi_device *sdev =NULL ; +struct scsi_target *starget; + +guard (mutex)(&shost->scan_mutex); + +if (!scsi_host_scan_allowed (shost)) + goto out; + + starget =scsi_alloc_target (&shost->shost_gendev,0 , shost->max_id); +if (!starget) + goto out; + + sdev =scsi_alloc_sdev (starget,U64_MAX ,NULL ); +if (!sdev) + goto reap_target; + + sdev->borken =0 ; +
drivers/scsi/scsi_scan.c void scsi_forget_host(struct Scsi_Host *shost) · line 2157
Quoted 10 lines
drivers/scsi/scsi_scan.c +10
-0+put_target: +/* See also the get_device(dev) call in scsi_alloc_target(). */ +put_device (&starget->dev); + +out: +return sdev; + +reap_target: +scsi_target_reap (starget); + goto put_target;
can we do better than jumping backwards? Maybe
sdev = scsi_alloc_sdev(starget, U64_MAX, NULL);
if (!sdev) {
scsi_target_reap(starget);
goto put_target;
}drivers/scsi/scsi_sysfs.c int scsi_sysfs_add_sdev(struct scsi_device *sdev) · line 1409
Quoted 10 lines
drivers/scsi/scsi_scan.c +1
-0+}drivers/scsi/scsi_sysfs.c +1
-0@@ -1406,6 +1406,8 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev)int error;struct scsi_target *starget = sdev->sdev_target; +WARN_ON_ONCE (scsi_device_is_pseudo_dev (sdev));
should we also error out?
Can we seem to be able to call this from scsi_add_lun() - is that proper?
Quoted 7 lines
drivers/scsi/scsi_sysfs.c +1
-0+ error =scsi_target_add (starget);if (error)return error;@@ -1513,7 +1515,7 @@ void __scsi_remove_device(struct scsi_device *sdev)kref_put (&sdev->host->tagset_refcnt, scsi_mq_free_tags);cancel_work_sync (&sdev->requeue_work);
Bart Van Assche
reviewerRe: [PATCH v3 02/26] scsi: core: Support allocating a pseudo SCSI device
Quoted 8 lines · John Garry
drivers/scsi/scsi_scan.c static struct scsi_device *scsi_alloc_sdev(struct · line 366
Quoted 5 lines · Bart Van Assche
drivers/scsi/scsi_scan.c +0
-0@@ -365,7 +365,7 @@ static struct scsi_device *scsi_alloc_sdev(structscsi_target *starget, scsi_sysfs_device_initialize(sdev);is the pseudo sdev visible in sysfs?
Pseudo SCSI devices are not visible in sysfs. scsi_sysfs_device_initialize() does not make a SCSI device visible in sysfs - it initializes multiple data structures and also includes some code that is important but not related to sysfs visibility. As an
example, the dev_set_name() calls in that function initialize names that
are used by SCSI tracing code.
Quoted 8 lines
drivers/scsi/scsi_scan.c static int scsi_add_lun(struct scsi_device · line 1080
Quoted 6 lines
drivers/scsi/scsi_scan.c +1
-1@@ -1077,7 +1077,7 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result, else if (*bflags & BLIST_MAX_1024) lim.max_hw_sectors = 1024; - if (hostt->sdev_configure) + if (!scsi_device_is_pseudo_dev(sdev) && hostt->sdev_configure)we also have an
sdev_configurecheck later for callingscsi_realloc_sdev_budget_map()(not shown) - should we have that call (toscsi_realloc_sdev_budget_map())?
No, since the budget map is not used for pseudo SCSI devices. I will make sure that scsi_realloc_sdev_budget_map() is not called for pseudo SCSI devices.
Quoted 6 lines
drivers/scsi/scsi_scan.c static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result, · line 1083
Quoted 4 lines
drivers/scsi/scsi_scan.c +0
-0ret = hostt->sdev_configure(sdev, &lim); if (ret) { queue_limits_cancel_update(sdev->request_queue);should we really be updating the queue limits (not shown) for the pseudo sdev?
As far as I can tell the queue limit update calls in scsi_scan.c are
only relevant if BLIST_MAX_512 and/or BLIST_MAX_1024 have been set. These flags are not set for pseudo SCSI devices and hence updating the queue limits for pseudo SCSI devices is not necessary.
Quoted 4 lines
don't we already have a pointer to
pseudo_sdevinshost->pseudo_sdev?Quoted 2 lines
+ if (
pseudo_sdev)
+__scsi_remove_device(pseudo_sdev);
Indeed. Let's use that pointer here.
Quoted 9 lines
can we do better than jumping backwards? Maybe
sdev =
scsi_alloc_sdev(starget, U64_MAX, NULL); if (!sdev) {
scsi_target_reap(starget);
gotoput_target;}
I will include this change.
Quoted 11 lines
drivers/scsi/scsi_sysfs.c int scsi_sysfs_add_sdev(struct scsi_device *sdev) · line 1408
Quoted 9 lines
drivers/scsi/scsi_scan.c +1
-0+}drivers/scsi/scsi_sysfs.c +1
-0@@ -1406,6 +1406,8 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev) int error; struct scsi_target *starget = sdev->sdev_target; + WARN_ON_ONCE(scsi_device_is_pseudo_dev(sdev));should we also error out?
That sounds good to me.
Quoted 1 line
Can we seem to be able to call this from
scsi_add_lun()- is that proper?
It's a bug that should be fixed. I didn't hit that code path because I only tested asynchronous scanning. I will include a fix when I repost this patch.
Thanks,
Bart.
Re: [PATCH v3 02/26] scsi: core: Support allocating a pseudo SCSI device
Quoted 11 lines · Bart Van Assche
Quoted 8 lines · John Garry
drivers/scsi/scsi_scan.c static struct scsi_device *scsi_alloc_sdev(struct · line 366
Quoted 5 lines · Bart Van Assche
drivers/scsi/scsi_scan.c +0
-0@@ -365,7 +365,7 @@ static struct scsi_device *scsi_alloc_sdev(structscsi_target *starget, scsi_sysfs_device_initialize(sdev);is the pseudo sdev visible in sysfs?
Pseudo SCSI devices are not visible in sysfs.
ok, good
Quoted 60 lines
scsi_sysfs_device_initialize()does not make a SCSI device visible in sysfs - it initializes multiple data structures and also includes some code that is important but not related to sysfs visibility. As an
example, thedev_set_name()calls in that function initialize names that
are used by SCSI tracing code.Quoted 8 lines
drivers/scsi/scsi_scan.c static int scsi_add_lun(struct scsi_device · line 1080
Quoted 6 lines
drivers/scsi/scsi_scan.c +1
-1@@ -1077,7 +1077,7 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result, else if (*bflags & BLIST_MAX_1024) lim.max_hw_sectors = 1024; - if (hostt->sdev_configure) + if (!scsi_device_is_pseudo_dev(sdev) && hostt->sdev_configure)we also have an
sdev_configurecheck later for callingscsi_realloc_sdev_budget_map()(not shown) - should we have that call (toscsi_realloc_sdev_budget_map())?No, since the budget map is not used for pseudo SCSI devices. I will make sure that
scsi_realloc_sdev_budget_map()is not called for pseudo SCSI devices.Quoted 6 lines
drivers/scsi/scsi_scan.c static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result, · line 1083
Quoted 4 lines
drivers/scsi/scsi_scan.c +0
-0ret = hostt->sdev_configure(sdev, &lim); if (ret) { queue_limits_cancel_update(sdev->request_queue);should we really be updating the queue limits (not shown) for the pseudo sdev?
As far as I can tell the queue limit update calls in
scsi_scan.care
only relevant ifBLIST_MAX_512and/orBLIST_MAX_1024have been set. These flags are not set for pseudo SCSI devices and hence updating the queue limits for pseudo SCSI devices is not necessary.Quoted 4 lines
don't we already have a pointer to
pseudo_sdevinshost->pseudo_sdev?Quoted 2 lines
+ if (
pseudo_sdev)
+__scsi_remove_device(pseudo_sdev);Indeed. Let's use that pointer here.
Quoted 9 lines
can we do better than jumping backwards? Maybe
sdev =
scsi_alloc_sdev(starget, U64_MAX, NULL); if (!sdev) {
scsi_target_reap(starget);
gotoput_target;}
I will include this change.
Quoted 11 lines
drivers/scsi/scsi_sysfs.c int scsi_sysfs_add_sdev(struct scsi_device *sdev) · line 1408
Quoted 9 lines
drivers/scsi/scsi_scan.c +1
-0+}drivers/scsi/scsi_sysfs.c +1
-0@@ -1406,6 +1406,8 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev) int error; struct scsi_target *starget = sdev->sdev_target; + WARN_ON_ONCE(scsi_device_is_pseudo_dev(sdev));should we also error out?
Quoted 6 lines
That sounds good to me.
Quoted 1 line
Can we seem to be able to call this from
scsi_add_lun()- is that proper?It's a bug that should be fixed. I didn't hit that code path because I only tested asynchronous scanning. I will include a fix when I repost this patch.
great, thanks!
Bart Van Assche
reviewerPATCH v3 3/26 scsi: core: Do not allocate a budget token for reserved commands
In reply to Bart Van AsscheThe SCSI budget mechanism is used to implement the host->cmd_per_lun limit. This limit does not apply to reserved commands. Hence, do not allocate a budget token for reserved commands.
--- drivers/scsi/scsi_lib.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +12 -2
@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)if (starget->can_queue >0 )atomic_dec (&starget->target_busy); -sbitmap_put (&sdev->budget_map, cmd->budget_token); +if (cmd->budget_token <INT_MAX ) +sbitmap_put (&sdev->budget_map, cmd->budget_token); cmd->budget_token =-1 ; }
@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct request_queue *q, {int token; +/* +* Do not allocate a budget token for reserved SCSI commands. Budget +* tokens are used to enforce the cmd_per_lun limit. That limit does not +* apply to reserved commands. +*/ +if (scsi_device_is_pseudo_dev (sdev)) +return INT_MAX ; + token =sbitmap_get (&sdev->budget_map);if (token <0 )return -1 ;
@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token) {struct scsi_device *sdev = q->queuedata; -sbitmap_put (&sdev->budget_map, budget_token); +if (budget_token <INT_MAX ) +sbitmap_put (&sdev->budget_map, budget_token); }/*
Re: [PATCH v3 03/26] scsi: core: Do not allocate a budget token for reserved commands
Quoted 27 lines · Bart Van Assche
The SCSI budget mechanism is used to implement the
host->cmd_per_lunlimit. This limit does not apply to reserved commands. Hence, do not allocate a budget token for reserved commands.Cc Hannes Reinecke <[email protected]>Cc John Garry <[email protected]>Cc Ming Lei <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- drivers/scsi/scsi_lib.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/scsi/scsi_lib.c +2
-1@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)if (starget->can_queue >0 )atomic_dec (&starget->target_busy); -sbitmap_put (&sdev->budget_map, cmd->budget_token); +if (cmd->budget_token <INT_MAX ) +sbitmap_put (&sdev->budget_map, cmd->budget_token); cmd->budget_token =-1 ; }
drivers/scsi/scsi_lib.c static inline int scsi_dev_queue_ready(struct request_queue *q, · line 1370
Quoted 10 lines
drivers/scsi/scsi_lib.c +7
-0@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct request_queue *q, {int token; +/* +* Do not allocate a budget token for reserved SCSI commands. Budget +* tokens are used to enforce the cmd_per_lun limit. That limit does not +* apply to reserved commands. +*/ +if (scsi_device_is_pseudo_dev (sdev)) +return INT_MAX ;
It's not ideal to have this extra check in the fast path always.
My original problem in this area was that we were trying to send reserved commands for real sdevs and those sdevs may have run out of budget. So I was suggesting to not get budget for reserved commands. But would it really be possible to run our of budget for the shost psuedo sdev? Can we just set a separate budget there?
BTW, you are only sending reserved commands to shost pseudo sdev in this series, right? I mean, I think that you don't send them to real sdevs.
Quoted 14 lines
drivers/scsi/scsi_lib.c +3
-1+ token =sbitmap_get (&sdev->budget_map);if (token <0 )return -1 ;@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token) {struct scsi_device *sdev = q->queuedata; -sbitmap_put (&sdev->budget_map, budget_token); +if (budget_token <INT_MAX ) +sbitmap_put (&sdev->budget_map, budget_token); }/*
Bart Van Assche
reviewerRe: [PATCH v3 03/26] scsi: core: Do not allocate a budget token for reserved commands
Quoted 5 lines · John Garry
Quoted 2 lines · Bart Van Assche
+ if (
scsi_device_is_pseudo_dev(sdev))
+ returnINT_MAX;It's not ideal to have this extra check in the fast path always.
Can you provide more detail about your concern? This is a simple integer check and hence the time this check takes is negligible compared to the atomic instructions required to update the budget map. BTW, if no budget map is allocated for pseudo SCSI devices, the above check can be changed into this:
if (!sdev->budget_map) return INT_MAX;
The performance impact of this test should be even smaller than the above test since the budget_map pointer is already used by scsi_dev_queue_ready().
Quoted 1 line
My original problem in this area was that we were trying to send reserved commands for real sdevs and those sdevs may have run out of budget. So I was suggesting to not get budget for reserved commands. But would it really be possible to run our of budget for the shost psuedo sdev?
Yes. The UFS error handler may be invoked if the SCSI budget has been exhausted and may have to allocate a reserved command to recover from the encountered error. The UFS error handler may e.g. submit a START
STOP UNIT command if the UFS device is in the suspended state to bring
it back to a fully powered state. See also the ufshcd_rpm_get_sync(hba)
call in ufshcd_err_handling_prepare().
Quoted 1 line
BTW, you are only sending reserved commands to shost pseudo sdev in this series, right? I mean, I think that you don't send them to real sdevs.
In this patch series all reserved commands are allocated from the pseudo SCSI device. From a later patch:
return scsi_get_internal_cmd_hctx( hba->host->pseudo_sdev, DMA_TO_DEVICE, BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT, 0);
Thanks,
Bart.
Re: [PATCH v3 03/26] scsi: core: Do not allocate a budget token for reserved commands
Quoted 10 lines · Bart Van Assche
Quoted 5 lines · John Garry
Quoted 2 lines · Bart Van Assche
+ if (
scsi_device_is_pseudo_dev(sdev))
+ returnINT_MAX;It's not ideal to have this extra check in the fast path always.
Can you provide more detail about your concern? This is a simple integer check and hence the time this check takes is negligible compared to the atomic instructions required to update the budget map.
Sure, it's a simple check. But it is in the fastpath and it will fail 99.99...% percent of the time. If we don't strictly need checks in the fastpath, then better not have them.
Quoted 11 lines
BTW, if no budget
map is allocated for pseudo SCSI devices, the above check can be changed into this:if (!sdev->budget_map)
returnINT_MAX;The performance impact of this test should be even smaller than the above test since the
budget_mappointer is already used byscsi_dev_queue_ready().
Quoted 7 lines
Quoted 1 line
My original problem in this area was that we were trying to send reserved commands for real sdevs and those sdevs may have run out of budget. So I was suggesting to not get budget for reserved commands. But would it really be possible to run our of budget for the shost psuedo sdev?
Yes. The UFS error handler may be invoked if the SCSI budget has been exhausted and may have to allocate a reserved command to recover from the encountered error. The UFS error handler may e.g. submit a START
STOP UNIT command if the UFS device is in the suspended state to bring
it back to a fully powered state. See also theufshcd_rpm_get_sync(hba)
call inufshcd_err_handling_prepare().
I assumed that the budget map depth for the psuedo sdev would be the same size as the number of reserved commands, right? If that is true, if we have a reserved command allocated, then we must have budget as well available.
Quoted 10 lines
Quoted 2 lines
BTW, you are only sending reserved commands to shost pseudo sdev in this series, right? I mean, I think that you don't send them to real sdevs.
In this patch series all reserved commands are allocated from the pseudo SCSI device. From a later patch:
return
scsi_get_internal_cmd_hctx( hba->host->pseudo_sdev, DMA_TO_DEVICE, BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT, 0);
understood
Bart Van Assche
reviewerRe: [PATCH v3 03/26] scsi: core: Do not allocate a budget token for reserved commands
Quoted 5 lines · John Garry
Quoted 4 lines · Bart Van Assche
Quoted 2 lines · John Garry
On 27/08/2025 01:06, Bart Van Assche wrote:
My original problem in this area was that we were trying to send reserved commands for real sdevs and those sdevs may have run out of budget. So I was suggesting to not get budget for reserved commands. But would it really be possible to run our of budget for the shost psuedo sdev?
Quoted 7 lines
Quoted 5 lines
Yes. The UFS error handler may be invoked if the SCSI budget has been exhausted and may have to allocate a reserved command to recover from the encountered error. The UFS error handler may e.g. submit a START
STOP UNIT command if the UFS device is in the suspended state to bring
it back to a fully powered state. See also theufshcd_rpm_get_sync(hba)
call inufshcd_err_handling_prepare().I assumed that the budget map depth for the psuedo sdev would be the same size as the number of reserved commands, right? If that is true, if we have a reserved command allocated, then we must have budget as well available.
A budget map is allocated by the SCSI core to enforce the host->cmd_per_lun limit. That limit does not apply to pseudo SCSI devices. Hence, there is no need to allocate a budget map for pseudo SCSI devices. The patch below shows a possible replacement for this patch, assuming that the previous patch is modified such that no budget map is allocated for pseudo devices:
drivers/scsi/scsi_lib.c +12 -2
@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)if (starget->can_queue >0 )atomic_dec (&starget->target_busy); -sbitmap_put (&sdev->budget_map, cmd->budget_token); +if (sdev->budget_map.map) +sbitmap_put (&sdev->budget_map, cmd->budget_token); cmd->budget_token =-1 ; }
@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct request_queue *q, {int token; +/* +* Do not allocate a budget token for reserved SCSI commands. Budget +* tokens are used to enforce the cmd_per_lun limit. That limit does not +* apply to reserved commands. +*/ +if (!sdev->budget_map.map) +return INT_MAX ; + token =sbitmap_get (&sdev->budget_map);if (token <0 )return -1 ;
@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token) {struct scsi_device *sdev = q->queuedata; -sbitmap_put (&sdev->budget_map, budget_token); +if (sdev->budget_map.map) +sbitmap_put (&sdev->budget_map, budget_token); }/*
Thanks,
Bart.
Re: [PATCH v3 03/26] scsi: core: Do not allocate a budget token for reserved commands
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Yes. The UFS error handler may be invoked if the SCSI budget has been exhausted and may have to allocate a reserved command to recover from the encountered error. The UFS error handler may e.g. submit a START
STOP UNIT command if the UFS device is in the suspended state to bring
it back to a fully powered state. See also theufshcd_rpm_get_sync(hba)
call inufshcd_err_handling_prepare().
Quoted 5 lines
Quoted 1 line
I assumed that the budget map depth for the psuedo sdev would be the same size as the number of reserved commands, right? If that is true, if we have a reserved command allocated, then we must have budget as well available.
A budget map is allocated by the SCSI core to enforce the
host->cmd_per_lunlimit. That limit does not apply to pseudo SCSI devices.
Sure, but if you had a budget map for the pseudo sdev then it would be simpler, as you don't require special checks, below. I appreciate that the budget map is not relevant to psuedo sdev to enforce any queue depth limit tracking. I am just suggesting to have consistency between all sdevs, to have less special cases.
Quoted 38 lines
Hence, there is no need to allocate a budget map for pseudo SCSI devices. The patch below shows a possible replacement for this patch, assuming that the previous patch is modified such that no budget map is allocated for pseudo devices:
drivers/scsi/scsi_lib.c +5
-1@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd) if (starget->can_queue > 0) atomic_dec(&starget->target_busy); - sbitmap_put(&sdev->budget_map, cmd->budget_token); + if (sdev->budget_map.map) + sbitmap_put(&sdev->budget_map, cmd->budget_token); cmd->budget_token = -1; }@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(structrequest_queue *q, { int token; + /* +* Do not allocate a budget token for reserved SCSI commands. Budget +* tokens are used to enforce the cmd_per_lun limit. That limit does not
+ * apply to reserved commands.
+ */
+ if (!sdev->budget_map.map)
+ returnINT_MAX;drivers/scsi/scsi_lib.c +1
-0+ token = sbitmap_get(&sdev->budget_map); if (token < 0) return -1;
drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct · line 1764
Quoted 8 lines
drivers/scsi/scsi_lib.c +2
-1@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token) { struct scsi_device *sdev = q->queuedata; - sbitmap_put(&sdev->budget_map, budget_token); + if (sdev->budget_map.map) + sbitmap_put(&sdev->budget_map, budget_token); }
Thanks,
John
Bart Van Assche
reviewerRe: [PATCH v3 03/26] scsi: core: Do not allocate a budget token for reserved commands
Quoted 7 lines · John Garry
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Yes. The UFS error handler may be invoked if the SCSI budget has been exhausted and may have to allocate a reserved command to recover from the encountered error. The UFS error handler may e.g. submit a START
STOP UNIT command if the UFS device is in the suspended state to bring
it back to a fully powered state. See also theufshcd_rpm_get_sync(hba)
call inufshcd_err_handling_prepare().
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I assumed that the budget map depth for the psuedo sdev would be the same size as the number of reserved commands, right? If that is true, if we have a reserved command allocated, then we must have budget as well available.
A budget map is allocated by the SCSI core to enforce the
host->cmd_per_lunlimit. That limit does not apply to pseudo SCSI devices.Sure, but if you had a budget map for the pseudo sdev then it would be simpler, as you don't require special checks, below. I appreciate that the budget map is not relevant to psuedo sdev to enforce any queue depth limit tracking. I am just suggesting to have consistency between all sdevs, to have less special cases.
Agreed, but a new patch series has been posted in which the sdev->budget_map.map test is essential. See also
https://lore.kernel.org/linux-scsi/[email protected].
Thanks,
Bart.
Bart Van Assche
reviewerPATCH v3 4/26 scsi: core: Bypass the queue limit checks for reserved commands
In reply to Bart Van AsscheFrom: John Garry <[email protected]>
Reserved commands will be used by SCSI LLDs for submitting internal commands. The SCSI host, target and device limits do not apply to these use cases. Additionally, do not activate the SCSI error handler if a reserved command fails such that reserved commands can be submitted from inside the SCSI error handler.
[ bvanassche: modified patch title and patch description. Renamed
.reserved_queuecommand() into .queue_reserved_command(). Changed
the second argument of __blk_mq_end_request() from 0 into error
code in the completion path if cmd->result != 0. Rewrote the
scsi_queue_rq() changes. See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]
--- drivers/scsi/hosts.c | 6 +++++ drivers/scsi/scsi_lib.c | 54 ++++++++++++++++++++++++++++------------ include/scsi/scsi_host.h | 6 +++++ 3 files changed, 50 insertions(+), 16 deletions(-)
drivers/scsi/hosts.c +6 -0
@@ -231,6 +231,12 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev, goto fail; } +if (shost->nr_reserved_cmds && !sht->queue_reserved_command) { +shost_printk (KERN_ERR , shost, +"nr_reserved_cmds set but no method to queue\n" ); + goto fail; + } +/* Use min_t(int, ...) in case shost->can_queue exceeds SHRT_MAX */ shost->cmd_per_lun =min_t (int, shost->cmd_per_lun, shost->can_queue);
drivers/scsi/scsi_lib.c +38 -16
75 lines not on this page · raw message
include/scsi/scsi_host.h +6 -0
@@ -86,6 +86,12 @@ struct scsi_host_template {*/ int (* queuecommand)(struct Scsi_Host *,struct scsi_cmnd *); +/* +* Queue a reserved command (BLK_MQ_REQ_RESERVED). The .queuecommand() +* documentation also applies to the .queue_reserved_command() callback. +*/ +int (*queue_reserved_command)(struct Scsi_Host *,struct scsi_cmnd *); +/* * The commit_rqs function is used to trigger a hardware * doorbell after some requests have been queued with
Re: [PATCH v3 04/26] scsi: core: Bypass the queue limit checks for reserved commands
Quoted 10 lines · Bart Van Assche
From: John Garry <[email protected]>
Reserved commands will be used by SCSI LLDs for submitting internal commands. The SCSI host, target and device limits do not apply to these use cases. Additionally, do not activate the SCSI error handler if a reserved command fails such that reserved commands can be submitted from inside the SCSI error handler.
Signed-off-by John Garry <[email protected]>[ bvanassche: modified patch title and patch description.
it's an odd name now... I don't know which queue limits we are bypassing
Quoted 18 lines
Renamed
.reserved_queuecommand() into .queue_reserved_command(). Changed the second argument of__blk_mq_end_request()from 0 into error code in the completion path ifcmd->result!= 0. Rewrote thescsi_queue_rq()changes. See alsohttps://urldefense.com/v3/__https://lore.kernel.org/linux-scsi/[email protected]/__;!!ACWV5N9M2RV99hQ!LlYERxyD9XQO4a934MwJG8q_NK4ySl31eOzkZOoyuPPNa1aEcS5n_T7NUuFFs0I8mBaBx0FgAZf7ghv6h06C5A$ ] Signed-off-by: Bart Van Assche <[email protected]> --- drivers/scsi/hosts.c | 6 +++++ drivers/scsi/scsi_lib.c | 54 ++++++++++++++++++++++++++++------------ include/scsi/scsi_host.h | 6 +++++ 3 files changed, 50 insertions(+), 16 deletions(-)drivers/scsi/hosts.c +0
-0
drivers/scsi/hosts.c int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev, · line 237
Quoted 7 lines
drivers/scsi/hosts.c +4
-0@@ -231,6 +231,12 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev, goto fail; } +if (shost->nr_reserved_cmds && !sht->queue_reserved_command) { +shost_printk (KERN_ERR , shost, +"nr_reserved_cmds set but no method to queue\n" ); + goto fail;
looks ok
Quoted 10 lines
drivers/scsi/hosts.c +2
-0+ } +/* Use min_t(int, ...) in case shost->can_queue exceeds SHRT_MAX */ shost->cmd_per_lun =min_t (int, shost->cmd_per_lun, shost->can_queue);drivers/scsi/scsi_lib.c +0
-0
drivers/scsi/scsi_lib.c static void scsi_complete(struct request *rq) · line 1544
Quoted 6 lines
drivers/scsi/scsi_lib.c +3
-0@@ -1539,6 +1539,14 @@ static void scsi_complete(struct request *rq)struct scsi_cmnd *cmd =blk_mq_rq_to_pdu (rq);enum scsi_disposition disposition; +if (blk_mq_is_reserved_rq (rq)) { +/* Only pass-through requests are supported in this code path. */ +WARN_ON_ONCE (!blk_rq_is_passthrough (scsi_cmd_to_rq (cmd)));
eh, do we really have passthough reserved command?
Quoted 18 lines
drivers/scsi/scsi_lib.c +7
-3+scsi_mq_uninit_cmd (cmd); +__blk_mq_end_request (rq,scsi_result_to_blk_status (cmd->result)); +return ; + } +INIT_LIST_HEAD (&cmd->eh_entry);atomic_inc (&cmd->device->iodone_cnt);@@ -1828,25 +1836,31 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,WARN_ON_ONCE (cmd->budget_token <0 );/* -* If the device is not in running state we will reject some or all -* commands. +* Bypass the SCSI device, SCSI target and SCSI host checks for +* reserved commands. */ -if (unlikely (sdev->sdev_state !=SDEV_RUNNING )) {
drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1847
Quoted 10 lines
drivers/scsi/scsi_lib.c +6
-4- ret =scsi_device_state_check (sdev, req); -if (ret !=BLK_STS_OK ) - goto out_put_budget; - } +if (!blk_mq_is_reserved_rq (req)) { +/* +* If the device is not in running state we will reject some or +* all commands. +*/ +if (unlikely (sdev->sdev_state !=SDEV_RUNNING )) {
I am curious about this. I mentioned previously if we only send reserved commands to the psuedo sdev (in this seris). If so, would the psuedo sdev not always be running state?
Quoted 61 lines
drivers/scsi/scsi_lib.c +22
-9+ ret =scsi_device_state_check (sdev, req); +if (ret !=BLK_STS_OK ) + goto out_put_budget; + } - ret =BLK_STS_RESOURCE ; -if (!scsi_target_queue_ready (shost, sdev)) - goto out_put_budget; -if (unlikely (scsi_host_in_recovery (shost))) { -if (cmd->flags &SCMD_FAIL_IF_RECOVERING ) - ret =BLK_STS_OFFLINE ; - goto out_dec_target_busy; + ret =BLK_STS_RESOURCE ; +if (!scsi_target_queue_ready (shost, sdev)) + goto out_put_budget; +if (unlikely (scsi_host_in_recovery (shost))) { +if (cmd->flags &SCMD_FAIL_IF_RECOVERING ) + ret =BLK_STS_OFFLINE ; + goto out_dec_target_busy; + } +if (!scsi_host_queue_ready (q, shost, sdev, cmd)) + goto out_dec_target_busy; } -if (!scsi_host_queue_ready (q, shost, sdev, cmd)) - goto out_dec_target_busy;/* * Only clear the driver-private command data if the LLD does not supply @@ -1875,6 +1889,14 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, cmd->submitter =SUBMITTED_BY_BLOCK_LAYER ;blk_mq_start_request (req); +if (blk_mq_is_reserved_rq (req)) { + reason = shost->hostt->queue_reserved_command (shost, cmd); +if (reason) { + ret =BLK_STS_RESOURCE ; + goto out_put_budget; + } +return BLK_STS_OK ; + } reason =scsi_dispatch_cmd (cmd);if (reason) {scsi_set_blocked (cmd, reason);include/scsi/scsi_host.h +6
-0@@ -86,6 +86,12 @@ struct scsi_host_template {*/ int (* queuecommand)(struct Scsi_Host *,struct scsi_cmnd *); +/* +* Queue a reserved command (BLK_MQ_REQ_RESERVED). The .queuecommand() +* documentation also applies to the .queue_reserved_command() callback. +*/ +int (*queue_reserved_command)(struct Scsi_Host *,struct scsi_cmnd *); +/* * The commit_rqs function is used to trigger a hardware * doorbell after some requests have been queued with
Bart Van Assche
reviewerRe: [PATCH v3 04/26] scsi: core: Bypass the queue limit checks for reserved commands
Quoted 4 lines · John Garry
Quoted 1 line · Bart Van Assche
[ bvanassche: modified patch title and patch description.
it's an odd name now... I don't know which queue limits we are bypassing
I can change "queue" into "SCSI host" in the patch title.
Quoted 11 lines
Quoted 9 lines
drivers/scsi/scsi_lib.c +2
-0@@ -1539,6 +1539,14 @@ static void scsi_complete(struct request *rq) struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq); enum scsi_disposition disposition; + if (blk_mq_is_reserved_rq(rq)) { + /* Only pass-through requests are supported in this codepath. */
+ WARN_ON_ONCE(!blk_rq_is_passthrough(scsi_cmd_to_rq(cmd)));eh, do we really have passthough reserved command?
All reserved commands that end up in scsi_complete() should be pass-through commands (REQ_OP_DRV_IN / REQ_OP_DRV_OUT), isn't it? I don't think that we should allow other request types for reserved commands.
Quoted 12 lines
Quoted 11 lines
- if (
unlikely(sdev->sdev_state != SDEV_RUNNING)) {
- ret =scsi_device_state_check(sdev, req);
- if (ret !=BLK_STS_OK)
- gotoout_put_budget;
- }
+ if (!blk_mq_is_reserved_rq(req)) {
+ /*
+ * If the device is not in running state we will reject some or + * all commands.
+ */
+ if (unlikely(sdev->sdev_state != SDEV_RUNNING)) {
Quoted 1 line
I am curious about this. I mentioned previously if we only send reserved commands to the psuedo sdev (in this seris). If so, would the psuedo sdev not always be running state?
Has the above code change perhaps been misread? The above change causes the sdev->sdev_state check to be skipped for pseudo SCSI devices.
Thanks,
Bart.
Re: [PATCH v3 04/26] scsi: core: Bypass the queue limit checks for reserved commands
Quoted 20 lines · Bart Van Assche
Quoted 4 lines · John Garry
Quoted 1 line · Bart Van Assche
[ bvanassche: modified patch title and patch description.
it's an odd name now... I don't know which queue limits we are bypassing
I can change "queue" into "SCSI host" in the patch title.
Quoted 11 lines
Quoted 9 lines
drivers/scsi/scsi_lib.c +2
-0@@ -1539,6 +1539,14 @@ static void scsi_complete(struct request *rq) struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq); enum scsi_disposition disposition; + if (blk_mq_is_reserved_rq(rq)) { + /* Only pass-through requests are supported in this codepath. */
+ WARN_ON_ONCE(!blk_rq_is_passthrough(scsi_cmd_to_rq(cmd)));eh, do we really have passthough reserved command?
Quoted 4 lines
All reserved commands that end up in
scsi_complete()should be pass-through commands (REQ_OP_DRV_IN/REQ_OP_DRV_OUT), isn't it? I don't think that we should allow other request types for reserved commands.
ok, I suppose, but it seems odd to say that a reserved command is a passthrough command.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
- if (
unlikely(sdev->sdev_state != SDEV_RUNNING)) {
- ret =scsi_device_state_check(sdev, req);
- if (ret !=BLK_STS_OK)
- gotoout_put_budget;
- }
+ if (!blk_mq_is_reserved_rq(req)) {
+ /*
+ * If the device is not in running state we will reject some or + * all commands.
+ */
+ if (unlikely(sdev->sdev_state != SDEV_RUNNING)) {
Quoted 5 lines
Quoted 1 line
I am curious about this. I mentioned previously if we only send reserved commands to the psuedo sdev (in this seris). If so, would the psuedo sdev not always be running state?
Has the above code change perhaps been misread? The above change causes the
sdev->sdev_statecheck to be skipped for pseudo SCSI devices.
Maybe I misread. I was just curious if we are doing this sdev state check for a psuedo sdev, which does not seem to be the case.
Thanks,
John
Bart Van Assche
reviewerPATCH v3 5/26 scsi: core: Add scsi_{get,put}_internal_cmd() helpers
In reply to Bart Van AsscheFrom: Hannes Reinecke <[email protected]>
Add helper functions to allow LLDDs to allocate and free internal commands.
[ bvanassche: changed the 'nowait' argument into a 'flags' argument /
added scsi_get_internal_cmd_hctx(). See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]
--- drivers/scsi/scsi_lib.c | 80 ++++++++++++++++++++++++++++++++++++++ include/scsi/scsi_cmnd.h | 2 + include/scsi/scsi_device.h | 7 ++++ 3 files changed, 89 insertions(+)
drivers/scsi/scsi_lib.c +80 -0
@@ -1247,6 +1247,18 @@ struct request *scsi_alloc_request(struct request_queue *q, blk_opf_t opf, }EXPORT_SYMBOL_GPL (scsi_alloc_request); +struct request *scsi_alloc_request_hctx (struct request_queue *q,blk_opf_t opf, +blk_mq_req_flags_t flags,unsigned int hctx_idx) +{ +struct request *rq; + + rq =blk_mq_alloc_request_hctx (q, opf, flags, hctx_idx); +if (!IS_ERR (rq)) +scsi_initialize_rq (rq); +return rq; +} +EXPORT_SYMBOL_GPL (scsi_alloc_request_hctx); +/* * Only called when the request isn't completed by SCSI, and not freed by * SCSI
@@ -2139,6 +2151,74 @@ void scsi_mq_free_tags(struct kref *kref)complete (&shost->tagset_freed); } +/** +* scsi_get_internal_cmd() - Allocate an internal SCSI command. +* @sdev: SCSI device from which to allocate the command +* @data_direction: Data direction for the allocated command +* @flags: request allocation flags, e.g. BLK_MQ_REQ_RESERVED or +* BLK_MQ_REQ_NOWAIT. +* +* Allocates a SCSI command for internal LLDD use. +*/ +struct scsi_cmnd *scsi_get_internal_cmd (struct scsi_device *sdev, +enum dma_data_direction data_direction, +blk_mq_req_flags_t flags) +{ +enum req_op op = data_direction ==DMA_TO_DEVICE ?REQ_OP_DRV_OUT : +REQ_OP_DRV_IN ; +struct scsi_cmnd *scmd; +struct request *rq; + + rq =scsi_alloc_request (sdev->request_queue, op, flags); +if (IS_ERR (rq)) +return NULL ; + scmd =blk_mq_rq_to_pdu (rq); + scmd->device = sdev; + +return scmd; +} +EXPORT_SYMBOL_GPL (scsi_get_internal_cmd); + +/** +* scsi_get_internal_cmd_hctx() - Allocate an internal SCSI command from a given +* hardware queue. +* @sdev: SCSI device from which to allocate the command +* @data_direction: Data direction for the allocated command +* @flags: request allocation flags, e.g. BLK_MQ_REQ_RESERVED or +* BLK_MQ_REQ_NOWAIT. +* @hctx_idx: Hardware queue index. +* +* Allocates a SCSI command for internal LLDD use. +*/ +struct scsi_cmnd *scsi_get_internal_cmd_hctx (struct scsi_device *sdev, +enum dma_data_direction data_direction, +blk_mq_req_flags_t flags,unsigned int hctx_idx) +{ +enum req_op op = data_direction ==DMA_TO_DEVICE ?REQ_OP_DRV_OUT : +REQ_OP_DRV_IN ; +struct scsi_cmnd *scmd; +struct request *rq; + + rq =scsi_alloc_request_hctx (sdev->request_queue, op, flags, hctx_idx); +if (IS_ERR (rq)) +return NULL ; + scmd =blk_mq_rq_to_pdu (rq); + scmd->device = sdev; + +return scmd; +} +EXPORT_SYMBOL_GPL (scsi_get_internal_cmd_hctx); + +/** +* scsi_put_internal_cmd() - Free an internal SCSI command. +* @scmd: SCSI command to be freed +*/ +void scsi_put_internal_cmd (struct scsi_cmnd *scmd) +{ +blk_mq_free_request (blk_mq_rq_from_pdu (scmd)); +} +EXPORT_SYMBOL_GPL (scsi_put_internal_cmd); +/** * scsi_device_from_queue - return sdev associated with a request_queue * @q: The request queue to return the sdev from
include/scsi/scsi_cmnd.h +2 -0
@@ -396,5 +396,7 @@ extern void scsi_build_sense(struct scsi_cmnd *scmd, int desc,struct request *scsi_alloc_request (struct request_queue *q,blk_opf_t opf,blk_mq_req_flags_t flags); +struct request *scsi_alloc_request_hctx (struct request_queue *q,blk_opf_t opf, +blk_mq_req_flags_t flags,unsigned int hctx_idx);#endif /* _SCSI_SCSI_CMND_H */
include/scsi/scsi_device.h +7 -0
@@ -558,6 +558,13 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd, const struct scsi_exec_args *args);void scsi_failures_reset_retries (struct scsi_failures *failures); +struct scsi_cmnd *scsi_get_internal_cmd (struct scsi_device *sdev, +enum dma_data_direction data_direction, +blk_mq_req_flags_t flags); +struct scsi_cmnd *scsi_get_internal_cmd_hctx (struct scsi_device *sdev, +enum dma_data_direction data_direction, +blk_mq_req_flags_t flags,unsigned int hctx_idx); +void scsi_put_internal_cmd (struct scsi_cmnd *scmd);extern void sdev_disable_disk_events (struct scsi_device *sdev);extern void sdev_enable_disk_events (struct scsi_device *sdev);extern int scsi_vpd_lun_id (struct scsi_device *,char *,size_t );
Re: [PATCH v3 05/26] scsi: core: Add scsi_{get,put}_internal_cmd() helpers
Quoted 10 lines · Bart Van Assche
From: Hannes Reinecke <[email protected]>
Add helper functions to allow LLDDs to allocate and free internal commands.
Cc John Garry <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>[ bvanassche: changed the 'nowait' argument into a 'flags' argument /
addedscsi_get_internal_cmd_hctx(). See also
https://lore.kernel.org/linux-scsi/[email protected]/ ]Signed-off-by Bart Van Assche <[email protected]>
did you consider reusing/refactoring scsi_execute_cmd() here? I'm just wondering if it is possible.
Quoted 77 lines
--- drivers/scsi/scsi_lib.c | 80 ++++++++++++++++++++++++++++++++++++++ include/scsi/scsi_cmnd.h | 2 + include/scsi/scsi_device.h | 7 ++++ 3 files changed, 89 insertions(+)drivers/scsi/scsi_lib.c +56
-0@@ -1247,6 +1247,18 @@ struct request *scsi_alloc_request(struct request_queue *q, blk_opf_t opf, }EXPORT_SYMBOL_GPL (scsi_alloc_request); +struct request *scsi_alloc_request_hctx (struct request_queue *q,blk_opf_t opf, +blk_mq_req_flags_t flags,unsigned int hctx_idx) +{ +struct request *rq; + + rq =blk_mq_alloc_request_hctx (q, opf, flags, hctx_idx); +if (!IS_ERR (rq)) +scsi_initialize_rq (rq); +return rq; +} +EXPORT_SYMBOL_GPL (scsi_alloc_request_hctx); +/* * Only called when the request isn't completed by SCSI, and not freed by * SCSI @@ -2139,6 +2151,74 @@ void scsi_mq_free_tags(struct kref *kref)complete (&shost->tagset_freed); } +/** +* scsi_get_internal_cmd() - Allocate an internal SCSI command. +* @sdev: SCSI device from which to allocate the command +* @data_direction: Data direction for the allocated command +* @flags: request allocation flags, e.g. BLK_MQ_REQ_RESERVED or +* BLK_MQ_REQ_NOWAIT. +* +* Allocates a SCSI command for internal LLDD use. +*/ +struct scsi_cmnd *scsi_get_internal_cmd (struct scsi_device *sdev, +enum dma_data_direction data_direction, +blk_mq_req_flags_t flags) +{ +enum req_op op = data_direction ==DMA_TO_DEVICE ?REQ_OP_DRV_OUT : +REQ_OP_DRV_IN ; +struct scsi_cmnd *scmd; +struct request *rq; + + rq =scsi_alloc_request (sdev->request_queue, op, flags); +if (IS_ERR (rq)) +return NULL ; + scmd =blk_mq_rq_to_pdu (rq); + scmd->device = sdev; + +return scmd; +} +EXPORT_SYMBOL_GPL (scsi_get_internal_cmd); + +/** +* scsi_get_internal_cmd_hctx() - Allocate an internal SCSI command from a given +* hardware queue. +* @sdev: SCSI device from which to allocate the command +* @data_direction: Data direction for the allocated command +* @flags: request allocation flags, e.g. BLK_MQ_REQ_RESERVED or +* BLK_MQ_REQ_NOWAIT. +* @hctx_idx: Hardware queue index. +* +* Allocates a SCSI command for internal LLDD use. +*/ +struct scsi_cmnd *scsi_get_internal_cmd_hctx (struct scsi_device *sdev, +enum dma_data_direction data_direction, +blk_mq_req_flags_t flags,unsigned int hctx_idx) +{ + enum req_op op = data_direction == DMA_TO_DEVICE ? REQ_OP_DRV_OUT :
Bart Van Assche
reviewerRe: [PATCH v3 05/26] scsi: core: Add scsi_{get,put}_internal_cmd() helpers
Quoted 4 lines · John Garry
Quoted 1 line · Bart Van Assche
Add helper functions to allow LLDDs to allocate and free internal commands.
did you consider reusing/refactoring
scsi_execute_cmd()here? I'm just wondering if it is possible.
That should be possible by adding more members into struct scsi_exec_args. However, I'm not sure this will result in a code reduction. As one can see in the last patch in this series, the amount of code shared between scsi_execute_cmd() and ufshcd_issue_dev_cmd() is minimal:
rq->timeout = timeout; blk_execute_rq(rq, true);
Thanks,
Bart.
Re: [PATCH v3 05/26] scsi: core: Add scsi_{get,put}_internal_cmd() helpers
Quoted 6 lines · Bart Van Assche
Quoted 4 lines · John Garry
Quoted 1 line · Bart Van Assche
Add helper functions to allow LLDDs to allocate and free internal commands.
did you consider reusing/refactoring
scsi_execute_cmd()here? I'm just wondering if it is possible.
Quoted 8 lines
That should be possible by adding more members into struct
scsi_exec_args. However, I'm not sure this will result in a code reduction. As one can see in the last patch in this series, the amount of code shared betweenscsi_execute_cmd()andufshcd_issue_dev_cmd()is minimal:rq->timeout = timeout;
blk_execute_rq(rq, true);
I think that it's a more a case of we would rather not send half-initialized requests or scsi commands though the block layer and scsi-midlayer. scsi_execute_cmd() looks to set all fields up appropriately, and not just the ones which we are interested in.
Thanks,
John
Bart Van Assche
reviewerRe: [PATCH v3 05/26] scsi: core: Add scsi_{get,put}_internal_cmd() helpers
Quoted 1 line · John Garry
I think that it's a more a case of we would rather not send half- initialized requests or scsi commands though the block layer and scsi- midlayer.
scsi_execute_cmd()looks to set all fields up appropriately, and not just the ones which we are interested in.
I will look into reusing scsi_execute_cmd().
Thanks,
Bart.
Bart Van Assche
reviewerPATCH v3 6/26 scsi_debug: Set .alloc_pseudo_sdev
In reply to Bart Van AsscheMake sure that the code for allocating a pseudo SCSI device gets triggered while running blktests.
--- drivers/scsi/scsi_debug.c | 1 + 1 file changed, 1 insertion(+)
drivers/scsi/scsi_debug.c +1 -0
@@ -9438,6 +9438,7 @@ static const struct scsi_host_template sdebug_driver_template = { .module = THIS_MODULE, .skip_settle_delay = 1, .track_queue_depth = 1, + .alloc_pseudo_sdev = 1, .cmd_size = sizeof(struct sdebug_scsi_cmd), .init_cmd_priv = sdebug_init_cmd_priv, .target_alloc = sdebug_target_alloc,
Re: [PATCH v3 06/26] scsi_debug: Set .alloc_pseudo_sdev
Quoted 13 lines · Bart Van Assche
Make sure that the code for allocating a pseudo SCSI device gets triggered while running blktests.
Signed-off-by Bart Van Assche <[email protected]>--- drivers/scsi/scsi_debug.c | 1 + 1 file changed, 1 insertion(+)drivers/scsi/scsi_debug.c +0
-0
drivers/scsi/scsi_debug.c static const struct scsi_host_template sdebug_driver_template = { · line 9441
Quoted 4 lines
drivers/scsi/scsi_debug.c +1
-0@@ -9438,6 +9438,7 @@ static const struct scsi_host_template sdebug_driver_template = { .module = THIS_MODULE, .skip_settle_delay = 1, .track_queue_depth = 1, + .alloc_pseudo_sdev = 1,
yeah, it's prob right to do this. But - as I mentioned earlier - we may not need this member if we can logically derive if we need to create the pseudo sdev from other sht/shost members
drivers/scsi/scsi_debug.c static const struct scsi_host_template sdebug_driver_template = { · line 9444
Quoted 4 lines
drivers/scsi/scsi_debug.c +0
-0.cmd_size = sizeof(struct sdebug_scsi_cmd), .init_cmd_priv = sdebug_init_cmd_priv, .target_alloc = sdebug_target_alloc,
for scsi_debug, maybe we can add reserved command handling as part of the driver abort handling, e.g. eh_abort_handler -> scsi_debug_abort should send a reserved command to "abort" a scmd, and the reserved command handler does the same as scsi_debug_abort currently does.
Just an idea. I think that I prototyped this a while ago. I would need to dig it up.
Thanks,
John
Bart Van Assche
reviewerRe: [PATCH v3 06/26] scsi_debug: Set .alloc_pseudo_sdev
Quoted 8 lines · John Garry
Quoted 4 lines · Bart Van Assche
.cmd_size =
sizeof(struct sdebug_scsi_cmd),
.init_cmd_priv =sdebug_init_cmd_priv,
.target_alloc =sdebug_target_alloc,for
scsi_debug, maybe we can add reserved command handling as part of the driver abort handling, e.g.eh_abort_handler->scsi_debug_abortshould send a reserved command to "abort" a scmd, and the reserved command handler does the same asscsi_debug_abortcurrently does.
Another possibility for getting rid of .alloc_pseudo_sdev is to allocate
a pseudo SCSI device if either .nr_reserved_cmds or .queue_reserved_command has been set in the SCSI host template. A dummy .queue_reserved_command method could be defined as follows in the SCSI
debug driver:
+/*
+ * The only purpose of this function is to make the SCSI core allocate a
+ * pseudo SCSI device.
+ */
+static int scsi_debug_queue_reserved_command(struct Scsi_Host *shost,
+ struct scsi_cmnd *scp)
+{
+ WARN_ON_ONCE(true);
+ scp->result = DID_ERROR << 16;
+ scsi_done(scp);
+ return 0;
+}
Thanks,
Bart.
Re: [PATCH v3 06/26] scsi_debug: Set .alloc_pseudo_sdev
Quoted 9 lines · Bart Van Assche
Quoted 1 line
for
scsi_debug, maybe we can add reserved command handling as part of the driver abort handling, e.g.eh_abort_handler->scsi_debug_abortshould send a reserved command to "abort" a scmd, and the reserved command handler does the same asscsi_debug_abortcurrently does.Another possibility for getting rid of .alloc_pseudo_sdev is to allocate
a pseudo SCSI device if either .nr_reserved_cmds or .queue_reserved_command has been set in the SCSI host template. A dummy .queue_reserved_command method could be defined as follows in the SCSI
debug driver:(quoted diff) +2
-0+/* + * The only purpose of this function is to make the SCSI core allocate a
Review
Quoted 10 lines
(quoted diff) +10
-0+ * pseudo SCSI device. + */ +static int scsi_debug_queue_reserved_command(struct Scsi_Host *shost, + struct scsi_cmnd *scp) +{ + WARN_ON_ONCE(true); + scp->result = DID_ERROR << 16; + scsi_done(scp); + return 0; +}
Sure, but it would be nice if this was not a stub, but rather actually did some work, like "aborting" a task (as I mentioned above). Then people like me who don't have UFS HW can actually test this functionality. I do appreciate that it is more work to implement this.
Thanks,
John
Bart Van Assche
reviewerPATCH v3 7/26 ufs: core: Move an assignment
In reply to Bart Van AsschePrepare for introducing a WARN_ON_ONCE() call in ufshcd_mcq_get_tag() if the tag lookup fails.
--- drivers/ufs/core/ufs-mcq.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/ufs/core/ufs-mcq.c +2 -1
@@ -303,9 +303,10 @@ static void ufshcd_mcq_process_cqe(struct ufs_hba *hba, struct ufs_hw_queue *hwq) {struct cq_entry *cqe =ufshcd_mcq_cur_cqe (hwq); -int tag =ufshcd_mcq_get_tag (hba, cqe);if (cqe->command_desc_base_addr) { +int tag =ufshcd_mcq_get_tag (hba, cqe); +ufshcd_compl_one_cqe (hba, tag, cqe);/* After processed the cqe, mark it empty (invalid) entry */ cqe->command_desc_base_addr =0 ;
Bart Van Assche
reviewerPATCH v3 8/26 ufs: core: Change the type of one ufshcd_add_cmd_upiu_trace() argument
In reply to Bart Van AsscheChange the 'tag' argument into an LRB pointer. This patch prepares for the removal of the hba->lrb[] array.
--- drivers/ufs/core/ufshcd.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +5 -4
@@ -405,10 +405,11 @@ static void ufshcd_configure_wb(struct ufs_hba *hba)ufshcd_wb_toggle_buf_flush (hba, true); } -static void ufshcd_add_cmd_upiu_trace (struct ufs_hba *hba,unsigned int tag, +static void ufshcd_add_cmd_upiu_trace (struct ufs_hba *hba, +struct ufshcd_lrb *lrb,enum ufs_trace_str_t str_t) { -struct utp_upiu_req *rq = hba->lrb[tag].ucd_req_ptr; +struct utp_upiu_req *rq = lrb->ucd_req_ptr;struct utp_upiu_header *header;if (!trace_ufshcd_upiu_enabled ())
@@ -417,7 +418,7 @@ static void ufshcd_add_cmd_upiu_trace(struct ufs_hba *hba, unsigned int tag,if (str_t ==UFS_CMD_SEND ) header = &rq->header;else - header = &hba->lrb[tag].ucd_rsp_ptr->header; + header = &lrb->ucd_rsp_ptr->header;trace_ufshcd_upiu (hba, str_t, header, &rq->sc.cdb,UFS_TSF_CDB );
@@ -491,7 +492,7 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,return ;/* trace UPIU also */ -ufshcd_add_cmd_upiu_trace (hba, tag, str_t); +ufshcd_add_cmd_upiu_trace (hba, lrbp, str_t);if (!trace_ufshcd_command_enabled ())return ;
Bart Van Assche
reviewerPATCH v3 9/26 ufs: core: Only call ufshcd_add_command_trace() for SCSI commands
In reply to Bart Van AsscheInstead of checking inside ufshcd_add_command_trace() whether 'cmd' points at a SCSI command, let the caller perform that check. This patch prepares for removing the lrbp->cmd pointer.
--- drivers/ufs/core/ufshcd.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +3 -5
@@ -488,9 +488,6 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,struct request *rq =scsi_cmd_to_rq (cmd);int transfer_len =-1 ; -if (!cmd) -return ; -/* trace UPIU also */ ufshcd_add_cmd_upiu_trace (hba, lrbp, str_t);if (!trace_ufshcd_command_enabled ())
@@ -2365,9 +2362,10 @@ void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag, lrbp->compl_time_stamp =ktime_set (0 ,0 ); lrbp->compl_time_stamp_local_clock =0 ; } -ufshcd_add_command_trace (hba, task_tag,UFS_CMD_SEND ); -if (lrbp->cmd) +if (lrbp->cmd) { +ufshcd_add_command_trace (hba, task_tag,UFS_CMD_SEND );ufshcd_clk_scaling_start_busy (hba); + }if (unlikely (ufshcd_should_inform_monitor (hba, lrbp)))ufshcd_start_monitor (hba, lrbp);
Bart Van Assche
reviewerPATCH v3 10/26 ufs: core: Change the type of one ufshcd_add_command_trace() argument
In reply to Bart Van AsscheChange the 'tag' argument into a SCSI command pointer. This patch prepares for the removal of the hba->lrb[] array.
--- drivers/ufs/core/ufshcd.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/ufs/core/ufshcd.c +6 -6
@@ -475,7 +475,7 @@ static void ufshcd_add_uic_command_trace(struct ufs_hba *hba, ufshcd_readl(hba, REG_UIC_COMMAND_ARG_3)); } -static void ufshcd_add_command_trace (struct ufs_hba *hba,unsigned int tag, +static void ufshcd_add_command_trace (struct ufs_hba *hba,struct scsi_cmnd *cmd,enum ufs_trace_str_t str_t) {u64 lba =0 ;
@@ -483,9 +483,9 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,u32 doorbell =0 ;u32 intr;u32 hwq_id =0 ; -struct ufshcd_lrb *lrbp = &hba->lrb[tag]; -struct scsi_cmnd *cmd = lrbp->cmd;struct request *rq =scsi_cmd_to_rq (cmd); +unsigned int tag = rq->tag; +struct ufshcd_lrb *lrbp = &hba->lrb[tag];int transfer_len =-1 ;/* trace UPIU also */
@@ -503,7 +503,7 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,be32_to_cpu (lrbp->ucd_req_ptr->sc.exp_data_transfer_len); lba =scsi_get_lba (cmd);if (opcode ==WRITE_10 ) - group_id = lrbp->cmd->cmnd[6 ]; + group_id = cmd->cmnd[6 ]; }else if (opcode ==UNMAP ) {/* * The number of Bytes to be unmapped beginning with the lba.
@@ -2363,7 +2363,7 @@ void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag, lrbp->compl_time_stamp_local_clock =0 ; }if (lrbp->cmd) { -ufshcd_add_command_trace (hba, task_tag,UFS_CMD_SEND ); +ufshcd_add_command_trace (hba, lrbp->cmd,UFS_CMD_SEND );ufshcd_clk_scaling_start_busy (hba); }if (unlikely (ufshcd_should_inform_monitor (hba, lrbp)))
@@ -5630,7 +5630,7 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,if (cmd) {if (unlikely (ufshcd_should_inform_monitor (hba, lrbp)))ufshcd_update_monitor (hba, lrbp); -ufshcd_add_command_trace (hba, task_tag,UFS_CMD_COMP ); +ufshcd_add_command_trace (hba, cmd,UFS_CMD_COMP ); cmd->result =ufshcd_transfer_rsp_status (hba, lrbp, cqe);ufshcd_release_scsi_cmd (hba, lrbp);/* Do not touch lrbp after scsi done */
Bart Van Assche
reviewerPATCH v3 11/26 ufs: core: Change the type of one ufshcd_send_command() argument
In reply to Bart Van AsscheChange the 'task_tag' argument into an LRB pointer. This patch prepares for the removal of the hba->lrb[] array.
--- drivers/ufs/core/ufshcd.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/ufs/core/ufshcd.c +6 -7
@@ -2346,14 +2346,13 @@ static void ufshcd_update_monitor(struct ufs_hba *hba, const struct ufshcd_lrb */** * ufshcd_send_command - Send SCSI or device management commands * @hba: per adapter instance -* @task_tag: Task tag of the command +* @lrbp: Local reference block of SCSI command * @hwq: pointer to hardware queue instance */ -static inline -void ufshcd_send_command (struct ufs_hba *hba,unsigned int task_tag, -struct ufs_hw_queue *hwq) +static inline void ufshcd_send_command (struct ufs_hba *hba, +struct ufshcd_lrb *lrbp, +struct ufs_hw_queue *hwq) { -struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];unsigned long flags;if (hba->monitor.enabled) {
@@ -3062,7 +3061,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)if (hba->mcq_enabled) hwq =ufshcd_mcq_req_to_hwq (hba,scsi_cmd_to_rq (cmd)); -ufshcd_send_command (hba, tag, hwq); +ufshcd_send_command (hba, lrbp, hwq); out:if (ufs_trigger_eh (hba)) {
@@ -3307,7 +3306,7 @@ static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,int err;ufshcd_add_query_upiu_trace (hba,UFS_QUERY_SEND , lrbp->ucd_req_ptr); -ufshcd_send_command (hba, tag, hba->dev_cmd_queue); +ufshcd_send_command (hba, lrbp, hba->dev_cmd_queue); err =ufshcd_wait_for_dev_cmd (hba, lrbp, timeout); ufshcd_add_query_upiu_trace(hba, err ? UFS_QUERY_ERR : UFS_QUERY_COMP,
Bart Van Assche
reviewerPATCH v3 12/26 ufs: core: Only call ufshcd_should_inform_monitor() for SCSI commands
In reply to Bart Van Asscheufshcd_should_inform_monitor() only returns 'true' for SCSI commands. Instead of checking inside ufshcd_should_inform_monitor() whether its second argument represents a SCSI command, only call this function for SCSI commands. This patch prepares for removing the lrbp->cmd member.
--- drivers/ufs/core/ufshcd.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c +4 -3
@@ -2289,12 +2289,13 @@ static inline int ufshcd_monitor_opcode2dir(u8 opcode)return -EINVAL ; } +/* Must only be called for SCSI commands. */ static inline bool ufshcd_should_inform_monitor (struct ufs_hba *hba,struct ufshcd_lrb *lrbp) {const struct ufs_hba_monitor *m = &hba->monitor; -return (m->enabled && lrbp && lrbp->cmd && +return (m->enabled && (!m->chunk_size || m->chunk_size == lrbp->cmd->sdb.length) &&ktime_before (hba->monitor.enabled_ts, lrbp->issue_time_stamp)); }
@@ -2364,9 +2365,9 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,if (lrbp->cmd) {ufshcd_add_command_trace (hba, lrbp->cmd,UFS_CMD_SEND );ufshcd_clk_scaling_start_busy (hba); +if (unlikely (ufshcd_should_inform_monitor (hba, lrbp))) +ufshcd_start_monitor (hba, lrbp); } -if (unlikely (ufshcd_should_inform_monitor (hba, lrbp))) -ufshcd_start_monitor (hba, lrbp);if (hba->mcq_enabled) {int utrd_size =sizeof (struct utp_transfer_req_desc );
Bart Van Assche
reviewerPATCH v3 13/26 ufs: core: Change the monitor function argument types
In reply to Bart Van AsschePass a SCSI command pointer instead of a struct ufshcd_lrb pointer. This patch prepares for combining the SCSI command and ufshcd_lrb data structures into a single data structure.
--- drivers/ufs/core/ufshcd.c | 44 +++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 22 deletions(-)
drivers/ufs/core/ufshcd.c +22 -22
@@ -2291,19 +2291,20 @@ static inline int ufshcd_monitor_opcode2dir(u8 opcode)/* Must only be called for SCSI commands. */ static inline bool ufshcd_should_inform_monitor (struct ufs_hba *hba, -struct ufshcd_lrb *lrbp) +struct scsi_cmnd *cmd) {const struct ufs_hba_monitor *m = &hba->monitor; +struct request *rq =scsi_cmd_to_rq (cmd); +struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag]; -return (m->enabled && - (!m->chunk_size || m->chunk_size == lrbp->cmd->sdb.length) && -ktime_before (hba->monitor.enabled_ts, lrbp->issue_time_stamp)); +return m->enabled && + (!m->chunk_size || m->chunk_size == cmd->sdb.length) && +ktime_before (hba->monitor.enabled_ts, lrbp->issue_time_stamp); } -static void ufshcd_start_monitor (struct ufs_hba *hba, -const struct ufshcd_lrb *lrbp) +static void ufshcd_start_monitor (struct ufs_hba *hba,struct scsi_cmnd *cmd) { -int dir =ufshcd_monitor_opcode2dir (*lrbp->cmd->cmnd); +int dir =ufshcd_monitor_opcode2dir (cmd->cmnd[0 ]);unsigned long flags;spin_lock_irqsave (hba->host->host_lock, flags);
@@ -2312,14 +2313,15 @@ static void ufshcd_start_monitor(struct ufs_hba *hba,spin_unlock_irqrestore (hba->host->host_lock, flags); } -static void ufshcd_update_monitor (struct ufs_hba *hba,const struct ufshcd_lrb *lrbp) +static void ufshcd_update_monitor (struct ufs_hba *hba,struct scsi_cmnd *cmd) { -int dir =ufshcd_monitor_opcode2dir (*lrbp->cmd->cmnd); +struct request *req =scsi_cmd_to_rq (cmd); +struct ufshcd_lrb *lrbp = &hba->lrb[req->tag]; +int dir =ufshcd_monitor_opcode2dir (cmd->cmnd[0 ]);unsigned long flags;spin_lock_irqsave (hba->host->host_lock, flags);if (dir >=0 && hba->monitor.nr_queued[dir] >0 ) { -const struct request *req =scsi_cmd_to_rq (lrbp->cmd);struct ufs_hba_monitor *m = &hba->monitor;ktime_t now, inc, lat;
@@ -2354,6 +2356,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,struct ufshcd_lrb *lrbp, struct ufs_hw_queue *hwq) { + struct scsi_cmnd *cmd = lrbp->cmd;unsigned long flags;if (hba->monitor.enabled) {
@@ -2362,11 +2365,11 @@ static inline void ufshcd_send_command(struct ufs_hba *hba, lrbp->compl_time_stamp =ktime_set (0 ,0 ); lrbp->compl_time_stamp_local_clock =0 ; } -if (lrbp->cmd) { -ufshcd_add_command_trace (hba, lrbp->cmd,UFS_CMD_SEND ); +if (cmd) { +ufshcd_add_command_trace (hba, cmd,UFS_CMD_SEND );ufshcd_clk_scaling_start_busy (hba); -if (unlikely (ufshcd_should_inform_monitor (hba, lrbp))) -ufshcd_start_monitor (hba, lrbp); +if (unlikely (ufshcd_should_inform_monitor (hba, cmd))) +ufshcd_start_monitor (hba, cmd); }if (hba->mcq_enabled) {
@@ -2382,8 +2385,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba, }else {spin_lock_irqsave (&hba->outstanding_lock, flags);if (hba->vops && hba->vops->setup_xfer_req) - hba->vops->setup_xfer_req (hba, lrbp->task_tag, - !!lrbp->cmd); + hba->vops->setup_xfer_req (hba, lrbp->task_tag, !!cmd);__set_bit (lrbp->task_tag, &hba->outstanding_reqs);ufshcd_writel (hba,1 << lrbp->task_tag,REG_UTP_TRANSFER_REQ_DOOR_BELL );
@@ -5617,19 +5619,17 @@ void ufshcd_release_scsi_cmd(struct ufs_hba *hba,void ufshcd_compl_one_cqe (struct ufs_hba *hba,int task_tag,struct cq_entry *cqe) { -struct ufshcd_lrb *lrbp; -struct scsi_cmnd *cmd; +struct ufshcd_lrb *lrbp = &hba->lrb[task_tag]; +struct scsi_cmnd *cmd = lrbp->cmd;enum utp_ocs ocs; - lrbp = &hba->lrb[task_tag];if (hba->monitor.enabled) { lrbp->compl_time_stamp =ktime_get (); lrbp->compl_time_stamp_local_clock =local_clock (); } - cmd = lrbp->cmd;if (cmd) { -if (unlikely (ufshcd_should_inform_monitor (hba, lrbp))) -ufshcd_update_monitor (hba, lrbp); +if (unlikely (ufshcd_should_inform_monitor (hba, cmd))) +ufshcd_update_monitor (hba, cmd);ufshcd_add_command_trace (hba, cmd,UFS_CMD_COMP ); cmd->result =ufshcd_transfer_rsp_status (hba, lrbp, cqe);ufshcd_release_scsi_cmd (hba, lrbp);
Bart Van Assche
reviewerPATCH v3 14/26 ufs: core: Rework ufshcd_mcq_compl_pending_transfer()
In reply to Bart Van AsscheReplace a tag loop with blk_mq_tagset_busy_iter(). This patch prepares for removing the hba->lrb[] array.
--- drivers/ufs/core/ufshcd.c | 80 ++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 34 deletions(-)
drivers/ufs/core/ufshcd.c +46 -34
@@ -5714,6 +5714,48 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)return completed_reqs !=0 ; } +static bool ufshcd_mcq_force_compl_one (struct request *rq,void *priv) +{ +struct scsi_cmnd *cmd =blk_mq_rq_to_pdu (rq); +struct scsi_device *sdev = rq->q->queuedata; +struct Scsi_Host *shost = sdev->host; +struct ufs_hba *hba =shost_priv (shost); +struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag]; +struct ufs_hw_queue *hwq =ufshcd_mcq_req_to_hwq (hba, rq); + +if (!hwq) +return true; + +ufshcd_mcq_compl_all_cqes_lock (hba, hwq); + +/* +* For those cmds of which the cqes are not present in the cq, complete +* them explicitly. +*/ + scoped_guard(spinlock_irqsave, &hwq->cq_lock) { +if (cmd && !test_bit (SCMD_STATE_COMPLETE , &cmd->state)) { +set_host_byte (cmd,DID_REQUEUE ); +ufshcd_release_scsi_cmd (hba, lrbp); +scsi_done (cmd); + } + } + +return true; +} + +static bool ufshcd_mcq_compl_one (struct request *rq,void *priv) +{ +struct scsi_device *sdev = rq->q->queuedata; +struct Scsi_Host *shost = sdev->host; +struct ufs_hba *hba =shost_priv (shost); +struct ufs_hw_queue *hwq =ufshcd_mcq_req_to_hwq (hba, rq); + +if (hwq) +ufshcd_mcq_poll_cqe_lock (hba, hwq); + +return true; +} +/** * ufshcd_mcq_compl_pending_transfer - MCQ mode function. It is * invoked from the error handler context or ufshcd_host_reset_and_restore()
@@ -5728,40 +5770,10 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)static void ufshcd_mcq_compl_pending_transfer (struct ufs_hba *hba,bool force_compl) { -struct ufs_hw_queue *hwq; -struct ufshcd_lrb *lrbp; -struct scsi_cmnd *cmd; -unsigned long flags; -int tag; - -for (tag =0 ; tag < hba->nutrs; tag++) { - lrbp = &hba->lrb[tag]; - cmd = lrbp->cmd; -if (!ufshcd_cmd_inflight (cmd) || -test_bit (SCMD_STATE_COMPLETE , &cmd->state)) -continue ; - - hwq =ufshcd_mcq_req_to_hwq (hba,scsi_cmd_to_rq (cmd)); -if (!hwq) -continue ; - -if (force_compl) { -ufshcd_mcq_compl_all_cqes_lock (hba, hwq); -/* -* For those cmds of which the cqes are not present -* in the cq, complete them explicitly. -*/ -spin_lock_irqsave (&hwq->cq_lock, flags); -if (cmd && !test_bit (SCMD_STATE_COMPLETE , &cmd->state)) { -set_host_byte (cmd,DID_REQUEUE ); -ufshcd_release_scsi_cmd (hba, lrbp); -scsi_done (cmd); - } -spin_unlock_irqrestore (&hwq->cq_lock, flags); - }else { -ufshcd_mcq_poll_cqe_lock (hba, hwq); - } - } +blk_mq_tagset_busy_iter (&hba->host->tag_set, + force_compl ? ufshcd_mcq_force_compl_one : + ufshcd_mcq_compl_one, +NULL ); }/**
Bart Van Assche
reviewerPATCH v3 15/26 ufs: core: Rework ufshcd_eh_device_reset_handler()
In reply to Bart Van AsscheMerge the MCQ mode and legacy mode loops into a single loop. This patch prepares for optimizing the hot path by removing the direct hba->lrb[] accesses from ufshcd_eh_device_reset_handler().
--- drivers/ufs/core/ufshcd.c | 84 ++++++++++++++++++--------------------- 1 file changed, 38 insertions(+), 46 deletions(-)
drivers/ufs/core/ufshcd.c +38 -46
@@ -7534,6 +7534,36 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *rreturn err ? : result; } +static bool ufshcd_clear_lu_cmds (struct request *req,void *priv) +{ +struct scsi_cmnd *cmd =blk_mq_rq_to_pdu (req); +struct scsi_device *sdev = cmd->device; +struct Scsi_Host *shost = sdev->host; +struct ufs_hba *hba =shost_priv (shost); +const u64 lun = *(u64 *)priv; +const u32 tag = req->tag; + +if (sdev->lun != lun) +return true; + +if (ufshcd_clear_cmd (hba, tag) <0 ) { +dev_err (hba->dev,"%s: failed to clear request %d\n" , __func__, + tag); +return true; + } + +if (hba->mcq_enabled) { +struct ufs_hw_queue *hwq =ufshcd_mcq_req_to_hwq (hba, req); + +if (hwq) +ufshcd_mcq_poll_cqe_lock (hba, hwq); +return true; + } + +ufshcd_compl_one_cqe (hba, tag,NULL ); +return true; +} +/** * ufshcd_eh_device_reset_handler() - Reset a single logical unit. * @cmd: SCSI command pointer
@@ -7542,12 +7572,8 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r*/ static int ufshcd_eh_device_reset_handler (struct scsi_cmnd *cmd) { -unsigned long flags, pending_reqs =0 , not_cleared =0 ;struct Scsi_Host *host;struct ufs_hba *hba; -struct ufs_hw_queue *hwq; -struct ufshcd_lrb *lrbp; -u32 pos, not_cleared_mask =0 ;int err;u8 resp =0xF , lun;
@@ -7556,50 +7582,16 @@ static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd) lun =ufshcd_scsi_to_upiu_lun (cmd->device->lun); err =ufshcd_issue_tm_cmd (hba, lun,0 ,UFS_LOGICAL_RESET , &resp); -if (err || resp !=UPIU_TASK_MANAGEMENT_FUNC_COMPL ) { -if (!err) - err = resp; - goto out; - } - -if (hba->mcq_enabled) { -for (pos =0 ; pos < hba->nutrs; pos++) { - lrbp = &hba->lrb[pos]; -if (ufshcd_cmd_inflight (lrbp->cmd) && - lrbp->lun == lun) { -ufshcd_clear_cmd (hba, pos); - hwq =ufshcd_mcq_req_to_hwq (hba,scsi_cmd_to_rq (lrbp->cmd)); -ufshcd_mcq_poll_cqe_lock (hba, hwq); - } - } - err =0 ; - goto out; - } - -/* clear the commands that were pending for corresponding LUN */ -spin_lock_irqsave (&hba->outstanding_lock, flags); -for_each_set_bit (pos, &hba->outstanding_reqs, hba->nutrs) -if (hba->lrb[pos].lun == lun) -__set_bit (pos, &pending_reqs); - hba->outstanding_reqs &= ~pending_reqs; -spin_unlock_irqrestore (&hba->outstanding_lock, flags); - -for_each_set_bit (pos, &pending_reqs, hba->nutrs) { -if (ufshcd_clear_cmd (hba, pos) <0 ) { -spin_lock_irqsave (&hba->outstanding_lock, flags); - not_cleared =1U << pos & -ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); - hba->outstanding_reqs |= not_cleared; - not_cleared_mask |= not_cleared; -spin_unlock_irqrestore (&hba->outstanding_lock, flags); - -dev_err (hba->dev,"%s: failed to clear request %d\n" , - __func__, pos); - } +if (err) { + }else if (resp !=UPIU_TASK_MANAGEMENT_FUNC_COMPL ) { + err = resp; + }else { +/* clear the commands that were pending for corresponding LUN */ +blk_mq_tagset_busy_iter (&hba->host->tag_set, + ufshcd_clear_lu_cmds, + &cmd->device->lun); } -__ufshcd_transfer_req_compl (hba, pending_reqs & ~not_cleared_mask); -out: hba->req_abort_count =0 ;ufshcd_update_evt_hist (hba,UFS_EVT_DEV_RESET , (u32 )err);if (!err) {
Bart Van Assche
reviewerPATCH v3 16/26 ufs: core: Allocate more commands for the SCSI host
In reply to Bart Van AsschePrepare for allocating the SCSI host earlier by making the SCSI host queue depth independent of the queue depth supported by the UFS device.
--- drivers/ufs/core/ufs-mcq.c | 20 +++++--------------- drivers/ufs/core/ufshcd-priv.h | 2 +- drivers/ufs/core/ufshcd.c | 17 +++++++++++++++-- 3 files changed, 21 insertions(+), 18 deletions(-)
drivers/ufs/core/ufs-mcq.c +5 -15
@@ -130,17 +130,15 @@ unsigned int ufshcd_mcq_queue_cfg_addr(struct ufs_hba *hba)EXPORT_SYMBOL_GPL (ufshcd_mcq_queue_cfg_addr);/** -* ufshcd_mcq_decide_queue_depth - decide the queue depth +* ufshcd_get_hba_mac - Maximum number of commands supported by the host +* controller. * @hba: per adapter instance * -* Return: queue-depth on success, non-zero on error +* Return: queue depth on success; negative upon error. * -* MAC - Max. Active Command of the Host Controller (HC) -* HC wouldn't send more than this commands to the device. -* Calculates and adjusts the queue depth based on the depth -* supported by the HC and ufs device. +* MAC = Maximum number of Active Commands supported by the Host Controller. */ -int ufshcd_mcq_decide_queue_depth (struct ufs_hba *hba) +int ufshcd_get_hba_mac (struct ufs_hba *hba) {int mac;
@@ -160,14 +158,6 @@ int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba)if (mac <0 ) goto err; -WARN_ON_ONCE (!hba->dev_info.bqueuedepth); -/* -* max. value of bqueuedepth = 256, mac is host dependent. -* It is mandatory for UFS device to define bQueueDepth if -* shared queuing architecture is enabled. -*/ -return min_t (int, mac, hba->dev_info.bqueuedepth); - err:dev_err (hba->dev,"Failed to get mac, err=%d\n" , mac);return mac;
drivers/ufs/core/ufshcd-priv.h +1 -1
@@ -65,7 +65,7 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag, struct cq_entry *cqe);int ufshcd_mcq_init (struct ufs_hba *hba);void ufshcd_mcq_disable (struct ufs_hba *hba); -int ufshcd_mcq_decide_queue_depth (struct ufs_hba *hba); +int ufshcd_get_hba_mac (struct ufs_hba *hba);int ufshcd_mcq_memory_alloc (struct ufs_hba *hba);struct ufs_hw_queue *ufshcd_mcq_req_to_hwq (struct ufs_hba *hba,struct request *req);
drivers/ufs/core/ufshcd.c +15 -2
@@ -8434,10 +8434,11 @@ static void ufs_init_rtc(struct ufs_hba *hba, u8 *desc_buf)static int ufs_get_device_desc (struct ufs_hba *hba) { +struct ufs_dev_info *dev_info = &hba->dev_info; +struct Scsi_Host *shost = hba->host;int err;u8 model_index;u8 *desc_buf; -struct ufs_dev_info *dev_info = &hba->dev_info; desc_buf =kzalloc (QUERY_DESC_MAX_SIZE ,GFP_KERNEL );if (!desc_buf) {
@@ -8465,6 +8466,18 @@ static int ufs_get_device_desc(struct ufs_hba *hba) desc_buf[DEVICE_DESC_PARAM_SPEC_VER +1 ]; dev_info->bqueuedepth = desc_buf[DEVICE_DESC_PARAM_Q_DPTH ]; +/* +* According to the UFS standard, the UFS device queue depth +* (bQueueDepth) must be in the range 1..255 if the shared queueing +* architecture is supported. bQueueDepth is zero if the shared queueing +* architecture is not supported. +*/ +if (dev_info->bqueuedepth) + shost->cmd_per_lun =min (hba->nutrs, dev_info->bqueuedepth) - +UFSHCD_NUM_RESERVED ; +else + shost->cmd_per_lun = shost->can_queue; + dev_info->rtt_cap = desc_buf[DEVICE_DESC_PARAM_RTT_CAP ]; dev_info->hid_sup = get_unaligned_be32(desc_buf +
@@ -8870,7 +8883,7 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba)int ret;int old_nutrs = hba->nutrs; - ret =ufshcd_mcq_decide_queue_depth (hba); + ret =ufshcd_get_hba_mac (hba);if (ret <0 )return ret;
Bart Van Assche
reviewerPATCH v3 17/26 ufs: core: Allocate the SCSI host earlier
In reply to Bart Van AsscheCall ufshcd_add_scsi_host() before any UPIU commands are sent to the UFS device. This patch prepares for letting ufshcd_add_scsi_host() allocate memory for both SCSI and UPIU commands.
--- drivers/ufs/core/ufshcd.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +12 -5
@@ -10576,6 +10576,9 @@ static int ufshcd_add_scsi_host(struct ufs_hba *hba) {int err; +WARN_ON_ONCE (!hba->host->can_queue); +WARN_ON_ONCE (!hba->host->cmd_per_lun); +if (is_mcq_supported (hba)) {ufshcd_mcq_enable (hba); err =ufshcd_alloc_mcq (hba);
@@ -10726,7 +10729,11 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)ufshcd_host_memory_configure (hba); host->can_queue = hba->nutrs -UFSHCD_NUM_RESERVED ; - host->cmd_per_lun = hba->nutrs -UFSHCD_NUM_RESERVED ; +/* +* Set the queue depth for WLUNs. ufs_get_device_desc() will increase +* host->cmd_per_lun to a larger value. +*/ + host->cmd_per_lun =1 ; host->max_id =UFSHCD_MAX_ID ; host->max_lun =UFS_MAX_LUNS ; host->max_channel =UFSHCD_MAX_CHANNEL ;
@@ -10817,6 +10824,10 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)FIELD_PREP (UFSHCI_AHIBERN8_SCALE_MASK ,3 ); } + err =ufshcd_add_scsi_host (hba); +if (err) + goto out_disable; +/* Hold auto suspend until async scan completes */ pm_runtime_get_sync (dev);
@@ -10867,10 +10878,6 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)if (err) goto out_disable; - err =ufshcd_add_scsi_host (hba); -if (err) - goto out_disable; -async_schedule (ufshcd_async_scan, hba);ufs_sysfs_add_nodes (hba->dev);
Bart Van Assche
reviewerPATCH v3 18/26 ufs: core: Call ufshcd_init_lrb() later
In reply to Bart Van AsscheCall ufshcd_init_lrb() from inside ufshcd_setup_dev_cmd() instead of ufshcd_host_memory_configure(). This patch prepares for calling ufshcd_host_memory_configure() before the information is available that is required to call ufshcd_setup_dev_cmd().
--- drivers/ufs/core/ufshcd.c | 53 ++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 26 deletions(-)
drivers/ufs/core/ufshcd.c +27 -26
@@ -2900,8 +2900,32 @@ static void ufshcd_comp_scsi_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)ufshcd_prepare_utp_scsi_cmd_upiu (lrbp, upiu_flags); } -static void __ufshcd_setup_cmd (struct ufshcd_lrb *lrbp,struct scsi_cmnd *cmd,u8 lun,int tag) +static void ufshcd_init_lrb (struct ufs_hba *hba,struct ufshcd_lrb *lrb,int i) +{ +struct utp_transfer_cmd_desc *cmd_descp = + (void *)hba->ucdl_base_addr + i *ufshcd_get_ucd_size (hba); +struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr; +dma_addr_t cmd_desc_element_addr = + hba->ucdl_dma_addr + i *ufshcd_get_ucd_size (hba); +u16 response_offset =le16_to_cpu (utrdlp[i].response_upiu_offset); +u16 prdt_offset =le16_to_cpu (utrdlp[i].prd_table_offset); + + lrb->utr_descriptor_ptr = utrdlp + i; + lrb->utrd_dma_addr = + hba->utrdl_dma_addr + i *sizeof (struct utp_transfer_req_desc ); + lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp->command_upiu; + lrb->ucd_req_dma_addr = cmd_desc_element_addr; + lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu; + lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset; + lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table; + lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset; +} + +static void __ufshcd_setup_cmd (struct ufs_hba *hba,struct ufshcd_lrb *lrbp, +struct scsi_cmnd *cmd,u8 lun,int tag) { +ufshcd_init_lrb (hba, lrbp, tag); +memset (lrbp->ucd_req_ptr,0 ,sizeof (*lrbp->ucd_req_ptr)); lrbp->cmd = cmd;
@@ -2913,7 +2937,7 @@ static void __ufshcd_setup_cmd(struct ufshcd_lrb *lrbp, struct scsi_cmnd *cmd, ustatic void ufshcd_setup_scsi_cmd (struct ufs_hba *hba,struct ufshcd_lrb *lrbp,struct scsi_cmnd *cmd,u8 lun,int tag) { -__ufshcd_setup_cmd (lrbp, cmd, lun, tag); +__ufshcd_setup_cmd (hba, lrbp, cmd, lun, tag); lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed (hba); lrbp->req_abort_skip = false;
@@ -2968,27 +2992,6 @@ static void ufshcd_map_queues(struct Scsi_Host *shost) } } -static void ufshcd_init_lrb (struct ufs_hba *hba,struct ufshcd_lrb *lrb,int i) -{ -struct utp_transfer_cmd_desc *cmd_descp = (void *)hba->ucdl_base_addr + - i *ufshcd_get_ucd_size (hba); -struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr; -dma_addr_t cmd_desc_element_addr = hba->ucdl_dma_addr + - i *ufshcd_get_ucd_size (hba); -u16 response_offset =le16_to_cpu (utrdlp[i].response_upiu_offset); -u16 prdt_offset =le16_to_cpu (utrdlp[i].prd_table_offset); - - lrb->utr_descriptor_ptr = utrdlp + i; - lrb->utrd_dma_addr = hba->utrdl_dma_addr + - i *sizeof (struct utp_transfer_req_desc ); - lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp->command_upiu; - lrb->ucd_req_dma_addr = cmd_desc_element_addr; - lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu; - lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset; - lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table; - lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset; -} -/** * ufshcd_queuecommand - main entry point for SCSI requests * @host: SCSI host pointer
@@ -3081,7 +3084,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)static void ufshcd_setup_dev_cmd (struct ufs_hba *hba,struct ufshcd_lrb *lrbp,enum dev_cmd_type cmd_type,u8 lun,int tag) { -__ufshcd_setup_cmd (lrbp,NULL , lun, tag); +__ufshcd_setup_cmd (hba, lrbp,NULL , lun, tag); lrbp->intr_cmd = true;/* No interrupt aggregation */ hba->dev_cmd.type = cmd_type; }
@@ -4039,8 +4042,6 @@ static void ufshcd_host_memory_configure(struct ufs_hba *hba) utrdlp[i].response_upiu_length =cpu_to_le16 (ALIGNED_UPIU_SIZE >>2 ); } - -ufshcd_init_lrb (hba, &hba->lrb[i], i); } }
Bart Van Assche
reviewerPATCH v3 19/26 ufs: core: Use hba->reserved_slot
In reply to Bart Van AsscheUse hba->reserved_slot instead of open-coding it. This patch prepares for changing the value of hba->reserved_slot.
--- drivers/ufs/core/ufs-mcq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ufs/core/ufs-mcq.c +1 -1
@@ -536,7 +536,7 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)if (hba->quirks &UFSHCD_QUIRK_MCQ_BROKEN_RTC )return -ETIMEDOUT ; -if (task_tag != hba->nutrs -UFSHCD_NUM_RESERVED ) { +if (task_tag != hba->reserved_slot) {if (!cmd)return -EINVAL ; hwq =ufshcd_mcq_req_to_hwq (hba,scsi_cmd_to_rq (cmd));
Bart Van Assche
reviewerPATCH v3 20/26 ufs: core: Make the reserved slot a reserved request
In reply to Bart Van AsscheInstead of letting the SCSI core allocate hba->nutrs - 1 commands, let the SCSI core allocate hba->nutrs commands, set the number of reserved tags to 1 and use the reserved tag for device management commands. This patch changes the 'reserved slot' from hba->nutrs - 1 into 0 because the block layer reserves the smallest tags for reserved commands.
--- drivers/ufs/core/ufshcd.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/ufs/core/ufshcd.c +4 -2
@@ -2472,7 +2472,7 @@ static inline int ufshcd_hba_capabilities(struct ufs_hba *hba) hba->nutrs = (hba->capabilities &MASK_TRANSFER_REQUESTS_SLOTS_SDB ) +1 ; hba->nutmrs = ((hba->capabilities &MASK_TASK_MANAGEMENT_REQUEST_SLOTS ) >>16 ) +1 ; - hba->reserved_slot = hba->nutrs -1 ; + hba->reserved_slot =0 ; hba->nortt =FIELD_GET (MASK_NUMBER_OUTSTANDING_RTT , hba->capabilities) +1 ;
@@ -8910,7 +8910,8 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba) goto err; hba->host->can_queue = hba->nutrs -UFSHCD_NUM_RESERVED ; - hba->reserved_slot = hba->nutrs -UFSHCD_NUM_RESERVED ; + hba->host->nr_reserved_cmds =UFSHCD_NUM_RESERVED ; + hba->reserved_slot =0 ;return 0 ; err:
@@ -10735,6 +10736,7 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)* host->cmd_per_lun to a larger value. */ host->cmd_per_lun =1 ; + host->nr_reserved_cmds =UFSHCD_NUM_RESERVED ; host->max_id =UFSHCD_MAX_ID ; host->max_lun =UFS_MAX_LUNS ; host->max_channel =UFSHCD_MAX_CHANNEL ;
Bart Van Assche
reviewerPATCH v3 21/26 ufs: core: Do not clear driver-private command data
In reply to Bart Van AsscheTell the SCSI core to skip the memset() call that clears driver-private data because __ufshcd_setup_cmd() performs all necessary initialization.
--- drivers/ufs/core/ufshcd.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/ufs/core/ufshcd.c +10 -0
@@ -2992,6 +2992,15 @@ static void ufshcd_map_queues(struct Scsi_Host *shost) } } +/* +* The only purpose of this function is to make the SCSI core skip the memset() +* call for the private command data. +*/ +static int ufshcd_init_cmd_priv (struct Scsi_Host *host,struct scsi_cmnd *cmd) +{ +return 0 ; +} +/** * ufshcd_queuecommand - main entry point for SCSI requests * @host: SCSI host pointer
@@ -9154,6 +9163,7 @@ static const struct scsi_host_template ufshcd_driver_template = { .name = UFSHCD, .proc_name = UFSHCD, .map_queues = ufshcd_map_queues, + .init_cmd_priv = ufshcd_init_cmd_priv, .queuecommand = ufshcd_queuecommand, .mq_poll = ufshcd_poll, .sdev_init = ufshcd_sdev_init,
Bart Van Assche
reviewerPATCH v3 22/26 ufs: core: Optimize the hot path
In reply to Bart Van AsscheSet .cmd_size in the SCSI host template such that the SCSI core makes struct scsi_cmnd and struct ufshcd_lrb adjacent. Convert the cmd->lrbp and lrbp->cmd memory loads into pointer offset calculations. Remove the data structure members that became superfluous, namely ufshcd_lrb.cmd and ufs_hba.lrb.
--- drivers/ufs/core/ufs-mcq.c | 9 +- drivers/ufs/core/ufshcd-crypto.h | 18 ++- drivers/ufs/core/ufshcd-priv.h | 41 +++++- drivers/ufs/core/ufshcd.c | 232 ++++++++++++++++--------------- include/ufs/ufshcd.h | 5 - 5 files changed, 177 insertions(+), 128 deletions(-)
drivers/ufs/core/ufs-mcq.c +5 -4
@@ -526,8 +526,8 @@ static int ufshcd_mcq_sq_start(struct ufs_hba *hba, struct ufs_hw_queue *hwq)*/ int ufshcd_mcq_sq_cleanup (struct ufs_hba *hba,int task_tag) { -struct ufshcd_lrb *lrbp = &hba->lrb[task_tag]; -struct scsi_cmnd *cmd = lrbp->cmd; +struct scsi_cmnd *cmd =ufshcd_tag_to_cmd (hba, task_tag); +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);struct ufs_hw_queue *hwq;void __iomem *reg, *opr_sqd_base;u32 nexus, id, val;
@@ -612,7 +612,8 @@ static void ufshcd_mcq_nullify_sqe(struct utp_transfer_req_desc *utrd)static bool ufshcd_mcq_sqe_search (struct ufs_hba *hba,struct ufs_hw_queue *hwq,int task_tag) { -struct ufshcd_lrb *lrbp = &hba->lrb[task_tag]; +struct scsi_cmnd *cmd =ufshcd_tag_to_cmd (hba, task_tag); +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);struct utp_transfer_req_desc *utrd;__le64 cmd_desc_base_addr;bool ret = false;
@@ -663,7 +664,7 @@ int ufshcd_mcq_abort(struct scsi_cmnd *cmd)struct Scsi_Host *host = cmd->device->host;struct ufs_hba *hba =shost_priv (host);int tag =scsi_cmd_to_rq (cmd)->tag; -struct ufshcd_lrb *lrbp = &hba->lrb[tag]; +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);struct ufs_hw_queue *hwq;int err;
drivers/ufs/core/ufshcd-crypto.h +11 -7
@@ -38,10 +38,10 @@ ufshcd_prepare_req_desc_hdr_crypto(struct ufshcd_lrb *lrbp, }static inline int ufshcd_crypto_fill_prdt (struct ufs_hba *hba, -struct ufshcd_lrb *lrbp) +struct scsi_cmnd *cmd) { -struct scsi_cmnd *cmd = lrbp->cmd;const struct bio_crypt_ctx *crypt_ctx =scsi_cmd_to_rq (cmd)->crypt_ctx; +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);if (crypt_ctx && hba->vops && hba->vops->fill_crypto_prdt) return hba->vops->fill_crypto_prdt(hba, crypt_ctx,
@@ -51,17 +51,19 @@ static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba, }static inline void ufshcd_crypto_clear_prdt (struct ufs_hba *hba, -struct ufshcd_lrb *lrbp) +struct scsi_cmnd *cmd) { +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd); +if (!(hba->quirks &UFSHCD_QUIRK_KEYS_IN_PRDT ))return ; -if (!(scsi_cmd_to_rq (lrbp->cmd)->crypt_ctx)) +if (!(scsi_cmd_to_rq (cmd)->crypt_ctx))return ;/* Zeroize the PRDT because it can contain cryptographic keys. */ memzero_explicit (lrbp->ucd_prdt_ptr, -ufshcd_sg_entry_size (hba) *scsi_sg_count (lrbp->cmd)); +ufshcd_sg_entry_size (hba) *scsi_sg_count (cmd)); }bool ufshcd_crypto_enable (struct ufs_hba *hba);
@@ -82,13 +84,15 @@ ufshcd_prepare_req_desc_hdr_crypto(struct ufshcd_lrb *lrbp, struct request_desc_header *h) { }static inline int ufshcd_crypto_fill_prdt (struct ufs_hba *hba, -struct ufshcd_lrb *lrbp) +struct scsi_cmnd *cmd) {return 0 ; }static inline void ufshcd_crypto_clear_prdt (struct ufs_hba *hba, -struct ufshcd_lrb *lrbp) { } +struct scsi_cmnd *cmd) +{ +}static inline bool ufshcd_crypto_enable (struct ufs_hba *hba) {
drivers/ufs/core/ufshcd-priv.h +39 -2
53 lines not on this page · raw message
drivers/ufs/core/ufshcd.c +122 -110
631 lines not on this page · raw message
include/ufs/ufshcd.h +0 -5
@@ -161,7 +161,6 @@ struct ufs_pm_lvl_states {* @ucd_prdt_dma_addr: PRDT dma address for debug * @ucd_rsp_dma_addr: UPIU response dma address for debug * @ucd_req_dma_addr: UPIU request dma address for debug -* @cmd: pointer to SCSI command * @scsi_status: SCSI status of the command * @command_type: SCSI, UFS, Query. * @task_tag: Task tag of the command
@@ -186,7 +185,6 @@ struct ufshcd_lrb {dma_addr_t ucd_rsp_dma_addr;dma_addr_t ucd_prdt_dma_addr; -struct scsi_cmnd *cmd;int scsi_status;int command_type;
@@ -857,7 +855,6 @@ enum ufshcd_mcq_opr {* @spm_lvl: desired UFS power management level during system PM. * @pm_op_in_progress: whether or not a PM operation is in progress. * @ahit: value of Auto-Hibernate Idle Timer register. -* @lrb: local reference block * @outstanding_tasks: Bits representing outstanding task requests * @outstanding_lock: Protects @outstanding_reqs. * @outstanding_reqs: Bits representing outstanding transfer requests
@@ -999,8 +996,6 @@ struct ufs_hba {/* Auto-Hibernate Idle Timer register value */ u32 ahit; -struct ufshcd_lrb *lrb; -unsigned long outstanding_tasks;spinlock_t outstanding_lock;unsigned long outstanding_reqs;
Bart Van Assche
reviewerPATCH v3 23/26 ufs: core: Pass a SCSI pointer instead of an LRB pointer
In reply to Bart Van AsschePrepare for removing the ufshcd_lrb task_tag member.
--- drivers/ufs/core/ufshcd.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
drivers/ufs/core/ufshcd.c +11 -10
@@ -2818,12 +2818,13 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd,/** * ufshcd_prepare_utp_query_req_upiu() - fill the utp_transfer_req_desc for query request * @hba: UFS hba -* @lrbp: local reference block pointer +* @cmd: SCSI command pointer * @upiu_flags: flags */ static void ufshcd_prepare_utp_query_req_upiu (struct ufs_hba *hba, -struct ufshcd_lrb *lrbp,u8 upiu_flags) +struct scsi_cmnd *cmd,u8 upiu_flags) { +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;struct ufs_query *query = &hba->dev_cmd.query;u16 len =be16_to_cpu (query->request.upiu_req.length);
@@ -2852,8 +2853,9 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,memcpy (ucd_req_ptr +1 , query->descriptor, len); } -static inline void ufshcd_prepare_utp_nop_upiu (struct ufshcd_lrb *lrbp) +static inline void ufshcd_prepare_utp_nop_upiu (struct scsi_cmnd *cmd) { +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;memset (ucd_req_ptr,0 ,sizeof (struct utp_upiu_req ));
@@ -2868,22 +2870,23 @@ static inline void ufshcd_prepare_utp_nop_upiu(struct ufshcd_lrb *lrbp)* ufshcd_compose_devman_upiu - UFS Protocol Information Unit(UPIU) * for Device Management Purposes * @hba: per adapter instance -* @lrbp: pointer to local reference block +* @cmd: SCSI command pointer * * Return: 0 upon success; < 0 upon failure. */ static int ufshcd_compose_devman_upiu (struct ufs_hba *hba, -struct ufshcd_lrb *lrbp) +struct scsi_cmnd *cmd) { +struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);u8 upiu_flags;int ret =0 ;ufshcd_prepare_req_desc_hdr (hba, lrbp, &upiu_flags,DMA_NONE ,0 );if (hba->dev_cmd.type ==DEV_CMD_TYPE_QUERY ) -ufshcd_prepare_utp_query_req_upiu (hba, lrbp, upiu_flags); +ufshcd_prepare_utp_query_req_upiu (hba, cmd, upiu_flags);else if (hba->dev_cmd.type ==DEV_CMD_TYPE_NOP ) -ufshcd_prepare_utp_nop_upiu (lrbp); +ufshcd_prepare_utp_nop_upiu (cmd);else ret = -EINVAL ;
@@ -3121,11 +3124,9 @@ static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,static int ufshcd_compose_dev_cmd (struct ufs_hba *hba,struct scsi_cmnd *cmd,enum dev_cmd_type cmd_type,int tag) { -struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd); -ufshcd_setup_dev_cmd (hba, cmd, cmd_type,0 , tag); -return ufshcd_compose_devman_upiu (hba, lrbp); +return ufshcd_compose_devman_upiu (hba, cmd); }/*
Bart Van Assche
reviewerPATCH v3 24/26 ufs: core: Remove the ufshcd_lrb task_tag member
In reply to Bart Van AsscheRemove the ufshcd_lrb task_tag member and use scsi_cmd_to_rq(cmd)->tag instead. Use rq->tag instead of lrbp->task_tag.
--- drivers/ufs/core/ufshcd.c | 62 +++++++++++++++++++-------------------- include/ufs/ufshcd.h | 1 - 2 files changed, 30 insertions(+), 33 deletions(-)
drivers/ufs/core/ufshcd.c +30 -32
@@ -601,7 +601,7 @@ static void ufshcd_print_tr(struct ufs_hba *hba, struct scsi_cmnd *cmd, bool pr_prdt) {struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd); -const int tag = lrbp->task_tag; +const int tag =scsi_cmd_to_rq (cmd)->tag;int prdt_length;if (hba->monitor.enabled) {
@@ -2365,6 +2365,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba, struct ufs_hw_queue *hwq) {struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd); +const int tag =scsi_cmd_to_rq (cmd)->tag;unsigned long flags;if (hba->monitor.enabled) {
@@ -2393,11 +2394,10 @@ static inline void ufshcd_send_command(struct ufs_hba *hba, }else {spin_lock_irqsave (&hba->outstanding_lock, flags);if (hba->vops && hba->vops->setup_xfer_req) - hba->vops->setup_xfer_req (hba, lrbp->task_tag, + hba->vops->setup_xfer_req (hba, tag,ufshcd_is_scsi_cmd (cmd)); -__set_bit (lrbp->task_tag, &hba->outstanding_reqs); -ufshcd_writel (hba,1 << lrbp->task_tag, -REG_UTP_TRANSFER_REQ_DOOR_BELL ); +__set_bit (tag, &hba->outstanding_reqs); +ufshcd_writel (hba,1 << tag,REG_UTP_TRANSFER_REQ_DOOR_BELL );spin_unlock_irqrestore (&hba->outstanding_lock, flags); } }
@@ -2794,6 +2794,7 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd, u8 upiu_flags) {struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd); +const int tag =scsi_cmd_to_rq (cmd)->tag;struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;unsigned short cdb_len;
@@ -2801,11 +2802,11 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd, .transaction_code = UPIU_TRANSACTION_COMMAND, .flags = upiu_flags, .lun = lrbp->lun, - .task_tag = lrbp->task_tag, + .task_tag = tag, .command_set_type = UPIU_COMMAND_SET_TYPE_SCSI, }; -WARN_ON_ONCE (ucd_req_ptr->header.task_tag != lrbp->task_tag); +WARN_ON_ONCE (ucd_req_ptr->header.task_tag != tag); ucd_req_ptr->sc.exp_data_transfer_len =cpu_to_be32 (cmd->sdb.length);
@@ -2826,6 +2827,7 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba, {struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr; +const int tag =scsi_cmd_to_rq (cmd)->tag;struct ufs_query *query = &hba->dev_cmd.query;u16 len =be16_to_cpu (query->request.upiu_req.length);
@@ -2834,7 +2836,7 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba, .transaction_code = UPIU_TRANSACTION_QUERY_REQ, .flags = upiu_flags, .lun = lrbp->lun, - .task_tag = lrbp->task_tag, + .task_tag = tag, .query_function = query->request.query_func,/* Data segment length only need for WRITE_DESC */ .data_segment_length =
@@ -2857,12 +2859,13 @@ static inline void ufshcd_prepare_utp_nop_upiu(struct scsi_cmnd *cmd) {struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr; +const int tag =scsi_cmd_to_rq (cmd)->tag;memset (ucd_req_ptr,0 ,sizeof (struct utp_upiu_req )); ucd_req_ptr->header = (struct utp_upiu_header ){ .transaction_code =UPIU_TRANSACTION_NOP_OUT , - .task_tag = lrbp->task_tag, + .task_tag = tag, }; }
@@ -2947,7 +2950,6 @@ static void __ufshcd_setup_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,memset (lrbp->ucd_req_ptr,0 ,sizeof (*lrbp->ucd_req_ptr)); - lrbp->task_tag = tag; lrbp->lun = lun;ufshcd_prepare_lrbp_crypto (cmd ?scsi_cmd_to_rq (cmd) :NULL , lrbp); }
@@ -3243,6 +3245,8 @@ ufshcd_dev_cmd_completion(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)static int ufshcd_wait_for_dev_cmd (struct ufs_hba *hba,struct ufshcd_lrb *lrbp,int max_timeout) { +struct scsi_cmnd *cmd = (struct scsi_cmnd *)lrbp -1 ; +const int tag =scsi_cmd_to_rq (cmd)->tag;unsigned long time_left =msecs_to_jiffies (max_timeout);unsigned long flags;bool pending;
@@ -3259,18 +3263,18 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba, }else { err = -ETIMEDOUT ;dev_dbg (hba->dev,"%s: dev_cmd request timedout, tag %d\n" , - __func__, lrbp->task_tag); + __func__, tag);/* MCQ mode */ if (hba->mcq_enabled) {/* successfully cleared the command, retry if needed */ -if (ufshcd_clear_cmd (hba, lrbp->task_tag) ==0 ) +if (ufshcd_clear_cmd (hba, tag) ==0 ) err = -EAGAIN ;return err; }/* SDB mode */ -if (ufshcd_clear_cmd (hba, lrbp->task_tag) ==0 ) { +if (ufshcd_clear_cmd (hba, tag) ==0 ) {/* successfully cleared the command, retry if needed */ err = -EAGAIN ;/*
@@ -3279,11 +3283,9 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,* variable. */ spin_lock_irqsave (&hba->outstanding_lock, flags); - pending =test_bit (lrbp->task_tag, - &hba->outstanding_reqs); + pending =test_bit (tag, &hba->outstanding_reqs);if (pending) -__clear_bit (lrbp->task_tag, - &hba->outstanding_reqs); +__clear_bit (tag, &hba->outstanding_reqs);spin_unlock_irqrestore (&hba->outstanding_lock, flags);if (!pending) {
@@ -3296,11 +3298,10 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba, } }else {dev_err (hba->dev,"%s: failed to clear tag %d\n" , - __func__, lrbp->task_tag); + __func__, tag);spin_lock_irqsave (&hba->outstanding_lock, flags); - pending =test_bit (lrbp->task_tag, - &hba->outstanding_reqs); + pending =test_bit (tag, &hba->outstanding_reqs);spin_unlock_irqrestore (&hba->outstanding_lock, flags);if (!pending) {
@@ -5456,6 +5457,7 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct cq_entry *cqe) {struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd); +const int tag =scsi_cmd_to_rq (cmd)->tag;int result =0 ;int scsi_status;enum utp_ocs ocs;
@@ -5527,10 +5529,8 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,case OCS_ABORTED :case OCS_INVALID_COMMAND_STATUS : result |=DID_REQUEUE <<16 ; -dev_warn (hba->dev, -"OCS %s from controller for tag %d\n" , - (ocs ==OCS_ABORTED ?"aborted" :"invalid" ), - lrbp->task_tag); +dev_warn (hba->dev,"OCS %s from controller for tag %d\n" , + ocs ==OCS_ABORTED ?"aborted" :"invalid" , tag);break ;case OCS_INVALID_CMD_TABLE_ATTR :case OCS_INVALID_PRDT_ATTR :
@@ -5543,9 +5543,8 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,case OCS_GENERAL_CRYPTO_ERROR :default : result |=DID_ERROR <<16 ; -dev_err (hba->dev, -"OCS error from controller = %x for tag %d\n" , - ocs, lrbp->task_tag); +dev_err (hba->dev,"OCS error from controller = %x for tag %d\n" , + ocs, tag);ufshcd_print_evt_hist (hba);ufshcd_print_host_state (hba);break ;
@@ -7659,8 +7658,8 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)u8 resp =0xF ;for (poll_cnt =100 ; poll_cnt; poll_cnt--) { - err =ufshcd_issue_tm_cmd (hba, lrbp->lun, lrbp->task_tag, -UFS_QUERY_TASK , &resp); + err =ufshcd_issue_tm_cmd (hba, lrbp->lun, tag,UFS_QUERY_TASK , + &resp);if (!err && resp ==UPIU_TASK_MANAGEMENT_FUNC_SUCCEEDED ) {/* cmd pending in the device */ dev_err(hba->dev, "%s: cmd pending in the device. tag = %d\n",
@@ -7693,8 +7692,7 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)if (!poll_cnt)return -EBUSY ; - err =ufshcd_issue_tm_cmd (hba, lrbp->lun, lrbp->task_tag, -UFS_ABORT_TASK , &resp); + err =ufshcd_issue_tm_cmd (hba, lrbp->lun, tag,UFS_ABORT_TASK , &resp);if (err || resp !=UPIU_TASK_MANAGEMENT_FUNC_COMPL ) {if (!err) { err = resp;/* service response error */
@@ -7804,7 +7802,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd) goto release; } - err =ufshcd_try_to_abort_task (hba, lrbp->task_tag); + err =ufshcd_try_to_abort_task (hba, tag);if (err) {dev_err (hba->dev,"%s: failed with err %d\n" , __func__, err);ufshcd_set_req_abort_skip (hba, hba->outstanding_reqs);
include/ufs/ufshcd.h +0 -1
@@ -188,7 +188,6 @@ struct ufshcd_lrb {int scsi_status;int command_type; -int task_tag;u8 lun;/* UPIU LUN id field is only 8-bit wide */ bool intr_cmd;bool req_abort_skip;
Bart Van Assche
reviewerPATCH v3 25/26 ufs: core: Make blk_mq_tagset_busy_iter() skip reserved requests
In reply to Bart Van AsscheA later patch will convert hba->reserved_slot into a reserved tag. Make blk_mq_tagset_busy_iter() skip reserved requests such that device management commands are skipped.
--- drivers/ufs/core/ufshcd.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +8 -4
@@ -646,7 +646,8 @@ static bool ufshcd_print_tr_iter(struct request *req, void *priv)struct Scsi_Host *shost = sdev->host;struct ufs_hba *hba =shost_priv (shost); -ufshcd_print_tr (hba,blk_mq_rq_to_pdu (req), *(bool *)priv); +if (!blk_mq_is_reserved_rq (req)) +ufshcd_print_tr (hba,blk_mq_rq_to_pdu (req), *(bool *)priv);return true; }
@@ -5741,7 +5742,7 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)struct ufs_hba *hba =shost_priv (shost);struct ufs_hw_queue *hwq =ufshcd_mcq_req_to_hwq (hba, rq); -if (!hwq) +if (blk_mq_is_reserved_rq (rq) || !hwq)return true;ufshcd_mcq_compl_all_cqes_lock (hba, hwq);
@@ -5768,7 +5769,7 @@ static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)struct ufs_hba *hba =shost_priv (shost);struct ufs_hw_queue *hwq =ufshcd_mcq_req_to_hwq (hba, rq); -if (hwq) +if (!blk_mq_is_reserved_rq (rq) && hwq)ufshcd_mcq_poll_cqe_lock (hba, hwq);return true;
@@ -6629,6 +6630,9 @@ static bool ufshcd_abort_one(struct request *rq, void *priv)struct Scsi_Host *shost = sdev->host;struct ufs_hba *hba =shost_priv (shost); +if (blk_mq_is_reserved_rq (rq)) +return true; + *ret =ufshcd_try_to_abort_task (hba, tag); dev_err(hba->dev, "Aborting tag %d / CDB %#02x %s\n", tag, ufshcd_is_scsi_cmd(cmd) ? cmd->cmnd[0] : -1,
@@ -7564,7 +7568,7 @@ static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)const u64 lun = *(u64 *)priv;const u32 tag = req->tag; -if (sdev->lun != lun) +if (blk_mq_is_reserved_rq (req) || sdev->lun != lun)return true;if (ufshcd_clear_cmd (hba, tag) <0 ) {
Bart Van Assche
reviewerPATCH v3 26/26 ufs: core: Switch to scsi_get_internal_cmd()
In reply to Bart Van AsscheInstead of storing the tag of the reserved command in hba->reserved_slot, use scsi_get_internal_cmd() and scsi_put_internal_cmd() to allocate the tag for the reserved command dynamically. Add support in ufshcd_queuecommand() for submitting reserved commands. Add support in ufshcd_abort() for device management commands. Use blk_execute_rq() for submitting reserved commands. Remove the code and data structures that became superfluous. This includes ufshcd_wait_for_dev_cmd(), hba->reserved_slot and ufs_dev_cmd.complete.
--- drivers/ufs/core/ufs-mcq.c | 19 +-- drivers/ufs/core/ufshcd-priv.h | 25 +--- drivers/ufs/core/ufshcd.c | 231 +++++++++++++++++---------------- include/ufs/ufshcd.h | 6 - 4 files changed, 125 insertions(+), 156 deletions(-)
drivers/ufs/core/ufs-mcq.c +7 -12
@@ -471,9 +471,6 @@ int ufshcd_mcq_init(struct ufs_hba *hba)mutex_init (&hwq->sq_mutex); } -/* The very first HW queue serves device commands */ - hba->dev_cmd_queue = &hba->uhq[0 ]; - host->host_tagset =1 ;return 0 ; }
@@ -528,6 +525,7 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag) {struct scsi_cmnd *cmd =ufshcd_tag_to_cmd (hba, task_tag);struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd); +struct request *rq =scsi_cmd_to_rq (cmd);struct ufs_hw_queue *hwq;void __iomem *reg, *opr_sqd_base;u32 nexus, id, val;
@@ -536,15 +534,12 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)if (hba->quirks &UFSHCD_QUIRK_MCQ_BROKEN_RTC )return -ETIMEDOUT ; -if (task_tag != hba->reserved_slot) { -if (!cmd) -return -EINVAL ; - hwq =ufshcd_mcq_req_to_hwq (hba,scsi_cmd_to_rq (cmd)); -if (!hwq) -return 0 ; - }else { - hwq = hba->dev_cmd_queue; - } +if (!cmd) +return -EINVAL ; + + hwq =ufshcd_mcq_req_to_hwq (hba, rq); +if (!hwq) +return 0 ; id = hwq->id;
drivers/ufs/core/ufshcd-priv.h +1 -24
@@ -367,30 +367,7 @@ static inline bool ufs_is_valid_unit_desc_lun(struct ufs_dev_info *dev_info, u8static inline struct scsi_cmnd *ufshcd_tag_to_cmd (struct ufs_hba *hba,u32 tag) {struct blk_mq_tags *tags = hba->host->tag_set.shared_tags; -struct request *rq; - -/* -* Handle reserved tags differently because the UFS driver does not -* call blk_mq_alloc_request() for allocating reserved requests. -* Allocating reserved tags with blk_mq_alloc_request() would require -* the following: -* - Allocate an additional request queue from &hba->host->tag_set for -* allocating reserved requests from. -* - For that request queue, allocate a SCSI device. -* - Calling blk_mq_alloc_request(hba->dev_mgmt_queue, REQ_OP_DRV_OUT, -* BLK_MQ_REQ_RESERVED) for allocating a reserved request and -* blk_mq_free_request() for freeing reserved requests. -* - Set the .device pointer for these reserved requests. -* - Submit reserved requests with blk_execute_rq(). -* - Modify ufshcd_queuecommand() such that it handles reserved requests -* in another way than SCSI requests. -* - Modify ufshcd_compl_one_cqe() such that it calls scsi_done() for -* device management commands. -* - Modify all callback functions called by blk_mq_tagset_busy_iter() -* calls in the UFS driver and skip device management commands. -*/ - rq = tag <UFSHCD_NUM_RESERVED ? tags->static_rqs[tag] : -blk_mq_tag_to_rq (tags, tag); +struct request *rq =blk_mq_tag_to_rq (tags, tag);if (WARN_ON_ONCE (!rq))return NULL ;
drivers/ufs/core/ufshcd.c +117 -114
385 lines not on this page · raw message
include/ufs/ufshcd.h +0 -6
@@ -236,13 +236,11 @@ struct ufs_query {* struct ufs_dev_cmd - all assosiated fields with device management commands * @type: device management command type - Query, NOP OUT * @lock: lock to allow one command at a time -* @complete: internal commands completion * @query: Device management query information */ struct ufs_dev_cmd {enum dev_cmd_type type;struct mutex lock; -struct completion complete;struct ufs_query query; };
@@ -862,7 +860,6 @@ enum ufshcd_mcq_opr {* @nutrs: Transfer Request Queue depth supported by controller * @nortt - Max outstanding RTTs supported by controller * @nutmrs: Task Management Queue depth supported by controller -* @reserved_slot: Used to submit device commands. Protected by @dev_cmd.lock. * @ufs_version: UFS Version to which controller complies * @vops: pointer to variant specific operations * @vps: pointer to variant specific parameters
@@ -953,7 +950,6 @@ enum ufshcd_mcq_opr {* @res: array of resource info of MCQ registers * @mcq_base: Multi circular queue registers base address * @uhq: array of supported hardware queues -* @dev_cmd_queue: Queue for issuing device management commands * @mcq_opr: MCQ operation and runtime registers * @ufs_rtc_update_work: A work for UFS RTC periodic update * @pm_qos_req: PM QoS request handle
@@ -1004,7 +1000,6 @@ struct ufs_hba {int nortt;u32 mcq_capabilities;int nutmrs; -u32 reserved_slot;u32 ufs_version;const struct ufs_hba_variant_ops *vops;struct ufs_hba_variant_params *vps;
@@ -1123,7 +1118,6 @@ struct ufs_hba {struct ufshcd_res_info res[RES_MAX ];void __iomem *mcq_base;struct ufs_hw_queue *uhq; -struct ufs_hw_queue *dev_cmd_queue;struct ufshcd_mcq_opr_info_t mcq_opr[OPR_MAX ];struct delayed_work ufs_rtc_update_work;