scsi: fixes for targets with many LUNs, and scsi_target_block rework
Subsystems (6)
v7 cover scsi: fixes for targets with many LUNs, and scsi_target_block rework
From: Martin Wilck <[email protected]>
This patch series addresses some issues we saw in a test setup with a large number of SCSI LUNs. The first two patches simply increase the number of available sg and bsg devices. 3-5 fix
a large delay we encountered between blocking a Fibre Channel remote port and the dev_loss_tmo. 6 renames scsi_target_block() to scsi_block_targets(), and makes additional changes to this API, as suggested in the review of the v2 series. 7 improves a warning message.
Changes v6 -> v7
- 6/7: fixed mistake I made inverting the argument order in v6 (I apologize for overlooking this dumb mistake!!)
Changes v5 -> v6
- 6/7: inverted order of arguments for scsi_block_targets (Christoph Hellwig), dropped "extern" (I took the liberty not to remove previous "Reviewed-by"'s because of this change)
- 5/7: wrapped one over-long comment line
- added tags
Changes v4 -> v5:
- added 7/7 to improve the WARN_ON_ONCE in scsi_device_block() (Bart van Assche)
- 6/7: added WARN_ON_ONCE in scsi_block_targets() (Bart van Assche)
- 4/7: improved comment (Bart van Assche)
- Added tags
Changes v3 -> v4:
- skipped 4/8: keep state_mutex held while quiescing queue (Bart van Assche), added a comment in 4/6 to explain the rationale
- renamed scsi_target_block() to scsi_block_targets() (Christoph Hellwig), and merged the previous patches 7/8 and 8/8 modifying this API into 6/6.
- rebased to latest mkp/queue branch
Changes v2 -> v3:
- Split previous 3/3 into 4 separate patches as suggested by Christoph Hellwig.
- Added 7/8 and 8/8, as suggested by Christoph and Bart van Assche.
- Added s-o-b and reviewed-by tags.
Changes v1 -> v2:
- call blk_mq_wait_quiesce_done() from scsi_target_block() to cover the case where BLK_MQ_F_BLOCKING is set (Bart van Assche)
Hannes Reinecke (2):
bsg: increase number of devices
scsi: sg: increase number of devices
Martin Wilck (5):
scsi: merge scsi_internal_device_block() and device_block()
scsi: don't wait for quiesce in scsi_stop_queue()
scsi: don't wait for quiesce in scsi_device_block()
scsi: replace scsi_target_block() by scsi_block_targets()
scsi: improve warning message in scsi_device_block()
block/bsg.c | 2 +- drivers/scsi/scsi_lib.c | 80 ++++++++++++++--------------- drivers/scsi/scsi_transport_fc.c | 2 +- drivers/scsi/scsi_transport_iscsi.c | 3 +- drivers/scsi/scsi_transport_srp.c | 6 +-- drivers/scsi/sg.c | 2 +- drivers/scsi/snic/snic_disc.c | 2 +- include/scsi/scsi_device.h | 2 +- 8 files changed, 50 insertions(+), 49 deletions(-)
Signature · 1 line
2.40.1
PATCH v7 1/7 bsg: increase number of devices
From: Hannes Reinecke <[email protected]>
Larger setups may need to allocate more than 32k bsg devices, so increase the number of devices to the full range of minor device numbers.
--- block/bsg.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/bsg.c +1 -1
@@ -36,7 +36,7 @@ static inline struct bsg_device *to_bsg_device(struct inode *inode) }#define BSG_DEFAULT_CMDS 64 -#define BSG_MAX_DEVS 32768 +#define BSG_MAX_DEVS (1 << MINORBITS)static DEFINE_IDA (bsg_minor_ida);static struct class *bsg_class;
Signature · 1 line
2.40.1
PATCH v7 2/7 scsi: sg: increase number of devices
In reply to [email protected]From: Hannes Reinecke <[email protected]>
Larger setups may need to allocate more than 32k sg devices, so increase the number of devices to the full range of minor device numbers.
--- drivers/scsi/sg.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/scsi/sg.c +1 -1
@@ -71,7 +71,7 @@ static int sg_proc_init(void);#define SG_ALLOW_DIO_DEF 0 -#define SG_MAX_DEVS 32768 +#define SG_MAX_DEVS (1 << MINORBITS)/* SG_MAX_CDB_SIZE should be 260 (spc4r37 section 3.1.30) however the type * of sg_io_hdr::cmd_len can only represent 255. All SCSI commands greater
Signature · 1 line
2.40.1
PATCH v7 3/7 scsi: merge scsi_internal_device_block() and device_block()
In reply to [email protected]From: Martin Wilck <[email protected]>
scsi_internal_device_block() is only called from device_block(). Merge the two functions, and call the result scsi_device_block(), as the name device_block() is confusingly generic.
--- drivers/scsi/scsi_lib.c | 27 ++++++++------------------- 1 file changed, 8 insertions(+), 19 deletions(-)
drivers/scsi/scsi_lib.c +8 -19
@@ -2781,13 +2781,12 @@ int scsi_internal_device_block_nowait(struct scsi_device *sdev)EXPORT_SYMBOL_GPL (scsi_internal_device_block_nowait);/** -* scsi_internal_device_block - try to transition to the SDEV_BLOCK state +* scsi_device_block - try to transition to the SDEV_BLOCK state * @sdev: device to block +* @data: dummy argument, ignored * * Pause SCSI command processing on the specified device and wait until all -* ongoing scsi_request_fn() / scsi_queue_rq() calls have finished. May sleep. -* -* Returns zero if successful or a negative error code upon failure. +* ongoing scsi_queue_rq() calls have finished. May sleep. * * Note: * This routine transitions the device to the SDEV_BLOCK state (which must be
@@ -2795,7 +2794,7 @@ EXPORT_SYMBOL_GPL(scsi_internal_device_block_nowait);* is paused until the device leaves the SDEV_BLOCK state. See also * scsi_internal_device_unblock(). */ -static int scsi_internal_device_block (struct scsi_device *sdev) +static void scsi_device_block (struct scsi_device *sdev,void *data) {int err;
@@ -2805,7 +2804,8 @@ static int scsi_internal_device_block(struct scsi_device *sdev)scsi_stop_queue (sdev, false);mutex_unlock (&sdev->state_mutex); -return err; +WARN_ONCE (err,"__scsi_internal_device_block_nowait(%s) failed: err = %d\n" , +dev_name (&sdev->sdev_gendev), err); }/**
@@ -2888,23 +2888,12 @@ static int scsi_internal_device_unblock(struct scsi_device *sdev,return ret; } -static void -device_block (struct scsi_device *sdev,void *data) -{ -int ret; - - ret =scsi_internal_device_block (sdev); - -WARN_ONCE (ret,"scsi_internal_device_block(%s) failed: ret = %d\n" , -dev_name (&sdev->sdev_gendev), ret); -} -static int target_block (struct device *dev,void *data) {if (scsi_is_target_device (dev))starget_for_each_device (to_scsi_target (dev),NULL , - device_block); + scsi_device_block);return 0 ; }
@@ -2913,7 +2902,7 @@ scsi_target_block(struct device *dev) {if (scsi_is_target_device (dev))starget_for_each_device (to_scsi_target (dev),NULL , - device_block); + scsi_device_block);else device_for_each_child (dev,NULL , target_block); }
Signature · 1 line
2.40.1
PATCH v7 4/7 scsi: don't wait for quiesce in scsi_stop_queue()
In reply to [email protected]From: Martin Wilck <[email protected]>
scsi_stop_queue() has just two callers, one with and one without "nowait". As blk_mq_quiesce_queue() comes down to blk_mq_quiesce_queue_nowait() followed by blk_mq_wait_quiesce_done(), we might as well open-code this in scsi_device_block().
Also, add a comment explaining why blk_mq_quiesce_queue_nowait() must be called with the state_mutex held, see
https://lore.kernel.org/linux-scsi/[email protected]/.
--- drivers/scsi/scsi_lib.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
drivers/scsi/scsi_lib.c +16 -16
@@ -2731,24 +2731,16 @@ void scsi_start_queue(struct scsi_device *sdev)blk_mq_unquiesce_queue (sdev->request_queue); } -static void scsi_stop_queue (struct scsi_device *sdev,bool nowait) +static void scsi_stop_queue (struct scsi_device *sdev) {/* * The atomic variable of ->queue_stopped covers that * blk_mq_quiesce_queue* is balanced with blk_mq_unquiesce_queue. * -* However, we still need to wait until quiesce is done -* in case that queue has been stopped. +* The caller needs to wait until quiesce is done. */ -if (!cmpxchg (&sdev->queue_stopped,0 ,1 )) { -if (nowait) -blk_mq_quiesce_queue_nowait (sdev->request_queue); -else -blk_mq_quiesce_queue (sdev->request_queue); - }else { -if (!nowait) -blk_mq_wait_quiesce_done (sdev->request_queue->tag_set); - } +if (!cmpxchg (&sdev->queue_stopped,0 ,1 )) +blk_mq_quiesce_queue_nowait (sdev->request_queue); }/**
@@ -2775,7 +2767,7 @@ int scsi_internal_device_block_nowait(struct scsi_device *sdev)* request queue. */ if (!ret) -scsi_stop_queue (sdev, true); +scsi_stop_queue (sdev);return ret; }EXPORT_SYMBOL_GPL (scsi_internal_device_block_nowait);
@@ -2800,9 +2792,17 @@ static void scsi_device_block(struct scsi_device *sdev, void *data)mutex_lock (&sdev->state_mutex); err =__scsi_internal_device_block_nowait (sdev); -if (err ==0 ) -scsi_stop_queue (sdev, false); -mutex_unlock (&sdev->state_mutex); +if (err ==0 ) { +/* +* scsi_stop_queue() must be called with the state_mutex +* held. Otherwise a simultaneous scsi_start_queue() call +* might unquiesce the queue before we quiesce it. +*/ +scsi_stop_queue (sdev); +mutex_unlock (&sdev->state_mutex); +blk_mq_wait_quiesce_done (sdev->request_queue->tag_set); + }else +mutex_unlock (&sdev->state_mutex);WARN_ONCE (err,"__scsi_internal_device_block_nowait(%s) failed: err = %d\n" ,dev_name (&sdev->sdev_gendev), err);
Signature · 1 line
2.40.1
PATCH v7 5/7 scsi: don't wait for quiesce in scsi_device_block()
In reply to [email protected]From: Martin Wilck <[email protected]>
scsi_device_block() is only called from scsi_target_block(), which calls it repeatedly for every child device. For targets with many devices, waiting for every queue to quiesce may cause a substantial delay (we measured more than 100s delay for blocking a FC rport with 2048 LUNs).
Just call blk_mq_wait_quiesce_done() once from scsi_target_block() after stopping all queues.
--- drivers/scsi/scsi_lib.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
drivers/scsi/scsi_lib.c +10 -7
@@ -2777,8 +2777,9 @@ EXPORT_SYMBOL_GPL(scsi_internal_device_block_nowait);* @sdev: device to block * @data: dummy argument, ignored * -* Pause SCSI command processing on the specified device and wait until all -* ongoing scsi_queue_rq() calls have finished. May sleep. +* Pause SCSI command processing on the specified device. Callers must wait +* until all ongoing scsi_queue_rq() calls have finished after this function +* returns. * * Note: * This routine transitions the device to the SDEV_BLOCK state (which must be
@@ -2792,17 +2793,15 @@ static void scsi_device_block(struct scsi_device *sdev, void *data)mutex_lock (&sdev->state_mutex); err =__scsi_internal_device_block_nowait (sdev); -if (err ==0 ) { +if (err ==0 )/* * scsi_stop_queue() must be called with the state_mutex * held. Otherwise a simultaneous scsi_start_queue() call * might unquiesce the queue before we quiesce it. */ scsi_stop_queue (sdev); -mutex_unlock (&sdev->state_mutex); -blk_mq_wait_quiesce_done (sdev->request_queue->tag_set); - }else -mutex_unlock (&sdev->state_mutex); + +mutex_unlock (&sdev->state_mutex);WARN_ONCE (err,"__scsi_internal_device_block_nowait(%s) failed: err = %d\n" ,dev_name (&sdev->sdev_gendev), err);
@@ -2900,11 +2899,15 @@ target_block(struct device *dev, void *data)void scsi_target_block (struct device *dev) { +struct Scsi_Host *shost =dev_to_shost (dev); +if (scsi_is_target_device (dev))starget_for_each_device (to_scsi_target (dev),NULL , scsi_device_block);else device_for_each_child (dev,NULL , target_block); + +blk_mq_wait_quiesce_done (&shost->tag_set); }EXPORT_SYMBOL_GPL (scsi_target_block);
Signature · 1 line
2.40.1
PATCH v7 6/7 scsi: replace scsi_target_block() by scsi_block_targets()
In reply to [email protected]From: Martin Wilck <[email protected]>
All callers (fc_remote_port_delete(), __iscsi_block_session(), __srp_start_tl_fail_timers(), srp_reconnect_rport(), snic_tgt_del()) pass parent devices of scsi_target devices to scsi_target_block().
Rename the function to scsi_block_targets(), and simplify it by assuming that it is always passed a parent device. Also, have callers pass the Scsi_Host pointer to scsi_block_targets(), as every caller has this pointer readily available.
--- drivers/scsi/scsi_lib.c | 26 ++++++++++++++++---------- drivers/scsi/scsi_transport_fc.c | 2 +- drivers/scsi/scsi_transport_iscsi.c | 3 ++- drivers/scsi/scsi_transport_srp.c | 6 +++--- drivers/scsi/snic/snic_disc.c | 2 +- include/scsi/scsi_device.h | 2 +- 6 files changed, 24 insertions(+), 17 deletions(-)
drivers/scsi/scsi_lib.c +16 -10
@@ -2896,20 +2896,26 @@ target_block(struct device *dev, void *data)return 0 ; } +/** +* scsi_block_targets - transition all SCSI child devices to SDEV_BLOCK state +* @dev: a parent device of one or more scsi_target devices +* @shost: the Scsi_Host to which this device belongs +* +* Iterate over all children of @dev, which should be scsi_target devices, +* and switch all subordinate scsi devices to SDEV_BLOCK state. Wait for +* ongoing scsi_queue_rq() calls to finish. May sleep. +* +* Note: +* @dev must not itself be a scsi_target device. +*/ void -scsi_target_block (struct device *dev) +scsi_block_targets (struct Scsi_Host *shost,struct device *dev) { -struct Scsi_Host *shost =dev_to_shost (dev); - -if (scsi_is_target_device (dev)) -starget_for_each_device (to_scsi_target (dev),NULL , - scsi_device_block); -else -device_for_each_child (dev,NULL , target_block); - +WARN_ON_ONCE (scsi_is_target_device (dev)); +device_for_each_child (dev,NULL , target_block);blk_mq_wait_quiesce_done (&shost->tag_set); } -EXPORT_SYMBOL_GPL (scsi_target_block); +EXPORT_SYMBOL_GPL (scsi_block_targets);static void device_unblock (struct scsi_device *sdev,void *data)
drivers/scsi/scsi_transport_fc.c +1 -1
@@ -3451,7 +3451,7 @@ fc_remote_port_delete(struct fc_rport *rport)spin_unlock_irqrestore (shost->host_lock, flags); -scsi_target_block (&rport->dev); +scsi_block_targets (shost, &rport->dev);/* see if we need to kill io faster than waiting for device loss */ if ((rport->fast_io_fail_tmo != -1) &&
drivers/scsi/scsi_transport_iscsi.c +2 -1
@@ -1943,13 +1943,14 @@ static void __iscsi_block_session(struct work_struct *work)struct iscsi_cls_session *session = container_of(work, struct iscsi_cls_session, block_work); +struct Scsi_Host *shost =iscsi_session_to_shost (session);unsigned long flags;ISCSI_DBG_TRANS_SESSION (session,"Blocking session\n" );spin_lock_irqsave (&session->lock, flags); session->state =ISCSI_SESSION_FAILED ;spin_unlock_irqrestore (&session->lock, flags); -scsi_target_block (&session->dev); +scsi_block_targets (shost, &session->dev);ISCSI_DBG_TRANS_SESSION (session,"Completed SCSI target blocking\n" );if (session->recovery_tmo >=0 ) queue_delayed_work(session->workq,
drivers/scsi/scsi_transport_srp.c +3 -3
@@ -396,7 +396,7 @@ static void srp_reconnect_work(struct work_struct *work) }/* -* scsi_target_block() must have been called before this function is +* scsi_block_targets() must have been called before this function is * called to guarantee that no .queuecommand() calls are in progress. */ static void __rport_fail_io_fast (struct srp_rport *rport)
@@ -480,7 +480,7 @@ static void __srp_start_tl_fail_timers(struct srp_rport *rport) srp_rport_set_state(rport, SRP_RPORT_BLOCKED) == 0) {pr_debug ("%s new state: %d\n" ,dev_name (&shost->shost_gendev), rport->state); -scsi_target_block (&shost->shost_gendev); +scsi_block_targets (shost, &shost->shost_gendev);if (fast_io_fail_tmo >=0 ) queue_delayed_work(system_long_wq, &rport->fast_io_fail_work,
@@ -548,7 +548,7 @@ int srp_reconnect_rport(struct srp_rport *rport)* later is ok though, scsi_internal_device_unblock_nowait() * treats SDEV_TRANSPORT_OFFLINE like SDEV_BLOCK. */ -scsi_target_block (&shost->shost_gendev); +scsi_block_targets (shost, &shost->shost_gendev); res = rport->state !=SRP_RPORT_LOST ? i->f->reconnect (rport) : -ENODEV ;pr_debug ("%s (state %d): transport.reconnect() returned %d\n" ,dev_name (&shost->shost_gendev), rport->state, res);
drivers/scsi/snic/snic_disc.c +1 -1
@@ -214,7 +214,7 @@ snic_tgt_del(struct work_struct *work)scsi_flush_work (shost);/* Block IOs on child devices, stops new IOs */ -scsi_target_block (&tgt->dev); +scsi_block_targets (shost, &tgt->dev);/* Cleanup IOs */ snic_tgt_scsi_abort_io (tgt);
include/scsi/scsi_device.h +1 -1
@@ -456,7 +456,7 @@ extern void scsi_scan_target(struct device *parent, unsigned int channel, unsigned int id, u64 lun, enum scsi_scan_mode rescan);extern void scsi_target_reap (struct scsi_target *); -extern void scsi_target_block (struct device *); +void scsi_block_targets (struct Scsi_Host *shost,struct device *dev);extern void scsi_target_unblock (struct device *,enum scsi_device_state );extern void scsi_remove_target (struct device *);extern const char *scsi_device_state_name (enum scsi_device_state );
Signature · 1 line
2.40.1
PATCH v7 7/7 scsi: improve warning message in scsi_device_block()
In reply to [email protected]From: Martin Wilck <[email protected]>
If __scsi_internal_device_block() returns an error, it is always -EINVAL because of an invalid state transition. For debugging purposes, it makes more sense to print the device state.
--- drivers/scsi/scsi_lib.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +4 -2
@@ -2790,9 +2790,11 @@ EXPORT_SYMBOL_GPL(scsi_internal_device_block_nowait);static void scsi_device_block (struct scsi_device *sdev,void *data) {int err; +enum scsi_device_state state;mutex_lock (&sdev->state_mutex); err =__scsi_internal_device_block_nowait (sdev); + state = sdev->sdev_state;if (err ==0 )/* * scsi_stop_queue() must be called with the state_mutex
@@ -2803,8 +2805,8 @@ static void scsi_device_block(struct scsi_device *sdev, void *data)mutex_unlock (&sdev->state_mutex); -WARN_ONCE (err,"__scsi_internal_device_block_nowait(%s) failed: err = %d\n" , -dev_name (&sdev->sdev_gendev), err); +WARN_ONCE (err,"%s: failed to block %s in state %d\n" , + __func__,dev_name (&sdev->sdev_gendev), state); }/**
Signature · 1 line
2.40.1
Martin K. Petersen
AppliedMartin,
Quoted 7 lines
This patch series addresses some issues we saw in a test setup with a large number of SCSI LUNs. The first two patches simply increase the number of available sg and bsg devices. 3-5 fix a large delay we encountered between blocking a Fibre Channel remote port and the
dev_loss_tmo. 6 renamesscsi_target_block()toscsi_block_targets(), and makes additional changes to this API, as suggested in the review of the v2 series. 7 improves a warning message.
Applied to 6.5/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin K. Petersen
AppliedQuoted 9 lines · On Wed, 14 Jun 2023 12:36:09 +0200, [email protected] wrote:
This patch series addresses some issues we saw in a test setup with a large number of SCSI LUNs. The first two patches simply increase the number of available sg and bsg devices. 3-5 fix
a large delay we encountered between blocking a Fibre Channel remote port and thedev_loss_tmo. 6 renamesscsi_target_block()toscsi_block_targets(), and makes additional changes to this API, as suggested in the review of the v2 series. 7 improves a warning message.[...]
Applied to 6.5/scsi-queue, thanks!
[1/7] bsg: increase number of devices
https://git.kernel.org/mkp/scsi/c/9077fb2ab78c [2/7] scsi: sg: increase number of devices https://git.kernel.org/mkp/scsi/c/37c918e03ef7 [3/7] scsi: merge scsi_internal_device_block() and device_block() https://git.kernel.org/mkp/scsi/c/c5e46f7ad43b [4/7] scsi: don't wait for quiesce in scsi_stop_queue() https://git.kernel.org/mkp/scsi/c/d7035b73a73a [5/7] scsi: don't wait for quiesce in scsi_device_block() https://git.kernel.org/mkp/scsi/c/e20fff8a1f49 [6/7] scsi: replace scsi_target_block() by scsi_block_targets() https://git.kernel.org/mkp/scsi/c/31950192d939 [7/7] scsi: improve warning message in scsi_device_block() https://git.kernel.org/mkp/scsi/c/6d7160c7da6f
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering