scsi: core: Restart the queue when unblocking a device set running
Status from this message: Applied to 7.5/scsi-queue, thanks!
Get series with b4
b4 am [email protected]Versions newest first
Changes in v2 from the cover letter
- v1 restarted the queue in store_state_field(). Sashiko pointed out that this unquiesces a device its transport still has blocked, which transports such as SRP and drivers such as aacraid rely on while they reconnect or reset the adapter. Drop that.
- Instead, let scsi_internal_device_unblock_nowait() finish the unblock of a device that is SDEV_RUNNING with its queue still stopped. The transport now decides when the queue restarts, and the device moves to the state the transport asked for. This also stops scsi_host_unblock() from bailing out at such a device.
- The cost is that the writer of "running" stays stuck holding the device lock until the transport unblocks, instead of for good.
v1: https://lore.kernel.org/r/[email protected]
Patches in v2 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/1 | scsi: core: Restart the queue when unblocking a device set runningdrivers/ | +9 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.
Cover letter [PATCH v2 0/1] scsi: core: Restart the queue when unblocking a device set running
A SCSI device can be left in SDEV_RUNNING with its request queue stopped for good. The task that put it there hangs holding the device lock, and anything else that takes the lock hangs behind it.
This was seen with iSCSI when a session drops again just after recovering. The error handler offlines a device while its transport has it blocked, which leaves the queue stopped. iscsid then writes "running" to the device's state attribute and store_state_field() moves it to SDEV_RUNNING without restarting the queue. The rescan it starts sends an INQUIRY that can never be dispatched. When the transport later unblocks the device, scsi_internal_device_unblock_nowait() rejects it with -EINVAL because it is already running, so nothing restarts the queue.
Changes in v2:
- v1 restarted the queue in store_state_field(). Sashiko pointed out that this unquiesces a device its transport still has blocked, which transports such as SRP and drivers such as aacraid rely on while they reconnect or reset the adapter. Drop that.
- Instead, let scsi_internal_device_unblock_nowait() finish the unblock of a device that is SDEV_RUNNING with its queue still stopped. The transport now decides when the queue restarts, and the device moves to the state the transport asked for. This also stops scsi_host_unblock() from bailing out at such a device.
- The cost is that the writer of "running" stays stuck holding the device lock until the transport unblocks, instead of for good.
v1: https://lore.kernel.org/r/[email protected]
Evidence from a vmcore of a hung initiator (a RHEL 9.6 5.14 kernel; the code involved is the same in mainline). The device is sd 0:0:0:1, and iscsid is the task that wrote "running" to it:
PID: 12087 TASK: ffff9635c9c80000 COMMAND: "iscsid"
#5 blk_execute_rq
#6 scsi_execute_cmd
#7 scsi_vpd_inquiry
#10 scsi_attach_vpd
#11 scsi_rescan_device_rh
#12 store_state_field
#13 kernfs_fop_write_iter
sdev_state = SDEV_RUNNING
queue_stopped = 1
request_queue->quiesce_depth = 1
sdev_gendev.mutex.owner = 0xffff9635c9c80001 (iscsid, plus a flag bit)
The INQUIRY iscsid is waiting on belongs to that device's queue and was never dispatched:
state = MQ_RQ_IDLE, start_time_ns = 0, deadline = 0
The other LUNs of the same session were unblocked by the transport and had their queues restarted:
lun 1: SDEV_RUNNING queue_stopped = 1
lun 2: SDEV_TRANSPORT_OFFLINE queue_stopped = 0
...
lun 9: SDEV_TRANSPORT_OFFLINE queue_stopped = 0
(scsi_rescan_device_rh is this distro kernel's name for scsi_rescan_device().)
The kernel log of the same crash shows the sequence for the stuck device (sd 0:0:0:1) and its siblings:
[50958.843822] connection9:0: detected conn error (1021) [50968.259428] connection9:0: detected conn error (1020) [50968.259456] sd 0:0:0:1: Device offlined - not ready after error recovery [50978.298947] session9: session recovery timed out after 10 secs [50978.298977] sd 0:0:0:2: rejecting I/O to offline device
The other LUNs are unblocked by the recovery timeout at 50978. The offlined device is not: by then it had already been written back to running.
Questions for review:
- The new case only runs for a device that is SDEV_RUNNING with queue_stopped set. As far as I can tell that only happens when something sets the device running while it is blocked, as the sysfs write does here. Is there any caller that relies on -EINVAL for that combination?
- scsi_internal_device_unblock_nowait() can be called without state_mutex by some drivers. Could a concurrent unblock see the transient SDEV_RUNNING-with-stopped-queue state between the state assignment and scsi_start_queue() and apply a different new_state?
I do not have a deterministic reproducer. The analysis comes from the vmcore, and initiators running the v1 change have not hit the hang again, but with a race this narrow that is not proof. Review of the state transitions is welcome.
Brian Bunker (1):
scsi: core: Restart the queue when unblocking a device set running
drivers/scsi/scsi_lib.c | 9 +++++++++ 1 file changed, 9 insertions(+)
Signature · 1 line
2.56.0