Skip to contentSkip to conversation

scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs

PATCH In review Jeff Layton 2 participants 17 messages Nov 18

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs

I posted v2 some time ago. Since then, I did a lot of adversarial review with AI agents. The review found a large number of bugs. Many of these bugs were preexisting.

This series fixes a deadlock that occurs regularly in Meta's fleet. The fix unblocks sdevs after a timeout, and this breaks the deadlock.

I don't do much SCSI driver work, but this problem came to me. I relied heavily on AI agents for this series.

If this series is acceptable, I would like to see it in linux-next soon.

Thanks,
Jeff

---
Changes in v3:
- New patches 1-8 and 10 fix pre-existing bugs in the code that this series changes:
- 1-2: fix UAFs and refcounting in the cleanup of the fw event queue. Do not leave requeued events pending at teardown.
- 3: requeue topology events only for busy devices. Do not requeue them for devices that the driver cannot add.
- 4-7: fix these TUR polling bugs:
- a pcie_device leak
- a lost unblock retry
- polling that continues after the scan is cancelled
- a retry count that is not per device
- 8: cache the missing delays that the IOC actually uses
- 10: do not use freed hostdata while the driver polls devices
- Patch 9: also publish sdev->hostdata under ioc->block_lock in scsih_sdev_init()
- Patch 11:
- Use the PCIe missing delay (from PCIe IO Unit Page 1) for NVMe devices.
- Arm dnr_unblock_work for the earliest deadline.
- Skip the timed unblock during host reset, but not during SCSI EH.
- Clear the deadline of an sdev under ioc->block_lock before the fw event worker polls the sdev.
- Do not put a timed block on offline sdevs.
- When a hard reset cancels polling, give new deadlines to the sdevs that stay blocked.
- Give deadlines to the sdevs that a broadcast primitive pass blocks. Prevent dnr_unblock_work from unblocking them while the pass runs.
- A repeat DNR restarts the deadline, at most MPT3SAS_DNR_BLOCK_MAX_EXTENDS times.
- At the next worker unblock, do a TUR check on expired sdevs.
- Keep the blocks that the driver makes during polling.
- Use system_dfl_wq instead of system_wq.
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- New patch 1: use a new ioc->block_lock to serialize the
interrupt-time sdev block with the hostdata teardown in scsih_sdev_destroy(). This fixes a pre-existing UAF.
- Do the timed unblock under ioc->block_lock. This prevents a race with an interrupt-time block.
- Clear block_deadline before the fw event worker polls a returning device with TURs. Then dnr_unblock_work cannot unblock the device during the poll.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Jeff Layton (11):
scsi: mpt3sas: Don't leave requeued firmware events pending at teardown
scsi: mpt3sas: Hold event references while cleaning up the event queue
scsi: mpt3sas: Requeue topology events only for busy devices
scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()
scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()
scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled
scsi: mpt3sas: Reset the TUR retry count for each device
scsi: mpt3sas: Cache the missing delays the IOC actually uses
scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown
scsi: mpt3sas: Don't use freed hostdata while polling devices
scsi: mpt3sas: Time out delayed not-responding device blocks

 drivers/scsi/mpt3sas/mpt3sas_base.c  |  62 ++-
 drivers/scsi/mpt3sas/mpt3sas_base.h  |  25 +
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 869 +++++++++++++++++++++++++++++------
 3 files changed, 810 insertions(+), 146 deletions(-)

---
base-commit: e8017d29067fd1310a65c3494ce1a1b492adbac6
change-id: 20261030-sas-abba-eacfcc45fe36

Best regards,

Signature · 1 line
Jeff Layton <[email protected]>
#1 lore raw

PATCH v3 1/11 scsi: mpt3sas: Don't leave requeued firmware events pending at teardown

When a topology change event finds a busy device, _scsih_fw_event_requeue() retries the event. It queues the event's delayed_work on firmware_event_thread, with a one-second delay. _scsih_fw_event_cleanup_queue() cancels only the event's work, not its delayed_work. Thus the timer stays armed. Then scsih_remove() destroys the workqueue. After the workqueue is freed, the timer queues _firmware_event_work_delayed() on it:

  BUG: KASAN: slab-use-after-free in __queue_work+0xa12/0xb50
  Read of size 4 at addr ffff88814fefd1c0 by task swapper/10/0
   __queue_work
   call_timer_fn
   __run_timer_base
   ...
  Allocated by task 11:
   __alloc_workqueue
   alloc_workqueue_noprof
   _scsih_probe [mpt3sas]
  Freed by task 0:
   kfree
   rcu_core
  ...
  workqueue: cannot queue _firmware_event_work_delayed [mpt3sas] on wq fw_event_mpt3sas0

The cleanup does not wait for a running topology change event. If such an event finds a busy device, it requeues itself. This can occur after the cleanup passes the event.

Fix this:

- In the cleanup, cancel the delayed_work of a requeued event.

- Do not requeue an event while the cleanup runs or while the driver removes the host. Do this check under fw_event_lock, which the cleanup also takes.

- Instead of a requeue, drop the reference that the work held. Previously, this reference leaked when firmware_event_thread was NULL.

- Also drop the reference if the delayed_work cannot be queued. Cleanups can overlap. Thus one cleanup can clear fw_events_cleanup while another cleanup cancels the delayed_work. The cancel disables the delayed_work.

- Initialize the delayed_work when the driver allocates the event, not at the first requeue. Thus a cleanup never cancels the delayed_work before it is initialized.

Fixes ad5957193107 ("scsi: mpt3sas: Add firmware event requeue support for busy devices")
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 49 ++++++++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 11 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +38 -11
index 04255286a2a6..545913123d01 100644
@@ -320,6 +320,11 @@ static struct fw_event_work *alloc_fw_event_work(int len)
 		return NULL;
 
 	kref_init(&fw_event->refcount);
+	/*
+	 * Initialize the delayed_work up front: a cleanup may cancel it
+	 * while the event is being requeued.
+	 */
+	INIT_DELAYED_WORK(&fw_event->delayed_work, _firmware_event_work_delayed);
 	return fw_event;
 }
 
@@ -3696,6 +3701,9 @@ _scsih_fw_event_del_from_list(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
  *
  * Context: This function will acquire ioc->fw_event_lock.
  *
+ * The reference held by the work that ran @fw_event passes to the delayed
+ * work.  If the event can't be requeued, that reference is dropped.
+ *
  * Return nothing.
  */
 static void
@@ -3704,20 +3712,31 @@ _scsih_fw_event_requeue(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
 {
 	unsigned long flags;
 
-	if (ioc->firmware_event_thread == NULL)
-		return;
-
 	spin_lock_irqsave(&ioc->fw_event_lock, flags);
+	/*
+	 * Don't rearm the event while the queue is being flushed or the
+	 * host is going away.  _scsih_fw_event_cleanup_queue() may already
+	 * be past it, and the timer could then fire after the workqueue
+	 * has been destroyed.
+	 */
+	if (!ioc->firmware_event_thread || ioc->remove_host ||
+	    ioc->fw_events_cleanup)
+		goto drop;
+	fw_event->delayed_work_active = 1;
+	/*
+	 * Cleanups can overlap, and one can clear fw_events_cleanup while
+	 * another is cancelling, and so disabling, the delayed work.
+	 */
+	if (!queue_delayed_work(ioc->firmware_event_thread,
+	    &fw_event->delayed_work, msecs_to_jiffies(delay)))
+		goto drop;
 	fw_event_work_get(fw_event);
 	list_add_tail(&fw_event->list, &ioc->fw_event_list);
-	if (!fw_event->delayed_work_active) {
-		fw_event->delayed_work_active = 1;
-		INIT_DELAYED_WORK(&fw_event->delayed_work,
-		    _firmware_event_work_delayed);
-	}
-	queue_delayed_work(ioc->firmware_event_thread, &fw_event->delayed_work,
-	    msecs_to_jiffies(delay));
 	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+	return;
+drop:
+	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+	fw_event_work_put(fw_event);
 }
 
  /**
@@ -3869,9 +3888,17 @@ _scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc)
 		 *
 		 * If it did execute, we wait for it to finish, and the put will
 		 * happen from _firmware_event_work()
+		 *
+		 * An event requeued by _scsih_fw_event_requeue() is pending
+		 * on its delayed_work instead, and its timer must not fire
+		 * after the workqueue is destroyed.
 		 */
-		if (cancel_work_sync(&fw_event->work))
+		if (fw_event->delayed_work_active) {
+			if (cancel_delayed_work_sync(&fw_event->delayed_work))
+				fw_event_work_put(fw_event);
+		} else if (cancel_work_sync(&fw_event->work)) {
 			fw_event_work_put(fw_event);
+		}
 
 	}
 	ioc->fw_events_cleanup = 0;
Signature · 1 line
2.56.1
#2 lore raw

Re: [PATCH v3 01/11] scsi: mpt3sas: Don't leave requeued firmware events pending at teardown

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] scsi: mpt3sas: races in _scsih_fw_event_requeue() and cleanup queue --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] scsi: mpt3sas: races in _scsih_fw_event_requeue() and cleanup queue
--

commit 462c12ccec856fed92f29e6abf5abd84f2b713ca
Author: Jeff Layton <[email protected]>

scsi: mpt3sas: Don't leave requeued firmware events pending at teardown

This commit initializes fw_event->delayed_work at allocation, skips requeuing events when the queue is being cleaned up or the host is being removed, and cancels pending delayed_work items during queue cleanup.

Quoted 4 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +0 -0
index 04255286a2a69..545913123d01f 100644

[ ... ]

Quoted 51 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +25 -10
@@ -3704,20 +3712,31 @@ _scsih_fw_event_requeue(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
 {
 	unsigned long flags;
 
-	if (ioc->firmware_event_thread == NULL)
-		return;
-
 	spin_lock_irqsave(&ioc->fw_event_lock, flags);
+	/*
+	 * Don't rearm the event while the queue is being flushed or the
+	 * host is going away.  _scsih_fw_event_cleanup_queue() may already
+	 * be past it, and the timer could then fire after the workqueue
+	 * has been destroyed.
+	 */
+	if (!ioc->firmware_event_thread || ioc->remove_host ||
+	    ioc->fw_events_cleanup)
+		goto drop;
+	fw_event->delayed_work_active = 1;
+	/*
+	 * Cleanups can overlap, and one can clear fw_events_cleanup while
+	 * another is cancelling, and so disabling, the delayed work.
+	 */
+	if (!queue_delayed_work(ioc->firmware_event_thread,
+	    &fw_event->delayed_work, msecs_to_jiffies(delay)))
+		goto drop;
 	fw_event_work_get(fw_event);
 	list_add_tail(&fw_event->list, &ioc->fw_event_list);
-	if (!fw_event->delayed_work_active) {
-		fw_event->delayed_work_active = 1;
-		INIT_DELAYED_WORK(&fw_event->delayed_work,
-		    _firmware_event_work_delayed);
-	}
-	queue_delayed_work(ioc->firmware_event_thread, &fw_event->delayed_work,
-	    msecs_to_jiffies(delay));
 	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+	return;
+drop:
+	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+	fw_event_work_put(fw_event);
 }
 
  /**
@@ -3869,9 +3888,17 @@ _scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc)
 		 *
 		 * If it did execute, we wait for it to finish, and the put will
 		 * happen from _firmware_event_work()
+		 *
+		 * An event requeued by _scsih_fw_event_requeue() is pending
+		 * on its delayed_work instead, and its timer must not fire
+		 * after the workqueue is destroyed.
 		 */

drivers/scsi/mpt3sas/mpt3sas_scsih.c _scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc) · line 3904

Quoted 10 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +5 -1
-		if (cancel_work_sync(&fw_event->work))
+		if (fw_event->delayed_work_active) {
+			if (cancel_delayed_work_sync(&fw_event->delayed_work))
+				fw_event_work_put(fw_event);
+		} else if (cancel_work_sync(&fw_event->work)) {
 			fw_event_work_put(fw_event);
+		}
 
 	}
 	ioc->fw_events_cleanup = 0;

[Severity: High]
Can a delayed_work timer still fire after ioc->firmware_event_thread is destroyed when two cleanups overlap, or can a stale topology change event still requeue itself after a host reset cleanup?

_scsih_fw_event_cleanup_queue() does not serialize concurrent callers and does not wait for a running topology change event (it skips waiting when ioc->current_event->event != MPT3SAS_REMOVE_UNRESPONDING_DEVICES and clears ioc->fw_events_cleanup = 0 as soon as the loop finishes). If a reset cleanup overlaps with teardown in scsih_remove() or scsih_shutdown():

CPU0 (reset cleanup) CPU1 (scsih_shutdown / scsih_remove)
_scsih_fw_event_cleanup_queue()
fw_event = dequeue_next_fw_event()

                                  _scsih_fw_event_cleanup_queue()
                                    /* list_empty && !current_event */
                                    return;
                                  wq = ioc->firmware_event_thread;
                                  ioc->firmware_event_thread = NULL;
                                  destroy_workqueue(wq);
                                  /* delayed_work timer fires on wq */
                                    __queue_work()  <-- use-after-free

cancel_delayed_work_sync()

In addition, when a host reset cleanup runs while a topology change event is executing, _scsih_fw_event_cleanup_queue() sets ioc->current_event->ignore = 1, clears ioc->current_event = NULL, and resets ioc->fw_events_cleanup = 0 without cancelling or disabling fw_event->delayed_work. If _scsih_add_device() or _scsih_pcie_add_device() fails on the final entry of the event (or an earlier entry already set requeue_event = 1), _scsih_sas_topology_change_event() exits its loop and returns requeue_event = 1 without re-checking fw_event->ignore. When _mpt3sas_fw_work() then calls _scsih_fw_event_requeue(), ioc->fw_events_cleanup is already 0 and ioc->remove_host is 0, and _scsih_fw_event_requeue() checks neither fw_event->ignore nor ioc->shost_recovery, so the stale pre-reset event still requeues itself.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

PATCH v3 2/11 scsi: mpt3sas: Hold event references while cleaning up the event queue

In reply to Jeff Layton

_scsih_fw_event_cleanup_queue() uses events without a reference to them. But the work of an event frees the event when the work ends. This causes these problems:

- dequeue_next_fw_event() drops the list's reference before the cleanup cancels the event's work. If the work runs and ends between these two steps, the cleanup calls cancel_work_sync() on a freed event.

- The cleanup holds no reference to an event that it finds through ioc->current_event. The worker drops its reference before it clears current_event. Thus the cleanup can find a freed event there, and set ->ignore on it or read its ->event. For a running MPT3SAS_REMOVE_UNRESPONDING_DEVICES event, cancel_work_sync() waits for the work. The work frees the event, and then cancel_work_sync() enables the work again in freed memory.

- To identify a running event, the cleanup only compares the event with current_event. It waits for the work of all other events that it finds. But the cleanup also clears current_event. Also, cleanups can overlap, because _base_fault_reset_work() calls a cleanup without reset_in_progress_mutex. Thus a cleanup that holds reset_in_progress_mutex can wait for a running event. That event is about to take the same mutex for a hard reset.

Originally, commit 146b16c8071f ("mpt3sas: Refcount fw_events and fix unsafe list usage") kept the list's reference until after the cancel. Then commit 9e73ed2e4cf5 ("scsi: mpt3sas: Cancel the running work during host reset") added the current_event path. Events on this path had no reference to drop. To fix the resulting underflow, commit 991df3dd5144 ("scsi: mpt3sas: Fix use-after-free warning") dropped the list's reference early.

Fix the reference handling:

- Let dequeue_next_fw_event() pass the list's reference to the cleanup again. The cleanup drops the reference after the cancel.

- Set and clear current_event under fw_event_lock. Clear it before the worker drops its reference. Then the cleanup can take a reference to the current event under the lock.

- When the worker requeues the event, clear current_event in _scsih_fw_event_requeue(). Do this under fw_event_lock, together with the requeue or the drop of the event. _scsih_fw_event_requeue() can now drop the reference.

- Also do the check for events to clean up under the lock.

Thus the cleanup finds the event either as the current event or on fw_event_list. It cannot miss the event between the two.

When the worker starts an event, remove the event from fw_event_list and make it the current event. Do both in one step under fw_event_lock. If the event is no longer on the list, a cleanup removed it, so drop it. Thus an event that the cleanup removes from the list never runs. The cancel of its work then never waits for a running event.

Do not change current_event in the cleanup. Let the worker manage it. Wait for the running event only if it is MPT3SAS_REMOVE_UNRESPONDING_DEVICES, as before.

Only the broadcast primitive handler clears broadcast_aen_busy. If the cleanup drops a broadcast primitive event, broadcast_aen_busy stays set. Later broadcast primitives then only count as pending. An event that the cleanup cancels before it runs already causes this. With this change, an event that the worker drops as it starts also causes it. When the cleanup drops such an event, clear broadcast_aen_busy and the stale pending count.

Fixes 9e73ed2e4cf5 ("scsi: mpt3sas: Cancel the running work during host reset")
Fixes 991df3dd5144 ("scsi: mpt3sas: Fix use-after-free warning")
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 211 ++++++++++++++++++++++++-----------
 1 file changed, 148 insertions(+), 63 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +148 -63
index 545913123d01..b455b7e5d8f3 100644
@@ -3672,25 +3672,37 @@ _scsih_fw_event_add(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
 }
 
 /**
- * _scsih_fw_event_del_from_list - delete fw_event from the list
+ * _scsih_fw_event_start - start running an event
  * @ioc: per adapter object
  * @fw_event: object describing the event
  * Context: This function will acquire ioc->fw_event_lock.
  *
- * If the fw_event is on the fw_event_list, remove it and do a put.
+ * An event is on fw_event_list from when it is queued until it starts.
+ * Take @fw_event off the list, dropping the list's reference, and make it
+ * the current event, in one step under fw_event_lock.  If it is no longer
+ * on the list, _scsih_fw_event_cleanup_queue() took it off, and may be
+ * waiting for its work to end.
+ *
+ * Return: true if @fw_event should run, false if it should be dropped.
  */
-static void
-_scsih_fw_event_del_from_list(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
-	*fw_event)
+static bool
+_scsih_fw_event_start(struct MPT3SAS_ADAPTER *ioc,
+		      struct fw_event_work *fw_event)
 {
 	unsigned long flags;
+	bool start = false;
 
 	spin_lock_irqsave(&ioc->fw_event_lock, flags);
 	if (!list_empty(&fw_event->list)) {
 		list_del_init(&fw_event->list);
+		/* the work still holds a reference */
 		fw_event_work_put(fw_event);
+		ioc->current_event = fw_event;
+		start = true;
 	}
 	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+
+	return start;
 }
 
 /**
@@ -3704,6 +3716,12 @@ _scsih_fw_event_del_from_list(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
  * The reference held by the work that ran @fw_event passes to the delayed
  * work.  If the event can't be requeued, that reference is dropped.
  *
+ * The worker is done with @fw_event, so this also clears
+ * ioc->current_event.  That happens under fw_event_lock together with
+ * requeueing or dropping the event, so that
+ * _scsih_fw_event_cleanup_queue() finds the event either as the current
+ * one or on fw_event_list.
+ *
  * Return nothing.
  */
 static void
@@ -3713,6 +3731,8 @@ _scsih_fw_event_requeue(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
 	unsigned long flags;
 
 	spin_lock_irqsave(&ioc->fw_event_lock, flags);
+	if (ioc->current_event == fw_event)
+		ioc->current_event = NULL;
 	/*
 	 * Don't rearm the event while the queue is being flushed or the
 	 * host is going away.  _scsih_fw_event_cleanup_queue() may already
@@ -3800,6 +3820,10 @@ mpt3sas_port_enable_complete(struct MPT3SAS_ADAPTER *ioc)
 	fw_event_work_put(fw_event);
 }
 
+/*
+ * Remove the first event from fw_event_list and return it, with the
+ * reference the list held.  The caller puts it.
+ */
 static struct fw_event_work *dequeue_next_fw_event(struct MPT3SAS_ADAPTER *ioc)
 {
 	unsigned long flags;
@@ -3810,13 +3834,47 @@ static struct fw_event_work *dequeue_next_fw_event(struct MPT3SAS_ADAPTER *ioc)
 		fw_event = list_first_entry(&ioc->fw_event_list,
 				struct fw_event_work, list);
 		list_del_init(&fw_event->list);
-		fw_event_work_put(fw_event);
 	}
 	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
 
 	return fw_event;
 }
 
+/*
+ * ioc->current_event is set and cleared under fw_event_lock, and cleared
+ * before the worker drops its reference to the event.  While it is set
+ * under the lock, the event is live.
+ */
+static void
+_scsih_set_current_event(struct MPT3SAS_ADAPTER *ioc,
+			 struct fw_event_work *fw_event)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->fw_event_lock, flags);
+	ioc->current_event = fw_event;
+	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+}
+
+/*
+ * Return the event the worker is running, with a reference that the
+ * caller puts, or NULL if there is none.
+ */
+static struct fw_event_work *
+_scsih_get_current_event(struct MPT3SAS_ADAPTER *ioc)
+{
+	struct fw_event_work *fw_event;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ioc->fw_event_lock, flags);
+	fw_event = ioc->current_event;
+	if (fw_event)
+		fw_event_work_get(fw_event);
+	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+
+	return fw_event;
+}
+
 /**
  * _scsih_fw_event_cleanup_queue - cleanup event queue
  * @ioc: per adapter object
@@ -3830,10 +3888,13 @@ static void
 _scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc)
 {
 	struct fw_event_work *fw_event;
+	unsigned long flags;
+	bool idle;
 
-	if ((list_empty(&ioc->fw_event_list) && !ioc->current_event) ||
-	    !ioc->firmware_event_thread)
+	if (!ioc->firmware_event_thread)
 		return;
+	spin_lock_irqsave(&ioc->fw_event_lock, flags);
+	idle = list_empty(&ioc->fw_event_list) && !ioc->current_event;
 	/*
 	 * Set current running event as ignore, so that
 	 * current running event will exit quickly.
@@ -3842,64 +3903,87 @@ _scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc)
 	 */
 	if (ioc->shost_recovery && ioc->current_event)
 		ioc->current_event->ignore = 1;
+	spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+	if (idle)
+		return;
 
+	/*
+	 * Hold a reference to each event while it is handled here: its work
+	 * can drop the last one as soon as it ends.
+	 *
+	 * Don't wait for the running event, other than
+	 * MPT3SAS_REMOVE_UNRESPONDING_DEVICES, which stops waiting for the
+	 * host reset when fw_events_cleanup is set.  Another event may be
+	 * issuing the hard reset this cleanup is part of, and cleanups can
+	 * overlap, so whether an event is running doesn't depend on another
+	 * cleanup.  An event taken off fw_event_list here doesn't run, see
+	 * _scsih_fw_event_start(), so cancelling it never waits for one.
+	 *
+	 * Orginal logic of cleaning the current_event is added
+	 * for handling the back to back host reset issued by the user.
+	 * i.e. during back to back host reset, driver use to process
+	 * the two instances of MPT3SAS_REMOVE_UNRESPONDING_DEVICES
+	 * event back to back and this made the drives to unregister
+	 * the devices from SML.
+	 */
 	ioc->fw_events_cleanup = 1;
-	while ((fw_event = dequeue_next_fw_event(ioc)) ||
-	     (fw_event = ioc->current_event)) {
+	for (;;) {
+		while ((fw_event = dequeue_next_fw_event(ioc))) {
+			/*
+			 * Driver has to clear ioc->start_scan flag when
+			 * it is cleaning up MPT3SAS_PORT_ENABLE_COMPLETE,
+			 * otherwise scsi_scan_host() API waits for the
+			 * 5 minute timer to expire. If we exit from
+			 * scsi_scan_host() early then we can issue the
+			 * new port enable request as part of current diag
+			 * reset.
+			 */
+			if (fw_event->event == MPT3SAS_PORT_ENABLE_COMPLETE) {
+				ioc->port_enable_cmds.status |= MPT3_CMD_RESET;
+				ioc->start_scan = 0;
+			}
 
-		/*
-		 * Don't call cancel_work_sync() for current_event
-		 * other than MPT3SAS_REMOVE_UNRESPONDING_DEVICES;
-		 * otherwise we may observe deadlock if current
-		 * hard reset issued as part of processing the current_event.
-		 *
-		 * Orginal logic of cleaning the current_event is added
-		 * for handling the back to back host reset issued by the user.
-		 * i.e. during back to back host reset, driver use to process
-		 * the two instances of MPT3SAS_REMOVE_UNRESPONDING_DEVICES
-		 * event back to back and this made the drives to unregister
-		 * the devices from SML.
-		 */
+			/*
+			 * If this returns 1, the event never ran, and the
+			 * reference its work held needs a put.  Otherwise
+			 * its work drops that reference.
+			 *
+			 * An event requeued by _scsih_fw_event_requeue() is
+			 * pending on its delayed_work instead, and its timer
+			 * must not fire after the workqueue is destroyed.
+			 */
+			if (fw_event->delayed_work_active) {
+				if (cancel_delayed_work_sync(&fw_event->delayed_work))
+					fw_event_work_put(fw_event);
+			} else if (cancel_work_sync(&fw_event->work)) {
+				fw_event_work_put(fw_event);
+			}
 
-		if (fw_event == ioc->current_event &&
-		    ioc->current_event->event !=
-		    MPT3SAS_REMOVE_UNRESPONDING_DEVICES) {
-			ioc->current_event = NULL;
-			continue;
-		}
+			/*
+			 * The broadcast callback queues one broadcast
+			 * primitive event at a time, and only its handler,
+			 * which won't run now, clears broadcast_aen_busy.
+			 */
+			if (fw_event->event ==
+			    MPI2_EVENT_SAS_BROADCAST_PRIMITIVE) {
+				ioc->broadcast_aen_busy = 0;
+				ioc->broadcast_aen_pending = 0;
+			}
 
-		/*
-		 * Driver has to clear ioc->start_scan flag when
-		 * it is cleaning up MPT3SAS_PORT_ENABLE_COMPLETE,
-		 * otherwise scsi_scan_host() API waits for the
-		 * 5 minute timer to expire. If we exit from
-		 * scsi_scan_host() early then we can issue the
-		 * new port enable request as part of current diag reset.
-		 */
-		if (fw_event->event == MPT3SAS_PORT_ENABLE_COMPLETE) {
-			ioc->port_enable_cmds.status |= MPT3_CMD_RESET;
-			ioc->start_scan = 0;
+			fw_event_work_put(fw_event);
 		}
 
-		/*
-		 * Wait on the fw_event to complete. If this returns 1, then
-		 * the event was never executed, and we need a put for the
-		 * reference the work had on the fw_event.
-		 *
-		 * If it did execute, we wait for it to finish, and the put will
-		 * happen from _firmware_event_work()
-		 *
-		 * An event requeued by _scsih_fw_event_requeue() is pending
-		 * on its delayed_work instead, and its timer must not fire
-		 * after the workqueue is destroyed.
-		 */
-		if (fw_event->delayed_work_active) {
-			if (cancel_delayed_work_sync(&fw_event->delayed_work))
-				fw_event_work_put(fw_event);
-		} else if (cancel_work_sync(&fw_event->work)) {
+		fw_event = _scsih_get_current_event(ioc);
+		if (!fw_event)
+			break;
+		if (fw_event->event != MPT3SAS_REMOVE_UNRESPONDING_DEVICES) {
 			fw_event_work_put(fw_event);
+			break;
 		}
-
+		/* wait for it, then flush what was queued meanwhile */
+		if (cancel_work_sync(&fw_event->work))
+			fw_event_work_put(fw_event);
+		fw_event_work_put(fw_event);
 	}
 	ioc->fw_events_cleanup = 0;
 }
@@ -11985,13 +12069,16 @@ mpt3sas_scsih_reset_done_handler(struct MPT3SAS_ADAPTER *ioc)
 static void
 _mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
 {
-	ioc->current_event = fw_event;
-	_scsih_fw_event_del_from_list(ioc, fw_event);
+	if (!_scsih_fw_event_start(ioc, fw_event)) {
+		/* _scsih_fw_event_cleanup_queue() took it off the list */
+		fw_event_work_put(fw_event);
+		return;
+	}
 
 	/* the queue is being flushed so ignore this event */
 	if (ioc->remove_host || ioc->pci_error_recovery) {
+		_scsih_set_current_event(ioc, NULL);
 		fw_event_work_put(fw_event);
-		ioc->current_event = NULL;
 		return;
 	}
 
@@ -12046,7 +12133,6 @@ _mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
 	case MPI2_EVENT_SAS_TOPOLOGY_CHANGE_LIST:
 		if (_scsih_sas_topology_change_event(ioc, fw_event)) {
 			_scsih_fw_event_requeue(ioc, fw_event, 1000);
-			ioc->current_event = NULL;
 			return;
 		}
 		break;
@@ -12090,14 +12176,13 @@ _mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
 	case MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST:
 		if (_scsih_pcie_topology_change_event(ioc, fw_event)) {
 			_scsih_fw_event_requeue(ioc, fw_event, 1000);
-			ioc->current_event = NULL;
 			return;
 		}
 		break;
 	}
 out:
+	_scsih_set_current_event(ioc, NULL);
 	fw_event_work_put(fw_event);
-	ioc->current_event = NULL;
 }
 
 /**
Signature · 1 line
2.56.1
#4 lore raw

PATCH v3 3/11 scsi: mpt3sas: Requeue topology events only for busy devices

In reply to Jeff Layton

Since commit ad5957193107 ("scsi: mpt3sas: Add firmware event requeue support for busy devices"), _scsih_sas_topology_change_event() requeues the event whenever _scsih_add_device() returns non-zero. It retries the event every second. But _scsih_add_device() returns 1 only for a busy device. It returns -1 when it cannot add the device at all. This occurs when:

- the driver cannot read the SAS Device Page 0 of the device
- the device is not an end device, or is not present
- discovery reported an access problem
- the device is already added

None of these conditions clear with time, and nothing limits the number of retries. In a stress test, the driver retried the add of a device that was not present 249 times.

A requeued event keeps the handle and phy number from its original report. Meanwhile, the handle may go to another device, because a host reset assigns all handles again. In one stress run, a requeued add first ran after a host reset. At that time, its handle was not in use. Thus the read of SAS Device Page 0 failed, and the driver requeued the event. One second later, at the retry, a device that was returning on another phy had that handle.

The retry then makes mpt3sas_transport_update_links() record the SAS address of that device on the phy of the event. The add of the device then makes _transport_sanity_check() delete the port of the drive on that phy. Thus the driver removed the drive, although the drive never disappeared.

When _scsih_add_device() cannot add the device, return 0, as _scsih_pcie_add_device() does. Then the driver retries only a busy device. Some failures of the page read do clear with time:

- -EAGAIN, when no internal command is free or after a reset
- an IOC status of BUSY or INSUFFICIENT_RESOURCES

Continue to retry these failures, up to command_retry_count times, as for a busy device. This change also prevents a spin on such a device in the loops that call _scsih_add_device() after a host reset.

Fixes ad5957193107 ("scsi: mpt3sas: Add firmware event requeue support for busy devices")
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +16 -9
index b455b7e5d8f3..2d98fcb8e135 100644
@@ -8509,7 +8509,9 @@ _scsih_check_device(struct MPT3SAS_ADAPTER *ioc,
  *
  * Creating end device object, stored in ioc->sas_device_list.
  *
- * Return: 0 for success, non-zero for failure.
+ * Return: 1 if adding the device should be retried: it is busy, or
+ * reading its SAS Device Page 0 failed in a way that can clear with time.
+ * 0 otherwise.
  */
 static int
 _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
@@ -8525,12 +8527,15 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
 	u32 device_info;
 	u8 connector_name[5];
 	u8 port_id;
+	int r;
 
-	if ((mpt3sas_config_get_sas_device_pg0(ioc, &mpi_reply, &sas_device_pg0,
-	    MPI2_SAS_DEVICE_PGAD_FORM_HANDLE, handle))) {
+	r = mpt3sas_config_get_sas_device_pg0(ioc, &mpi_reply, &sas_device_pg0,
+	    MPI2_SAS_DEVICE_PGAD_FORM_HANDLE, handle);
+	if (r) {
 		ioc_err(ioc, "failure at %s:%d/%s()!\n",
 			__FILE__, __LINE__, __func__);
-		return -1;
+		/* e.g. no free internal command, or a reset intervened */
+		return r == -EAGAIN && retry_count < command_retry_count;
 	}
 
 	ioc_status = le16_to_cpu(mpi_reply.IOCStatus) &
@@ -8538,13 +8543,15 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
 	if (ioc_status != MPI2_IOCSTATUS_SUCCESS) {
 		ioc_err(ioc, "failure at %s:%d/%s()!\n",
 			__FILE__, __LINE__, __func__);
-		return -1;
+		return (ioc_status == MPI2_IOCSTATUS_BUSY ||
+		    ioc_status == MPI2_IOCSTATUS_INSUFFICIENT_RESOURCES) &&
+		    retry_count < command_retry_count;
 	}
 
 	/* check if this is end device */
 	device_info = le32_to_cpu(sas_device_pg0.DeviceInfo);
 	if (!(_scsih_is_end_device(device_info)))
-		return -1;
+		return 0;
 	set_bit(handle, ioc->pend_os_device_add);
 	sas_address = le64_to_cpu(sas_device_pg0.SASAddress);
 
@@ -8553,13 +8560,13 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
 	    MPI2_SAS_DEVICE0_FLAGS_DEVICE_PRESENT)) {
 		ioc_err(ioc, "device is not present handle(0x04%x)!!!\n",
 			handle);
-		return -1;
+		return 0;
 	}
 
 	/* check if there were any issues with discovery */
 	if (_scsih_check_access_status(ioc, sas_address, handle,
 	    sas_device_pg0.AccessStatus))
-		return -1;
+		return 0;
 
 	port_id = sas_device_pg0.PhysicalPort;
 	sas_device = mpt3sas_get_sdev_by_addr(ioc,
@@ -8567,7 +8574,7 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
 	if (sas_device) {
 		clear_bit(handle, ioc->pend_os_device_add);
 		sas_device_put(sas_device);
-		return -1;
+		return 0;
 	}
 
 	if (sas_device_pg0.EnclosureHandle) {
Signature · 1 line
2.56.1
#5 lore raw

PATCH v3 4/11 scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()

In reply to Jeff Layton

_scsih_ublock_io_device_wait() gets the pcie_device on each pass of its TEST UNIT READY retry loop. But it puts the pcie_device only at the bottom of the loop. The DEVICE_RETRY/START_UNIT/STOP_UNIT/RETRY_UA branch ends in a continue, which skips the put. Thus each retry of an NVMe device leaks a reference, and the driver can never free the _pcie_device.

The function needs the pcie_device only for the reset timeout and the reset method. Put the pcie_device immediately after the function reads these values.

Fixes aee682fad6cd ("scsi: mpt3sas: Improve device discovery and readiness handling for slow devices")
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +2 -2
index 2d98fcb8e135..45419f0ebedc 100644
@@ -4220,6 +4220,8 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
 				tr_timeout = pcie_device->reset_timeout;
 				tr_method = MPI26_SCSITASKMGMT_MSGFLAGS_PROTOCOL_LVL_RST_PCIE;
 			}
+			if (pcie_device)
+				pcie_device_put(pcie_device);
 			rc = _scsih_wait_for_device_to_become_ready(ioc,
 			      sas_target->handle, 0, (sas_target->flags &
 			      MPT_TARGET_FLAGS_RAID_COMPONENT), sdev->lun, tr_timeout, tr_method);
@@ -4241,8 +4243,6 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
 				}
 				continue;
 			}
-			if (pcie_device)
-				pcie_device_put(pcie_device);
 		} while ((rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
 		    rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
 			&& count++ <= command_retry_count);
Signature · 1 line
2.56.1
#6 lore raw

PATCH v3 5/11 scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()

In reply to Jeff Layton

When the firmware reports that a device responds again, the event worker unblocks the sdevs of the device. Previously, the worker used _scsih_ublock_io_device() for this, which uses _scsih_internal_device_unblock(). The unblock can fail with -EINVAL, if the SD layer set the device to SDEV_RUNNING while our block still stopped its queue. Then that helper blocks the device again and unblocks it, to restart the queue.

_scsih_ublock_io_device_wait(), which does a TUR check, replaced _scsih_ublock_io_device(). It open-codes the unblock without that retry, so such a device can stay stalled. Use _scsih_internal_device_unblock() in _scsih_ublock_io_device_wait() too.

Fixes aee682fad6cd ("scsi: mpt3sas: Improve device discovery and readiness handling for slow devices")
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +1 -2
index 45419f0ebedc..5d494ad12c54 100644
@@ -4247,10 +4247,9 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
 		    rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
 			&& count++ <= command_retry_count);
 
-		sas_device_priv_data->block = 0;
 		if (rc != DEVICE_READY)
 			sas_device_priv_data->deleted = 1;
-		scsi_internal_device_unblock_nowait(sdev, SDEV_RUNNING);
+		_scsih_internal_device_unblock(sdev, sas_device_priv_data);
 
 		if (rc != DEVICE_READY) {
 			sdev_printk(KERN_WARNING, sdev,
Signature · 1 line
2.56.1
#7 lore raw

PATCH v3 6/11 scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled

In reply to Jeff Layton

After a hard reset, the MPT3SAS_REMOVE_UNRESPONDING_DEVICES event polls each blocked device with TEST UNIT READY in _scsih_ublock_io_all_device(). It retries a device for up to command_retry_count seconds (144 by default).

If another hard reset starts during this time, it sets shost_recovery. Then _scsih_fw_event_cleanup_queue() waits for the event in cancel_work_sync(). While shost_recovery is set, _scsih_wait_for_device_to_become_ready() returns DEVICE_RETRY and does not test the device. Thus, before the reset can continue, the event consumes all the retries of the device that it polls. The event then marks that device deleted and offline, although it never tested the device.

If the event is cancelled, or if the host is being removed, the event already returns before it polls. Do the same check during polling, and stop polling if the check is true. The remaining devices stay blocked. The reset queues a new MPT3SAS_REMOVE_UNRESPONDING_DEVICES event, which polls them after recovery ends. If the host is removed, the removal deletes them.

Fixes 72340fecd0c8 ("scsi: mpt3sas: Add configurable command retry limit for slow-to-respond devices")
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +23 -1
index 5d494ad12c54..5657bda241f7 100644
@@ -4057,6 +4057,22 @@ _scsih_internal_device_unblock(struct scsi_device *sdev,
 	}
 }
 
+/**
+ * _scsih_unresponding_scan_cancelled - should polling devices stop
+ * @ioc: per adapter object
+ *
+ * While the MPT3SAS_REMOVE_UNRESPONDING_DEVICES event polls devices, a
+ * hard reset or the removal of the host waits for it to finish, and the
+ * readiness check doesn't test devices during a reset.
+ *
+ * Return: true if the event is being cancelled, false otherwise.
+ */
+static bool
+_scsih_unresponding_scan_cancelled(struct MPT3SAS_ADAPTER *ioc)
+{
+	return READ_ONCE(ioc->remove_host) || READ_ONCE(ioc->fw_events_cleanup);
+}
+
 /**
  * _scsih_ublock_io_all_device - unblock every device
  * @ioc: per adapter object
@@ -4114,7 +4130,13 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
 				pcie_device_put(pcie_device);
 		} while ((rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
 		    rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
-			&& count++ < command_retry_count);
+			&& count++ < command_retry_count &&
+			!_scsih_unresponding_scan_cancelled(ioc));
+		if (_scsih_unresponding_scan_cancelled(ioc)) {
+			/* a reset queues a new scan, which polls the rest */
+			scsi_device_put(sdev);
+			return;
+		}
 		sas_device_priv_data->block = 0;
 		if (rc != DEVICE_READY)
 			sas_device_priv_data->deleted = 1;
Signature · 1 line
2.56.1
#8 lore raw

Re: [PATCH v3 06/11] scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [High] scsi: mpt3sas: _mpt3sas_fw_work() continues scan after cancellation - [High] scsi: mpt3sas…

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] scsi: mpt3sas: _mpt3sas_fw_work() continues scan after cancellation
- [High] scsi: mpt3sas: _scsih_unresponding_scan_cancelled() ignores shost_recovery
--

commit 35d95180d221b7f75fd9660f0fc687ebab407910
Author: Jeff Layton <[email protected]>

scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled

This commit stops the device readiness polling loop in _scsih_ublock_io_all_device() and returns early when ioc->remove_host or ioc->fw_events_cleanup is set.

Quoted 13 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +5 -0
index 5d494ad12c542..5657bda241f77 100644
@@ -4057,6 +4057,22 @@ _scsih_internal_device_unblock(struct scsi_device *sdev,
 	}
 }
 
+/**
+ * _scsih_unresponding_scan_cancelled - should polling devices stop
+ * @ioc: per adapter object
+ *
+ * While the MPT3SAS_REMOVE_UNRESPONDING_DEVICES event polls devices, a

drivers/scsi/mpt3sas/mpt3sas_scsih.c _scsih_internal_device_unblock(struct scsi_device *sdev, · line 4074

Quoted 10 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +10 -0
+ * hard reset or the removal of the host waits for it to finish, and the
+ * readiness check doesn't test devices during a reset.
+ *
+ * Return: true if the event is being cancelled, false otherwise.
+ */
+static bool
+_scsih_unresponding_scan_cancelled(struct MPT3SAS_ADAPTER *ioc)
+{
+	return READ_ONCE(ioc->remove_host) || READ_ONCE(ioc->fw_events_cleanup);
+}

[Severity: High]
Can a device still exhaust its retries and be offlined untested while ioc->shost_recovery is set before ioc->fw_events_cleanup is set?

mpt3sas_base_hard_reset_handler() sets ioc->shost_recovery = 1 before running mpt3sas_wait_for_commands_to_complete() (up to 10 seconds) and mpt3sas_base_make_ioc_ready() (15 to 45+ seconds), whereas ioc->fw_events_cleanup is only set later inside _base_clear_outstanding_commands() -> ... -> _scsih_fw_event_cleanup_queue(). During that initial reset window, _scsih_wait_for_device_to_become_ready() immediately returns DEVICE_RETRY without testing the device, while _scsih_unresponding_scan_cancelled() still returns false. If count reaches command_retry_count before _scsih_fw_event_cleanup_queue() runs (or if mpt3sas_base_make_ioc_ready() fails and skips _base_clear_outstanding_commands()), the loop still marks the untested device deleted and SDEV_OFFLINE.

Quoted 5 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +0 -0
 /**
  * _scsih_ublock_io_all_device - unblock every device
  * @ioc: per adapter object
@@ -4114,7 +4130,13 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
 				pcie_device_put(pcie_device);

drivers/scsi/mpt3sas/mpt3sas_scsih.c _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs) · line 4139

Quoted 10 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +7 -1
 		} while ((rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
 		    rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
-			&& count++ < command_retry_count);
+			&& count++ < command_retry_count &&
+			!_scsih_unresponding_scan_cancelled(ioc));
+		if (_scsih_unresponding_scan_cancelled(ioc)) {
+			/* a reset queues a new scan, which polls the rest */
+			scsi_device_put(sdev);
+			return;
+		}

[Severity: High]
When this returns early, doesn't _mpt3sas_fw_work() continue running the rest of the MPT3SAS_REMOVE_UNRESPONDING_DEVICES handler while the reset or removal is waiting in cancel_work_sync()?

drivers/scsi/mpt3sas/mpt3sas_scsih.c:_mpt3sas_fw_work() {
	...
	case MPT3SAS_REMOVE_UNRESPONDING_DEVICES:
		...
		_scsih_remove_unresponding_devices(ioc);
		_scsih_del_dirty_vphy(ioc);
		_scsih_del_dirty_port_entries(ioc);
		if (ioc->is_gen35_ioc)
			_scsih_update_device_qdepth(ioc);
		_scsih_scan_for_devices_after_reset(ioc);
		...
		if (ioc->is_driver_loading)
			_scsih_complete_devices_scanning(ioc);
		_scsih_set_nvme_max_shutdown_latency(ioc);
		break;
}

When cancelled by a hard reset, the reset thread holds ioc->reset_in_progress_mutex with the controller in MPI2_IOC_STATE_READY and blocks in _scsih_fw_event_cleanup_queue() -> cancel_work_sync() before _base_make_ioc_operational() is reached. _scsih_scan_for_devices_after_reset() then issues config requests that wait for MPI2_IOC_STATE_OPERATIONAL in mpt3sas_wait_for_ioc(): if ioc->is_driver_loading is set, mpt3sas_wait_for_ioc() returns -ETIME and _config_request() calls mpt3sas_base_hard_reset_handler(), deadlocking on ioc->reset_in_progress_mutex; otherwise each config request stalls the reset for its 15-second timeout. Should _mpt3sas_fw_work() also bail out to out after _scsih_remove_unresponding_devices() when _scsih_unresponding_scan_cancelled(ioc) is true?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#9 lore raw

PATCH v3 7/11 scsi: mpt3sas: Reset the TUR retry count for each device

In reply to Jeff Layton

_scsih_ublock_io_all_device() and _scsih_ublock_io_device_wait() poll each blocked sdev with TEST UNIT READY, for up to command_retry_count retries. These functions set the retry counter to 0 once per call, not once per sdev. Thus retries on one device consume the retry budget of all later devices. When the budget is empty, the driver marks a device deleted after its first TUR and sets it SDEV_OFFLINE. This occurs even for a device that needs only one retry.

Start the count from 0 for each sdev.

Fixes 72340fecd0c8 ("scsi: mpt3sas: Add configurable command retry limit for slow-to-respond devices")
---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +4 -2
index 5657bda241f7..1269446eec4e 100644
@@ -4089,7 +4089,7 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
 	enum device_responsive_state rc;
 	struct _sas_device *sas_device = NULL;
 	struct _pcie_device *pcie_device = NULL;
-	int count = 0;
+	int count;
 	u8 tr_method = 0;
 	u8 tr_timeout = 30;
 
@@ -4113,6 +4113,7 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
 			continue;
 		}
 
+		count = 0;
 		do {
 			pcie_device = mpt3sas_get_pdev_by_handle(ioc, sas_target->handle);
 			if (pcie_device && (!ioc->tm_custom_handling) &&
@@ -4199,7 +4200,7 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
 	struct _pcie_device *pcie_device;
 	u8 tr_timeout = 30;
 	u8 tr_method = 0;
-	int count = 0;
+	int count;
 
 	/* moving devices from SDEV_OFFLINE to SDEV_BLOCK */
 	shost_for_each_device(sdev, ioc->shost) {
@@ -4234,6 +4235,7 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
 		if (!sas_device_priv_data->block)
 			continue;
 
+		count = 0;
 		do {
 			host_reset_completion_count = 0;
 			pcie_device = mpt3sas_get_pdev_by_handle(ioc, sas_target->handle);
Signature · 1 line
2.56.1
#10 lore raw

PATCH v3 8/11 scsi: mpt3sas: Cache the missing delays the IOC actually uses

In reply to Jeff Layton

When the missing_delay module parameter is set, mpt3sas_base_update_missing_delay() writes the new delays to SAS IO Unit Page 1. If the write succeeds, the function caches the requested values in ioc->device_missing_delay and ioc->io_missing_delay.

mpt3sas_config_set_sas_iounit_pg1() ignores the result of the WRITE_CURRENT request. It returns the result of the subsequent WRITE_NVRAM request. If the WRITE_CURRENT request fails and the WRITE_NVRAM request succeeds, the cached delays do not match the delays that the IOC uses. The device_delay sysfs attribute then reports the wrong value.

After the write, read the page again and cache the delays that it reports. If this read fails, cache the requested delays if the NVRAM update succeeded, as before.

Fixes f92363d12359 ("[SCSI] mpt3sas: add new driver supporting 12GB SAS")
---
 drivers/scsi/mpt3sas/mpt3sas_base.c | 46 +++++++++++++++++++++++++------------
 1 file changed, 31 insertions(+), 15 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_base.c +31 -15
index 5c957044eded..07e16acc8abc 100644
@@ -4923,6 +4923,7 @@ mpt3sas_base_update_missing_delay(struct MPT3SAS_ADAPTER *ioc,
 	Mpi2ConfigReply_t mpi_reply;
 	u8 num_phys = 0;
 	u16 ioc_status;
+	int rc;
 
 	mpt3sas_config_get_number_hba_phys(ioc, &num_phys);
 	if (!num_phys)
@@ -4969,23 +4970,38 @@ mpt3sas_base_update_missing_delay(struct MPT3SAS_ADAPTER *ioc,
 	io_missing_delay_original = sas_iounit_pg1->IODeviceMissingDelay;
 	sas_iounit_pg1->IODeviceMissingDelay = io_missing_delay;
 
-	if (!mpt3sas_config_set_sas_iounit_pg1(ioc, &mpi_reply, sas_iounit_pg1,
-	    sz)) {
-		if (dmd & MPI2_SASIOUNIT1_REPORT_MISSING_UNIT_16)
-			dmd_new = (dmd &
-			    MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK) * 16;
-		else
-			dmd_new =
-		    dmd & MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK;
-		ioc_info(ioc, "device_missing_delay: old(%d), new(%d)\n",
-			 dmd_orignal, dmd_new);
-		ioc_info(ioc, "ioc_missing_delay: old(%d), new(%d)\n",
-			 io_missing_delay_original,
-			 io_missing_delay);
-		ioc->device_missing_delay = dmd_new;
-		ioc->io_missing_delay = io_missing_delay;
+	/*
+	 * This returns the result of the NVRAM update, not of the update
+	 * to the current page, so read back the delays the IOC now uses.
+	 * If that fails, assume the requested delays are used if the NVRAM
+	 * update succeeded.
+	 */
+	rc = mpt3sas_config_set_sas_iounit_pg1(ioc, &mpi_reply,
+	    sas_iounit_pg1, sz);
+	if (!mpt3sas_config_get_sas_iounit_pg1(ioc, &mpi_reply,
+	    sas_iounit_pg1, sz) &&
+	    (le16_to_cpu(mpi_reply.IOCStatus) & MPI2_IOCSTATUS_MASK) ==
+	    MPI2_IOCSTATUS_SUCCESS) {
+		dmd = sas_iounit_pg1->ReportDeviceMissingDelay;
+		io_missing_delay = sas_iounit_pg1->IODeviceMissingDelay;
+	} else {
+		ioc_warn(ioc, "failed to read back the missing delays\n");
+		if (rc)
+			goto out;
 	}
 
+	if (dmd & MPI2_SASIOUNIT1_REPORT_MISSING_UNIT_16)
+		dmd_new = (dmd & MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK) * 16;
+	else
+		dmd_new = dmd & MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK;
+	ioc_info(ioc, "device_missing_delay: old(%d), new(%d)\n",
+		 dmd_orignal, dmd_new);
+	ioc_info(ioc, "ioc_missing_delay: old(%d), new(%d)\n",
+		 io_missing_delay_original,
+		 io_missing_delay);
+	ioc->device_missing_delay = dmd_new;
+	ioc->io_missing_delay = io_missing_delay;
+
 out:
 	kfree(sas_iounit_pg1);
 }
Signature · 1 line
2.56.1
#11 lore raw

PATCH v3 9/11 scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown

In reply to Jeff Layton

On DELAY_NOT_RESPONDING, the event callback runs _scsih_block_io_device() at interrupt time. This function walks the sdevs with shost_for_each_device(). It uses sdev->hostdata and its sas_target, and holds no lock.

The sdev reference pins only the scsi_device. __scsi_remove_device() can still call ->sdev_destroy() at the same time, for example when a scan fails to probe a new LUN. scsih_sdev_destroy() frees hostdata, which causes a use-after-free. scsi_target_reap() frees the hostdata of the target, after sdev_destroy.

Add ioc->block_lock. Take it:

- in _scsih_block_io_device(), for each sdev
- around the code that sets and clears sdev->hostdata in scsih_sdev_init() and scsih_sdev_destroy()

The sdev is on shost->__devices before sdev_init. Because the driver publishes hostdata under the lock, the lock also orders the sas_target store. While the lock is held and hostdata is non-NULL, hostdata and its target are live.

Fixes f92363d12359 ("[SCSI] mpt3sas: add new driver supporting 12GB SAS")
---
 drivers/scsi/mpt3sas/mpt3sas_base.h  |  2 ++
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 30 ++++++++++++++++++++++--------
 2 files changed, 24 insertions(+), 8 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_base.h +2 -0
index 47255bf9cdda..419fdf164fc6 100644
@@ -1249,6 +1249,7 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  * @pcie_device_list: pcie device object list
  * @pcie_device_init_list: pcie device object list (used only at init time)
  * @pcie_device_lock:
+ * @block_lock: serializes sdev blocking against sdev hostdata teardown
  * @io_missing_delay: time for IO completed by fw when PDR enabled
  * @device_missing_delay: time for device missing by fw when PDR enabled
  * @sas_id : used for setting volume target IDs
@@ -1482,6 +1483,7 @@ struct MPT3SAS_ADAPTER {
 	struct list_head pcie_device_list;
 	struct list_head pcie_device_init_list;
 	spinlock_t      pcie_device_lock;
+	spinlock_t	block_lock;
 
 	struct list_head raid_device_list;
 	spinlock_t	raid_device_lock;
drivers/scsi/mpt3sas/mpt3sas_scsih.c +22 -8
index 1269446eec4e..09dfa3124ebf 100644
@@ -2132,12 +2132,17 @@ scsih_sdev_init(struct scsi_device *sdev)
 	sas_target_priv_data = starget->hostdata;
 	sas_target_priv_data->num_luns++;
 	sas_device_priv_data->sas_target = sas_target_priv_data;
+
+	shost = dev_to_shost(&starget->dev);
+	ioc = shost_priv(shost);
+
+	/* _scsih_block_io_device() may already see this sdev */
+	spin_lock_irqsave(&ioc->block_lock, flags);
 	sdev->hostdata = sas_device_priv_data;
+	spin_unlock_irqrestore(&ioc->block_lock, flags);
 	if ((sas_target_priv_data->flags & MPT_TARGET_FLAGS_RAID_COMPONENT))
 		sdev->no_uld_attach = 1;
 
-	shost = dev_to_shost(&starget->dev);
-	ioc = shost_priv(shost);
 	if (starget->channel == RAID_CHANNEL) {
 		spin_lock_irqsave(&ioc->raid_device_lock, flags);
 		raid_device = _scsih_raid_device_find_by_id(ioc,
@@ -2189,6 +2194,7 @@ scsih_sdev_init(struct scsi_device *sdev)
 static void
 scsih_sdev_destroy(struct scsi_device *sdev)
 {
+	struct MPT3SAS_DEVICE *sas_device_priv_data;
 	struct MPT3SAS_TARGET *sas_target_priv_data;
 	struct scsi_target *starget;
 	struct Scsi_Host *shost;
@@ -2231,8 +2237,11 @@ scsih_sdev_destroy(struct scsi_device *sdev)
 		spin_unlock_irqrestore(&ioc->sas_device_lock, flags);
 	}
 
-	kfree(sdev->hostdata);
+	spin_lock_irqsave(&ioc->block_lock, flags);
+	sas_device_priv_data = sdev->hostdata;
 	sdev->hostdata = NULL;
+	spin_unlock_irqrestore(&ioc->block_lock, flags);
+	kfree(sas_device_priv_data);
 }
 
 /**
@@ -4384,26 +4393,30 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 	struct MPT3SAS_DEVICE *sas_device_priv_data;
 	struct scsi_device *sdev;
 	struct _sas_device *sas_device;
+	unsigned long flags;
 
 	sas_device = mpt3sas_get_sdev_by_handle(ioc, handle);
 
 	shost_for_each_device(sdev, ioc->shost) {
+		spin_lock_irqsave(&ioc->block_lock, flags);
 		sas_device_priv_data = sdev->hostdata;
 		if (!sas_device_priv_data)
-			continue;
+			goto next;
 		if (sas_device_priv_data->sas_target->handle != handle)
-			continue;
+			goto next;
 		if (sas_device_priv_data->block)
-			continue;
+			goto next;
 		if (sas_device && sas_device->pend_sas_rphy_add)
-			continue;
+			goto next;
 		if (sas_device_priv_data->ignore_delay_remove) {
 			sdev_printk(KERN_INFO, sdev,
 			"%s skip device_block for SES handle(0x%04x)\n",
 			__func__, sas_device_priv_data->sas_target->handle);
-			continue;
+			goto next;
 		}
 		_scsih_internal_device_block(sdev, sas_device_priv_data);
+next:
+		spin_unlock_irqrestore(&ioc->block_lock, flags);
 	}
 
 	if (sas_device)
@@ -13697,6 +13710,7 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	spin_lock_init(&ioc->raid_device_lock);
 	spin_lock_init(&ioc->pcie_device_lock);
 	spin_lock_init(&ioc->diag_trigger_lock);
+	spin_lock_init(&ioc->block_lock);
 
 	INIT_LIST_HEAD(&ioc->sas_device_list);
 	INIT_LIST_HEAD(&ioc->sas_device_init_list);
Signature · 1 line
2.56.1
#12 lore raw

PATCH v3 10/11 scsi: mpt3sas: Don't use freed hostdata while polling devices

In reply to Jeff Layton

_scsih_ublock_io_all_device() and _scsih_ublock_io_device_wait() send TURs to each blocked sdev until it is ready, for up to command_retry_count rounds. They sleep between retries. They read sdev->hostdata and its sas_target once. Then they use both during polling and after it.

The sdev reference pins only the scsi_device. __scsi_remove_device() can call ->sdev_destroy() during this time. For example, this occurs when a scan fails to probe a new LUN, or when the device is deleted through sysfs. ->sdev_destroy() frees the hostdata. If the sdev was the last LUN, scsi_target_reap() then frees the hostdata of the target. Thus the worker reads and writes freed memory.

Before each TUR, read the handle and the target flags under block_lock. If the hostdata is gone, stop polling. Do the final unblock under the lock, and only if the hostdata is unchanged. Copy the target fields that the offline logging needs. Take block_lock in the other worker-side walkers that block or unblock sdevs too.

Fixes aee682fad6cd ("scsi: mpt3sas: Improve device discovery and readiness handling for slow devices")
---
 drivers/scsi/mpt3sas/mpt3sas_base.h  |   3 +-
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 210 ++++++++++++++++++++++++++++-------
 2 files changed, 173 insertions(+), 40 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_base.h +2 -1
index 419fdf164fc6..f5e1edf1a280 100644
@@ -1249,7 +1249,8 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  * @pcie_device_list: pcie device object list
  * @pcie_device_init_list: pcie device object list (used only at init time)
  * @pcie_device_lock:
- * @block_lock: serializes sdev blocking against sdev hostdata teardown
+ * @block_lock: serializes sdev blocking and unblocking against sdev
+ *	hostdata teardown
  * @io_missing_delay: time for IO completed by fw when PDR enabled
  * @device_missing_delay: time for device missing by fw when PDR enabled
  * @sas_id : used for setting volume target IDs
drivers/scsi/mpt3sas/mpt3sas_scsih.c +171 -39

367 lines not on this page · raw message

Signature · 1 line
2.56.1
#13 lore raw

Re: [PATCH v3 10/11] scsi: mpt3sas: Don't use freed hostdata while polling devices

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] scsi: mpt3sas: unlocked hostdata dereference in device unblock loops --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] scsi: mpt3sas: unlocked hostdata dereference in device unblock loops
--

commit a3557f5a5117e28396670a7dd01d7199c96743db
Author: Jeff Layton <[email protected]>

scsi: mpt3sas: Don't use freed hostdata while polling devices

This commit serializes device polling and unblocking against sdev hostdata teardown using ioc->block_lock so that polling loops stop when hostdata is freed.

Quoted 4 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +0 -0
index 09dfa3124ebf9..814c18289cf1c 100644

[ ... ]

Quoted 34 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +14 -4
@@ -4098,10 +4191,11 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
 	enum device_responsive_state rc;
 	struct _sas_device *sas_device = NULL;
 	struct _pcie_device *pcie_device = NULL;
+	struct _scsih_target_ids ids;
 	int count;
 	u8 tr_method = 0;
 	u8 tr_timeout = 30;
-
+	unsigned long flags;
 
 	shost_for_each_device(sdev, ioc->shost) {
 		sas_device_priv_data = sdev->hostdata;
@@ -4116,23 +4210,40 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
 			continue;
 
 		if ((no_turs) || (!issue_scsi_cmd_to_bringup_drive)) {
-			sdev_printk(KERN_WARNING, sdev, "device_unblocked handle(0x%04x)\n",
-				sas_device_priv_data->sas_target->handle);
-			_scsih_internal_device_unblock(sdev, sas_device_priv_data);
+			spin_lock_irqsave(&ioc->block_lock, flags);
+			if (sdev->hostdata != sas_device_priv_data)
+				goto next;
+			if (sas_device_priv_data->block) {
+				sdev_printk(KERN_WARNING, sdev,
+				    "device_unblocked handle(0x%04x)\n",
+				    sas_target->handle);
+				_scsih_internal_device_unblock(sdev,
+				    sas_device_priv_data);
+			}
+next:
+			spin_unlock_irqrestore(&ioc->block_lock, flags);
 			continue;
 		}

[ ... ]

Quoted 1 line
drivers/scsi/mpt3sas/mpt3sas_scsih.c +0 -0
@@ -4207,27 +4315,32 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,

[ ... ]

Quoted 16 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +2 -0
 	/* moving devices from SDEV_BLOCK to SDEV_RUNNING state */
 	shost_for_each_device(sdev, ioc->shost) {
 		sas_device_priv_data = sdev->hostdata;
 		if (!sas_device_priv_data)
 			continue;
 		sas_target = sas_device_priv_data->sas_target;
 		if (!sas_target)
 			continue;
 		if (sas_target->sas_address != sas_address ||
 		    sas_target->port != port)
 			continue;
 		if (!sas_device_priv_data->block)
 			continue;
 
+		/*
+		 * Polling can take minutes, and sdev_destroy() and

drivers/scsi/mpt3sas/mpt3sas_scsih.c

Quoted 10 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +6 -1
+		 * scsi_target_reap() can free the hostdata meanwhile.
+		 */
+		rc = DEVICE_ERROR;
 		count = 0;
 		do {
 			host_reset_completion_count = 0;
-			pcie_device = mpt3sas_get_pdev_by_handle(ioc, sas_target->handle);
+			if (!_scsih_get_target_ids(ioc, sdev,
+			    sas_device_priv_data, &ids))
+				break;

[Severity: High]
Can the initial checks at the top of _scsih_ublock_io_all_device() and the second loop of _scsih_ublock_io_device_wait() still read freed sas_device_priv_data and sas_target before ioc->block_lock is taken?

Unlike the first loop in _scsih_ublock_io_device_wait(), both of these loops read sdev->hostdata and dereference sas_device_priv_data->sas_target, sas_target->deleted (or sas_address and port), and sas_device_priv_data->block without holding ioc->block_lock:

CPU0 CPU1
_scsih_ublock_io_all_device()
sas_device_priv_data = sdev->hostdata

                                        __scsi_remove_device(sdev)
                                          scsih_sdev_destroy(sdev)
                                            spin_lock(&ioc->block_lock)
                                            sdev->hostdata = NULL
                                            spin_unlock(&ioc->block_lock)
                                            kfree(sas_device_priv_data)
                                          scsi_target_reap()
                                            ...
                                              kfree(sas_target)
  sas_target =
    sas_device_priv_data->sas_target    <-- use-after-free

if (sas_target->deleted) <-- use-after-free

Taking ioc->block_lock at the start of each iteration to validate sdev->hostdata, sas_target, and sas_device_priv_data->block before deciding whether to unblock or poll the device would close this window.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
#14 lore raw

PATCH v3 11/11 scsi: mpt3sas: Time out delayed not-responding device blocks

In reply to Jeff Layton

We have an agent that writes "- - -" to the sysfs scan attribute at regular intervals. Several of these scans deadlocked:

1. A drive in reset failed its initial scan. Thus the worker started to remove the drive.
2. A sysfs scan took sas_host->lock, allocated an sdev for the drive and sent an INQUIRY.
3. A DELAY_NOT_RESPONDING event blocked the new sdev.
4. The worker waited for sas_host->lock in sas_rphy_free():

      __mutex_lock
      sas_port_delete+0x75 [scsi_transport_sas]
      mpt3sas_transport_port_remove
      _scsih_add_device

After that, all later scan attempts hung while they waited for sas_host->lock.

On DELAY_NOT_RESPONDING, the event callback blocks every sdev on the handle. Only these two paths release the block:
- the callback, if the firmware later reports TARG_NOT_RESPONDING
- the firmware event worker, when it processes a later event

Commit e4bc7f5c21a1 ("mpt3sas: Don't block the drive when drive addition under the control of SML") partially fixed a similar issue. That commit stops the driver from blocking sdevs that a scan still probes. This is not sufficient. When LUN 0 is visible, the scan also sends REPORT LUNS to it. LUN 0 may be blocked before or during the scan.

Within the report device missing delay, one of two things occurs:
- The firmware reports the device as not responding, and the callback unblocks it.
- The device returns.

After that delay, the block has no further use.

Schedule delayed_work on system_dfl_wq for the missing delay plus 30s. When the work runs, unblock all expired sdevs, except during a host reset. Do not wait for SCSI EH to end. scsi_send_eh_cmnd() waits for SDEV_BLOCK to end only up to the EH command timeout. Thus EH would fail its TURs to a device that is back, and set it offline. Meanwhile, the unblock by the event worker waits for EH to end.

The driver tracked only the SAS missing delay. Also read the PCIe missing delay from PCIe IO Unit Page 1. The driver already reads this page for the NVMe queue depth. Use the delay that applies to the device type. Until the first read of the page, assume the longest PCIe delay that the firmware supports. Because the timeouts differ, always queue the work for the earliest deadline.

Before the worker polls a blocked sdev, clear the deadline of the sdev under ioc->block_lock. Then dnr_unblock_work leaves the sdev to the worker. Do not put a timed block on offline sdevs. Expiry would set them running again with no TUR check. This includes sdevs that failed the TUR check of the worker. They are marked deleted under block_lock, but set offline only after the lock is released.

If a hard reset cancels the polling, give new deadlines to the sdevs that are still blocked. A failed reset queues no new scan. Also, the next scan removes missing devices before it polls the others. That removal can deadlock again.

If a host reset starts during a broadcast primitive pass, the pass skips its unblock. For the same reason as above, give deadlines to the sdevs that the pass blocks. Let sdevs that are already blocked keep their deadlines. While the pass queries and aborts the commands of these sdevs, prevent dnr_unblock_work from unblocking them.

Exclude offline sdevs from the pass. They take no I/O. The unblock by the pass, or an expiry, would set them running again without a TUR check.

A repeat DNR on a blocked sdev restarts its deadline, but at most MPT3SAS_DNR_BLOCK_MAX_EXTENDS times. Thus a drive that continues to flap cannot make a scan wait on it for as long as the flapping lasts.

An expired block may end before the device returns. Thus, at its next unblock, the worker still sends TURs to those sdevs. It sets them offline if they are not ready. If the driver makes a new delayed not-responding block during polling, keep that block until its own expiry or the next event.

Fixes f92363d12359 ("[SCSI] mpt3sas: add new driver supporting 12GB SAS")
---
 drivers/scsi/mpt3sas/mpt3sas_base.c  |  16 ++
 drivers/scsi/mpt3sas/mpt3sas_base.h  |  26 ++-
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 339 +++++++++++++++++++++++++++++++++--
 3 files changed, 369 insertions(+), 12 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_base.c +16 -0
index 07e16acc8abc..b7169253ca6e 100644
@@ -5421,6 +5421,7 @@ _base_update_diag_trigger_pages(struct MPT3SAS_ADAPTER *ioc)
 /**
  * _base_assign_fw_reported_qd	- Get FW reported QD for SAS/SATA devices.
  *				- On failure set default QD values.
+ *				- Also get the PCIe device missing delay.
  * @ioc : per adapter object
  *
  * Returns 0 for success, non-zero for failure.
@@ -5432,6 +5433,7 @@ static int _base_assign_fw_reported_qd(struct MPT3SAS_ADAPTER *ioc)
 	Mpi2SasIOUnitPage1_t sas_iounit_pg1;
 	Mpi26PCIeIOUnitPage1_t pcie_iounit_pg1;
 	u16 depth;
+	u8 dmd;
 	int rc = 0;
 
 	ioc->max_wideport_qd = MPT3SAS_SAS_QUEUE_DEPTH;
@@ -5469,6 +5471,20 @@ static int _base_assign_fw_reported_qd(struct MPT3SAS_ADAPTER *ioc)
 	ioc->max_nvme_qd = (le16_to_cpu(pcie_iounit_pg1.NVMeMaxQueueDepth)) ?
 	    (le16_to_cpu(pcie_iounit_pg1.NVMeMaxQueueDepth)) :
 	    MPT3SAS_NVME_QUEUE_DEPTH;
+
+	if ((le16_to_cpu(mpi_reply.IOCStatus) & MPI2_IOCSTATUS_MASK) !=
+	    MPI2_IOCSTATUS_SUCCESS) {
+		ioc_warn(ioc, "PCIe device missing delay unknown, using %us\n",
+		    ioc->pcie_device_missing_delay);
+		goto out;
+	}
+	dmd = pcie_iounit_pg1.DMDReportPCIe;
+	if (dmd & MPI26_PCIEIOUNIT1_DMDRPT_UNIT_16_SEC)
+		ioc->pcie_device_missing_delay =
+		    (dmd & MPI26_PCIEIOUNIT1_DMDRPT_DELAY_TIME_MASK) * 16;
+	else
+		ioc->pcie_device_missing_delay =
+		    dmd & MPI26_PCIEIOUNIT1_DMDRPT_DELAY_TIME_MASK;
 out:
 	dinitprintk(ioc, pr_err(
 	    "MaxWidePortQD: 0x%x MaxNarrowPortQD: 0x%x MaxSataQD: 0x%x MaxNvmeQD: 0x%x\n",
drivers/scsi/mpt3sas/mpt3sas_base.h +24 -2
index f5e1edf1a280..6d3ded18c3a2 100644
@@ -469,6 +469,14 @@ struct MPT3SAS_TARGET {
  * @flags: MPT_DEVICE_XXX flags
  * @configured_lun: lun is configured
  * @block: device is in SDEV_BLOCK state
+ * @block_deadline: jiffies after which a delayed not-responding block is
+ *	released without waiting for the firmware event worker, 0 if none
+ * @dnr_expired: a delayed not-responding block expired before the firmware
+ *	event worker checked the device; it still needs a readiness check
+ * @block_extends: times a repeat delayed not-responding event has pushed
+ *	out @block_deadline
+ * @block_seq: incremented on each new block, so the firmware event worker
+ *	can tell whether the device was blocked again while it polled it
  * @tlr_snoop_check: flag used in determining whether to disable TLR
  * @eedp_enable: eedp support enable bit
  * @eedp_type: 0(type_1), 1(type_2), 2(type_3)
@@ -486,6 +494,10 @@ struct MPT3SAS_DEVICE {
 	u8	ignore_delay_remove;
 	/* Iopriority Command Handling */
 	u8	ncq_prio_enable;
+	u8	dnr_expired;
+	u8	block_extends;
+	unsigned int	block_seq;
+	unsigned long	block_deadline;
 	/*
 	 * Bug workaround for SATL handling: the mpt2/3sas firmware
 	 * doesn't return BUSY or TASK_SET_FULL for subsequent
@@ -1165,6 +1177,9 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  *			pci resource handling
  * @fault_reset_work_q: fw fault workqueue
  * @fault_reset_work: fw fault work
+ * @dnr_unblock_work: releases expired delayed not-responding blocks
+ * @dnr_unblock_deadline: jiffies dnr_unblock_work is set to run at, 0 if
+ *	it isn't pending.  Protected by block_lock.
  * @firmware_event_thread: fw event work queue
  * @fw_event_lock:
  * @fw_event_list: list of fw events
@@ -1172,6 +1187,8 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  * @fw_event_cleanup: set to one while cleaning up the fw events
  * @aen_event_read_flag: event log was read
  * @broadcast_aen_busy: broadcast aen waiting to be serviced
+ * @broadcast_tm_active: broadcast primitive pass running, with every device
+ *	blocked; dnr_unblock_work waits for it
  * @shost_recovery: host reset in progress
  * @ioc_reset_in_progress_lock:
  * @ioc_link_reset_in_progress: phy/hard reset in progress
@@ -1249,10 +1266,11 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  * @pcie_device_list: pcie device object list
  * @pcie_device_init_list: pcie device object list (used only at init time)
  * @pcie_device_lock:
- * @block_lock: serializes sdev blocking and unblocking against sdev
- *	hostdata teardown
+ * @block_lock: serializes sdev blocking and unblocking against
+ *	dnr_unblock_work and sdev hostdata teardown
  * @io_missing_delay: time for IO completed by fw when PDR enabled
  * @device_missing_delay: time for device missing by fw when PDR enabled
+ * @pcie_device_missing_delay: time for PCIe device missing by fw
  * @sas_id : used for setting volume target IDs
  * @pcie_target_id: used for setting pcie target IDs
  * @blocking_handles: bitmask used to identify which devices need blocking
@@ -1352,6 +1370,8 @@ struct MPT3SAS_ADAPTER {
 	/* fw fault handler */
 	struct workqueue_struct *fault_reset_work_q;
 	struct delayed_work fault_reset_work;
+	struct delayed_work dnr_unblock_work;
+	unsigned long dnr_unblock_deadline;
 
 	/* fw event handler */
 	struct workqueue_struct	*firmware_event_thread;
@@ -1363,6 +1383,7 @@ struct MPT3SAS_ADAPTER {
 	 /* misc flags */
 	int		aen_event_read_flag;
 	u8		broadcast_aen_busy;
+	u8		broadcast_tm_active;
 	u16		broadcast_aen_pending;
 	u8		shost_recovery;
 	u8		got_task_abort_from_ioctl;
@@ -1490,6 +1511,7 @@ struct MPT3SAS_ADAPTER {
 	spinlock_t	raid_device_lock;
 	u8		io_missing_delay;
 	u16		device_missing_delay;
+	u16		pcie_device_missing_delay;
 	int		sas_id;
 	int		pcie_target_id;
 
drivers/scsi/mpt3sas/mpt3sas_scsih.c +329 -10

544 lines not on this page · raw message

Signature · 1 line
2.56.1
#15 lore raw

Re: [PATCH v3 11/11] scsi: mpt3sas: Time out delayed not-responding device blocks

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] scsi: mpt3sas: new DNR events ignored while polling in _scsih_block_io_device() - [Medium] scsi…

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] scsi: mpt3sas: new DNR events ignored while polling in _scsih_block_io_device()
- [Medium] scsi: mpt3sas: sticky deleted flag on sdev prevents future device blocking
--

commit bc92a124ea93f9bf48382ebe6742a2dde2541670
Author: Jeff Layton <[email protected]>

scsi: mpt3sas: Time out delayed not-responding device blocks

This commit adds a delayed work item to time out and release delayed not-responding SDEV_BLOCK states after the firmware device missing delay plus a 30-second margin, and coordinates that timeout with device readiness polling and broadcast primitive handling.

Quoted 4 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +0 -0
index 814c18289cf1c..6eb92bd4f1225 100644

[ ... ]

Quoted 1 line
drivers/scsi/mpt3sas/mpt3sas_scsih.c +0 -0
@@ -4125,23 +4128,106 @@ _scsih_get_target_ids(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,

[ ... ]

Quoted 19 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +19 -0
+static bool
+_scsih_claim_block(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
+		   struct MPT3SAS_DEVICE *sas_device_priv_data,
+		   unsigned int *block_seq)
+{
+	unsigned long flags;
+	bool claimed = false;
+
+	spin_lock_irqsave(&ioc->block_lock, flags);
+	if (sdev->hostdata == sas_device_priv_data &&
+	    (sas_device_priv_data->block || sas_device_priv_data->dnr_expired)) {
+		sas_device_priv_data->block_deadline = 0;
+		sas_device_priv_data->dnr_expired = 0;
+		*block_seq = sas_device_priv_data->block_seq;
+		claimed = true;
+	}
+	spin_unlock_irqrestore(&ioc->block_lock, flags);
+	return claimed;
+}

[ ... ]

Quoted 16 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +7 -0
@@ -4149,8 +4235,15 @@ _scsih_end_poll(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
 	spin_lock_irqsave(&ioc->block_lock, flags);
 	if (!__scsih_get_target_ids(sdev, sas_device_priv_data, ids))
 		goto out;
+	if (sas_device_priv_data->block_seq != block_seq) {
+		sdev_printk(KERN_INFO, sdev,
+		    "blocked again while polled, handle(0x%04x)\n",
+		    ids->handle);
+		goto out;
+	}
 	if (!ready)
 		sas_device_priv_data->deleted = 1;
+	/* not blocked if dnr_unblock_work released it before polling */
 	if (sas_device_priv_data->block)
 		_scsih_internal_device_unblock(sdev, sas_device_priv_data);
 	ret = true;

[ ... ]

Quoted 8 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +3 -1
@@ -4536,10 +4818,30 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
 			goto next;
 		if (sas_device_priv_data->sas_target->handle != handle)
 			goto next;
-		if (sas_device_priv_data->block)
+		deadline = (jiffies + _scsih_dnr_block_timeout(ioc,
+		    sas_device_priv_data->sas_target)) ?: 1;
+		if (sas_device_priv_data->block) {

drivers/scsi/mpt3sas/mpt3sas_scsih.c _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle) · line 4833

Quoted 10 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +9 -0
+			/* a repeat DNR restarts the firmware's missing delay */
+			if (sas_device_priv_data->block_deadline &&
+			    sas_device_priv_data->block_extends <
+			    MPT3SAS_DNR_BLOCK_MAX_EXTENDS) {
+				sas_device_priv_data->block_extends++;
+				sas_device_priv_data->block_deadline = deadline;
+				__scsih_dnr_unblock_work_by(ioc, deadline);
+			}
 			goto next;
+		}

[Severity: Medium]
Does _scsih_block_io_device() drop a new DELAY_NOT_RESPONDING event that arrives while the firmware event worker is polling a still-blocked sdev?

Before polling, _scsih_ublock_io_all_device(), _scsih_ublock_io_device_wait() and _scsih_claim_block() clear sas_device_priv_data->block_deadline to 0 while leaving sas_device_priv_data->block set to 1:

kworker <hardirq>
_scsih_ublock_io_device_wait()
block_deadline = 0
_scsih_claim_block()

    *block_seq = priv->block_seq
                                  _scsih_block_io_device()
                                    if (priv->block)
                                      if (priv->block_deadline && ...)
                                      goto next;

_scsih_end_poll()
if (priv->block_seq != block_seq) _scsih_internal_device_unblock() <-- unblocks or offlines during DNR

Because sas_device_priv_data->block is still 1 and block_deadline is 0, _scsih_block_io_device() neither increments block_seq nor sets a new block_deadline. _scsih_end_poll() then sees an unchanged block_seq and either unblocks the non-responding device to SDEV_RUNNING or marks it deleted and offlines it during the active device missing delay.

Also, in the second loop of _scsih_ublock_io_all_device() and _scsih_ublock_io_device_wait(), _scsih_claim_block() is called lazily per sdev. If an sdev whose previous block already expired (block == 0, dnr_expired == 1) receives a new DELAY_NOT_RESPONDING block while an earlier sdev is being polled, _scsih_claim_block() will claim that new block, clear its block_deadline and poll it immediately.

drivers/scsi/mpt3sas/mpt3sas_scsih.c _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle) · line 4844

Quoted 11 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +9 -0
 		if (sas_device && sas_device->pend_sas_rphy_add)
 			goto next;
+		/*
+		 * Expiry would bring it back running without a TUR check.
+		 * _scsih_ublock_io_device_wait() polls offline devices anyway.
+		 * A device that failed its TUR check is marked deleted under
+		 * block_lock, before it is set offline.
+		 */
+		if (sdev->sdev_state == SDEV_OFFLINE ||
+		    sas_device_priv_data->deleted)
+			goto next;

[Severity: Medium]
Can sas_device_priv_data->deleted remain set to 1 after an offlined sdev is returned to SDEV_RUNNING?

When a readiness check fails, _scsih_end_poll() sets sas_device_priv_data->deleted = 1 under ioc->block_lock before the caller sets sdev->sdev_state to SDEV_OFFLINE. The only place that clears sas_device_priv_data->deleted back to 0 is the first loop of _scsih_ublock_io_device_wait(), and only when sdev->sdev_state is still SDEV_OFFLINE.

If an administrator brings the offlined device back to SDEV_RUNNING via sysfs, sas_device_priv_data->deleted stays 1, so _scsih_block_io_device() and _scsih_block_io_all_device() will skip blocking that sdev on all future DELAY_NOT_RESPONDING and broadcast primitive events.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
#16 lore raw
In reply to Jeff Layton
Quoted 82 lines · Jeff Layton

I posted v2 some time ago. Since then, I did a lot of adversarial review with AI agents. The review found a large number of bugs. Many of these bugs were preexisting.

This series fixes a deadlock that occurs regularly in Meta's fleet. The fix unblocks sdevs after a timeout, and this breaks the deadlock.

I don't do much SCSI driver work, but this problem came to me. I relied heavily on AI agents for this series.

If this series is acceptable, I would like to see it in linux-next soon.

Thanks,
Jeff

---
Changes in v3:
- New patches 1-8 and 10 fix pre-existing bugs in the code that this series changes:
- 1-2: fix UAFs and refcounting in the cleanup of the fw event queue. Do not leave requeued events pending at teardown.
- 3: requeue topology events only for busy devices. Do not requeue them for devices that the driver cannot add.
- 4-7: fix these TUR polling bugs:
- a pcie_device leak
- a lost unblock retry
- polling that continues after the scan is cancelled
- a retry count that is not per device
- 8: cache the missing delays that the IOC actually uses
- 10: do not use freed hostdata while the driver polls devices
- Patch 9: also publish sdev->hostdata under ioc->block_lock in scsih_sdev_init()
- Patch 11:
- Use the PCIe missing delay (from PCIe IO Unit Page 1) for NVMe devices.
- Arm dnr_unblock_work for the earliest deadline.
- Skip the timed unblock during host reset, but not during SCSI EH.
- Clear the deadline of an sdev under ioc->block_lock before the fw event worker polls the sdev.
- Do not put a timed block on offline sdevs.
- When a hard reset cancels polling, give new deadlines to the sdevs that stay blocked.
- Give deadlines to the sdevs that a broadcast primitive pass blocks. Prevent dnr_unblock_work from unblocking them while the pass runs.
- A repeat DNR restarts the deadline, at most MPT3SAS_DNR_BLOCK_MAX_EXTENDS times.
- At the next worker unblock, do a TUR check on expired sdevs.
- Keep the blocks that the driver makes during polling.
- Use system_dfl_wq instead of system_wq.
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- New patch 1: use a new ioc->block_lock to serialize the
interrupt-time sdev block with the hostdata teardown in scsih_sdev_destroy(). This fixes a pre-existing UAF.
- Do the timed unblock under ioc->block_lock. This prevents a race with an interrupt-time block.
- Clear block_deadline before the fw event worker polls a returning device with TURs. Then dnr_unblock_work cannot unblock the device during the poll.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Jeff Layton (11):
scsi: mpt3sas: Don't leave requeued firmware events pending at teardown
scsi: mpt3sas: Hold event references while cleaning up the event queue
scsi: mpt3sas: Requeue topology events only for busy devices
scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()
scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()
scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled
scsi: mpt3sas: Reset the TUR retry count for each device
scsi: mpt3sas: Cache the missing delays the IOC actually uses
scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown
scsi: mpt3sas: Don't use freed hostdata while polling devices
scsi: mpt3sas: Time out delayed not-responding device blocks

 drivers/scsi/mpt3sas/mpt3sas_base.c  |  62 ++-
 drivers/scsi/mpt3sas/mpt3sas_base.h  |  25 +
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 869 +++++++++++++++++++++++++++++------
 3 files changed, 810 insertions(+), 146 deletions(-)
Quoted 5 lines

---
base-commit: e8017d29067fd1310a65c3494ce1a1b492adbac6
change-id: 20261030-sas-abba-eacfcc45fe36

Best regards,

My apologies, this should have been sent as v4 of the series. I had forgotten that I had sent a v3 a couple of weeks ago. Please treat this series as a v4. The v3 series is here:

https://lore.kernel.org/linux-scsi/[email protected]/

Signature · 1 line
Jeff Layton <[email protected]>