Skip to content

scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs

Jeff Layton linux-scsi lkml 4 versions – In review

Get series with b4
b4 am [email protected] change-id 20261030-sas-abba-eacfcc45fe36

Versions newest first

  1. v3 Latest posting 2/211 patches 1 replyIn review
  2. v3 posting 1/22 patches 1 replySuperseded
  3. v22 patches 1 replySuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v11 patch0 repliesSuperseded

Changes in v3 from the cover letter · 24 lines

- New patches 1-8 and 10 fix pre-existing bugs in the code that this series changes:
- 1-2: fix UAFs and refcounting in the cleanup of the fw event queue. Do not leave requeued events pending at teardown.
- 3: requeue topology events only for busy devices. Do not requeue them for devices that the driver cannot add.
- 4-7: fix these TUR polling bugs:
- a pcie_device leak
- a lost unblock retry
- polling that continues after the scan is cancelled
- a retry count that is not per device
- 8: cache the missing delays that the IOC actually uses
- 10: do not use freed hostdata while the driver polls devices
- Patch 9: also publish sdev->hostdata under ioc->block_lock in scsih_sdev_init()
- Patch 11:
- Use the PCIe missing delay (from PCIe IO Unit Page 1) for NVMe devices.
- Arm dnr_unblock_work for the earliest deadline.
- Skip the timed unblock during host reset, but not during SCSI EH.
- Clear the deadline of an sdev under ioc->block_lock before the fw event worker polls the sdev.
- Do not put a timed block on offline sdevs.
- When a hard reset cancels polling, give new deadlines to the sdevs that stay blocked.
- Give deadlines to the sdevs that a broadcast primitive pass blocks. Prevent dnr_unblock_work from unblocking them while the pass runs.
- A repeat DNR restarts the deadline, at most MPT3SAS_DNR_BLOCK_MAX_EXTENDS times.
- At the next worker unblock, do a TUR check on expired sdevs.
- Keep the blocks that the driver makes during polling.
- Use system_dfl_wq instead of system_wq.
- Link to v2: https://lore.kernel.org/r/[email protected]

Patches in v3 posting 2/2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11scsi: mpt3sas: Don't leave requeued firmware events pending at teardowndrivers/scsi/mpt3sas/mpt3sas_scsih.c+38 -110 replies
2/11scsi: mpt3sas: Hold event references while cleaning up the event queuedrivers/scsi/mpt3sas/mpt3sas_scsih.c+148 -630 replies
3/11scsi: mpt3sas: Requeue topology events only for busy devicesdrivers/scsi/mpt3sas/mpt3sas_scsih.c+16 -90 replies
4/11scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()drivers/scsi/mpt3sas/mpt3sas_scsih.c+2 -20 replies
5/11scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()drivers/scsi/mpt3sas/mpt3sas_scsih.c+1 -20 replies
6/11scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelleddrivers/scsi/mpt3sas/mpt3sas_scsih.c+23 -10 replies
7/11scsi: mpt3sas: Reset the TUR retry count for each devicedrivers/scsi/mpt3sas/mpt3sas_scsih.c+4 -20 replies
8/11scsi: mpt3sas: Cache the missing delays the IOC actually usesdrivers/scsi/mpt3sas/mpt3sas_base.c+31 -150 replies
9/11scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardowndrivers/scsi/mpt3sas/mpt3sas_base.h drivers/scsi/mpt3sas/mpt3sas_scsih.c+24 -80 replies
10/11scsi: mpt3sas: Don't use freed hostdata while polling devicesdrivers/scsi/mpt3sas/mpt3sas_base.h drivers/scsi/mpt3sas/mpt3sas_scsih.c+173 -400 replies
11/11scsi: mpt3sas: Time out delayed not-responding device blocksdrivers/scsi/mpt3sas/mpt3sas_base.c drivers/scsi/mpt3sas/mpt3sas_base.h drivers/scsi/mpt3sas/mpt3sas_scsih.c+369 -120 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs

I posted v2 some time ago. Since then, I did a lot of adversarial review with AI agents. The review found a large number of bugs. Many of these bugs were preexisting.

This series fixes a deadlock that occurs regularly in Meta's fleet. The fix unblocks sdevs after a timeout, and this breaks the deadlock.

I don't do much SCSI driver work, but this problem came to me. I relied heavily on AI agents for this series.

If this series is acceptable, I would like to see it in linux-next soon.

Thanks,
Jeff

---
Changes in v3:
- New patches 1-8 and 10 fix pre-existing bugs in the code that this series changes:
- 1-2: fix UAFs and refcounting in the cleanup of the fw event queue. Do not leave requeued events pending at teardown.
- 3: requeue topology events only for busy devices. Do not requeue them for devices that the driver cannot add.
- 4-7: fix these TUR polling bugs:
- a pcie_device leak
- a lost unblock retry
- polling that continues after the scan is cancelled
- a retry count that is not per device
- 8: cache the missing delays that the IOC actually uses
- 10: do not use freed hostdata while the driver polls devices
- Patch 9: also publish sdev->hostdata under ioc->block_lock in scsih_sdev_init()
- Patch 11:
- Use the PCIe missing delay (from PCIe IO Unit Page 1) for NVMe devices.
- Arm dnr_unblock_work for the earliest deadline.
- Skip the timed unblock during host reset, but not during SCSI EH.
- Clear the deadline of an sdev under ioc->block_lock before the fw event worker polls the sdev.
- Do not put a timed block on offline sdevs.
- When a hard reset cancels polling, give new deadlines to the sdevs that stay blocked.
- Give deadlines to the sdevs that a broadcast primitive pass blocks. Prevent dnr_unblock_work from unblocking them while the pass runs.
- A repeat DNR restarts the deadline, at most MPT3SAS_DNR_BLOCK_MAX_EXTENDS times.
- At the next worker unblock, do a TUR check on expired sdevs.
- Keep the blocks that the driver makes during polling.
- Use system_dfl_wq instead of system_wq.
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- New patch 1: use a new ioc->block_lock to serialize the
interrupt-time sdev block with the hostdata teardown in scsih_sdev_destroy(). This fixes a pre-existing UAF.
- Do the timed unblock under ioc->block_lock. This prevents a race with an interrupt-time block.
- Clear block_deadline before the fw event worker polls a returning device with TURs. Then dnr_unblock_work cannot unblock the device during the poll.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Jeff Layton (11):
scsi: mpt3sas: Don't leave requeued firmware events pending at teardown
scsi: mpt3sas: Hold event references while cleaning up the event queue
scsi: mpt3sas: Requeue topology events only for busy devices
scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()
scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()
scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled
scsi: mpt3sas: Reset the TUR retry count for each device
scsi: mpt3sas: Cache the missing delays the IOC actually uses
scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown
scsi: mpt3sas: Don't use freed hostdata while polling devices
scsi: mpt3sas: Time out delayed not-responding device blocks

 drivers/scsi/mpt3sas/mpt3sas_base.c  |  62 ++-
 drivers/scsi/mpt3sas/mpt3sas_base.h  |  25 +
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 869 +++++++++++++++++++++++++++++------
 3 files changed, 810 insertions(+), 146 deletions(-)

---
base-commit: e8017d29067fd1310a65c3494ce1a1b492adbac6
change-id: 20261030-sas-abba-eacfcc45fe36

Best regards,

Signature · 1 line
Jeff Layton <[email protected]>