iommu/arm-smmu-v3: Quarantine device upon ATC invalidation timeout

Hi all,

This series addresses a critical vulnerability and stability issue where an unresponsive PCIe device failing to process ATC (Address Translation Cache) invalidation requests leads to silent data corruption and continuous SMMU CMDQ error spam.

[ As Jason pointed out, because this series fundamentally introduces a new
RAS feature to quarantine and recover from hardware faults and relies on a recently accepted SMMU driver rework, it is not treated as a standard bug fix. Thus, most of the patches here don't carry a "Fixes" tag. ]

Currently, when an ATC invalidation times out, the SMMUv3 driver skips the CMDQ_ERR_CERROR_ATC_INV_IDX error. This leaves the device's ATS cache state desynchronized from the SMMU: the device cache may retain stale ATC entries for memory pages that the OS has already reclaimed and reassigned, creating a direct vector for data corruption. Furthermore, the driver might continue issuing ATC_INV commands, resulting in constant CMDQ errors:

    unexpected global error reported (0x00000001), this could be serious
    CMDQ error (cons 0x0302bb84): ATC invalidate timeout
    unexpected global error reported (0x00000001), this could be serious
    CMDQ error (cons 0x0302bb88): ATC invalidate timeout
    unexpected global error reported (0x00000001), this could be serious
    CMDQ error (cons 0x0302bb8c): ATC invalidate timeout
    ...

To resolve this, introduce a mechanism to quarantine a broken device in the SMMUv3 driver and the IOMMU core. To achieve this, add preparatory changes:
- Pass in PCI reset result to pci_dev_reset_iommu_done()
- Co-clear pending CMDQ_ERR from the cmdq issuer under a raw_spinlock_t, so an ATC_INV timeout flagged in cmdq->atc_sync_timeouts is definitive when the issuer reads its bit after CMD_SYNC poll

On the SMMUv3 driver side, retry the timedout ATC_INV batch to identify the faulty device(s). Perform a surgical STE update, and flag the ATS as broken to reject further ATS/ATC requests at HW level and suppress timeout spam.

This is on Github:
https://github.com/nicolinc/iommufd/commits/smmuv3_atc_timeout-v6

Changelog
v6:
* Rebase on v7.3-rc1
* Use "LLM" for the Assisted-by tags
* Add Reviewed-by tags from Baolu and Jason
* [PCI] Move the port register read to the probe path
* [PCI] Drop the patch propagating error code in quirks.c
* [iommu] Rename enum gdev_blocked to enum blocked_reason
* [iommu] Pass the correct old domain when releasing a blocked device
* [smmuv3] Drop the patch adding arm_smmu_cmdq_batch_force_sync (merged)
v5:
https://lore.kernel.org/all/[email protected]/
* Rebase on v7.2-rc1
* [PCI] Probe the underlying bus reset in cxl_reset_bus_function()
* [PCI] Add quirk_flr_err() to stop the reset cascade on FLR timeout
* [iommu] Drop iommu_report_device_broken() and its preparatory patches
* [smmuv3] Drop master->ats_broken bool
* [smmuv3] Drop master->ats_broken_lock
* [smmuv3] Drop master->ats_invs scratch
* [smmuv3] Introduce INV_TYPE_ATS_BROKEN marker
* [smmuv3] Add arm_smmu_cmdq_batch_force_sync()
* [smmuv3] Don't rb_erase() a never-inserted stream node
* [smmuv3] Add streams_lock for atomic SID->master lookup
* [smmuv3] Drop "Serialize STE.EATS and ats_broken updates"
* [smmuv3] Drop "Move arm_smmu_invs_for_each_entry to header"
* [smmuv3] Recheck CMDQ_ERR in tegra241_vintf0_handle_error()
* [smmuv3] Thread arm_smmu_master_domain on a per-master list
* [smmuv3] Drop pci_disable_ats() from arm_smmu_quarantine_ats()
* [smmuv3] Limit the quarantine() to ARM_SMMU_FEAT_COHERENCY only
* [smmuv3] Rework arm_smmu_quarantine_ats() to walk master_domains
* [smmuv3] Rework arm_smmu_cmdq_batch_retry(): per-unique-SID retry
* [smmuv3] Rework arm_smmu_inv_cmp() to treat ATS variants as one class
* [smmuv3] Rework the issuer-side atc_sync_timeouts test with smp_rmb()
* [smmuv3] Drop "Co-clear pending CMDQ_ERR when queue_has_space() fails"
* [smmuv3] Drop "Keep smmu pointer in arm_smmu_inv but add master for ATS"
v4:
https://lore.kernel.org/all/[email protected]/
* Rebase on Joerg's IOMMU "fixes" branch
* Rebase on Jason's SMMUv3 cmd_ent series
https://lore.kernel.org/all/[email protected]/
* [PCI] Don't suspend IOMMU in probe mode
* [iommu] kfree_rcu() iommu_group
* [iommu] Convert gdev->blocked to enum gdev_blocked
* [iommu] Use disable_work_sync() to fix UAF and ref leak
* [iommu] Gate done() transitions to preserve BLOCKED_BROKEN
* [iommu] Decrement recovery_cnt when unplugging a blocked gdev
* [iommu] Drop racy dev_has_iommu() in iommu_report_device_broken()
* [iommu] Add gdev->broken_pending to skip worker after racing recovery
* [smmuv3] Add master->ats_invs scratch
* [smmuv3] Add arm_smmu_cmdq_batch_issue() wrapper
* [smmuv3] Force per-flush sync for has_ats batches
* [smmuv3] Serialize STE.EATS and ats_broken updates
* [smmuv3] Co-clear pending CMDQ_ERR from cmdq issuer
* [smmuv3] Add invs and has_ats to arm_smmu_cmdq_batch
* [smmuv3] Move arm_smmu_invs_for_each_entry to header
* [smmuv3] Set master->ats_broken after clearing STE.EATS
* [smmuv3] Issue CFGI_STE via arm_smmu_cmdq_issue_cmd_with_sync()
* [smmuv3] Keep "smmu" pointer in arm_smmu_inv but add "master" for ATS
v3:
https://lore.kernel.org/all/[email protected]/
* Rebase on arm/smmu/updates branch + bug fix
* Update commit messages and inline comments
* [iommu] Drop unnecessary ops validation
* [iommu] Add missed function stub when !CONFIG_IOMMU_API
* [iommu] Change iommu_report_device_broken() to per gdev
* [iommu] Separate quarantine from pci_dev_reset_prepare()
* [iommu] Check reset failure in pci_dev_reset_iommu_done()
* [smmuv3] Fix STE update with try_cmpxchg64()
* [smmuv3] Fix "continue" bug when skipping ATC commands
* [smmuv3] Replace atomic_t prod_err with a lockless bitmap
* [smmuv3] Drop master->invs_domain; disable ATS per-master directly
* [smmuv3] Return -EIO for ATC timeout v.s. -ETIMEDOUT for poll timeout
* [smmuv3] Replace INV_TYPE_ATS_DISABLED with per-master ats_broken flag
v2:
https://lore.kernel.org/all/[email protected]/
* Rebase on arm_smmu_invs-v13 series
* Bisect batched atc invalidation commands
* Drop the direct pci_reset_function() call
* Move the work queue from SMMUv3 to the core
* Proceed a surgical STE update to disable EATS
* Wait for pci_dev_reset_iommu_done() to signal a recovery
v1:
https://lore.kernel.org/all/[email protected]/

Thanks
Nicolin

Nicolin Chen (17):
PCI: Don't suspend IOMMU when probing reset capability PCI/CXL: Probe the underlying bus reset in cxl_reset_bus_function()
iommu: Convert gdev->blocked from bool to enum blocked_reason
iommu: Pass in gdev's blocked state to iommu_deinit_device()
iommu: Pass in reset result to pci_dev_reset_iommu_done() iommu/arm-smmu-v3: Don't rb_erase() a never-inserted stream node iommu/arm-smmu-v3: Track ATC invalidation timeouts in a bitmap iommu/arm-smmu-v3: Skip remaining GERROR causes on SFM iommu/arm-smmu-v3: Introduce per-cmdq cmdq_err_handler callback iommu/arm-smmu-v3: Recheck CMDQ_ERR in tegra241_vintf0_handle_error() iommu/arm-smmu-v3: Co-clear pending CMDQ_ERR when CMD_SYNC times out iommu/arm-smmu-v3: Introduce arm_smmu_cmdq_batch_issue() wrapper iommu/arm-smmu-v3: Add streams_lock for atomic-context SID->master
lookup
iommu/arm-smmu-v3: Add has_ats to struct arm_smmu_cmdq_batch iommu/arm-smmu-v3: Add INV_TYPE_ATS_BROKEN for quarantined masters iommu/arm-smmu-v3: Thread arm_smmu_master_domain on a per-master list iommu/arm-smmu-v3: Quarantine ATS after an ATC invalidation timeout

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  33 +-
 include/linux/iommu.h                         |   5 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 477 ++++++++++++++++--
 .../iommu/arm/arm-smmu-v3/tegra241-cmdqv.c    |  41 +-
 drivers/iommu/iommu.c                         |  87 +++-
 drivers/pci/pci-acpi.c                        |   2 +-
 drivers/pci/pci.c                             |  37 +-
 drivers/pci/quirks.c                          |  13 +-
 8 files changed, 606 insertions(+), 89 deletions(-)

base-commit: cee9395acd8043be0644b25c34bfa86623f2b935

2.43.0