iommu/arm-smmu-v3: Adopt the crashed kernel's stream table for kdump

When transitioning to a kdump kernel, the primary kernel might have crashed while endpoint devices were actively bus-mastering DMA. Currently, the SMMU driver aggressively resets the hardware during probe by clearing CR0_SMMUEN and setting the Global Bypass Attribute (GBPA) to ABORT.

In a kdump scenario, this aggressive reset is highly destructive:
a) If GBPA is set to ABORT, in-flight DMA will be aborted, generating fatal PCIe AER or SErrors that may panic the kdump kernel
b) If GBPA is set to BYPASS, in-flight DMA targeting some IOVAs will bypass the SMMU and corrupt the physical memory at those 1:1 mapped IOVAs.

To safely absorb in-flight DMA, the kdump kernel must leave SMMUEN=1 intact and avoid modifying STRTAB_BASE. This allows HW to continue translating in- flight DMA using the crashed kernel's page tables until the endpoint device drivers probe and quiesce their respective hardware.

However, the ARM SMMUv3 architecture specification states that updating the SMMU_STRTAB_BASE register while SMMUEN == 1 is UNPREDICTABLE or ignored.

This leaves a kdump kernel no choice but to adopt the stream table from the crashed kernel.

In this series:
- Introduce an ARM_SMMU_OPT_KDUMP_ADOPT
- Skip SMMUEN and STRTAB_BASE resets in arm_smmu_device_reset()
- Skip EVENTQ/PRIQ setup including interrupts and their handlers
- Memremap the crashed kernel's stream tables into the kdump kernel [*]
- Reserve the crashed kernel's in-use ASIDs and VMIDs, preventing any TLB aliasing with the kdump kernel's own domains
- Defer any default domain attachment to retain STEs until device drivers explicitly request it.
Most of the new code is in arm-smmu-v3-kexec.c. A hidden ARM_SMMU_V3_KEXEC config symbol (def_bool CRASH_DUMP) controls its build.

The common helpers parse and walk the crashed kernel's stream/CD tables and reserve its in-use IDs. The kdump code is guarded by CONFIG_CRASH_DUMP. The helpers can also support the proposed SMMUv3 Live Update work:
https://lore.kernel.org/all/[email protected]/

[*] For verification reasons, this series only fixes coherent SMMUs.

For non-ARM_SMMU_OPT_KDUMP_ADOPT cases, keep a status quo since the commit 3f54c447df34f ("iommu/arm-smmu-v3: Don't disable SMMU in kdump kernel"):
full reset followed by driver-initiated reattach, potentially rejecting any in-flight DMA.

Note that this series is no longer treated as a bug fix, since it has grown fairly big and most of the kdump code now resides in a separate C file. For folks interested in back-porting the change: a v6.12+ kernel (since commit 85196f54743d ("iommu/arm-smmu-v3: Reorganize struct arm_smmu_strtab_cfg")) would be still compatible with this series.

Patches here are assisted by LLM. v7.4-rc1 removed the requirement for the per-patch tag. So, adding a side note here.

This is on Github:
https://github.com/nicolinc/iommufd/commits/smmuv3_kdump-v12

Changelog
v12
* Rebase on v7.4-rc1
* Add Tested-by from Shuai
* Drop "Assisted-by: LLM" tags
* Add Reviewed-by from Jason and Mostafa
* Add a core patch to fix old_domain for attach_deferred
* Check IDR0.ST_LVL before adopting a 2-level stream table
* Look up MSI index instead of swapping EVTQ/GERROR indexes
* Reuse the L1 descriptor check for the deferred L2 adoption
v11
https://lore.kernel.org/linux-iommu/[email protected]/
* Rebase on arm/smmu/updates branch
* Add Tested-by from Breno and Cristian
* Add Reviewed/Acked-by from Jason and Kiryl
* Update commit messages and Assisted-by tags
* New prep patch: swap EVTQ/GERROR MSI indexes
* Skip EVTQ's CFG0 write and vector allocation
* Merge arm-smmu-v3-kdump code into arm-smmu-v3-kexec
* Fold patches accordingly as static functions require callers
v10
https://lore.kernel.org/linux-iommu/[email protected]/
* Rebase on v7.3-rc1
* Bound the 2-level entry count before the shift
* Drop the vmid_map devres action that landed upstream
* Validate the CD table base alignments during the scan
* New prep patch: make the ASID space per SMMU instance
* Gate the EVTQ/PRIQ on feature bits, so kdump skips both entirely
v9
https://lore.kernel.org/all/[email protected]/
* Reject valid CD L1 descriptors carrying a null L2 pointer
* Move the ida devres prep before the stream table adoption
* Reject a 2-level CD table on hardware without FEAT_2_LVL_CDTAB
* Cap the linear table log2size by sid_bits on 2-level capable HW
* Add a hidden ARM_SMMU_V3_KEXEC config for Live Update to extend
* Factor the table walkers and ID reservation into arm-smmu-v3-kexec.c
v8
https://lore.kernel.org/all/[email protected]/
* Move the kdump code into a new arm-smmu-v3-kdump.c
* Move the EVTQ/PRIQ patches to the front of the series
* Prefix "kdump: " to prints via dev_fmt in the new file
* Add a prep patch destroying the vmid_map ida via devres
* Reject valid-span L1 descriptors with a null L2 pointer
* Document notes/limitations at the top of arm-smmu-v3-kdump.c
* Rename arm_smmu_kdump_adopt_l2_strtab() to a deferred variant
* Add a new patch reserving the crashed kernel's ASIDs and VMIDs
* Make arm_smmu_get_step_for_sid() a static inline in the header
* Validate alignments of the adopted stream table base addresses
* Retarget to the merge window; drop the Fixes and Cc-stable tags
* Rename the kdump probe function to arm_smmu_device_kdump_probe()
* Document that a disabled event queue discards new events silently
* Add a common arm_smmu_is_attach_deferred() calling a kdump helper
* Document that acking SFM_ERR is defined but does not exit the SFM
* Document that CR0 queue enables can be cleared while SMMUEN is set
* Clear only the CR0 queue enables in kdump reset, keeping other fields
v7
https://lore.kernel.org/all/[email protected]/
* Rebase v7.2-rc1
* Add Reviewed-by from Pranjal
* Reword the linear stream table adoption comment
* Use dev_dbg for the stream table adoption message
* Document why the lazy L2 adoption uses devm_memremap()
* Drop redundant FEAT_COHERENCY checks in the adopt functions
* Use feature bit instead of STRTAB_BASE_CFG in adopt cleanup
* Skip CR0_ATSCHK update in adopt mode to retain the crashed policy
* Restore FEAT_2_LVL_STRTAB if the cleanup action fails to register
v6
https://lore.kernel.org/all/[email protected]/
* Rebase v7.1-rc3
* Add Reviewed-by from Jason
* Replace dma_addr_t with phys_addr_t
* Drop arm_smmu_kdump_phys_is_corrupted()
* Skip threaded IRQ handlers for EVTQ and PRIQ
* Bypass arm_smmu_rmr_install_bypass_ste() in kdump case
* Drop devm_ for adopt-time allocations; set up cleanup function via devm_add_action_or_reset()
v5
https://lore.kernel.org/all/[email protected]/
* Add Reviewed-by from Kevin
* Drop READ_ONCE on lazy-attach L1 read
* Split "Skip EVTQ/PRIQ setup" into two patches
* Tighten kdump probe comment and dev_warn message
* Use MEM + BUSY in arm_smmu_kdump_phys_is_corrupted
v4
https://lore.kernel.org/all/[email protected]/
* Rebase v7.1-rc1
* s/arm_smmu_adopt/arm_smmu_kdump_adopt
* Revert alloc/memremap/fmt on fallback
* Reorder patches to avoid bisect regression
* Use IRQ_NONE for spurious evtq/priq entries
* Cap linear log2size by kdump's allocation bound
* Defer clearing FEAT_2_LVL_STRTAB on linear adopt
* Add arm_smmu_kdump_phys_is_corrupted() validation
* Defer l2 stream table memremap till master inserts
* Re-validate L1 desc on master insert with READ_ONCE
v3
https://lore.kernel.org/all/[email protected]/
* s/OPT_KDUMP/OPT_KDUMP_ADOPT
* Do not adopt if GERROR_SFM_ERR
* Retain CR0_ATSCHK beside CR0_SMMUEN
* Clear latched GERROR bits (e.g. CMDQ_ERR)
* Assert ARM_SMMU_FEAT_COHERENCY in adopt functions
* Add STE.Cfg check in arm_smmu_is_attach_deferred()
* Fix validations on return codes from devm_memremap()
* Sanitize crashed kernel register values in adopt functions
* Drop unnecessary l2ptrs guard in arm_smmu_is_attach_deferred()
* Don't enable PRIQ/EVTQ irqs and guard the irq functions for combined irq cases
v2
https://lore.kernel.org/all/[email protected]/
* Add warning in non-coherent SMMU cases
* Keep eventq/priq disabled vs. enabling-and-disabling-later
* Check KDUMP option in the beginning of arm_smmu_device_reset()
* Validate STRTAB format matches HW capability instead of forcing flags
v1:
https://lore.kernel.org/all/[email protected]/

Nicolin Chen (12):
iommu: Pass in NULL old_domain when attaching a deferred device iommu/arm-smmu-v3: Init the vmid_map ida before the stream table setup iommu/arm-smmu-v3: Make the ASID space per SMMU instance iommu/arm-smmu-v3: Pack the MSI vectors of the used sources iommu/arm-smmu-v3: Add ARM_SMMU_FEAT_EVTQ for the event queue iommu/arm-smmu-v3: Disable the EVTQ and the PRIQ in a kdump kernel iommu/arm-smmu-v3: Add ARM_SMMU_OPT_KDUMP_ADOPT for kdump kernel iommu/arm-smmu-v3-kexec: Reserve crashed kernel's ASIDs and VMIDs iommu/arm-smmu-v3-kexec: Implement is_attach_deferred() iommu/arm-smmu-v3: Retain CR0_SMMUEN during kdump device reset iommu/arm-smmu-v3: Skip RMR bypass for kdump adoption iommu/arm-smmu-v3: Detect ARM_SMMU_OPT_KDUMP_ADOPT in probe()

 drivers/iommu/arm/Kconfig                     |   3 +
 drivers/iommu/arm/arm-smmu-v3/Makefile        |   1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  61 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kexec.c | 735 ++++++++++++++++++
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |   6 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 287 +++++--
 drivers/iommu/iommu.c                         |   9 +-
 7 files changed, 1011 insertions(+), 91 deletions(-)
 create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kexec.c

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4

2.43.0