iommu/arm-smmu-v3: Fix device crash on kdump kernel

When transitioning to a kdump kernel, the primary kernel might have crashed while endpoint devices were actively bus-mastering DMA. Currently, the SMMU driver aggressively resets the hardware during probe by clearing CR0_SMMUEN and setting the Global Bypass Attribute (GBPA) to ABORT.

In a kdump scenario, this aggressive reset is highly destructive:
a) If GBPA is set to ABORT, in-flight DMA will be aborted, generating fatal PCIe AER or SErrors that may panic the kdump kernel

b) If GBPA is set to BYPASS, in-flight DMA targeting some IOVAs will bypass the SMMU and corrupt the physical memory at those 1:1 mapped IOVAs.

To safely absorb in-flight DMA, the kdump kernel must leave SMMUEN=1 intact and avoid modifying STRTAB_BASE. This allows HW to continue translating in- flight DMA using the crashed kernel's page tables until the endpoint device drivers probe and quiesce their respective hardware.

However, the ARM SMMUv3 architecture specification states that updating the SMMU_STRTAB_BASE register while SMMUEN == 1 is UNPREDICTABLE or ignored.

This leaves a kdump kernel no choice but to adopt the stream table from the crashed kernel.

In this series:
- Introduce an ARM_SMMU_OPT_KDUMP_ADOPT
- Skip SMMUEN and STRTAB_BASE resets in arm_smmu_device_reset()
- Skip EVENTQ/PRIQ setup including interrupts and their handlers
- Memremap the crashed kernel's stream tables into the kdump kernel [*]
- Defer any default domain attachment to retain STEs until device drivers explicitly request it.

[*] For verification reasons, this series only fixes coherent SMMUs.

For non-ARM_SMMU_OPT_KDUMP_ADOPT cases, keep a status quo since the commit 3f54c447df34f ("iommu/arm-smmu-v3: Don't disable SMMU in kdump kernel"):
full reset followed by driver-initiated reattach, potentially rejecting any in-flight DMA.

Note that the series requires Jason's work that was merged in v6.12: commit 85196f54743d ("iommu/arm-smmu-v3: Reorganize struct arm_smmu_strtab_cfg"). I have a backported version that is verified with a v6.8 kernel. I can send if we see a strong need after this version is accepted.

This is on Github:
https://github.com/nicolinc/iommufd/commits/smmuv3_kdump-v7

Changelog
v7
* Rebase v7.2-rc1
* Add Reviewed-by from Pranjal
* Reword the linear stream table adoption comment
* Use dev_dbg for the stream table adoption message
* Document why the lazy L2 adoption uses devm_memremap()
* Drop redundant FEAT_COHERENCY checks in the adopt functions
* Use feature bit instead of STRTAB_BASE_CFG in adopt cleanup
* Skip CR0_ATSCHK update in adopt mode to retain the crashed policy
* Restore FEAT_2_LVL_STRTAB if the cleanup action fails to register
v6
https://lore.kernel.org/all/[email protected]/
* Rebase v7.1-rc3
* Add Reviewed-by from Jason
* Replace dma_addr_t with phys_addr_t
* Drop arm_smmu_kdump_phys_is_corrupted()
* Skip threaded IRQ handlers for EVTQ and PRIQ
* Bypass arm_smmu_rmr_install_bypass_ste() in kdump case
* Drop devm_ for adopt-time allocations; set up cleanup function via devm_add_action_or_reset()
v5
https://lore.kernel.org/all/[email protected]/
* Add Reviewed-by from Kevin
* Drop READ_ONCE on lazy-attach L1 read
* Split "Skip EVTQ/PRIQ setup" into two patches
* Tighten kdump probe comment and dev_warn message
* Use MEM + BUSY in arm_smmu_kdump_phys_is_corrupted
v4
https://lore.kernel.org/all/[email protected]/
* Rebase v7.1-rc1
* s/arm_smmu_adopt/arm_smmu_kdump_adopt
* Revert alloc/memremap/fmt on fallback
* Reorder patches to avoid bisect regression
* Use IRQ_NONE for spurious evtq/priq entries
* Cap linear log2size by kdump's allocation bound
* Defer clearing FEAT_2_LVL_STRTAB on linear adopt
* Add arm_smmu_kdump_phys_is_corrupted() validation
* Defer l2 stream table memremap till master inserts
* Re-validate L1 desc on master insert with READ_ONCE
v3
https://lore.kernel.org/all/[email protected]/
* s/OPT_KDUMP/OPT_KDUMP_ADOPT
* Do not adopt if GERROR_SFM_ERR
* Retain CR0_ATSCHK beside CR0_SMMUEN
* Clear latched GERROR bits (e.g. CMDQ_ERR)
* Assert ARM_SMMU_FEAT_COHERENCY in adopt functions
* Add STE.Cfg check in arm_smmu_is_attach_deferred()
* Fix validations on return codes from devm_memremap()
* Sanitize crashed kernel register values in adopt functions
* Drop unnecessary l2ptrs guard in arm_smmu_is_attach_deferred()
* Don't enable PRIQ/EVTQ irqs and guard the irq functions for combined irq cases
v2
https://lore.kernel.org/all/[email protected]/
* Add warning in non-coherent SMMU cases
* Keep eventq/priq disabled vs. enabling-and-disabling-later
* Check KDUMP option in the beginning of arm_smmu_device_reset()
* Validate STRTAB format matches HW capability instead of forcing flags
v1:
https://lore.kernel.org/all/[email protected]/

Nicolin Chen (7):
iommu/arm-smmu-v3: Add arm_smmu_kdump_adopt_strtab() for kdump iommu/arm-smmu-v3: Implement is_attach_deferred() for kdump iommu/arm-smmu-v3: Do not enable EVTQ/PRIQ interrupts in kdump kernel iommu/arm-smmu-v3: Skip EVTQ/PRIQ setup in kdump kernel iommu/arm-smmu-v3: Retain CR0_SMMUEN during kdump device reset iommu/arm-smmu-v3: Skip RMR bypass for kdump adoption iommu/arm-smmu-v3: Detect ARM_SMMU_OPT_KDUMP_ADOPT in probe()

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |   1 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 467 ++++++++++++++++++--
 2 files changed, 422 insertions(+), 46 deletions(-)

Signature · 1 line

2.43.0

Quoted 8 lines · Nicolin Chen

When transitioning to a kdump kernel, the primary kernel might have crashed while endpoint devices were actively bus-mastering DMA. Currently, the SMMU driver aggressively resets the hardware during probe by clearing CR0_SMMUEN and setting the Global Bypass Attribute (GBPA) to ABORT.

In a kdump scenario, this aggressive reset is highly destructive:
a) If GBPA is set to ABORT, in-flight DMA will be aborted, generating fatal PCIe AER or SErrors that may panic the kdump kernel

Can you please clarify more on those errors, what conditions will trigger that?
For example, patch 4 disables the EVTQ to avoid events as there might be a lot, why are they not fatal also?

Quoted 8 lines

b) If GBPA is set to BYPASS, in-flight DMA targeting some IOVAs will bypass the SMMU and corrupt the physical memory at those 1:1 mapped IOVAs.

To safely absorb in-flight DMA, the kdump kernel must leave SMMUEN=1 intact and avoid modifying STRTAB_BASE. This allows HW to continue translating in- flight DMA using the crashed kernel's page tables until the endpoint device drivers probe and quiesce their respective hardware.

Quoted 5 lines

However, the ARM SMMUv3 architecture specification states that updating the SMMU_STRTAB_BASE register while SMMUEN == 1 is UNPREDICTABLE or ignored.

This leaves a kdump kernel no choice but to adopt the stream table from the crashed kernel.

In many cases the patches assume that the CDs/STE might be corrupted, but still attempt to retrieve them with some validation (log2size/split...)
However, the base address might be broken, TLBs state is unknown...

IMO, although that might improve the status quo, there are still heuristics, in addition to noticeable complexity to transition the stream tables. I wonder if FW can deal with AER in that case before booting the kdump kernel.

Quoted 34 lines · Mostafa Saleh

Quoted 8 lines · Nicolin Chen

When transitioning to a kdump kernel, the primary kernel might have crashed while endpoint devices were actively bus-mastering DMA. Currently, the SMMU driver aggressively resets the hardware during probe by clearing CR0_SMMUEN and setting the Global Bypass Attribute (GBPA) to ABORT.

In a kdump scenario, this aggressive reset is highly destructive:
a) If GBPA is set to ABORT, in-flight DMA will be aborted, generating fatal PCIe AER or SErrors that may panic the kdump kernel

Can you please clarify more on those errors, what conditions will trigger that?
For example, patch 4 disables the EVTQ to avoid events as there might be a lot, why are they not fatal also?

Quoted 8 lines

b) If GBPA is set to BYPASS, in-flight DMA targeting some IOVAs will bypass the SMMU and corrupt the physical memory at those 1:1 mapped IOVAs.

To safely absorb in-flight DMA, the kdump kernel must leave SMMUEN=1 intact and avoid modifying STRTAB_BASE. This allows HW to continue translating in- flight DMA using the crashed kernel's page tables until the endpoint device drivers probe and quiesce their respective hardware.

Quoted 5 lines

However, the ARM SMMUv3 architecture specification states that updating the SMMU_STRTAB_BASE register while SMMUEN == 1 is UNPREDICTABLE or ignored.

This leaves a kdump kernel no choice but to adopt the stream table from the crashed kernel.

In many cases the patches assume that the CDs/STE might be corrupted, but still attempt to retrieve them with some validation (log2size/split...)
However, the base address might be broken, TLBs state is unknown...

Quoted 4 lines

IMO, although that might improve the status quo, there are still heuristics, in addition to noticeable complexity to transition the stream tables. I wonder if FW can deal with AER in that case before booting the kdump kernel.

I guess we're reading the base address from the HW register itself so that should be fine? CDs are in-memory so that's why they could be corrupted?

About the TLB state, I'm not sure what might pollute it, since this is a kexec, I don't expect any non-kernel entity to gain program control before the kdump kernel.. Hence, IMO, we can't configure FW to deal with AER here..

Thanks,
Praan

Quoted 35 lines · Pranjal Shrivastava

Quoted 34 lines · Mostafa Saleh

Quoted 8 lines · Nicolin Chen

When transitioning to a kdump kernel, the primary kernel might have crashed while endpoint devices were actively bus-mastering DMA. Currently, the SMMU driver aggressively resets the hardware during probe by clearing CR0_SMMUEN and setting the Global Bypass Attribute (GBPA) to ABORT.

In a kdump scenario, this aggressive reset is highly destructive:
a) If GBPA is set to ABORT, in-flight DMA will be aborted, generating fatal PCIe AER or SErrors that may panic the kdump kernel

Can you please clarify more on those errors, what conditions will trigger that?
For example, patch 4 disables the EVTQ to avoid events as there might be a lot, why are they not fatal also?

Quoted 8 lines

b) If GBPA is set to BYPASS, in-flight DMA targeting some IOVAs will bypass the SMMU and corrupt the physical memory at those 1:1 mapped IOVAs.

To safely absorb in-flight DMA, the kdump kernel must leave SMMUEN=1 intact and avoid modifying STRTAB_BASE. This allows HW to continue translating in- flight DMA using the crashed kernel's page tables until the endpoint device drivers probe and quiesce their respective hardware.

Quoted 5 lines

However, the ARM SMMUv3 architecture specification states that updating the SMMU_STRTAB_BASE register while SMMUEN == 1 is UNPREDICTABLE or ignored.

This leaves a kdump kernel no choice but to adopt the stream table from the crashed kernel.

In many cases the patches assume that the CDs/STE might be corrupted, but still attempt to retrieve them with some validation (log2size/split...)
However, the base address might be broken, TLBs state is unknown...

Quoted 8 lines

Quoted 4 lines

IMO, although that might improve the status quo, there are still heuristics, in addition to noticeable complexity to transition the stream tables. I wonder if FW can deal with AER in that case before booting the kdump kernel.

I guess we're reading the base address from the HW register itself so that should be fine? CDs are in-memory so that's why they could be corrupted?

For example patch#1 verifies log2size and split and both are read from HW registers. Same for the base address or other addresses as the page tables, they might be corrupted due to a buggy driver. My point is that, it is really hard to assume that the previous state of registers/STE/page-tables were valid or even consistent, when the kernel crashed and did not transition the state gracefully.

Quoted 5 lines

About the TLB state, I'm not sure what might pollute it, since this is a kexec, I don't expect any non-kernel entity to gain program control before the kdump kernel.. Hence, IMO, we can't configure FW to deal with AER here..

Similarly for TLBs, the kernel might have panicked in the middle of an unmap or free domain. (not to mention what that means for RPM where a device reset with unknown TLBs?

Why can't the FW deal with it?

As I mentioned above in the previous
reply I am not sure I understand what situation leads into this, when does a device trigger SError to the system vs when not which is observed as an event in that case.

Quoted 4 lines

As I mentioned above in the previous
reply I am not sure I understand what situation leads into this, when does a device trigger SError to the system vs when not which is observed as an event in that case.

Ack. I see what you mean now.. How does a DMA fault raise an SError?

Quoted 5 lines · Pranjal Shrivastava

Quoted 4 lines

As I mentioned above in the previous
reply I am not sure I understand what situation leads into this, when does a device trigger SError to the system vs when not which is observed as an event in that case.

Quoted 1 line

Ack. I see what you mean now.. How does a DMA fault raise an SError?

As I gave an example to Robin if the unhandled failure escalates into RAS emergency unplugging CXL memory then the system is going to explode when kdump touches that CXL memory as part of the dumping. It is not quite so simple that a DMA abort is triggering SError.

I don't know exactly the sequence of events that lead up to the kdump kernel crashing (I imagine it is hard to debug that one), but it is something related to the new kernel not participating in the RAS and the RAS flow escalating to something fatal.

Quoted 4 lines

I don't know exactly the sequence of events that lead up to the kdump kernel crashing (I imagine it is hard to debug that one), but it is something related to the new kernel not participating in the RAS and the RAS flow escalating to something fatal.

Here is the original bug report:
- kernel boots into a crash kernel
- crash kernel hits OOM do to insufficient reserved memory and panics
- PCIe errors are observed during this failure flow

For example patch#1 verifies log2size and split and both are read from HW registers. Same for the base address or other addresses as the page tables, they might be corrupted due to a buggy driver. My point is that, it is really hard to assume that the previous state of registers/STE/page-tables were valid or even consistent, when the kernel crashed and did not transition the state gracefully.

Similarly for TLBs, the kernel might have panicked in the middle of an unmap or free domain. (not to mention what that means for RPM where a device reset with unknown TLBs)

Quoted 6 lines · Mostafa Saleh

For example patch#1 verifies log2size and split and both are read from HW registers. Same for the base address or other addresses as the page tables, they might be corrupted due to a buggy driver. My point is that, it is really hard to assume that the previous state of registers/STE/page-tables were valid or even consistent, when the kernel crashed and did not transition the state gracefully.

Sure, and this mechanism is probably not very useful for debugging these kinds of errors in the SMMU driver. Oh well, that isn't a common source of kernel crashes :)

Quoted 4 lines

Similarly for TLBs, the kernel might have panicked in the middle of an unmap or free domain. (not to mention what that means for RPM where a device reset with unknown TLBs)

TLB is fine. kdump works by carving out a chunk of memory for the future crash kernel. When the kernel boots it ignores all the memory used by the prior kernel. So DMA can keep running into the old kernels memory with no issue. It doesn't matter if the TLBs are inconsistent or not.

Jason

Quoted 9 lines · Jason Gunthorpe

Quoted 6 lines · Mostafa Saleh

For example patch#1 verifies log2size and split and both are read from HW registers. Same for the base address or other addresses as the page tables, they might be corrupted due to a buggy driver. My point is that, it is really hard to assume that the previous state of registers/STE/page-tables were valid or even consistent, when the kernel crashed and did not transition the state gracefully.

Quoted 3 lines

Sure, and this mechanism is probably not very useful for debugging these kinds of errors in the SMMU driver. Oh well, that isn't a common source of kernel crashes :)

I hope not! Although memory corruption can happen due to many other reasons :/

I am not trying to bikeshed, but I wondering if there is a more reliable way rather than doing archaeology from a panicked kernel SMMUv3 configuration, as I am worried that will be even harder to debug if it goes wrong.

Quoted 10 lines

Quoted 4 lines

Similarly for TLBs, the kernel might have panicked in the middle of an unmap or free domain. (not to mention what that means for RPM where a device reset with unknown TLBs)

TLB is fine. kdump works by carving out a chunk of memory for the future crash kernel. When the kernel boots it ignores all the memory used by the prior kernel. So DMA can keep running into the old kernels memory with no issue. It doesn't matter if the TLBs are inconsistent or not.

Ideally if a TLB is to be missed (because of the panic), it should not point to kdump memory as it is carved-out. However, it is still a leap to assume that the TLBs are in a good shape as I mentioned with RPM (or even if the device resets transiently for some reason) it can end up with garbage in its TLBs.

However, I agree with the overall problem, i.e. IF an active device unmaps the DMA addr after the transaction in the previous kernel, (with the SMMU powered ON) but the TLBI was missed due to a crash/panic, Any new DMA in the new kernel may alias onto a memory in the previous (crashed) kernel, not the kdump kernel.

Quoted 5 lines · Pranjal Shrivastava

However, I agree with the overall problem, i.e. IF an active device unmaps the DMA addr after the transaction in the previous kernel, (with the SMMU powered ON) but the TLBI was missed due to a crash/panic, Any new DMA in the new kernel may alias onto a memory in the previous (crashed) kernel, not the kdump kernel.

It looks like there is an issue in this series, it isn't doing anything with the VMIDs.

The VMIDs that are in-used by the adopted stream table have to be removed from the idr as well (and similarly for ASID if we don't have VMID HW support).

Then the VMIDs that may be dirtied by the prior kernel remain isolated and are never re-used by the new kernel. When the new kernel wants to do DMA it will replace the STE with a new, clean VMID, and there is no problem.

Quoted 15 lines · Jason Gunthorpe

Quoted 5 lines · Pranjal Shrivastava

However, I agree with the overall problem, i.e. IF an active device unmaps the DMA addr after the transaction in the previous kernel, (with the SMMU powered ON) but the TLBI was missed due to a crash/panic, Any new DMA in the new kernel may alias onto a memory in the previous (crashed) kernel, not the kdump kernel.

It looks like there is an issue in this series, it isn't doing anything with the VMIDs.

The VMIDs that are in-used by the adopted stream table have to be removed from the idr as well (and similarly for ASID if we don't have VMID HW support).

Quoted 4 lines

Then the VMIDs that may be dirtied by the prior kernel remain isolated and are never re-used by the new kernel. When the new kernel wants to do DMA it will replace the STE with a new, clean VMID, and there is no problem.

I see. I assume the reserved VMID for the kdump kernel will be a clean VMID (!=0). That should guarantee different cache tags.

But, do we have to scan CDs for ASID? I wonder if we could limit to ARM_SMMU_FEAT_TRANS_S2 only, as this series does not memremap CDs at all..

Quoted 10 lines · Nicolin Chen

Quoted 8 lines · Jason Gunthorpe

The VMIDs that are in-used by the adopted stream table have to be removed from the idr as well (and similarly for ASID if we don't have VMID HW support).

Then the VMIDs that may be dirtied by the prior kernel remain isolated and are never re-used by the new kernel. When the new kernel wants to do DMA it will replace the STE with a new, clean VMID, and there is no problem.

Quoted 2 lines

I see. I assume the reserved VMID for the kdump kernel will be a clean VMID (!=0). That should guarantee different cache tags.

You will also have to change things to allocate the kernel global vmid from the IDR, it will usually be 0 but not for kdump. Then you have to find all the places where the 0 is implicitly placed and put in the actual value.

Quoted 14 lines · Jason Gunthorpe

Quoted 12 lines · Nicolin Chen

Quoted 8 lines · Jason Gunthorpe

The VMIDs that are in-used by the adopted stream table have to be removed from the idr as well (and similarly for ASID if we don't have VMID HW support).

Then the VMIDs that may be dirtied by the prior kernel remain isolated and are never re-used by the new kernel. When the new kernel wants to do DMA it will replace the STE with a new, clean VMID, and there is no problem.

I see. I assume the reserved VMID for the kdump kernel will be a clean VMID (!=0). That should guarantee different cache tags.

Quoted 4 lines

You will also have to change things to allocate the kernel global vmid from the IDR, it will usually be 0 but not for kdump. Then you have to find all the places where the 0 is implicitly placed and put in the actual value.

Hmm, I just realized that all the EL2 commands do not take VMID.

Quoted 20 lines · Nicolin Chen

Quoted 18 lines · Jason Gunthorpe

Quoted 5 lines · Nicolin Chen

Quoted 4 lines · Jason Gunthorpe

The VMIDs that are in-used by the adopted stream table have to be removed from the idr as well (and similarly for ASID if we don't have VMID HW support).

Quoted 7 lines

Quoted 4 lines

Then the VMIDs that may be dirtied by the prior kernel remain isolated and are never re-used by the new kernel. When the new kernel wants to do DMA it will replace the STE with a new, clean VMID, and there is no problem.

I see. I assume the reserved VMID for the kdump kernel will be a clean VMID (!=0). That should guarantee different cache tags.

You will also have to change things to allocate the kernel global vmid from the IDR, it will usually be 0 but not for kdump. Then you have to find all the places where the 0 is implicitly placed and put in the actual value.

Quoted 1 line

Hmm, I just realized that all the EL2 commands do not take VMID.

If the EL2 commands are being used then there is no S2 support and no VMID support.

So if you disable kdump when S2 is not supported it also disables it when EL2 would be used, which effectively means it is not supported in a VM.

Quoted 3 lines

So if you disable kdump when S2 is not supported it also disables it when EL2 would be used, which effectively means it is not supported in a VM.

Hmm, they are actually not exclusive: e.g. Grace has both.

EL2 is from FEAT_E2H, which is set when
1. IDR0.HYP=1
2. cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)

VMID is from FEAT_TRANS_S2, which is set When
1. IDR0_S2P=1

So, host-level stage-1 TLBIs use EL2 commands (no VMID); guest-level TLBIs use NH_ commands (with VMID).

In many cases the patches assume that the CDs/STE might be corrupted, but still attempt to retrieve them with some validation (log2size/split...)
However, the base address might be broken, TLBs state is unknown...

IMO, although that might improve the status quo, there are still heuristics, in addition to noticeable complexity to transition the stream tables.

For example, patch 4 disables the EVTQ to avoid events as there might be a lot, why are they not fatal also?

(I think Jason has answered most of the questions here.)

Quoted 2 lines · Mostafa Saleh

For example, patch 4 disables the EVTQ to avoid events as there might be a lot, why are they not fatal also?

FWIW, the PATCH-4 doesn't disable the EVTQ: EVTQ is disabled in kdump case prior to the series; PATCH-4 just makes sure it won't get enabled transiently.

Nicolin