iommu/vt-d: Prevent boot failure with devices requiring ATS
Subsystems (3)
Lu Baolu
maintainerPATCH 1/1 iommu/vt-d: Prevent boot failure with devices requiring ATS
SOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the ATC_REQUIRED bit set in the Flags field.
The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS can cause a conflict, leading to boot failure, especially if the device is a graphics device.
To prevent this issue, check PCI ATS support before enumerating the IOMMU devices. If any device requires PCI ATS, but PCI ATS is disabled by 'pci=noats', switch the IOMMU to operate in legacy mode to ensure successful booting.
--- drivers/iommu/intel/iommu.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.c +19 -3
@@ -3127,10 +3127,26 @@ int dmar_iommu_notify_scope_dev(struct dmar_pci_notify_info *info) (void *)satc + satc->header.length, satc->segment, satcu->devices, satcu->devices_cnt); -if (ret >0 ) -break ; -else if (ret <0 ) +if (ret <0 )return ret; + +if (ret >0 ) { +/* +* The device requires PCI/ATS when the IOMMU +* works in the scalable mode. If PCI/ATS is +* disabled using the pci=noats kernel parameter, +* the IOMMU will default to legacy mode. Users +* are informed of this change. +*/ +if (intel_iommu_sm && satcu->atc_required && + !pci_ats_supported (info->dev)) { +pci_warn (info->dev, +"PCI/ATS not supported, system working in IOMMU legacy mode\n" ); + intel_iommu_sm =0 ; + } + +break ; + } }else if (info->event ==BUS_NOTIFY_REMOVED_DEVICE ) {if (dmar_remove_dev_scope (info, satc->segment, satcu->devices, satcu->devices_cnt))
Signature · 1 line
2.34.1
Quoted 13 lines
From: Lu Baolu <[email protected]>
Sent: Wednesday, September 4, 2024 2:07 PMSOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the
ATC_REQUIREDbit set in the Flags field.The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS can cause a conflict, leading to boot failure, especially if the device is a graphics device.
Quoted 4 lines
To prevent this issue, check PCI ATS support before enumerating the IOMMU devices. If any device requires PCI ATS, but PCI ATS is disabled by 'pci=noats', switch the IOMMU to operate in legacy mode to ensure successful booting.
I guess the reason of switching to legacy mode is because the platform automatically enables ATS in this mode, as the comment says in dmar_ats_supported(). This should be explained otherwise it's unclear why switching the mode can make ATS working for those devices.
But then doesn't it break the meaning of 'pci=noats' which means disabling ATS physically? It's described as "do not use PCIe ATS and IOMMU device IOTLB" in kernel doc, which is not equivalent to "leave PCIe ATS to be managed by HW".
and why would one want to use 'pci=noats' on a platform which requires ats?
Baolu Lu
maintainerQuoted 18 lines · Kevin
Quoted 13 lines
From: Lu Baolu <[email protected]>
Sent: Wednesday, September 4, 2024 2:07 PMSOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the
ATC_REQUIREDbit set in the Flags field.The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS can cause a conflict, leading to boot failure, especially if the device is a graphics device.
Quoted 4 lines
To prevent this issue, check PCI ATS support before enumerating the IOMMU devices. If any device requires PCI ATS, but PCI ATS is disabled by 'pci=noats', switch the IOMMU to operate in legacy mode to ensure successful booting.
Quoted 4 lines
I guess the reason of switching to legacy mode is because the platform automatically enables ATS in this mode, as the comment says in
dmar_ats_supported(). This should be explained otherwise it's unclear why switching the mode can make ATS working for those devices.
Not 'automatically enable ATS,' but hardware provides something that is equivalent to PCI ATS. The ATS capability on the device is still disabled. That's the reason why such device must be an SOC-integrated one.
Quoted 5 lines
But then doesn't it break the meaning of 'pci=noats' which means disabling ATS physically? It's described as "do not use PCIe ATS and IOMMU device IOTLB" in kernel doc, which is not equivalent to "leave PCIe ATS to be managed by HW".
Therefore, the PCI ATS is not used and the syntax of pci=noats is not broken.
Quoted 2 lines
and why would one want to use 'pci=noats' on a platform which requires ats?
We don't recommend users to disable ATS on a platform which has devices that rely on it. But nothing can prevent users from doing so. I am not sure why it is needed. One possible reason that I can think of is about security. Sometimes, people don't trust ATS because it allows devices to access the memory with translated requests directly without any permission check on the IOMMU end.
Thanks,
baolu
Quoted 28 lines
From: Baolu Lu <[email protected]>
Sent: Wednesday, September 4, 2024 3:50 PMQuoted 8 lines · Kevin
Quoted 8 lines
From: Lu Baolu <[email protected]>
Sent: Wednesday, September 4, 2024 2:07 PMSOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the
ATC_REQUIREDbit set in the Flags field.Quoted 6 lines
Quoted 6 lines
The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS can cause a conflict, leading to boot failure, especially if the device is a graphics device.
To prevent this issue, check PCI ATS support before enumerating the
IOMMU
Quoted 8 lines
Quoted 3 lines
devices. If any device requires PCI ATS, but PCI ATS is disabled by 'pci=noats', switch the IOMMU to operate in legacy mode to ensure successful booting.
I guess the reason of switching to legacy mode is because the platform automatically enables ATS in this mode, as the comment says in
dmar_ats_supported(). This should be explained otherwise it's unclear why switching the mode can make ATS working for those devices.
Quoted 4 lines
Not 'automatically enable ATS,' but hardware provides something that is equivalent to PCI ATS. The ATS capability on the device is still disabled. That's the reason why such device must be an SOC-integrated one.
well does that equivalent means use PCI ATS protocol at all (i.e. do untranslated request followed by translated request based on device TLB)?
If yes it's still ATS under the hood.
If not could you elaborate how it works in PCI world?
Quoted 9 lines
Quoted 6 lines
But then doesn't it break the meaning of 'pci=noats' which means disabling ATS physically? It's described as "do not use PCIe ATS and IOMMU device IOTLB" in kernel doc, which is not equivalent to "leave PCIe ATS to be managed by HW".
Therefore, the PCI ATS is not used and the syntax of pci=noats is not broken.
I'm not sure the point of noats is to just disable the PCI capability while allowing the underlying hw to continue sending ATS protocol...
Quoted 11 lines
Quoted 3 lines
and why would one want to use 'pci=noats' on a platform which requires ats?
We don't recommend users to disable ATS on a platform which has devices that rely on it. But nothing can prevent users from doing so. I am not sure why it is needed. One possible reason that I can think of is about security. Sometimes, people don't trust ATS because it allows devices to access the memory with translated requests directly without any permission check on the IOMMU end.
but this doesn't make sense. If the user doesn't trust ATS and deliberately wants to disable ats then it should be followed and whatever usage requiring ATS is then broken. The user should decide which is more favored between security vs. usage to make the right call.
Baolu Lu
maintainerQuoted 33 lines · Kevin
Quoted 28 lines
From: Baolu Lu <[email protected]>
Sent: Wednesday, September 4, 2024 3:50 PMQuoted 8 lines · Kevin
Quoted 8 lines
From: Lu Baolu <[email protected]>
Sent: Wednesday, September 4, 2024 2:07 PMSOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the
ATC_REQUIREDbit set in the Flags field.Quoted 6 lines
Quoted 6 lines
The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS can cause a conflict, leading to boot failure, especially if the device is a graphics device.
To prevent this issue, check PCI ATS support before enumerating the
IOMMU
Quoted 8 lines
Quoted 3 lines
devices. If any device requires PCI ATS, but PCI ATS is disabled by 'pci=noats', switch the IOMMU to operate in legacy mode to ensure successful booting.
I guess the reason of switching to legacy mode is because the platform automatically enables ATS in this mode, as the comment says in
dmar_ats_supported(). This should be explained otherwise it's unclear why switching the mode can make ATS working for those devices.Quoted 4 lines
Not 'automatically enable ATS,' but hardware provides something that is equivalent to PCI ATS. The ATS capability on the device is still disabled. That's the reason why such device must be an SOC-integrated one.
Quoted 7 lines
well does that equivalent means use PCI ATS protocol at all (i.e. do untranslated request followed by translated request based on device TLB)?
If yes it's still ATS under the hood.
If not could you elaborate how it works in PCI world?
I'm not a hardware expert, so I can't provide specific details. :-)
Anyway, from the Linux box's perspective, if 'pci=noats' is used on a Meteorlake device, the 'lspci' tool shows that PCI ATS is disabled:
# dmesg [...] [ 2.419806] pci 0000:00:02.0: DMAR: PCI/ATS not supported, system working in IOMMU legacy mode [...]
# lspci -s 0000:00:02.0 -vv
00:02.0 VGA compatible controller: Intel Corporation Meteor Lake-M [Intel Graphics] (prog-if 00 [VGA controller])
[...]
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable-, Smallest Translation Unit: 00
[...]As for how hardware works, it appears to be transparent to the software.
Quoted 26 lines
Quoted 10 lines
Quoted 7 lines
But then doesn't it break the meaning of 'pci=noats' which means disabling ATS physically? It's described as "do not use PCIe ATS and IOMMU device IOTLB" in kernel doc, which is not equivalent to "leave PCIe ATS to be managed by HW".
Therefore, the PCI ATS is not used and the syntax of pci=noats is not broken.
I'm not sure the point of noats is to just disable the PCI capability while allowing the underlying hw to continue sending ATS protocol...
Quoted 11 lines
Quoted 3 lines
and why would one want to use 'pci=noats' on a platform which requires ats?
We don't recommend users to disable ATS on a platform which has devices that rely on it. But nothing can prevent users from doing so. I am not sure why it is needed. One possible reason that I can think of is about security. Sometimes, people don't trust ATS because it allows devices to access the memory with translated requests directly without any permission check on the IOMMU end.
Quoted 4 lines
but this doesn't make sense. If the user doesn't trust ATS and deliberately wants to disable ats then it should be followed and whatever usage requiring ATS is then broken. The user should decide which is more favored between security vs. usage to make the right call.
Yes. That's just a reason that I could think of.
Anyway, for a client platform, we should avoid any boot failure no matter what kernel parameters the users are using.
Instead, perhaps we could emit a big fat warning to the user, informing them that a device relies on ATS for functionality, thus 'pci=noats' might be compromised.
Thanks,
baolu
Quoted 24 lines · Baolu Lu
Quoted 18 lines · Kevin
Quoted 13 lines
From: Lu Baolu <[email protected]>
Sent: Wednesday, September 4, 2024 2:07 PMSOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the
ATC_REQUIREDbit set in the Flags field.The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS can cause a conflict, leading to boot failure, especially if the device is a graphics device.
Quoted 4 lines
To prevent this issue, check PCI ATS support before enumerating the IOMMU devices. If any device requires PCI ATS, but PCI ATS is disabled by 'pci=noats', switch the IOMMU to operate in legacy mode to ensure successful booting.
Quoted 4 lines
I guess the reason of switching to legacy mode is because the platform automatically enables ATS in this mode, as the comment says in
dmar_ats_supported(). This should be explained otherwise it's unclear why switching the mode can make ATS working for those devices.
Quoted 4 lines
Not 'automatically enable ATS,' but hardware provides something that is equivalent to PCI ATS. The ATS capability on the device is still disabled. That's the reason why such device must be an SOC-integrated one.
That is confusing, how to know the "hardware provides something that is equivalent to PCI ATS" ? any public docs to say that ?
Quoted 13 lines
Quoted 6 lines
But then doesn't it break the meaning of 'pci=noats' which means disabling ATS physically? It's described as "do not use PCIe ATS and IOMMU device IOTLB" in kernel doc, which is not equivalent to "leave PCIe ATS to be managed by HW".
Therefore, the PCI ATS is not used and the syntax of pci=noats is not broken.
Quoted 2 lines
and why would one want to use 'pci=noats' on a platform which requires ats?
Quoted 6 lines
We don't recommend users to disable ATS on a platform which has devices that rely on it. But nothing can prevent users from doing so. I am not sure why it is needed. One possible reason that I can think of is about security. Sometimes, people don't trust ATS because it allows devices to access the memory with translated requests directly without any permission check on the IOMMU end.
Appears that would happen with CXL link, while PCI link still will do some checking (per VT-d spec sec 4.2.4). I have question here, such behaviour happens with HW passthrough, also does to software passthrough (removed identity mapping) ?
Thanks,
Ethan
Quoted 4 lines
Thanks,
baolu
Hi Lu,
Tested this on an X1 Carbon G12 with a kernel built form drm-tip and this patch - and was able to boot successfully with pci=noats
Mark
PS - note on small typo below.
Quoted 7 lines · Lu Baolu
SOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the
ATC_REQUIREDbit set in the Flags field.The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS
pci=noats
Quoted 53 lines
can cause a conflict, leading to boot failure, especially if the device is a graphics device.
To prevent this issue, check PCI ATS support before enumerating the IOMMU devices. If any device requires PCI ATS, but PCI ATS is disabled by 'pci=noats', switch the IOMMU to operate in legacy mode to ensure successful booting.
Fixes 97f2f2c5317f ("iommu/vt-d: Enable ATS for the devices in SATC table")Signed-off-by Lu Baolu <[email protected]>--- drivers/iommu/intel/iommu.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)drivers/iommu/intel/iommu.c +19
-3@@ -3127,10 +3127,26 @@ int dmar_iommu_notify_scope_dev(struct dmar_pci_notify_info *info) (void *)satc + satc->header.length, satc->segment, satcu->devices, satcu->devices_cnt); -if (ret >0 ) -break ; -else if (ret <0 ) +if (ret <0 )return ret; + +if (ret >0 ) { +/* +* The device requires PCI/ATS when the IOMMU +* works in the scalable mode. If PCI/ATS is +* disabled using the pci=noats kernel parameter, +* the IOMMU will default to legacy mode. Users +* are informed of this change. +*/ +if (intel_iommu_sm && satcu->atc_required && + !pci_ats_supported (info->dev)) { +pci_warn (info->dev, +"PCI/ATS not supported, system working in IOMMU legacy mode\n" ); + intel_iommu_sm =0 ; + } + +break ; + } }else if (info->event ==BUS_NOTIFY_REMOVED_DEVICE ) { if (dmar_remove_dev_scope(info, satc->segment,
satcu->devices,satcu->devices_cnt))Signature · 1 line
2.34.1
Baolu Lu
maintainerQuoted 5 lines · Mark Pearson
Hi Lu,
Tested this on an X1 Carbon G12 with a kernel built form drm-tip and this patch - and was able to boot successfully with pci=noats
Tested-by Mark Pearson<[email protected]>
Thank you!
Quoted 6 lines
Mark
PS - note on small typo below.
On Wed, Sep 4, 2024, at 2:07 AM, Lu Baolu wrote:
Quoted 8 lines
Quoted 7 lines
SOC-integrated devices on some platforms require their PCI ATS enabled for operation when the IOMMU is in scalable mode. Those devices are reported via ACPI/SATC table with the
ATC_REQUIREDbit set in the Flags field.The PCI subsystem offers the 'pci=noats' kernel command to disable PCI ATS on all devices. Using 'pci=noat' with devices that require PCI ATS
pci=noats
Fixed.
Thanks,
baolu