Skip to contentSkip to conversation

vfio/pci: add vfio-pci ATS control property

PATCH Superseded Nathan Chen 3 participants 8 messages Jun 23

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover vfio/pci: add vfio-pci ATS control property

Hi,

This is a follow-up to the series [0] that adds an ATS control property to vfio-pci and wires iommufd support reporting so vfio-pci can decide whether to expose the ATS PCIe extended capability to the guest.

The new vfio-pci property is:
ats=on|off|auto (default: auto)

Behavior:
- off: always masks ATS from guest view
- on: requires ATS support; fails realize if kernel reports no ATS support via IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- auto: masks ATS only when kernel reports no ATS support via IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED

We intentionally omit a legacy ats=on compat override because forcing the value to "on" on old machine types can turn a host-kernel capability update (IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED) into a new realize-time boot failure. So we keep the uniform default ats=auto to preserve bootability and follow effective kernel ATS support rather than preserving the old behavior of always advertising ATS when available.

Please note that no linux-headers sync patch is included in this series because IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is already present in linux-headers/linux/iommufd.h in the current base.

Testing:
Verified ATS capability visibility changes with vfio-pci ats=off/on/auto on an NVIDIA Grace system:
- ats=on + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=auto + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=on + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => realize failure
- ats=auto + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => ATS masked
- ats=off => ATS masked
- ats=on + no ATS support in physical device => warning + no ATS ext cap

Changes from v2:
- Added R-by tags, thanks all!
- Fixed alignment of two object_class_property_set_description() blocks in vfio_pci_properties[]
- Removed compat override setting ats=on and added a warning to the case where ats=auto and kernel reports ATS is not supported.
- Squash patch for warning when ats=on and there is no ATS support in physical device into patch #2
- Emit warning when ats=auto and no ATS support in physical device

Changes from v1:
- Set ats_needed to false by default in vfio_pci_ats_requested_and_supported()
- Revise log prefix from "vfio" to "vfio-pci" in vfio_pci_ats_requested_and_supported()
- Squash compat update patch into patch for introducing ats property for bisectibility
- Revise comments and descriptions for support_ats(), vfio_pci_ats_requested_and_supported(), and vfio_pci_class_init() to be more generic, removing references to IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- Emit a warning when ats=on and no ATS support in physical device

Changes from "Resolve AUTO properties" series:
- Split into separate patches for support_ats() handler introduction and iommufd implementation, "ats" property addition, and compat property fix
- Rebase on vfio-next
- Remove unnecessary includes for hw/vfio/pci.c
- Rename ats_need to ats_needed for vfio_pci_ats_requested_and_supported()
- Move the vfio_pci_ats_requested_and_supported() call to vfio_pci_add_capabilities() so we can propagate the Error **, and then pass ats_needed as a new input arg to vfio_add_ext_cap
- Fix indent error in vfio_add_ext_cap()
- Break and add a pcie_add_capability() call for the PASID case in vfio_add_ext_cap()
- Update comment about return value for support_ats()

Thanks,
Nathan

[0] https://lore.kernel.org/all/[email protected]/

Nathan Chen (2):
iommufd: Introduce handler for device ATS support vfio/pci: Add ats property

 hw/vfio/pci.h                      |  1 +
 include/system/host_iommu_device.h |  9 +++
 backends/iommufd.c                 |  8 +++
 hw/vfio/pci.c                      | 88 ++++++++++++++++++++++++++++--
 4 files changed, 102 insertions(+), 4 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v3 1/2 iommufd: Introduce handler for device ATS support

From: Nathan Chen <[email protected]>

Introduce a support_ats() handler that returns whether ATS is supported for a device associated with a host IOMMU device, checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.

Implement this handler in a new function for iommufd.

---
 include/system/host_iommu_device.h | 9 +++++++++
 backends/iommufd.c                 | 8 ++++++++
 2 files changed, 17 insertions(+)
include/system/host_iommu_device.h +9 -0
index f000301583..2cb5365340 100644
@@ -133,6 +133,15 @@ struct HostIOMMUDeviceClass {
      * Returns: true on success, false on failure.
      */
     bool (*get_pasid_info)(HostIOMMUDevice *hiod, PasidInfo *pasid_info);
+    /**
+     *  @support_ats: Returns true if ATS can be used by the device,
+     *  false if the host IOMMU reports it is unavailable.
+     *
+     *  @hiod: handle to the host IOMMU device
+     *
+     *  Returns: true if ATS is supported, false otherwise
+     */
+    bool (*support_ats)(HostIOMMUDevice *hiod);
 };
 
 /*
backends/iommufd.c +8 -0
index 09624cd652..4b40aaebe6 100644
@@ -638,6 +638,13 @@ static int hiod_iommufd_get_cap(HostIOMMUDevice *hiod, int cap, Error **errp)
     }
 }
 
+static bool hiod_iommufd_support_ats(HostIOMMUDevice *hiod)
+{
+    HostIOMMUDeviceCaps *caps = &hiod->caps;
+
+    return !(caps->hw_caps & IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED);
+}
+
 static bool hiod_iommufd_get_pasid_info(HostIOMMUDevice *hiod,
                                         PasidInfo *pasid_info)
 {
@@ -660,6 +667,7 @@ static void hiod_iommufd_class_init(ObjectClass *oc, const void *data)
 
     hiodc->get_cap = hiod_iommufd_get_cap;
     hiodc->get_pasid_info = hiod_iommufd_get_pasid_info;
+    hiodc->support_ats = hiod_iommufd_support_ats;
 };
 
 static const TypeInfo types[] = {
Signature · 1 line
2.43.0
#2 lore raw

PATCH v3 2/2 vfio/pci: Add ats property

In reply to Nathan Chen

From: Nathan Chen <[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

Emit a warning when ats=on is requested but the physical device does not advertise ATS, since ATS cannot be exposed to the guest in this case.

---
 hw/vfio/pci.h |  1 +
 hw/vfio/pci.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 85 insertions(+), 4 deletions(-)
hw/vfio/pci.h +1 -0
index c3a1f53d35..f2934f2d84 100644
@@ -188,6 +188,7 @@ struct VFIOPCIDevice {
     bool clear_parent_atomics_on_exit;
     bool skip_vsc_check;
     uint16_t vpasid_cap_offset;
+    OnOffAuto ats;
     VFIODisplay *dpy;
     Notifier irqchip_change_notifier;
     VFIOPCICPR cpr;
hw/vfio/pci.c +84 -4

139 lines not on this page · raw message

Signature · 1 line
2.43.0
#3 lore raw

RE: [PATCH v3 2/2] vfio/pci: Add ats property

Quoted 94 lines

-----Original Message-----
From: Nathan Chen <[email protected]>
Sent: 23 June 2026 03:36
To: [email protected]
Cc: Yi Liu <[email protected]>; Eric Auger <[email protected]>; Zhenzhong Duan <[email protected]>; Alex Williamson <[email protected]>; Cédric Le Goater <[email protected]>; Matt Ochs <[email protected]>; Nicolin Chen <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Nathan Chen <[email protected]>
Subject: [PATCH v3 2/2] vfio/pci: Add ats property

From: Nathan Chen <[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask
the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

Emit a warning when ats=on is requested but the physical device does not advertise ATS, since ATS cannot be exposed to the guest in this case.

---
 hw/vfio/pci.h |  1 +
 hw/vfio/pci.c | 88
++++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 85 insertions(+), 4 deletions(-)
hw/vfio/pci.h +1 -0
index c3a1f53d35..f2934f2d84 100644
@@ -188,6 +188,7 @@ struct VFIOPCIDevice {
     bool clear_parent_atomics_on_exit;
     bool skip_vsc_check;
     uint16_t vpasid_cap_offset;
+    OnOffAuto ats;
     VFIODisplay *dpy;
     Notifier irqchip_change_notifier;
     VFIOPCICPR cpr;
hw/vfio/pci.c +2 -1
index 9c06b25e63..c0436b4c04 100644
@@ -2546,10 +2546,58 @@ static bool
 vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)
     return true;
 }
 
-static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
+/*
+ * Determine whether ATS capability should be advertised for @vdev, based

on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reports it is not supported. Returns true in all other cases.
+ */

hw/vfio/pci.c +22 -0
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev,
+                                                 bool *ats_needed, Error **errp)
+{
+    HostIOMMUDevice *hiod = vdev->vbasedev.hiod;
+    HostIOMMUDeviceClass *hiodc;
+    bool ats_supported;
+    *ats_needed = false;
+
+    if (vdev->ats == ON_OFF_AUTO_OFF) {
+        return true;
+    }
+
+    *ats_needed = true;
+    if (!hiod) {
+        return true;
+    }
+    hiodc = HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+    if (!hiodc || !hiodc->support_ats) {
+        return true;
+    }
+
+    ats_supported = hiodc->support_ats(hiod);

hw/vfio/pci.c static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) · line 2590

Quoted 10 lines
hw/vfio/pci.c +10 -0
+    if (vdev->ats == ON_OFF_AUTO_ON && !ats_supported) {
+        error_setg(errp, "vfio-pci: ATS requested but not supported by kernel");
+        *ats_needed = false;
+        return false;
+    }
+
+    if (vdev->ats == ON_OFF_AUTO_AUTO && !ats_supported) {
+        warn_report("vfio-pci: host kernel reports ATS unsupported; "
+                    "ATS capability will be masked");
+    }

This will be slightly misleading if dev doesn't have the ATS CAP at all. Isn't it? I think, the best place to do this warning is below where we know ats_cap_present is set or not.

Thanks,
Shameer

Quoted 115 lines
hw/vfio/pci.c +20 -1
+
+    *ats_needed = ats_supported;
+    return true;
+}
+
+static void vfio_add_ext_cap(VFIOPCIDevice *vdev, bool ats_needed)
 {
     PCIDevice *pdev = PCI_DEVICE(vdev);
     bool pasid_cap_added = false;
+    bool ats_cap_present = false;
     Error *err = NULL;
     uint32_t header;
     uint16_t cap_id, next, size;
@@ -2635,7 +2683,19 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
          */
         case PCI_EXT_CAP_ID_PASID:
             pasid_cap_added = true;
-            /* fallthrough */
+            pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            break;
+        case PCI_EXT_CAP_ID_ATS:
+            ats_cap_present = true;
+            /*
+             * If ATS is requested and supported according to the kernel, add
+             * the ATS capability. If not supported according to the kernel or
+             * disabled on the qemu command line, omit the ATS cap.
+             */
+            if (ats_needed) {
+                pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            }
+            break;
         default:
             pcie_add_capability(pdev, cap_id, cap_ver, next, size);
         }
hw/vfio/pci.c +5 -0
@@ -2646,6 +2706,11 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
         error_report_err(err);
     }
 
+    if (ats_needed && !ats_cap_present) {
+        warn_report("vfio-pci: ats=on requested, but host device has no "
+                    "ATS extended capability");
+    }
+
     /* Cleanup chain head ID if necessary */
     if (pci_get_word(pdev->config + PCI_CONFIG_SPACE_SIZE) == 0xFFFF) {
         pci_set_word(pdev->config + PCI_CONFIG_SPACE_SIZE, 0);
hw/vfio/pci.c +1 -0
@@ -2657,6 +2722,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
 bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp)
 {
     PCIDevice *pdev = PCI_DEVICE(vdev);
+    bool ats_needed = false;
 
     if (!(pdev->config[PCI_STATUS] & PCI_STATUS_CAP_LIST) ||
         !pdev->config[PCI_CAPABILITY_LIST]) {
hw/vfio/pci.c +6 -1
@@ -2667,7 +2733,11 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice
 *vdev, Error **errp)
         return false;
     }
 
-    vfio_add_ext_cap(vdev);
+    if (!vfio_pci_ats_requested_and_supported(vdev, &ats_needed, errp)) {
+        return false;
+    }
+
+    vfio_add_ext_cap(vdev, ats_needed);
     return true;
 }
@@ -3815,6 +3885,7 @@ static const Property vfio_pci_properties[] = {
     DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check,
 true),
     DEFINE_PROP_UINT16("x-vpasid-cap-offset", VFIOPCIDevice,
                        vpasid_cap_offset, 0),
+    DEFINE_PROP_ON_OFF_AUTO("ats", VFIOPCIDevice, ats,
 ON_OFF_AUTO_AUTO),
 };

static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp)

hw/vfio/pci.c +1 -1
@@ -3966,13 +4037,22 @@ static void vfio_pci_class_init(ObjectClass *klass,
 const void *data)
                                           "destination when doing live "
                                           "migration of device state via "
                                           "multifd channels");
-   object_class_property_set_description(klass, /* 11.0 */
+    object_class_property_set_description(klass, /* 11.0 */
                                           "x-vpasid-cap-offset",
                                           "PCIe extended configuration space offset at which to

place a "
"synthetic PASID extended capability when PASID is
enabled via "
"a vIOMMU. A value of 0 (default) places the capability
at the "
"end of the extended configuration space. The offset
must be "
"4-byte aligned and within the PCIe extended
configuration space");

hw/vfio/pci.c +3 -0
+    object_class_property_set_description(klass, /* 11.1 */
+                                          "ats",
+                                          "Control guest visibility of the ATS PCIe extended

capability. "

hw/vfio/pci.c +3 -0
+                                          "Valid values are on, off, and auto (default). "
+                                          "'off' always masks ATS. "
+                                          "'on' requires ATS support for the device and fails realize

if the "

hw/vfio/pci.c +3 -1
+                                          "host kernel reports ATS as unavailable for this device. "
+                                          "'auto' masks ATS only when the host kernel reports "
+                                          "ATS as unavailable");
 }
 
 static const TypeInfo vfio_pci_info = {
--

2.43.0

#4 lore raw

Re: [PATCH v3 2/2] vfio/pci: Add ats property

Quoted 152 lines · Shameer Kolothum Thodi
Quoted 94 lines

-----Original Message-----
From: Nathan Chen <[email protected]>
Sent: 23 June 2026 03:36
To: [email protected]
Cc: Yi Liu <[email protected]>; Eric Auger <[email protected]>; Zhenzhong Duan <[email protected]>; Alex Williamson <[email protected]>; Cédric Le Goater <[email protected]>; Matt Ochs <[email protected]>; Nicolin Chen <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Nathan Chen <[email protected]>
Subject: [PATCH v3 2/2] vfio/pci: Add ats property

From: Nathan Chen <[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask
the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

Emit a warning when ats=on is requested but the physical device does not advertise ATS, since ATS cannot be exposed to the guest in this case.

---
  hw/vfio/pci.h |  1 +
  hw/vfio/pci.c | 88
++++++++++++++++++++++++++++++++++++++++++++++++---
  2 files changed, 85 insertions(+), 4 deletions(-)
hw/vfio/pci.h +1 -0
index c3a1f53d35..f2934f2d84 100644
@@ -188,6 +188,7 @@ struct VFIOPCIDevice {
      bool clear_parent_atomics_on_exit;
      bool skip_vsc_check;
      uint16_t vpasid_cap_offset;
+    OnOffAuto ats;
      VFIODisplay *dpy;
      Notifier irqchip_change_notifier;
      VFIOPCICPR cpr;
hw/vfio/pci.c +2 -1
index 9c06b25e63..c0436b4c04 100644
@@ -2546,10 +2546,58 @@ static bool
 vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)
      return true;
  }
 
-static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
+/*
+ * Determine whether ATS capability should be advertised for @vdev, based

on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reports it is not supported. Returns true in all other cases.
+ */

hw/vfio/pci.c +22 -0
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev,
+                                                 bool *ats_needed, Error **errp)
+{
+    HostIOMMUDevice *hiod = vdev->vbasedev.hiod;
+    HostIOMMUDeviceClass *hiodc;
+    bool ats_supported;
+    *ats_needed = false;
+
+    if (vdev->ats == ON_OFF_AUTO_OFF) {
+        return true;
+    }
+
+    *ats_needed = true;
+    if (!hiod) {
+        return true;
+    }
+    hiodc = HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+    if (!hiodc || !hiodc->support_ats) {
+        return true;
+    }
+
+    ats_supported = hiodc->support_ats(hiod);

hw/vfio/pci.c static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) · line 2590

Quoted 10 lines
hw/vfio/pci.c +10 -0
+    if (vdev->ats == ON_OFF_AUTO_ON && !ats_supported) {
+        error_setg(errp, "vfio-pci: ATS requested but not supported by kernel");
+        *ats_needed = false;
+        return false;
+    }
+
+    if (vdev->ats == ON_OFF_AUTO_AUTO && !ats_supported) {
+        warn_report("vfio-pci: host kernel reports ATS unsupported; "
+                    "ATS capability will be masked");
+    }

This will be slightly misleading if dev doesn't have the ATS CAP at all. Isn't it? I think, the best place to do this warning is below where we know ats_cap_present is set or not.

Thanks,
Shameer

Quoted 40 lines
hw/vfio/pci.c +20 -1
+
+    *ats_needed = ats_supported;
+    return true;
+}
+
+static void vfio_add_ext_cap(VFIOPCIDevice *vdev, bool ats_needed)
  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
      bool pasid_cap_added = false;
+    bool ats_cap_present = false;
      Error *err = NULL;
      uint32_t header;
      uint16_t cap_id, next, size;
@@ -2635,7 +2683,19 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
           */
          case PCI_EXT_CAP_ID_PASID:
              pasid_cap_added = true;
-            /* fallthrough */
+            pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            break;
+        case PCI_EXT_CAP_ID_ATS:
+            ats_cap_present = true;
+            /*
+             * If ATS is requested and supported according to the kernel, add
+             * the ATS capability. If not supported according to the kernel or
+             * disabled on the qemu command line, omit the ATS cap.
+             */
+            if (ats_needed) {
+                pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            }
+            break;
          default:
              pcie_add_capability(pdev, cap_id, cap_ver, next, size);
          }
hw/vfio/pci.c +0 -0
@@ -2646,6 +2706,11 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
          error_report_err(err);
      }
 
Quoted 4 lines

hw/vfio/pci.c static void vfio_add_ext_cap(VFIOPCIDevice *vdev) · line 2712

Quoted 4 lines
hw/vfio/pci.c +4 -0
+    if (ats_needed && !ats_cap_present) {
+        warn_report("vfio-pci: ats=on requested, but host device has no "
+                    "ATS extended capability");
+    }

"ats=on requested" is wrong here. OnOffAuto_str() could be used instead.

Also, ats_needed is true for "ats=auto" too, and since it's the default setting, and most devices lack ATS and any modern host kernel support it, this warning will be emitted for nearly all devices.

How about :

  if (vdev->ats == ON_OFF_AUTO_ON && !ats_cap_present) {
      warn_report("vfio-pci: ats=on requested, but host device has no "
                  "ATS extended capability");
  }

?

Thanks,

C.

Quoted 72 lines
Quoted 71 lines
hw/vfio/pci.c +1 -0
+
      /* Cleanup chain head ID if necessary */
      if (pci_get_word(pdev->config + PCI_CONFIG_SPACE_SIZE) == 0xFFFF) {
          pci_set_word(pdev->config + PCI_CONFIG_SPACE_SIZE, 0);
hw/vfio/pci.c +1 -0
@@ -2657,6 +2722,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
  bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp)
  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
+    bool ats_needed = false;
 
      if (!(pdev->config[PCI_STATUS] & PCI_STATUS_CAP_LIST) ||
          !pdev->config[PCI_CAPABILITY_LIST]) {
hw/vfio/pci.c +6 -1
@@ -2667,7 +2733,11 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice
 *vdev, Error **errp)
          return false;
      }
 
-    vfio_add_ext_cap(vdev);
+    if (!vfio_pci_ats_requested_and_supported(vdev, &ats_needed, errp)) {
+        return false;
+    }
+
+    vfio_add_ext_cap(vdev, ats_needed);
      return true;
  }
@@ -3815,6 +3885,7 @@ static const Property vfio_pci_properties[] = {
      DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check,
 true),
      DEFINE_PROP_UINT16("x-vpasid-cap-offset", VFIOPCIDevice,
                         vpasid_cap_offset, 0),
+    DEFINE_PROP_ON_OFF_AUTO("ats", VFIOPCIDevice, ats,
 ON_OFF_AUTO_AUTO),
  };

static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp)

hw/vfio/pci.c +1 -1
@@ -3966,13 +4037,22 @@ static void vfio_pci_class_init(ObjectClass *klass,
 const void *data)
                                            "destination when doing live "
                                            "migration of device state via "
                                            "multifd channels");
-   object_class_property_set_description(klass, /* 11.0 */
+    object_class_property_set_description(klass, /* 11.0 */
                                            "x-vpasid-cap-offset",
                                            "PCIe extended configuration space offset at which to

place a "
"synthetic PASID extended capability when PASID is
enabled via "
"a vIOMMU. A value of 0 (default) places the capability
at the "
"end of the extended configuration space. The offset
must be "
"4-byte aligned and within the PCIe extended
configuration space");

hw/vfio/pci.c +3 -0
+    object_class_property_set_description(klass, /* 11.1 */
+                                          "ats",
+                                          "Control guest visibility of the ATS PCIe extended

capability. "

hw/vfio/pci.c +3 -0
+                                          "Valid values are on, off, and auto (default). "
+                                          "'off' always masks ATS. "
+                                          "'on' requires ATS support for the device and fails realize

if the "

hw/vfio/pci.c +3 -1
+                                          "host kernel reports ATS as unavailable for this device. "
+                                          "'auto' masks ATS only when the host kernel reports "
+                                          "ATS as unavailable");
  }
 
  static const TypeInfo vfio_pci_info = {
--

2.43.0

#5 lore raw

Re: [PATCH v3 2/2] vfio/pci: Add ats property

Quoted 172 lines · Cédric Le Goater
Quoted 159 lines · Shameer Kolothum Thodi
Quoted 101 lines

-----Original Message-----
From: Nathan Chen <[email protected]>
Sent: 23 June 2026 03:36
To: [email protected]
Cc: Yi Liu <[email protected]>; Eric Auger <[email protected]>; Zhenzhong Duan <[email protected]>; Alex Williamson <[email protected]>; Cédric Le Goater <[email protected]>; Matt Ochs <[email protected]>; Nicolin Chen <[email protected]>; Shameer Kolothum Thodi <[email protected]>; Nathan Chen <[email protected]>
Subject: [PATCH v3 2/2] vfio/pci: Add ats property

From: Nathan Chen <[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask
the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

Emit a warning when ats=on is requested but the physical device does not advertise ATS, since ATS cannot be exposed to the guest in this case.

---
  hw/vfio/pci.h |  1 +
  hw/vfio/pci.c | 88
++++++++++++++++++++++++++++++++++++++++++++++++---

  2 files changed, 85 insertions(+), 4 deletions(-)
hw/vfio/pci.h +1 -0
index c3a1f53d35..f2934f2d84 100644
@@ -188,6 +188,7 @@ struct VFIOPCIDevice {
       bool clear_parent_atomics_on_exit;
       bool skip_vsc_check;
       uint16_t vpasid_cap_offset;
+    OnOffAuto ats;
       VFIODisplay *dpy;
       Notifier irqchip_change_notifier;
       VFIOPCICPR cpr;
hw/vfio/pci.c +2 -1
index 9c06b25e63..c0436b4c04 100644
@@ -2546,10 +2546,58 @@ static bool
 vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)
       return true;
   }
 
-static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
+/*
+ * Determine whether ATS capability should be advertised for @vdev, 

based
on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reports it is not supported. Returns true in all other cases.
+ */
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev, +                                                 bool *ats_needed, Error **errp)

hw/vfio/pci.c +22 -0
+{
+    HostIOMMUDevice *hiod = vdev->vbasedev.hiod;
+    HostIOMMUDeviceClass *hiodc;
+    bool ats_supported;
+    *ats_needed = false;
+
+    if (vdev->ats == ON_OFF_AUTO_OFF) {
+        return true;
+    }
+
+    *ats_needed = true;
+    if (!hiod) {
+        return true;
+    }
+    hiodc = HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+    if (!hiodc || !hiodc->support_ats) {
+        return true;
+    }
+
+    ats_supported = hiodc->support_ats(hiod);
+    if (vdev->ats == ON_OFF_AUTO_ON && !ats_supported) {
+        error_setg(errp, "vfio-pci: ATS requested but not supported 

by kernel");

Quoted 8 lines

+        *ats_needed = false;
+        return false;
+    }
+
+    if (vdev->ats == ON_OFF_AUTO_AUTO && !ats_supported) { +        warn_report("vfio-pci: host kernel reports ATS unsupported; " +                    "ATS capability will be masked");
+    }

This will be slightly misleading if dev doesn't have the ATS CAP at all. Isn't it?  I think, the best place to do this warning is below where we know ats_cap_present is set or not.

Thanks,
Shameer

Quoted 42 lines
hw/vfio/pci.c +6 -0
+
+    *ats_needed = ats_supported;
+    return true;
+}
+
+static void vfio_add_ext_cap(VFIOPCIDevice *vdev, bool ats_needed)

  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
      bool pasid_cap_added = false;
+    bool ats_cap_present = false;
      Error *err = NULL;
      uint32_t header;
      uint16_t cap_id, next, size;

hw/vfio/pci.c +0 -0
@@ -2635,7 +2683,19 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
            */

          case PCI_EXT_CAP_ID_PASID:
              pasid_cap_added = true;
-            /* fallthrough */
+            pcie_add_capability(pdev, cap_id, cap_ver, next, size); +            break;
+        case PCI_EXT_CAP_ID_ATS:
+            ats_cap_present = true;
+            /*
+             * If ATS is requested and supported according to the kernel, add
+             * the ATS capability. If not supported according to the kernel or
+             * disabled on the qemu command line, omit the ATS cap. +             */
+            if (ats_needed) {
+                pcie_add_capability(pdev, cap_id, cap_ver, next, size); +            }
+            break;
          default:
              pcie_add_capability(pdev, cap_id, cap_ver, next, size); }

hw/vfio/pci.c +0 -0
@@ -2646,6 +2706,11 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
           error_report_err(err);
       }
 
Quoted 5 lines
Quoted 5 lines
hw/vfio/pci.c +2 -0
+    if (ats_needed && !ats_cap_present) {
+        warn_report("vfio-pci: ats=on requested, but host device has 

no "
+                    "ATS extended capability");
+    }

"ats=on requested" is wrong here. OnOffAuto_str() could be used instead.

Also, ats_needed is true for "ats=auto" too, and since it's the default setting, and most devices lack ATS and any modern host kernel support it, this warning will be emitted for nearly all devices.

Quoted 6 lines

How about :

  if (vdev->ats == ON_OFF_AUTO_ON && !ats_cap_present) { warn_report("vfio-pci: ats=on requested, but host device has no " "ATS extended capability");
  }

Yes that is a good point about the warning being emitted for many other devices that lack ATS. I will revert this back to checking vdev->ats == ON_OFF_AUTO_ON. Originally I was thinking it may be useful to emit a warning for the auto case as well when host device has no ATS. But I think that is outweighed by your point about most devices lacking ATS, and users can just check the ATS cap's presence directly from the host beforehand.

Thanks,
Nathan

#6 lore raw

Re: [PATCH v3 2/2] vfio/pci: Add ats property

In reply to Shameer Kolothum Thodi
Quoted 107 lines · Shameer Kolothum Thodi
Quoted 93 lines

-----Original Message-----
From: Nathan Chen<[email protected]>
Sent: 23 June 2026 03:36
To:[email protected]
Cc: Yi Liu<[email protected]>; Eric Auger<[email protected]>; Zhenzhong Duan<[email protected]>; Alex Williamson <[email protected]>; Cédric Le Goater<[email protected]>; Matt Ochs <[email protected]>; Nicolin Chen<[email protected]>; Shameer Kolothum Thodi<[email protected]>; Nathan Chen <[email protected]>
Subject: [PATCH v3 2/2] vfio/pci: Add ats property

From: Nathan Chen<[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask
the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

Emit a warning when ats=on is requested but the physical device does not advertise ATS, since ATS cannot be exposed to the guest in this case.

---
  hw/vfio/pci.h |  1 +
  hw/vfio/pci.c | 88
++++++++++++++++++++++++++++++++++++++++++++++++---
  2 files changed, 85 insertions(+), 4 deletions(-)
hw/vfio/pci.h +1 -0
index c3a1f53d35..f2934f2d84 100644
@@ -188,6 +188,7 @@ struct VFIOPCIDevice {
      bool clear_parent_atomics_on_exit;
      bool skip_vsc_check;
      uint16_t vpasid_cap_offset;
+    OnOffAuto ats;
      VFIODisplay *dpy;
      Notifier irqchip_change_notifier;
      VFIOPCICPR cpr;
hw/vfio/pci.c +2 -1
index 9c06b25e63..c0436b4c04 100644
@@ -2546,10 +2546,58 @@ static bool
 vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)
      return true;
  }
 
-static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
+/*
+ * Determine whether ATS capability should be advertised for @vdev, based

on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reports it is not supported. Returns true in all other cases.
+ */

hw/vfio/pci.c +22 -0
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev,
+                                                 bool *ats_needed, Error **errp)
+{
+    HostIOMMUDevice *hiod = vdev->vbasedev.hiod;
+    HostIOMMUDeviceClass *hiodc;
+    bool ats_supported;
+    *ats_needed = false;
+
+    if (vdev->ats == ON_OFF_AUTO_OFF) {
+        return true;
+    }
+
+    *ats_needed = true;
+    if (!hiod) {
+        return true;
+    }
+    hiodc = HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+    if (!hiodc || !hiodc->support_ats) {
+        return true;
+    }
+
+    ats_supported = hiodc->support_ats(hiod);

hw/vfio/pci.c static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) · line 2590

Quoted 10 lines
hw/vfio/pci.c +10 -0
+    if (vdev->ats == ON_OFF_AUTO_ON && !ats_supported) {
+        error_setg(errp, "vfio-pci: ATS requested but not supported by kernel");
+        *ats_needed = false;
+        return false;
+    }
+
+    if (vdev->ats == ON_OFF_AUTO_AUTO && !ats_supported) {
+        warn_report("vfio-pci: host kernel reports ATS unsupported; "
+                    "ATS capability will be masked");
+    }

This will be slightly misleading if dev doesn't have the ATS CAP at all. Isn't it? I think, the best place to do this warning is below where we know ats_cap_present is set or not.

Quoted 3 lines

Thanks,
Shameer

Yes, I agree it would be better to put this in an else statement after it is verified that ats cap is present.

Thanks,
Nathan>> +

Quoted 114 lines
Quoted 114 lines
hw/vfio/pci.c +19 -1
+    *ats_needed = ats_supported;
+    return true;
+}
+
+static void vfio_add_ext_cap(VFIOPCIDevice *vdev, bool ats_needed)
  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
      bool pasid_cap_added = false;
+    bool ats_cap_present = false;
      Error *err = NULL;
      uint32_t header;
      uint16_t cap_id, next, size;
@@ -2635,7 +2683,19 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
           */
          case PCI_EXT_CAP_ID_PASID:
              pasid_cap_added = true;
-            /* fallthrough */
+            pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            break;
+        case PCI_EXT_CAP_ID_ATS:
+            ats_cap_present = true;
+            /*
+             * If ATS is requested and supported according to the kernel, add
+             * the ATS capability. If not supported according to the kernel or
+             * disabled on the qemu command line, omit the ATS cap.
+             */
+            if (ats_needed) {
+                pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            }
+            break;
          default:
              pcie_add_capability(pdev, cap_id, cap_ver, next, size);
          }
hw/vfio/pci.c +5 -0
@@ -2646,6 +2706,11 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
          error_report_err(err);
      }
 
+    if (ats_needed && !ats_cap_present) {
+        warn_report("vfio-pci: ats=on requested, but host device has no "
+                    "ATS extended capability");
+    }
+
      /* Cleanup chain head ID if necessary */
      if (pci_get_word(pdev->config + PCI_CONFIG_SPACE_SIZE) == 0xFFFF) {
          pci_set_word(pdev->config + PCI_CONFIG_SPACE_SIZE, 0);
hw/vfio/pci.c +1 -0
@@ -2657,6 +2722,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice
 *vdev)
  bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp)
  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
+    bool ats_needed = false;
 
      if (!(pdev->config[PCI_STATUS] & PCI_STATUS_CAP_LIST) ||
          !pdev->config[PCI_CAPABILITY_LIST]) {
hw/vfio/pci.c +6 -1
@@ -2667,7 +2733,11 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice
 *vdev, Error **errp)
          return false;
      }
 
-    vfio_add_ext_cap(vdev);
+    if (!vfio_pci_ats_requested_and_supported(vdev, &ats_needed, errp)) {
+        return false;
+    }
+
+    vfio_add_ext_cap(vdev, ats_needed);
      return true;
  }
@@ -3815,6 +3885,7 @@ static const Property vfio_pci_properties[] = {
      DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check,
 true),
      DEFINE_PROP_UINT16("x-vpasid-cap-offset", VFIOPCIDevice,
                         vpasid_cap_offset, 0),
+    DEFINE_PROP_ON_OFF_AUTO("ats", VFIOPCIDevice, ats,
 ON_OFF_AUTO_AUTO),
  };

static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp)

hw/vfio/pci.c +1 -1
@@ -3966,13 +4037,22 @@ static void vfio_pci_class_init(ObjectClass *klass,
 const void *data)
                                            "destination when doing live "
                                            "migration of device state via "
                                            "multifd channels");
-   object_class_property_set_description(klass, /* 11.0 */
+    object_class_property_set_description(klass, /* 11.0 */
                                            "x-vpasid-cap-offset",
                                            "PCIe extended configuration space offset at which to

place a "
"synthetic PASID extended capability when PASID is
enabled via "
"a vIOMMU. A value of 0 (default) places the capability
at the "
"end of the extended configuration space. The offset
must be "
"4-byte aligned and within the PCIe extended
configuration space");

hw/vfio/pci.c +3 -0
+    object_class_property_set_description(klass, /* 11.1 */
+                                          "ats",
+                                          "Control guest visibility of the ATS PCIe extended

capability. "

hw/vfio/pci.c +3 -0
+                                          "Valid values are on, off, and auto (default). "
+                                          "'off' always masks ATS. "
+                                          "'on' requires ATS support for the device and fails realize

if the "

hw/vfio/pci.c +3 -1
+                                          "host kernel reports ATS as unavailable for this device. "
+                                          "'auto' masks ATS only when the host kernel reports "
+                                          "ATS as unavailable");
  }
 
  static const TypeInfo vfio_pci_info = {
--

2.43.0

#7 lore raw

Re: [PATCH v3 2/2] vfio/pci: Add ats property

Quoted 115 lines · Nathan Chen
Quoted 112 lines · Shameer Kolothum Thodi
Quoted 100 lines

-----Original Message-----
From: Nathan Chen<[email protected]>
Sent: 23 June 2026 03:36
To:[email protected]
Cc: Yi Liu<[email protected]>; Eric Auger<[email protected]>; Zhenzhong Duan<[email protected]>; Alex Williamson <[email protected]>; Cédric Le Goater<[email protected]>; Matt Ochs <[email protected]>; Nicolin Chen<[email protected]>; Shameer Kolothum Thodi<[email protected]>; Nathan Chen <[email protected]>
Subject: [PATCH v3 2/2] vfio/pci: Add ats property

From: Nathan Chen<[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask
the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

Emit a warning when ats=on is requested but the physical device does not advertise ATS, since ATS cannot be exposed to the guest in this case.

---
  hw/vfio/pci.h |  1 +
  hw/vfio/pci.c | 88
++++++++++++++++++++++++++++++++++++++++++++++++---

  2 files changed, 85 insertions(+), 4 deletions(-)
hw/vfio/pci.h +1 -0
index c3a1f53d35..f2934f2d84 100644
@@ -188,6 +188,7 @@ struct VFIOPCIDevice {
       bool clear_parent_atomics_on_exit;
       bool skip_vsc_check;
       uint16_t vpasid_cap_offset;
+    OnOffAuto ats;
       VFIODisplay *dpy;
       Notifier irqchip_change_notifier;
       VFIOPCICPR cpr;
hw/vfio/pci.c +2 -1
index 9c06b25e63..c0436b4c04 100644
@@ -2546,10 +2546,58 @@ static bool
 vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)
       return true;
   }
 
-static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
+/*
+ * Determine whether ATS capability should be advertised for @vdev, 

based
on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reports it is not supported. Returns true in all other cases.
+ */
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev, +                                                 bool *ats_needed, Error **errp)

hw/vfio/pci.c +22 -0
+{
+    HostIOMMUDevice *hiod = vdev->vbasedev.hiod;
+    HostIOMMUDeviceClass *hiodc;
+    bool ats_supported;
+    *ats_needed = false;
+
+    if (vdev->ats == ON_OFF_AUTO_OFF) {
+        return true;
+    }
+
+    *ats_needed = true;
+    if (!hiod) {
+        return true;
+    }
+    hiodc = HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+    if (!hiodc || !hiodc->support_ats) {
+        return true;
+    }
+
+    ats_supported = hiodc->support_ats(hiod);
+    if (vdev->ats == ON_OFF_AUTO_ON && !ats_supported) {
+        error_setg(errp, "vfio-pci: ATS requested but not supported 

by kernel");

Quoted 8 lines

+        *ats_needed = false;
+        return false;
+    }
+
+    if (vdev->ats == ON_OFF_AUTO_AUTO && !ats_supported) { +        warn_report("vfio-pci: host kernel reports ATS unsupported; " +                    "ATS capability will be masked");
+    }

This will be slightly misleading if dev doesn't have the ATS CAP at all. Isn't it?  I think, the best place to do this warning is below where we know ats_cap_present is set or not.

Quoted 4 lines
Quoted 3 lines

Thanks,
Shameer

Yes, I agree it would be better to put this in an else statement after it is verified that ats cap is present.

On second thought, placing this after verifying ats cap presence would be better:

if (vdev->ats == ON_OFF_AUTO_AUTO && ats_cap_present && !ats_needed) {
warn_report("vfio-pci: host kernel reports ATS unsupported; "
"ATS capability will be masked");
}

When ats=auto, ats_needed would only be false when kernel reports ATS is unsupported. And if ATS is not present, there will be no warning about masking it.

Thanks,
Nathan