Skip to contentSkip to conversation

vfio/pci: add vfio-pci ATS control property

PATCH Superseded Nathan Chen 3 participants 14 messages Jun 12

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover vfio/pci: add vfio-pci ATS control property

Hi,

This series adds an ATS control property to vfio-pci and wires iommufd support reporting so vfio-pci can decide whether to expose the ATS PCIe extended capability to the guest. This series is a follow-up to the previous patch included in the "hw/arm/smmuv3-accel: Resolve AUTO properties" series [0].

The new vfio-pci property is:
ats=on|off|auto (default: auto)

Behavior:
- off: always masks ATS from guest view
- on: requires ATS support; fails realize if kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- auto: masks ATS only when kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED

To preserve machine compatibility for older machine types, this series adds:
{ "vfio-pci", "ats", "on" } to hw_compat_11_0.

Please note that no linux-headers sync patch is included in this series because IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is already present in linux-headers/linux/iommufd.h in the current base.

Testing:
Verified ATS capability visibility changes with vfio-pci ats=off/on/auto on an NVIDIA Grace system:
- ats=on + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=auto + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=on + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => realize failure
- ats=auto + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => ATS masked
- ats=off => ATS masked

Changes from "Resolve AUTO properties" series:
- Split into separate patches for support_ats() handler introduction and iommufd implementation, "ats" property addition, and compat property fix
- Rebase on vfio-next
- Remove unnecessary includes for hw/vfio/pci.c
- Rename ats_need to ats_needed for vfio_pci_ats_requested_and_supported()
- Move the vfio_pci_ats_requested_and_supported() call to vfio_pci_add_capabilities() so we can propagate the Error **, and then pass ats_needed as a new input arg to vfio_add_ext_cap
- Fix indent error in vfio_add_ext_cap()
- Break and add a pcie_add_capability() call for the PASID case in vfio_add_ext_cap()
- Update comment about return value for support_ats()

Thanks,
Nathan

[0] https://lore.kernel.org/qemu-devel/[email protected]/

Nathan Chen (3):
iommufd: Introduce handler for device ATS support vfio/pci: Add ats property
hw/core/machine: Add new compat for vfio-pci ats property

 hw/vfio/pci.h                      |  1 +
 include/system/host_iommu_device.h | 10 ++++
 backends/iommufd.c                 |  8 ++++
 hw/core/machine.c                  |  1 +
 hw/vfio/pci.c                      | 74 ++++++++++++++++++++++++++++--
 5 files changed, 91 insertions(+), 3 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/3 iommufd: Introduce handler for device ATS support

From: Nathan Chen <[email protected]>

Introduce a support_ats() handler that returns whether ATS is supported for a device associated with a host IOMMU device, checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.

Implement this handler in a new function for iommufd.

---
 include/system/host_iommu_device.h | 10 ++++++++++
 backends/iommufd.c                 |  8 ++++++++
 2 files changed, 18 insertions(+)
include/system/host_iommu_device.h +10 -0
index f000301583..5615d6cb57 100644
@@ -133,6 +133,16 @@ struct HostIOMMUDeviceClass {
      * Returns: true on success, false on failure.
      */
     bool (*get_pasid_info)(HostIOMMUDevice *hiod, PasidInfo *pasid_info);
+    /**
+     *  @support_ats: Return whether ATS is supported for the device
+     *  associated with @hiod host IOMMU device, checking if the
+     *  IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.
+     *
+     *  @hiod: handle to the host IOMMU device
+     *
+     *  Returns: true if ATS is supported, false otherwise
+     */
+    bool (*support_ats)(HostIOMMUDevice *hiod);
 };
 
 /*
backends/iommufd.c +8 -0
index 410b044370..7cc6767b4b 100644
@@ -573,6 +573,13 @@ static int hiod_iommufd_get_cap(HostIOMMUDevice *hiod, int cap, Error **errp)
     }
 }
 
+static bool hiod_iommufd_support_ats(HostIOMMUDevice *hiod)
+{
+    HostIOMMUDeviceCaps *caps = &hiod->caps;
+
+    return !(caps->hw_caps & IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED);
+}
+
 static bool hiod_iommufd_get_pasid_info(HostIOMMUDevice *hiod,
                                         PasidInfo *pasid_info)
 {
@@ -595,6 +602,7 @@ static void hiod_iommufd_class_init(ObjectClass *oc, const void *data)
 
     hiodc->get_cap = hiod_iommufd_get_cap;
     hiodc->get_pasid_info = hiod_iommufd_get_pasid_info;
+    hiodc->support_ats = hiod_iommufd_support_ats;
 };
 
 static const TypeInfo types[] = {
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/3] iommufd: Introduce handler for device ATS support

Quoted 20 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Introduce a support_ats() handler that returns whether ATS is supported for a device associated with a host IOMMU device, checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.

Implement this handler in a new function for iommufd.

---
  include/system/host_iommu_device.h | 10 ++++++++++
  backends/iommufd.c                 |  8 ++++++++
  2 files changed, 18 insertions(+)
include/system/host_iommu_device.h +0 -0
index f000301583..5615d6cb57 100644

include/system/host_iommu_device.h struct HostIOMMUDeviceClass { · line 139

Quoted 7 lines
include/system/host_iommu_device.h +4 -0
@@ -133,6 +133,16 @@ struct HostIOMMUDeviceClass {
       * Returns: true on success, false on failure.
       */
      bool (*get_pasid_info)(HostIOMMUDevice *hiod, PasidInfo *pasid_info);
+    /**
+     *  @support_ats: Return whether ATS is supported for the device
+     *  associated with @hiod host IOMMU device, checking if the
+     *  IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.

IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is low level implementation detail, which could differ on a future host IOMMU backend.

I would remove "checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set."

Something like this is more generic :

Returns true if ATS can be used by the device, false if the host IOMMU reports it is unavailable.

Thanks,

C.

Quoted 35 lines
include/system/host_iommu_device.h +6 -0
+     *
+     *  @hiod: handle to the host IOMMU device
+     *
+     *  Returns: true if ATS is supported, false otherwise
+     */
+    bool (*support_ats)(HostIOMMUDevice *hiod);
  };
 
  /*
backends/iommufd.c +8 -0
index 410b044370..7cc6767b4b 100644
@@ -573,6 +573,13 @@ static int hiod_iommufd_get_cap(HostIOMMUDevice *hiod, int cap, Error **errp)
      }
  }
 
+static bool hiod_iommufd_support_ats(HostIOMMUDevice *hiod)
+{
+    HostIOMMUDeviceCaps *caps = &hiod->caps;
+
+    return !(caps->hw_caps & IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED);
+}
+
  static bool hiod_iommufd_get_pasid_info(HostIOMMUDevice *hiod,
                                          PasidInfo *pasid_info)
  {
@@ -595,6 +602,7 @@ static void hiod_iommufd_class_init(ObjectClass *oc, const void *data)
 
      hiodc->get_cap = hiod_iommufd_get_cap;
      hiodc->get_pasid_info = hiod_iommufd_get_pasid_info;
+    hiodc->support_ats = hiod_iommufd_support_ats;
  };
 
  static const TypeInfo types[] = {
#3 lore raw

Re: [PATCH 1/3] iommufd: Introduce handler for device ATS support

Quoted 32 lines · Cédric Le Goater
Quoted 20 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Introduce a support_ats() handler that returns whether ATS is supported for a device associated with a host IOMMU device, checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.

Implement this handler in a new function for iommufd.

---
  include/system/host_iommu_device.h | 10 ++++++++++ backends/iommufd.c                 |  8 ++++++++

  2 files changed, 18 insertions(+)
include/system/host_iommu_device.h +0 -0
index f000301583..5615d6cb57 100644

include/system/host_iommu_device.h struct HostIOMMUDeviceClass { · line 139

Quoted 7 lines
include/system/host_iommu_device.h +4 -0
@@ -133,6 +133,16 @@ struct HostIOMMUDeviceClass {
        * Returns: true on success, false on failure.
        */
       bool (*get_pasid_info)(HostIOMMUDevice *hiod, PasidInfo *pasid_info);
+    /**
+     *  @support_ats: Return whether ATS is supported for the device
+     *  associated with @hiod host IOMMU device, checking if the
+     *  IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.

IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is low level implementation detail, which could differ on a future host IOMMU backend.

Quoted 7 lines

I would remove "checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set."

Something like this is more generic :

  Returns true if ATS can be used by the device, false if the host IOMMU reports it is unavailable.

Ok, I will re-word this comment on the next revision.

Thanks,
Nathan

#4 lore raw

PATCH 2/3 vfio/pci: Add ats property

In reply to Nathan Chen

From: Nathan Chen <[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

---
 hw/vfio/pci.h |  1 +
 hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 72 insertions(+), 3 deletions(-)
hw/vfio/pci.h +1 -0
index c3a1f53d35..b48138870e 100644
@@ -191,6 +191,7 @@ struct VFIOPCIDevice {
     VFIODisplay *dpy;
     Notifier irqchip_change_notifier;
     VFIOPCICPR cpr;
+    OnOffAuto ats;
 };
 
 /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
hw/vfio/pci.c +71 -3

110 lines not on this page · raw message

Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH 2/3] vfio/pci: Add ats property

Quoted 25 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

---
  hw/vfio/pci.h |  1 +
  hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++---
  2 files changed, 72 insertions(+), 3 deletions(-)
hw/vfio/pci.h +0 -0
index c3a1f53d35..b48138870e 100644

hw/vfio/pci.h struct VFIOPCIDevice { · line 194

Quoted 4 lines
hw/vfio/pci.h +1 -0
@@ -191,6 +191,7 @@ struct VFIOPCIDevice {
      VFIODisplay *dpy;
      Notifier irqchip_change_notifier;
      VFIOPCICPR cpr;
+    OnOffAuto ats;

I would move this new property field close to the other property fields in VFIOPCIDevice.

Quoted 21 lines
hw/vfio/pci.h +0 -0
  };
 
  /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
hw/vfio/pci.c +9 -1
index 9c06b25e63..6edb3bf5db 100644
@@ -2546,7 +2546,49 @@ static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)
      return true;
  }
 
-static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
+/*
+ * Determine whether ATS capability should be advertised for @vdev, based on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel's IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED bit.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. Returns true in all other cases.

hw/vfio/pci.c static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) · line 2567

Quoted 10 lines
hw/vfio/pci.c +10 -0
+ */
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev,
+                                                 bool *ats_needed, Error **errp)
+{
+    HostIOMMUDevice *hiod = vdev->vbasedev.hiod;
+    HostIOMMUDeviceClass *hiodc;
+    bool ats_supported;
+
+    if (vdev->ats == ON_OFF_AUTO_OFF) {
+        *ats_needed = false;

Why not set '*ats_needed' to false by default here too. I see it is done in vfio_pci_add_capabilities() but it looks safer.

Quoted 5 lines
hw/vfio/pci.c +5 -0
+        return true;
+    }
+
+    *ats_needed = true;
+    if (!hiod) {

hw/vfio/pci.c static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) · line 2582

Quoted 10 lines
hw/vfio/pci.c +10 -0
+        return true;
+    }
+    hiodc = HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+    if (!hiodc || !hiodc->support_ats) {
+        return true;
+    }
+
+    ats_supported = hiodc->support_ats(hiod);
+    if (vdev->ats == ON_OFF_AUTO_ON && !ats_supported) {
+        error_setg(errp, "vfio: ATS requested but not supported by kernel");

"vfio-pci: ..."

Quoted 62 lines
hw/vfio/pci.c +28 -2
+        *ats_needed = false;
+        return false;
+    }
+
+    *ats_needed = ats_supported;
+    return true;
+}
+
+static void vfio_add_ext_cap(VFIOPCIDevice *vdev, bool ats_needed)
  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
      bool pasid_cap_added = false;
@@ -2635,7 +2677,18 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
           */
          case PCI_EXT_CAP_ID_PASID:
              pasid_cap_added = true;
-            /* fallthrough */
+            pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            break;
+        case PCI_EXT_CAP_ID_ATS:
+            /*
+             * If ATS is requested and supported according to the kernel, add
+             * the ATS capability. If not supported according to the kernel or
+             * disabled on the qemu command line, omit the ATS cap.
+             */
+            if (ats_needed) {
+                pcie_add_capability(pdev, cap_id, cap_ver, next, size);
+            }
+            break;
          default:
              pcie_add_capability(pdev, cap_id, cap_ver, next, size);
          }
@@ -2657,6 +2710,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
  bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp)
  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
+    bool ats_needed = false;
 
      if (!(pdev->config[PCI_STATUS] & PCI_STATUS_CAP_LIST) ||
          !pdev->config[PCI_CAPABILITY_LIST]) {
@@ -2667,7 +2721,11 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp)
          return false;
      }
 
-    vfio_add_ext_cap(vdev);
+    if (!vfio_pci_ats_requested_and_supported(vdev, &ats_needed, errp)) {
+        return false;
+    }
+
+    vfio_add_ext_cap(vdev, ats_needed);
      return true;
  }
 
@@ -3815,6 +3873,7 @@ static const Property vfio_pci_properties[] = {
      DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check, true),
      DEFINE_PROP_UINT16("x-vpasid-cap-offset", VFIOPCIDevice,
                         vpasid_cap_offset, 0),
+    DEFINE_PROP_ON_OFF_AUTO("ats", VFIOPCIDevice, ats, ON_OFF_AUTO_AUTO),
  };
 
  static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp)

hw/vfio/pci.c static void vfio_pci_class_init(ObjectClass *klass, const void *data) · line 4041

Quoted 10 lines
hw/vfio/pci.c +7 -0
@@ -3973,6 +4032,15 @@ static void vfio_pci_class_init(ObjectClass *klass, const void *data)
                                            "a vIOMMU. A value of 0 (default) places the capability at the "
                                            "end of the extended configuration space. The offset must be "
                                            "4-byte aligned and within the PCIe extended configuration space");
+   object_class_property_set_description(klass, /* 11.1 */
+                                          "ats",
+                                          "Control guest visibility of the ATS PCIe extended capability. "
+                                          "Valid values are on, off, and auto (default). "
+                                          "'off' always masks ATS. "
+                                          "'on' requires ATS support for the device and fails realize if the "
+                                          "host kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. "

Again, IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is a kernel UAPI bit and it is a bit too low-level/internal a QEMU for property documentation.

What about :

"'on' requires ATS support for the device and fails realize if the " "host kernel reports ATS as unavailable for this device. " "'auto' masks ATS only when the host kernel reports ATS as unavailable."

?

Thanks,

C.

Quoted 5 lines
hw/vfio/pci.c +2 -0
+                                          "'auto' masks ATS only when the host kernel reports "
+                                          "IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED.");
  }
 
  static const TypeInfo vfio_pci_info = {
#6 lore raw

Re: [PATCH 2/3] vfio/pci: Add ats property

Quoted 21 lines · Cédric Le Goater
Quoted 20 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask the capability so the guest does not see it.

If ATS is explicitly requested but not supported by the kernel, fail device realize.

This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.

---
  hw/vfio/pci.h |  1 +
  hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++---

  2 files changed, 72 insertions(+), 3 deletions(-)
Quoted 13 lines

hw/vfio/pci.h struct VFIOPCIDevice { · line 194

Quoted 9 lines
hw/vfio/pci.h +1 -0
index c3a1f53d35..b48138870e 100644
@@ -191,6 +191,7 @@ struct VFIOPCIDevice {
       VFIODisplay *dpy;
       Notifier irqchip_change_notifier;
       VFIOPCICPR cpr;
+    OnOffAuto ats;

I would move this new property field close to the other property fields in VFIOPCIDevice.

Ok, I will move this closer.

Quoted 29 lines
Quoted 29 lines
hw/vfio/pci.h +0 -0
   };
   /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
hw/vfio/pci.c +2 -1
index 9c06b25e63..6edb3bf5db 100644
@@ -2546,7 +2546,49 @@ static bool 
 vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)
       return true;
   }
-static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
+/*
+ * Determine whether ATS capability should be advertised for @vdev, 

based on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel's IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED bit.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. Returns true in all other cases.
+ */
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev, +                                                 bool *ats_needed, Error **errp)
+{
+    HostIOMMUDevice *hiod = vdev->vbasedev.hiod;

Quoted 9 lines
Quoted 5 lines

+    HostIOMMUDeviceClass *hiodc;
+    bool ats_supported;
+
+    if (vdev->ats == ON_OFF_AUTO_OFF) {
+        *ats_needed = false;

Why not set '*ats_needed' to false by default here too. I see it is done in vfio_pci_add_capabilities() but it looks safer.

Right, I'll set it to false at the top of this function in case it is given a non-initialized ats_needed input.

Quoted 11 lines
Quoted 11 lines

+        return true;
+    }
+
+    *ats_needed = true;
+    if (!hiod) {
+        return true;
+    }
+    hiodc = HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+    if (!hiodc || !hiodc->support_ats) {
+        return true;

Quoted 8 lines
Quoted 6 lines

+    }
+
+    ats_supported = hiodc->support_ats(hiod);
+    if (vdev->ats == ON_OFF_AUTO_ON && !ats_supported) { +        error_setg(errp, "vfio: ATS requested but not supported by kernel");

 "vfio-pci: ..."

Noted, I will update this in the next refresh.

Quoted 83 lines
Quoted 71 lines

+        *ats_needed = false;
+        return false;
+    }

hw/vfio/pci.c +6 -0
+
+    *ats_needed = ats_supported;
+    return true;
+}
+
+static void vfio_add_ext_cap(VFIOPCIDevice *vdev, bool ats_needed)

  {
      PCIDevice *pdev = PCI_DEVICE(vdev);
      bool pasid_cap_added = false;

hw/vfio/pci.c +0 -0
@@ -2635,7 +2677,18 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
            */

          case PCI_EXT_CAP_ID_PASID:
              pasid_cap_added = true;
-            /* fallthrough */
+            pcie_add_capability(pdev, cap_id, cap_ver, next, size); +            break;
+        case PCI_EXT_CAP_ID_ATS:
+            /*
+             * If ATS is requested and supported according to the kernel, add
+             * the ATS capability. If not supported according to the kernel or
+             * disabled on the qemu command line, omit the ATS cap. +             */
+            if (ats_needed) {
+                pcie_add_capability(pdev, cap_id, cap_ver, next, size); +            }
+            break;
          default:
              pcie_add_capability(pdev, cap_id, cap_ver, next, size); }

hw/vfio/pci.c +1 -0
@@ -2657,6 +2710,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)
   bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp)
   {
       PCIDevice *pdev = PCI_DEVICE(vdev);
+    bool ats_needed = false;

      if (!(pdev->config[PCI_STATUS] & PCI_STATUS_CAP_LIST) || !pdev->config[PCI_CAPABILITY_LIST]) {

hw/vfio/pci.c +6 -1
@@ -2667,7 +2721,11 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice 
 *vdev, Error **errp)
           return false;
       }
-    vfio_add_ext_cap(vdev);
+    if (!vfio_pci_ats_requested_and_supported(vdev, &ats_needed, 
 errp)) {
+        return false;
+    }
+
+    vfio_add_ext_cap(vdev, ats_needed);
       return true;
   }
@@ -3815,6 +3873,7 @@ static const Property vfio_pci_properties[] = {
       DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check, true),
       DEFINE_PROP_UINT16("x-vpasid-cap-offset", VFIOPCIDevice,
                          vpasid_cap_offset, 0),
+    DEFINE_PROP_ON_OFF_AUTO("ats", VFIOPCIDevice, ats, 
 ON_OFF_AUTO_AUTO),
   };
   static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp)
@@ -3973,6 +4032,15 @@ static void vfio_pci_class_init(ObjectClass 
 *klass, const void *data)

                                            "a vIOMMU. A value of 0 (default) places the capability at the " "end of the extended configuration space. The offset must be " "4-byte aligned and within the PCIe extended configuration space"); +   object_class_property_set_description(klass, /* 11.1 */
+                                          "ats",
+                                          "Control guest visibility

Quoted 8 lines

of the ATS PCIe extended capability. "
+                                          "Valid values are on, off, and auto (default). "
+                                          "'off' always masks ATS. " +                                          "'on' requires ATS support for the device and fails realize if the "
+                                          "host kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. "

Again, IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is a kernel UAPI bit and it is a bit too low-level/internal a QEMU for property documentation.

Quoted 7 lines

What about :

  "'on' requires ATS support for the device and fails realize if the " "host kernel reports ATS as unavailable for this device. " "'auto' masks ATS only when the host kernel reports ATS as unavailable."

?

Yes I agree, I will update the descriptions here accordingly.

Thanks,
Nathan

#7 lore raw

PATCH 3/3 hw/core/machine: Add new compat for vfio-pci ats property

In reply to Nathan Chen

From: Nathan Chen <[email protected]>

Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".

---
 hw/core/machine.c | 1 +
 1 file changed, 1 insertion(+)
hw/core/machine.c +1 -0
index 4d8b15d99e..c0cdf7c9da 100644
@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = {
     { "chardev-vc", "encoding", "cp437" },
     { "tpm-crb", "cap-chunk", "off" },
     { "tpm-crb", "x-allow-chunk-migration", "off" },
+    { "vfio-pci", "ats", "on" },
 };
 const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
 
Signature · 1 line
2.43.0
Reviewed-by on v1 3/3
#8 lore raw

Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property

Quoted 15 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".

---
  hw/core/machine.c | 1 +
  1 file changed, 1 insertion(+)
hw/core/machine.c +0 -0
index 4d8b15d99e..c0cdf7c9da 100644

hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 49

Quoted 7 lines
hw/core/machine.c +1 -0
@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = {
      { "chardev-vc", "encoding", "cp437" },
      { "tpm-crb", "cap-chunk", "off" },
      { "tpm-crb", "x-allow-chunk-migration", "off" },
+    { "vfio-pci", "ats", "on" },
  };
  const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);
 

Thanks,

C.

#9 lore raw

Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property

In reply to Nathan Chen

Hi Nathan,

Quoted 15 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".

---
  hw/core/machine.c | 1 +
  1 file changed, 1 insertion(+)
hw/core/machine.c +0 -0
index 4d8b15d99e..c0cdf7c9da 100644

hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 48

Quoted 6 lines
hw/core/machine.c +1 -0
@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = {
      { "chardev-vc", "encoding", "cp437" },
      { "tpm-crb", "cap-chunk", "off" },
      { "tpm-crb", "x-allow-chunk-migration", "off" },
+    { "vfio-pci", "ats", "on" },
  };
  const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);

Shouldn't this be squashed with previous patch to keep bisectability?

#10 lore raw

Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property

Quoted 13 lines · Philippe Mathieu-Daudé

Hi Nathan,

Quoted 10 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".

---
  hw/core/machine.c | 1 +

  1 file changed, 1 insertion(+)
Quoted 13 lines

hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 48

Quoted 11 lines
hw/core/machine.c +1 -0
index 4d8b15d99e..c0cdf7c9da 100644
@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = {
       { "chardev-vc", "encoding", "cp437" },
       { "tpm-crb", "cap-chunk", "off" },
       { "tpm-crb", "x-allow-chunk-migration", "off" },
+    { "vfio-pci", "ats", "on" },
   };
   const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);

Shouldn't this be squashed with previous patch to keep bisectability?

Indeed. Patch 3 restores the old "always expose ATS" semantics for <machine>-11.0 and older. With patch 2, ATS could be exposed OFF. My bad.

Thanks,

C.

#11 lore raw

Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property

In reply to Philippe Mathieu-Daudé

Hi Philippe,

Quoted 13 lines · Philippe Mathieu-Daudé

Hi Nathan,

Quoted 10 lines · Nathan Chen

From: Nathan Chen <[email protected]>

Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".

---
  hw/core/machine.c | 1 +

  1 file changed, 1 insertion(+)
Quoted 13 lines

hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 48

Quoted 11 lines
hw/core/machine.c +1 -0
index 4d8b15d99e..c0cdf7c9da 100644
@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = {
       { "chardev-vc", "encoding", "cp437" },
       { "tpm-crb", "cap-chunk", "off" },
       { "tpm-crb", "x-allow-chunk-migration", "off" },
+    { "vfio-pci", "ats", "on" },
   };
   const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);

Shouldn't this be squashed with previous patch to keep bisectability?

Yes, I will squash this with patch 2 to keep bisectability.

Thanks,
Nathan

#12 lore raw
In reply to Nathan Chen

Hello Nathan,

Quoted 25 lines · Nathan Chen

Hi,

This series adds an ATS control property to vfio-pci and wires iommufd support reporting so vfio-pci can decide whether to expose the ATS PCIe extended capability to the guest. This series is a follow-up to the previous patch included in the "hw/arm/smmuv3-accel: Resolve AUTO properties" series [0].

The new vfio-pci property is:
ats=on|off|auto (default: auto)

Behavior:
- off: always masks ATS from guest view
- on: requires ATS support; fails realize if kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- auto: masks ATS only when kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED

To preserve machine compatibility for older machine types, this series adds:

   { "vfio-pci", "ats", "on" } to hw_compat_11_0.

Please note that no linux-headers sync patch is included in this series because IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is already present in linux-headers/linux/iommufd.h in the current base.

Quoted 12 lines

Testing:
Verified ATS capability visibility changes with vfio-pci ats=off/on/auto on an NVIDIA Grace system:
- ats=on + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=auto + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=on + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => realize failure
- ats=auto + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => ATS masked
- ats=off => ATS masked

What about ats=on and no ATS support in the physical device ?

I agree it is a bit of a corner case. A warning would be nice, it can be addressed as a follow up change.

Thanks,

C.

Quoted 31 lines

Changes from "Resolve AUTO properties" series:
- Split into separate patches for support_ats() handler introduction and iommufd implementation, "ats" property addition, and compat property fix
- Rebase on vfio-next
- Remove unnecessary includes for hw/vfio/pci.c
- Rename ats_need to ats_needed for vfio_pci_ats_requested_and_supported()
- Move the vfio_pci_ats_requested_and_supported() call to vfio_pci_add_capabilities() so we can propagate the Error **, and then pass ats_needed as a new input arg to vfio_add_ext_cap
- Fix indent error in vfio_add_ext_cap()
- Break and add a pcie_add_capability() call for the PASID case in vfio_add_ext_cap()
- Update comment about return value for support_ats()

Thanks,
Nathan

[0] https://lore.kernel.org/qemu-devel/[email protected]/

Nathan Chen (3):
iommufd: Introduce handler for device ATS support vfio/pci: Add ats property
hw/core/machine: Add new compat for vfio-pci ats property

  hw/vfio/pci.h                      |  1 +
  include/system/host_iommu_device.h | 10 ++++
  backends/iommufd.c                 |  8 ++++
  hw/core/machine.c                  |  1 +
  hw/vfio/pci.c                      | 74 ++++++++++++++++++++++++++++--
  5 files changed, 91 insertions(+), 3 deletions(-)
#13 lore raw

Hi Cédric,

Quoted 41 lines · Cédric Le Goater

Hello Nathan,

Quoted 25 lines · Nathan Chen

Hi,

This series adds an ATS control property to vfio-pci and wires iommufd support reporting so vfio-pci can decide whether to expose the ATS PCIe extended capability to the guest. This series is a follow-up to the previous patch included in the "hw/arm/smmuv3-accel: Resolve AUTO properties" series [0].

The new vfio-pci property is:
   ats=on|off|auto (default: auto)

Behavior:
- off: always masks ATS from guest view
- on: requires ATS support; fails realize if kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- auto: masks ATS only when kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED

To preserve machine compatibility for older machine types, this series adds:
   { "vfio-pci", "ats", "on" } to hw_compat_11_0.

Please note that no linux-headers sync patch is included in this series because IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is already present in linux-headers/linux/iommufd.h in the current base.

Quoted 12 lines

Testing:
Verified ATS capability visibility changes with vfio-pci ats=off/on/auto on an NVIDIA Grace system:
- ats=on + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=auto + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=on + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => realize failure
- ats=auto + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => ATS masked
- ats=off => ATS masked

Quoted 4 lines

What about ats=on and no ATS support in the physical device ?

I agree it is a bit of a corner case. A warning would be nice, it can be addressed as a follow up change.

Right, I had missed that case. I will look into accounting for it in the next refresh.

Thanks,
Nathan