vfio/pci: add vfio-pci ATS control property
cover vfio/pci: add vfio-pci ATS control property
Hi,
This series adds an ATS control property to vfio-pci and wires iommufd support reporting so vfio-pci can decide whether to expose the ATS PCIe extended capability to the guest. This series is a follow-up to the previous patch included in the "hw/arm/smmuv3-accel: Resolve AUTO properties" series [0].
The new vfio-pci property is:
ats=on|off|auto (default: auto)
Behavior:
- off: always masks ATS from guest view
- on: requires ATS support; fails realize if kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- auto: masks ATS only when kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
To preserve machine compatibility for older machine types, this series adds:
{ "vfio-pci", "ats", "on" } to hw_compat_11_0.
Please note that no linux-headers sync patch is included in this series because IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is already present in linux-headers/linux/iommufd.h in the current base.
Testing:
Verified ATS capability visibility changes with vfio-pci ats=off/on/auto on an NVIDIA Grace system:
- ats=on + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=auto + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=on + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => realize failure
- ats=auto + hiod_iommufd_support_ats modified to return false (mimic kernel-reported unsupported ATS) => ATS masked
- ats=off => ATS masked
Changes from "Resolve AUTO properties" series:
- Split into separate patches for support_ats() handler introduction and iommufd implementation, "ats" property addition, and compat property fix
- Rebase on vfio-next
- Remove unnecessary includes for hw/vfio/pci.c
- Rename ats_need to ats_needed for vfio_pci_ats_requested_and_supported()
- Move the vfio_pci_ats_requested_and_supported() call to vfio_pci_add_capabilities() so we can propagate the Error **, and then pass ats_needed as a new input arg to vfio_add_ext_cap
- Fix indent error in vfio_add_ext_cap()
- Break and add a pcie_add_capability() call for the PASID case in vfio_add_ext_cap()
- Update comment about return value for support_ats()
Thanks,
Nathan
[0] https://lore.kernel.org/qemu-devel/[email protected]/
Nathan Chen (3):
iommufd: Introduce handler for device ATS support vfio/pci: Add ats property
hw/core/machine: Add new compat for vfio-pci ats property
hw/vfio/pci.h | 1 + include/system/host_iommu_device.h | 10 ++++ backends/iommufd.c | 8 ++++ hw/core/machine.c | 1 + hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++-- 5 files changed, 91 insertions(+), 3 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/3 iommufd: Introduce handler for device ATS support
From: Nathan Chen <[email protected]>
Introduce a support_ats() handler that returns whether ATS is supported for a device associated with a host IOMMU device, checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.
Implement this handler in a new function for iommufd.
--- include/system/host_iommu_device.h | 10 ++++++++++ backends/iommufd.c | 8 ++++++++ 2 files changed, 18 insertions(+)
include/system/host_iommu_device.h +10 -0
@@ -133,6 +133,16 @@ struct HostIOMMUDeviceClass {* Returns: true on success, false on failure. */ bool (*get_pasid_info)(HostIOMMUDevice *hiod,PasidInfo *pasid_info); +/** +* @support_ats: Return whether ATS is supported for the device +* associated with @hiod host IOMMU device, checking if the +* IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set. +* +* @hiod: handle to the host IOMMU device +* +* Returns: true if ATS is supported, false otherwise +*/ +bool (*support_ats)(HostIOMMUDevice *hiod); };/*
backends/iommufd.c +8 -0
@@ -573,6 +573,13 @@ static int hiod_iommufd_get_cap(HostIOMMUDevice *hiod, int cap, Error **errp) } } +static bool hiod_iommufd_support_ats (HostIOMMUDevice *hiod) +{ +HostIOMMUDeviceCaps *caps = &hiod->caps; + +return !(caps->hw_caps &IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED ); +} +static bool hiod_iommufd_get_pasid_info (HostIOMMUDevice *hiod,PasidInfo *pasid_info) {
@@ -595,6 +602,7 @@ static void hiod_iommufd_class_init(ObjectClass *oc, const void *data) hiodc->get_cap = hiod_iommufd_get_cap; hiodc->get_pasid_info = hiod_iommufd_get_pasid_info; + hiodc->support_ats = hiod_iommufd_support_ats; };static const TypeInfo types[] = {
Signature · 1 line
2.43.0
Re: [PATCH 1/3] iommufd: Introduce handler for device ATS support
Quoted 20 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Introduce a
support_ats()handler that returns whether ATS is supported for a device associated with a host IOMMU device, checking if theIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDcapability bit is set.Implement this handler in a new function for iommufd.
Suggested-by Shameer Kolothum <[email protected]>Signed-off-by Nathan Chen <[email protected]>--- include/system/host_iommu_device.h | 10 ++++++++++ backends/iommufd.c | 8 ++++++++ 2 files changed, 18 insertions(+)include/system/host_iommu_device.h +0
-0
include/system/host_iommu_device.h struct HostIOMMUDeviceClass { · line 139
Quoted 7 lines
include/system/host_iommu_device.h +4
-0@@ -133,6 +133,16 @@ struct HostIOMMUDeviceClass {* Returns: true on success, false on failure. */ bool (*get_pasid_info)(HostIOMMUDevice *hiod,PasidInfo *pasid_info); +/** +* @support_ats: Return whether ATS is supported for the device +* associated with @hiod host IOMMU device, checking if the +* IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is low level implementation detail, which could differ on a future host IOMMU backend.
I would remove "checking if the IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set."
Something like this is more generic :
Returns true if ATS can be used by the device, false if the host IOMMU reports it is unavailable.
Thanks,
C.
Quoted 35 lines
include/system/host_iommu_device.h +6
-0+* +* @hiod: handle to the host IOMMU device +* +* Returns: true if ATS is supported, false otherwise +*/ +bool (*support_ats)(HostIOMMUDevice *hiod); };/* backends/iommufd.c +8
-0@@ -573,6 +573,13 @@ static int hiod_iommufd_get_cap(HostIOMMUDevice *hiod, int cap, Error **errp) } } +static bool hiod_iommufd_support_ats (HostIOMMUDevice *hiod) +{ +HostIOMMUDeviceCaps *caps = &hiod->caps; + +return !(caps->hw_caps &IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED ); +} +static bool hiod_iommufd_get_pasid_info (HostIOMMUDevice *hiod,PasidInfo *pasid_info) {@@ -595,6 +602,7 @@ static void hiod_iommufd_class_init(ObjectClass *oc, const void *data) hiodc->get_cap = hiod_iommufd_get_cap; hiodc->get_pasid_info = hiod_iommufd_get_pasid_info; + hiodc->support_ats = hiod_iommufd_support_ats; };static const TypeInfo types[] = {
Re: [PATCH 1/3] iommufd: Introduce handler for device ATS support
Quoted 32 lines · Cédric Le Goater
Quoted 20 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Introduce a
support_ats()handler that returns whether ATS is supported for a device associated with a host IOMMU device, checking if theIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDcapability bit is set.Implement this handler in a new function for iommufd.
Suggested-by Shameer Kolothum <[email protected]>Signed-off-by Nathan Chen <[email protected]>---
include/system/host_iommu_device.h| 10 ++++++++++ backends/iommufd.c | 8 ++++++++2 files changed, 18 insertions(+)include/system/host_iommu_device.h +0
-0include/system/host_iommu_device.h struct HostIOMMUDeviceClass { · line 139
Quoted 7 lines
include/system/host_iommu_device.h +4
-0@@ -133,6 +133,16 @@ struct HostIOMMUDeviceClass {* Returns: true on success, false on failure. */ bool (*get_pasid_info)(HostIOMMUDevice *hiod, PasidInfo *pasid_info); + /** +* @support_ats: Return whether ATS is supported for the device +* associated with @hiod host IOMMU device, checking if the +* IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED capability bit is set.
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDis low level implementation detail, which could differ on a future host IOMMU backend.
Quoted 7 lines
I would remove "checking if the
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDcapability bit is set."Something like this is more generic :
Returns true if ATS can be used by the device, false if the host IOMMU reports it is unavailable.
Ok, I will re-word this comment on the next revision.
Thanks,
Nathan
PATCH 2/3 vfio/pci: Add ats property
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask the capability so the guest does not see it.
If ATS is explicitly requested but not supported by the kernel, fail device realize.
This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.
--- hw/vfio/pci.h | 1 + hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 72 insertions(+), 3 deletions(-)
hw/vfio/pci.h +1 -0
@@ -191,6 +191,7 @@ struct VFIOPCIDevice {VFIODisplay *dpy;Notifier irqchip_change_notifier;VFIOPCICPR cpr; +OnOffAuto ats; };/* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */
hw/vfio/pci.c +71 -3
110 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH 2/3] vfio/pci: Add ats property
Quoted 25 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask the capability so the guest does not see it.If ATS is explicitly requested but not supported by the kernel, fail device realize.
This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.
Suggested-by Shameer Kolothum <[email protected]>Signed-off-by Nathan Chen <[email protected]>--- hw/vfio/pci.h | 1 + hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 72 insertions(+), 3 deletions(-)hw/vfio/pci.h +0
-0
hw/vfio/pci.h struct VFIOPCIDevice { · line 194
Quoted 4 lines
hw/vfio/pci.h +1
-0@@ -191,6 +191,7 @@ struct VFIOPCIDevice {VFIODisplay *dpy;Notifier irqchip_change_notifier;VFIOPCICPR cpr; +OnOffAuto ats;
I would move this new property field close to the other property fields in VFIOPCIDevice.
Quoted 21 lines
hw/vfio/pci.h +0
-0};/* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ hw/vfio/pci.c +9
-1@@ -2546,7 +2546,49 @@ static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp)return true; } -static void vfio_add_ext_cap (VFIOPCIDevice *vdev) +/* +* Determine whether ATS capability should be advertised for @vdev, based on +* whether it was enabled on the command line and whether it is supported +* according to the kernel's IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED bit. +* +* Store whether ATS capability should be advertised in @ats_needed. +* +* Returns false only when ats=on is explicitly requested but the kernel +* reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. Returns true in all other cases.
hw/vfio/pci.c static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) · line 2567
Quoted 10 lines
hw/vfio/pci.c +10
-0+*/ +static bool vfio_pci_ats_requested_and_supported (VFIOPCIDevice *vdev, +bool *ats_needed,Error **errp) +{ +HostIOMMUDevice *hiod = vdev->vbasedev.hiod; +HostIOMMUDeviceClass *hiodc; +bool ats_supported; + +if (vdev->ats ==ON_OFF_AUTO_OFF ) { + *ats_needed = false;
Why not set '*ats_needed' to false by default here too. I see it is done in vfio_pci_add_capabilities() but it looks safer.
Quoted 5 lines
hw/vfio/pci.c +5
-0+return true; + } + + *ats_needed = true; +if (!hiod) {
hw/vfio/pci.c static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) · line 2582
Quoted 10 lines
hw/vfio/pci.c +10
-0+return true; + } + hiodc =HOST_IOMMU_DEVICE_GET_CLASS (hiod); +if (!hiodc || !hiodc->support_ats) { +return true; + } + + ats_supported = hiodc->support_ats (hiod); +if (vdev->ats ==ON_OFF_AUTO_ON && !ats_supported) { +error_setg (errp,"vfio: ATS requested but not supported by kernel" );
"vfio-pci: ..."
Quoted 62 lines
hw/vfio/pci.c +28
-2+ *ats_needed = false; +return false; + } + + *ats_needed = ats_supported; +return true; +} + +static void vfio_add_ext_cap (VFIOPCIDevice *vdev,bool ats_needed) {PCIDevice *pdev =PCI_DEVICE (vdev);bool pasid_cap_added = false;@@ -2635,7 +2677,18 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)*/ case PCI_EXT_CAP_ID_PASID : pasid_cap_added = true; -/* fallthrough */ +pcie_add_capability (pdev, cap_id, cap_ver, next, size); +break ; +case PCI_EXT_CAP_ID_ATS : +/* +* If ATS is requested and supported according to the kernel, add +* the ATS capability. If not supported according to the kernel or +* disabled on the qemu command line, omit the ATS cap. +*/ +if (ats_needed) { +pcie_add_capability (pdev, cap_id, cap_ver, next, size); + } +break ;default :pcie_add_capability (pdev, cap_id, cap_ver, next, size); }@@ -2657,6 +2710,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)bool vfio_pci_add_capabilities (VFIOPCIDevice *vdev,Error **errp) {PCIDevice *pdev =PCI_DEVICE (vdev); +bool ats_needed = false;if (!(pdev->config[PCI_STATUS ] &PCI_STATUS_CAP_LIST ) || !pdev->config[PCI_CAPABILITY_LIST ]) {@@ -2667,7 +2721,11 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp)return false; } -vfio_add_ext_cap (vdev); +if (!vfio_pci_ats_requested_and_supported (vdev, &ats_needed, errp)) { +return false; + } + +vfio_add_ext_cap (vdev, ats_needed);return true; }@@ -3815,6 +3873,7 @@ static const Property vfio_pci_properties[] = {DEFINE_PROP_BOOL ("skip-vsc-check" , VFIOPCIDevice, skip_vsc_check, true),DEFINE_PROP_UINT16 ("x-vpasid-cap-offset" , VFIOPCIDevice, vpasid_cap_offset,0 ), + DEFINE_PROP_ON_OFF_AUTO("ats", VFIOPCIDevice, ats, ON_OFF_AUTO_AUTO), };static void vfio_pci_set_fd (Object *obj,const char *str,Error **errp)
hw/vfio/pci.c static void vfio_pci_class_init(ObjectClass *klass, const void *data) · line 4041
Quoted 10 lines
hw/vfio/pci.c +7
-0@@ -3973,6 +4032,15 @@ static void vfio_pci_class_init(ObjectClass *klass, const void *data)"a vIOMMU. A value of 0 (default) places the capability at the " "end of the extended configuration space. The offset must be " "4-byte aligned and within the PCIe extended configuration space"); +object_class_property_set_description (klass,/* 11.1 */ +"ats" , +"Control guest visibility of the ATS PCIe extended capability. " +"Valid values are on, off, and auto (default). " +"'off' always masks ATS. " +"'on' requires ATS support for the device and fails realize if the " + "host kernel reports IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. "
Again, IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED is a kernel UAPI bit and it is a bit too low-level/internal a QEMU for property documentation.
What about :
"'on' requires ATS support for the device and fails realize if the " "host kernel reports ATS as unavailable for this device. " "'auto' masks ATS only when the host kernel reports ATS as unavailable."
?
Thanks,
C.
Quoted 5 lines
hw/vfio/pci.c +2
-0+"'auto' masks ATS only when the host kernel reports " + "IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED."); }static const TypeInfo vfio_pci_info = {
Re: [PATCH 2/3] vfio/pci: Add ats property
Quoted 21 lines · Cédric Le Goater
Quoted 20 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Add an "ats" OnOffAuto property to vfio-pci. When the device has an ATS extended capability in config space but we should not expose it (ats=off, or ats=auto and kernel reports
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED), mask the capability so the guest does not see it.If ATS is explicitly requested but not supported by the kernel, fail device realize.
This aligns with the kernel's per-device effective ATS reporting and allows vfio-pci to mask ATS when the host kernel reports ATS as unsupported.
Suggested-by Shameer Kolothum <[email protected]>Signed-off-by Nathan Chen <[email protected]>---
hw/vfio/pci.h| 1 +
hw/vfio/pci.c| 74 ++++++++++++++++++++++++++++++++++++++++++++++++---2 files changed, 72 insertions(+), 3 deletions(-)
Quoted 13 lines
hw/vfio/pci.h struct VFIOPCIDevice { · line 194
Quoted 9 lines
hw/vfio/pci.h +1
-0@@ -191,6 +191,7 @@ struct VFIOPCIDevice { VFIODisplay *dpy; Notifier irqchip_change_notifier; VFIOPCICPR cpr; + OnOffAuto ats;I would move this new property field close to the other property fields in VFIOPCIDevice.
Ok, I will move this closer.
Quoted 29 lines
Quoted 29 lines
hw/vfio/pci.h +0
-0}; /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */hw/vfio/pci.c +2
-1@@ -2546,7 +2546,49 @@ static bool vfio_pci_synthesize_pasid_cap(VFIOPCIDevice *vdev, Error **errp) return true; } -static void vfio_add_ext_cap (VFIOPCIDevice *vdev) +/* +* Determine whether ATS capability should be advertised for @vdev, based on
+ * whether it was enabled on the command line and whether it is supported
+ * according to the kernel'sIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDbit.
+ *
+ * Store whether ATS capability should be advertised in @ats_needed.
+ *
+ * Returns false only when ats=on is explicitly requested but the kernel
+ * reportsIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. Returns true in all other cases.
+ */
+static bool vfio_pci_ats_requested_and_supported(VFIOPCIDevice *vdev, + bool*ats_needed, Error **errp)
+{
+ HostIOMMUDevice *hiod =vdev->vbasedev.hiod;
Quoted 9 lines
Quoted 5 lines
+ HostIOMMUDeviceClass *hiodc;
+ boolats_supported;
+
+ if (vdev->ats==ON_OFF_AUTO_OFF) {
+*ats_needed= false;Why not set '
*ats_needed' to false by default here too. I see it is done invfio_pci_add_capabilities()but it looks safer.
Right, I'll set it to false at the top of this function in case it is given a non-initialized ats_needed input.
Quoted 11 lines
Quoted 11 lines
+ return true;
+ }
+
+*ats_needed= true;
+ if (!hiod) {
+ return true;
+ }
+ hiodc =HOST_IOMMU_DEVICE_GET_CLASS(hiod);
+ if (!hiodc || !hiodc->support_ats) {
+ return true;
Quoted 8 lines
Quoted 6 lines
+ }
+
+ats_supported=hiodc->support_ats(hiod);
+ if (vdev->ats==ON_OFF_AUTO_ON&& !ats_supported) { +error_setg(errp, "vfio: ATS requested but not supported by kernel");"vfio-pci: ..."
Noted, I will update this in the next refresh.
Quoted 83 lines
Quoted 71 lines
+
*ats_needed= false;
+ return false;
+ }hw/vfio/pci.c +6
-0+ + *ats_needed = ats_supported; + return true; +} + +static void vfio_add_ext_cap (VFIOPCIDevice *vdev,bool ats_needed){
PCIDevice *pdev =PCI_DEVICE(vdev);
boolpasid_cap_added= false;hw/vfio/pci.c +0
-0@@ -2635,7 +2677,18 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)*/ case
PCI_EXT_CAP_ID_PASID:
pasid_cap_added= true;
- /* fallthrough */
+pcie_add_capability(pdev, cap_id, cap_ver, next, size); + break;
+ casePCI_EXT_CAP_ID_ATS:
+ /*
+ * If ATS is requested and supported according to the kernel, add
+ * the ATS capability. If not supported according to the kernel or
+ * disabled on the qemu command line, omit the ATS cap. + */
+ if (ats_needed) {
+pcie_add_capability(pdev, cap_id, cap_ver, next, size); + }
+ break;
default:
pcie_add_capability(pdev, cap_id, cap_ver, next, size); }hw/vfio/pci.c +1
-0@@ -2657,6 +2710,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev) bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp) { PCIDevice *pdev = PCI_DEVICE(vdev); + bool ats_needed = false;if (!(pdev->config[PCI_STATUS] &
PCI_STATUS_CAP_LIST) || !pdev->config[PCI_CAPABILITY_LIST]) {hw/vfio/pci.c +6
-1@@ -2667,7 +2721,11 @@ bool vfio_pci_add_capabilities(VFIOPCIDevice *vdev, Error **errp) return false; } - vfio_add_ext_cap(vdev); + if (!vfio_pci_ats_requested_and_supported(vdev, &ats_needed, errp)) { + return false; + } + + vfio_add_ext_cap(vdev, ats_needed); return true; }@@ -3815,6 +3873,7 @@ static const Property vfio_pci_properties[] = { DEFINE_PROP_BOOL("skip-vsc-check", VFIOPCIDevice, skip_vsc_check, true), DEFINE_PROP_UINT16("x-vpasid-cap-offset", VFIOPCIDevice, vpasid_cap_offset, 0), + DEFINE_PROP_ON_OFF_AUTO("ats", VFIOPCIDevice, ats,ON_OFF_AUTO_AUTO ), }; static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp)@@ -3973,6 +4032,15 @@ static void vfio_pci_class_init(ObjectClass *klass, const void *data)"a vIOMMU. A value of 0 (default) places the capability at the " "end of the extended configuration space. The offset must be " "4-byte aligned and within the PCIe extended configuration space"); + object_class_property_set_description(klass, /* 11.1 */
+ "ats",
+ "Control guest visibilityQuoted 8 lines
of the ATS PCIe extended capability. "
+ "Valid values are on, off, and auto (default). "
+ "'off' always masks ATS. " + "'on' requires ATS support for the device and fails realize if the "
+ "host kernel reportsIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED. "Again,
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDis a kernel UAPI bit and it is a bit too low-level/internal a QEMU for property documentation.
Quoted 7 lines
What about :
"'on' requires ATS support for the device and fails realize if the " "host kernel reports ATS as unavailable for this device. " "'auto' masks ATS only when the host kernel reports ATS as unavailable."
?
Yes I agree, I will update the descriptions here accordingly.
Thanks,
Nathan
PATCH 3/3 hw/core/machine: Add new compat for vfio-pci ats property
In reply to Nathan ChenFrom: Nathan Chen <[email protected]>
Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".
--- hw/core/machine.c | 1 + 1 file changed, 1 insertion(+)
hw/core/machine.c +1 -0
@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = { { "chardev-vc", "encoding", "cp437" }, { "tpm-crb", "cap-chunk", "off" }, { "tpm-crb", "x-allow-chunk-migration", "off" }, + { "vfio-pci", "ats", "on" }, };const size_t hw_compat_11_0_len =G_N_ELEMENTS (hw_compat_11_0);
Signature · 1 line
2.43.0
Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property
Quoted 15 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".
Signed-off-by Nathan Chen <[email protected]>--- hw/core/machine.c | 1 + 1 file changed, 1 insertion(+)hw/core/machine.c +0
-0
hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 49
Quoted 7 lines
hw/core/machine.c +1
-0@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = { { "chardev-vc", "encoding", "cp437" }, { "tpm-crb", "cap-chunk", "off" }, { "tpm-crb", "x-allow-chunk-migration", "off" }, + { "vfio-pci", "ats", "on" }, };const size_t hw_compat_11_0_len =G_N_ELEMENTS (hw_compat_11_0);
Thanks,
C.
Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property
In reply to Nathan ChenHi Nathan,
Quoted 15 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".
Signed-off-by Nathan Chen <[email protected]>--- hw/core/machine.c | 1 + 1 file changed, 1 insertion(+)hw/core/machine.c +0
-0
hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 48
Quoted 6 lines
hw/core/machine.c +1
-0@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = { { "chardev-vc", "encoding", "cp437" }, { "tpm-crb", "cap-chunk", "off" }, { "tpm-crb", "x-allow-chunk-migration", "off" }, + { "vfio-pci", "ats", "on" }, };const size_t hw_compat_11_0_len =G_N_ELEMENTS (hw_compat_11_0);
Shouldn't this be squashed with previous patch to keep bisectability?
Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property
Quoted 13 lines · Philippe Mathieu-Daudé
Hi Nathan,
Quoted 10 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".
Signed-off-by Nathan Chen <[email protected]>---
hw/core/machine.c| 1 +1 file changed, 1 insertion(+)
Quoted 13 lines
hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 48
Quoted 11 lines
hw/core/machine.c +1
-0@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = { { "chardev-vc", "encoding", "cp437" }, { "tpm-crb", "cap-chunk", "off" }, { "tpm-crb", "x-allow-chunk-migration", "off" }, + { "vfio-pci", "ats", "on" }, }; const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);Shouldn't this be squashed with previous patch to keep bisectability?
Indeed. Patch 3 restores the old "always expose ATS" semantics for <machine>-11.0 and older. With patch 2, ATS could be exposed OFF. My bad.
Thanks,
C.
Re: [PATCH 3/3] hw/core/machine: Add new compat for vfio-pci ats property
In reply to Philippe Mathieu-DaudéHi Philippe,
Quoted 13 lines · Philippe Mathieu-Daudé
Hi Nathan,
Quoted 10 lines · Nathan Chen
From: Nathan Chen <[email protected]>
Introduce a new compat for the changed default for vfio-pci device ATS property, from "on" to "auto".
Signed-off-by Nathan Chen <[email protected]>---
hw/core/machine.c| 1 +1 file changed, 1 insertion(+)
Quoted 13 lines
hw/core/machine.c GlobalProperty hw_compat_11_0[] = { · line 48
Quoted 11 lines
hw/core/machine.c +1
-0@@ -43,6 +43,7 @@ GlobalProperty hw_compat_11_0[] = { { "chardev-vc", "encoding", "cp437" }, { "tpm-crb", "cap-chunk", "off" }, { "tpm-crb", "x-allow-chunk-migration", "off" }, + { "vfio-pci", "ats", "on" }, }; const size_t hw_compat_11_0_len = G_N_ELEMENTS(hw_compat_11_0);Shouldn't this be squashed with previous patch to keep bisectability?
Yes, I will squash this with patch 2 to keep bisectability.
Thanks,
Nathan
Hello Nathan,
Quoted 25 lines · Nathan Chen
Hi,
This series adds an ATS control property to vfio-pci and wires iommufd support reporting so vfio-pci can decide whether to expose the ATS PCIe extended capability to the guest. This series is a follow-up to the previous patch included in the "hw/arm/smmuv3-accel: Resolve AUTO properties" series [0].
The new vfio-pci property is:
ats=on|off|auto (default: auto)Behavior:
- off: always masks ATS from guest view
- on: requires ATS support; fails realize if kernel reportsIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- auto: masks ATS only when kernel reportsIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDTo preserve machine compatibility for older machine types, this series adds:
{ "vfio-pci", "ats", "on" } to hw_compat_11_0.Please note that no linux-headers sync patch is included in this series because
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDis already present inlinux-headers/linux/iommufd.hin the current base.
Quoted 12 lines
Testing:
Verified ATS capability visibility changes with vfio-pci ats=off/on/auto on an NVIDIA Grace system:
- ats=on + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=auto + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=on +hiod_iommufd_support_atsmodified to return false (mimic kernel-reported unsupported ATS) => realize failure
- ats=auto +hiod_iommufd_support_atsmodified to return false (mimic kernel-reported unsupported ATS) => ATS masked
- ats=off => ATS masked
What about ats=on and no ATS support in the physical device ?
I agree it is a bit of a corner case. A warning would be nice, it can be addressed as a follow up change.
Thanks,
C.
Quoted 31 lines
Changes from "Resolve AUTO properties" series:
- Split into separate patches forsupport_ats()handler introduction and iommufd implementation, "ats" property addition, and compat property fix
- Rebase on vfio-next
- Remove unnecessary includes forhw/vfio/pci.c
- Renameats_needtoats_neededforvfio_pci_ats_requested_and_supported()
- Move thevfio_pci_ats_requested_and_supported()call tovfio_pci_add_capabilities()so we can propagate the Error **, and then passats_neededas a new input arg tovfio_add_ext_cap
- Fix indent error invfio_add_ext_cap()
- Break and add apcie_add_capability()call for the PASID case invfio_add_ext_cap()
- Update comment about return value forsupport_ats()Thanks,
Nathan[0] https://lore.kernel.org/qemu-devel/[email protected]/
Nathan Chen (3):
iommufd: Introduce handler for device ATS support vfio/pci: Add ats property
hw/core/machine: Add new compat for vfio-pci ats propertyhw/vfio/pci.h | 1 + include/system/host_iommu_device.h | 10 ++++ backends/iommufd.c | 8 ++++ hw/core/machine.c | 1 + hw/vfio/pci.c | 74 ++++++++++++++++++++++++++++-- 5 files changed, 91 insertions(+), 3 deletions(-)
Hi Cédric,
Quoted 41 lines · Cédric Le Goater
Hello Nathan,
Quoted 25 lines · Nathan Chen
Hi,
This series adds an ATS control property to vfio-pci and wires iommufd support reporting so vfio-pci can decide whether to expose the ATS PCIe extended capability to the guest. This series is a follow-up to the previous patch included in the "hw/arm/smmuv3-accel: Resolve AUTO properties" series [0].
The new vfio-pci property is:
ats=on|off|auto (default: auto)Behavior:
- off: always masks ATS from guest view
- on: requires ATS support; fails realize if kernel reportsIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED
- auto: masks ATS only when kernel reportsIOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDTo preserve machine compatibility for older machine types, this series adds:
{ "vfio-pci", "ats", "on" } tohw_compat_11_0.Please note that no linux-headers sync patch is included in this series because
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTEDis already present inlinux-headers/linux/iommufd.hin the current base.Quoted 12 lines
Testing:
Verified ATS capability visibility changes with vfio-pci ats=off/on/auto on an NVIDIA Grace system:
- ats=on + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=auto + ATS supported by kernel => propagate ATS extended cap to guest view
- ats=on +hiod_iommufd_support_atsmodified to return false (mimic kernel-reported unsupported ATS) => realize failure
- ats=auto +hiod_iommufd_support_atsmodified to return false (mimic kernel-reported unsupported ATS) => ATS masked
- ats=off => ATS masked
Quoted 4 lines
What about ats=on and no ATS support in the physical device ?
I agree it is a bit of a corner case. A warning would be nice, it can be addressed as a follow up change.
Right, I had missed that case. I will look into accounting for it in the next refresh.
Thanks,
Nathan