Skip to contentSkip to conversation

ATS/PASID SVA support and PRI-less fault servicing

PATCH In review Nareshkumar Gollakoti 5 participants 15 messages Oct 21 – Oct 22

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover ATS/PASID SVA support and PRI-less fault servicing

This series adds initial Shared Virtual Addressing (SVA) support for xe on platforms where ATS and PASID are available but PRI is not enabled or not supported.

For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series adds the xe side infrastructure needed for that flow: VM support for SVA PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID for ATS and PASID enabled contexts.

On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.

To support that case, this series introduces a CPU mirror VMA and uses it as part of a driver managed SVM path for SVA faults without PRI.The purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the driver can rely on the kernel MM/VMA machinery to validate mappings and resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely enough to let faults be service through normal MM paths.

At a high level, this series does the following:

1. log ATS/PASID enablement state during probe,
2. add VM infrastructure needed for SVA PASID support,
3. add a helper to retrieve the VM SVA PASID,
4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
5. add a PCI helper for ATS/PASID operation without PRI,
6. add a CPU mirror VMA for handling SVA faults on non PRI platforms

Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.

Himal Prasad Ghimiray (1):
drm/xe/vm: Setup SVA infrastructure for PASID support

Nareshkumar Gollakoti (5):
drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

 drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
 drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
 drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
 drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
 drivers/gpu/drm/xe/xe_vm.c              | 56 +++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h              |  2 +
 drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
 9 files changed, 184 insertions(+)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/6 drm/xe/pci: log ATS/PASID enablement state at probe

Add debug logging in xe_pci_probe() for ATS and PASID enablement state when the corresponding capabilities are present.
This makes platform and IOMMU related bring up issues easier to diagnose.

Also add a small helper to report weather ATS and PASID are both enabled, for use follow up checks that depend on both features.

---
 drivers/gpu/drm/xe/xe_pci.c | 7 +++++++
 drivers/gpu/drm/xe/xe_pci.h | 6 ++++++
 2 files changed, 13 insertions(+)
drivers/gpu/drm/xe/xe_pci.c +7 -0
index e44b74b9aa27..40c0f4c42ed3 100644
@@ -1275,6 +1275,13 @@ static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *des
 		xe_step_name(xe->info.step.media),
 		xe_step_name(xe->info.step.basedie));
 
+	if (pdev->ats_cap)
+		drm_dbg(&xe->drm, "Arch ATS = %s",
+			str_enabled_disabled(pdev->ats_enabled));
+	if (pdev->pasid_cap)
+		drm_dbg(&xe->drm, "Arch PASID = %s",
+			str_enabled_disabled(pdev->pasid_enabled));
+
 	drm_dbg(&xe->drm, "SR-IOV support: %s (mode: %s)\n",
 		str_yes_no(xe_device_has_sriov(xe)),
 		xe_sriov_mode_to_string(xe_device_sriov_mode(xe)));
drivers/gpu/drm/xe/xe_pci.h +6 -0
index 11bcc5fe2c5b..5f784229a5ec 100644
@@ -5,6 +5,7 @@
 
 #ifndef _XE_PCI_H_
 #define _XE_PCI_H_
+#include <linux/pci.h>
 
 struct pci_dev;
 
@@ -12,4 +13,9 @@ int xe_register_pci_driver(void);
 void xe_unregister_pci_driver(void);
 struct xe_device *xe_pci_to_pf_device(struct pci_dev *pdev);
 
+static inline bool xe_pci_ats_pasid_enabled(struct pci_dev *pdev)
+{
+	return pdev->ats_enabled && pdev->pasid_enabled;
+}
+
 #endif
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/6] drm/xe/pci: log ATS/PASID enablement state at probe

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] drm/xe/pci: xe_pci_ats_pasid_enabled() always returns false on SR-IOV VFs - [Low] drm/xe/pci…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] drm/xe/pci: xe_pci_ats_pasid_enabled() always returns false on SR-IOV VFs
- [Low] drm/xe/pci: build failure in __xe_pci_probe() without CONFIG_PCI_ATS/PASID --

commit 9613705bbc43ffcbfae8d92b0e6cff920caffa87
Author: Nareshkumar Gollakoti <[email protected]>

drm/xe/pci: log ATS/PASID enablement state at probe

This commit logs ATS and PASID enablement states during __xe_pci_probe() when their PCI capabilities are present, and introduces xe_pci_ats_pasid_enabled() to check whether both features are enabled on a PCI device.

Quoted 5 lines
drivers/gpu/drm/xe/xe_pci.c +0 -0
index 44587b5f4df60..7c8a4286b657a 100644

drivers/gpu/drm/xe/xe_pci.c static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *des · line 1289

Quoted 9 lines
drivers/gpu/drm/xe/xe_pci.c +6 -0
@@ -1281,6 +1281,13 @@ static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *des
 		xe_step_name(xe->info.step.media),
 		xe_step_name(xe->info.step.basedie));
 
+	if (pdev->ats_cap)
+		drm_dbg(&xe->drm, "Arch ATS = %s",
+			str_enabled_disabled(pdev->ats_enabled));
+	if (pdev->pasid_cap)
+		drm_dbg(&xe->drm, "Arch PASID = %s",
+			str_enabled_disabled(pdev->pasid_enabled));

[Severity: Low]
Will __xe_pci_probe() in drivers/gpu/drm/xe/xe_pci.c fail to build when CONFIG_PCI_ATS or CONFIG_PCI_PASID is disabled?

In include/linux/pci.h, struct pci_dev only defines ats_cap and pasid_cap when their respective Kconfig options are enabled:

#ifdef CONFIG_PCI_ATS
	...
	u16		ats_cap;	/* ATS Capability offset */
	u8		ats_stu;	/* ATS Smallest Translation Unit */
#endif
...
#ifdef CONFIG_PCI_PASID
	u16		pasid_cap;	/* PASID Capability offset */
	u16		pasid_features;

#endif

Since CONFIG_DRM_XE in drivers/gpu/drm/xe/Kconfig only depends on DRM and PCI without selecting or depending on PCI_ATS or PCI_PASID (for example, in drivers/gpu/drm/xe/.kunitconfig), accessing pdev->ats_cap and pdev->pasid_cap directly without IS_ENABLED() guards breaks the build when either option is not set.

Quoted 17 lines
drivers/gpu/drm/xe/xe_pci.c +1 -0
+
 	drm_dbg(&xe->drm, "SR-IOV support: %s (mode: %s)\n",
 		str_yes_no(xe_device_has_sriov(xe)),
 		xe_sriov_mode_to_string(xe_device_sriov_mode(xe)));
drivers/gpu/drm/xe/xe_pci.h +1 -0
index 11bcc5fe2c5b9..5f784229a5ecf 100644
@@ -5,6 +5,7 @@
 
 #ifndef _XE_PCI_H_
 #define _XE_PCI_H_
+#include <linux/pci.h>
 
 struct pci_dev;
 

drivers/gpu/drm/xe/xe_pci.h int xe_register_pci_driver(void); · line 19

Quoted 7 lines
drivers/gpu/drm/xe/xe_pci.h +4 -0
@@ -12,4 +13,9 @@ int xe_register_pci_driver(void);
 void xe_unregister_pci_driver(void);
 struct xe_device *xe_pci_to_pf_device(struct pci_dev *pdev);
 
+static inline bool xe_pci_ats_pasid_enabled(struct pci_dev *pdev)
+{
+	return pdev->ats_enabled && pdev->pasid_enabled;
+}

[Severity: Medium]
How does xe_pci_ats_pasid_enabled() in drivers/gpu/drm/xe/xe_pci.h (as well as the PASID check in __xe_pci_probe()) behave on SR-IOV Virtual Functions?

In drivers/pci/ats.c, pci_enable_pasid() notes that VFs do not implement their own PASID capability (so pdev->pasid_cap is 0) and instead share the PF's PASID configuration:

	if (pdev->is_virtfn) {
		if (pci_physfn(pdev)->pasid_enabled)
			return 0;
		return -EINVAL;
	}

When the IOMMU enables PASID on a VF via pci_enable_pasid(), it checks pci_physfn(pdev)->pasid_enabled and returns 0 without setting pdev->pasid_enabled on the VF's pci_dev (whereas pci_enable_ats() sets pdev->ats_enabled on the VF).

Because xe_pci_ats_pasid_enabled() and __xe_pci_probe() inspect pdev->pasid_enabled and pdev->pasid_cap directly instead of checking pci_physfn(pdev) (or calling pci_pasid_status(pdev)), won't xe_pci_ats_pasid_enabled() always return false on VFs and __xe_pci_probe() always skip logging the PASID state on VFs?

Quoted 2 lines
drivers/gpu/drm/xe/xe_pci.h +1 -0
+
 #endif
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

PATCH 2/6 drm/xe/vm: Setup SVA infrastructure for PASID support

In reply to Nareshkumar Gollakoti

From: Himal Prasad Ghimiray <[email protected]>

Add sva_handle to struct xe_vm to represent the device-mm bond established via iommu_sva_bind_device(). This handle will be used to retrieve the PASID for SVM-enabled VMs

Introduce xe_vm_sva_bind() to bind the VM to the current process address space using Shared Virtual Addressing(SVA) when ATS and PASID are enabled on the device. This allows the GPU to shate the same virtual address space as the CPU.

If sva binding fails, the VM continues without PASID support. Also introduce a cleanup helper to unbind during VM teardown.

---
 drivers/gpu/drm/xe/xe_vm.c       | 40 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm_types.h |  6 +++++
 2 files changed, 46 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +40 -0
index 761bc136145f..b1d37e5a673f 100644
@@ -45,6 +45,7 @@
 #include "xe_trace_bo.h"
 #include "xe_vm_madvise.h"
 #include "xe_wa.h"
+#include "xe_pci.h"
 
 static struct drm_gem_object *xe_vm_obj(struct xe_vm *vm)
 {
@@ -1648,6 +1649,41 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)
 	up_read(&vm->exec_queues.lock);
 }
 
+static void xe_vm_sva_handle_init(struct xe_vm *vm)
+{
+	struct device *dev = vm->xe->drm.dev;
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct iommu_sva *sva_handle;
+
+	/*
+	 * Assumes the IOMMU driver has already decided and latched the
+	 * device's ATS/PASID enablement by the time any VM is created (true
+	 * for Intel VT-d, which enables both eagerly at IOMMU probe). If a
+	 * backend instead enables them lazily on first SVA bind, this check
+	 * will never see them set and SVA/PASID support is silently skipped.
+	 */
+	if (!xe_pci_ats_pasid_enabled(pdev))
+		return;
+
+	sva_handle = iommu_sva_bind_device(dev, current->mm);
+	if (IS_ERR(sva_handle)) {
+		drm_err(&vm->xe->drm, "SVA binding failed, preventing PASID creation.\n");
+		return;
+	}
+
+	vm->sva_handle = sva_handle;
+}
+
+static void xe_vm_sva_handle_fini(struct xe_vm *vm)
+{
+	xe_assert(vm->xe, xe_vm_is_closed(vm));
+	if (!vm->sva_handle)
+		return;
+
+	iommu_sva_unbind_device(vm->sva_handle);
+	vm->sva_handle = NULL;
+}
+
 struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)
 {
 	struct drm_gem_object *vm_resv_obj;
@@ -1994,6 +2030,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 
 	xe_svm_fini(vm);
 
+	xe_vm_sva_handle_fini(vm);
 	up_write(&vm->lock);
 
 	down_write(&xe->usm.lock);
@@ -2159,6 +2196,9 @@ int xe_vm_create_ioctl(struct drm_device *dev, void *data,
 	if (IS_ERR(vm))
 		return PTR_ERR(vm);
 
+	if (flags & XE_VM_FLAG_FAULT_MODE)
+		xe_vm_sva_handle_init(vm);
+
 #if IS_ENABLED(CONFIG_DRM_XE_DEBUG_MEM)
 	/* Warning: Security issue - never enable by default */
 	args->reserved[0] = xe_bo_main_addr(vm->pt_root[0]->bo, XE_PAGE_SIZE);
drivers/gpu/drm/xe/xe_vm_types.h +6 -0
index fb9305124679..113807fa4ea0 100644
@@ -11,6 +11,7 @@
 #include <drm/drm_pagemap_util.h>
 
 #include <linux/dma-resv.h>
+#include <linux/iommu.h>
 #include <linux/kref.h>
 #include <linux/mmu_notifier.h>
 #include <linux/scatterlist.h>
@@ -423,6 +424,11 @@ struct xe_vm {
 	bool batch_invalidate_tlb;
 	/** @xef: Xe file handle for tracking this VM's drm client */
 	struct xe_file *xef;
+	/**
+	 * @sva_handle: the sva_handle to a device-mm bond, populated using
+	 * iommu_sva_bind_device(). To be used to determine pasid in case of svm vm.
+	 */
+	struct iommu_sva *sva_handle;
 };
 
 /** struct xe_vma_op_map - VMA map operation */
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH 2/6] drm/xe/vm: Setup SVA infrastructure for PASID support

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] drm/xe/vm: missing SVA unbind on device removal leaks DMA owner and crashes --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] drm/xe/vm: missing SVA unbind on device removal leaks DMA owner and crashes --

commit 08c0c9b32bbf8f35bf925eec1c7c7d7bb127ed1a
Author: Himal Prasad Ghimiray <[email protected]>

drm/xe/vm: Setup SVA infrastructure for PASID support

This commit adds an sva_handle field to struct xe_vm and introduces helpers to bind and unbind a fault-mode VM to the current process address space via iommu_sva_bind_device() and iommu_sva_unbind_device().

Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0 -0
index b98f2e56bf6f4..3504b890ababf 100644

[ ... ]

Quoted 28 lines
drivers/gpu/drm/xe/xe_vm.c +24 -0
@@ -1674,6 +1675,41 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)
 	up_read(&vm->exec_queues.lock);
 }
 
+static void xe_vm_sva_handle_init(struct xe_vm *vm)
+{
+	struct device *dev = vm->xe->drm.dev;
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct iommu_sva *sva_handle;
+
+	/*
+	 * Assumes the IOMMU driver has already decided and latched the
+	 * device's ATS/PASID enablement by the time any VM is created (true
+	 * for Intel VT-d, which enables both eagerly at IOMMU probe). If a
+	 * backend instead enables them lazily on first SVA bind, this check
+	 * will never see them set and SVA/PASID support is silently skipped.
+	 */
+	if (!xe_pci_ats_pasid_enabled(pdev))
+		return;
+
+	sva_handle = iommu_sva_bind_device(dev, current->mm);
+	if (IS_ERR(sva_handle)) {
+		drm_err(&vm->xe->drm, "SVA binding failed, preventing PASID creation.\n");
+		return;
+	}
+
+	vm->sva_handle = sva_handle;
+}

drivers/gpu/drm/xe/xe_vm.c static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm) · line 1711

Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10 -0
+
+static void xe_vm_sva_handle_fini(struct xe_vm *vm)
+{
+	xe_assert(vm->xe, xe_vm_is_closed(vm));
+	if (!vm->sva_handle)
+		return;
+
+	iommu_sva_unbind_device(vm->sva_handle);
+	vm->sva_handle = NULL;
+}

[Severity: High]
What happens if the pci device is unbound from the driver or hot-unplugged while a userspace process still holds an open fault-mode xe_vm?

When xe_vm_create_ioctl() calls xe_vm_sva_handle_init(), iommu_sva_bind_device() attaches a pasid domain and inserts an entry into dev->iommu_group->pasid_array. However, vm->sva_handle is only unbound in xe_vm_sva_handle_fini() during xe_vm_close_and_put(), which is tied to the userspace file descriptor and vm lifetime rather than device removal in drivers/gpu/drm/xe/xe_device.c:

void xe_device_remove(struct xe_device *xe)

{
	xe_display_unregister(xe);

	drm_dev_unplug(&xe->drm);

	xe_bo_pci_dev_remove_all(xe);
}

In addition, xe_vm_sva_handle_init() does not guard the iommu_sva_bind_device() call with drm_dev_enter(), so an in-flight xe_vm_create_ioctl() can also establish a new bond after drm_dev_unplug() completes.

When __device_release_driver() finishes xe_pci_remove() -> xe_device_remove(), it immediately calls pci_dma_cleanup() -> iommu_device_unuse_default_domain() in drivers/iommu/iommu.c:

	mutex_lock(&group->mutex);
	if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array)))
		group->owner_cnt--;

	mutex_unlock(&group->mutex);

Because group->pasid_array is not empty while vm->sva_handle remains bound, does this trigger the WARN_ON() and leak group->owner_cnt?

Furthermore, if the pci device is hot-unplugged and removed from the bus, iommu_release_device() -> __iommu_group_remove_device() -> iommu_deinit_device() clears dev->iommu_group to NULL. When userspace later closes the file descriptor or destroys the vm:

xe_vm_close_and_put()
xe_vm_sva_handle_fini()
iommu_sva_unbind_device()
iommu_detach_device_pasid()

drivers/iommu/iommu.c:iommu_detach_device_pasid() requires the caller to still be a probed driver on dev and dereferences dev->iommu_group directly:

void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev,
ioasid_t pasid)

{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;

	mutex_lock(&group->mutex);

Wouldn't calling iommu_sva_unbind_device() after device removal dereference a NULL group pointer here? Should xe_device_remove() unbind active sva_handle bonds before driver detach completes, and should xe_vm_sva_handle_init() protect the bind with drm_dev_enter() / drm_dev_exit()?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

PATCH 3/6 drm/xe/vm: add helper to retrieve VM SVA PASID

In reply to Nareshkumar Gollakoti

Add xe_vm_get_sva_pasid() to return the PASID associated with a VM SVA binding.

The helper checks whether the VM has a valid SVA handle and, if so, returns the PASID obtained from iommu_sva_get_pasid(). If no SVA handle is present, return -EINVAL.

Export the helper in xe_vm.h so other XE components can query the VM's PASID when programming per-context state.

---
 drivers/gpu/drm/xe/xe_vm.c | 16 ++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h |  2 ++
 2 files changed, 18 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +16 -0
index b1d37e5a673f..dcaeeeb94721 100644
@@ -1649,6 +1649,22 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)
 	up_read(&vm->exec_queues.lock);
 }
 
+/**
+ * xe_vm_get_sva_pasid() - get sva pasid for the vm if allocated
+ * @vm: The vm to get valid sva_handle
+ *
+ * Helper to get associated pasid for the vm which uses svm.
+ *
+ * Return: %pasid value on Success. -EINVAL if not pasid available.
+ */
+int xe_vm_get_sva_pasid(struct xe_vm *vm)
+{
+	if (!vm || !vm->sva_handle)
+		return -EINVAL;
+
+	return iommu_sva_get_pasid(vm->sva_handle);
+}
+
 static void xe_vm_sva_handle_init(struct xe_vm *vm)
 {
 	struct device *dev = vm->xe->drm.dev;
drivers/gpu/drm/xe/xe_vm.h +2 -0
index 89c4f95984a2..fe10fd91cdfa 100644
@@ -290,6 +290,8 @@ struct dma_fence *xe_vm_bind_kernel_bo(struct xe_vm *vm, struct xe_bo *bo,
 
 void xe_vm_resume_rebind_worker(struct xe_vm *vm);
 
+int xe_vm_get_sva_pasid(struct xe_vm *vm);
+
 /**
  * xe_vm_resv() - Return's the vm's reservation object
  * @vm: The vm
Signature · 1 line
2.43.0
#6 lore raw

PATCH 4/6 drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts

In reply to Nareshkumar Gollakoti

Add CTX_SYS_PASID definitions to the LRC layout and program the field during LRC context initialization when ATS and PASID are both enabled on the PCI device.

When ATS/PASID support is available, fetch the VM SVA PASID and write it to CTX_SYS_PASID together with the ATS enable and PASID valid bits. This ensures the GPU context carries the system PASID state required for address translation services.

Bspec: 67168

---
 drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 ++++
 drivers/gpu/drm/xe/xe_lrc.c             | 37 +++++++++++++++++++++++++
 2 files changed, 43 insertions(+)
drivers/gpu/drm/xe/regs/xe_lrc_layout.h +6 -0
index 6f7abc0181b5..9394e3d5aecb 100644
@@ -18,6 +18,7 @@
 #define CTX_TIMESTAMP_UDW		(0x24 + 1)
 #define CTX_INDIRECT_RING_STATE		(0x26 + 1)
 #define CTX_ACC_CTR_THOLD		(0x2a + 1)
+#define CTX_SYS_PASID			(0x2c + 1)
 #define CTX_ASID			(0x2e + 1)
 #define CTX_PDP0_UDW			(0x30 + 1)
 #define CTX_PDP0_LDW			(0x32 + 1)
@@ -45,4 +46,9 @@
 #define INDIRECT_CTX_RING_START_UDW	(0x08 + 1)
 #define INDIRECT_CTX_RING_CTL		(0x0a + 1)
 
+/* Fields for CTX_SYS_PASID */
+#define CTX_SYS_ATS_ENABLED                    REG_BIT(31)
+#define CTX_SYS_PASID_VALID                    REG_BIT(30)
+#define CTX_SYS_SYSTEM_PASID_MASK	       REG_GENMASK(19, 0)
+
 #endif
drivers/gpu/drm/xe/xe_lrc.c +37 -0
index de9a9560ecf3..a4f7f0492262 100644
@@ -9,6 +9,7 @@
 
 #include <linux/ascii85.h>
 #include <linux/panic.h>
+#include <linux/pci.h>
 
 #include "instructions/xe_mi_commands.h"
 #include "instructions/xe_gfxpipe_commands.h"
@@ -35,6 +36,7 @@
 #include "xe_trace_lrc.h"
 #include "xe_vm.h"
 #include "xe_wa.h"
+#include "xe_pci.h"
 
 #define LRC_VALID				BIT_ULL(0)
 #define LRC_PRIVILEGE				BIT_ULL(8)
@@ -998,6 +1000,16 @@ void xe_lrc_write_ctx_reg(struct xe_lrc *lrc, int reg_nr, u32 val)
 	xe_map_write32(xe, &map, val);
 }
 
+static void xe_lrc_rmw_ctx_reg(struct xe_lrc *lrc,
+			       int reg_nr, u32 mask, u32 val)
+{
+	u32 oldval = xe_lrc_read_ctx_reg(lrc, reg_nr);
+	u32 newval = (oldval & ~mask) | (val & mask);
+
+	if (newval != oldval)
+		xe_lrc_write_ctx_reg(lrc, reg_nr, newval);
+}
+
 static void *empty_lrc_data(struct xe_hw_engine *hwe)
 {
 	struct xe_gt *gt = hwe->gt;
@@ -1493,6 +1505,26 @@ static u8 xe_multi_queue_prio_to_lrc(struct xe_lrc *lrc, enum xe_multi_queue_pri
 	return priority;
 }
 
+static void xe_lrc_program_ctx_sys_pasid(struct xe_lrc *lrc, struct xe_vm *vm)
+{
+	int pasid;
+	u32 mask = CTX_SYS_ATS_ENABLED | CTX_SYS_PASID_VALID |
+		   CTX_SYS_SYSTEM_PASID_MASK;
+	struct xe_device *xe = lrc_to_xe(lrc);
+
+	pasid = xe_vm_get_sva_pasid(vm);
+	/* PASID 0 is reserved by the IOMMU for non-PASID-capable devices. */
+	if (pasid <= 0)
+		return;
+
+	xe_lrc_rmw_ctx_reg(lrc, CTX_SYS_PASID, mask,
+			   CTX_SYS_ATS_ENABLED |
+			   CTX_SYS_PASID_VALID |
+			   (pasid & CTX_SYS_SYSTEM_PASID_MASK));
+	drm_dbg(&xe->drm, "LRC: context enabled with ATS and PASID=%d for VM %p and lrc %p\n",
+		pasid, vm, lrc);
+}
+
 /**
  * xe_lrc_set_multi_queue_priority() - Set multi queue priority in LRC
  * @lrc: Logical Ring Context
@@ -1542,6 +1574,7 @@ static int xe_lrc_ctx_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct
 	struct xe_gt *gt = hwe->gt;
 	struct xe_tile *tile = gt_to_tile(gt);
 	struct xe_device *xe = gt_to_xe(gt);
+	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
 	struct iosys_map map;
 	u32 arb_enable;
 	u32 state_cache_perf_fix[3];
@@ -1642,6 +1675,10 @@ static int xe_lrc_ctx_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct
 	if (GRAPHICS_VER(xe) >= 20 && hwe->class == XE_ENGINE_CLASS_RENDER)
 		xe_lrc_set_gpgpu_preemption_level(lrc, gt);
 
+	/* Program CTX_SYS_PASID when ATS and PASID is supported. */
+	if (xe_pci_ats_pasid_enabled(pdev))
+		xe_lrc_program_ctx_sys_pasid(lrc, vm);
+
 	lrc->desc = LRC_VALID;
 	lrc->desc |= FIELD_PREP(LRC_ADDRESSING_MODE, LRC_LEGACY_64B_CONTEXT);
 	/* TODO: Priority */
Signature · 1 line
2.43.0
#7 lore raw

PATCH 5/6 drm/xe/pci: add helper for ATS/PASID without PRI

In reply to Nareshkumar Gollakoti

On platforms where ATS and PASID are enabled but PRI is not available, SVA faults cannot rely on the IOMMU to fault in missing CPU mirror bindings. Those faults must instead be recovered in driver.

Add a small PCI helper to report this condition so fault-handling code can use a single predicate for the PRI-less SVA case.

---
 drivers/gpu/drm/xe/xe_pci.h | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/gpu/drm/xe/xe_pci.h +9 -0
index 5f784229a5ec..cd22d316883b 100644
@@ -18,4 +18,13 @@ static inline bool xe_pci_ats_pasid_enabled(struct pci_dev *pdev)
 	return pdev->ats_enabled && pdev->pasid_enabled;
 }
 
+/*
+ * True when SVA faults must be serviced in software: ATS/PASID is
+ * enabled but PRI is not available, so the IOMMU has no self-healing path.
+ */
+static inline bool xe_pci_svm_needs_sw_iopf(struct pci_dev *pdev)
+{
+	return xe_pci_ats_pasid_enabled(pdev) && !pdev->pri_enabled;
+}
+
 #endif
Signature · 1 line
2.43.0
#8 lore raw

PATCH 6/6 drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

In reply to Nareshkumar Gollakoti

On ATS/PASID platforms without PRI, xe cannot rely on the hardware page request path to populate missing CPU mirror mappings for SVA faults. This can leave xe_pagefault_service() without a VMA even when the faulting address is valid in the process address space.

Create a CPU-address-mirror VMA on demand for such faults so they can proceed through the existing SVM fault handling path.

Faults against unmapped addresses or unusable VMs continue to fail.

---
 drivers/gpu/drm/xe/xe_pagefault.c       | 49 +++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
 2 files changed, 55 insertions(+)
drivers/gpu/drm/xe/xe_pagefault.c +49 -0
index 98cce0f12b12..c3acec28e115 100644
@@ -17,6 +17,7 @@
 #include "xe_migrate.h"
 #include "xe_pagefault.h"
 #include "xe_pagefault_types.h"
+#include "xe_pci.h"
 #include "xe_pm.h"
 #include "xe_svm.h"
 #include "xe_trace_bo.h"
@@ -268,6 +269,7 @@ static int xe_pagefault_service(struct xe_pagefault *pf)
 {
 	struct xe_gt *gt = pf->gt;
 	struct xe_device *xe = gt_to_xe(gt);
+	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
 	struct xe_vm *vm;
 	struct xe_vma *vma = NULL;
 	int err;
@@ -298,6 +300,51 @@ static int xe_pagefault_service(struct xe_pagefault *pf)
 	}
 
 	vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
+
+	/*
+	 * ATS/PASID is negotiated but this hardware has no PRI, so there is
+	 * no IOMMU self-healing path to fault the page in - the driver has
+	 * to create the missing CPU-address-mirror binding itself and let
+	 * the SVM path below service the fault.
+	 */
+	if (!vma && vm->sva_handle && xe_pci_svm_needs_sw_iopf(pdev)) {
+		u64 page_size = vm->flags & XE_VM_FLAG_64K ? SZ_64K : SZ_4K;
+		u64 start = ALIGN_DOWN(xe_pagefault_addr(pf), page_size);
+		u64 end = start + page_size;
+		int alloc_err = 0;
+
+		up_read(&vm->lock);
+		down_write(&vm->lock);
+		/* Lock was dropped, re-check: a racing fault may have won */
+		vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
+		if (!vma) {
+			if (xe_vm_is_closed_or_banned(vm)) {
+				alloc_err = -ENOENT;
+			} else {
+				/* Merge with adjacent mirror VMAs instead of leaving singletons */
+				xe_vm_find_cpu_addr_mirror_vma_range(vm, &start, &end);
+				alloc_err = xe_vm_alloc_cpu_addr_mirror_vma(vm, start, end - start);
+			}
+			if (!alloc_err)
+				vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
+		}
+		downgrade_write(&vm->lock);
+
+		if (alloc_err || !vma || !xe_vma_is_cpu_addr_mirror(vma)) {
+			/*
+			 * err is what actually tells guc_ack_fault() to NACK;
+			 * distinguish a transient allocation failure from a
+			 * genuinely unmapped/closed address for diagnostics.
+			 */
+			err = alloc_err ? : -EINVAL;
+			if (alloc_err == -ENOMEM)
+				xe_pagefault_set_error(pf, XE_PAGEFAULT_ERROR_VMA_ALLOC);
+			else
+				xe_pagefault_set_error(pf, XE_PAGEFAULT_ERROR_VMA_NOT_FOUND);
+			goto unlock_vm;
+		}
+	}
+
 	if (!vma) {
 		err = -EINVAL;
 		xe_pagefault_set_error(pf, XE_PAGEFAULT_ERROR_VMA_NOT_FOUND);
@@ -589,6 +636,8 @@ static const char *xe_pagefault_error_to_str(enum xe_pagefault_error error)
 		return "VM_CLOSED";
 	case XE_PAGEFAULT_ERROR_VMA_NOT_FOUND:
 		return "VMA_NOT_FOUND";
+	case XE_PAGEFAULT_ERROR_VMA_ALLOC:
+		return "VMA_ALLOC";
 	case XE_PAGEFAULT_ERROR_READ_ONLY_VIOLATION:
 		return "READ_ONLY_VIOLATION";
 	case XE_PAGEFAULT_ERROR_VMA_NEEDS_VRAM_CHECK:
drivers/gpu/drm/xe/xe_pagefault_types.h +6 -0
index 56defc05922e..d95d47ee0c0c 100644
@@ -59,6 +59,12 @@ enum xe_pagefault_error {
 	XE_PAGEFAULT_ERROR_VM_CLOSED,
 	/** @XE_PAGEFAULT_ERROR_VMA_NOT_FOUND: No VMA covers the faulted address */
 	XE_PAGEFAULT_ERROR_VMA_NOT_FOUND,
+	/**
+	 * @XE_PAGEFAULT_ERROR_VMA_ALLOC: Failed to allocate a CPU-address-mirror
+	 * VMA to service a PRI-less SVA fault (e.g. -ENOMEM); distinct from a
+	 * genuinely unmapped address
+	 */
+	XE_PAGEFAULT_ERROR_VMA_ALLOC,
 	/**
 	 * @XE_PAGEFAULT_ERROR_READ_ONLY_VIOLATION: Write/atomic fault on a
 	 * read-only VMA
Signature · 1 line
2.43.0
#9 lore raw

Re: [PATCH 6/6] drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] drm/xe: NULL deref and uninitialized PAT in xe_vm_alloc_cpu_addr_mirror_vma() - [High] drm/xe…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] drm/xe: NULL deref and uninitialized PAT in xe_vm_alloc_cpu_addr_mirror_vma()
- [High] drm/xe: 4K VMA check before 64K mirror alloc in xe_pagefault_service() --

commit c441331b0c386c437e9ad479c73465c57bd86054
Author: Nareshkumar Gollakoti <[email protected]>

drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

This commit adds on-demand creation of a cpu-address-mirror vma in xe_pagefault_service() when servicing sva faults on platforms with ats/pasid enabled but without pri support. It also introduces the XE_PAGEFAULT_ERROR_VMA_ALLOC error code to report vma allocation failures.

Quoted 4 lines
drivers/gpu/drm/xe/xe_pagefault.c +0 -0
index 5247321c805ec..fbdf12d3d5580 100644

[ ... ]

Quoted 11 lines
drivers/gpu/drm/xe/xe_pagefault.c +7 -0
@@ -298,6 +300,51 @@ static int xe_pagefault_service(struct xe_pagefault *pf)
 	}
 
 	vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
+
+	/*
+	 * ATS/PASID is negotiated but this hardware has no PRI, so there is
+	 * no IOMMU self-healing path to fault the page in - the driver has
+	 * to create the missing CPU-address-mirror binding itself and let
+	 * the SVM path below service the fault.
+	 */

drivers/gpu/drm/xe/xe_pagefault.c static int xe_pagefault_service(struct xe_pagefault *pf) · line 319

Quoted 10 lines
drivers/gpu/drm/xe/xe_pagefault.c +10 -0
+	if (!vma && vm->sva_handle && xe_pci_svm_needs_sw_iopf(pdev)) {
+		u64 page_size = vm->flags & XE_VM_FLAG_64K ? SZ_64K : SZ_4K;
+		u64 start = ALIGN_DOWN(xe_pagefault_addr(pf), page_size);
+		u64 end = start + page_size;
+		int alloc_err = 0;
+
+		up_read(&vm->lock);
+		down_write(&vm->lock);
+		/* Lock was dropped, re-check: a racing fault may have won */
+		vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));

[Severity: High]
When vm->flags & XE_VM_FLAG_64K is set, xe_pagefault_service() aligns start and end to SZ_64K, but xe_vm_find_vma_by_addr() in drivers/gpu/drm/xe/xe_vm.c only checks a 4k range at the faulting address:

drivers/gpu/drm/xe/xe_vm.c:xe_vm_find_vma_by_addr() {
	return xe_vm_find_overlapping_vma(vm, page_addr, SZ_4K);
}

Because vm_bind_ioctl_check_args() in drivers/gpu/drm/xe/xe_vm.c and madvise_args_are_sane() in drivers/gpu/drm/xe/xe_vm_madvise.c allow 4k-aligned mappings on 64k vms (xe_vm_bind_ioctl_validate_bo() only requires 64k alignment for bos with XE_BO_FLAG_INTERNAL_64K), another 4k-aligned vma (such as a system-memory bo, userptr, sparse vma, or madvise-split vma) can already exist elsewhere within [start, end).

If a fault hits an unmapped 4k page in that 64k window without first checking xe_vm_find_overlapping_vma(vm, start, page_size), won't xe_vm_alloc_cpu_addr_mirror_vma() -> xe_vm_alloc_vma() -> drm_gpuvm_sm_map_ops_create() generate unmap or remap operations that destroy the overlapping vma via xe_vma_destroy() without unbinding its page tables or holding bo->ttm.base.resv, triggering xe_bo_assert_held(bo) in xe_vma_destroy()?

drivers/gpu/drm/xe/xe_pagefault.c static int xe_pagefault_service(struct xe_pagefault *pf) · line 326

Quoted 7 lines
drivers/gpu/drm/xe/xe_pagefault.c +7 -0
+		if (!vma) {
+			if (xe_vm_is_closed_or_banned(vm)) {
+				alloc_err = -ENOENT;
+			} else {
+				/* Merge with adjacent mirror VMAs instead of leaving singletons */
+				xe_vm_find_cpu_addr_mirror_vma_range(vm, &start, &end);
+				alloc_err = xe_vm_alloc_cpu_addr_mirror_vma(vm, start, end - start);

[Severity: Critical]
What happens in xe_pagefault_service() when a fault occurs at an address with no immediately adjacent default-attribute mirror vma at start - SZ_4K or end + 1 (such as the first sva page fault in the vm)?

In that case, xe_vm_find_cpu_addr_mirror_vma_range() leaves [start, end) unchanged so the range does not overlap any existing gpuva in vm->gpuvm.

When xe_vm_alloc_cpu_addr_mirror_vma() calls xe_vm_alloc_vma(vm, &map_req, false) in drivers/gpu/drm/xe/xe_vm.c, vma_flags is initialized to 0 and default_pat is left uninitialized on the stack, because both are only populated from an existing vma during DRM_GPUVA_OP_UNMAP or DRM_GPUVA_OP_REMAP:

drivers/gpu/drm/xe/xe_vm.c:xe_vm_alloc_vma() {
	unsigned int vma_flags = 0;
	bool remap_op = false;
	struct xe_vma_mem_attr tmp_attr = {};
	u16 default_pat;
	...
		if (!is_madvise) {
			if (__op->op == DRM_GPUVA_OP_UNMAP) {
				vma = gpuva_to_vma(op->base.unmap.va);
				XE_WARN_ON(!xe_vma_has_default_mem_attrs(vma));
				default_pat = vma->attr.default_pat_index;
				vma_flags = vma->gpuva.flags;
			}

			if (__op->op == DRM_GPUVA_OP_REMAP) {
				vma = gpuva_to_vma(op->base.remap.unmap->va);
				default_pat = vma->attr.default_pat_index;
				vma_flags = vma->gpuva.flags;
			}

			if (__op->op == DRM_GPUVA_OP_MAP) {
				op->map.vma_flags |= vma_flags & XE_VMA_CREATE_MASK;
				op->map.pat_index = default_pat;
			}
		}
}

Because drm_gpuvm_sm_map_ops_create() only emits DRM_GPUVA_OP_MAP when no existing gpuva overlaps [start, end), op->map.vma_flags remains 0 (missing XE_VMA_SYSTEM_ALLOCATOR) and op->map.pat_index uses the uninitialized default_pat variable.

From there, vm_bind_ioctl_ops_parse() -> new_vma() -> xe_vma_create() in drivers/gpu/drm/xe/xe_vm.c sees !bo && !is_null && !is_cpu_addr_mirror, allocates a struct xe_userptr_vma instead of a mirror vma, and calls xe_userptr_setup() in drivers/gpu/drm/xe/xe_userptr.c:

drivers/gpu/drm/xe/xe_userptr.c:xe_userptr_setup() {
	...
	err = mmu_interval_notifier_insert(&userptr->notifier, current->mm,
					   start, range,
					   &vma_userptr_notifier_ops);
}

Since xe_pagefault_service() runs on a workqueue kthread where current->mm is NULL, won't this cause a null pointer dereference inside mmu_interval_notifier_insert()?

Quoted 1 line
drivers/gpu/drm/xe/xe_pagefault.c +1 -0
+			}

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#10 lore raw

✗ CI.KUnit: failure for ATS/PASID SVA support and PRI-less fault servicing

In reply to Nareshkumar Gollakoti
== Series Details ==

== Series Details ==

Series: ATS/PASID SVA support and PRI-less fault servicing
URL : https://patchwork.freedesktop.org/series/176519/
State : failure

== Summary ==

+ trap cleanup EXIT
+ kunitconfigs=('/kernel/drivers/gpu/tests/.kunitconfig' '/kernel/drivers/gpu/drm/xe/.kunitconfig' '/kernel/drivers/gpu/drm/tests/.kunitconfig' '/kernel/drivers/gpu/drm/ttm/tests/.kunitconfig' '/kernel/drivers/dma-buf/.kunitconfig')
+ for kcfg in "${kunitconfigs[@]}"
+ [[ ! -f /kernel/drivers/gpu/tests/.kunitconfig ]]
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/tests/.kunitconfig
[04:20:43] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[04:20:47] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48
[04:21:08] Starting KUnit Kernel (1/1)...
[04:21:08] ============================================================
Running tests with:
$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[04:21:08] ================= gpu_buddy (14 subtests) ==================
[04:21:08] [PASSED] gpu_test_buddy_alloc_limit
[04:21:08] [PASSED] gpu_test_buddy_alloc_optimistic
[04:21:08] [PASSED] gpu_test_buddy_alloc_pessimistic
[04:21:08] [PASSED] gpu_test_buddy_alloc_pathological
[04:21:08] [PASSED] gpu_test_buddy_alloc_contiguous
[04:21:08] [PASSED] gpu_test_buddy_alloc_clear
[04:21:08] [PASSED] gpu_test_buddy_alloc_range
[04:21:09] [PASSED] gpu_test_buddy_alloc_range_bias
[04:21:10] [PASSED] gpu_test_buddy_fragmentation_performance
[04:21:10] [PASSED] gpu_test_buddy_dirty_tracker_performance
[04:21:10] [PASSED] gpu_test_buddy_alloc_exceeds_max_order
[04:21:10] [PASSED] gpu_test_buddy_offset_aligned_allocation
[04:21:10] [PASSED] gpu_test_buddy_subtree_offset_alignment_stress
[04:21:10] [PASSED] gpu_test_buddy_addr_to_block
[04:21:10] ==================== [PASSED] gpu_buddy ====================
[04:21:10] ============================================================
[04:21:10] Testing complete. Ran 14 tests: passed: 14
[04:21:10] Elapsed time: 27.450s total, 4.472s configuring, 20.860s building, 2.054s running

+ for kcfg in "${kunitconfigs[@]}"
+ [[ ! -f /kernel/drivers/gpu/drm/xe/.kunitconfig ]]
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig ERROR:root:../drivers/gpu/drm/xe/xe_pci.c: In function ‘__xe_pci_probe’:
../drivers/gpu/drm/xe/xe_pci.c:1288:19: error: ‘struct pci_dev’ has no member named ‘pasid_cap’; did you mean ‘pcie_cap’?

 1288 |         if (pdev->pasid_cap)
      |                   ^~~~~~~~~
      |                   pcie_cap

make[7]: *** [../scripts/Makefile.build:290: drivers/gpu/drm/xe/xe_pci.o] Error 1 make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:551: drivers/gpu/drm/xe] Error 2 make[5]: *** [../scripts/Makefile.build:551: drivers/gpu/drm] Error 2 make[4]: *** [../scripts/Makefile.build:551: drivers/gpu] Error 2 make[3]: *** [../scripts/Makefile.build:551: drivers] Error 2 make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/kernel/Makefile:2229: .] Error 2
make[1]: *** [/kernel/Makefile:248: __sub-make] Error 2
make: *** [Makefile:248: __sub-make] Error 2

[04:21:10] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[04:21:12] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48
+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel
#11 lore raw
In reply to Nareshkumar Gollakoti

Hi!

Quoted 49 lines · Nareshkumar Gollakoti

This series adds initial Shared Virtual Addressing (SVA) support for xe
on platforms where ATS and PASID are available but PRI is not enabled or
not supported.

For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series
adds the xe side infrastructure needed for that flow: VM support for SVA
PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID
for ATS and PASID enabled contexts.

On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need
to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In
that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.

To support that case, this series introduces a CPU mirror VMA and uses
it as part of a driver managed SVM path for SVA faults without PRI.The
purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the
driver can rely on the kernel MM/VMA machinery to validate mappings and
resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely
enough to let faults be service through normal MM paths.

At a high level, this series does the following:

 1. log ATS/PASID enablement state during probe,
 2. add VM infrastructure needed for SVA PASID support,
 3. add a helper to retrieve the VM SVA PASID,
 4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
 5. add a PCI helper for ATS/PASID operation without PRI,
 6. add a CPU mirror VMA for handling SVA faults on non PRI platforms

Quoted 4 lines

Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.

I was under the impression that we'd integrate this with current SVM so that we can still migrate to VRAM and use page-table mirroring or keep data in system memory and use SVA page-table sharing *OR* page-table- mirroring depending on user-space's wish?

Thanks,
Thomas

Quoted 22 lines

Himal Prasad Ghimiray (1):
  drm/xe/vm: Setup SVA infrastructure for PASID support

Nareshkumar Gollakoti (5):
  drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

 drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++ drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++ drivers/gpu/drm/xe/xe_pci.c             |  7 ++++ drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++ drivers/gpu/drm/xe/xe_vm.c              | 56
+++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h              |  2 + drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++

 9 files changed, 184 insertions(+)
#12 lore raw

Hi Thomas,

Quoted 56 lines · Thomas Hellström

Hi!

Quoted 49 lines · Nareshkumar Gollakoti

This series adds initial Shared Virtual Addressing (SVA) support for xe
on platforms where ATS and PASID are available but PRI is not enabled or
not supported.

For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series
adds the xe side infrastructure needed for that flow: VM support for SVA
PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID
for ATS and PASID enabled contexts.

On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need
to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In
that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.

To support that case, this series introduces a CPU mirror VMA and uses
it as part of a driver managed SVM path for SVA faults without PRI.The
purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the
driver can rely on the kernel MM/VMA machinery to validate mappings and
resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely
enough to let faults be service through normal MM paths.

At a high level, this series does the following:

 1. log ATS/PASID enablement state during probe,
 2. add VM infrastructure needed for SVA PASID support,
 3. add a helper to retrieve the VM SVA PASID,
 4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
 5. add a PCI helper for ATS/PASID operation without PRI,
 6. add a CPU mirror VMA for handling SVA faults on non PRI platforms

Quoted 4 lines

Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.

Quoted 7 lines

I was under the impression that we'd integrate this with current SVM so that we can still migrate to VRAM and use page-table mirroring or keep data in system memory and use SVA page-table sharing *OR* page-table- mirroring depending on user-space's wish?

Thanks,
Thomas

IMO, Yes, that is the intended model. Userspace picks between the two approaches with the bind.

Bind with CPU_ADDR_MIRROR → existing SVM. GPU page fault, optional VRAM migrate, PPGTT mirroring and this path is unchanged.

Do not bind → SVA page-table sharing. IOMMU walks the CPU page tables with ATS+PASID. Backing has to stay in system memory. This is the end-to-end share flow, and it needs ATS+PASID+PRI.

If PRI is missing, an ATS miss cannot be healed by the IOMMU. The device then reports a GPU page fault. Instead of NACKing that unbound fault, we create a CPU-mirror VMA and service it with the same SVM path so the GPU gets a PPGTT mapping.

So, ATS+PASID+PRI → share works as user space asked (no bind). Bind with CPU_ADDR_MIRROR → SVM, independent of PRI. *PRI-less is only the recovery path when share cannot complete at the IOMMU*.(This is one of the proposal)

Let me know if anything am missing.

Thanks,

Naresh

Quoted 22 lines
Quoted 22 lines

Himal Prasad Ghimiray (1):
  drm/xe/vm: Setup SVA infrastructure for PASID support

Nareshkumar Gollakoti (5):
  drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

  drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
  drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
  drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++
  drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
  drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
  drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
  drivers/gpu/drm/xe/xe_vm.c              | 56
+++++++++++++++++++++++++
  drivers/gpu/drm/xe/xe_vm.h              |  2 +
  drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
  9 files changed, 184 insertions(+)
#13 lore raw
Quoted 105 lines · Naresh

Hi Thomas,

Quoted 79 lines · Thomas Hellström

Hi!

Quoted 61 lines · Nareshkumar Gollakoti

This series adds initial Shared Virtual Addressing (SVA) support for
xe
on platforms where ATS and PASID are available but PRI is not enabled
or
not supported.

For SVA, the device must be able to issue memory transactions tagged
with a PASID so that address translation is performed in the process
address space rather than in a driver owned address space. This series
adds the xe side infrastructure needed for that flow: VM support for
SVA
PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID
for ATS and PASID enabled contexts.

On systems with PRI, device page faults can be surfaced and resolved
through the hardware page request interface, so the driver does not
need
to implement a separate fault servicing path for SVA. On non PRI systems, that hardware assisted fault path is unavailable.In
that case, the driver still needs a way to resolve faults against the
backing process address space so execution can proceed.

To support that case, this series introduces a CPU mirror VMA and uses
it as part of a driver managed SVM path for SVA faults without PRI.The
purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so
the
driver can rely on the kernel MM/VMA machinery to validate mappings
and
resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver
needs a CPU visible VMA context that mirrors the GPU SVA range closely
enough to let faults be service through normal MM paths.

At a high level, this series does the following:

  1. log ATS/PASID enablement state during probe,
  2. add VM infrastructure needed for SVA PASID support,
  3. add a helper to retrieve the VM SVA PASID,
  4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
  5. add a PCI helper for ATS/PASID operation without PRI,
  6. add a CPU mirror VMA for handling SVA faults on non PRI platforms

Quoted 7 lines

Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for
this flow and have been validated internally.Patch 6 adds the non PRI
fault servicing path, and feedback on that overall approach would be
especially appreciated.

I was under the impression that we'd integrate this with current SVM so
that we can still migrate to VRAM and use page-table mirroring or keep
data in system memory and use SVA page-table sharing *OR* page- table-
mirroring depending on user-space's wish?

Quoted 2 lines

Thanks,
Thomas

IMO, Yes, that is the intended model. Userspace picks between the two approaches with the bind.

Bind with CPU_ADDR_MIRROR → existing SVM. GPU page fault, optional VRAM
migrate, PPGTT mirroring and this path is unchanged.

Do not bind → SVA page-table sharing. IOMMU walks the CPU page tables with ATS+PASID. Backing has to stay in system memory. This is the end-to-end share flow, and it needs ATS+PASID+PRI.

If PRI is missing, an ATS miss cannot be healed by the IOMMU. The device
then reports a GPU page fault. Instead of NACKing that unbound fault, we
create a CPU-mirror VMA and service it with the same SVM path so the GPU
gets a PPGTT mapping.

Quoted 8 lines

So, ATS+PASID+PRI → share works as user space asked (no bind). Bind with CPU_ADDR_MIRROR → SVM, independent of PRI. *PRI-less is only the recovery path when share cannot complete at the IOMMU*.(This is one of
the proposal)

Let me know if anything am missing.

Yes, first on the (implicit) UAPI, KMD must never implicitly create a GPU_VMA. This is done to be able to handle page-faults for non-PRI capable hardware. Couldn't that be done much easier with a separate page-fault handler that just calls fixup_user_fault() or a similar in- kernel API and mark the gpu fault as handled? Then I assume the ATS+PASID request will be retried?

On the implicit "Everything not having a GPU vma will be using ATS/PASID" I strongly oppose that. Was this discussed on the UAPI channel? Does NEO have any patches published to back this up? ATS/PASID enables us to HW optimize a subset of the existing SVM functionality, and IMO KMD is the best judge of whether to use it or not, possibly with a UMD madvise hint. IMO we should require a CPU_ADDR_MIRROR vma covering also the ATS/PASID regions and add an MADVISE setting for user-space to prefer ATS/PASID lookups over current hmm mirroring. If user-space then selects an MADVISE setting that is incompatible with ATS/PASID (like different PAT, VRAM preferred region etc.) Then KMD can transparently move over to that. We can also reuse existing igts and user-space. If UMD needs to detect whether ATS/PASID is usable on the VM then I figure we also must consider that.

Finally, it looks like you don't use the XE_PAGE_FORCE_FAULT bit, like in here:

https://patchwork.freedesktop.org/patch/657641/?series=149886&rev=3

How would you then stop the hardware from attempting an ATS/PASID lookup rather than faulting in mapped VMA?

Thanks,
Thomas

Quoted 31 lines

Thanks,

Naresh

Quoted 25 lines
Quoted 25 lines

Himal Prasad Ghimiray (1):
   drm/xe/vm: Setup SVA infrastructure for PASID support

Nareshkumar Gollakoti (5):
   drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts
   drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

  drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++ drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c       | 49
++++++++++++++++++++++
  drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++ drivers/gpu/drm/xe/xe_pci.c             |  7 ++++ drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++ drivers/gpu/drm/xe/xe_vm.c              | 56
+++++++++++++++++++++++++
  drivers/gpu/drm/xe/xe_vm.h              |  2 + drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++

  9 files changed, 184 insertions(+)
maintainer
#14 lore raw
In reply to Nareshkumar Gollakoti
Quoted 37 lines · Nareshkumar Gollakoti

This series adds initial Shared Virtual Addressing (SVA) support for xe on platforms where ATS and PASID are available but PRI is not enabled or not supported.

For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series adds the xe side infrastructure needed for that flow: VM support for SVA PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID for ATS and PASID enabled contexts.

On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.

To support that case, this series introduces a CPU mirror VMA and uses it as part of a driver managed SVM path for SVA faults without PRI.The purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the driver can rely on the kernel MM/VMA machinery to validate mappings and resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely enough to let faults be service through normal MM paths.

At a high level, this series does the following:

1. log ATS/PASID enablement state during probe,
2. add VM infrastructure needed for SVA PASID support,
3. add a helper to retrieve the VM SVA PASID,
4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
5. add a PCI helper for ATS/PASID operation without PRI,
6. add a CPU mirror VMA for handling SVA faults on non PRI platforms

Quoted 5 lines

Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.

Many concerns with this series, and I'm also generally quite confused about the flows here. Let's take a step back and dig into what you're trying to accomplish and why.

Prior to this series:

VM bind sets up PPGTTs.
- System memory uses DMA addresses (via dma-map) with DM=0 in the entry. These are GPAs (no pasid) translated by the IOMMU (or via ATS).
- Device memory uses DPAs with DM=1 in the entry.
SVM is configured via VM bind and CPU page-table mirroring.
- Pagefault or prefetches create mappings, data may or may not migrate.

After this series, all I have is questions and none are explained by the cover-letter or patch commits:

1. Is a user expected to call VM bind? It seems the answer is yes.

2. Do we need any changes to the PPGTT setup? bspec53912 seems to indicate that we do, but I'm not seeing any PPGTT setup changes in this series. At the very minimum, I'd expect userptr to not use dma-address and rely a PPGTT entry with P=0, FF=0 so the PASID walk occurs.

3. Are any programming model changes required in the UMD? If so, what are they? I'm also concerned that this becomes a blind opt-in whenever ATS + PASID are supported, as this would surely affect performance and a UMD should have an explicit opt-in or opt-out mechanism.

4. Is SVM just implicitly enabled? Patch 6 seems to suggest that it is. However, if I'm reading bspec53912 correctly and force-fault is not set in the PPGTT, non-present PPGTT entries that correspond to valid CPU pages would simply be translated and never migrated. If we miss in the CPU page tables (e.g., the CPU has never touched the page), migration could still occur, but only in 4K chunks, which would be terrible for performance.

5. If SVM is explicitly enabled (e.g., through CPU page-table mirroring via a CPU bind), we still have all the same issues described in #4 with migration largely being bypassed. However, we'd at least hit the CPU mirror and could perform larger migrations.

6. The most important question is: why is this being posted at all? Unless we have hard data showing a performance benefit, I'd prefer not to carry this code in the upstream tree. I could see some very narrow use cases (e.g., userptr or SVM with a preferred system-memory location) where avoiding a DMA map might provide a small performance win. However, we'd likely need to rework things so that FF=1 is the default SVM configuration. Alternatively, if we wanted access-counter-driven migration rather than first-touch migration, this might make sense.

Matt

Quoted 24 lines

Himal Prasad Ghimiray (1):
drm/xe/vm: Setup SVA infrastructure for PASID support

Nareshkumar Gollakoti (5):
drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

 drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
 drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
 drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
 drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
 drivers/gpu/drm/xe/xe_vm.c              | 56 +++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h              |  2 +
 drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
 9 files changed, 184 insertions(+)
Signature · 1 line
2.43.0