ATS/PASID SVA support and PRI-less fault servicing
Subsystems (2)
cover ATS/PASID SVA support and PRI-less fault servicing
This series adds initial Shared Virtual Addressing (SVA) support for xe on platforms where ATS and PASID are available but PRI is not enabled or not supported.
For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series adds the xe side infrastructure needed for that flow: VM support for SVA PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID for ATS and PASID enabled contexts.
On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.
To support that case, this series introduces a CPU mirror VMA and uses it as part of a driver managed SVM path for SVA faults without PRI.The purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the driver can rely on the kernel MM/VMA machinery to validate mappings and resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely enough to let faults be service through normal MM paths.
At a high level, this series does the following:
1. log ATS/PASID enablement state during probe,
2. add VM infrastructure needed for SVA PASID support,
3. add a helper to retrieve the VM SVA PASID,
4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
5. add a PCI helper for ATS/PASID operation without PRI,
6. add a CPU mirror VMA for handling SVA faults on non PRI platforms
Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.
Himal Prasad Ghimiray (1):
drm/xe/vm: Setup SVA infrastructure for PASID support
Nareshkumar Gollakoti (5):
drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI
drivers/gpu/drm/xe/regs/xe_lrc_layout.h | 6 +++ drivers/gpu/drm/xe/xe_lrc.c | 37 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c | 49 ++++++++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 6 +++ drivers/gpu/drm/xe/xe_pci.c | 7 ++++ drivers/gpu/drm/xe/xe_pci.h | 15 +++++++ drivers/gpu/drm/xe/xe_vm.c | 56 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 6 +++ 9 files changed, 184 insertions(+)
Signature · 1 line
2.43.0
PATCH 1/6 drm/xe/pci: log ATS/PASID enablement state at probe
Add debug logging in xe_pci_probe() for ATS and PASID enablement state when the corresponding capabilities are present.
This makes platform and IOMMU related bring up issues easier to diagnose.
Also add a small helper to report weather ATS and PASID are both enabled, for use follow up checks that depend on both features.
--- drivers/gpu/drm/xe/xe_pci.c | 7 +++++++ drivers/gpu/drm/xe/xe_pci.h | 6 ++++++ 2 files changed, 13 insertions(+)
drivers/gpu/drm/xe/xe_pci.c +7 -0
@@ -1275,6 +1275,13 @@ static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *desxe_step_name (xe->info.step.media), xe_step_name(xe->info.step.basedie)); +if (pdev->ats_cap) +drm_dbg (&xe->drm,"Arch ATS = %s" , +str_enabled_disabled (pdev->ats_enabled)); +if (pdev->pasid_cap) +drm_dbg (&xe->drm,"Arch PASID = %s" , +str_enabled_disabled (pdev->pasid_enabled)); +drm_dbg (&xe->drm,"SR-IOV support: %s (mode: %s)\n" ,str_yes_no (xe_device_has_sriov (xe)),xe_sriov_mode_to_string (xe_device_sriov_mode (xe)));
drivers/gpu/drm/xe/xe_pci.h +6 -0
@@ -5,6 +5,7 @@#ifndef _XE_PCI_H_#define _XE_PCI_H_ +#include <linux/pci.h> struct pci_dev ;
@@ -12,4 +13,9 @@ int xe_register_pci_driver(void);void xe_unregister_pci_driver (void );struct xe_device *xe_pci_to_pf_device (struct pci_dev *pdev); +static inline bool xe_pci_ats_pasid_enabled (struct pci_dev *pdev) +{ +return pdev->ats_enabled && pdev->pasid_enabled; +} +#endif
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH 1/6] drm/xe/pci: log ATS/PASID enablement state at probe
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] drm/xe/pci: xe_pci_ats_pasid_enabled() always returns false on SR-IOV VFs - [Low] drm/xe/pci…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] drm/xe/pci: xe_pci_ats_pasid_enabled() always returns false on SR-IOV VFs
- [Low] drm/xe/pci: build failure in __xe_pci_probe() without CONFIG_PCI_ATS/PASID --
commit 9613705bbc43ffcbfae8d92b0e6cff920caffa87
Author: Nareshkumar Gollakoti <[email protected]>
drm/xe/pci: log ATS/PASID enablement state at probe
This commit logs ATS and PASID enablement states during __xe_pci_probe() when their PCI capabilities are present, and introduces xe_pci_ats_pasid_enabled() to check whether both features are enabled on a PCI device.
Quoted 5 lines
drivers/gpu/drm/xe/xe_pci.c +0
-0
drivers/gpu/drm/xe/xe_pci.c static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *des · line 1289
Quoted 9 lines
drivers/gpu/drm/xe/xe_pci.c +6
-0@@ -1281,6 +1281,13 @@ static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *desxe_step_name (xe->info.step.media), xe_step_name(xe->info.step.basedie)); +if (pdev->ats_cap) +drm_dbg (&xe->drm,"Arch ATS = %s" , +str_enabled_disabled (pdev->ats_enabled)); +if (pdev->pasid_cap) +drm_dbg (&xe->drm,"Arch PASID = %s" , +str_enabled_disabled (pdev->pasid_enabled));
[Severity: Low]
Will __xe_pci_probe() in drivers/gpu/drm/xe/xe_pci.c fail to build when CONFIG_PCI_ATS or CONFIG_PCI_PASID is disabled?
In include/linux/pci.h, struct pci_dev only defines ats_cap and pasid_cap when their respective Kconfig options are enabled:
#ifdef CONFIG_PCI_ATS ... u16 ats_cap; /* ATS Capability offset */ u8 ats_stu; /* ATS Smallest Translation Unit */ #endif ... #ifdef CONFIG_PCI_PASID u16 pasid_cap; /* PASID Capability offset */ u16 pasid_features;
#endif
Since CONFIG_DRM_XE in drivers/gpu/drm/xe/Kconfig only depends on DRM and PCI without selecting or depending on PCI_ATS or PCI_PASID (for example, in drivers/gpu/drm/xe/.kunitconfig), accessing pdev->ats_cap and pdev->pasid_cap directly without IS_ENABLED() guards breaks the build when either option is not set.
Quoted 17 lines
drivers/gpu/drm/xe/xe_pci.c +1
-0+drm_dbg (&xe->drm,"SR-IOV support: %s (mode: %s)\n" ,str_yes_no (xe_device_has_sriov (xe)),xe_sriov_mode_to_string (xe_device_sriov_mode (xe)));drivers/gpu/drm/xe/xe_pci.h +1
-0@@ -5,6 +5,7 @@#ifndef _XE_PCI_H_#define _XE_PCI_H_ +#include <linux/pci.h> struct pci_dev ;
drivers/gpu/drm/xe/xe_pci.h int xe_register_pci_driver(void); · line 19
Quoted 7 lines
drivers/gpu/drm/xe/xe_pci.h +4
-0@@ -12,4 +13,9 @@ int xe_register_pci_driver(void);void xe_unregister_pci_driver (void );struct xe_device *xe_pci_to_pf_device (struct pci_dev *pdev); +static inline bool xe_pci_ats_pasid_enabled (struct pci_dev *pdev) +{ +return pdev->ats_enabled && pdev->pasid_enabled; +}
[Severity: Medium]
How does xe_pci_ats_pasid_enabled() in drivers/gpu/drm/xe/xe_pci.h (as well as the PASID check in __xe_pci_probe()) behave on SR-IOV Virtual Functions?
In drivers/pci/ats.c, pci_enable_pasid() notes that VFs do not implement their own PASID capability (so pdev->pasid_cap is 0) and instead share the PF's PASID configuration:
if (pdev->is_virtfn) {
if (pci_physfn(pdev)->pasid_enabled)
return 0;
return -EINVAL;
}When the IOMMU enables PASID on a VF via pci_enable_pasid(), it checks pci_physfn(pdev)->pasid_enabled and returns 0 without setting pdev->pasid_enabled on the VF's pci_dev (whereas pci_enable_ats() sets pdev->ats_enabled on the VF).
Because xe_pci_ats_pasid_enabled() and __xe_pci_probe() inspect pdev->pasid_enabled and pdev->pasid_cap directly instead of checking pci_physfn(pdev) (or calling pci_pasid_status(pdev)), won't xe_pci_ats_pasid_enabled() always return false on VFs and __xe_pci_probe() always skip logging the PASID state on VFs?
Quoted 2 lines
drivers/gpu/drm/xe/xe_pci.h +1
-0+#endif
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH 2/6 drm/xe/vm: Setup SVA infrastructure for PASID support
In reply to Nareshkumar GollakotiFrom: Himal Prasad Ghimiray <[email protected]>
Add sva_handle to struct xe_vm to represent the device-mm bond established via iommu_sva_bind_device(). This handle will be used to retrieve the PASID for SVM-enabled VMs
Introduce xe_vm_sva_bind() to bind the VM to the current process address space using Shared Virtual Addressing(SVA) when ATS and PASID are enabled on the device. This allows the GPU to shate the same virtual address space as the CPU.
If sva binding fails, the VM continues without PASID support. Also introduce a cleanup helper to unbind during VM teardown.
--- drivers/gpu/drm/xe/xe_vm.c | 40 ++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm_types.h | 6 +++++ 2 files changed, 46 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +40 -0
@@ -45,6 +45,7 @@#include "xe_trace_bo.h" #include "xe_vm_madvise.h" #include "xe_wa.h" +#include "xe_pci.h" static struct drm_gem_object *xe_vm_obj (struct xe_vm *vm) {
@@ -1648,6 +1649,41 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)up_read (&vm->exec_queues.lock); } +static void xe_vm_sva_handle_init (struct xe_vm *vm) +{ +struct device *dev = vm->xe->drm.dev; +struct pci_dev *pdev =to_pci_dev (dev); +struct iommu_sva *sva_handle; + +/* +* Assumes the IOMMU driver has already decided and latched the +* device's ATS/PASID enablement by the time any VM is created (true +* for Intel VT-d, which enables both eagerly at IOMMU probe). If a +* backend instead enables them lazily on first SVA bind, this check +* will never see them set and SVA/PASID support is silently skipped. +*/ +if (!xe_pci_ats_pasid_enabled (pdev)) +return ; + + sva_handle =iommu_sva_bind_device (dev, current->mm); +if (IS_ERR (sva_handle)) { +drm_err (&vm->xe->drm,"SVA binding failed, preventing PASID creation.\n" ); +return ; + } + + vm->sva_handle = sva_handle; +} + +static void xe_vm_sva_handle_fini (struct xe_vm *vm) +{ +xe_assert (vm->xe,xe_vm_is_closed (vm)); +if (!vm->sva_handle) +return ; + +iommu_sva_unbind_device (vm->sva_handle); + vm->sva_handle =NULL ; +} +struct xe_vm *xe_vm_create (struct xe_device *xe,u32 flags,struct xe_file *xef) {struct drm_gem_object *vm_resv_obj;
@@ -1994,6 +2030,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)xe_svm_fini (vm); +xe_vm_sva_handle_fini (vm);up_write (&vm->lock);down_write (&xe->usm.lock);
@@ -2159,6 +2196,9 @@ int xe_vm_create_ioctl(struct drm_device *dev, void *data,if (IS_ERR (vm))return PTR_ERR (vm); +if (flags &XE_VM_FLAG_FAULT_MODE ) +xe_vm_sva_handle_init (vm); +#if IS_ENABLED (CONFIG_DRM_XE_DEBUG_MEM )/* Warning: Security issue - never enable by default */ args->reserved[0 ] =xe_bo_main_addr (vm->pt_root[0 ]->bo,XE_PAGE_SIZE );
drivers/gpu/drm/xe/xe_vm_types.h +6 -0
@@ -11,6 +11,7 @@#include <drm/drm_pagemap_util.h> #include <linux/dma-resv.h> +#include <linux/iommu.h> #include <linux/kref.h> #include <linux/mmu_notifier.h> #include <linux/scatterlist.h>
@@ -423,6 +424,11 @@ struct xe_vm {bool batch_invalidate_tlb;/** @xef: Xe file handle for tracking this VM's drm client */ struct xe_file *xef; +/** +* @sva_handle: the sva_handle to a device-mm bond, populated using +* iommu_sva_bind_device(). To be used to determine pasid in case of svm vm. +*/ +struct iommu_sva *sva_handle; };/** struct xe_vma_op_map - VMA map operation */
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH 2/6] drm/xe/vm: Setup SVA infrastructure for PASID support
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] drm/xe/vm: missing SVA unbind on device removal leaks DMA owner and crashes --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] drm/xe/vm: missing SVA unbind on device removal leaks DMA owner and crashes --
commit 08c0c9b32bbf8f35bf925eec1c7c7d7bb127ed1a
Author: Himal Prasad Ghimiray <[email protected]>
drm/xe/vm: Setup SVA infrastructure for PASID support
This commit adds an sva_handle field to struct xe_vm and introduces helpers to bind and unbind a fault-mode VM to the current process address space via iommu_sva_bind_device() and iommu_sva_unbind_device().
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0
[ ... ]
Quoted 28 lines
drivers/gpu/drm/xe/xe_vm.c +24
-0@@ -1674,6 +1675,41 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)up_read (&vm->exec_queues.lock); } +static void xe_vm_sva_handle_init (struct xe_vm *vm) +{ +struct device *dev = vm->xe->drm.dev; +struct pci_dev *pdev =to_pci_dev (dev); +struct iommu_sva *sva_handle; + +/* +* Assumes the IOMMU driver has already decided and latched the +* device's ATS/PASID enablement by the time any VM is created (true +* for Intel VT-d, which enables both eagerly at IOMMU probe). If a +* backend instead enables them lazily on first SVA bind, this check +* will never see them set and SVA/PASID support is silently skipped. +*/ +if (!xe_pci_ats_pasid_enabled (pdev)) +return ; + + sva_handle =iommu_sva_bind_device (dev, current->mm); +if (IS_ERR (sva_handle)) { +drm_err (&vm->xe->drm,"SVA binding failed, preventing PASID creation.\n" ); +return ; + } + + vm->sva_handle = sva_handle; +}
drivers/gpu/drm/xe/xe_vm.c static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm) · line 1711
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+ +static void xe_vm_sva_handle_fini (struct xe_vm *vm) +{ +xe_assert (vm->xe,xe_vm_is_closed (vm)); +if (!vm->sva_handle) +return ; + +iommu_sva_unbind_device (vm->sva_handle); + vm->sva_handle =NULL ; +}
[Severity: High]
What happens if the pci device is unbound from the driver or hot-unplugged while a userspace process still holds an open fault-mode xe_vm?
When xe_vm_create_ioctl() calls xe_vm_sva_handle_init(), iommu_sva_bind_device() attaches a pasid domain and inserts an entry into dev->iommu_group->pasid_array. However, vm->sva_handle is only unbound in xe_vm_sva_handle_fini() during xe_vm_close_and_put(), which is tied to the userspace file descriptor and vm lifetime rather than device removal in drivers/gpu/drm/xe/xe_device.c:
void xe_device_remove(struct xe_device *xe)
{
xe_display_unregister(xe);
drm_dev_unplug(&xe->drm);
xe_bo_pci_dev_remove_all(xe);
}In addition, xe_vm_sva_handle_init() does not guard the iommu_sva_bind_device() call with drm_dev_enter(), so an in-flight xe_vm_create_ioctl() can also establish a new bond after drm_dev_unplug() completes.
When __device_release_driver() finishes xe_pci_remove() -> xe_device_remove(), it immediately calls pci_dma_cleanup() -> iommu_device_unuse_default_domain() in drivers/iommu/iommu.c:
mutex_lock(&group->mutex); if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array))) group->owner_cnt--; mutex_unlock(&group->mutex);
Because group->pasid_array is not empty while vm->sva_handle remains bound, does this trigger the WARN_ON() and leak group->owner_cnt?
Furthermore, if the pci device is hot-unplugged and removed from the bus, iommu_release_device() -> __iommu_group_remove_device() -> iommu_deinit_device() clears dev->iommu_group to NULL. When userspace later closes the file descriptor or destroys the vm:
xe_vm_close_and_put()
xe_vm_sva_handle_fini()
iommu_sva_unbind_device()
iommu_detach_device_pasid()
drivers/iommu/iommu.c:iommu_detach_device_pasid() requires the caller to still be a probed driver on dev and dereferences dev->iommu_group directly:
void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev,
ioasid_t pasid)
{
/* Caller must be a probed driver on dev */
struct iommu_group *group = dev->iommu_group;
mutex_lock(&group->mutex);Wouldn't calling iommu_sva_unbind_device() after device removal dereference a NULL group pointer here? Should xe_device_remove() unbind active sva_handle bonds before driver detach completes, and should xe_vm_sva_handle_init() protect the bind with drm_dev_enter() / drm_dev_exit()?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH 3/6 drm/xe/vm: add helper to retrieve VM SVA PASID
In reply to Nareshkumar GollakotiAdd xe_vm_get_sva_pasid() to return the PASID associated with a VM SVA binding.
The helper checks whether the VM has a valid SVA handle and, if so, returns the PASID obtained from iommu_sva_get_pasid(). If no SVA handle is present, return -EINVAL.
Export the helper in xe_vm.h so other XE components can query the VM's PASID when programming per-context state.
--- drivers/gpu/drm/xe/xe_vm.c | 16 ++++++++++++++++ drivers/gpu/drm/xe/xe_vm.h | 2 ++ 2 files changed, 18 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +16 -0
@@ -1649,6 +1649,22 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)up_read (&vm->exec_queues.lock); } +/** +* xe_vm_get_sva_pasid() - get sva pasid for the vm if allocated +* @vm: The vm to get valid sva_handle +* +* Helper to get associated pasid for the vm which uses svm. +* +* Return: %pasid value on Success. -EINVAL if not pasid available. +*/ +int xe_vm_get_sva_pasid (struct xe_vm *vm) +{ +if (!vm || !vm->sva_handle) +return -EINVAL ; + +return iommu_sva_get_pasid (vm->sva_handle); +} +static void xe_vm_sva_handle_init (struct xe_vm *vm) {struct device *dev = vm->xe->drm.dev;
drivers/gpu/drm/xe/xe_vm.h +2 -0
@@ -290,6 +290,8 @@ struct dma_fence *xe_vm_bind_kernel_bo(struct xe_vm *vm, struct xe_bo *bo,void xe_vm_resume_rebind_worker (struct xe_vm *vm); +int xe_vm_get_sva_pasid (struct xe_vm *vm); +/** * xe_vm_resv() - Return's the vm's reservation object * @vm: The vm
Signature · 1 line
2.43.0
PATCH 4/6 drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts
In reply to Nareshkumar GollakotiAdd CTX_SYS_PASID definitions to the LRC layout and program the field during LRC context initialization when ATS and PASID are both enabled on the PCI device.
When ATS/PASID support is available, fetch the VM SVA PASID and write it to CTX_SYS_PASID together with the ATS enable and PASID valid bits. This ensures the GPU context carries the system PASID state required for address translation services.
Bspec: 67168
--- drivers/gpu/drm/xe/regs/xe_lrc_layout.h | 6 ++++ drivers/gpu/drm/xe/xe_lrc.c | 37 +++++++++++++++++++++++++ 2 files changed, 43 insertions(+)
drivers/gpu/drm/xe/regs/xe_lrc_layout.h +6 -0
@@ -18,6 +18,7 @@#define CTX_TIMESTAMP_UDW (0x24 + 1)#define CTX_INDIRECT_RING_STATE (0x26 + 1)#define CTX_ACC_CTR_THOLD (0x2a + 1) +#define CTX_SYS_PASID (0x2c + 1)#define CTX_ASID (0x2e + 1)#define CTX_PDP0_UDW (0x30 + 1)#define CTX_PDP0_LDW (0x32 + 1)
@@ -45,4 +46,9 @@#define INDIRECT_CTX_RING_START_UDW (0x08 + 1)#define INDIRECT_CTX_RING_CTL (0x0a + 1) +/* Fields for CTX_SYS_PASID */ +#define CTX_SYS_ATS_ENABLED REG_BIT(31) +#define CTX_SYS_PASID_VALID REG_BIT(30) +#define CTX_SYS_SYSTEM_PASID_MASK REG_GENMASK(19, 0) +#endif
drivers/gpu/drm/xe/xe_lrc.c +37 -0
@@ -9,6 +9,7 @@#include <linux/ascii85.h> #include <linux/panic.h> +#include <linux/pci.h> #include "instructions/xe_mi_commands.h" #include "instructions/xe_gfxpipe_commands.h"
@@ -35,6 +36,7 @@#include "xe_trace_lrc.h" #include "xe_vm.h" #include "xe_wa.h" +#include "xe_pci.h" #define LRC_VALID BIT_ULL(0)#define LRC_PRIVILEGE BIT_ULL(8)
@@ -998,6 +1000,16 @@ void xe_lrc_write_ctx_reg(struct xe_lrc *lrc, int reg_nr, u32 val)xe_map_write32 (xe, &map, val); } +static void xe_lrc_rmw_ctx_reg (struct xe_lrc *lrc, +int reg_nr,u32 mask,u32 val) +{ +u32 oldval =xe_lrc_read_ctx_reg (lrc, reg_nr); +u32 newval = (oldval & ~mask) | (val & mask); + +if (newval != oldval) +xe_lrc_write_ctx_reg (lrc, reg_nr, newval); +} +static void *empty_lrc_data (struct xe_hw_engine *hwe) {struct xe_gt *gt = hwe->gt;
@@ -1493,6 +1505,26 @@ static u8 xe_multi_queue_prio_to_lrc(struct xe_lrc *lrc, enum xe_multi_queue_prireturn priority; } +static void xe_lrc_program_ctx_sys_pasid (struct xe_lrc *lrc,struct xe_vm *vm) +{ +int pasid; +u32 mask =CTX_SYS_ATS_ENABLED |CTX_SYS_PASID_VALID | +CTX_SYS_SYSTEM_PASID_MASK ; +struct xe_device *xe =lrc_to_xe (lrc); + + pasid =xe_vm_get_sva_pasid (vm); +/* PASID 0 is reserved by the IOMMU for non-PASID-capable devices. */ +if (pasid <=0 ) +return ; + +xe_lrc_rmw_ctx_reg (lrc,CTX_SYS_PASID , mask, +CTX_SYS_ATS_ENABLED | +CTX_SYS_PASID_VALID | + (pasid &CTX_SYS_SYSTEM_PASID_MASK )); +drm_dbg (&xe->drm,"LRC: context enabled with ATS and PASID=%d for VM %p and lrc %p\n" , + pasid, vm, lrc); +} +/** * xe_lrc_set_multi_queue_priority() - Set multi queue priority in LRC * @lrc: Logical Ring Context
@@ -1542,6 +1574,7 @@ static int xe_lrc_ctx_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, structstruct xe_gt *gt = hwe->gt;struct xe_tile *tile =gt_to_tile (gt);struct xe_device *xe =gt_to_xe (gt); +struct pci_dev *pdev =to_pci_dev (xe->drm.dev);struct iosys_map map;u32 arb_enable;u32 state_cache_perf_fix[3 ];
@@ -1642,6 +1675,10 @@ static int xe_lrc_ctx_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, structif (GRAPHICS_VER (xe) >=20 && hwe->class ==XE_ENGINE_CLASS_RENDER )xe_lrc_set_gpgpu_preemption_level (lrc, gt); +/* Program CTX_SYS_PASID when ATS and PASID is supported. */ +if (xe_pci_ats_pasid_enabled (pdev)) +xe_lrc_program_ctx_sys_pasid (lrc, vm); + lrc->desc =LRC_VALID ; lrc->desc |=FIELD_PREP (LRC_ADDRESSING_MODE ,LRC_LEGACY_64B_CONTEXT );/* TODO: Priority */
Signature · 1 line
2.43.0
PATCH 5/6 drm/xe/pci: add helper for ATS/PASID without PRI
In reply to Nareshkumar GollakotiOn platforms where ATS and PASID are enabled but PRI is not available, SVA faults cannot rely on the IOMMU to fault in missing CPU mirror bindings. Those faults must instead be recovered in driver.
Add a small PCI helper to report this condition so fault-handling code can use a single predicate for the PRI-less SVA case.
--- drivers/gpu/drm/xe/xe_pci.h | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/gpu/drm/xe/xe_pci.h +9 -0
@@ -18,4 +18,13 @@ static inline bool xe_pci_ats_pasid_enabled(struct pci_dev *pdev) return pdev->ats_enabled && pdev->pasid_enabled; } +/* +* True when SVA faults must be serviced in software: ATS/PASID is +* enabled but PRI is not available, so the IOMMU has no self-healing path. +*/ +static inline bool xe_pci_svm_needs_sw_iopf (struct pci_dev *pdev) +{ +return xe_pci_ats_pasid_enabled (pdev) && !pdev->pri_enabled; +} +#endif
Signature · 1 line
2.43.0
PATCH 6/6 drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI
In reply to Nareshkumar GollakotiOn ATS/PASID platforms without PRI, xe cannot rely on the hardware page request path to populate missing CPU mirror mappings for SVA faults. This can leave xe_pagefault_service() without a VMA even when the faulting address is valid in the process address space.
Create a CPU-address-mirror VMA on demand for such faults so they can proceed through the existing SVM fault handling path.
Faults against unmapped addresses or unusable VMs continue to fail.
--- drivers/gpu/drm/xe/xe_pagefault.c | 49 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 6 +++ 2 files changed, 55 insertions(+)
drivers/gpu/drm/xe/xe_pagefault.c +49 -0
@@ -17,6 +17,7 @@#include "xe_migrate.h" #include "xe_pagefault.h" #include "xe_pagefault_types.h" +#include "xe_pci.h" #include "xe_pm.h" #include "xe_svm.h" #include "xe_trace_bo.h"
@@ -268,6 +269,7 @@ static int xe_pagefault_service(struct xe_pagefault *pf) {struct xe_gt *gt = pf->gt;struct xe_device *xe =gt_to_xe (gt); +struct pci_dev *pdev =to_pci_dev (xe->drm.dev);struct xe_vm *vm;struct xe_vma *vma =NULL ;int err;
@@ -298,6 +300,51 @@ static int xe_pagefault_service(struct xe_pagefault *pf) } vma =xe_vm_find_vma_by_addr (vm,xe_pagefault_addr (pf)); + +/* +* ATS/PASID is negotiated but this hardware has no PRI, so there is +* no IOMMU self-healing path to fault the page in - the driver has +* to create the missing CPU-address-mirror binding itself and let +* the SVM path below service the fault. +*/ +if (!vma && vm->sva_handle &&xe_pci_svm_needs_sw_iopf (pdev)) { +u64 page_size = vm->flags &XE_VM_FLAG_64K ?SZ_64K :SZ_4K ; +u64 start =ALIGN_DOWN (xe_pagefault_addr (pf), page_size); +u64 end = start + page_size; +int alloc_err =0 ; + +up_read (&vm->lock); +down_write (&vm->lock); +/* Lock was dropped, re-check: a racing fault may have won */ + vma =xe_vm_find_vma_by_addr (vm,xe_pagefault_addr (pf)); +if (!vma) { +if (xe_vm_is_closed_or_banned (vm)) { + alloc_err = -ENOENT ; + }else { +/* Merge with adjacent mirror VMAs instead of leaving singletons */ +xe_vm_find_cpu_addr_mirror_vma_range (vm, &start, &end); + alloc_err =xe_vm_alloc_cpu_addr_mirror_vma (vm, start, end - start); + } +if (!alloc_err) + vma =xe_vm_find_vma_by_addr (vm,xe_pagefault_addr (pf)); + } +downgrade_write (&vm->lock); + +if (alloc_err || !vma || !xe_vma_is_cpu_addr_mirror (vma)) { +/* +* err is what actually tells guc_ack_fault() to NACK; +* distinguish a transient allocation failure from a +* genuinely unmapped/closed address for diagnostics. +*/ + err = alloc_err ? : -EINVAL ; +if (alloc_err == -ENOMEM ) +xe_pagefault_set_error (pf,XE_PAGEFAULT_ERROR_VMA_ALLOC ); +else +xe_pagefault_set_error (pf,XE_PAGEFAULT_ERROR_VMA_NOT_FOUND ); + goto unlock_vm; + } + } +if (!vma) { err = -EINVAL ;xe_pagefault_set_error (pf,XE_PAGEFAULT_ERROR_VMA_NOT_FOUND );
@@ -589,6 +636,8 @@ static const char *xe_pagefault_error_to_str(enum xe_pagefault_error error)return "VM_CLOSED" ;case XE_PAGEFAULT_ERROR_VMA_NOT_FOUND :return "VMA_NOT_FOUND" ; +case XE_PAGEFAULT_ERROR_VMA_ALLOC : +return "VMA_ALLOC" ;case XE_PAGEFAULT_ERROR_READ_ONLY_VIOLATION :return "READ_ONLY_VIOLATION" ;case XE_PAGEFAULT_ERROR_VMA_NEEDS_VRAM_CHECK :
drivers/gpu/drm/xe/xe_pagefault_types.h +6 -0
@@ -59,6 +59,12 @@ enum xe_pagefault_error {XE_PAGEFAULT_ERROR_VM_CLOSED ,/** @XE_PAGEFAULT_ERROR_VMA_NOT_FOUND: No VMA covers the faulted address */ XE_PAGEFAULT_ERROR_VMA_NOT_FOUND , +/** +* @XE_PAGEFAULT_ERROR_VMA_ALLOC: Failed to allocate a CPU-address-mirror +* VMA to service a PRI-less SVA fault (e.g. -ENOMEM); distinct from a +* genuinely unmapped address +*/ + XE_PAGEFAULT_ERROR_VMA_ALLOC,/** * @XE_PAGEFAULT_ERROR_READ_ONLY_VIOLATION: Write/atomic fault on a * read-only VMA
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH 6/6] drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] drm/xe: NULL deref and uninitialized PAT in xe_vm_alloc_cpu_addr_mirror_vma() - [High] drm/xe…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] drm/xe: NULL deref and uninitialized PAT in xe_vm_alloc_cpu_addr_mirror_vma()
- [High] drm/xe: 4K VMA check before 64K mirror alloc in xe_pagefault_service() --
commit c441331b0c386c437e9ad479c73465c57bd86054
Author: Nareshkumar Gollakoti <[email protected]>
drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI
This commit adds on-demand creation of a cpu-address-mirror vma in xe_pagefault_service() when servicing sva faults on platforms with ats/pasid enabled but without pri support. It also introduces the XE_PAGEFAULT_ERROR_VMA_ALLOC error code to report vma allocation failures.
Quoted 4 lines
drivers/gpu/drm/xe/xe_pagefault.c +0
-0
[ ... ]
Quoted 11 lines
drivers/gpu/drm/xe/xe_pagefault.c +7
-0@@ -298,6 +300,51 @@ static int xe_pagefault_service(struct xe_pagefault *pf) } vma =xe_vm_find_vma_by_addr (vm,xe_pagefault_addr (pf)); + +/* +* ATS/PASID is negotiated but this hardware has no PRI, so there is +* no IOMMU self-healing path to fault the page in - the driver has +* to create the missing CPU-address-mirror binding itself and let +* the SVM path below service the fault. +*/
drivers/gpu/drm/xe/xe_pagefault.c static int xe_pagefault_service(struct xe_pagefault *pf) · line 319
Quoted 10 lines
drivers/gpu/drm/xe/xe_pagefault.c +10
-0+if (!vma && vm->sva_handle &&xe_pci_svm_needs_sw_iopf (pdev)) { +u64 page_size = vm->flags &XE_VM_FLAG_64K ?SZ_64K :SZ_4K ; +u64 start =ALIGN_DOWN (xe_pagefault_addr (pf), page_size); +u64 end = start + page_size; +int alloc_err =0 ; + +up_read (&vm->lock); +down_write (&vm->lock); +/* Lock was dropped, re-check: a racing fault may have won */ + vma =xe_vm_find_vma_by_addr (vm,xe_pagefault_addr (pf));
[Severity: High]
When vm->flags & XE_VM_FLAG_64K is set, xe_pagefault_service() aligns start and end to SZ_64K, but xe_vm_find_vma_by_addr() in drivers/gpu/drm/xe/xe_vm.c only checks a 4k range at the faulting address:
drivers/gpu/drm/xe/xe_vm.c:xe_vm_find_vma_by_addr() {
return xe_vm_find_overlapping_vma(vm, page_addr, SZ_4K);
}Because vm_bind_ioctl_check_args() in drivers/gpu/drm/xe/xe_vm.c and madvise_args_are_sane() in drivers/gpu/drm/xe/xe_vm_madvise.c allow 4k-aligned mappings on 64k vms (xe_vm_bind_ioctl_validate_bo() only requires 64k alignment for bos with XE_BO_FLAG_INTERNAL_64K), another 4k-aligned vma (such as a system-memory bo, userptr, sparse vma, or madvise-split vma) can already exist elsewhere within [start, end).
If a fault hits an unmapped 4k page in that 64k window without first checking xe_vm_find_overlapping_vma(vm, start, page_size), won't xe_vm_alloc_cpu_addr_mirror_vma() -> xe_vm_alloc_vma() -> drm_gpuvm_sm_map_ops_create() generate unmap or remap operations that destroy the overlapping vma via xe_vma_destroy() without unbinding its page tables or holding bo->ttm.base.resv, triggering xe_bo_assert_held(bo) in xe_vma_destroy()?
drivers/gpu/drm/xe/xe_pagefault.c static int xe_pagefault_service(struct xe_pagefault *pf) · line 326
Quoted 7 lines
drivers/gpu/drm/xe/xe_pagefault.c +7
-0+if (!vma) { +if (xe_vm_is_closed_or_banned (vm)) { + alloc_err = -ENOENT ; + }else { +/* Merge with adjacent mirror VMAs instead of leaving singletons */ +xe_vm_find_cpu_addr_mirror_vma_range (vm, &start, &end); + alloc_err =xe_vm_alloc_cpu_addr_mirror_vma (vm, start, end - start);
[Severity: Critical]
What happens in xe_pagefault_service() when a fault occurs at an address with no immediately adjacent default-attribute mirror vma at start - SZ_4K or end + 1 (such as the first sva page fault in the vm)?
In that case, xe_vm_find_cpu_addr_mirror_vma_range() leaves [start, end) unchanged so the range does not overlap any existing gpuva in vm->gpuvm.
When xe_vm_alloc_cpu_addr_mirror_vma() calls xe_vm_alloc_vma(vm, &map_req, false) in drivers/gpu/drm/xe/xe_vm.c, vma_flags is initialized to 0 and default_pat is left uninitialized on the stack, because both are only populated from an existing vma during DRM_GPUVA_OP_UNMAP or DRM_GPUVA_OP_REMAP:
drivers/gpu/drm/xe/xe_vm.c:xe_vm_alloc_vma() {
unsigned int vma_flags = 0;
bool remap_op = false;
struct xe_vma_mem_attr tmp_attr = {};
u16 default_pat;
...
if (!is_madvise) {
if (__op->op == DRM_GPUVA_OP_UNMAP) {
vma = gpuva_to_vma(op->base.unmap.va);
XE_WARN_ON(!xe_vma_has_default_mem_attrs(vma));
default_pat = vma->attr.default_pat_index;
vma_flags = vma->gpuva.flags;
}
if (__op->op == DRM_GPUVA_OP_REMAP) {
vma = gpuva_to_vma(op->base.remap.unmap->va);
default_pat = vma->attr.default_pat_index;
vma_flags = vma->gpuva.flags;
}
if (__op->op == DRM_GPUVA_OP_MAP) {
op->map.vma_flags |= vma_flags & XE_VMA_CREATE_MASK;
op->map.pat_index = default_pat;
}
}
}Because drm_gpuvm_sm_map_ops_create() only emits DRM_GPUVA_OP_MAP when no existing gpuva overlaps [start, end), op->map.vma_flags remains 0 (missing XE_VMA_SYSTEM_ALLOCATOR) and op->map.pat_index uses the uninitialized default_pat variable.
From there, vm_bind_ioctl_ops_parse() -> new_vma() -> xe_vma_create() in drivers/gpu/drm/xe/xe_vm.c sees !bo && !is_null && !is_cpu_addr_mirror, allocates a struct xe_userptr_vma instead of a mirror vma, and calls xe_userptr_setup() in drivers/gpu/drm/xe/xe_userptr.c:
drivers/gpu/drm/xe/xe_userptr.c:xe_userptr_setup() {
...
err = mmu_interval_notifier_insert(&userptr->notifier, current->mm,
start, range,
&vma_userptr_notifier_ops);
}Since xe_pagefault_service() runs on a workqueue kthread where current->mm is NULL, won't this cause a null pointer dereference inside mmu_interval_notifier_insert()?
Quoted 1 line
drivers/gpu/drm/xe/xe_pagefault.c +1
-0+ }
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Patchwork
CI✗ CI.KUnit: failure for ATS/PASID SVA support and PRI-less fault servicing
In reply to Nareshkumar Gollakoti== Series Details ==
== Series Details ==
Series: ATS/PASID SVA support and PRI-less fault servicing
URL : https://patchwork.freedesktop.org/series/176519/
State : failure
== Summary ==
+ trap cleanup EXIT
+ kunitconfigs=('/kernel/drivers/gpu/tests/.kunitconfig' '/kernel/drivers/gpu/drm/xe/.kunitconfig' '/kernel/drivers/gpu/drm/tests/.kunitconfig' '/kernel/drivers/gpu/drm/ttm/tests/.kunitconfig' '/kernel/drivers/dma-buf/.kunitconfig')
+ for kcfg in "${kunitconfigs[@]}"
+ [[ ! -f /kernel/drivers/gpu/tests/.kunitconfig ]]
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/tests/.kunitconfig
[04:20:43] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [04:20:47] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48 [04:21:08] Starting KUnit Kernel (1/1)... [04:21:08] ============================================================ Running tests with: $ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt [04:21:08] ================= gpu_buddy (14 subtests) ================== [04:21:08] [PASSED] gpu_test_buddy_alloc_limit [04:21:08] [PASSED] gpu_test_buddy_alloc_optimistic [04:21:08] [PASSED] gpu_test_buddy_alloc_pessimistic [04:21:08] [PASSED] gpu_test_buddy_alloc_pathological [04:21:08] [PASSED] gpu_test_buddy_alloc_contiguous [04:21:08] [PASSED] gpu_test_buddy_alloc_clear [04:21:08] [PASSED] gpu_test_buddy_alloc_range [04:21:09] [PASSED] gpu_test_buddy_alloc_range_bias [04:21:10] [PASSED] gpu_test_buddy_fragmentation_performance [04:21:10] [PASSED] gpu_test_buddy_dirty_tracker_performance [04:21:10] [PASSED] gpu_test_buddy_alloc_exceeds_max_order [04:21:10] [PASSED] gpu_test_buddy_offset_aligned_allocation [04:21:10] [PASSED] gpu_test_buddy_subtree_offset_alignment_stress [04:21:10] [PASSED] gpu_test_buddy_addr_to_block [04:21:10] ==================== [PASSED] gpu_buddy ==================== [04:21:10] ============================================================ [04:21:10] Testing complete. Ran 14 tests: passed: 14 [04:21:10] Elapsed time: 27.450s total, 4.472s configuring, 20.860s building, 2.054s running
+ for kcfg in "${kunitconfigs[@]}"
+ [[ ! -f /kernel/drivers/gpu/drm/xe/.kunitconfig ]]
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig ERROR:root:../drivers/gpu/drm/xe/xe_pci.c: In function ‘__xe_pci_probe’:../drivers/gpu/drm/xe/xe_pci.c:1288:19: error: ‘struct pci_dev’ has no member named ‘pasid_cap’; did you mean ‘pcie_cap’?
1288 | if (pdev->pasid_cap)
| ^~~~~~~~~
| pcie_capmake[7]: *** [../scripts/Makefile.build:290: drivers/gpu/drm/xe/xe_pci.o] Error 1 make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:551: drivers/gpu/drm/xe] Error 2 make[5]: *** [../scripts/Makefile.build:551: drivers/gpu/drm] Error 2 make[4]: *** [../scripts/Makefile.build:551: drivers/gpu] Error 2 make[3]: *** [../scripts/Makefile.build:551: drivers] Error 2 make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/kernel/Makefile:2229: .] Error 2
make[1]: *** [/kernel/Makefile:248: __sub-make] Error 2
make: *** [Makefile:248: __sub-make] Error 2
[04:21:10] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [04:21:12] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48 + cleanup ++ stat -c %u:%g /kernel + chown -R 1003:1003 /kernel
Thomas Hellström
maintainerHi!
Quoted 49 lines · Nareshkumar Gollakoti
This series adds initial Shared Virtual Addressing (SVA) support for xe
on platforms where ATS and PASID are available but PRI is not enabled or
not supported.For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series
adds the xe side infrastructure needed for that flow: VM support for SVA
PASID state, PASID retriveal helpers, and programming ofCTX_SYS_PASID
for ATS and PASID enabled contexts.On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need
to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In
that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.To support that case, this series introduces a CPU mirror VMA and uses
it as part of a driver managed SVM path for SVA faults without PRI.The
purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the
driver can rely on the kernel MM/VMA machinery to validate mappings and
resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely
enough to let faults be service through normal MM paths.At a high level, this series does the following:
1. log ATS/PASID enablement state during probe,
2. add VM infrastructure needed for SVA PASID support,
3. add a helper to retrieve the VM SVA PASID,
4. programCTX_SYS_PASIDfor ATS/PASID-enabled contexts,
5. add a PCI helper for ATS/PASID operation without PRI,
6. add a CPU mirror VMA for handling SVA faults on non PRI platforms
Quoted 4 lines
Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.
I was under the impression that we'd integrate this with current SVM so that we can still migrate to VRAM and use page-table mirroring or keep data in system memory and use SVA page-table sharing *OR* page-table- mirroring depending on user-space's wish?
Thanks,
Thomas
Quoted 22 lines
Himal Prasad Ghimiray (1):
drm/xe/vm: Setup SVA infrastructure for PASID supportNareshkumar Gollakoti (5):
drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: programCTX_SYS_PASIDfor ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRIdrivers/gpu/drm/xe/regs/xe_lrc_layout.h | 6 +++ drivers/gpu/drm/xe/xe_lrc.c | 37 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c | 49 ++++++++++++++++++++++
drivers/gpu/drm/xe/xe_pagefault_types.h| 6 +++ drivers/gpu/drm/xe/xe_pci.c | 7 ++++ drivers/gpu/drm/xe/xe_pci.h | 15 +++++++ drivers/gpu/drm/xe/xe_vm.c | 56
+++++++++++++++++++++++++
drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 6 +++9 files changed, 184 insertions(+)
Hi Thomas,
Quoted 56 lines · Thomas Hellström
Hi!
Quoted 49 lines · Nareshkumar Gollakoti
This series adds initial Shared Virtual Addressing (SVA) support for xe
on platforms where ATS and PASID are available but PRI is not enabled or
not supported.For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series
adds the xe side infrastructure needed for that flow: VM support for SVA
PASID state, PASID retriveal helpers, and programming ofCTX_SYS_PASID
for ATS and PASID enabled contexts.On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need
to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In
that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.To support that case, this series introduces a CPU mirror VMA and uses
it as part of a driver managed SVM path for SVA faults without PRI.The
purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the
driver can rely on the kernel MM/VMA machinery to validate mappings and
resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely
enough to let faults be service through normal MM paths.At a high level, this series does the following:
1. log ATS/PASID enablement state during probe,
2. add VM infrastructure needed for SVA PASID support,
3. add a helper to retrieve the VM SVA PASID,
4. programCTX_SYS_PASIDfor ATS/PASID-enabled contexts,
5. add a PCI helper for ATS/PASID operation without PRI,
6. add a CPU mirror VMA for handling SVA faults on non PRI platformsQuoted 4 lines
Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.
Quoted 7 lines
I was under the impression that we'd integrate this with current SVM so that we can still migrate to VRAM and use page-table mirroring or keep data in system memory and use SVA page-table sharing *OR* page-table- mirroring depending on user-space's wish?
Thanks,
Thomas
IMO, Yes, that is the intended model. Userspace picks between the two approaches with the bind.
Bind with CPU_ADDR_MIRROR → existing SVM. GPU page fault, optional VRAM migrate, PPGTT mirroring and this path is unchanged.
Do not bind → SVA page-table sharing. IOMMU walks the CPU page tables with ATS+PASID. Backing has to stay in system memory. This is the end-to-end share flow, and it needs ATS+PASID+PRI.
If PRI is missing, an ATS miss cannot be healed by the IOMMU. The device then reports a GPU page fault. Instead of NACKing that unbound fault, we create a CPU-mirror VMA and service it with the same SVM path so the GPU gets a PPGTT mapping.
So, ATS+PASID+PRI → share works as user space asked (no bind). Bind with CPU_ADDR_MIRROR → SVM, independent of PRI. *PRI-less is only the recovery path when share cannot complete at the IOMMU*.(This is one of the proposal)
Let me know if anything am missing.
Thanks,
Naresh
Quoted 22 lines
Quoted 22 lines
Himal Prasad Ghimiray (1):
drm/xe/vm: Setup SVA infrastructure for PASID supportNareshkumar Gollakoti (5):
drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: programCTX_SYS_PASIDfor ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRIdrivers/gpu/drm/xe/regs/xe_lrc_layout.h | 6 +++ drivers/gpu/drm/xe/xe_lrc.c | 37 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c | 49 ++++++++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 6 +++ drivers/gpu/drm/xe/xe_pci.c | 7 ++++ drivers/gpu/drm/xe/xe_pci.h | 15 +++++++ drivers/gpu/drm/xe/xe_vm.c | 56 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 6 +++ 9 files changed, 184 insertions(+)
Thomas Hellström
maintainerQuoted 105 lines · Naresh
Hi Thomas,
Quoted 79 lines · Thomas Hellström
Hi!
Quoted 61 lines · Nareshkumar Gollakoti
This series adds initial Shared Virtual Addressing (SVA) support for
xe
on platforms where ATS and PASID are available but PRI is not enabled
or
not supported.For SVA, the device must be able to issue memory transactions tagged
with a PASID so that address translation is performed in the process
address space rather than in a driver owned address space. This series
adds the xe side infrastructure needed for that flow: VM support for
SVA
PASID state, PASID retriveal helpers, and programming ofCTX_SYS_PASID
for ATS and PASID enabled contexts.On systems with PRI, device page faults can be surfaced and resolved
through the hardware page request interface, so the driver does not
need
to implement a separate fault servicing path for SVA. On non PRI systems, that hardware assisted fault path is unavailable.In
that case, the driver still needs a way to resolve faults against the
backing process address space so execution can proceed.To support that case, this series introduces a CPU mirror VMA and uses
it as part of a driver managed SVM path for SVA faults without PRI.The
purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so
the
driver can rely on the kernel MM/VMA machinery to validate mappings
and
resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver
needs a CPU visible VMA context that mirrors the GPU SVA range closely
enough to let faults be service through normal MM paths.At a high level, this series does the following:
1. log ATS/PASID enablement state during probe,
2. add VM infrastructure needed for SVA PASID support,
3. add a helper to retrieve the VM SVA PASID,
4. programCTX_SYS_PASIDfor ATS/PASID-enabled contexts,
5. add a PCI helper for ATS/PASID operation without PRI,
6. add a CPU mirror VMA for handling SVA faults on non PRI platformsQuoted 7 lines
Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for
this flow and have been validated internally.Patch 6 adds the non PRI
fault servicing path, and feedback on that overall approach would be
especially appreciated.I was under the impression that we'd integrate this with current SVM so
that we can still migrate to VRAM and use page-table mirroring or keep
data in system memory and use SVA page-table sharing *OR* page- table-
mirroring depending on user-space's wish?Quoted 2 lines
Thanks,
ThomasIMO, Yes, that is the intended model. Userspace picks between the two approaches with the bind.
Bind with CPU_ADDR_MIRROR → existing SVM. GPU page fault, optional VRAM
migrate, PPGTT mirroring and this path is unchanged.Do not bind → SVA page-table sharing. IOMMU walks the CPU page tables with ATS+PASID. Backing has to stay in system memory. This is the end-to-end share flow, and it needs ATS+PASID+PRI.
If PRI is missing, an ATS miss cannot be healed by the IOMMU. The device
then reports a GPU page fault. Instead of NACKing that unbound fault, we
create a CPU-mirror VMA and service it with the same SVM path so the GPU
gets a PPGTT mapping.
Quoted 8 lines
So, ATS+PASID+PRI → share works as user space asked (no bind). Bind with CPU_ADDR_MIRROR → SVM, independent of PRI. *PRI-less is only the recovery path when share cannot complete at the IOMMU*.(This is one of
the proposal)Let me know if anything am missing.
Yes, first on the (implicit) UAPI, KMD must never implicitly create a GPU_VMA. This is done to be able to handle page-faults for non-PRI capable hardware. Couldn't that be done much easier with a separate page-fault handler that just calls fixup_user_fault() or a similar in- kernel API and mark the gpu fault as handled? Then I assume the ATS+PASID request will be retried?
On the implicit "Everything not having a GPU vma will be using ATS/PASID" I strongly oppose that. Was this discussed on the UAPI channel? Does NEO have any patches published to back this up? ATS/PASID enables us to HW optimize a subset of the existing SVM functionality, and IMO KMD is the best judge of whether to use it or not, possibly with a UMD madvise hint. IMO we should require a CPU_ADDR_MIRROR vma covering also the ATS/PASID regions and add an MADVISE setting for user-space to prefer ATS/PASID lookups over current hmm mirroring. If user-space then selects an MADVISE setting that is incompatible with ATS/PASID (like different PAT, VRAM preferred region etc.) Then KMD can transparently move over to that. We can also reuse existing igts and user-space. If UMD needs to detect whether ATS/PASID is usable on the VM then I figure we also must consider that.
Finally, it looks like you don't use the XE_PAGE_FORCE_FAULT bit, like in here:
https://patchwork.freedesktop.org/patch/657641/?series=149886&rev=3
How would you then stop the hardware from attempting an ATS/PASID lookup rather than faulting in mapped VMA?
Thanks,
Thomas
Quoted 31 lines
Thanks,
Naresh
Quoted 25 lines
Quoted 25 lines
Himal Prasad Ghimiray (1):
drm/xe/vm: Setup SVA infrastructure for PASID supportNareshkumar Gollakoti (5):
drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: programCTX_SYS_PASIDfor ATS/PASID-enabled contexts
drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRIdrivers/gpu/drm/xe/regs/xe_lrc_layout.h | 6 +++ drivers/gpu/drm/xe/xe_lrc.c | 37 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c | 49
++++++++++++++++++++++
drivers/gpu/drm/xe/xe_pagefault_types.h | 6 +++ drivers/gpu/drm/xe/xe_pci.c | 7 ++++ drivers/gpu/drm/xe/xe_pci.h | 15 +++++++ drivers/gpu/drm/xe/xe_vm.c | 56
+++++++++++++++++++++++++
drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 6 +++9 files changed, 184 insertions(+)
Matthew Brost
maintainerQuoted 37 lines · Nareshkumar Gollakoti
This series adds initial Shared Virtual Addressing (SVA) support for xe on platforms where ATS and PASID are available but PRI is not enabled or not supported.
For SVA, the device must be able to issue memory transactions tagged with a PASID so that address translation is performed in the process address space rather than in a driver owned address space. This series adds the xe side infrastructure needed for that flow: VM support for SVA PASID state, PASID retriveal helpers, and programming of
CTX_SYS_PASIDfor ATS and PASID enabled contexts.On systems with PRI, device page faults can be surfaced and resolved through the hardware page request interface, so the driver does not need to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In that case, the driver still needs a way to resolve faults against the backing process address space so execution can proceed.To support that case, this series introduces a CPU mirror VMA and uses it as part of a driver managed SVM path for SVA faults without PRI.The purpose of the CPU mirror VMA is to provide a CPU side VMA representation for the address range being accesses through SVA,so the driver can rely on the kernel MM/VMA machinery to validate mappings and resolve backing pages for fault handling in the absence of PRI.In other words, since the device cannot issue PRI requests, the driver needs a CPU visible VMA context that mirrors the GPU SVA range closely enough to let faults be service through normal MM paths.
At a high level, this series does the following:
1. log ATS/PASID enablement state during probe,
2. add VM infrastructure needed for SVA PASID support,
3. add a helper to retrieve the VM SVA PASID,
4. programCTX_SYS_PASIDfor ATS/PASID-enabled contexts,
5. add a PCI helper for ATS/PASID operation without PRI,
6. add a CPU mirror VMA for handling SVA faults on non PRI platforms
Quoted 5 lines
Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for this flow and have been validated internally.Patch 6 adds the non PRI fault servicing path, and feedback on that overall approach would be especially appreciated.
Many concerns with this series, and I'm also generally quite confused about the flows here. Let's take a step back and dig into what you're trying to accomplish and why.
Prior to this series:
VM bind sets up PPGTTs.
- System memory uses DMA addresses (via dma-map) with DM=0 in the entry. These are GPAs (no pasid) translated by the IOMMU (or via ATS).
- Device memory uses DPAs with DM=1 in the entry.
SVM is configured via VM bind and CPU page-table mirroring.
- Pagefault or prefetches create mappings, data may or may not migrate.
After this series, all I have is questions and none are explained by the cover-letter or patch commits:
1. Is a user expected to call VM bind? It seems the answer is yes.
2. Do we need any changes to the PPGTT setup? bspec53912 seems to indicate that we do, but I'm not seeing any PPGTT setup changes in this series. At the very minimum, I'd expect userptr to not use dma-address and rely a PPGTT entry with P=0, FF=0 so the PASID walk occurs.
3. Are any programming model changes required in the UMD? If so, what are they? I'm also concerned that this becomes a blind opt-in whenever ATS + PASID are supported, as this would surely affect performance and a UMD should have an explicit opt-in or opt-out mechanism.
4. Is SVM just implicitly enabled? Patch 6 seems to suggest that it is. However, if I'm reading bspec53912 correctly and force-fault is not set in the PPGTT, non-present PPGTT entries that correspond to valid CPU pages would simply be translated and never migrated. If we miss in the CPU page tables (e.g., the CPU has never touched the page), migration could still occur, but only in 4K chunks, which would be terrible for performance.
5. If SVM is explicitly enabled (e.g., through CPU page-table mirroring via a CPU bind), we still have all the same issues described in #4 with migration largely being bypassed. However, we'd at least hit the CPU mirror and could perform larger migrations.
6. The most important question is: why is this being posted at all? Unless we have hard data showing a performance benefit, I'd prefer not to carry this code in the upstream tree. I could see some very narrow use cases (e.g., userptr or SVM with a preferred system-memory location) where avoiding a DMA map might provide a small performance win. However, we'd likely need to rework things so that FF=1 is the default SVM configuration. Alternatively, if we wanted access-counter-driven migration rather than first-touch migration, this might make sense.
Matt
Quoted 24 lines
Himal Prasad Ghimiray (1):
drm/xe/vm: Setup SVA infrastructure for PASID supportNareshkumar Gollakoti (5):
drm/xe/pci: log ATS/PASID enablement state at probe drm/xe/vm: add helper to retrieve VM SVA PASID drm/xe/lrc: programCTX_SYS_PASIDfor ATS/PASID-enabled contexts drm/xe/pci: add helper for ATS/PASID without PRI drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRIdrivers/gpu/drm/xe/regs/xe_lrc_layout.h | 6 +++ drivers/gpu/drm/xe/xe_lrc.c | 37 ++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault.c | 49 ++++++++++++++++++++++ drivers/gpu/drm/xe/xe_pagefault_types.h | 6 +++ drivers/gpu/drm/xe/xe_pci.c | 7 ++++ drivers/gpu/drm/xe/xe_pci.h | 15 +++++++ drivers/gpu/drm/xe/xe_vm.c | 56 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 6 +++ 9 files changed, 184 insertions(+)Signature · 1 line
2.43.0