drm/xe/madvise: Add support for purgeable buffer objects
Subsystems (3)
v2 cover drm/xe/madvise: Add support for purgeable buffer objects
This patch series introduces comprehensive support for purgeable buffer objects in the Xe driver, enabling userspace to provide memory usage hints for better memory management under system pressure.
Overview:
Purgeable memory allows applications to mark buffer objects as "not currently needed" (DONTNEED), making them eligible for kernel reclamation during memory pressure. This helps prevent OOM conditions and enables more efficient GPU memory utilization for workloads with temporary or regeneratable data (caches, intermediate results, decoded frames, etc.).
Purgeable BO Lifecycle:
1. WILLNEED (default): BO actively needed, kernel preserves backing store
2. DONTNEED (user hint): BO contents discardable, eligible for purging
3. PURGED (kernel action): Backing store reclaimed during memory pressure
Key Design Principles:
- i915 compatibility: "Once purged, always purged" semantics - purged BOs remain permanently invalid and must be destroyed/recreated
- Safety first: Only non-shared BOs can be marked DONTNEED to prevent multi-process data corruption
- Multiple protection layers: Validation in madvise, VM bind, mmap, and fault handlers
- Async TLB invalidation: Uses xe_bo_trigger_rebind() for non-blocking GPU mapping invalidation
- Scratch PTE support: Fault-mode VMs use scratch pages for safe zero reads on purged BO access
Error Handling:
- CPU access (mmap): Returns VM_FAULT_SIGBUS (SIGBUS signal to process)
- GPU access (non-scratch VM): Page fault fails with -EACCES, GPU context reset
- GPU access (scratch VM): Page fault succeeds, rebinds with scratch PTEs
- VM_BIND operations: MAP/PREFETCH rejected with -EINVAL
- Mmap offset ioctl: Rejected with -EINVAL for early error detection
v2 Changes:
- Reordered patches: Moved shared BO helper before main implementation for proper dependency order
- Fixed reference counting in mmap offset validation (use drm_gem_object_put)
- Removed incorrect claims about madvise(WILLNEED) restoring purged BOs
- Fixed error code documentation inconsistencies
- Initialize purge_state_val fields to prevent kernel memory leaks
- Use xe_bo_trigger_rebind() for async TLB invalidation (Thomas Hellström)
- Add NULL rebind with scratch PTEs for fault mode (Thomas Hellström)
- Implement i915-compatible retained field logic (Thomas Hellström)
- Skip BO validation for purged BOs in page fault handler (crash fix)
- Add scratch VM check in page fault path (non-scratch VMs fail fault)
Arvind Yadav (7):
drm/xe/bo: Add purgeable bo state tracking and field madv to xe_bo drm/xe/bo: Prevent purging of shared buffer objects drm/xe/madvise: Implement purgeable buffer object support drm/xe/bo: Handle CPU faults on purged buffer objects drm/xe/bo: Prevent mmap of purged buffer objects drm/xe/vm: Prevent binding of purged buffer objects drm/xe: Add support for querying purgeable BO states
Himal Prasad Ghimiray (2):
drm/xe/uapi: Add UAPI support for purgeable buffer objects drm/xe/uapi: Add UAPI for purgeable bo state to madvise query response
drivers/gpu/drm/xe/xe_bo.c | 92 ++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_bo.h | 57 +++++++++++++++++ drivers/gpu/drm/xe/xe_bo_types.h | 3 + drivers/gpu/drm/xe/xe_gt_pagefault.c | 19 ++++++ drivers/gpu/drm/xe/xe_pt.c | 13 +++- drivers/gpu/drm/xe/xe_vm.c | 12 ++++ drivers/gpu/drm/xe/xe_vm_madvise.c | 76 +++++++++++++++++++++++ include/uapi/drm/xe_drm.h | 50 ++++++++++++++- 8 files changed, 308 insertions(+), 14 deletions(-)
Signature · 1 line
2.43.0
RFC v2 1/9 drm/xe/uapi: Add UAPI support for purgeable buffer objects
From: Himal Prasad Ghimiray <[email protected]>
Extend the DRM_XE_MADVISE ioctl to support purgeable buffer object management by adding DRM_XE_VMA_ATTR_PURGEABLE_STATE attribute type.
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
- WILLNEED: Buffer is needed and should not be purged. If the BO was previously purged, retained field returns 0 indicating backing store was lost (once purged, always purged semantics matching i915).
- DONTNEED: Buffer is not currently needed and may be purged by the kernel under memory pressure to free resources. Only applies to non-shared BOs.
The implementation includes a 'retained' output field (matching i915's drm_i915_gem_madvise.retained) that indicates whether the BO's backing store still exists (1) or has been purged (0).
v2: Add PURGED state for read-only status, change ioctl to DRM_IOWR,
add retained field for i915 compatibility
--- include/uapi/drm/xe_drm.h | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
include/uapi/drm/xe_drm.h +36 -1
@@ -121,7 +121,7 @@ extern "C" {#define DRM_IOCTL_XE_EXEC DRM_IOW(DRM_COMMAND_BASE + DRM_XE_EXEC, struct drm_xe_exec)#define DRM_IOCTL_XE_WAIT_USER_FENCE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_WAIT_USER_FENCE, struct drm_xe_wait_user_fence)#define DRM_IOCTL_XE_OBSERVATION DRM_IOW(DRM_COMMAND_BASE + DRM_XE_OBSERVATION, struct drm_xe_observation_param) -#define DRM_IOCTL_XE_MADVISE DRM_IOW(DRM_COMMAND_BASE + DRM_XE_MADVISE, struct drm_xe_madvise) +#define DRM_IOCTL_XE_MADVISE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_MADVISE, struct drm_xe_madvise)#define DRM_IOCTL_XE_VM_QUERY_MEM_RANGE_ATTRS DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_VM_QUERY_MEM_RANGE_ATTRS, struct drm_xe_vm_query_mem_range_attr)/**
@@ -2051,6 +2051,7 @@ struct drm_xe_madvise {#define DRM_XE_MEM_RANGE_ATTR_PREFERRED_LOC 0#define DRM_XE_MEM_RANGE_ATTR_ATOMIC 1#define DRM_XE_MEM_RANGE_ATTR_PAT 2 +#define DRM_XE_VMA_ATTR_PURGEABLE_STATE 3/** @type: type of attribute */ __u32 type;
@@ -2129,6 +2130,40 @@ struct drm_xe_madvise {/** @pat_index.reserved: Reserved */ __u64 reserved; } pat_index; + +/** +* @purge_state_val: Purgeable state configuration +* +* Used when @type == DRM_XE_VMA_ATTR_PURGEABLE_STATE. +* +* Configures the purgeable state of buffer objects in the specified +* virtual address range. This allows applications to hint to the kernel +* about bo's usage patterns for better memory management. +* +* Supported values for @purge_state_val.val: +* - DRM_XE_VMA_PURGEABLE_STATE_WILLNEED (0): Marks BO as needed. +* If BO was purged, returns retained=0 (backing store lost). +* +* - DRM_XE_VMA_PURGEABLE_STATE_DONTNEED (1): Hints that BO is not +* currently needed. Kernel may purge it under memory pressure. +* Only applies to non-shared BOs. Returns retained=1 if not purged. +*/ +struct { +#define DRM_XE_VMA_PURGEABLE_STATE_WILLNEED 0 +#define DRM_XE_VMA_PURGEABLE_STATE_DONTNEED 1 +/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ +__u32 val; +/** +* @purge_state_val.retained: Whether the backing store still exists. +* +* Output field indicating if the BO's backing store is retained. +* Set to 1 if backing store exists, 0 if it has been purged. +* Similar to i915's drm_i915_gem_madvise.retained field. +*/ +__u32 retained; +/** @purge_state_val.reserved: Reserved */ +__u64 reserved; + } purge_state_val; };/** @reserved: Reserved */
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 1/9] drm/xe/uapi: Add UAPI support for purgeable buffer objects
Quoted 39 lines · Arvind Yadav
From: Himal Prasad Ghimiray <[email protected]>
Extend the
DRM_XE_MADVISEioctl to support purgeable buffer object management by addingDRM_XE_VMA_ATTR_PURGEABLE_STATEattribute type.This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
- WILLNEED: Buffer is needed and should not be purged. If the BO was previously purged, retained field returns 0 indicating backing store was lost (once purged, always purged semantics matching i915).
- DONTNEED: Buffer is not currently needed and may be purged by the kernel under memory pressure to free resources. Only applies to non-shared BOs.
The implementation includes a 'retained' output field (matching i915's drm_i915_gem_madvise.retained) that indicates whether the BO's backing store still exists (1) or has been purged (0).
v2: Add PURGED state for read-only status, change ioctl to
DRM_IOWR,
add retained field for i915 compatibilityCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Signed-off-by Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- include/uapi/drm/xe_drm.h | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)include/uapi/drm/xe_drm.h +0
-0
include/uapi/drm/xe_drm.h extern "C" { · line 124
Quoted 5 lines
include/uapi/drm/xe_drm.h +1
-1@@ -121,7 +121,7 @@ extern "C" {#define DRM_IOCTL_XE_EXEC DRM_IOW(DRM_COMMAND_BASE + DRM_XE_EXEC, struct drm_xe_exec)#define DRM_IOCTL_XE_WAIT_USER_FENCE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_WAIT_USER_FENCE, struct drm_xe_wait_user_fence)#define DRM_IOCTL_XE_OBSERVATION DRM_IOW(DRM_COMMAND_BASE + DRM_XE_OBSERVATION, struct drm_xe_observation_param) -#define DRM_IOCTL_XE_MADVISE DRM_IOW(DRM_COMMAND_BASE + DRM_XE_MADVISE, struct drm_xe_madvise) +#define DRM_IOCTL_XE_MADVISE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_MADVISE, struct drm_xe_madvise)
I'm not sure if this is something we are allowed to change per Linux uAPI rules. I'd check with our maintainers (Thomas, Rodrigo) on this one.
Quoted 36 lines
include/uapi/drm/xe_drm.h +22
-0#define DRM_IOCTL_XE_VM_QUERY_MEM_RANGE_ATTRS DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_VM_QUERY_MEM_RANGE_ATTRS, struct drm_xe_vm_query_mem_range_attr)/** @@ -2051,6 +2051,7 @@ struct drm_xe_madvise {#define DRM_XE_MEM_RANGE_ATTR_PREFERRED_LOC 0#define DRM_XE_MEM_RANGE_ATTR_ATOMIC 1#define DRM_XE_MEM_RANGE_ATTR_PAT 2 +#define DRM_XE_VMA_ATTR_PURGEABLE_STATE 3/** @type: type of attribute */ __u32 type;@@ -2129,6 +2130,40 @@ struct drm_xe_madvise {/** @pat_index.reserved: Reserved */ __u64 reserved; } pat_index; + +/** +* @purge_state_val: Purgeable state configuration +* +* Used when @type == DRM_XE_VMA_ATTR_PURGEABLE_STATE. +* +* Configures the purgeable state of buffer objects in the specified +* virtual address range. This allows applications to hint to the kernel +* about bo's usage patterns for better memory management. +* +* Supported values for @purge_state_val.val: +* - DRM_XE_VMA_PURGEABLE_STATE_WILLNEED (0): Marks BO as needed. +* If BO was purged, returns retained=0 (backing store lost). +* +* - DRM_XE_VMA_PURGEABLE_STATE_DONTNEED (1): Hints that BO is not +* currently needed. Kernel may purge it under memory pressure. +* Only applies to non-shared BOs. Returns retained=1 if not purged. +*/ +struct { +#define DRM_XE_VMA_PURGEABLE_STATE_WILLNEED 0 +#define DRM_XE_VMA_PURGEABLE_STATE_DONTNEED 1
include/uapi/drm/xe_drm.h struct drm_xe_madvise { · line 2163
Quoted 10 lines
include/uapi/drm/xe_drm.h +10
-0+/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ +__u32 val; +/** +* @purge_state_val.retained: Whether the backing store still exists. +* +* Output field indicating if the BO's backing store is retained. +* Set to 1 if backing store exists, 0 if it has been purged. +* Similar to i915's drm_i915_gem_madvise.retained field. +*/ +__u32 retained;
If we can't change the IOCTL to DRM_IOWR, then we could hack around this restriction with making 'retained' a userptr which the madvise IOCTL explictly copies back into rather than relying DRM IOCTL core to implement the copy_to_user.
Matt
Quoted 9 lines
include/uapi/drm/xe_drm.h +3
-0+/** @purge_state_val.reserved: Reserved */ +__u64 reserved; + } purge_state_val; };/** @reserved: Reserved */ Signature · 1 line
2.43.0
Re: [RFC v2 1/9] drm/xe/uapi: Add UAPI support for purgeable buffer objects
Quoted 91 lines · Matthew Brost
Quoted 39 lines · Arvind Yadav
From: Himal Prasad Ghimiray <[email protected]>
Extend the
DRM_XE_MADVISEioctl to support purgeable buffer object management by addingDRM_XE_VMA_ATTR_PURGEABLE_STATEattribute type.This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
- WILLNEED: Buffer is needed and should not be purged. If the BO was previously purged, retained field returns 0 indicating backing store was lost (once purged, always purged semantics matching i915).
- DONTNEED: Buffer is not currently needed and may be purged by the kernel under memory pressure to free resources. Only applies to non-shared BOs.
The implementation includes a 'retained' output field (matching i915's drm_i915_gem_madvise.retained) that indicates whether the BO's backing store still exists (1) or has been purged (0).
v2: Add PURGED state for read-only status, change ioctl to
DRM_IOWR,
add retained field for i915 compatibilityCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Signed-off-by Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- include/uapi/drm/xe_drm.h | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)include/uapi/drm/xe_drm.h +0
-0include/uapi/drm/xe_drm.h extern "C" { · line 124
Quoted 5 lines
include/uapi/drm/xe_drm.h +1
-1@@ -121,7 +121,7 @@ extern "C" {#define DRM_IOCTL_XE_EXEC DRM_IOW(DRM_COMMAND_BASE + DRM_XE_EXEC, struct drm_xe_exec)#define DRM_IOCTL_XE_WAIT_USER_FENCE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_WAIT_USER_FENCE, struct drm_xe_wait_user_fence)#define DRM_IOCTL_XE_OBSERVATION DRM_IOW(DRM_COMMAND_BASE + DRM_XE_OBSERVATION, struct drm_xe_observation_param) -#define DRM_IOCTL_XE_MADVISE DRM_IOW(DRM_COMMAND_BASE + DRM_XE_MADVISE, struct drm_xe_madvise) +#define DRM_IOCTL_XE_MADVISE DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_MADVISE, struct drm_xe_madvise)I'm not sure if this is something we are allowed to change per Linux uAPI rules. I'd check with our maintainers (Thomas, Rodrigo) on this one.
Quoted 42 lines
include/uapi/drm/xe_drm.h +28
-0#define DRM_IOCTL_XE_VM_QUERY_MEM_RANGE_ATTRS DRM_IOWR(DRM_COMMAND_BASE + DRM_XE_VM_QUERY_MEM_RANGE_ATTRS, struct drm_xe_vm_query_mem_range_attr)/** @@ -2051,6 +2051,7 @@ struct drm_xe_madvise {#define DRM_XE_MEM_RANGE_ATTR_PREFERRED_LOC 0#define DRM_XE_MEM_RANGE_ATTR_ATOMIC 1#define DRM_XE_MEM_RANGE_ATTR_PAT 2 +#define DRM_XE_VMA_ATTR_PURGEABLE_STATE 3/** @type: type of attribute */ __u32 type;@@ -2129,6 +2130,40 @@ struct drm_xe_madvise {/** @pat_index.reserved: Reserved */ __u64 reserved; } pat_index; + +/** +* @purge_state_val: Purgeable state configuration +* +* Used when @type == DRM_XE_VMA_ATTR_PURGEABLE_STATE. +* +* Configures the purgeable state of buffer objects in the specified +* virtual address range. This allows applications to hint to the kernel +* about bo's usage patterns for better memory management. +* +* Supported values for @purge_state_val.val: +* - DRM_XE_VMA_PURGEABLE_STATE_WILLNEED (0): Marks BO as needed. +* If BO was purged, returns retained=0 (backing store lost). +* +* - DRM_XE_VMA_PURGEABLE_STATE_DONTNEED (1): Hints that BO is not +* currently needed. Kernel may purge it under memory pressure. +* Only applies to non-shared BOs. Returns retained=1 if not purged. +*/ +struct { +#define DRM_XE_VMA_PURGEABLE_STATE_WILLNEED 0 +#define DRM_XE_VMA_PURGEABLE_STATE_DONTNEED 1 +/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ +__u32 val; +/** +* @purge_state_val.retained: Whether the backing store still exists. +* +* Output field indicating if the BO's backing store is retained.
Quoted 8 lines
include/uapi/drm/xe_drm.h struct drm_xe_madvise { · line 2163
Quoted 4 lines
include/uapi/drm/xe_drm.h +4
-0+* Set to 1 if backing store exists, 0 if it has been purged. +* Similar to i915's drm_i915_gem_madvise.retained field. +*/ +__u32 retained;If we can't change the IOCTL to
DRM_IOWR, then we could hack around this restriction with making 'retained' a userptr which the madvise IOCTL explictly copies back into rather than relying DRM IOCTL core to implement thecopy_to_user.
Noted. I will do the changes as per your suggestion.
~Arvind
Quoted 12 lines
Matt
Quoted 9 lines
include/uapi/drm/xe_drm.h +3
-0+/** @purge_state_val.reserved: Reserved */ +__u64 reserved; + } purge_state_val; };/** @reserved: Reserved */ Signature · 1 line
2.43.0
RFC v2 2/9 drm/xe/bo: Add purgeable bo state tracking and field madv to xe_bo
In reply to Arvind YadavAdd infrastructure for tracking purgeable state of buffer objects. This includes:
Introduce enum xe_madv_purgeable_state with three states:
- XE_MADV_PURGEABLE_WILLNEED (0): BO is needed and should not be purged. This is the default state for all BOs.
- XE_MADV_PURGEABLE_DONTNEED (1): BO is not currently needed and can be purged by the kernel under memory pressure to reclaim resources. Only non-shared BOs can be marked as DONTNEED.
- XE_MADV_PURGEABLE_PURGED (2): BO has been purged by the kernel. Accessing a purged BO results in error. Follows i915 semantics where once purged, the BO remains permanently invalid ("once purged, always purged").
Add atomic_t madv field to struct xe_bo for state tracking
of purgeable state across concurrent access paths
v2: Add xe_bo_is_purged() helper, improve state documentation
--- drivers/gpu/drm/xe/xe_bo.h | 27 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo_types.h | 3 +++ 2 files changed, 30 insertions(+)
drivers/gpu/drm/xe/xe_bo.h +27 -0
@@ -85,6 +85,28 @@#define XE_PCI_BARRIER_MMAP_OFFSET (0x50 << XE_PTE_SHIFT) +/** +* enum xe_madv_purgeable_state - Buffer object purgeable state enumeration +* +* This enum defines the possible purgeable states for a buffer object, +* allowing userspace to provide memory usage hints to the kernel for +* better memory management under pressure. +* +* @XE_MADV_PURGEABLE_WILLNEED: The buffer object is needed and should not be purged. +* This is the default state. +* @XE_MADV_PURGEABLE_DONTNEED: The buffer object is not currently needed and can be +* purged by the kernel under memory pressure. +* @XE_MADV_PURGEABLE_PURGED: The buffer object has been purged by the kernel. +* +* Accessing a purged buffer will result in an error. Per i915 semantics, +* once purged, a BO remains permanently invalid and must be destroyed and recreated. +*/ +enum xe_madv_purgeable_state { +XE_MADV_PURGEABLE_WILLNEED , +XE_MADV_PURGEABLE_DONTNEED , +XE_MADV_PURGEABLE_PURGED , +}; +struct sg_table ;struct xe_bo *xe_bo_alloc (void );
@@ -213,6 +235,11 @@ static inline bool xe_bo_is_protected(const struct xe_bo *bo)return bo->pxp_key_instance; } +static inline bool xe_bo_is_purged (struct xe_bo *bo) +{ +return atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_PURGED ; +} +static inline void xe_bo_unpin_map_no_vm (struct xe_bo *bo) {if (likely (bo)) {
drivers/gpu/drm/xe/xe_bo_types.h +3 -0
@@ -108,6 +108,9 @@ struct xe_bo {* from default */ u64 min_align; + +/** @madv_purgeable: user space advise on BO purgeability */ + atomic_t madv_purgeable; };#endif
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 2/9] drm/xe/bo: Add purgeable bo state tracking and field madv to xe_bo
Quoted 64 lines · Arvind Yadav
Add infrastructure for tracking purgeable state of buffer objects. This includes:
Introduce enum
xe_madv_purgeable_statewith three states:
-XE_MADV_PURGEABLE_WILLNEED(0): BO is needed and should not be purged. This is the default state for all BOs.-
XE_MADV_PURGEABLE_DONTNEED(1): BO is not currently needed and can be purged by the kernel under memory pressure to reclaim resources. Only non-shared BOs can be marked as DONTNEED.-
XE_MADV_PURGEABLE_PURGED(2): BO has been purged by the kernel. Accessing a purged BO results in error. Follows i915 semantics where once purged, the BO remains permanently invalid ("once purged, always purged").Add
atomic_tmadv field to structxe_bofor state tracking
of purgeable state across concurrent access pathsv2: Add
xe_bo_is_purged()helper, improve state documentationCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.h | 27 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo_types.h | 3 +++ 2 files changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +22
-0@@ -85,6 +85,28 @@#define XE_PCI_BARRIER_MMAP_OFFSET (0x50 << XE_PTE_SHIFT) +/** +* enum xe_madv_purgeable_state - Buffer object purgeable state enumeration +* +* This enum defines the possible purgeable states for a buffer object, +* allowing userspace to provide memory usage hints to the kernel for +* better memory management under pressure. +* +* @XE_MADV_PURGEABLE_WILLNEED: The buffer object is needed and should not be purged. +* This is the default state. +* @XE_MADV_PURGEABLE_DONTNEED: The buffer object is not currently needed and can be +* purged by the kernel under memory pressure. +* @XE_MADV_PURGEABLE_PURGED: The buffer object has been purged by the kernel. +* +* Accessing a purged buffer will result in an error. Per i915 semantics, +* once purged, a BO remains permanently invalid and must be destroyed and recreated. +*/ +enum xe_madv_purgeable_state { +XE_MADV_PURGEABLE_WILLNEED , +XE_MADV_PURGEABLE_DONTNEED , +XE_MADV_PURGEABLE_PURGED , +}; +struct sg_table ;struct xe_bo *xe_bo_alloc (void );
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_protected(const struct xe_bo *bo) · line 237
Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.h +0
-0@@ -213,6 +235,11 @@ static inline bool xe_bo_is_protected(const struct xe_bo *bo)return bo->pxp_key_instance; }
Kernel doc.
Matt
Quoted 25 lines
drivers/gpu/drm/xe/xe_bo.h +5
-0+static inline bool xe_bo_is_purged (struct xe_bo *bo) +{ +return atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_PURGED ; +} +static inline void xe_bo_unpin_map_no_vm (struct xe_bo *bo) {if (likely (bo)) {drivers/gpu/drm/xe/xe_bo_types.h +3
-0@@ -108,6 +108,9 @@ struct xe_bo {* from default */ u64 min_align; + +/** @madv_purgeable: user space advise on BO purgeability */ + atomic_t madv_purgeable; };#endif Signature · 1 line
2.43.0
Re: [RFC v2 2/9] drm/xe/bo: Add purgeable bo state tracking and field madv to xe_bo
Quoted 63 lines · Matthew Brost
Quoted 62 lines · Arvind Yadav
Add infrastructure for tracking purgeable state of buffer objects. This includes:
Introduce enum
xe_madv_purgeable_statewith three states:
-XE_MADV_PURGEABLE_WILLNEED(0): BO is needed and should not be purged. This is the default state for all BOs.-
XE_MADV_PURGEABLE_DONTNEED(1): BO is not currently needed and can be purged by the kernel under memory pressure to reclaim resources. Only non-shared BOs can be marked as DONTNEED.-
XE_MADV_PURGEABLE_PURGED(2): BO has been purged by the kernel. Accessing a purged BO results in error. Follows i915 semantics where once purged, the BO remains permanently invalid ("once purged, always purged").Add
atomic_tmadv field to structxe_bofor state tracking
of purgeable state across concurrent access pathsv2: Add
xe_bo_is_purged()helper, improve state documentationCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.h | 27 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo_types.h | 3 +++ 2 files changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +22
-0@@ -85,6 +85,28 @@#define XE_PCI_BARRIER_MMAP_OFFSET (0x50 << XE_PTE_SHIFT) +/** +* enum xe_madv_purgeable_state - Buffer object purgeable state enumeration +* +* This enum defines the possible purgeable states for a buffer object, +* allowing userspace to provide memory usage hints to the kernel for +* better memory management under pressure. +* +* @XE_MADV_PURGEABLE_WILLNEED: The buffer object is needed and should not be purged. +* This is the default state. +* @XE_MADV_PURGEABLE_DONTNEED: The buffer object is not currently needed and can be +* purged by the kernel under memory pressure. +* @XE_MADV_PURGEABLE_PURGED: The buffer object has been purged by the kernel. +* +* Accessing a purged buffer will result in an error. Per i915 semantics, +* once purged, a BO remains permanently invalid and must be destroyed and recreated. +*/ +enum xe_madv_purgeable_state { +XE_MADV_PURGEABLE_WILLNEED , +XE_MADV_PURGEABLE_DONTNEED , +XE_MADV_PURGEABLE_PURGED , +}; +struct sg_table ;
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_protected(const struct xe_bo *bo) · line 237
Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.h +0
-0struct xe_bo *xe_bo_alloc (void );@@ -213,6 +235,11 @@ static inline bool xe_bo_is_protected(const struct xe_bo *bo)return bo->pxp_key_instance; }Kernel doc.
Noted.
~Arvind
Quoted 28 lines
Matt
Quoted 25 lines
drivers/gpu/drm/xe/xe_bo.h +5
-0+static inline bool xe_bo_is_purged (struct xe_bo *bo) +{ +return atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_PURGED ; +} +static inline void xe_bo_unpin_map_no_vm (struct xe_bo *bo) {if (likely (bo)) {drivers/gpu/drm/xe/xe_bo_types.h +3
-0@@ -108,6 +108,9 @@ struct xe_bo {* from default */ u64 min_align; + +/** @madv_purgeable: user space advise on BO purgeability */ + atomic_t madv_purgeable; };#endif Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 2/9] drm/xe/bo: Add purgeable bo state tracking and field madv to xe_bo
In reply to Arvind YadavQuoted 64 lines · Arvind Yadav
Add infrastructure for tracking purgeable state of buffer objects. This includes:
Introduce enum
xe_madv_purgeable_statewith three states:
-XE_MADV_PURGEABLE_WILLNEED(0): BO is needed and should not be purged. This is the default state for all BOs.-
XE_MADV_PURGEABLE_DONTNEED(1): BO is not currently needed and can be purged by the kernel under memory pressure to reclaim resources. Only non-shared BOs can be marked as DONTNEED.-
XE_MADV_PURGEABLE_PURGED(2): BO has been purged by the kernel. Accessing a purged BO results in error. Follows i915 semantics where once purged, the BO remains permanently invalid ("once purged, always purged").Add
atomic_tmadv field to structxe_bofor state tracking
of purgeable state across concurrent access pathsv2: Add
xe_bo_is_purged()helper, improve state documentationCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.h | 27 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo_types.h | 3 +++ 2 files changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +22
-0@@ -85,6 +85,28 @@#define XE_PCI_BARRIER_MMAP_OFFSET (0x50 << XE_PTE_SHIFT) +/** +* enum xe_madv_purgeable_state - Buffer object purgeable state enumeration +* +* This enum defines the possible purgeable states for a buffer object, +* allowing userspace to provide memory usage hints to the kernel for +* better memory management under pressure. +* +* @XE_MADV_PURGEABLE_WILLNEED: The buffer object is needed and should not be purged. +* This is the default state. +* @XE_MADV_PURGEABLE_DONTNEED: The buffer object is not currently needed and can be +* purged by the kernel under memory pressure. +* @XE_MADV_PURGEABLE_PURGED: The buffer object has been purged by the kernel. +* +* Accessing a purged buffer will result in an error. Per i915 semantics, +* once purged, a BO remains permanently invalid and must be destroyed and recreated. +*/ +enum xe_madv_purgeable_state { +XE_MADV_PURGEABLE_WILLNEED , +XE_MADV_PURGEABLE_DONTNEED , +XE_MADV_PURGEABLE_PURGED , +}; +struct sg_table ;struct xe_bo *xe_bo_alloc (void );
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_protected(const struct xe_bo *bo) · line 239
Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.h +2
-0@@ -213,6 +235,11 @@ static inline bool xe_bo_is_protected(const struct xe_bo *bo)return bo->pxp_key_instance; } +static inline bool xe_bo_is_purged (struct xe_bo *bo) +{
I think you want to assert the BO's dma-resv lock is held here. I suggest this in patch #5.
Quoted 11 lines
drivers/gpu/drm/xe/xe_bo.h +3
-0+return atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_PURGED ; +} +static inline void xe_bo_unpin_map_no_vm (struct xe_bo *bo) {if (likely (bo)) {drivers/gpu/drm/xe/xe_bo_types.h +0
-0
drivers/gpu/drm/xe/xe_bo_types.h struct xe_bo { · line 113
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo_types.h +3
-0@@ -108,6 +108,9 @@ struct xe_bo {* from default */ u64 min_align; + +/** @madv_purgeable: user space advise on BO purgeability */ +atomic_t madv_purgeable;
Does this really need to be atomic? From a grep on the final result, all critical accesses (ones where functionality would break) are done under (or should be done under) the BO's dma-resv lock, right? The madvise reading the attribute is likely the expection where it is ok to just read this value without the dma-resv lock.
So with that, I'd suggest drop the atomic and also add a helper to set this value that also asserts the BO's dma-resv lock is held. Use a WRITE_ONCE which would pair with a READ_ONCE in the madvise attribute query.
Matt
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo_types.h +0
-0};#endif Signature · 1 line
2.43.0
RFC v2 3/9 drm/xe/bo: Prevent purging of shared buffer objects
In reply to Arvind YadavIntroduce the xe_bo_is_shared_locked() inline helper to determine if a buffer object is shared across multiple clients or drivers. A buffer is considered shared if it is exported via dma-buf, imported, or has a handle count greater than one.
This check is critical for safely implementing purgeable memory. Purging a buffer that is shared would lead to data corruption for other clients that still hold a reference to it.
The kernel cannot safely determine when all clients are done with a shared buffer, so shared BOs must never be marked DONTNEED or purged.
The new helper is used in two key locations:
1. In xe_vm_madvise_purgeable_bo(), to prevent userspace from successfully marking a shared buffer as DONTNEED. This is the primary safeguard against incorrect usage.
2. In xe_bo_move(), as a final safety check before the kernel initiates a purge during eviction. This ensures that even if a shared buffer were somehow marked DONTNEED, it would not be purged.
--- drivers/gpu/drm/xe/xe_bo.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
drivers/gpu/drm/xe/xe_bo.h +30 -0
@@ -478,4 +478,34 @@ static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type)xe_bo_assert_held (bo); return bo->ttm.resource->mem_type == mem_type; } + +/** +* xe_bo_is_shared_locked - Check if a buffer object is shared +* @bo: The buffer object to check +* +* Determines if a buffer object is considered shared, which includes: +* - Exported via dma-buf (obj->dma_buf is set) +* - Imported from another driver (obj->import_attach is set) +* - Referenced by multiple clients (handle_count > 1) +* +* This check is used to prevent data loss on shared content by avoiding +* certain operations like purging on buffers that other processes or +* drivers might still be using. +* +* Return: true if the buffer object is shared, false otherwise. +*/ +static inline bool xe_bo_is_shared_locked (const struct xe_bo *bo) +{ +const struct drm_gem_object *obj = &bo->ttm.base; + +dma_resv_assert_held (obj->resv); + +if (obj->dma_buf || obj->import_attach) +return true; + +if (obj->handle_count >1 ) +return true; + +return false; +}#endif
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 3/9] drm/xe/bo: Prevent purging of shared buffer objects
Quoted 48 lines · Arvind Yadav
Introduce the
xe_bo_is_shared_locked()inline helper to determine if a buffer object is shared across multiple clients or drivers. A buffer is considered shared if it is exported via dma-buf, imported, or has a handle count greater than one.This check is critical for safely implementing purgeable memory. Purging a buffer that is shared would lead to data corruption for other clients that still hold a reference to it.
The kernel cannot safely determine when all clients are done with a shared buffer, so shared BOs must never be marked DONTNEED or purged.
The new helper is used in two key locations:
1. Inxe_vm_madvise_purgeable_bo(), to prevent userspace from successfully marking a shared buffer asDONTNEED. This is the primary safeguard against incorrect usage.2. In
xe_bo_move(), as a final safety check before the kernel initiates a purge during eviction. This ensures that even if a shared buffer were somehow markedDONTNEED, it would not be purged.Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +10
-0@@ -478,4 +478,34 @@ static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type)xe_bo_assert_held (bo);return bo->ttm.resource->mem_type == mem_type; } + +/** +* xe_bo_is_shared_locked - Check if a buffer object is shared +* @bo: The buffer object to check +* +* Determines if a buffer object is considered shared, which includes: +* - Exported via dma-buf (obj->dma_buf is set) +* - Imported from another driver (obj->import_attach is set) +* - Referenced by multiple clients (handle_count > 1) +*
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 500
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.h +10
-0+* This check is used to prevent data loss on shared content by avoiding +* certain operations like purging on buffers that other processes or +* drivers might still be using. +* +* Return: true if the buffer object is shared, false otherwise. +*/ +static inline bool xe_bo_is_shared_locked (const struct xe_bo *bo) +{ +const struct drm_gem_object *obj = &bo->ttm.base; +
It seems like everything below here should be a new drm gem helper.
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 506
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.h +6
-0+dma_resv_assert_held (obj->resv); + +if (obj->dma_buf || obj->import_attach) +return true; + +if (obj->handle_count >1 )
So this covers the case when we prime fd to handle but we resolve to a BO (i.e., we don't do a dma-buf attach, rather just take reference on BO as the BO is from the same device)? I just want to make sure I'm understanding this part correctly. If so, maybe throw a comment in here or update the functions kernel doc a bit with a better explaination.
Matt
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h +4
-0+return true; + + return false; +}#endif Signature · 1 line
2.43.0
Re: [RFC v2 3/9] drm/xe/bo: Prevent purging of shared buffer objects
Quoted 52 lines · Matthew Brost
Quoted 51 lines · Arvind Yadav
Introduce the
xe_bo_is_shared_locked()inline helper to determine if a buffer object is shared across multiple clients or drivers. A buffer is considered shared if it is exported via dma-buf, imported, or has a handle count greater than one.This check is critical for safely implementing purgeable memory. Purging a buffer that is shared would lead to data corruption for other clients that still hold a reference to it.
The kernel cannot safely determine when all clients are done with a shared buffer, so shared BOs must never be marked DONTNEED or purged.
The new helper is used in two key locations:
1. Inxe_vm_madvise_purgeable_bo(), to prevent userspace from successfully marking a shared buffer asDONTNEED. This is the primary safeguard against incorrect usage.2. In
xe_bo_move(), as a final safety check before the kernel initiates a purge during eviction. This ensures that even if a shared buffer were somehow markedDONTNEED, it would not be purged.Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +13
-0@@ -478,4 +478,34 @@ static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type)xe_bo_assert_held (bo);return bo->ttm.resource->mem_type == mem_type; } + +/** +* xe_bo_is_shared_locked - Check if a buffer object is shared +* @bo: The buffer object to check +* +* Determines if a buffer object is considered shared, which includes: +* - Exported via dma-buf (obj->dma_buf is set) +* - Imported from another driver (obj->import_attach is set) +* - Referenced by multiple clients (handle_count > 1) +* +* This check is used to prevent data loss on shared content by avoiding +* certain operations like purging on buffers that other processes or +* drivers might still be using.
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 500
Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.h +7
-0+* +* Return: true if the buffer object is shared, false otherwise. +*/ +static inline bool xe_bo_is_shared_locked (const struct xe_bo *bo) +{ +const struct drm_gem_object *obj = &bo->ttm.base; +It seems like everything below here should be a new drm gem helper.
There is a DRM helper 'drm_gem_object_is_shared_for_memory_stats()', but it's specifically scoped for fdinfo memory accounting and doesn't check import_attach.
Quoted 11 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 506
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.h +6
-0+dma_resv_assert_held (obj->resv); + +if (obj->dma_buf || obj->import_attach) +return true; + +if (obj->handle_count >1 )So this covers the case when we prime fd to handle but we resolve to a BO (i.e., we don't do a dma-buf attach, rather just take reference on BO as the BO is from the same device)? I just want to make sure I'm understanding this part correctly. If so, maybe throw a comment in here or update the functions kernel doc a bit with a better explaination.
Yes, that's correct! The handle_count > 1 check covers exactly that scenario:
When we do prime fd-to-handle but both processes are using the same xe device, we don't do a dma-buf attach. Instead, we just increment the reference count and handle_count on the same xe_bo.
I will add and update the function with comments.
~Arvind
Quoted 10 lines
Matt
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h +4
-0+return true; + + return false; +}#endif Signature · 1 line
2.43.0
Thomas Hellström
maintainerRe: [RFC v2 3/9] drm/xe/bo: Prevent purging of shared buffer objects
Quoted 98 lines · Arvind
Quoted 68 lines · Matthew Brost
Quoted 67 lines · Arvind Yadav
Introduce the
xe_bo_is_shared_locked()inline helper to determine if a
buffer object is shared across multiple clients or drivers. A buffer is
considered shared if it is exported via dma-buf, imported, or has a
handle count greater than one.This check is critical for safely implementing purgeable memory. Purging
a buffer that is shared would lead to data corruption for other clients
that still hold a reference to it.The kernel cannot safely determine when all clients are done with a
shared buffer, so shared BOs must never be marked DONTNEED or purged.The new helper is used in two key locations:
1. Inxe_vm_madvise_purgeable_bo(), to prevent userspace from successfully marking a shared buffer asDONTNEED. This is the
primary safeguard against incorrect usage.2. In
xe_bo_move(), as a final safety check before the kernel initiates a purge during eviction. This ensures that even if a
shared buffer were somehow markedDONTNEED, it would not be
purged.Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>---
drivers/gpu/drm/xe/xe_bo.h| 30 ++++++++++++++++++++++++++++++1 file changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +6
-0@@ -478,4 +478,34 @@ static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) xe_bo_assert_held(bo); return bo->ttm.resource->mem_type == mem_type; } + +/** +* xe_bo_is_shared_locked - Check if a buffer object is shared +* @bo: The buffer object to check +* +* Determines if a buffer object is considered shared, which includes:
+ * - Exported via dma-buf (obj->dma_bufis set)
+ * - Imported from another driver (obj->import_attachis set)
+ * - Referenced by multiple clients (handle_count> 1)
+ *
+ * This check is used to prevent data loss on shared content by avoiding
+ * certain operations like purging on buffers that other processes or
+ * drivers might still be using.
+ *Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 500
Quoted 7 lines
+ * Return: true if the buffer object is shared, false otherwise.
+ */
+static inline boolxe_bo_is_shared_locked(const struct xe_bo *bo)drivers/gpu/drm/xe/xe_bo.h +3
-0+{ +const struct drm_gem_object *obj = &bo->ttm.base; +It seems like everything below here should be a new drm gem helper.
There is a DRM helper '
drm_gem_object_is_shared_for_memory_stats()', but
it's
specifically scoped for fdinfo memory accounting and doesn't checkimport_attach.Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 506
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.h +6
-0+dma_resv_assert_held (obj->resv); + +if (obj->dma_buf || obj->import_attach) +return true; + +if (obj->handle_count >1 )So this covers the case when we prime fd to handle but we resolve
Quoted 8 lines
to a
BO (i.e., we don't do a dma-buf attach, rather just take reference on BO
as the BO is from the same device)? I just want to make sure I'm understanding this part correctly. If so, maybe throw a comment in here
or update the functions kernel doc a bit with a better explaination.
Quoted 10 lines
Yes, that's correct! The
handle_count> 1 check covers exactly that scenario:
When we do prime fd-to-handle but both processes are using the same xe
device,
we don't do a dma-buf attach. Instead, we just increment the reference
count
andhandle_counton the samexe_bo.
I will add and update the function with comments.
Arvind, Matt
I think the correct way to check purgability support for shared buffers is to loop over all vmas attached to the bo and check that they all say WONTNEED? If they don't, the bo is not purgeable. This will also need a check at VMA unbinding.
/Thomas
Quoted 12 lines
~Arvind
Quoted 10 lines
Matt
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h +4
-0+return true; + +return false; +}#endif
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 3/9] drm/xe/bo: Prevent purging of shared buffer objects
Quoted 110 lines · Thomas Hellström
Quoted 98 lines · Arvind
Quoted 68 lines · Matthew Brost
Quoted 67 lines · Arvind Yadav
Introduce the
xe_bo_is_shared_locked()inline helper to determine if a
buffer object is shared across multiple clients or drivers. A buffer is
considered shared if it is exported via dma-buf, imported, or has a
handle count greater than one.This check is critical for safely implementing purgeable memory. Purging
a buffer that is shared would lead to data corruption for other clients
that still hold a reference to it.The kernel cannot safely determine when all clients are done with a
shared buffer, so shared BOs must never be marked DONTNEED or purged.The new helper is used in two key locations:
1. Inxe_vm_madvise_purgeable_bo(), to prevent userspace from successfully marking a shared buffer asDONTNEED. This is the
primary safeguard against incorrect usage.2. In
xe_bo_move(), as a final safety check before the kernel initiates a purge during eviction. This ensures that even if a
shared buffer were somehow markedDONTNEED, it would not be
purged.Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>---
drivers/gpu/drm/xe/xe_bo.h| 30 ++++++++++++++++++++++++++++++1 file changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +6
-0@@ -478,4 +478,34 @@ static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) xe_bo_assert_held(bo); return bo->ttm.resource->mem_type == mem_type; } + +/** +* xe_bo_is_shared_locked - Check if a buffer object is shared +* @bo: The buffer object to check +* +* Determines if a buffer object is considered shared, which includes:
+ * - Exported via dma-buf (obj->dma_bufis set)
+ * - Imported from another driver (obj->import_attachis set)
+ * - Referenced by multiple clients (handle_count> 1)
+ *
+ * This check is used to prevent data loss on shared content by avoiding
+ * certain operations like purging on buffers that other processes or
+ * drivers might still be using.
+ *Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 500
Quoted 7 lines
+ * Return: true if the buffer object is shared, false otherwise.
+ */
+static inline boolxe_bo_is_shared_locked(const struct xe_bo *bo)drivers/gpu/drm/xe/xe_bo.h +3
-0+{ +const struct drm_gem_object *obj = &bo->ttm.base; +It seems like everything below here should be a new drm gem helper.
There is a DRM helper '
drm_gem_object_is_shared_for_memory_stats()', but
it's
specifically scoped for fdinfo memory accounting and doesn't checkimport_attach.Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 506
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.h +6
-0+dma_resv_assert_held (obj->resv); + +if (obj->dma_buf || obj->import_attach) +return true; + +if (obj->handle_count >1 )So this covers the case when we prime fd to handle but we resolve
Quoted 8 lines
to a
BO (i.e., we don't do a dma-buf attach, rather just take reference on BO
as the BO is from the same device)? I just want to make sure I'm understanding this part correctly. If so, maybe throw a comment in here
or update the functions kernel doc a bit with a better explaination.Quoted 10 lines
Yes, that's correct! The
handle_count> 1 check covers exactly that scenario:
When we do prime fd-to-handle but both processes are using the same xe
device,
we don't do a dma-buf attach. Instead, we just increment the reference
count
andhandle_counton the samexe_bo.
I will add and update the function with comments.
Quoted 7 lines
Arvind, Matt
I think the correct way to check purgability support for shared buffers is to loop over all vmas attached to the bo and check that they all say WONTNEED? If they don't, the bo is not purgeable. This will also need a check at VMA unbinding.
I think this makes sense. I haven't fully gotten through this series yet, but will consider this during the code reviews. I probably should apply this series in full before providing feedback.
Matt
Quoted 20 lines
/Thomas
Quoted 12 lines
~Arvind
Quoted 10 lines
Matt
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h +4
-0+return true; + +return false; +}#endif
Signature · 1 line
2.43.0
Re: [RFC v2 3/9] drm/xe/bo: Prevent purging of shared buffer objects
Quoted 111 lines · Matthew Brost
Quoted 110 lines · Thomas Hellström
Quoted 97 lines · Arvind
Quoted 68 lines · Matthew Brost
Quoted 67 lines · Arvind Yadav
Introduce the
xe_bo_is_shared_locked()inline helper to determine if a
buffer object is shared across multiple clients or drivers. A buffer is
considered shared if it is exported via dma-buf, imported, or has a
handle count greater than one.This check is critical for safely implementing purgeable memory. Purging
a buffer that is shared would lead to data corruption for other clients
that still hold a reference to it.The kernel cannot safely determine when all clients are done with a
shared buffer, so shared BOs must never be marked DONTNEED or purged.The new helper is used in two key locations:
1. Inxe_vm_madvise_purgeable_bo(), to prevent userspace from
successfully marking a shared buffer asDONTNEED. This is
the
primary safeguard against incorrect usage.2. In
xe_bo_move(), as a final safety check before the kernel
initiates a purge during eviction. This ensures that even if
a
shared buffer were somehow markedDONTNEED, it would not
be
purged.Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +6
-0@@ -478,4 +478,34 @@ static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) xe_bo_assert_held(bo); return bo->ttm.resource->mem_type == mem_type; } + +/** +* xe_bo_is_shared_locked - Check if a buffer object is shared +* @bo: The buffer object to check +* +* Determines if a buffer object is considered shared, which includes:
+ * - Exported via dma-buf (obj->dma_bufis set)
+ * - Imported from another driver (obj->import_attachis set)
+ * - Referenced by multiple clients (handle_count> 1)
+ *
+ * This check is used to prevent data loss on shared content by avoiding
+ * certain operations like purging on buffers that other processes or
+ * drivers might still be using.
+ *Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 500
Quoted 7 lines
+ * Return: true if the buffer object is shared, false otherwise.
+ */
+static inline boolxe_bo_is_shared_locked(const struct xe_bo *bo)drivers/gpu/drm/xe/xe_bo.h +3
-0+{ +const struct drm_gem_object *obj = &bo->ttm.base; +It seems like everything below here should be a new drm gem helper.
There is a DRM helper '
drm_gem_object_is_shared_for_memory_stats()', but
it's
specifically scoped for fdinfo memory accounting and doesn't checkimport_attach.Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 506
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.h +6
-0+dma_resv_assert_held (obj->resv); + +if (obj->dma_buf || obj->import_attach) +return true; + +if (obj->handle_count >1 )So this covers the case when we prime fd to handle but we resolve
Quoted 8 lines
to a
BO (i.e., we don't do a dma-buf attach, rather just take reference on BO
as the BO is from the same device)? I just want to make sure I'm understanding this part correctly. If so, maybe throw a comment in here
or update the functions kernel doc a bit with a better explaination.Quoted 10 lines
Yes, that's correct! The
handle_count> 1 check covers exactly that scenario:
When we do prime fd-to-handle but both processes are using the same xe
device,
we don't do a dma-buf attach. Instead, we just increment the reference
count
andhandle_counton the samexe_bo.
I will add and update the function with comments.Arvind, Matt
Quoted 8 lines
Quoted 5 lines
I think the correct way to check purgability support for shared buffers is to loop over all vmas attached to the bo and check that they all say WONTNEED? If they don't, the bo is not purgeable. This will also need a check at VMA unbinding.
I think this makes sense. I haven't fully gotten through this series yet, but will consider this during the code reviews. I probably should apply this series in full before providing feedback.
Thomas, Matt
Agreed. I will reworked the purgeability logic exactly as you suggested. Instead of relying on a “shared BO” helper, the driver now keeps a per-VMA purgeable_state, and a BO becomes DONTNEED only when all VMAs attached to it report DONTNEED.
~Arvind
Quoted 20 lines
Matt
Quoted 18 lines
/Thomas
Quoted 11 lines
~Arvind
Quoted 10 lines
Matt
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h +4
-0+return true; + +return false; +} #endifSignature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 3/9] drm/xe/bo: Prevent purging of shared buffer objects
Quoted 122 lines · Arvind
Quoted 111 lines · Matthew Brost
Quoted 110 lines · Thomas Hellström
Quoted 97 lines · Arvind
Quoted 68 lines · Matthew Brost
Quoted 67 lines · Arvind Yadav
Introduce the
xe_bo_is_shared_locked()inline helper to determine if a
buffer object is shared across multiple clients or drivers. A buffer is
considered shared if it is exported via dma-buf, imported, or has a
handle count greater than one.This check is critical for safely implementing purgeable memory. Purging
a buffer that is shared would lead to data corruption for other clients
that still hold a reference to it.The kernel cannot safely determine when all clients are done with a
shared buffer, so shared BOs must never be marked DONTNEED or purged.The new helper is used in two key locations:
1. Inxe_vm_madvise_purgeable_bo(), to prevent userspace from
successfully marking a shared buffer asDONTNEED. This is
the
primary safeguard against incorrect usage.2. In
xe_bo_move(), as a final safety check before the kernel
initiates a purge during eviction. This ensures that even if
a
shared buffer were somehow markedDONTNEED, it would not
be
purged.Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)drivers/gpu/drm/xe/xe_bo.h +6
-0@@ -478,4 +478,34 @@ static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) xe_bo_assert_held(bo); return bo->ttm.resource->mem_type == mem_type; } + +/** +* xe_bo_is_shared_locked - Check if a buffer object is shared +* @bo: The buffer object to check +* +* Determines if a buffer object is considered shared, which includes:
+ * - Exported via dma-buf (obj->dma_bufis set)
+ * - Imported from another driver (obj->import_attachis set)
+ * - Referenced by multiple clients (handle_count> 1)
+ *
+ * This check is used to prevent data loss on shared content by avoiding
+ * certain operations like purging on buffers that other processes or
+ * drivers might still be using.
+ *Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 500
Quoted 7 lines
+ * Return: true if the buffer object is shared, false otherwise.
+ */
+static inline boolxe_bo_is_shared_locked(const struct xe_bo *bo)drivers/gpu/drm/xe/xe_bo.h +3
-0+{ +const struct drm_gem_object *obj = &bo->ttm.base; +It seems like everything below here should be a new drm gem helper.
There is a DRM helper '
drm_gem_object_is_shared_for_memory_stats()', but
it's
specifically scoped for fdinfo memory accounting and doesn't checkimport_attach.Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.h static inline bool xe_bo_is_mem_type(struct xe_bo *bo, u32 mem_type) · line 506
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.h +6
-0+dma_resv_assert_held (obj->resv); + +if (obj->dma_buf || obj->import_attach) +return true; + +if (obj->handle_count >1 )So this covers the case when we prime fd to handle but we resolve
Quoted 8 lines
to a
BO (i.e., we don't do a dma-buf attach, rather just take reference on BO
as the BO is from the same device)? I just want to make sure I'm understanding this part correctly. If so, maybe throw a comment in here
or update the functions kernel doc a bit with a better explaination.Quoted 10 lines
Yes, that's correct! The
handle_count> 1 check covers exactly that scenario:
When we do prime fd-to-handle but both processes are using the same xe
device,
we don't do a dma-buf attach. Instead, we just increment the reference
count
andhandle_counton the samexe_bo.
I will add and update the function with comments.Arvind, Matt
Quoted 8 lines
Quoted 5 lines
I think the correct way to check purgability support for shared buffers is to loop over all vmas attached to the bo and check that they all say WONTNEED? If they don't, the bo is not purgeable. This will also need a check at VMA unbinding.
I think this makes sense. I haven't fully gotten through this series yet, but will consider this during the code reviews. I probably should apply this series in full before providing feedback.
Quoted 7 lines
Thomas, Matt
Agreed. I will reworked the purgeability logic exactly as you suggested. Instead of relying on a “shared BO” helper, the driver now keeps a per-VMA
purgeable_state,
and a BO becomes DONTNEED only when all VMAs attached to it report DONTNEED.
+1, let me get through the rest of the series today and see if I spot anything low hanging that should be fixed in the next rev.
Matt
Quoted 22 lines
~Arvind
Quoted 20 lines
Matt
Quoted 18 lines
/Thomas
Quoted 11 lines
~Arvind
Quoted 10 lines
Matt
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.h +4
-0+return true; + +return false; +} #endifSignature · 1 line
2.43.0
RFC v2 4/9 drm/xe/madvise: Implement purgeable buffer object support
In reply to Arvind YadavThis allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
Add the core implementation for purgeable buffer objects, enabling memory reclamation of user-designated DONTNEED buffers during eviction.
This patch implements the purge operation and state machine transitions:
Purgeable States (from xe_madv_purgeable_state):
- WILLNEED (0): BO should be retained, actively used
- DONTNEED (1): BO eligible for purging, not currently needed
- PURGED (2): BO backing store reclaimed, permanently invalid
Design Rationale:
- Async TLB invalidation via trigger_rebind (no blocking xe_vm_invalidate_vma)
- i915 compatibility: retained field, "once purged always purged" semantics
- Shared BO protection prevents multi-process memory corruption
- Scratch PTE reuse avoids new infrastructure, safe for fault mode
v2:
- Use xe_bo_trigger_rebind() for async TLB invalidation (Thomas Hellström)
- Add NULL rebind with scratch PTEs for fault mode (Thomas Hellström)
- Implement i915-compatible retained field logic (Thomas Hellström)
- Skip BO validation for purged BOs in page fault handler (crash fix)
- Add scratch VM check in page fault path (non-scratch VMs fail fault)
- Force clear_pt for non-scratch VMs to avoid phys addr 0 mapping (review fix)
- Add !is_purged check to resource cursor setup to prevent stale access
--- drivers/gpu/drm/xe/xe_bo.c | 72 ++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_gt_pagefault.c | 19 ++++++++ drivers/gpu/drm/xe/xe_pt.c | 36 ++++++++++++-- drivers/gpu/drm/xe/xe_vm.c | 11 ++++- drivers/gpu/drm/xe/xe_vm_madvise.c | 73 ++++++++++++++++++++++++++++ 5 files changed, 193 insertions(+), 18 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +60 -12
@@ -836,6 +836,53 @@ static int xe_bo_move_notify(struct xe_bo *bo,return 0 ; } +static void xe_bo_set_purged (struct xe_bo *bo) +{ +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv); + +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_PURGED ); +} + +/** +* xe_ttm_bo_purge() - Purge buffer object backing store +* @ttm_bo: The TTM buffer object to purge +* @ctx: TTM operation context +* +* This function purges the backing store of a BO marked as DONTNEED and +* triggers rebind to invalidate stale GPU mappings. For fault-mode VMs, +* this zaps the PTEs. The next GPU access will trigger a page fault and +* perform NULL rebind (scratch pages or clear PTEs based on VM config). +*/ +static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) +{ +struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); +struct xe_bo *bo =ttm_to_xe_bo (ttm_bo); + +if (ttm_bo->ttm) { +struct ttm_placement place = {}; +int ret =ttm_bo_validate (ttm_bo, &place, ctx); + +drm_WARN_ON (&xe->drm, ret); +if (!ret && bo) { +if (atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_DONTNEED ) { +xe_bo_set_purged (bo); + +/* +* Trigger rebind to invalidate stale GPU mappings. +* - Non-fault mode: Marks VMAs for rebind +* - Fault mode: Zaps PTEs (sets to 0), next access triggers fault +* and NULL rebind with scratch/clear PTEs per VM config +*/ + ret =xe_bo_trigger_rebind (xe, bo, ctx); +if (ret) +drm_warn (&xe->drm, +"Failed to invalidate purged BO: %d\n" , ret); + } + } + } +} + static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_resource *new_mem,
@@ -853,8 +900,18 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,bool needs_clear;bool handle_system_ccs = (!IS_DGFX (xe) &&xe_bo_needs_ccs_pages (bo) && ttm &&ttm_tt_is_populated (ttm)) ? true : false; +int state =atomic_read (&bo->madv_purgeable);int ret =0 ; +/* +* Purge only non-shared BOs explicitly marked DONTNEED by userspace. +* The move_notify callback will handle invalidation asynchronously. +*/ +if (evict && state ==XE_MADV_PURGEABLE_DONTNEED && !xe_bo_is_shared_locked (bo)) { +xe_ttm_bo_purge (ttm_bo, ctx); +return 0 ; + } +/* Bo creation path, moving to system or TT. */ if ((!old_mem && ttm) && !handle_system_ccs) {if (new_mem->mem_type ==XE_PL_TT )
@@ -1606,18 +1663,6 @@ static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo) } } -static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) -{ -struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); - -if (ttm_bo->ttm) { -struct ttm_placement place = {}; -int ret =ttm_bo_validate (ttm_bo, &place, ctx); - -drm_WARN_ON (&xe->drm, ret); - } -} -static void xe_ttm_bo_swap_notify (struct ttm_buffer_object *ttm_bo) {struct ttm_operation_ctx ctx = {
@@ -2202,6 +2247,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,#endif INIT_LIST_HEAD (&bo->vram_userfault_link); +/* Initialize purge advisory state */ +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +drm_gem_private_object_init (&xe->drm, &bo->ttm.base, size);if (resv) {
drivers/gpu/drm/xe/xe_gt_pagefault.c +19 -0
@@ -87,6 +87,13 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,if (!bo)return 0 ; +/* +* Skip validation/migration for purged BOs - they have no backing pages. +* Rebind will use scratch PTEs instead. +*/ +if (xe_bo_is_purged (bo)) +return 0 ; +return need_vram_move ?xe_bo_migrate (bo, vram->placement,NULL , exec) :xe_bo_validate (bo, vm, true, exec); }
@@ -100,9 +107,21 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,struct drm_exec exec;struct dma_fence *fence;int err, needs_vram; +struct xe_bo *bo;lockdep_assert_held_write (&vm->lock); +/* +* Check if BO is purged. For purged BOs: +* - Scratch VMs: Allow rebind with scratch PTEs (safe zero reads) +* - Non-scratch VMs: FAIL the page fault (no scratch page available) +*/ + bo =xe_vma_bo (vma); +if (bo &&xe_bo_is_purged (bo)) { +if (!xe_vm_has_scratch (vm)) +return -EACCES ; + } + needs_vram =xe_vma_need_vram_for_atomic (vm->xe, vma, atomic);if (needs_vram <0 || (needs_vram &&xe_vma_is_userptr (vma)))return needs_vram <0 ? needs_vram : -EACCES ;
drivers/gpu/drm/xe/xe_pt.c +31 -5
@@ -533,20 +533,26 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,/* Is this a leaf entry ?*/ if (level ==0 ||xe_pt_hugepte_possible (addr, next, level, xe_walk)) {struct xe_res_cursor *curs = xe_walk->curs; +struct xe_bo *bo =xe_vma_bo (xe_walk->vma);bool is_null =xe_vma_is_null (xe_walk->vma); -bool is_vram = is_null ? false :xe_res_is_vram (curs); +bool is_purged = bo &&xe_bo_is_purged (bo); +bool is_vram = (is_null || is_purged) ? false :xe_res_is_vram (curs);XE_WARN_ON (xe_walk->va_curs_start != addr);if (xe_walk->clear_pt) { pte =0 ; }else { - pte = vm->pt_ops->pte_encode_vma (is_null ?0 : +/* +* For purged BOs, treat like null VMAs - pass address 0. +* The pte_encode_vma will set XE_PTE_NULL flag for scratch mapping. +*/ + pte = vm->pt_ops->pte_encode_vma ((is_null || is_purged) ?0 :xe_res_dma (curs) + xe_walk->dma_offset, xe_walk->vma, pat_index, level); -if (!is_null) +if (!is_null && !is_purged) pte |= is_vram ? xe_walk->default_vram_pte : xe_walk->default_system_pte;
@@ -570,7 +576,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,if (unlikely (ret))return ret; -if (!is_null && !xe_walk->clear_pt) +if (!is_null && !is_purged && !xe_walk->clear_pt)xe_res_next (curs, next - addr); xe_walk->va_curs_start = next; xe_walk->vma->gpuva.flags |= (XE_VMA_PTE_4K << level);
@@ -723,6 +729,26 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, };struct xe_pt *pt = vm->pt_root[tile->id];int ret; +bool is_purged = false; + +/* +* Check if BO is purged: +* - Scratch VMs: Use scratch PTEs (XE_PTE_NULL) for safe zero reads +* - Non-scratch VMs: Clear PTEs to zero (non-present) to avoid mapping to phys addr 0 +* +* For non-scratch VMs, we force clear_pt=true so leaf PTEs become completely +* zero instead of creating a PRESENT mapping to physical address 0. +*/ +if (bo &&xe_bo_is_purged (bo)) { + is_purged = true; + +/* +* For non-scratch VMs, a NULL rebind should use zero PTEs +* (non-present), not a present PTE to phys 0. +*/ +if (!xe_vm_has_scratch (vm)) + xe_walk.clear_pt = true; + }if (range) {/* Move this entire thing to xe_svm.c? */
@@ -762,7 +788,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (!range)xe_bo_assert_held (bo); -if (!xe_vma_is_null (vma) && !range) { +if (!xe_vma_is_null (vma) && !range && !is_purged) {if (xe_vma_is_userptr (vma))xe_res_first_dma (to_userptr_vma (vma)->userptr.pages.dma_addr,0 ,xe_vma_size (vma), &curs);
drivers/gpu/drm/xe/xe_vm.c +10 -1
@@ -1336,6 +1336,9 @@ static u64 xelp_pte_encode_bo(struct xe_bo *bo, u64 bo_offset,static u64 xelp_pte_encode_vma (u64 pte,struct xe_vma *vma,u16 pat_index,u32 pt_level) { +struct xe_bo *bo =xe_vma_bo (vma); +struct xe_vm *vm =xe_vma_vm (vma); + pte |=XE_PAGE_PRESENT ;if (likely (!xe_vma_read_only (vma)))
@@ -1344,7 +1347,13 @@ static u64 xelp_pte_encode_vma(u64 pte, struct xe_vma *vma, pte |=pte_encode_pat_index (pat_index, pt_level); pte |=pte_encode_ps (pt_level); -if (unlikely (xe_vma_is_null (vma))) +/* +* NULL PTEs redirect to scratch page (return zeros on read). +* Set for: 1) explicit null VMAs, 2) purged BOs on scratch VMs. +* Never set NULL flag without scratch page - causes undefined behavior. +*/ +if (unlikely (xe_vma_is_null (vma) || + (bo &&xe_bo_is_purged (bo) &&xe_vm_has_scratch (vm)))) pte |=XE_PTE_NULL ;return pte;
drivers/gpu/drm/xe/xe_vm_madvise.c +73 -0
91 lines not on this page · raw message
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 4/9] drm/xe/madvise: Implement purgeable buffer object support
Quoted 11 lines · Arvind Yadav
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
Add the core implementation for purgeable buffer objects, enabling memory reclamation of user-designated DONTNEED buffers during eviction.
This patch implements the purge operation and state machine transitions:
Purgeable States (from
xe_madv_purgeable_state):
- WILLNEED (0): BO should be retained, actively used
- DONTNEED (1): BO eligible for purging, not currently needed
Quick comment - should we use TTM priority levels so WILLNEED is a higher priority (less likely to be evicted) than DONTNEED (more likely to be evicted).
Expect more comments but just a quick thought.
Matt
Quoted 384 lines
- PURGED (2): BO backing store reclaimed, permanently invalid
Design Rationale:
- Async TLB invalidation viatrigger_rebind(no blockingxe_vm_invalidate_vma)
- i915 compatibility: retained field, "once purged always purged" semantics
- Shared BO protection prevents multi-process memory corruption
- Scratch PTE reuse avoids new infrastructure, safe for fault modev2:
- Usexe_bo_trigger_rebind()for async TLB invalidation (Thomas Hellström)
- Add NULL rebind with scratch PTEs for fault mode (Thomas Hellström)
- Implement i915-compatible retained field logic (Thomas Hellström)
- Skip BO validation for purged BOs in page fault handler (crash fix)
- Add scratch VM check in page fault path (non-scratch VMs fail fault)
- Forceclear_ptfor non-scratch VMs to avoid phys addr 0 mapping (review fix)
- Add !is_purged check to resource cursor setup to prevent stale accessCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 72 ++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_gt_pagefault.c | 19 ++++++++ drivers/gpu/drm/xe/xe_pt.c | 36 ++++++++++++-- drivers/gpu/drm/xe/xe_vm.c | 11 ++++- drivers/gpu/drm/xe/xe_vm_madvise.c | 73 ++++++++++++++++++++++++++++ 5 files changed, 193 insertions(+), 18 deletions(-)drivers/gpu/drm/xe/xe_bo.c +60
-12@@ -836,6 +836,53 @@ static int xe_bo_move_notify(struct xe_bo *bo,return 0 ; } +static void xe_bo_set_purged (struct xe_bo *bo) +{ +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv); + +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_PURGED ); +} + +/** +* xe_ttm_bo_purge() - Purge buffer object backing store +* @ttm_bo: The TTM buffer object to purge +* @ctx: TTM operation context +* +* This function purges the backing store of a BO marked as DONTNEED and +* triggers rebind to invalidate stale GPU mappings. For fault-mode VMs, +* this zaps the PTEs. The next GPU access will trigger a page fault and +* perform NULL rebind (scratch pages or clear PTEs based on VM config). +*/ +static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) +{ +struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); +struct xe_bo *bo =ttm_to_xe_bo (ttm_bo); + +if (ttm_bo->ttm) { +struct ttm_placement place = {}; +int ret =ttm_bo_validate (ttm_bo, &place, ctx); + +drm_WARN_ON (&xe->drm, ret); +if (!ret && bo) { +if (atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_DONTNEED ) { +xe_bo_set_purged (bo); + +/* +* Trigger rebind to invalidate stale GPU mappings. +* - Non-fault mode: Marks VMAs for rebind +* - Fault mode: Zaps PTEs (sets to 0), next access triggers fault +* and NULL rebind with scratch/clear PTEs per VM config +*/ + ret =xe_bo_trigger_rebind (xe, bo, ctx); +if (ret) +drm_warn (&xe->drm, +"Failed to invalidate purged BO: %d\n" , ret); + } + } + } +} + static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_resource *new_mem,@@ -853,8 +900,18 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,bool needs_clear;bool handle_system_ccs = (!IS_DGFX (xe) &&xe_bo_needs_ccs_pages (bo) && ttm &&ttm_tt_is_populated (ttm)) ? true : false; +int state =atomic_read (&bo->madv_purgeable);int ret =0 ; +/* +* Purge only non-shared BOs explicitly marked DONTNEED by userspace. +* The move_notify callback will handle invalidation asynchronously. +*/ +if (evict && state ==XE_MADV_PURGEABLE_DONTNEED && !xe_bo_is_shared_locked (bo)) { +xe_ttm_bo_purge (ttm_bo, ctx); +return 0 ; + } +/* Bo creation path, moving to system or TT. */ if ((!old_mem && ttm) && !handle_system_ccs) {if (new_mem->mem_type ==XE_PL_TT )@@ -1606,18 +1663,6 @@ static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo) } } -static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) -{ -struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); - -if (ttm_bo->ttm) { -struct ttm_placement place = {}; -int ret =ttm_bo_validate (ttm_bo, &place, ctx); - -drm_WARN_ON (&xe->drm, ret); - } -} -static void xe_ttm_bo_swap_notify (struct ttm_buffer_object *ttm_bo) {struct ttm_operation_ctx ctx = {@@ -2202,6 +2247,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,#endif INIT_LIST_HEAD (&bo->vram_userfault_link); +/* Initialize purge advisory state */ +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +drm_gem_private_object_init (&xe->drm, &bo->ttm.base, size);if (resv) {drivers/gpu/drm/xe/xe_gt_pagefault.c +19
-0@@ -87,6 +87,13 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,if (!bo)return 0 ; +/* +* Skip validation/migration for purged BOs - they have no backing pages. +* Rebind will use scratch PTEs instead. +*/ +if (xe_bo_is_purged (bo)) +return 0 ; +return need_vram_move ?xe_bo_migrate (bo, vram->placement,NULL , exec) :xe_bo_validate (bo, vm, true, exec); }@@ -100,9 +107,21 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,struct drm_exec exec;struct dma_fence *fence;int err, needs_vram; +struct xe_bo *bo;lockdep_assert_held_write (&vm->lock); +/* +* Check if BO is purged. For purged BOs: +* - Scratch VMs: Allow rebind with scratch PTEs (safe zero reads) +* - Non-scratch VMs: FAIL the page fault (no scratch page available) +*/ + bo =xe_vma_bo (vma); +if (bo &&xe_bo_is_purged (bo)) { +if (!xe_vm_has_scratch (vm)) +return -EACCES ; + } + needs_vram =xe_vma_need_vram_for_atomic (vm->xe, vma, atomic);if (needs_vram <0 || (needs_vram &&xe_vma_is_userptr (vma)))return needs_vram <0 ? needs_vram : -EACCES ;drivers/gpu/drm/xe/xe_pt.c +31
-5@@ -533,20 +533,26 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,/* Is this a leaf entry ?*/ if (level ==0 ||xe_pt_hugepte_possible (addr, next, level, xe_walk)) {struct xe_res_cursor *curs = xe_walk->curs; +struct xe_bo *bo =xe_vma_bo (xe_walk->vma);bool is_null =xe_vma_is_null (xe_walk->vma); -bool is_vram = is_null ? false :xe_res_is_vram (curs); +bool is_purged = bo &&xe_bo_is_purged (bo); +bool is_vram = (is_null || is_purged) ? false :xe_res_is_vram (curs);XE_WARN_ON (xe_walk->va_curs_start != addr);if (xe_walk->clear_pt) { pte =0 ; }else { - pte = vm->pt_ops->pte_encode_vma (is_null ?0 : +/* +* For purged BOs, treat like null VMAs - pass address 0. +* The pte_encode_vma will set XE_PTE_NULL flag for scratch mapping. +*/ + pte = vm->pt_ops->pte_encode_vma ((is_null || is_purged) ?0 :xe_res_dma (curs) + xe_walk->dma_offset, xe_walk->vma, pat_index, level); -if (!is_null) +if (!is_null && !is_purged) pte |= is_vram ? xe_walk->default_vram_pte : xe_walk->default_system_pte;@@ -570,7 +576,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,if (unlikely (ret))return ret; -if (!is_null && !xe_walk->clear_pt) +if (!is_null && !is_purged && !xe_walk->clear_pt)xe_res_next (curs, next - addr); xe_walk->va_curs_start = next; xe_walk->vma->gpuva.flags |= (XE_VMA_PTE_4K << level);@@ -723,6 +729,26 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, };struct xe_pt *pt = vm->pt_root[tile->id];int ret; +bool is_purged = false; + +/* +* Check if BO is purged: +* - Scratch VMs: Use scratch PTEs (XE_PTE_NULL) for safe zero reads +* - Non-scratch VMs: Clear PTEs to zero (non-present) to avoid mapping to phys addr 0 +* +* For non-scratch VMs, we force clear_pt=true so leaf PTEs become completely +* zero instead of creating a PRESENT mapping to physical address 0. +*/ +if (bo &&xe_bo_is_purged (bo)) { + is_purged = true; + +/* +* For non-scratch VMs, a NULL rebind should use zero PTEs +* (non-present), not a present PTE to phys 0. +*/ +if (!xe_vm_has_scratch (vm)) + xe_walk.clear_pt = true; + }if (range) {/* Move this entire thing to xe_svm.c? */ @@ -762,7 +788,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (!range)xe_bo_assert_held (bo); -if (!xe_vma_is_null (vma) && !range) { +if (!xe_vma_is_null (vma) && !range && !is_purged) {if (xe_vma_is_userptr (vma))xe_res_first_dma (to_userptr_vma (vma)->userptr.pages.dma_addr,0 ,xe_vma_size (vma), &curs);drivers/gpu/drm/xe/xe_vm.c +10
-1@@ -1336,6 +1336,9 @@ static u64 xelp_pte_encode_bo(struct xe_bo *bo, u64 bo_offset,static u64 xelp_pte_encode_vma (u64 pte,struct xe_vma *vma,u16 pat_index,u32 pt_level) { +struct xe_bo *bo =xe_vma_bo (vma); +struct xe_vm *vm =xe_vma_vm (vma); + pte |=XE_PAGE_PRESENT ;if (likely (!xe_vma_read_only (vma)))@@ -1344,7 +1347,13 @@ static u64 xelp_pte_encode_vma(u64 pte, struct xe_vma *vma, pte |=pte_encode_pat_index (pat_index, pt_level); pte |=pte_encode_ps (pt_level); -if (unlikely (xe_vma_is_null (vma))) +/* +* NULL PTEs redirect to scratch page (return zeros on read). +* Set for: 1) explicit null VMAs, 2) purged BOs on scratch VMs. +* Never set NULL flag without scratch page - causes undefined behavior. +*/ +if (unlikely (xe_vma_is_null (vma) || + (bo &&xe_bo_is_purged (bo) &&xe_vm_has_scratch (vm)))) pte |=XE_PTE_NULL ;return pte;drivers/gpu/drm/xe/xe_vm_madvise.c +73
-0@@ -158,6 +158,60 @@ static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, } } +/* +* Handle purgeable buffer object advice for DONTNEED/WILLNEED/PURGED. +* Updates op->purge_state_val.retained to indicate if backing store +* exists (matches i915's retained). +*/ +static void xe_vm_madvise_purgeable_bo (struct xe_device *xe,struct xe_vm *vm, +struct xe_vma **vmas,int num_vmas, +struct drm_xe_madvise *op) +{ +bool has_purged_bo = false; +int i; + +xe_assert (vm->xe, op->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ); + +for (i =0 ; i < num_vmas; i++) { +struct xe_bo *bo =xe_vma_bo (vmas[i]); + +if (!bo) +continue ; + +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv); + +/* +* Once purged, always purged. Cannot transition back to WILLNEED. +* This matches i915 semantics where purged BOs are permanently invalid. +*/ +if (xe_bo_is_purged (bo)) { + has_purged_bo = true; +continue ; + } + +switch (op->purge_state_val.val) { +case DRM_XE_VMA_PURGEABLE_STATE_WILLNEED : +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +break ; +case DRM_XE_VMA_PURGEABLE_STATE_DONTNEED : +if (!xe_bo_is_shared_locked (bo)) +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_DONTNEED ); +break ; +default : +drm_warn (&vm->xe->drm,"Invalid madvice value = %d\n" , + op->purge_state_val.val); +return ; + } + } + +/* +* Set retained flag to indicate if backing store still exists. +* Matches i915: retained = 1 if not purged, 0 if purged. +*/ + op->purge_state_val.retained = !has_purged_bo; +} +typedef void (*madvise_func )(struct xe_device *xe,struct xe_vm *vm,struct xe_vma **vmas,int num_vmas,struct drm_xe_madvise *op);@@ -283,6 +337,19 @@ static bool madvise_args_are_sane(struct xe_device *xe, const struct drm_xe_madvreturn false;break ; } +case DRM_XE_VMA_ATTR_PURGEABLE_STATE : + { +u32 val = args->purge_state_val.val; + +if (XE_IOCTL_DBG (xe, !((val ==DRM_XE_VMA_PURGEABLE_STATE_WILLNEED ) || + (val ==DRM_XE_VMA_PURGEABLE_STATE_DONTNEED )))) +return false; + +if (XE_IOCTL_DBG (xe, args->purge_state_val.reserved)) +return false; + +break ; + }default :if (XE_IOCTL_DBG (xe,1 ))return false;@@ -402,6 +469,12 @@ int xe_vm_madvise_ioctl(struct drm_device *dev, void *data, struct drm_file *fil goto err_fini; } } +if (args->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ) { +xe_vm_madvise_purgeable_bo (xe, vm, madvise_range.vmas, + madvise_range.num_vmas, args); + goto err_fini; + + } }if (madvise_range.has_svm_userptr_vmas) {Signature · 1 line
2.43.0
Re: [RFC v2 4/9] drm/xe/madvise: Implement purgeable buffer object support
Quoted 9 lines · Matthew Brost
Quoted 8 lines · Arvind Yadav
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
Add the core implementation for purgeable buffer objects, enabling memory reclamation of user-designated DONTNEED buffers during eviction.
This patch implements the purge operation and state machine transitions:
Quoted 8 lines
Quoted 3 lines
Purgeable States (from
xe_madv_purgeable_state):
- WILLNEED (0): BO should be retained, actively used
- DONTNEED (1): BO eligible for purging, not currently neededQuick comment - should we use TTM priority levels so WILLNEED is a higher priority (less likely to be evicted) than DONTNEED (more likely to be evicted).
Expect more comments but just a quick thought.
Yes, we should leverage TTM priority levels for better eviction ordering. Currently TTM has separate LRU lists per priority (man->lru[TTM_MAX_BO_PRIORITY]), and eviction walks the lists starting from lower priority BOs first.
1. Set DONTNEED BOs to priority 0 (evicted first, before normal BOs)
2. Keep WILLNEED BOs at priority ' XE_BO_PRIORITY_NORMAL' (normal eviction order)
~Arvind
Quoted 386 lines
Matt
Quoted 384 lines
- PURGED (2): BO backing store reclaimed, permanently invalid
Design Rationale:
- Async TLB invalidation viatrigger_rebind(no blockingxe_vm_invalidate_vma)
- i915 compatibility: retained field, "once purged always purged" semantics
- Shared BO protection prevents multi-process memory corruption
- Scratch PTE reuse avoids new infrastructure, safe for fault modev2:
- Usexe_bo_trigger_rebind()for async TLB invalidation (Thomas Hellström)
- Add NULL rebind with scratch PTEs for fault mode (Thomas Hellström)
- Implement i915-compatible retained field logic (Thomas Hellström)
- Skip BO validation for purged BOs in page fault handler (crash fix)
- Add scratch VM check in page fault path (non-scratch VMs fail fault)
- Forceclear_ptfor non-scratch VMs to avoid phys addr 0 mapping (review fix)
- Add !is_purged check to resource cursor setup to prevent stale accessCc Matthew Brost<[email protected]>Cc Thomas Hellström<[email protected]>Cc Himal Prasad Ghimiray<[email protected]>Signed-off-by Arvind Yadav<[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 72 ++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_gt_pagefault.c | 19 ++++++++ drivers/gpu/drm/xe/xe_pt.c | 36 ++++++++++++-- drivers/gpu/drm/xe/xe_vm.c | 11 ++++- drivers/gpu/drm/xe/xe_vm_madvise.c | 73 ++++++++++++++++++++++++++++ 5 files changed, 193 insertions(+), 18 deletions(-)drivers/gpu/drm/xe/xe_bo.c +60
-12@@ -836,6 +836,53 @@ static int xe_bo_move_notify(struct xe_bo *bo,return 0 ; } +static void xe_bo_set_purged (struct xe_bo *bo) +{ +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv); + +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_PURGED ); +} + +/** +* xe_ttm_bo_purge() - Purge buffer object backing store +* @ttm_bo: The TTM buffer object to purge +* @ctx: TTM operation context +* +* This function purges the backing store of a BO marked as DONTNEED and +* triggers rebind to invalidate stale GPU mappings. For fault-mode VMs, +* this zaps the PTEs. The next GPU access will trigger a page fault and +* perform NULL rebind (scratch pages or clear PTEs based on VM config). +*/ +static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) +{ +struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); +struct xe_bo *bo =ttm_to_xe_bo (ttm_bo); + +if (ttm_bo->ttm) { +struct ttm_placement place = {}; +int ret =ttm_bo_validate (ttm_bo, &place, ctx); + +drm_WARN_ON (&xe->drm, ret); +if (!ret && bo) { +if (atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_DONTNEED ) { +xe_bo_set_purged (bo); + +/* +* Trigger rebind to invalidate stale GPU mappings. +* - Non-fault mode: Marks VMAs for rebind +* - Fault mode: Zaps PTEs (sets to 0), next access triggers fault +* and NULL rebind with scratch/clear PTEs per VM config +*/ + ret =xe_bo_trigger_rebind (xe, bo, ctx); +if (ret) +drm_warn (&xe->drm, +"Failed to invalidate purged BO: %d\n" , ret); + } + } + } +} + static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_resource *new_mem,@@ -853,8 +900,18 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,bool needs_clear;bool handle_system_ccs = (!IS_DGFX (xe) &&xe_bo_needs_ccs_pages (bo) && ttm &&ttm_tt_is_populated (ttm)) ? true : false; +int state =atomic_read (&bo->madv_purgeable);int ret =0 ; +/* +* Purge only non-shared BOs explicitly marked DONTNEED by userspace. +* The move_notify callback will handle invalidation asynchronously. +*/ +if (evict && state ==XE_MADV_PURGEABLE_DONTNEED && !xe_bo_is_shared_locked (bo)) { +xe_ttm_bo_purge (ttm_bo, ctx); +return 0 ; + } +/* Bo creation path, moving to system or TT. */ if ((!old_mem && ttm) && !handle_system_ccs) {if (new_mem->mem_type ==XE_PL_TT )@@ -1606,18 +1663,6 @@ static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo) } } -static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) -{ -struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); - -if (ttm_bo->ttm) { -struct ttm_placement place = {}; -int ret =ttm_bo_validate (ttm_bo, &place, ctx); - -drm_WARN_ON (&xe->drm, ret); - } -} -static void xe_ttm_bo_swap_notify (struct ttm_buffer_object *ttm_bo) {struct ttm_operation_ctx ctx = {@@ -2202,6 +2247,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,#endif INIT_LIST_HEAD (&bo->vram_userfault_link); +/* Initialize purge advisory state */ +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +drm_gem_private_object_init (&xe->drm, &bo->ttm.base, size);if (resv) {drivers/gpu/drm/xe/xe_gt_pagefault.c +19
-0@@ -87,6 +87,13 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,if (!bo)return 0 ; +/* +* Skip validation/migration for purged BOs - they have no backing pages. +* Rebind will use scratch PTEs instead. +*/ +if (xe_bo_is_purged (bo)) +return 0 ; +return need_vram_move ?xe_bo_migrate (bo, vram->placement,NULL , exec) :xe_bo_validate (bo, vm, true, exec); }@@ -100,9 +107,21 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,struct drm_exec exec;struct dma_fence *fence;int err, needs_vram; +struct xe_bo *bo;lockdep_assert_held_write (&vm->lock); +/* +* Check if BO is purged. For purged BOs: +* - Scratch VMs: Allow rebind with scratch PTEs (safe zero reads) +* - Non-scratch VMs: FAIL the page fault (no scratch page available) +*/ + bo =xe_vma_bo (vma); +if (bo &&xe_bo_is_purged (bo)) { +if (!xe_vm_has_scratch (vm)) +return -EACCES ; + } + needs_vram =xe_vma_need_vram_for_atomic (vm->xe, vma, atomic);if (needs_vram <0 || (needs_vram &&xe_vma_is_userptr (vma)))return needs_vram <0 ? needs_vram : -EACCES ;drivers/gpu/drm/xe/xe_pt.c +31
-5@@ -533,20 +533,26 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,/* Is this a leaf entry ?*/ if (level ==0 ||xe_pt_hugepte_possible (addr, next, level, xe_walk)) {struct xe_res_cursor *curs = xe_walk->curs; +struct xe_bo *bo =xe_vma_bo (xe_walk->vma);bool is_null =xe_vma_is_null (xe_walk->vma); -bool is_vram = is_null ? false :xe_res_is_vram (curs); +bool is_purged = bo &&xe_bo_is_purged (bo); +bool is_vram = (is_null || is_purged) ? false :xe_res_is_vram (curs);XE_WARN_ON (xe_walk->va_curs_start != addr);if (xe_walk->clear_pt) { pte =0 ; }else { - pte = vm->pt_ops->pte_encode_vma (is_null ?0 : +/* +* For purged BOs, treat like null VMAs - pass address 0. +* The pte_encode_vma will set XE_PTE_NULL flag for scratch mapping. +*/ + pte = vm->pt_ops->pte_encode_vma ((is_null || is_purged) ?0 :xe_res_dma (curs) + xe_walk->dma_offset, xe_walk->vma, pat_index, level); -if (!is_null) +if (!is_null && !is_purged) pte |= is_vram ? xe_walk->default_vram_pte : xe_walk->default_system_pte;@@ -570,7 +576,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,if (unlikely (ret))return ret; -if (!is_null && !xe_walk->clear_pt) +if (!is_null && !is_purged && !xe_walk->clear_pt)xe_res_next (curs, next - addr); xe_walk->va_curs_start = next; xe_walk->vma->gpuva.flags |= (XE_VMA_PTE_4K << level);@@ -723,6 +729,26 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, };struct xe_pt *pt = vm->pt_root[tile->id];int ret; +bool is_purged = false; + +/* +* Check if BO is purged: +* - Scratch VMs: Use scratch PTEs (XE_PTE_NULL) for safe zero reads +* - Non-scratch VMs: Clear PTEs to zero (non-present) to avoid mapping to phys addr 0 +* +* For non-scratch VMs, we force clear_pt=true so leaf PTEs become completely +* zero instead of creating a PRESENT mapping to physical address 0. +*/ +if (bo &&xe_bo_is_purged (bo)) { + is_purged = true; + +/* +* For non-scratch VMs, a NULL rebind should use zero PTEs +* (non-present), not a present PTE to phys 0. +*/ +if (!xe_vm_has_scratch (vm)) + xe_walk.clear_pt = true; + }if (range) {/* Move this entire thing to xe_svm.c? */ @@ -762,7 +788,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (!range)xe_bo_assert_held (bo); -if (!xe_vma_is_null (vma) && !range) { +if (!xe_vma_is_null (vma) && !range && !is_purged) {if (xe_vma_is_userptr (vma))xe_res_first_dma (to_userptr_vma (vma)->userptr.pages.dma_addr,0 ,xe_vma_size (vma), &curs);drivers/gpu/drm/xe/xe_vm.c +10
-1@@ -1336,6 +1336,9 @@ static u64 xelp_pte_encode_bo(struct xe_bo *bo, u64 bo_offset,static u64 xelp_pte_encode_vma (u64 pte,struct xe_vma *vma,u16 pat_index,u32 pt_level) { +struct xe_bo *bo =xe_vma_bo (vma); +struct xe_vm *vm =xe_vma_vm (vma); + pte |=XE_PAGE_PRESENT ;if (likely (!xe_vma_read_only (vma)))@@ -1344,7 +1347,13 @@ static u64 xelp_pte_encode_vma(u64 pte, struct xe_vma *vma, pte |=pte_encode_pat_index (pat_index, pt_level); pte |=pte_encode_ps (pt_level); -if (unlikely (xe_vma_is_null (vma))) +/* +* NULL PTEs redirect to scratch page (return zeros on read). +* Set for: 1) explicit null VMAs, 2) purged BOs on scratch VMs. +* Never set NULL flag without scratch page - causes undefined behavior. +*/ +if (unlikely (xe_vma_is_null (vma) || + (bo &&xe_bo_is_purged (bo) &&xe_vm_has_scratch (vm)))) pte |=XE_PTE_NULL ;return pte;drivers/gpu/drm/xe/xe_vm_madvise.c +73
-0@@ -158,6 +158,60 @@ static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, } } +/* +* Handle purgeable buffer object advice for DONTNEED/WILLNEED/PURGED. +* Updates op->purge_state_val.retained to indicate if backing store +* exists (matches i915's retained). +*/ +static void xe_vm_madvise_purgeable_bo (struct xe_device *xe,struct xe_vm *vm, +struct xe_vma **vmas,int num_vmas, +struct drm_xe_madvise *op) +{ +bool has_purged_bo = false; +int i; + +xe_assert (vm->xe, op->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ); + +for (i =0 ; i < num_vmas; i++) { +struct xe_bo *bo =xe_vma_bo (vmas[i]); + +if (!bo) +continue ; + +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv); + +/* +* Once purged, always purged. Cannot transition back to WILLNEED. +* This matches i915 semantics where purged BOs are permanently invalid. +*/ +if (xe_bo_is_purged (bo)) { + has_purged_bo = true; +continue ; + } + +switch (op->purge_state_val.val) { +case DRM_XE_VMA_PURGEABLE_STATE_WILLNEED : +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +break ; +case DRM_XE_VMA_PURGEABLE_STATE_DONTNEED : +if (!xe_bo_is_shared_locked (bo)) +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_DONTNEED ); +break ; +default : +drm_warn (&vm->xe->drm,"Invalid madvice value = %d\n" , + op->purge_state_val.val); +return ; + } + } + +/* +* Set retained flag to indicate if backing store still exists. +* Matches i915: retained = 1 if not purged, 0 if purged. +*/ + op->purge_state_val.retained = !has_purged_bo; +} +typedef void (*madvise_func )(struct xe_device *xe,struct xe_vm *vm,struct xe_vma **vmas,int num_vmas,struct drm_xe_madvise *op);@@ -283,6 +337,19 @@ static bool madvise_args_are_sane(struct xe_device *xe, const struct drm_xe_madvreturn false;break ; } +case DRM_XE_VMA_ATTR_PURGEABLE_STATE : + { +u32 val = args->purge_state_val.val; + +if (XE_IOCTL_DBG (xe, !((val ==DRM_XE_VMA_PURGEABLE_STATE_WILLNEED ) || + (val ==DRM_XE_VMA_PURGEABLE_STATE_DONTNEED )))) +return false; + +if (XE_IOCTL_DBG (xe, args->purge_state_val.reserved)) +return false; + +break ; + }default :if (XE_IOCTL_DBG (xe,1 ))return false;@@ -402,6 +469,12 @@ int xe_vm_madvise_ioctl(struct drm_device *dev, void *data, struct drm_file *fil goto err_fini; } } +if (args->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ) { +xe_vm_madvise_purgeable_bo (xe, vm, madvise_range.vmas, + madvise_range.num_vmas, args); + goto err_fini; + + } }if (madvise_range.has_svm_userptr_vmas) {Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 4/9] drm/xe/madvise: Implement purgeable buffer object support
In reply to Arvind YadavQuoted 45 lines · Arvind Yadav
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
Add the core implementation for purgeable buffer objects, enabling memory reclamation of user-designated DONTNEED buffers during eviction.
This patch implements the purge operation and state machine transitions:
Purgeable States (from
xe_madv_purgeable_state):
- WILLNEED (0): BO should be retained, actively used
- DONTNEED (1): BO eligible for purging, not currently needed
- PURGED (2): BO backing store reclaimed, permanently invalidDesign Rationale:
- Async TLB invalidation viatrigger_rebind(no blockingxe_vm_invalidate_vma)
- i915 compatibility: retained field, "once purged always purged" semantics
- Shared BO protection prevents multi-process memory corruption
- Scratch PTE reuse avoids new infrastructure, safe for fault modev2:
- Usexe_bo_trigger_rebind()for async TLB invalidation (Thomas Hellström)
- Add NULL rebind with scratch PTEs for fault mode (Thomas Hellström)
- Implement i915-compatible retained field logic (Thomas Hellström)
- Skip BO validation for purged BOs in page fault handler (crash fix)
- Add scratch VM check in page fault path (non-scratch VMs fail fault)
- Forceclear_ptfor non-scratch VMs to avoid phys addr 0 mapping (review fix)
- Add !is_purged check to resource cursor setup to prevent stale accessCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 72 ++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_gt_pagefault.c | 19 ++++++++ drivers/gpu/drm/xe/xe_pt.c | 36 ++++++++++++-- drivers/gpu/drm/xe/xe_vm.c | 11 ++++- drivers/gpu/drm/xe/xe_vm_madvise.c | 73 ++++++++++++++++++++++++++++ 5 files changed, 193 insertions(+), 18 deletions(-)drivers/gpu/drm/xe/xe_bo.c +0
-0
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 843
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c +5
-0@@ -836,6 +836,53 @@ static int xe_bo_move_notify(struct xe_bo *bo,return 0 ; } +static void xe_bo_set_purged (struct xe_bo *bo) +{ +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv); +
+1 to assert, but I think xe_bo_assert_held can be used.
Quoted 11 lines
drivers/gpu/drm/xe/xe_bo.c +11
-0+atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_PURGED ); +} + +/** +* xe_ttm_bo_purge() - Purge buffer object backing store +* @ttm_bo: The TTM buffer object to purge +* @ctx: TTM operation context +* +* This function purges the backing store of a BO marked as DONTNEED and +* triggers rebind to invalidate stale GPU mappings. For fault-mode VMs, +* this zaps the PTEs. The next GPU access will trigger a page fault and
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 864
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10
-0+* perform NULL rebind (scratch pages or clear PTEs based on VM config). +*/ +static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) +{ +struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); +struct xe_bo *bo =ttm_to_xe_bo (ttm_bo); + +if (ttm_bo->ttm) { +struct ttm_placement place = {}; +int ret =ttm_bo_validate (ttm_bo, &place, ctx);
Do we need to the eviction to complete? e.g., wait on DMA_RESV_USAGE_KERNEL slots?
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 866
Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +2
-0+ +drm_WARN_ON (&xe->drm, ret);
Likely an specific warn on function or assert.
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 867
Quoted 1 line
drivers/gpu/drm/xe/xe_bo.c +1
-0+if (!ret && bo) {
When would BO be NULL? I don't think ever.
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 868
Quoted 1 line
drivers/gpu/drm/xe/xe_bo.c +1
-0+if (atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_DONTNEED ) {
Can we helpers for madv_purgeable access /w an assert?
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 880
Quoted 12 lines
drivers/gpu/drm/xe/xe_bo.c +12
-0+xe_bo_set_purged (bo); + +/* +* Trigger rebind to invalidate stale GPU mappings. +* - Non-fault mode: Marks VMAs for rebind +* - Fault mode: Zaps PTEs (sets to 0), next access triggers fault +* and NULL rebind with scratch/clear PTEs per VM config +*/ + ret =xe_bo_trigger_rebind (xe, bo, ctx); +if (ret) +drm_warn (&xe->drm, +"Failed to invalidate purged BO: %d\n" , ret);
Xe specific warn on or assert?
Also maybe this function just returns an error if something fails too and xe_bo_move would return that failure.
Quoted 61 lines
drivers/gpu/drm/xe/xe_bo.c +18
-12+ } + } + } +} + static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_resource *new_mem,@@ -853,8 +900,18 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,bool needs_clear;bool handle_system_ccs = (!IS_DGFX (xe) &&xe_bo_needs_ccs_pages (bo) && ttm &&ttm_tt_is_populated (ttm)) ? true : false; +int state =atomic_read (&bo->madv_purgeable);int ret =0 ; +/* +* Purge only non-shared BOs explicitly marked DONTNEED by userspace. +* The move_notify callback will handle invalidation asynchronously. +*/ +if (evict && state ==XE_MADV_PURGEABLE_DONTNEED && !xe_bo_is_shared_locked (bo)) { +xe_ttm_bo_purge (ttm_bo, ctx); +return 0 ; + } +/* Bo creation path, moving to system or TT. */ if ((!old_mem && ttm) && !handle_system_ccs) {if (new_mem->mem_type ==XE_PL_TT )@@ -1606,18 +1663,6 @@ static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo) } } -static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) -{ -struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); - -if (ttm_bo->ttm) { -struct ttm_placement place = {}; -int ret =ttm_bo_validate (ttm_bo, &place, ctx); - -drm_WARN_ON (&xe->drm, ret); - } -} -static void xe_ttm_bo_swap_notify (struct ttm_buffer_object *ttm_bo) {struct ttm_operation_ctx ctx = {@@ -2202,6 +2247,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,#endif INIT_LIST_HEAD (&bo->vram_userfault_link); +/* Initialize purge advisory state */ +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +drm_gem_private_object_init (&xe->drm, &bo->ttm.base, size);if (resv) {drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0
drivers/gpu/drm/xe/xe_gt_pagefault.c static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma, · line 96
Quoted 10 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +7
-0@@ -87,6 +87,13 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,if (!bo)return 0 ; +/* +* Skip validation/migration for purged BOs - they have no backing pages. +* Rebind will use scratch PTEs instead. +*/ +if (xe_bo_is_purged (bo)) +return 0 ; +
This needs a rebase as xe_gt_pagefault.c is gone upstream and replaced with xe_pagefault.c.
Quoted 11 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-0return need_vram_move ?xe_bo_migrate (bo, vram->placement,NULL , exec) :xe_bo_validate (bo, vm, true, exec); }@@ -100,9 +107,21 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,struct drm_exec exec;struct dma_fence *fence;int err, needs_vram; +struct xe_bo *bo;lockdep_assert_held_write (&vm->lock);
drivers/gpu/drm/xe/xe_gt_pagefault.c static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma, · line 123
Quoted 10 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +10
-0+/* +* Check if BO is purged. For purged BOs: +* - Scratch VMs: Allow rebind with scratch PTEs (safe zero reads) +* - Non-scratch VMs: FAIL the page fault (no scratch page available) +*/ + bo =xe_vma_bo (vma); +if (bo &&xe_bo_is_purged (bo)) { +if (!xe_vm_has_scratch (vm)) +return -EACCES ; + }
As discussed in other patches, move the xe_bo_is_purged check under the dma-resv lock.
Quoted 9 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-0+ needs_vram =xe_vma_need_vram_for_atomic (vm->xe, vma, atomic);if (needs_vram <0 || (needs_vram &&xe_vma_is_userptr (vma)))return needs_vram <0 ? needs_vram : -EACCES ;drivers/gpu/drm/xe/xe_pt.c +0
-0
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, · line 538
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +2
-1@@ -533,20 +533,26 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,/* Is this a leaf entry ?*/ if (level ==0 ||xe_pt_hugepte_possible (addr, next, level, xe_walk)) {struct xe_res_cursor *curs = xe_walk->curs; +struct xe_bo *bo =xe_vma_bo (xe_walk->vma);bool is_null =xe_vma_is_null (xe_walk->vma); -bool is_vram = is_null ? false :xe_res_is_vram (curs); +bool is_purged = bo &&xe_bo_is_purged (bo);
Can we drop is_purged and just set is_null to true for purged BOs?
Or rename s/is_null/is_null_or_purged for clarity?
Quoted 44 lines
drivers/gpu/drm/xe/xe_pt.c +17
-3+bool is_vram = (is_null || is_purged) ? false :xe_res_is_vram (curs);XE_WARN_ON (xe_walk->va_curs_start != addr);if (xe_walk->clear_pt) { pte =0 ; }else { - pte = vm->pt_ops->pte_encode_vma (is_null ?0 : +/* +* For purged BOs, treat like null VMAs - pass address 0. +* The pte_encode_vma will set XE_PTE_NULL flag for scratch mapping. +*/ + pte = vm->pt_ops->pte_encode_vma ((is_null || is_purged) ?0 :xe_res_dma (curs) + xe_walk->dma_offset, xe_walk->vma, pat_index, level); -if (!is_null) +if (!is_null && !is_purged) pte |= is_vram ? xe_walk->default_vram_pte : xe_walk->default_system_pte;@@ -570,7 +576,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,if (unlikely (ret))return ret; -if (!is_null && !xe_walk->clear_pt) +if (!is_null && !is_purged && !xe_walk->clear_pt)xe_res_next (curs, next - addr); xe_walk->va_curs_start = next; xe_walk->vma->gpuva.flags |= (XE_VMA_PTE_4K << level);@@ -723,6 +729,26 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, };struct xe_pt *pt = vm->pt_root[tile->id];int ret; +bool is_purged = false; + +/* +* Check if BO is purged: +* - Scratch VMs: Use scratch PTEs (XE_PTE_NULL) for safe zero reads +* - Non-scratch VMs: Clear PTEs to zero (non-present) to avoid mapping to phys addr 0 +* +* For non-scratch VMs, we force clear_pt=true so leaf PTEs become completely +* zero instead of creating a PRESENT mapping to physical address 0.
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, · line 750
Quoted 10 lines
drivers/gpu/drm/xe/xe_pt.c +10
-0+*/ +if (bo &&xe_bo_is_purged (bo)) { + is_purged = true; + +/* +* For non-scratch VMs, a NULL rebind should use zero PTEs +* (non-present), not a present PTE to phys 0. +*/ +if (!xe_vm_has_scratch (vm)) + xe_walk.clear_pt = true;
So the idea is purged BOs will fault if the VMA is accessed on non-scratch VMs?
Quoted 62 lines
drivers/gpu/drm/xe/xe_pt.c +2
-1+ }if (range) {/* Move this entire thing to xe_svm.c? */ @@ -762,7 +788,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (!range)xe_bo_assert_held (bo); -if (!xe_vma_is_null (vma) && !range) { +if (!xe_vma_is_null (vma) && !range && !is_purged) {if (xe_vma_is_userptr (vma))xe_res_first_dma (to_userptr_vma (vma)->userptr.pages.dma_addr,0 ,xe_vma_size (vma), &curs);drivers/gpu/drm/xe/xe_vm.c +10
-1@@ -1336,6 +1336,9 @@ static u64 xelp_pte_encode_bo(struct xe_bo *bo, u64 bo_offset,static u64 xelp_pte_encode_vma (u64 pte,struct xe_vma *vma,u16 pat_index,u32 pt_level) { +struct xe_bo *bo =xe_vma_bo (vma); +struct xe_vm *vm =xe_vma_vm (vma); + pte |=XE_PAGE_PRESENT ;if (likely (!xe_vma_read_only (vma)))@@ -1344,7 +1347,13 @@ static u64 xelp_pte_encode_vma(u64 pte, struct xe_vma *vma, pte |=pte_encode_pat_index (pat_index, pt_level); pte |=pte_encode_ps (pt_level); -if (unlikely (xe_vma_is_null (vma))) +/* +* NULL PTEs redirect to scratch page (return zeros on read). +* Set for: 1) explicit null VMAs, 2) purged BOs on scratch VMs. +* Never set NULL flag without scratch page - causes undefined behavior. +*/ +if (unlikely (xe_vma_is_null (vma) || + (bo &&xe_bo_is_purged (bo) &&xe_vm_has_scratch (vm)))) pte |=XE_PTE_NULL ;return pte;drivers/gpu/drm/xe/xe_vm_madvise.c +12
-0@@ -158,6 +158,60 @@ static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, } } +/* +* Handle purgeable buffer object advice for DONTNEED/WILLNEED/PURGED. +* Updates op->purge_state_val.retained to indicate if backing store +* exists (matches i915's retained). +*/ +static void xe_vm_madvise_purgeable_bo (struct xe_device *xe,struct xe_vm *vm, +struct xe_vma **vmas,int num_vmas, +struct drm_xe_madvise *op) +{ +bool has_purged_bo = false; +int i; +
drivers/gpu/drm/xe/xe_vm_madvise.c static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, · line 182
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +10
-0+xe_assert (vm->xe, op->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ); + +for (i =0 ; i < num_vmas; i++) { +struct xe_bo *bo =xe_vma_bo (vmas[i]); + +if (!bo) +continue ; + +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv);
xe_bo_assert_held
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +6
-0+ +/* +* Once purged, always purged. Cannot transition back to WILLNEED. +* This matches i915 semantics where purged BOs are permanently invalid. +*/ +if (xe_bo_is_purged (bo)) {
drivers/gpu/drm/xe/xe_vm_madvise.c static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, · line 198
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +10
-0+ has_purged_bo = true; +continue ; + } + +switch (op->purge_state_val.val) { +case DRM_XE_VMA_PURGEABLE_STATE_WILLNEED : +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +break ; +case DRM_XE_VMA_PURGEABLE_STATE_DONTNEED : +if (!xe_bo_is_shared_locked (bo))
I think we need a complete VMA check here per Thomas's feedback (e.g., all VMAs attached to the BO must be in DRM_XE_VMA_PURGEABLE_STATE_DONTNEED to flip the BO state). I think the BO is an import or exported dma-buf we can never flip the state as we don't know what an external device is doing with it.
Matt
Quoted 55 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +35
-0+atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_DONTNEED ); +break ; +default : +drm_warn (&vm->xe->drm,"Invalid madvice value = %d\n" , + op->purge_state_val.val); +return ; + } + } + +/* +* Set retained flag to indicate if backing store still exists. +* Matches i915: retained = 1 if not purged, 0 if purged. +*/ + op->purge_state_val.retained = !has_purged_bo; +} +typedef void (*madvise_func )(struct xe_device *xe,struct xe_vm *vm,struct xe_vma **vmas,int num_vmas,struct drm_xe_madvise *op);@@ -283,6 +337,19 @@ static bool madvise_args_are_sane(struct xe_device *xe, const struct drm_xe_madvreturn false;break ; } +case DRM_XE_VMA_ATTR_PURGEABLE_STATE : + { +u32 val = args->purge_state_val.val; + +if (XE_IOCTL_DBG (xe, !((val ==DRM_XE_VMA_PURGEABLE_STATE_WILLNEED ) || + (val ==DRM_XE_VMA_PURGEABLE_STATE_DONTNEED )))) +return false; + +if (XE_IOCTL_DBG (xe, args->purge_state_val.reserved)) +return false; + +break ; + }default :if (XE_IOCTL_DBG (xe,1 ))return false;@@ -402,6 +469,12 @@ int xe_vm_madvise_ioctl(struct drm_device *dev, void *data, struct drm_file *fil goto err_fini; } } +if (args->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ) { +xe_vm_madvise_purgeable_bo (xe, vm, madvise_range.vmas, + madvise_range.num_vmas, args); + goto err_fini; + + } }if (madvise_range.has_svm_userptr_vmas) {Signature · 1 line
2.43.0
Re: [RFC v2 4/9] drm/xe/madvise: Implement purgeable buffer object support
Quoted 49 lines · Matthew Brost
Quoted 48 lines · Arvind Yadav
This allows userspace applications to provide memory usage hints to the kernel for better memory management under pressure:
Add the core implementation for purgeable buffer objects, enabling memory reclamation of user-designated DONTNEED buffers during eviction.
This patch implements the purge operation and state machine transitions:
Purgeable States (from
xe_madv_purgeable_state):
- WILLNEED (0): BO should be retained, actively used
- DONTNEED (1): BO eligible for purging, not currently needed
- PURGED (2): BO backing store reclaimed, permanently invalidDesign Rationale:
- Async TLB invalidation viatrigger_rebind(no blockingxe_vm_invalidate_vma)
- i915 compatibility: retained field, "once purged always purged" semantics
- Shared BO protection prevents multi-process memory corruption
- Scratch PTE reuse avoids new infrastructure, safe for fault modev2:
- Usexe_bo_trigger_rebind()for async TLB invalidation (Thomas Hellström)
- Add NULL rebind with scratch PTEs for fault mode (Thomas Hellström)
- Implement i915-compatible retained field logic (Thomas Hellström)
- Skip BO validation for purged BOs in page fault handler (crash fix)
- Add scratch VM check in page fault path (non-scratch VMs fail fault)
- Forceclear_ptfor non-scratch VMs to avoid phys addr 0 mapping (review fix)
- Add !is_purged check to resource cursor setup to prevent stale accessCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 72 ++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_gt_pagefault.c | 19 ++++++++ drivers/gpu/drm/xe/xe_pt.c | 36 ++++++++++++-- drivers/gpu/drm/xe/xe_vm.c | 11 ++++- drivers/gpu/drm/xe/xe_vm_madvise.c | 73 ++++++++++++++++++++++++++++ 5 files changed, 193 insertions(+), 18 deletions(-)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -836,6 +836,53 @@ static int xe_bo_move_notify(struct xe_bo *bo,return 0 ; }
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 843
Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +5
-0+static void xe_bo_set_purged (struct xe_bo *bo) +{ +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv); ++1 to assert, but I think
xe_bo_assert_heldcan be used.
Noted.
Quoted 15 lines
Quoted 15 lines
drivers/gpu/drm/xe/xe_bo.c +14
-0+atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_PURGED ); +} + +/** +* xe_ttm_bo_purge() - Purge buffer object backing store +* @ttm_bo: The TTM buffer object to purge +* @ctx: TTM operation context +* +* This function purges the backing store of a BO marked as DONTNEED and +* triggers rebind to invalidate stale GPU mappings. For fault-mode VMs, +* this zaps the PTEs. The next GPU access will trigger a page fault and +* perform NULL rebind (scratch pages or clear PTEs based on VM config). +*/ +static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx)
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 864
Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +7
-0+{ +struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); +struct xe_bo *bo =ttm_to_xe_bo (ttm_bo); + +if (ttm_bo->ttm) { +struct ttm_placement place = {}; +int ret =ttm_bo_validate (ttm_bo, &place, ctx);Do we need to the eviction to complete? e.g., wait on
DMA_RESV_USAGE_KERNELslots?
xe_ttm_bo_purge is existing code, not written by me. I just relocated the function to this section.
Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 866
Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +2
-0+ +drm_WARN_ON (&xe->drm, ret);Likely an specific warn on function or assert.
same, xe_ttm_bo_purge is existing code, not written by me. I just relocated the function to this section.
Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 867
Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0+if (!ret && bo) {When would BO be NULL? I don't think ever.
Noted.
Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 868
Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0+if (atomic_read (&bo->madv_purgeable) ==XE_MADV_PURGEABLE_DONTNEED ) {Can we helpers for
madv_purgeableaccess /w an assert?
Noted.
Quoted 14 lines
drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move_notify(struct xe_bo *bo, · line 880
Quoted 13 lines
drivers/gpu/drm/xe/xe_bo.c +12
-0+xe_bo_set_purged (bo); + +/* +* Trigger rebind to invalidate stale GPU mappings. +* - Non-fault mode: Marks VMAs for rebind +* - Fault mode: Zaps PTEs (sets to 0), next access triggers fault +* and NULL rebind with scratch/clear PTEs per VM config +*/ + ret =xe_bo_trigger_rebind (xe, bo, ctx); +if (ret) +drm_warn (&xe->drm, +"Failed to invalidate purged BO: %d\n" , ret);Xe specific warn on or assert?
Noted.
Quoted 3 lines
Also maybe this function just returns an error if something fails too and
xe_bo_movewould return that failure.
same, xe_ttm_bo_purge is existing code, not written by me. I just relocated the function to this section
Quoted 65 lines
Quoted 65 lines
drivers/gpu/drm/xe/xe_bo.c +18
-12+ } + } + } +} + static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_resource *new_mem,@@ -853,8 +900,18 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,bool needs_clear;bool handle_system_ccs = (!IS_DGFX (xe) &&xe_bo_needs_ccs_pages (bo) && ttm &&ttm_tt_is_populated (ttm)) ? true : false; +int state =atomic_read (&bo->madv_purgeable);int ret =0 ; +/* +* Purge only non-shared BOs explicitly marked DONTNEED by userspace. +* The move_notify callback will handle invalidation asynchronously. +*/ +if (evict && state ==XE_MADV_PURGEABLE_DONTNEED && !xe_bo_is_shared_locked (bo)) { +xe_ttm_bo_purge (ttm_bo, ctx); +return 0 ; + } +/* Bo creation path, moving to system or TT. */ if ((!old_mem && ttm) && !handle_system_ccs) {if (new_mem->mem_type ==XE_PL_TT )@@ -1606,18 +1663,6 @@ static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo) } } -static void xe_ttm_bo_purge (struct ttm_buffer_object *ttm_bo,struct ttm_operation_ctx *ctx) -{ -struct xe_device *xe =ttm_to_xe_device (ttm_bo->bdev); - -if (ttm_bo->ttm) { -struct ttm_placement place = {}; -int ret =ttm_bo_validate (ttm_bo, &place, ctx); - -drm_WARN_ON (&xe->drm, ret); - } -} -static void xe_ttm_bo_swap_notify (struct ttm_buffer_object *ttm_bo) {struct ttm_operation_ctx ctx = {@@ -2202,6 +2247,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,#endif INIT_LIST_HEAD (&bo->vram_userfault_link); +/* Initialize purge advisory state */ +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED ); +drm_gem_private_object_init (&xe->drm, &bo->ttm.base, size);if (resv) {drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -87,6 +87,13 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,if (!bo)return 0 ;
Quoted 9 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma, · line 96
Quoted 7 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +7
-0+/* +* Skip validation/migration for purged BOs - they have no backing pages. +* Rebind will use scratch PTEs instead. +*/ +if (xe_bo_is_purged (bo)) +return 0 ; +This needs a rebase as
xe_gt_pagefault.cis gone upstream and replaced withxe_pagefault.c.
Noted.
Quoted 12 lines
Quoted 12 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-0return need_vram_move ?xe_bo_migrate (bo, vram->placement,NULL , exec) :xe_bo_validate (bo, vm, true, exec); }@@ -100,9 +107,21 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,struct drm_exec exec;struct dma_fence *fence;int err, needs_vram; +struct xe_bo *bo;lockdep_assert_held_write (&vm->lock);
Quoted 12 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma, · line 123
Quoted 10 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +10
-0+/* +* Check if BO is purged. For purged BOs: +* - Scratch VMs: Allow rebind with scratch PTEs (safe zero reads) +* - Non-scratch VMs: FAIL the page fault (no scratch page available) +*/ + bo =xe_vma_bo (vma); +if (bo &&xe_bo_is_purged (bo)) { +if (!xe_vm_has_scratch (vm)) +return -EACCES ; + }As discussed in other patches, move the
xe_bo_is_purgedcheck under the dma-resv lock.
Noted.
Quoted 19 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-0+ needs_vram =xe_vma_need_vram_for_atomic (vm->xe, vma, atomic);if (needs_vram <0 || (needs_vram &&xe_vma_is_userptr (vma)))return needs_vram <0 ? needs_vram : -EACCES ;drivers/gpu/drm/xe/xe_pt.c +0
-0drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, · line 538
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +2
-1@@ -533,20 +533,26 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,/* Is this a leaf entry ?*/ if (level ==0 ||xe_pt_hugepte_possible (addr, next, level, xe_walk)) {struct xe_res_cursor *curs = xe_walk->curs; +struct xe_bo *bo =xe_vma_bo (xe_walk->vma);bool is_null =xe_vma_is_null (xe_walk->vma); -bool is_vram = is_null ? false :xe_res_is_vram (curs); +bool is_purged = bo &&xe_bo_is_purged (bo);Can we drop
is_purgedand just setis_nullto true for purged BOs?
Quoted 1 line
Or rename s/is_null/is_null_or_purged for clarity?
Agreed, I will do the changes as per suggestion.
Quoted 49 lines
Quoted 49 lines
drivers/gpu/drm/xe/xe_pt.c +21
-3+bool is_vram = (is_null || is_purged) ? false :xe_res_is_vram (curs);XE_WARN_ON (xe_walk->va_curs_start != addr);if (xe_walk->clear_pt) { pte =0 ; }else { - pte = vm->pt_ops->pte_encode_vma (is_null ?0 : +/* +* For purged BOs, treat like null VMAs - pass address 0. +* The pte_encode_vma will set XE_PTE_NULL flag for scratch mapping. +*/ + pte = vm->pt_ops->pte_encode_vma ((is_null || is_purged) ?0 :xe_res_dma (curs) + xe_walk->dma_offset, xe_walk->vma, pat_index, level); -if (!is_null) +if (!is_null && !is_purged) pte |= is_vram ? xe_walk->default_vram_pte : xe_walk->default_system_pte;@@ -570,7 +576,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset,if (unlikely (ret))return ret; -if (!is_null && !xe_walk->clear_pt) +if (!is_null && !is_purged && !xe_walk->clear_pt)xe_res_next (curs, next - addr); xe_walk->va_curs_start = next; xe_walk->vma->gpuva.flags |= (XE_VMA_PTE_4K << level);@@ -723,6 +729,26 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, };struct xe_pt *pt = vm->pt_root[tile->id];int ret; +bool is_purged = false; + +/* +* Check if BO is purged: +* - Scratch VMs: Use scratch PTEs (XE_PTE_NULL) for safe zero reads +* - Non-scratch VMs: Clear PTEs to zero (non-present) to avoid mapping to phys addr 0 +* +* For non-scratch VMs, we force clear_pt=true so leaf PTEs become completely +* zero instead of creating a PRESENT mapping to physical address 0. +*/ +if (bo &&xe_bo_is_purged (bo)) { + is_purged = true; +
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, · line 750
Quoted 6 lines
drivers/gpu/drm/xe/xe_pt.c +6
-0+/* +* For non-scratch VMs, a NULL rebind should use zero PTEs +* (non-present), not a present PTE to phys 0. +*/ +if (!xe_vm_has_scratch (vm)) + xe_walk.clear_pt = true;So the idea is purged BOs will fault if the VMA is accessed on non-scratch VMs?
Yes, exactly. The behavior differs based on VM type:
- Scratch VMs: Purged BOs use scratch PTEs (XE_PTE_NULL flag set by pte_encode_vma when addr=0). GPU reads return zeros, no fault.
- Non-scratch VMs: Purged BOs get completely zero PTEs (non-present). Any GPU access will fault, which is the expected behavior for non-scratch VMs - they don't have scratch pages to fall back to.
Quoted 66 lines
Quoted 66 lines
drivers/gpu/drm/xe/xe_pt.c +2
-1+ }if (range) {/* Move this entire thing to xe_svm.c? */ @@ -762,7 +788,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (!range)xe_bo_assert_held (bo); -if (!xe_vma_is_null (vma) && !range) { +if (!xe_vma_is_null (vma) && !range && !is_purged) {if (xe_vma_is_userptr (vma))xe_res_first_dma (to_userptr_vma (vma)->userptr.pages.dma_addr,0 ,xe_vma_size (vma), &curs);drivers/gpu/drm/xe/xe_vm.c +10
-1@@ -1336,6 +1336,9 @@ static u64 xelp_pte_encode_bo(struct xe_bo *bo, u64 bo_offset,static u64 xelp_pte_encode_vma (u64 pte,struct xe_vma *vma,u16 pat_index,u32 pt_level) { +struct xe_bo *bo =xe_vma_bo (vma); +struct xe_vm *vm =xe_vma_vm (vma); + pte |=XE_PAGE_PRESENT ;if (likely (!xe_vma_read_only (vma)))@@ -1344,7 +1347,13 @@ static u64 xelp_pte_encode_vma(u64 pte, struct xe_vma *vma, pte |=pte_encode_pat_index (pat_index, pt_level); pte |=pte_encode_ps (pt_level); -if (unlikely (xe_vma_is_null (vma))) +/* +* NULL PTEs redirect to scratch page (return zeros on read). +* Set for: 1) explicit null VMAs, 2) purged BOs on scratch VMs. +* Never set NULL flag without scratch page - causes undefined behavior. +*/ +if (unlikely (xe_vma_is_null (vma) || + (bo &&xe_bo_is_purged (bo) &&xe_vm_has_scratch (vm)))) pte |=XE_PTE_NULL ;return pte;drivers/gpu/drm/xe/xe_vm_madvise.c +15
-0@@ -158,6 +158,60 @@ static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, } } +/* +* Handle purgeable buffer object advice for DONTNEED/WILLNEED/PURGED. +* Updates op->purge_state_val.retained to indicate if backing store +* exists (matches i915's retained). +*/ +static void xe_vm_madvise_purgeable_bo (struct xe_device *xe,struct xe_vm *vm, +struct xe_vma **vmas,int num_vmas, +struct drm_xe_madvise *op) +{ +bool has_purged_bo = false; +int i; + +xe_assert (vm->xe, op->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ); + +for (i =0 ; i < num_vmas; i++) {
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm_madvise.c static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, · line 182
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +7
-0+struct xe_bo *bo =xe_vma_bo (vmas[i]); + +if (!bo) +continue ; + +/* BO must be locked before modifying madv state */ +dma_resv_assert_held (bo->ttm.base.resv);
xe_bo_assert_held
Noted.
Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +13
-0+ +/* +* Once purged, always purged. Cannot transition back to WILLNEED. +* This matches i915 semantics where purged BOs are permanently invalid. +*/ +if (xe_bo_is_purged (bo)) { + has_purged_bo = true; +continue ; + } + +switch (op->purge_state_val.val) { +case DRM_XE_VMA_PURGEABLE_STATE_WILLNEED : +atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_WILLNEED );
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm_madvise.c static void madvise_pat_index(struct xe_device *xe, struct xe_vm *vm, · line 198
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +3
-0+break ; +case DRM_XE_VMA_PURGEABLE_STATE_DONTNEED : +if (!xe_bo_is_shared_locked (bo))I think we need a complete VMA check here per Thomas's feedback (e.g., all VMAs attached to the BO must be in
DRM_XE_VMA_PURGEABLE_STATE_DONTNEEDto flip the BO state). I think the BO is an import or exported dma-buf we can never flip the state as we don't know what an external device is doing with it.
I will add VMA checK as per suggestion.
Good catch on the dma-buf case.
However, you're right that we're missing the imported/exported dma-buf check. I'll add a check to reject DONTNEED for:
- Imported BOs (bo->ttm.base.import_attach != NULL)
- Exported BOs (bo->ttm.base.dma_buf != NULL && bo->ttm.base.dma_buf->file != NULL)
~Arvind
Quoted 58 lines
Matt
Quoted 55 lines
drivers/gpu/drm/xe/xe_vm_madvise.c +35
-0+atomic_set (&bo->madv_purgeable,XE_MADV_PURGEABLE_DONTNEED ); +break ; +default : +drm_warn (&vm->xe->drm,"Invalid madvice value = %d\n" , + op->purge_state_val.val); +return ; + } + } + +/* +* Set retained flag to indicate if backing store still exists. +* Matches i915: retained = 1 if not purged, 0 if purged. +*/ + op->purge_state_val.retained = !has_purged_bo; +} +typedef void (*madvise_func )(struct xe_device *xe,struct xe_vm *vm,struct xe_vma **vmas,int num_vmas,struct drm_xe_madvise *op);@@ -283,6 +337,19 @@ static bool madvise_args_are_sane(struct xe_device *xe, const struct drm_xe_madvreturn false;break ; } +case DRM_XE_VMA_ATTR_PURGEABLE_STATE : + { +u32 val = args->purge_state_val.val; + +if (XE_IOCTL_DBG (xe, !((val ==DRM_XE_VMA_PURGEABLE_STATE_WILLNEED ) || + (val ==DRM_XE_VMA_PURGEABLE_STATE_DONTNEED )))) +return false; + +if (XE_IOCTL_DBG (xe, args->purge_state_val.reserved)) +return false; + +break ; + }default :if (XE_IOCTL_DBG (xe,1 ))return false;@@ -402,6 +469,12 @@ int xe_vm_madvise_ioctl(struct drm_device *dev, void *data, struct drm_file *fil goto err_fini; } } +if (args->type ==DRM_XE_VMA_ATTR_PURGEABLE_STATE ) { +xe_vm_madvise_purgeable_bo (xe, vm, madvise_range.vmas, + madvise_range.num_vmas, args); + goto err_fini; + + } }if (madvise_range.has_svm_userptr_vmas) {Signature · 1 line
2.43.0
RFC v2 5/9 drm/xe/bo: Handle CPU faults on purged buffer objects
In reply to Arvind YadavModify the CPU page fault handler, xe_bo_cpu_fault(), to correctly handle access to buffer objects that have been purged.
When a buffer object is in the XE_MADV_PURGED state, its backing store has been reclaimed by the kernel. If the CPU attempts to access this memory, it is an error that should be reported to the application.
v2:
- Added xe_bo_is_purged(bo) instead of atomic_read.
- Avoids leaks and keeps drm_dev_exit() while returning.
--- drivers/gpu/drm/xe/xe_bo.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +10 -0
@@ -1992,6 +1992,16 @@ static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf)if (!drm_dev_enter (&xe->drm, &idx))return ttm_bo_vm_dummy_page (vmf, vmf->vma->vm_page_prot); +/* +* BO content is gone. Signal the user process. +* Once purged, BO remains permanently invalid (i915 semantics). +* Application must destroy and recreate the BO. +*/ +if (xe_bo_is_purged (bo)) { + ret =VM_FAULT_SIGBUS ; + goto out; + } + ret =xe_bo_cpu_fault_fastpath (vmf, xe, bo, needs_rpm);if (ret !=VM_FAULT_RETRY ) goto out;
Signature · 1 line
2.43.0
Re: [RFC v2 5/9] drm/xe/bo: Handle CPU faults on purged buffer objects
Quoted 7 lines · Arvind Yadav
Modify the CPU page fault handler,
xe_bo_cpu_fault(), to correctly handle access to buffer objects that have been purged.When a buffer object is in the
XE_MADV_PURGEDstate, its backing store has been reclaimed by the kernel. If the CPU attempts to access this memory, it is an error that should be reported to the application.
Quoted 5 lines
v2:
- Addedxe_bo_is_purged(bo)instead ofatomic_read.
- Avoids leaks and keepsdrm_dev_exit()while returning.Cc Matthew Brost <[email protected]>
Quoted 31 lines
Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/gpu/drm/xe/xe_bo.c +10
-0@@ -1992,6 +1992,16 @@ static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf)if (!drm_dev_enter (&xe->drm, &idx))return ttm_bo_vm_dummy_page (vmf, vmf->vma->vm_page_prot); +/* +* BO content is gone. Signal the user process. +* Once purged, BO remains permanently invalid (i915 semantics). +* Application must destroy and recreate the BO. +*/ +if (xe_bo_is_purged (bo)) { + ret =VM_FAULT_SIGBUS ; + goto out; + } + ret =xe_bo_cpu_fault_fastpath (vmf, xe, bo, needs_rpm);if (ret !=VM_FAULT_RETRY ) goto out;Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 5/9] drm/xe/bo: Handle CPU faults on purged buffer objects
Quoted 8 lines · Matthew Brost
Quoted 7 lines · Arvind Yadav
Modify the CPU page fault handler,
xe_bo_cpu_fault(), to correctly handle access to buffer objects that have been purged.When a buffer object is in the
XE_MADV_PURGEDstate, its backing store has been reclaimed by the kernel. If the CPU attempts to access this memory, it is an error that should be reported to the application.
Quoted 8 lines
Quoted 5 lines
v2:
- Addedxe_bo_is_purged(bo)instead ofatomic_read.
- Avoids leaks and keepsdrm_dev_exit()while returning.Cc Matthew Brost <[email protected]>Reviewed-by Matthew Brost <[email protected]>
Ah, actually I think I made a mistake here.
Quoted 15 lines
Quoted 15 lines
Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -1992,6 +1992,16 @@ static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf)if (!drm_dev_enter (&xe->drm, &idx))return ttm_bo_vm_dummy_page (vmf, vmf->vma->vm_page_prot);
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf) · line 2000
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +6
-0+/* +* BO content is gone. Signal the user process. +* Once purged, BO remains permanently invalid (i915 semantics). +* Application must destroy and recreate the BO. +*/ +if (xe_bo_is_purged (bo)) {
Doesn't this need to done under the BO's dma-resv lock to avoid a race? Consider the case where after this check, TTM evicts this BO changing the state purged. Now we grab the BO's dma-resv lock and try to get pages on purged BO. Seems like an issue.
Also with that, xe_bo_is_purged likely should have lockdep annotation asserting the BOs dma-resv lock is held.
Matt
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +4
-0+ ret =VM_FAULT_SIGBUS ; + goto out; + } + ret =xe_bo_cpu_fault_fastpath (vmf, xe, bo, needs_rpm);if (ret !=VM_FAULT_RETRY ) goto out;Signature · 1 line
2.43.0
Re: [RFC v2 5/9] drm/xe/bo: Handle CPU faults on purged buffer objects
Quoted 33 lines · Matthew Brost
Quoted 8 lines · Matthew Brost
Quoted 7 lines · Arvind Yadav
Modify the CPU page fault handler,
xe_bo_cpu_fault(), to correctly handle access to buffer objects that have been purged.When a buffer object is in the
XE_MADV_PURGEDstate, its backing store has been reclaimed by the kernel. If the CPU attempts to access this memory, it is an error that should be reported to the application.Quoted 7 lines
Quoted 5 lines
v2:
- Addedxe_bo_is_purged(bo)instead ofatomic_read.
- Avoids leaks and keepsdrm_dev_exit()while returning.Cc Matthew Brost <[email protected]>Reviewed-by Matthew Brost <[email protected]>Ah, actually I think I made a mistake here.
Quoted 15 lines
Quoted 15 lines
Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -1992,6 +1992,16 @@ static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf)if (!drm_dev_enter (&xe->drm, &idx))return ttm_bo_vm_dummy_page (vmf, vmf->vma->vm_page_prot);
Quoted 10 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf) · line 2000
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +6
-0+/* +* BO content is gone. Signal the user process. +* Once purged, BO remains permanently invalid (i915 semantics). +* Application must destroy and recreate the BO. +*/ +if (xe_bo_is_purged (bo)) {Doesn't this need to done under the BO's dma-resv lock to avoid a race? Consider the case where after this check, TTM evicts this BO changing the state purged. Now we grab the BO's dma-resv lock and try to get pages on purged BO. Seems like an issue.
Thanks for catching these issues!.
Quoted 3 lines
Also with that,
xe_bo_is_purgedlikely should have lockdep annotation asserting the BOs dma-resv lock is held.
I initially added xe_bo_assert_held() to xe_bo_is_purged(), but it causes crashes because many callers don't hold the lock. For example, in xe_pagefault.c (this early check), no lock is held. I’ll recheck the call and update accordingly.
~Arvind
Quoted 13 lines
Matt
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +4
-0+ ret =VM_FAULT_SIGBUS ; + goto out; + } + ret =xe_bo_cpu_fault_fastpath (vmf, xe, bo, needs_rpm);if (ret !=VM_FAULT_RETRY ) goto out;Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 5/9] drm/xe/bo: Handle CPU faults on purged buffer objects
Quoted 48 lines · Arvind
Quoted 34 lines · Matthew Brost
Quoted 8 lines · Matthew Brost
Quoted 7 lines · Arvind Yadav
Modify the CPU page fault handler,
xe_bo_cpu_fault(), to correctly handle access to buffer objects that have been purged.When a buffer object is in the
XE_MADV_PURGEDstate, its backing store has been reclaimed by the kernel. If the CPU attempts to access this memory, it is an error that should be reported to the application.Quoted 7 lines
Quoted 5 lines
v2:
- Addedxe_bo_is_purged(bo)instead ofatomic_read.
- Avoids leaks and keepsdrm_dev_exit()while returning.Cc Matthew Brost <[email protected]>Reviewed-by Matthew Brost <[email protected]>Ah, actually I think I made a mistake here.
Quoted 16 lines
Quoted 16 lines
Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 10 ++++++++++ 1 file changed, 10 insertions(+)drivers/gpu/drm/xe/xe_bo.c +2
-0@@ -1992,6 +1992,16 @@ static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf)if (!drm_dev_enter (&xe->drm, &idx))return ttm_bo_vm_dummy_page (vmf, vmf->vma->vm_page_prot); +/* +* BO content is gone. Signal the user process. Quoted 8 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c static vm_fault_t xe_bo_cpu_fault(struct vm_fault *vmf) · line 2000
Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +4
-0+* Once purged, BO remains permanently invalid (i915 semantics). +* Application must destroy and recreate the BO. +*/ +if (xe_bo_is_purged (bo)) {Doesn't this need to done under the BO's dma-resv lock to avoid a race? Consider the case where after this check, TTM evicts this BO changing the state purged. Now we grab the BO's dma-resv lock and try to get pages on purged BO. Seems like an issue.
Thanks for catching these issues!.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Also with that,
xe_bo_is_purgedlikely should have lockdep annotation asserting the BOs dma-resv lock is held.I initially added
xe_bo_assert_held()toxe_bo_is_purged(), but it causes crashes because many callers don't hold the lock.
For example, inxe_pagefault.c(this early check), no lock is held. I’ll recheck the call and update accordingly.
I've touched on this in other patches - sorry my reviews sometimes come in scattered bursts as I look at code - but I think the point is all purging state changes / critical checks should be done under the BO dma-resv lock to avoid races. Sure user space shouldn't touching a BO in WONTNEED state but it could and if purging races with a check outside a lock it seems like bad things could happen in the kernel.
Matt
Quoted 14 lines
~Arvind
Quoted 13 lines
Matt
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +4
-0+ ret =VM_FAULT_SIGBUS ; + goto out; + } + ret =xe_bo_cpu_fault_fastpath (vmf, xe, bo, needs_rpm);if (ret !=VM_FAULT_RETRY ) goto out;Signature · 1 line
2.43.0
RFC v2 6/9 drm/xe/bo: Prevent mmap of purged buffer objects
In reply to Arvind YadavFail DRM_IOCTL_XE_GEM_MMAP_OFFSET with -EINVAL when called on purged buffer objects to provide early error detection instead of allowing deferred SIGBUS on memory access.
Problem:
The mmap offset ioctl (DRM_IOCTL_XE_GEM_MMAP_OFFSET) returns a file offset that userspace can pass to mmap() to map GPU memory into its address space. For purged BOs, the backing store has been freed, but the VMA node offset remains valid. Without this check:
1. Userspace successfully gets mmap offset for purged BO
2. mmap() succeeds (VMA is created but has no backing pages)
3. Any memory access triggers CPU page fault
4. xe_bo_cpu_fault() detects purged state and returns VM_FAULT_SIGBUS
v2:
- Fix reference counting: use drm_gem_object_put() instead of xe_bo_put() to properly balance drm_gem_object_lookup() (review feedback).
- Added xe_bo_is_purged(bo) instead of atomic_read.
--- drivers/gpu/drm/xe/xe_bo.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +11 -0
@@ -3346,6 +3346,7 @@ int xe_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,struct xe_device *xe =to_xe_device (dev);struct drm_xe_gem_mmap_offset *args = data;struct drm_gem_object *gem_obj; +struct xe_bo *bo;if (XE_IOCTL_DBG (xe, args->extensions) ||XE_IOCTL_DBG (xe, args->reserved[0 ] || args->reserved[1 ]))
@@ -3375,6 +3376,16 @@ int xe_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,if (XE_IOCTL_DBG (xe, !gem_obj))return -ENOENT ; + bo =gem_to_xe_bo (gem_obj); + +/* +* Reject mmap offset requests for purged BOs. +*/ +if (xe_bo_is_purged (bo)) { +drm_gem_object_put (gem_obj); +return -EINVAL ; + } +/* The mmap offset was set up at BO allocation time. */ args->offset =drm_vma_node_offset_addr (&gem_obj->vma_node);
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 6/9] drm/xe/bo: Prevent mmap of purged buffer objects
Quoted 41 lines · Arvind Yadav
Fail
DRM_IOCTL_XE_GEM_MMAP_OFFSETwith-EINVALwhen called on purged buffer objects to provide early error detection instead of allowing deferred SIGBUS on memory access.Problem:
The mmap offset ioctl (DRM_IOCTL_XE_GEM_MMAP_OFFSET) returns a file offset that userspace can pass tommap()to map GPU memory into its address space. For purged BOs, the backing store has been freed, but the VMA node offset remains valid. Without this check:1. Userspace successfully gets mmap offset for purged BO
2.mmap()succeeds (VMA is created but has no backing pages)
3. Any memory access triggers CPU page fault
4.xe_bo_cpu_fault()detects purged state and returnsVM_FAULT_SIGBUSv2:
- Fix reference counting: usedrm_gem_object_put()instead ofxe_bo_put()to properly balancedrm_gem_object_lookup()(review feedback).
- Addedxe_bo_is_purged(bo)instead ofatomic_read.Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 11 +++++++++++ 1 file changed, 11 insertions(+)drivers/gpu/drm/xe/xe_bo.c +1
-0@@ -3346,6 +3346,7 @@ int xe_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,struct xe_device *xe =to_xe_device (dev);struct drm_xe_gem_mmap_offset *args = data;struct drm_gem_object *gem_obj; +struct xe_bo *bo;if (XE_IOCTL_DBG (xe, args->extensions) ||XE_IOCTL_DBG (xe, args->reserved[0 ] || args->reserved[1 ]))
drivers/gpu/drm/xe/xe_bo.c int xe_gem_mmap_offset_ioctl(struct drm_device *dev, void *data, · line 3384
Quoted 9 lines
drivers/gpu/drm/xe/xe_bo.c +6
-0@@ -3375,6 +3376,16 @@ int xe_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,if (XE_IOCTL_DBG (xe, !gem_obj))return -ENOENT ; + bo =gem_to_xe_bo (gem_obj); + +/* +* Reject mmap offset requests for purged BOs. +*/ +if (xe_bo_is_purged (bo)) {
I don't think this is needed. A subsequent CPU page fault would just fail with SIGBUS. As discussed in patch 2/5 we'd have the BO's dma-resv lock there too making this check race free.
Matt
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +4
-0+drm_gem_object_put (gem_obj); +return -EINVAL ; + } +/* The mmap offset was set up at BO allocation time. */ args->offset =drm_vma_node_offset_addr (&gem_obj->vma_node);Signature · 1 line
2.43.0
RFC v2 7/9 drm/xe/vm: Prevent binding of purged buffer objects
In reply to Arvind YadavAdd validation in xe_vm_bind_ioctl_validate_bo() to reject MAP and PREFETCH operations on purged buffer objects with -EINVAL.
Problem:
When a BO is purged (XE_MADV_PURGEABLE_PURGED state), its backing pages have been freed by the kernel. Without this check, VM_BIND operations would proceed:
1. DRM_XE_VM_BIND_OP_MAP: Attempts to create GPU mappings to freed memory
- xe_vma_ops_alloc() creates VMA pointing to invalid BO
- Page tables populated with stale/invalid addresses
- GPU access leads to undefined behavior or hangs
2. DRM_XE_VM_BIND_OP_PREFETCH: Attempts to migrate non-existent pages
- Triggers BO validation in TTM
- ttm_bo_validate() fails or crashes (no backing store)
- Wasted work for permanently invalid BO
With this check:
- MAP/PREFETCH immediately fail with -EINVAL at ioctl boundary
- Clear error message at syscall (better UX than deferred GPU hang)
- Prevents creation of invalid GPU page table entries
v2:
- Clarify that purged BOs are permanently invalid (i915 semantics)
- Remove incorrect claim about madvise(WILLNEED) restoring purged BOs
--- drivers/gpu/drm/xe/xe_vm.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +7 -0
@@ -3482,6 +3482,13 @@ static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo,return -EINVAL ; } +/* Purged BOs are permanently invalid; reject new MAP/PREFETCH. */ +if (XE_IOCTL_DBG (xe, +xe_bo_is_purged (bo) && + (op ==DRM_XE_VM_BIND_OP_MAP || + op ==DRM_XE_VM_BIND_OP_PREFETCH ))) +return -EINVAL ; +/* * Some platforms require 64k VM_BIND alignment, * specifically those with XE_VRAM_FLAGS_NEED64K.
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 7/9] drm/xe/vm: Prevent binding of purged buffer objects
Quoted 40 lines · Arvind Yadav
Add validation in
xe_vm_bind_ioctl_validate_bo()to reject MAP and PREFETCH operations on purged buffer objects with-EINVAL.Problem:
When a BO is purged (XE_MADV_PURGEABLE_PURGEDstate), its backing pages have been freed by the kernel. Without this check,VM_BINDoperations would proceed:1.
DRM_XE_VM_BIND_OP_MAP: Attempts to create GPU mappings to freed memory
-xe_vma_ops_alloc()creates VMA pointing to invalid BO
- Page tables populated with stale/invalid addresses
- GPU access leads to undefined behavior or hangs2.
DRM_XE_VM_BIND_OP_PREFETCH: Attempts to migrate non-existent pages
- Triggers BO validation in TTM
-ttm_bo_validate()fails or crashes (no backing store)
- Wasted work for permanently invalid BOWith this check:
- MAP/PREFETCH immediately fail with-EINVALat ioctl boundary
- Clear error message at syscall (better UX than deferred GPU hang)
- Prevents creation of invalid GPU page table entriesv2:
- Clarify that purged BOs are permanently invalid (i915 semantics)
- Remove incorrect claim about madvise(WILLNEED) restoring purged BOsCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_vm.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/gpu/drm/xe/xe_vm.c +0
-0
drivers/gpu/drm/xe/xe_vm.c static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo, · line 3490
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +6
-0@@ -3482,6 +3482,13 @@ static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo,return -EINVAL ; } +/* Purged BOs are permanently invalid; reject new MAP/PREFETCH. */ +if (XE_IOCTL_DBG (xe, +xe_bo_is_purged (bo) && + (op ==DRM_XE_VM_BIND_OP_MAP || + op ==DRM_XE_VM_BIND_OP_PREFETCH ))) +return -EINVAL ;
I'd move this check to later in the bind pipeline once we have BO's dma-resv lock to make this race free. I think vma_lock_and_validate is likely the correct function after drm_exec_lock_obj but before xe_bo_validate. Or you could just make xe_bo_validate fail if the object is purged.
Matt
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0+/* * Some platforms require 64k VM_BIND alignment, * specifically those with XE_VRAM_FLAGS_NEED64K. Signature · 1 line
2.43.0
Re: [RFC v2 7/9] drm/xe/vm: Prevent binding of purged buffer objects
Quoted 44 lines · Matthew Brost
Quoted 43 lines · Arvind Yadav
Add validation in
xe_vm_bind_ioctl_validate_bo()to reject MAP and PREFETCH operations on purged buffer objects with-EINVAL.Problem:
When a BO is purged (XE_MADV_PURGEABLE_PURGEDstate), its backing pages have been freed by the kernel. Without this check,VM_BINDoperations would proceed:1.
DRM_XE_VM_BIND_OP_MAP: Attempts to create GPU mappings to freed memory
-xe_vma_ops_alloc()creates VMA pointing to invalid BO
- Page tables populated with stale/invalid addresses
- GPU access leads to undefined behavior or hangs2.
DRM_XE_VM_BIND_OP_PREFETCH: Attempts to migrate non-existent pages
- Triggers BO validation in TTM
-ttm_bo_validate()fails or crashes (no backing store)
- Wasted work for permanently invalid BOWith this check:
- MAP/PREFETCH immediately fail with-EINVALat ioctl boundary
- Clear error message at syscall (better UX than deferred GPU hang)
- Prevents creation of invalid GPU page table entriesv2:
- Clarify that purged BOs are permanently invalid (i915 semantics)
- Remove incorrect claim about madvise(WILLNEED) restoring purged BOsCc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- drivers/gpu/drm/xe/xe_vm.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -3482,6 +3482,13 @@ static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo,return -EINVAL ; }
Quoted 11 lines
drivers/gpu/drm/xe/xe_vm.c static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo, · line 3490
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +6
-0+/* Purged BOs are permanently invalid; reject new MAP/PREFETCH. */ +if (XE_IOCTL_DBG (xe, +xe_bo_is_purged (bo) && + (op ==DRM_XE_VM_BIND_OP_MAP || + op ==DRM_XE_VM_BIND_OP_PREFETCH ))) +return -EINVAL ;I'd move this check to later in the bind pipeline once we have BO's dma-resv lock to make this race free. I think
vma_lock_and_validateis likely the correct function afterdrm_exec_lock_objbut beforexe_bo_validate. Or you could just makexe_bo_validatefail if the object is purged.
Noted. I will move this check under vm_lock_and_validate.
Moving the purge check to xe_bo_validate() would break GPU page fault recovery for purged BO-backed VMAs (preventing scratch PTE rebinds).
~Arvind
Quoted 10 lines
Matt
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0+/* * Some platforms require 64k VM_BIND alignment, * specifically those with XE_VRAM_FLAGS_NEED64K. Signature · 1 line
2.43.0
RFC v2 8/9 drm/xe/uapi: Add UAPI for purgeable bo state to madvise query response
In reply to Arvind YadavFrom: Himal Prasad Ghimiray <[email protected]>
Complete the purgeable buffer object UAPI by adding the response structure to drm_xe_mem_range_attr for querying current purgeable state of buffer objects within a memory range.
This allows userspace to determine the current state of BOs:
- DRM_XE_VMA_PURGEABLE_STATE_WILLNEED (0): BO actively needed, has backing store
- DRM_XE_VMA_PURGEABLE_STATE_DONTNEED (1): BO eligible for purging, still has backing
- DRM_XE_VMA_PURGEABLE_STATE_PURGED (2): BO purged, backing store freed (read-only)
--- include/uapi/drm/xe_drm.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
include/uapi/drm/xe_drm.h +13 -0
@@ -2147,10 +2147,14 @@ struct drm_xe_madvise {* - DRM_XE_VMA_PURGEABLE_STATE_DONTNEED (1): Hints that BO is not * currently needed. Kernel may purge it under memory pressure. * Only applies to non-shared BOs. Returns retained=1 if not purged. +* +* - DRM_XE_VMA_PURGEABLE_STATE_PURGED: Read-only state indicating +* the BO purge state. */ struct {#define DRM_XE_VMA_PURGEABLE_STATE_WILLNEED 0#define DRM_XE_VMA_PURGEABLE_STATE_DONTNEED 1 +#define DRM_XE_VMA_PURGEABLE_STATE_PURGED 2/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ __u32 val;/**
@@ -2224,6 +2228,15 @@ struct drm_xe_mem_range_attr {__u32 reserved; } pat_index; +/** @purge_state_val: Purgeable state configuration */ +struct { +/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ +__u32 val; + +/** @purge_state_val.reserved: Reserved */ +__u32 reserved; + } purge_state_val; +/** @reserved: Reserved */ __u64 reserved[2 ]; };
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [RFC v2 8/9] drm/xe/uapi: Add UAPI for purgeable bo state to madvise query response
Quoted 24 lines · Arvind Yadav
From: Himal Prasad Ghimiray <[email protected]>
Complete the purgeable buffer object UAPI by adding the response structure to
drm_xe_mem_range_attrfor querying current purgeable state of buffer objects within a memory range.This allows userspace to determine the current state of BOs:
-DRM_XE_VMA_PURGEABLE_STATE_WILLNEED(0): BO actively needed, has backing store
-DRM_XE_VMA_PURGEABLE_STATE_DONTNEED(1): BO eligible for purging, still has backing
-DRM_XE_VMA_PURGEABLE_STATE_PURGED(2): BO purged, backing store freed (read-only)Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Signed-off-by Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- include/uapi/drm/xe_drm.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/uapi/drm/xe_drm.h +0
-0
include/uapi/drm/xe_drm.h struct drm_xe_madvise { · line 2152
Quoted 6 lines
include/uapi/drm/xe_drm.h +3
-0@@ -2147,10 +2147,14 @@ struct drm_xe_madvise {* - DRM_XE_VMA_PURGEABLE_STATE_DONTNEED (1): Hints that BO is not * currently needed. Kernel may purge it under memory pressure. * Only applies to non-shared BOs. Returns retained=1 if not purged. +* +* - DRM_XE_VMA_PURGEABLE_STATE_PURGED: Read-only state indicating +* the BO purge state.
This is tricky one. Since the value of purgable state can immediately change after getting populated, is there any value in reporting this to user space? My guess is no. So with that, I'd probably remove this uAPI.
Matt
Quoted 27 lines
include/uapi/drm/xe_drm.h +10
-0*/ struct {#define DRM_XE_VMA_PURGEABLE_STATE_WILLNEED 0#define DRM_XE_VMA_PURGEABLE_STATE_DONTNEED 1 +#define DRM_XE_VMA_PURGEABLE_STATE_PURGED 2/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ __u32 val;/** @@ -2224,6 +2228,15 @@ struct drm_xe_mem_range_attr {__u32 reserved; } pat_index; +/** @purge_state_val: Purgeable state configuration */ +struct { +/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ +__u32 val; + +/** @purge_state_val.reserved: Reserved */ +__u32 reserved; + } purge_state_val; +/** @reserved: Reserved */ __u64 reserved[2 ]; };Signature · 1 line
2.43.0
Re: [RFC v2 8/9] drm/xe/uapi: Add UAPI for purgeable bo state to madvise query response
Quoted 20 lines · Matthew Brost
Quoted 19 lines · Arvind Yadav
From: Himal Prasad Ghimiray <[email protected]>
Complete the purgeable buffer object UAPI by adding the response structure to
drm_xe_mem_range_attrfor querying current purgeable state of buffer objects within a memory range.This allows userspace to determine the current state of BOs:
-DRM_XE_VMA_PURGEABLE_STATE_WILLNEED(0): BO actively needed, has backing store
-DRM_XE_VMA_PURGEABLE_STATE_DONTNEED(1): BO eligible for purging, still has backing
-DRM_XE_VMA_PURGEABLE_STATE_PURGED(2): BO purged, backing store freed (read-only)Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Signed-off-by Himal Prasad Ghimiray <[email protected]>Signed-off-by Arvind Yadav <[email protected]>--- include/uapi/drm/xe_drm.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
Quoted 14 lines
include/uapi/drm/xe_drm.h struct drm_xe_madvise { · line 2152
Quoted 11 lines
include/uapi/drm/xe_drm.h +3
-0@@ -2147,10 +2147,14 @@ struct drm_xe_madvise {* - DRM_XE_VMA_PURGEABLE_STATE_DONTNEED (1): Hints that BO is not * currently needed. Kernel may purge it under memory pressure. * Only applies to non-shared BOs. Returns retained=1 if not purged. +* +* - DRM_XE_VMA_PURGEABLE_STATE_PURGED: Read-only state indicating +* the BO purge state. This is tricky one. Since the value of purgable state can immediately change after getting populated, is there any value in reporting this to user space? My guess is no. So with that, I'd probably remove this uAPI.
Noted. I will drop this patch along with next following patch both.
~Arvind
Quoted 30 lines
Matt
Quoted 27 lines
include/uapi/drm/xe_drm.h +10
-0*/ struct {#define DRM_XE_VMA_PURGEABLE_STATE_WILLNEED 0#define DRM_XE_VMA_PURGEABLE_STATE_DONTNEED 1 +#define DRM_XE_VMA_PURGEABLE_STATE_PURGED 2/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ __u32 val;/** @@ -2224,6 +2228,15 @@ struct drm_xe_mem_range_attr {__u32 reserved; } pat_index; +/** @purge_state_val: Purgeable state configuration */ +struct { +/** @purge_state_val.val: value for DRM_XE_VMA_ATTR_PURGEABLE_STATE */ +__u32 val; + +/** @purge_state_val.reserved: Reserved */ +__u32 reserved; + } purge_state_val; +/** @reserved: Reserved */ __u64 reserved[2 ]; };Signature · 1 line
2.43.0
RFC v2 9/9 drm/xe: Add support for querying purgeable BO states
In reply to Arvind YadavAdd support for querying purgeable buffer object states through the XE_VM_QUERY_MEM_RANGE_ATTRS ioctl. This allows userspace to determine the current purgeable state of BO:
- WILLNEED (0): BO actively needed, has backing store
- DONTNEED (1): BO eligible for purging, still valid
- PURGED (2): BO purged by kernel, backing store freed
v2:
- Initialize purge_state_val for non-BO VMAs to avoid leaking kernel data.
--- drivers/gpu/drm/xe/xe_vm.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +7 -0
@@ -2023,6 +2023,7 @@ static int get_mem_attrs(struct xe_vm *vm, u32 *num_vmas, u64 start,drm_gpuvm_for_each_va_range (gpuva, &vm->gpuvm, start, end) {struct xe_vma *vma =gpuva_to_vma (gpuva); +struct xe_bo *bo;if (i == *num_vmas)return -ENOSPC ;
@@ -2035,6 +2036,12 @@ static int get_mem_attrs(struct xe_vm *vm, u32 *num_vmas, u64 start, attrs[i].preferred_mem_loc.migration_policy = vma->attr.preferred_loc.migration_policy; + bo =xe_vma_bo (vma); +if (bo) + attrs[i].purge_state_val.val =atomic_read (&bo->madv_purgeable); +else /* Non-BO VMAs (userptr, null) have no purgeable state */ + attrs[i].purge_state_val.val =DRM_XE_VMA_PURGEABLE_STATE_WILLNEED ; + i++; }
Signature · 1 line
2.43.0
Quoted 43 lines · Arvind Yadav
This patch series introduces comprehensive support for purgeable buffer objects
in the Xe driver, enabling userspace to provide memory usage hints for better
memory management under system pressure.Overview:
Purgeable memory allows applications to mark buffer objects as "not currently
needed" (DONTNEED), making them eligible for kernel reclamation during memory
pressure. This helps prevent OOM conditions and enables more efficient GPU
memory utilization for workloads with temporary or regeneratable data (caches,
intermediate results, decoded frames, etc.).Purgeable BO Lifecycle:
1. WILLNEED (default): BO actively needed, kernel preserves backing store
2. DONTNEED (user hint): BO contents discardable, eligible for purging
3. PURGED (kernel action): Backing store reclaimed during memory pressureKey Design Principles:
- i915 compatibility: "Once purged, always purged" semantics - purged BOs
remain permanently invalid and must be destroyed/recreated
- Safety first: Only non-shared BOs can be marked DONTNEED to prevent
multi-process data corruption
- Multiple protection layers: Validation in madvise, VM bind, mmap, and
fault handlers
- Async TLB invalidation: Usesxe_bo_trigger_rebind()for non- blocking
GPU mapping invalidation
- Scratch PTE support: Fault-mode VMs use scratch pages for safe zero reads
on purged BO access
Quoted 10 lines
Error Handling:
- CPU access (mmap): ReturnsVM_FAULT_SIGBUS(SIGBUS signal to process)
- GPU access (non-scratch VM): Page fault fails with-EACCES, GPU context reset
- GPU access (scratch VM): Page fault succeeds, rebinds with scratch PTEs
-VM_BINDoperations: MAP/PREFETCH rejected with-EINVAL
- Mmap offset ioctl: Rejected with-EINVALfor early error detection
uAPI and overal this lgtm, please let me know when you think this is ready to be implemented on Mesa side.
Quoted 46 lines
v2 Changes:
- Reordered patches: Moved shared BO helper before main implementation for
proper dependency order
- Fixed reference counting in mmap offset validation (usedrm_gem_object_put)
- Removed incorrect claims about madvise(WILLNEED) restoring purged BOs
- Fixed error code documentation inconsistencies
- Initializepurge_state_valfields to prevent kernel memory leaks
- Usexe_bo_trigger_rebind()for async TLB invalidation (Thomas Hellström)
- Add NULL rebind with scratch PTEs for fault mode (Thomas Hellström)
- Implement i915-compatible retained field logic (Thomas Hellström)
- Skip BO validation for purged BOs in page fault handler (crash fix)
- Add scratch VM check in page fault path (non-scratch VMs fail fault)Arvind Yadav (7):
drm/xe/bo: Add purgeable bo state tracking and field madv toxe_bodrm/xe/bo: Prevent purging of shared buffer objects drm/xe/madvise: Implement purgeable buffer object support drm/xe/bo: Handle CPU faults on purged buffer objects drm/xe/bo: Prevent mmap of purged buffer objects drm/xe/vm: Prevent binding of purged buffer objects drm/xe: Add support for querying purgeable BO statesHimal Prasad Ghimiray (2):
drm/xe/uapi: Add UAPI support for purgeable buffer objects drm/xe/uapi: Add UAPI for purgeable bo state to madvise query responsedrivers/gpu/drm/xe/xe_bo.c | 92 ++++++++++++++++++++++++-- --
drivers/gpu/drm/xe/xe_bo.h | 57 +++++++++++++++++ drivers/gpu/drm/xe/xe_bo_types.h | 3 +drivers/gpu/drm/xe/xe_gt_pagefault.c| 19 ++++++ drivers/gpu/drm/xe/xe_pt.c | 13 +++- drivers/gpu/drm/xe/xe_vm.c | 12 ++++ drivers/gpu/drm/xe/xe_vm_madvise.c | 76 +++++++++++++++++++++++ include/uapi/drm/xe_drm.h | 50 ++++++++++++++-8 files changed, 308 insertions(+), 14 deletions(-)