Upstreaming prep / all of mbrosts patches
Subsystems (16)
Matthew Brost
maintainercover Upstreaming prep / all of mbrosts patches
Series includes:
- DRM scheduler changes for firmware backends (1 to 1 entity to scheduler)
- LR workload story
- VM LRU handling
- GuC doorbell submission
- Basic GPUVA
- Sparse binding support
- GPUVA + extobj + drm exec (collaboration with dakr + Francois Dugast)
- GPUVA + userptr (minimal, more can be once Nouveua has userptr)
- Fix fencing rules for compute / fault mode
- Remove async worker for VM + error handling updates
Series is not fully ready for upstream and some of these things need to get merged upstream first but overall it is largely correct and certainly step in the right direction. Based on its size and the fact it took me 8 hours to rebase this today I'd say let's get this tree and fixup everything else in place.
Minor uAPI breakage, IGT series will be posted early tomorrow.
gitlab link:
https://gitlab.freedesktop.org/drm/xe/kernel/-/merge_requests/344
Christian König (1):
drm: execution context for GEM buffers v3
Danilo Krummrich (2):
maple_tree: split up MA_STATE() macro
drm: manager to keep track of GPUs VA mappings
Matthew Brost (27):
drm/sched: Add run_wq argument to drm_sched_init drm/sched: Move schedule policy to scheduler drm/sched: Add DRM_SCHED_POLICY_SINGLE_ENTITY scheduling policy drm/xe: Use DRM_SCHED_POLICY_SINGLE_ENTITY mode drm/xe: Long running job update
drm/xe: Ensure LR engines are not persistent
drm/xe: Only try to lock external BOs in VM bind drm/xe: VM LRU bulk move
drm/xe/guc: Read HXG fields from DW1 of G2H response drm/xe/guc: Return the lower part of blocking H2G message drm/xe/guc: Use doorbells for submission if possible drm/xe/guc: Print doorbell ID in GuC engine debugfs entry
maple_tree: Export mas_preallocate
drm/xe: Port Xe to GPUVA
drm/xe: NULL binding implementation
drm/xe: Avoid doing rebinds
drm/xe: Reduce the number list links in xe_vma drm/xe: Optimize size of xe_vma allocation
drm/gpuva: Add drm device to GPUVA manager
drm/gpuva: Move dma-resv to GPUVA manager
drm/gpuva: Add support for extobj
drm/xe: Userptr refactor
drm/exec: Always compile drm_exec
drm/xe: Use drm_exec for locking rather than TTM exec helpers drm/xe: Allow dma-fences as in-syncs for compute / faulting VM drm/xe: Allow compute VMs to output dma-fences on binds drm/xe: remove async worker, sync binds, new error handling
Documentation/gpu/drm-mm.rst | 43 + drivers/gpu/drm/Kconfig | 6 + drivers/gpu/drm/Makefile | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +- drivers/gpu/drm/drm_debugfs.c | 41 + drivers/gpu/drm/drm_exec.c | 248 ++ drivers/gpu/drm/drm_gem.c | 3 + drivers/gpu/drm/drm_gpuva_mgr.c | 1779 ++++++++++++ drivers/gpu/drm/etnaviv/etnaviv_sched.c | 5 +- drivers/gpu/drm/i915/display/intel_display.c | 6 +- drivers/gpu/drm/lima/lima_sched.c | 5 +- drivers/gpu/drm/msm/msm_ringbuffer.c | 5 +- drivers/gpu/drm/panfrost/panfrost_job.c | 5 +- drivers/gpu/drm/scheduler/sched_entity.c | 84 +- drivers/gpu/drm/scheduler/sched_fence.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 88 +- drivers/gpu/drm/v3d/v3d_sched.c | 25 +- drivers/gpu/drm/xe/Kconfig | 1 + drivers/gpu/drm/xe/regs/xe_guc_regs.h | 1 + drivers/gpu/drm/xe/xe_bo.c | 100 +- drivers/gpu/drm/xe/xe_bo.h | 13 +- drivers/gpu/drm/xe/xe_bo_evict.c | 24 +- drivers/gpu/drm/xe/xe_bo_types.h | 1 - drivers/gpu/drm/xe/xe_device.c | 2 +- drivers/gpu/drm/xe/xe_dma_buf.c | 2 +- drivers/gpu/drm/xe/xe_engine.c | 50 +- drivers/gpu/drm/xe/xe_engine.h | 4 + drivers/gpu/drm/xe/xe_engine_types.h | 1 + drivers/gpu/drm/xe/xe_exec.c | 117 +- drivers/gpu/drm/xe/xe_execlist.c | 3 +- drivers/gpu/drm/xe/xe_gt_pagefault.c | 84 +- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 14 +- drivers/gpu/drm/xe/xe_guc.c | 6 + drivers/gpu/drm/xe/xe_guc_ct.c | 12 +- drivers/gpu/drm/xe/xe_guc_engine_types.h | 9 + drivers/gpu/drm/xe/xe_guc_pc.c | 6 +- drivers/gpu/drm/xe/xe_guc_submit.c | 398 ++- drivers/gpu/drm/xe/xe_guc_submit.h | 1 + drivers/gpu/drm/xe/xe_guc_types.h | 4 + drivers/gpu/drm/xe/xe_huc.c | 2 +- drivers/gpu/drm/xe/xe_lrc.c | 8 +- drivers/gpu/drm/xe/xe_migrate.c | 31 +- drivers/gpu/drm/xe/xe_pt.c | 198 +- drivers/gpu/drm/xe/xe_sync.c | 26 +- drivers/gpu/drm/xe/xe_sync.h | 2 +- drivers/gpu/drm/xe/xe_trace.h | 20 +- drivers/gpu/drm/xe/xe_vm.c | 2565 +++++++----------- drivers/gpu/drm/xe/xe_vm.h | 135 +- drivers/gpu/drm/xe/xe_vm_madvise.c | 125 +- drivers/gpu/drm/xe/xe_vm_types.h | 324 ++- drivers/gpu/drm/xe/xe_wait_user_fence.c | 43 +- include/drm/drm_debugfs.h | 24 + include/drm/drm_drv.h | 7 + include/drm/drm_exec.h | 115 + include/drm/drm_gem.h | 75 + include/drm/drm_gpuva_mgr.h | 759 ++++++ include/drm/gpu_scheduler.h | 29 +- include/linux/maple_tree.h | 7 +- include/uapi/drm/xe_drm.h | 87 +- lib/maple_tree.c | 1 + 60 files changed, 5484 insertions(+), 2304 deletions(-) create mode 100644 drivers/gpu/drm/drm_exec.c create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c create mode 100644 include/drm/drm_exec.h create mode 100644 include/drm/drm_gpuva_mgr.h
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 1/30 drm/sched: Add run_wq argument to drm_sched_init
We will have this argument upstream, lets pull into the Xe repo.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 2 +- drivers/gpu/drm/msm/msm_ringbuffer.c | 2 +- drivers/gpu/drm/panfrost/panfrost_job.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 4 +++- drivers/gpu/drm/v3d/v3d_sched.c | 10 +++++----- drivers/gpu/drm/xe/xe_execlist.c | 2 +- drivers/gpu/drm/xe/xe_guc_submit.c | 2 +- include/drm/gpu_scheduler.h | 1 + 10 files changed, 16 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -1
@@ -2364,7 +2364,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev)break ; } - r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, + r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, NULL, ring->num_hw_submission, amdgpu_job_hang_limit, timeout, adev->reset_domain->wq, ring->sched_score, ring->name,
drivers/gpu/drm/etnaviv/etnaviv_sched.c +1 -1
@@ -133,7 +133,7 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) {int ret; - ret =drm_sched_init (&gpu->sched, &etnaviv_sched_ops, + ret =drm_sched_init (&gpu->sched, &etnaviv_sched_ops,NULL , etnaviv_hw_jobs_limit, etnaviv_job_hang_limit,msecs_to_jiffies (500 ),NULL ,NULL ,dev_name (gpu->dev), gpu->dev);
drivers/gpu/drm/lima/lima_sched.c +1 -1
@@ -488,7 +488,7 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name)INIT_WORK (&pipe->recover_work, lima_sched_recover_work); -return drm_sched_init (&pipe->base, &lima_sched_ops,1 , +return drm_sched_init (&pipe->base, &lima_sched_ops,NULL ,1 , lima_job_hang_limit,msecs_to_jiffies (timeout),NULL ,NULL , name, pipe->ldev->dev);
drivers/gpu/drm/msm/msm_ringbuffer.c +1 -1
@@ -95,7 +95,7 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,/* currently managing hangcheck ourselves: */ sched_timeout =MAX_SCHEDULE_TIMEOUT ; - ret =drm_sched_init (&ring->sched, &msm_sched_ops, + ret =drm_sched_init (&ring->sched, &msm_sched_ops,NULL , num_hw_submissions,0 , sched_timeout,NULL ,NULL ,to_msm_bo (ring->bo)->name, gpu->dev->dev);if (ret) {
drivers/gpu/drm/panfrost/panfrost_job.c +1 -1
@@ -815,7 +815,7 @@ int panfrost_job_init(struct panfrost_device *pfdev) js->queue[j].fence_context =dma_fence_context_alloc (1 ); ret = drm_sched_init(&js->queue[j].sched, - &panfrost_sched_ops, + &panfrost_sched_ops, NULL, nentries, 0, msecs_to_jiffies(JOB_TIMEOUT_MS), pfdev->reset.wq,
drivers/gpu/drm/scheduler/sched_main.c +3 -1
@@ -1182,6 +1182,7 @@ static void drm_sched_main(struct work_struct *w)* * @sched: scheduler instance * @ops: backend operations for this scheduler +* @run_wq: workqueue to use for run work. If NULL, the system_wq is used * @hw_submission: number of hw submissions that can be in flight * @hang_limit: number of times to allow a job to hang before dropping it * @timeout: timeout value in jiffies for the scheduler
@@ -1195,6 +1196,7 @@ static void drm_sched_main(struct work_struct *w)*/ int drm_sched_init (struct drm_gpu_scheduler *sched,const struct drm_sched_backend_ops *ops, +struct workqueue_struct *run_wq,unsigned hw_submission,unsigned hang_limit,long timeout,struct workqueue_struct *timeout_wq,atomic_t *score,const char *name,struct device *dev)
@@ -1203,9 +1205,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name; + sched->run_wq = run_wq ? : system_wq; sched->timeout = timeout; sched->timeout_wq = timeout_wq ? : system_wq; - sched->run_wq = system_wq;/* FIXME: Let user pass this in */ sched->hang_limit = hang_limit; sched->score = score ? score : &sched->_score; sched->dev = dev;
drivers/gpu/drm/v3d/v3d_sched.c +5 -5
@@ -388,7 +388,7 @@ v3d_sched_init(struct v3d_dev *v3d)int ret; ret =drm_sched_init (&v3d->queue[V3D_BIN ].sched, - &v3d_bin_sched_ops, + &v3d_bin_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL ,NULL ,"v3d_bin" , v3d->drm.dev);
@@ -396,7 +396,7 @@ v3d_sched_init(struct v3d_dev *v3d)return ret; ret =drm_sched_init (&v3d->queue[V3D_RENDER ].sched, - &v3d_render_sched_ops, + &v3d_render_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL ,NULL ,"v3d_render" , v3d->drm.dev);
@@ -404,7 +404,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret =drm_sched_init (&v3d->queue[V3D_TFU ].sched, - &v3d_tfu_sched_ops, + &v3d_tfu_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL ,NULL ,"v3d_tfu" , v3d->drm.dev);
@@ -413,7 +413,7 @@ v3d_sched_init(struct v3d_dev *v3d)if (v3d_has_csd (v3d)) { ret =drm_sched_init (&v3d->queue[V3D_CSD ].sched, - &v3d_csd_sched_ops, + &v3d_csd_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL ,NULL ,"v3d_csd" , v3d->drm.dev);
@@ -421,7 +421,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret =drm_sched_init (&v3d->queue[V3D_CACHE_CLEAN ].sched, - &v3d_cache_clean_sched_ops, + &v3d_cache_clean_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL ,NULL ,"v3d_cache_clean" , v3d->drm.dev);
drivers/gpu/drm/xe/xe_execlist.c +1 -1
@@ -336,7 +336,7 @@ static int execlist_engine_init(struct xe_engine *e) exl->engine = e; - err = drm_sched_init(&exl->sched, &drm_sched_ops, + err = drm_sched_init(&exl->sched, &drm_sched_ops, NULL, e->lrc[0].ring.size / MAX_JOB_SIZE_BYTES, XE_SCHED_HANG_LIMIT, XE_SCHED_JOB_TIMEOUT, NULL, NULL, e->hwe->name,
drivers/gpu/drm/xe/xe_guc_submit.c +1 -1
@@ -1081,7 +1081,7 @@ static int guc_engine_init(struct xe_engine *e)init_waitqueue_head (&ge->suspend_wait); timeout =xe_vm_no_dma_fences (e->vm) ?MAX_SCHEDULE_TIMEOUT :HZ *5 ; - err =drm_sched_init (&ge->sched, &drm_sched_ops, + err =drm_sched_init (&ge->sched, &drm_sched_ops,NULL , e->lrc[0 ].ring.size /MAX_JOB_SIZE_BYTES ,64 , timeout,guc_to_gt (guc)->ordered_wq,NULL , e->name,gt_to_xe (e->gt)->drm.dev);
include/drm/gpu_scheduler.h +1 -0
@@ -539,6 +539,7 @@ struct drm_gpu_scheduler {int drm_sched_init (struct drm_gpu_scheduler *sched,const struct drm_sched_backend_ops *ops, +struct workqueue_struct *run_wq,uint32_t hw_submission,unsigned hang_limit,long timeout,struct workqueue_struct *timeout_wq,atomic_t *score,const char *name,struct device *dev);
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 2/30 drm/sched: Move schedule policy to scheduler
In reply to Matthew BrostRather than a global modparam for scheduling policy, move the scheduling policy to scheduler so user can control each scheduler policy.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/etnaviv/etnaviv_sched.c | 3 ++- drivers/gpu/drm/lima/lima_sched.c | 3 ++- drivers/gpu/drm/msm/msm_ringbuffer.c | 3 ++- drivers/gpu/drm/panfrost/panfrost_job.c | 3 ++- drivers/gpu/drm/scheduler/sched_entity.c | 24 ++++++++++++++++++---- drivers/gpu/drm/scheduler/sched_main.c | 21 ++++++++++++++----- drivers/gpu/drm/v3d/v3d_sched.c | 15 +++++++++----- drivers/gpu/drm/xe/xe_execlist.c | 2 +- drivers/gpu/drm/xe/xe_guc_submit.c | 3 ++- include/drm/gpu_scheduler.h | 20 ++++++++++++------ 11 files changed, 72 insertions(+), 26 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
@@ -2368,6 +2368,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) ring->num_hw_submission, amdgpu_job_hang_limit, timeout, adev->reset_domain->wq, ring->sched_score, ring->name, +DRM_SCHED_POLICY_DEFAULT , adev->dev);if (r) { DRM_ERROR("Failed to create scheduler on ring %s.\n",
drivers/gpu/drm/etnaviv/etnaviv_sched.c +2 -1
@@ -136,7 +136,8 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) ret =drm_sched_init (&gpu->sched, &etnaviv_sched_ops,NULL , etnaviv_hw_jobs_limit, etnaviv_job_hang_limit,msecs_to_jiffies (500 ),NULL ,NULL , -dev_name (gpu->dev), gpu->dev); +dev_name (gpu->dev),DRM_SCHED_POLICY_DEFAULT , + gpu->dev);if (ret)return ret;
drivers/gpu/drm/lima/lima_sched.c +2 -1
@@ -491,7 +491,8 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name)return drm_sched_init (&pipe->base, &lima_sched_ops,NULL ,1 , lima_job_hang_limit,msecs_to_jiffies (timeout),NULL , -NULL , name, pipe->ldev->dev); +NULL , name,DRM_SCHED_POLICY_DEFAULT , + pipe->ldev->dev); }void lima_sched_pipe_fini (struct lima_sched_pipe *pipe)
drivers/gpu/drm/msm/msm_ringbuffer.c +2 -1
@@ -97,7 +97,8 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, ret =drm_sched_init (&ring->sched, &msm_sched_ops,NULL , num_hw_submissions,0 , sched_timeout, -NULL ,NULL ,to_msm_bo (ring->bo)->name, gpu->dev->dev); +NULL ,NULL ,to_msm_bo (ring->bo)->name, +DRM_SCHED_POLICY_DEFAULT , gpu->dev->dev);if (ret) { goto fail; }
drivers/gpu/drm/panfrost/panfrost_job.c +2 -1
@@ -819,7 +819,8 @@ int panfrost_job_init(struct panfrost_device *pfdev) nentries,0 ,msecs_to_jiffies (JOB_TIMEOUT_MS ), pfdev->reset.wq, - NULL, "pan_js", pfdev->dev); +NULL ,"pan_js" ,DRM_SCHED_POLICY_DEFAULT , + pfdev->dev);if (ret) {dev_err (pfdev->dev,"Failed to create scheduler: %d." , ret); goto err_sched;
drivers/gpu/drm/scheduler/sched_entity.c +20 -4
@@ -33,6 +33,20 @@#define to_drm_sched_job (sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) +static bool bad_policies (struct drm_gpu_scheduler **sched_list, +unsigned int num_sched_list) +{ +enum drm_sched_policy sched_policy = sched_list[0 ]->sched_policy; +unsigned int i; + +/* All scdedule policies must match */ +for (i =1 ; i < num_sched_list; ++i) +if (sched_policy != sched_list[i]->sched_policy) +return true; + +return false; +} +/** * drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring.
@@ -62,7 +76,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, unsigned int num_sched_list, atomic_t *guilty) { -if (!(entity && sched_list && (num_sched_list ==0 || sched_list[0 ]))) +if (!(entity && sched_list && (num_sched_list ==0 || sched_list[0 ])) || +bad_policies (sched_list, num_sched_list))return -EINVAL ;memset (entity,0 ,sizeof (struct drm_sched_entity ));
@@ -75,8 +90,9 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, entity->last_scheduled =NULL ;RB_CLEAR_NODE (&entity->rb_tree_node); -if (num_sched_list) +if (num_sched_list) { entity->rq = &sched_list[0 ]->sched_rq[entity->priority]; + }init_completion (&entity->entity_idle);
@@ -440,7 +456,7 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)* Update the entity's location in the min heap according to * the timestamp of the next job, if any. */ -if (drm_sched_policy ==DRM_SCHED_POLICY_FIFO ) { +if (entity->rq->sched->sched_policy ==DRM_SCHED_POLICY_FIFO ) {struct drm_sched_job *next; next =to_drm_sched_job (spsc_queue_peek (&entity->job_queue));
@@ -528,7 +544,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)drm_sched_rq_add_entity (entity->rq, entity);spin_unlock (&entity->rq_lock); -if (drm_sched_policy ==DRM_SCHED_POLICY_FIFO ) +if (entity->rq->sched->sched_policy ==DRM_SCHED_POLICY_FIFO )drm_sched_rq_update_fifo (entity, sched_job->submit_ts);drm_sched_wakeup (entity->rq->sched);
drivers/gpu/drm/scheduler/sched_main.c +16 -5
@@ -62,14 +62,14 @@#define to_drm_sched_job (sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) -int drm_sched_policy =DRM_SCHED_POLICY_FIFO ; +int default_drm_sched_policy =DRM_SCHED_POLICY_FIFO ;/** * DOC: sched_policy (int) * Used to override default entities scheduling policy in a run queue. */ MODULE_PARM_DESC(sched_policy, "Specify the scheduling policy for entities on a run-queue, " __stringify(DRM_SCHED_POLICY_RR) " = Round Robin, " __stringify(DRM_SCHED_POLICY_FIFO) " = FIFO (default)."); -module_param_named (sched_policy, drm_sched_policy, int,0444 ); +module_param_named (sched_policy, default_drm_sched_policy, int,0444 ); static __always_inline bool drm_sched_entity_compare_before(struct rb_node *a,const struct rb_node *b)
@@ -173,7 +173,7 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,if (rq->current_entity == entity) rq->current_entity =NULL ; -if (drm_sched_policy ==DRM_SCHED_POLICY_FIFO ) +if (rq->sched->sched_policy ==DRM_SCHED_POLICY_FIFO )drm_sched_rq_remove_fifo_locked (entity);spin_unlock (&rq->lock);
@@ -956,7 +956,7 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched)/* Kernel run queue has higher priority than normal run queue*/ for (i =DRM_SCHED_PRIORITY_COUNT -1 ; i >=DRM_SCHED_PRIORITY_MIN ; i--) { - entity = drm_sched_policy ==DRM_SCHED_POLICY_FIFO ? + entity = sched->sched_policy ==DRM_SCHED_POLICY_FIFO ?drm_sched_rq_select_entity_fifo (&sched->sched_rq[i]) :drm_sched_rq_select_entity_rr (&sched->sched_rq[i]);if (entity)
@@ -1190,6 +1190,7 @@ static void drm_sched_main(struct work_struct *w)* used * @score: optional score atomic shared with other schedulers * @name: name used for debugging +* @sched_policy: schedule policy * @dev: target &struct device * * Return 0 on success, otherwise error code.
@@ -1199,9 +1200,15 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,struct workqueue_struct *run_wq, unsigned hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev) + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev) { int i; + +if (sched_policy >=DRM_SCHED_POLICY_COUNT ) +return -EINVAL ; + sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name;
@@ -1211,6 +1218,10 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->hang_limit = hang_limit; sched->score = score ? score : &sched->_score; sched->dev = dev; +if (sched_policy ==DRM_SCHED_POLICY_DEFAULT ) + sched->sched_policy = default_drm_sched_policy; +else + sched->sched_policy = sched_policy;for (i =DRM_SCHED_PRIORITY_MIN ; i <DRM_SCHED_PRIORITY_COUNT ; i++)drm_sched_rq_init (sched, &sched->sched_rq[i]);
drivers/gpu/drm/v3d/v3d_sched.c +10 -5
@@ -391,7 +391,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_bin_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL , - NULL, "v3d_bin", v3d->drm.dev); +NULL ,"v3d_bin" ,DRM_SCHED_POLICY_DEFAULT , + v3d->drm.dev);if (ret)return ret;
@@ -399,7 +400,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_render_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL , - NULL, "v3d_render", v3d->drm.dev); +ULL ,"v3d_render" ,DRM_SCHED_POLICY_DEFAULT , + v3d->drm.dev);if (ret) goto fail;
@@ -407,7 +409,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_tfu_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL , - NULL, "v3d_tfu", v3d->drm.dev); +NULL ,"v3d_tfu" ,DRM_SCHED_POLICY_DEFAULT , + v3d->drm.dev);if (ret) goto fail;
@@ -416,7 +419,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_csd_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL , - NULL, "v3d_csd", v3d->drm.dev); +NULL ,"v3d_csd" ,DRM_SCHED_POLICY_DEFAULT , + v3d->drm.dev);if (ret) goto fail;
@@ -424,7 +428,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_cache_clean_sched_ops,NULL , hw_jobs_limit, job_hang_limit,msecs_to_jiffies (hang_limit_ms),NULL , - NULL, "v3d_cache_clean", v3d->drm.dev); +NULL ,"v3d_cache_clean" , + DRM_SCHED_POLICY_DEFAULT, v3d->drm.dev);if (ret) goto fail; }
drivers/gpu/drm/xe/xe_execlist.c +1 -1
@@ -339,7 +339,7 @@ static int execlist_engine_init(struct xe_engine *e) err =drm_sched_init (&exl->sched, &drm_sched_ops,NULL , e->lrc[0 ].ring.size /MAX_JOB_SIZE_BYTES ,XE_SCHED_HANG_LIMIT ,XE_SCHED_JOB_TIMEOUT , -NULL ,NULL , e->hwe->name, +NULL ,NULL , e->hwe->name,DRM_SCHED_POLICY_DEFAULT ,gt_to_xe (e->gt)->drm.dev);if (err) goto err_free;
drivers/gpu/drm/xe/xe_guc_submit.c +2 -1
@@ -1084,7 +1084,8 @@ static int guc_engine_init(struct xe_engine *e) err =drm_sched_init (&ge->sched, &drm_sched_ops,NULL , e->lrc[0 ].ring.size /MAX_JOB_SIZE_BYTES ,64 , timeout,guc_to_gt (guc)->ordered_wq,NULL , - e->name,gt_to_xe (e->gt)->drm.dev); + e->name,DRM_SCHED_POLICY_DEFAULT , +gt_to_xe (e->gt)->drm.dev);if (err) goto err_free;
include/drm/gpu_scheduler.h +14 -6
@@ -63,11 +63,15 @@ enum drm_sched_priority { DRM_SCHED_PRIORITY_UNSET = -2 }; -/* Used to chose between FIFO and RR jobs scheduling */ -extern int drm_sched_policy; - -#define DRM_SCHED_POLICY_RR 0 -#define DRM_SCHED_POLICY_FIFO 1 +/* Used to chose default scheduling policy*/ +extern int default_drm_sched_policy; + +enum drm_sched_policy { +DRM_SCHED_POLICY_DEFAULT , +DRM_SCHED_POLICY_RR , +DRM_SCHED_POLICY_FIFO , +DRM_SCHED_POLICY_COUNT , +};/** * struct drm_sched_entity - A wrapper around a job queue (typically
@@ -505,6 +509,7 @@ struct drm_sched_backend_ops {* guilty and it will no longer be considered for scheduling. * @score: score to help loadbalancer pick a idle sched * @_score: score used when the driver doesn't provide one +* @sched_policy: Schedule policy for scheduler * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. * @pause_run_wq: pause queuing of @work_run on @run_wq
@@ -531,6 +536,7 @@ struct drm_gpu_scheduler {int hang_limit;atomic_t *score;atomic_t _score; +enum drm_sched_policy sched_policy;bool ready;bool free_guilty;bool pause_run_wq;
@@ -542,7 +548,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,struct workqueue_struct *run_wq, uint32_t hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev); + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev);void drm_sched_fini (struct drm_gpu_scheduler *sched); int drm_sched_job_init(struct drm_sched_job *job,
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 3/30 drm/sched: Add DRM_SCHED_POLICY_SINGLE_ENTITY scheduling policy
In reply to Matthew BrostDRM_SCHED_POLICY_SINGLE_ENTITY creates a 1 to 1 relationship between scheduler and entity. No priorities or run queue used in this mode. Intended for devices with firmware schedulers.
--- drivers/gpu/drm/scheduler/sched_entity.c | 64 +++++++++++++++++++----- drivers/gpu/drm/scheduler/sched_fence.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 63 ++++++++++++++++++++--- include/drm/gpu_scheduler.h | 8 +++ 4 files changed, 115 insertions(+), 22 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +51 -13
@@ -83,6 +83,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,memset (entity,0 ,sizeof (struct drm_sched_entity ));INIT_LIST_HEAD (&entity->list); entity->rq =NULL ; + entity->single_sched =NULL ; entity->guilty = guilty; entity->num_sched_list = num_sched_list; entity->priority = priority;
@@ -91,7 +92,15 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,RB_CLEAR_NODE (&entity->rb_tree_node);if (num_sched_list) { - entity->rq = &sched_list[0 ]->sched_rq[entity->priority]; +if (sched_list[0 ]->sched_policy != +DRM_SCHED_POLICY_SINGLE_ENTITY ) { + entity->rq = &sched_list[0 ]->sched_rq[entity->priority]; + }else { +if (num_sched_list !=1 || sched_list[0 ]->single_entity) +return -EINVAL ; + sched_list[0 ]->single_entity = entity; + entity->single_sched = sched_list[0 ]; + } }init_completion (&entity->entity_idle);
@@ -125,7 +134,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity,struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list) { - WARN_ON(!num_sched_list || !sched_list); + WARN_ON(!num_sched_list || !sched_list || + !!entity->single_sched); entity->sched_list = sched_list; entity->num_sched_list = num_sched_list;
@@ -195,13 +205,15 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) {struct drm_sched_job *job;struct dma_fence *prev; +bool single_entity = !!entity->single_sched; -if (!entity->rq) +if (!entity->rq && !single_entity)return ;spin_lock (&entity->rq_lock); entity->stopped = true; -drm_sched_rq_remove_entity (entity->rq, entity); +if (!single_entity) +drm_sched_rq_remove_entity (entity->rq, entity);spin_unlock (&entity->rq_lock);/* Make sure this entity is not used by the scheduler at the moment */
@@ -223,6 +235,20 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity)dma_fence_put (prev); } +/** +* drm_sched_entity_to_scheduler - Schedule entity to GPU scheduler +* @entity: scheduler entity +* +* Returns GPU scheduler for the entity +*/ +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler (struct drm_sched_entity *entity) +{ +bool single_entity = !!entity->single_sched; + +return single_entity ? entity->single_sched : entity->rq->sched; +} +/** * drm_sched_entity_flush - Flush a context entity *
@@ -240,11 +266,12 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout)struct drm_gpu_scheduler *sched;struct task_struct *last_user;long ret = timeout; +bool single_entity = !!entity->single_sched; -if (!entity->rq) +if (!entity->rq && !single_entity)return 0 ; - sched = entity->rq->sched; + sched =drm_sched_entity_to_scheduler (entity);/** * The client will not queue more IBs during this fini, consume existing * queued IBs or discard them on SIGKILL
@@ -337,7 +364,7 @@ static void drm_sched_entity_wakeup(struct dma_fence *f, container_of(cb, struct drm_sched_entity, cb);drm_sched_entity_clear_dep (f, cb); -drm_sched_wakeup (entity->rq->sched); +drm_sched_wakeup (drm_sched_entity_to_scheduler (entity)); }/**
@@ -351,6 +378,8 @@ static void drm_sched_entity_wakeup(struct dma_fence *f,void drm_sched_entity_set_priority (struct drm_sched_entity *entity,enum drm_sched_priority priority) { +WARN_ON (!!entity->single_sched); +spin_lock (&entity->rq_lock); entity->priority = priority;spin_unlock (&entity->rq_lock);
@@ -363,7 +392,7 @@ EXPORT_SYMBOL(drm_sched_entity_set_priority);*/ static bool drm_sched_entity_add_dependency_cb (struct drm_sched_entity *entity) { -struct drm_gpu_scheduler *sched = entity->rq->sched; +struct drm_gpu_scheduler *sched =drm_sched_entity_to_scheduler (entity);struct dma_fence *fence = entity->dependency;struct drm_sched_fence *s_fence;
@@ -456,7 +485,8 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)* Update the entity's location in the min heap according to * the timestamp of the next job, if any. */ -if (entity->rq->sched->sched_policy ==DRM_SCHED_POLICY_FIFO ) { +if (drm_sched_entity_to_scheduler (entity)->sched_policy == +DRM_SCHED_POLICY_FIFO ) {struct drm_sched_job *next; next =to_drm_sched_job (spsc_queue_peek (&entity->job_queue));
@@ -473,6 +503,8 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)struct drm_gpu_scheduler *sched;struct drm_sched_rq *rq; +WARN_ON (!!entity->single_sched); +/* single possible engine and already selected */ if (!entity->sched_list)return ;
@@ -522,16 +554,21 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)void drm_sched_entity_push_job (struct drm_sched_job *sched_job) {struct drm_sched_entity *entity = sched_job->entity; +bool single_entity = !!entity->single_sched;bool first;trace_drm_sched_job (sched_job, entity); -atomic_inc (entity->rq->sched->score); +if (!single_entity) +atomic_inc (entity->rq->sched->score);WRITE_ONCE (entity->last_user, current->group_leader); first =spsc_queue_push (&entity->job_queue, &sched_job->queue_node); sched_job->submit_ts =ktime_get ();/* first job wakes up scheduler */ if (first) { +struct drm_gpu_scheduler *sched = +drm_sched_entity_to_scheduler (entity); +/* Add the entity to the run queue */ spin_lock (&entity->rq_lock);if (entity->stopped) {
@@ -541,13 +578,14 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)return ; } -drm_sched_rq_add_entity (entity->rq, entity); +if (!single_entity) +drm_sched_rq_add_entity (entity->rq, entity);spin_unlock (&entity->rq_lock); -if (entity->rq->sched->sched_policy ==DRM_SCHED_POLICY_FIFO ) +if (sched->sched_policy ==DRM_SCHED_POLICY_FIFO )drm_sched_rq_update_fifo (entity, sched_job->submit_ts); -drm_sched_wakeup (entity->rq->sched); +drm_sched_wakeup (sched); } }EXPORT_SYMBOL (drm_sched_entity_push_job);
drivers/gpu/drm/scheduler/sched_fence.c +1 -1
@@ -167,7 +167,7 @@ void drm_sched_fence_init(struct drm_sched_fence *fence, {unsigned seq; - fence->sched = entity->rq->sched; + fence->sched =drm_sched_entity_to_scheduler (entity); seq =atomic_inc_return (&entity->fence_seq);dma_fence_init (&fence->scheduled, &drm_sched_fence_ops_scheduled, &fence->lock, entity->fence_context, seq);
drivers/gpu/drm/scheduler/sched_main.c +55 -8
145 lines not on this page · raw message
include/drm/gpu_scheduler.h +8 -0
@@ -70,6 +70,7 @@ enum drm_sched_policy {DRM_SCHED_POLICY_DEFAULT ,DRM_SCHED_POLICY_RR ,DRM_SCHED_POLICY_FIFO , +DRM_SCHED_POLICY_SINGLE_ENTITY , DRM_SCHED_POLICY_COUNT, };
@@ -103,6 +104,9 @@ struct drm_sched_entity {*/ struct drm_sched_rq *rq; +/** @single_sched: Single scheduler */ +struct drm_gpu_scheduler *single_sched; +/** * @sched_list: *
@@ -488,6 +492,7 @@ struct drm_sched_backend_ops {* struct drm_gpu_scheduler - scheduler instance-specific data * * @ops: backend operations provided by the driver. +* @single_entity: Single entity for the scheduler * @hw_submission_limit: the max size of the hardware queue. * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used.
@@ -519,6 +524,7 @@ struct drm_sched_backend_ops {*/ struct drm_gpu_scheduler {const struct drm_sched_backend_ops *ops; +struct drm_sched_entity *single_entity;uint32_t hw_submission_limit;long timeout;const char *name;
@@ -604,6 +610,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list, atomic_t *guilty); +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler (struct drm_sched_entity *entity);long drm_sched_entity_flush (struct drm_sched_entity *entity,long timeout);void drm_sched_entity_fini (struct drm_sched_entity *entity);void drm_sched_entity_destroy (struct drm_sched_entity *entity);
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 4/30 drm/xe: Use DRM_SCHED_POLICY_SINGLE_ENTITY mode
In reply to Matthew BrostWe create 1 GPU scheduler per entity in Xe, use DRM_SCHED_POLICY_SINGLE_ENTITY scheduling which is designed for that paradigm.
--- drivers/gpu/drm/xe/xe_execlist.c | 3 ++- drivers/gpu/drm/xe/xe_guc_submit.c | 3 +-- 2 files changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/xe/xe_execlist.c +2 -1
@@ -339,7 +339,8 @@ static int execlist_engine_init(struct xe_engine *e) err =drm_sched_init (&exl->sched, &drm_sched_ops,NULL , e->lrc[0 ].ring.size /MAX_JOB_SIZE_BYTES ,XE_SCHED_HANG_LIMIT ,XE_SCHED_JOB_TIMEOUT , -NULL ,NULL , e->hwe->name,DRM_SCHED_POLICY_DEFAULT , +NULL ,NULL , e->hwe->name, +DRM_SCHED_POLICY_SINGLE_ENTITY ,gt_to_xe (e->gt)->drm.dev);if (err) goto err_free;
drivers/gpu/drm/xe/xe_guc_submit.c +1 -2
@@ -1084,7 +1084,7 @@ static int guc_engine_init(struct xe_engine *e) err =drm_sched_init (&ge->sched, &drm_sched_ops,NULL , e->lrc[0 ].ring.size /MAX_JOB_SIZE_BYTES ,64 , timeout,guc_to_gt (guc)->ordered_wq,NULL , - e->name,DRM_SCHED_POLICY_DEFAULT , + e->name,DRM_SCHED_POLICY_SINGLE_ENTITY ,gt_to_xe (e->gt)->drm.dev);if (err) goto err_free;
@@ -1185,7 +1185,6 @@ static int guc_engine_set_priority(struct xe_engine *e,if (!msg)return -ENOMEM ; -drm_sched_entity_set_priority (e->entity, priority);guc_engine_add_msg (e, msg,SET_SCHED_PROPS );return 0 ;
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 5/30 drm/xe: Long running job update
In reply to Matthew BrostFlow control + write ring in exec, return NULL in run_job, siganl xe_hw_fence immediately, and override TDR for LR jobs.
--- drivers/gpu/drm/xe/xe_engine.c | 32 ++++++++ drivers/gpu/drm/xe/xe_engine.h | 4 + drivers/gpu/drm/xe/xe_exec.c | 8 ++ drivers/gpu/drm/xe/xe_guc_engine_types.h | 2 + drivers/gpu/drm/xe/xe_guc_submit.c | 95 +++++++++++++++++++++--- drivers/gpu/drm/xe/xe_trace.h | 5 ++ 6 files changed, 137 insertions(+), 9 deletions(-)
drivers/gpu/drm/xe/xe_engine.c +32 -0
@@ -18,6 +18,7 @@#include "xe_macros.h" #include "xe_migrate.h" #include "xe_pm.h" +#include "xe_ring_ops_types.h" #include "xe_trace.h" #include "xe_vm.h"
@@ -673,6 +674,37 @@ static void engine_kill_compute(struct xe_engine *e)up_write (&e->vm->lock); } +/** +* xe_engine_is_lr() - Whether an engine is long-running +* @e: The engine +* +* Return: True if the engine is long-running, false otherwise. +*/ +bool xe_engine_is_lr (struct xe_engine *e) +{ +return e->vm &&xe_vm_no_dma_fences (e->vm) && + !(e->flags &ENGINE_FLAG_VM ); +} + +static s32 xe_engine_num_job_inflight (struct xe_engine *e) +{ +return e->lrc->fence_ctx.next_seqno -xe_lrc_seqno (e->lrc) -1 ; +} + +/** +* xe_engine_ring_full() - Whether an engine's ring is full +* @e: The engine +* +* Return: True if the engine's ring is full, false otherwise. +*/ +bool xe_engine_ring_full (struct xe_engine *e) +{ +struct xe_lrc *lrc = e->lrc; +s32 max_job = lrc->ring.size /MAX_JOB_SIZE_BYTES ; + +return xe_engine_num_job_inflight (e) >= max_job; +} +/** * xe_engine_is_idle() - Whether an engine is idle. * @engine: The engine
drivers/gpu/drm/xe/xe_engine.h +4 -0
@@ -42,6 +42,10 @@ static inline bool xe_engine_is_parallel(struct xe_engine *engine)return engine->width >1 ; } +bool xe_engine_is_lr (struct xe_engine *e); + +bool xe_engine_ring_full (struct xe_engine *e); +bool xe_engine_is_idle (struct xe_engine *engine);void xe_engine_kill (struct xe_engine *e);
drivers/gpu/drm/xe/xe_exec.c +8 -0
@@ -13,6 +13,7 @@#include "xe_device.h" #include "xe_engine.h" #include "xe_macros.h" +#include "xe_ring_ops_types.h" #include "xe_sched_job.h" #include "xe_sync.h" #include "xe_vm.h"
@@ -277,6 +278,11 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto err_engine_end; } +if (xe_engine_is_lr (engine) &&xe_engine_ring_full (engine)) { + err = -EWOULDBLOCK ; + goto err_engine_end; + } + job =xe_sched_job_create (engine,xe_engine_is_parallel (engine) ? addresses : &args->address);if (IS_ERR (job)) {
@@ -363,6 +369,8 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)xe_sync_entry_signal (&syncs[i], job, &job->drm.s_fence->finished); +if (xe_engine_is_lr (engine)) + engine->ring_ops->emit_job (job);xe_sched_job_push (job);xe_vm_reactivate_rebind (vm);
drivers/gpu/drm/xe/xe_guc_engine_types.h +2 -0
@@ -31,6 +31,8 @@ struct xe_guc_engine {*/ #define MAX_STATIC_MSG_TYPE 3struct drm_sched_msg static_msgs[MAX_STATIC_MSG_TYPE ]; +/** @lr_tdr: long running TDR worker */ +struct work_struct lr_tdr;/** @fini_async: do final fini async from this worker */ struct work_struct fini_async;/** @resume_time: time of last resume */
drivers/gpu/drm/xe/xe_guc_submit.c +86 -9
190 lines not on this page · raw message
drivers/gpu/drm/xe/xe_trace.h +5 -0
@@ -219,6 +219,11 @@ DEFINE_EVENT(xe_engine, xe_engine_resubmit,TP_ARGS (e) ); +DEFINE_EVENT (xe_engine, xe_engine_lr_cleanup, + TP_PROTO(struct xe_engine *e), +TP_ARGS (e) +); + DECLARE_EVENT_CLASS(xe_sched_job, TP_PROTO(struct xe_sched_job *job), TP_ARGS(job),
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 6/30 drm/xe: Ensure LR engines are not persistent
In reply to Matthew BrostWith our ref counting scheme LR engines only close properly if not persistent, ensure that LR engines are non-persistent.
--- drivers/gpu/drm/xe/xe_engine.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_engine.c +3 -1
@@ -596,7 +596,9 @@ int xe_engine_create_ioctl(struct drm_device *dev, void *data,return -ENOENT ; e =xe_engine_create (xe, vm, logical_mask, - args->width, hwe,ENGINE_FLAG_PERSISTENT ); + args->width, hwe, +xe_vm_no_dma_fences (vm) ?0 : +ENGINE_FLAG_PERSISTENT );xe_vm_put (vm);if (IS_ERR (e))return PTR_ERR (e);
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 7/30 drm/xe: Only try to lock external BOs in VM bind
In reply to Matthew BrostNot needed and causes some issues with bulk LRU moves.
--- drivers/gpu/drm/xe/xe_vm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +5 -3
@@ -2064,9 +2064,11 @@ static int vm_bind_ioctl(struct xe_vm *vm, struct xe_vma *vma,*/ xe_bo_get (vbo); - tv_bo.bo = &vbo->ttm; - tv_bo.num_shared =1 ; -list_add (&tv_bo.head, &objs); +if (!vbo->vm) { + tv_bo.bo = &vbo->ttm; + tv_bo.num_shared =1 ; +list_add (&tv_bo.head, &objs); + } } again:
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 8/30 drm/xe: VM LRU bulk move
In reply to Matthew BrostUse the TTM LRU bulk move for BOs tied to a VM. Update the bulk moves LRU position on every exec.
--- drivers/gpu/drm/xe/xe_bo.c | 32 ++++++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_bo.h | 4 ++-- drivers/gpu/drm/xe/xe_dma_buf.c | 2 +- drivers/gpu/drm/xe/xe_exec.c | 6 ++++++ drivers/gpu/drm/xe/xe_vm_types.h | 3 +++ 5 files changed, 40 insertions(+), 7 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +28 -4
@@ -985,6 +985,23 @@ static void xe_gem_object_free(struct drm_gem_object *obj)ttm_bo_put (container_of(obj ,struct ttm_buffer_object ,base )); } +static void xe_gem_object_close (struct drm_gem_object *obj, +struct drm_file *file_priv) +{ +struct xe_bo *bo =gem_to_xe_bo (obj); + +if (bo->vm && !xe_vm_no_dma_fences (bo->vm)) { +struct ww_acquire_ctx ww; + +XE_BUG_ON (!xe_bo_is_user (bo)); + +xe_bo_lock (bo, &ww,0 , false); +ttm_bo_set_bulk_move (&bo->ttm,NULL ); +xe_bo_unlock (bo, &ww); + } +} + +static bool should_migrate_to_system (struct xe_bo *bo) {struct xe_device *xe =xe_bo_device (bo);
@@ -1040,6 +1057,7 @@ static const struct vm_operations_struct xe_gem_vm_ops = {static const struct drm_gem_object_funcs xe_gem_object_funcs = { .free = xe_gem_object_free, + .close = xe_gem_object_close, .mmap = drm_gem_ttm_mmap, .export = xe_gem_prime_export, .vm_ops = &xe_gem_vm_ops,
@@ -1081,8 +1099,8 @@ void xe_bo_free(struct xe_bo *bo)struct xe_bo *__xe_bo_create_locked (struct xe_device *xe,struct xe_bo *bo,struct xe_gt *gt,struct dma_resv *resv, -size_t size,enum ttm_bo_type type, -u32 flags) +struct ttm_lru_bulk_move *bulk,size_t size, +enum ttm_bo_type type,u32 flags) {struct ttm_operation_ctx ctx = { .interruptible = true,
@@ -1149,7 +1167,10 @@ struct xe_bo *__xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo,return ERR_PTR (err); bo->created = true; -ttm_bo_move_to_lru_tail_unlocked (&bo->ttm); +if (bulk) +ttm_bo_set_bulk_move (&bo->ttm, bulk); +else +ttm_bo_move_to_lru_tail_unlocked (&bo->ttm);return bo; }
@@ -1219,7 +1240,10 @@ xe_bo_create_locked_range(struct xe_device *xe, } } - bo =__xe_bo_create_locked (xe, bo, gt, vm ? &vm->resv :NULL , size, + bo =__xe_bo_create_locked (xe, bo, gt, vm ? &vm->resv :NULL , + vm && !xe_vm_no_dma_fences (vm) && + flags &XE_BO_CREATE_USER_BIT ? + &vm->lru_bulk_move :NULL , size, type, flags);if (IS_ERR (bo))return bo;
drivers/gpu/drm/xe/xe_bo.h +2 -2
@@ -81,8 +81,8 @@ void xe_bo_free(struct xe_bo *bo);struct xe_bo *__xe_bo_create_locked (struct xe_device *xe,struct xe_bo *bo,struct xe_gt *gt,struct dma_resv *resv, -size_t size,enum ttm_bo_type type, -u32 flags); +struct ttm_lru_bulk_move *bulk,size_t size, +enum ttm_bo_type type,u32 flags);struct xe_bo * xe_bo_create_locked_range(struct xe_device *xe, struct xe_gt *gt, struct xe_vm *vm,
drivers/gpu/drm/xe/xe_dma_buf.c +1 -1
@@ -199,7 +199,7 @@ xe_dma_buf_init_obj(struct drm_device *dev, struct xe_bo *storage,int ret;dma_resv_lock (resv,NULL ); - bo =__xe_bo_create_locked (xe, storage,NULL , resv, dma_buf->size, + bo =__xe_bo_create_locked (xe, storage,NULL , resv,NULL , dma_buf->size, ttm_bo_type_sg,XE_BO_CREATE_SYSTEM_BIT );if (IS_ERR (bo)) { ret =PTR_ERR (bo);
drivers/gpu/drm/xe/xe_exec.c +6 -0
@@ -374,6 +374,12 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)xe_sched_job_push (job);xe_vm_reactivate_rebind (vm); +if (!err && !xe_vm_no_dma_fences (vm)) { +spin_lock (&xe->ttm.lru_lock); +ttm_lru_bulk_move_tail (&vm->lru_bulk_move); +spin_unlock (&xe->ttm.lru_lock); + } + err_repin:if (!xe_vm_no_dma_fences (vm))up_read (&vm->userptr.notifier_lock);
drivers/gpu/drm/xe/xe_vm_types.h +3 -0
@@ -164,6 +164,9 @@ struct xe_vm {/** Protects @rebind_list and the page-table structures */ struct dma_resv resv; +/** @lru_bulk_move: Bulk LRU move list for this VM's BOs */ +struct ttm_lru_bulk_move lru_bulk_move; +u64 size;struct rb_root vmas;
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 9/30 drm/xe/guc: Read HXG fields from DW1 of G2H response
In reply to Matthew BrostThe HXG fields are DW1 not DW0, fix this.
--- drivers/gpu/drm/xe/xe_guc_ct.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/xe/xe_guc_ct.c +3 -3
@@ -782,13 +782,13 @@ static int parse_g2h_response(struct xe_guc_ct *ct, u32 *msg, u32 len)if (type ==GUC_HXG_TYPE_RESPONSE_FAILURE ) { g2h_fence->fail = true; g2h_fence->error = -FIELD_GET (GUC_HXG_FAILURE_MSG_0_ERROR , msg[0 ]); +FIELD_GET (GUC_HXG_FAILURE_MSG_0_ERROR , msg[1 ]); g2h_fence->hint = -FIELD_GET (GUC_HXG_FAILURE_MSG_0_HINT , msg[0 ]); +FIELD_GET (GUC_HXG_FAILURE_MSG_0_HINT , msg[1 ]); }else if (type ==GUC_HXG_TYPE_NO_RESPONSE_RETRY ) { g2h_fence->retry = true; g2h_fence->reason = -FIELD_GET (GUC_HXG_RETRY_MSG_0_REASON , msg[0 ]); +FIELD_GET (GUC_HXG_RETRY_MSG_0_REASON , msg[1 ]); }else if (g2h_fence->response_buffer) { g2h_fence->response_len = response_len; memcpy(g2h_fence->response_buffer, msg + GUC_CTB_MSG_MIN_LEN,
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 10/30 drm/xe/guc: Return the lower part of blocking H2G message
In reply to Matthew BrostThe upper layers may need this data, an example of this is allocating DIST doorbell.
--- drivers/gpu/drm/xe/xe_guc_ct.c | 6 +++++- drivers/gpu/drm/xe/xe_guc_pc.c | 6 ++++-- drivers/gpu/drm/xe/xe_huc.c | 2 +- 3 files changed, 10 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/xe_guc_ct.c +5 -1
@@ -25,6 +25,7 @@struct g2h_fence {u32 *response_buffer;u32 seqno; +u32 status;u16 response_len;u16 error;u16 hint;
@@ -727,7 +728,7 @@ static int guc_ct_send_recv(struct xe_guc_ct *ct, const u32 *action, u32 len, ret = -EIO ; } -return ret >0 ?0 : ret; +return ret >0 ? g2h_fence.status : ret; } int xe_guc_ct_send_recv(struct xe_guc_ct *ct, const u32 *action, u32 len,
@@ -793,6 +794,9 @@ static int parse_g2h_response(struct xe_guc_ct *ct, u32 *msg, u32 len) g2h_fence->response_len = response_len;memcpy (g2h_fence->response_buffer, msg +GUC_CTB_MSG_MIN_LEN , response_len *sizeof (u32)); + }else { + g2h_fence->status = +FIELD_GET (GUC_HXG_RESPONSE_MSG_0_DATA0 , msg[1 ]); }g2h_release_space (ct,GUC_CTB_HXG_MSG_MAX_LEN );
drivers/gpu/drm/xe/xe_guc_pc.c +4 -2
@@ -204,11 +204,13 @@ static int pc_action_query_task_state(struct xe_guc_pc *pc)/* Blocking here to ensure the results are ready before reading them */ ret =xe_guc_ct_send_block (ct, action,ARRAY_SIZE (action)); -if (ret) +if (ret <0 ) {drm_err (&pc_to_xe (pc)->drm,"GuC PC query task state failed: %pe" ,ERR_PTR (ret)); +return ret; + } -return ret; +return 0 ; }static int pc_action_set_param (struct xe_guc_pc *pc,u8 id,u32 value)
drivers/gpu/drm/xe/xe_huc.c +1 -1
@@ -39,7 +39,7 @@ int xe_huc_init(struct xe_huc *huc) huc->fw.type =XE_UC_FW_TYPE_HUC ; ret =xe_uc_fw_init (&huc->fw); -if (ret) +if (ret <0 ) goto out;xe_uc_fw_change_status (&huc->fw,XE_UC_FIRMWARE_LOADABLE );
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 11/30 drm/xe/guc: Use doorbells for submission if possible
In reply to Matthew BrostWe have 256 doorbells (on most platforms) that we can allocate to bypass using the H2G channel for submission. This will avoid contention on the CT mutex.
--- drivers/gpu/drm/xe/regs/xe_guc_regs.h | 1 + drivers/gpu/drm/xe/xe_guc.c | 6 + drivers/gpu/drm/xe/xe_guc_engine_types.h | 7 + drivers/gpu/drm/xe/xe_guc_submit.c | 295 ++++++++++++++++++++++- drivers/gpu/drm/xe/xe_guc_submit.h | 1 + drivers/gpu/drm/xe/xe_guc_types.h | 4 + drivers/gpu/drm/xe/xe_trace.h | 5 + 7 files changed, 315 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/regs/xe_guc_regs.h +1 -0
@@ -109,6 +109,7 @@ struct guc_doorbell_info {#define DIST_DBS_POPULATED XE_REG(0xd08)#define DOORBELLS_PER_SQIDI_MASK REG_GENMASK(23, 16) +#define DOORBELLS_PER_SQIDI_SHIFT 16#define SQIDIS_DOORBELL_EXIST_MASK REG_GENMASK(15, 0)#define GUC_BCS_RCS_IER XE_REG(0xC550)
drivers/gpu/drm/xe/xe_guc.c +6 -0
@@ -297,6 +297,12 @@ int xe_guc_init(struct xe_guc *guc)*/ int xe_guc_init_post_hwconfig (struct xe_guc *guc) { +int ret; + + ret =xe_guc_submit_init_post_hwconfig (guc); +if (ret) +return ret; +return xe_guc_ads_init_post_hwconfig (&guc->ads); }
drivers/gpu/drm/xe/xe_guc_engine_types.h +7 -0
@@ -12,6 +12,7 @@#include <drm/gpu_scheduler.h> struct dma_fence ; +struct xe_bo ;struct xe_engine ;/**
@@ -37,6 +38,10 @@ struct xe_guc_engine {struct work_struct fini_async;/** @resume_time: time of last resume */ u64 resume_time; +/** @doorbell_bo: BO for memory doorbell */ +struct xe_bo *doorbell_bo; +/** @doorbell_offset: MMIO doorbell offset */ +u32 doorbell_offset;/** @state: GuC specific state for this xe_engine */ atomic_t state;/** @wqi_head: work queue item tail */
@@ -45,6 +50,8 @@ struct xe_guc_engine {u32 wqi_tail;/** @id: GuC id for this xe_engine */ u16 id; +/** @doorbell_id: doorbell id */ +u16 doorbell_id;/** @suspend_wait: wait queue used to wait on pending suspends */ wait_queue_head_t suspend_wait;/** @suspend_pending: a suspend of the engine is pending */
drivers/gpu/drm/xe/xe_guc_submit.c +291 -4
400 lines not on this page · raw message
drivers/gpu/drm/xe/xe_guc_submit.h +1 -0
@@ -13,6 +13,7 @@ struct xe_engine;struct xe_guc ;int xe_guc_submit_init (struct xe_guc *guc); +int xe_guc_submit_init_post_hwconfig (struct xe_guc *guc);void xe_guc_submit_print (struct xe_guc *guc,struct drm_printer *p);int xe_guc_submit_reset_prepare (struct xe_guc *guc);
drivers/gpu/drm/xe/xe_guc_types.h +4 -0
@@ -36,10 +36,14 @@ struct xe_guc {struct xarray engine_lookup;/** @guc_ids: used to allocate new guc_ids, single-lrc */ struct ida guc_ids; +/** @doorbell_ids: use to allocate new doorbells */ +struct ida doorbell_ids;/** @guc_ids_bitmap: used to allocate new guc_ids, multi-lrc */ unsigned long *guc_ids_bitmap;/** @stopped: submissions are stopped */ atomic_t stopped; +/** @num_doorbells: number of doorbels */ +int num_doorbells;/** @lock: protects submission state */ struct mutex lock;/** @suspend: suspend fence state */
drivers/gpu/drm/xe/xe_trace.h +5 -0
@@ -149,6 +149,11 @@ DEFINE_EVENT(xe_engine, xe_engine_submit,TP_ARGS (e) ); +DEFINE_EVENT (xe_engine, xe_engine_ring_db, + TP_PROTO(struct xe_engine *e), +TP_ARGS (e) +); +DEFINE_EVENT (xe_engine, xe_engine_scheduling_enable, TP_PROTO(struct xe_engine *e), TP_ARGS(e)
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 12/30 drm/xe/guc: Print doorbell ID in GuC engine debugfs entry
In reply to Matthew BrostThis information is helpful so print it.
--- drivers/gpu/drm/xe/xe_guc_submit.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/gpu/drm/xe/xe_guc_submit.c +2 -0
@@ -2016,6 +2016,8 @@ static void guc_engine_print(struct xe_engine *e, struct drm_printer *p)drm_printf (p,"\tTimeslice: %u (us)\n" , e->sched_props.timeslice_us);drm_printf (p,"\tPreempt timeout: %u (us)\n" , e->sched_props.preempt_timeout_us); +drm_printf (p,"\tDoorbell ID: %u\n" , + e->guc->doorbell_id);for (i =0 ; i < e->width; ++i ) {struct xe_lrc *lrc = e->lrc + i;
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 13/30 maple_tree: split up MA_STATE() macro
In reply to Matthew BrostFrom: Danilo Krummrich <[email protected]>
Split up the MA_STATE() macro such that components using the maple tree can easily inherit from struct ma_state and build custom tree walk macros to hide their internals from users.
Example:
struct sample_iterator {
struct ma_state mas;
struct sample_mgr *mgr;
};\#define SAMPLE_ITERATOR(name, __mgr, start) \
struct sample_iterator name = { \
.mas = MA_STATE_INIT(&(__mgr)->mt, start, 0), \
.mgr = __mgr, \
}\#define sample_iter_for_each_range(it__, entry__, end__) \
mas_for_each(&(it__).mas, entry__, end__)--
struct sample *sample;SAMPLE_ITERATOR(si, min);
sample_iter_for_each_range(&si, sample, max) {
frob(mgr, sample);
}--- include/linux/maple_tree.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
include/linux/maple_tree.h +5 -2
@@ -423,8 +423,8 @@ struct ma_wr_state {#define MA_ERROR (err) \ ((struct maple_enode *)(((unsigned long)err << 2) | 2UL)) -#define MA_STATE (name, mt, first, end) \ - struct ma_state name = { \ +#define MA_STATE_INIT (mt, first, end) \ + { \ .tree = mt, \ .index = first, \ .last = end, \
@@ -435,6 +435,9 @@ struct ma_wr_state { .mas_flags = 0, \ } +#define MA_STATE (name, mt, first, end) \ + struct ma_state name = MA_STATE_INIT(mt, first, end) +#define MA_WR_STATE (name, ma_state, wr_entry) \ struct ma_wr_state name = { \ .mas = ma_state, \
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 14/30 maple_tree: Export mas_preallocate
In reply to Matthew BrostThe DRM GPUVA implementation needs this function.
--- lib/maple_tree.c | 1 + 1 file changed, 1 insertion(+)
lib/maple_tree.c +1 -0
@@ -5726,6 +5726,7 @@ int mas_preallocate(struct ma_state *mas, gfp_t gfp)mas_reset (mas);return ret; } +EXPORT_SYMBOL_GPL (mas_preallocate);/* * mas_destroy() - destroy a maple state.
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 15/30 drm: manager to keep track of GPUs VA mappings
In reply to Matthew BrostFrom: Danilo Krummrich <[email protected]>
Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation.
New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context.
1) Provide infrastructure to track GPU VA allocations and mappings, making use of the maple_tree.
2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects.
3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests.
--- Documentation/gpu/drm-mm.rst | 31 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_debugfs.c | 41 + drivers/gpu/drm/drm_gem.c | 3 + drivers/gpu/drm/drm_gpuva_mgr.c | 1686 +++++++++++++++++++++++++++++++ include/drm/drm_debugfs.h | 24 + include/drm/drm_drv.h | 7 + include/drm/drm_gem.h | 75 ++ include/drm/drm_gpuva_mgr.h | 681 +++++++++++++ 9 files changed, 2549 insertions(+) create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c create mode 100644 include/drm/drm_gpuva_mgr.h
Documentation/gpu/drm-mm.rst +31 -0
@@ -466,6 +466,37 @@ DRM MM Range Allocator Function References .. kernel-doc:: drivers/gpu/drm/drm_mm.c :export: +DRM GPU VA Manager +================== + +Overview +-------- + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :doc: Overview + +Split and Merge +--------------- + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :doc: Split and Merge + +Locking +------- + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :doc: Locking + + +DRM GPU VA Manager Function References +-------------------------------------- + +.. kernel-doc:: include/drm/drm_gpuva_mgr.h + :internal: + +.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c + :export: + DRM Buddy Allocator ===================
drivers/gpu/drm/Makefile +1 -0
@@ -46,6 +46,7 @@ drm-y := \ drm_vblank.o \ drm_vblank_work.o \ drm_vma_manager.o \ + drm_gpuva_mgr.o \ drm_writeback.o drm-$(CONFIG_DRM_LEGACY) += \ drm_agpsupport.o \
drivers/gpu/drm/drm_debugfs.c +41 -0
53 lines not on this page · raw message
drivers/gpu/drm/drm_gem.c +3 -0
@@ -164,6 +164,9 @@ void drm_gem_private_object_init(struct drm_device *dev,if (!obj->resv) obj->resv = &obj->_resv; +if (drm_core_check_feature (dev,DRIVER_GEM_GPUVA )) +drm_gem_gpuva_init (obj); +drm_vma_node_reset (&obj->vma_node);INIT_LIST_HEAD (&obj->lru_node); }
drivers/gpu/drm/drm_gpuva_mgr.c new file +1686 -0
1,686 lines not on this page · raw message
include/drm/drm_debugfs.h +24 -0
@@ -34,6 +34,22 @@#include <linux/types.h> #include <linux/seq_file.h> + +#include <drm/drm_gpuva_mgr.h> + +/** +* DRM_DEBUGFS_GPUVA_INFO - &drm_info_list entry to dump a GPU VA space +* @show: the &drm_info_list's show callback +* @data: driver private data +* +* Drivers should use this macro to define a &drm_info_list entry to provide a +* debugfs file for dumping the GPU VA space regions and mappings. +* +* For each DRM GPU VA space drivers should call drm_debugfs_gpuva_info() from +* their @show callback. +*/ +#define DRM_DEBUGFS_GPUVA_INFO (show, data) {"gpuvas", show, DRIVER_GEM_GPUVA, data} +/** * struct drm_info_list - debugfs info list entry *
@@ -134,6 +150,8 @@ void drm_debugfs_add_file(struct drm_device *dev, const char *name,void drm_debugfs_add_files (struct drm_device *dev,const struct drm_debugfs_info *files,int count); +int drm_debugfs_gpuva_info (struct seq_file *m, +struct drm_gpuva_manager *mgr);#else static inline void drm_debugfs_create_files(const struct drm_info_list *files, int count, struct dentry *root,
@@ -155,6 +173,12 @@ static inline void drm_debugfs_add_files(struct drm_device *dev, const struct drm_debugfs_info *files, int count) {} + +static inline int drm_debugfs_gpuva_info (struct seq_file *m, +struct drm_gpuva_manager *mgr) +{ +return 0 ; +}#endif #endif /* _DRM_DEBUGFS_H_ */
include/drm/drm_drv.h +7 -0
@@ -105,6 +105,13 @@ enum drm_driver_feature {*/ DRIVER_COMPUTE_ACCEL =BIT (7 ), +/** +* @DRIVER_GEM_GPUVA: +* +* Driver supports user defined GPU VA bindings for GEM objects. +*/ + DRIVER_GEM_GPUVA = BIT(8), +/* IMPORTANT: Below are all the legacy flags, add new ones above. */ /**
include/drm/drm_gem.h +75 -0
91 lines not on this page · raw message
include/drm/drm_gpuva_mgr.h new file +681 -0
681 lines not on this page · raw message
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 16/30 drm/xe: Port Xe to GPUVA
In reply to Matthew BrostRather than open coding VM binds and VMA tracking, use the GPUVA library. GPUVA provides a common infrastructure for VM binds to use mmap / munmap semantics and support for VK sparse bindings.
The concepts are:
1) xe_vm inherits from drm_gpuva_manager
2) xe_vma inherits from drm_gpuva
3) xe_vma_op inherits from drm_gpuva_op
4) VM bind operations (MAP, UNMAP, PREFETCH, UNMAP_ALL) call into the GPUVA code to generate an VMA operations list which is parsed, commited, and executed.
v2 (CI): Add break after default in case statement.
v3: Rebase
--- drivers/gpu/drm/xe/xe_bo.c | 10 +- drivers/gpu/drm/xe/xe_device.c | 2 +- drivers/gpu/drm/xe/xe_exec.c | 2 +- drivers/gpu/drm/xe/xe_gt_pagefault.c | 23 +- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 14 +- drivers/gpu/drm/xe/xe_migrate.c | 8 +- drivers/gpu/drm/xe/xe_pt.c | 106 +- drivers/gpu/drm/xe/xe_trace.h | 10 +- drivers/gpu/drm/xe/xe_vm.c | 1847 +++++++++---------- drivers/gpu/drm/xe/xe_vm.h | 66 +- drivers/gpu/drm/xe/xe_vm_madvise.c | 88 +- drivers/gpu/drm/xe/xe_vm_types.h | 167 +- 12 files changed, 1168 insertions(+), 1175 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +5 -5
@@ -391,7 +391,8 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo, {struct dma_resv_iter cursor;struct dma_fence *fence; -struct xe_vma *vma; +struct drm_gpuva *gpuva; +struct drm_gem_object *obj = &bo->ttm.base;int ret =0 ;dma_resv_assert_held (bo->ttm.base.resv);
@@ -404,8 +405,9 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,dma_resv_iter_end (&cursor); } -list_for_each_entry (vma, &bo->vmas, bo_link) { -struct xe_vm *vm = vma->vm; +drm_gem_for_each_gpuva (gpuva, obj) { +struct xe_vma *vma =gpuva_to_vma (gpuva); +struct xe_vm *vm =xe_vma_vm (vma);trace_xe_vma_evict (vma);
@@ -430,10 +432,8 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo, }else { ret = timeout; } - }else {bool vm_resv_locked = false; -struct xe_vm *vm = vma->vm;/* * We need to put the vma on the vm's rebind_list,
drivers/gpu/drm/xe/xe_device.c +1 -1
@@ -131,7 +131,7 @@ static struct drm_driver driver = { .driver_features =DRIVER_GEM |DRIVER_RENDER |DRIVER_SYNCOBJ | - DRIVER_SYNCOBJ_TIMELINE, + DRIVER_SYNCOBJ_TIMELINE | DRIVER_GEM_GPUVA, .open = xe_file_open, .postclose = xe_file_close,
drivers/gpu/drm/xe/xe_exec.c +1 -1
@@ -119,7 +119,7 @@ static int xe_exec_begin(struct xe_engine *e, struct ww_acquire_ctx *ww,if (xe_vma_is_userptr (vma))continue ; - err =xe_bo_validate (vma->bo, vm, false); + err =xe_bo_validate (xe_vma_bo (vma), vm, false);if (err) {xe_vm_unlock_dma_resv (vm, tv_onstack, *tv, ww, objs); *tv =NULL ;
drivers/gpu/drm/xe/xe_gt_pagefault.c +9 -14
@@ -75,9 +75,10 @@ static bool vma_is_valid(struct xe_gt *gt, struct xe_vma *vma) !(BIT (gt->info.id) & vma->usm.gt_invalidated); } -static bool vma_matches (struct xe_vma *vma,struct xe_vma *lookup) +static bool vma_matches (struct xe_vma *vma,u64 page_addr) { -if (lookup->start > vma->end || lookup->end < vma->start) +if (page_addr >xe_vma_end (vma) -1 || + page_addr +SZ_4K <xe_vma_start (vma))return false;return true;
@@ -90,16 +91,14 @@ static bool only_needs_bo_lock(struct xe_bo *bo)static struct xe_vma *lookup_vma (struct xe_vm *vm,u64 page_addr) { -struct xe_vma *vma =NULL , lookup; +struct xe_vma *vma =NULL ; - lookup.start = page_addr; - lookup.end = lookup.start +SZ_4K -1 ;if (vm->usm.last_fault_vma) {/* Fast lookup */ -if (vma_matches (vm->usm.last_fault_vma, &lookup)) +if (vma_matches (vm->usm.last_fault_vma, page_addr)) vma = vm->usm.last_fault_vma; }if (!vma) - vma =xe_vm_find_overlapping_vma (vm, &lookup); + vma =xe_vm_find_overlapping_vma (vm, page_addr,SZ_4K );return vma; }
@@ -170,7 +169,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) }/* Lock VM and BOs dma-resv */ - bo = vma->bo; + bo =xe_vma_bo (vma);if (only_needs_bo_lock (bo)) {/* This path ensures the BO's LRU is updated */ ret =xe_bo_lock (bo, &ww, xe->info.tile_count, false);
@@ -487,12 +486,8 @@ static struct xe_vma *get_acc_vma(struct xe_vm *vm, struct acc *acc) {u64 page_va = acc->va_range_base + (ffs (acc->sub_granularity) -1 ) *sub_granularity_in_byte (acc->granularity); -struct xe_vma lookup; - - lookup.start = page_va; - lookup.end = lookup.start +SZ_4K -1 ; -return xe_vm_find_overlapping_vma (vm, &lookup); +return xe_vm_find_overlapping_vma (vm, page_va,SZ_4K ); }static int handle_acc (struct xe_gt *gt,struct acc *acc)
@@ -536,7 +531,7 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc) goto unlock_vm;/* Lock VM and BOs dma-resv */ - bo = vma->bo; + bo =xe_vma_bo (vma);if (only_needs_bo_lock (bo)) {/* This path ensures the BO's LRU is updated */ ret =xe_bo_lock (bo, &ww, xe->info.tile_count, false);
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +7 -7
@@ -201,8 +201,8 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,if (!xe->info.has_range_tlb_invalidation) { action[len++] =MAKE_INVAL_OP (XE_GUC_TLB_INVAL_FULL ); }else { -u64 start = vma->start; -u64 length = vma->end - vma->start +1 ; +u64 start =xe_vma_start (vma); +u64 length =xe_vma_size (vma);u64 align, end;if (length <SZ_4K )
@@ -215,12 +215,12 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,* address mask covering the required range. */ align =roundup_pow_of_two (length); - start =ALIGN_DOWN (vma->start, align); - end =ALIGN (vma->start + length, align); + start =ALIGN_DOWN (xe_vma_start (vma), align); + end =ALIGN (xe_vma_start (vma) + length, align); length = align;while (start + length < end) { length <<=1 ; - start =ALIGN_DOWN (vma->start, length); + start =ALIGN_DOWN (xe_vma_start (vma), length); }/*
@@ -229,7 +229,7 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,*/ if (length >=SZ_2M ) { length =max_t (u64,SZ_16M , length); - start =ALIGN_DOWN (vma->start, length); + start =ALIGN_DOWN (xe_vma_start (vma), length); }XE_BUG_ON (length <SZ_4K );
@@ -238,7 +238,7 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,XE_BUG_ON (!IS_ALIGNED (start, length)); action[len++] =MAKE_INVAL_OP (XE_GUC_TLB_INVAL_PAGE_SELECTIVE ); - action[len++] = vma->vm->usm.asid; + action[len++] =xe_vma_vm (vma)->usm.asid; action[len++] =lower_32_bits (start); action[len++] =upper_32_bits (start); action[len++] =ilog2 (length) -ilog2 (SZ_4K );
drivers/gpu/drm/xe/xe_migrate.c +6 -2
@@ -1050,8 +1050,10 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m,return ERR_PTR (-ETIME );if (wait_vm && !dma_resv_test_signaled (&vm->resv, -DMA_RESV_USAGE_BOOKKEEP )) +DMA_RESV_USAGE_BOOKKEEP )) { +vm_dbg (&vm->xe->drm,"wait on VM for munmap" );return ERR_PTR (-ETIME ); + }if (ops->pre_commit) { err = ops->pre_commit (pt_update);
@@ -1139,7 +1141,8 @@ xe_migrate_update_pgtables(struct xe_migrate *m,u64 addr;int err =0 ;bool usm = !eng && xe->info.supports_usm; -bool first_munmap_rebind = vma && vma->first_munmap_rebind; +bool first_munmap_rebind = vma && + vma->gpuva.flags &XE_VMA_FIRST_REBIND ;/* Use the CPU if no in syncs and engine is idle */ if (no_in_syncs (syncs, num_syncs) && (!eng ||xe_engine_is_idle (eng))) {
@@ -1260,6 +1263,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,* trigger preempts before moving forward */ if (first_munmap_rebind) { +vm_dbg (&vm->xe->drm,"wait on first_munmap_rebind" ); err =job_add_deps (job, &vm->resv,DMA_RESV_USAGE_BOOKKEEP );if (err)
drivers/gpu/drm/xe/xe_pt.c +55 -51
288 lines not on this page · raw message
drivers/gpu/drm/xe/xe_trace.h +5 -5
@@ -18,7 +18,7 @@#include "xe_gt_types.h" #include "xe_guc_engine_types.h" #include "xe_sched_job.h" -#include "xe_vm_types.h" +#include "xe_vm.h" DECLARE_EVENT_CLASS(xe_gt_tlb_invalidation_fence, TP_PROTO(struct xe_gt_tlb_invalidation_fence *fence),
@@ -378,10 +378,10 @@ DECLARE_EVENT_CLASS(xe_vma,TP_fast_assign ( __entry->vma = (unsigned long)vma; - __entry->asid = vma->vm->usm.asid; - __entry->start = vma->start; - __entry->end = vma->end; - __entry->ptr = (u64 )vma->userptr.ptr; + __entry->asid =xe_vma_vm (vma)->usm.asid; + __entry->start =xe_vma_start (vma); + __entry->end =xe_vma_end (vma) -1 ; + __entry->ptr =xe_vma_userptr (vma); ), TP_printk("vma=0x%016llx, asid=0x%05x, start=0x%012llx, end=0x%012llx, ptr=0x%012llx,",
drivers/gpu/drm/xe/xe_vm.c +882 -965
2,471 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +63 -3
45 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_madvise.c +21 -67
88 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +113 -54
167 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1
Part of the diff is not shown here. View the raw message.
Matthew Brost
maintainerPATCH 17/30 drm/xe: NULL binding implementation
In reply to Matthew BrostAdd uAPI and implementation for NULL bindings. A NULL binding is defined as writes dropped and read zero. A single bit in the uAPI has been added which results in a single bit in the PTEs being set.
NULL bindings are indended to be used to implement VK sparse bindings.
--- drivers/gpu/drm/xe/xe_bo.h | 1 + drivers/gpu/drm/xe/xe_exec.c | 2 + drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +- drivers/gpu/drm/xe/xe_pt.c | 77 ++++++++++++++++------- drivers/gpu/drm/xe/xe_vm.c | 92 ++++++++++++++++++---------- drivers/gpu/drm/xe/xe_vm.h | 10 +++ drivers/gpu/drm/xe/xe_vm_madvise.c | 2 +- drivers/gpu/drm/xe/xe_vm_types.h | 3 + include/uapi/drm/xe_drm.h | 8 +++ 9 files changed, 144 insertions(+), 55 deletions(-)
drivers/gpu/drm/xe/xe_bo.h +1 -0
@@ -56,6 +56,7 @@#define XE_PDE_IPS_64K BIT_ULL(11)#define XE_GGTT_PTE_LM BIT_ULL(1) +#define XE_PTE_NULL BIT_ULL(9)#define XE_USM_PPGTT_PTE_AE BIT_ULL(10)#define XE_PPGTT_PTE_LM BIT_ULL(11)#define XE_PDE_64K BIT_ULL(6)
drivers/gpu/drm/xe/xe_exec.c +2 -0
@@ -116,6 +116,8 @@ static int xe_exec_begin(struct xe_engine *e, struct ww_acquire_ctx *ww,* to a location where the GPU can access it). */ list_for_each_entry (vma, &vm->rebind_list, rebind_link) { +XE_BUG_ON (xe_vma_is_null (vma)); +if (xe_vma_is_userptr (vma))continue ;
drivers/gpu/drm/xe/xe_gt_pagefault.c +2 -2
@@ -526,8 +526,8 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc)trace_xe_vma_acc (vma); -/* Userptr can't be migrated, nothing to do */ -if (xe_vma_is_userptr (vma)) +/* Userptr or null can't be migrated, nothing to do */ +if (xe_vma_has_no_bo (vma)) goto unlock_vm;/* Lock VM and BOs dma-resv */
drivers/gpu/drm/xe/xe_pt.c +56 -21
157 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.c +61 -31
241 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +10 -0
@@ -175,7 +175,17 @@ static inline void xe_vm_reactivate_rebind(struct xe_vm *vm) } } +static inline bool xe_vma_is_null (struct xe_vma *vma) +{ +return vma->gpuva.flags &XE_VMA_NULL ; +} +static inline bool xe_vma_is_userptr (struct xe_vma *vma) +{ +return !xe_vma_bo (vma) && !xe_vma_is_null (vma); +} + +static inline bool xe_vma_has_no_bo (struct xe_vma *vma) {return !xe_vma_bo (vma); }
drivers/gpu/drm/xe/xe_vm_madvise.c +1 -1
@@ -227,7 +227,7 @@ get_vmas(struct xe_vm *vm, int *num_vmas, u64 addr, u64 range)drm_gpuva_iter_for_each_range (gpuva, it, addr + range) {struct xe_vma *vma =gpuva_to_vma (gpuva); -if (xe_vma_is_userptr (vma)) +if (xe_vma_has_no_bo (vma))continue ;if (*num_vmas == max_vmas) {
drivers/gpu/drm/xe/xe_vm_types.h +3 -0
@@ -29,6 +29,7 @@ struct xe_vm;#define XE_VMA_ATOMIC_PTE_BIT (DRM_GPUVA_USERBITS << 2)#define XE_VMA_FIRST_REBIND (DRM_GPUVA_USERBITS << 3)#define XE_VMA_LAST_REBIND (DRM_GPUVA_USERBITS << 4) +#define XE_VMA_NULL (DRM_GPUVA_USERBITS << 5)struct xe_vma {/** @gpuva: Base GPUVA object */
@@ -315,6 +316,8 @@ struct xe_vma_op_map {bool immediate;/** @read_only: Read only */ bool read_only; +/** @null: NULL (writes dropped, read zero) */ +bool null; };/** struct xe_vma_op_unmap - VMA unmap operation */
include/uapi/drm/xe_drm.h +8 -0
@@ -447,6 +447,14 @@ struct drm_xe_vm_bind_op {* than differing the MAP to the page fault handler. */ #define XE_VM_BIND_FLAG_IMMEDIATE (0x1 << 18) +/* +* When the NULL flag is set, the page tables are setup with a special +* bit which indicates writes are dropped and all reads return zero. The +* NULL flags is only valid for XE_VM_BIND_OP_MAP operations, the BO +* handle MBZ, and the BO offset MBZ. This flag is intended to implement +* VK sparse bindings. +*/ +#define XE_VM_BIND_FLAG_NULL (0x1 << 19)/** @reserved: Reserved */ __u64 reserved[2 ];
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 18/30 drm/xe: Avoid doing rebinds
In reply to Matthew BrostIf we dont change page sizes we can avoid doing rebinds rather just do a partial unbind. The algorithm to determine is page size is greedy as we assume all pages in the removed VMA are the largest page used in the VMA.
--- drivers/gpu/drm/xe/xe_pt.c | 4 ++ drivers/gpu/drm/xe/xe_vm.c | 71 +++++++++++++++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 17 ++++---- 3 files changed, 67 insertions(+), 25 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +4 -0
@@ -412,6 +412,8 @@ struct xe_pt_stage_bind_walk {/* Input parameters for the walk */ /** @vm: The vm we're building for. */ struct xe_vm *vm; +/** @vma: The vma we are binding for. */ +struct xe_vma *vma;/** @gt: The gt we're building for. */ struct xe_gt *gt;/** @cache: Desired cache level for the ptes */
@@ -688,6 +690,7 @@ xe_pt_stage_bind_entry(struct drm_pt *parent, pgoff_t offset,if (!null)xe_res_next (curs, next - addr); xe_walk->va_curs_start = next; + xe_walk->vma->gpuva.flags |= (XE_VMA_PTE_4K << level); *action =ACTION_CONTINUE ;return ret;
@@ -776,6 +779,7 @@ xe_pt_stage_bind(struct xe_gt *gt, struct xe_vma *vma, .max_level = XE_PT_HIGHEST_LEVEL, }, .vm = xe_vma_vm(vma), + .vma = vma, .gt = gt, .curs = &curs, .va_curs_start = xe_vma_start(vma),
drivers/gpu/drm/xe/xe_vm.c +56 -15
113 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +7 -10
@@ -30,6 +30,9 @@ struct xe_vm;#define XE_VMA_FIRST_REBIND (DRM_GPUVA_USERBITS << 3)#define XE_VMA_LAST_REBIND (DRM_GPUVA_USERBITS << 4)#define XE_VMA_NULL (DRM_GPUVA_USERBITS << 5) +#define XE_VMA_PTE_4K (DRM_GPUVA_USERBITS << 6) +#define XE_VMA_PTE_2M (DRM_GPUVA_USERBITS << 7) +#define XE_VMA_PTE_1G (DRM_GPUVA_USERBITS << 8)struct xe_vma {/** @gpuva: Base GPUVA object */
@@ -320,14 +323,6 @@ struct xe_vma_op_map {bool null; }; -/** struct xe_vma_op_unmap - VMA unmap operation */ -struct xe_vma_op_unmap { -/** @start: start of the VMA unmap */ -u64 start; -/** @range: range of the VMA unmap */ -u64 range; -}; -/** struct xe_vma_op_remap - VMA remap operation */ struct xe_vma_op_remap {/** @prev: VMA preceding part of a split mapping */
@@ -338,6 +333,10 @@ struct xe_vma_op_remap {u64 start;/** @range: range of the VMA unmap */ u64 range; +/** @skip_prev: skip prev rebind */ +bool skip_prev; +/** @skip_next: skip next rebind */ +bool skip_next;/** @unmap_done: unmap operation in done */ bool unmap_done; };
@@ -395,8 +394,6 @@ struct xe_vma_op {union {/** @map: VMA map operation specific data */ struct xe_vma_op_map map; -/** @unmap: VMA unmap operation specific data */ -struct xe_vma_op_unmap unmap;/** @remap: VMA remap operation specific data */ struct xe_vma_op_remap remap;/** @prefetch: VMA prefetch operation specific data */
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 19/30 drm/xe: Reduce the number list links in xe_vma
In reply to Matthew Brost5 list links in can be squashed into a union in xe_vma as being on the various list is mutually exclusive.
--- drivers/gpu/drm/xe/xe_gt_pagefault.c | 2 +- drivers/gpu/drm/xe/xe_pt.c | 5 +- drivers/gpu/drm/xe/xe_vm.c | 29 ++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 71 +++++++++++++++------------- 4 files changed, 55 insertions(+), 52 deletions(-)
drivers/gpu/drm/xe/xe_gt_pagefault.c +1 -1
@@ -157,7 +157,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf)if (xe_vma_is_userptr (vma) && write_locked) {spin_lock (&vm->userptr.invalidated_lock); -list_del_init (&vma->userptr.invalidate_link); +list_del_init (&vma->invalidate_link);spin_unlock (&vm->userptr.invalidated_lock); ret =xe_vma_userptr_pin_pages (vma);
drivers/gpu/drm/xe/xe_pt.c +2 -3
@@ -1116,8 +1116,7 @@ static int xe_pt_userptr_inject_eagain(struct xe_vma *vma) vma->userptr.divisor = divisor <<1 ;spin_lock (&vm->userptr.invalidated_lock); -list_move_tail (&vma->userptr.invalidate_link, - &vm->userptr.invalidated); +list_move_tail (&vma->invalidate_link, &vm->userptr.invalidated);spin_unlock (&vm->userptr.invalidated_lock);return true; }
@@ -1724,7 +1723,7 @@ __xe_pt_unbind_vma(struct xe_gt *gt, struct xe_vma *vma, struct xe_engine *e,if (!vma->gt_present) {spin_lock (&vm->userptr.invalidated_lock); -list_del_init (&vma->userptr.invalidate_link); +list_del_init (&vma->invalidate_link);spin_unlock (&vm->userptr.invalidated_lock); }up_read (&vm->userptr.notifier_lock);
drivers/gpu/drm/xe/xe_vm.c +14 -15
@@ -677,8 +677,7 @@ static bool vma_userptr_invalidate(struct mmu_interval_notifier *mni,if (!xe_vm_in_fault_mode (vm) && !(vma->gpuva.flags &XE_VMA_DESTROYED ) && vma->gt_present) {spin_lock (&vm->userptr.invalidated_lock); -list_move_tail (&vma->userptr.invalidate_link, - &vm->userptr.invalidated); +list_move_tail (&vma->invalidate_link, &vm->userptr.invalidated);spin_unlock (&vm->userptr.invalidated_lock); }
@@ -726,8 +725,8 @@ int xe_vm_userptr_pin(struct xe_vm *vm)/* Collect invalidated userptrs */ spin_lock (&vm->userptr.invalidated_lock);list_for_each_entry_safe (vma, next, &vm->userptr.invalidated, - userptr.invalidate_link) { -list_del_init (&vma->userptr.invalidate_link); + invalidate_link) { +list_del_init (&vma->invalidate_link);list_move_tail (&vma->userptr_link, &vm->userptr.repin_list); }spin_unlock (&vm->userptr.invalidated_lock);
@@ -830,12 +829,11 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm,return vma; } -/* FIXME: Way to many lists, should be able to reduce this */ +/* +* userptr_link, destroy_link, notifier.rebind_link, +* invalidate_link +*/ INIT_LIST_HEAD (&vma->rebind_link); -INIT_LIST_HEAD (&vma->unbind_link); -INIT_LIST_HEAD (&vma->userptr_link); -INIT_LIST_HEAD (&vma->userptr.invalidate_link); -INIT_LIST_HEAD (&vma->notifier.rebind_link);INIT_LIST_HEAD (&vma->extobj.link);INIT_LIST_HEAD (&vma->gpuva.gem.entry);
@@ -953,15 +951,14 @@ static void xe_vma_destroy(struct xe_vma *vma, struct dma_fence *fence)struct xe_vm *vm =xe_vma_vm (vma);lockdep_assert_held_write (&vm->lock); -XE_BUG_ON (!list_empty (&vma->unbind_link));if (xe_vma_is_userptr (vma)) {XE_WARN_ON (!(vma->gpuva.flags &XE_VMA_DESTROYED ));spin_lock (&vm->userptr.invalidated_lock); -list_del_init (&vma->userptr.invalidate_link); +if (!list_empty (&vma->invalidate_link)) +list_del_init (&vma->invalidate_link);spin_unlock (&vm->userptr.invalidated_lock); -list_del (&vma->userptr_link); }else if (!xe_vma_is_null (vma)) {xe_bo_assert_held (xe_vma_bo (vma));drm_gpuva_unlink (&vma->gpuva);
@@ -1328,7 +1325,9 @@ void xe_vm_close_and_put(struct xe_vm *vm)continue ; } -list_add_tail (&vma->unbind_link, &contested); +if (!list_empty (&vma->destroy_link)) +list_del_init (&vma->destroy_link); +list_add_tail (&vma->destroy_link, &contested); }/*
@@ -1356,8 +1355,8 @@ void xe_vm_close_and_put(struct xe_vm *vm)* Since we hold a refcount to the bo, we can remove and free * the members safely without locking. */ -list_for_each_entry_safe (vma, next_vma, &contested, unbind_link) { -list_del_init (&vma->unbind_link); +list_for_each_entry_safe (vma, next_vma, &contested, destroy_link) { +list_del_init (&vma->destroy_link);xe_vma_destroy_unlocked (vma); }
drivers/gpu/drm/xe/xe_vm_types.h +38 -33
93 lines not on this page · raw message
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 20/30 drm/xe: Optimize size of xe_vma allocation
In reply to Matthew BrostReduce gt_mask to a u8 from a u64, only allocate userptr state if VMA is a userptr, and union of destroy callback and worker.
--- drivers/gpu/drm/xe/xe_vm.c | 14 +++-- drivers/gpu/drm/xe/xe_vm_types.h | 88 +++++++++++++++++--------------- 2 files changed, 57 insertions(+), 45 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +9 -5
@@ -814,7 +814,7 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm,u64 bo_offset_or_userptr, u64 start, u64 end, bool read_only, bool null, - u64 gt_mask) + u8 gt_mask) { struct xe_vma *vma;struct xe_gt *gt;
@@ -823,7 +823,11 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm,XE_BUG_ON (start >= end);XE_BUG_ON (end >= vm->size); - vma =kzalloc (sizeof (*vma),GFP_KERNEL ); +if (!bo && !null)/* userptr */ + vma =kzalloc (sizeof (*vma),GFP_KERNEL ); +else + vma =kzalloc (sizeof (*vma) -sizeof (struct xe_userptr ), +GFP_KERNEL );if (!vma) { vma =ERR_PTR (-ENOMEM );return vma;
@@ -2149,7 +2153,7 @@ static void print_op(struct xe_device *xe, struct drm_gpuva_op *op)static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo,u64 bo_offset_or_userptr,u64 addr,u64 range, -u32 operation,u64 gt_mask,u32 region) +u32 operation,u8 gt_mask,u32 region) {struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww;
@@ -2234,7 +2238,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, }static struct xe_vma *new_vma (struct xe_vm *vm,struct drm_gpuva_op_map *op, -u64 gt_mask,bool read_only,bool null) +u8 gt_mask,bool read_only,bool null) {struct xe_bo *bo = op->gem.obj ?gem_to_xe_bo (op->gem.obj) :NULL ;struct xe_vma *vma;
@@ -3213,8 +3217,8 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u64 addr = bind_ops[i].addr;u32 op = bind_ops[i].op;u64 obj_offset = bind_ops[i].obj_offset; -u64 gt_mask = bind_ops[i].gt_mask;u32 region = bind_ops[i].region; +u8 gt_mask = bind_ops[i].gt_mask; ops[i] = vm_bind_ioctl_ops_create(vm, bos[i], obj_offset, addr, range, op, gt_mask,
drivers/gpu/drm/xe/xe_vm_types.h +48 -40
121 lines not on this page · raw message
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 21/30 drm/gpuva: Add drm device to GPUVA manager
In reply to Matthew BrostThis is the logical place for this, will help with upcoming changes too.
--- drivers/gpu/drm/drm_gpuva_mgr.c | 3 +++ drivers/gpu/drm/xe/xe_migrate.c | 10 +++++----- drivers/gpu/drm/xe/xe_pt.c | 18 +++++++++--------- drivers/gpu/drm/xe/xe_vm.c | 31 +++++++++++++++---------------- drivers/gpu/drm/xe/xe_vm.h | 10 ++++++++++ drivers/gpu/drm/xe/xe_vm_types.h | 2 -- include/drm/drm_gpuva_mgr.h | 4 ++++ 7 files changed, 46 insertions(+), 32 deletions(-)
drivers/gpu/drm/drm_gpuva_mgr.c +3 -0
@@ -413,6 +413,7 @@ static void __drm_gpuva_remove(struct drm_gpuva *va);/** * drm_gpuva_manager_init - initialize a &drm_gpuva_manager * @mgr: pointer to the &drm_gpuva_manager to initialize +* @drm: drm device * @name: the name of the GPU VA space * @start_offset: the start offset of the GPU VA space * @range: the size of the GPU VA space
@@ -427,6 +428,7 @@ static void __drm_gpuva_remove(struct drm_gpuva *va);*/ void drm_gpuva_manager_init(struct drm_gpuva_manager *mgr, + struct drm_device *drm, const char *name, u64 start_offset, u64 range, u64 reserve_offset, u64 reserve_range,
@@ -437,6 +439,7 @@ drm_gpuva_manager_init(struct drm_gpuva_manager *mgr, mgr->mm_start = start_offset; mgr->mm_range = range; + mgr->drm = drm; mgr->name = name ? name :"unknown" ; mgr->ops = ops;
drivers/gpu/drm/xe/xe_migrate.c +5 -5
@@ -129,7 +129,7 @@ static u64 xe_migrate_vram_ofs(u64 addr)static int xe_migrate_create_cleared_bo (struct xe_migrate *m,struct xe_vm *vm) {struct xe_gt *gt = m->gt; -struct xe_device *xe = vm->xe; +struct xe_device *xe =xe_vm_device (vm);size_t cleared_size;u64 vram_addr;bool is_vram;
@@ -175,7 +175,7 @@ static int xe_migrate_prepare_vm(struct xe_gt *gt, struct xe_migrate *m,/* Need to be sure everything fits in the first PT, or create more */ XE_BUG_ON (m->batch_base_ofs + batch->size >=SZ_2M ); - bo = xe_bo_create_pin_map(vm->xe, m->gt, vm, + bo = xe_bo_create_pin_map(xe_vm_device(vm), m->gt, vm, num_entries * XE_PAGE_SIZE, ttm_bo_type_kernel, XE_BO_CREATE_VRAM_IF_DGFX(m->gt) |
@@ -1051,7 +1051,7 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m,if (wait_vm && !dma_resv_test_signaled (&vm->resv,DMA_RESV_USAGE_BOOKKEEP )) { -vm_dbg (&vm->xe->drm,"wait on VM for munmap" ); +vm_dbg (&xe_vm_device (vm)->drm,"wait on VM for munmap" );return ERR_PTR (-ETIME ); }
@@ -1069,7 +1069,7 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m,if (vm) {trace_xe_vm_cpu_bind (vm); -xe_device_wmb (vm->xe); +xe_device_wmb (xe_vm_device (vm)); } fence =dma_fence_get_stub ();
@@ -1263,7 +1263,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,* trigger preempts before moving forward */ if (first_munmap_rebind) { -vm_dbg (&vm->xe->drm,"wait on first_munmap_rebind" ); +vm_dbg (&xe_vm_device (vm)->drm,"wait on first_munmap_rebind" ); err =job_add_deps (job, &vm->resv,DMA_RESV_USAGE_BOOKKEEP );if (err)
drivers/gpu/drm/xe/xe_pt.c +9 -9
@@ -218,7 +218,7 @@ struct xe_pt *xe_pt_create(struct xe_vm *vm, struct xe_gt *gt,if (!pt)return ERR_PTR (-ENOMEM ); - bo = xe_bo_create_pin_map(vm->xe, gt, vm, SZ_4K, + bo = xe_bo_create_pin_map(xe_vm_device(vm), gt, vm, SZ_4K, ttm_bo_type_kernel, XE_BO_CREATE_VRAM_IF_DGFX(gt) | XE_BO_CREATE_IGNORE_MIN_PAGE_SIZE_BIT |
@@ -264,11 +264,11 @@ void xe_pt_populate_empty(struct xe_gt *gt, struct xe_vm *vm,* FIXME: Some memory is allocated already allocated to zero? * Find out which memory that is and avoid this memset... */ -xe_map_memset (vm->xe, map,0 ,0 ,SZ_4K ); +xe_map_memset (xe_vm_device (vm), map,0 ,0 ,SZ_4K ); }else { empty =__xe_pt_empty_pte (gt, vm, pt->level);for (i =0 ; i <XE_PDES ; i++) -xe_pt_write (vm->xe, map, i, empty); +xe_pt_write (xe_vm_device (vm), map, i, empty); } }
@@ -355,7 +355,7 @@ int xe_pt_create_scratch(struct xe_device *xe, struct xe_gt *gt,if (IS_ERR (vm->scratch_bo[id]))return PTR_ERR (vm->scratch_bo[id]); -xe_map_memset (vm->xe, &vm->scratch_bo[id]->vmap,0 ,0 , +xe_map_memset (xe_vm_device (vm), &vm->scratch_bo[id]->vmap,0 ,0 , vm->scratch_bo[id]->size);for (i =0 ; i < vm->pt_root[id]->level; i++) {
@@ -538,7 +538,7 @@ xe_pt_insert_entry(struct xe_pt_stage_bind_walk *xe_walk, struct xe_pt *parent,if (unlikely (xe_child)) parent->drm.dir->entries[offset] = &xe_child->drm; -xe_pt_write (xe_walk->vm->xe, map, offset, pte); +xe_pt_write (xe_vm_device (xe_walk->vm), map, offset, pte); parent->num_live++; }else {/* Shared pt. Stage update. */
@@ -1337,7 +1337,7 @@ __xe_pt_bind_vma(struct xe_gt *gt, struct xe_vma *vma, struct xe_engine *e,xe_vm_assert_held (vm);XE_BUG_ON (xe_gt_is_media_type (gt)); -vm_dbg (&xe_vma_vm (vma)->xe->drm, +vm_dbg (&xe_vma_device (vma)->drm,"Preparing bind, with range [%llx...%llx) engine %p.\n" ,xe_vma_start (vma),xe_vma_end (vma) -1 , e);
@@ -1366,7 +1366,7 @@ __xe_pt_bind_vma(struct xe_gt *gt, struct xe_vma *vma, struct xe_engine *e,if (last_munmap_rebind) -vm_dbg (&vm->xe->drm,"last_munmap_rebind" ); +vm_dbg (&xe_vm_device (vm)->drm,"last_munmap_rebind" );/* TLB invalidation must be done before signaling rebind */ if (rebind && !xe_vm_no_dma_fences (xe_vma_vm (vma))) {
@@ -1401,7 +1401,7 @@ __xe_pt_bind_vma(struct xe_gt *gt, struct xe_vma *vma, struct xe_engine *e,xe_bo_put_commit (&deferred); }if (!rebind && last_munmap_rebind &&xe_vm_in_compute_mode (vm)) -queue_work (vm->xe->ordered_wq, +queue_work (xe_vm_device (vm)->ordered_wq, &vm->preempt.rebind_work); }else {kfree (ifence);
@@ -1664,7 +1664,7 @@ __xe_pt_unbind_vma(struct xe_gt *gt, struct xe_vma *vma, struct xe_engine *e,xe_vm_assert_held (vm);XE_BUG_ON (xe_gt_is_media_type (gt)); -vm_dbg (&xe_vma_vm (vma)->xe->drm, +vm_dbg (&xe_vma_device (vma)->drm,"Preparing unbind, with range [%llx...%llx) engine %p.\n" ,xe_vma_start (vma),xe_vma_end (vma) -1 , e);
drivers/gpu/drm/xe/xe_vm.c +15 -16
@@ -50,7 +50,7 @@ int xe_vma_userptr_check_repin(struct xe_vma *vma)int xe_vma_userptr_pin_pages (struct xe_vma *vma) {struct xe_vm *vm =xe_vma_vm (vma); -struct xe_device *xe = vm->xe; +struct xe_device *xe =xe_vm_device (vm);const unsigned long num_pages =xe_vma_size (vma) >>PAGE_SHIFT ;struct page **pages;bool in_kthread = !current->mm;
@@ -852,12 +852,12 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm,if (gt_mask) { vma->gt_mask = gt_mask; }else { -for_each_gt (gt, vm->xe, id) +for_each_gt (gt,xe_vm_device (vm), id)if (!xe_gt_is_media_type (gt)) vma->gt_mask |=0x1 << id; } -if (vm->xe->info.platform ==XE_PVC ) +if (xe_vm_device (vm)->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;if (bo) {
@@ -904,7 +904,7 @@ static void vm_remove_extobj(struct xe_vma *vma)static void xe_vma_destroy_late (struct xe_vma *vma) {struct xe_vm *vm =xe_vma_vm (vma); -struct xe_device *xe = vm->xe; +struct xe_device *xe =xe_vm_device (vm);bool read_only =xe_vma_read_only (vma);if (xe_vma_is_userptr (vma)) {
@@ -1084,7 +1084,6 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)if (!vm)return ERR_PTR (-ENOMEM ); - vm->xe = xe;kref_init (&vm->refcount);dma_resv_init (&vm->resv);
@@ -1125,7 +1124,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)if (err) goto err_put; -drm_gpuva_manager_init (&vm->mgr,"Xe VM" ,0 , vm->size,0 ,0 , +drm_gpuva_manager_init (&vm->mgr, &xe->drm,"Xe VM" ,0 , vm->size,0 ,0 , &gpuva_ops);if (IS_DGFX (xe) && xe->info.vram_flags &XE_VRAM_FLAGS_NEED64K ) vm->flags |=XE_VM_FLAGS_64K ;
@@ -1284,7 +1283,7 @@ void xe_vm_close_and_put(struct xe_vm *vm) {struct list_head contested;struct ww_acquire_ctx ww; -struct xe_device *xe = vm->xe; +struct xe_device *xe =xe_vm_device (vm);struct xe_gt *gt;struct xe_vma *vma, *next_vma;struct drm_gpuva *gpuva;
@@ -1387,7 +1386,7 @@ static void vm_destroy_work_func(struct work_struct *w)struct xe_vm *vm = container_of(w, struct xe_vm, destroy_work);struct ww_acquire_ctx ww; -struct xe_device *xe = vm->xe; +struct xe_device *xe =xe_vm_device (vm);struct xe_gt *gt;u8 id;void *lookup;
@@ -1481,7 +1480,7 @@ xe_vm_unbind_vma(struct xe_vma *vma, struct xe_engine *e,return ERR_PTR (-ENOMEM ); } -for_each_gt (gt, vm->xe, id) { +for_each_gt (gt,xe_vm_device (vm), id) {if (!(vma->gt_present &BIT (id))) goto next;
@@ -1555,7 +1554,7 @@ xe_vm_bind_vma(struct xe_vma *vma, struct xe_engine *e,return ERR_PTR (-ENOMEM ); } -for_each_gt (gt, vm->xe, id) { +for_each_gt (gt,xe_vm_device (vm), id) {if (!(vma->gt_mask &BIT (id))) goto next;
@@ -2061,7 +2060,7 @@ static int vm_insert_extobj(struct xe_vm *vm, struct xe_vma *vma)static int vm_bind_ioctl_lookup_vma (struct xe_vm *vm,struct xe_bo *bo,u64 addr,u64 range,u32 op) { -struct xe_device *xe = vm->xe; +struct xe_device *xe =xe_vm_device (vm);struct xe_vma *vma;bool async = !!(op &XE_VM_BIND_FLAG_ASYNC );
@@ -2164,7 +2163,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,lockdep_assert_held_write (&vm->lock); -vm_dbg (&vm->xe->drm, +vm_dbg (&xe_vm_device (vm)->drm,"op=%d, addr=0x%016llx, range=0x%016llx, bo_offset_or_userptr=0x%016llx" ,VM_BIND_OP (operation), addr, range, bo_offset_or_userptr);
@@ -2232,7 +2231,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,if (!IS_ERR (ops))drm_gpuva_for_each_op (__op, ops) -print_op (vm->xe, __op); +print_op (xe_vm_device (vm), __op);return ops; }
@@ -2779,7 +2778,7 @@ static void xe_vma_op_work_func(struct work_struct *w)down_write (&vm->lock); err =xe_vma_op_execute (vm, op);if (err) { -drm_warn (&vm->xe->drm, +drm_warn (&xe_vm_device (vm)->drm,"Async VM op(%d) failed with %d" , op->base.op, err);vm_set_async_error (vm, err);
@@ -3099,7 +3098,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)/* Rebinds may have been blocked, give worker a kick */ if (xe_vm_in_compute_mode (vm)) -queue_work (vm->xe->ordered_wq, +queue_work (xe_vm_device (vm)->ordered_wq, &vm->preempt.rebind_work); }
@@ -3311,7 +3310,7 @@ void xe_vm_unlock(struct xe_vm *vm, struct ww_acquire_ctx *ww)*/ int xe_vm_invalidate_vma (struct xe_vma *vma) { -struct xe_device *xe =xe_vma_vm (vma)->xe; +struct xe_device *xe =xe_vm_device (xe_vma_vm (vma));struct xe_gt *gt;u32 gt_needs_invalidate =0 ;int seqno[XE_MAX_GT ];
drivers/gpu/drm/xe/xe_vm.h +10 -0
@@ -52,6 +52,11 @@ static inline bool xe_vm_is_closed(struct xe_vm *vm)struct xe_vma *xe_vm_find_overlapping_vma (struct xe_vm *vm,u64 start,u64 range); +static inline struct xe_device *xe_vm_device (struct xe_vm *vm) +{ + return container_of(vm->mgr.drm, struct xe_device, drm); +} +static inline struct xe_vm *gpuva_to_vm (struct drm_gpuva *gpuva) { return container_of(gpuva->mgr, struct xe_vm, mgr);
@@ -102,6 +107,11 @@ static inline struct xe_vm *xe_vma_vm(struct xe_vma *vma) return container_of(vma->gpuva.mgr, struct xe_vm, mgr); } +static inline struct xe_device *xe_vma_device (struct xe_vma *vma) +{ +return xe_vm_device (xe_vma_vm (vma)); +} +static inline bool xe_vma_read_only (struct xe_vma *vma) {return vma->gpuva.flags &XE_VMA_READ_ONLY ;
drivers/gpu/drm/xe/xe_vm_types.h +0 -2
@@ -137,8 +137,6 @@ struct xe_vm {/** @mgr: base GPUVA used to track VMAs */ struct drm_gpuva_manager mgr; -struct xe_device *xe; -struct kref refcount;/* engine used for (un)binding vma's */
include/drm/drm_gpuva_mgr.h +4 -0
@@ -169,6 +169,9 @@ static inline bool drm_gpuva_evicted(struct drm_gpuva *va)* There should be one manager instance per GPU virtual address space. */ struct drm_gpuva_manager { +/** @drm: drm device */ +struct drm_device *drm; +/** * @name: the name of the DRM GPU VA space */
@@ -204,6 +207,7 @@ struct drm_gpuva_manager { }; void drm_gpuva_manager_init(struct drm_gpuva_manager *mgr, + struct drm_device *drm, const char *name, u64 start_offset, u64 range, u64 reserve_offset, u64 reserve_range,
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 22/30 drm/gpuva: Move dma-resv to GPUVA manager
In reply to Matthew BrostLogical place for this, will help with upcoming patches.
--- drivers/gpu/drm/drm_gpuva_mgr.c | 2 ++ drivers/gpu/drm/xe/xe_bo.c | 10 +++++----- drivers/gpu/drm/xe/xe_bo.h | 2 +- drivers/gpu/drm/xe/xe_exec.c | 4 ++-- drivers/gpu/drm/xe/xe_migrate.c | 4 ++-- drivers/gpu/drm/xe/xe_pt.c | 6 +++--- drivers/gpu/drm/xe/xe_vm.c | 34 ++++++++++++++++---------------- drivers/gpu/drm/xe/xe_vm.h | 12 ++++++++++- drivers/gpu/drm/xe/xe_vm_types.h | 6 +----- include/drm/drm_gpuva_mgr.h | 6 ++++++ 10 files changed, 50 insertions(+), 36 deletions(-)
drivers/gpu/drm/drm_gpuva_mgr.c +2 -0
@@ -443,6 +443,8 @@ drm_gpuva_manager_init(struct drm_gpuva_manager *mgr, mgr->name = name ? name :"unknown" ; mgr->ops = ops; +dma_resv_init (&mgr->resv); +memset (&mgr->kernel_alloc_node,0 ,sizeof (struct drm_gpuva ));if (reserve_range) {
drivers/gpu/drm/xe/xe_bo.c +5 -5
@@ -441,9 +441,9 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,* that we indeed have it locked, put the vma an the * vm's notifier.rebind_list instead and scoop later. */ -if (dma_resv_trylock (&vm->resv)) +if (dma_resv_trylock (xe_vm_resv (vm))) vm_resv_locked = true; -else if (ctx->resv != &vm->resv) { +else if (ctx->resv !=xe_vm_resv (vm)) {spin_lock (&vm->notifier.list_lock);list_move_tail (&vma->notifier.rebind_link, &vm->notifier.rebind_list);
@@ -456,7 +456,7 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,list_add_tail (&vma->rebind_link, &vm->rebind_list);if (vm_resv_locked) -dma_resv_unlock (&vm->resv); +dma_resv_unlock (xe_vm_resv (vm)); } }
@@ -1240,7 +1240,7 @@ xe_bo_create_locked_range(struct xe_device *xe, } } - bo = __xe_bo_create_locked(xe, bo, gt, vm ? &vm->resv : NULL, + bo = __xe_bo_create_locked(xe, bo, gt, vm ? xe_vm_resv(vm) : NULL, vm && !xe_vm_no_dma_fences(vm) && flags & XE_BO_CREATE_USER_BIT ? &vm->lru_bulk_move : NULL, size,
@@ -1555,7 +1555,7 @@ int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict)xe_vm_assert_held (vm); ctx.allow_res_evict = allow_res_evict; - ctx.resv = &vm->resv; + ctx.resv =xe_vm_resv (vm); }return ttm_bo_validate (&bo->ttm, &bo->placement, &ctx);
drivers/gpu/drm/xe/xe_bo.h +1 -1
@@ -150,7 +150,7 @@ void xe_bo_unlock(struct xe_bo *bo, struct ww_acquire_ctx *ww);static inline void xe_bo_unlock_vm_held (struct xe_bo *bo) {if (bo) { -XE_BUG_ON (bo->vm && bo->ttm.base.resv != &bo->vm->resv); +XE_BUG_ON (bo->vm && bo->ttm.base.resv != &bo->vm->mgr.resv);if (bo->vm)xe_vm_assert_held (bo->vm);else
drivers/gpu/drm/xe/xe_exec.c +2 -2
@@ -327,7 +327,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)/* Wait behind munmap style rebinds */ if (!xe_vm_no_dma_fences (vm)) { err =drm_sched_job_add_resv_dependencies (&job->drm, - &vm->resv, +xe_vm_resv (vm),DMA_RESV_USAGE_KERNEL );if (err) goto err_put_job;
@@ -355,7 +355,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)xe_sched_job_arm (job);if (!xe_vm_no_dma_fences (vm)) {/* Block userptr invalidations / BO eviction */ -dma_resv_add_fence (&vm->resv, +dma_resv_add_fence (xe_vm_resv (vm), &job->drm.s_fence->finished,DMA_RESV_USAGE_BOOKKEEP );
drivers/gpu/drm/xe/xe_migrate.c +2 -2
@@ -1049,7 +1049,7 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m, DMA_RESV_USAGE_KERNEL))return ERR_PTR (-ETIME ); -if (wait_vm && !dma_resv_test_signaled (&vm->resv, +if (wait_vm && !dma_resv_test_signaled (xe_vm_resv (vm),DMA_RESV_USAGE_BOOKKEEP )) {vm_dbg (&xe_vm_device (vm)->drm,"wait on VM for munmap" );return ERR_PTR (-ETIME );
@@ -1264,7 +1264,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,*/ if (first_munmap_rebind) {vm_dbg (&xe_vm_device (vm)->drm,"wait on first_munmap_rebind" ); - err =job_add_deps (job, &vm->resv, + err =job_add_deps (job,xe_vm_resv (vm),DMA_RESV_USAGE_BOOKKEEP );if (err) goto err_job;
drivers/gpu/drm/xe/xe_pt.c +3 -3
@@ -1020,7 +1020,7 @@ static void xe_pt_commit_locks_assert(struct xe_vma *vma)else if (!xe_vma_is_null (vma))dma_resv_assert_held (xe_vma_bo (vma)->ttm.base.resv); -dma_resv_assert_held (&vm->resv); +dma_resv_assert_held (xe_vm_resv (vm)); } static void xe_pt_commit_bind(struct xe_vma *vma,
@@ -1381,7 +1381,7 @@ __xe_pt_bind_vma(struct xe_gt *gt, struct xe_vma *vma, struct xe_engine *e, }/* add shared fence now for pagetable delayed destroy */ -dma_resv_add_fence (&vm->resv, fence, !rebind && +dma_resv_add_fence (xe_vm_resv (vm), fence, !rebind && last_munmap_rebind ?DMA_RESV_USAGE_KERNEL :DMA_RESV_USAGE_BOOKKEEP );
@@ -1701,7 +1701,7 @@ __xe_pt_unbind_vma(struct xe_gt *gt, struct xe_vma *vma, struct xe_engine *e, fence = &ifence->base.base;/* add shared fence now for pagetable delayed destroy */ -dma_resv_add_fence (&vm->resv, fence, +dma_resv_add_fence (xe_vm_resv (vm), fence,DMA_RESV_USAGE_BOOKKEEP );/* This fence will be installed by caller when doing eviction */
drivers/gpu/drm/xe/xe_vm.c +17 -17
@@ -307,7 +307,7 @@ static void resume_and_reinstall_preempt_fences(struct xe_vm *vm)list_for_each_entry (e, &vm->preempt.engines, compute.link) { e->ops->resume (e); -dma_resv_add_fence (&vm->resv, e->compute.pfence, +dma_resv_add_fence (xe_vm_resv (vm), e->compute.pfence,DMA_RESV_USAGE_BOOKKEEP );xe_vm_fence_all_extobjs (vm, e->compute.pfence,DMA_RESV_USAGE_BOOKKEEP );
@@ -345,7 +345,7 @@ int xe_vm_add_compute_engine(struct xe_vm *vm, struct xe_engine *e)down_read (&vm->userptr.notifier_lock); -dma_resv_add_fence (&vm->resv, pfence, +dma_resv_add_fence (xe_vm_resv (vm), pfence,DMA_RESV_USAGE_BOOKKEEP );xe_vm_fence_all_extobjs (vm, pfence,DMA_RESV_USAGE_BOOKKEEP );
@@ -603,7 +603,7 @@ static void preempt_rebind_work_func(struct work_struct *w) }/* Wait on munmap style VM unbinds */ - wait =dma_resv_wait_timeout (&vm->resv, + wait =dma_resv_wait_timeout (xe_vm_resv (vm),DMA_RESV_USAGE_KERNEL , false,MAX_SCHEDULE_TIMEOUT );if (wait <=0 ) {
@@ -689,13 +689,13 @@ static bool vma_userptr_invalidate(struct mmu_interval_notifier *mni,* unbinds to complete, and those are attached as BOOKMARK fences * to the vm. */ -dma_resv_iter_begin (&cursor, &vm->resv, +dma_resv_iter_begin (&cursor,xe_vm_resv (vm),DMA_RESV_USAGE_BOOKKEEP );dma_resv_for_each_fence_unlocked (&cursor, fence)dma_fence_enable_sw_signaling (fence);dma_resv_iter_end (&cursor); - err =dma_resv_wait_timeout (&vm->resv, + err =dma_resv_wait_timeout (xe_vm_resv (vm),DMA_RESV_USAGE_BOOKKEEP , false,MAX_SCHEDULE_TIMEOUT );XE_WARN_ON (err <=0 );
@@ -742,12 +742,12 @@ int xe_vm_userptr_pin(struct xe_vm *vm) }/* Take lock and move to rebind_list for rebinding. */ - err =dma_resv_lock_interruptible (&vm->resv,NULL ); + err =dma_resv_lock_interruptible (xe_vm_resv (vm),NULL );if (err) goto out_err;list_splice_tail (&tmp_evict, &vm->rebind_list); -dma_resv_unlock (&vm->resv); +dma_resv_unlock (xe_vm_resv (vm));return 0 ;
@@ -1085,7 +1085,6 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)return ERR_PTR (-ENOMEM );kref_init (&vm->refcount); -dma_resv_init (&vm->resv); vm->size =1ull <<xe_pt_shift (xe->info.vm_max_level +1 );
@@ -1120,12 +1119,13 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)xe_device_mem_access_get (xe); } - err =dma_resv_lock_interruptible (&vm->resv,NULL ); +drm_gpuva_manager_init (&vm->mgr, &xe->drm,"Xe VM" ,0 , vm->size,0 ,0 , + &gpuva_ops); + + err =dma_resv_lock_interruptible (xe_vm_resv (vm),NULL );if (err) goto err_put; -drm_gpuva_manager_init (&vm->mgr, &xe->drm,"Xe VM" ,0 , vm->size,0 ,0 , - &gpuva_ops);if (IS_DGFX (xe) && xe->info.vram_flags &XE_VRAM_FLAGS_NEED64K ) vm->flags |=XE_VM_FLAGS_64K ;
@@ -1173,7 +1173,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)xe_pt_populate_empty (gt, vm, vm->pt_root[id]); } -dma_resv_unlock (&vm->resv); +dma_resv_unlock (xe_vm_resv (vm));/* Kernel migration VM shouldn't have a circular loop.. */ if (!(flags &XE_VM_FLAG_MIGRATION )) {
@@ -1230,10 +1230,10 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)if (vm->pt_root[id])xe_pt_destroy (vm->pt_root[id], vm->flags,NULL ); } -dma_resv_unlock (&vm->resv); +dma_resv_unlock (xe_vm_resv (vm));drm_gpuva_manager_destroy (&vm->mgr); err_put: -dma_resv_fini (&vm->resv); +dma_resv_fini (xe_vm_resv (vm));kfree (vm);if (!(flags &XE_VM_FLAG_MIGRATION )) {xe_device_mem_access_put (xe);
@@ -1422,7 +1422,7 @@ static void vm_destroy_work_func(struct work_struct *w)trace_xe_vm_free (vm);dma_fence_put (vm->rebind_fence); -dma_resv_fini (&vm->resv); +dma_resv_fini (xe_vm_resv (vm));kfree (vm); }
@@ -3294,7 +3294,7 @@ int xe_vm_lock(struct xe_vm *vm, struct ww_acquire_ctx *ww,void xe_vm_unlock (struct xe_vm *vm,struct ww_acquire_ctx *ww) { -dma_resv_unlock (&vm->resv); +dma_resv_unlock (xe_vm_resv (vm));ww_acquire_fini (ww); }
@@ -3327,7 +3327,7 @@ int xe_vm_invalidate_vma(struct xe_vma *vma)WARN_ON_ONCE (!mmu_interval_check_retry (&vma->userptr.notifier, vma->userptr.notifier_seq)); -WARN_ON_ONCE (!dma_resv_test_signaled (&xe_vma_vm (vma)->resv, +WARN_ON_ONCE (!dma_resv_test_signaled (xe_vma_resv (vma),DMA_RESV_USAGE_BOOKKEEP )); }else {
drivers/gpu/drm/xe/xe_vm.h +11 -1
@@ -57,6 +57,11 @@ static inline struct xe_device *xe_vm_device(struct xe_vm *vm) return container_of(vm->mgr.drm, struct xe_device, drm); } +static inline struct dma_resv *xe_vm_resv (struct xe_vm *vm) +{ +return &vm->mgr.resv; +} +static inline struct xe_vm *gpuva_to_vm (struct drm_gpuva *gpuva) { return container_of(gpuva->mgr, struct xe_vm, mgr);
@@ -112,6 +117,11 @@ static inline struct xe_device *xe_vma_device(struct xe_vma *vma)return xe_vm_device (xe_vma_vm (vma)); } +static inline struct dma_resv *xe_vma_resv (struct xe_vma *vma) +{ +return xe_vm_resv (xe_vma_vm (vma)); +} +static inline bool xe_vma_read_only (struct xe_vma *vma) {return vma->gpuva.flags &XE_VMA_READ_ONLY ;
@@ -122,7 +132,7 @@ static inline u64 xe_vma_userptr(struct xe_vma *vma)return vma->gpuva.gem.offset; } -#define xe_vm_assert_held (vm) dma_resv_assert_held(&(vm)->resv) +#define xe_vm_assert_held (vm) dma_resv_assert_held(&(vm)->mgr.resv)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_gt *full_gt);
drivers/gpu/drm/xe/xe_vm_types.h +1 -5
@@ -8,7 +8,6 @@#include <drm/drm_gpuva_mgr.h> -#include <linux/dma-resv.h> #include <linux/kref.h> #include <linux/mmu_notifier.h> #include <linux/scatterlist.h>
@@ -131,7 +130,7 @@ struct xe_vma {struct xe_device ; -#define xe_vm_assert_held (vm) dma_resv_assert_held(&(vm)->resv) +#define xe_vm_assert_held (vm) dma_resv_assert_held(&(vm)->mgr.resv)struct xe_vm {/** @mgr: base GPUVA used to track VMAs */
@@ -142,9 +141,6 @@ struct xe_vm {/* engine used for (un)binding vma's */ struct xe_engine *eng[XE_MAX_GT ]; -/** Protects @rebind_list and the page-table structures */ -struct dma_resv resv; -/** @lru_bulk_move: Bulk LRU move list for this VM's BOs */ struct ttm_lru_bulk_move lru_bulk_move;
include/drm/drm_gpuva_mgr.h +6 -0
@@ -25,6 +25,7 @@* OTHER DEALINGS IN THE SOFTWARE. */ +#include <linux/dma-resv.h> #include <linux/maple_tree.h> #include <linux/mm.h> #include <linux/rbtree.h>
@@ -177,6 +178,11 @@ struct drm_gpuva_manager {*/ const char *name; +/** +* @resv: dma-resv for all private GEMs mapped in this address space +*/ +struct dma_resv resv; +/** * @mm_start: start of the VA space */
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 23/30 drm/gpuva: Add support for extobj
In reply to Matthew BrostManager maintains lists of GPUVA with extobjs.
--- drivers/gpu/drm/drm_gpuva_mgr.c | 45 +++++++++++++-- drivers/gpu/drm/xe/xe_exec.c | 24 ++++---- drivers/gpu/drm/xe/xe_vm.c | 99 +++++--------------------------- drivers/gpu/drm/xe/xe_vm.h | 3 - drivers/gpu/drm/xe/xe_vm_types.h | 16 ------ include/drm/drm_gpuva_mgr.h | 39 ++++++++++++- 6 files changed, 105 insertions(+), 121 deletions(-)
drivers/gpu/drm/drm_gpuva_mgr.c +41 -4
@@ -447,6 +447,9 @@ drm_gpuva_manager_init(struct drm_gpuva_manager *mgr,memset (&mgr->kernel_alloc_node,0 ,sizeof (struct drm_gpuva )); + mgr->extobj.entries =0 ; +INIT_LIST_HEAD (&mgr->extobj.list); +if (reserve_range) { mgr->kernel_alloc_node.va.addr = reserve_offset; mgr->kernel_alloc_node.va.range = reserve_range;
@@ -706,7 +709,8 @@ EXPORT_SYMBOL(drm_gpuva_remove);* @va: the &drm_gpuva to link * * This adds the given &va to the GPU VA list of the &drm_gem_object it is -* associated with. +* associated with and to &drm_gpuva_manager.extobj.list if GPUVA maps an +* extobj. * * This function expects the caller to protect the GEM's GPUVA list against * concurrent access.
@@ -714,8 +718,14 @@ EXPORT_SYMBOL(drm_gpuva_remove);void drm_gpuva_link (struct drm_gpuva *va) { -if (likely (va->gem.obj)) +if (likely (va->gem.obj)) {list_add_tail (&va->gem.entry, &va->gem.obj->gpuva.list); +if (va->flags &DRM_GPUVA_EXTOBJ ) { +list_add_tail (&va->gem.extobj_link, + &va->mgr->extobj.list); + ++va->mgr->extobj.entries; + } + } }EXPORT_SYMBOL (drm_gpuva_link);
@@ -724,7 +734,8 @@ EXPORT_SYMBOL(drm_gpuva_link);* @va: the &drm_gpuva to unlink * * This removes the given &va from the GPU VA list of the &drm_gem_object it is -* associated with. +* associated with and from &drm_gpuva_manager.extobj.list if GPUVA maps an +* extobj. * * This function expects the caller to protect the GEM's GPUVA list against * concurrent access.
@@ -732,8 +743,13 @@ EXPORT_SYMBOL(drm_gpuva_link);void drm_gpuva_unlink (struct drm_gpuva *va) { -if (likely (va->gem.obj)) +if (likely (va->gem.obj)) {list_del_init (&va->gem.entry); +if (va->flags &DRM_GPUVA_EXTOBJ ) { +list_del (&va->gem.extobj_link); + --va->mgr->extobj.entries; + } + } }EXPORT_SYMBOL (drm_gpuva_unlink);
@@ -871,6 +887,27 @@ drm_gpuva_interval_empty(struct drm_gpuva_manager *mgr, u64 addr, u64 range) }EXPORT_SYMBOL (drm_gpuva_interval_empty); +/** +* drm_gpuva_add_fence - add fence to private and all extobj dma-resv +* @mgr: the &drm_gpuva_manager to add a fence to +* @fence: fence to add +* @private_usage: private dma-resv usage +* @extobj_usage: extobj dma-resv usage +* +* Returns: true if the interval is empty, false otherwise +*/ +void drm_gpuva_add_fence (struct drm_gpuva_manager *mgr,struct dma_fence *fence, +enum dma_resv_usage private_usage, +enum dma_resv_usage extobj_usage) +{ +struct drm_gpuva *gpuva; + +dma_resv_add_fence (&mgr->resv, fence, private_usage); +drm_gpuva_for_each_extobj (gpuva, mgr) +dma_resv_add_fence (gpuva->gem.obj->resv, fence, extobj_usage); +} +EXPORT_SYMBOL (drm_gpuva_add_fence); +/** * drm_gpuva_map - helper to insert a &drm_gpuva from &drm_gpuva_fn_ops * callbacks
drivers/gpu/drm/xe/xe_exec.c +11 -13
@@ -353,19 +353,17 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)* the job and let the DRM scheduler / backend clean up the job. */ xe_sched_job_arm (job); -if (!xe_vm_no_dma_fences (vm)) { -/* Block userptr invalidations / BO eviction */ -dma_resv_add_fence (xe_vm_resv (vm), - &job->drm.s_fence->finished, -DMA_RESV_USAGE_BOOKKEEP ); - -/* -* Make implicit sync work across drivers, assuming all external -* BOs are written as we don't pass in a read / write list. -*/ -xe_vm_fence_all_extobjs (vm, &job->drm.s_fence->finished, -DMA_RESV_USAGE_WRITE ); - } + +/* +* Block userptr invalidations / BO eviction +* +* Make implicit sync work across drivers, assuming all external BOss +* are written as we don't pass in a read / write list. +*/ +if (!xe_vm_no_dma_fences (vm)) +drm_gpuva_add_fence (&vm->mgr, &job->drm.s_fence->finished, +DMA_RESV_USAGE_BOOKKEEP , +DMA_RESV_USAGE_WRITE );for (i =0 ; i < num_syncs; i++) xe_sync_entry_signal(&syncs[i], job,
drivers/gpu/drm/xe/xe_vm.c +15 -84
187 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +0 -3
@@ -232,9 +232,6 @@ void xe_vm_unlock_dma_resv(struct xe_vm *vm,struct ww_acquire_ctx *ww, struct list_head *objs); -void xe_vm_fence_all_extobjs (struct xe_vm *vm,struct dma_fence *fence, -enum dma_resv_usage usage); -int xe_analyze_vm (struct drm_printer *p,struct xe_vm *vm,int gt_id);#if IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )
drivers/gpu/drm/xe/xe_vm_types.h +0 -16
@@ -113,14 +113,6 @@ struct xe_vma {u8 gt_invalidated; } usm; -struct { -/** -* @extobj.link: Link into vm's external object list. -* protected by the vm lock. -*/ -struct list_head link; - } extobj; -/** * @userptr: user pointer state, only allocated for VMAs that are * user pointers
@@ -189,14 +181,6 @@ struct xe_vm {*/ struct work_struct destroy_work; -/** @extobj: bookkeeping for external objects. Protected by the vm lock */ -struct { -/** @enties: number of external BOs attached this VM */ -u32 entries; -/** @list: list of vmas with external bos attached */ -struct list_head list; - } extobj; -/** @async_ops: async VM operations (bind / unbinds) */ struct {/** @list: list of pending async VM ops */
include/drm/drm_gpuva_mgr.h +38 -1
@@ -54,10 +54,18 @@ enum drm_gpuva_flags {*/ DRM_GPUVA_SPARSE = (1 <<1 ), +/** +* @DRM_GPUVA_EXTOBJ: +* +* Flag indicating that the &drm_gpuva is a mapping of an extobj (GEN +* not tied to a single address space). +*/ +DRM_GPUVA_EXTOBJ = (1 <<2 ), +/** * @DRM_GPUVA_USERBITS: user defined bits */ - DRM_GPUVA_USERBITS = (1 << 2), + DRM_GPUVA_USERBITS = (1 << 3), };/**
@@ -112,6 +120,12 @@ struct drm_gpuva {* @entry: the &list_head to attach this object to a &drm_gem_object */ struct list_head entry; + +/** +* @extobj_link: the &list_head to attach this object to a +* @drm_gpuva_manager.extobj.list +*/ +struct list_head extobj_link; } gem; };
@@ -134,6 +148,10 @@ struct drm_gpuva *drm_gpuva_find_next(struct drm_gpuva_manager *mgr, u64 end);bool drm_gpuva_interval_empty (struct drm_gpuva_manager *mgr,u64 addr,u64 range); +void drm_gpuva_add_fence (struct drm_gpuva_manager *mgr,struct dma_fence *fence, +enum dma_resv_usage private_usage, +enum dma_resv_usage extobj_usage); +/** * drm_gpuva_evict - sets whether the backing GEM of this &drm_gpuva is evicted * @va: the &drm_gpuva to set the evict flag for
@@ -206,6 +224,17 @@ struct drm_gpuva_manager {*/ struct drm_gpuva kernel_alloc_node; +/** @extobj: bookkeeping for external GEMs */ +struct { +/** +* @entries: number of external GEMs attached this address +* space +*/ +u32 entries; +/** @list: list of GPUVAs with external GEMs attached */ +struct list_head list; + } extobj; +/** * @ops: &drm_gpuva_fn_ops providing the split/merge steps to drivers */
@@ -509,6 +538,14 @@ struct drm_gpuva_ops {struct list_head list; }; +/** +* drm_gpuva_for_each_op - iterator to walk over &drm_gpuva of extobjs +* @va: &drm_gpuva to assign in each iteration step +* @mgr: &drm_gpuva_manager to walk extobj lisy +*/ +#define drm_gpuva_for_each_extobj (va, mgr) \ + list_for_each_entry(va, &(mgr)->extobj.list, gem.extobj_link) +/** * drm_gpuva_for_each_op - iterator to walk over &drm_gpuva_ops * @op: &drm_gpuva_op to assign in each iteration step
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 24/30 drm/xe: Userptr refactor
In reply to Matthew BrostAdd GPUVA userptr flag, add GPUVA userptr sub-struct, and drop sg pointer. A larger follow on cleanup may push more of userptr implementation to GPUVA.
--- drivers/gpu/drm/xe/xe_pt.c | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 41 +++++++++++---------- drivers/gpu/drm/xe/xe_vm.h | 23 +++++++----- drivers/gpu/drm/xe/xe_vm_types.h | 20 +++++----- include/drm/drm_gpuva_mgr.h | 63 +++++++++++++++++++++----------- 5 files changed, 89 insertions(+), 64 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +3 -3
@@ -92,8 +92,8 @@ static dma_addr_t vma_addr(struct xe_vma *vma, u64 offset, page = offset >>PAGE_SHIFT ; offset &= (PAGE_SIZE -1 ); -xe_res_first_sg (vma->userptr.sg, page <<PAGE_SHIFT , page_size, - &cur); +xe_res_first_sg (&vma->userptr.sgt, page <<PAGE_SHIFT , + page_size, &cur);return xe_res_dma (&cur) + offset; }else {return xe_bo_addr (xe_vma_bo (vma), offset, page_size, is_vram);
@@ -813,7 +813,7 @@ xe_pt_stage_bind(struct xe_gt *gt, struct xe_vma *vma,xe_bo_assert_held (bo);if (!xe_vma_is_null (vma)) {if (xe_vma_is_userptr (vma)) -xe_res_first_sg (vma->userptr.sg,0 ,xe_vma_size (vma), +xe_res_first_sg (&vma->userptr.sgt,0 ,xe_vma_size (vma), &curs);else if (xe_bo_is_vram (bo) ||xe_bo_is_stolen (bo)) xe_res_first(bo->ttm.resource, xe_vma_bo_offset(vma),
drivers/gpu/drm/xe/xe_vm.c +21 -20
115 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +14 -9
@@ -101,12 +101,6 @@ static inline u64 xe_vma_bo_offset(struct xe_vma *vma)return vma->gpuva.gem.offset; } -static inline struct xe_bo *xe_vma_bo (struct xe_vma *vma) -{ -return !vma->gpuva.gem.obj ?NULL : - container_of(vma->gpuva.gem.obj, struct xe_bo, ttm.base); -} -static inline struct xe_vm *xe_vma_vm (struct xe_vma *vma) { return container_of(vma->gpuva.mgr, struct xe_vm, mgr);
@@ -129,7 +123,7 @@ static inline bool xe_vma_read_only(struct xe_vma *vma)static inline u64 xe_vma_userptr (struct xe_vma *vma) { -return vma->gpuva.gem.offset; +return vma->gpuva.userptr.address; }#define xe_vm_assert_held (vm) dma_resv_assert_held(&(vm)->mgr.resv)
@@ -197,12 +191,18 @@ static inline void xe_vm_reactivate_rebind(struct xe_vm *vm)static inline bool xe_vma_is_null (struct xe_vma *vma) { -return vma->gpuva.flags &XE_VMA_NULL ; +return vma->gpuva.flags &DRM_GPUVA_SPARSE ; }static inline bool xe_vma_is_userptr (struct xe_vma *vma) { -return !xe_vma_bo (vma) && !xe_vma_is_null (vma); +return vma->gpuva.flags &DRM_GPUVA_USERPTR ; +} + +static inline struct xe_bo *xe_vma_bo (struct xe_vma *vma) +{ +return xe_vma_is_null (vma) ||xe_vma_is_userptr (vma) ?NULL : + container_of(vma->gpuva.gem.obj, struct xe_bo, ttm.base); }static inline bool xe_vma_has_no_bo (struct xe_vma *vma)
@@ -210,6 +210,11 @@ static inline bool xe_vma_has_no_bo(struct xe_vma *vma)return !xe_vma_bo (vma); } +static inline bool xe_vma_userptr_sg_mapped (struct xe_vma *vma) +{ +return vma->gpuva.flags &XE_VMA_USERPTR_SG_MAPPED ; +} +int xe_vma_userptr_pin_pages (struct xe_vma *vma);int xe_vma_userptr_check_repin (struct xe_vma *vma);
drivers/gpu/drm/xe/xe_vm_types.h +9 -11
@@ -23,15 +23,15 @@ struct xe_vm;#define TEST_VM_ASYNC_OPS_ERROR #define FORCE_ASYNC_OP_ERROR BIT(31) -#define XE_VMA_READ_ONLY DRM_GPUVA_USERBITS -#define XE_VMA_DESTROYED (DRM_GPUVA_USERBITS << 1) -#define XE_VMA_ATOMIC_PTE_BIT (DRM_GPUVA_USERBITS << 2) -#define XE_VMA_FIRST_REBIND (DRM_GPUVA_USERBITS << 3) -#define XE_VMA_LAST_REBIND (DRM_GPUVA_USERBITS << 4) -#define XE_VMA_NULL (DRM_GPUVA_USERBITS << 5) -#define XE_VMA_PTE_4K (DRM_GPUVA_USERBITS << 6) -#define XE_VMA_PTE_2M (DRM_GPUVA_USERBITS << 7) -#define XE_VMA_PTE_1G (DRM_GPUVA_USERBITS << 8) +#define XE_VMA_READ_ONLY DRM_GPUVA_USERBITS +#define XE_VMA_DESTROYED (DRM_GPUVA_USERBITS << 1) +#define XE_VMA_ATOMIC_PTE_BIT (DRM_GPUVA_USERBITS << 2) +#define XE_VMA_FIRST_REBIND (DRM_GPUVA_USERBITS << 3) +#define XE_VMA_LAST_REBIND (DRM_GPUVA_USERBITS << 4) +#define XE_VMA_USERPTR_SG_MAPPED (DRM_GPUVA_USERBITS << 5) +#define XE_VMA_PTE_4K (DRM_GPUVA_USERBITS << 6) +#define XE_VMA_PTE_2M (DRM_GPUVA_USERBITS << 7) +#define XE_VMA_PTE_1G (DRM_GPUVA_USERBITS << 8)/** struct xe_userptr - User pointer */ struct xe_userptr {
@@ -41,8 +41,6 @@ struct xe_userptr {struct mmu_interval_notifier notifier;/** @sgt: storage for a scatter gather table */ struct sg_table sgt; -/** @sg: allocated scatter gather table */ -struct sg_table *sg;/** @notifier_seq: notifier sequence number */ unsigned long notifier_seq;/**
include/drm/drm_gpuva_mgr.h +42 -21
83 lines not on this page · raw message
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 25/30 drm: execution context for GEM buffers v3
In reply to Matthew BrostFrom: Christian König <[email protected]>
This adds the infrastructure for an execution context for GEM buffers which is similar to the existinc TTMs execbuf util and intended to replace it in the long term.
The basic functionality is that we abstracts the necessary loop to lock many different GEM buffers with automated deadlock and duplicate handling.
v2: drop xarray and use dynamic resized array instead, the locking
overhead is unecessary and measureable.
v3: drop duplicate tracking, radeon is really the only one needing that.
--- Documentation/gpu/drm-mm.rst | 12 ++ drivers/gpu/drm/Kconfig | 6 + drivers/gpu/drm/Makefile | 2 + drivers/gpu/drm/drm_exec.c | 248 +++++++++++++++++++++++++++++++++++ include/drm/drm_exec.h | 115 ++++++++++++++++ 5 files changed, 383 insertions(+) create mode 100644 drivers/gpu/drm/drm_exec.c create mode 100644 include/drm/drm_exec.h
Documentation/gpu/drm-mm.rst +12 -0
@@ -524,6 +524,18 @@ DRM Sync Objects .. kernel-doc:: drivers/gpu/drm/drm_syncobj.c :export: +DRM Execution context +===================== + +.. kernel-doc:: drivers/gpu/drm/drm_exec.c + :doc: Overview + +.. kernel-doc:: include/drm/drm_exec.h + :internal: + +.. kernel-doc:: drivers/gpu/drm/drm_exec.c + :export: + GPU Scheduler =============
drivers/gpu/drm/Kconfig +6 -0
@@ -201,6 +201,12 @@ config DRM_TTM GPU memory types. Will be enabled automatically if a device driver uses it. +config DRM_EXEC + tristate + depends on DRM + help + Execution context for command submissions + config DRM_BUDDY tristate depends on DRM
drivers/gpu/drm/Makefile +2 -0
@@ -80,6 +80,8 @@ obj-$(CONFIG_DRM_PANEL_ORIENTATION_QUIRKS) += drm_panel_orientation_quirks.o # # Memory-management helpers # +# +obj-$(CONFIG_DRM_EXEC) += drm_exec.o obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o
drivers/gpu/drm/drm_exec.c new file +248 -0
248 lines not on this page · raw message
include/drm/drm_exec.h new file +115 -0
115 lines not on this page · raw message
Signature · 1 line
2.34.1
Christian König
maintainerRe: [Intel-xe] [PATCH 25/30] drm: execution context for GEM buffers v3
Just FYI I found time to work a bit on this last week and will probably send out the next version this week.
Christian.
Am 01.05.23 um 09:51 schrieb Matthew Brost:
Quoted 452 lines
From: Christian König <[email protected]>
This adds the infrastructure for an execution context for GEM buffers which is similar to the existinc TTMs execbuf util and intended to replace it in the long term.
The basic functionality is that we abstracts the necessary loop to lock many different GEM buffers with automated deadlock and duplicate handling.
v2: drop xarray and use dynamic resized array instead, the locking
overhead is unecessary and measureable.
v3: drop duplicate tracking, radeon is really the only one needing that.Signed-off-by Christian König <[email protected]>--- Documentation/gpu/drm-mm.rst | 12 ++ drivers/gpu/drm/Kconfig | 6 + drivers/gpu/drm/Makefile | 2 + drivers/gpu/drm/drm_exec.c | 248 +++++++++++++++++++++++++++++++++++ include/drm/drm_exec.h | 115 ++++++++++++++++ 5 files changed, 383 insertions(+)create mode 100644
drivers/gpu/drm/drm_exec.c
create mode 100644include/drm/drm_exec.hDocumentation/gpu/drm-mm.rst +12
-0@@ -524,6 +524,18 @@ DRM Sync Objects .. kernel-doc:: drivers/gpu/drm/drm_syncobj.c :export: +DRM Execution context +===================== + +.. kernel-doc:: drivers/gpu/drm/drm_exec.c + :doc: Overview + +.. kernel-doc:: include/drm/drm_exec.h + :internal: + +.. kernel-doc:: drivers/gpu/drm/drm_exec.c + :export: + GPU Scheduler =============drivers/gpu/drm/Kconfig +6
-0@@ -201,6 +201,12 @@ config DRM_TTM GPU memory types. Will be enabled automatically if a device driver uses it. +config DRM_EXEC + tristate + depends on DRM + help + Execution context for command submissions + config DRM_BUDDY tristate depends on DRMdrivers/gpu/drm/Makefile +2
-0@@ -80,6 +80,8 @@ obj-$(CONFIG_DRM_PANEL_ORIENTATION_QUIRKS) += drm_panel_orientation_quirks.o # # Memory-management helpers # +# +obj-$(CONFIG_DRM_EXEC) += drm_exec.o obj-$(CONFIG_DRM_BUDDY) += drm_buddy.odrivers/gpu/drm/drm_exec.c new file +248
-0@@ -0,0 +1,248 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ + +#include <drm/drm_exec.h> +#include <drm/drm_gem.h> +#include <linux/dma-resv.h> + +/** +* DOC: Overview +* +* This component mainly abstracts the retry loop necessary for locking +* multiple GEM objects while preparing hardware operations (e.g. command +* submissions, page table updates etc..). +* +* If a contention is detected while locking a GEM object the cleanup procedure +* unlocks all previously locked GEM objects and locks the contended one first +* before locking any further objects. +* +* After an object is locked fences slots can optionally be reserved on the +* dma_resv object inside the GEM object. +* +* A typical usage pattern should look like this:: +* +* struct drm_gem_object *obj; +* struct drm_exec exec; +* unsigned long index; +* int ret; +* +* drm_exec_init(&exec, true); +* drm_exec_while_not_all_locked(&exec) { +* ret = drm_exec_prepare_obj(&exec, boA, 1); +* drm_exec_continue_on_contention(&exec); +* if (ret) +* goto error; +* +* ret = drm_exec_lock(&exec, boB, 1); +* drm_exec_continue_on_contention(&exec); +* if (ret) +* goto error; +* } +* +* drm_exec_for_each_locked_object(&exec, index, obj) { +* dma_resv_add_fence(obj->resv, fence, DMA_RESV_USAGE_READ); +* ... +* } +* drm_exec_fini(&exec); +* +* See struct dma_exec for more details. +*/ + +/* Dummy value used to initially enter the retry loop */ +#define DRM_EXEC_DUMMY (void*)~0 + +/* Unlock all objects and drop references */ +static void drm_exec_unlock_all (struct drm_exec *exec) +{ +struct drm_gem_object *obj; +unsigned long index; + +drm_exec_for_each_locked_object (exec, index, obj) { +dma_resv_unlock (obj->resv); +drm_gem_object_put (obj); + } + +if (exec->prelocked) { +drm_gem_object_put (exec->prelocked); + exec->prelocked =NULL ; + } +} + +/** +* drm_exec_init - initialize a drm_exec object +* @exec: the drm_exec object to initialize +* @interruptible: if locks should be acquired interruptible +* +* Initialize the object and make sure that we can track locked and duplicate +* objects. +*/ +void drm_exec_init (struct drm_exec *exec,bool interruptible) +{ + exec->interruptible = interruptible; + exec->objects =kmalloc (PAGE_SIZE ,GFP_KERNEL ); + +/* If allocation here fails, just delay that till the first use */ + exec->max_objects = exec->objects ?PAGE_SIZE /sizeof (void *) :0 ; + exec->num_objects =0 ; + exec->contended =DRM_EXEC_DUMMY ; + exec->prelocked =NULL ; +} +EXPORT_SYMBOL (drm_exec_init); + +/** +* drm_exec_fini - finalize a drm_exec object +* @exec: the drm_exec object to finilize +* +* Unlock all locked objects, drop the references to objects and free all memory +* used for tracking the state. +*/ +void drm_exec_fini (struct drm_exec *exec) +{ +drm_exec_unlock_all (exec); +kvfree (exec->objects); +if (exec->contended !=DRM_EXEC_DUMMY ) { +drm_gem_object_put (exec->contended); +ww_acquire_fini (&exec->ticket); + } +} +EXPORT_SYMBOL (drm_exec_fini); + +/** +* drm_exec_cleanup - cleanup when contention is detected +* @exec: the drm_exec object to cleanup +* +* Cleanup the current state and return true if we should stay inside the retry +* loop, false if there wasn't any contention detected and we can keep the +* objects locked. +*/ +bool drm_exec_cleanup (struct drm_exec *exec) +{ +if (likely (!exec->contended)) { +ww_acquire_done (&exec->ticket); +return false; + } + +if (likely (exec->contended ==DRM_EXEC_DUMMY )) { + exec->contended =NULL ; +ww_acquire_init (&exec->ticket, &reservation_ww_class); +return true; + } + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; +return true; +} +EXPORT_SYMBOL (drm_exec_cleanup); + +/* Track the locked object in the xa and reserve fences */ +static int drm_exec_obj_locked (struct drm_exec *exec, +struct drm_gem_object *obj) +{ +if (unlikely (exec->num_objects == exec->max_objects)) { +size_t size = exec->max_objects *sizeof (void *); +void *tmp; + + tmp =kvrealloc (exec->objects, size, size +PAGE_SIZE , +GFP_KERNEL ); +if (!tmp) +return -ENOMEM ; + + exec->objects = tmp; + exec->max_objects +=PAGE_SIZE /sizeof (void *); + } +drm_gem_object_get (obj); + exec->objects[exec->num_objects++] = obj; + +return 0 ; +} + +/* Make sure the contended object is locked first */ +static int drm_exec_lock_contended (struct drm_exec *exec) +{ +struct drm_gem_object *obj = exec->contended; +int ret; + +if (likely (!obj)) +return 0 ; + +if (exec->interruptible) { + ret =dma_resv_lock_slow_interruptible (obj->resv, + &exec->ticket); +if (unlikely (ret)) + goto error_dropref; + }else { +dma_resv_lock_slow (obj->resv, &exec->ticket); + } + + ret =drm_exec_obj_locked (exec, obj); +if (unlikely (ret)) { +dma_resv_unlock (obj->resv); + goto error_dropref; + } + +swap (exec->prelocked, obj); + +error_dropref: +/* Always cleanup the contention so that error handling can kick in */ +drm_gem_object_put (obj); + exec->contended =NULL ; +return ret; +} + +/** +* drm_exec_prepare_obj - prepare a GEM object for use +* @exec: the drm_exec object with the state +* @obj: the GEM object to prepare +* @num_fences: how many fences to reserve +* +* Prepare a GEM object for use by locking it and reserving fence slots. All +* succesfully locked objects are put into the locked container. Duplicates +* detected as well and automatically moved into the duplicates container. +* +* Returns: -EDEADLK if a contention is detected, -ENOMEM when memory +* allocation failed and zero for success. +*/ +int drm_exec_prepare_obj (struct drm_exec *exec,struct drm_gem_object *obj, +unsigned int num_fences) +{ +int ret; + + ret =drm_exec_lock_contended (exec); +if (unlikely (ret)) +return ret; + +if (exec->prelocked == obj) { +drm_gem_object_put (exec->prelocked); + exec->prelocked =NULL ; + +return dma_resv_reserve_fences (obj->resv, num_fences); + } + +if (exec->interruptible) + ret =dma_resv_lock_interruptible (obj->resv, &exec->ticket); +else + ret =dma_resv_lock (obj->resv, &exec->ticket); + +if (unlikely (ret == -EDEADLK )) { +drm_gem_object_get (obj); + exec->contended = obj; +return -EDEADLK ; + } + +if (unlikely (ret)) +return ret; + + ret =drm_exec_obj_locked (exec, obj); +if (ret) + goto error_unlock; + +/* Keep locked when reserving fences fails */ +return dma_resv_reserve_fences (obj->resv, num_fences); + +error_unlock: +dma_resv_unlock (obj->resv); +return ret; +} +EXPORT_SYMBOL (drm_exec_prepare_obj); + +MODULE_DESCRIPTION ("DRM execution context" ); +MODULE_LICENSE ("Dual MIT/GPL" );include/drm/drm_exec.h new file +115
-0@@ -0,0 +1,115 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ + +#ifndef __DRM_EXEC_H__ +#define __DRM_EXEC_H__ + +#include <linux/ww_mutex.h> + +struct drm_gem_object ; + +/** +* struct drm_exec - Execution context +*/ +struct drm_exec { +/** +* @interruptible: If locks should be taken interruptible +*/ +bool interruptible; + +/** +* @ticket: WW ticket used for acquiring locks +*/ +struct ww_acquire_ctx ticket; + +/** +* @num_objects: number of objects locked +*/ +unsigned int num_objects; + +/** +* @max_objects: maximum objects in array +*/ +unsigned int max_objects; + +/** +* @objects: array of the locked objects +*/ +struct drm_gem_object **objects; + +/** +* @contended: contended GEM object we backet of for +*/ +struct drm_gem_object *contended; + +/** +* @prelocked: already locked GEM object because of contention +*/ +struct drm_gem_object *prelocked; +}; + +/** +* drm_exec_for_each_locked_object - iterate over all the locked objects +* @exec: drm_exec object +* @index: unsigned long index for the iteration +* @obj: the current GEM object +* +* Iterate over all the locked GEM objects inside the drm_exec object. +*/ +#define drm_exec_for_each_locked_object (exec, index, obj) \ + for (index = 0, obj = (exec)->objects[0]; \ + index < (exec)->num_objects; \ + ++index, obj = (exec)->objects[index]) + +/** +* drm_exec_while_not_all_locked - loop until all GEM objects are prepared +* @exec: drm_exec object +* +* Core functionality of the drm_exec object. Loops until all GEM objects are +* prepared and no more contention exists. +* +* At the beginning of the loop it is guaranteed that no GEM object is locked. +*/ +#define drm_exec_while_not_all_locked (exec) \ + while (drm_exec_cleanup(exec)) + +/** +* drm_exec_continue_on_contention - continue the loop when we need to cleanup +* @exec: drm_exec object +* +* Control flow helper to continue when a contention was detected and we need to +* clean up and re-start the loop to prepare all GEM objects. +*/ +#define drm_exec_continue_on_contention (exec) \ + if (unlikely(drm_exec_is_contended(exec))) \ + continue + +/** +* drm_exec_break_on_contention - break a subordinal loop on contention +* @exec: drm_exec object +* +* Control flow helper to break a subordinal loop when a contention was detected +* and we need to clean up and re-start the loop to prepare all GEM objects. +*/ +#define drm_exec_break_on_contention (exec) \ + if (unlikely(drm_exec_is_contended(exec))) \ + break + +/** +* drm_exec_is_contended - check for contention +* @exec: drm_exec object +* +* Returns true if the drm_exec object has run into some contention while +* locking a GEM object and needs to clean up. +*/ +static inline bool drm_exec_is_contended (struct drm_exec *exec) +{ +return !!exec->contended; +} + +void drm_exec_init (struct drm_exec *exec,bool interruptible); +void drm_exec_fini (struct drm_exec *exec); +bool drm_exec_cleanup (struct drm_exec *exec); +int drm_exec_prepare_obj (struct drm_exec *exec,struct drm_gem_object *obj, +unsigned int num_fences); + +#endif
Matthew Brost
maintainerPATCH 26/30 drm/exec: Always compile drm_exec
In reply to Matthew BrostWe want some helpers for DRM exec in gpuva, alway compile this.
--- drivers/gpu/drm/Makefile | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/Makefile +1 -2
@@ -23,6 +23,7 @@ drm-y := \ drm_dumb_buffers.o \ drm_edid.o \ drm_encoder.o \ + drm_exec.o \ drm_file.o \ drm_fourcc.o \ drm_framebuffer.o \
@@ -81,8 +82,6 @@ obj-$(CONFIG_DRM_PANEL_ORIENTATION_QUIRKS) += drm_panel_orientation_quirks.o # Memory-management helpers # # -obj-$(CONFIG_DRM_EXEC) += drm_exec.o - obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o drm_dma_helper-y := drm_gem_dma_helper.o
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 27/30 drm/xe: Use drm_exec for locking rather than TTM exec helpers
In reply to Matthew Brostdrm_exec is intended to replace TTM exec helpers, use drm_exec.
--- drivers/gpu/drm/drm_gpuva_mgr.c | 67 ++++- drivers/gpu/drm/i915/display/intel_display.c | 6 +- drivers/gpu/drm/xe/Kconfig | 1 + drivers/gpu/drm/xe/xe_bo.c | 56 ++-- drivers/gpu/drm/xe/xe_bo.h | 6 +- drivers/gpu/drm/xe/xe_bo_evict.c | 24 +- drivers/gpu/drm/xe/xe_bo_types.h | 1 - drivers/gpu/drm/xe/xe_engine.c | 7 +- drivers/gpu/drm/xe/xe_exec.c | 37 +-- drivers/gpu/drm/xe/xe_gt_pagefault.c | 55 +--- drivers/gpu/drm/xe/xe_lrc.c | 8 +- drivers/gpu/drm/xe/xe_migrate.c | 13 +- drivers/gpu/drm/xe/xe_vm.c | 283 ++++++++----------- drivers/gpu/drm/xe/xe_vm.h | 27 +- drivers/gpu/drm/xe/xe_vm_madvise.c | 37 +-- include/drm/drm_gpuva_mgr.h | 16 +- 16 files changed, 299 insertions(+), 345 deletions(-)
drivers/gpu/drm/drm_gpuva_mgr.c +59 -8
@@ -483,6 +483,50 @@ drm_gpuva_manager_destroy(struct drm_gpuva_manager *mgr) }EXPORT_SYMBOL (drm_gpuva_manager_destroy); +/** +* TODO +*/ +int drm_gpuva_manager_lock (struct drm_gpuva_manager *mgr,struct drm_exec *exec, +struct drm_gem_object *mgr_obj,bool intr, +unsigned int num_fences) +{ +struct drm_gpuva *gpuva; +int ret; + +drm_exec_init (exec, intr); +drm_exec_while_not_all_locked (exec) { + ret =drm_exec_prepare_obj (exec, mgr_obj, num_fences); +drm_exec_continue_on_contention (exec); +if (ret && ret != -EALREADY ) + goto err_exec; + +drm_gpuva_for_each_extobj (gpuva, mgr) { + ret =drm_exec_prepare_obj (exec, gpuva->gem.obj, + num_fences); +drm_exec_break_on_contention (exec); +if (ret && ret != -EALREADY ) + goto err_exec; + } + } + +return 0 ; + +err_exec: +drm_exec_fini (exec); +return ret; +} +EXPORT_SYMBOL (drm_gpuva_manager_lock); + +/** +* TODO +*/ +void drm_gpuva_manager_unlock (struct drm_gpuva_manager *mgr, +struct drm_exec *exec) +{ +drm_exec_fini (exec); +} +EXPORT_SYMBOL (drm_gpuva_manager_unlock); +static inline bool drm_gpuva_in_mm_range (struct drm_gpuva_manager *mgr,u64 addr,u64 range) {
@@ -888,7 +932,7 @@ drm_gpuva_interval_empty(struct drm_gpuva_manager *mgr, u64 addr, u64 range)EXPORT_SYMBOL (drm_gpuva_interval_empty);/** -* drm_gpuva_add_fence - add fence to private and all extobj dma-resv +* drm_gpuva_manager_add_fence - add fence to private and all extobj dma-resv * @mgr: the &drm_gpuva_manager to add a fence to * @fence: fence to add * @private_usage: private dma-resv usage
@@ -896,17 +940,24 @@ EXPORT_SYMBOL(drm_gpuva_interval_empty);* * Returns: true if the interval is empty, false otherwise */ -void drm_gpuva_add_fence (struct drm_gpuva_manager *mgr,struct dma_fence *fence, -enum dma_resv_usage private_usage, -enum dma_resv_usage extobj_usage) +void drm_gpuva_manager_add_fence (struct drm_gpuva_manager *mgr, +struct drm_exec *exec, +struct dma_fence *fence, +enum dma_resv_usage private_usage, +enum dma_resv_usage extobj_usage) { -struct drm_gpuva *gpuva; +struct drm_gem_object *obj; +unsigned long index; + +dma_resv_assert_held (&mgr->resv);dma_resv_add_fence (&mgr->resv, fence, private_usage); -drm_gpuva_for_each_extobj (gpuva, mgr) -dma_resv_add_fence (gpuva->gem.obj->resv, fence, extobj_usage); +drm_exec_for_each_locked_object (exec, index, obj) +if (likely (&mgr->resv != obj->resv)) +dma_resv_add_fence (obj->resv, fence, extobj_usage); } -EXPORT_SYMBOL (drm_gpuva_add_fence); +EXPORT_SYMBOL (drm_gpuva_manager_add_fence); +/** * drm_gpuva_map - helper to insert a &drm_gpuva from &drm_gpuva_fn_ops
drivers/gpu/drm/i915/display/intel_display.c +3 -3
@@ -7340,11 +7340,11 @@ static int i915_gem_object_read_from_page(struct xe_bo *bo,void *virtual;bool is_iomem;int ret; -struct ww_acquire_ctx ww; +struct drm_exec exec;XE_BUG_ON (size !=8 ); - ret =xe_bo_lock (bo, &ww,0 , true); + ret =xe_bo_lock (bo, &exec,0 , true);if (ret)return ret;
@@ -7361,7 +7361,7 @@ static int i915_gem_object_read_from_page(struct xe_bo *bo,ttm_bo_kunmap (&map); out_unlock: -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec); return ret; }#endif
drivers/gpu/drm/xe/Kconfig +1 -0
@@ -8,6 +8,7 @@ config DRM_XE select SHMEM select TMPFS select DRM_BUDDY + select DRM_EXEC select DRM_KMS_HELPER select DRM_PANEL select DRM_SUBALLOC_HELPER
drivers/gpu/drm/xe/xe_bo.c +23 -33
122 lines not on this page · raw message
drivers/gpu/drm/xe/xe_bo.h +3 -3
@@ -75,6 +75,7 @@#define XE_BO_PROPS_INVALID (-1) +struct drm_exec ;struct sg_table ;struct xe_bo *xe_bo_alloc (void );
@@ -142,10 +143,9 @@ static inline void xe_bo_assert_held(struct xe_bo *bo)dma_resv_assert_held ((bo)->ttm.base.resv); } -int xe_bo_lock (struct xe_bo *bo,struct ww_acquire_ctx *ww, +int xe_bo_lock (struct xe_bo *bo,struct drm_exec *exec,int num_resv,bool intr); - -void xe_bo_unlock (struct xe_bo *bo,struct ww_acquire_ctx *ww); +void xe_bo_unlock (struct xe_bo *bo,struct drm_exec *exec);static inline void xe_bo_unlock_vm_held (struct xe_bo *bo) {
drivers/gpu/drm/xe/xe_bo_evict.c +13 -11
@@ -3,6 +3,8 @@* Copyright © 2022 Intel Corporation */ +#include <drm/drm_exec.h> +#include "xe_bo_evict.h" #include "xe_bo.h"
@@ -27,7 +29,7 @@int xe_bo_evict_all (struct xe_device *xe) {struct ttm_device *bdev = &xe->ttm; -struct ww_acquire_ctx ww; +struct drm_exec exec;struct xe_bo *bo;struct xe_gt *gt;struct list_head still_in_list;
@@ -62,9 +64,9 @@ int xe_bo_evict_all(struct xe_device *xe)list_move_tail (&bo->pinned_link, &still_in_list);spin_unlock (&xe->pinned.lock); -xe_bo_lock (bo, &ww,0 , false); +xe_bo_lock (bo, &exec,0 , false); ret =xe_bo_evict_pinned (bo); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec);xe_bo_put (bo);if (ret) {spin_lock (&xe->pinned.lock);
@@ -96,9 +98,9 @@ int xe_bo_evict_all(struct xe_device *xe)list_move_tail (&bo->pinned_link, &xe->pinned.evicted);spin_unlock (&xe->pinned.lock); -xe_bo_lock (bo, &ww,0 , false); +xe_bo_lock (bo, &exec,0 , false); ret =xe_bo_evict_pinned (bo); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec);xe_bo_put (bo);if (ret)return ret;
@@ -123,7 +125,7 @@ int xe_bo_evict_all(struct xe_device *xe)*/ int xe_bo_restore_kernel (struct xe_device *xe) { -struct ww_acquire_ctx ww; +struct drm_exec exec;struct xe_bo *bo;int ret;
@@ -140,9 +142,9 @@ int xe_bo_restore_kernel(struct xe_device *xe)list_move_tail (&bo->pinned_link, &xe->pinned.kernel_bo_present);spin_unlock (&xe->pinned.lock); -xe_bo_lock (bo, &ww,0 , false); +xe_bo_lock (bo, &exec,0 , false); ret =xe_bo_restore_pinned (bo); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec);if (ret) {xe_bo_put (bo);return ret;
@@ -182,7 +184,7 @@ int xe_bo_restore_kernel(struct xe_device *xe)*/ int xe_bo_restore_user (struct xe_device *xe) { -struct ww_acquire_ctx ww; +struct drm_exec exec;struct xe_bo *bo;struct xe_gt *gt;struct list_head still_in_list;
@@ -204,9 +206,9 @@ int xe_bo_restore_user(struct xe_device *xe)xe_bo_get (bo);spin_unlock (&xe->pinned.lock); -xe_bo_lock (bo, &ww,0 , false); +xe_bo_lock (bo, &exec,0 , false); ret =xe_bo_restore_pinned (bo); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec);xe_bo_put (bo);if (ret) {spin_lock (&xe->pinned.lock);
drivers/gpu/drm/xe/xe_bo_types.h +0 -1
@@ -11,7 +11,6 @@#include <drm/drm_mm.h> #include <drm/ttm/ttm_bo.h> #include <drm/ttm/ttm_device.h> -#include <drm/ttm/ttm_execbuf_util.h> #include <drm/ttm/ttm_placement.h> struct xe_device ;
drivers/gpu/drm/xe/xe_engine.c +4 -3
@@ -8,6 +8,7 @@#include <linux/nospec.h> #include <drm/drm_device.h> +#include <drm/drm_exec.h> #include <drm/drm_file.h> #include <drm/xe_drm.h>
@@ -89,18 +90,18 @@ struct xe_engine *xe_engine_create(struct xe_device *xe, struct xe_vm *vm, u32 logical_mask, u16 width, struct xe_hw_engine *hwe, u32 flags) { - struct ww_acquire_ctx ww; + struct drm_exec exec;struct xe_engine *e;int err;if (vm) { - err =xe_vm_lock (vm, &ww,0 , true); + err =xe_vm_lock (vm, &exec,0 , true);if (err)return ERR_PTR (err); } e =__xe_engine_create (xe, vm, logical_mask, width, hwe, flags);if (vm) -xe_vm_unlock (vm, &ww); +xe_vm_unlock (vm, &exec);return e; }
drivers/gpu/drm/xe/xe_exec.c +13 -24
@@ -6,6 +6,7 @@#include "xe_exec.h" #include <drm/drm_device.h> +#include <drm/drm_exec.h> #include <drm/drm_file.h> #include <drm/xe_drm.h>
@@ -92,21 +93,16 @@* Unlock all */ -static int xe_exec_begin (struct xe_engine *e,struct ww_acquire_ctx *ww, -struct ttm_validate_buffer tv_onstack[], -struct ttm_validate_buffer **tv, -struct list_head *objs) +static int xe_exec_begin (struct xe_engine *e,struct drm_exec *exec) {struct xe_vm *vm = e->vm;struct xe_vma *vma; -LIST_HEAD (dups);int err; - *tv =NULL ;if (xe_vm_no_dma_fences (e->vm))return 0 ; - err =xe_vm_lock_dma_resv (vm, ww, tv_onstack, tv, objs, true,1 ); + err =xe_vm_lock_dma_resv (vm, exec, true,1 );if (err)return err;
@@ -123,8 +119,7 @@ static int xe_exec_begin(struct xe_engine *e, struct ww_acquire_ctx *ww, err =xe_bo_validate (xe_vma_bo (vma), vm, false);if (err) { -xe_vm_unlock_dma_resv (vm, tv_onstack, *tv, ww, objs); - *tv =NULL ; +xe_vm_unlock_dma_resv (vm, exec);return err; } }
@@ -132,14 +127,10 @@ static int xe_exec_begin(struct xe_engine *e, struct ww_acquire_ctx *ww,return 0 ; } -static void xe_exec_end (struct xe_engine *e, -struct ttm_validate_buffer *tv_onstack, -struct ttm_validate_buffer *tv, -struct ww_acquire_ctx *ww, -struct list_head *objs) +static void xe_exec_end (struct xe_engine *e,struct drm_exec *exec) {if (!xe_vm_no_dma_fences (e->vm)) -xe_vm_unlock_dma_resv (e->vm, tv_onstack, tv, ww, objs); +xe_vm_unlock_dma_resv (e->vm, exec); }int xe_exec_ioctl (struct drm_device *dev,void *data,struct drm_file *file)
@@ -149,17 +140,14 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)struct drm_xe_exec *args = data;struct drm_xe_sync __user *syncs_user =u64_to_user_ptr (args->syncs);u64 __user *addresses_user =u64_to_user_ptr (args->address); +struct drm_exec exec;struct xe_engine *engine;struct xe_sync_entry *syncs =NULL ;u64 addresses[XE_HW_ENGINE_MAX_INSTANCE ]; -struct ttm_validate_buffer tv_onstack[XE_ONSTACK_TV ]; -struct ttm_validate_buffer *tv =NULL ;u32 i, num_syncs =0 ;struct xe_sched_job *job;struct dma_fence *rebind_fence;struct xe_vm *vm; -struct ww_acquire_ctx ww; -struct list_head objs;bool write_locked;int err =0 ;
@@ -270,7 +258,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto err_unlock_list; } - err =xe_exec_begin (engine, &ww, tv_onstack, &tv, &objs); + err =xe_exec_begin (engine, &exec);if (err) goto err_unlock_list;
@@ -361,9 +349,10 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)* are written as we don't pass in a read / write list. */ if (!xe_vm_no_dma_fences (vm)) -drm_gpuva_add_fence (&vm->mgr, &job->drm.s_fence->finished, -DMA_RESV_USAGE_BOOKKEEP , -DMA_RESV_USAGE_WRITE ); +drm_gpuva_manager_add_fence (&vm->mgr, &exec, + &job->drm.s_fence->finished, +DMA_RESV_USAGE_BOOKKEEP , +DMA_RESV_USAGE_WRITE );for (i =0 ; i < num_syncs; i++) xe_sync_entry_signal(&syncs[i], job,
@@ -387,7 +376,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)if (err)xe_sched_job_put (job); err_engine_end: -xe_exec_end (engine, tv_onstack, tv, &ww, &objs); +xe_exec_end (engine, &exec); err_unlock_list:if (write_locked)up_write (&vm->lock);
drivers/gpu/drm/xe/xe_gt_pagefault.c +7 -48
103 lines not on this page · raw message
drivers/gpu/drm/xe/xe_lrc.c +5 -3
@@ -3,6 +3,8 @@* Copyright © 2021 Intel Corporation */ +#include <drm/drm_exec.h> +#include "xe_lrc.h" #include "regs/xe_engine_regs.h"
@@ -712,16 +714,16 @@ int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,void xe_lrc_finish (struct xe_lrc *lrc) { -struct ww_acquire_ctx ww; +struct drm_exec exec;xe_hw_fence_ctx_finish (&lrc->fence_ctx);if (lrc->bo->vm) -xe_vm_lock (lrc->bo->vm, &ww,0 , false); +xe_vm_lock (lrc->bo->vm, &exec,0 , false);else xe_bo_lock_no_vm (lrc->bo,NULL );xe_bo_unpin (lrc->bo);if (lrc->bo->vm) -xe_vm_unlock (lrc->bo->vm, &ww); +xe_vm_unlock (lrc->bo->vm, &exec);else xe_bo_unlock_no_vm (lrc->bo);xe_bo_put (lrc->bo);
drivers/gpu/drm/xe/xe_migrate.c +7 -6
@@ -9,6 +9,7 @@#include <linux/sizes.h> #include <drm/drm_managed.h> +#include <drm/drm_exec.h> #include <drm/ttm/ttm_tt.h> #include <drm/xe_drm.h>
@@ -86,13 +87,13 @@ struct xe_engine *xe_gt_migrate_engine(struct xe_gt *gt)static void xe_migrate_fini (struct drm_device *dev,void *arg) {struct xe_migrate *m = arg; -struct ww_acquire_ctx ww; +struct drm_exec exec; -xe_vm_lock (m->eng->vm, &ww,0 , false); +xe_vm_lock (m->eng->vm, &exec,0 , false);xe_bo_unpin (m->pt_bo);if (m->cleared_bo)xe_bo_unpin (m->cleared_bo); -xe_vm_unlock (m->eng->vm, &ww); +xe_vm_unlock (m->eng->vm, &exec);dma_fence_put (m->fence);if (m->cleared_bo)
@@ -315,7 +316,7 @@ struct xe_migrate *xe_migrate_init(struct xe_gt *gt)struct xe_device *xe =gt_to_xe (gt);struct xe_migrate *m;struct xe_vm *vm; -struct ww_acquire_ctx ww; +struct drm_exec exec;int err;XE_BUG_ON (xe_gt_is_media_type (gt));
@@ -332,9 +333,9 @@ struct xe_migrate *xe_migrate_init(struct xe_gt *gt)if (IS_ERR (vm))return ERR_CAST (vm); -xe_vm_lock (vm, &ww,0 , false); +xe_vm_lock (vm, &exec,0 , false); err =xe_migrate_prepare_vm (gt, m, vm); -xe_vm_unlock (vm, &ww); +xe_vm_unlock (vm, &exec);if (err) {xe_vm_close_and_put (vm);return ERR_PTR (err);
drivers/gpu/drm/xe/xe_vm.c +120 -163
538 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +9 -18
@@ -12,6 +12,7 @@#include "xe_vm_types.h" struct drm_device ; +struct drm_exec ;struct drm_printer ;struct drm_file ;
@@ -38,10 +39,14 @@ static inline void xe_vm_put(struct xe_vm *vm)kref_put (&vm->refcount, xe_vm_free); } -int xe_vm_lock (struct xe_vm *vm,struct ww_acquire_ctx *ww, +int xe_vm_lock (struct xe_vm *vm,struct drm_exec *exec,int num_resv,bool intr); +void xe_vm_unlock (struct xe_vm *vm,struct drm_exec *exec); -void xe_vm_unlock (struct xe_vm *vm,struct ww_acquire_ctx *ww); +int xe_vm_bo_lock (struct xe_vm *vm,struct xe_bo *bo,struct drm_exec *exec, +int num_resv,bool intr); +void xe_vm_bo_unlock (struct xe_vm *vm,struct xe_bo *bo,struct drm_exec *exec, +bool lru_update);static inline bool xe_vm_is_closed (struct xe_vm *vm) {
@@ -219,23 +224,9 @@ int xe_vma_userptr_pin_pages(struct xe_vma *vma);int xe_vma_userptr_check_repin (struct xe_vma *vma); -/* -* XE_ONSTACK_TV is used to size the tv_onstack array that is input -* to xe_vm_lock_dma_resv() and xe_vm_unlock_dma_resv(). -*/ -#define XE_ONSTACK_TV 20 -int xe_vm_lock_dma_resv (struct xe_vm *vm,struct ww_acquire_ctx *ww, -struct ttm_validate_buffer *tv_onstack, -struct ttm_validate_buffer **tv, -struct list_head *objs, -bool intr, +int xe_vm_lock_dma_resv (struct xe_vm *vm,struct drm_exec *exec,bool intr,unsigned int num_shared); - -void xe_vm_unlock_dma_resv (struct xe_vm *vm, -struct ttm_validate_buffer *tv_onstack, -struct ttm_validate_buffer *tv, -struct ww_acquire_ctx *ww, -struct list_head *objs); +void xe_vm_unlock_dma_resv (struct xe_vm *vm,struct drm_exec *exec);int xe_analyze_vm (struct drm_printer *p,struct xe_vm *vm,int gt_id);
drivers/gpu/drm/xe/xe_vm_madvise.c +19 -18
@@ -7,6 +7,7 @@#include <linux/nospec.h> +#include <drm/drm_exec.h> #include <drm/ttm/ttm_tt.h> #include <drm/xe_drm.h>
@@ -28,16 +29,16 @@ static int madvise_preferred_mem_class(struct xe_device *xe, struct xe_vm *vm,for (i =0 ; i < num_vmas; ++i) {struct xe_bo *bo; -struct ww_acquire_ctx ww; +struct drm_exec exec; bo =xe_vma_bo (vmas[i]); - err =xe_bo_lock (bo, &ww,0 , true); + err =xe_bo_lock (bo, &exec,0 , true);if (err)return err; bo->props.preferred_mem_class = value;xe_bo_placement_for_flags (xe, bo, bo->flags); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec); }return 0 ;
@@ -53,16 +54,16 @@ static int madvise_preferred_gt(struct xe_device *xe, struct xe_vm *vm,for (i =0 ; i < num_vmas; ++i) {struct xe_bo *bo; -struct ww_acquire_ctx ww; +struct drm_exec exec; bo =xe_vma_bo (vmas[i]); - err =xe_bo_lock (bo, &ww,0 , true); + err =xe_bo_lock (bo, &exec,0 , true);if (err)return err; bo->props.preferred_gt = value;xe_bo_placement_for_flags (xe, bo, bo->flags); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec); }return 0 ;
@@ -89,17 +90,17 @@ static int madvise_preferred_mem_class_gt(struct xe_device *xe,for (i =0 ; i < num_vmas; ++i) {struct xe_bo *bo; -struct ww_acquire_ctx ww; +struct drm_exec exec; bo =xe_vma_bo (vmas[i]); - err =xe_bo_lock (bo, &ww,0 , true); + err =xe_bo_lock (bo, &exec,0 , true);if (err)return err; bo->props.preferred_mem_class = mem_class; bo->props.preferred_gt = gt_id;xe_bo_placement_for_flags (xe, bo, bo->flags); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec); }return 0 ;
@@ -112,13 +113,13 @@ static int madvise_cpu_atomic(struct xe_device *xe, struct xe_vm *vm,for (i =0 ; i < num_vmas; ++i) {struct xe_bo *bo; -struct ww_acquire_ctx ww; +struct drm_exec exec; bo =xe_vma_bo (vmas[i]);if (XE_IOCTL_ERR (xe, !(bo->flags &XE_BO_CREATE_SYSTEM_BIT )))return -EINVAL ; - err =xe_bo_lock (bo, &ww,0 , true); + err =xe_bo_lock (bo, &exec,0 , true);if (err)return err; bo->props.cpu_atomic = !!value;
@@ -130,7 +131,7 @@ static int madvise_cpu_atomic(struct xe_device *xe, struct xe_vm *vm,*/ if (bo->props.cpu_atomic)ttm_bo_unmap_virtual (&bo->ttm); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec); }return 0 ;
@@ -143,18 +144,18 @@ static int madvise_device_atomic(struct xe_device *xe, struct xe_vm *vm,for (i =0 ; i < num_vmas; ++i) {struct xe_bo *bo; -struct ww_acquire_ctx ww; +struct drm_exec exec; bo =xe_vma_bo (vmas[i]);if (XE_IOCTL_ERR (xe, !(bo->flags &XE_BO_CREATE_VRAM0_BIT ) && !(bo->flags &XE_BO_CREATE_VRAM1_BIT )))return -EINVAL ; - err =xe_bo_lock (bo, &ww,0 , true); + err =xe_bo_lock (bo, &exec,0 , true);if (err)return err; bo->props.device_atomic = !!value; -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec); }return 0 ;
@@ -174,16 +175,16 @@ static int madvise_priority(struct xe_device *xe, struct xe_vm *vm,for (i =0 ; i < num_vmas; ++i) {struct xe_bo *bo; -struct ww_acquire_ctx ww; +struct drm_exec exec; bo =xe_vma_bo (vmas[i]); - err =xe_bo_lock (bo, &ww,0 , true); + err =xe_bo_lock (bo, &exec,0 , true);if (err)return err; bo->ttm.priority = value;ttm_bo_move_to_lru_tail (&bo->ttm); -xe_bo_unlock (bo, &ww); +xe_bo_unlock (bo, &exec); }return 0 ;
include/drm/drm_gpuva_mgr.h +13 -3
@@ -32,6 +32,8 @@#include <linux/spinlock.h> #include <linux/types.h> +#include <drm/drm_exec.h> +struct drm_gpuva_manager ;struct drm_gpuva_fn_ops ;struct drm_gpuva_prealloc ;
@@ -169,9 +171,17 @@ struct drm_gpuva *drm_gpuva_find_next(struct drm_gpuva_manager *mgr, u64 end);bool drm_gpuva_interval_empty (struct drm_gpuva_manager *mgr,u64 addr,u64 range); -void drm_gpuva_add_fence (struct drm_gpuva_manager *mgr,struct dma_fence *fence, -enum dma_resv_usage private_usage, -enum dma_resv_usage extobj_usage); +int drm_gpuva_manager_lock (struct drm_gpuva_manager *mgr,struct drm_exec *exec, +struct drm_gem_object *mgr_obj,bool intr, +unsigned int num_fences); +void drm_gpuva_manager_unlock (struct drm_gpuva_manager *mgr, +struct drm_exec *exec); + +void drm_gpuva_manager_add_fence (struct drm_gpuva_manager *mgr, +struct drm_exec *exec, +struct dma_fence *fence, +enum dma_resv_usage private_usage, +enum dma_resv_usage extobj_usage);/** * drm_gpuva_evict - sets whether the backing GEM of this &drm_gpuva is evicted
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 28/30 drm/xe: Allow dma-fences as in-syncs for compute / faulting VM
In reply to Matthew BrostThis is allowed per the dma-fencing rules.
--- drivers/gpu/drm/xe/xe_sync.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/gpu/drm/xe/xe_sync.c +7 -5
@@ -105,6 +105,7 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef, {struct drm_xe_sync sync_in;int err; +bool signal;if (copy_from_user (&sync_in, sync_user,sizeof (*sync_user)))return -EFAULT ;
@@ -113,9 +114,10 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef, ~(SYNC_FLAGS_TYPE_MASK | DRM_XE_SYNC_SIGNAL)))return -EINVAL ; + signal = sync_in.flags &DRM_XE_SYNC_SIGNAL ;switch (sync_in.flags &SYNC_FLAGS_TYPE_MASK ) {case DRM_XE_SYNC_SYNCOBJ : -if (XE_IOCTL_ERR (xe, no_dma_fences)) +if (XE_IOCTL_ERR (xe, no_dma_fences && signal))return -ENOTSUPP ;if (XE_IOCTL_ERR (xe,upper_32_bits (sync_in.addr)))
@@ -125,7 +127,7 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef,if (XE_IOCTL_ERR (xe, !sync->syncobj))return -ENOENT ; -if (!(sync_in.flags &DRM_XE_SYNC_SIGNAL )) { +if (!signal) { sync->fence =drm_syncobj_fence_get (sync->syncobj);if (XE_IOCTL_ERR (xe, !sync->fence))return -EINVAL ;
@@ -133,7 +135,7 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef,break ;case DRM_XE_SYNC_TIMELINE_SYNCOBJ : -if (XE_IOCTL_ERR (xe, no_dma_fences)) +if (XE_IOCTL_ERR (xe, no_dma_fences && signal))return -ENOTSUPP ;if (XE_IOCTL_ERR (xe,upper_32_bits (sync_in.addr)))
@@ -146,7 +148,7 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef,if (XE_IOCTL_ERR (xe, !sync->syncobj))return -ENOENT ; -if (sync_in.flags &DRM_XE_SYNC_SIGNAL ) { +if (signal) { sync->chain_fence =dma_fence_chain_alloc ();if (!sync->chain_fence)return -ENOMEM ;
@@ -168,7 +170,7 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef,break ;case DRM_XE_SYNC_USER_FENCE : -if (XE_IOCTL_ERR (xe, !(sync_in.flags &DRM_XE_SYNC_SIGNAL ))) +if (XE_IOCTL_ERR (xe, !signal))return -ENOTSUPP ;if (XE_IOCTL_ERR (xe, sync_in.addr &0x7 ))
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 29/30 drm/xe: Allow compute VMs to output dma-fences on binds
In reply to Matthew BrostBinds are not long running jobs thus we can export dma-fences even if a VM is in compute mode.
--- drivers/gpu/drm/xe/xe_vm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_vm.c +1 -1
@@ -3043,7 +3043,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)for (num_syncs =0 ; num_syncs < args->num_syncs; num_syncs++) { err =xe_sync_entry_parse (xe, xef, &syncs[num_syncs], &syncs_user[num_syncs], false, -xe_vm_no_dma_fences (vm)); +xe_vm_in_fault_mode (vm));if (err) goto free_syncs; }
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH 30/30 drm/xe: remove async worker, sync binds, new error handling
In reply to Matthew BrostAsync worker is gone, all jobs and memory allocations done in IOCTL.
Async vs. sync now means when do bind operations complete relative to the IOCTL. Async completes when out-syncs signal while sync completes when the IOCTL returns. In-syncs and out-syncs are only allowed in async mode.
The error handling is similar to before, on memory allocation errors binds are pause, VM is put in an error state, and the bind IOCTL returns -ENOSPC. The user is allowed to issue sync unbinds, with the reclaim bit set, while in an error state. Bind operations without the reclaim bit set are rejected with -EALREADY until the exits the error state. To exit the error issue a restart bind operation which will pick up where the original failure left off.
TODO: Update kernel doc
--- drivers/gpu/drm/xe/xe_engine.c | 7 +- drivers/gpu/drm/xe/xe_engine_types.h | 1 + drivers/gpu/drm/xe/xe_exec.c | 42 -- drivers/gpu/drm/xe/xe_sync.c | 14 +- drivers/gpu/drm/xe/xe_sync.h | 2 +- drivers/gpu/drm/xe/xe_vm.c | 712 ++++++------------------ drivers/gpu/drm/xe/xe_vm.h | 2 - drivers/gpu/drm/xe/xe_vm_types.h | 37 +- drivers/gpu/drm/xe/xe_wait_user_fence.c | 43 +- include/uapi/drm/xe_drm.h | 79 +-- 10 files changed, 213 insertions(+), 726 deletions(-)
drivers/gpu/drm/xe/xe_engine.c +6 -1
@@ -541,7 +541,10 @@ int xe_engine_create_ioctl(struct drm_device *dev, void *data,if (XE_IOCTL_ERR (xe, eci[0 ].gt_id >= xe->info.tile_count))return -EINVAL ; -if (eci[0 ].engine_class ==DRM_XE_ENGINE_CLASS_VM_BIND ) { +if (eci[0 ].engine_class >=DRM_XE_ENGINE_CLASS_VM_BIND_ASYNC ) { +bool sync = eci[0 ].engine_class == +DRM_XE_ENGINE_CLASS_VM_BIND_SYNC ; +for_each_gt (gt, xe, id) {struct xe_engine *new;
@@ -564,6 +567,8 @@ int xe_engine_create_ioctl(struct drm_device *dev, void *data, args->width, hwe,ENGINE_FLAG_PERSISTENT |ENGINE_FLAG_VM | + (sync ?0 : +ENGINE_FLAG_VM_ASYNC ) | (id ?ENGINE_FLAG_BIND_ENGINE_CHILD : 0));
drivers/gpu/drm/xe/xe_engine_types.h +1 -0
@@ -59,6 +59,7 @@ struct xe_engine {#define ENGINE_FLAG_VM BIT(4)#define ENGINE_FLAG_BIND_ENGINE_CHILD BIT(5)#define ENGINE_FLAG_WA BIT(6) +#define ENGINE_FLAG_VM_ASYNC BIT(7)/** * @flags: flags for this engine, should statically setup aside from ban
drivers/gpu/drm/xe/xe_exec.c +0 -42
54 lines not on this page · raw message
drivers/gpu/drm/xe/xe_sync.c +3 -11
@@ -18,7 +18,6 @@#include "xe_sched_job_types.h" #define SYNC_FLAGS_TYPE_MASK 0x3 -#define SYNC_FLAGS_FENCE_INSTALLED 0x10000struct user_fence {struct xe_device *xe;
@@ -221,12 +220,11 @@ int xe_sync_entry_add_deps(struct xe_sync_entry *sync, struct xe_sched_job *job)return 0 ; } -bool xe_sync_entry_signal (struct xe_sync_entry *sync,struct xe_sched_job *job, +void xe_sync_entry_signal (struct xe_sync_entry *sync,struct xe_sched_job *job,struct dma_fence *fence) { -if (!(sync->flags &DRM_XE_SYNC_SIGNAL ) || - sync->flags &SYNC_FLAGS_FENCE_INSTALLED ) -return false; +if (!(sync->flags &DRM_XE_SYNC_SIGNAL )) +return ;if (sync->chain_fence) { drm_syncobj_add_point(sync->syncobj, sync->chain_fence,
@@ -258,12 +256,6 @@ bool xe_sync_entry_signal(struct xe_sync_entry *sync, struct xe_sched_job *job, job->user_fence.addr = sync->addr; job->user_fence.value = sync->timeline_value; } - -/* TODO: external BO? */ - - sync->flags |=SYNC_FLAGS_FENCE_INSTALLED ; - -return true; }void xe_sync_entry_cleanup (struct xe_sync_entry *sync)
drivers/gpu/drm/xe/xe_sync.h +1 -1
@@ -19,7 +19,7 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef,int xe_sync_entry_wait (struct xe_sync_entry *sync);int xe_sync_entry_add_deps (struct xe_sync_entry *sync,struct xe_sched_job *job); -bool xe_sync_entry_signal (struct xe_sync_entry *sync, +void xe_sync_entry_signal (struct xe_sync_entry *sync,struct xe_sched_job *job,struct dma_fence *fence);void xe_sync_entry_cleanup (struct xe_sync_entry *sync);
drivers/gpu/drm/xe/xe_vm.c +183 -529
1,084 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +0 -2
@@ -171,8 +171,6 @@ struct dma_fence *xe_vm_rebind(struct xe_vm *vm, bool rebind_worker);int xe_vm_invalidate_vma (struct xe_vma *vma); -int xe_vm_async_fence_wait_start (struct dma_fence *fence); -extern struct ttm_device_funcs xe_ttm_funcs;struct ttm_buffer_object *xe_vm_ttm_bo (struct xe_vm *vm);
drivers/gpu/drm/xe/xe_vm_types.h +5 -32
@@ -15,7 +15,6 @@#include "xe_device_types.h" #include "xe_pt_types.h" -struct async_op_fence ;struct xe_bo ;struct xe_sync_entry ;struct xe_vm ;
@@ -143,7 +142,7 @@ struct xe_vm {/** @flags: flags for this VM, statically setup a creation time */ #define XE_VM_FLAGS_64K BIT(0)#define XE_VM_FLAG_COMPUTE_MODE BIT(1) -#define XE_VM_FLAG_ASYNC_BIND_OPS BIT(2) +#define XE_VM_FLAG_ASYNC_DEFAULT BIT(2)#define XE_VM_FLAG_MIGRATION BIT(3)#define XE_VM_FLAG_SCRATCH_PAGE BIT(4)#define XE_VM_FLAG_FAULT_MODE BIT(5)
@@ -179,40 +178,18 @@ struct xe_vm {*/ struct work_struct destroy_work; -/** @async_ops: async VM operations (bind / unbinds) */ +/** @ops_state: VM operations (bind / unbinds) state*/ struct { -/** @list: list of pending async VM ops */ +/** @list: list of pending VM ops */ struct list_head pending; -/** @work: worker to execute async VM ops */ -struct work_struct work; -/** @lock: protects list of pending async VM ops and fences */ -spinlock_t lock; -/** @error_capture: error capture state */ -struct { -/** @mm: user MM */ -struct mm_struct *mm; -/** -* @addr: user pointer to copy error capture state too -*/ -u64 addr; -/** @wq: user fence wait queue for VM errors */ -wait_queue_head_t wq; - } error_capture; -/** @fence: fence state */ -struct { -/** @context: context of async fence */ -u64 context; -/** @seqno: seqno of async fence */ -u32 seqno; - } fence; -/** @error: error state for async VM ops */ +/** @error: error state for VM ops */ int error;/** * @munmap_rebind_inflight: an munmap style VM bind is in the * middle of a set of ops which requires a rebind at the end. */ bool munmap_rebind_inflight; - } async_ops; + } ops_state;/** @userptr: user pointer state */ struct {
@@ -366,10 +343,6 @@ struct xe_vma_op {u32 num_syncs;/** @link: async operation link */ struct list_head link; -/** -* @fence: async operation fence, signaled on last operation complete -*/ -struct async_op_fence *fence;/** @gt_mask: gt mask for this operation */ u8 gt_mask;/** @flags: operation flags */
drivers/gpu/drm/xe/xe_wait_user_fence.c +5 -38
88 lines not on this page · raw message
include/uapi/drm/xe_drm.h +9 -70
125 lines not on this page · raw message
Signature · 1 line
2.34.1
Patchwork
CI[Intel-xe] ✓ CI.Patch_applied: success for Upstreaming prep / all of mbrosts patches
In reply to Matthew Brost== Series Details ==
== Series Details ==
Series: Upstreaming prep / all of mbrosts patches
URL : https://patchwork.freedesktop.org/series/117156/
State : success
== Summary ==
=== Applying kernel patches on branch 'drm-xe-next' with base: === Base commit: f7b51a4d5 drm/xe: Fix media detection for pre-GMD_ID platforms === git am output follows ===
.git/rebase-apply/patch:26: trailing whitespace.
warning: 1 line adds whitespace errors.
Applying: drm/sched: Add run_wq argument to drm_sched_init
Applying: drm/sched: Move schedule policy to scheduler
Applying: drm/sched: Add DRM_SCHED_POLICY_SINGLE_ENTITY scheduling policy
Applying: drm/xe: Use DRM_SCHED_POLICY_SINGLE_ENTITY mode
Applying: drm/xe: Long running job update
Applying: drm/xe: Ensure LR engines are not persistent
Applying: drm/xe: Only try to lock external BOs in VM bind
Applying: drm/xe: VM LRU bulk move
Applying: drm/xe/guc: Read HXG fields from DW1 of G2H response
Applying: drm/xe/guc: Return the lower part of blocking H2G message
Applying: drm/xe/guc: Use doorbells for submission if possible
Applying: drm/xe/guc: Print doorbell ID in GuC engine debugfs entry
Applying: maple_tree: split up MA_STATE() macro
Applying: maple_tree: Export mas_preallocate
Applying: drm: manager to keep track of GPUs VA mappings
Applying: drm/xe: Port Xe to GPUVA
Applying: drm/xe: NULL binding implementation
Applying: drm/xe: Avoid doing rebinds
Applying: drm/xe: Reduce the number list links in xe_vma
Applying: drm/xe: Optimize size of xe_vma allocation
Applying: drm/gpuva: Add drm device to GPUVA manager
Applying: drm/gpuva: Move dma-resv to GPUVA manager
Applying: drm/gpuva: Add support for extobj
Applying: drm/xe: Userptr refactor
Applying: drm: execution context for GEM buffers v3
Applying: drm/exec: Always compile drm_exec
Applying: drm/xe: Use drm_exec for locking rather than TTM exec helpers
Applying: drm/xe: Allow dma-fences as in-syncs for compute / faulting VM
Applying: drm/xe: Allow compute VMs to output dma-fences on binds
Applying: drm/xe: remove async worker, sync binds, new error handling
Patchwork
CI[Intel-xe] ✗ CI.KUnit: failure for Upstreaming prep / all of mbrosts patches
In reply to Matthew Brost== Series Details ==
== Series Details ==
Series: Upstreaming prep / all of mbrosts patches
URL : https://patchwork.freedesktop.org/series/117156/
State : failure
== Summary ==
+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig
ERROR:root:In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:../drivers/gpu/drm/xe/tests/xe_bo.c: In function ‘evict_test_run_gt’:../drivers/gpu/drm/xe/tests/xe_bo.c:185:18: error: passing argument 2 of ‘xe_vm_lock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
185 | xe_vm_lock(vm, &ww, 0, false);
| ^~~
| |
| struct ww_acquire_ctx *
In file included from ../drivers/gpu/drm/xe/xe_trace.h:21,
from ../drivers/gpu/drm/xe/xe_bo.c:26:
../drivers/gpu/drm/xe/xe_vm.h:42:51: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
42 | int xe_vm_lock(struct xe_vm *vm, struct drm_exec *exec,
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:188:20: error: passing argument 2 of ‘xe_vm_unlock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
188 | xe_vm_unlock(vm, &ww);
| ^~~
| |
| struct ww_acquire_ctx *
In file included from ../drivers/gpu/drm/xe/xe_trace.h:21,
from ../drivers/gpu/drm/xe/xe_bo.c:26:
../drivers/gpu/drm/xe/xe_vm.h:44:54: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
44 | void xe_vm_unlock(struct xe_vm *vm, struct drm_exec *exec);
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:201:24: error: passing argument 2 of ‘xe_bo_lock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
201 | xe_bo_lock(external, &ww, 0, false);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1733:51: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1733 | int xe_bo_lock(struct xe_bo *bo, struct drm_exec *exec,
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:203:26: error: passing argument 2 of ‘xe_bo_unlock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
203 | xe_bo_unlock(external, &ww);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1754:54: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1754 | void xe_bo_unlock(struct xe_bo *bo, struct drm_exec *exec)
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:243:19: error: passing argument 2 of ‘xe_vm_lock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
243 | xe_vm_lock(vm, &ww, 0, false);
| ^~~
| |
| struct ww_acquire_ctx *
In file included from ../drivers/gpu/drm/xe/xe_trace.h:21,
from ../drivers/gpu/drm/xe/xe_bo.c:26:
../drivers/gpu/drm/xe/xe_vm.h:42:51: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
42 | int xe_vm_lock(struct xe_vm *vm, struct drm_exec *exec,
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:245:21: error: passing argument 2 of ‘xe_vm_unlock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
245 | xe_vm_unlock(vm, &ww);
| ^~~
| |
| struct ww_acquire_ctx *
In file included from ../drivers/gpu/drm/xe/xe_trace.h:21,
from ../drivers/gpu/drm/xe/xe_bo.c:26:
../drivers/gpu/drm/xe/xe_vm.h:44:54: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
44 | void xe_vm_unlock(struct xe_vm *vm, struct drm_exec *exec);
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:252:25: error: passing argument 2 of ‘xe_bo_lock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
252 | xe_bo_lock(external, &ww, 0, false);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1733:51: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1733 | int xe_bo_lock(struct xe_bo *bo, struct drm_exec *exec,
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:254:27: error: passing argument 2 of ‘xe_bo_unlock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
254 | xe_bo_unlock(external, &ww);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1754:54: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1754 | void xe_bo_unlock(struct xe_bo *bo, struct drm_exec *exec)
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:262:24: error: passing argument 2 of ‘xe_bo_lock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
262 | xe_bo_lock(external, &ww, 0, false);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1733:51: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1733 | int xe_bo_lock(struct xe_bo *bo, struct drm_exec *exec,
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:264:26: error: passing argument 2 of ‘xe_bo_unlock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
264 | xe_bo_unlock(external, &ww);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1754:54: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1754 | void xe_bo_unlock(struct xe_bo *bo, struct drm_exec *exec)
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:271:24: error: passing argument 2 of ‘xe_bo_lock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
271 | xe_bo_lock(external, &ww, 0, false);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1733:51: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1733 | int xe_bo_lock(struct xe_bo *bo, struct drm_exec *exec,
| ~~~~~~~~~~~~~~~~~^~~~
In file included from ../drivers/gpu/drm/xe/xe_bo.c:1969:
../drivers/gpu/drm/xe/tests/xe_bo.c:273:26: error: passing argument 2 of ‘xe_bo_unlock’ from incompatible pointer type [-Werror=incompatible-pointer-types]
273 | xe_bo_unlock(external, &ww);
| ^~~
| |
| struct ww_acquire_ctx *
../drivers/gpu/drm/xe/xe_bo.c:1754:54: note: expected ‘struct drm_exec *’ but argument is of type ‘struct ww_acquire_ctx *’
1754 | void xe_bo_unlock(struct xe_bo *bo, struct drm_exec *exec)
| ~~~~~~~~~~~~~~~~~^~~~cc1: some warnings being treated as errors
make[6]: *** [../scripts/Makefile.build:252: drivers/gpu/drm/xe/xe_bo.o] Error 1 make[6]: *** Waiting for unfinished jobs....
make[5]: *** [../scripts/Makefile.build:494: drivers/gpu/drm/xe] Error 2 make[5]: *** Waiting for unfinished jobs....
make[4]: *** [../scripts/Makefile.build:494: drivers/gpu/drm] Error 2 make[3]: *** [../scripts/Makefile.build:494: drivers/gpu] Error 2 make[3]: *** Waiting for unfinished jobs....
make[2]: *** [../scripts/Makefile.build:494: drivers] Error 2 make[2]: *** Waiting for unfinished jobs....
make[1]: *** [/kernel/Makefile:2025: .] Error 2
make: *** [Makefile:226: __sub-make] Error 2
[07:54:32] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [07:54:36] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make ARCH=um O=.kunit --jobs=48 + cleanup ++ stat -c %u:%g /kernel + chown -R 1003:1003 /kernel