Skip to contentSkip to conversation

*** Dedicated vmid per process ***

PATCH In review Chunming Zhou 3 participants 12 messages Apr 21, 2017 6 messages not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover *** Dedicated vmid per process ***

The current kernel implementation, which grabs the idle VMID from pool when emitting the job may:

The back-to-back submission from one process could use different VMID. The submission to different queues from single process could use different VMID

It works well in most case but cannot work for the SQ thread trace capture.

The VMID for the submission that set the {SQTT}_BASE, which refers to the address of the trace buffer, is stored in shader engine.

If the profiling application have to use different VMIDs to submit IBs in its life cycle:

Some trace is not captured since it actually uses different VMID to submit jobs.
Some part of captured trace may come from different application since they are accidentally uses the owner’s VMID to submit jobs.

ChunmingrZhour(5):
drm/amdgpu: handle flags when context alloc drm/amdgpu: add dedicated vmid field in vm struct drm/amdgpu: add context flagn for dedicated vmid reqest drm/amdgpu: implement grab dedicated vmid drm/amdgpu: add limitation for dedicated vm number

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c    |  14 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     | 137 ++++++++++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h     |   6 ++
 include/uapi/drm/amdgpu_drm.h              |   3 +
 6 files changed, 158 insertions(+), 4 deletions(-)
Signature · 1 line
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1492769142-31180-1-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#2 lore raw

PATCH 1/5 drm/amdgpu: handle flags when context alloc

Change-Id I652d89200c38a238ff50092107fa1ee31317728a
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +6 -3
index ff20e11..c2b2896 100644
@@ -103,12 +103,14 @@ static void amdgpu_ctx_fini(struct amdgpu_ctx *ctx)
 
 static int amdgpu_ctx_alloc(struct amdgpu_device *adev,
 			    struct amdgpu_fpriv *fpriv,
-			    uint32_t *id)
+			    uint32_t *id, uint32_t flags)
 {
 	struct amdgpu_ctx_mgr *mgr = &fpriv->ctx_mgr;
 	struct amdgpu_ctx *ctx;
 	int r;
 
+	if (flags)
+		return -EINVAL;
 	ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);
 	if (!ctx)
 		return -ENOMEM;
@@ -199,7 +201,7 @@ int amdgpu_ctx_ioctl(struct drm_device *dev, void *data,
 		     struct drm_file *filp)
 {
 	int r;
-	uint32_t id;
+	uint32_t id, flags;
 
 	union drm_amdgpu_ctx *args = data;
 	struct amdgpu_device *adev = dev->dev_private;
@@ -210,7 +212,8 @@ int amdgpu_ctx_ioctl(struct drm_device *dev, void *data,
 
 	switch (args->in.op) {
 	case AMDGPU_CTX_OP_ALLOC_CTX:
-		r = amdgpu_ctx_alloc(adev, fpriv, &id);
+		flags = args->in.flags;
+		r = amdgpu_ctx_alloc(adev, fpriv, &id, flags);
 		args->out.alloc.ctx_id = id;
 		break;
 	case AMDGPU_CTX_OP_FREE_CTX:
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1492769142-31180-2-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

Reviewed-by
#4 lore raw

Re: [PATCH 1/5] drm/amdgpu: handle flags when context alloc

Am 21.04.2017 um 12:05 schrieb Chunming Zhou:

Quoted 2 lines
Change-Id I652d89200c38a238ff50092107fa1ee31317728a
Quoted 34 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 9 ++++++---
  1 file changed, 6 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +4 -2
index ff20e11..c2b2896 100644
@@ -103,12 +103,14 @@ static void amdgpu_ctx_fini(struct amdgpu_ctx *ctx)
 
  static int amdgpu_ctx_alloc(struct amdgpu_device *adev,
  			    struct amdgpu_fpriv *fpriv,
-			    uint32_t *id)
+			    uint32_t *id, uint32_t flags)
  {
  	struct amdgpu_ctx_mgr *mgr = &fpriv->ctx_mgr;
  	struct amdgpu_ctx *ctx;
  	int r;
 
+	if (flags)
+		return -EINVAL;
  	ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);
  	if (!ctx)
  		return -ENOMEM;
@@ -199,7 +201,7 @@ int amdgpu_ctx_ioctl(struct drm_device *dev, void *data,
  		     struct drm_file *filp)
  {
  	int r;
-	uint32_t id;
+	uint32_t id, flags;
 
  	union drm_amdgpu_ctx *args = data;
  	struct amdgpu_device *adev = dev->dev_private;

drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c int amdgpu_ctx_ioctl(struct drm_device *dev, void *data, · line 219

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +2 -1
@@ -210,7 +212,8 @@ int amdgpu_ctx_ioctl(struct drm_device *dev, void *data,
 
  	switch (args->in.op) {
  	case AMDGPU_CTX_OP_ALLOC_CTX:
-		r = amdgpu_ctx_alloc(adev, fpriv, &id);
+		flags = args->in.flags;
+		r = amdgpu_ctx_alloc(adev, fpriv, &id, flags);
  		args->out.alloc.ctx_id = id;
  		break;
  	case AMDGPU_CTX_OP_FREE_CTX:

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#5 lore raw

PATCH 2/5 drm/amdgpu: add dedicated vmid field in vm struct

In reply to a message not in the archive
Change-Id Id728e20366c8a1ae90d4e901dc80e136e2a613bb
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 17 ++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h |  2 ++
 2 files changed, 18 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +16 -1
index f804d38..e7dd8d1 100644
@@ -2144,10 +2144,12 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	unsigned ring_instance;
 	struct amdgpu_ring *ring;
 	struct amd_sched_rq *rq;
-	int r;
+	int r, i;
 
 	vm->va = RB_ROOT;
 	vm->client_id = atomic64_inc_return(&adev->vm_manager.client_counter);
+	for (i = 0; i < AMDGPU_MAX_VMHUBS; i++)
+		vm->dedicated_vmid[i] = NULL;
 	spin_lock_init(&vm->status_lock);
 	INIT_LIST_HEAD(&vm->invalidated);
 	INIT_LIST_HEAD(&vm->cleared);
@@ -2250,6 +2252,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 {
 	struct amdgpu_bo_va_mapping *mapping, *tmp;
 	bool prt_fini_needed = !!adev->gart.gart_funcs->set_prt;
+	int i;
 
 	if (vm->is_kfd_vm) {
 		struct amdgpu_vm_id_manager *id_mgr =
@@ -2292,6 +2295,18 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 
 	amdgpu_vm_free_levels(&vm->root);
 	fence_put(vm->last_dir_update);
+	for (i = 0; i < AMDGPU_MAX_VMHUBS; i++) {
+		struct amdgpu_vm_id_manager *id_mgr =
+			&adev->vm_manager.id_mgr[i];
+
+		mutex_lock(&id_mgr->lock);
+		if (vm->dedicated_vmid[i]) {
+			list_add(&vm->dedicated_vmid[i]->list,
+				 &id_mgr->ids_lru);
+			vm->dedicated_vmid[i] = NULL;
+		}
+		mutex_unlock(&id_mgr->lock);
+	}
 }
 
 /**
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +2 -0
index 0f547c6..a056e01 100644
@@ -122,6 +122,8 @@ struct amdgpu_vm {
 
 	/* client id */
 	u64                     client_id;
+	/* dedicated vmid */
+	struct amdgpu_vm_id	*dedicated_vmid[AMDGPU_MAX_VMHUBS];
 	/* each VM will map on CSA */
 	struct amdgpu_bo_va *csa_bo_va;
 
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#6 lore raw

PATCH 3/5 drm/amdgpu: add context flags for dedicated vmid reqest

In reply to a message not in the archive

ctx_alloc will check dedicated flag for its process by this flag.

Change-Id I5f80dc39dc9d44660a96a2b710b0dbb4d3b9039d
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c |  9 ++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 50 +++++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  4 +++
 include/uapi/drm/amdgpu_drm.h           |  3 ++
 4 files changed, 64 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +7 -2
index c2b2896..b620923 100644
@@ -109,8 +109,13 @@ static int amdgpu_ctx_alloc(struct amdgpu_device *adev,
 	struct amdgpu_ctx *ctx;
 	int r;
 
-	if (flags)
-		return -EINVAL;
+	if ((flags & AMDGPU_CTX_FLAGS_DEDICATED_VMID) &&
+	    !amdgpu_vm_dedicated_vmid_ready(&fpriv->vm)) {
+		r = amdgpu_vm_alloc_dedicated_vmid(adev, &fpriv->vm);
+		if (r)
+			return r;
+	}
+
 	ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);
 	if (!ctx)
 		return -ENOMEM;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +50 -0

62 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +4 -0
index a056e01..90453af 100644
@@ -212,6 +212,10 @@ int amdgpu_vm_alloc_pts(struct amdgpu_device *adev,
 int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
 		      struct amdgpu_sync *sync, struct fence *fence,
 		      struct amdgpu_job *job);
+int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
+				   struct amdgpu_vm *vm);
+bool amdgpu_vm_dedicated_vmid_ready(struct amdgpu_vm *vm);
+
 int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job);
 void amdgpu_vm_reset_id(struct amdgpu_device *adev, unsigned vmhub,
 			unsigned vmid);
include/uapi/drm/amdgpu_drm.h +3 -0
index 56b7a2f3..754208e 100644
@@ -182,6 +182,9 @@ struct drm_amdgpu_bo_list_out {
 /* unknown cause */
 #define AMDGPU_CTX_UNKNOWN_RESET	3
 
+/* context flags */
+#define AMDGPU_CTX_FLAGS_DEDICATED_VMID	(1ULL << 0)
+
 struct drm_amdgpu_ctx_in {
 	/** AMDGPU_CTX_OP_* */
 	__u32	op;
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1492769142-31180-4-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#8 lore raw

Re: [PATCH 3/5] drm/amdgpu: add context flags for dedicated vmid reqest

Am 21.04.2017 um 12:05 schrieb Chunming Zhou:

Quoted 119 lines

ctx_alloc will check dedicated flag for its process by this flag.

Change-Id I5f80dc39dc9d44660a96a2b710b0dbb4d3b9039d
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c |  9 ++++--
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 50 +++++++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  4 +++
  include/uapi/drm/amdgpu_drm.h           |  3 ++
  4 files changed, 64 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +7 -2
index c2b2896..b620923 100644
@@ -109,8 +109,13 @@ static int amdgpu_ctx_alloc(struct amdgpu_device *adev,
  	struct amdgpu_ctx *ctx;
  	int r;
 
-	if (flags)
-		return -EINVAL;
+	if ((flags & AMDGPU_CTX_FLAGS_DEDICATED_VMID) &&
+	    !amdgpu_vm_dedicated_vmid_ready(&fpriv->vm)) {
+		r = amdgpu_vm_alloc_dedicated_vmid(adev, &fpriv->vm);
+		if (r)
+			return r;
+	}
+
  	ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);
  	if (!ctx)
  		return -ENOMEM;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +50 -0
index e7dd8d1..21cca99 100644
@@ -397,6 +397,17 @@ static bool amdgpu_vm_had_gpu_reset(struct amdgpu_device *adev,
  		atomic_read(&adev->gpu_reset_counter);
  }
 
+bool amdgpu_vm_dedicated_vmid_ready(struct amdgpu_vm *vm)
+{
+	unsigned vmhub;
+
+	for (vmhub = 0; vmhub < AMDGPU_MAX_VMHUBS; vmhub++) {
+		if (!vm->dedicated_vmid[vmhub])
+			return false;
+	}
+	return true;
+}
+
  /**
   * amdgpu_vm_grab_id - allocate the next free VMID
   *
@@ -546,6 +557,45 @@ int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
  	return r;
  }
 
+int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
+				   struct amdgpu_vm *vm)
+{
+	struct amdgpu_vm_id_manager *id_mgr;
+	struct amdgpu_vm_id *idle;
+	unsigned vmhub;
+	int r;
+
+	for (vmhub = 0; vmhub < AMDGPU_MAX_VMHUBS; vmhub++) {
+		id_mgr = &adev->vm_manager.id_mgr[vmhub];
+
+		mutex_lock(&id_mgr->lock);
+		/* Select the first entry VMID */
+		idle = list_first_entry(&id_mgr->ids_lru, struct amdgpu_vm_id,
+					list);
+		list_del_init(&idle->list);
+		vm->dedicated_vmid[vmhub] = idle;
+		mutex_unlock(&id_mgr->lock);
+
+		r = amdgpu_sync_wait(&idle->active);
+		if (r)
+			goto err;
+	}
+
+	return 0;
+err:
+	for (vmhub = 0; vmhub < AMDGPU_MAX_VMHUBS; vmhub++) {
+		id_mgr = &adev->vm_manager.id_mgr[vmhub];
+
+		mutex_lock(&id_mgr->lock);
+		if (vm->dedicated_vmid[vmhub])
+			list_add(&vm->dedicated_vmid[vmhub]->list,
+				 &id_mgr->ids_lru);
+		vm->dedicated_vmid[vmhub] = NULL;
+		mutex_unlock(&id_mgr->lock);
+	}
+	return r;
+}
+
  static bool amdgpu_vm_ring_has_compute_vm_bug(struct amdgpu_ring *ring)
  {
  	struct amdgpu_device *adev = ring->adev;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +4 -0
index a056e01..90453af 100644
@@ -212,6 +212,10 @@ int amdgpu_vm_alloc_pts(struct amdgpu_device *adev,
  int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
  		      struct amdgpu_sync *sync, struct fence *fence,
  		      struct amdgpu_job *job);
+int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
+				   struct amdgpu_vm *vm);
+bool amdgpu_vm_dedicated_vmid_ready(struct amdgpu_vm *vm);
+
  int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job);
  void amdgpu_vm_reset_id(struct amdgpu_device *adev, unsigned vmhub,
  			unsigned vmid);
include/uapi/drm/amdgpu_drm.h +0 -0
index 56b7a2f3..754208e 100644

include/uapi/drm/amdgpu_drm.h struct drm_amdgpu_bo_list_out { · line 186

Quoted 5 lines
include/uapi/drm/amdgpu_drm.h +2 -0
@@ -182,6 +182,9 @@ struct drm_amdgpu_bo_list_out {
  /* unknown cause */
  #define AMDGPU_CTX_UNKNOWN_RESET	3
 
+/* context flags */
+#define AMDGPU_CTX_FLAGS_DEDICATED_VMID	(1ULL << 0)

At minimum we should call this AMDGPU_CTX_FLAGS_TRACED or something like that.

Allocating a static VMID is the effect it has currently, but not necessary in the future.

On the other hand putting it on the context looks more and more like a bad idea to me.

Christian.

include/uapi/drm/amdgpu_drm.h struct drm_amdgpu_bo_list_out { · line 190

Quoted 4 lines
include/uapi/drm/amdgpu_drm.h +1 -0
+
  struct drm_amdgpu_ctx_in {
  	/** AMDGPU_CTX_OP_* */
  	__u32	op;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#9 lore raw

PATCH 4/5 drm/amdgpu: implement grab dedicated vmid

In reply to a message not in the archive
Change-Id I64da2701c9fdcf986afb90ba1492a78d5bef1b6c
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 60 ++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +60 -0
index 21cca99..5764a14 100644
@@ -408,6 +408,62 @@ bool amdgpu_vm_dedicated_vmid_ready(struct amdgpu_vm *vm)
 	return true;
 }
 
+int amdgpu_vm_grab_dedicated_vmid(struct amdgpu_vm *vm,
+				  struct amdgpu_ring *ring,
+				  struct amdgpu_sync *sync,
+				  struct fence *fence,
+				  struct amdgpu_job *job)
+{
+	struct amdgpu_device *adev = ring->adev;
+	unsigned vmhub = ring->funcs->vmhub;
+	struct amdgpu_vm_id *id = vm->dedicated_vmid[vmhub];
+	struct amdgpu_vm_id_manager *id_mgr = &adev->vm_manager.id_mgr[vmhub];
+	struct fence *updates = sync->last_vm_update;
+	int r = 0;
+	struct fence *flushed, *tmp;
+	bool needs_flush = false;
+
+	mutex_lock(&id_mgr->lock);
+	tmp = amdgpu_sync_get_fence(&id->active);
+	if (tmp) {
+		r = amdgpu_sync_fence(adev, sync, tmp);
+		fence_put(tmp);
+		mutex_unlock(&id_mgr->lock);
+		return r;
+	}
+
+	if (amdgpu_vm_had_gpu_reset(adev, id))
+		needs_flush = true;
+
+	flushed  = id->flushed_updates;
+	if (updates && (!flushed || updates->context != flushed->context ||
+			fence_is_later(updates, flushed)))
+		needs_flush = true;
+
+	/* Good we can use this VMID. Remember this submission as
+	* user of the VMID.
+	*/
+	r = amdgpu_sync_fence(ring->adev, &id->active, fence);
+	if (r)
+		goto out;
+
+	if (updates && (!flushed || updates->context != flushed->context ||
+			fence_is_later(updates, flushed))) {
+		fence_put(id->flushed_updates);
+		id->flushed_updates = fence_get(updates);
+	}
+	id->pd_gpu_addr = job->vm_pd_addr;
+	id->current_gpu_reset_count = atomic_read(&adev->gpu_reset_counter);
+	atomic64_set(&id->owner, vm->client_id);
+	job->vm_needs_flush = needs_flush;
+
+	job->vm_id = id - id_mgr->ids;
+	trace_amdgpu_vm_grab_id(vm, ring, job);
+out:
+	mutex_unlock(&id_mgr->lock);
+	return r;
+}
+
 /**
  * amdgpu_vm_grab_id - allocate the next free VMID
  *
@@ -432,6 +488,10 @@ int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
 	unsigned i;
 	int r = 0;
 
+	if (amdgpu_vm_dedicated_vmid_ready(vm))
+		return amdgpu_vm_grab_dedicated_vmid(vm, ring, sync,
+						     fence, job);
+
 	fences = kmalloc_array(sizeof(void *), id_mgr->num_ids, GFP_KERNEL);
 	if (!fences)
 		return -ENOMEM;
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1492769142-31180-5-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#11 lore raw

Re: [PATCH 4/5] drm/amdgpu: implement grab dedicated vmid

Am 21.04.2017 um 12:05 schrieb Chunming Zhou:

Quoted 27 lines
Change-Id I64da2701c9fdcf986afb90ba1492a78d5bef1b6c
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 60 ++++++++++++++++++++++++++++++++++
  1 file changed, 60 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +13 -0
index 21cca99..5764a14 100644
@@ -408,6 +408,62 @@ bool amdgpu_vm_dedicated_vmid_ready(struct amdgpu_vm *vm)
  	return true;
  }
 
+int amdgpu_vm_grab_dedicated_vmid(struct amdgpu_vm *vm,
+				  struct amdgpu_ring *ring,
+				  struct amdgpu_sync *sync,
+				  struct fence *fence,
+				  struct amdgpu_job *job)
+{
+	struct amdgpu_device *adev = ring->adev;
+	unsigned vmhub = ring->funcs->vmhub;
+	struct amdgpu_vm_id *id = vm->dedicated_vmid[vmhub];
+	struct amdgpu_vm_id_manager *id_mgr = &adev->vm_manager.id_mgr[vmhub];
+	struct fence *updates = sync->last_vm_update;
+	int r = 0;
+	struct fence *flushed, *tmp;

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c bool amdgpu_vm_dedicated_vmid_ready(struct amdgpu_vm *vm) · line 433

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +10 -0
+	bool needs_flush = false;
+
+	mutex_lock(&id_mgr->lock);
+	tmp = amdgpu_sync_get_fence(&id->active);
+	if (tmp) {
+		r = amdgpu_sync_fence(adev, sync, tmp);
+		fence_put(tmp);
+		mutex_unlock(&id_mgr->lock);
+		return r;
+	}

This only needs to be done when we flush the VMID, otherwise it is unnecessary and really kills performance.

Christian.

Quoted 37 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +33 -0
+
+	if (amdgpu_vm_had_gpu_reset(adev, id))
+		needs_flush = true;
+
+	flushed  = id->flushed_updates;
+	if (updates && (!flushed || updates->context != flushed->context ||
+			fence_is_later(updates, flushed)))
+		needs_flush = true;
+
+	/* Good we can use this VMID. Remember this submission as
+	* user of the VMID.
+	*/
+	r = amdgpu_sync_fence(ring->adev, &id->active, fence);
+	if (r)
+		goto out;
+
+	if (updates && (!flushed || updates->context != flushed->context ||
+			fence_is_later(updates, flushed))) {
+		fence_put(id->flushed_updates);
+		id->flushed_updates = fence_get(updates);
+	}
+	id->pd_gpu_addr = job->vm_pd_addr;
+	id->current_gpu_reset_count = atomic_read(&adev->gpu_reset_counter);
+	atomic64_set(&id->owner, vm->client_id);
+	job->vm_needs_flush = needs_flush;
+
+	job->vm_id = id - id_mgr->ids;
+	trace_amdgpu_vm_grab_id(vm, ring, job);
+out:
+	mutex_unlock(&id_mgr->lock);
+	return r;
+}
+
  /**
   * amdgpu_vm_grab_id - allocate the next free VMID
   *

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring, · line 497

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
@@ -432,6 +488,10 @@ int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
  	unsigned i;
  	int r = 0;
 
+	if (amdgpu_vm_dedicated_vmid_ready(vm))
+		return amdgpu_vm_grab_dedicated_vmid(vm, ring, sync,
+						     fence, job);
+
  	fences = kmalloc_array(sizeof(void *), id_mgr->num_ids, GFP_KERNEL);
  	if (!fences)
  		return -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#12 lore raw

PATCH 5/5 drm/amdgpu: add limitation for dedicated vm number

In reply to a message not in the archive
Change-Id Ie5958cf6dbdc1c8278e61d9158483472d6f5c6e3
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     | 10 ++++++++++
 3 files changed, 12 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 0831cd2..8034500 100644
@@ -1591,6 +1591,7 @@ struct amdgpu_device {
 	struct amdgpu_dummy_page	dummy_page;
 	struct amdgpu_vm_manager	vm_manager;
 	struct amdgpu_vmhub             vmhub[AMDGPU_MAX_VMHUBS];
+	atomic_t			dedicated_vm;
 
 	/* memory management */
 	struct amdgpu_mman		mman;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
index a175dfd..4c0c0ad 100644
@@ -1889,6 +1889,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	adev->vm_manager.vm_pte_num_rings = 0;
 	adev->gart.gart_funcs = NULL;
 	adev->fence_context = kcl_fence_context_alloc(AMDGPU_MAX_RINGS);
+	atomic_set(&adev->dedicated_vm, 0);
 
 	adev->smc_rreg = &amdgpu_invalid_rreg;
 	adev->smc_wreg = &amdgpu_invalid_wreg;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +10 -0
index 5764a14..26bbdfc 100644
@@ -617,6 +617,7 @@ int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
 	return r;
 }
 
+#define AMDGPU_VM_MAX_DEDICATED 2
 int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
 				   struct amdgpu_vm *vm)
 {
@@ -625,6 +626,10 @@ int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
 	unsigned vmhub;
 	int r;
 
+	if (atomic_read(&adev->dedicated_vm) >= AMDGPU_VM_MAX_DEDICATED) {
+		DRM_ERROR("Over limit dedicated vm num\n");
+		return -EINVAL;
+	}
 	for (vmhub = 0; vmhub < AMDGPU_MAX_VMHUBS; vmhub++) {
 		id_mgr = &adev->vm_manager.id_mgr[vmhub];
 
@@ -640,6 +645,7 @@ int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
 		if (r)
 			goto err;
 	}
+	atomic_inc(&adev->dedicated_vm);
 
 	return 0;
 err:
@@ -2362,6 +2368,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 {
 	struct amdgpu_bo_va_mapping *mapping, *tmp;
 	bool prt_fini_needed = !!adev->gart.gart_funcs->set_prt;
+	bool dedicated = false;
 	int i;
 
 	if (vm->is_kfd_vm) {
@@ -2414,9 +2421,12 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 			list_add(&vm->dedicated_vmid[i]->list,
 				 &id_mgr->ids_lru);
 			vm->dedicated_vmid[i] = NULL;
+			dedicated = true;
 		}
 		mutex_unlock(&id_mgr->lock);
 	}
+	if (dedicated)
+		atomic_dec(&adev->dedicated_vm);
 }
 
 /**
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1492769142-31180-6-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#14 lore raw

Re: [PATCH 5/5] drm/amdgpu: add limitation for dedicated vm number

Am 21.04.2017 um 12:05 schrieb Chunming Zhou:

Quoted 2 lines
Change-Id Ie5958cf6dbdc1c8278e61d9158483472d6f5c6e3

That patch should come earlier in the series.

Quoted 11 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     | 10 ++++++++++
  3 files changed, 12 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index 0831cd2..8034500 100644

drivers/gpu/drm/amd/amdgpu/amdgpu.h struct amdgpu_device { · line 1594

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
@@ -1591,6 +1591,7 @@ struct amdgpu_device {
  	struct amdgpu_dummy_page	dummy_page;
  	struct amdgpu_vm_manager	vm_manager;
  	struct amdgpu_vmhub             vmhub[AMDGPU_MAX_VMHUBS];
+	atomic_t			dedicated_vm;

Please put that into the vm_manager structure.

Quoted 20 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
  	/* memory management */
  	struct amdgpu_mman		mman;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
index a175dfd..4c0c0ad 100644
@@ -1889,6 +1889,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
  	adev->vm_manager.vm_pte_num_rings = 0;
  	adev->gart.gart_funcs = NULL;
  	adev->fence_context = kcl_fence_context_alloc(AMDGPU_MAX_RINGS);
+	atomic_set(&adev->dedicated_vm, 0);
 
  	adev->smc_rreg = &amdgpu_invalid_rreg;
  	adev->smc_wreg = &amdgpu_invalid_wreg;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index 5764a14..26bbdfc 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring, · line 620

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
@@ -617,6 +617,7 @@ int amdgpu_vm_grab_id(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
  	return r;
  }
 
+#define AMDGPU_VM_MAX_DEDICATED 2

That should go into into amdgpu_vm.h and maybe not call it dedicated, but rather reserved VMID.

Regards,
Christian.

Quoted 33 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +6 -0
  int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
  				   struct amdgpu_vm *vm)
  {
@@ -625,6 +626,10 @@ int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
  	unsigned vmhub;
  	int r;
 
+	if (atomic_read(&adev->dedicated_vm) >= AMDGPU_VM_MAX_DEDICATED) {
+		DRM_ERROR("Over limit dedicated vm num\n");
+		return -EINVAL;
+	}
  	for (vmhub = 0; vmhub < AMDGPU_MAX_VMHUBS; vmhub++) {
  		id_mgr = &adev->vm_manager.id_mgr[vmhub];
 
@@ -640,6 +645,7 @@ int amdgpu_vm_alloc_dedicated_vmid(struct amdgpu_device *adev,
  		if (r)
  			goto err;
  	}
+	atomic_inc(&adev->dedicated_vm);
 
  	return 0;
  err:
@@ -2362,6 +2368,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
  {
  	struct amdgpu_bo_va_mapping *mapping, *tmp;
  	bool prt_fini_needed = !!adev->gart.gart_funcs->set_prt;
+	bool dedicated = false;
  	int i;
 
  	if (vm->is_kfd_vm) {
@@ -2414,9 +2421,12 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
  			list_add(&vm->dedicated_vmid[i]->list,
  				 &id_mgr->ids_lru);

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm) · line 2432

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +3 -0
  			vm->dedicated_vmid[i] = NULL;
+			dedicated = true;
  		}
  		mutex_unlock(&id_mgr->lock);
  	}
+	if (dedicated)
+		atomic_dec(&adev->dedicated_vm);
  }
 
  /**

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#15 lore raw
In reply to a message not in the archive
Quoted 20 lines

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Chunming Zhou
Sent: Friday, April 21, 2017 6:06 AM
To: [email protected]
Cc: Zhou, David(ChunMing)
Subject: [PATCH 0/5] *** Dedicated vmid per process ***

The current kernel implementation, which grabs the idle VMID from pool when emitting the job may:

The back-to-back submission from one process could use different VMID. The submission to different queues from single process could use different
VMID

It works well in most case but cannot work for the SQ thread trace capture.

The VMID for the submission that set the {SQTT}_BASE, which refers to the address of the trace buffer, is stored in shader engine.

Quoted 7 lines

If the profiling application have to use different VMIDs to submit IBs in its life cycle:

Some trace is not captured since it actually uses different VMID to submit
jobs.
Some part of captured trace may come from different application since
they are accidentally uses the owner’s VMID to submit jobs.

Please add a patch at the end to bump the module version so we can track when the flag is available.

Alex

Quoted 18 lines

ChunmingrZhour(5):
drm/amdgpu: handle flags when context alloc drm/amdgpu: add dedicated vmid field in vm struct drm/amdgpu: add context flagn for dedicated vmid reqest drm/amdgpu: implement grab dedicated vmid drm/amdgpu: add limitation for dedicated vm number

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c    |  14 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     | 137
++++++++++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h     |   6 ++
 include/uapi/drm/amdgpu_drm.h              |   3 +
 6 files changed, 158 insertions(+), 4 deletions(-)
Quoted 2 lines

--
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CY4PR12MB16535C5FAB9CCC54116EEA93F71A0-rpdhrqHFk06apTa93KjAaQdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org

#17 lore raw

Am 21.04.2017 um 18:27 schrieb Deucher, Alexander:

Quoted 20 lines
Quoted 20 lines

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Chunming Zhou
Sent: Friday, April 21, 2017 6:06 AM
To: [email protected]
Cc: Zhou, David(ChunMing)
Subject: [PATCH 0/5] *** Dedicated vmid per process ***

The current kernel implementation, which grabs the idle VMID from pool when emitting the job may:

The back-to-back submission from one process could use different VMID. The submission to different queues from single process could use different
VMID

It works well in most case but cannot work for the SQ thread trace capture.

The VMID for the submission that set the {SQTT}_BASE, which refers to the address of the trace buffer, is stored in shader engine.

Quoted 8 lines
Quoted 7 lines

If the profiling application have to use different VMIDs to submit IBs in its life cycle:

Some trace is not captured since it actually uses different VMID to submit
jobs.
Some part of captured trace may come from different application since
they are accidentally uses the owner’s VMID to submit jobs.

Please add a patch at the end to bump the module version so we can track when the flag is available.

Additional to that we might want to reconsider the interface once more.

When we see a context with that flag we assign a permanent ID to the VM and never free until the process exits. That is quite ugly.

I would either suggest to add an atomic and alloc/free the VMID when the first/last context needing it for a trace in a process is created/freed.

Or completely drop the per context flag and add a new IOCTL to enable it for the whole process.

I prefer option #2, but that is clearly more work.

Regards,
Christian.

Quoted 20 lines

Alex

Quoted 17 lines

ChunmingrZhour(5):
drm/amdgpu: handle flags when context alloc drm/amdgpu: add dedicated vmid field in vm struct drm/amdgpu: add context flagn for dedicated vmid reqest drm/amdgpu: implement grab dedicated vmid drm/amdgpu: add limitation for dedicated vm number

  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c    |  14 ++-
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     | 137
++++++++++++++++++++++++++++-
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h     |   6 ++
  include/uapi/drm/amdgpu_drm.h              |   3 +
  6 files changed, 158 insertions(+), 4 deletions(-)
Quoted 6 lines
Quoted 2 lines

--
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx