Skip to contentSkip to conversation

drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

PATCH In review Jacob He 3 participants 10 messages Mar 2, 2020 – Mar 3, 2020

Subsystems (2)
lore
Share & tools
lore

Jacob He

author
#0 lore raw

PATCH 1/1 drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
index c00696f3017e..c761d3a0b6e8 100644
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
 	struct dma_fence *fence = NULL;
 	bool pasid_mapping_needed = false;
 	unsigned patch_offset = 0;
+	bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
 	int r;
 
+	if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+		adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
 	if (amdgpu_vmid_had_gpu_reset(adev, id)) {
 		gds_switch_needed = true;
 		vm_flush_needed = true;
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#1 lore raw

[AMD Official Use Only - Internal Distribution Only]

Could anyone help to review the patch?

Thanks
Jacob

________________________________
From: He, Jacob <[email protected]>
Sent: Monday, March 2, 2020 1:35:29 PM
To: [email protected] <[email protected]>
Cc: He, Jacob <[email protected]>
Subject: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
index c00696f3017e..c761d3a0b6e8 100644
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
         struct dma_fence *fence = NULL;
         bool pasid_mapping_needed = false;
         unsigned patch_offset = 0;
+       bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
         int r;
 
+       if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+               adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
         if (amdgpu_vmid_had_gpu_reset(adev, id)) {
                 gds_switch_needed = true;
                 vm_flush_needed = true;

--
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#2 lore raw
In reply to Jacob He

Am 02.03.20 um 06:35 schrieb Jacob He:

Quoted 15 lines

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index c00696f3017e..c761d3a0b6e8 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1088

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
  	struct dma_fence *fence = NULL;
  	bool pasid_mapping_needed = false;
  	unsigned patch_offset = 0;
+	bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
  	int r;
 
+	if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+		adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+

It would be better if we could do that asynchronously with a register write on the ring.

The alternative is that we block for the VM to be idle in amdgpu_vm_ioctl() before unreserving the VMID.

In other words lock the reservation object of the root PD and call amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

Regards,
Christian.

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1091

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
  	if (amdgpu_vmid_had_gpu_reset(adev, id)) {
  		gds_switch_needed = true;
  		vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#3 lore raw

[AMD Official Use Only - Internal Distribution Only]

It would be better if we could do that asynchronously with a register write on the ring.

Sorry, I don’t get your point. Could you please elaborate more?

Thanks
Jacob

From: Christian König<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 10:16 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 02.03.20 um 06:35 schrieb Jacob He:

Quoted 15 lines

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index c00696f3017e..c761d3a0b6e8 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1088

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
       struct dma_fence *fence = NULL;
       bool pasid_mapping_needed = false;
       unsigned patch_offset = 0;
+     bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
       int r;
 
+     if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+             adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+

It would be better if we could do that asynchronously with a register write on the ring.

The alternative is that we block for the VM to be idle in amdgpu_vm_ioctl() before unreserving the VMID.

In other words lock the reservation object of the root PD and call amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

Regards,
Christian.

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1091

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
       if (amdgpu_vmid_had_gpu_reset(adev, id)) {
               gds_switch_needed = true;
               vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#4 lore raw

Am 03.03.20 um 15:34 schrieb He, Jacob:

Quoted 11 lines

[AMD Official Use Only - Internal Distribution Only]

/It would be better if we could do that asynchronously with a register write on the ring.

/

Sorry, I don’t get your point. Could you please elaborate more?

You pass the ring from amdgpu_vm_flush() to the *_update_spm_vmid() functions.

And then instead of using WREG32() you call amdgpu_ring_emit_wreg() to make the write asynchronously on the ring buffer using a CP command.

This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.

Regards,
Christian.

Quoted 42 lines

Thanks

Jacob

*From: *Christian König <mailto:[email protected]>
*Sent: *Tuesday, March 3, 2020 10:16 PM
*To: *He, Jacob <mailto:[email protected]>; [email protected] <mailto:[email protected]> *Subject: *Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 02.03.20 um 06:35 schrieb Jacob He:

Review

Quoted 11 lines

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700

---
   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++

   1 file changed, 4 insertions(+)
(quoted diff) +0 -0

b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring,

Quoted 4 lines

index c00696f3017e..c761d3a0b6e8 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, 

struct amdgpu_job *job,

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
         struct dma_fence *fence = NULL;
         bool pasid_mapping_needed = false;
         unsigned patch_offset = 0;
+     bool update_spm_vmid_needed = (job->vm && 

(job->vm->reserved_vmid[vmhub] != NULL));

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, · line 1086

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
         int r;
 
+     if (update_spm_vmid_needed && 

adev->gfx.rlc.funcs->update_spm_vmid)

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1088

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+

It would be better if we could do that asynchronously with a register write on the ring.

Quoted 13 lines

The alternative is that we block for the VM to be idle in amdgpu_vm_ioctl() before unreserving the VMID.

In other words lock the reservation object of the root PD and call amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

Regards,
Christian.

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
         if (amdgpu_vmid_had_gpu_reset(adev, id)) {
                 gds_switch_needed = true;
                 vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#5 lore raw

[AMD Official Use Only - Internal Distribution Only]

Thanks! Could you please take an example of trouble “This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.”?

Thanks
Jacob

From: Koenig, Christian<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 11:03 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 03.03.20 um 15:34 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

It would be better if we could do that asynchronously with a register write on the ring.
Sorry, I don’t get your point. Could you please elaborate more?

You pass the ring from amdgpu_vm_flush() to the *_update_spm_vmid() functions.

And then instead of using WREG32() you call amdgpu_ring_emit_wreg() to make the write asynchronously on the ring buffer using a CP command.

This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.

Regards,
Christian.

Thanks
Jacob

From: Christian König<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 10:16 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 02.03.20 um 06:35 schrieb Jacob He:

Quoted 15 lines

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index c00696f3017e..c761d3a0b6e8 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1088

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
       struct dma_fence *fence = NULL;
       bool pasid_mapping_needed = false;
       unsigned patch_offset = 0;
+     bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
       int r;
 
+     if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+             adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+

It would be better if we could do that asynchronously with a register write on the ring.

The alternative is that we block for the VM to be idle in amdgpu_vm_ioctl() before unreserving the VMID.

In other words lock the reservation object of the root PD and call amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

Regards,
Christian.

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1091

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
       if (amdgpu_vmid_had_gpu_reset(adev, id)) {
               gds_switch_needed = true;
               vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#6 lore raw

See the SPM buffer address is set using CP commands as well, right? And those execute asynchronously.

When we now synchronously update the SPM VMID we risk that we switch from one process to another while the new process is not ready yet with its setup.

That could have quite a bunch of unforeseen consequences, including accidentally writing SPM data into the new process address space at whatever buffer address was used before.

This is something we at least should try to avoid.

Regards,
Christian.

Am 03.03.20 um 16:28 schrieb He, Jacob:

Quoted 85 lines

[AMD Official Use Only - Internal Distribution Only]

Thanks!  Could you please take an example of trouble  “This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.”?

Thanks

Jacob

*From: *Koenig, Christian <mailto:[email protected]>
*Sent: *Tuesday, March 3, 2020 11:03 PM
*To: *He, Jacob <mailto:[email protected]>; [email protected] <mailto:[email protected]> *Subject: *Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 03.03.20 um 15:34 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

/It would be better if we could do that asynchronously with a register
write on the ring./

Sorry, I don’t get your point. Could you please elaborate more?

You pass the ring from amdgpu_vm_flush() to the *_update_spm_vmid() functions.

And then instead of using WREG32() you call amdgpu_ring_emit_wreg() to make the write asynchronously on the ring buffer using a CP command.

This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.

Regards,
Christian.

    Thanks

    Jacob

    *From: *Christian König <mailto:[email protected]>
    *Sent: *Tuesday, March 3, 2020 10:16 PM
    *To: *He, Jacob <mailto:[email protected]>;
    [email protected] <mailto:[email protected]>
    *Subject: *Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's
    vmid when application reserves the vmid

    Am 02.03.20 um 06:35 schrieb Jacob He:
    > SPM access the video memory according to SPM_VMID. It should be
    updated
    > with the job's vmid right before the job is scheduled. SPM_VMID is a
    > global resource
    >
    > Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
    > Signed-off-by: Jacob He <[email protected]> <mailto:[email protected]>
    > ---
    >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
    >   1 file changed, 4 insertions(+)
    >
    > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
    b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
    > index c00696f3017e..c761d3a0b6e8 100644
    > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
    > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
    > @@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring
    *ring, struct amdgpu_job *job,
    >        struct dma_fence *fence = NULL;
    >        bool pasid_mapping_needed = false;
    >        unsigned patch_offset = 0;
    > +     bool update_spm_vmid_needed = (job->vm &&
    (job->vm->reserved_vmid[vmhub] != NULL));
    >        int r;
    >
    > +     if (update_spm_vmid_needed &&
    adev->gfx.rlc.funcs->update_spm_vmid)
    > + adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
    > +

    It would be better if we could do that asynchronously with a register
    write on the ring.

    The alternative is that we block for the VM to be idle in
    amdgpu_vm_ioctl() before unreserving the VMID.

    In other words lock the reservation object of the root PD and call
Quoted 9 lines
    amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

    Regards,
    Christian.

    >        if (amdgpu_vmid_had_gpu_reset(adev, id)) {
    >                gds_switch_needed = true;
    >                vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#7 lore raw

[AMD Official Use Only - Internal Distribution Only]

Oh, you are right! If SPM_VMID is updated by other process while the SPM enabled commands is executing, that will cause VM fault.

Is the wait vm idle right before unreserve vmid still necessary if using asynchroneously setting SPM_VMID?

Thanks
Jacob

From: Koenig, Christian<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 11:36 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

See the SPM buffer address is set using CP commands as well, right? And those execute asynchronously.

When we now synchronously update the SPM VMID we risk that we switch from one process to another while the new process is not ready yet with its setup.

That could have quite a bunch of unforeseen consequences, including accidentally writing SPM data into the new process address space at whatever buffer address was used before.

This is something we at least should try to avoid.

Regards,
Christian.

Am 03.03.20 um 16:28 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

Thanks! Could you please take an example of trouble “This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.”?

Thanks
Jacob

From: Koenig, Christian<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 11:03 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 03.03.20 um 15:34 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

It would be better if we could do that asynchronously with a register write on the ring.
Sorry, I don’t get your point. Could you please elaborate more?

You pass the ring from amdgpu_vm_flush() to the *_update_spm_vmid() functions.

And then instead of using WREG32() you call amdgpu_ring_emit_wreg() to make the write asynchronously on the ring buffer using a CP command.

This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.

Regards,
Christian.

Thanks
Jacob

From: Christian König<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 10:16 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 02.03.20 um 06:35 schrieb Jacob He:

Quoted 15 lines

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index c00696f3017e..c761d3a0b6e8 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1088

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
       struct dma_fence *fence = NULL;
       bool pasid_mapping_needed = false;
       unsigned patch_offset = 0;
+     bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
       int r;
 
+     if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+             adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+

It would be better if we could do that asynchronously with a register write on the ring.

The alternative is that we block for the VM to be idle in amdgpu_vm_ioctl() before unreserving the VMID.

In other words lock the reservation object of the root PD and call amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

Regards,
Christian.

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1091

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
       if (amdgpu_vmid_had_gpu_reset(adev, id)) {
               gds_switch_needed = true;
               vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#8 lore raw

Am 03.03.20 um 17:07 schrieb He, Jacob:

Quoted 8 lines

[AMD Official Use Only - Internal Distribution Only]

Oh, you are right! If SPM_VMID is updated by other process while the SPM enabled commands is executing, that will cause VM fault.

Is the wait vm idle right before unreserve vmid still necessary if using asynchroneously setting SPM_VMID?

No, that are alternative approaches.

Updating the VMID asynchronously sounds a bit cleaner to me, but feel free to pick whatever is easier for you to implement.

Regards,
Christian.

Quoted 121 lines

Thanks

Jacob

*From: *Koenig, Christian <mailto:[email protected]>
*Sent: *Tuesday, March 3, 2020 11:36 PM
*To: *He, Jacob <mailto:[email protected]>; [email protected] <mailto:[email protected]> *Subject: *Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

See the SPM buffer address is set using CP commands as well, right? And those execute asynchronously.

When we now synchronously update the SPM VMID we risk that we switch from one process to another while the new process is not ready yet with its setup.

That could have quite a bunch of unforeseen consequences, including accidentally writing SPM data into the new process address space at whatever buffer address was used before.

This is something we at least should try to avoid.

Regards,
Christian.

Am 03.03.20 um 16:28 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

Thanks!  Could you please take an example of trouble  “This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.”?

Thanks

Jacob

*From: *Koenig, Christian <mailto:[email protected]> *Sent: *Tuesday, March 3, 2020 11:03 PM
*To: *He, Jacob <mailto:[email protected]>; [email protected] <mailto:[email protected]> *Subject: *Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 03.03.20 um 15:34 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

/It would be better if we could do that asynchronously with a register
write on the ring./

Sorry, I don’t get your point. Could you please elaborate more?

You pass the ring from amdgpu_vm_flush() to the *_update_spm_vmid() functions.

And then instead of using WREG32() you call amdgpu_ring_emit_wreg() to make the write asynchronously on the ring buffer using a CP command.

This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.

Regards,
Christian.

Thanks

Jacob

*From: *Christian König <mailto:[email protected]> *Sent: *Tuesday, March 3, 2020 10:16 PM
*To: *He, Jacob <mailto:[email protected]>; [email protected] <mailto:[email protected]>
*Subject: *Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 02.03.20 um 06:35 schrieb Jacob He:
> SPM access the video memory according to SPM_VMID. It should be updated
> with the job's vmid right before the job is scheduled. SPM_VMID is a
> global resource
>
> Change-Id: Id3881908960398f87e7c95026a54ff83ff826700 > Signed-off-by: Jacob He <[email protected]> <mailto:[email protected]>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++ >   1 file changed, 4 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index c00696f3017e..c761d3a0b6e8 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
>        struct dma_fence *fence = NULL;
>        bool pasid_mapping_needed = false;
>        unsigned patch_offset = 0;
> +     bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
>        int r;
>
> +     if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
> + adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid); > +

It would be better if we could do that asynchronously with a register
write on the ring.

The alternative is that we block for the VM to be idle in amdgpu_vm_ioctl() before unreserving the VMID.

In other words lock the reservation object of the root PD and call
amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

Quoted 7 lines

Regards,
Christian.

>        if (amdgpu_vmid_had_gpu_reset(adev, id)) { >                gds_switch_needed = true; >                vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#9 lore raw

[AMD Official Use Only - Internal Distribution Only]

Thanks million! I’ll change the patch.

-Jacob

________________________________
From: Koenig, Christian <[email protected]>
Sent: Wednesday, March 4, 2020 12:09:36 AM
To: He, Jacob <[email protected]>; [email protected] <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 03.03.20 um 17:07 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

Oh, you are right! If SPM_VMID is updated by other process while the SPM enabled commands is executing, that will cause VM fault.

Is the wait vm idle right before unreserve vmid still necessary if using asynchroneously setting SPM_VMID?

No, that are alternative approaches.

Updating the VMID asynchronously sounds a bit cleaner to me, but feel free to pick whatever is easier for you to implement.

Regards,
Christian.

Thanks

Jacob

From: Koenig, Christian<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 11:36 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

See the SPM buffer address is set using CP commands as well, right? And those execute asynchronously.

When we now synchronously update the SPM VMID we risk that we switch from one process to another while the new process is not ready yet with its setup.

That could have quite a bunch of unforeseen consequences, including accidentally writing SPM data into the new process address space at whatever buffer address was used before.

This is something we at least should try to avoid.

Regards,
Christian.

Am 03.03.20 um 16:28 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

Thanks! Could you please take an example of trouble “This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.”?

Thanks

Jacob

From: Koenig, Christian<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 11:03 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 03.03.20 um 15:34 schrieb He, Jacob:

[AMD Official Use Only - Internal Distribution Only]

It would be better if we could do that asynchronously with a register write on the ring.

Sorry, I don’t get your point. Could you please elaborate more?

You pass the ring from amdgpu_vm_flush() to the *_update_spm_vmid() functions.

And then instead of using WREG32() you call amdgpu_ring_emit_wreg() to make the write asynchronously on the ring buffer using a CP command.

This way we avoid a bunch of trouble when one process drops the VMID reservation and another one grabs it.

Regards,
Christian.

Thanks

Jacob

From: Christian König<mailto:[email protected]>
Sent: Tuesday, March 3, 2020 10:16 PM
To: He, Jacob<mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid

Am 02.03.20 um 06:35 schrieb Jacob He:

Quoted 15 lines

SPM access the video memory according to SPM_VMID. It should be updated with the job's vmid right before the job is scheduled. SPM_VMID is a global resource

Change-Id Id3881908960398f87e7c95026a54ff83ff826700
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index c00696f3017e..c761d3a0b6e8 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1088

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
@@ -1080,8 +1080,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
       struct dma_fence *fence = NULL;
       bool pasid_mapping_needed = false;
       unsigned patch_offset = 0;
+     bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
       int r;
 
+     if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+             adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+

It would be better if we could do that asynchronously with a register write on the ring.

The alternative is that we block for the VM to be idle in amdgpu_vm_ioctl() before unreserving the VMID.

In other words lock the reservation object of the root PD and call amdgpu_vm_wait_idle() before calling amdgpu_vmid_free_reserved().

Regards,
Christian.

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · line 1091

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
       if (amdgpu_vmid_had_gpu_reset(adev, id)) {
               gds_switch_needed = true;
               vm_flush_needed = true;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx