drm/amdgpu: Unmap CSA under SR-IOV in KFD path
Subsystems (3)
PATCH 1/1 drm/amdgpu: Unmap CSA under SR-IOV in KFD path
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling mdgpu_vm_make_compute will fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM object
The fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10 -0
@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1 -1
@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va); -if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));amdgpu_vm_bo_rmv (adev, fpriv->csa_va);
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1556447115-1689-1-git-send-email-Trigger.Huang-5C7GfCeVMHo@public.gmane.org
Am 28.04.19 um 12:25 schrieb Trigger Huang:
Quoted 7 lines
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Well we should rather try to avoid that.
The plan with that was that we do this so that one VM can be used both by compute and gfx.
I would rather lower the check in amdgpu_vm_make_compute() to make sure that we don't have page tables filled, instead of checking for a mapping.
Christian.
Quoted 33 lines
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c void amdgpu_driver_postclose_kms(struct drm_device *dev, · line 1075
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va); -if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));amdgpu_vm_bo_rmv (adev, fpriv->csa_va);
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
I remember a past discussion to change the CSA allocation/mapping scheme to avoid this issue in the first place. Can adding the CSA to the VM be delayed a little to a point after the VM gets converted to a compute VM? Maybe the first command submission?
Regards,
Felix
Quoted 40 lines · Trigger Huang
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c void amdgpu_driver_postclose_kms(struct drm_device *dev, · line 1075
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va); -if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));amdgpu_vm_bo_rmv (adev, fpriv->csa_va);
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): be96ca90-23c2-e442-42d0-7889436b0393-5C7GfCeVMHo@public.gmane.org
Well that is exactly what we already do here. The only problem is we do the wrong check amdgpu_vm_make_compute().
Instead of checking if some page tables are already filled we check if some mapping is already made.
Regards,
Christian.
Am 30.04.19 um 01:34 schrieb Kuehling, Felix:
Quoted 52 lines
I remember a past discussion to change the CSA allocation/mapping scheme to avoid this issue in the first place. Can adding the CSA to the VM be delayed a little to a point after the VM gets converted to a compute VM? Maybe the first command submission?
Regards,
FelixQuoted 43 lines · Trigger Huang
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0
-0@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va);
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c void amdgpu_driver_postclose_kms(struct drm_device *dev, · line 1075
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1-if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));amdgpu_vm_bo_rmv (adev, fpriv->csa_va);_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 749d808b-5afc-488e-6d31-fae5e8dac208-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
Hi Christian & Felix,
Thanks for the information.
But actually currently CSA is mapped only in amdgpu_driver_open_kms under SR-IOV.
So would you give more information about ' Well that is exactly what we already do here'? Where I can find its implementation?
On the other hand, I will try the method 'Instead of checking if some page tables are already filled we check if some mapping is already made'
Thanks & Best Wishes,
Trigger Huang
-----Original Message-----
From: Christian König <[email protected]>
Sent: Tuesday, April 30, 2019 5:23 PM
To: Kuehling, Felix <[email protected]>; Huang, Trigger <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD path
[CAUTION: External Email]
Well that is exactly what we already do here. The only problem is we do the wrong check amdgpu_vm_make_compute().
Instead of checking if some page tables are already filled we check if some mapping is already made.
Regards,
Christian.
Am 30.04.19 um 01:34 schrieb Kuehling, Felix:
Quoted 55 lines
I remember a past discussion to change the CSA allocation/mapping scheme to avoid this issue in the first place. Can adding the CSA to the VM be delayed a little to a point after the VM gets converted to a compute VM? Maybe the first command submission?
Regards,
FelixQuoted 46 lines · Trigger Huang
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0
-0@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va);
Quoted 9 lines
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1-if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));amdgpu_vm_bo_rmv(adev, fpriv->csa_va);_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MN2PR12MB3408BADD8375F5B6D6090244FE3A0-rweVpJHSKTo4ZTiO+qW6TgdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org
Am 30.04.19 um 11:53 schrieb Huang, Trigger:
Quoted 6 lines
Hi Christian & Felix,
Thanks for the information.
But actually currently CSA is mapped only in
amdgpu_driver_open_kmsunder SR-IOV.
So would you give more information about ' Well that is exactly what we already do here'? Where I can find its implementation?
Well the plan was to move this to when the actually execution context is created. But Rex left the company, could be that the patches for this where never committed.
Quoted 2 lines
On the other hand, I will try the method 'Instead of checking if some page tables are already filled we check if some mapping is already made'
Yeah, that should certainly work as well. Just check all entries of the root PD in a for loop if any subsequent PDs are allocated and abort if you find one.
Christian.
Quoted 12 lines
Thanks & Best Wishes,
Trigger Huang-----Original Message-----
From: Christian König <[email protected]>
Sent: Tuesday, April 30, 2019 5:23 PM
To: Kuehling, Felix <[email protected]>; Huang, Trigger <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD path[CAUTION: External Email]
Quoted 72 lines
Well that is exactly what we already do here. The only problem is we do the wrong check
amdgpu_vm_make_compute().Instead of checking if some page tables are already filled we check if some mapping is already made.
Regards,
Christian.Am 30.04.19 um 01:34 schrieb Kuehling, Felix:
Quoted 64 lines
I remember a past discussion to change the CSA allocation/mapping scheme to avoid this issue in the first place. Can adding the CSA to the VM be delayed a little to a point after the VM gets converted to a compute VM? Maybe the first command submission?
Regards,
FelixQuoted 50 lines · Trigger Huang
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va); -if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));Quoted 1 line
amdgpu_vm_bo_rmv(adev, fpriv->csa_va);_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 0df2a92e-c021-66f7-f6ba-4984e24d5d6c-5C7GfCeVMHo@public.gmane.org
Hi Christian,
Thanks for the quick response.
One thing I want to confirm, per my understanding that CSA is only used by CP preemption for KCQ and KGQ path, but in KFD user mode queue path, can we delete CSA? If yes, then maybe we can split this topic into two patches
1, the original one that unmap CSA in KFD path is still needed
2, Add new patch to use new method for VM checking.
Thanks & Best Wishes,
Trigger Huang
-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Tuesday, April 30, 2019 5:59 PM
To: Huang, Trigger <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD path
Am 30.04.19 um 11:53 schrieb Huang, Trigger:
Quoted 6 lines
Hi Christian & Felix,
Thanks for the information.
But actually currently CSA is mapped only in
amdgpu_driver_open_kmsunder SR-IOV.
So would you give more information about ' Well that is exactly what we already do here'? Where I can find its implementation?
Well the plan was to move this to when the actually execution context is created. But Rex left the company, could be that the patches for this where never committed.
Quoted 2 lines
On the other hand, I will try the method 'Instead of checking if some page tables are already filled we check if some mapping is already made'
Yeah, that should certainly work as well. Just check all entries of the root PD in a for loop if any subsequent PDs are allocated and abort if you find one.
Christian.
Quoted 12 lines
Thanks & Best Wishes,
Trigger Huang-----Original Message-----
From: Christian König <[email protected]>
Sent: Tuesday, April 30, 2019 5:23 PM
To: Kuehling, Felix <[email protected]>; Huang, Trigger <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD path[CAUTION: External Email]
Quoted 72 lines
Well that is exactly what we already do here. The only problem is we do the wrong check
amdgpu_vm_make_compute().Instead of checking if some page tables are already filled we check if some mapping is already made.
Regards,
Christian.Am 30.04.19 um 01:34 schrieb Kuehling, Felix:
Quoted 64 lines
I remember a past discussion to change the CSA allocation/mapping scheme to avoid this issue in the first place. Can adding the CSA to the VM be delayed a little to a point after the VM gets converted to a compute VM? Maybe the first command submission?
Regards,
FelixQuoted 50 lines · Trigger Huang
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va); -if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));Quoted 1 line
amdgpu_vm_bo_rmv(adev, fpriv->csa_va);_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MN2PR12MB3408B8667025191A35674764FE3A0-rweVpJHSKTo4ZTiO+qW6TgdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org
Am 30.04.19 um 12:28 schrieb Huang, Trigger:
Quoted 7 lines
Hi Christian,
Thanks for the quick response.
One thing I want to confirm, per my understanding that CSA is only used by CP preemption for KCQ and KGQ path, but in KFD user mode queue path, can we delete CSA? If yes, then maybe we can split this topic into two patches
1, the original one that unmap CSA in KFD path is still needed
I won't go down that route anyway, we should probably move the CSA mapping into the CTX handling completely.
Quoted 1 line
2, Add new patch to use new method for VM checking.
That should actually be the first patch, cause this checking is certainly incorrect.
Christian.
Quoted 109 lines
Thanks & Best Wishes,
Trigger Huang-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Tuesday, April 30, 2019 5:59 PM
To: Huang, Trigger <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD pathAm 30.04.19 um 11:53 schrieb Huang, Trigger:
Quoted 6 lines
Hi Christian & Felix,
Thanks for the information.
But actually currently CSA is mapped only in
amdgpu_driver_open_kmsunder SR-IOV.
So would you give more information about ' Well that is exactly what we already do here'? Where I can find its implementation?Well the plan was to move this to when the actually execution context is created. But Rex left the company, could be that the patches for this where never committed.
Quoted 1 line
On the other hand, I will try the method 'Instead of checking if some page tables are already filled we check if some mapping is already made'
Yeah, that should certainly work as well. Just check all entries of the root PD in a for loop if any subsequent PDs are allocated and abort if you find one.
Christian.
Quoted 11 lines
Thanks & Best Wishes,
Trigger Huang-----Original Message-----
From: Christian König <[email protected]>
Sent: Tuesday, April 30, 2019 5:23 PM
To: Kuehling, Felix <[email protected]>; Huang, Trigger <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD path[CAUTION: External Email]
Quoted 72 lines
Well that is exactly what we already do here. The only problem is we do the wrong check
amdgpu_vm_make_compute().Instead of checking if some page tables are already filled we check if some mapping is already made.
Regards,
Christian.Am 30.04.19 um 01:34 schrieb Kuehling, Felix:
Quoted 64 lines
I remember a past discussion to change the CSA allocation/mapping scheme to avoid this issue in the first place. Can adding the CSA to the VM be delayed a little to a point after the VM gets converted to a compute VM? Maybe the first command submission?
Regards,
FelixQuoted 50 lines · Trigger Huang
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va); -if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));Quoted 1 line
amdgpu_vm_bo_rmv(adev, fpriv->csa_va);_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): f3ed2a39-920a-9f32-9f92-342fec294c77-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
Thanks, Ok, I got it.
That makes more clear.
Thanks & Best Wishes,
Trigger Huang
-----Original Message-----
From: Christian König <[email protected]>
Sent: Tuesday, April 30, 2019 6:32 PM
To: Huang, Trigger <[email protected]>; Koenig, Christian <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD path
[CAUTION: External Email]
Am 30.04.19 um 12:28 schrieb Huang, Trigger:
Quoted 7 lines
Hi Christian,
Thanks for the quick response.
One thing I want to confirm, per my understanding that CSA is only used by CP preemption for KCQ and KGQ path, but in KFD user mode queue path, can we delete CSA? If yes, then maybe we can split this topic into two patches 1, the
original one that unmap CSA in KFD path is still needed
I won't go down that route anyway, we should probably move the CSA mapping into the CTX handling completely.
Quoted 1 line
2, Add new patch to use new method for VM checking.
That should actually be the first patch, cause this checking is certainly incorrect.
Christian.
Quoted 111 lines
Thanks & Best Wishes,
Trigger Huang-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Tuesday, April 30, 2019 5:59 PM
To: Huang, Trigger <[email protected]>; Kuehling, Felix <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD pathAm 30.04.19 um 11:53 schrieb Huang, Trigger:
Quoted 6 lines
Hi Christian & Felix,
Thanks for the information.
But actually currently CSA is mapped only in
amdgpu_driver_open_kmsunder SR-IOV.
So would you give more information about ' Well that is exactly what we already do here'? Where I can find its implementation?Well the plan was to move this to when the actually execution context is created. But Rex left the company, could be that the patches for this where never committed.
Quoted 1 line
On the other hand, I will try the method 'Instead of checking if some page tables are already filled we check if some mapping is already made'
Yeah, that should certainly work as well. Just check all entries of the root PD in a for loop if any subsequent PDs are allocated and abort if you find one.
Christian.
Quoted 12 lines
Thanks & Best Wishes,
Trigger Huang-----Original Message-----
From: Christian König <[email protected]>
Sent: Tuesday, April 30, 2019 5:23 PM
To: Kuehling, Felix <[email protected]>; Huang, Trigger <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Unmap CSA under SR-IOV in KFD path[CAUTION: External Email]
Quoted 72 lines
Well that is exactly what we already do here. The only problem is we do the wrong check
amdgpu_vm_make_compute().Instead of checking if some page tables are already filled we check if some mapping is already made.
Regards,
Christian.Am 30.04.19 um 01:34 schrieb Kuehling, Felix:
Quoted 64 lines
I remember a past discussion to change the CSA allocation/mapping scheme to avoid this issue in the first place. Can adding the CSA to the VM be delayed a little to a point after the VM gets converted to a compute VM? Maybe the first command submission?
Regards,
FelixQuoted 50 lines · Trigger Huang
In amdgpu open path, CSA will be mappened in VM, so when opening KFD, calling
mdgpu_vm_make_computewill fail because it found this VM is not a clean VM with some mappings, as a result, it will lead to failed to create process VM objectThe fix is try to unmap CSA, and actually CSA is not needed in compute VF world switch
Signed-off-by Trigger Huang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +10
-0@@ -956,6 +956,16 @@ int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd,if (avm->process_info)return -EINVAL ; +/* Delete CSA mapping to make sure this VM is a clean VM before +* converting VM +*/ +if (amdgpu_sriov_vf (adev) && drv_priv->csa_va) { +amdgpu_bo_reserve (adev->virt.csa_obj, true); +amdgpu_vm_bo_rmv (adev, drv_priv->csa_va); + drv_priv->csa_va =NULL ; +amdgpu_bo_unreserve (adev->virt.csa_obj); + } +/* Convert VM into a compute VM */ ret =amdgpu_vm_make_compute (adev, avm, pasid);if (ret)drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +1
-1@@ -1069,7 +1069,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev,amdgpu_vm_bo_rmv (adev, fpriv->prt_va); -if (amdgpu_sriov_vf (adev)) { +if (amdgpu_sriov_vf (adev) && fpriv->csa_va) {/* TODO: how to handle reserve failure */ BUG_ON (amdgpu_bo_reserve (adev->virt.csa_obj, true));Quoted 1 line
amdgpu_vm_bo_rmv(adev, fpriv->csa_va);_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx