handle userptr corner cases with HMM path
Subsystems (3)
cover handle userptr corner cases with HMM path
Those corner cases are found by kfdtest.KFDIPCTest.
Philip Yang (3):
drm/amdkfd: support concurrent userptr update for HMM drm/amdgpu: support userptr cross VMAs case with HMM drm/amdgpu: more descriptive message if HMM not enabled
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 28 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 122 +++++++++++++----- 3 files changed, 110 insertions(+), 42 deletions(-)
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/3 drm/amdkfd: support concurrent userptr update for HMM
Userptr restore may have concurrent userptr invalidation after hmm_vma_fault adds the range to the hmm->ranges list, needs call hmm_vma_range_done to remove the range from hmm->ranges list first, then reschedule the restore worker. Otherwise hmm_vma_fault will add same range to the list, this will cause loop in the list because range->next point to range itself.
Add function untrack_invalid_user_pages to reduce code duplication.
--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 28 ++++++++++++++----- 1 file changed, 21 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +21 -7
@@ -1731,6 +1731,23 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info,return 0 ; } +/* Remove invalid userptr BOs from hmm track list +* +* Stop HMM track the userptr update +*/ +static void untrack_invalid_user_pages (struct amdkfd_process_info *process_info) +{ +struct kgd_mem *mem, *tmp_mem; +struct amdgpu_bo *bo; + +list_for_each_entry_safe (mem, tmp_mem, + &process_info->userptr_inval_list, + validate_list.head) { + bo = mem->bo; +amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm); + } +} +/* Validate invalid userptr BOs * * Validates BOs on the userptr_inval_list, and moves them back to the
@@ -1848,12 +1865,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) out_free:kfree (pd_bo_list_entries); out_no_mem: -list_for_each_entry_safe (mem, tmp_mem, - &process_info->userptr_inval_list, - validate_list.head) { - bo = mem->bo; -amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm); - } +untrack_invalid_user_pages (process_info);return ret; }
@@ -1897,8 +1909,10 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work)* and we can just restart the queues. */ if (!list_empty (&process_info->userptr_inval_list)) { -if (atomic_read (&process_info->evicted_bos) != evicted_bos) +if (atomic_read (&process_info->evicted_bos) != evicted_bos) { +untrack_invalid_user_pages (process_info); goto unlock_out;/* Concurrent eviction, try again */ + }if (validate_invalid_user_pages (process_info)) goto unlock_out;
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 1/3] drm/amdkfd: support concurrent userptr update for HMM
Hmm, I'm not sure. This change probably fixes this issue, but there may be other similar corner cases in other situations where the restore worker fails and needs to retry. The better place to call untrack in amdgpu_amdkfd_restore_userptr_worker would be at the very end. Anything that's left in the userptr_inval_list at that point needs to be untracked.
For now as a quick fix for an urgent bug, this change is Reviewed-by:
Felix Kuehling <[email protected]>. But please revisit this and check if there are similar corner cases as I explained above.
Regards,
Felix
Quoted 59 lines · Philip
Userptr restore may have concurrent userptr invalidation after
hmm_vma_faultadds the range to thehmm->rangeslist, needs callhmm_vma_range_doneto remove the range fromhmm->rangeslist first, then reschedule the restore worker. Otherwisehmm_vma_faultwill add same range to the list, this will cause loop in the list becauserange->nextpoint to range itself.Add function
untrack_invalid_user_pagesto reduce code duplication.Change-Id I31407739dc10554f8e418c7a0e0415d3d95552f1Signed-off-by Philip Yang <[email protected]>--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 28 ++++++++++++++----- 1 file changed, 21 insertions(+), 7 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +18
-6@@ -1731,6 +1731,23 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info,return 0 ; } +/* Remove invalid userptr BOs from hmm track list +* +* Stop HMM track the userptr update +*/ +static void untrack_invalid_user_pages (struct amdkfd_process_info *process_info) +{ +struct kgd_mem *mem, *tmp_mem; +struct amdgpu_bo *bo; + +list_for_each_entry_safe (mem, tmp_mem, + &process_info->userptr_inval_list, + validate_list.head) { + bo = mem->bo; +amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm); + } +} +/* Validate invalid userptr BOs * * Validates BOs on the userptr_inval_list, and moves them back to the @@ -1848,12 +1865,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) out_free:kfree (pd_bo_list_entries); out_no_mem: -list_for_each_entry_safe (mem, tmp_mem, - &process_info->userptr_inval_list, - validate_list.head) { - bo = mem->bo; -amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm); - } +untrack_invalid_user_pages (process_info);return ret; }@@ -1897,8 +1909,10 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work)* and we can just restart the queues.
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) · line 1918
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +3
-1*/ if (!list_empty (&process_info->userptr_inval_list)) { -if (atomic_read (&process_info->evicted_bos) != evicted_bos) +if (atomic_read (&process_info->evicted_bos) != evicted_bos) { +untrack_invalid_user_pages (process_info); goto unlock_out;/* Concurrent eviction, try again */ + }if (validate_invalid_user_pages (process_info)) goto unlock_out;
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 772b1109-b6db-05cc-4f36-6428724ae4a3-5C7GfCeVMHo@public.gmane.org
Re: [PATCH 1/3] drm/amdkfd: support concurrent userptr update for HMM
Hi Felix,
Thanks, there are other corner cases, call untrack at end of restore userptr worker is better place to cleanup. I will submit v2 patch, to fix this issue completely.
Philip
Quoted 81 lines · Felix
Hmm, I'm not sure. This change probably fixes this issue, but there may be other similar corner cases in other situations where the restore worker fails and needs to retry. The better place to call untrack in
amdgpu_amdkfd_restore_userptr_workerwould be at the very end. Anything that's left in theuserptr_inval_listat that point needs to be untracked.For now as a quick fix for an urgent bug, this change is Reviewed-by:
Felix Kuehling <[email protected]>. But please revisit this and check if there are similar corner cases as I explained above.Regards,
FelixQuoted 67 lines · Philip
Userptr restore may have concurrent userptr invalidation after
hmm_vma_faultadds the range to thehmm->rangeslist, needs callhmm_vma_range_doneto remove the range fromhmm->rangeslist first, then reschedule the restore worker. Otherwisehmm_vma_faultwill add same range to the list, this will cause loop in the list becauserange->nextpoint to range itself.Add function
untrack_invalid_user_pagesto reduce code duplication.Change-Id I31407739dc10554f8e418c7a0e0415d3d95552f1Signed-off-by Philip Yang <[email protected]>--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 28 ++++++++++++++----- 1 file changed, 21 insertions(+), 7 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +21
-7@@ -1731,6 +1731,23 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info,return 0 ; } +/* Remove invalid userptr BOs from hmm track list +* +* Stop HMM track the userptr update +*/ +static void untrack_invalid_user_pages (struct amdkfd_process_info *process_info) +{ +struct kgd_mem *mem, *tmp_mem; +struct amdgpu_bo *bo; + +list_for_each_entry_safe (mem, tmp_mem, + &process_info->userptr_inval_list, + validate_list.head) { + bo = mem->bo; +amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm); + } +} +/* Validate invalid userptr BOs * * Validates BOs on the userptr_inval_list, and moves them back to the @@ -1848,12 +1865,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) out_free:kfree (pd_bo_list_entries); out_no_mem: -list_for_each_entry_safe (mem, tmp_mem, - &process_info->userptr_inval_list, - validate_list.head) { - bo = mem->bo; -amdgpu_ttm_tt_get_user_pages_done (bo->tbo.ttm); - } +untrack_invalid_user_pages (process_info);return ret; }@@ -1897,8 +1909,10 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work)* and we can just restart the queues. */ if (!list_empty (&process_info->userptr_inval_list)) { -if (atomic_read (&process_info->evicted_bos) != evicted_bos) +if (atomic_read (&process_info->evicted_bos) != evicted_bos) { +untrack_invalid_user_pages (process_info); goto unlock_out;/* Concurrent eviction, try again */ + }
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) · line 1918
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0if (validate_invalid_user_pages (process_info)) goto unlock_out;
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 2/3 drm/amdgpu: support userptr cross VMAs case with HMM
In reply to a message not in the archiveuserptr may cross two VMAs if the forked child process (not call exec after fork) malloc buffer, then free it, and then malloc larger size buf, kerenl will create new VMA adjacent to old VMA which was cloned from parent process, some pages of userptr are in the first VMA, the rest pages are in the second VMA.
HMM expects range only have one VMA, loop over all VMAs in the address range, create multiple ranges to handle this case. See is_mergeable_anon_vma in mm/mmap.c for details.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 122 +++++++++++++++++------- 1 file changed, 87 insertions(+), 35 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +87 -35
@@ -711,7 +711,8 @@ struct amdgpu_ttm_tt {struct task_struct *usertask;uint32_t userflags;#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -struct hmm_range range; +struct hmm_range *ranges; +int nr_ranges;#endif };
@@ -723,62 +724,104 @@ struct amdgpu_ttm_tt {* once afterwards to stop HMM tracking */ #if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) + +/* Support Userptr pages cross max 16 vmas */ +#define MAX_NR_VMAS (16) +int amdgpu_ttm_tt_get_user_pages (struct ttm_tt *ttm,struct page **pages) {struct amdgpu_ttm_tt *gtt = (void *)ttm;struct mm_struct *mm = gtt->usertask->mm; -unsigned long end = gtt->userptr + ttm->num_pages *PAGE_SIZE ; -struct hmm_range *range = >t->range; -int r =0 , i; +unsigned long start = gtt->userptr; +unsigned long end = start + ttm->num_pages *PAGE_SIZE ; +struct hmm_range *ranges; +struct vm_area_struct *vma =NULL , *vmas[MAX_NR_VMAS ]; +uint64_t *pfns, f; +int r =0 , i, nr_pages;if (!mm)/* Happens during process shutdown */ return -ESRCH ; -amdgpu_hmm_init_range (range); -down_read (&mm->mmap_sem); - range->vma =find_vma (mm, gtt->userptr); -if (!range_in_vma (range->vma, gtt->userptr, end)) - r = -EFAULT ; -else if ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - range->vma->vm_file) +/* user pages may cross multiple VMAs */ + gtt->nr_ranges =0 ; +do { + vma =find_vma (mm, vma ? vma->vm_end : start); +if (unlikely (!vma)) { + r = -EFAULT ; + goto out; + } + vmas[gtt->nr_ranges++] = vma; +if (gtt->nr_ranges >=MAX_NR_VMAS ) { +DRM_ERROR ("invalid userptr range\n" ); + r = -EFAULT ; + goto out; + } + }while (end > vma->vm_end); + +DRM_DEBUG_DRIVER ("0x%lx nr_ranges %d pages 0x%lx\n" , + start, gtt->nr_ranges, ttm->num_pages); + +if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && + vmas[0 ]->vm_file)) { r = -EPERM ; -if (r) goto out; + } - range->pfns =kvmalloc_array (ttm->num_pages,sizeof (uint64_t ), -GFP_KERNEL ); -if (range->pfns ==NULL ) { + ranges =kvmalloc_array (gtt->nr_ranges,sizeof (*ranges),GFP_KERNEL ); +if (unlikely (!ranges)) { r = -ENOMEM ; goto out; } - range->start = gtt->userptr; - range->end = end; - range->pfns[0 ] = range->flags[HMM_PFN_VALID ]; - range->pfns[0 ] |=amdgpu_ttm_tt_is_readonly (ttm) ? -0 : range->flags[HMM_PFN_WRITE ]; -for (i =1 ; i < ttm->num_pages; i++) - range->pfns[i] = range->pfns[0 ]; + pfns =kvmalloc_array (ttm->num_pages,sizeof (*pfns),GFP_KERNEL ); +if (unlikely (!pfns)) { + r = -ENOMEM ; + goto out_free_ranges; + } + +for (i =0 ; i < gtt->nr_ranges; i++) +amdgpu_hmm_init_range (&ranges[i]); + + f = ranges[0 ].flags[HMM_PFN_VALID ]; + f |=amdgpu_ttm_tt_is_readonly (ttm) ? +0 : ranges[0 ].flags[HMM_PFN_WRITE ]; +memset64 (pfns, f, ttm->num_pages); + +for (nr_pages =0 , i =0 ; i < gtt->nr_ranges; i++) { + ranges[i].vma = vmas[i]; + ranges[i].start =max (start, vmas[i]->vm_start); + ranges[i].end =min (end, vmas[i]->vm_end); + ranges[i].pfns = pfns + nr_pages; + nr_pages += (ranges[i].end - ranges[i].start) /PAGE_SIZE ; + + r =hmm_vma_fault (&ranges[i], true); +if (unlikely (r)) +break ; + } +if (unlikely (r)) { +while (i--) +hmm_vma_range_done (&ranges[i]); -/* This may trigger page table update */ - r =hmm_vma_fault (range, true); -if (r) goto out_free_pfns; + }up_read (&mm->mmap_sem);for (i =0 ; i < ttm->num_pages; i++) - pages[i] =hmm_pfn_to_page (range, range->pfns[i]); + pages[i] =hmm_pfn_to_page (&ranges[0 ], pfns[i]); + gtt->ranges = ranges;return 0 ; out_free_pfns: -kvfree (range->pfns); - range->pfns =NULL ; +kvfree (pfns); +out_free_ranges: +kvfree (ranges); out:up_read (&mm->mmap_sem); +return r; }
@@ -792,15 +835,23 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) {struct amdgpu_ttm_tt *gtt = (void *)ttm;bool r = false; +int i;if (!gtt || !gtt->userptr)return false; -WARN_ONCE (!gtt->range.pfns,"No user pages to check\n" ); -if (gtt->range.pfns) { - r =hmm_vma_range_done (>t->range); -kvfree (gtt->range.pfns); - gtt->range.pfns =NULL ; +DRM_DEBUG_DRIVER ("user_pages_done 0x%llx nr_ranges %d pages 0x%lx\n" , + gtt->userptr, gtt->nr_ranges, ttm->num_pages); + +WARN_ONCE (!gtt->ranges || !gtt->ranges[0 ].pfns, +"No user pages to check\n" ); + +if (gtt->ranges) { +for (i =0 ; i < gtt->nr_ranges; i++) + r |=hmm_vma_range_done (>t->ranges[i]); +kvfree (gtt->ranges[0 ].pfns); +kvfree (gtt->ranges); + gtt->ranges =NULL ; }return r;
@@ -884,8 +935,9 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range.pfns && - ttm->pages[0 ] ==hmm_pfn_to_page (>t->range, gtt->range.pfns[0 ])) +if (gtt->ranges && + ttm->pages[0 ] ==hmm_pfn_to_page (>t->ranges[0 ], + gtt->ranges[0 ].pfns[0 ]))WARN_ONCE (1 ,"Missing get_user_page_done\n" );#endif }
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 2/3] drm/amdgpu: support userptr cross VMAs case with HMM
Some comments inline ...
Quoted 64 lines · Philip
userptr may cross two VMAs if the forked child process (not call exec after fork) malloc buffer, then free it, and then malloc larger size buf, kerenl will create new VMA adjacent to old VMA which was cloned from parent process, some pages of userptr are in the first VMA, the rest pages are in the second VMA.
HMM expects range only have one VMA, loop over all VMAs in the address range, create multiple ranges to handle this case. See
is_mergeable_anon_vmainmm/mmap.cfor details.Change-Id I0ca8c77e28deabccc139906f9ffee04b7e383314Signed-off-by Philip Yang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 122 +++++++++++++++++------- 1 file changed, 87 insertions(+), 35 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +12
-11@@ -711,7 +711,8 @@ struct amdgpu_ttm_tt {struct task_struct *usertask;uint32_t userflags;#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -struct hmm_range range; +struct hmm_range *ranges; +int nr_ranges;#endif };@@ -723,62 +724,104 @@ struct amdgpu_ttm_tt {* once afterwards to stop HMM tracking */ #if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) + +/* Support Userptr pages cross max 16 vmas */ +#define MAX_NR_VMAS (16) +int amdgpu_ttm_tt_get_user_pages (struct ttm_tt *ttm,struct page **pages) {struct amdgpu_ttm_tt *gtt = (void *)ttm;struct mm_struct *mm = gtt->usertask->mm; -unsigned long end = gtt->userptr + ttm->num_pages *PAGE_SIZE ; -struct hmm_range *range = >t->range; -int r =0 , i; +unsigned long start = gtt->userptr; +unsigned long end = start + ttm->num_pages *PAGE_SIZE ; +struct hmm_range *ranges; +struct vm_area_struct *vma =NULL , *vmas[MAX_NR_VMAS ]; +uint64_t *pfns, f; +int r =0 , i, nr_pages;if (!mm)/* Happens during process shutdown */ return -ESRCH ; -amdgpu_hmm_init_range (range); -down_read (&mm->mmap_sem); - range->vma =find_vma (mm, gtt->userptr); -if (!range_in_vma (range->vma, gtt->userptr, end)) - r = -EFAULT ; -else if ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - range->vma->vm_file)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c struct amdgpu_ttm_tt { · line 756
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +10
-0+/* user pages may cross multiple VMAs */ + gtt->nr_ranges =0 ; + do { + vma = find_vma(mm, vma ? vma->vm_end : start); + if (unlikely(!vma)) { + r = -EFAULT; + goto out; + } + vmas[gtt->nr_ranges++] = vma; + if (gtt->nr_ranges >= MAX_NR_VMAS) {
This will lead to a failure when you have exactly 16 VMAs. If you move the check to the start of the loop, it will only trigger when you exceed the limit not just after you reach it.
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c struct amdgpu_ttm_tt { · line 757
Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0+DRM_ERROR ("invalid userptr range\n" );
The userptr range is not really invalid. It only exceeds some artificial limitation in this code. A message like "Too many VMAs in userptr range" would be more appropriate.
Quoted 61 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +41
-12+ r = -EFAULT ; + goto out; + } + }while (end > vma->vm_end); + +DRM_DEBUG_DRIVER ("0x%lx nr_ranges %d pages 0x%lx\n" , + start, gtt->nr_ranges, ttm->num_pages); + +if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && + vmas[0 ]->vm_file)) { r = -EPERM ; -if (r) goto out; + } - range->pfns =kvmalloc_array (ttm->num_pages,sizeof (uint64_t ), -GFP_KERNEL ); -if (range->pfns ==NULL ) { + ranges =kvmalloc_array (gtt->nr_ranges,sizeof (*ranges),GFP_KERNEL ); +if (unlikely (!ranges)) { r = -ENOMEM ; goto out; } - range->start = gtt->userptr; - range->end = end; - range->pfns[0 ] = range->flags[HMM_PFN_VALID ]; - range->pfns[0 ] |=amdgpu_ttm_tt_is_readonly (ttm) ? -0 : range->flags[HMM_PFN_WRITE ]; -for (i =1 ; i < ttm->num_pages; i++) - range->pfns[i] = range->pfns[0 ]; + pfns =kvmalloc_array (ttm->num_pages,sizeof (*pfns),GFP_KERNEL ); +if (unlikely (!pfns)) { + r = -ENOMEM ; + goto out_free_ranges; + } + +for (i =0 ; i < gtt->nr_ranges; i++) +amdgpu_hmm_init_range (&ranges[i]); + + f = ranges[0 ].flags[HMM_PFN_VALID ]; + f |=amdgpu_ttm_tt_is_readonly (ttm) ? +0 : ranges[0 ].flags[HMM_PFN_WRITE ]; +memset64 (pfns, f, ttm->num_pages); + +for (nr_pages =0 , i =0 ; i < gtt->nr_ranges; i++) { + ranges[i].vma = vmas[i]; + ranges[i].start =max (start, vmas[i]->vm_start); + ranges[i].end =min (end, vmas[i]->vm_end); + ranges[i].pfns = pfns + nr_pages; + nr_pages += (ranges[i].end - ranges[i].start) /PAGE_SIZE ; + + r =hmm_vma_fault (&ranges[i], true); +if (unlikely (r)) +break ; + } +if (unlikely (r)) { +while (i--) +hmm_vma_range_done (&ranges[i]); -/* This may trigger page table update */
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c struct amdgpu_ttm_tt { · line 813
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-3- r =hmm_vma_fault (range, true); -if (r) goto out_free_pfns; + }up_read (&mm->mmap_sem);for (i =0 ; i < ttm->num_pages; i++) - pages[i] =hmm_pfn_to_page (range, range->pfns[i]); + pages[i] =hmm_pfn_to_page (&ranges[0 ], pfns[i]);
This looks wrong. Don't you need to use the correct range here instead of range[0]?
Quoted 30 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +6
-7+ gtt->ranges = ranges;return 0 ; out_free_pfns: -kvfree (range->pfns); - range->pfns =NULL ; +kvfree (pfns); +out_free_ranges: +kvfree (ranges); out:up_read (&mm->mmap_sem); +return r; }@@ -792,15 +835,23 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) {struct amdgpu_ttm_tt *gtt = (void *)ttm;bool r = false; +int i;if (!gtt || !gtt->userptr)return false; -WARN_ONCE (!gtt->range.pfns,"No user pages to check\n" ); -if (gtt->range.pfns) { - r =hmm_vma_range_done (>t->range); -kvfree (gtt->range.pfns); - gtt->range.pfns =NULL ;
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) · line 852
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +10
-0+DRM_DEBUG_DRIVER ("user_pages_done 0x%llx nr_ranges %d pages 0x%lx\n" , + gtt->userptr, gtt->nr_ranges, ttm->num_pages); + +WARN_ONCE (!gtt->ranges || !gtt->ranges[0 ].pfns, +"No user pages to check\n" ); + +if (gtt->ranges) { +for (i =0 ; i < gtt->nr_ranges; i++) + r |=hmm_vma_range_done (>t->ranges[i]); +kvfree (gtt->ranges[0 ].pfns);
Shouldn't this be in the loop and apply to gtt->ranges[i].pfns?
Regards,
Felix
Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-0+kvfree (gtt->ranges); + gtt->ranges =NULL ; }return r;@@ -884,8 +935,9 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 943
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3
-2#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range.pfns && - ttm->pages[0 ] ==hmm_pfn_to_page (>t->range, gtt->range.pfns[0 ])) +if (gtt->ranges && + ttm->pages[0 ] ==hmm_pfn_to_page (>t->ranges[0 ], + gtt->ranges[0 ].pfns[0 ]))WARN_ONCE (1 ,"Missing get_user_page_done\n" );#endif }
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): f51eaecd-8d50-0eea-aac6-2905f945a2c2-5C7GfCeVMHo@public.gmane.org
Re: [PATCH 2/3] drm/amdgpu: support userptr cross VMAs case with HMM
I will submit v2 to fix those issues. Some comments inline...
Quoted 73 lines · Felix
Some comments inline ...
Quoted 70 lines · Philip
userptr may cross two VMAs if the forked child process (not call exec after fork) malloc buffer, then free it, and then malloc larger size buf, kerenl will create new VMA adjacent to old VMA which was cloned from parent process, some pages of userptr are in the first VMA, the rest pages are in the second VMA.
HMM expects range only have one VMA, loop over all VMAs in the address range, create multiple ranges to handle this case. See
is_mergeable_anon_vmainmm/mmap.cfor details.Change-Id I0ca8c77e28deabccc139906f9ffee04b7e383314Signed-off-by Philip Yang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 122 +++++++++++++++++------- 1 file changed, 87 insertions(+), 35 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +18
-11@@ -711,7 +711,8 @@ struct amdgpu_ttm_tt {struct task_struct *usertask;uint32_t userflags;#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -struct hmm_range range; +struct hmm_range *ranges; +int nr_ranges;#endif };@@ -723,62 +724,104 @@ struct amdgpu_ttm_tt {* once afterwards to stop HMM tracking */ #if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) + +/* Support Userptr pages cross max 16 vmas */ +#define MAX_NR_VMAS (16) +int amdgpu_ttm_tt_get_user_pages (struct ttm_tt *ttm,struct page **pages) {struct amdgpu_ttm_tt *gtt = (void *)ttm;struct mm_struct *mm = gtt->usertask->mm; -unsigned long end = gtt->userptr + ttm->num_pages *PAGE_SIZE ; -struct hmm_range *range = >t->range; -int r =0 , i; +unsigned long start = gtt->userptr; +unsigned long end = start + ttm->num_pages *PAGE_SIZE ; +struct hmm_range *ranges; +struct vm_area_struct *vma =NULL , *vmas[MAX_NR_VMAS ]; +uint64_t *pfns, f; +int r =0 , i, nr_pages;if (!mm)/* Happens during process shutdown */ return -ESRCH ; -amdgpu_hmm_init_range (range); -down_read (&mm->mmap_sem); - range->vma =find_vma (mm, gtt->userptr); -if (!range_in_vma (range->vma, gtt->userptr, end)) - r = -EFAULT ; -else if ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && - range->vma->vm_file) +/* user pages may cross multiple VMAs */ + gtt->nr_ranges =0 ; + do { + vma = find_vma(mm, vma ? vma->vm_end : start); + if (unlikely(!vma)) { + r = -EFAULT;
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c struct amdgpu_ttm_tt { · line 756
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +4
-0+ goto out; + } + vmas[gtt->nr_ranges++] = vma; +if (gtt->nr_ranges >=MAX_NR_VMAS ) {This will lead to a failure when you have exactly 16 VMAs. If you move the check to the start of the loop, it will only trigger when you exceed the limit not just after you reach it.
Ok
Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c struct amdgpu_ttm_tt { · line 757
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0+DRM_ERROR ("invalid userptr range\n" );The userptr range is not really invalid. It only exceeds some artificial limitation in this code. A message like "Too many VMAs in userptr range" would be more appropriate.
Ok
Quoted 67 lines
Quoted 67 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +42
-14+ r = -EFAULT ; + goto out; + } + }while (end > vma->vm_end); + +DRM_DEBUG_DRIVER ("0x%lx nr_ranges %d pages 0x%lx\n" , + start, gtt->nr_ranges, ttm->num_pages); + +if (unlikely ((gtt->userflags &AMDGPU_GEM_USERPTR_ANONONLY ) && + vmas[0 ]->vm_file)) { r = -EPERM ; -if (r) goto out; + } - range->pfns =kvmalloc_array (ttm->num_pages,sizeof (uint64_t ), -GFP_KERNEL ); -if (range->pfns ==NULL ) { + ranges =kvmalloc_array (gtt->nr_ranges,sizeof (*ranges),GFP_KERNEL ); +if (unlikely (!ranges)) { r = -ENOMEM ; goto out; } - range->start = gtt->userptr; - range->end = end; - range->pfns[0 ] = range->flags[HMM_PFN_VALID ]; - range->pfns[0 ] |=amdgpu_ttm_tt_is_readonly (ttm) ? -0 : range->flags[HMM_PFN_WRITE ]; -for (i =1 ; i < ttm->num_pages; i++) - range->pfns[i] = range->pfns[0 ]; + pfns =kvmalloc_array (ttm->num_pages,sizeof (*pfns),GFP_KERNEL ); +if (unlikely (!pfns)) { + r = -ENOMEM ; + goto out_free_ranges; + } + +for (i =0 ; i < gtt->nr_ranges; i++) +amdgpu_hmm_init_range (&ranges[i]); + + f = ranges[0 ].flags[HMM_PFN_VALID ]; + f |=amdgpu_ttm_tt_is_readonly (ttm) ? +0 : ranges[0 ].flags[HMM_PFN_WRITE ]; +memset64 (pfns, f, ttm->num_pages); + +for (nr_pages =0 , i =0 ; i < gtt->nr_ranges; i++) { + ranges[i].vma = vmas[i]; + ranges[i].start =max (start, vmas[i]->vm_start); + ranges[i].end =min (end, vmas[i]->vm_end); + ranges[i].pfns = pfns + nr_pages; + nr_pages += (ranges[i].end - ranges[i].start) /PAGE_SIZE ; + + r =hmm_vma_fault (&ranges[i], true); +if (unlikely (r)) +break ; + } +if (unlikely (r)) { +while (i--) +hmm_vma_range_done (&ranges[i]); -/* This may trigger page table update */ - r =hmm_vma_fault (range, true); -if (r) goto out_free_pfns; + }
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c struct amdgpu_ttm_tt { · line 813
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-1up_read (&mm->mmap_sem);for (i =0 ; i < ttm->num_pages; i++) - pages[i] =hmm_pfn_to_page (range, range->pfns[i]); + pages[i] =hmm_pfn_to_page (&ranges[0 ], pfns[i]);This looks wrong. Don't you need to use the correct range here instead of range[0]?
hmm_pfn_to_page uses flags HMM_PFN_ERROR etc and pfn_shift, those flags and pfn_shift are same for all ranges. To use range corresponding to the page, we need another calculation from page to range.
Quoted 35 lines
Quoted 35 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +10
-7+ gtt->ranges = ranges;return 0 ; out_free_pfns: -kvfree (range->pfns); - range->pfns =NULL ; +kvfree (pfns); +out_free_ranges: +kvfree (ranges); out:up_read (&mm->mmap_sem); +return r; }@@ -792,15 +835,23 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) {struct amdgpu_ttm_tt *gtt = (void *)ttm;bool r = false; +int i;if (!gtt || !gtt->userptr)return false; -WARN_ONCE (!gtt->range.pfns,"No user pages to check\n" ); -if (gtt->range.pfns) { - r =hmm_vma_range_done (>t->range); -kvfree (gtt->range.pfns); - gtt->range.pfns =NULL ; +DRM_DEBUG_DRIVER ("user_pages_done 0x%llx nr_ranges %d pages 0x%lx\n" , + gtt->userptr, gtt->nr_ranges, ttm->num_pages); + + WARN_ONCE(!gtt->ranges || !gtt->ranges[0].pfns,
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) · line 852
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +6
-0+ "No user pages to check\n"); + +if (gtt->ranges) { +for (i =0 ; i < gtt->nr_ranges; i++) + r |=hmm_vma_range_done (>t->ranges[i]); +kvfree (gtt->ranges[0 ].pfns);Shouldn't this be in the loop and apply to gtt->ranges[i].pfns?
pfns is allocated once based on ttm->num_pages, ranges[0].pfns is the pfns, other ranges pfns is pfns plus nr_pages offset. So only need to free ranges[0].pfns.
Quoted 11 lines
Regards,
FelixQuoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-0+kvfree (gtt->ranges); + gtt->ranges =NULL ; }return r;@@ -884,8 +935,9 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)sg_free_table (ttm->sg);
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 943
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3
-2#if IS_ENABLED (CONFIG_DRM_AMDGPU_USERPTR ) -if (gtt->range.pfns && - ttm->pages[0 ] ==hmm_pfn_to_page (>t->range, gtt->range.pfns[0 ])) +if (gtt->ranges && + ttm->pages[0 ] ==hmm_pfn_to_page (>t->ranges[0 ], + gtt->ranges[0 ].pfns[0 ]))WARN_ONCE (1 ,"Missing get_user_page_done\n" );#endif }
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 3/3 drm/amdgpu: more descriptive message if HMM not enabled
In reply to a message not in the archiveIf using old kernel config file, CONFIG_ZONE_DEVICE is not selected, so CONFIG_HMM and CONFIG_HMM_MIRROR is not enabled, the current driver error message "Failed to register MMU notifier" is not clear. Inform user with more descriptive message on how to fix the missing kernel config option.
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=109808
--- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 2 ++ 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +2 -0
@@ -53,6 +53,8 @@ static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, }static inline int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr) { +DRM_ERROR ("HMM_MIRROR kernel config option is not enabled\n" ); +DRM_ERROR ("add CONFIG_ZONE_DEVICE=y in config file to fix this\n" );return -ENODEV ; }static inline void amdgpu_mn_unregister (struct amdgpu_bo *bo) {}
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 3/3] drm/amdgpu: more descriptive message if HMM not enabled
Quoted 20 lines · Philip
If using old kernel config file,
CONFIG_ZONE_DEVICEis not selected, soCONFIG_HMMandCONFIG_HMM_MIRRORis not enabled, the current driver error message "Failed to register MMU notifier" is not clear. Inform user with more descriptive message on how to fix the missing kernel config option.Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=109808
Change-Id Idfebaeababa4c37c1ef093c2b91a26910a167585Signed-off-by Philip Yang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, · line 60
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +2
-0@@ -53,6 +53,8 @@ static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, }static inline int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr) { +DRM_ERROR ("HMM_MIRROR kernel config option is not enabled\n" ); +DRM_ERROR ("add CONFIG_ZONE_DEVICE=y in config file to fix this\n" );return -ENODEV ; }static inline void amdgpu_mn_unregister (struct amdgpu_bo *bo) {}
amdgpu_gem_userptr_ioctl calls amdgpu_mn_register if userspace sets the AMDGPU_GEM_USERPTR_REGISTER flag, so buggy/malicious userspace could spam dmesg with these errors. I'm afraid a different solution is needed.
Signature · 6 lines
Earthling Michel Dänzer | https://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 9ac0b1ab-891e-9fe6-667e-f77a244dd862-otUistvHUpPR7s880joybQ@public.gmane.org
Re: [PATCH 3/3] drm/amdgpu: more descriptive message if HMM not enabled
Quoted 34 lines · Michel Dänzer
Quoted 20 lines · Philip
If using old kernel config file,
CONFIG_ZONE_DEVICEis not selected, soCONFIG_HMMandCONFIG_HMM_MIRRORis not enabled, the current driver error message "Failed to register MMU notifier" is not clear. Inform user with more descriptive message on how to fix the missing kernel config option.Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=109808
Change-Id Idfebaeababa4c37c1ef093c2b91a26910a167585Signed-off-by Philip Yang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0
-0drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, · line 60
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +2
-0@@ -53,6 +53,8 @@ static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, }static inline int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr) { +DRM_ERROR ("HMM_MIRROR kernel config option is not enabled\n" ); +DRM_ERROR ("add CONFIG_ZONE_DEVICE=y in config file to fix this\n" );return -ENODEV ; }static inline void amdgpu_mn_unregister (struct amdgpu_bo *bo) {}
amdgpu_gem_userptr_ioctlcallsamdgpu_mn_registerif userspace sets theAMDGPU_GEM_USERPTR_REGISTERflag, so buggy/malicious userspace could spam dmesg with these errors. I'm afraid a different solution is needed.
Quoted 1 line
The error message is inside dummy function
amdgpu_mn_register, it only
shows if kernel config option HMM is not enabled. Driver does need HMM kernel option to support userptr. We want to provide obvious message to users to update kernel config file from old kernel, otherwise all userptr related functions are broken. I will change this to WARN_ONCE to avoid spam dmesg.
Regards,
Philip
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): baa08fbf-8bb0-6ad7-9e62-b9ebdf88cf72-5C7GfCeVMHo@public.gmane.org
Re: [PATCH 3/3] drm/amdgpu: more descriptive message if HMM not enabled
Quoted 35 lines · Philip
Quoted 25 lines · Michel Dänzer
Quoted 24 lines · Philip
If using old kernel config file,
CONFIG_ZONE_DEVICEis not selected, soCONFIG_HMMandCONFIG_HMM_MIRRORis not enabled, the current driver error message "Failed to register MMU notifier" is not clear. Inform user with more descriptive message on how to fix the missing kernel config option.Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=109808
Change-Id Idfebaeababa4c37c1ef093c2b91a26910a167585Signed-off-by Philip Yang <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +2
-0@@ -53,6 +53,8 @@ static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, }static inline int amdgpu_mn_register (struct amdgpu_bo *bo,unsigned long addr) { +DRM_ERROR ("HMM_MIRROR kernel config option is not enabled\n" ); +DRM_ERROR ("add CONFIG_ZONE_DEVICE=y in config file to fix this\n" );Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, · line 60
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +0
-0return -ENODEV ; }static inline void amdgpu_mn_unregister (struct amdgpu_bo *bo) {}
amdgpu_gem_userptr_ioctlcallsamdgpu_mn_registerif userspace sets theAMDGPU_GEM_USERPTR_REGISTERflag, so buggy/malicious userspace could spam dmesg with these errors. I'm afraid a different solution is needed.
Quoted 6 lines
The error message is inside dummy function
amdgpu_mn_register, it only shows if kernel config option HMM is not enabled. Driver does need HMM kernel option to support userptr. We want to provide obvious message to users to update kernel config file from old kernel, otherwise all userptr related functions are broken. I will change this toWARN_ONCEto avoid spam dmesg.
I hope you mean DRM_WARN_ONCE. :) Plain WARN_ONCE is probably also too noisy/scary for something that can be triggered by userspace.
Signature · 6 lines
Earthling Michel Dänzer | https://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx