Skip to contentSkip to conversation

drm/amdgpu: use new HMM APIs and helpers

PATCH In review Yang, Philip 3 participants 9 messages May 30, 2019 – Jun 3, 2019 8 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: use new HMM APIs and helpers

HMM provides new APIs and helps in kernel 5.2-rc1 to simplify driver path. The old hmm APIs are deprecated and will be removed in future.

Below are changes in driver:

1. Change hmm_vma_fault to hmm_range_register and hmm_range_fault which supports range with multiple vmas, remove the multiple vmas handle path and data structure.
2. Change hmm_vma_range_done to hmm_range_unregister.
3. Use default flags to avoid pre-fill pfn arrays.
4. Use new hmm_device_ helpers.

Change-Id I1a00f88459f3b96c9536933e9a17eb1ebaa78113
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c  |   1 -
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 140 +++++++++---------------
 2 files changed, 53 insertions(+), 88 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +0 -1
index 41ccee49a224..e0bb47ce570b 100644
@@ -519,7 +519,6 @@ void amdgpu_hmm_init_range(struct hmm_range *range)
 		range->flags = hmm_range_flags;
 		range->values = hmm_range_values;
 		range->pfn_shift = PAGE_SHIFT;
-		range->pfns = NULL;
 		INIT_LIST_HEAD(&range->list);
 	}
 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +53 -87

211 lines not on this page · raw message

Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

This is a nice simplification. See a few comments inline.

Quoted 101 lines · Philip

HMM provides new APIs and helps in kernel 5.2-rc1 to simplify driver path. The old hmm APIs are deprecated and will be removed in future.

Below are changes in driver:

1. Change hmm_vma_fault to hmm_range_register and hmm_range_fault which supports range with multiple vmas, remove the multiple vmas handle path and data structure.
2. Change hmm_vma_range_done to hmm_range_unregister.
3. Use default flags to avoid pre-fill pfn arrays.
4. Use new hmm_device_ helpers.

Change-Id I1a00f88459f3b96c9536933e9a17eb1ebaa78113
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c  |   1 -
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 140 +++++++++---------------
  2 files changed, 53 insertions(+), 88 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +0 -1
index 41ccee49a224..e0bb47ce570b 100644
@@ -519,7 +519,6 @@ void amdgpu_hmm_init_range(struct hmm_range *range)
  		range->flags = hmm_range_flags;
  		range->values = hmm_range_values;
  		range->pfn_shift = PAGE_SHIFT;
-		range->pfns = NULL;
  		INIT_LIST_HEAD(&range->list);
  	}
  }
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +10 -35
index 7138dc1dd1f4..25a9fcb409c6 100644
@@ -711,8 +711,7 @@ struct amdgpu_ttm_tt {
  	struct task_struct	*usertask;
  	uint32_t		userflags;
  #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
-	struct hmm_range	*ranges;
-	int			nr_ranges;
+	struct hmm_range	*range;
  #endif
  };
 
@@ -725,57 +724,33 @@ struct amdgpu_ttm_tt {
   */
  #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
 
-/* Support Userptr pages cross max 16 vmas */
-#define MAX_NR_VMAS	(16)
-
  int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages)
  {
  	struct amdgpu_ttm_tt *gtt = (void *)ttm;
  	struct mm_struct *mm = gtt->usertask->mm;
  	unsigned long start = gtt->userptr;
-	unsigned long end = start + ttm->num_pages * PAGE_SIZE;
-	struct vm_area_struct *vma = NULL, *vmas[MAX_NR_VMAS];
-	struct hmm_range *ranges;
-	unsigned long nr_pages, i;
-	uint64_t *pfns, f;
+	struct vm_area_struct *vma;
+	struct hmm_range *range;
+	unsigned long i;
+	uint64_t *pfns;
  	int r = 0;
 
  	if (!mm) /* Happens during process shutdown */
  		return -ESRCH;
 
-	down_read(&mm->mmap_sem);
-
-	/* user pages may cross multiple VMAs */
-	gtt->nr_ranges = 0;
-	do {
-		unsigned long vm_start;
-
-		if (gtt->nr_ranges >= MAX_NR_VMAS) {
-			DRM_ERROR("Too many VMAs in userptr range\n");
-			r = -EFAULT;
-			goto out;
-		}
-
-		vm_start = vma ? vma->vm_end : start;
-		vma = find_vma(mm, vm_start);
-		if (unlikely(!vma || vm_start < vma->vm_start)) {
-			r = -EFAULT;
-			goto out;
-		}
-		vmas[gtt->nr_ranges++] = vma;
-	} while (end > vma->vm_end);
-
-	DRM_DEBUG_DRIVER("0x%lx nr_ranges %d pages 0x%lx\n",
-		start, gtt->nr_ranges, ttm->num_pages);
-
+	vma = find_vma(mm, start);
+	if (unlikely(!vma || start < vma->vm_start)) {
+		r = -EFAULT;
+		goto out;
+	}

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c struct amdgpu_ttm_tt { · line 752

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2 -3
  	if (unlikely((gtt->userflags & AMDGPU_GEM_USERPTR_ANONONLY) &&
-		vmas[0]->vm_file)) {
+		vma->vm_file)) {
  		r = -EPERM;
  		goto out;
  	}
 
-	ranges = kvmalloc_array(gtt->nr_ranges, sizeof(*ranges), GFP_KERNEL);
-	if (unlikely(!ranges)) {
+	range = kvmalloc(sizeof(*range), GFP_KERNEL | __GFP_ZERO);

A single range is pretty small. You can probably allocate that with kmalloc now (or kcalloc/kzalloc since you specified GFP_ZERO).

Quoted 25 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +9 -8
+	if (unlikely(!range)) {
  		r = -ENOMEM;
  		goto out;
  	}
@@ -786,61 +761,52 @@ int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages)
  		goto out_free_ranges;
  	}
 
-	for (i = 0; i < gtt->nr_ranges; i++)
-		amdgpu_hmm_init_range(&ranges[i]);
+	amdgpu_hmm_init_range(range);
+	range->default_flags = range->flags[HMM_PFN_VALID];
+	range->default_flags |= amdgpu_ttm_tt_is_readonly(ttm) ?
+				0 : range->flags[HMM_PFN_WRITE];
+	range->pfn_flags_mask = 0;
+	range->pfns = pfns;
+	hmm_range_register(range, mm, start,
+			   start + ttm->num_pages * PAGE_SIZE, PAGE_SHIFT);
 
-	f = ranges[0].flags[HMM_PFN_VALID];
-	f |= amdgpu_ttm_tt_is_readonly(ttm) ?
-				0 : ranges[0].flags[HMM_PFN_WRITE];
-	memset64(pfns, f, ttm->num_pages);
-
-	for (nr_pages = 0, i = 0; i < gtt->nr_ranges; i++) {

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages) · line 777

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5 -5
-		ranges[i].vma = vmas[i];
-		ranges[i].start = max(start, vmas[i]->vm_start);
-		ranges[i].end = min(end, vmas[i]->vm_end);
-		ranges[i].pfns = pfns + nr_pages;
-		nr_pages += (ranges[i].end - ranges[i].start) / PAGE_SIZE;
+	/*
+	 * Just wait for range to be valid, safe to ignore return value as we
+	 * will use the return value of hmm_range_fault() below under the
+	 * mmap_sem to ascertain the validity of the range.
+	*/

Documentation/vm/hmm.rst suggests a retry if hmm_range_fault (or snapshot) returns -EAGAIN. I think this would be the right place to put the again: label.

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -7
+	hmm_range_wait_until_valid(range, HMM_RANGE_DEFAULT_TIMEOUT);
 
-		r = hmm_vma_fault(&ranges[i], true);
-		if (unlikely(r))
-			break;
-	}
-	if (unlikely(r)) {
-		while (i--)
-			hmm_vma_range_done(&ranges[i]);

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages) · line 787

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +4 -2
+	down_read(&mm->mmap_sem);
 
-		goto out_free_pfns;
-	}
+	r = hmm_range_fault(range, true);
 
  	up_read(&mm->mmap_sem);
 
+	if (unlikely(r < 0))
+		goto out_free_pfns;

If r == -EAGAIN, you could goto again for a retry here as suggested by the HMM documentation.

Quoted 15 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +6 -4
+
  	for (i = 0; i < ttm->num_pages; i++) {
-		pages[i] = hmm_pfn_to_page(&ranges[0], pfns[i]);
-		if (!pages[i]) {
+		pages[i] = hmm_device_entry_to_page(range, pfns[i]);
+		if (unlikely(!pages[i])) {
  			pr_err("Page fault failed for pfn[%lu] = 0x%llx\n",
  			       i, pfns[i]);
-			goto out_invalid_pfn;
+			r = -ENOMEM;
+
+			goto out_free_pfns;
  		}
  	}
-	gtt->ranges = ranges;

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages) · line 807

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3 -1
+	gtt->range = range;
 
  	return 0;
 
  out_free_pfns:
+	hmm_range_unregister(range);
  	kvfree(pfns);
  out_free_ranges:
-	kvfree(ranges);
+	kvfree(range);

If you use kmalloc/kcalloc/kzalloc above, change this to kfree.

Quoted 49 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +10 -19
  out:
-	up_read(&mm->mmap_sem);
-
  	return r;
-
-out_invalid_pfn:
-	for (i = 0; i < gtt->nr_ranges; i++)
-		hmm_vma_range_done(&ranges[i]);
-	kvfree(pfns);
-	kvfree(ranges);
-	return -ENOMEM;
  }
 
  /**
@@ -853,23 +819,23 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm)
  {
  	struct amdgpu_ttm_tt *gtt = (void *)ttm;
  	bool r = false;
-	int i;
 
  	if (!gtt || !gtt->userptr)
  		return false;
 
-	DRM_DEBUG_DRIVER("user_pages_done 0x%llx nr_ranges %d pages 0x%lx\n",
-		gtt->userptr, gtt->nr_ranges, ttm->num_pages);
+	DRM_DEBUG_DRIVER("user_pages_done 0x%llx pages 0x%lx\n",
+		gtt->userptr, ttm->num_pages);
 
-	WARN_ONCE(!gtt->ranges || !gtt->ranges[0].pfns,
+	WARN_ONCE(!gtt->range || !gtt->range->pfns,
  		"No user pages to check\n");
 
-	if (gtt->ranges) {
-		for (i = 0; i < gtt->nr_ranges; i++)
-			r |= hmm_vma_range_done(&gtt->ranges[i]);
-		kvfree(gtt->ranges[0].pfns);
-		kvfree(gtt->ranges);
-		gtt->ranges = NULL;
+	if (gtt->range) {
+		r = hmm_range_valid(gtt->range);
+		hmm_range_unregister(gtt->range);
+
+		kvfree(gtt->range->pfns);
+		kvfree(gtt->range);
+		gtt->range = NULL;
  	}
 
  	return r;

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 924

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3 -3
@@ -953,9 +919,9 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm)
  	sg_free_table(ttm->sg);
 
  #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
-	if (gtt->ranges &&
-	    ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0],
-					     gtt->ranges[0].pfns[0]))
+	if (gtt->range &&
+	    ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+						      gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not NULL here, we're leaking memory.

Regards,
  Felix

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 927

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -0
  		WARN_ONCE(1, "Missing get_user_page_done\n");
  #endif
  }

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 704410a5-be01-f423-11ef-01a640cd469c-5C7GfCeVMHo@public.gmane.org

#4 lore raw
Quoted 11 lines · Felix

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 924

Quoted 8 lines
   
   #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3 -3
-	if (gtt->ranges &&
-	    ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0],
-					     gtt->ranges[0].pfns[0]))
+	if (gtt->range &&
+	    ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+						      gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not NULL here, we're leaking memory.

If just checking gtt->range, there is a false warning in amdgpu_test userptr case in amdgpu_cs_list_validate path. If userptr is invalidated, then ttm->pages[0] is outdated pages, lobj->user_pages is new pages, it goes to ttm_tt_unbind first to unpin old pages (this causes false warning) then call amdgpu_ttm_tt_set_user_pages.

I will submit patch v2, to add retry if hmm_range_fault returns -EAGAIN. use kzalloc to allocate small size range.

Thanks,
Philip

Quoted 4 lines

Regards,
  Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): bd8f1fd6-f6c4-66fc-c0b9-c8ed36cd6027-5C7GfCeVMHo@public.gmane.org

#6 lore raw
Quoted 13 lines · Philip
Quoted 11 lines · Felix

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) · line 924

Quoted 8 lines
    
    #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3 -3
-	if (gtt->ranges &&
-	    ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0],
-					     gtt->ranges[0].pfns[0]))
+	if (gtt->range &&
+	    ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+						      gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not NULL here, we're leaking memory.

Quoted 5 lines

If just checking gtt->range, there is a false warning in amdgpu_test userptr case in amdgpu_cs_list_validate path. If userptr is invalidated, then ttm->pages[0] is outdated pages, lobj->user_pages is new pages, it goes to ttm_tt_unbind first to unpin old pages (this causes false warning) then call amdgpu_ttm_tt_set_user_pages.

But doesn't that mean we're leaking the gtt->range somewhere?

Regards,
  Felix

Quoted 11 lines

I will submit patch v2, to add retry if hmm_range_fault returns -EAGAIN. use kzalloc to allocate small size range.

Thanks,
Philip

Quoted 4 lines

Regards,
  Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): befbe7fa-0bd3-ffcf-d2eb-36f15053d1a5-5C7GfCeVMHo@public.gmane.org

#8 lore raw
Quoted 14 lines · Felix
Quoted 13 lines · Philip
Quoted 11 lines · Felix

Review

Quoted 8 lines
     
     #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
(quoted diff) +3 -3
-	if (gtt->ranges &&
-	    ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0],
-					     gtt->ranges[0].pfns[0]))
+	if (gtt->range &&
+	    ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+						      gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not NULL here, we're leaking memory.

Quoted 8 lines
Quoted 5 lines

If just checking gtt->range, there is a false warning in amdgpu_test userptr case in amdgpu_cs_list_validate path. If userptr is invalidated, then ttm->pages[0] is outdated pages, lobj->user_pages is new pages, it goes to ttm_tt_unbind first to unpin old pages (this causes false warning) then call amdgpu_ttm_tt_set_user_pages.

But doesn't that mean we're leaking the gtt->range somewhere?

ttm_tt_unbind is called from ttm_tt_destroy and amdgpu_cs_list_validate, the later one causes false warning. ttm_ttm_destory path is fine to only check gtt->range.

Double checked, amdgpu_ttm_tt_get_user_pages and amdgpu_ttm_tt_get_user_pages_done always match in both paths, so no leak gtt->range.

1. amdgpu_gem_userptr_ioctl

       amdgpu_ttm_tt_get_user_pages
           ttm->pages for userptr pages
       amdgpu_ttm_tt_get_user_pages_done

2. amdgpu_cs_ioctl

       amdgpu_cs_parser_bos
           amdgpu_ttm_tt_get_user_pages
           if (userpage_invalidated)
               e->user_pages for new pages
           amdgpu_cs_list_validate
               if (userpage_invalidated)
                  ttm_tt_unbind ttm->pages // this causes warning
                  amdgpu_ttm_tt_set_user_pages(ttm->pages, e->user_pages)
       amdgpu_cs_submit
           amdgpu_ttm_tt_get_user_pages_done
Quoted 16 lines

Regards,
  Felix

Quoted 12 lines

I will submit patch v2, to add retry if hmm_range_fault returns -EAGAIN. use kzalloc to allocate small size range.

Thanks,
Philip

Quoted 4 lines

Regards,
  Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 9ae26883-4326-c60f-9a8e-d95d0d458e31-5C7GfCeVMHo@public.gmane.org

#10 lore raw
Quoted 36 lines · Philip
Quoted 13 lines · Felix
Quoted 12 lines · Philip
Quoted 11 lines · Felix

Review

Quoted 8 lines
      
      #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
(quoted diff) +3 -3
-	if (gtt->ranges &&
-	    ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0],
-					     gtt->ranges[0].pfns[0]))
+	if (gtt->range &&
+	    ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+						      gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not NULL here, we're leaking memory.

Quoted 7 lines
Quoted 5 lines

If just checking gtt->range, there is a false warning in amdgpu_test userptr case in amdgpu_cs_list_validate path. If userptr is invalidated, then ttm->pages[0] is outdated pages, lobj->user_pages is new pages, it goes to ttm_tt_unbind first to unpin old pages (this causes false warning) then call amdgpu_ttm_tt_set_user_pages.

But doesn't that mean we're leaking the gtt->range somewhere?

ttm_tt_unbind is called from ttm_tt_destroy and amdgpu_cs_list_validate, the later one causes false warning. ttm_ttm_destory path is fine to only check gtt->range.

Double checked, amdgpu_ttm_tt_get_user_pages and amdgpu_ttm_tt_get_user_pages_done always match in both paths, so no leak gtt->range.

1. amdgpu_gem_userptr_ioctl

        amdgpu_ttm_tt_get_user_pages
            ttm->pages for userptr pages
        amdgpu_ttm_tt_get_user_pages_done

2. amdgpu_cs_ioctl

Quoted 8 lines
        amdgpu_cs_parser_bos
            amdgpu_ttm_tt_get_user_pages
            if (userpage_invalidated)
                e->user_pages for new pages
            amdgpu_cs_list_validate
                if (userpage_invalidated)
                   ttm_tt_unbind ttm->pages // this causes warning
                   amdgpu_ttm_tt_set_user_pages(ttm->pages, e->user_pages)

Hmm, I think amdgpu_cs is doing something weird there. It does some double book-keeping of the user pages in the BO list and the TTM BO. We did something similar for KFD and simplified it when moving to HMM. It could probably be simplified for amdgpu_cs as well. But not in this patch series.

I'll review your updated change.

Thanks,
  Felix

Quoted 17 lines

amdgpu_cs_submit
amdgpu_ttm_tt_get_user_pages_done

Quoted 14 lines

Regards,
  Felix

Quoted 10 lines

I will submit patch v2, to add retry if hmm_range_fault returns -EAGAIN. use kzalloc to allocate small size range.

Thanks,
Philip

Quoted 4 lines

Regards,
  Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 9f5f4060-5f8f-b688-2cc2-39aca92c7505-5C7GfCeVMHo@public.gmane.org

#12 lore raw

Am 01.06.19 um 00:01 schrieb Kuehling, Felix:

Quoted 44 lines · Philip
Quoted 35 lines · Felix
Quoted 13 lines · Philip
Quoted 12 lines · Felix
Quoted 11 lines

Review

Quoted 8 lines
       
       #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
(quoted diff) +3 -3
-	if (gtt->ranges &&
-	    ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0],
-					     gtt->ranges[0].pfns[0]))
+	if (gtt->range &&
+	    ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+						      gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not NULL here, we're leaking memory.

Quoted 7 lines
Quoted 5 lines

If just checking gtt->range, there is a false warning in amdgpu_test userptr case in amdgpu_cs_list_validate path. If userptr is invalidated, then ttm->pages[0] is outdated pages, lobj->user_pages is new pages, it goes to ttm_tt_unbind first to unpin old pages (this causes false warning) then call amdgpu_ttm_tt_set_user_pages.

But doesn't that mean we're leaking the gtt->range somewhere?

ttm_tt_unbind is called from ttm_tt_destroy and amdgpu_cs_list_validate, the later one causes false warning. ttm_ttm_destory path is fine to only check gtt->range.

Double checked, amdgpu_ttm_tt_get_user_pages and amdgpu_ttm_tt_get_user_pages_done always match in both paths, so no leak gtt->range.

1. amdgpu_gem_userptr_ioctl

         amdgpu_ttm_tt_get_user_pages
             ttm->pages for userptr pages
         amdgpu_ttm_tt_get_user_pages_done

2. amdgpu_cs_ioctl

Quoted 8 lines
         amdgpu_cs_parser_bos
             amdgpu_ttm_tt_get_user_pages
             if (userpage_invalidated)
                 e->user_pages for new pages
             amdgpu_cs_list_validate
                 if (userpage_invalidated)
                    ttm_tt_unbind ttm->pages // this causes warning
                    amdgpu_ttm_tt_set_user_pages(ttm->pages, e->user_pages)
Quoted 5 lines

Hmm, I think amdgpu_cs is doing something weird there. It does some double book-keeping of the user pages in the BO list and the TTM BO. We did something similar for KFD and simplified it when moving to HMM. It could probably be simplified for amdgpu_cs as well. But not in this patch series.

That actually sounds like a bug to me.

It is most likely a leftover from the time when we couldn't get the pages for a BO while the BO was reserved.

Going to take a closer look,
Christian.

Quoted 20 lines

I'll review your updated change.

Thanks,
  Felix

Quoted 13 lines

amdgpu_cs_submit
amdgpu_ttm_tt_get_user_pages_done

Quoted 10 lines

Regards,
  Felix

Quoted 6 lines

I will submit patch v2, to add retry if hmm_range_fault returns -EAGAIN. use kzalloc to allocate small size range.

Thanks,
Philip

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Regards,
  Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 972ef129-3dae-d5eb-100f-b9be83d0afcc-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#14 lore raw

Am 03.06.19 um 12:17 schrieb Christian König:

Quoted 50 lines

Am 01.06.19 um 00:01 schrieb Kuehling, Felix:

Quoted 43 lines · Philip
Quoted 33 lines · Felix
Quoted 12 lines · Philip
Quoted 11 lines · Felix
Quoted 10 lines
Quoted 7 lines

              #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR) -    if (gtt->ranges &&
-        ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0], -                         gtt->ranges[0].pfns[0]))
+    if (gtt->range &&
+        ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+ gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not NULL here, we're leaking memory.

Quoted 7 lines
Quoted 5 lines

If just checking gtt->range, there is a false warning in amdgpu_test userptr case in amdgpu_cs_list_validate path. If userptr is invalidated, then ttm->pages[0] is outdated pages, lobj->user_pages is new pages, it goes to ttm_tt_unbind first to unpin old pages (this causes false warning) then call amdgpu_ttm_tt_set_user_pages.

But doesn't that mean we're leaking the gtt->range somewhere?

ttm_tt_unbind is called from ttm_tt_destroy and amdgpu_cs_list_validate, the later one causes false warning. ttm_ttm_destory path is fine to only check gtt->range.

Double checked, amdgpu_ttm_tt_get_user_pages and amdgpu_ttm_tt_get_user_pages_done always match in both paths, so no leak gtt->range.

1. amdgpu_gem_userptr_ioctl
         amdgpu_ttm_tt_get_user_pages
             ttm->pages for userptr pages
         amdgpu_ttm_tt_get_user_pages_done

Quoted 9 lines

2. amdgpu_cs_ioctl
         amdgpu_cs_parser_bos amdgpu_ttm_tt_get_user_pages
             if (userpage_invalidated)
                 e->user_pages for new pages amdgpu_cs_list_validate
                 if (userpage_invalidated)
                    ttm_tt_unbind ttm->pages // this causes warning amdgpu_ttm_tt_set_user_pages(ttm->pages, e->user_pages)

Quoted 5 lines

Hmm, I think amdgpu_cs is doing something weird there. It does some double book-keeping of the user pages in the BO list and the TTM BO. We did something similar for KFD and simplified it when moving to HMM. It could probably be simplified for amdgpu_cs as well. But not in this patch series.

Quoted 3 lines

That actually sounds like a bug to me.

It is most likely a leftover from the time when we couldn't get the pages for a BO while the BO was reserved.

Mhm, at least it's not racy in the way I thought it would be. But it is certainly still overkill and should be simplified.

Philip are you taking a look or should I tackle this?

Thanks,
Christian.

Quoted 35 lines

Going to take a closer look,
Christian.

Quoted 29 lines

I'll review your updated change.

Thanks,
    Felix

Quoted 18 lines

         amdgpu_cs_submit
             amdgpu_ttm_tt_get_user_pages_done

Quoted 15 lines

Regards,
      Felix

Quoted 11 lines

I will submit patch v2, to add retry if hmm_range_fault returns -EAGAIN.
use kzalloc to allocate small size range.

Thanks,
Philip

Quoted 4 lines

Regards,
        Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): c20fba4e-f303-75b9-4bba-bdede43237aa-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#16 lore raw
Quoted 59 lines · Christian König

Am 03.06.19 um 12:17 schrieb Christian König:

Quoted 58 lines

Am 01.06.19 um 00:01 schrieb Kuehling, Felix:

Quoted 49 lines · Philip
Quoted 39 lines · Felix
Quoted 13 lines · Philip
Quoted 12 lines · Felix
Quoted 11 lines
Quoted 7 lines

              #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR) -    if (gtt->ranges &&
-        ttm->pages[0] == hmm_pfn_to_page(&gtt->ranges[0], -                         gtt->ranges[0].pfns[0]))
+    if (gtt->range &&
+        ttm->pages[0] == hmm_device_entry_to_page(gtt->range,
+ gtt->range->pfns[0]))

I think just checking gtt->range here is enough. If gtt->range is not
NULL here, we're leaking memory.

Quoted 9 lines
Quoted 7 lines

If just checking gtt->range, there is a false warning in amdgpu_test userptr case in amdgpu_cs_list_validate path. If userptr is invalidated,
then ttm->pages[0] is outdated pages, lobj->user_pages is new pages, it
goes to ttm_tt_unbind first to unpin old pages (this causes false warning) then call amdgpu_ttm_tt_set_user_pages.

But doesn't that mean we're leaking the gtt->range somewhere?

ttm_tt_unbind is called from ttm_tt_destroy and amdgpu_cs_list_validate,
the later one causes false warning. ttm_ttm_destory path is fine to only
check gtt->range.

Double checked, amdgpu_ttm_tt_get_user_pages and amdgpu_ttm_tt_get_user_pages_done always match in both paths, so no leak
gtt->range.

1. amdgpu_gem_userptr_ioctl
         amdgpu_ttm_tt_get_user_pages
             ttm->pages for userptr pages
         amdgpu_ttm_tt_get_user_pages_done

Quoted 9 lines

2. amdgpu_cs_ioctl
         amdgpu_cs_parser_bos amdgpu_ttm_tt_get_user_pages
             if (userpage_invalidated)
                 e->user_pages for new pages amdgpu_cs_list_validate
                 if (userpage_invalidated)
                    ttm_tt_unbind ttm->pages // this causes warning amdgpu_ttm_tt_set_user_pages(ttm->pages, e->user_pages)

Quoted 5 lines

Hmm, I think amdgpu_cs is doing something weird there. It does some double book-keeping of the user pages in the BO list and the TTM BO. We did something similar for KFD and simplified it when moving to HMM. It could probably be simplified for amdgpu_cs as well. But not in this patch series.

That actually sounds like a bug to me.

Quoted 8 lines
Quoted 2 lines

It is most likely a leftover from the time when we couldn't get the pages for a BO while the BO was reserved.

Mhm, at least it's not racy in the way I thought it would be. But it is certainly still overkill and should be simplified.

Philip are you taking a look or should I tackle this?

Hi Christian,

I will submit another patch to simplify amdgpu_cs_ioctl path, please help review it.

Thanks,
Philip

Quoted 39 lines

Thanks,
Christian.

Quoted 35 lines

Going to take a closer look,
Christian.

Quoted 29 lines

I'll review your updated change.

Thanks,
    Felix

Quoted 18 lines

         amdgpu_cs_submit
             amdgpu_ttm_tt_get_user_pages_done

Quoted 15 lines

Regards,
      Felix

Quoted 11 lines

I will submit patch v2, to add retry if hmm_range_fault returns -EAGAIN.
use kzalloc to allocate small size range.

Thanks,
Philip

Quoted 4 lines

Regards,
        Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx