Skip to contentSkip to conversation

drm/amdkfd: Fix svm_bo and vram page refcount

PATCH In review Philip Yang 7 participants 13 messages Sep 26, 2025 – Oct 6, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdkfd: Fix svm_bo and vram page refcount

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Afterwards, this causes use-after-free bug and page refcount warning.

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +13 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
 	page = pfn_to_page(pfn);
 	svm_range_bo_ref(prange->svm_bo);
 	page->zone_device_data = prange->svm_bo;
-	zone_device_page_init(page);
+	get_page(page);
+	lock_page(page);
 }
 
 static void
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
 	if (mpages) {
 		prange->actual_loc = best_loc;
 		prange->vram_pages += mpages;
+		/*
+		 * To guarent we hold correct page refcount for all prange vram
+		 * pages and svm_bo refcount.
+		 * After prange migrated to VRAM, each vram page refcount hold
+		 * one svm_bo refcount, and vram node hold one refcount.
+		 * After page migrated to system memory, vram page refcount
+		 * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+		 * svm_range_vram_node_free will free the svm_bo.
+		 */
+		WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+			  "svm_bo refcount leaking\n");
 	} else if (!prange->actual_loc) {
 		/* if no page migrated and all pages from prange are at
 		 * sys ram drop svm_bo got from svm_range_vram_node_new
Signature · 1 line
2.49.0
Acked-by on v1 1/1
#1 lore raw

[AMD Official Use Only - AMD Internal Distribution Only]

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Philip Yang
Sent: Friday, September 26, 2025 5:04 PM
To: [email protected]
Cc: Kuehling, Felix <[email protected]>; Yang, Philip <[email protected]>
Subject: [PATCH] drm/amdkfd: Fix svm_bo and vram page refcount

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Afterwards, this causes use-after-free bug and page refcount warning.

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +13 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
        page = pfn_to_page(pfn);
        svm_range_bo_ref(prange->svm_bo);
        page->zone_device_data = prange->svm_bo;
-       zone_device_page_init(page);
+       get_page(page);
+       lock_page(page);
 }
 
 static void
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
        if (mpages) {
                prange->actual_loc = best_loc;
                prange->vram_pages += mpages;
+               /*
+                * To guarent we hold correct page refcount for all prange vram
+                * pages and svm_bo refcount.
+                * After prange migrated to VRAM, each vram page refcount hold
+                * one svm_bo refcount, and vram node hold one refcount.
+                * After page migrated to system memory, vram page refcount
+                * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+                * svm_range_vram_node_free will free the svm_bo.
+                */
+               WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+                         "svm_bo refcount leaking\n");
        } else if (!prange->actual_loc) {
                /* if no page migrated and all pages from prange are at
                 * sys ram drop svm_bo got from svm_range_vram_node_new

--
2.49.0

#2 lore raw
In reply to Philip Yang
ThispatchisReviewed-by JamesZhu<[email protected]>

James Zhu

Quoted 55 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Afterwards, this causes use-after-free bug and page refcount warning.

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-
  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +13 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
  	page = pfn_to_page(pfn);
  	svm_range_bo_ref(prange->svm_bo);
  	page->zone_device_data = prange->svm_bo;
-	zone_device_page_init(page);
+	get_page(page);
+	lock_page(page);
  }
 
  static void
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
  	if (mpages) {
  		prange->actual_loc = best_loc;
  		prange->vram_pages += mpages;
+		/*
+		 * To guarent we hold correct page refcount for all prange vram
+		 * pages and svm_bo refcount.
+		 * After prange migrated to VRAM, each vram page refcount hold
+		 * one svm_bo refcount, and vram node hold one refcount.
+		 * After page migrated to system memory, vram page refcount
+		 * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+		 * svm_range_vram_node_free will free the svm_bo.
+		 */
+		WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+			  "svm_bo refcount leaking\n");
  	} else if (!prange->actual_loc) {
  		/* if no page migrated and all pages from prange are at
  		 * sys ram drop svm_bo got from svm_range_vram_node_new
#3 lore raw
In reply to Philip Yang
Quoted 38 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Afterwards, this causes use-after-free bug and page refcount warning.

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-
  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
  	page = pfn_to_page(pfn);
  	svm_range_bo_ref(prange->svm_bo);
  	page->zone_device_data = prange->svm_bo;
-	zone_device_page_init(page);
+	get_page(page);
+	lock_page(page);
  }
 
  static void

drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, · line 557

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -0
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
  	if (mpages) {
  		prange->actual_loc = best_loc;
  		prange->vram_pages += mpages;
+		/*
+		 * To guarent we hold correct page refcount for all prange vram

guarantee> + * pages and svm_bo refcount.

Quoted 11 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +8 -0
+		 * After prange migrated to VRAM, each vram page refcount hold
+		 * one svm_bo refcount, and vram node hold one refcount.
+		 * After page migrated to system memory, vram page refcount
+		 * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+		 * svm_range_vram_node_free will free the svm_bo.
+		 */
+		WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+			  "svm_bo refcount leaking\n");
  	} else if (!prange->actual_loc) {
  		/* if no page migrated and all pages from prange are at
  		 * sys ram drop svm_bo got from svm_range_vram_node_new
maintainer
#4 lore raw
In reply to Philip Yang
Quoted 11 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Afterwards, this causes use-after-free bug and page refcount warning.

This implies that migration to RAM and to VRAM of the same range are happening at the same time. Isn't that a bigger problem? It means someone doing a migration is not holding the prange->migrate_mutex.

Regards,
  Felix

Quoted 44 lines

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-
  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +13 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
  	page = pfn_to_page(pfn);
  	svm_range_bo_ref(prange->svm_bo);
  	page->zone_device_data = prange->svm_bo;
-	zone_device_page_init(page);
+	get_page(page);
+	lock_page(page);
  }
 
  static void
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
  	if (mpages) {
  		prange->actual_loc = best_loc;
  		prange->vram_pages += mpages;
+		/*
+		 * To guarent we hold correct page refcount for all prange vram
+		 * pages and svm_bo refcount.
+		 * After prange migrated to VRAM, each vram page refcount hold
+		 * one svm_bo refcount, and vram node hold one refcount.
+		 * After page migrated to system memory, vram page refcount
+		 * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+		 * svm_range_vram_node_free will free the svm_bo.
+		 */
+		WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+			  "svm_bo refcount leaking\n");
  	} else if (!prange->actual_loc) {
  		/* if no page migrated and all pages from prange are at
  		 * sys ram drop svm_bo got from svm_range_vram_node_new
#5 lore raw
Quoted 11 lines · Felix Kuehling
Quoted 10 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Quoted 3 lines
Quoted 1 line

Afterwards, this causes use-after-free bug and page refcount warning.

This implies that migration to RAM and to VRAM of the same range are happening at the same time. Isn't that a bigger problem? It means someone doing a migration is not holding the prange->migrate_mutex.

Migration hold prange->migrate_mutex so we don't have migration to RAM and VRAM of same range at same time, the issue is in step 3, CPU page fault handler do_swap_page put_page after pgmap->ops->migrate_to_ram() returns and during migate_to_vram.

Regards,

Philip

Quoted 51 lines

Regards,
  Felix

Quoted 46 lines

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-

  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range 
 *prange, unsigned long pfn)
       page = pfn_to_page(pfn);
       svm_range_bo_ref(prange->svm_bo);
       page->zone_device_data = prange->svm_bo;
-    zone_device_page_init(page);
+    get_page(page);
+    lock_page(page);
   }

    static void

drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +11 -0
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range 
 *prange, uint32_t best_loc,
       if (mpages) {
           prange->actual_loc = best_loc;
           prange->vram_pages += mpages;
+        /*
+         * To guarent we hold correct page refcount for all prange vram
+         * pages and svm_bo refcount.
+         * After prange migrated to VRAM, each vram page refcount hold
+         * one svm_bo refcount, and vram node hold one refcount.
+         * After page migrated to system memory, vram page refcount
+         * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+         * svm_range_vram_node_free will free the svm_bo.
+         */
+        WARN_ONCE(prange->vram_pages == 
 kref_read(&prange->svm_bo->kref),
+              "svm_bo refcount leaking\n");
       } else if (!prange->actual_loc) {

          /* if no page migrated and all pages from prange are at
           * sys ram drop svm_bo got from svm_range_vram_node_new

maintainer
#6 lore raw
Quoted 13 lines · Philip Yang
Quoted 11 lines · Felix Kuehling
Quoted 10 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Afterwards, this causes use-after-free bug and page refcount warning.

This implies that migration to RAM and to VRAM of the same range are happening at the same time. Isn't that a bigger problem? It means someone doing a migration is not holding the prange->migrate_mutex.

Migration hold prange->migrate_mutex so we don't have migration to RAM and VRAM of same range at same time, the issue is in step 3, CPU page fault handler do_swap_page put_page after pgmap->ops->migrate_to_ram() returns and during migate_to_vram.

That's the part I don't understand. The CPU page fault handler (svm_migrate_to_ram) is holding prange->migrate_mutex until the very end. Where do we have a put_page for a zone_device page outside the prange->migrate_mutex? Do you have a backtrace?

Regards,
  Felix

Quoted 57 lines

Regards,

Philip

Quoted 52 lines

Regards,
  Felix

Quoted 47 lines

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-

  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range 
 *prange, unsigned long pfn)
       page = pfn_to_page(pfn);
       svm_range_bo_ref(prange->svm_bo);
       page->zone_device_data = prange->svm_bo;
-    zone_device_page_init(page);
+    get_page(page);
+    lock_page(page);
   }

    static void

drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -0
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range 
 *prange, uint32_t best_loc,
       if (mpages) {
           prange->actual_loc = best_loc;
           prange->vram_pages += mpages;
+        /*
+         * To guarent we hold correct page refcount for all prange 

vram
+         * pages and svm_bo refcount.
+         * After prange migrated to VRAM, each vram page refcount hold +         * one svm_bo refcount, and vram node hold one refcount. +         * After page migrated to system memory, vram page refcount +         * reduced to 0, svm_migrate_page_free reduce svm_bo refcount. +         * svm_range_vram_node_free will free the svm_bo.
+         */
+        WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+              "svm_bo refcount leaking\n");
      } else if (!prange->actual_loc) {
          /* if no page migrated and all pages from prange are at
           * sys ram drop svm_bo got from svm_range_vram_node_new

#7 lore raw
Quoted 15 lines · Felix Kuehling
Quoted 13 lines · Philip Yang
Quoted 11 lines · Felix Kuehling
Quoted 10 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Afterwards, this causes use-after-free bug and page refcount warning.

This implies that migration to RAM and to VRAM of the same range are happening at the same time. Isn't that a bigger problem? It means someone doing a migration is not holding the prange->migrate_mutex.

Migration hold prange->migrate_mutex so we don't have migration to RAM and VRAM of same range at same time, the issue is in step 3, CPU page fault handler do_swap_page put_page after pgmap->ops->migrate_to_ram() returns and during migate_to_vram.

That's the part I don't understand. The CPU page fault handler (svm_migrate_to_ram) is holding prange->migrate_mutex until the very end. Where do we have a put_page for a zone_device page outside the prange->migrate_mutex? Do you have a backtrace?

do_swap_page() {
   .......
        } else if (is_device_private_entry(entry)) {
   ........

            /*
             * Get a page reference while we know the page can't be
             * freed.
             */
            if (trylock_page(vmf->page)) {
                struct dev_pagemap *pgmap;

                get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); pgmap = page_pgmap(vmf->page); ret = pgmap->ops->migrate_to_ram(vmf); unlock_page(vmf->page); put_page(vmf->page);

This put_page reduce the vram page refcount to zero if migrate_to_vram -> svm_migrate_get_vram_page already call zone_device_page_init set page refcount to 1.

put_page must be after unlock_page as put_page may free the page, svm_migrate_get_vram_page can lock the page, but page refcount becomes 0.

Regards,

Philip

Quoted 64 lines

Regards,
  Felix

Quoted 59 lines

Regards,

Philip

Quoted 54 lines

Regards,
  Felix

Quoted 49 lines

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-

  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range 
 *prange, unsigned long pfn)
       page = pfn_to_page(pfn);
       svm_range_bo_ref(prange->svm_bo);
       page->zone_device_data = prange->svm_bo;
-    zone_device_page_init(page);
+    get_page(page);
+    lock_page(page);
   }

    static void

drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -0
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range 
 *prange, uint32_t best_loc,
       if (mpages) {
           prange->actual_loc = best_loc;
           prange->vram_pages += mpages;
+        /*
+         * To guarent we hold correct page refcount for all prange 

vram
+         * pages and svm_bo refcount.
+         * After prange migrated to VRAM, each vram page refcount hold
+         * one svm_bo refcount, and vram node hold one refcount. +         * After page migrated to system memory, vram page refcount +         * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+         * svm_range_vram_node_free will free the svm_bo. +         */
+        WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+              "svm_bo refcount leaking\n");
      } else if (!prange->actual_loc) {
          /* if no page migrated and all pages from prange are at
           * sys ram drop svm_bo got from svm_range_vram_node_new

maintainer
#8 lore raw

[+Linux MM and HMM maintainers]

Please see below my question about the safety of using zone_device_page_init.

Quoted 43 lines · Philip Yang
Quoted 15 lines · Felix Kuehling
Quoted 13 lines · Philip Yang
Quoted 11 lines · Felix Kuehling
Quoted 10 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Afterwards, this causes use-after-free bug and page refcount warning.

This implies that migration to RAM and to VRAM of the same range are happening at the same time. Isn't that a bigger problem? It means someone doing a migration is not holding the prange->migrate_mutex.

Migration hold prange->migrate_mutex so we don't have migration to RAM and VRAM of same range at same time, the issue is in step 3, CPU page fault handler do_swap_page put_page after pgmap->ops->migrate_to_ram() returns and during migate_to_vram.

That's the part I don't understand. The CPU page fault handler (svm_migrate_to_ram) is holding prange->migrate_mutex until the very end. Where do we have a put_page for a zone_device page outside the prange->migrate_mutex? Do you have a backtrace?

do_swap_page() {
   .......
        } else if (is_device_private_entry(entry)) {
   ........

            /*
             * Get a page reference while we know the page can't be
             * freed.
             */
            if (trylock_page(vmf->page)) {
                struct dev_pagemap *pgmap;

                get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); pgmap = page_pgmap(vmf->page); ret = pgmap->ops->migrate_to_ram(vmf); unlock_page(vmf->page); put_page(vmf->page);

Quoted 5 lines

This put_page reduce the vram page refcount to zero if migrate_to_vram -> svm_migrate_get_vram_page already call zone_device_page_init set page refcount to 1.

put_page must be after unlock_page as put_page may free the page, svm_migrate_get_vram_page can lock the page, but page refcount becomes 0.

OK. Then you must have hit the WARN_ON_ONCE(!percpu_ref_tryget_live(&page_pgmap(page)->ref)) in that function.

It sounds like zone_device_page_init is just unsafe to use in general. It assumes that pages have a 0 refcount. But I don't see a good way for drivers to guarantee that, because they are not in control of when the page refcounts for their zone-device pages get decremented.

Regards,
  Felix

Quoted 69 lines

Regards,

Philip

Quoted 64 lines

Regards,
  Felix

Quoted 59 lines

Regards,

Philip

Quoted 54 lines

Regards,
  Felix

Quoted 49 lines

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-

  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range 
 *prange, unsigned long pfn)
       page = pfn_to_page(pfn);
       svm_range_bo_ref(prange->svm_bo);
       page->zone_device_data = prange->svm_bo;
-    zone_device_page_init(page);
+    get_page(page);
+    lock_page(page);
   }

    static void

drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -0
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range 
 *prange, uint32_t best_loc,
       if (mpages) {
           prange->actual_loc = best_loc;
           prange->vram_pages += mpages;
+        /*
+         * To guarent we hold correct page refcount for all 

prange vram
+         * pages and svm_bo refcount.
+         * After prange migrated to VRAM, each vram page refcount hold
+         * one svm_bo refcount, and vram node hold one refcount. +         * After page migrated to system memory, vram page refcount +         * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+         * svm_range_vram_node_free will free the svm_bo. +         */
+        WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+              "svm_bo refcount leaking\n");
      } else if (!prange->actual_loc) {
          /* if no page migrated and all pages from prange are at
           * sys ram drop svm_bo got from svm_range_vram_node_new

#9 lore raw
Quoted 48 lines · Felix Kuehling

[+Linux MM and HMM maintainers]

Please see below my question about the safety of using zone_device_page_init.

Quoted 43 lines · Philip Yang
Quoted 15 lines · Felix Kuehling
Quoted 13 lines · Philip Yang
Quoted 11 lines · Felix Kuehling
Quoted 10 lines · Philip Yang

zone_device_page_init uses set_page_count to set vram page refcount to 1, there is race if step 2 happens between step 1 and 3.

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1
3. CPU page fault handler put vram page, the vram page refcount is 0 and reduce the vram_bo refcount
4. vram_bo refcount is 1 off because the vram page is still used.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Afterwards, this causes use-after-free bug and page refcount warning.

This implies that migration to RAM and to VRAM of the same range are happening at the same time. Isn't that a bigger problem? It means someone doing a migration is not holding the prange->migrate_mutex.

Migration hold prange->migrate_mutex so we don't have migration to RAM and VRAM of same range at same time, the issue is in step 3, CPU page fault handler do_swap_page put_page after pgmap->ops->migrate_to_ram() returns and during migate_to_vram.

That's the part I don't understand. The CPU page fault handler (svm_migrate_to_ram) is holding prange->migrate_mutex until the very end. Where do we have a put_page for a zone_device page outside the prange->migrate_mutex? Do you have a backtrace?

do_swap_page() {
   .......
        } else if (is_device_private_entry(entry)) {
   ........

            /*
             * Get a page reference while we know the page can't be
             * freed.
             */
            if (trylock_page(vmf->page)) {
                struct dev_pagemap *pgmap;

                get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); pgmap = page_pgmap(vmf->page); ret = pgmap->ops->migrate_to_ram(vmf); unlock_page(vmf->page); put_page(vmf->page);

Quoted 5 lines
Quoted 3 lines

This put_page reduce the vram page refcount to zero if migrate_to_vram -> svm_migrate_get_vram_page already call zone_device_page_init set page refcount to 1.

put_page must be after unlock_page as put_page may free the page, svm_migrate_get_vram_page can lock the page, but page refcount becomes 0.

OK. Then you must have hit the WARN_ON_ONCE(!percpu_ref_tryget_live(&page_pgmap(page)->ref)) in that function.

This warning is for pgmap percpu_refcount, not for page refcount, I didn't see this warning.

Quoted 2 lines

It sounds like zone_device_page_init is just unsafe to use in general. It assumes that pages have a 0 refcount. But I don't see a good way for drivers to guarantee that, because they are not in control of when the page refcounts for their zone-device pages get decremented.

Seems this issue is caused by the change in commit 1afaeb8293c9 "mm/migrate: Trylock device page in do_swap_page", I am not sure if the same fix is needed in several drivers calling zone_device_page_init.

Regards,

Philip

Quoted 74 lines

Regards,
  Felix

Quoted 69 lines

Regards,

Philip

Quoted 64 lines

Regards,
  Felix

Quoted 59 lines

Regards,

Philip

Quoted 54 lines

Regards,
  Felix

Quoted 49 lines

zone_device_page_init should not use in page migration, change to get_page fix the race bug.

Add WARN_ONCE to report this issue early because the refcount bug is hard to investigate.

---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 +++++++++++++-

  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -1
index d10c6673f4de..15ab2db4af1d 100644
@@ -217,7 +217,8 @@ svm_migrate_get_vram_page(struct svm_range 
 *prange, unsigned long pfn)
       page = pfn_to_page(pfn);
       svm_range_bo_ref(prange->svm_bo);
       page->zone_device_data = prange->svm_bo;
-    zone_device_page_init(page);
+    get_page(page);
+    lock_page(page);
   }

    static void

drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -0
@@ -552,6 +553,17 @@ svm_migrate_ram_to_vram(struct svm_range 
 *prange, uint32_t best_loc,
       if (mpages) {
           prange->actual_loc = best_loc;
           prange->vram_pages += mpages;
+        /*
+         * To guarent we hold correct page refcount for all 

prange vram
+         * pages and svm_bo refcount.
+         * After prange migrated to VRAM, each vram page refcount hold
+         * one svm_bo refcount, and vram node hold one refcount. +         * After page migrated to system memory, vram page refcount +         * reduced to 0, svm_migrate_page_free reduce svm_bo refcount.
+         * svm_range_vram_node_free will free the svm_bo. +         */
+        WARN_ONCE(prange->vram_pages == kref_read(&prange->svm_bo->kref),
+              "svm_bo refcount leaking\n");
      } else if (!prange->actual_loc) {
          /* if no page migrated and all pages from prange are at
           * sys ram drop svm_bo got from svm_range_vram_node_new

#10 lore raw
In reply to Felix Kuehling
Quoted 2 lines · Felix Kuehling

It sounds like zone_device_page_init is just unsafe to use in general.

It can only be used if you know the page is freed.

Quoted 1 line

It assumes that pages have a 0 refcount.

Yes

Quoted 3 lines

But I don't see a good way for drivers to guarantee that, because they are not in control of when the page refcounts for their zone-device pages get decremented.

?? Drivers are supposed to hoook pgmap->ops->page_free() and keep track.

There is no way to write a driver without calling zone_device_page_init() as there is no other defined way to re-use a page that has been returned through page_free().

It is completely wrong to call get_page() on a 0 refcount folio, we don't have a debugging crash for this, but we really should. If you think the refcount could be 0 you have to use a try_get().

So this patch looks wrong to me, I see a page_free() implementation and this is the only call to zone_device_page_init(). If you remove it the driver is absolutely broken.

I would expect migration should be writing to freed memory and zone_device_page_init() is the correct and only way to make freed memory usable again.

Therefore, I expect the refcount to be 0 when
svm_migrate_ram_to_vram() picks a dst.

If it is not true, and you are tring to migrate to already allocated VRAM, then WTF?

And if you really want to do that then yes you need to use get_page but you need a different path to handle already allocated vs page_free() called. get_page() MUST NOT be used to unfree page_free'd memory.

The explanation in the commit doesn't really have enough detail:

Quoted 3 lines

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1

So why is the same vram page being used for both? For #1 the VRAM page is installed in a swap entry so it is has an elevated refcount.

The implication is that #2 is targeting already allocated VRAM memory that is NOT FREE.

Jason

maintainer
#11 lore raw
Quoted 6 lines · Jason Gunthorpe
Quoted 2 lines · Felix Kuehling

It sounds like zone_device_page_init is just unsafe to use in general.

It can only be used if you know the page is freed.

Quoted 8 lines
Quoted 1 line

It assumes that pages have a 0 refcount.

Yes

Quoted 3 lines

But I don't see a good way for drivers to guarantee that, because they are not in control of when the page refcounts for their zone-device pages get decremented.

?? Drivers are supposed to hoook pgmap->ops->page_free() and keep track.

Right, we have that callback, it's currently only used to track references to our buffer objects used to back device pages.

Quoted 9 lines

There is no way to write a driver without calling zone_device_page_init() as there is no other defined way to re-use a page that has been returned through page_free().

It is completely wrong to call get_page() on a 0 refcount folio, we don't have a debugging crash for this, but we really should. If you think the refcount could be 0 you have to use a try_get().

Quoted 7 lines

So this patch looks wrong to me, I see a page_free() implementation and this is the only call to zone_device_page_init(). If you remove it the driver is absolutely broken.

I would expect migration should be writing to freed memory and zone_device_page_init() is the correct and only way to make freed memory usable again.

OK. We made an incorrect assumption that we can reuse a page if the driver isn't tracking it as allocated to any of our SVM ranges (i.e., after dev_pagemap_ops.migrate_to_ram() migrated all data out of the page). However, we neglected that other parts of the kernel can still hold references to a page even after that.

Would something like this work:

static void
svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)

{
	...
	if (!try_get_page(page)) {
		page->zone_device_data = prange->svm_bo;
		zone_device_page_init(page);
	}
}
Quoted 6 lines

Therefore, I expect the refcount to be 0 when
svm_migrate_ram_to_vram() picks a dst.

If it is not true, and you are tring to migrate to already allocated VRAM, then WTF?

As I understand it, it's a race condition. The driver is done with the page and its migrate_to_ram() call has completed. But do_swap_page hasn't called put_page yet. At the same time, another thread is trying to reuse the page, migrating data back to VRAM.

Regards,
  Felix

Quoted 19 lines

And if you really want to do that then yes you need to use get_page but you need a different path to handle already allocated vs page_free() called. get_page() MUST NOT be used to unfree page_free'd memory.

The explanation in the commit doesn't really have enough detail:

Quoted 3 lines

1. CPU page fault handler get vram page, migrate the vram page to system page
2. GPU page fault migrate to the vram page, set page refcount to 1

So why is the same vram page being used for both? For #1 the VRAM page is installed in a swap entry so it is has an elevated refcount.

The implication is that #2 is targeting already allocated VRAM memory that is NOT FREE.

Jason

#12 lore raw
Quoted 5 lines · Felix Kuehling

OK. We made an incorrect assumption that we can reuse a page if the driver isn't tracking it as allocated to any of our SVM ranges (i.e., after dev_pagemap_ops.migrate_to_ram() migrated all data out of the page). However, we neglected that other parts of the kernel can still hold references to a page even after that.

Yes, that sounds completely incorrect.

Quoted 4 lines

As I understand it, it's a race condition. The driver is done with the page and its migrate_to_ram() call has completed. But do_swap_page hasn't called put_page yet. At the same time, another thread is trying to reuse the page, migrating data back to VRAM.

Which means the driver is not properly tracking freed pages.

I don't think the code you showed makes alot of sense, if someone else has a reference on the page it could be for many reasons. If you take a non-free page and treat it as free and safe to use you probably are adding a security bug.

Jason