drm/amdkfd: Fix partial migrate issue
Subsystems (3)
PATCH v4 1/1 drm/amdkfd: Fix partial migrate issue
For partial migrate from ram to vram, the migrate->cpages is not equal to migrate->npages, should use migrate->npages to check all needed migrate pages which could be copied or not.
And only need to set those pages could be migrated to migrate->dst[i], or the migrate_vma_pages will migrate the wrong pages based on the migrate->dst[i].
v2:
Add mpages to break the loop earlier.
v3:
Uses MIGRATE_PFN_MIGRATE to identify whether page could be migrated.
--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +10 -7
@@ -278,10 +278,11 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, struct migrate_vma *migrate, struct dma_fence **mfence, dma_addr_t *scratch, uint64_t ttm_res_offset) { - uint64_t npages = migrate->cpages; + uint64_t npages = migrate->npages;struct amdgpu_device *adev = node->adev;struct device *dev = adev->dev;struct amdgpu_res_cursor cursor; +uint64_t mpages =0 ;dma_addr_t *src;uint64_t *dst;uint64_t i, j;
@@ -295,14 +296,16 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,amdgpu_res_first (prange->ttm_res, ttm_res_offset, npages <<PAGE_SHIFT , &cursor); -for (i = j =0 ; i < npages; i++) { +for (i = j =0 ; (i < npages) && (mpages < migrate->cpages); i++) {struct page *spage; - dst[i] = cursor.start + (j <<PAGE_SHIFT ); - migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); -svm_migrate_get_vram_page (prange, migrate->dst[i]); - migrate->dst[i] =migrate_pfn (migrate->dst[i]); - +if (migrate->src[i] &MIGRATE_PFN_MIGRATE ) { + dst[i] = cursor.start + (j <<PAGE_SHIFT ); + migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); +svm_migrate_get_vram_page (prange, migrate->dst[i]); + migrate->dst[i] =migrate_pfn (migrate->dst[i]); + mpages++; + } spage =migrate_pfn_to_page (migrate->src[i]);if (spage && !is_zone_device_page (spage)) { src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
Signature · 1 line
2.34.1
On 2025-01-10 02:49, Emily Deng wrote:
For partial migrate from ram to vram, the migrate->cpages is not equal to migrate->npages, should use migrate->npages to check all needed migrate pages which could be copied or not. And only need to set those pages could be migrated to migrate->dst[i], or the migrate_vma_pages will migrate the wrong pages based on the migrate->dst[i]. v2: Add mpages to break the loop earlier. v3: Uses MIGRATE_PFN_MIGRATE to identify whether page could be migrated. The error handling need below change, with that fixed, this patch is
if (r) {
pr_debug("failed %d to copy memory to vram\n", r); - while (i--) {
+
+ for (i = 0; i < npages && mpages; i++) { + if (!dst[i])
+ continue; svm_migrate_put_vram_page(adev, dst[i]); migrate->dst[i] = 0;
+ mpages--;
}
}
Quoted 70 lines · Philip Yang
Quoted 4 lines · Emily Deng
For partial migrate from ram to vram, the
migrate->cpagesis not equal tomigrate->npages, should usemigrate->npagesto check all needed migrate pages which could be copied or not.Quoted 8 lines
And only need to set those pages could be migrated to migrate->dst[i], or the
migrate_vma_pageswill migrate the wrong pages based on the migrate->dst[i].v2:
Add mpages to break the loop earlier.v3:
UsesMIGRATE_PFN_MIGRATEto identify whether page could be migrated.The error handling need below change, with that fixed, this patch is
Reviewed-by Philip Yang<[email protected]>Quoted 35 lines
Signed-off-by Emily Deng<[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +3
-7@@ -278,10 +278,11 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, struct migrate_vma *migrate, struct dma_fence **mfence, dma_addr_t *scratch, uint64_t ttm_res_offset) { - uint64_t npages = migrate->cpages; + uint64_t npages = migrate->npages;struct amdgpu_device *adev = node->adev;struct device *dev = adev->dev;struct amdgpu_res_cursor cursor; +uint64_t mpages =0 ;dma_addr_t *src;uint64_t *dst;uint64_t i, j;@@ -295,14 +296,16 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,amdgpu_res_first (prange->ttm_res, ttm_res_offset, npages <<PAGE_SHIFT , &cursor); -for (i = j =0 ; i < npages; i++) { +for (i = j =0 ; (i < npages) && (mpages < migrate->cpages); i++) {struct page *spage; - dst[i] = cursor.start + (j <<PAGE_SHIFT ); - migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); -svm_migrate_get_vram_page (prange, migrate->dst[i]); - migrate->dst[i] =migrate_pfn (migrate->dst[i]); -drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, · line 311
Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +7
-0+if (migrate->src[i] &MIGRATE_PFN_MIGRATE ) { + dst[i] = cursor.start + (j <<PAGE_SHIFT ); + migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); +svm_migrate_get_vram_page (prange, migrate->dst[i]); + migrate->dst[i] =migrate_pfn (migrate->dst[i]); + mpages++; + } spage =migrate_pfn_to_page (migrate->src[i]);if (spage && !is_zone_device_page (spage)) { src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,out_free_vram_pages:
if (r) {
pr_debug("failed %d to copy memory to vram\n", r); - while (i--) {
+
Quoted 8 lines
+ for (i = 0; i < npages && mpages; i++) { + if (!dst[i])
+ continue;svm_migrate_put_vram_page(adev, dst[i]); migrate->dst[i] = 0;
+ mpages--;
}
}
This error handing not need recover all vram pages as error happened at middle. Can use se do {....} while(i--);
Regards
Xiaogang
Philip Yang
Reviewed-byOn 2025-01-10 11:23, Chen, Xiaogang wrote:
On 1/10/2025 8:37 AM, Philip Yang wrote:
On 2025-01-10 02:49, Emily Deng wrote:
For partial migrate from ram to vram, the migrate->cpages is not equal to migrate->npages, should use migrate->npages to check all needed migrate pages which could be copied or not.
And only need to set those pages could be migrated to migrate->dst[i], or the migrate_vma_pages will migrate the wrong pages based on the migrate->dst[i].
v2:
Add mpages to break the loop earlier.
v3:
Uses MIGRATE_PFN_MIGRATE to identify whether page could be migrated.
The error handling need below change, with that fixed, this patch is
---
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +0 -0
@@ -278,10 +278,11 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
struct migrate_vma *migrate, struct dma_fence **mfence, dma_addr_t *scratch, uint64_t ttm_res_offset)
{
- uint64_t npages = migrate->cpages;
+ uint64_t npages = migrate->npages;
struct amdgpu_device *adev = node->adev;
struct device *dev = adev->dev;
struct amdgpu_res_cursor cursor;
+ uint64_t mpages = 0;
dma_addr_t *src;
uint64_t *dst;
uint64_t i, j;
@@ -295,14 +296,16 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, amdgpu_res_first(prange->ttm_res, ttm_res_offset,
npages << PAGE_SHIFT, &cursor);
- for (i = j = 0; i < npages; i++) {
+ for (i = j = 0; (i < npages) && (mpages < migrate->cpages); i++) {
struct page *spage;
- dst[i] = cursor.start + (j << PAGE_SHIFT);
- migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
- svm_migrate_get_vram_page(prange, migrate->dst[i]);
- migrate->dst[i] = migrate_pfn(migrate->dst[i]);
-
+ if (migrate->src[i] & MIGRATE_PFN_MIGRATE) {
+ dst[i] = cursor.start + (j << PAGE_SHIFT);
+ migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
+ svm_migrate_get_vram_page(prange, migrate->dst[i]);
+ migrate->dst[i] = migrate_pfn(migrate->dst[i]);
+ mpages++;
+ }
spage = migrate_pfn_to_page(migrate->src[i]);
if (spage && !is_zone_device_page(spage)) {
src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, out_free_vram_pages:
if (r) {
pr_debug("failed %d to copy memory to vram\n", r);
- while (i--) {
+
+ for (i = 0; i < npages && mpages; i++) {
+ if (!dst[i])
+ continue;
svm_migrate_put_vram_page(adev, dst[i]);
migrate->dst[i] = 0;
+ mpages--;
}
}
This error handing not need recover all vram pages as error happened at middle. Can use se do {....} while(i--); no, for example migrate npage=cpage=4, and outside for loop, svm_migrate_copy_memory_gart failed, dst[4] is out of range access.
Regards
Xiaogang
Quoted 85 lines · Philip Yang
Quoted 82 lines · Xiaogang
Quoted 79 lines · Philip Yang
Quoted 4 lines · Emily Deng
For partial migrate from ram to vram, the
migrate->cpagesis not equal tomigrate->npages, should usemigrate->npagesto check all needed migrate pages which could be copied or not.Quoted 8 lines
And only need to set those pages could be migrated to migrate->dst[i], or the
migrate_vma_pageswill migrate the wrong pages based on the migrate->dst[i].v2:
Add mpages to break the loop earlier.v3:
UsesMIGRATE_PFN_MIGRATEto identify whether page could be migrated.The error handling need below change, with that fixed, this patch is
Reviewed-by Philip Yang<[email protected]>Quoted 45 lines
Signed-off-by Emily Deng<[email protected]>---
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c| 17 ++++++++++-------1 file changed, 10 insertions(+), 7 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2
-1@@ -278,10 +278,11 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, struct migrate_vma *migrate, struct dma_fence **mfence, dma_addr_t *scratch, uint64_t ttm_res_offset) { - uint64_t npages = migrate->cpages; + uint64_t npages = migrate->npages; struct amdgpu_device *adev = node->adev; struct device *dev = adev->dev; struct amdgpu_res_cursor cursor; + uint64_t mpages = 0; dma_addr_t *src; uint64_t *dst;
uint64_ti, j;drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +8
-5@@ -295,14 +296,16 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, amdgpu_res_first(prange->ttm_res, ttm_res_offset, npages << PAGE_SHIFT, &cursor); - for (i = j = 0; i < npages; i++) { + for (i = j = 0; (i < npages) && (mpages < migrate->cpages); i++) { struct page *spage; - dst[i] = cursor.start + (j << PAGE_SHIFT); - migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); - svm_migrate_get_vram_page(prange, migrate->dst[i]); - migrate->dst[i] = migrate_pfn(migrate->dst[i]); - + if (migrate->src[i] & MIGRATE_PFN_MIGRATE) { + dst[i] = cursor.start + (j << PAGE_SHIFT); + migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); + svm_migrate_get_vram_page(prange, migrate->dst[i]); + migrate->dst[i] = migrate_pfn(migrate->dst[i]); + mpages++; + } spage = migrate_pfn_to_page(migrate->src[i]); if (spage && !is_zone_device_page(spage)) {Quoted 1 line
src[i] = dma_map_page(dev, spage, 0,
PAGE_SIZE,out_free_vram_pages:
if (r) {
pr_debug("failed %d to copy memory to vram\n", r); - while (i--) {
+
+ for (i = 0; i < npages && mpages; i++) { + if (!dst[i])
+ continue;svm_migrate_put_vram_page(adev, dst[i]); migrate->dst[i] = 0;
+ mpages--;
}
}
Quoted 2 lines
Quoted 1 line
This error handing not need recover all vram pages as error happened at middle. Can use se do {....} while(i--);
no, for example migrate npage=cpage=4, and outside for loop,
svm_migrate_copy_memory_gartfailed, dst[4] is out of range access.
You are right, but it is awkward. Inside loop we update dst[i] before do sdam that does not include the dst[i] just updated.
Regard
Xiaogang
Quoted 5 lines
Quoted 5 lines
Regards
Xiaogang
[AMD Official Use Only - AMD Internal Distribution Only]
Quoted 99 lines
-----Original Message-----
From: Chen, Xiaogang <[email protected]>
Sent: Saturday, January 11, 2025 2:13 AM
To: Yang, Philip <[email protected]>; Deng, Emily <[email protected]>; [email protected]
Subject: Re: [PATCH v4] drm/amdkfd: Fix partial migrate issueQuoted 90 lines · Philip Yang
Quoted 87 lines · Xiaogang
Quoted 84 lines · Philip Yang
Quoted 9 lines · Emily Deng
For partial migrate from ram to vram, the
migrate->cpagesis not equal tomigrate->npages, should usemigrate->npagesto check all needed migrate pages which could be copied or not.And only need to set those pages could be migrated to migrate->dst[i], or
themigrate_vma_pageswill migrate the wrong pages based on the migrate->dst[i].Quoted 5 lines
v2:
Add mpages to break the loop earlier.v3:
UsesMIGRATE_PFN_MIGRATEto identify whether page could be migrated.The error handling need below change, with that fixed, this patch is
Reviewed-by Philip Yang<[email protected]>Quoted 39 lines
Signed-off-by Emily Deng<[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2
-1@@ -278,10 +278,11 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, struct migrate_vma *migrate, struct dma_fence **mfence, dma_addr_t *scratch, uint64_t ttm_res_offset) { -uint64_t npages = migrate->cpages; +uint64_t npages = migrate->npages;struct amdgpu_device *adev = node->adev;struct device *dev = adev->dev;struct amdgpu_res_cursor cursor; +uint64_t mpages =0 ;dma_addr_t *src;uint64_t *dst; uint64_t i, j;drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2
-5@@ -295,14 +296,16 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,amdgpu_res_first (prange->ttm_res, ttm_res_offset, npages <<PAGE_SHIFT , &cursor); -for (i = j =0 ; i < npages; i++) { +for (i = j =0 ; (i < npages) && (mpages < migrate->cpages); i++) {struct page *spage; - dst[i] = cursor.start + (j <<PAGE_SHIFT ); - migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); -svm_migrate_get_vram_page (prange, migrate->dst[i]); - migrate->dst[i] =migrate_pfn (migrate->dst[i]); - +if (migrate->src[i] &MIGRATE_PFN_MIGRATE ) {drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_copy_to_vram(struct kfd_node
Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +7
-0+ dst[i] = cursor.start + (j <<PAGE_SHIFT ); + migrate->dst[i] =svm_migrate_addr_to_pfn (adev, +dst[i]); +svm_migrate_get_vram_page (prange, migrate->dst[i]); + migrate->dst[i] =migrate_pfn (migrate->dst[i]); + mpages++; + } spage =migrate_pfn_to_page (migrate->src[i]);if (spage && !is_zone_device_page (spage)) { src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
out_free_vram_pages:if (r) { pr_debug("failed %d to copy memory to vram\n", r);(quoted diff) +5
-1- while (i--) { + + for (i = 0; i < npages && mpages; i++) { + if (!dst[i]) + continue; svm_migrate_put_vram_page(adev, dst[i]); migrate->dst[i] = 0; + mpages--; } }
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
This error handing not need recover all vram pages as error happened at middle. Can use se do {....} while(i--);
no, for example migrate npage=cpage=4, and outside for loop,
svm_migrate_copy_memory_gartfailed, dst[4] is out of range access.You are right, but it is awkward. Inside loop we update dst[i] before do sdam that does not include the dst[i] just updated.
As the dst[i] is already initialized to NULL, so I think it doesn't matter if use follow:
while (i--) {
if (dst[i])
svm_migrate_put_vram_page(adev, dst[i]);
migrate->dst[i] = 0;
}Emily Deng
Best Wishes
Quoted 10 lines
Regard
Xiaogang
Quoted 5 lines
Quoted 5 lines
Regards
Xiaogang
Quoted 100 lines · Emily
[AMD Official Use Only - AMD Internal Distribution Only]
Quoted 98 lines
-----Original Message-----
From: Chen, Xiaogang <[email protected]>
Sent: Saturday, January 11, 2025 2:13 AM
To: Yang, Philip <[email protected]>; Deng, Emily <[email protected]>; [email protected]
Subject: Re: [PATCH v4] drm/amdkfd: Fix partial migrate issueQuoted 89 lines · Philip Yang
Quoted 83 lines · Xiaogang
Quoted 72 lines · Philip Yang
Quoted 9 lines · Emily Deng
For partial migrate from ram to vram, the
migrate->cpagesis not equal tomigrate->npages, should usemigrate->npagesto check all needed migrate pages which could be copied or not.And only need to set those pages could be migrated to migrate->dst[i], or
themigrate_vma_pageswill migrate the wrong pages based on the migrate->dst[i].Quoted 5 lines
v2:
Add mpages to break the loop earlier.v3:
UsesMIGRATE_PFN_MIGRATEto identify whether page could be migrated.The error handling need below change, with that fixed, this patch is
Reviewed-by Philip Yang<[email protected]>Quoted 39 lines
Signed-off-by Emily Deng<[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2
-1@@ -278,10 +278,11 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, struct migrate_vma *migrate, struct dma_fence **mfence, dma_addr_t *scratch, uint64_t ttm_res_offset) { -uint64_t npages = migrate->cpages; +uint64_t npages = migrate->npages;struct amdgpu_device *adev = node->adev;struct device *dev = adev->dev;struct amdgpu_res_cursor cursor; +uint64_t mpages =0 ;dma_addr_t *src;uint64_t *dst; uint64_t i, j;drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2
-5@@ -295,14 +296,16 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,amdgpu_res_first (prange->ttm_res, ttm_res_offset, npages <<PAGE_SHIFT , &cursor); -for (i = j =0 ; i < npages; i++) { +for (i = j =0 ; (i < npages) && (mpages < migrate->cpages); i++) {struct page *spage; - dst[i] = cursor.start + (j <<PAGE_SHIFT ); - migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); -svm_migrate_get_vram_page (prange, migrate->dst[i]); - migrate->dst[i] =migrate_pfn (migrate->dst[i]); - +if (migrate->src[i] &MIGRATE_PFN_MIGRATE ) {drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_copy_to_vram(struct kfd_node
Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +7
-0+ dst[i] = cursor.start + (j <<PAGE_SHIFT ); + migrate->dst[i] =svm_migrate_addr_to_pfn (adev, +dst[i]); +svm_migrate_get_vram_page (prange, migrate->dst[i]); + migrate->dst[i] =migrate_pfn (migrate->dst[i]); + mpages++; + } spage =migrate_pfn_to_page (migrate->src[i]);if (spage && !is_zone_device_page (spage)) { src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
out_free_vram_pages:if (r) { pr_debug("failed %d to copy memory to vram\n", r);Review
Quoted 10 lines
(quoted diff) +5
-1- while (i--) { + + for (i = 0; i < npages && mpages; i++) { + if (!dst[i]) + continue; svm_migrate_put_vram_page(adev, dst[i]); migrate->dst[i] = 0; + mpages--; } }Quoted 2 lines
This error handing not need recover all vram pages as error happened at middle. Can use se do {....} while(i--);
no, for example migrate npage=cpage=4, and outside for loop,
svm_migrate_copy_memory_gartfailed, dst[4] is out of range access.
Quoted 8 lines
Quoted 2 lines
You are right, but it is awkward. Inside loop we update dst[i] before do sdam that does not include the dst[i] just updated.
As the dst[i] is already initialized to NULL, so I think it doesn't matter if use follow:
while (i--) { if (dst[i]) svm_migrate_put_vram_page(adev, dst[i]); migrate->dst[i] = 0; }
The issue at error recovery path is due to we handled sys ram(src) and dst(vram) discontinuation in different way. When src got discontinuation we migrate j pages that page ith is not migrated. When dst(vram) got discontinuation we migrate j+1 pages that page ith is migrated. That cause error path has to iterate all pages to find which page got migrated before error happened. Also makes code more difficult to read.
Besides your change I think we can also change inside loop that handle src and dst discontinuation in consistent way.
Quoted 12 lines
Emily Deng
Best WishesQuoted 8 lines
Regard
Xiaogang
Quoted 4 lines
Quoted 4 lines
Regards
Xiaogang