Remove device private pages from physical address space
Subsystems (26)
v2 cover Remove device private pages from physical address space
Today, when creating these device private struct pages, the first step is to use request_free_mem_region() to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory using memremap_pages.
Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
This series changes device private memory so that it does not require allocation of physical address space and these problems are avoided. Instead of using the physical address space, we introduce a "device private address space" and allocate from there.
A consequence of placing the device private pages outside of the physical address space is that they no longer have a PFN. However, it is still necessary to be able to look up a corresponding device private page from a device private PTE entry, which means that we still require some way to index into this device private address space. Instead of a PFN, device private pages use an offset into this device private address space to look up device private struct pages.
The problem that then needs to be addressed is how to avoid confusing these device private offsets with PFNs. It is the inherent limited usage of the device private pages themselves which make this possible. A device private page is only used for userspace mappings, we do not need to be concerned with them being used within the mm more broadly. This means that the only way that the core kernel looks up these pages is via the page table, where their PTE already indicates if they refer to a device private page via their swap type, e.g. SWP_DEVICE_WRITE. We can use this information to determine if the PTE contains a PFN which should be looked up in the page map, or a device private offset which should be looked up elsewhere.
This applies when we are creating PTE entries for device private pages - because they have their own type there are already must be handled separately, so it is a small step to convert them to a device private PFN now too.
The first part of the series updates callers where device private offsets might now be encountered to track this extra state.
The last patch contains the bulk of the work where we change how we convert between device private pages to device private offsets and then use a new interface for allocating device private pages without the need for reserving physical address space.
By removing the device private pages from the physical address space, this series also opens up the possibility to moving away from tracking device private memory using struct pages in the future. This is desirable as on systems with large amounts of memory these device private struct pages use a signifiant amount of memory and take a significant amount of time to initialize.
*** Changes in v2 ***
The most significant change in v2 is addressing code paths that are common between MEMORY_DEVICE_PRIVATE and MEMORY_DEVICE_COHERENT devices.
This had been overlooked in previous revisions.
To do this we introduce a migrate_pfn_from_page() helper which will call device_private_offset_to_page() and set the MIGRATE_PFN_DEVICE_PRIVATE flag if required.
In places where we could have a device private offset (MEMORY_DEVICE_PRIVATE) or a pfn (MEMORY_DEVICE_COHERENT) we update to use an mpfn to disambiguate. This includes some users in the drivers and migrate_device_{pfns,range}().
Seeking opinions on using the mpfns like this or if a new type would be preferred.
- mm/migrate_device: Introduce migrate_pfn_from_page() helper
- New to series
- drm/amdkfd: Use migrate pfns internally
- New to series
- mm/migrate_device: Make migrate_device_{pfns,range}() take mpfns
- New to series
- mm/migrate_device: Add migrate PFN flag to track device private pages
- Update for migrate_pfn_from_page()
- Rename to MIGRATE_PFN_DEVICE_PRIVATE
- drm/amd: Check adev->gmc.xgmi.connected_to_cpu
- lib/test_hmm.c: Check chunk->pagemap.type == MEMORY_DEVICE_PRIVATE
- mm: Add helpers to create migration entries from struct pages
- Add a flags param
- mm: Add a new swap type for migration entries of device private pages
- Add softleaf_is_migration_device_private_read()
- mm: Add helpers to create device private entries from struct pages
- Add a flags param
- mm: Remove device private pages from the physical address space
- Make sure last member of struct dev_pagemap remains DECLARE_FLEX_ARRAY(struct range, ranges);
Testing:
- selftests/mm/hmm-tests on an amd64 VM
* NOTE: I will need help in testing the driver changes *
Revisions:
- RFC: https://lore.kernel.org/all/[email protected]/
- v1: https://lore.kernel.org/all/[email protected]/
[0] https://lore.kernel.org/lkml/CAMj1kXFZ=4hLL1w6iCV5O5uVoVLHAJbc0rr40j24ObenAjXe9w@mail.gmail.com/
Jordan Niethe (11):
mm/migrate_device: Introduce migrate_pfn_from_page() helper drm/amdkfd: Use migrate pfns internally
mm/migrate_device: Make migrate_device_{pfns,range}() take mpfns mm/migrate_device: Add migrate PFN flag to track device private pages mm/page_vma_mapped: Add flags to page_vma_mapped_walk::pfn to track
device private pages
mm: Add helpers to create migration entries from struct pages
mm: Add a new swap type for migration entries of device private pages
mm: Add helpers to create device private entries from struct pages mm/util: Add flag to track device private pages in page snapshots mm/hmm: Add flag to track device private pages
mm: Remove device private pages from the physical address space
Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 43 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 45 +++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 2 +- drivers/gpu/drm/drm_pagemap.c | 11 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 45 ++---- drivers/gpu/drm/xe/xe_svm.c | 37 ++--- fs/proc/page.c | 6 +- include/drm/drm_pagemap.h | 8 +- include/linux/hmm.h | 7 +- include/linux/leafops.h | 116 ++++++++++++-- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 23 ++- include/linux/mm.h | 9 +- include/linux/rmap.h | 33 +++- include/linux/swap.h | 8 +- include/linux/swapops.h | 136 ++++++++++++++++ lib/test_hmm.c | 86 ++++++---- mm/debug.c | 9 +- mm/hmm.c | 5 +- mm/huge_memory.c | 43 ++--- mm/hugetlb.c | 15 +- mm/memory.c | 5 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/migrate.c | 6 +- mm/migrate_device.c | 76 +++++---- mm/mm_init.c | 8 +- mm/mprotect.c | 10 +- mm/page_vma_mapped.c | 32 +++- mm/rmap.c | 59 ++++--- mm/util.c | 8 +- mm/vmscan.c | 2 +- 32 files changed, 822 insertions(+), 339 deletions(-)
base-commit: f8f9c1f4d0c7a64600e2ca312dec824a0bc2f1da
Signature · 1 line
2.34.1
PATCH v2 1/11 mm/migrate_device: Introduce migrate_pfn_from_page() helper
To create a migrate from a given struct page, that page is first converted to its pfn, before passing the pfn to migrate_pfn().
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn and must be handled separately.
Prepare for this with a new helper:
- migrate_pfn_from_page()
This helper takes a struct page as parameter instead of a pfn. This will allow more flexibility for handling the mpfn differently for device private pages.
---
v2: New to series
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 2 +- drivers/gpu/drm/drm_pagemap.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 4 ++-- include/linux/migrate.h | 5 +++++ lib/test_hmm.c | 11 ++++++----- mm/migrate_device.c | 7 +++---- 7 files changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
@@ -784,7 +784,7 @@ static int kvmppc_svm_page_in(struct vm_area_struct *vma, } } - *mig.dst =migrate_pfn (page_to_pfn (dpage)); + *mig.dst =migrate_pfn_from_page (dpage);migrate_vma_pages (&mig); out_finalize:migrate_vma_finalize (&mig);
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1 -1
@@ -646,7 +646,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,pr_debug_ratelimited ("dma mapping dst to 0x%llx, pfn 0x%lx\n" , dst[i] >>PAGE_SHIFT ,page_to_pfn (dpage)); - migrate->dst[i] =migrate_pfn (page_to_pfn (dpage)); + migrate->dst[i] =migrate_pfn_from_page (dpage); j++; }
drivers/gpu/drm/drm_pagemap.c +1 -1
@@ -490,7 +490,7 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas, goto free_pages; page =folio_page (folio,0 ); - mpfn[i] =migrate_pfn (page_to_pfn (page)); + mpfn[i] =migrate_pfn_from_page (page); next:if (page)
drivers/gpu/drm/nouveau/nouveau_dmem.c +2 -2
@@ -249,7 +249,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) goto done; } - args.dst[0 ] =migrate_pfn (page_to_pfn (dpage)); + args.dst[0 ] =migrate_pfn_from_page (dpage);if (order) args.dst[0 ] |=MIGRATE_PFN_COMPOUND ; dfolio =page_folio (dpage);
@@ -766,7 +766,7 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm, ((paddr >>PAGE_SHIFT ) <<NVIF_VMM_PFNMAP_V0_ADDR_SHIFT );if (src &MIGRATE_PFN_WRITE ) *pfn |=NVIF_VMM_PFNMAP_V0_W ; - mpfn =migrate_pfn (page_to_pfn (dpage)); + mpfn =migrate_pfn_from_page (dpage);if (folio_order (page_folio (dpage))) mpfn |=MIGRATE_PFN_COMPOUND ;return mpfn;
include/linux/migrate.h +5 -0
@@ -140,6 +140,11 @@ static inline unsigned long migrate_pfn(unsigned long pfn)return (pfn <<MIGRATE_PFN_SHIFT ) |MIGRATE_PFN_VALID ; } +static inline unsigned long migrate_pfn_from_page (struct page *page) +{ +return migrate_pfn (page_to_pfn (page)); +} +enum migrate_vma_direction {MIGRATE_VMA_SELECT_SYSTEM =1 <<0 ,MIGRATE_VMA_SELECT_DEVICE_PRIVATE =1 <<1 ,
lib/test_hmm.c +6 -5
@@ -727,7 +727,8 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, rpage =BACKING_PAGE (dpage); rpage->zone_device_data = dmirror; - *dst =migrate_pfn (page_to_pfn (dpage)) | write; + *dst =migrate_pfn_from_page (dpage) | + write; src_page =pfn_to_page (spfn + i);if (spage)
@@ -754,7 +755,7 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args,pr_debug ("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n" ,page_to_pfn (spage),page_to_pfn (dpage)); - *dst =migrate_pfn (page_to_pfn (dpage)) | write; + *dst =migrate_pfn_from_page (dpage) | write;if (is_large) {int i;
@@ -989,7 +990,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args,if (dpage) {lock_page (dpage); - *dst |=migrate_pfn (page_to_pfn (dpage)); + *dst |=migrate_pfn_from_page (dpage); }for (i =0 ; i < (1 << order); i++) {
@@ -1000,7 +1001,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args,if (!dpage && order) { dpage =alloc_page_vma (GFP_HIGHUSER_MOVABLE , args->vma, addr);lock_page (dpage); - dst[i] =migrate_pfn (page_to_pfn (dpage)); + dst[i] =migrate_pfn_from_page (dpage); dst_page =pfn_to_page (page_to_pfn (dpage)); dpage =NULL ;/* For the next iteration */ }else {
@@ -1412,7 +1413,7 @@ static void dmirror_device_evict_chunk(struct dmirror_chunk *chunk)/* TODO Support splitting here */ lock_page (dpage); - dst_pfns[i] =migrate_pfn (page_to_pfn (dpage)); + dst_pfns[i] =migrate_pfn_from_page (dpage);if (src_pfns[i] &MIGRATE_PFN_WRITE ) dst_pfns[i] |=MIGRATE_PFN_WRITE ;if (order)
mm/migrate_device.c +3 -4
@@ -207,9 +207,8 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, .vma = walk->vma, }; -unsigned long pfn =page_to_pfn (folio_page (folio,0 )); - - migrate->src[migrate->npages] =migrate_pfn (pfn) | write + migrate->src[migrate->npages] =migrate_pfn_from_page (folio_page (folio,0 )) + | write |MIGRATE_PFN_MIGRATE |MIGRATE_PFN_COMPOUND ; migrate->dst[migrate->npages++] =0 ;
@@ -328,7 +327,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, goto again; } - mpfn =migrate_pfn (page_to_pfn (page)) | + mpfn =migrate_pfn_from_page (page) |MIGRATE_PFN_MIGRATE ;if (softleaf_is_device_private_write (entry)) mpfn |=MIGRATE_PFN_WRITE ;
Signature · 1 line
2.34.1
Felix Kuehling
maintainerRe: [PATCH v2 01/11] mm/migrate_device: Introduce migrate_pfn_from_page() helper
Quoted 46 lines · Jordan Niethe
To create a migrate from a given struct page, that page is first converted to its pfn, before passing the pfn to
migrate_pfn().A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn and must be handled separately.
Prepare for this with a new helper:
-
migrate_pfn_from_page()This helper takes a struct page as parameter instead of a pfn. This will allow more flexibility for handling the mpfn differently for device private pages.
Signed-off-by Jordan Niethe <[email protected]>---
v2: New to series--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 2 +- drivers/gpu/drm/drm_pagemap.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 4 ++-- include/linux/migrate.h | 5 +++++ lib/test_hmm.c | 11 ++++++----- mm/migrate_device.c | 7 +++---- 7 files changed, 19 insertions(+), 14 deletions(-)arch/powerpc/kvm/book3s_hv_uvmem.c +1
-1@@ -784,7 +784,7 @@ static int kvmppc_svm_page_in(struct vm_area_struct *vma, } } - *mig.dst =migrate_pfn (page_to_pfn (dpage)); + *mig.dst =migrate_pfn_from_page (dpage);migrate_vma_pages (&mig); out_finalize:migrate_vma_finalize (&mig);drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +0
-0
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, · line 649
Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1
-1@@ -646,7 +646,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,pr_debug_ratelimited ("dma mapping dst to 0x%llx, pfn 0x%lx\n" , dst[i] >>PAGE_SHIFT ,page_to_pfn (dpage)); - migrate->dst[i] =migrate_pfn (page_to_pfn (dpage)); + migrate->dst[i] =migrate_pfn_from_page (dpage);
You missed another instance of this in svm_migrate_copy_to_vram.
Regards,
Felix
Quoted 129 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +0
-0j++; }drivers/gpu/drm/drm_pagemap.c +1
-1@@ -490,7 +490,7 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas, goto free_pages; page =folio_page (folio,0 ); - mpfn[i] =migrate_pfn (page_to_pfn (page)); + mpfn[i] =migrate_pfn_from_page (page); next:if (page)drivers/gpu/drm/nouveau/nouveau_dmem.c +2
-2@@ -249,7 +249,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) goto done; } - args.dst[0 ] =migrate_pfn (page_to_pfn (dpage)); + args.dst[0 ] =migrate_pfn_from_page (dpage);if (order) args.dst[0 ] |=MIGRATE_PFN_COMPOUND ; dfolio =page_folio (dpage);@@ -766,7 +766,7 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm, ((paddr >>PAGE_SHIFT ) <<NVIF_VMM_PFNMAP_V0_ADDR_SHIFT );if (src &MIGRATE_PFN_WRITE ) *pfn |=NVIF_VMM_PFNMAP_V0_W ; - mpfn =migrate_pfn (page_to_pfn (dpage)); + mpfn =migrate_pfn_from_page (dpage);if (folio_order (page_folio (dpage))) mpfn |=MIGRATE_PFN_COMPOUND ;return mpfn;include/linux/migrate.h +5
-0@@ -140,6 +140,11 @@ static inline unsigned long migrate_pfn(unsigned long pfn)return (pfn <<MIGRATE_PFN_SHIFT ) |MIGRATE_PFN_VALID ; } +static inline unsigned long migrate_pfn_from_page (struct page *page) +{ +return migrate_pfn (page_to_pfn (page)); +} +enum migrate_vma_direction {MIGRATE_VMA_SELECT_SYSTEM =1 <<0 ,MIGRATE_VMA_SELECT_DEVICE_PRIVATE =1 <<1 ,lib/test_hmm.c +6
-5@@ -727,7 +727,8 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, rpage =BACKING_PAGE (dpage); rpage->zone_device_data = dmirror; - *dst =migrate_pfn (page_to_pfn (dpage)) | write; + *dst =migrate_pfn_from_page (dpage) | + write; src_page =pfn_to_page (spfn + i);if (spage)@@ -754,7 +755,7 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args,pr_debug ("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n" ,page_to_pfn (spage),page_to_pfn (dpage)); - *dst =migrate_pfn (page_to_pfn (dpage)) | write; + *dst =migrate_pfn_from_page (dpage) | write;if (is_large) {int i;@@ -989,7 +990,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args,if (dpage) {lock_page (dpage); - *dst |=migrate_pfn (page_to_pfn (dpage)); + *dst |=migrate_pfn_from_page (dpage); }for (i =0 ; i < (1 << order); i++) {@@ -1000,7 +1001,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args,if (!dpage && order) { dpage =alloc_page_vma (GFP_HIGHUSER_MOVABLE , args->vma, addr);lock_page (dpage); - dst[i] =migrate_pfn (page_to_pfn (dpage)); + dst[i] =migrate_pfn_from_page (dpage); dst_page =pfn_to_page (page_to_pfn (dpage)); dpage =NULL ;/* For the next iteration */ }else {@@ -1412,7 +1413,7 @@ static void dmirror_device_evict_chunk(struct dmirror_chunk *chunk)/* TODO Support splitting here */ lock_page (dpage); - dst_pfns[i] =migrate_pfn (page_to_pfn (dpage)); + dst_pfns[i] =migrate_pfn_from_page (dpage);if (src_pfns[i] &MIGRATE_PFN_WRITE ) dst_pfns[i] |=MIGRATE_PFN_WRITE ;if (order)mm/migrate_device.c +3
-4@@ -207,9 +207,8 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, .vma = walk->vma, }; -unsigned long pfn =page_to_pfn (folio_page (folio,0 )); - - migrate->src[migrate->npages] =migrate_pfn (pfn) | write + migrate->src[migrate->npages] =migrate_pfn_from_page (folio_page (folio,0 )) + | write |MIGRATE_PFN_MIGRATE |MIGRATE_PFN_COMPOUND ; migrate->dst[migrate->npages++] =0 ;@@ -328,7 +327,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, goto again; } - mpfn =migrate_pfn (page_to_pfn (page)) | + mpfn =migrate_pfn_from_page (page) |MIGRATE_PFN_MIGRATE ;if (softleaf_is_device_private_write (entry)) mpfn |=MIGRATE_PFN_WRITE ;
Re: [PATCH v2 01/11] mm/migrate_device: Introduce migrate_pfn_from_page() helper
Hi,
Quoted 9 lines · Felix Kuehling
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_copy_to_ram(struct amdgpu_device · line 649
Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1
-1@@ -646,7 +646,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, pr_debug_ratelimited("dma mapping dst to 0x%llx, pfn 0x%lx\n", dst[i] >> PAGE_SHIFT, page_to_pfn(dpage)); - migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); + migrate->dst[i] = migrate_pfn_from_page(dpage);You missed another instance of this in
svm_migrate_copy_to_vram.
I might be missing something, but is there call to migrate_pfn() in svm_migrate_copy_to_vram()? I'm seeing svm_migrate_copy_to_vram() calls svm_migrate_addr_to_mpfn() - that should be handled already.
Thanks for reviewing,
Jordan.
Quoted 5 lines
Regards,
Felix
Kuehling, Felix
maintainerRe: [PATCH v2 01/11] mm/migrate_device: Introduce migrate_pfn_from_page() helper
Quoted 13 lines · Jordan Niethe
Hi,
Quoted 9 lines · Felix Kuehling
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_copy_to_ram(struct amdgpu_device · line 649
Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1
-1@@ -646,7 +646,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, pr_debug_ratelimited("dma mapping dst to 0x%llx, pfn 0x%lx\n", dst[i] >> PAGE_SHIFT, page_to_pfn(dpage)); - migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); + migrate->dst[i] = migrate_pfn_from_page(dpage);You missed another instance of this in
svm_migrate_copy_to_vram.
Quoted 2 lines
I might be missing something, but is there call to
migrate_pfn()insvm_migrate_copy_to_vram()?
I see what's happening. You removed that call in patch 2. Never mind.
Regards,
Felix
Quoted 11 lines
I'm seeing
svm_migrate_copy_to_vram()callssvm_migrate_addr_to_mpfn()- that should be handled already.Thanks for reviewing,
Jordan.Quoted 5 lines
Regards,
Felix
Re: [PATCH v2 01/11] mm/migrate_device: Introduce migrate_pfn_from_page() helper
In reply to Jordan NietheQuoted 172 lines · Jordan Niethe
To create a migrate from a given struct page, that page is first converted to its pfn, before passing the pfn to
migrate_pfn().A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn and must be handled separately.
Prepare for this with a new helper:
-
migrate_pfn_from_page()This helper takes a struct page as parameter instead of a pfn. This will allow more flexibility for handling the mpfn differently for device private pages.
Signed-off-by Jordan Niethe <[email protected]>---
v2: New to series--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 2 +- drivers/gpu/drm/drm_pagemap.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 4 ++-- include/linux/migrate.h | 5 +++++ lib/test_hmm.c | 11 ++++++----- mm/migrate_device.c | 7 +++---- 7 files changed, 19 insertions(+), 14 deletions(-)arch/powerpc/kvm/book3s_hv_uvmem.c +1
-1@@ -784,7 +784,7 @@ static int kvmppc_svm_page_in(struct vm_area_struct *vma, } } - *mig.dst =migrate_pfn (page_to_pfn (dpage)); + *mig.dst =migrate_pfn_from_page (dpage);migrate_vma_pages (&mig); out_finalize:migrate_vma_finalize (&mig);drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1
-1@@ -646,7 +646,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,pr_debug_ratelimited ("dma mapping dst to 0x%llx, pfn 0x%lx\n" , dst[i] >>PAGE_SHIFT ,page_to_pfn (dpage)); - migrate->dst[i] =migrate_pfn (page_to_pfn (dpage)); + migrate->dst[i] =migrate_pfn_from_page (dpage); j++; }drivers/gpu/drm/drm_pagemap.c +1
-1@@ -490,7 +490,7 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas, goto free_pages; page =folio_page (folio,0 ); - mpfn[i] =migrate_pfn (page_to_pfn (page)); + mpfn[i] =migrate_pfn_from_page (page); next:if (page)drivers/gpu/drm/nouveau/nouveau_dmem.c +2
-2@@ -249,7 +249,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) goto done; } - args.dst[0 ] =migrate_pfn (page_to_pfn (dpage)); + args.dst[0 ] =migrate_pfn_from_page (dpage);if (order) args.dst[0 ] |=MIGRATE_PFN_COMPOUND ; dfolio =page_folio (dpage);@@ -766,7 +766,7 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm, ((paddr >>PAGE_SHIFT ) <<NVIF_VMM_PFNMAP_V0_ADDR_SHIFT );if (src &MIGRATE_PFN_WRITE ) *pfn |=NVIF_VMM_PFNMAP_V0_W ; - mpfn =migrate_pfn (page_to_pfn (dpage)); + mpfn =migrate_pfn_from_page (dpage);if (folio_order (page_folio (dpage))) mpfn |=MIGRATE_PFN_COMPOUND ;return mpfn;include/linux/migrate.h +5
-0@@ -140,6 +140,11 @@ static inline unsigned long migrate_pfn(unsigned long pfn)return (pfn <<MIGRATE_PFN_SHIFT ) |MIGRATE_PFN_VALID ; } +static inline unsigned long migrate_pfn_from_page (struct page *page) +{ +return migrate_pfn (page_to_pfn (page)); +} +enum migrate_vma_direction {MIGRATE_VMA_SELECT_SYSTEM =1 <<0 ,MIGRATE_VMA_SELECT_DEVICE_PRIVATE =1 <<1 ,lib/test_hmm.c +6
-5@@ -727,7 +727,8 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, rpage =BACKING_PAGE (dpage); rpage->zone_device_data = dmirror; - *dst =migrate_pfn (page_to_pfn (dpage)) | write; + *dst =migrate_pfn_from_page (dpage) | + write; src_page =pfn_to_page (spfn + i);if (spage)@@ -754,7 +755,7 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args,pr_debug ("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n" ,page_to_pfn (spage),page_to_pfn (dpage)); - *dst =migrate_pfn (page_to_pfn (dpage)) | write; + *dst =migrate_pfn_from_page (dpage) | write;if (is_large) {int i;@@ -989,7 +990,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args,if (dpage) {lock_page (dpage); - *dst |=migrate_pfn (page_to_pfn (dpage)); + *dst |=migrate_pfn_from_page (dpage); }for (i =0 ; i < (1 << order); i++) {@@ -1000,7 +1001,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args,if (!dpage && order) { dpage =alloc_page_vma (GFP_HIGHUSER_MOVABLE , args->vma, addr);lock_page (dpage); - dst[i] =migrate_pfn (page_to_pfn (dpage)); + dst[i] =migrate_pfn_from_page (dpage); dst_page =pfn_to_page (page_to_pfn (dpage)); dpage =NULL ;/* For the next iteration */ }else {@@ -1412,7 +1413,7 @@ static void dmirror_device_evict_chunk(struct dmirror_chunk *chunk)/* TODO Support splitting here */ lock_page (dpage); - dst_pfns[i] =migrate_pfn (page_to_pfn (dpage)); + dst_pfns[i] =migrate_pfn_from_page (dpage);if (src_pfns[i] &MIGRATE_PFN_WRITE ) dst_pfns[i] |=MIGRATE_PFN_WRITE ;if (order)mm/migrate_device.c +2
-3@@ -207,9 +207,8 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, .vma = walk->vma, }; -unsigned long pfn =page_to_pfn (folio_page (folio,0 )); - - migrate->src[migrate->npages] =migrate_pfn (pfn) | write + migrate->src[migrate->npages] =migrate_pfn_from_page (folio_page (folio,0 )) + | write |MIGRATE_PFN_MIGRATE |MIGRATE_PFN_COMPOUND ; migrate->dst[migrate->npages++] =0 ;
mm/migrate_device.c static int migrate_vma_collect_pmd(pmd_t *pmdp, · line 333
Quoted 8 lines
mm/migrate_device.c +1
-1@@ -328,7 +327,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, goto again; } - mpfn =migrate_pfn (page_to_pfn (page)) | + mpfn =migrate_pfn_from_page (page) |MIGRATE_PFN_MIGRATE ;if (softleaf_is_device_private_write (entry)) mpfn |=MIGRATE_PFN_WRITE ;
Makes sense
PATCH v2 2/11 drm/amdkfd: Use migrate pfns internally
In reply to Jordan NietheA future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn.
A MIGRATE_PFN flag will be introduced that distinguishes between mpfns that contain a pfn vs an offset into device private memory.
Replace usages of pfns and page_to_pfn() to mpfns and migrate_pfn_to_page() to prepare for handling this distinction. This will assist in continuing to use the same code paths for both MEMORY_DEVICE_PRIVATE and MEMORY_DEVICE_COHERENT devices.
---
v2:
- New to series
--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 15 +++++++-------- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 2 +- 2 files changed, 8 insertions(+), 9 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +7 -8
@@ -204,17 +204,17 @@ svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence) }unsigned long -svm_migrate_addr_to_pfn (struct amdgpu_device *adev,unsigned long addr) +svm_migrate_addr_to_mpfn (struct amdgpu_device *adev,unsigned long addr) { -return (addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ; +return migrate_pfn ((addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ); }static void -svm_migrate_get_vram_page (struct svm_range *prange,unsigned long pfn) +svm_migrate_get_vram_page (struct svm_range *prange,unsigned long mpfn) {struct page *page; - page =pfn_to_page (pfn); + page =migrate_pfn_to_page (mpfn);svm_range_bo_ref (prange->svm_bo); page->zone_device_data = prange->svm_bo;zone_device_page_init (page,0 );
@@ -225,7 +225,7 @@ svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long addr) {struct page *page; - page =pfn_to_page (svm_migrate_addr_to_pfn (adev, addr)); + page =migrate_pfn_to_page (svm_migrate_addr_to_mpfn (adev, addr));unlock_page (page);put_page (page); }
@@ -235,7 +235,7 @@ svm_migrate_addr(struct amdgpu_device *adev, struct page *page) {unsigned long addr; - addr =page_to_pfn (page) <<PAGE_SHIFT ; + addr = (migrate_pfn_from_page (page) >>MIGRATE_PFN_SHIFT ) <<PAGE_SHIFT ;return (addr - adev->kfd.pgmap.range.start); }
@@ -301,9 +301,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,if (migrate->src[i] &MIGRATE_PFN_MIGRATE ) { dst[i] = cursor.start + (j <<PAGE_SHIFT ); - migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); + migrate->dst[i] =svm_migrate_addr_to_mpfn (adev, dst[i]);svm_migrate_get_vram_page (prange, migrate->dst[i]); - migrate->dst[i] =migrate_pfn (migrate->dst[i]); mpages++; } spage =migrate_pfn_to_page (migrate->src[i]);
drivers/gpu/drm/amd/amdkfd/kfd_migrate.h +1 -1
@@ -48,7 +48,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm, uint32_t trigger, struct page *fault_page);unsigned long -svm_migrate_addr_to_pfn (struct amdgpu_device *adev,unsigned long addr); +svm_migrate_addr_to_mpfn (struct amdgpu_device *adev,unsigned long addr);#endif /* IS_ENABLED(CONFIG_HSA_AMD_SVM) */
Signature · 1 line
2.34.1
Felix Kuehling
maintainerRe: [PATCH v2 02/11] drm/amdkfd: Use migrate pfns internally
Quoted 57 lines · Jordan Niethe
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn.
A
MIGRATE_PFNflag will be introduced that distinguishes between mpfns that contain a pfn vs an offset into device private memory.Replace usages of pfns and
page_to_pfn()to mpfns andmigrate_pfn_to_page()to prepare for handling this distinction. This will assist in continuing to use the same code paths for bothMEMORY_DEVICE_PRIVATEandMEMORY_DEVICE_COHERENTdevices.Signed-off-by Jordan Niethe <[email protected]>---
v2:
- New to series--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 15 +++++++-------- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 2 +- 2 files changed, 8 insertions(+), 9 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +5
-5@@ -204,17 +204,17 @@ svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence) }unsigned long -svm_migrate_addr_to_pfn (struct amdgpu_device *adev,unsigned long addr) +svm_migrate_addr_to_mpfn (struct amdgpu_device *adev,unsigned long addr) { -return (addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ; +return migrate_pfn ((addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ); }static void -svm_migrate_get_vram_page (struct svm_range *prange,unsigned long pfn) +svm_migrate_get_vram_page (struct svm_range *prange,unsigned long mpfn) {struct page *page; - page =pfn_to_page (pfn); + page =migrate_pfn_to_page (mpfn);svm_range_bo_ref (prange->svm_bo); page->zone_device_data = prange->svm_bo;zone_device_page_init (page,0 );@@ -225,7 +225,7 @@ svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long addr) {struct page *page; - page =pfn_to_page (svm_migrate_addr_to_pfn (adev, addr)); + page =migrate_pfn_to_page (svm_migrate_addr_to_mpfn (adev, addr));unlock_page (page);put_page (page); }
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_addr(struct amdgpu_device *adev, struct page *page) · line 239
Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1
-1@@ -235,7 +235,7 @@ svm_migrate_addr(struct amdgpu_device *adev, struct page *page) {unsigned long addr; - addr =page_to_pfn (page) <<PAGE_SHIFT ; + addr = (migrate_pfn_from_page (page) >>MIGRATE_PFN_SHIFT ) <<PAGE_SHIFT ;return (addr - adev->kfd.pgmap.range.start);
I guess we rely on the fact that for DEVICE_PRIVATE memory, adev->kfd.pgmap.range.start will be 0 after your patch 11. So we don't need a special condition here to handle DEVICE_PRIVATE differently.
In general, I like the way you handle mpfns as it keeps all the special casing out of the drivers.
Regards,
Felix
Quoted 26 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1
-2}@@ -301,9 +301,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,if (migrate->src[i] &MIGRATE_PFN_MIGRATE ) { dst[i] = cursor.start + (j <<PAGE_SHIFT ); - migrate->dst[i] =svm_migrate_addr_to_pfn (adev, dst[i]); + migrate->dst[i] =svm_migrate_addr_to_mpfn (adev, dst[i]);svm_migrate_get_vram_page (prange, migrate->dst[i]); - migrate->dst[i] =migrate_pfn (migrate->dst[i]); mpages++; } spage =migrate_pfn_to_page (migrate->src[i]);drivers/gpu/drm/amd/amdkfd/kfd_migrate.h +1
-1@@ -48,7 +48,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm, uint32_t trigger, struct page *fault_page);unsigned long -svm_migrate_addr_to_pfn (struct amdgpu_device *adev,unsigned long addr); +svm_migrate_addr_to_mpfn (struct amdgpu_device *adev,unsigned long addr);#endif /* IS_ENABLED(CONFIG_HSA_AMD_SVM) */
Re: [PATCH v2 02/11] drm/amdkfd: Use migrate pfns internally
Hi,
Quoted 11 lines · Felix Kuehling
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_addr(struct amdgpu_device *adev, · line 240
Quoted 9 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1
-1@@ -235,7 +235,7 @@ svm_migrate_addr(struct amdgpu_device *adev, struct page *page) { unsigned long addr; - addr = page_to_pfn(page) << PAGE_SHIFT; + addr = (migrate_pfn_from_page(page) >> MIGRATE_PFN_SHIFT) <<PAGE_SHIFT ; return (addr - adev->kfd.pgmap.range.start);I guess we rely on the fact that for
DEVICE_PRIVATEmemory, adev-
Quoted 2 lines
Quoted 1 line
kfd.pgmap.range.start will be 0 after your patch 11. So we don't need
a special condition here to handle
DEVICE_PRIVATEdifferently.
Actually pgmap.range.start won't be zero - part of the change to the memremap_device_private_pagemap() in patch 11 is that range is used as an output parameter. It returns the range we allocate for the pagemap from the device_private_pgmap_tree maple tree, representing "device private address space".
But it's correct that means we don't need special handling here.
Quoted 2 lines
In general, I like the way you handle mpfns as it keeps all the special casing out of the drivers.
Yeah, it does turn out quite neat.
Thanks for review,
Jordan.
Quoted 5 lines
Regards,
Felix
PATCH v2 3/11 mm/migrate_device: Make migrate_device_{pfns,range}() take mpfns
In reply to Jordan NietheA future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn.
This causes an issue for migrate_device_{pfns,range}() which take pfn parameters because depending on of the device is MEMORY_DEVICE_PRIVATE or MEMORY_DEVICE_COHERENT will effect how that parameter should be interpreted.
A MIGRATE_PFN flag will be introduced that distinguishes between mpfns that contain a pfn vs an offset into device private memory, we will take advantage of that here.
Update migrate_device_{pfns,range}() to take a mpfn instead of pfn.
Update the users of migrate_device_{pfns,range}() to pass in an mpfn.
To support this change, update the dpagemap_devmem_ops::populate_devmem_pfn() to instead return mpfns and rename accordingly.
---
v2: New to series
--- drivers/gpu/drm/drm_pagemap.c | 9 +++--- drivers/gpu/drm/nouveau/nouveau_dmem.c | 5 +-- drivers/gpu/drm/xe/xe_svm.c | 9 +++--- include/drm/drm_pagemap.h | 8 ++--- lib/test_hmm.c | 2 +- mm/migrate_device.c | 45 ++++++++++++++------------ 6 files changed, 41 insertions(+), 37 deletions(-)
drivers/gpu/drm/drm_pagemap.c +4 -5
@@ -337,7 +337,7 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,mmap_assert_locked (mm); -if (!ops->populate_devmem_pfn || !ops->copy_to_devmem || +if (!ops->populate_devmem_mpfn || !ops->copy_to_devmem || !ops->copy_to_ram)return -EOPNOTSUPP ;
@@ -390,7 +390,7 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation, goto err_finalize; } - err = ops->populate_devmem_pfn (devmem_allocation, npages, migrate.dst); + err = ops->populate_devmem_mpfn (devmem_allocation, npages, migrate.dst);if (err) goto err_finalize;
@@ -401,10 +401,9 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation, goto err_finalize;for (i =0 ; i < npages; ++i) { -struct page *page =pfn_to_page (migrate.dst[i]); +struct page *page =migrate_pfn_to_page (migrate.dst[i]); pages[i] = page; - migrate.dst[i] =migrate_pfn (migrate.dst[i]);drm_pagemap_get_devmem_page (page, zdd); }
@@ -575,7 +574,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation) pagemap_addr = buf + (2 *sizeof (*src) * npages); pages = buf + (2 *sizeof (*src) +sizeof (*pagemap_addr)) * npages; - err = ops->populate_devmem_pfn (devmem_allocation, npages, src); + err = ops->populate_devmem_mpfn (devmem_allocation, npages, src);if (err) goto err_free;
drivers/gpu/drm/nouveau/nouveau_dmem.c +3 -2
@@ -483,8 +483,9 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) dst_pfns =kvcalloc (npages,sizeof (*dst_pfns),GFP_KERNEL | __GFP_NOFAIL); dma_info =kvcalloc (npages,sizeof (*dma_info),GFP_KERNEL | __GFP_NOFAIL); -migrate_device_range (src_pfns, chunk->pagemap.range.start >>PAGE_SHIFT , - npages); +migrate_device_range (src_pfns, +migrate_pfn (chunk->pagemap.range.start >>PAGE_SHIFT ), + npages);for (i =0 ; i < npages; i++) {if (src_pfns[i] &MIGRATE_PFN_MIGRATE ) {
drivers/gpu/drm/xe/xe_svm.c +5 -4
@@ -5,6 +5,7 @@#include <drm/drm_drv.h> +#include <linux/migrate.h> #include "xe_bo.h" #include "xe_exec_queue_types.h" #include "xe_gt_stats.h"
@@ -681,8 +682,8 @@ static struct drm_buddy *vram_to_buddy(struct xe_vram_region *vram)return &vram->ttm.mm; } -static int xe_svm_populate_devmem_pfn (struct drm_pagemap_devmem *devmem_allocation, -unsigned long npages,unsigned long *pfn) +static int xe_svm_populate_devmem_mpfn (struct drm_pagemap_devmem *devmem_allocation, +unsigned long npages,unsigned long *pfn) {struct xe_bo *bo =to_xe_bo (devmem_allocation);struct ttm_resource *res = bo->ttm.resource;
@@ -697,7 +698,7 @@ static int xe_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocatiint i;for (i =0 ; i <drm_buddy_block_size (buddy, block) >>PAGE_SHIFT ; ++i) - pfn[j++] = block_pfn + i; + pfn[j++] =migrate_pfn (block_pfn + i); }return 0 ;
@@ -705,7 +706,7 @@ static int xe_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocatistatic const struct drm_pagemap_devmem_ops dpagemap_devmem_ops = { .devmem_release = xe_svm_devmem_release, - .populate_devmem_pfn = xe_svm_populate_devmem_pfn, + .populate_devmem_mpfn = xe_svm_populate_devmem_mpfn, .copy_to_devmem = xe_svm_copy_to_devmem, .copy_to_ram = xe_svm_copy_to_ram, };
include/drm/drm_pagemap.h +4 -4
@@ -157,17 +157,17 @@ struct drm_pagemap_devmem_ops {void (*devmem_release)(struct drm_pagemap_devmem *devmem_allocation);/** -* @populate_devmem_pfn: Populate device memory PFN (required for migration) +* @populate_devmem_mpfn: Populate device memory PFN (required for migration) * @devmem_allocation: device memory allocation * @npages: Number of pages to populate -* @pfn: Array of page frame numbers to populate +* @mpfn: Array of migrate page frame numbers to populate * * Populate device memory page frame numbers (PFN). * * Return: 0 on success, a negative error code on failure. */ -int (*populate_devmem_pfn)(struct drm_pagemap_devmem *devmem_allocation, -unsigned long npages,unsigned long *pfn); +int (*populate_devmem_mpfn)(struct drm_pagemap_devmem *devmem_allocation, +unsigned long npages,unsigned long *pfn);/** * @copy_to_devmem: Copy to device memory (required for migration)
lib/test_hmm.c +1 -1
@@ -1389,7 +1389,7 @@ static void dmirror_device_evict_chunk(struct dmirror_chunk *chunk) src_pfns =kvcalloc (npages,sizeof (*src_pfns),GFP_KERNEL | __GFP_NOFAIL); dst_pfns =kvcalloc (npages,sizeof (*dst_pfns),GFP_KERNEL | __GFP_NOFAIL); -migrate_device_range (src_pfns, start_pfn, npages); +migrate_device_range (src_pfns,migrate_pfn (start_pfn), npages);for (i =0 ; i < npages; i++) {struct page *dpage, *spage;
mm/migrate_device.c +24 -21
108 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v2 4/11 mm/migrate_device: Add migrate PFN flag to track device private pages
In reply to Jordan NietheA future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal PFN and must be handled separately.
Prepare for this by adding a MIGRATE_PFN_DEVICE_PRIVATE flag to indicate that a migrate pfn contains a PFN for a device private page.
---
v1:
- Update for HMM huge page support
- Update existing drivers to use MIGRATE_PFN_DEVICE
v2:
- Include changes to migrate_pfn_from_page()
- Rename to MIGRATE_PFN_DEVICE_PRIVATE
- drm/amd: Check adev->gmc.xgmi.connected_to_cpu
- lib/test_hmm.c: Check chunk->pagemap.type == MEMORY_DEVICE_PRIVATE
--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 7 ++++++- drivers/gpu/drm/nouveau/nouveau_dmem.c | 3 ++- drivers/gpu/drm/xe/xe_svm.c | 2 +- include/linux/migrate.h | 14 +++++++++----- lib/test_hmm.c | 6 +++++- 5 files changed, 23 insertions(+), 9 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +6 -1
@@ -206,7 +206,12 @@ svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence)unsigned long svm_migrate_addr_to_mpfn (struct amdgpu_device *adev,unsigned long addr) { -return migrate_pfn ((addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ); +unsigned long flags =0 ; + +if (!adev->gmc.xgmi.connected_to_cpu) + flags |=MIGRATE_PFN_DEVICE_PRIVATE ; +return migrate_pfn ((addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ) | + flags; } static void
drivers/gpu/drm/nouveau/nouveau_dmem.c +2 -1
@@ -484,7 +484,8 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) dma_info =kvcalloc (npages,sizeof (*dma_info),GFP_KERNEL | __GFP_NOFAIL);migrate_device_range (src_pfns, -migrate_pfn (chunk->pagemap.range.start >>PAGE_SHIFT ), +migrate_pfn (chunk->pagemap.range.start >>PAGE_SHIFT ) | +MIGRATE_PFN_DEVICE_PRIVATE , npages);for (i =0 ; i < npages; i++) {
drivers/gpu/drm/xe/xe_svm.c +1 -1
@@ -698,7 +698,7 @@ static int xe_svm_populate_devmem_mpfn(struct drm_pagemap_devmem *devmem_allocatint i;for (i =0 ; i <drm_buddy_block_size (buddy, block) >>PAGE_SHIFT ; ++i) - pfn[j++] =migrate_pfn (block_pfn + i); + pfn[j++] =migrate_pfn (block_pfn + i) |MIGRATE_PFN_DEVICE_PRIVATE ; }return 0 ;
include/linux/migrate.h +9 -5
@@ -122,11 +122,12 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)* have enough bits to store all physical address and flags. So far we have * enough room for all our flags. */ -#define MIGRATE_PFN_VALID (1UL << 0) -#define MIGRATE_PFN_MIGRATE (1UL << 1) -#define MIGRATE_PFN_WRITE (1UL << 3) -#define MIGRATE_PFN_COMPOUND (1UL << 4) -#define MIGRATE_PFN_SHIFT 6 +#define MIGRATE_PFN_VALID (1UL << 0) +#define MIGRATE_PFN_MIGRATE (1UL << 1) +#define MIGRATE_PFN_WRITE (1UL << 3) +#define MIGRATE_PFN_COMPOUND (1UL << 4) +#define MIGRATE_PFN_DEVICE_PRIVATE (1UL << 5) +#define MIGRATE_PFN_SHIFT 6static inline struct page *migrate_pfn_to_page (unsigned long mpfn) {
@@ -142,6 +143,9 @@ static inline unsigned long migrate_pfn(unsigned long pfn)static inline unsigned long migrate_pfn_from_page (struct page *page) { +if (is_device_private_page (page)) +return migrate_pfn (page_to_pfn (page)) | +MIGRATE_PFN_DEVICE_PRIVATE ;return migrate_pfn (page_to_pfn (page)); }
lib/test_hmm.c +5 -1
@@ -1385,11 +1385,15 @@ static void dmirror_device_evict_chunk(struct dmirror_chunk *chunk)unsigned long *src_pfns;unsigned long *dst_pfns;unsigned int order =0 ; +unsigned long flags =0 ; src_pfns =kvcalloc (npages,sizeof (*src_pfns),GFP_KERNEL | __GFP_NOFAIL); dst_pfns =kvcalloc (npages,sizeof (*dst_pfns),GFP_KERNEL | __GFP_NOFAIL); -migrate_device_range (src_pfns,migrate_pfn (start_pfn), npages); +if (chunk->pagemap.type ==MEMORY_DEVICE_PRIVATE ) + flags |=MIGRATE_PFN_DEVICE_PRIVATE ; + +migrate_device_range (src_pfns,migrate_pfn (start_pfn) | flags, npages);for (i =0 ; i < npages; i++) {struct page *dpage, *spage;
Signature · 1 line
2.34.1
Felix Kuehling
maintainerRe: [PATCH v2 04/11] mm/migrate_device: Add migrate PFN flag to track device private pages
Quoted 32 lines · Jordan Niethe
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal PFN and must be handled separately.
Prepare for this by adding a
MIGRATE_PFN_DEVICE_PRIVATEflag to indicate that a migrate pfn contains a PFN for a device private page.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
v1:
- Update for HMM huge page support
- Update existing drivers to useMIGRATE_PFN_DEVICE
v2:
- Include changes tomigrate_pfn_from_page()
- Rename toMIGRATE_PFN_DEVICE_PRIVATE
- drm/amd: Checkadev->gmc.xgmi.connected_to_cpu
-lib/test_hmm.c: Checkchunk->pagemap.type==MEMORY_DEVICE_PRIVATE--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 7 ++++++- drivers/gpu/drm/nouveau/nouveau_dmem.c | 3 ++- drivers/gpu/drm/xe/xe_svm.c | 2 +- include/linux/migrate.h | 14 +++++++++----- lib/test_hmm.c | 6 +++++- 5 files changed, 23 insertions(+), 9 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +0
-0
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence) · line 211
Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +3
-1@@ -206,7 +206,12 @@ svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence)unsigned long svm_migrate_addr_to_mpfn (struct amdgpu_device *adev,unsigned long addr) { -return migrate_pfn ((addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ); +unsigned long flags =0 ; + +if (!adev->gmc.xgmi.connected_to_cpu)
We could probably use adev->kfd.pgmap.type == MEMORY_DEVICE_PRIVATE here. This avoids making any assumptions about how KFD decides device page type it wants to use, which may change on future HW generations.
Other than that, this looks good to me.
Thanks,
Felix
Quoted 86 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +3
-0+ flags |=MIGRATE_PFN_DEVICE_PRIVATE ; +return migrate_pfn ((addr + adev->kfd.pgmap.range.start) >>PAGE_SHIFT ) | + flags; } static voiddrivers/gpu/drm/nouveau/nouveau_dmem.c +2
-1@@ -484,7 +484,8 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) dma_info =kvcalloc (npages,sizeof (*dma_info),GFP_KERNEL | __GFP_NOFAIL);migrate_device_range (src_pfns, -migrate_pfn (chunk->pagemap.range.start >>PAGE_SHIFT ), +migrate_pfn (chunk->pagemap.range.start >>PAGE_SHIFT ) | +MIGRATE_PFN_DEVICE_PRIVATE , npages);for (i =0 ; i < npages; i++) {drivers/gpu/drm/xe/xe_svm.c +1
-1@@ -698,7 +698,7 @@ static int xe_svm_populate_devmem_mpfn(struct drm_pagemap_devmem *devmem_allocatint i;for (i =0 ; i <drm_buddy_block_size (buddy, block) >>PAGE_SHIFT ; ++i) - pfn[j++] =migrate_pfn (block_pfn + i); + pfn[j++] =migrate_pfn (block_pfn + i) |MIGRATE_PFN_DEVICE_PRIVATE ; }return 0 ;include/linux/migrate.h +9
-5@@ -122,11 +122,12 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)* have enough bits to store all physical address and flags. So far we have * enough room for all our flags. */ -#define MIGRATE_PFN_VALID (1UL << 0) -#define MIGRATE_PFN_MIGRATE (1UL << 1) -#define MIGRATE_PFN_WRITE (1UL << 3) -#define MIGRATE_PFN_COMPOUND (1UL << 4) -#define MIGRATE_PFN_SHIFT 6 +#define MIGRATE_PFN_VALID (1UL << 0) +#define MIGRATE_PFN_MIGRATE (1UL << 1) +#define MIGRATE_PFN_WRITE (1UL << 3) +#define MIGRATE_PFN_COMPOUND (1UL << 4) +#define MIGRATE_PFN_DEVICE_PRIVATE (1UL << 5) +#define MIGRATE_PFN_SHIFT 6static inline struct page *migrate_pfn_to_page (unsigned long mpfn) {@@ -142,6 +143,9 @@ static inline unsigned long migrate_pfn(unsigned long pfn)static inline unsigned long migrate_pfn_from_page (struct page *page) { +if (is_device_private_page (page)) +return migrate_pfn (page_to_pfn (page)) | +MIGRATE_PFN_DEVICE_PRIVATE ;return migrate_pfn (page_to_pfn (page)); }lib/test_hmm.c +5
-1@@ -1385,11 +1385,15 @@ static void dmirror_device_evict_chunk(struct dmirror_chunk *chunk)unsigned long *src_pfns;unsigned long *dst_pfns;unsigned int order =0 ; +unsigned long flags =0 ; src_pfns =kvcalloc (npages,sizeof (*src_pfns),GFP_KERNEL | __GFP_NOFAIL); dst_pfns =kvcalloc (npages,sizeof (*dst_pfns),GFP_KERNEL | __GFP_NOFAIL); -migrate_device_range (src_pfns,migrate_pfn (start_pfn), npages); +if (chunk->pagemap.type ==MEMORY_DEVICE_PRIVATE ) + flags |=MIGRATE_PFN_DEVICE_PRIVATE ; + +migrate_device_range (src_pfns,migrate_pfn (start_pfn) | flags, npages);for (i =0 ; i < npages; i++) {struct page *dpage, *spage;
Re: [PATCH v2 04/11] mm/migrate_device: Add migrate PFN flag to track device private pages
Hi,
Quoted 18 lines · Felix Kuehling
Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +0
-0@@ -206,7 +206,12 @@ svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence)Quoted 8 lines
unsigned long
svm_migrate_addr_to_mpfn(struct amdgpu_device *adev, unsigned long addr){
- returnmigrate_pfn((addr + adev->kfd.pgmap.range.start) >> PAGE_SHIFT);
+ unsigned long flags = 0;
+
+ if (!adev->gmc.xgmi.connected_to_cpu)We could probably use
adev->kfd.pgmap.type==MEMORY_DEVICE_PRIVATEhere. This avoids making any assumptions about how KFD decides device page type it wants to use, which may change on future HW generations.
Quoted 1 line
Other than that, this looks good to me.
That's a good point - I'll update.
Thanks for review,
Jordan.
Quoted 5 lines
Thanks,
Felix
PATCH v2 5/11 mm/page_vma_mapped: Add flags to page_vma_mapped_walk::pfn to track device private pages
In reply to Jordan NietheA future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal PFN and must be handled separately.
Prepare for this by modifying page_vma_mapped_walk::pfn to contain flags as well as a PFN. Introduce a PVMW_PFN_DEVICE_PRIVATE flag to indicate that a page_vma_mapped_walk::pfn contains a PFN for a device private page.
---
v1:
- Update for HMM huge page support
v2:
- Move adding device_private param to check_pmd() until final patch
--- include/linux/rmap.h | 30 +++++++++++++++++++++++++++++- mm/page_vma_mapped.c | 13 +++++++------ mm/rmap.c | 4 ++-- mm/vmscan.c | 2 +- 4 files changed, 39 insertions(+), 10 deletions(-)
include/linux/rmap.h +29 -1
@@ -939,9 +939,37 @@ struct page_vma_mapped_walk {unsigned int flags; }; +/* pfn is a device private offset */ +#define PVMW_PFN_DEVICE_PRIVATE (1UL << 0) +#define PVMW_PFN_SHIFT 1 + +static inline unsigned long page_vma_walk_pfn (unsigned long pfn) +{ +return (pfn <<PVMW_PFN_SHIFT ); +} + +static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) +{ +if (folio_is_device_private (folio)) +return page_vma_walk_pfn (folio_pfn (folio)) | +PVMW_PFN_DEVICE_PRIVATE ; + +return page_vma_walk_pfn (folio_pfn (folio)); +} + +static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) +{ +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +} + +static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +{ +return page_folio (page_vma_walk_pfn_to_page (pvmw_pfn)); +} +#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \ struct page_vma_mapped_walk name = { \ - .pfn = folio_pfn(_folio), \ + .pfn = folio_page_vma_walk_pfn(_folio), \ .nr_pages = folio_nr_pages(_folio), \ .pgoff = folio_pgoff(_folio), \ .vma = _vma, \
mm/page_vma_mapped.c +7 -6
@@ -129,9 +129,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) pfn =softleaf_to_pfn (entry); } -if ((pfn + pte_nr -1 ) < pvmw->pfn) +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false; -if (pfn > (pvmw->pfn + pvmw->nr_pages -1 )) +if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))return false;return true; }
@@ -139,9 +139,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)/* Returns true if the two ranges overlap. Careful to not overflow. */ static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) { -if ((pfn +HPAGE_PMD_NR -1 ) < pvmw->pfn) +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false; -if (pfn > pvmw->pfn + pvmw->nr_pages -1 ) +if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )return false;return true; }
@@ -254,7 +254,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) entry =softleaf_from_pmd (pmde);if (!softleaf_is_migration (entry) || - !check_pmd (softleaf_to_pfn (entry), pvmw)) + !check_pmd (softleaf_to_pfn (entry), + pvmw))return not_found (pvmw);return true; }
@@ -350,7 +351,7 @@ unsigned long page_mapped_in_vma(const struct page *page, {const struct folio *folio =page_folio (page);struct page_vma_mapped_walk pvmw = { - .pfn =page_to_pfn (page), + .pfn =folio_page_vma_walk_pfn (folio), .nr_pages =1 , .vma = vma, .flags =PVMW_SYNC ,
mm/rmap.c +2 -2
@@ -1112,7 +1112,7 @@ static bool mapping_wrprotect_range_one(struct folio *folio, {struct wrprotect_file_state *state = (struct wrprotect_file_state *)arg;struct page_vma_mapped_walk pvmw = { - .pfn = state->pfn, + .pfn =page_vma_walk_pfn (state->pfn), .nr_pages = state->nr_pages, .pgoff = state->pgoff, .vma = vma,
@@ -1190,7 +1190,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, struct vm_area_struct *vma) {struct page_vma_mapped_walk pvmw = { - .pfn = pfn, + .pfn =page_vma_walk_pfn (pfn), .nr_pages = nr_pages, .pgoff = pgoff, .vma = vma,
mm/vmscan.c +1 -1
@@ -4203,7 +4203,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)pte_t *pte = pvmw->pte;unsigned long addr = pvmw->address;struct vm_area_struct *vma = pvmw->vma; -struct folio *folio =pfn_folio (pvmw->pfn); +struct folio *folio =page_vma_walk_pfn_to_folio (pvmw->pfn);struct mem_cgroup *memcg =folio_memcg (folio);struct pglist_data *pgdat =folio_pgdat (folio);struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat);
Signature · 1 line
2.34.1
Zi Yan
reviewerRe: [PATCH v2 05/11] mm/page_vma_mapped: Add flags to page_vma_mapped_walk::pfn to track device private pages
Quoted 71 lines · Jordan Niethe
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal PFN and must be handled separately.
Prepare for this by modifying
page_vma_mapped_walk::pfnto contain flags as well as a PFN. Introduce aPVMW_PFN_DEVICE_PRIVATEflag to indicate that apage_vma_mapped_walk::pfncontains a PFN for a device private page.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
v1:
- Update for HMM huge page support
v2:
- Move addingdevice_privateparam tocheck_pmd()until final patch--- include/linux/rmap.h | 30 +++++++++++++++++++++++++++++- mm/page_vma_mapped.c | 13 +++++++------ mm/rmap.c | 4 ++-- mm/vmscan.c | 2 +- 4 files changed, 39 insertions(+), 10 deletions(-)include/linux/rmap.h +29
-1@@ -939,9 +939,37 @@ struct page_vma_mapped_walk {unsigned int flags; }; +/* pfn is a device private offset */ +#define PVMW_PFN_DEVICE_PRIVATE (1UL << 0) +#define PVMW_PFN_SHIFT 1 + +static inline unsigned long page_vma_walk_pfn (unsigned long pfn) +{ +return (pfn <<PVMW_PFN_SHIFT ); +} + +static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) +{ +if (folio_is_device_private (folio)) +return page_vma_walk_pfn (folio_pfn (folio)) | +PVMW_PFN_DEVICE_PRIVATE ; + +return page_vma_walk_pfn (folio_pfn (folio)); +} + +static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) +{ +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +} + +static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +{ +return page_folio (page_vma_walk_pfn_to_page (pvmw_pfn)); +} +#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \ struct page_vma_mapped_walk name = { \ - .pfn = folio_pfn(_folio), \ + .pfn = folio_page_vma_walk_pfn(_folio), \ .nr_pages = folio_nr_pages(_folio), \ .pgoff = folio_pgoff(_folio), \ .vma = _vma, \mm/page_vma_mapped.c +0
-0
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 132
Quoted 5 lines
mm/page_vma_mapped.c +1
-1@@ -129,9 +129,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) pfn =softleaf_to_pfn (entry); } -if ((pfn + pte_nr -1 ) < pvmw->pfn) +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))
Can you add a helper function for (pvmw->pfn >> PVMW_PFN_SHIFT)? It is impossible to tell why pfn does not need >> PVMW_PFN_SHIFT.
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 134
Quoted 3 lines
mm/page_vma_mapped.c +1
-1return false; -if (pfn > (pvmw->pfn + pvmw->nr_pages -1 )) +if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))
Ditto.
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 142
Quoted 9 lines
mm/page_vma_mapped.c +1
-1return false;return true; }@@ -139,9 +139,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)/* Returns true if the two ranges overlap. Careful to not overflow. */ static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) { -if ((pfn +HPAGE_PMD_NR -1 ) < pvmw->pfn) +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))
Ditto.
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 144
Quoted 3 lines
mm/page_vma_mapped.c +1
-1return false; -if (pfn > pvmw->pfn + pvmw->nr_pages -1 ) +if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )
Ditto.
Quoted 59 lines
mm/page_vma_mapped.c +3
-2return false;return true; }@@ -254,7 +254,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) entry =softleaf_from_pmd (pmde);if (!softleaf_is_migration (entry) || - !check_pmd (softleaf_to_pfn (entry), pvmw)) + !check_pmd (softleaf_to_pfn (entry), + pvmw))return not_found (pvmw);return true; }@@ -350,7 +351,7 @@ unsigned long page_mapped_in_vma(const struct page *page, {const struct folio *folio =page_folio (page);struct page_vma_mapped_walk pvmw = { - .pfn =page_to_pfn (page), + .pfn =folio_page_vma_walk_pfn (folio), .nr_pages =1 , .vma = vma, .flags =PVMW_SYNC ,mm/rmap.c +2
-2@@ -1112,7 +1112,7 @@ static bool mapping_wrprotect_range_one(struct folio *folio, {struct wrprotect_file_state *state = (struct wrprotect_file_state *)arg;struct page_vma_mapped_walk pvmw = { - .pfn = state->pfn, + .pfn =page_vma_walk_pfn (state->pfn), .nr_pages = state->nr_pages, .pgoff = state->pgoff, .vma = vma,@@ -1190,7 +1190,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, struct vm_area_struct *vma) {struct page_vma_mapped_walk pvmw = { - .pfn = pfn, + .pfn =page_vma_walk_pfn (pfn), .nr_pages = nr_pages, .pgoff = pgoff, .vma = vma,mm/vmscan.c +1
-1@@ -4203,7 +4203,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)pte_t *pte = pvmw->pte;unsigned long addr = pvmw->address;struct vm_area_struct *vma = pvmw->vma; -struct folio *folio =pfn_folio (pvmw->pfn); +struct folio *folio =page_vma_walk_pfn_to_folio (pvmw->pfn);struct mem_cgroup *memcg =folio_memcg (folio);struct pglist_data *pgdat =folio_pgdat (folio);struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat);Signature · 1 line
2.34.1
Best Regards,
Yan, Zi
Re: [PATCH v2 05/11] mm/page_vma_mapped: Add flags to page_vma_mapped_walk::pfn to track device private pages
Hi,
Quoted 76 lines · Zi Yan
Quoted 74 lines · Jordan Niethe
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal PFN and must be handled separately.
Prepare for this by modifying
page_vma_mapped_walk::pfnto contain flags as well as a PFN. Introduce aPVMW_PFN_DEVICE_PRIVATEflag to indicate that apage_vma_mapped_walk::pfncontains a PFN for a device private page.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
v1:
- Update for HMM huge page support
v2:
- Move addingdevice_privateparam tocheck_pmd()until final patch--- include/linux/rmap.h | 30 +++++++++++++++++++++++++++++- mm/page_vma_mapped.c | 13 +++++++------ mm/rmap.c | 4 ++-- mm/vmscan.c | 2 +- 4 files changed, 39 insertions(+), 10 deletions(-)include/linux/rmap.h +29
-1@@ -939,9 +939,37 @@ struct page_vma_mapped_walk {unsigned int flags; }; +/* pfn is a device private offset */ +#define PVMW_PFN_DEVICE_PRIVATE (1UL << 0) +#define PVMW_PFN_SHIFT 1 + +static inline unsigned long page_vma_walk_pfn (unsigned long pfn) +{ +return (pfn <<PVMW_PFN_SHIFT ); +} + +static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) +{ +if (folio_is_device_private (folio)) +return page_vma_walk_pfn (folio_pfn (folio)) | +PVMW_PFN_DEVICE_PRIVATE ; + +return page_vma_walk_pfn (folio_pfn (folio)); +} + +static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) +{ +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +} + +static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +{ +return page_folio (page_vma_walk_pfn_to_page (pvmw_pfn)); +} +#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \ struct page_vma_mapped_walk name = { \ - .pfn = folio_pfn(_folio), \ + .pfn = folio_page_vma_walk_pfn(_folio), \ .nr_pages = folio_nr_pages(_folio), \ .pgoff = folio_pgoff(_folio), \ .vma = _vma, \mm/page_vma_mapped.c +0
-0@@ -129,9 +129,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) pfn =softleaf_to_pfn (entry); }
Quoted 5 lines
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 132
Quoted 2 lines
mm/page_vma_mapped.c +1
-1-if ((pfn + pte_nr -1 ) < pvmw->pfn) +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))Can you add a helper function for (
pvmw->pfn>>PVMW_PFN_SHIFT)? It is impossible to tell why pfn does not need >>PVMW_PFN_SHIFT.
Sure, something like page_vma_walk_pfn_to_offset()?
Thanks,
Jordan.
Quoted 86 lines
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 134
Quoted 4 lines
mm/page_vma_mapped.c +1
-1return false; -if (pfn > (pvmw->pfn + pvmw->nr_pages -1 )) +if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))Ditto.
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 142
Quoted 9 lines
mm/page_vma_mapped.c +1
-1return false;return true; }@@ -139,9 +139,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)/* Returns true if the two ranges overlap. Careful to not overflow. */ static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) { -if ((pfn +HPAGE_PMD_NR -1 ) < pvmw->pfn) +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))Ditto.
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 144
Quoted 3 lines
mm/page_vma_mapped.c +1
-1return false; -if (pfn > pvmw->pfn + pvmw->nr_pages -1 ) +if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )Ditto.
Quoted 59 lines
mm/page_vma_mapped.c +3
-2return false;return true; }@@ -254,7 +254,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) entry =softleaf_from_pmd (pmde);if (!softleaf_is_migration (entry) || - !check_pmd (softleaf_to_pfn (entry), pvmw)) + !check_pmd (softleaf_to_pfn (entry), + pvmw))return not_found (pvmw);return true; }@@ -350,7 +351,7 @@ unsigned long page_mapped_in_vma(const struct page *page, {const struct folio *folio =page_folio (page);struct page_vma_mapped_walk pvmw = { - .pfn =page_to_pfn (page), + .pfn =folio_page_vma_walk_pfn (folio), .nr_pages =1 , .vma = vma, .flags =PVMW_SYNC ,mm/rmap.c +2
-2@@ -1112,7 +1112,7 @@ static bool mapping_wrprotect_range_one(struct folio *folio, {struct wrprotect_file_state *state = (struct wrprotect_file_state *)arg;struct page_vma_mapped_walk pvmw = { - .pfn = state->pfn, + .pfn =page_vma_walk_pfn (state->pfn), .nr_pages = state->nr_pages, .pgoff = state->pgoff, .vma = vma,@@ -1190,7 +1190,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, struct vm_area_struct *vma) {struct page_vma_mapped_walk pvmw = { - .pfn = pfn, + .pfn =page_vma_walk_pfn (pfn), .nr_pages = nr_pages, .pgoff = pgoff, .vma = vma,mm/vmscan.c +1
-1@@ -4203,7 +4203,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)pte_t *pte = pvmw->pte;unsigned long addr = pvmw->address;struct vm_area_struct *vma = pvmw->vma; -struct folio *folio =pfn_folio (pvmw->pfn); +struct folio *folio =page_vma_walk_pfn_to_folio (pvmw->pfn);struct mem_cgroup *memcg =folio_memcg (folio);struct pglist_data *pgdat =folio_pgdat (folio);struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat);Signature · 1 line
2.34.1Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH v2 05/11] mm/page_vma_mapped: Add flags to page_vma_mapped_walk::pfn to track device private pages
Quoted 79 lines · Jordan Niethe
Hi,
Quoted 76 lines · Zi Yan
Quoted 74 lines · Jordan Niethe
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal PFN and must be handled separately.
Prepare for this by modifying
page_vma_mapped_walk::pfnto contain flags as well as a PFN. Introduce aPVMW_PFN_DEVICE_PRIVATEflag to indicate that apage_vma_mapped_walk::pfncontains a PFN for a device private page.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
v1:
- Update for HMM huge page support
v2:
- Move addingdevice_privateparam tocheck_pmd()until final patch--- include/linux/rmap.h | 30 +++++++++++++++++++++++++++++- mm/page_vma_mapped.c | 13 +++++++------ mm/rmap.c | 4 ++-- mm/vmscan.c | 2 +- 4 files changed, 39 insertions(+), 10 deletions(-)include/linux/rmap.h +29
-1@@ -939,9 +939,37 @@ struct page_vma_mapped_walk {unsigned int flags; }; +/* pfn is a device private offset */ +#define PVMW_PFN_DEVICE_PRIVATE (1UL << 0) +#define PVMW_PFN_SHIFT 1 + +static inline unsigned long page_vma_walk_pfn (unsigned long pfn) +{ +return (pfn <<PVMW_PFN_SHIFT ); +} + +static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) +{ +if (folio_is_device_private (folio)) +return page_vma_walk_pfn (folio_pfn (folio)) | +PVMW_PFN_DEVICE_PRIVATE ; + +return page_vma_walk_pfn (folio_pfn (folio)); +} + +static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) +{ +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +} + +static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +{ +return page_folio (page_vma_walk_pfn_to_page (pvmw_pfn)); +} +#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \ struct page_vma_mapped_walk name = { \ - .pfn = folio_pfn(_folio), \ + .pfn = folio_page_vma_walk_pfn(_folio), \ .nr_pages = folio_nr_pages(_folio), \ .pgoff = folio_pgoff(_folio), \ .vma = _vma, \mm/page_vma_mapped.c +0
-0@@ -129,9 +129,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) pfn =softleaf_to_pfn (entry); }
Quoted 8 lines
Quoted 5 lines
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 132
Quoted 2 lines
mm/page_vma_mapped.c +1
-1-if ((pfn + pte_nr -1 ) < pvmw->pfn) +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))Can you add a helper function for (
pvmw->pfn>>PVMW_PFN_SHIFT)? It is impossible to tell why pfn does not need >>PVMW_PFN_SHIFT.Sure, something like
page_vma_walk_pfn_to_offset()?
Just page_vma_walk_pfn(pvmw)? Since the code is comparing with pfn.
Best Regards,
Yan, Zi
Re: [PATCH v2 05/11] mm/page_vma_mapped: Add flags to page_vma_mapped_walk::pfn to track device private pages
Hi,
Quoted 87 lines · Zi Yan
Quoted 85 lines · Jordan Niethe
Hi,
Quoted 81 lines · Zi Yan
Quoted 71 lines · Jordan Niethe
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal PFN and must be handled separately.
Prepare for this by modifying
page_vma_mapped_walk::pfnto contain flags as well as a PFN. Introduce aPVMW_PFN_DEVICE_PRIVATEflag to indicate that apage_vma_mapped_walk::pfncontains a PFN for a device private page.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
v1:
- Update for HMM huge page support
v2:
- Move addingdevice_privateparam tocheck_pmd()until final patch--- include/linux/rmap.h | 30 +++++++++++++++++++++++++++++- mm/page_vma_mapped.c | 13 +++++++------ mm/rmap.c | 4 ++-- mm/vmscan.c | 2 +- 4 files changed, 39 insertions(+), 10 deletions(-)include/linux/rmap.h +29
-1@@ -939,9 +939,37 @@ struct page_vma_mapped_walk {unsigned int flags; }; +/* pfn is a device private offset */ +#define PVMW_PFN_DEVICE_PRIVATE (1UL << 0) +#define PVMW_PFN_SHIFT 1 + +static inline unsigned long page_vma_walk_pfn (unsigned long pfn) +{ +return (pfn <<PVMW_PFN_SHIFT ); +} + +static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) +{ +if (folio_is_device_private (folio)) +return page_vma_walk_pfn (folio_pfn (folio)) | +PVMW_PFN_DEVICE_PRIVATE ; + +return page_vma_walk_pfn (folio_pfn (folio)); +} + +static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) +{ +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +} + +static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +{ +return page_folio (page_vma_walk_pfn_to_page (pvmw_pfn)); +} +#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \ struct page_vma_mapped_walk name = { \ - .pfn = folio_pfn(_folio), \ + .pfn = folio_page_vma_walk_pfn(_folio), \ .nr_pages = folio_nr_pages(_folio), \ .pgoff = folio_pgoff(_folio), \ .vma = _vma, \mm/page_vma_mapped.c +0
-0mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 132
Quoted 5 lines
mm/page_vma_mapped.c +1
-1@@ -129,9 +129,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) pfn =softleaf_to_pfn (entry); } -if ((pfn + pte_nr -1 ) < pvmw->pfn) +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))Can you add a helper function for (
pvmw->pfn>>PVMW_PFN_SHIFT)? It is impossible to tell why pfn does not need >>PVMW_PFN_SHIFT.
Quoted 4 lines
Quoted 2 lines
Sure, something like
page_vma_walk_pfn_to_offset()?Just
page_vma_walk_pfn(pvmw)? Since the code is comparing with pfn.
Sure, that works.
Thanks,
Jordan.
Quoted 3 lines
Best Regards,
Yan, Zi
PATCH v2 6/11 mm: Add helpers to create migration entries from struct pages
In reply to Jordan NietheTo create a new migration entry for a given struct page, that page is first converted to its pfn, before passing the pfn to make_readable_migration_entry() (and friends).
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn and must be handled separately.
Prepare for this with a new set of helpers:
- make_readable_migration_entry_from_page()
- make_readable_exclusive_migration_entry_from_page()
- make_writable_migration_entry_from_page()
These helpers take a struct page as parameter instead of a pfn. This will allow more flexibility for handling the swap offset field differently for device private pages.
---
v1:
- New to series
v2:
- Add flags param
--- include/linux/leafops.h | 14 ++++++++++++++ include/linux/swapops.h | 33 +++++++++++++++++++++++++++++++++ mm/huge_memory.c | 29 +++++++++++++++++------------ mm/hugetlb.c | 15 +++++++++------ mm/memory.c | 5 +++-- mm/migrate_device.c | 12 ++++++------ mm/mprotect.c | 10 +++++++--- mm/rmap.c | 12 ++++++------ 8 files changed, 95 insertions(+), 35 deletions(-)
include/linux/leafops.h +14 -0
@@ -363,6 +363,20 @@ static inline unsigned long softleaf_to_pfn(softleaf_t entry)return swp_offset (entry) &SWP_PFN_MASK ; } +/** +* softleaf_to_flags() - Obtain flags encoded within leaf entry. +* @entry: Leaf entry, softleaf_has_pfn(@entry) must return true. +* +* Returns: The flags associated with the leaf entry. +*/ +static inline unsigned long softleaf_to_flags (softleaf_t entry) +{ +VM_WARN_ON_ONCE (!softleaf_has_pfn (entry)); + +/* Temporary until swp_entry_t eliminated. */ +return swp_offset (entry) & (SWP_MIG_YOUNG |SWP_MIG_DIRTY ); +} +/** * softleaf_to_page() - Obtains struct page for PFN encoded within leaf entry. * @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
include/linux/swapops.h +33 -0
@@ -173,16 +173,33 @@ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)return swp_entry (SWP_MIGRATION_READ , offset); } +static inline swp_entry_t make_readable_migration_entry_from_page (struct page *page,pgoff_t flags) +{ +return swp_entry (SWP_MIGRATION_READ ,page_to_pfn (page) | flags); +} +static inline swp_entry_t make_readable_exclusive_migration_entry (pgoff_t offset) {return swp_entry (SWP_MIGRATION_READ_EXCLUSIVE , offset); } +static inline swp_entry_t make_readable_exclusive_migration_entry_from_page (struct page *page, +pgoff_t flags) +{ +return swp_entry (SWP_MIGRATION_READ_EXCLUSIVE ,page_to_pfn (page) | flags); +} +static inline swp_entry_t make_writable_migration_entry (pgoff_t offset) {return swp_entry (SWP_MIGRATION_WRITE , offset); } +static inline swp_entry_t make_writable_migration_entry_from_page (struct page *page, +pgoff_t flags) +{ +return swp_entry (SWP_MIGRATION_WRITE ,page_to_pfn (page) | flags); +} +/* * Returns whether the host has large enough swap offset field to support * carrying over pgtable A/D bits for page migrations. The result is
@@ -222,11 +239,27 @@ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)return swp_entry (0 ,0 ); } +static inline swp_entry_t make_readable_migration_entry_from_page (struct page *page,pgoff_t flags) +{ +return swp_entry (0 ,0 ); +} + +static inline swp_entry_t make_writeable_migration_entry_from_page (struct page *page,pgoff_t flags) +{ +return swp_entry (0 ,0 ); +} +static inline swp_entry_t make_readable_exclusive_migration_entry (pgoff_t offset) {return swp_entry (0 ,0 ); } +static inline swp_entry_t make_readable_exclusive_migration_entry_from_page (struct page *page, +pgoff_t flags) +{ +return swp_entry (0 ,0 ); +} +static inline swp_entry_t make_writable_migration_entry (pgoff_t offset) {return swp_entry (0 ,0 );
mm/huge_memory.c +17 -12
@@ -1800,7 +1800,8 @@ static void copy_huge_non_present_pmd(if (softleaf_is_migration_write (entry) ||softleaf_is_migration_read_exclusive (entry)) { - entry =make_readable_migration_entry (swp_offset (entry)); + entry =make_readable_migration_entry_from_page (softleaf_to_page (entry), +softleaf_to_flags (entry)); pmd =swp_entry_to_pmd (entry);if (pmd_swp_soft_dirty (*src_pmd)) pmd =pmd_swp_mksoft_dirty (pmd);
@@ -2524,9 +2525,13 @@ static void change_non_present_huge_pmd(struct mm_struct *mm,* just be safe and disable write */ if (folio_test_anon (folio)) - entry =make_readable_exclusive_migration_entry (swp_offset (entry)); + entry =make_readable_exclusive_migration_entry_from_page ( +softleaf_to_page (entry), +softleaf_to_flags (entry));else - entry =make_readable_migration_entry (swp_offset (entry)); + entry =make_readable_migration_entry_from_page ( +softleaf_to_page (entry), +softleaf_to_flags (entry)); newpmd =swp_entry_to_pmd (entry);if (pmd_swp_soft_dirty (*pmd)) newpmd =pmd_swp_mksoft_dirty (newpmd);
@@ -3183,14 +3188,14 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,for (i =0 , addr = haddr; i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE ) {if (write) - swp_entry =make_writable_migration_entry ( -page_to_pfn (page + i)); + swp_entry =make_writable_migration_entry_from_page ( + page + i,0 );else if (anon_exclusive) - swp_entry =make_readable_exclusive_migration_entry ( -page_to_pfn (page + i)); + swp_entry =make_readable_exclusive_migration_entry_from_page ( + page + i,0 );else - swp_entry =make_readable_migration_entry ( -page_to_pfn (page + i)); + swp_entry =make_readable_migration_entry_from_page ( + page + i,0 );if (young) swp_entry =make_migration_entry_young (swp_entry);if (dirty)
@@ -4890,11 +4895,11 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,if (pmd_dirty (pmdval))folio_mark_dirty (folio);if (pmd_write (pmdval)) - entry =make_writable_migration_entry (page_to_pfn (page)); + entry =make_writable_migration_entry_from_page (page,0 );else if (anon_exclusive) - entry =make_readable_exclusive_migration_entry (page_to_pfn (page)); + entry =make_readable_exclusive_migration_entry_from_page (page,0 );else - entry =make_readable_migration_entry (page_to_pfn (page)); + entry =make_readable_migration_entry_from_page (page,0 );if (pmd_young (pmdval)) entry =make_migration_entry_young (entry);if (pmd_dirty (pmdval))
mm/hugetlb.c +9 -6
@@ -4939,8 +4939,9 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,* COW mappings require pages in both * parent and child to be set to read. */ - softleaf =make_readable_migration_entry ( -swp_offset (softleaf)); + softleaf =make_readable_migration_entry_from_page ( +softleaf_to_page (softleaf), +softleaf_to_flags (softleaf)); entry =swp_entry_to_pte (softleaf);if (userfaultfd_wp (src_vma) && uffd_wp) entry =pte_swp_mkuffd_wp (entry);
@@ -6491,11 +6492,13 @@ long hugetlb_change_protection(struct vm_area_struct *vma,if (softleaf_is_migration_write (entry)) {if (folio_test_anon (folio)) - entry =make_readable_exclusive_migration_entry ( -swp_offset (entry)); + entry =make_readable_exclusive_migration_entry_from_page ( +softleaf_to_page (entry), +softleaf_to_flags (entry));else - entry =make_readable_migration_entry ( -swp_offset (entry)); + entry =make_readable_migration_entry_from_page ( +softleaf_to_page (entry), +softleaf_to_flags (entry)); newpte =swp_entry_to_pte (entry); pages++; }
mm/memory.c +3 -2
@@ -963,8 +963,9 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,* to be set to read. A previously exclusive entry is * now shared. */ - entry =make_readable_migration_entry ( -swp_offset (entry)); + entry =make_readable_migration_entry_from_page ( +softleaf_to_page (entry), +softleaf_to_flags (entry)); pte =softleaf_to_pte (entry);if (pte_swp_soft_dirty (orig_pte)) pte =pte_swp_mksoft_dirty (pte);
mm/migrate_device.c +6 -6
@@ -432,14 +432,14 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,/* Setup special migration page table entry */ if (mpfn &MIGRATE_PFN_WRITE ) - entry =make_writable_migration_entry ( -page_to_pfn (page)); + entry =make_writable_migration_entry_from_page ( + page,0 );else if (anon_exclusive) - entry =make_readable_exclusive_migration_entry ( -page_to_pfn (page)); + entry =make_readable_exclusive_migration_entry_from_page ( + page,0 );else - entry =make_readable_migration_entry ( -page_to_pfn (page)); + entry =make_readable_migration_entry_from_page ( + page,0 );if (pte_present (pte)) {if (pte_young (pte)) entry =make_migration_entry_young (entry);
mm/mprotect.c +7 -3
@@ -328,10 +328,14 @@ static long change_pte_range(struct mmu_gather *tlb,* just be safe and disable write */ if (folio_test_anon (folio)) - entry =make_readable_exclusive_migration_entry ( -swp_offset (entry)); + entry =make_readable_exclusive_migration_entry_from_page ( +softleaf_to_page (entry), +softleaf_to_flags (entry));else - entry =make_readable_migration_entry (swp_offset (entry)); + entry =make_readable_migration_entry_from_page ( +softleaf_to_page (entry), +softleaf_to_flags (entry)); + newpte =swp_entry_to_pte (entry);if (pte_swp_soft_dirty (oldpte)) newpte =pte_swp_mksoft_dirty (newpte);
mm/rmap.c +6 -6
@@ -2539,14 +2539,14 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* pte is removed and then restart fault handling. */ if (writable) - entry =make_writable_migration_entry ( -page_to_pfn (subpage)); + entry =make_writable_migration_entry_from_page ( + subpage,0 );else if (anon_exclusive) - entry =make_readable_exclusive_migration_entry ( -page_to_pfn (subpage)); + entry =make_readable_exclusive_migration_entry_from_page ( + subpage,0 );else - entry =make_readable_migration_entry ( -page_to_pfn (subpage)); + entry =make_readable_migration_entry_from_page ( + subpage,0 );if (likely (pte_present (pteval))) {if (pte_young (pteval)) entry =make_migration_entry_young (entry);
Signature · 1 line
2.34.1
PATCH v2 7/11 mm: Add a new swap type for migration entries of device private pages
In reply to Jordan NietheA future change will remove device private pages from the physical address space. This will mean that device private pages no longer have pfns and must be handled separately.
When migrating a device private page a migration entry is created for that page - this includes the pfn for that page. Once device private pages begin using device memory offsets instead of pfns we will need to be able to determine which kind of value is in the entry so we can associate it with the correct page.
Introduce new swap types:
- SWP_MIGRATION_DEVICE_READ
- SWP_MIGRATION_DEVICE_WRITE
- SWP_MIGRATION_DEVICE_READ_EXCLUSIVE
These correspond to
- SWP_MIGRATION_READ
- SWP_MIGRATION_WRITE
- SWP_MIGRATION_READ_EXCLUSIVE
except the swap entry contains a device private offset.
The SWP_MIGRATION_DEVICE swap types are treated as specializations of the SWP_MIGRATION types. That is, the existing helpers such as is_writable_migration_entry() will still return true for a SWP_MIGRATION_DEVICE_WRITE entry. Likewise, the make_*__migration_entry_from_page() helpers will determine create either a SWP_MIGRATION_DEVICE or a SWP_MIGRATION type as the page requires.
Introduce new helpers such as is_writable_device_migration_private_entry() to disambiguate between a SWP_MIGRATION_WRITE and a SWP_MIGRATION_DEVICE_WRITE entry.
Introduce corresponding softleaf types and helpers.
---
v1:
- Update for softleaf infrastructure
- Handle make_readable_migration_entry_from_page() and friends
- s/make_device_migration_readable_exclusive_migration_entry/make_readable_exclusive_migration_device_private_entry
- s/is_device_migration_readable_exclusive_entry/is_readable_exclusive_device_private_migration_entry/
v2:
- Add softleaf_is_migration_device_private_read()
--- include/linux/leafops.h | 86 +++++++++++++++++++++++++++++++++++++---- include/linux/swap.h | 8 +++- include/linux/swapops.h | 79 +++++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+), 9 deletions(-)
include/linux/leafops.h +78 -8
@@ -28,6 +28,9 @@ enum softleaf_type {SOFTLEAF_DEVICE_PRIVATE_READ ,SOFTLEAF_DEVICE_PRIVATE_WRITE ,SOFTLEAF_DEVICE_EXCLUSIVE , +SOFTLEAF_MIGRATION_DEVICE_READ , +SOFTLEAF_MIGRATION_DEVICE_READ_EXCLUSIVE , +SOFTLEAF_MIGRATION_DEVICE_WRITE ,/* H/W posion types. */ SOFTLEAF_HWPOISON,/* Marker types. */
@@ -165,6 +168,12 @@ static inline enum softleaf_type softleaf_type(softleaf_t entry)return SOFTLEAF_DEVICE_PRIVATE_READ ;case SWP_DEVICE_EXCLUSIVE :return SOFTLEAF_DEVICE_EXCLUSIVE ; +case SWP_MIGRATION_DEVICE_READ : +return SOFTLEAF_MIGRATION_DEVICE_READ ; +case SWP_MIGRATION_DEVICE_WRITE : +return SOFTLEAF_MIGRATION_DEVICE_WRITE ; +case SWP_MIGRATION_DEVICE_READ_EXCLUSIVE : +return SOFTLEAF_MIGRATION_DEVICE_READ_EXCLUSIVE ;#endif #ifdef CONFIG_MEMORY_FAILURE case SWP_HWPOISON :
@@ -190,16 +199,75 @@ static inline bool softleaf_is_swap(softleaf_t entry)return softleaf_type (entry) ==SOFTLEAF_SWAP ; } +/** +* softleaf_is_migration_device_private() - Is this leaf entry a migration +* device private entry? +* @entry: Leaf entry. +* +* Returns: true if the leaf entry is a device private entry, otherwise false. +*/ +static inline bool softleaf_is_migration_device_private (softleaf_t entry) +{ +switch (softleaf_type (entry)) { +case SOFTLEAF_MIGRATION_DEVICE_READ : +case SOFTLEAF_MIGRATION_DEVICE_WRITE : +case SOFTLEAF_MIGRATION_DEVICE_READ_EXCLUSIVE : +return true; +default : +return false; + } +} + +/** +* softleaf_is_migration_device_private_write() - Is this leaf entry a writable +* device private migration entry? +* @entry: Leaf entry. +* +* Returns: true if the leaf entry is a writable device private migration entry, +* otherwise false. +*/ +static inline bool softleaf_is_migration_device_private_write (softleaf_t entry) +{ +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_DEVICE_WRITE ; +} + +/** +* softleaf_is_migration_device_private_read() - Is this leaf entry a readable +* device private migration entry? +* @entry: Leaf entry. +* +* Returns: true if the leaf entry is an readable device private migration +* entry, otherwise false. +*/ +static inline bool softleaf_is_migration_device_private_read (softleaf_t entry) +{ +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_DEVICE_READ ; +} + +/** +* softleaf_is_migration_read_exclusive() - Is this leaf entry an exclusive +* readable device private migration entry? +* @entry: Leaf entry. +* +* Returns: true if the leaf entry is an exclusive readable device private +* migration entry, otherwise false. +*/ +static inline bool softleaf_is_migration_device_private_read_exclusive (softleaf_t entry) +{ +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_DEVICE_READ_EXCLUSIVE ; +} +/** * softleaf_is_migration_write() - Is this leaf entry a writable migration entry? * @entry: Leaf entry. * -* Returns: true if the leaf entry is a writable migration entry, otherwise -* false. +* Returns: true if the leaf entry is a writable migration entry or a writable +* device private migration entry, otherwise false. */ static inline bool softleaf_is_migration_write (softleaf_t entry) { -return softleaf_type (entry) ==SOFTLEAF_MIGRATION_WRITE ; +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_WRITE || +softleaf_is_migration_device_private_write (entry); }/**
@@ -211,7 +279,8 @@ static inline bool softleaf_is_migration_write(softleaf_t entry)*/ static inline bool softleaf_is_migration_read (softleaf_t entry) { -return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ ; +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ || +softleaf_is_migration_device_private_read (entry); }/**
@@ -219,12 +288,13 @@ static inline bool softleaf_is_migration_read(softleaf_t entry)* readable migration entry? * @entry: Leaf entry. * -* Returns: true if the leaf entry is an exclusive readable migration entry, -* otherwise false. +* Returns: true if the leaf entry is an exclusive readable migration entry or +* exclusive readable device private migration entry, otherwise false. */ static inline bool softleaf_is_migration_read_exclusive (softleaf_t entry) { -return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ_EXCLUSIVE ; +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ_EXCLUSIVE || +softleaf_is_migration_device_private_read_exclusive (entry); }/**
@@ -241,7 +311,7 @@ static inline bool softleaf_is_migration(softleaf_t entry)case SOFTLEAF_MIGRATION_WRITE :return true;default : -return false; +return softleaf_is_migration_device_private (entry); } }
include/linux/swap.h +7 -1
@@ -74,12 +74,18 @@ static inline int current_is_kswapd(void)* * When a page is mapped by the device for exclusive access we set the CPU page * table entries to a special SWP_DEVICE_EXCLUSIVE entry. +* +* Because device private pages do not use regular PFNs, special migration +* entries are also needed. */ #ifdef CONFIG_DEVICE_PRIVATE -#define SWP_DEVICE_NUM 3 +#define SWP_DEVICE_NUM 6#define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM)#define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1)#define SWP_DEVICE_EXCLUSIVE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2) +#define SWP_MIGRATION_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+3) +#define SWP_MIGRATION_DEVICE_READ_EXCLUSIVE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+4) +#define SWP_MIGRATION_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+5)#else #define SWP_DEVICE_NUM 0#endif
include/linux/swapops.h +79 -0
109 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v2 07/11] mm: Add a new swap type for migration entries of device private pages
Hi,
I am copying across this discussion that occurred on a resend of this series on the intel-xe list for a wider audience [0].
[0] https://lore.kernel.org/all/[email protected]/
Quoted 10 lines · Lorenzo Stoakes
include/linux/leafops.h enum softleaf_type { · line 33
Quoted 7 lines
include/linux/leafops.h +3
-0@@ -28,6 +28,9 @@ enum softleaf_type {SOFTLEAF_DEVICE_PRIVATE_READ ,SOFTLEAF_DEVICE_PRIVATE_WRITE ,SOFTLEAF_DEVICE_EXCLUSIVE , +SOFTLEAF_MIGRATION_DEVICE_READ , +SOFTLEAF_MIGRATION_DEVICE_READ_EXCLUSIVE , + SOFTLEAF_MIGRATION_DEVICE_WRITE,I think these should be
SOFTLEAF_MIGRATION_DEVICE_PRIVATE_xxx
Quoted 3 lines
And I realise that's a mouthful 🙂
But to be consistent with other naming, including SOFTLEAF_DEVICE_PRIVATE_*.
Sure.
Quoted 6 lines
I don't see any users of this, do you definitely use it? I mean presumably you might in a subsequent patch, not checked.
Otherwise I'd wrap it into
softleaf_is_migration().
It gets used in the final patch of the series when we begin doing things like:
static inline struct page *softleaf_to_page(softleaf_t entry)
{
struct page *page;
if (softleaf_is_migration_device_private(entry) ||
softleaf_is_device_private(entry))
page = device_private_entry_to_page(entry);
else
page = pfn_to_page(softleaf_to_pfn(entry));Quoted 11 lines
include/linux/leafops.h static inline bool softleaf_is_migration_write(softleaf_t entry) · line 281
Quoted 7 lines
include/linux/leafops.h +0
-0/** @@ -211,7 +279,8 @@ static inline bool softleaf_is_migration_write(softleaf_t entry)*/ static inline bool softleaf_is_migration_read (softleaf_t entry) {For these ones that you are making sort of compound now, can you please update the kdoc to reflect it? You've done it for others but not this one.
Sure, sorry I missed this one.
Quoted 35 lines
Quoted 24 lines
include/linux/leafops.h +6
-4-return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ ; +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ || +softleaf_is_migration_device_private_read (entry); }/** @@ -219,12 +288,13 @@ static inline bool softleaf_is_migration_read(softleaf_t entry)* readable migration entry? * @entry: Leaf entry. * -* Returns: true if the leaf entry is an exclusive readable migration entry, -* otherwise false. +* Returns: true if the leaf entry is an exclusive readable migration entry or +* exclusive readable device private migration entry, otherwise false. */ static inline bool softleaf_is_migration_read_exclusive (softleaf_t entry) { -return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ_EXCLUSIVE ; +return softleaf_type (entry) ==SOFTLEAF_MIGRATION_READ_EXCLUSIVE || +softleaf_is_migration_device_private_read_exclusive (entry); }/** include/linux/leafops.h static inline bool softleaf_is_migration(softleaf_t entry) · line 316
Quoted 7 lines
include/linux/leafops.h +1
-1@@ -241,7 +311,7 @@ static inline bool softleaf_is_migration(softleaf_t entry)case SOFTLEAF_MIGRATION_WRITE :return true;default : -return false; +return softleaf_is_migration_device_private (entry); } }So all of these above ^^^ are making it so you can't determine if an entry is 'migration xxx' vs. 'migration xxx device private'. Is this ok?
Quoted 4 lines
Does anything need to exclusively determine if something is a 'migration xxx'?
If not then fine, but just wanted to check.
The new entry types have the following relationship:
SOFTLEAF_MIGRATION_DEVICE_READ is-a SOFTLEAF_MIGRATION_READ SOFTLEAF_MIGRATION_READ !is-a SOFTLEAF_MIGRATION_DEVICE_READ
So it is remains possible the distinguish the types using the softleaf_is_migration_device_private_read() check
In practice, the only reason for introducing this new type is so we know when we can not call pfn_to_page() on the swap entry offset. That is the only time that the difference matters.
Rather than introducing a new type we could accomplish this by adding a new flag like SWP_MIG_DIRTY, SWP_MIG_YOUNG but my concern was how we handle the !migration_entry_supports_ad() case.
Quoted 18 lines
Quoted 18 lines
include/linux/swap.h +4
-1@@ -74,12 +74,18 @@ static inline int current_is_kswapd(void)* * When a page is mapped by the device for exclusive access we set the CPU page * table entries to a special SWP_DEVICE_EXCLUSIVE entry. +* +* Because device private pages do not use regular PFNs, special migration +* entries are also needed. */ #ifdef CONFIG_DEVICE_PRIVATE -#define SWP_DEVICE_NUM 3 +#define SWP_DEVICE_NUM 6#define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM)
Quoted 8 lines
include/linux/swap.h static inline int current_is_kswapd(void) · line 88
Quoted 5 lines
include/linux/swap.h +3
-0#define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1)#define SWP_DEVICE_EXCLUSIVE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2) +#define SWP_MIGRATION_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+3) +#define SWP_MIGRATION_DEVICE_READ_EXCLUSIVE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+4) +#define SWP_MIGRATION_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+5)I've lost track on how many entries we have left, have you tested this with a config that maximises the number?
Good point - let me double check that.
Quoted 14 lines
I really hate how we do this by the way, that's another thing to fix... 🙂
include/linux/swapops.h
Quoted 7 lines
include/linux/swap.h +0
-0#else #define SWP_DEVICE_NUM 0#endif include/linux/swapops.h +0
-0Since this is pure softleaf stuff can we please put it all in
leafops.hif possible? I know we already have some stuff here rather than there, but this really doesn't seem to belong here.
The reason for adding the swapops.h entries to correspond with the softleaf entries was it looked like the swapops were still required for making the entries currently.
That is, there aren't softleaf equivalents to make_readable_device_private_entry() and friends yet.
Would it be better if I introduced the swapops.h changes in a proceeding patch?
Quoted 3 lines
Cheers, Lorenzo
Thanks for reviewing.
Jordan.
PATCH v2 8/11 mm: Add helpers to create device private entries from struct pages
In reply to Jordan NietheTo create a new device private entry for a given struct page, that page is first converted to its pfn, before passing the pfn to make_writable_device_private_entry() (and friends).
A future change will remove device private pages from the physical address space. This will mean that device private pages no longer have a pfn and must be handled separately.
Prepare for this with a new set of helpers:
- make_readable_device_private_entry_from_page()
- make_writable_device_private_entry_from_page()
These helpers take a struct page as parameter instead of a pfn. This will allow more flexibility for handling the swap offset field differently for device private pages.
---
v1:
- New to series
v2:
- Add flag param
--- include/linux/swapops.h | 24 ++++++++++++++++++++++++ mm/huge_memory.c | 14 ++++++-------- mm/migrate.c | 6 ++---- mm/migrate_device.c | 12 ++++-------- 4 files changed, 36 insertions(+), 20 deletions(-)
include/linux/swapops.h +24 -0
@@ -138,11 +138,23 @@ static inline swp_entry_t make_readable_device_private_entry(pgoff_t offset)return swp_entry (SWP_DEVICE_READ , offset); } +static inline swp_entry_t make_readable_device_private_entry_from_page (struct page *page, +pgoff_t flags) +{ +return swp_entry (SWP_DEVICE_READ ,page_to_pfn (page) | flags); +} +static inline swp_entry_t make_writable_device_private_entry (pgoff_t offset) {return swp_entry (SWP_DEVICE_WRITE , offset); } +static inline swp_entry_t make_writable_device_private_entry_from_page (struct page *page, +pgoff_t flags) +{ +return swp_entry (SWP_DEVICE_WRITE ,page_to_pfn (page) | flags); +} +static inline swp_entry_t make_device_exclusive_entry (pgoff_t offset) {return swp_entry (SWP_DEVICE_EXCLUSIVE , offset);
@@ -191,11 +203,23 @@ static inline swp_entry_t make_readable_device_private_entry(pgoff_t offset)return swp_entry (0 ,0 ); } +static inline swp_entry_t make_readable_device_private_entry_from_page (struct page *page, +pgoff_t flags) +{ +return swp_entry (0 ,0 ); +} +static inline swp_entry_t make_writable_device_private_entry (pgoff_t offset) {return swp_entry (0 ,0 ); } +static inline swp_entry_t make_writable_device_private_entry_from_page (struct page *page, +pgoff_t flags) +{ +return swp_entry (0 ,0 ); +} +static inline swp_entry_t make_device_exclusive_entry (pgoff_t offset) {return swp_entry (0 ,0 );
mm/huge_memory.c +6 -8
@@ -3219,11 +3219,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* is false. */ if (write) - swp_entry =make_writable_device_private_entry ( -page_to_pfn (page + i)); + swp_entry =make_writable_device_private_entry_from_page ( + page + i,0 );else - swp_entry =make_readable_device_private_entry ( -page_to_pfn (page + i)); + swp_entry =make_readable_device_private_entry_from_page ( + page + i,0 );/* * Young and dirty bits are not progated via swp_entry */
@@ -4950,11 +4950,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)swp_entry_t entry;if (pmd_write (pmde)) - entry =make_writable_device_private_entry ( -page_to_pfn (new)); + entry =make_writable_device_private_entry_from_page (new,0 );else - entry =make_readable_device_private_entry ( -page_to_pfn (new)); + entry =make_readable_device_private_entry_from_page (new,0 ); pmde =swp_entry_to_pmd (entry);if (pmd_swp_soft_dirty (*pvmw->pmd))
mm/migrate.c +2 -4
@@ -399,11 +399,9 @@ static bool remove_migration_pte(struct folio *folio,if (unlikely (is_device_private_page (new))) {if (pte_write (pte)) - entry =make_writable_device_private_entry ( -page_to_pfn (new)); + entry =make_writable_device_private_entry_from_page (new,0 );else - entry =make_readable_device_private_entry ( -page_to_pfn (new)); + entry =make_readable_device_private_entry_from_page (new,0 ); pte =softleaf_to_pte (entry);if (pte_swp_soft_dirty (old_pte)) pte =pte_swp_mksoft_dirty (pte);
mm/migrate_device.c +4 -8
@@ -836,11 +836,9 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate,swp_entry_t swp_entry;if (vma->vm_flags &VM_WRITE ) - swp_entry =make_writable_device_private_entry ( -page_to_pfn (page)); + swp_entry =make_writable_device_private_entry_from_page (page,0 );else - swp_entry =make_readable_device_private_entry ( -page_to_pfn (page)); + swp_entry =make_readable_device_private_entry_from_page (page,0 ); entry =swp_entry_to_pmd (swp_entry); }else { if (folio_is_zone_device(folio) &&
@@ -1033,11 +1031,9 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,swp_entry_t swp_entry;if (vma->vm_flags &VM_WRITE ) - swp_entry =make_writable_device_private_entry ( -page_to_pfn (page)); + swp_entry =make_writable_device_private_entry_from_page (page,0 );else - swp_entry =make_readable_device_private_entry ( -page_to_pfn (page)); + swp_entry =make_readable_device_private_entry_from_page (page,0 ); entry =swp_entry_to_pte (swp_entry); }else { if (folio_is_zone_device(folio) &&
Signature · 1 line
2.34.1
PATCH v2 9/11 mm/util: Add flag to track device private pages in page snapshots
In reply to Jordan NietheA future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal pfns and must be handled separately.
Add a new flag PAGE_SNAPSHOT_DEVICE_PRIVATE to track when the pfn of a page snapshot is a device private page.
---
v1:
- No change
v2:
- No change
--- fs/proc/page.c | 6 ++++-- include/linux/mm.h | 7 ++++--- mm/util.c | 3 +++ 3 files changed, 11 insertions(+), 5 deletions(-)
fs/proc/page.c +4 -2
@@ -191,10 +191,12 @@ u64 stable_page_flags(const struct page *page) folio_test_large_rmappable(folio)) {/* Note: we indicate any THPs here, not just PMD-sized ones */ u |=1 <<KPF_THP ; - } else if (is_huge_zero_pfn(ps.pfn)) { + } else if (!(ps.flags & PAGE_SNAPSHOT_DEVICE_PRIVATE) && + is_huge_zero_pfn(ps.pfn)) { u |=1 <<KPF_ZERO_PAGE ; u |=1 <<KPF_THP ; - } else if (is_zero_pfn(ps.pfn)) { + } else if (!(ps.flags & PAGE_SNAPSHOT_DEVICE_PRIVATE) + && is_zero_pfn(ps.pfn)) { u |=1 <<KPF_ZERO_PAGE ; }
include/linux/mm.h +4 -3
@@ -4623,9 +4623,10 @@ static inline bool page_pool_page_is_pp(const struct page *page) }#endif -#define PAGE_SNAPSHOT_FAITHFUL (1 << 0) -#define PAGE_SNAPSHOT_PG_BUDDY (1 << 1) -#define PAGE_SNAPSHOT_PG_IDLE (1 << 2) +#define PAGE_SNAPSHOT_FAITHFUL (1 << 0) +#define PAGE_SNAPSHOT_PG_BUDDY (1 << 1) +#define PAGE_SNAPSHOT_PG_IDLE (1 << 2) +#define PAGE_SNAPSHOT_DEVICE_PRIVATE (1 << 3)struct page_snapshot {struct folio folio_snapshot;
mm/util.c +3 -0
@@ -1218,6 +1218,9 @@ static void set_ps_flags(struct page_snapshot *ps, const struct folio *folio,if (folio_test_idle (folio)) ps->flags |=PAGE_SNAPSHOT_PG_IDLE ; + +if (is_device_private_page (page)) + ps->flags |=PAGE_SNAPSHOT_DEVICE_PRIVATE ; }/**
Signature · 1 line
2.34.1
PATCH v2 10/11 mm/hmm: Add flag to track device private pages
In reply to Jordan NietheA future change will remove device private pages from the physical address space. This will mean that device private pages no longer have normal pfns and must be handled separately.
Prepare for this by adding a HMM_PFN_DEVICE_PRIVATE flag to indicate that a hmm_pfn contains a PFN for a device private page.
---
v1:
- Update HMM_PFN_ORDER_SHIFT
- Handle hmm_vma_handle_absent_pmd()
v2:
- No change
--- include/linux/hmm.h | 4 +++- mm/hmm.c | 5 +++-- 2 files changed, 6 insertions(+), 3 deletions(-)
include/linux/hmm.h +3 -1
@@ -23,6 +23,7 @@ struct mmu_interval_notifier;* HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc +* HMM_PFN_DEVICE_PRIVATE - the pfn field contains a DEVICE_PRIVATE pfn. * HMM_PFN_P2PDMA - P2P page * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation
@@ -40,6 +41,7 @@ enum hmm_pfn_flags {HMM_PFN_VALID =1UL << (BITS_PER_LONG -1 ),HMM_PFN_WRITE =1UL << (BITS_PER_LONG -2 ),HMM_PFN_ERROR =1UL << (BITS_PER_LONG -3 ), +HMM_PFN_DEVICE_PRIVATE =1UL << (BITS_PER_LONG -7 ),/* * Sticky flags, carried from input to output, * don't forget to update HMM_PFN_INOUT_FLAGS
@@ -48,7 +50,7 @@ enum hmm_pfn_flags {HMM_PFN_P2PDMA =1UL << (BITS_PER_LONG -5 ),HMM_PFN_P2PDMA_BUS =1UL << (BITS_PER_LONG -6 ), -HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG -11 ), +HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG -12 ),/* Input flags */ HMM_PFN_REQ_FAULT = HMM_PFN_VALID,
mm/hmm.c +3 -2
@@ -267,7 +267,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,if (softleaf_is_device_private (entry) &&page_pgmap (softleaf_to_page (entry))->owner == range->dev_private_owner) { - cpu_flags =HMM_PFN_VALID ; + cpu_flags =HMM_PFN_VALID |HMM_PFN_DEVICE_PRIVATE ;if (softleaf_is_device_private_write (entry)) cpu_flags |=HMM_PFN_WRITE ; new_pfn_flags =softleaf_to_pfn (entry) | cpu_flags;
@@ -347,7 +347,8 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start, softleaf_to_folio(entry)->pgmap->owner == range->dev_private_owner) {unsigned long cpu_flags =HMM_PFN_VALID | -hmm_pfn_flags_order (PMD_SHIFT -PAGE_SHIFT ); +hmm_pfn_flags_order (PMD_SHIFT -PAGE_SHIFT ) | +HMM_PFN_DEVICE_PRIVATE ;unsigned long pfn =softleaf_to_pfn (entry);unsigned long i;
Signature · 1 line
2.34.1
PATCH v2 11/11 mm: Remove device private pages from the physical address space
In reply to Jordan NietheCurrently when creating these device private struct pages, the first step is to use request_free_mem_region() to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory using memremap_pages().
Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface memremap_device_private_pagemap() that allocates a requested amount of device private address space and creates the necessary device private pages.
To support this new interface, struct dev_pagemap needs some changes:
- Add a new dev_pagemap::nr_pages field as an input parameter.
- Add a new dev_pagemap::pages array to store the device private pages.
When using memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped, dev_pagemap::nr_ranges will always be 1, and the device private range that is reserved is returned in dev_pagemap::range.
Forbid calling memremap_pages() with dev_pagemap::ranges::type = MEMORY_DEVICE_PRIVATE.
Represent this device private address space using a new device_private_pgmap_tree maple tree. This tree maps a given device private address to a struct dev_pagemap, where a specific device private page may then be looked up in that dev_pagemap::pages array.
Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new memunmap_device_private_pagemap() interface.
Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that page_to_pfn(), et al. are no longer meaningful.
Introduce helpers:
- device_private_page_to_offset()
- device_private_folio_to_offset()
to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use device_private_offset_to_page() instead.
Update the existing users:
- lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
to use the new memremap_device_private_pagemap() interface.
---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater for memremap_device_private_pagemap()
- Add devm_memremap_device_private_pagemap() and friends
- Update existing users of memremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guard device_private_offset_to_page and friends with CONFIG_ZONE_DEVICE
v2:
- Make sure last member of struct dev_pagemap remains DECLARE_FLEX_ARRAY(struct range, ranges);
--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)
Documentation/mm/hmm.rst +3 -8
@@ -276,17 +276,12 @@ These can be allocated and freed with:: struct resource *res; struct dev_pagemap pagemap; - res = request_free_mem_region(&iomem_resource, /* number of bytes */, - "name of driver resource"); pagemap.type = MEMORY_DEVICE_PRIVATE; - pagemap.range.start = res->start; - pagemap.range.end = res->end; - pagemap.nr_range = 1; + pagemap.nr_pages = /* number of pages */; pagemap.ops = &device_devmem_ops; - memremap_pages(&pagemap, numa_node_id()); + memremap_device_private_pagemap(&pagemap, numa_node_id()); - memunmap_pages(&pagemap); - release_mem_region(pagemap.range.start, range_len(&pagemap.range)); + memunmap_device_private_pagemap(&pagemap); There are also devm_request_free_mem_region(), devm_memremap_pages(), devm_memunmap_pages(), and devm_release_mem_region() when the resources can
arch/powerpc/kvm/book3s_hv_uvmem.c +13 -28
100 lines not on this page · raw message
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +12 -11
@@ -1024,9 +1024,9 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev) {struct amdgpu_kfd_dev *kfddev = &adev->kfd;struct dev_pagemap *pgmap; -struct resource *res =NULL ;unsigned long size;void *r; +int ret;/* Page migration works on gfx9 or newer */ if (amdgpu_ip_version (adev,GC_HWIP ,0 ) <IP_VERSION (9 ,0 ,1 ))
@@ -1047,11 +1047,7 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev) pgmap->range.end = adev->gmc.aper_base + adev->gmc.aper_size -1 ; pgmap->type =MEMORY_DEVICE_COHERENT ; }else { - res =devm_request_free_mem_region (adev->dev, &iomem_resource, size); -if (IS_ERR (res)) -return PTR_ERR (res); - pgmap->range.start = res->start; - pgmap->range.end = res->end; + pgmap->nr_pages = size /PAGE_SIZE ; pgmap->type =MEMORY_DEVICE_PRIVATE ; }
@@ -1062,14 +1058,19 @@ int kgd2kfd_init_zone_device(struct amdgpu_device *adev)/* Device manager releases device-specific resources, memory region and * pgmap when driver disconnects from device. */ - r =devm_memremap_pages (adev->dev, pgmap); -if (IS_ERR (r)) { +if (pgmap->type ==MEMORY_DEVICE_PRIVATE ) { + ret =devm_memremap_device_private_pagemap (adev->dev, pgmap); + }else { + r =devm_memremap_pages (adev->dev, pgmap); +if (IS_ERR (r)) + ret =PTR_ERR (r); + } + +if (ret) {pr_err ("failed to register HMM device memory\n" ); -if (pgmap->type ==MEMORY_DEVICE_PRIVATE ) -devm_release_mem_region (adev->dev, res->start,resource_size (res));/* Disable SVM support capability */ pgmap->type =0 ; -return PTR_ERR (r); +return ret; } pr_debug("reserve %ldMB system memory for VRAM pages struct\n",
drivers/gpu/drm/nouveau/nouveau_dmem.c +9 -26
@@ -109,7 +109,7 @@ static struct nouveau_drm *page_to_drm(struct page *page)unsigned long nouveau_dmem_page_addr (struct page *page) {struct nouveau_dmem_chunk *chunk =nouveau_page_to_chunk (page); -unsigned long off = (page_to_pfn (page) <<PAGE_SHIFT ) - +unsigned long off = (device_private_page_to_offset (page) <<PAGE_SHIFT ) - chunk->pagemap.range.start;return chunk->bo->offset + off;
@@ -297,9 +297,7 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage, bool is_large) {struct nouveau_dmem_chunk *chunk; -struct resource *res;struct page *page; -void *ptr;unsigned long i, pfn_first, pfn;int ret;
@@ -309,39 +307,28 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage, goto out; } -/* Allocate unused physical address space for device private pages. */ - res =request_free_mem_region (&iomem_resource,DMEM_CHUNK_SIZE *NR_CHUNKS , -"nouveau_dmem" ); -if (IS_ERR (res)) { - ret =PTR_ERR (res); - goto out_free; - } - chunk->drm = drm; chunk->pagemap.type =MEMORY_DEVICE_PRIVATE ; - chunk->pagemap.range.start = res->start; - chunk->pagemap.range.end = res->end; chunk->pagemap.nr_range =1 ; + chunk->pagemap.nr_pages =DMEM_CHUNK_SIZE *NR_CHUNKS /PAGE_SIZE ; chunk->pagemap.ops = &nouveau_dmem_pagemap_ops; chunk->pagemap.owner = drm->dev; ret =nouveau_bo_new_pin (&drm->client,NOUVEAU_GEM_DOMAIN_VRAM ,DMEM_CHUNK_SIZE , &chunk->bo);if (ret) - goto out_release; + goto out_free; - ptr =memremap_pages (&chunk->pagemap,numa_node_id ()); -if (IS_ERR (ptr)) { - ret =PTR_ERR (ptr); + ret =memremap_device_private_pagemap (&chunk->pagemap,numa_node_id ()); +if (ret) goto out_bo_free; - }mutex_lock (&drm->dmem->mutex);list_add (&chunk->list, &drm->dmem->chunks);mutex_unlock (&drm->dmem->mutex); pfn_first = chunk->pagemap.range.start >>PAGE_SHIFT ; - page =pfn_to_page (pfn_first); + page =device_private_offset_to_page (pfn_first);spin_lock (&drm->dmem->lock); pfn = pfn_first;
@@ -350,12 +337,12 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage,if (!IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) || !is_large) {for (j =0 ; j <DMEM_CHUNK_NPAGES -1 ; j++, pfn++) { - page =pfn_to_page (pfn); + page =device_private_offset_to_page (pfn); page->zone_device_data = drm->dmem->free_pages; drm->dmem->free_pages = page; } }else { - page =pfn_to_page (pfn); + page =device_private_offset_to_page (pfn); page->zone_device_data = drm->dmem->free_folios; drm->dmem->free_folios =page_folio (page); pfn +=DMEM_CHUNK_NPAGES ;
@@ -382,8 +369,6 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage, out_bo_free:nouveau_bo_unpin_del (&chunk->bo); -out_release: -release_mem_region (chunk->pagemap.range.start,range_len (&chunk->pagemap.range)); out_free:kfree (chunk); out:
@@ -543,9 +528,7 @@ nouveau_dmem_fini(struct nouveau_drm *drm)nouveau_bo_unpin_del (&chunk->bo);WARN_ON (chunk->callocated);list_del (&chunk->list); -memunmap_pages (&chunk->pagemap); -release_mem_region (chunk->pagemap.range.start, -range_len (&chunk->pagemap.range)); +memunmap_device_private_pagemap (&chunk->pagemap);kfree (chunk); }
drivers/gpu/drm/xe/xe_svm.c +8 -20
@@ -404,7 +404,7 @@ static u64 xe_vram_region_page_to_dpa(struct xe_vram_region *vr, struct page *page) {u64 dpa; -u64 pfn =page_to_pfn (page); +u64 pfn =device_private_page_to_offset (page);u64 offset;xe_assert (vr->xe,is_device_private_page (page));
@@ -1471,39 +1471,27 @@ int xe_devm_add(struct xe_tile *tile, struct xe_vram_region *vr) {struct xe_device *xe =tile_to_xe (tile);struct device *dev = &to_pci_dev (xe->drm.dev)->dev; -struct resource *res; -void *addr;int ret; - res =devm_request_free_mem_region (dev, &iomem_resource, - vr->usable_size); -if (IS_ERR (res)) { - ret =PTR_ERR (res); -return ret; - } - vr->pagemap.type =MEMORY_DEVICE_PRIVATE ; - vr->pagemap.range.start = res->start; - vr->pagemap.range.end = res->end; vr->pagemap.nr_range =1 ; + vr->pagemap.nr_pages = vr->usable_size /PAGE_SIZE ; vr->pagemap.ops =drm_pagemap_pagemap_ops_get (); vr->pagemap.owner =xe_svm_devm_owner (xe); - addr =devm_memremap_pages (dev, &vr->pagemap); + ret =devm_memremap_device_private_pagemap (dev, &vr->pagemap); vr->dpagemap.dev = dev; vr->dpagemap.ops = &xe_drm_pagemap_ops; -if (IS_ERR (addr)) { -devm_release_mem_region (dev, res->start,resource_size (res)); - ret =PTR_ERR (addr); -drm_err (&xe->drm,"Failed to remap tile %d memory, errno %pe\n" , - tile->id,ERR_PTR (ret)); +if (ret) { +drm_err (&xe->drm,"Failed to remap tile %d memory, errno %d\n" , + tile->id, ret);return ret; } - vr->hpa_base = res->start; + vr->hpa_base = vr->pagemap.range.start;drm_dbg (&xe->drm,"Added tile %d memory [%llx-%llx] to devm, remapped to %pr\n" , - tile->id, vr->io_start, vr->io_start + vr->usable_size, res); + tile->id, vr->io_start, vr->io_start + vr->usable_size, &vr->pagemap.range);return 0 ; }#else
include/linux/hmm.h +3 -0
@@ -68,6 +68,9 @@ enum hmm_pfn_flags {*/ static inline struct page *hmm_pfn_to_page (unsigned long hmm_pfn) { +if (hmm_pfn &HMM_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (hmm_pfn & ~HMM_PFN_FLAGS ); +return pfn_to_page (hmm_pfn & ~HMM_PFN_FLAGS ); }
include/linux/leafops.h +14 -2
@@ -455,7 +455,13 @@ static inline unsigned long softleaf_to_flags(softleaf_t entry)*/ static inline struct page *softleaf_to_page (softleaf_t entry) { -struct page *page =pfn_to_page (softleaf_to_pfn (entry)); +struct page *page; + +if (softleaf_is_migration_device_private (entry) || +softleaf_is_device_private (entry)) + page =device_private_entry_to_page (entry); +else + page =pfn_to_page (softleaf_to_pfn (entry));VM_WARN_ON_ONCE (!softleaf_has_pfn (entry));/*
@@ -475,7 +481,13 @@ static inline struct page *softleaf_to_page(softleaf_t entry)*/ static inline struct folio *softleaf_to_folio (softleaf_t entry) { -struct folio *folio =pfn_folio (softleaf_to_pfn (entry)); +struct folio *folio; + +if (softleaf_is_migration_device_private (entry) || +softleaf_is_device_private (entry)) + folio =page_folio (device_private_entry_to_page (entry)); +else + folio =pfn_folio (softleaf_to_pfn (entry));VM_WARN_ON_ONCE (!softleaf_has_pfn (entry));/*
include/linux/memremap.h +62 -2
107 lines not on this page · raw message
include/linux/migrate.h +5 -1
@@ -133,6 +133,10 @@ static inline struct page *migrate_pfn_to_page(unsigned long mpfn) {if (!(mpfn &MIGRATE_PFN_VALID ))return NULL ; + +if (mpfn &MIGRATE_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (mpfn >>MIGRATE_PFN_SHIFT ); +return pfn_to_page (mpfn >>MIGRATE_PFN_SHIFT ); }
@@ -144,7 +148,7 @@ static inline unsigned long migrate_pfn(unsigned long pfn)static inline unsigned long migrate_pfn_from_page (struct page *page) {if (is_device_private_page (page)) -return migrate_pfn (page_to_pfn (page)) | +return migrate_pfn (device_private_page_to_offset (page)) |MIGRATE_PFN_DEVICE_PRIVATE ;return migrate_pfn (page_to_pfn (page)); }
include/linux/mm.h +2 -0
@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio)); +return page_to_pfn (&folio->page); }
include/linux/rmap.h +4 -1
@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));
@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }
include/linux/swapops.h +5 -5
@@ -141,7 +141,7 @@ static inline swp_entry_t make_readable_device_private_entry(pgoff_t offset)static inline swp_entry_t make_readable_device_private_entry_from_page (struct page *page,pgoff_t flags) { -return swp_entry (SWP_DEVICE_READ ,page_to_pfn (page) | flags); +return swp_entry (SWP_DEVICE_READ ,device_private_page_to_offset (page) | flags); }static inline swp_entry_t make_writable_device_private_entry (pgoff_t offset)
@@ -152,7 +152,7 @@ static inline swp_entry_t make_writable_device_private_entry(pgoff_t offset)static inline swp_entry_t make_writable_device_private_entry_from_page (struct page *page,pgoff_t flags) { -return swp_entry (SWP_DEVICE_WRITE ,page_to_pfn (page) | flags); +return swp_entry (SWP_DEVICE_WRITE ,device_private_page_to_offset (page) | flags); }static inline swp_entry_t make_device_exclusive_entry (pgoff_t offset)
@@ -268,7 +268,7 @@ static inline swp_entry_t make_readable_migration_entry_from_page(struct page *p {if (is_device_private_page (page))return make_readable_migration_device_private_entry ( -page_to_pfn (page) | flags); +device_private_page_to_offset (page) | flags);return swp_entry (SWP_MIGRATION_READ ,page_to_pfn (page) | flags); }
@@ -283,7 +283,7 @@ static inline swp_entry_t make_readable_exclusive_migration_entry_from_page(stru {if (is_device_private_page (page))return make_readable_exclusive_migration_device_private_entry ( -page_to_pfn (page) | flags); +device_private_page_to_offset (page) | flags);return swp_entry (SWP_MIGRATION_READ_EXCLUSIVE ,page_to_pfn (page) | flags); }
@@ -298,7 +298,7 @@ static inline swp_entry_t make_writable_migration_entry_from_page(struct page *p {if (is_device_private_page (page))return make_writable_migration_device_private_entry ( -page_to_pfn (page) | flags); +device_private_page_to_offset (page) | flags);return swp_entry (SWP_MIGRATION_WRITE ,page_to_pfn (page) | flags); }
lib/test_hmm.c +40 -29
150 lines not on this page · raw message
mm/debug.c +6 -3
@@ -77,9 +77,11 @@ static void __dump_folio(const struct folio *folio, const struct page *page,if (page_mapcount_is_type (mapcount)) mapcount =0 ; -pr_warn ("page: refcount:%d mapcount:%d mapping:%p index:%#lx pfn:%#lx\n" , +pr_warn ("page: refcount:%d mapcount:%d mapping:%p index:%#lx %spfn:%#lx\n" ,folio_ref_count (folio), mapcount, mapping, - folio->index + idx, pfn); + folio->index + idx, +folio_is_device_private (folio) ?"device " :"" , + pfn);if (folio_test_large (folio)) {int pincount =0 ;
@@ -113,7 +115,8 @@ static void __dump_folio(const struct folio *folio, const struct page *page,* inaccuracy here due to racing. */ pr_warn ("%sflags: %pGp%s\n" , type, &folio->flags, -is_migrate_cma_folio (folio, pfn) ?" CMA" :"" ); + (!folio_is_device_private (folio) && +is_migrate_cma_folio (folio, pfn)) ?" CMA" :"" );if (page_has_type (&folio->page))pr_warn ("page_type: %x(%s)\n" , folio->page.page_type >>24 ,page_type_name (folio->page.page_type));
mm/memremap.c +151 -42
252 lines not on this page · raw message
mm/mm_init.c +4 -4
@@ -1004,9 +1004,9 @@ static void __init memmap_init(void) }#ifdef CONFIG_ZONE_DEVICE -static void __ref __init_zone_device_page (struct page *page,unsigned long pfn, -unsigned long zone_idx,int nid, -struct dev_pagemap *pgmap) +void __ref __init_zone_device_page (struct page *page,unsigned long pfn, +unsigned long zone_idx,int nid, +struct dev_pagemap *pgmap) {__init_single_page (page, pfn, zone_idx, nid);
@@ -1038,7 +1038,7 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,* Please note that MEMINIT_HOTPLUG path doesn't clear memmap * because this is done early in section_activate() */ -if (pageblock_aligned (pfn)) { +if (pgmap->type !=MEMORY_DEVICE_PRIVATE &&pageblock_aligned (pfn)) {init_pageblock_migratetype (page,MIGRATE_MOVABLE , false);cond_resched (); }
mm/page_vma_mapped.c +17 -2
@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {
@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);
@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))
@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )
@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;
@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw)) +if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }
mm/rmap.c +29 -14
99 lines not on this page · raw message
mm/util.c +4 -1
@@ -1244,7 +1244,10 @@ void snapshot_page(struct page_snapshot *ps, const struct page *page)struct folio *foliop;int loops =5 ; - ps->pfn =page_to_pfn (page); +if (is_device_private_page (page)) + ps->pfn =device_private_page_to_offset (page); +else + ps->pfn =page_to_pfn (page); ps->flags =PAGE_SNAPSHOT_FAITHFUL ; again:
Signature · 1 line
2.34.1
Zi Yan
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)
<snip>
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));
Please use VM_WARN_ON instead.
Quoted 26 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }
<snip>
Quoted 60 lines
mm/page_vma_mapped.c +16
-1@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 8 lines
mm/page_vma_mapped.c +1
-1@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw)) +if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }
It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating pvmw->pfn itself.
Best Regards,
Yan, Zi
Re: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.
ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))
Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.
We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
Thanks,
Jordan.
Quoted 3 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 241 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.
Quoted 6 lines
We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like page_vma_walk_pfn(pvmw) to return the converted pfn instead of pvmw->pfn directly. You can add a comment to ask people to use helper function and even mark pvmw->pfn /* do not use directly */.
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Best Regards,
Yan, Zi
Re: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.
Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Currently for migrate pfns we left shift by pfns by MIGRATE_PFN_SHIFT (6), so I thought doing something similiar here should be safe.
Thanks,
Jordan.
Quoted 4 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 261 lines · Jordan Niethe
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.
Yeah, but that limits to archs supporting HMM. page_vma_mapped_walk is used by almost every arch, so it has a broader impact.
Best Regards,
Yan, Zi
Re: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Hi,
Quoted 267 lines · Zi Yan
Quoted 265 lines · Jordan Niethe
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.
That is a good point.
I see a few options:
- On every arch we can assume SWP_PFN_BITS? I could add a sanity check that we have an extra bit on top of SWP_PFN_BITS within an unsigned long.
- We could define PVMW_PFN_SHIFT as 0 if !CONFIG_MIGRATION as the flag is not required.
- Instead of modifying pvmw->pfn we could use pvmw->flags but that has the issues of separating the offset type and offset.
Thanks,
Jordan.
Quoted 3 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 276 lines · Jordan Niethe
Hi,
Quoted 267 lines · Zi Yan
Quoted 265 lines · Jordan Niethe
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.
Quoted 6 lines
That is a good point.
I see a few options:
- On every arch we can assume
SWP_PFN_BITS? I could add a sanity check that we have an extra bit on top ofSWP_PFN_BITSwithin an unsigned long.
Yes, but if there is no extra bit, are you going to disable device private pages?
Quoted 2 lines
- We could define
PVMW_PFN_SHIFTas 0 if !CONFIG_MIGRATION as the flag is not required.
Sure, or !CONFIG_DEVICE_MIGRATION
Quoted 2 lines
- Instead of modifying
pvmw->pfnwe could usepvmw->flagsbut that has the issues of separating the offset type and offset.
It seems that I was not clear on my proposal. Here is the patch on top of your patchset and it compiles.
Basically, pvmw->pfn stores either PFN or device private offset without additional shift. Caller interprets pvmw->pfn based on pvmw->flags & PVMW_DEVICE_PRIVATE. And you can ignore my helper function of pvmw->pfn suggestion, since my patch below can use pvmw->pfn directly.
Let me know if my patch works. Thanks.
include/linux/rmap.h +15 -14
@@ -921,6 +921,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,#define PVMW_SYNC (1 << 0)/* Look for migration entries rather than present PTEs */ #define PVMW_MIGRATION (1 << 1) +#define PVMW_DEVICE_PRIVATE (1 << 2)/* Result flags */
@@ -943,6 +944,13 @@ struct page_vma_mapped_walk {#define PVMW_PFN_DEVICE_PRIVATE (1UL << 0)#define PVMW_PFN_SHIFT 1 +static inline unsigned long page_vma_walk_flags (struct folio *folio,unsigned long flags) +{ +if (folio_is_device_private (folio)) +return flags |PVMW_DEVICE_PRIVATE ; +return flags; +} +static inline unsigned long page_vma_walk_pfn (unsigned long pfn) {return (pfn <<PVMW_PFN_SHIFT );
@@ -951,23 +959,16 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (device_private_folio_to_offset (folio)) | -PVMW_PFN_DEVICE_PRIVATE ; - -return page_vma_walk_pfn (folio_pfn (folio)); -} - -static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) -{ -if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) -return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return device_private_folio_to_offset (folio); -return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return (folio_pfn (folio)); } -static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +static inline struct folio *page_vma_walk_pfn_to_folio (struct page_vma_mapped_walk *pvmw) { -return page_folio (page_vma_walk_pfn_to_page (pvmw_pfn)); +if (pvmw->flags &PVMW_DEVICE_PRIVATE ) +return page_folio (device_private_offset_to_page (pvmw->pfn)); +return pfn_folio (pvmw->pfn); }#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \
@@ -977,7 +978,7 @@ static inline struct folio *page_vma_walk_pfn_to_folio(unsigned long pvmw_pfn) .pgoff = folio_pgoff(_folio), \ .vma = _vma, \ .address = _address, \ - .flags = _flags, \ + .flags = page_vma_walk_flags(_folio, _flags), \ }static inline void page_vma_mapped_walk_done (struct page_vma_mapped_walk *pvmw)
mm/page_vma_mapped.c +7 -7
@@ -136,12 +136,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) device_private = true; } -if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +if ((device_private) ^ !!(pvmw->flags &PVMW_DEVICE_PRIVATE ))return false; -if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT )) +if ((pfn + pte_nr -1 ) < pvmw->pfn)return false; -if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )) +if (pfn > (pvmw->pfn + pvmw->nr_pages -1 ))return false;return true; }
@@ -149,12 +149,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)/* Returns true if the two ranges overlap. Careful to not overflow. */ static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { -if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +if ((device_private) ^ !!(pvmw->flags &PVMW_DEVICE_PRIVATE ))return false; -if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT )) +if ((pfn +HPAGE_PMD_NR -1 ) < pvmw->pfn)return false; -if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ) +if (pfn > pvmw->pfn + pvmw->nr_pages -1 )return false;return true; }
@@ -369,7 +369,7 @@ unsigned long page_mapped_in_vma(const struct page *page, .pfn = folio_page_vma_walk_pfn(folio), .nr_pages = 1, .vma = vma, - .flags = PVMW_SYNC, + .flags = page_vma_walk_flags(folio, PVMW_SYNC), }; pvmw.address =vma_address (vma,page_pgoff (folio, page),1 );
mm/vmscan.c +1 -1
@@ -4203,7 +4203,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)pte_t *pte = pvmw->pte;unsigned long addr = pvmw->address;struct vm_area_struct *vma = pvmw->vma; -struct folio *folio =page_vma_walk_pfn_to_folio (pvmw->pfn); +struct folio *folio =page_vma_walk_pfn_to_folio (pvmw);struct mem_cgroup *memcg =folio_memcg (folio);struct pglist_data *pgdat =folio_pgdat (folio);struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat);
Best Regards,
Yan, Zi
Re: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 280 lines · Zi Yan
Quoted 278 lines · Jordan Niethe
Hi,
Quoted 267 lines · Zi Yan
Quoted 265 lines · Jordan Niethe
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) · line 147
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.That is a good point.
Quoted 7 lines
Quoted 4 lines
I see a few options:
- On every arch we can assume
SWP_PFN_BITS? I could add a sanity check that we have an extra bit on top ofSWP_PFN_BITSwithin an unsigned long.Yes, but if there is no extra bit, are you going to disable device private pages?
In this case, migrate PFNs would also be broken (due to MIGRATE_PFN_SHIFT) so we'd have to.
Quoted 11 lines
Quoted 3 lines
- We could define
PVMW_PFN_SHIFTas 0 if !CONFIG_MIGRATION as the flag is not required.Sure, or !CONFIG_DEVICE_MIGRATION
Quoted 2 lines
- Instead of modifying
pvmw->pfnwe could usepvmw->flagsbut that has the issues of separating the offset type and offset.It seems that I was not clear on my proposal. Here is the patch on top of your patchset and it compiles.
Oh I'd interpreted “bool is_device_private” as adding a new field to pvmw.
Quoted 5 lines
Basically,
pvmw->pfnstores either PFN or device private offset without additional shift. Caller interpretspvmw->pfnbased onpvmw->flags&PVMW_DEVICE_PRIVATE. And you can ignore my helper function ofpvmw->pfnsuggestion, since my patch below can usepvmw->pfndirectly.
Thanks, looks reasonable. I'll try it.
Thanks,
Jordan.
Quoted 133 lines
Let me know if my patch works. Thanks.
include/linux/rmap.h +15
-14@@ -921,6 +921,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,#define PVMW_SYNC (1 << 0)/* Look for migration entries rather than present PTEs */ #define PVMW_MIGRATION (1 << 1) +#define PVMW_DEVICE_PRIVATE (1 << 2)/* Result flags */ @@ -943,6 +944,13 @@ struct page_vma_mapped_walk {#define PVMW_PFN_DEVICE_PRIVATE (1UL << 0)#define PVMW_PFN_SHIFT 1 +static inline unsigned long page_vma_walk_flags (struct folio *folio,unsigned long flags) +{ +if (folio_is_device_private (folio)) +return flags |PVMW_DEVICE_PRIVATE ; +return flags; +} +static inline unsigned long page_vma_walk_pfn (unsigned long pfn) {return (pfn <<PVMW_PFN_SHIFT );@@ -951,23 +959,16 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (device_private_folio_to_offset (folio)) | -PVMW_PFN_DEVICE_PRIVATE ; - -return page_vma_walk_pfn (folio_pfn (folio)); -} - -static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) -{ -if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) -return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return device_private_folio_to_offset (folio); -return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return (folio_pfn (folio)); } -static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +static inline struct folio *page_vma_walk_pfn_to_folio (struct page_vma_mapped_walk *pvmw) { -return page_folio (page_vma_walk_pfn_to_page (pvmw_pfn)); +if (pvmw->flags &PVMW_DEVICE_PRIVATE ) +return page_folio (device_private_offset_to_page (pvmw->pfn)); +return pfn_folio (pvmw->pfn); }#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \@@ -977,7 +978,7 @@ static inline struct folio *page_vma_walk_pfn_to_folio(unsigned long pvmw_pfn) .pgoff = folio_pgoff(_folio), \ .vma = _vma, \ .address = _address, \ - .flags = _flags, \ + .flags = page_vma_walk_flags(_folio, _flags), \ }static inline void page_vma_mapped_walk_done (struct page_vma_mapped_walk *pvmw)mm/page_vma_mapped.c +7
-7@@ -136,12 +136,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) device_private = true; } -if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +if ((device_private) ^ !!(pvmw->flags &PVMW_DEVICE_PRIVATE ))return false; -if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT )) +if ((pfn + pte_nr -1 ) < pvmw->pfn)return false; -if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )) +if (pfn > (pvmw->pfn + pvmw->nr_pages -1 ))return false;return true; }@@ -149,12 +149,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)/* Returns true if the two ranges overlap. Careful to not overflow. */ static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { -if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +if ((device_private) ^ !!(pvmw->flags &PVMW_DEVICE_PRIVATE ))return false; -if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT )) +if ((pfn +HPAGE_PMD_NR -1 ) < pvmw->pfn)return false; -if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ) +if (pfn > pvmw->pfn + pvmw->nr_pages -1 )return false;return true; }@@ -369,7 +369,7 @@ unsigned long page_mapped_in_vma(const struct page *page, .pfn = folio_page_vma_walk_pfn(folio), .nr_pages = 1, .vma = vma, - .flags = PVMW_SYNC, + .flags = page_vma_walk_flags(folio, PVMW_SYNC), }; pvmw.address =vma_address (vma,page_pgoff (folio, page),1 );mm/vmscan.c +1
-1@@ -4203,7 +4203,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)pte_t *pte = pvmw->pte;unsigned long addr = pvmw->address;struct vm_area_struct *vma = pvmw->vma; -struct folio *folio =page_vma_walk_pfn_to_folio (pvmw->pfn); +struct folio *folio =page_vma_walk_pfn_to_folio (pvmw);struct mem_cgroup *memcg =folio_memcg (folio);struct pglist_data *pgdat =folio_pgdat (folio);struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat);Best Regards,
Yan, Zi
Re: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Hi,
Quoted 311 lines · Jordan Niethe
Quoted 280 lines · Zi Yan
Quoted 278 lines · Jordan Niethe
Hi,
Quoted 267 lines · Zi Yan
Quoted 265 lines · Jordan Niethe
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);
---
Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++---drivers/gpu/drm/amd/amdkfd/kfd_migrate.c| 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +---
include/linux/hmm.h | 3 +
include/linux/leafops.h | 16 +-
include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +-
include/linux/mm.h | 2 +
include/linux/rmap.h | 5 +-
include/linux/swapops.h | 10 +-
lib/test_hmm.c | 69 ++++----
mm/debug.c | 9 +-
mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +-
mm/page_vma_mapped.c | 19 ++-
mm/rmap.c | 43 +++--
mm/util.c | 5 +-19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn(const struct folio *folio) { + VM_BUG_ON(folio_is_device_private(folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+ return page_to_pfn(&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn) static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio) { if (folio_is_device_private(folio)) - return page_vma_walk_pfn(folio_pfn(folio)) | + return page_vma_walk_pfn(device_private_folio_to_offset(folio)) | PVMW_PFN_DEVICE_PRIVATE; return page_vma_walk_pfn(folio_pfn(folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio) static inline struct page *page_vma_walk_pfn_to_page(unsigned long pvmw_pfn) { + if (pvmw_pfn & PVMW_PFN_DEVICE_PRIVATE) + return device_private_offset_to_page(pvmw_pfn >> PVMW_PFN_SHIFT); + return pfn_to_page(pvmw_pfn >> PVMW_PFN_SHIFT); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +14
-1@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) { unsigned long pfn; + bool device_private = false; pte_t ptent = ptep_get(pvmw->pte); if (pvmw->flags & PVMW_MIGRATION) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) if (!softleaf_is_migration(entry)) return false; + if (softleaf_is_migration_device_private(entry)) + device_private = true; + pfn = softleaf_to_pfn(entry); } else if (pte_present(ptent)) { pfn = pte_pfn(ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) return false; pfn = softleaf_to_pfn(entry); + + if (softleaf_is_device_private(entry)) + device_private = true; } + if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE)) + return false; + if ((pfn + pte_nr - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT)) return false; if (pfn > ((pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) } /* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { + if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE)) + return false; + if ((pfn + HPAGE_PMD_NR - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT)) return false; if (pfn > (pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1)@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration(entry) || !check_pmd(softleaf_to_pfn(entry), +
softleaf_is_device_private(entry)|| +softleaf_is_migration_device_private(entry), pvmw))
returnnot_found(pvmw);
return true;mm/page_vma_mapped.c +0
-1@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (likely(pmd_trans_huge(pmde))) { if (pvmw->flags & PVMW_MIGRATION) return not_found(pvmw); - if (!check_pmd(pmd_pfn(pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+ if (!check_pmd(pmd_pfn(pmde), false, pvmw)) return not_found(pvmw); return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.That is a good point.
Quoted 7 lines
Quoted 4 lines
I see a few options:
- On every arch we can assume
SWP_PFN_BITS? I could add a sanity check that we have an extra bit on top ofSWP_PFN_BITSwithin an unsigned long.Yes, but if there is no extra bit, are you going to disable device private pages?
In this case, migrate PFNs would also be broken (due to
MIGRATE_PFN_SHIFT) so we'd have to.Quoted 11 lines
Quoted 3 lines
- We could define
PVMW_PFN_SHIFTas 0 if !CONFIG_MIGRATION as the flag is not required.Sure, or !CONFIG_DEVICE_MIGRATION
Quoted 2 lines
- Instead of modifying
pvmw->pfnwe could usepvmw->flagsbut that has the issues of separating the offset type and offset.It seems that I was not clear on my proposal. Here is the patch on top of your patchset and it compiles.
Oh I'd interpreted “bool is_device_private” as adding a new field to pvmw.
Quoted 5 lines
Basically,
pvmw->pfnstores either PFN or device private offset without additional shift. Caller interpretspvmw->pfnbased onpvmw->flags&PVMW_DEVICE_PRIVATE. And you can ignore my helper function ofpvmw->pfnsuggestion, since my patch below can usepvmw->pfndirectly.
Quoted 7 lines
Thanks, looks reasonable. I'll try it.
Thanks,
Jordan.Quoted 2 lines
Let me know if my patch works. Thanks.
We need to be careful now to ensure the PVMW_DEVICE_PRIVATE flag doesn't get overwritten:
mm/rmap.c +2 -2
@@ -1871,7 +1871,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* if page table locking is skipped: use TTU_SYNC to wait for that. */ if (flags &TTU_SYNC ) - pvmw.flags =PVMW_SYNC ; + pvmw.flags =page_vma_walk_flags (folio,PVMW_SYNC );/* * For THP, we have to assume the worse case ie pmd for invalidation.
@@ -2304,7 +2304,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* if page table locking is skipped: use TTU_SYNC to wait for that. */ if (flags &TTU_SYNC ) - pvmw.flags =PVMW_SYNC ; + pvmw.flags =page_vma_walk_flags (folio,PVMW_SYNC );/* * For THP, we have to assume the worse case ie pmd for invalidation.
Other than that tests ok.
Thanks,
Jordan.
Quoted 132 lines
Quoted 131 lines
include/linux/rmap.h +15
-14@@ -921,6 +921,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, #define PVMW_SYNC (1 << 0) /* Look for migration entries rather than present PTEs */ #define PVMW_MIGRATION (1 << 1) +#define PVMW_DEVICE_PRIVATE (1 << 2) /* Result flags */@@ -943,6 +944,13 @@ struct page_vma_mapped_walk { #define PVMW_PFN_DEVICE_PRIVATE (1UL << 0) #define PVMW_PFN_SHIFT 1 +static inline unsigned long page_vma_walk_flags (struct folio *folio,unsigned long flags) +{ + if (folio_is_device_private(folio)) + return flags | PVMW_DEVICE_PRIVATE; + return flags; +} + static inline unsigned long page_vma_walk_pfn(unsigned long pfn) { return (pfn << PVMW_PFN_SHIFT);@@ -951,23 +959,16 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn) static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio) { if (folio_is_device_private(folio)) - return page_vma_walk_pfn(device_private_folio_to_offset(folio)) | - PVMW_PFN_DEVICE_PRIVATE; - - return page_vma_walk_pfn(folio_pfn(folio)); -} - -static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) -{ - if (pvmw_pfn & PVMW_PFN_DEVICE_PRIVATE) - return device_private_offset_to_page(pvmw_pfn >> PVMW_PFN_SHIFT); + return device_private_folio_to_offset(folio); - return pfn_to_page(pvmw_pfn >> PVMW_PFN_SHIFT); + return (folio_pfn(folio)); } -static inline struct folio *page_vma_walk_pfn_to_folio (unsigned long pvmw_pfn) +static inline struct folio *page_vma_walk_pfn_to_folio (struct page_vma_mapped_walk *pvmw) { - return page_folio(page_vma_walk_pfn_to_page(pvmw_pfn)); + if (pvmw->flags & PVMW_DEVICE_PRIVATE) + return page_folio(device_private_offset_to_page(pvmw->pfn)); + return pfn_folio(pvmw->pfn); } #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags) \@@ -977,7 +978,7 @@ static inline struct folio *page_vma_walk_pfn_to_folio(unsigned long pvmw_pfn) .pgoff = folio_pgoff(_folio), \ .vma = _vma, \ .address = _address, \ - .flags = _flags, \ + .flags = page_vma_walk_flags(_folio, _flags), \ } static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw)mm/page_vma_mapped.c +7
-7@@ -136,12 +136,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) device_private = true; } - if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE)) + if ((device_private) ^ !!(pvmw->flags & PVMW_DEVICE_PRIVATE)) return false; - if ((pfn + pte_nr - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT)) + if ((pfn + pte_nr - 1) < pvmw->pfn) return false; - if (pfn > ((pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1)) + if (pfn > (pvmw->pfn + pvmw->nr_pages - 1)) return false; return true; }@@ -149,12 +149,12 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) /* Returns true if the two ranges overlap. Careful to not overflow. */ static bool check_pmd(unsigned long pfn, bool device_private, struct page_vma_mapped_walk *pvmw) { - if ((device_private) ^ !!(pvmw->pfn & PVMW_PFN_DEVICE_PRIVATE)) + if ((device_private) ^ !!(pvmw->flags & PVMW_DEVICE_PRIVATE)) return false; - if ((pfn + HPAGE_PMD_NR - 1) < (pvmw->pfn >> PVMW_PFN_SHIFT)) + if ((pfn + HPAGE_PMD_NR - 1) < pvmw->pfn) return false; - if (pfn > (pvmw->pfn >> PVMW_PFN_SHIFT) + pvmw->nr_pages - 1) + if (pfn > pvmw->pfn + pvmw->nr_pages - 1) return false; return true; }@@ -369,7 +369,7 @@ unsigned long page_mapped_in_vma(const struct page *page, .pfn = folio_page_vma_walk_pfn(folio), .nr_pages = 1, .vma = vma, - .flags = PVMW_SYNC, + .flags = page_vma_walk_flags(folio, PVMW_SYNC), }; pvmw.address = vma_address(vma, page_pgoff(folio, page), 1);mm/vmscan.c +1
-1@@ -4203,7 +4203,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; struct vm_area_struct *vma = pvmw->vma; - struct folio *folio = page_vma_walk_pfn_to_folio(pvmw->pfn); + struct folio *folio = page_vma_walk_pfn_to_folio(pvmw); struct mem_cgroup *memcg = folio_memcg(folio); struct pglist_data *pgdat = folio_pgdat(folio); struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);Best Regards,
Yan, Zi
Alistair Popple
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
In reply to Zi YanOn 2026-01-21 at 10:06 +1100, Zi Yan <[email protected]> wrote...
Quoted 267 lines · Jordan Niethe
Quoted 266 lines
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.
We need to be a bit careful by what we mean when we say "HMM" in the kernel.
Specifically MIGRATE_PFN_SHIFT is used with migrate_vma/migrate_device, which is the migration half of "HMM" which does depend on CONFIG_DEVICE_MIGRATION or really just CONFIG_ZONE_DEVICE making it somewhat arch specific.
However hmm_range_fault() does something similar - see the definition of hmm_pfn_flags - it actually steals the top 11 bits of a pfn for flags, and it is not architecture specific. It only depends on CONFIG_MMU.
Now I'm not saying this implies it actually works on all architectures as I agree the page_vma_mapped_walk code is used much more widely. Rather I'm just pointing out if there are issues with some architectures using high PFN bits then we likely have a problem here too :-)
- Alistair
Quoted 2 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 276 lines · Alistair Popple
On 2026-01-21 at 10:06 +1100, Zi Yan <[email protected]> wrote...
Quoted 267 lines · Jordan Niethe
Quoted 266 lines
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.We need to be a bit careful by what we mean when we say "HMM" in the kernel.
Quoted 7 lines
Specifically
MIGRATE_PFN_SHIFTis used with migrate_vma/migrate_device, which is the migration half of "HMM" which does depend onCONFIG_DEVICE_MIGRATIONor really justCONFIG_ZONE_DEVICEmaking it somewhat arch specific.However
hmm_range_fault()does something similar - see the definition ofhmm_pfn_flags- it actually steals the top 11 bits of a pfn for flags, and it is not architecture specific. It only depends onCONFIG_MMU.
Oh, that is hacky. But are HMM PFNs with any flag exposed to code outside HMM? Currently, device private needs to reserve PFNs for struct page, so I assume only the reserved PFNs are seen by outsiders. Otherwise, when outsiders see a HMM PFN with a flag, pfn_to_page() on such a PFN will read non exist struct page, right?
For this page_vma_mapped_walk code, it is manipulating PFNs used by everyone, not just HMM, and can potentially (might be very rare) alter their values after shifts. And if an HMM PFN with HMM_PFN_VALID is processed by the code, the HMM PFN will lose HMM_PFN_VALID bit. So I guess HMM PFN is not showing outside HMM code.
Quoted 5 lines
Now I'm not saying this implies it actually works on all architectures as I agree the
page_vma_mapped_walkcode is used much more widely. Rather I'm just pointing out if there are issues with some architectures using high PFN bits then we likely have a problem here too :-)
Best Regards,
Yan, Zi
Zi Yan
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 292 lines · Zi Yan
Quoted 276 lines · Alistair Popple
On 2026-01-21 at 10:06 +1100, Zi Yan <[email protected]> wrote...
Quoted 267 lines · Jordan Niethe
Quoted 266 lines
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.We need to be a bit careful by what we mean when we say "HMM" in the kernel.
Quoted 7 lines
Specifically
MIGRATE_PFN_SHIFTis used with migrate_vma/migrate_device, which is the migration half of "HMM" which does depend onCONFIG_DEVICE_MIGRATIONor really justCONFIG_ZONE_DEVICEmaking it somewhat arch specific.However
hmm_range_fault()does something similar - see the definition ofhmm_pfn_flags- it actually steals the top 11 bits of a pfn for flags, and it is not architecture specific. It only depends onCONFIG_MMU.Oh, that is hacky. But are HMM PFNs with any flag exposed to code outside HMM? Currently, device private needs to reserve PFNs for struct page, so I assume only the reserved PFNs are seen by outsiders. Otherwise, when outsiders see a HMM PFN with a flag,
pfn_to_page()on such a PFN will read non exist struct page, right?
Quoted 8 lines
For this
page_vma_mapped_walkcode, it is manipulating PFNs used by everyone, not just HMM, and can potentially (might be very rare) alter their values after shifts.And if an HMM PFN with
HMM_PFN_VALIDis processed by the code, the HMM PFN will loseHMM_PFN_VALIDbit. So I guess HMM PFN is not showing outside HMM code.
Oops, this code is removing PFN reservation mechanism, so please ignore the above two sentences.
Quoted 10 lines
Quoted 6 lines
Now I'm not saying this implies it actually works on all architectures as I agree the
page_vma_mapped_walkcode is used much more widely. Rather I'm just pointing out if there are issues with some architectures using high PFN bits then we likely have a problem here too :-)Best Regards,
Yan, Zi
Best Regards,
Yan, Zi
Alistair Popple
reviewerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
On 2026-01-23 at 14:09 +1100, Zi Yan <[email protected]> wrote...
Quoted 288 lines · Zi Yan
Quoted 287 lines · Alistair Popple
Quoted 284 lines
On 2026-01-21 at 10:06 +1100, Zi Yan <[email protected]> wrote...
Quoted 267 lines · Jordan Niethe
Quoted 266 lines
Hi,
Quoted 247 lines · Zi Yan
Quoted 245 lines · Jordan Niethe
Quoted 123 lines · Zi Yan
Quoted 119 lines · Jordan Niethe
Currently when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages().Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 meaning the feature does not work there.
Instead of using the physical address space, introduce a device private address space and allocate devices regions from there to represent the device private pages.
Introduce a new interface
memremap_device_private_pagemap()that allocates a requested amount of device private address space and creates the necessary device private pages.To support this new interface, struct
dev_pagemapneeds some changes:- Add a new
dev_pagemap::nr_pagesfield as an input parameter.
- Add a newdev_pagemap::pagesarray to store the device private pages.When using
memremap_device_private_pagemap(), rather then passing in dev_pagemap::ranges[dev_pagemap::nr_ranges] of physical address space to be remapped,dev_pagemap::nr_rangeswill always be 1, and the device private range that is reserved is returned indev_pagemap::range.Forbid calling
memremap_pages()withdev_pagemap::ranges::type=MEMORY_DEVICE_PRIVATE.Represent this device private address space using a new
device_private_pgmap_treemaple tree. This tree maps a given device private address to a structdev_pagemap, where a specific device private page may then be looked up in thatdev_pagemap::pagesarray.Device private address space can be reclaimed and the assoicated device private pages freed using the corresponding new
memunmap_device_private_pagemap()interface.Because the device private pages now live outside the physical address space, they no longer have a normal PFN. This means that
page_to_pfn(), et al. are no longer meaningful.Introduce helpers:
-
device_private_page_to_offset()
-device_private_folio_to_offset()to take a given device private page / folio and return its offset within the device private address space.
Update the places where we previously converted a device private page to a PFN to use these new helpers. When we encounter a device private offset, instead of looking up its page within the pagemap use
device_private_offset_to_page()instead.Update the existing users:
-
lib/test_hmm.c
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveauto use the new
memremap_device_private_pagemap()interface.Signed-off-by Jordan Niethe <[email protected]>Signed-off-by Alistair Popple <[email protected]>---
NOTE: The updates to the existing drivers have only been compile tested. I'll need some help in testing these drivers.
v1:
- Include NUMA node paramater formemremap_device_private_pagemap()
- Adddevm_memremap_device_private_pagemap()and friends
- Update existing users ofmemremap_pages():
- ppc ultravisor
- drm/amd/amdkfd
- gpu/drm/xe
- gpu/drm/nouveau
- Update for HMM huge page support
- Guarddevice_private_offset_to_pageand friends withCONFIG_ZONE_DEVICEv2:
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);--- Documentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 41 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 23 +-- drivers/gpu/drm/nouveau/nouveau_dmem.c | 35 ++-- drivers/gpu/drm/xe/xe_svm.c | 28 +--- include/linux/hmm.h | 3 + include/linux/leafops.h | 16 +- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 6 +- include/linux/mm.h | 2 + include/linux/rmap.h | 5 +- include/linux/swapops.h | 10 +- lib/test_hmm.c | 69 ++++---- mm/debug.c | 9 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/mm_init.c | 8 +- mm/page_vma_mapped.c | 19 ++- mm/rmap.c | 43 +++-- mm/util.c | 5 +- 19 files changed, 391 insertions(+), 199 deletions(-)<snip>
Quoted 11 lines
include/linux/mm.h static inline unsigned long memdesc_section(memdesc_flags_t mdf) · line 2041
Quoted 9 lines
include/linux/mm.h +1
-0@@ -2038,6 +2038,8 @@ static inline unsigned long memdesc_section(memdesc_flags_t mdf)*/ static inline unsigned long folio_pfn (const struct folio *folio) { +VM_BUG_ON (folio_is_device_private (folio));Please use
VM_WARN_ONinstead.ack.
Quoted 94 lines
Quoted 27 lines
include/linux/mm.h +1
-0+return page_to_pfn (&folio->page); }include/linux/rmap.h +4
-1@@ -951,7 +951,7 @@ static inline unsigned long page_vma_walk_pfn(unsigned long pfn)static inline unsigned long folio_page_vma_walk_pfn (const struct folio *folio) {if (folio_is_device_private (folio)) -return page_vma_walk_pfn (folio_pfn (folio)) | +return page_vma_walk_pfn (device_private_folio_to_offset (folio)) |PVMW_PFN_DEVICE_PRIVATE ;return page_vma_walk_pfn (folio_pfn (folio));@@ -959,6 +959,9 @@ static inline unsigned long folio_page_vma_walk_pfn(const struct folio *folio)static inline struct page *page_vma_walk_pfn_to_page (unsigned long pvmw_pfn) { +if (pvmw_pfn &PVMW_PFN_DEVICE_PRIVATE ) +return device_private_offset_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); +return pfn_to_page (pvmw_pfn >>PVMW_PFN_SHIFT ); }<snip>
Quoted 64 lines
mm/page_vma_mapped.c +16
-2@@ -107,6 +107,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,static bool check_pte (struct page_vma_mapped_walk *pvmw,unsigned long pte_nr) {unsigned long pfn; +bool device_private = false;pte_t ptent =ptep_get (pvmw->pte);if (pvmw->flags &PVMW_MIGRATION ) {@@ -115,6 +116,9 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)if (!softleaf_is_migration (entry))return false; +if (softleaf_is_migration_device_private (entry)) + device_private = true; + pfn =softleaf_to_pfn (entry); }else if (pte_present (ptent)) { pfn =pte_pfn (ptent);@@ -127,8 +131,14 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)return false; pfn =softleaf_to_pfn (entry); + +if (softleaf_is_device_private (entry)) + device_private = true; } +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn + pte_nr -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > ((pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 ))@@ -137,8 +147,11 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr) }/* Returns true if the two ranges overlap. Careful to not overflow. */ -static bool check_pmd (unsigned long pfn,struct page_vma_mapped_walk *pvmw) +static bool check_pmd (unsigned long pfn,bool device_private,struct page_vma_mapped_walk *pvmw) { +if ((device_private) ^ !!(pvmw->pfn &PVMW_PFN_DEVICE_PRIVATE )) +return false; +if ((pfn +HPAGE_PMD_NR -1 ) < (pvmw->pfn >>PVMW_PFN_SHIFT ))return false;if (pfn > (pvmw->pfn >>PVMW_PFN_SHIFT ) + pvmw->nr_pages -1 )@@ -255,6 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!softleaf_is_migration (entry) || !check_pmd (softleaf_to_pfn (entry), +softleaf_is_device_private (entry) || +softleaf_is_migration_device_private (entry), pvmw))return not_found (pvmw);return true;@@ -262,7 +277,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (likely (pmd_trans_huge (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw); -if (!check_pmd (pmd_pfn (pmde), pvmw))Quoted 8 lines
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 283
Quoted 4 lines
mm/page_vma_mapped.c +1
-0+if (!check_pmd (pmd_pfn (pmde), false, pvmw))return not_found (pvmw);return true; }It seems to me that you can add a new flag like “bool is_device_private” to indicate whether pfn is a device private index instead of pfn without manipulating
pvmw->pfnitself.We could do it like that, however my concern with using a new param was that storing this info seperately might make it easier to misuse a device private index as a regular pfn.
Quoted 7 lines
Quoted 2 lines
It seemed like it could be easy to overlook both when creating the pvmw and then when accessing the pfn.
That is why I asked for a helper function like
page_vma_walk_pfn(pvmw)to return the converted pfn instead ofpvmw->pfndirectly. You can add a comment to ask people to use helper function and even markpvmw->pfn/* do not use directly */.Yeah I agree that is a good idea.
Quoted 4 lines
In addition, your patch manipulates pfn by left shifting it by 1. Are you sure there is no weird arch having pfns with bit 63 being 1? Your change could break it, right?
Quoted 5 lines
Quoted 2 lines
Currently for migrate pfns we left shift by pfns by
MIGRATE_PFN_SHIFT(6), so I thought doing something similiar here should be safe.Yeah, but that limits to archs supporting HMM.
page_vma_mapped_walkis used by almost every arch, so it has a broader impact.We need to be a bit careful by what we mean when we say "HMM" in the kernel.
Specifically
MIGRATE_PFN_SHIFTis used with migrate_vma/migrate_device, which is the migration half of "HMM" which does depend onCONFIG_DEVICE_MIGRATIONor really justCONFIG_ZONE_DEVICEmaking it somewhat arch specific.However
hmm_range_fault()does something similar - see the definition ofhmm_pfn_flags- it actually steals the top 11 bits of a pfn for flags, and it is not architecture specific. It only depends onCONFIG_MMU.
Quoted 5 lines
Quoted 5 lines
Oh, that is hacky. But are HMM PFNs with any flag exposed to code outside HMM? Currently, device private needs to reserve PFNs for struct page, so I assume only the reserved PFNs are seen by outsiders. Otherwise, when outsiders see a HMM PFN with a flag,
pfn_to_page()on such a PFN will read non exist struct page, right?
Any user of hmm_range_fault() would be exposed to an issue - most users of hmm_range_fault() use it to grab a PFN (ie. physical address) to map into some remote page table. So potentially if some important bit in the PFN is dropped that could potentially result in users mapping the wrong physical address or page.
And just to be clear this is completely orthogonal to any DEVICE_PRIVATE specific issue - it existed well before any changes here. Of course it may not actually be an issue - do we know if there are any architectures that actually use upper physical address bits? I assume not because how would they fit in a page table entry. But I don't really know.
Quoted 26 lines
Quoted 8 lines
For this
page_vma_mapped_walkcode, it is manipulating PFNs used by everyone, not just HMM, and can potentially (might be very rare) alter their values after shifts.And if an HMM PFN with
HMM_PFN_VALIDis processed by the code, the HMM PFN will loseHMM_PFN_VALIDbit. So I guess HMM PFN is not showing outside HMM code.Oops, this code is removing PFN reservation mechanism, so please ignore the above two sentences.
Quoted 10 lines
Quoted 6 lines
Now I'm not saying this implies it actually works on all architectures as I agree the
page_vma_mapped_walkcode is used much more widely. Rather I'm just pointing out if there are issues with some architectures using high PFN bits then we likely have a problem here too :-)Best Regards,
Yan, ZiBest Regards,
Yan, Zi
Jason Gunthorpe
maintainerRe: [PATCH v2 11/11] mm: Remove device private pages from the physical address space
Quoted 16 lines · Alistair Popple
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
We need to be a bit careful by what we mean when we say "HMM" in the kernel.
Specifically
MIGRATE_PFN_SHIFTis used with migrate_vma/migrate_device, which is the migration half of "HMM" which does depend onCONFIG_DEVICE_MIGRATIONor really justCONFIG_ZONE_DEVICEmaking it somewhat arch specific.However
hmm_range_fault()does something similar - see the definition ofhmm_pfn_flags- it actually steals the top 11 bits of a pfn for flags, and it is not architecture specific. It only depends onCONFIG_MMU.Quoted 5 lines
Quoted 5 lines
Oh, that is hacky. But are HMM PFNs with any flag exposed to code outside HMM? Currently, device private needs to reserve PFNs for struct page, so I assume only the reserved PFNs are seen by outsiders. Otherwise, when outsiders see a HMM PFN with a flag,
pfn_to_page()on such a PFN will read non exist struct page, right?
Quoted 5 lines
Any user of
hmm_range_fault()would be exposed to an issue - most users ofhmm_range_fault()use it to grab a PFN (ie. physical address) to map into some remote page table. So potentially if some important bit in the PFN is dropped that could potentially result in users mapping the wrong physical address or page.
Trim the quotes guys..
hmm is arguably returning phys_addr_t >> PAGE_SHIFT. This is a lossless translation because everything is aligned, it isn't hacky.
The value it returns is not a "pfn", it is a hmm structure that has to be decoded to something else using a hmm helper function.
I think we take a number of liberties going between pte, phys_addr_t, pfn. If there are arches that use a special encoding for the mm PFN then range_fault would need to call converter functions to get to/from phys_addr_t.
Jason
Matthew Brost
maintainerQuoted 106 lines · Jordan Niethe
Today, when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages.Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
This series changes device private memory so that it does not require allocation of physical address space and these problems are avoided. Instead of using the physical address space, we introduce a "device private address space" and allocate from there.
A consequence of placing the device private pages outside of the physical address space is that they no longer have a PFN. However, it is still necessary to be able to look up a corresponding device private page from a device private PTE entry, which means that we still require some way to index into this device private address space. Instead of a PFN, device private pages use an offset into this device private address space to look up device private struct pages.
The problem that then needs to be addressed is how to avoid confusing these device private offsets with PFNs. It is the inherent limited usage of the device private pages themselves which make this possible. A device private page is only used for userspace mappings, we do not need to be concerned with them being used within the mm more broadly. This means that the only way that the core kernel looks up these pages is via the page table, where their PTE already indicates if they refer to a device private page via their swap type, e.g.
SWP_DEVICE_WRITE. We can use this information to determine if the PTE contains a PFN which should be looked up in the page map, or a device private offset which should be looked up elsewhere.This applies when we are creating PTE entries for device private pages - because they have their own type there are already must be handled separately, so it is a small step to convert them to a device private PFN now too.
The first part of the series updates callers where device private offsets might now be encountered to track this extra state.
The last patch contains the bulk of the work where we change how we convert between device private pages to device private offsets and then use a new interface for allocating device private pages without the need for reserving physical address space.
By removing the device private pages from the physical address space, this series also opens up the possibility to moving away from tracking device private memory using struct pages in the future. This is desirable as on systems with large amounts of memory these device private struct pages use a signifiant amount of memory and take a significant amount of time to initialize.
*** Changes in v2 ***
The most significant change in v2 is addressing code paths that are common between
MEMORY_DEVICE_PRIVATEandMEMORY_DEVICE_COHERENTdevices.This had been overlooked in previous revisions.
To do this we introduce a
migrate_pfn_from_page()helper which will calldevice_private_offset_to_page()and set theMIGRATE_PFN_DEVICE_PRIVATEflag if required.In places where we could have a device private offset (
MEMORY_DEVICE_PRIVATE) or a pfn (MEMORY_DEVICE_COHERENT) we update to use an mpfn to disambiguate. This includes some users in the drivers and migrate_device_{pfns,range}().Seeking opinions on using the mpfns like this or if a new type would be preferred.
-
mm/migrate_device: Introducemigrate_pfn_from_page()helper
- New to series- drm/amdkfd: Use migrate pfns internally
- New to series-
mm/migrate_device: Make migrate_device_{pfns,range}() take mpfns
- New to series-
mm/migrate_device: Add migrate PFN flag to track device private pages
- Update formigrate_pfn_from_page()
- Rename toMIGRATE_PFN_DEVICE_PRIVATE
- drm/amd: Checkadev->gmc.xgmi.connected_to_cpu
-lib/test_hmm.c: Checkchunk->pagemap.type==MEMORY_DEVICE_PRIVATE- mm: Add helpers to create migration entries from struct pages
- Add a flags param- mm: Add a new swap type for migration entries of device private pages
- Addsoftleaf_is_migration_device_private_read()- mm: Add helpers to create device private entries from struct pages
- Add a flags param
Quoted 8 lines
- mm: Remove device private pages from the physical address space
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);Testing:
- selftests/mm/hmm-tests on an amd64 VM* NOTE: I will need help in testing the driver changes *
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CI results.
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?
Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Quoted 59 lines
Revisions:
- RFC: https://lore.kernel.org/all/[email protected]/
- v1: https://lore.kernel.org/all/[email protected]/[0] https://lore.kernel.org/lkml/CAMj1kXFZ=4hLL1w6iCV5O5uVoVLHAJbc0rr40j24ObenAjXe9w@mail.gmail.com/
Jordan Niethe (11):
mm/migrate_device: Introducemigrate_pfn_from_page()helper drm/amdkfd: Use migrate pfns internally
mm/migrate_device: Make migrate_device_{pfns,range}() take mpfnsmm/migrate_device: Add migrate PFN flag to track device private pagesmm/page_vma_mapped: Add flags topage_vma_mapped_walk::pfnto track
device private pages
mm: Add helpers to create migration entries from struct pages
mm: Add a new swap type for migration entries of device private pages
mm: Add helpers to create device private entries from struct pages mm/util: Add flag to track device private pages in page snapshots mm/hmm: Add flag to track device private pages
mm: Remove device private pages from the physical address spaceDocumentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 43 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 45 +++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 2 +- drivers/gpu/drm/drm_pagemap.c | 11 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 45 ++---- drivers/gpu/drm/xe/xe_svm.c | 37 ++--- fs/proc/page.c | 6 +- include/drm/drm_pagemap.h | 8 +- include/linux/hmm.h | 7 +- include/linux/leafops.h | 116 ++++++++++++-- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 23 ++- include/linux/mm.h | 9 +- include/linux/rmap.h | 33 +++- include/linux/swap.h | 8 +- include/linux/swapops.h | 136 ++++++++++++++++ lib/test_hmm.c | 86 ++++++---- mm/debug.c | 9 +- mm/hmm.c | 5 +- mm/huge_memory.c | 43 ++--- mm/hugetlb.c | 15 +- mm/memory.c | 5 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/migrate.c | 6 +- mm/migrate_device.c | 76 +++++---- mm/mm_init.c | 8 +- mm/mprotect.c | 10 +- mm/page_vma_mapped.c | 32 +++- mm/rmap.c | 59 ++++--- mm/util.c | 8 +- mm/vmscan.c | 2 +- 32 files changed, 822 insertions(+), 339 deletions(-)base-commit:
f8f9c1f4d0c7a64600e2ca312dec824a0bc2f1daSignature · 1 line
2.34.1
Zi Yan
reviewerQuoted 123 lines · Matthew Brost
Quoted 106 lines · Jordan Niethe
Today, when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages.Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
This series changes device private memory so that it does not require allocation of physical address space and these problems are avoided. Instead of using the physical address space, we introduce a "device private address space" and allocate from there.
A consequence of placing the device private pages outside of the physical address space is that they no longer have a PFN. However, it is still necessary to be able to look up a corresponding device private page from a device private PTE entry, which means that we still require some way to index into this device private address space. Instead of a PFN, device private pages use an offset into this device private address space to look up device private struct pages.
The problem that then needs to be addressed is how to avoid confusing these device private offsets with PFNs. It is the inherent limited usage of the device private pages themselves which make this possible. A device private page is only used for userspace mappings, we do not need to be concerned with them being used within the mm more broadly. This means that the only way that the core kernel looks up these pages is via the page table, where their PTE already indicates if they refer to a device private page via their swap type, e.g.
SWP_DEVICE_WRITE. We can use this information to determine if the PTE contains a PFN which should be looked up in the page map, or a device private offset which should be looked up elsewhere.This applies when we are creating PTE entries for device private pages - because they have their own type there are already must be handled separately, so it is a small step to convert them to a device private PFN now too.
The first part of the series updates callers where device private offsets might now be encountered to track this extra state.
The last patch contains the bulk of the work where we change how we convert between device private pages to device private offsets and then use a new interface for allocating device private pages without the need for reserving physical address space.
By removing the device private pages from the physical address space, this series also opens up the possibility to moving away from tracking device private memory using struct pages in the future. This is desirable as on systems with large amounts of memory these device private struct pages use a signifiant amount of memory and take a significant amount of time to initialize.
*** Changes in v2 ***
The most significant change in v2 is addressing code paths that are common between
MEMORY_DEVICE_PRIVATEandMEMORY_DEVICE_COHERENTdevices.This had been overlooked in previous revisions.
To do this we introduce a
migrate_pfn_from_page()helper which will calldevice_private_offset_to_page()and set theMIGRATE_PFN_DEVICE_PRIVATEflag if required.In places where we could have a device private offset (
MEMORY_DEVICE_PRIVATE) or a pfn (MEMORY_DEVICE_COHERENT) we update to use an mpfn to disambiguate. This includes some users in the drivers and migrate_device_{pfns,range}().Seeking opinions on using the mpfns like this or if a new type would be preferred.
-
mm/migrate_device: Introducemigrate_pfn_from_page()helper
- New to series- drm/amdkfd: Use migrate pfns internally
- New to series-
mm/migrate_device: Make migrate_device_{pfns,range}() take mpfns
- New to series-
mm/migrate_device: Add migrate PFN flag to track device private pages
- Update formigrate_pfn_from_page()
- Rename toMIGRATE_PFN_DEVICE_PRIVATE
- drm/amd: Checkadev->gmc.xgmi.connected_to_cpu
-lib/test_hmm.c: Checkchunk->pagemap.type==MEMORY_DEVICE_PRIVATE- mm: Add helpers to create migration entries from struct pages
- Add a flags param- mm: Add a new swap type for migration entries of device private pages
- Addsoftleaf_is_migration_device_private_read()- mm: Add helpers to create device private entries from struct pages
- Add a flags paramQuoted 8 lines
- mm: Remove device private pages from the physical address space
- Make sure last member of structdev_pagemapremainsDECLARE_FLEX_ARRAY(struct range, ranges);Testing:
- selftests/mm/hmm-tests on an amd64 VM* NOTE: I will need help in testing the driver changes *
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CI results.
Quoted 2 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?
I will take a look. But next time, do you mind Ccing MM maintainers and reviewers based on MAINTAINERS file? Otherwise, it is hard for people to check every email from linux-mm.
Thanks.
Quoted 65 lines
Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1Quoted 59 lines
Revisions:
- RFC: https://lore.kernel.org/all/[email protected]/
- v1: https://lore.kernel.org/all/[email protected]/[0] https://lore.kernel.org/lkml/CAMj1kXFZ=4hLL1w6iCV5O5uVoVLHAJbc0rr40j24ObenAjXe9w@mail.gmail.com/
Jordan Niethe (11):
mm/migrate_device: Introducemigrate_pfn_from_page()helper drm/amdkfd: Use migrate pfns internally
mm/migrate_device: Make migrate_device_{pfns,range}() take mpfnsmm/migrate_device: Add migrate PFN flag to track device private pagesmm/page_vma_mapped: Add flags topage_vma_mapped_walk::pfnto track
device private pages
mm: Add helpers to create migration entries from struct pages
mm: Add a new swap type for migration entries of device private pages
mm: Add helpers to create device private entries from struct pages mm/util: Add flag to track device private pages in page snapshots mm/hmm: Add flag to track device private pages
mm: Remove device private pages from the physical address spaceDocumentation/mm/hmm.rst | 11 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 43 ++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 45 +++--- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 2 +- drivers/gpu/drm/drm_pagemap.c | 11 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 45 ++---- drivers/gpu/drm/xe/xe_svm.c | 37 ++--- fs/proc/page.c | 6 +- include/drm/drm_pagemap.h | 8 +- include/linux/hmm.h | 7 +- include/linux/leafops.h | 116 ++++++++++++-- include/linux/memremap.h | 64 +++++++- include/linux/migrate.h | 23 ++- include/linux/mm.h | 9 +- include/linux/rmap.h | 33 +++- include/linux/swap.h | 8 +- include/linux/swapops.h | 136 ++++++++++++++++ lib/test_hmm.c | 86 ++++++---- mm/debug.c | 9 +- mm/hmm.c | 5 +- mm/huge_memory.c | 43 ++--- mm/hugetlb.c | 15 +- mm/memory.c | 5 +- mm/memremap.c | 193 ++++++++++++++++++----- mm/migrate.c | 6 +- mm/migrate_device.c | 76 +++++---- mm/mm_init.c | 8 +- mm/mprotect.c | 10 +- mm/page_vma_mapped.c | 32 +++- mm/rmap.c | 59 ++++--- mm/util.c | 8 +- mm/vmscan.c | 2 +- 32 files changed, 822 insertions(+), 339 deletions(-)base-commit:
f8f9c1f4d0c7a64600e2ca312dec824a0bc2f1daSignature · 1 line
2.34.1
Best Regards,
Yan, Zi
Hi,
Quoted 7 lines · Matthew Brost
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CI
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Hi,
Quoted 4 lines · Jordan Niethe
Hi,
Quoted 1 line · Matthew Brost
Quoted 8 lines
Quoted 6 lines
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CI
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
For reference the rebase on drm-tip on the intel-xe list:
https://patchwork.freedesktop.org/series/159738/
Will watch the CI results.
Thanks,
Jordan.
Quoted 13 lines
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Hi,
Quoted 14 lines · Jordan Niethe
Hi,
Quoted 11 lines · Jordan Niethe
Hi,
Quoted 7 lines · Matthew Brost
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CI
Quoted 7 lines
Quoted 1 line
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
For reference the rebase on drm-tip on the intel-xe list:
https://patchwork.freedesktop.org/series/159738/
Will watch the CI results.
The series causes some failures in the intel-xe tests:
https://patchwork.freedesktop.org/series/159738/#rev4
Working through the failures now.
Thanks,
Jordan.
Quoted 18 lines
Thanks,
Jordan.Quoted 13 lines
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Matthew Brost
maintainerQuoted 22 lines · Jordan Niethe
Hi,
Quoted 19 lines · Jordan Niethe
Hi,
Quoted 12 lines · Jordan Niethe
Hi,
Quoted 8 lines · Matthew Brost
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all
our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CIQuoted 1 line
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
For reference the rebase on drm-tip on the intel-xe list:
Quoted 9 lines
Quoted 3 lines
https://patchwork.freedesktop.org/series/159738/
Will watch the CI results.
The series causes some failures in the intel-xe tests:
https://patchwork.freedesktop.org/series/159738/#rev4Working through the failures now.
Yea, I saw the failures. I haven't had time look at the patches on my end quite yet. Scrabling to get a few things in 6.20/7.0 PR, so I may not have bandwidth to look in depth until mid next week but digging is on my TODO list.
Matt
Quoted 22 lines
Thanks,
Jordan.Quoted 18 lines
Thanks,
Jordan.Quoted 13 lines
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Hi
Quoted 30 lines · Matthew Brost
Quoted 29 lines · Jordan Niethe
Hi,
Quoted 15 lines · Jordan Niethe
Hi,
Quoted 12 lines · Jordan Niethe
Hi,
Quoted 8 lines · Matthew Brost
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all
our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CIQuoted 7 lines
Quoted 1 line
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
For reference the rebase on drm-tip on the intel-xe list:
https://patchwork.freedesktop.org/series/159738/
Will watch the CI results.
The series causes some failures in the intel-xe tests:
https://patchwork.freedesktop.org/series/159738/#rev4
Quoted 7 lines
Quoted 2 lines
Working through the failures now.
Yea, I saw the failures. I haven't had time look at the patches on my end quite yet. Scrabling to get a few things in 6.20/7.0 PR, so I may not have bandwidth to look in depth until mid next week but digging is on my TODO list.
Sure, that's completely fine. The failures seem pretty directly related to the series so I think I'll be able to make good progress.
For example https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-159738v4/bat-bmg-2/igt@[email protected]
It looks like I missed that xe_pagemap_destroy_work() needs to be updated to remove the call to devm_release_mem_region() now we are no longer reserving a mem region.
Thanks,
Jordan.
Quoted 25 lines
Matt
Quoted 22 lines
Thanks,
Jordan.Quoted 18 lines
Thanks,
Jordan.Quoted 13 lines
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Matthew Brost
maintainerQuoted 44 lines · Jordan Niethe
Hi
Quoted 30 lines · Matthew Brost
Quoted 29 lines · Jordan Niethe
Hi,
Quoted 15 lines · Jordan Niethe
Hi,
Quoted 12 lines · Jordan Niethe
Hi,
Quoted 8 lines · Matthew Brost
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all
our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CIQuoted 7 lines
Quoted 1 line
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
For reference the rebase on drm-tip on the intel-xe list:
https://patchwork.freedesktop.org/series/159738/
Will watch the CI results.
The series causes some failures in the intel-xe tests:
https://patchwork.freedesktop.org/series/159738/#rev4Quoted 7 lines
Quoted 2 lines
Working through the failures now.
Yea, I saw the failures. I haven't had time look at the patches on my end quite yet. Scrabling to get a few things in 6.20/7.0 PR, so I may not have bandwidth to look in depth until mid next week but digging is on my TODO list.
Sure, that's completely fine. The failures seem pretty directly related to the
series so I think I'll be able to make good progress.
Quoted 6 lines
For example https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-159738v4/bat-bmg-2/igt@[email protected]
It looks like I missed that
xe_pagemap_destroy_work()needs to be updated to remove the call todevm_release_mem_region()now we are no longer reserving a mem
region.
+1
So this is the one I’d be most concerned about [1]. xe_exec_system_allocator is our SVM test, which does almost all the ridiculous things possible in user space to stress SVM. It’s blowing up in the core MM—but the source of the bug could be anywhere (e.g., Xe SVM, GPU SVM, migrate device layer, or core MM). I’ll try to help when I have bandwidth.
Matt
Quoted 31 lines
Thanks,
Jordan.Quoted 25 lines
Matt
Quoted 22 lines
Thanks,
Jordan.Quoted 18 lines
Thanks,
Jordan.Quoted 13 lines
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Hi,
Quoted 61 lines · Matthew Brost
Quoted 44 lines · Jordan Niethe
Hi
Quoted 30 lines · Matthew Brost
Quoted 29 lines · Jordan Niethe
Hi,
Quoted 15 lines · Jordan Niethe
Hi,
Quoted 12 lines · Jordan Niethe
Hi,
Quoted 8 lines · Matthew Brost
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all
our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CIQuoted 7 lines
Quoted 1 line
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
For reference the rebase on drm-tip on the intel-xe list:
https://patchwork.freedesktop.org/series/159738/
Will watch the CI results.
The series causes some failures in the intel-xe tests:
https://patchwork.freedesktop.org/series/159738/#rev4Quoted 7 lines
Quoted 2 lines
Working through the failures now.
Yea, I saw the failures. I haven't had time look at the patches on my end quite yet. Scrabling to get a few things in 6.20/7.0 PR, so I may not have bandwidth to look in depth until mid next week but digging is on my TODO list.
Sure, that's completely fine. The failures seem pretty directly related to the
series so I think I'll be able to make good progress.Quoted 6 lines
For example https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-159738v4/bat-bmg-2/igt@[email protected]
It looks like I missed that
xe_pagemap_destroy_work()needs to be updated to remove the call todevm_release_mem_region()now we are no longer reserving a mem
region.+1
So this is the one I’d be most concerned about [1].
xe_exec_system_allocatoris our SVM test, which does almost all the ridiculous things possible in user space to stress SVM. It’s blowing up in the core MM—but the source of the bug could be anywhere (e.g., Xe SVM, GPU SVM, migrate device layer, or core MM). I’ll try to help when I have bandwidth.
Quoted 3 lines
Matt
A similar fault in lruvec_stat_mod_folio can be repro'd if memremap_device_private_pagemap() is called with NUMA_NO_NODE instead of (say) numa_node_id() for the nid parameter.
The xe_svm driver uses devm_memremap_device_private_pagemap() which uses dev_to_node() for the nid parameter. Suspect this is causing something similar to happen.
When memremap_pages() calls pagemap_range() we have the following logic:
if (nid < 0)
nid = numa_mem_id();I think we might need to add this to memremap_device_private_pagemap() to handle the NUMA_NO_NODE case. Still confirming.
Thanks,
Jordan.
Quoted 32 lines
Quoted 32 lines
Thanks,
Jordan.Quoted 25 lines
Matt
Quoted 22 lines
Thanks,
Jordan.Quoted 18 lines
Thanks,
Jordan.Quoted 13 lines
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Hi,
Quoted 76 lines · Jordan Niethe
Hi,
Quoted 61 lines · Matthew Brost
Quoted 44 lines · Jordan Niethe
Hi
Quoted 30 lines · Matthew Brost
Quoted 29 lines · Jordan Niethe
Hi,
Quoted 15 lines · Jordan Niethe
Hi,
Quoted 12 lines · Jordan Niethe
Hi,
Quoted 8 lines · Matthew Brost
Thanks for the series. For some reason Intel's CI couldn't apply this series to drm-tip to get results [1]. I'll manually apply this and run all
our SVM tests and get back you on results + review the changes here. For future reference if you want to use our CI system, the series must apply to drm-tip, feel free to rebase this series and just send to intel-xe list if you want CIQuoted 7 lines
Quoted 1 line
Thanks, I'll rebase on drm-tip and send to the intel-xe list.
For reference the rebase on drm-tip on the intel-xe list:
https://patchwork.freedesktop.org/series/159738/
Will watch the CI results.
The series causes some failures in the intel-xe tests:
https://patchwork.freedesktop.org/series/159738/#rev4Quoted 7 lines
Quoted 2 lines
Working through the failures now.
Yea, I saw the failures. I haven't had time look at the patches on my end quite yet. Scrabling to get a few things in 6.20/7.0 PR, so I may not have bandwidth to look in depth until mid next week but digging is on my TODO list.
Sure, that's completely fine. The failures seem pretty directly related to the
series so I think I'll be able to make good progress.Quoted 6 lines
For example https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-159738v4/bat-bmg-2/igt@[email protected]
It looks like I missed that
xe_pagemap_destroy_work()needs to be updated to remove the call todevm_release_mem_region()now we are no longer reserving a mem
region.+1
So this is the one I’d be most concerned about [1].
xe_exec_system_allocatoris our SVM test, which does almost all the ridiculous things possible in user space to stress SVM. It’s blowing up in the core MM—but the source of the bug could be anywhere (e.g., Xe SVM, GPU SVM, migrate device layer, or core MM). I’ll try to help when I have bandwidth.Quoted 3 lines
Matt
A similar fault in
lruvec_stat_mod_foliocan be repro'd ifmemremap_device_private_pagemap()is called withNUMA_NO_NODEinstead of (say)numa_node_id()for the nid parameter.The
xe_svmdriver usesdevm_memremap_device_private_pagemap()which usesdev_to_node()for the nid parameter. Suspect this is causing something similar to happen.
Quoted 7 lines
When
memremap_pages()callspagemap_range()we have the following logic:if (nid < 0)
nid =numa_mem_id();I think we might need to add this to
memremap_device_private_pagemap()to handle theNUMA_NO_NODEcase. Still confirming.
This was the problem, fixed in v3.
Quoted 37 lines
Thanks,
Jordan.Quoted 32 lines
Quoted 32 lines
Thanks,
Jordan.Quoted 25 lines
Matt
Quoted 22 lines
Thanks,
Jordan.Quoted 18 lines
Thanks,
Jordan.Quoted 13 lines
Jordan.
Quoted 8 lines
I was also wondering if Nvidia could help review one our core MM patches
[2] which is gating enabling 2M device pages too?Matt
[1] https://patchwork.freedesktop.org/series/159738/
[2] https://patchwork.freedesktop.org/patch/694775/?series=159119&rev=1
Andrew Morton
maintainerQuoted 5 lines · On Wed, 7 Jan 2026 20:18:12 +1100 Jordan Niethe <[email protected]> wrote:
Today, when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages.
Welcome to Linux MM. That's a heck of an opening salvo ;)
Quoted 11 lines
Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
Can you better help us understand the seriousness of these problems? How much are our users really hurting from this?
Quoted 2 lines
Seeking opinions on using the mpfns like this or if a new type would be preferred.
Whose opinions? IOW, can you suggest who you'd like to see review this work?
Quoted 3 lines
* NOTE: I will need help in testing the driver changes *
Again, please name names ;) I'm not afraid to prod.
I'm reluctant to add this to mm.git's development/testing branches at this time. Your advice on when you think we're ready for that step would be valuable, thanks.
Jason Gunthorpe
maintainerQuoted 6 lines · Andrew Morton
Quoted 3 lines
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
Can you better help us understand the seriousness of these problems? How much are our users really hurting from this?
We think it is pretty serious, in the future HW support sense, as it means real systems being built do not work :)
Also Willy and others were cheering this work on at LPC. I think the possible followup to move DEVICE_PRIVATE from struct page and reduce the memory allocation would be well celebrated.
The Intel Xe and AMD GPU teams are the two drivers most important to be testing this as they consume the feature.
Jason
Quoted 12 lines · Jason Gunthorpe
Quoted 6 lines · Andrew Morton
Quoted 3 lines
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
Can you better help us understand the seriousness of these problems? How much are our users really hurting from this?
We think it is pretty serious, in the future HW support sense, as it means real systems being built do not work :)
Quoted 7 lines
Also Willy and others were cheering this work on at LPC. I think the possible followup to move
DEVICE_PRIVATEfrom struct page and reduce the memory allocation would be well celebrated.The Intel Xe and AMD GPU teams are the two drivers most important to be testing this as they consume the feature.
And the ultravisor usage in powerpc as well (book3s_hv_uvmem).
Balbir
Alistair Popple
reviewerOn 2026-01-08 at 08:02 +1100, Balbir Singh <[email protected]> wrote...
Quoted 7 lines · Jason Gunthorpe
Quoted 6 lines · Andrew Morton
Quoted 5 lines
Quoted 4 lines
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Can you better help us understand the seriousness of these problems? How much are our users really hurting from this?
We think it is pretty serious, in the future HW support sense, as it means real systems being built do not work :)
There's actually existing HW that could benefit from this support - after all there is nothing stopping someone plugging a Intel/AMD/NVIDIA GPU into an ARM machine today :-)
So it would be nice if we could support this feature there as it results in really sub-optimal performance compared with x86 when using the SVM (shared virtual memory) feature because data has to be remote mapped (ie. accessed via PCIe link) rather than migrated to local GPU video memory.
Having the kernel steal physical address space has also caused problems on x86 - we have encountered virtualised environments which depending on specific firmware/BIOS don't have enough free physical address space to support device private pages and hence migration of memory to the GPU device, again leading to sub-optmial performance.
Quoted 3 lines
Quoted 3 lines
Also Willy and others were cheering this work on at LPC. I think the possible followup to move
DEVICE_PRIVATEfrom struct page and reduce the memory allocation would be well celebrated.
For reference the recording of my LPC presentation covering both this series and the above is here - https://www.youtube.com/watch?v=CFe_c8-tEuM
The hope is that in addition to enabling support for this more broadly across other platforms/architectures that it will also enable further clean-ups to reduce memory allocation overhead (I almost convinced myself we wouldn't need a struct at all ... almost)
Quoted 5 lines
Quoted 3 lines
The Intel Xe and AMD GPU teams are the two drivers most important to be testing this as they consume the feature.
And the ultravisor usage in powerpc as well (
book3s_hv_uvmem).
As does Nouveau (which I've tested). But I agree AMD GPU and Intel Xe are the most important drivers here. I would be surprised if anyone was actually using the powerpc ultravisor, and I don't have access to a setup for this, so unless some PPC folk can offer to help I wouldn't like to see testing there hold up the series.
Especially as I believe most of the driver side changes are relatively straight forward.
- Alistair
Quoted 1 line
Balbir
Quoted 1 line · Andrew Morton
On Wed, 7 Jan 2026 20:18:12 +1100 Jordan Niethe <[email protected]> wrote:
...
Quoted 3 lines
Can you better help us understand the seriousness of these problems? How much are our users really hurting from this?
A lot! We have been involved in escalations from various customers who have attempted to enable, say, KASLR and HMM at the same time. And they ran out of phys address space, forcing them into an awkward ugly choice of one or the other, often.
This is a huge pain point and a barrier to HMM adoption.
thanks,
Signature · 1 line
John Hubbard
Alistair Popple
reviewerOn 2026-01-08 at 07:06 +1100, Andrew Morton <[email protected]> wrote...
Quoted 18 lines · On Wed, 7 Jan 2026 20:18:12 +1100 Jordan Niethe <[email protected]> wrote:
Quoted 6 lines
Today, when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages.Welcome to Linux MM. That's a heck of an opening salvo ;)
Quoted 8 lines
Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
Quoted 6 lines
Quoted 3 lines
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
Can you better help us understand the seriousness of these problems? How much are our users really hurting from this?
Hopefully the rest of the thread helps address this.
Quoted 5 lines
Quoted 2 lines
Seeking opinions on using the mpfns like this or if a new type would be preferred.
Whose opinions? IOW, can you suggest who you'd like to see review this work?
I was going to see if I could find Lorenzo on IRC as I think it would be good to get his opinion on the softleaf changes. And probably Felix's (and my) opinion for the mpfn changes (I don't think Intel currently uses DEVICE_COHERENT which this bit has the biggest impact on).
Quoted 5 lines
Quoted 3 lines
* NOTE: I will need help in testing the driver changes *
Again, please name names ;) I'm not afraid to prod.
As noted in the other thread Intel Xe and AMD GPU are the biggest. Matthew has already offered to help test Intel (thanks!) and Felix saw the v1 posting so hoping he can help with testing there.
Quoted 3 lines
I'm reluctant to add this to mm.git's development/testing branches at this time. Your advice on when you think we're ready for that step would be valuable, thanks.
Will leave the readiness call to Jordan, but we were hoping to get this in for the v6.20 merge window if at all possible. I realise we're probably running late given we generally like to let stuff settle in development/testing branches for a while prior to the merge window, but it did have an early round of review last year
(https://lore.kernel.org/linux-mm/[email protected]/)
and I reviewed it internally and it looked very reasonable.
I will take a look at this latest version later today.
- Alistair
Hi,
Quoted 31 lines · Alistair Popple
On 2026-01-08 at 07:06 +1100, Andrew Morton <[email protected]> wrote...
Quoted 18 lines · On Wed, 7 Jan 2026 20:18:12 +1100 Jordan Niethe <[email protected]> wrote:
Quoted 6 lines
Today, when creating these device private struct pages, the first step is to use
request_free_mem_region()to get a range of physical address space large enough to represent the devices memory. This allocated physical address range is then remapped as device private memory usingmemremap_pages.Welcome to Linux MM. That's a heck of an opening salvo ;)
Quoted 8 lines
Needing allocation of physical address space has some problems:
1) There may be insufficient physical address space to represent the device memory. KASLR reducing the physical address space and VM configurations with limited physical address space increase the likelihood of hitting this especially as device memory increases. This has been observed to prevent device private from being initialized.
Quoted 6 lines
Quoted 3 lines
2) Attempting to add the device private pages to the linear map at addresses beyond the actual physical memory causes issues on architectures like aarch64 - meaning the feature does not work there [0].
Can you better help us understand the seriousness of these problems? How much are our users really hurting from this?
Hopefully the rest of the thread helps address this.
Quoted 3 lines
Quoted 2 lines
Seeking opinions on using the mpfns like this or if a new type would be preferred.
Quoted 7 lines
Quoted 2 lines
Whose opinions? IOW, can you suggest who you'd like to see review this work?
I was going to see if I could find Lorenzo on IRC as I think it would be good to get his opinion on the softleaf changes. And probably Felix's (and my) opinion for the mpfn changes (I don't think Intel currently uses
DEVICE_COHERENTwhich this bit has the biggest impact on).
It also effects intel's driver because the mpfn changes also touch migrate_device_pfns() which gets used there.
So also looking for Matthew's thoughts here as well as Felix's.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
* NOTE: I will need help in testing the driver changes *
Again, please name names ;) I'm not afraid to prod.
As noted in the other thread Intel Xe and AMD GPU are the biggest. Matthew has already offered to help test Intel (thanks!) and Felix saw the v1 posting so hoping he can help with testing there.
Yes, I should also be able to get run this through the intel-xe CI. The other area that needs testing is the powerpc ultravisor. (+cc) Madhavan Srinivasan - are you able to help here?
Quoted 5 lines
Quoted 4 lines
I'm reluctant to add this to mm.git's development/testing branches at this time. Your advice on when you think we're ready for that step would be valuable, thanks.
Quoted 7 lines
Will leave the readiness call to Jordan, but we were hoping to get this in for the v6.20 merge window if at all possible. I realise we're probably running late given we generally like to let stuff settle in development/testing branches for a while prior to the merge window, but it did have an early round of review last year
(https://lore.kernel.org/linux-mm/[email protected]/)
and I reviewed it internally and it looked very reasonable.
Matt has kindly said that he is reviewing the patches so will wait for his feedback. I'd also like to get the results from the intel-xe CI first.
Andrew, I'll advise on including in mm.git after these steps - but I don't expect any major issues at this stage. The changes have been solid with the hmm selftests and with updating our out of tree driver to use the new interface.
Thanks,
Jordan.
Quoted 4 lines
I will take a look at this latest version later today.
- Alistair