ensure device private pages have an owner v2
Subsystems (12)
When acting on device private mappings a driver needs to know if the device (or other entity in case of kvmppc) actually owns this private mapping. This series adds an owner field and converts the migrate_vma code over to check it. I looked into doing the same for hmm_range_fault, but as far as I can tell that code has never been wired up to actually work for device private memory, so instead of trying to fix some unused code the second patch just remove the code. We can add it back once we have a working and fully tested code, and then should pass the expected owner in the hmm_range structure.
Changes since v1:
- split out the pgmap->owner addition into a separate patch
- check pgmap->owner is set for device private mappings
- rename the dev_private_owner field in struct migrate_vma to src_owner
- refuse to migrate private pages if src_owner is not set
- keep the non-fault device private handling in hmm_range_fault
PATCH 1/4 memremap: add an owner field to struct dev_pagemap
Add a new opaque owner field to struct dev_pagemap, which will allow the hmm and migrate_vma code to identify who owns ZONE_DEVICE memory, and refuse to work on mappings not owned by the calling entity.
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 ++ drivers/gpu/drm/nouveau/nouveau_dmem.c | 1 + include/linux/memremap.h | 4 ++++ mm/memremap.c | 4 ++++ 4 files changed, 11 insertions(+)
arch/powerpc/kvm/book3s_hv_uvmem.c +2 -0
@@ -779,6 +779,8 @@ int kvmppc_uvmem_init(void) kvmppc_uvmem_pgmap.type =MEMORY_DEVICE_PRIVATE ; kvmppc_uvmem_pgmap.res = *res; kvmppc_uvmem_pgmap.ops = &kvmppc_uvmem_ops; +/* just one global instance: */ + kvmppc_uvmem_pgmap.owner = &kvmppc_uvmem_pgmap; addr =memremap_pages (&kvmppc_uvmem_pgmap,NUMA_NO_NODE );if (IS_ERR (addr)) { ret =PTR_ERR (addr);
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
@@ -526,6 +526,7 @@ nouveau_dmem_init(struct nouveau_drm *drm) drm->dmem->pagemap.type =MEMORY_DEVICE_PRIVATE ; drm->dmem->pagemap.res = *res; drm->dmem->pagemap.ops = &nouveau_dmem_pagemap_ops; + drm->dmem->pagemap.owner = drm->dev;if (IS_ERR (devm_memremap_pages (device, &drm->dmem->pagemap))) goto out_free;
include/linux/memremap.h +4 -0
@@ -103,6 +103,9 @@ struct dev_pagemap_ops {* @type: memory type: see MEMORY_* in memory_hotplug.h * @flags: PGMAP_* flags to specify defailed behavior * @ops: method table +* @owner: an opaque pointer identifying the entity that manages this +* instance. Used by various helpers to make sure that no +* foreign ZONE_DEVICE memory is accessed. */ struct dev_pagemap {struct vmem_altmap altmap;
@@ -113,6 +116,7 @@ struct dev_pagemap {enum memory_type type;unsigned int flags;const struct dev_pagemap_ops *ops; +void *owner; };static inline struct vmem_altmap *pgmap_altmap (struct dev_pagemap *pgmap)
mm/memremap.c +4 -0
@@ -181,6 +181,10 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)WARN (1 ,"Missing migrate_to_ram method\n" );return ERR_PTR (-EINVAL ); } +if (!pgmap->owner) { +WARN (1 ,"Missing owner\n" ); +return ERR_PTR (-EINVAL ); + }break ;case MEMORY_DEVICE_FS_DAX : if (!IS_ENABLED(CONFIG_ZONE_DEVICE) ||
Signature · 1 line
2.24.1
Re: [PATCH 1/4] memremap: add an owner field to struct dev_pagemap
Quoted 5 lines · Christoph Hellwig
Add a new opaque owner field to struct
dev_pagemap, which will allow the hmm andmigrate_vmacode to identify who ownsZONE_DEVICEmemory, and refuse to work on mappings not owned by the calling entity.Signed-off-by Christoph Hellwig <[email protected]>
This looks like a reasonable approach to take.
Quoted 70 lines
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 ++ drivers/gpu/drm/nouveau/nouveau_dmem.c | 1 + include/linux/memremap.h | 4 ++++ mm/memremap.c | 4 ++++ 4 files changed, 11 insertions(+)arch/powerpc/kvm/book3s_hv_uvmem.c +2
-0@@ -779,6 +779,8 @@ int kvmppc_uvmem_init(void) kvmppc_uvmem_pgmap.type =MEMORY_DEVICE_PRIVATE ; kvmppc_uvmem_pgmap.res = *res; kvmppc_uvmem_pgmap.ops = &kvmppc_uvmem_ops; +/* just one global instance: */ + kvmppc_uvmem_pgmap.owner = &kvmppc_uvmem_pgmap; addr =memremap_pages (&kvmppc_uvmem_pgmap,NUMA_NO_NODE );if (IS_ERR (addr)) { ret =PTR_ERR (addr);drivers/gpu/drm/nouveau/nouveau_dmem.c +1
-0@@ -526,6 +526,7 @@ nouveau_dmem_init(struct nouveau_drm *drm) drm->dmem->pagemap.type =MEMORY_DEVICE_PRIVATE ; drm->dmem->pagemap.res = *res; drm->dmem->pagemap.ops = &nouveau_dmem_pagemap_ops; + drm->dmem->pagemap.owner = drm->dev;if (IS_ERR (devm_memremap_pages (device, &drm->dmem->pagemap))) goto out_free;include/linux/memremap.h +4
-0@@ -103,6 +103,9 @@ struct dev_pagemap_ops {* @type: memory type: see MEMORY_* in memory_hotplug.h * @flags: PGMAP_* flags to specify defailed behavior * @ops: method table +* @owner: an opaque pointer identifying the entity that manages this +* instance. Used by various helpers to make sure that no +* foreign ZONE_DEVICE memory is accessed. */ struct dev_pagemap {struct vmem_altmap altmap;@@ -113,6 +116,7 @@ struct dev_pagemap {enum memory_type type;unsigned int flags;const struct dev_pagemap_ops *ops; +void *owner; };static inline struct vmem_altmap *pgmap_altmap (struct dev_pagemap *pgmap)mm/memremap.c +4
-0@@ -181,6 +181,10 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)WARN (1 ,"Missing migrate_to_ram method\n" );return ERR_PTR (-EINVAL ); } +if (!pgmap->owner) { +WARN (1 ,"Missing owner\n" ); +return ERR_PTR (-EINVAL ); + }break ;case MEMORY_DEVICE_FS_DAX : if (!IS_ENABLED(CONFIG_ZONE_DEVICE) ||
PATCH 2/4 mm: handle multiple owners of device private pages in migrate_vma
In reply to Christoph HellwigAdd a new src_owner field to struct migrate_vma. If the field is set, only device private pages with page->pgmap->owner equal to that field are migrated. If the field is not set only "normal" pages are migrated.
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + drivers/gpu/drm/nouveau/nouveau_dmem.c | 1 + include/linux/migrate.h | 8 ++++++++ mm/migrate.c | 9 ++++++--- 4 files changed, 16 insertions(+), 3 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -0
@@ -563,6 +563,7 @@ kvmppc_svm_page_out(struct vm_area_struct *vma, unsigned long start, mig.end = end; mig.src = &src_pfn; mig.dst = &dst_pfn; + mig.src_owner = &kvmppc_uvmem_pgmap;mutex_lock (&kvm->arch.uvmem_lock);/* The requested page is already paged-out, nothing to do */
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
@@ -176,6 +176,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) .end = vmf->address + PAGE_SIZE, .src = &src, .dst = &dst, + .src_owner = drm->dev, };/*
include/linux/migrate.h +8 -0
@@ -196,6 +196,14 @@ struct migrate_vma {unsigned long npages;unsigned long start;unsigned long end; + +/* +* Set to the owner value also stored in page->pgmap->owner for +* migrating out of device private memory. If set only device +* private pages with this owner are migrated. If not set +* device private pages are not migrated at all. +*/ +void *src_owner; };int migrate_vma_setup (struct migrate_vma *args);
mm/migrate.c +6 -3
@@ -2241,7 +2241,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,arch_enter_lazy_mmu_mode ();for (; addr < end; addr +=PAGE_SIZE , ptep++) { -unsigned long mpfn, pfn; +unsigned long mpfn =0 , pfn;struct page *page;swp_entry_t entry;pte_t pte;
@@ -2255,8 +2255,6 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, }if (!pte_present (pte)) { - mpfn =0 ; -/* * Only care about unaddressable device page special * page table entry. Other special swap entries are not
@@ -2267,11 +2265,16 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, goto next; page =device_private_entry_to_page (entry); +if (page->pgmap->owner != migrate->src_owner) + goto next; + mpfn =migrate_pfn (page_to_pfn (page)) |MIGRATE_PFN_MIGRATE ;if (is_write_device_private_entry (entry)) mpfn |=MIGRATE_PFN_WRITE ; }else { +if (migrate->src_owner) + goto next; pfn =pte_pfn (pte);if (is_zero_pfn (pfn)) { mpfn =MIGRATE_PFN_MIGRATE ;
Signature · 1 line
2.24.1
Re: [PATCH 2/4] mm: handle multiple owners of device private pages in migrate_vma
Quoted 6 lines · Christoph Hellwig
Add a new
src_ownerfield to structmigrate_vma. If the field is set, only device private pages withpage->pgmap->ownerequal to that field are migrated. If the field is not set only "normal" pages are migrated.Signed-off-by Christoph Hellwig <[email protected]>Fixes df6ad69838fc ("mm/device-public-memory: device memory cache coherent with CPU")
When migrating to device private memory, setting the src_owner lets the caller know about source pages that are already migrated and skips pages migrated to a different device similar to pages swapped out an actual swap device. But, it prevents normal pages from being migrated to device private memory. It can still be useful for the driver to know that a page is owned by a different device if it has a device to device way of migrating data. nouveau_dmem_migrate_vma() isn't setting args.src_owner so only normal pages will be migrated which I guess is OK for now since nouveau doesn't handle direct GPU to GPU migration currently.
When migrating device private memory to system memory due to a CPU fault, the source page should be the device's device private struct page so if it isn't, then it does make sense to not migrate whatever normal page is there. nouveau_dmem_migrate_to_ram() sets src_owner so this case looks OK.
Just had to think this through.
Quoted 90 lines
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + drivers/gpu/drm/nouveau/nouveau_dmem.c | 1 + include/linux/migrate.h | 8 ++++++++ mm/migrate.c | 9 ++++++--- 4 files changed, 16 insertions(+), 3 deletions(-)arch/powerpc/kvm/book3s_hv_uvmem.c +1
-0@@ -563,6 +563,7 @@ kvmppc_svm_page_out(struct vm_area_struct *vma, unsigned long start, mig.end = end; mig.src = &src_pfn; mig.dst = &dst_pfn; + mig.src_owner = &kvmppc_uvmem_pgmap;mutex_lock (&kvm->arch.uvmem_lock);/* The requested page is already paged-out, nothing to do */ drivers/gpu/drm/nouveau/nouveau_dmem.c +1
-0@@ -176,6 +176,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) .end = vmf->address + PAGE_SIZE, .src = &src, .dst = &dst, + .src_owner = drm->dev, };/* include/linux/migrate.h +8
-0@@ -196,6 +196,14 @@ struct migrate_vma {unsigned long npages;unsigned long start;unsigned long end; + +/* +* Set to the owner value also stored in page->pgmap->owner for +* migrating out of device private memory. If set only device +* private pages with this owner are migrated. If not set +* device private pages are not migrated at all. +*/ +void *src_owner; };int migrate_vma_setup (struct migrate_vma *args);mm/migrate.c +6
-3@@ -2241,7 +2241,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,arch_enter_lazy_mmu_mode ();for (; addr < end; addr +=PAGE_SIZE , ptep++) { -unsigned long mpfn, pfn; +unsigned long mpfn =0 , pfn;struct page *page;swp_entry_t entry;pte_t pte;@@ -2255,8 +2255,6 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, }if (!pte_present (pte)) { - mpfn =0 ; -/* * Only care about unaddressable device page special * page table entry. Other special swap entries are not @@ -2267,11 +2265,16 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, goto next; page =device_private_entry_to_page (entry); +if (page->pgmap->owner != migrate->src_owner) + goto next; + mpfn =migrate_pfn (page_to_pfn (page)) |MIGRATE_PFN_MIGRATE ;if (is_write_device_private_entry (entry)) mpfn |=MIGRATE_PFN_WRITE ; }else { +if (migrate->src_owner) + goto next; pfn =pte_pfn (pte);if (is_zero_pfn (pfn)) { mpfn =MIGRATE_PFN_MIGRATE ;
PATCH 3/4 mm: simplify device private page handling in hmm_range_fault
In reply to Christoph HellwigRemove the code to fault device private pages back into system memory that has never been used by any driver. Also replace the usage of the HMM_PFN_DEVICE_PRIVATE flag in the pfns array with a simple is_device_private_page check in nouveau.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 1 - drivers/gpu/drm/nouveau/nouveau_dmem.c | 5 +++-- drivers/gpu/drm/nouveau/nouveau_svm.c | 1 - include/linux/hmm.h | 2 -- mm/hmm.c | 25 +++++-------------------- 5 files changed, 8 insertions(+), 26 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
@@ -776,7 +776,6 @@ struct amdgpu_ttm_tt {static const uint64_t hmm_range_flags[HMM_PFN_FLAG_MAX ] = { (1 <<0 ),/* HMM_PFN_VALID */ (1 <<1 ),/* HMM_PFN_WRITE */ -0 /* HMM_PFN_DEVICE_PRIVATE */ };static const uint64_t hmm_range_values[HMM_PFN_VALUE_MAX ] = {
drivers/gpu/drm/nouveau/nouveau_dmem.c +3 -2
@@ -28,6 +28,7 @@#include <nvif/class.h> #include <nvif/object.h> +#include <nvif/if000c.h> #include <nvif/if500b.h> #include <nvif/if900b.h>
@@ -692,9 +693,8 @@ nouveau_dmem_convert_pfn(struct nouveau_drm *drm,if (page =NULL )continue ; -if (!(range->pfns[i] & range->flags[HMM_PFN_DEVICE_PRIVATE ])) { +if (!is_device_private_page (page))continue ; - }if (!nouveau_dmem_page (drm, page)) {WARN (1 ,"Some unknown device memory !\n" );
@@ -705,5 +705,6 @@ nouveau_dmem_convert_pfn(struct nouveau_drm *drm, addr =nouveau_dmem_page_addr (page); range->pfns[i] &= ((1UL << range->pfn_shift) -1 ); range->pfns[i] |= (addr >>PAGE_SHIFT ) << range->pfn_shift; + range->pfns[i] |=NVIF_VMM_PFNMAP_V0_VRAM ; } }
drivers/gpu/drm/nouveau/nouveau_svm.c +0 -1
@@ -367,7 +367,6 @@ static const u64 nouveau_svm_pfn_flags[HMM_PFN_FLAG_MAX] = { [HMM_PFN_VALID ] = NVIF_VMM_PFNMAP_V0_V, [HMM_PFN_WRITE ] = NVIF_VMM_PFNMAP_V0_W, - [HMM_PFN_DEVICE_PRIVATE] = NVIF_VMM_PFNMAP_V0_VRAM, }; static const u64
include/linux/hmm.h +0 -2
@@ -74,7 +74,6 @@* Flags: * HMM_PFN_VALID: pfn is valid. It has, at least, read permission. * HMM_PFN_WRITE: CPU page table has write permission set -* HMM_PFN_DEVICE_PRIVATE: private device memory (ZONE_DEVICE) * * The driver provides a flags array for mapping page protections to device * PTE bits. If the driver valid bit for an entry is bit 3,
@@ -86,7 +85,6 @@enum hmm_pfn_flag_e {HMM_PFN_VALID =0 ,HMM_PFN_WRITE , -HMM_PFN_DEVICE_PRIVATE ,HMM_PFN_FLAG_MAX };
mm/hmm.c +5 -20
@@ -118,15 +118,6 @@ static inline void hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk,/* We aren't ask to do anything ... */ if (!(pfns & range->flags[HMM_PFN_VALID ]))return ; -/* If this is device memory then only fault if explicitly requested */ -if ((cpu_flags & range->flags[HMM_PFN_DEVICE_PRIVATE ])) { -/* Do we fault on device memory ? */ -if (pfns & range->flags[HMM_PFN_DEVICE_PRIVATE ]) { - *write_fault = pfns & range->flags[HMM_PFN_WRITE ]; - *fault = true; - } -return ; - }/* If CPU page table is not valid then we need to fault */ *fault = !(cpu_flags & range->flags[HMM_PFN_VALID ]);
@@ -260,21 +251,15 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,swp_entry_t entry =pte_to_swp_entry (pte);/* -* This is a special swap entry, ignore migration, use -* device and report anything else as error. +* Never fault in device private pages pages, but just report +* the PFN even if not present. */ if (is_device_private_entry (entry)) { - cpu_flags = range->flags[HMM_PFN_VALID ] | - range->flags[HMM_PFN_DEVICE_PRIVATE ]; - cpu_flags |=is_write_device_private_entry (entry) ? - range->flags[HMM_PFN_WRITE ] :0 ; -hmm_pte_need_fault (hmm_vma_walk, orig_pfn, cpu_flags, - &fault, &write_fault); -if (fault || write_fault) - goto fault; *pfn =hmm_device_entry_from_pfn (range,swp_offset (entry)); - *pfn |= cpu_flags; + *pfn |= range->flags[HMM_PFN_VALID ]; +if (is_write_device_private_entry (entry)) + *pfn |= range->flags[HMM_PFN_WRITE ];return 0 ; }
Signature · 1 line
2.24.1
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 6 lines · Christoph Hellwig
mm/hmm.c +1
-0+++ b/mm/hmm.c@@ -118,15 +118,6 @@ static inline void hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk,/* We aren't ask to do anything ... */ if (!(pfns & range->flags[HMM_PFN_VALID ]))
mm/hmm.c static inline void hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk, · line 129
Quoted 10 lines
mm/hmm.c +0
-9return ; -/* If this is device memory then only fault if explicitly requested */ -if ((cpu_flags & range->flags[HMM_PFN_DEVICE_PRIVATE ])) { -/* Do we fault on device memory ? */ -if (pfns & range->flags[HMM_PFN_DEVICE_PRIVATE ]) { - *write_fault = pfns & range->flags[HMM_PFN_WRITE ]; - *fault = true; - } -return ; - }
Yes, this is an elegant solution to the input flags.
However, between patch 3 and 4 doesn't this break amd gpu as it will return device_private pages now if not requested? Squash the two?
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 2 lines · Jason Gunthorpe
However, between patch 3 and 4 doesn't this break amd gpu as it will return
device_privatepages now if not requested? Squash the two?
No change in behavior in this patch as long as HMM_PFN_DEVICE_PRIVATE isn't set in ->pfns or ->default_flags, which is the case for both nouveau and amdgpu. The existing behavior is broken for private pages not known to the driver, but that is fixed in the next patch.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
In reply to Christoph HellwigQuoted 6 lines · Christoph Hellwig
Remove the code to fault device private pages back into system memory that has never been used by any driver. Also replace the usage of the
HMM_PFN_DEVICE_PRIVATEflag in the pfns array with a simpleis_device_private_pagecheck in nouveau.Signed-off-by Christoph Hellwig <[email protected]>
Getting rid of HMM_PFN_DEVICE_PRIVATE seems reasonable to me since a driver can look at the struct page but what if a driver needs to fault in a page from another device's private memory? Should it call handle_mm_fault()?
Quoted 130 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 1 - drivers/gpu/drm/nouveau/nouveau_dmem.c | 5 +++-- drivers/gpu/drm/nouveau/nouveau_svm.c | 1 - include/linux/hmm.h | 2 -- mm/hmm.c | 25 +++++-------------------- 5 files changed, 8 insertions(+), 26 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0
-1@@ -776,7 +776,6 @@ struct amdgpu_ttm_tt {static const uint64_t hmm_range_flags[HMM_PFN_FLAG_MAX ] = { (1 <<0 ),/* HMM_PFN_VALID */ (1 <<1 ),/* HMM_PFN_WRITE */ -0 /* HMM_PFN_DEVICE_PRIVATE */ };static const uint64_t hmm_range_values[HMM_PFN_VALUE_MAX ] = {drivers/gpu/drm/nouveau/nouveau_dmem.c +3
-2@@ -28,6 +28,7 @@#include <nvif/class.h> #include <nvif/object.h> +#include <nvif/if000c.h> #include <nvif/if500b.h> #include <nvif/if900b.h> @@ -692,9 +693,8 @@ nouveau_dmem_convert_pfn(struct nouveau_drm *drm,if (page =NULL )continue ; -if (!(range->pfns[i] & range->flags[HMM_PFN_DEVICE_PRIVATE ])) { +if (!is_device_private_page (page))continue ; - }if (!nouveau_dmem_page (drm, page)) {WARN (1 ,"Some unknown device memory !\n" );@@ -705,5 +705,6 @@ nouveau_dmem_convert_pfn(struct nouveau_drm *drm, addr =nouveau_dmem_page_addr (page); range->pfns[i] &= ((1UL << range->pfn_shift) -1 ); range->pfns[i] |= (addr >>PAGE_SHIFT ) << range->pfn_shift; + range->pfns[i] |=NVIF_VMM_PFNMAP_V0_VRAM ; } }drivers/gpu/drm/nouveau/nouveau_svm.c +0
-1@@ -367,7 +367,6 @@ static const u64 nouveau_svm_pfn_flags[HMM_PFN_FLAG_MAX] = { [HMM_PFN_VALID ] = NVIF_VMM_PFNMAP_V0_V, [HMM_PFN_WRITE ] = NVIF_VMM_PFNMAP_V0_W, - [HMM_PFN_DEVICE_PRIVATE] = NVIF_VMM_PFNMAP_V0_VRAM, }; static const u64include/linux/hmm.h +0
-2@@ -74,7 +74,6 @@* Flags: * HMM_PFN_VALID: pfn is valid. It has, at least, read permission. * HMM_PFN_WRITE: CPU page table has write permission set -* HMM_PFN_DEVICE_PRIVATE: private device memory (ZONE_DEVICE) * * The driver provides a flags array for mapping page protections to device * PTE bits. If the driver valid bit for an entry is bit 3, @@ -86,7 +85,6 @@enum hmm_pfn_flag_e {HMM_PFN_VALID =0 ,HMM_PFN_WRITE , -HMM_PFN_DEVICE_PRIVATE ,HMM_PFN_FLAG_MAX };mm/hmm.c +5
-20@@ -118,15 +118,6 @@ static inline void hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk,/* We aren't ask to do anything ... */ if (!(pfns & range->flags[HMM_PFN_VALID ]))return ; -/* If this is device memory then only fault if explicitly requested */ -if ((cpu_flags & range->flags[HMM_PFN_DEVICE_PRIVATE ])) { -/* Do we fault on device memory ? */ -if (pfns & range->flags[HMM_PFN_DEVICE_PRIVATE ]) { - *write_fault = pfns & range->flags[HMM_PFN_WRITE ]; - *fault = true; - } -return ; - }/* If CPU page table is not valid then we need to fault */ *fault = !(cpu_flags & range->flags[HMM_PFN_VALID ]);@@ -260,21 +251,15 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,swp_entry_t entry =pte_to_swp_entry (pte);/* -* This is a special swap entry, ignore migration, use -* device and report anything else as error. +* Never fault in device private pages pages, but just report +* the PFN even if not present. */ if (is_device_private_entry (entry)) { - cpu_flags = range->flags[HMM_PFN_VALID ] | - range->flags[HMM_PFN_DEVICE_PRIVATE ]; - cpu_flags |=is_write_device_private_entry (entry) ? - range->flags[HMM_PFN_WRITE ] :0 ; -hmm_pte_need_fault (hmm_vma_walk, orig_pfn, cpu_flags, - &fault, &write_fault); -if (fault || write_fault) - goto fault; *pfn =hmm_device_entry_from_pfn (range,swp_offset (entry)); - *pfn |= cpu_flags; + *pfn |= range->flags[HMM_PFN_VALID ]; +if (is_write_device_private_entry (entry)) + *pfn |= range->flags[HMM_PFN_WRITE ];return 0 ; }
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 11 lines · Ralph Campbell
Quoted 6 lines · Christoph Hellwig
Remove the code to fault device private pages back into system memory that has never been used by any driver. Also replace the usage of the
HMM_PFN_DEVICE_PRIVATEflag in the pfns array with a simpleis_device_private_pagecheck in nouveau.Signed-off-by Christoph Hellwig <[email protected]>Getting rid of
HMM_PFN_DEVICE_PRIVATEseems reasonable to me since a driver can look at the struct page but what if a driver needs to fault in a page from another device's private memory? Should it callhandle_mm_fault()?
Obviously no driver cared for that so far. Once we have test cases for that and thus testable code we can add code to fault it in from hmm_vma_handle_pte.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Ralph Campbell
Quoted 6 lines · Christoph Hellwig
Remove the code to fault device private pages back into system memory that has never been used by any driver. Also replace the usage of the
HMM_PFN_DEVICE_PRIVATEflag in the pfns array with a simpleis_device_private_pagecheck in nouveau.Signed-off-by Christoph Hellwig <[email protected]>
Quoted 8 lines
Quoted 3 lines
Getting rid of
HMM_PFN_DEVICE_PRIVATEseems reasonable to me since a driver can look at the struct page but what if a driver needs to fault in a page from another device's private memory? Should it callhandle_mm_fault()?Obviously no driver cared for that so far. Once we have test cases for that and thus testable code we can add code to fault it in from
hmm_vma_handle_pte.
I'm OK with the series. I think I would have been less confused if I looked at patch 4 then 3.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 7 lines · Ralph Campbell
Quoted 4 lines
Obviously no driver cared for that so far. Once we have test cases for that and thus testable code we can add code to fault it in from
hmm_vma_handle_pte.I'm OK with the series. I think I would have been less confused if I looked at patch 4 then 3.
I guess I could just merge 3 and 4 if it is too confusing otherwise.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
In reply to Ralph CampbellQuoted 7 lines · Ralph Campbell
Quoted 5 lines · Christoph Hellwig
Remove the code to fault device private pages back into system memory that has never been used by any driver. Also replace the usage of the
HMM_PFN_DEVICE_PRIVATEflag in the pfns array with a simpleis_device_private_pagecheck in nouveau.
Quoted 5 lines
Quoted 1 line
Signed-off-by Christoph Hellwig <[email protected]>Getting rid of
HMM_PFN_DEVICE_PRIVATEseems reasonable to me since a driver can look at the struct page but what if a driver needs to fault in a page from another device's private memory? Should it callhandle_mm_fault()?
Isn't that what this series basically does?
The dev_private_owner is set to the type of pgmap the device knows how to handle, and everything else is automatically faulted for the device.
If the device does not know how to handle device_private then it sets dev_private_owner to NULL and it never gets device_private pfns.
Since the device_private pfn cannot be dma mapped, drivers must have explicit support for them.
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 10 lines · Jason Gunthorpe
Quoted 3 lines
Getting rid of
HMM_PFN_DEVICE_PRIVATEseems reasonable to me since a driver can look at the struct page but what if a driver needs to fault in a page from another device's private memory? Should it callhandle_mm_fault()?Isn't that what this series basically does?
The
dev_private_owneris set to the type of pgmap the device knows how to handle, and everything else is automatically faulted for the device.
Quoted 5 lines
If the device does not know how to handle
device_privatethen it setsdev_private_ownerto NULL and it never getsdevice_privatepfns.Since the
device_privatepfn cannot be dma mapped, drivers must have explicit support for them.
No, with this series (and all actual callers before this series) we never fault in device private pages.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jason Gunthorpe
Quoted 3 lines
Getting rid of
HMM_PFN_DEVICE_PRIVATEseems reasonable to me since a driver can look at the struct page but what if a driver needs to fault in a page from another device's private memory? Should it callhandle_mm_fault()?Isn't that what this series basically does?
The
dev_private_owneris set to the type of pgmap the device knows how to handle, and everything else is automatically faulted for the device.
Quoted 8 lines
Quoted 5 lines
If the device does not know how to handle
device_privatethen it setsdev_private_ownerto NULL and it never getsdevice_privatepfns.Since the
device_privatepfn cannot be dma mapped, drivers must have explicit support for them.No, with this series (and all actual callers before this series) we never fault in device private pages.
IFF we want to fault it in we'd need something like this. But I'd really prefer to see test cases for that first.
mm/hmm.c +1 -1
@@ -276,7 +276,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,if (!fault && !write_fault)return 0 ; -if (!non_swap_entry (entry)) +if (!non_swap_entry (entry) ||is_device_private_entry (entry)) goto fault;if (is_migration_entry (entry)) {
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 15 lines · Christoph Hellwig
Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · Jason Gunthorpe
Quoted 3 lines
Getting rid of
HMM_PFN_DEVICE_PRIVATEseems reasonable to me since a driver can look at the struct page but what if a driver needs to fault in a page from another device's private memory? Should it callhandle_mm_fault()?Isn't that what this series basically does?
The
dev_private_owneris set to the type of pgmap the device knows how to handle, and everything else is automatically faulted for the device.If the device does not know how to handle
device_privatethen it setsdev_private_ownerto NULL and it never getsdevice_privatepfns.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Since the
device_privatepfn cannot be dma mapped, drivers must have explicit support for them.No, with this series (and all actual callers before this series) we never fault in device private pages.
IFF we want to fault it in we'd need something like this. But I'd really prefer to see test cases for that first.
In general I think hmm_range_fault should have a mode that is the same as get_user_pages in terms of when it returns a hard failure, and generates faults. AFAIK, GUP will fault in this case?
I need this for making ODP use this API. ODP is the one that is highly likely to see other driver's device_private pages and must have them always fault to CPU.
mm/hmm.c static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, · line 280
Quoted 10 lines
mm/hmm.c +2
-1+++ b/mm/hmm.c@@ -276,7 +276,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,if (!fault && !write_fault)return 0 ; -if (!non_swap_entry (entry)) +if (!non_swap_entry (entry) ||is_device_private_entry (entry)) goto fault;
Yes, OK, makes sense.
I've been using v7 of Ralph's tester and it is working well - it has DEVICE_PRIVATE support so I think it can test this flow too. Ralph are you able?
This hunk seems trivial enough to me, can we include it now?
Thanks,
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 5 lines · Jason Gunthorpe
I've been using v7 of Ralph's tester and it is working well - it has
DEVICE_PRIVATEsupport so I think it can test this flow too. Ralph are you able?This hunk seems trivial enough to me, can we include it now?
I can send a separate patch for it once the tester covers it. I don't want to add it to the original patch as it is a significant behavior change compared to the existing code.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 10 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
I've been using v7 of Ralph's tester and it is working well - it has
DEVICE_PRIVATEsupport so I think it can test this flow too. Ralph are you able?This hunk seems trivial enough to me, can we include it now?
I can send a separate patch for it once the tester covers it. I don't want to add it to the original patch as it is a significant behavior change compared to the existing code.
Okay. I'm happy enough for now that amdgpu will get ERROR on device_private pages. That is a bug fix in of itself.
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
In reply to Christoph HellwigQuoted 11 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
I've been using v7 of Ralph's tester and it is working well - it has
DEVICE_PRIVATEsupport so I think it can test this flow too. Ralph are you able?This hunk seems trivial enough to me, can we include it now?
I can send a separate patch for it once the tester covers it. I don't want to add it to the original patch as it is a significant behavior change compared to the existing code.
Attached is an updated version of my HMM tests based on linux-5.6.0-rc6. I ran this OK with Jason's 8+1 HMM patches, Christoph's 1-5 misc HMM clean ups, and Christoph's 1-4 device private page changes applied.
I'm working on getting my nouveau tests running again on a different test machine and will report on that when ready.
From d499fb343bfa9764695ecdcd759fb16bc1ca3c93 Mon Sep 17 00:00:00 2001
From: Ralph Campbell <[email protected]>
Date: Tue, 17 Mar 2020 11:10:38 -0700
Subject: [PATCH] mm/hmm/test: add self tests for HMM
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add some basic stand alone self tests for HMM.
--- MAINTAINERS | 3 + include/uapi/linux/test_hmm.h | 59 ++ lib/Kconfig.debug | 12 + lib/Makefile | 1 + lib/test_hmm.c | 1210 +++++++++++++++++++++ tools/testing/selftests/vm/.gitignore | 1 + tools/testing/selftests/vm/Makefile | 3 + tools/testing/selftests/vm/config | 2 + tools/testing/selftests/vm/hmm-tests.c | 1353 ++++++++++++++++++++++++ tools/testing/selftests/vm/run_vmtests | 16 + tools/testing/selftests/vm/test_hmm.sh | 97 ++ 11 files changed, 2757 insertions(+) create mode 100644 include/uapi/linux/test_hmm.h create mode 100644 lib/test_hmm.c create mode 100644 tools/testing/selftests/vm/hmm-tests.c create mode 100755 tools/testing/selftests/vm/test_hmm.sh
MAINTAINERS +3 -0
@@ -7608,7 +7608,10 @@ L: [email protected] S: Maintained F: mm/hmm* F: include/linux/hmm* +F: include/uapi/linux/test_hmm* F: Documentation/vm/hmm.rst +F: lib/test_hmm* +F: tools/testing/selftests/vm/*hmm* HOST AP DRIVER M: Jouni Malinen <[email protected]>
include/uapi/linux/test_hmm.h new file +59 -0
59 lines not on this page · raw message
lib/Kconfig.debug +12 -0
@@ -2162,6 +2162,18 @@ config TEST_MEMINIT If unsure, say N. +config TEST_HMM + tristate "Test HMM (Heterogeneous Memory Management)" + depends on DEVICE_PRIVATE + select HMM_MIRROR + select MMU_NOTIFIER + help + This is a pseudo device driver solely for testing HMM. + Say M here if you want to build the HMM test module. + Doing so will allow you to run tools/testing/selftest/vm/hmm-tests. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTEST
lib/Makefile +1 -0
@@ -89,6 +89,7 @@ obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o obj-$(CONFIG_TEST_BLACKHOLE_DEV) += test_blackhole_dev.o obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o +obj-$(CONFIG_TEST_HMM) += test_hmm.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/
lib/test_hmm.c new file +1210 -0
1,210 lines not on this page · raw message
tools/testing/selftests/vm/.gitignore +1 -0
@@ -14,3 +14,4 @@ virtual_address_range gup_benchmark va_128TBswitch map_fixed_noreplace +hmm-tests
tools/testing/selftests/vm/Makefile +3 -0
@@ -7,6 +7,7 @@ CFLAGS = -Wall -I ../../../../usr/include $(EXTRA_CFLAGS) LDLIBS = -lrt TEST_GEN_FILES = compaction_test TEST_GEN_FILES += gup_benchmark +TEST_GEN_FILES += hmm-tests TEST_GEN_FILES += hugepage-mmap TEST_GEN_FILES += hugepage-shm TEST_GEN_FILES += map_hugetlb
@@ -31,6 +32,8 @@ TEST_FILES := test_vmalloc.sh KSFT_KHDR_INSTALL := 1 include ../lib.mk +$(OUTPUT)/hmm-tests: LDLIBS += -lhugetlbfs -lpthread + $(OUTPUT)/userfaultfd: LDLIBS += -lpthread $(OUTPUT)/mlock-random-test: LDLIBS += -lcap
tools/testing/selftests/vm/config +2 -0
@@ -1,3 +1,5 @@ CONFIG_SYSVIPC=y CONFIG_USERFAULTFD=y CONFIG_TEST_VMALLOC=m +CONFIG_HMM_MIRROR=y +CONFIG_DEVICE_PRIVATE=y
tools/testing/selftests/vm/hmm-tests.c new file +1353 -0
1,353 lines not on this page · raw message
tools/testing/selftests/vm/run_vmtests +16 -0
@@ -270,4 +270,20 @@ else exitcode=1 fi +echo "------------------------------------" +echo "running HMM smoke test" +echo "------------------------------------" +./test_hmm.sh smoke +ret_val=$? + +if [ $ret_val -eq 0 ]; then + echo "[PASS]" +elif [ $ret_val -eq $ksft_skip ]; then + echo "[SKIP]" + exitcode=$ksft_skip +else + echo "[FAIL]" + exitcode=1 +fi + exit $exitcode
tools/testing/selftests/vm/test_hmm.sh new file +97 -0
97 lines not on this page · raw message
Signature · 1 line
2.23.0
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 9 lines · Ralph Campbell
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
I've been using v7 of Ralph's tester and it is working well - it has
DEVICE_PRIVATEsupport so I think it can test this flow too. Ralph are you able?This hunk seems trivial enough to me, can we include it now?
Quoted 8 lines
Quoted 4 lines
I can send a separate patch for it once the tester covers it. I don't want to add it to the original patch as it is a significant behavior change compared to the existing code.
Attached is an updated version of my HMM tests based on linux-5.6.0-rc6. I ran this OK with Jason's 8+1 HMM patches, Christoph's 1-5 misc HMM clean ups, and Christoph's 1-4 device private page changes applied.
I'd like to get this to mergable, it looks pretty good now, but I have no idea about selftests - and I'm struggling to even compile the tools dir
lib/Kconfig.debug config TEST_MEMINIT · line 2169
Quoted 12 lines
lib/Kconfig.debug +6
-0+++ b/lib/Kconfig.debug@@ -2162,6 +2162,18 @@ config TEST_MEMINIT If unsure, say N. +config TEST_HMM + tristate "Test HMM (Heterogeneous Memory Management)" + depends on DEVICE_PRIVATE + select HMM_MIRROR + select MMU_NOTIFIER
extra spaces
In general I wonder if it even makes sense that DEVICE_PRIVATE is user selectable?
Quoted 15 lines
lib/Kconfig.debug +15
-0+static int dmirror_fops_open(struct inode *inode, struct file *filp) +{ + struct cdev *cdev = inode->i_cdev; + struct dmirror *dmirror; + int ret; + + /* Mirror this process address space */ + dmirror = kzalloc(sizeof(*dmirror), GFP_KERNEL); + if (dmirror = NULL) + return -ENOMEM; + + dmirror->mdevice = container_of(cdev, struct dmirror_device, cdevice); + mutex_init(&dmirror->mutex); + xa_init(&dmirror->pt); +
lib/Kconfig.debug config TEST_MEMINIT
Quoted 10 lines
lib/Kconfig.debug +10
-0+ ret = mmu_interval_notifier_insert(&dmirror->notifier, current->mm, + 0, ULONG_MAX & PAGE_MASK, &dmirror_min_ops); + if (ret) { + kfree(dmirror); + return ret; + } + + /* Pairs with the mmdrop() in dmirror_fops_release(). */ + mmgrab(current->mm); + dmirror->mm = current->mm;
The notifier holds a mmgrab, no need for another one
lib/Kconfig.debug
Quoted 2 lines
lib/Kconfig.debug +2
-0+ /* Only the first open registers the address space. */ + filp->private_data = dmirror;
Not sure what this comment means
lib/Kconfig.debug
Quoted 8 lines
lib/Kconfig.debug +8
-0+static inline struct dmirror_device *dmirror_page_to_device(struct page *page) + +{ + struct dmirror_chunk *devmem; + + devmem = container_of(page->pgmap, struct dmirror_chunk, pagemap); + return devmem->mdevice; +}
extra devmem var is not really needed
lib/Kconfig.debug
Quoted 9 lines
lib/Kconfig.debug +9
-0+ +static bool dmirror_device_is_mine(struct dmirror_device *mdevice, + struct page *page) +{ + if (!is_zone_device_page(page)) + return false; + return page->pgmap->ops = &dmirror_devmem_ops && + dmirror_page_to_device(page) = mdevice; +}
Use new owner stuff, right? Actually this is redunant now, the check should be just WARN_ON pageowner != self owner
Quoted 9 lines
lib/Kconfig.debug +9
-0+static int dmirror_do_fault(struct dmirror *dmirror, struct hmm_range *range) +{ + uint64_t *pfns = range->pfns; + unsigned long pfn; + + for (pfn = (range->start >> PAGE_SHIFT); + pfn < (range->end >> PAGE_SHIFT); + pfn++, pfns++) { + struct page *page;
lib/Kconfig.debug
Quoted 10 lines
lib/Kconfig.debug +10
-0+ void *entry; + + /* + * HMM_PFN_ERROR is returned if it is accessing invalid memory + * either because of memory error (hardware detected memory + * corruption) or more likely because of truncate on mmap + * file. + */ + if (*pfns = range->values[HMM_PFN_ERROR]) + return -EFAULT;
Unless that snapshot is use hmm_range_fault() never returns success and sets PFN_ERROR, so this should be a WARN_ON
lib/Kconfig.debug
Quoted 2 lines
lib/Kconfig.debug +2
-0+ if (!(*pfns & range->flags[HMM_PFN_VALID])) + return -EFAULT;
Same with valid.
lib/Kconfig.debug
Quoted 4 lines
lib/Kconfig.debug +4
-0+ page = hmm_device_entry_to_page(range, *pfns); + /* We asked for pages to be populated but check anyway. */ + if (!page) + return -EFAULT;
WARN_ON
lib/Kconfig.debug
Quoted 7 lines
lib/Kconfig.debug +7
-0+ if (is_zone_device_page(page)) { + /* + * TODO: need a way to ask HMM to fault foreign zone + * device private pages. + */ + if (!dmirror_device_is_mine(dmirror->mdevice, page)) + continue;
Actually re
lib/Kconfig.debug
Quoted 13 lines
lib/Kconfig.debug +13
-0+static bool dmirror_interval_invalidate(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq) +{ + struct dmirror *dmirror = container_of(mni, struct dmirror, notifier); + struct mm_struct *mm = dmirror->mm; + + /* + * If the process doesn't exist, we don't need to invalidate the + * device page table since the address space will be torn down. + */ + if (!mmget_not_zero(mm)) + return true;
Why? Don't the notifiers provide for this already.
mmget_not_zero() is required before calling hmm_range_fault() though
Quoted 13 lines
lib/Kconfig.debug +13
-0+static int dmirror_fault(struct dmirror *dmirror, unsigned long start, + unsigned long end, bool write) +{ + struct mm_struct *mm = dmirror->mm; + unsigned long addr; + uint64_t pfns[64]; + struct hmm_range range = { + .notifier = &dmirror->notifier, + .pfns = pfns, + .flags = dmirror_hmm_flags, + .values = dmirror_hmm_values, + .pfn_shift = DPT_SHIFT, + .pfn_flags_mask = ~(dmirror_hmm_flags[HMM_PFN_VALID] |
lib/Kconfig.debug
Quoted 10 lines
lib/Kconfig.debug +10
-0+ dmirror_hmm_flags[HMM_PFN_WRITE]), + .default_flags = dmirror_hmm_flags[HMM_PFN_VALID] | + (write ? dmirror_hmm_flags[HMM_PFN_WRITE] : 0), + .dev_private_owner = dmirror->mdevice, + }; + int ret = 0; + + /* Since the mm is for the mirrored process, get a reference first. */ + if (!mmget_not_zero(mm)) + return 0;
Right
Quoted 55 lines
lib/Kconfig.debug +55
-0+ for (addr = start; addr < end; addr = range.end) { + range.start = addr; + range.end = min(addr + (ARRAY_SIZE(pfns) << PAGE_SHIFT), end); + + ret = dmirror_range_fault(dmirror, &range); + if (ret) + break; + } + + mmput(mm); + return ret; +} + +static int dmirror_do_read(struct dmirror *dmirror, unsigned long start, + unsigned long end, struct dmirror_bounce *bounce) +{ + unsigned long pfn; + void *ptr; + + ptr = bounce->ptr + ((start - bounce->addr) & PAGE_MASK); + + for (pfn = start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++) { + void *entry; + struct page *page; + void *tmp; + + entry = xa_load(&dmirror->pt, pfn); + page = xa_untag_pointer(entry); + if (!page) + return -ENOENT; + + tmp = kmap(page); + memcpy(ptr, tmp, PAGE_SIZE); + kunmap(page); + + ptr += PAGE_SIZE; + bounce->cpages++; + } + + return 0; +} + +static int dmirror_read(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) +{ + struct dmirror_bounce bounce; + unsigned long start, end; + unsigned long size = cmd->npages << PAGE_SHIFT; + int ret; + + start = cmd->addr; + end = start + size; + if (end < start) + return -EINVAL; + + ret = dmirror_bounce_init(&bounce, start, size);
lib/Kconfig.debug
Quoted 10 lines
lib/Kconfig.debug +10
-0+ if (ret) + return ret; + +again: + mutex_lock(&dmirror->mutex); + ret = dmirror_do_read(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret = 0) + ret = copy_to_user((void __user *)cmd->ptr, bounce.ptr, + bounce.size);
Use u64_to_user_ptr() instead of the cast
Quoted 7 lines
lib/Kconfig.debug +7
-0+static int dmirror_write(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) +{ + struct dmirror_bounce bounce; + unsigned long start, end; + unsigned long size = cmd->npages << PAGE_SHIFT; + int ret; +
lib/Kconfig.debug
Quoted 10 lines
lib/Kconfig.debug +10
-0+ start = cmd->addr; + end = start + size; + if (end < start) + return -EINVAL; + + ret = dmirror_bounce_init(&bounce, start, size); + if (ret) + return ret; + ret = copy_from_user(bounce.ptr, (void __user *)cmd->ptr, + bounce.size);
ditto
lib/Kconfig.debug
Quoted 13 lines
lib/Kconfig.debug +13
-0+ if (ret) + return ret; + +again: + mutex_lock(&dmirror->mutex); + ret = dmirror_do_write(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret = -ENOENT) { + start = cmd->addr + (bounce.cpages << PAGE_SHIFT); + ret = dmirror_fault(dmirror, start, end, true); + if (ret = 0) { + cmd->faults++; + goto again;
Use a loop instead of goto?
Also I get this:
lib/test_hmm.c: In function ‘dmirror_devmem_fault_alloc_and_copy’:lib/test_hmm.c:1041:25: warning: unused variable ‘vma’ [-Wunused-variable]
1041 | struct vm_area_struct *vma = args->vma;
But this is a kernel bug, due to alloc_page_vma being a #define not a static inline and me having CONFIG_NUMA off in this .config
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Adding [email protected] for the test config question.
Quoted 30 lines · Jason Gunthorpe
Quoted 9 lines · Ralph Campbell
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
I've been using v7 of Ralph's tester and it is working well - it has
DEVICE_PRIVATEsupport so I think it can test this flow too. Ralph are you able?This hunk seems trivial enough to me, can we include it now?
Quoted 8 lines
Quoted 4 lines
I can send a separate patch for it once the tester covers it. I don't want to add it to the original patch as it is a significant behavior change compared to the existing code.
Attached is an updated version of my HMM tests based on linux-5.6.0-rc6. I ran this OK with Jason's 8+1 HMM patches, Christoph's 1-5 misc HMM clean ups, and Christoph's 1-4 device private page changes applied.
I'd like to get this to mergable, it looks pretty good now, but I have no idea about selftests - and I'm struggling to even compile the tools dir
Quoted 7 lines
lib/Kconfig.debug +1
-0+++ b/lib/Kconfig.debug@@ -2162,6 +2162,18 @@ config TEST_MEMINIT If unsure, say N.
Quoted 7 lines
lib/Kconfig.debug config TEST_MEMINIT · line 2169
Quoted 5 lines
lib/Kconfig.debug +5
-0+config TEST_HMM + tristate "Test HMM (Heterogeneous Memory Management)" + depends on DEVICE_PRIVATE + select HMM_MIRROR + select MMU_NOTIFIERextra spaces
Will fix in v8.
Quoted 2 lines
In general I wonder if it even makes sense that
DEVICE_PRIVATEis user selectable?
Should tests enable the feature or the feature enable the test? IMHO, if the feature is being compiled into the kernel, that should enable the menu item for the test. If the feature isn't selected, no need to test it :-)
Quoted 19 lines
Quoted 19 lines
lib/Kconfig.debug +19
-0+static int dmirror_fops_open(struct inode *inode, struct file *filp) +{ + struct cdev *cdev = inode->i_cdev; + struct dmirror *dmirror; + int ret; + + /* Mirror this process address space */ + dmirror = kzalloc(sizeof(*dmirror), GFP_KERNEL); + if (dmirror == NULL) + return -ENOMEM; + + dmirror->mdevice = container_of(cdev, struct dmirror_device, cdevice); + mutex_init(&dmirror->mutex); + xa_init(&dmirror->pt); + + ret = mmu_interval_notifier_insert(&dmirror->notifier, current->mm, + 0, ULONG_MAX & PAGE_MASK, &dmirror_min_ops); + if (ret) { + kfree(dmirror);
Quoted 8 lines
lib/Kconfig.debug
Quoted 6 lines
lib/Kconfig.debug +6
-0+ return ret; + } + + /* Pairs with the mmdrop() in dmirror_fops_release(). */ + mmgrab(current->mm); + dmirror->mm = current->mm;The notifier holds a mmgrab, no need for another one
OK. I'll replace dmirror->mm with dmirror->notifier.mm.
Quoted 4 lines
lib/Kconfig.debug
Quoted 2 lines
lib/Kconfig.debug +2
-0+ /* Only the first open registers the address space. */ + filp->private_data = dmirror;Not sure what this comment means
I'll change the comment to:
/*
* The first open of the device character file registers the address
* space of the process doing the open() system call with the device.
* Subsequent file opens by other processes will have access to the
* first process' address space.
*/
Quoted 10 lines
lib/Kconfig.debug
Quoted 8 lines
lib/Kconfig.debug +8
-0+static inline struct dmirror_device *dmirror_page_to_device(struct page *page) + +{ + struct dmirror_chunk *devmem; + + devmem = container_of(page->pgmap, struct dmirror_chunk, pagemap); + return devmem->mdevice; +}extra devmem var is not really needed
I'll change this to:
return container_of(page->pgmap, struct dmirror_chunk, pagemap)->mdevice;
Quoted 4 lines
Quoted 4 lines
lib/Kconfig.debug +4
-0+ +static bool dmirror_device_is_mine(struct dmirror_device *mdevice, + struct page *page) +{
Quoted 8 lines
lib/Kconfig.debug
Quoted 5 lines
lib/Kconfig.debug +5
-0+ if (!is_zone_device_page(page)) + return false; + return page->pgmap->ops == &dmirror_devmem_ops && + dmirror_page_to_device(page) == mdevice; +}Use new owner stuff, right? Actually this is redunant now, the check should be just
WARN_ONpageowner != self owner
I'll clean this up. dmirror_device_is_mine() isn't needed now.
Quoted 61 lines
Quoted 9 lines
lib/Kconfig.debug +9
-0+static int dmirror_do_fault(struct dmirror *dmirror, struct hmm_range *range) +{ + uint64_t *pfns = range->pfns; + unsigned long pfn; + + for (pfn = (range->start >> PAGE_SHIFT); + pfn < (range->end >> PAGE_SHIFT); + pfn++, pfns++) { + struct page *page;lib/Kconfig.debug
Quoted 10 lines
lib/Kconfig.debug +10
-0+ void *entry; + + /* + * HMM_PFN_ERROR is returned if it is accessing invalid memory + * either because of memory error (hardware detected memory + * corruption) or more likely because of truncate on mmap + * file. + */ + if (*pfns == range->values[HMM_PFN_ERROR]) + return -EFAULT;Unless that snapshot is use
hmm_range_fault()never returns success and setsPFN_ERROR, so this should be aWARN_ONlib/Kconfig.debug
Quoted 2 lines
lib/Kconfig.debug +2
-0+ if (!(*pfns & range->flags[HMM_PFN_VALID])) + return -EFAULT;Same with valid.
lib/Kconfig.debug
Quoted 4 lines
lib/Kconfig.debug +4
-0+ page = hmm_device_entry_to_page(range, *pfns); + /* We asked for pages to be populated but check anyway. */ + if (!page) + return -EFAULT;
WARN_ONlib/Kconfig.debug
Quoted 7 lines
lib/Kconfig.debug +7
-0+ if (is_zone_device_page(page)) { + /* + * TODO: need a way to ask HMM to fault foreign zone + * device private pages. + */ + if (!dmirror_device_is_mine(dmirror->mdevice, page)) + continue;Actually re
lib/Kconfig.debug
Quoted 13 lines
lib/Kconfig.debug +13
-0+static bool dmirror_interval_invalidate(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq) +{ + struct dmirror *dmirror = container_of(mni, struct dmirror, notifier); + struct mm_struct *mm = dmirror->mm; + + /* + * If the process doesn't exist, we don't need to invalidate the + * device page table since the address space will be torn down. + */ + if (!mmget_not_zero(mm)) + return true;Why? Don't the notifiers provide for this already.
Quoted 1 line
mmget_not_zero()is required before callinghmm_range_fault()though
This is a workaround for a problem I don't quite understand. If you change tools/testing/selftests/vm/hmm-tests.c line 868 to
ASSERT_EQ(ret, -1);
Then the test will abort, core dump, and cause two problems,
1) the migrated page will be faulted back to system memory in order to write
it to the core dump. This triggers lockdep_assert_held(&walk.mm->mmap_sem)
in walk_page_range().
2) Then after a delay, I get:
[ 137.852986] rcu: INFO: rcu_sched self-detected stall on CPU
[ 137.858594] rcu: 0-....: (26000 ticks this GP) idle=69e/1/0x4000000000000002 softirq=34555/34555 fqs=6497
[ 137.868439] (t=26007 jiffies g=14653 q=271)
[ 137.872711] NMI backtrace for cpu 0
[ 137.876205] CPU: 0 PID: 6228 Comm: hmm-tests Not tainted 5.6.0-rc6+ #2
[ 137.882730] Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4302 08/29/2013
[ 137.892115] Call Trace:
[ 137.894570] <IRQ>
[ 137.896593] dump_stack+0x97/0xe0
[ 137.899920] nmi_cpu_backtrace.cold+0x14/0x68
[ 137.904287] ? lapic_can_unplug_cpu.cold+0x39/0x39
[ 137.909091] nmi_trigger_cpumask_backtrace+0xf1/0x10e
[ 137.914152] rcu_dump_cpu_stacks+0xe2/0x125
[ 137.918348] rcu_sched_clock_irq.cold+0x393/0x610
[ 137.923069] update_process_times+0x24/0x50
[ 137.927263] tick_sched_handle+0x68/0x90
[ 137.931196] tick_sched_timer+0x38/0xa0
[ 137.935037] __hrtimer_run_queues+0x1f9/0x6c0
[ 137.939403] ? tick_sched_do_timer+0x90/0x90
[ 137.943683] ? enqueue_hrtimer+0x1a0/0x1a0
[ 137.947790] ? recalibrate_cpu_khz+0x10/0x10
[ 137.952064] ? ktime_get_update_offsets_now+0xed/0x1c0
[ 137.957209] hrtimer_interrupt+0x1a5/0x340
[ 137.961315] ? rcu_read_lock_sched_held+0xa1/0xd0
[ 137.966038] smp_apic_timer_interrupt+0xbb/0x320
[ 137.970665] apic_timer_interrupt+0xf/0x20
[ 137.974771] </IRQ>
[ 137.976876] RIP: 0010:xas_load+0x7/0x80
[ 137.980718] Code: 80 2f 1a 83 c6 05 e9 8d 7b 01 01 e8 3e b1 b1 fe e9 05 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 41 56 41 55 41 54 55 <48> 89 fd 53 4c 8d 6d 10 e8 3c fc ff ff 49 89 c4 4c 89 e0 83 e0 03
[ 137.999461] RSP: 0018:ffffc900015e77c8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 138.007028] RAX: ffff8886e508c408 RBX: 0000000000000000 RCX: ffffffff82626c89
[ 138.014159] RDX: dffffc0000000000 RSI: 0000000000000000 RDI: ffffc900015e78a0
[ 138.021293] RBP: ffffc900015e78a0 R08: ffffffff811461c4 R09: fffff520002bcf17
[ 138.028426] R10: fffff520002bcf16 R11: 0000000000000003 R12: 0000000002606d10
[ 138.035557] R13: ffff8886e508c448 R14: 0000000000000031 R15: ffffffffa06546a0
[ 138.042701] ? do_raw_spin_lock+0x104/0x1d0
[ 138.046888] ? xas_store+0x19/0xa60
[ 138.050390] xas_store+0x5b3/0xa60
[ 138.053806] ? register_lock_class+0x860/0x860
[ 138.058267] __xa_erase+0x96/0x110
[ 138.061673] ? xas_store+0xa60/0xa60
[ 138.065267] xa_erase+0x19/0x30
[ 138.068418] dmirror_interval_invalidate+0x7d/0xc0 [test_hmm]
[ 138.074174] __mmu_notifier_release+0x1a6/0x370
[ 138.078714] ? mmu_notifier_unregister+0x1e0/0x1e0
[ 138.083520] ? lock_downgrade+0x380/0x380
[ 138.087535] ? uprobe_clear_state+0x2e/0x150
[ 138.091823] exit_mmap+0x24d/0x2a0
[ 138.095229] ? do_munmap+0x10/0x10
[ 138.098635] ? __x64_sys_io_setup+0x200/0x200
[ 138.102995] ? __mutex_unlock_slowpath+0xb4/0x3f0
[ 138.107704] ? wait_for_completion+0x250/0x250
[ 138.112158] ? lock_downgrade+0x380/0x380
[ 138.116176] ? check_flags.part.0+0x82/0x210
[ 138.120463] mmput+0xb5/0x210
[ 138.123444] do_exit+0x602/0x14c0
[ 138.126776] ? mm_update_next_owner+0x400/0x400
[ 138.131329] do_group_exit+0x8a/0x140
[ 138.135006] get_signal+0x25b/0x1080
[ 138.138606] do_signal+0x8c/0xa90
[ 138.141928] ? _raw_spin_unlock_irq+0x24/0x30
[ 138.146292] ? mark_held_locks+0x24/0x90
[ 138.150219] ? _raw_spin_unlock_irq+0x24/0x30
[ 138.154580] ? lockdep_hardirqs_on+0x190/0x280
[ 138.159026] ? setup_sigcontext+0x260/0x260
[ 138.163210] ? sigprocmask+0x10b/0x150
[ 138.166965] ? __x64_sys_rt_sigsuspend+0xe0/0xe0
[ 138.171594] ? __x64_sys_rt_sigprocmask+0xfb/0x180
[ 138.176394] ? __ia32_compat_sys_rt_sigprocmask+0x190/0x190
[ 138.181965] ? entry_SYSCALL_64_after_hwframe+0x3e/0xbe
[ 138.187192] ? exit_to_usermode_loop+0x60/0x100
[ 138.191723] ? mark_held_locks+0x24/0x90
[ 138.195656] exit_to_usermode_loop+0x85/0x100
[ 138.200023] do_syscall_64+0x20b/0x290
[ 138.203782] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 138.208839] RIP: 0033:0x7f2d349f1625
[ 138.212420] Code: c2 b8 ea 00 00 00 0f 05 48 3d 00 f0 ff ff 77 3d 41 89 c0 41 ba 08 00 00 00 31 d2 4c 89 ce bf 02 00 00 00 b8 0e 00 00 00 0f 05 <48> 8b 84 24 08 01 00 00 64 48 33 04 25 28 00 00 00 75 24 44 89 c0
[ 138.231163] RSP: 002b:00007ffe84228c60 EFLAGS: 00000246 ORIG_RAX: 000000000000000e
[ 138.238731] RAX: 0000000000000000 RBX: 00007f2d349ad040 RCX: 00007f2d349f1625
[ 138.245862] RDX: 0000000000000000 RSI: 00007ffe84228c60 RDI: 0000000000000002
[ 138.252997] RBP: 00007ffe84228ec0 R08: 0000000000000000 R09: 00007ffe84228c60
[ 138.260127] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000401240
[ 138.267261] R13: 00007ffe84229190 R14: 0000000000000000 R15: 0000000000000000
[ 138.274554] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 0-... } 26428 jiffies s: 277 root: 0x1/.
[ 138.285167] rcu: blocking rcu_node structures:
[ 138.289641] Task dump for CPU 0:
[ 138.292890] hmm-tests R running task 27560 6228 6211 0x8000400a
[ 138.300002] Call Trace:
[ 138.302486] ? check_chain_key+0x1d1/0x2c0
[ 138.306627] ? __lock_acquire+0x61c/0x2820
[ 138.310760] ? match_held_lock+0x1b/0x230
[ 138.314800] ? check_chain_key+0x1d1/0x2c0
[ 138.318937] ? lock_downgrade+0x380/0x380
[ 138.323008] ? lock_acquire+0xff/0x220
[ 138.326795] ? stack_depot_save+0x137/0x450
[ 138.331026] ? _raw_spin_unlock_irqrestore+0x3e/0x50
[ 138.336026] ? mark_held_locks+0x24/0x90
[ 138.340012] ? _raw_spin_unlock_irqrestore+0x3e/0x50
[ 138.345022] ? lockdep_hardirqs_on+0x190/0x280
[ 138.349504] ? stack_depot_save+0x253/0x450
[ 138.353731] ? check_chain_key+0x1d1/0x2c0
[ 138.357871] ? __lock_acquire+0x61c/0x2820
[ 138.362033] ? match_held_lock+0x1b/0x230
[ 138.366070] ? check_chain_key+0x1d1/0x2c0
[ 138.370209] ? mark_lock+0xac/0x9e0
[ 138.373734] ? mark_lock+0xac/0x9e0
[ 138.377261] ? mark_lock+0xac/0x9e0
[ 138.380772] ? mark_held_locks+0x65/0x90
[ 138.384730] ? mark_lock+0xac/0x9e0
[ 138.388249] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 138.392895] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 138.397542] ? lockdep_hardirqs_on+0x190/0x280
[ 138.402029] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 138.406676] ? mark_lock+0xac/0x9e0
[ 138.410209] ? lock_is_held_type+0x110/0x140
[ 138.414509] ? mark_held_locks+0x65/0x90
[ 138.418461] ? match_held_lock+0x1b/0x230
[ 138.422500] ? match_held_lock+0x1b/0x230
[ 138.426551] ? __lock_acquire+0x11c3/0x2820
[ 138.430773] ? xas_load+0x64/0x80
[ 138.434126] ? xas_store+0xac/0xa60
[ 138.437652] ? register_lock_class+0x860/0x860
[ 138.442140] ? lock_downgrade+0x380/0x380
[ 138.446193] ? _raw_spin_lock+0x2c/0x40
[ 138.450059] ? _raw_spin_unlock+0x17/0x30
[ 138.454100] ? xa_erase+0xe/0x30
[ 138.457370] ? dmirror_interval_invalidate+0x7d/0xc0 [test_hmm]
[ 138.463325] ? __mmu_notifier_release+0x1a6/0x370
[ 138.468065] ? mmu_notifier_unregister+0x1e0/0x1e0
[ 138.472896] ? lock_downgrade+0x380/0x380
[ 138.476937] ? uprobe_clear_state+0x2e/0x150
[ 138.481252] ? exit_mmap+0x24d/0x2a0
[ 138.484859] ? do_munmap+0x10/0x10
[ 138.488298] ? __x64_sys_io_setup+0x200/0x200
[ 138.492683] ? __mutex_unlock_slowpath+0xb4/0x3f0
[ 138.497418] ? wait_for_completion+0x250/0x250
[ 138.501897] ? lock_downgrade+0x380/0x380
[ 138.505944] ? check_flags.part.0+0x82/0x210
[ 138.510254] ? mmput+0xb5/0x210
[ 138.513435] ? do_exit+0x602/0x14c0
[ 138.516967] ? mm_update_next_owner+0x400/0x400
[ 138.521545] ? do_group_exit+0x8a/0x140
[ 138.525423] ? get_signal+0x25b/0x1080
[ 138.529225] ? do_signal+0x8c/0xa90
[ 138.532755] ? _raw_spin_unlock_irq+0x24/0x30
[ 138.537143] ? mark_held_locks+0x24/0x90
[ 138.541096] ? _raw_spin_unlock_irq+0x24/0x30
[ 138.545492] ? lockdep_hardirqs_on+0x190/0x280
[ 138.549971] ? setup_sigcontext+0x260/0x260
[ 138.554190] ? sigprocmask+0x10b/0x150
[ 138.557969] ? __x64_sys_rt_sigsuspend+0xe0/0xe0
[ 138.562627] ? __x64_sys_rt_sigprocmask+0xfb/0x180
[ 138.567452] ? __ia32_compat_sys_rt_sigprocmask+0x190/0x190
[ 138.573057] ? entry_SYSCALL_64_after_hwframe+0x3e/0xbe
[ 138.578311] ? exit_to_usermode_loop+0x60/0x100
[ 138.582875] ? mark_held_locks+0x24/0x90
[ 138.586837] ? exit_to_usermode_loop+0x85/0x100
[ 138.591400] ? do_syscall_64+0x20b/0x290
[ 138.595360] ? entry_SYSCALL_64_after_hwframe+0x49/0xbe./test_hmm.sh: line 58: 6211 Alarm clock ./hmm-tests
Quoted 85 lines
Quoted 13 lines
lib/Kconfig.debug +13
-0+static int dmirror_fault(struct dmirror *dmirror, unsigned long start, + unsigned long end, bool write) +{ + struct mm_struct *mm = dmirror->mm; + unsigned long addr; + uint64_t pfns[64]; + struct hmm_range range = { + .notifier = &dmirror->notifier, + .pfns = pfns, + .flags = dmirror_hmm_flags, + .values = dmirror_hmm_values, + .pfn_shift = DPT_SHIFT, + .pfn_flags_mask = ~(dmirror_hmm_flags[HMM_PFN_VALID] |lib/Kconfig.debug
Quoted 10 lines
lib/Kconfig.debug +10
-0+ dmirror_hmm_flags[HMM_PFN_WRITE]), + .default_flags = dmirror_hmm_flags[HMM_PFN_VALID] | + (write ? dmirror_hmm_flags[HMM_PFN_WRITE] : 0), + .dev_private_owner = dmirror->mdevice, + }; + int ret = 0; + + /* Since the mm is for the mirrored process, get a reference first. */ + if (!mmget_not_zero(mm)) + return 0;Right
Quoted 59 lines
lib/Kconfig.debug +59
-0+ for (addr = start; addr < end; addr = range.end) { + range.start = addr; + range.end = min(addr + (ARRAY_SIZE(pfns) << PAGE_SHIFT), end); + + ret = dmirror_range_fault(dmirror, &range); + if (ret) + break; + } + + mmput(mm); + return ret; +} + +static int dmirror_do_read(struct dmirror *dmirror, unsigned long start, + unsigned long end, struct dmirror_bounce *bounce) +{ + unsigned long pfn; + void *ptr; + + ptr = bounce->ptr + ((start - bounce->addr) & PAGE_MASK); + + for (pfn = start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++) { + void *entry; + struct page *page; + void *tmp; + + entry = xa_load(&dmirror->pt, pfn); + page = xa_untag_pointer(entry); + if (!page) + return -ENOENT; + + tmp = kmap(page); + memcpy(ptr, tmp, PAGE_SIZE); + kunmap(page); + + ptr += PAGE_SIZE; + bounce->cpages++; + } + + return 0; +} + +static int dmirror_read(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) +{ + struct dmirror_bounce bounce; + unsigned long start, end; + unsigned long size = cmd->npages << PAGE_SHIFT; + int ret; + + start = cmd->addr; + end = start + size; + if (end < start) + return -EINVAL; + + ret = dmirror_bounce_init(&bounce, start, size); + if (ret) + return ret; + +again:
Quoted 8 lines
lib/Kconfig.debug
Quoted 6 lines
lib/Kconfig.debug +6
-0+ mutex_lock(&dmirror->mutex); + ret = dmirror_do_read(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret == 0) + ret = copy_to_user((void __user *)cmd->ptr, bounce.ptr, + bounce.size);Use
u64_to_user_ptr()instead of the cast
Will do.
Quoted 20 lines
Quoted 7 lines
lib/Kconfig.debug +7
-0+static int dmirror_write(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) +{ + struct dmirror_bounce bounce; + unsigned long start, end; + unsigned long size = cmd->npages << PAGE_SHIFT; + int ret; +lib/Kconfig.debug
Quoted 10 lines
lib/Kconfig.debug +10
-0+ start = cmd->addr; + end = start + size; + if (end < start) + return -EINVAL; + + ret = dmirror_bounce_init(&bounce, start, size); + if (ret) + return ret; + ret = copy_from_user(bounce.ptr, (void __user *)cmd->ptr, + bounce.size);ditto
Quoted 15 lines
lib/Kconfig.debug
Quoted 13 lines
lib/Kconfig.debug +13
-0+ if (ret) + return ret; + +again: + mutex_lock(&dmirror->mutex); + ret = dmirror_do_write(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret == -ENOENT) { + start = cmd->addr + (bounce.cpages << PAGE_SHIFT); + ret = dmirror_fault(dmirror, start, end, true); + if (ret == 0) { + cmd->faults++; + goto again;Use a loop instead of goto?
OK.
Quoted 8 lines
Also I get this:
lib/test_hmm.c: In function ‘dmirror_devmem_fault_alloc_and_copy’:lib/test_hmm.c:1041:25: warning: unused variable ‘vma’ [-Wunused-variable]
1041 | structvm_area_struct*vma =args->vma;But this is a kernel bug, due to
alloc_page_vmabeing a #define not a static inline and me havingCONFIG_NUMAoff in this .config
Fixed.
I'll repost as a proper series shortly.
Quoted 2 lines
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 41 lines · Ralph Campbell
Adding [email protected] for the test config question.
Quoted 23 lines · Jason Gunthorpe
Quoted 9 lines · Ralph Campbell
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Jason Gunthorpe
I've been using v7 of Ralph's tester and it is working well - it has
DEVICE_PRIVATEsupport so I think it can test this flow too. Ralph are you able?This hunk seems trivial enough to me, can we include it now?
Quoted 8 lines
Quoted 4 lines
I can send a separate patch for it once the tester covers it. I don't want to add it to the original patch as it is a significant behavior change compared to the existing code.
Attached is an updated version of my HMM tests based on linux-5.6.0-rc6. I ran this OK with Jason's 8+1 HMM patches, Christoph's 1-5 misc HMM clean ups, and Christoph's 1-4 device private page changes applied.
I'd like to get this to mergable, it looks pretty good now, but I have no idea about selftests - and I'm struggling to even compile the tools dir
Quoted 12 lines
lib/Kconfig.debug config TEST_MEMINIT · line 2167
Quoted 10 lines
lib/Kconfig.debug +6
-0+++ b/lib/Kconfig.debug@@ -2162,6 +2162,18 @@ config TEST_MEMINIT If unsure, say N. +config TEST_HMM + tristate "Test HMM (Heterogeneous Memory Management)" + depends on DEVICE_PRIVATE + select HMM_MIRROR + select MMU_NOTIFIERextra spaces
Will fix in v8.
Quoted 7 lines
Quoted 2 lines
In general I wonder if it even makes sense that
DEVICE_PRIVATEis user selectable?Should tests enable the feature or the feature enable the test? IMHO, if the feature is being compiled into the kernel, that should enable the menu item for the test. If the feature isn't selected, no need to test it :-)
I ment if DEVICE_PRIVATE should be a user selectable option at all, or should it be turned on when a driver like nouveau is selected.
Is there some downside to enabling DEVICE_PRIVATE?
Quoted 3 lines
Quoted 1 line
The notifier holds a mmgrab, no need for another one
OK. I'll replace
dmirror->mmwithdmirror->notifier.mm.
Right that is good too
Quoted 11 lines
Quoted 3 lines
lib/Kconfig.debug config TEST_MEMINIT · line 2168
Quoted 1 line
lib/Kconfig.debug +1
-0+ filp->private_data = dmirror;Not sure what this comment means
I'll change the comment to:
/*
* The first open of the device character file registers the address
* space of the process doing theopen()system call with the device.
* Subsequent file opens by other processes will have access to the
* first process' address space.
*/
How does this happen? The function looks like it always does the same thing
Quoted 16 lines
Quoted 16 lines
lib/Kconfig.debug config TEST_MEMINIT
Quoted 13 lines
lib/Kconfig.debug +13
-0+static bool dmirror_interval_invalidate(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq) +{ + struct dmirror *dmirror = container_of(mni, struct dmirror, notifier); + struct mm_struct *mm = dmirror->mm; + + /* + * If the process doesn't exist, we don't need to invalidate the + * device page table since the address space will be torn down. + */ + if (!mmget_not_zero(mm)) + return true;Why? Don't the notifiers provide for this already.
Quoted 1 line
Quoted 1 line
mmget_not_zero()is required before callinghmm_range_fault()though
Oh... This is the invalidate_all path during invalidation
IMHO you should test the invalidation reason in the range to exclude this.
But xa_erase looks totally safe so there should be no reason to do that.
Quoted 7 lines
This is a workaround for a problem I don't quite understand. If you change
tools/testing/selftests/vm/hmm-tests.cline 868 toASSERT_EQ(ret, -1);Then the test will abort, core dump, and cause two problems,
1) the migrated page will be faulted back to system memory in order to write it to the core dump. This triggerslockdep_assert_held(&walk.mm->mmap_sem)inwalk_page_range().
Has the migration stuff become entangled with the xarray?
Quoted 6 lines
[ 137.980718] Code: 80 2f 1a 83 c6 05 e9 8d 7b 01 01 e8 3e b1 b1 fe e9 05 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 41 56 41 55 41 54 55 <48> 89 fd 53 4c 8d 6d 10 e8 3c fc ff ff 49 89 c4 4c 89 e0 83 e0 03 [ 137.999461] RSP: 0018:ffffc900015e77c8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 138.007028] RAX: ffff8886e508c408 RBX: 0000000000000000 RCX: ffffffff82626c89 [ 138.014159] RDX: dffffc0000000000 RSI: 0000000000000000 RDI: ffffc900015e78a0 [ 138.021293] RBP: ffffc900015e78a0 R08: ffffffff811461c4 R09: fffff520002bcf17 [ 138.028426] R10: fffff520002bcf16 R11: 0000000000000003 R12: 0000000002606d10
Quoted 8 lines
[ 138.035557] R13: ffff8886e508c448 R14: 0000000000000031 R15: ffffffffa06546a0 [ 138.042701] ? do_raw_spin_lock+0x104/0x1d0 [ 138.046888] ? xas_store+0x19/0xa60 [ 138.050390] xas_store+0x5b3/0xa60 [ 138.053806] ? register_lock_class+0x860/0x860 [ 138.058267] __xa_erase+0x96/0x110 [ 138.061673] ? xas_store+0xa60/0xa60 [ 138.065267] xa_erase+0x19/0x30
oh, it is doing this:
static void mn_itree_release(struct mmu_notifier_subscriptions *subscriptions,
struct mm_struct *mm)
{
struct mmu_notifier_range range = {
.flags = MMU_NOTIFIER_RANGE_BLOCKABLE,
.event = MMU_NOTIFY_RELEASE,
.mm = mm,
.start = 0,
.end = ULONG_MAX,
};ie it is sitting doing a huge number of xa_erases, I suppose. Probably in normal exit the notifier is removed before the mm is destroyed.
The xa_erase needs to be a bit smarter to jump over gaps in the tree perhaps some
xa_for_each()
xa_erase()
pattern?
Quoted 10 lines
Quoted 8 lines
Also I get this:
lib/test_hmm.c: In function ‘dmirror_devmem_fault_alloc_and_copy’:lib/test_hmm.c:1041:25: warning: unused variable ‘vma’ [-Wunused-variable]
1041 | structvm_area_struct*vma =args->vma;But this is a kernel bug, due to
alloc_page_vmabeing a #define not a static inline and me havingCONFIG_NUMAoff in this .configFixed.
in gfp.h?
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines
Should tests enable the feature or the feature enable the test? IMHO, if the feature is being compiled into the kernel, that should enable the menu item for the test. If the feature isn't selected, no need to test it :-)
I ment if
DEVICE_PRIVATEshould be a user selectable option at all, or should it be turned on when a driver like nouveau is selected.
I don't think it should be user selectable. This is an implementation detail users can't know about.
Quoted 1 line
Is there some downside to enabling
DEVICE_PRIVATE?
The option itself adds a little more code to the core kernel, and introduces a few additional branches in core mm code.
But more importantly it pulls in the whole pgmap infrastructure.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
In reply to Ralph CampbellQuoted 4 lines · Ralph Campbell
lib/test_hmm.c new file +4
-0+static int dmirror_fault (struct dmirror *dmirror,unsigned long start, +unsigned long end,bool write) +{ +struct mm_struct *mm = dmirror->mm;
lib/test_hmm.c line 341
Quoted 10 lines
lib/test_hmm.c new file +10
-0+unsigned long addr; +uint64_t pfns[64 ]; +struct hmm_range range = { + .notifier = &dmirror->notifier, + .pfns = pfns, + .flags = dmirror_hmm_flags, + .values = dmirror_hmm_values, + .pfn_shift =DPT_SHIFT , + .pfn_flags_mask = ~(dmirror_hmm_flags[HMM_PFN_VALID ] | + dmirror_hmm_flags[HMM_PFN_WRITE ]),
Since pfns is not initialized pfn_flags_mask should be 0.
Quoted 13 lines
lib/test_hmm.c new file +12
-0+ .default_flags = dmirror_hmm_flags[HMM_PFN_VALID] | + (write ? dmirror_hmm_flags[HMM_PFN_WRITE] : 0), + .dev_private_owner = dmirror->mdevice, + }; +int ret =0 ; +static int dmirror_snapshot (struct dmirror *dmirror, +struct hmm_dmirror_cmd *cmd) +{ +struct mm_struct *mm = dmirror->mm; +unsigned long start, end; +unsigned long size = cmd->npages <<PAGE_SHIFT ; +unsigned long addr;
lib/test_hmm.c line 919
Quoted 10 lines
lib/test_hmm.c new file +10
-0+unsigned long next; +uint64_t pfns[64 ]; +unsigned char perm[64 ]; +char __user *uptr; +struct hmm_range range = { + .pfns = pfns, + .flags = dmirror_hmm_flags, + .values = dmirror_hmm_values, + .pfn_shift =DPT_SHIFT , + .pfn_flags_mask = ~0ULL ,
Same here, especially since this is snapshot
Jason
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 18 lines · Jason Gunthorpe
Quoted 4 lines · Ralph Campbell
lib/test_hmm.c new file +4
-0+static int dmirror_fault (struct dmirror *dmirror,unsigned long start, +unsigned long end,bool write) +{ +struct mm_struct *mm = dmirror->mm;lib/test_hmm.c line 341
Quoted 10 lines
lib/test_hmm.c new file +10
-0+unsigned long addr; +uint64_t pfns[64 ]; +struct hmm_range range = { + .notifier = &dmirror->notifier, + .pfns = pfns, + .flags = dmirror_hmm_flags, + .values = dmirror_hmm_values, + .pfn_shift =DPT_SHIFT , + .pfn_flags_mask = ~(dmirror_hmm_flags[HMM_PFN_VALID ] | + dmirror_hmm_flags[HMM_PFN_WRITE ]),Since pfns is not initialized
pfn_flags_maskshould be 0.
Good point.
Quoted 26 lines
Quoted 13 lines
lib/test_hmm.c new file +12
-0+ .default_flags = dmirror_hmm_flags[HMM_PFN_VALID] | + (write ? dmirror_hmm_flags[HMM_PFN_WRITE] : 0), + .dev_private_owner = dmirror->mdevice, + }; +int ret =0 ; +static int dmirror_snapshot (struct dmirror *dmirror, +struct hmm_dmirror_cmd *cmd) +{ +struct mm_struct *mm = dmirror->mm; +unsigned long start, end; +unsigned long size = cmd->npages <<PAGE_SHIFT ; +unsigned long addr;lib/test_hmm.c line 919
Quoted 10 lines
lib/test_hmm.c new file +10
-0+unsigned long next; +uint64_t pfns[64 ]; +unsigned char perm[64 ]; +char __user *uptr; +struct hmm_range range = { + .pfns = pfns, + .flags = dmirror_hmm_flags, + .values = dmirror_hmm_values, + .pfn_shift =DPT_SHIFT , + .pfn_flags_mask = ~0ULL ,Same here, especially since this is snapshot
Quoted 1 line
Jason
Actually, snapshot ignores pfn_flags_mask and default_flags.
In hmm_pte_need_fault(), HMM_FAULT_SNAPSHOT is checked and returns early before checking pfn_flags_mask and default_flags since no faults are being requested.
Re: [PATCH 3/4] mm: simplify device private page handling in hmm_range_fault
Quoted 26 lines · Ralph Campbell
Quoted 26 lines
Quoted 13 lines
lib/test_hmm.c new file +12
-0+ .default_flags = dmirror_hmm_flags[HMM_PFN_VALID] | + (write ? dmirror_hmm_flags[HMM_PFN_WRITE] : 0), + .dev_private_owner = dmirror->mdevice, + }; +int ret =0 ; +static int dmirror_snapshot (struct dmirror *dmirror, +struct hmm_dmirror_cmd *cmd) +{ +struct mm_struct *mm = dmirror->mm; +unsigned long start, end; +unsigned long size = cmd->npages <<PAGE_SHIFT ; +unsigned long addr;lib/test_hmm.c line 919
Quoted 10 lines
lib/test_hmm.c new file +10
-0+unsigned long next; +uint64_t pfns[64 ]; +unsigned char perm[64 ]; +char __user *uptr; +struct hmm_range range = { + .pfns = pfns, + .flags = dmirror_hmm_flags, + .values = dmirror_hmm_values, + .pfn_shift =DPT_SHIFT , + .pfn_flags_mask = ~0ULL ,Same here, especially since this is snapshot
Quoted 3 lines
Quoted 1 line
Jason
Actually, snapshot ignores
pfn_flags_maskanddefault_flags.
Yes, so no reason to set them to not 0..
Jason
PATCH 4/4 mm: check the device private page owner in hmm_range_fault
In reply to Christoph HellwigHmm range fault will succeed for any kind of device private memory, even if it doesn't belong to the calling entity. While nouveau has some crude checks for that, they are broken because they assume nouveau is the only user of device private memory. Fix this by passing in an expected pgmap owner in the hmm_range_fault structure.
--- drivers/gpu/drm/nouveau/nouveau_dmem.c | 12 ------------ include/linux/hmm.h | 2 ++ mm/hmm.c | 10 +++++++++- 3 files changed, 11 insertions(+), 13 deletions(-)
drivers/gpu/drm/nouveau/nouveau_dmem.c +0 -12
@@ -672,12 +672,6 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm,return ret; } -static inline bool -nouveau_dmem_page (struct nouveau_drm *drm,struct page *page) -{ -return is_device_private_page (page) && drm->dmem =page_to_dmem (page); -} -void nouveau_dmem_convert_pfn (struct nouveau_drm *drm,struct hmm_range *range)
@@ -696,12 +690,6 @@ nouveau_dmem_convert_pfn(struct nouveau_drm *drm,if (!is_device_private_page (page))continue ; -if (!nouveau_dmem_page (drm, page)) { -WARN (1 ,"Some unknown device memory !\n" ); - range->pfns[i] =0 ; -continue ; - } - addr =nouveau_dmem_page_addr (page); range->pfns[i] &= ((1UL << range->pfn_shift) -1 ); range->pfns[i] |= (addr >>PAGE_SHIFT ) << range->pfn_shift;
include/linux/hmm.h +2 -0
@@ -132,6 +132,7 @@ enum hmm_pfn_value_e {* @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter * @pfn_shifts: pfn shift value (should be <= PAGE_SHIFT) * @valid: pfns array did not change since it has been fill by an HMM function +* @dev_private_owner: owner of device private pages */ struct hmm_range {struct mmu_interval_notifier *notifier;
@@ -144,6 +145,7 @@ struct hmm_range {uint64_t default_flags;uint64_t pfn_flags_mask;uint8_t pfn_shift; +void *dev_private_owner; };/*
mm/hmm.c +9 -1
@@ -216,6 +216,14 @@ int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, unsigned long end, uint64_t *pfns, pmd_t pmd);#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline bool hmm_is_device_private_entry (struct hmm_range *range, +swp_entry_t entry) +{ +return is_device_private_entry (entry) && +device_private_entry_to_page (entry)->pgmap->owner = + range->dev_private_owner; +} +static inline uint64_t pte_to_hmm_pfn_flags (struct hmm_range *range,pte_t pte) {if (pte_none (pte) || !pte_present (pte) ||pte_protnone (pte))
@@ -254,7 +262,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,* Never fault in device private pages pages, but just report * the PFN even if not present. */ -if (is_device_private_entry (entry)) { +if (hmm_is_device_private_entry (range, entry)) { *pfn =hmm_device_entry_from_pfn (range,swp_offset (entry)); *pfn |= range->flags[HMM_PFN_VALID ];
Signature · 1 line
2.24.1
Re: [PATCH 4/4] mm: check the device private page owner in hmm_range_fault
Quoted 6 lines · Christoph Hellwig
Hmm range fault will succeed for any kind of device private memory, even if it doesn't belong to the calling entity. While nouveau has some crude checks for that, they are broken because they assume nouveau is the only user of device private memory. Fix this by passing in an expected pgmap owner in the
hmm_range_faultstructure.
Quoted 7 lines
Signed-off-by Christoph Hellwig <[email protected]>Fixes 4ef589dc9b10 ("mm/hmm/devmem: device memory hotplug using ZONE_DEVICE")--- drivers/gpu/drm/nouveau/nouveau_dmem.c | 12 ------------ include/linux/hmm.h | 2 ++ mm/hmm.c | 10 +++++++++- 3 files changed, 11 insertions(+), 13 deletions(-)
Nice
Jason
Re: [PATCH 4/4] mm: check the device private page owner in hmm_range_fault
In reply to Christoph HellwigQuoted 8 lines · Christoph Hellwig
Hmm range fault will succeed for any kind of device private memory, even if it doesn't belong to the calling entity. While nouveau has some crude checks for that, they are broken because they assume nouveau is the only user of device private memory. Fix this by passing in an expected pgmap owner in the
hmm_range_faultstructure.Signed-off-by Christoph Hellwig <[email protected]>Fixes 4ef589dc9b10 ("mm/hmm/devmem: device memory hotplug using ZONE_DEVICE")
Looks good.
Quoted 85 lines
--- drivers/gpu/drm/nouveau/nouveau_dmem.c | 12 ------------ include/linux/hmm.h | 2 ++ mm/hmm.c | 10 +++++++++- 3 files changed, 11 insertions(+), 13 deletions(-)drivers/gpu/drm/nouveau/nouveau_dmem.c +0
-12@@ -672,12 +672,6 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm,return ret; } -static inline bool -nouveau_dmem_page (struct nouveau_drm *drm,struct page *page) -{ -return is_device_private_page (page) && drm->dmem =page_to_dmem (page); -} -void nouveau_dmem_convert_pfn (struct nouveau_drm *drm,struct hmm_range *range)@@ -696,12 +690,6 @@ nouveau_dmem_convert_pfn(struct nouveau_drm *drm,if (!is_device_private_page (page))continue ; -if (!nouveau_dmem_page (drm, page)) { -WARN (1 ,"Some unknown device memory !\n" ); - range->pfns[i] =0 ; -continue ; - } - addr =nouveau_dmem_page_addr (page); range->pfns[i] &= ((1UL << range->pfn_shift) -1 ); range->pfns[i] |= (addr >>PAGE_SHIFT ) << range->pfn_shift;include/linux/hmm.h +2
-0@@ -132,6 +132,7 @@ enum hmm_pfn_value_e {* @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter * @pfn_shifts: pfn shift value (should be <= PAGE_SHIFT) * @valid: pfns array did not change since it has been fill by an HMM function +* @dev_private_owner: owner of device private pages */ struct hmm_range {struct mmu_interval_notifier *notifier;@@ -144,6 +145,7 @@ struct hmm_range {uint64_t default_flags;uint64_t pfn_flags_mask;uint8_t pfn_shift; +void *dev_private_owner; };/* mm/hmm.c +9
-1@@ -216,6 +216,14 @@ int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, unsigned long end, uint64_t *pfns, pmd_t pmd);#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline bool hmm_is_device_private_entry (struct hmm_range *range, +swp_entry_t entry) +{ +return is_device_private_entry (entry) && +device_private_entry_to_page (entry)->pgmap->owner = + range->dev_private_owner; +} +static inline uint64_t pte_to_hmm_pfn_flags (struct hmm_range *range,pte_t pte) {if (pte_none (pte) || !pte_present (pte) ||pte_protnone (pte))@@ -254,7 +262,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,* Never fault in device private pages pages, but just report * the PFN even if not present. */ -if (is_device_private_entry (entry)) { +if (hmm_is_device_private_entry (range, entry)) { *pfn =hmm_device_entry_from_pfn (range,swp_offset (entry)); *pfn |= range->flags[HMM_PFN_VALID ];
Re: [PATCH 4/4] mm: check the device private page owner in hmm_range_fault
In reply to Christoph HellwigQuoted 4 lines · Christoph Hellwig
mm/hmm.c +1
-0+++ b/mm/hmm.c
mm/hmm.c int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, · line 225
Quoted 10 lines
mm/hmm.c +7
-0@@ -216,6 +216,14 @@ int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, unsigned long end, uint64_t *pfns, pmd_t pmd);#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline bool hmm_is_device_private_entry (struct hmm_range *range, +swp_entry_t entry) +{ +return is_device_private_entry (entry) && +device_private_entry_to_page (entry)->pgmap->owner = + range->dev_private_owner; +}
Thinking about this some more, does the locking work out here?
hmm_range_fault() runs with mmap_sem in read, and does not lock any of the page table levels.
So it relies on accessing stale pte data being safe, and here we introduce for the first time a page pointer dereference and a pgmap dereference without any locking/refcounting.
The get_dev_pagemap() worked on the PFN and obtained a refcount, so it created safety.
Is there some tricky reason this is safe, eg a DEVICE_PRIVATE page cannot be removed from the vma without holding mmap_sem in write or something?
Jason
Re: [PATCH 4/4] mm: check the device private page owner in hmm_range_fault
Quoted 9 lines · Jason Gunthorpe
Thinking about this some more, does the locking work out here?
hmm_range_fault()runs withmmap_semin read, and does not lock any of the page table levels.So it relies on accessing stale pte data being safe, and here we introduce for the first time a page pointer dereference and a pgmap dereference without any locking/refcounting.
Quoted 6 lines
The
get_dev_pagemap()worked on the PFN and obtained a refcount, so it created safety.Is there some tricky reason this is safe, eg a
DEVICE_PRIVATEpage cannot be removed from the vma without holdingmmap_semin write or something?
I don't think there is any specific protection. Let me see if we can throw in a get_dev_pagemap here - note that current mainline doesn't even use it for this path..
Re: [PATCH 4/4] mm: check the device private page owner in hmm_range_fault
Quoted 13 lines · Christoph Hellwig
Quoted 12 lines · Jason Gunthorpe
Thinking about this some more, does the locking work out here?
hmm_range_fault()runs withmmap_semin read, and does not lock any of the page table levels.So it relies on accessing stale pte data being safe, and here we introduce for the first time a page pointer dereference and a pgmap dereference without any locking/refcounting.
The
get_dev_pagemap()worked on the PFN and obtained a refcount, so it created safety.
Quoted 6 lines
Quoted 3 lines
Is there some tricky reason this is safe, eg a
DEVICE_PRIVATEpage cannot be removed from the vma without holdingmmap_semin write or something?I don't think there is any specific protection. Let me see if we can throw in a
get_dev_pagemaphere
The page tables are RCU protected right? could we do something like
if (is_device_private_entry()) {
rcu_read_lock()
if (READ_ONCE(*ptep) != pte)
return -EBUSY;
hmm_is_device_private_entry()
rcu_read_unlock()}
?
Then pgmap needs a synchronize_rcu before the struct page's are destroyed (possibly gup_fast already requires this?)
I've got some other patches trying to close some of these styles of bugs, but
Quoted 1 line
note that current mainline doesn't even use it for this path..
Don't follow?
Jason
Re: [PATCH 4/4] mm: check the device private page owner in hmm_range_fault
Quoted 11 lines · Jason Gunthorpe
Quoted 2 lines
I don't think there is any specific protection. Let me see if we can throw in a
get_dev_pagemaphereThe page tables are RCU protected right? could we do something like
if (is_device_private_entry()) { rcu_read_lock() if (READ_ONCE(*ptep) != pte) return -EBUSY; hmm_is_device_private_entry() rcu_read_unlock()
Quoted 3 lines
}
?
Are they everywhere? I'd really love to hear from people that really know this ara..
Quoted 10 lines
Then pgmap needs a
synchronize_rcubefore the struct page's are destroyed (possiblygup_fastalready requires this?)I've got some other patches trying to close some of these styles of bugs, but
Quoted 1 line
note that current mainline doesn't even use it for this path..
Don't follow?
If you look at mainline (or any other tree), we only do a get_dev_pagemap for devmap ptes. But device private pages are encoded as non-present swap ptes.
Quoted 9 lines · Christoph Hellwig
When acting on device private mappings a driver needs to know if the device (or other entity in case of kvmppc) actually owns this private mapping. This series adds an owner field and converts the
migrate_vmacode over to check it. I looked into doing the same forhmm_range_fault, but as far as I can tell that code has never been wired up to actually work for device private memory, so instead of trying to fix some unused code the second patch just remove the code. We can add it back once we have a working and fully tested code, and then should pass the expected owner in thehmm_rangestructure.
Boot-tested a pseries secure guest with this change (1/4 and 2/4 only)
So Tested-by: Bharata B Rao <[email protected]> for the above two patches.
Regards,
Bharata.
Quoted 10 lines · Christoph Hellwig
When acting on device private mappings a driver needs to know if the device (or other entity in case of kvmppc) actually owns this private mapping. This series adds an owner field and converts the
migrate_vmacode over to check it. I looked into doing the same forhmm_range_fault, but as far as I can tell that code has never been wired up to actually work for device private memory, so instead of trying to fix some unused code the second patch just remove the code. We can add it back once we have a working and fully tested code, and then should pass the expected owner in thehmm_rangestructure.
Quoted 6 lines
Changes since v1:
- split out thepgmap->owneraddition into a separate patch
- checkpgmap->owneris set for device private mappings
- rename thedev_private_ownerfield in structmigrate_vmatosrc_owner
- refuse to migrate private pages ifsrc_owneris not set
- keep the non-fault device private handling inhmm_range_fault
I'm happy enough to take this, did you have plans for a v3?
Thanks,
Jason
Quoted 8 lines · Jason Gunthorpe
Quoted 6 lines
Changes since v1:
- split out thepgmap->owneraddition into a separate patch
- checkpgmap->owneris set for device private mappings
- rename thedev_private_ownerfield in structmigrate_vmatosrc_owner
- refuse to migrate private pages ifsrc_owneris not set
- keep the non-fault device private handling inhmm_range_faultI'm happy enough to take this, did you have plans for a v3?
I think the only open question is if merging 3 and 4 might make sense. It's up to you if you want it resent that way or not.
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Jason Gunthorpe
Quoted 6 lines
Changes since v1:
- split out thepgmap->owneraddition into a separate patch
- checkpgmap->owneris set for device private mappings
- rename thedev_private_ownerfield in structmigrate_vmatosrc_owner
- refuse to migrate private pages ifsrc_owneris not set
- keep the non-fault device private handling inhmm_range_fault
Quoted 4 lines
Quoted 1 line
I'm happy enough to take this, did you have plans for a v3?
I think the only open question is if merging 3 and 4 might make sense. It's up to you if you want it resent that way or not.
Now that I understand that amdgpu doesn't set the 'do not return device_private pages' flag, I think the split is fine, I'll grab it as is then today
Thanks,
Jason
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Jason Gunthorpe
Quoted 6 lines
Changes since v1:
- split out thepgmap->owneraddition into a separate patch
- checkpgmap->owneris set for device private mappings
- rename thedev_private_ownerfield in structmigrate_vmatosrc_owner
- refuse to migrate private pages ifsrc_owneris not set
- keep the non-fault device private handling inhmm_range_fault
Quoted 4 lines
Quoted 1 line
I'm happy enough to take this, did you have plans for a v3?
I think the only open question is if merging 3 and 4 might make sense. It's up to you if you want it resent that way or not.
Okay, I kept it as is and elaborated the commit messages a bit based on the discussion
It doesn't seem like the changes outside hmm are significant enough to need more acks
Thanks,
Jason