mm: support device-private THP
Subsystems (15)
v4 cover mm: support device-private THP
This patch series introduces support for Transparent Huge Page (THP) migration in zone device-private memory. The implementation enables efficient migration of large folios between system memory and device-private memory
Background
Current zone device-private memory implementation only supports PAGE_SIZE granularity, leading to:
- Increased TLB pressure
- Inefficient migration between CPU and GPU memory
This series extends the existing zone device-private infrastructure to support THP, leading to:
- Reduced page table overhead
- Improved memory bandwidth utilization
- Seamless fallback to base pages when needed
In my local testing (using lib/test_hmm) and a throughput test, the series shows a 4x improvement in data transfer throughput and a 5x improvement in latency
These patches build on the earlier posts by Ralph Campbell [1]
Two new flags are added in vma_migration to select and mark compound pages. migrate_vma_setup(), migrate_vma_pages() and migrate_vma_finalize() support migration of these pages when MIGRATE_VMA_SELECT_COMPOUND is passed in as arguments.
The series also adds zone device awareness to (m)THP pages along with fault handling of large zone device private pages. page vma walk and the rmap code is also zone device aware. Support has also been added for folios that might need to be split in the middle of migration (when the src and dst do not agree on MIGRATE_PFN_COMPOUND), that occurs when src side of the migration can migrate large pages, but the destination has not been able to allocate large pages. The code supported and used folio_split() when migrating THP pages, this is used when MIGRATE_VMA_SELECT_COMPOUND is not passed as an argument to migrate_vma_setup().
The test infrastructure lib/test_hmm.c has been enhanced to support THP migration. A new ioctl to emulate failure of large page allocations has been added to test the folio split code path. hmm-tests.c has new test cases for huge page migration and to test the folio split path. A new throughput test has been added as well.
The nouveau dmem code has been enhanced to use the new THP migration capability.
mTHP support:
The patches hard code, HPAGE_PMD_NR in a few places, but the code has been kept generic to support various order sizes. With additional refactoring of the code support of different order sizes should be possible.
The future plan is to post enhancements to support mTHP with a rough design as follows:
1. Add the notion of allowable thp orders to the HMM based test driver
2. For non PMD based THP paths in migrate_device.c, check to see if a suitable order is found and supported by the driver
3. Iterate across orders to check the highest supported order for migration
4. Migrate and finalize
The mTHP patches can be built on top of this series, the key design elements that need to be worked out are infrastructure and driver support for multiple ordered pages and their migration.
HMM support for large folios:
Currently in mm-unstable [4]
References:
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/T/
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://lkml.kernel.org/r/[email protected]
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/[email protected]/
These patches are built on top of mm/mm-stable
Changelog v4 [6] :
- Addressed review comments
- Split patch 2 into a smaller set of patches
- PVMW_THP_DEVICE_PRIVATE flag is no longer present
- damon/page_idle and other page_vma_mapped_walk paths are aware of device-private folios
- No more flush for non-present entries in set_pmd_migration_entry
- Implemented a helper function for migrate_vma_split_folio() which splits large folios if seen during a pte walk
- Removed the controversial change for folio_ref_freeze using folio_expected_ref_count()
- Removed functions invoked from with VM_WARN_ON
- New test cases and fixes from Matthew Brost
- Fixed bugs reported by kernel test robot (Thanks!)
- Several fixes for THP support in nouveau driver
Changelog v3 [5] :
- Addressed review comments
- No more split_device_private_folio() helper
- Device private large folios do not end up on deferred scan lists
- Removed THP size order checks when initializing zone device folio
- Fixed bugs reported by kernel test robot (Thanks!)
Changelog v2 [3] :
- Several review comments from David Hildenbrand were addressed, Mika, Zi, Matthew also provided helpful review comments
- In paths where it makes sense a new helper is_pmd_device_private_entry() is used
- anon_exclusive handling of zone device private pages in split_huge_pmd_locked() has been fixed
- Patches that introduced helpers have been folded into where they are used
- Zone device handling in mm/huge_memory.c has benefited from the code and testing of Matthew Brost, he helped find bugs related to copy_huge_pmd() and partial unmapping of folios.
- Zone device THP PMD support via page_vma_mapped_walk() is restricted to try_to_migrate_one()
- There is a new dedicated helper to split large zone device folios
Changelog v1 [2]:
- Support for handling fault_folio and using trylock in the fault path
- A new test case has been added to measure the throughput improvement
- General refactoring of code to keep up with the changes in mm
- New split folio callback when the entire split is complete/done. The callback is used to know when the head order needs to be reset.
Testing:
- Testing was done with ZONE_DEVICE private pages on an x86 VM
Balbir Singh (14):
mm/zone_device: support large zone device private folios mm/huge_memory: add device-private THP support to PMD operations mm/rmap: extend rmap and migration support device-private entries mm/huge_memory: implement device-private THP splitting mm/migrate_device: handle partially mapped folios during collection mm/migrate_device: implement THP migration of zone device pages mm/memory/fault: add THP fault handling for zone device private pages lib/test_hmm: add zone device private THP test infrastructure mm/memremap: add driver callback support for folio splitting mm/migrate_device: add THP splitting during migration lib/test_hmm: add large page allocation failure testing selftests/mm/hmm-tests: new tests for zone device THP migration selftests/mm/hmm-tests: new throughput tests including THP gpu/drm/nouveau: enable THP support for GPU memory migration
Matthew Brost (1):
selftests/mm/hmm-tests: partial unmap, mremap and anon_write tests
drivers/gpu/drm/nouveau/nouveau_dmem.c | 306 +++++--- drivers/gpu/drm/nouveau/nouveau_svm.c | 6 +- drivers/gpu/drm/nouveau/nouveau_svm.h | 3 +- include/linux/huge_mm.h | 18 +- include/linux/memremap.h | 51 +- include/linux/migrate.h | 2 + include/linux/mm.h | 1 + include/linux/swapops.h | 27 + lib/test_hmm.c | 443 +++++++++--- lib/test_hmm_uapi.h | 3 + mm/damon/ops-common.c | 20 +- mm/huge_memory.c | 288 ++++++-- mm/memory.c | 6 +- mm/memremap.c | 38 +- mm/migrate_device.c | 614 +++++++++++++++-- mm/page_idle.c | 5 +- mm/page_vma_mapped.c | 12 +- mm/pgtable-generic.c | 6 + mm/rmap.c | 25 +- tools/testing/selftests/mm/hmm-tests.c | 919 +++++++++++++++++++++++-- 20 files changed, 2399 insertions(+), 394 deletions(-)
Signature · 1 line
2.50.1
PATCH v4 1/15 mm/zone_device: support large zone device private folios
Add routines to support allocation of large order zone device folios and helper functions for zone device folios, to check if a folio is device private and helpers for setting zone device data.
When large folios are used, the existing page_free() callback in pgmap is called when the folio is freed, this is true for both PAGE_SIZE and higher order pages.
Zone device private large folios do not support deferred split and scan like normal THP folios.
--- include/linux/memremap.h | 10 +++++++++- mm/memremap.c | 38 +++++++++++++++++++++++++------------- mm/rmap.c | 6 +++++- 3 files changed, 39 insertions(+), 15 deletions(-)
include/linux/memremap.h +9 -1
@@ -199,7 +199,7 @@ static inline bool folio_is_fsdax(const struct folio *folio) }#ifdef CONFIG_ZONE_DEVICE -void zone_device_page_init (struct page *page); +void zone_device_folio_init (struct folio *folio,unsigned int order);void *memremap_pages (struct dev_pagemap *pgmap,int nid);void memunmap_pages (struct dev_pagemap *pgmap);void *devm_memremap_pages (struct device *dev,struct dev_pagemap *pgmap);
@@ -209,6 +209,14 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn,bool pgmap_pfn_valid (struct dev_pagemap *pgmap,unsigned long pfn);unsigned long memremap_compat_align (void ); + +static inline void zone_device_page_init (struct page *page) +{ +struct folio *folio =page_folio (page); + +zone_device_folio_init (folio,0 ); +} +#else static inline void *devm_memremap_pages (struct device *dev,struct dev_pagemap *pgmap)
mm/memremap.c +25 -13
@@ -427,20 +427,19 @@ EXPORT_SYMBOL_GPL(get_dev_pagemap);void free_zone_device_folio (struct folio *folio) {struct dev_pagemap *pgmap = folio->pgmap; +unsigned long nr =folio_nr_pages (folio); +int i;if (WARN_ON_ONCE (!pgmap))return ;mem_cgroup_uncharge (folio); -/* -* Note: we don't expect anonymous compound pages yet. Once supported -* and we could PTE-map them similar to THP, we'd have to clear -* PG_anon_exclusive on all tail pages. -*/ if (folio_test_anon (folio)) { -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); -__ClearPageAnonExclusive (folio_page (folio,0 )); +for (i =0 ; i < nr; i++) +__ClearPageAnonExclusive (folio_page (folio, i)); + }else { +VM_WARN_ON_ONCE (folio_test_large (folio)); }/*
@@ -464,11 +463,15 @@ void free_zone_device_folio(struct folio *folio)switch (pgmap->type) {case MEMORY_DEVICE_PRIVATE : +percpu_ref_put_many (&folio->pgmap->ref, nr); + pgmap->ops->page_free (&folio->page); + folio->page.mapping =NULL ; +break ;case MEMORY_DEVICE_COHERENT :if (WARN_ON_ONCE (!pgmap->ops || !pgmap->ops->page_free))break ; - pgmap->ops->page_free (folio_page (folio,0 )); -put_dev_pagemap (pgmap); + pgmap->ops->page_free (&folio->page); +percpu_ref_put (&folio->pgmap->ref);break ;case MEMORY_DEVICE_GENERIC :
@@ -491,14 +494,23 @@ void free_zone_device_folio(struct folio *folio) } } -void zone_device_page_init (struct page *page) +void zone_device_folio_init (struct folio *folio,unsigned int order) { +struct page *page =folio_page (folio,0 ); + +VM_WARN_ON_ONCE (order >MAX_ORDER_NR_PAGES ); +/* * Drivers shouldn't be allocating pages after calling * memunmap_pages(). */ -WARN_ON_ONCE (!percpu_ref_tryget_live (&page_pgmap (page)->ref)); -set_page_count (page,1 ); +WARN_ON_ONCE (!percpu_ref_tryget_many (&page_pgmap (page)->ref,1 << order)); +folio_set_count (folio,1 );lock_page (page); + +if (order >1 ) { +prep_compound_page (page, order); +folio_set_large_rmappable (folio); + } } -EXPORT_SYMBOL_GPL (zone_device_page_init); +EXPORT_SYMBOL_GPL (zone_device_folio_init);
mm/rmap.c +5 -1
@@ -1769,9 +1769,13 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,* the folio is unmapped and at least one page is still mapped. * * Check partially_mapped first to ensure it is a large folio. +* +* Device private folios do not support deferred splitting and +* shrinker based scanning of the folios to free. */ if (partially_mapped &&folio_test_anon (folio) && - !folio_test_partially_mapped (folio)) + !folio_test_partially_mapped (folio) && + !folio_is_device_private (folio))deferred_split_folio (folio, true);__folio_mod_stat (folio, -nr, -nr_pmdmapped);
Signature · 1 line
2.50.1
Re: [v4 01/15] mm/zone_device: support large zone device private folios
Quoted 37 lines · Balbir Singh
Add routines to support allocation of large order zone device folios and helper functions for zone device folios, to check if a folio is device private and helpers for setting zone device data.
When large folios are used, the existing
page_free()callback in pgmap is called when the folio is freed, this is true for bothPAGE_SIZEand higher order pages.Zone device private large folios do not support deferred split and scan like normal THP folios.
Cc Andrew Morton <[email protected]>Cc David Hildenbrand <[email protected]>Cc Zi Yan <[email protected]>Cc Joshua Hahn <[email protected]>Cc Rakie Kim <[email protected]>Cc Byungchul Park <[email protected]>Cc Gregory Price <[email protected]>Cc Ying Huang <[email protected]>Cc Alistair Popple <[email protected]>Cc Oscar Salvador <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Baolin Wang <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc Nico Pache <[email protected]>Cc Ryan Roberts <[email protected]>Cc Dev Jain <[email protected]>Cc Barry Song <[email protected]>Cc Lyude Paul <[email protected]>Cc Danilo Krummrich <[email protected]>Cc David Airlie <[email protected]>Cc Simona Vetter <[email protected]>Cc Ralph Campbell <[email protected]>Cc Mika Penttilä <[email protected]>Cc Matthew Brost <[email protected]>Cc Francois Dugast <[email protected]>
Quoted 2 lines
Signed-off-by Balbir Singh <[email protected]>---
I think you missed my review of this patch in the last revision.
Signature · 3 lines
Cheers David / dhildenb
PATCH v4 2/15 mm/huge_memory: add device-private THP support to PMD operations
In reply to Balbir SinghExtend core huge page management functions to handle device-private THP entries. This enables proper handling of large device-private folios in fundamental MM operations.
The following functions have been updated:
- copy_huge_pmd(): Handle device-private entries during fork/clone
- zap_huge_pmd(): Properly free device-private THP during munmap
- change_huge_pmd(): Support protection changes on device-private THP
- __pte_offset_map(): Add device-private entry awareness
--- include/linux/swapops.h | 27 +++++++++++++++++++ mm/huge_memory.c | 60 ++++++++++++++++++++++++++++++++++++----- mm/pgtable-generic.c | 6 +++++ 3 files changed, 86 insertions(+), 7 deletions(-)
include/linux/swapops.h +27 -0
@@ -594,6 +594,33 @@ static inline int is_pmd_migration_entry(pmd_t pmd) }#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#if defined(CONFIG_ZONE_DEVICE ) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATION ) + +/** +* is_pmd_device_private_entry() - Check if PMD contains a device private swap entry +* @pmd: The PMD to check +* +* Returns true if the PMD contains a swap entry that represents a device private +* page mapping. This is used for zone device private pages that have been +* swapped out but still need special handling during various memory management +* operations. +* +* Return: 1 if PMD contains device private entry, 0 otherwise +*/ +static inline int is_pmd_device_private_entry (pmd_t pmd) +{ +return is_swap_pmd (pmd) &&is_device_private_entry (pmd_to_swp_entry (pmd)); +} + +#else /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ + +static inline int is_pmd_device_private_entry (pmd_t pmd) +{ +return 0 ; +} + +#endif /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ +static inline int non_swap_entry (swp_entry_t entry) {return swp_type (entry) >=MAX_SWAPFILES ;
mm/huge_memory.c +53 -7
102 lines not on this page · raw message
mm/pgtable-generic.c +6 -0
@@ -292,6 +292,12 @@ pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) *pmdvalp = pmdval;if (unlikely (pmd_none (pmdval) ||is_pmd_migration_entry (pmdval))) goto nomap; +if (is_swap_pmd (pmdval)) { +swp_entry_t entry =pmd_to_swp_entry (pmdval); + +if (is_device_private_entry (entry)) + goto nomap; + }if (unlikely (pmd_trans_huge (pmdval))) goto nomap;if (unlikely (pmd_bad (pmdval))) {
Signature · 1 line
2.50.1
PATCH v4 3/15 mm/rmap: extend rmap and migration support device-private entries
In reply to Balbir SinghAdd device-private THP support to reverse mapping infrastructure, enabling proper handling during migration and walk operations.
The key changes are:
- add_migration_pmd()/remove_migration_pmd(): Handle device-private entries during folio migration and splitting
- page_vma_mapped_walk(): Recognize device-private THP entries during VMA traversal operations
This change supports folio splitting and migration operations on device-private entries.
--- mm/damon/ops-common.c | 20 +++++++++++++++++--- mm/huge_memory.c | 16 +++++++++++++++- mm/page_idle.c | 5 +++-- mm/page_vma_mapped.c | 12 ++++++++++-- mm/rmap.c | 19 ++++++++++++++++--- 5 files changed, 61 insertions(+), 11 deletions(-)
mm/damon/ops-common.c +17 -3
@@ -75,12 +75,24 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addrvoid damon_pmdp_mkold (pmd_t *pmd,struct vm_area_struct *vma,unsigned long addr) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE -struct folio *folio =damon_get_folio (pmd_pfn (pmdp_get (pmd))); +pmd_t pmdval =pmdp_get (pmd); +struct folio *folio; +bool young = false; +unsigned long pfn; + +if (likely (pmd_present (pmdval))) + pfn =pmd_pfn (pmdval); +else + pfn =swp_offset_pfn (pmd_to_swp_entry (pmdval)); + folio =damon_get_folio (pfn);if (!folio)return ; -if (pmdp_clear_young_notify (vma, addr, pmd)) +if (likely (pmd_present (pmdval))) + young |=pmdp_clear_young_notify (vma, addr, pmd); + young |=mmu_notifier_clear_young (vma->vm_mm, addr, addr +PAGE_SIZE ); +if (young)folio_set_young (folio);folio_set_idle (folio);
@@ -203,7 +215,9 @@ static bool damon_folio_young_one(struct folio *folio,mmu_notifier_test_young (vma->vm_mm, addr); }else {#ifdef CONFIG_TRANSPARENT_HUGEPAGE - *accessed =pmd_young (pmdp_get (pvmw.pmd)) || +pmd_t pmd =pmdp_get (pvmw.pmd); + + *accessed = (pmd_present (pmd) &&pmd_young (pmd)) || !folio_test_idle (folio) ||mmu_notifier_test_young (vma->vm_mm, addr);#else
mm/huge_memory.c +15 -1
@@ -4649,7 +4649,10 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,return 0 ;flush_cache_range (vma, address, address +HPAGE_PMD_SIZE ); - pmdval =pmdp_invalidate (vma, address, pvmw->pmd); +if (unlikely (!pmd_present (*pvmw->pmd))) + pmdval =pmdp_huge_get_and_clear (vma->vm_mm, address, pvmw->pmd); +else + pmdval =pmdp_invalidate (vma, address, pvmw->pmd);/* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive =folio_test_anon (folio) &&PageAnonExclusive (page);
@@ -4699,6 +4702,17 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) entry =pmd_to_swp_entry (*pvmw->pmd);folio_get (folio); pmde =folio_mk_pmd (folio,READ_ONCE (vma->vm_page_prot)); + +if (folio_is_device_private (folio)) { +if (pmd_write (pmde)) + entry =make_writable_device_private_entry ( +page_to_pfn (new)); +else + entry =make_readable_device_private_entry ( +page_to_pfn (new)); + pmde =swp_entry_to_pmd (entry); + } +if (pmd_swp_soft_dirty (*pvmw->pmd)) pmde =pmd_mksoft_dirty (pmde);if (is_writable_migration_entry (entry))
mm/page_idle.c +3 -2
@@ -71,8 +71,9 @@ static bool page_idle_clear_pte_refs_one(struct folio *folio, referenced |=ptep_test_and_clear_young (vma, addr, pvmw.pte); referenced |=mmu_notifier_clear_young (vma->vm_mm, addr, addr +PAGE_SIZE ); }else if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { -if (pmdp_clear_young_notify (vma, addr, pvmw.pmd)) - referenced = true; +if (likely (pmd_present (pmdp_get (pvmw.pmd)))) + referenced |=pmdp_clear_young_notify (vma, addr, pvmw.pmd); + referenced |=mmu_notifier_clear_young (vma->vm_mm, addr, addr +PAGE_SIZE ); }else {/* unexpected pmd-mapped page? */ WARN_ON_ONCE (1 );
mm/page_vma_mapped.c +10 -2
@@ -250,12 +250,11 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) pvmw->ptl =pmd_lock (mm, pvmw->pmd); pmde = *pvmw->pmd;if (!pmd_present (pmde)) { -swp_entry_t entry; +swp_entry_t entry =pmd_to_swp_entry (pmde);if (!thp_migration_supported () || !(pvmw->flags &PVMW_MIGRATION ))return not_found (pvmw); - entry =pmd_to_swp_entry (pmde);if (!is_migration_entry (entry) || !check_pmd (swp_offset_pfn (entry), pvmw))return not_found (pvmw);
@@ -277,6 +276,15 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)* cannot return prematurely, while zap_huge_pmd() has * cleared *pmd but not decremented compound_mapcount(). */ +swp_entry_t entry; + + entry =pmd_to_swp_entry (pmde); + +if (is_device_private_entry (entry)) { + pvmw->ptl =pmd_lock (mm, pvmw->pmd); +return true; + } + if ((pvmw->flags & PVMW_SYNC) && thp_vma_suitable_order(vma, pvmw->address, PMD_ORDER) &&
mm/rmap.c +16 -3
@@ -1064,8 +1064,10 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) }else {#ifdef CONFIG_TRANSPARENT_HUGEPAGE pmd_t *pmd = pvmw->pmd; -pmd_t entry; +pmd_t entry =pmdp_get (pmd); +if (!pmd_present (entry)) +continue ;if (!pmd_dirty (*pmd) && !pmd_write (*pmd))continue ;
@@ -2330,6 +2332,11 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,while (page_vma_mapped_walk (&pvmw)) {/* PMD-mapped THP migration entry */ if (!pvmw.pte) { +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +unsigned long pfn; +pmd_t pmdval; +#endif +if (flags &TTU_SPLIT_HUGE_PMD ) {split_huge_pmd_locked (vma, pvmw.address, pvmw.pmd, true);
@@ -2338,8 +2345,14 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,break ; }#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION - subpage =folio_page (folio, -pmd_pfn (*pvmw.pmd) -folio_pfn (folio)); + pmdval =pmdp_get (pvmw.pmd); +if (likely (pmd_present (pmdval))) + pfn =pmd_pfn (pmdval); +else + pfn =swp_offset_pfn (pmd_to_swp_entry (pmdval)); + + subpage =folio_page (folio, pfn -folio_pfn (folio)); +VM_BUG_ON_FOLIO (folio_test_hugetlb (folio) || !folio_test_pmd_mappable (folio), folio);
Signature · 1 line
2.50.1
PATCH v4 4/15 mm/huge_memory: implement device-private THP splitting
In reply to Balbir SinghAdd support for splitting device-private THP folios, enabling fallback to smaller page sizes when large page allocation or migration fails.
Key changes:
- split_huge_pmd(): Handle device-private PMD entries during splitting
- Preserve RMAP_EXCLUSIVE semantics for anonymous exclusive folios
- Skip RMP_USE_SHARED_ZEROPAGE for device-private entries as they don't support shared zero page semantics
--- mm/huge_memory.c | 129 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 91 insertions(+), 38 deletions(-)
mm/huge_memory.c +91 -38
@@ -2888,16 +2888,19 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,struct page *page;pgtable_t pgtable;pmd_t old_pmd, _pmd; -bool young, write, soft_dirty, pmd_migration = false, uffd_wp = false; -bool anon_exclusive = false, dirty = false; +bool young, write, soft_dirty, uffd_wp = false; +bool anon_exclusive = false, dirty = false, present = false;unsigned long addr;pte_t *pte;int i; +swp_entry_t swp_entry;VM_BUG_ON (haddr & ~HPAGE_PMD_MASK );VM_BUG_ON_VMA (vma->vm_start > haddr, vma);VM_BUG_ON_VMA (vma->vm_end < haddr +HPAGE_PMD_SIZE , vma); -VM_BUG_ON (!is_pmd_migration_entry (*pmd) && !pmd_trans_huge (*pmd)); + +VM_WARN_ON (!is_pmd_migration_entry (*pmd) && !pmd_trans_huge (*pmd) && + !is_pmd_device_private_entry (*pmd));count_vm_event (THP_SPLIT_PMD );
@@ -2945,18 +2948,43 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,return __split_huge_zero_page_pmd (vma, haddr, pmd); } - pmd_migration =is_pmd_migration_entry (*pmd); -if (unlikely (pmd_migration)) { -swp_entry_t entry; + present =pmd_present (*pmd); +if (unlikely (!present)) { + swp_entry =pmd_to_swp_entry (*pmd); old_pmd = *pmd; - entry =pmd_to_swp_entry (old_pmd); - page =pfn_swap_entry_to_page (entry); - write =is_writable_migration_entry (entry); -if (PageAnon (page)) - anon_exclusive =is_readable_exclusive_migration_entry (entry); - young =is_migration_entry_young (entry); - dirty =is_migration_entry_dirty (entry); + + folio =pfn_swap_entry_folio (swp_entry); +VM_WARN_ON (!is_migration_entry (swp_entry) && + !is_device_private_entry (swp_entry)); + page =pfn_swap_entry_to_page (swp_entry); + +if (is_pmd_migration_entry (old_pmd)) { + write =is_writable_migration_entry (swp_entry); +if (PageAnon (page)) + anon_exclusive = +is_readable_exclusive_migration_entry ( + swp_entry); + young =is_migration_entry_young (swp_entry); + dirty =is_migration_entry_dirty (swp_entry); + }else if (is_pmd_device_private_entry (old_pmd)) { + write =is_writable_device_private_entry (swp_entry); + anon_exclusive =PageAnonExclusive (page); +if (freeze && anon_exclusive && +folio_try_share_anon_rmap_pmd (folio, page)) + freeze = false; +if (!freeze) { +rmap_t rmap_flags =RMAP_NONE ; + +folio_ref_add (folio,HPAGE_PMD_NR -1 ); +if (anon_exclusive) + rmap_flags |=RMAP_EXCLUSIVE ; + +folio_add_anon_rmap_ptes (folio, page,HPAGE_PMD_NR , + vma, haddr, rmap_flags); + } + } + soft_dirty =pmd_swp_soft_dirty (old_pmd); uffd_wp =pmd_swp_uffd_wp (old_pmd); }else {
@@ -3042,30 +3070,49 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Note that NUMA hinting access restrictions are not transferred to * avoid any possibility of altering permissions across VMAs. */ -if (freeze || pmd_migration) { +if (freeze || !present) {for (i =0 , addr = haddr; i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE ) {pte_t entry; -swp_entry_t swp_entry; - -if (write) - swp_entry =make_writable_migration_entry ( -page_to_pfn (page + i)); -else if (anon_exclusive) - swp_entry =make_readable_exclusive_migration_entry ( -page_to_pfn (page + i)); -else - swp_entry =make_readable_migration_entry ( -page_to_pfn (page + i)); -if (young) - swp_entry =make_migration_entry_young (swp_entry); -if (dirty) - swp_entry =make_migration_entry_dirty (swp_entry); - entry =swp_entry_to_pte (swp_entry); -if (soft_dirty) - entry =pte_swp_mksoft_dirty (entry); -if (uffd_wp) - entry =pte_swp_mkuffd_wp (entry); - +if (freeze ||is_migration_entry (swp_entry)) { +if (write) + swp_entry =make_writable_migration_entry ( +page_to_pfn (page + i)); +else if (anon_exclusive) + swp_entry =make_readable_exclusive_migration_entry ( +page_to_pfn (page + i)); +else + swp_entry =make_readable_migration_entry ( +page_to_pfn (page + i)); +if (young) + swp_entry =make_migration_entry_young (swp_entry); +if (dirty) + swp_entry =make_migration_entry_dirty (swp_entry); + entry =swp_entry_to_pte (swp_entry); +if (soft_dirty) + entry =pte_swp_mksoft_dirty (entry); +if (uffd_wp) + entry =pte_swp_mkuffd_wp (entry); + }else { +/* +* anon_exclusive was already propagated to the relevant +* pages corresponding to the pte entries when freeze +* is false. +*/ +if (write) + swp_entry =make_writable_device_private_entry ( +page_to_pfn (page + i)); +else + swp_entry =make_readable_device_private_entry ( +page_to_pfn (page + i)); +/* +* Young and dirty bits are not progated via swp_entry +*/ + entry =swp_entry_to_pte (swp_entry); +if (soft_dirty) + entry =pte_swp_mksoft_dirty (entry); +if (uffd_wp) + entry =pte_swp_mkuffd_wp (entry); + }VM_WARN_ON (!pte_none (ptep_get (pte + i)));set_pte_at (mm, addr, pte + i, entry); }
@@ -3092,7 +3139,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, }pte_unmap (pte); -if (!pmd_migration) +if (!is_pmd_migration_entry (*pmd))folio_remove_rmap_pmd (folio, page, vma);if (freeze)put_page (page);
@@ -3104,8 +3151,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,void split_huge_pmd_locked (struct vm_area_struct *vma,unsigned long address,pmd_t *pmd,bool freeze) { +VM_WARN_ON_ONCE (!IS_ALIGNED (address,HPAGE_PMD_SIZE )); -if (pmd_trans_huge (*pmd) ||is_pmd_migration_entry (*pmd)) +if (pmd_trans_huge (*pmd) ||is_pmd_migration_entry (*pmd) || +is_pmd_device_private_entry (*pmd))__split_huge_pmd_locked (vma, pmd, address, freeze); }
@@ -3284,6 +3333,9 @@ static void lru_add_split_folio(struct folio *folio, struct folio *new_folio,VM_BUG_ON_FOLIO (folio_test_lru (new_folio), folio);lockdep_assert_held (&lruvec->lru_lock); +if (folio_is_device_private (folio)) +return ; +if (list) {/* page reclaim is reclaiming a huge page */ VM_WARN_ON (folio_test_lru (folio));
@@ -3904,8 +3956,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,if (nr_shmem_dropped)shmem_uncharge (mapping->host, nr_shmem_dropped); -if (!ret && is_anon) +if (!ret && is_anon && !folio_is_device_private (folio)) remap_flags =RMP_USE_SHARED_ZEROPAGE ; +remap_page (folio,1 << order, remap_flags);/*
Signature · 1 line
2.50.1
PATCH v4 5/15 mm/migrate_device: handle partially mapped folios during collection
In reply to Balbir SinghExtend migrate_vma_collect_pmd() to handle partially mapped large folios that require splitting before migration can proceed.
During PTE walk in the collection phase, if a large folio is only partially mapped in the migration range, it must be split to ensure the folio is correctly migrated.
--- mm/migrate_device.c | 95 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)
mm/migrate_device.c +95 -0
@@ -54,6 +54,54 @@ static int migrate_vma_collect_hole(unsigned long start,return 0 ; } +/** +* migrate_vma_split_folio - Helper function to split a(n) (m)THP folio +* +* @folio - the folio to split +* @fault_page - struct page associated with the fault if any +* +* Returns 0 on success +*/ +static int migrate_vma_split_folio (struct folio *folio, +struct page *fault_page) +{ +int ret; +struct folio *fault_folio = fault_page ?page_folio (fault_page) :NULL ; +struct folio *new_fault_folio =NULL ; + +if (folio != fault_folio) { +folio_get (folio); +folio_lock (folio); + } + + ret =split_folio (folio); +if (ret) { +if (folio != fault_folio) { +folio_unlock (folio); +folio_put (folio); + } +return ret; + } + + new_fault_folio = fault_page ?page_folio (fault_page) :NULL ; + +/* +* Ensure the lock is held on the correct +* folio after the split +*/ +if (!new_fault_folio) { +folio_unlock (folio); +folio_put (folio); + }else if (folio != new_fault_folio) { +folio_get (new_fault_folio); +folio_lock (new_fault_folio); +folio_unlock (folio); +folio_put (folio); + } + +return 0 ; +} + static int migrate_vma_collect_pmd(pmd_t *pmdp, unsigned long start, unsigned long end,
@@ -136,6 +184,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,* page table entry. Other special swap entries are not * migratable, and we ignore regular swapped page. */ +struct folio *folio; + entry =pte_to_swp_entry (pte);if (!is_device_private_entry (entry)) goto next;
@@ -147,6 +197,29 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, pgmap->owner != migrate->pgmap_owner) goto next; + folio =page_folio (page); +if (folio_test_large (folio)) { +int ret; + +/* +* The reason for finding pmd present with a +* large folio for the pte is partial unmaps. +* Split the folio now for the migration to be +* handled correctly +*/ +pte_unmap_unlock (ptep, ptl); + ret =migrate_vma_split_folio (folio, + migrate->fault_page); + +if (ret) { + ptep =pte_offset_map_lock (mm, pmdp, addr, &ptl); + goto next; + } + + addr = start; + goto again; + } + mpfn =migrate_pfn (page_to_pfn (page)) |MIGRATE_PFN_MIGRATE ;if (is_writable_device_private_entry (entry))
@@ -171,6 +244,28 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, pgmap->owner != migrate->pgmap_owner) goto next; } + folio =page_folio (page); +if (folio_test_large (folio)) { +int ret; + +/* +* The reason for finding pmd present with a +* large folio for the pte is partial unmaps. +* Split the folio now for the migration to be +* handled correctly +*/ +pte_unmap_unlock (ptep, ptl); + ret =migrate_vma_split_folio (folio, + migrate->fault_page); + +if (ret) { + ptep =pte_offset_map_lock (mm, pmdp, addr, &ptl); + goto next; + } + + addr = start; + goto again; + } mpfn =migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; mpfn |=pte_write (pte) ?MIGRATE_PFN_WRITE :0 ; }
Signature · 1 line
2.50.1
Re: [v4 05/15] mm/migrate_device: handle partially mapped folios during collection
Hi,
Quoted 140 lines · Balbir Singh
Extend
migrate_vma_collect_pmd()to handle partially mapped large folios that require splitting before migration can proceed.During PTE walk in the collection phase, if a large folio is only partially mapped in the migration range, it must be split to ensure the folio is correctly migrated.
Cc Andrew Morton <[email protected]>Cc David Hildenbrand <[email protected]>Cc Zi Yan <[email protected]>Cc Joshua Hahn <[email protected]>Cc Rakie Kim <[email protected]>Cc Byungchul Park <[email protected]>Cc Gregory Price <[email protected]>Cc Ying Huang <[email protected]>Cc Alistair Popple <[email protected]>Cc Oscar Salvador <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Baolin Wang <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc Nico Pache <[email protected]>Cc Ryan Roberts <[email protected]>Cc Dev Jain <[email protected]>Cc Barry Song <[email protected]>Cc Lyude Paul <[email protected]>Cc Danilo Krummrich <[email protected]>Cc David Airlie <[email protected]>Cc Simona Vetter <[email protected]>Cc Ralph Campbell <[email protected]>Cc Mika Penttilä <[email protected]>Cc Matthew Brost <[email protected]>Cc Francois Dugast <[email protected]>Signed-off-by Balbir Singh <[email protected]>--- mm/migrate_device.c | 95 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)mm/migrate_device.c +73
-0@@ -54,6 +54,54 @@ static int migrate_vma_collect_hole(unsigned long start,return 0 ; } +/** +* migrate_vma_split_folio - Helper function to split a(n) (m)THP folio +* +* @folio - the folio to split +* @fault_page - struct page associated with the fault if any +* +* Returns 0 on success +*/ +static int migrate_vma_split_folio (struct folio *folio, +struct page *fault_page) +{ +int ret; +struct folio *fault_folio = fault_page ?page_folio (fault_page) :NULL ; +struct folio *new_fault_folio =NULL ; + +if (folio != fault_folio) { +folio_get (folio); +folio_lock (folio); + } + + ret =split_folio (folio); +if (ret) { +if (folio != fault_folio) { +folio_unlock (folio); +folio_put (folio); + } +return ret; + } + + new_fault_folio = fault_page ?page_folio (fault_page) :NULL ; + +/* +* Ensure the lock is held on the correct +* folio after the split +*/ +if (!new_fault_folio) { +folio_unlock (folio); +folio_put (folio); + }else if (folio != new_fault_folio) { +folio_get (new_fault_folio); +folio_lock (new_fault_folio); +folio_unlock (folio); +folio_put (folio); + } + +return 0 ; +} + static int migrate_vma_collect_pmd(pmd_t *pmdp, unsigned long start, unsigned long end,@@ -136,6 +184,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,* page table entry. Other special swap entries are not * migratable, and we ignore regular swapped page. */ +struct folio *folio; + entry =pte_to_swp_entry (pte);if (!is_device_private_entry (entry)) goto next;@@ -147,6 +197,29 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, pgmap->owner != migrate->pgmap_owner) goto next; + folio =page_folio (page); +if (folio_test_large (folio)) { +int ret; + +/* +* The reason for finding pmd present with a +* large folio for the pte is partial unmaps. +* Split the folio now for the migration to be +* handled correctly +*/ +pte_unmap_unlock (ptep, ptl); + ret =migrate_vma_split_folio (folio, + migrate->fault_page); + +if (ret) { + ptep =pte_offset_map_lock (mm, pmdp, addr, &ptl); + goto next; + } + + addr = start; + goto again; + } + mpfn =migrate_pfn (page_to_pfn (page)) |MIGRATE_PFN_MIGRATE ;if (is_writable_device_private_entry (entry))@@ -171,6 +244,28 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, pgmap->owner != migrate->pgmap_owner) goto next; }
mm/migrate_device.c static int migrate_vma_collect_pmd(pmd_t *pmdp, · line 256
Quoted 10 lines
mm/migrate_device.c +10
-0+ folio =page_folio (page); +if (folio_test_large (folio)) { +int ret; + +/* +* The reason for finding pmd present with a +* large folio for the pte is partial unmaps. +* Split the folio now for the migration to be +* handled correctly +*/
There are other reasons like vma splits for various reasons.
Quoted 15 lines
mm/migrate_device.c +12
-0+pte_unmap_unlock (ptep, ptl); + ret =migrate_vma_split_folio (folio, + migrate->fault_page); + +if (ret) { + ptep =pte_offset_map_lock (mm, pmdp, addr, &ptl); + goto next; + } + + addr = start; + goto again; + } mpfn =migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; mpfn |=pte_write (pte) ?MIGRATE_PFN_WRITE :0 ; }
Re: [v4 05/15] mm/migrate_device: handle partially mapped folios during collection
Quoted 11 lines · Mika Penttilä
Hi,
Quoted 7 lines · Balbir Singh
Extend
migrate_vma_collect_pmd()to handle partially mapped large folios that require splitting before migration can proceed.During PTE walk in the collection phase, if a large folio is only partially mapped in the migration range, it must be split to ensure the folio is correctly migrated.
<snip>
Quoted 10 lines
mm/migrate_device.c static int migrate_vma_collect_pmd(pmd_t *pmdp, · line 209
Quoted 7 lines
mm/migrate_device.c +7
-0+ +/* +* The reason for finding pmd present with a +* large folio for the pte is partial unmaps. +* Split the folio now for the migration to be +* handled correctly +*/ There are other reasons like vma splits for various reasons.
Yes, David had pointed that out as well, I meant to cleanup the comment change "The" to "One", I missed addressing it in the refactor, but easy to do
Thanks,
Balbir Singh
Re: [v4 05/15] mm/migrate_device: handle partially mapped folios during collection
Quoted 23 lines · Balbir Singh
Quoted 11 lines · Mika Penttilä
Hi,
Quoted 7 lines · Balbir Singh
Extend
migrate_vma_collect_pmd()to handle partially mapped large folios that require splitting before migration can proceed.During PTE walk in the collection phase, if a large folio is only partially mapped in the migration range, it must be split to ensure the folio is correctly migrated.
<snip>
Quoted 9 lines
mm/migrate_device.c static int migrate_vma_collect_pmd(pmd_t *pmdp, · line 209
Quoted 7 lines
mm/migrate_device.c +7
-0+ +/* +* The reason for finding pmd present with a +* large folio for the pte is partial unmaps. +* Split the folio now for the migration to be +* handled correctly +*/ There are other reasons like vma splits for various reasons.
Quoted 2 lines
Yes, David had pointed that out as well, I meant to cleanup the comment change "The" to "One", I missed addressing it in the refactor, but easy to do
And of course now you split all mTHPs as well, which is different what we do today (ignoring). Splitting might be the right thing to do, but maybe worth mentioning.
Quoted 3 lines
Thanks,
Balbir Singh
--Mika
kernel test robot
build botRe: [v4 05/15] mm/migrate_device: handle partially mapped folios during collection
In reply to Balbir Singhkernel test robot noticed the following build warnings:
Hi Balbir,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-nonmm-unstable]
[also build test WARNING on sj/damon/next linus/master v6.17-rc4 next-20250904] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Balbir-Singh/mm-zone_device-support-large-zone-device-private-folios/20250903-092241 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-nonmm-unstable patch link: https://lore.kernel.org/r/20250903011900.3657435-6-balbirs%40nvidia.com patch subject: [v4 05/15] mm/migrate_device: handle partially mapped folios during collection config: arm64-randconfig-001-20250904 (https://download.01.org/0day-ci/archive/20250904/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 8.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250904/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> Warning: mm/migrate_device.c:66 function parameter 'folio' not described in 'migrate_vma_split_folio' >> Warning: mm/migrate_device.c:66 function parameter 'fault_page' not described in 'migrate_vma_split_folio'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v4 6/15 mm/migrate_device: implement THP migration of zone device pages
In reply to Balbir SinghMIGRATE_VMA_SELECT_COMPOUND will be used to select THP pages during migrate_vma_setup() and MIGRATE_PFN_COMPOUND will make migrating device pages as compound pages during device pfn migration.
migrate_device code paths go through the collect, setup and finalize phases of migration.
The entries in src and dst arrays passed to these functions still remain at a PAGE_SIZE granularity. When a compound page is passed, the first entry has the PFN along with MIGRATE_PFN_COMPOUND and other flags set (MIGRATE_PFN_MIGRATE, MIGRATE_PFN_VALID), the remaining entries (HPAGE_PMD_NR - 1) are filled with 0's. This representation allows for the compound page to be split into smaller page sizes.
migrate_vma_collect_hole(), migrate_vma_collect_pmd() are now THP page aware. Two new helper functions migrate_vma_collect_huge_pmd() and migrate_vma_insert_huge_pmd_page() have been added.
migrate_vma_collect_huge_pmd() can collect THP pages, but if for some reason this fails, there is fallback support to split the folio and migrate it.
migrate_vma_insert_huge_pmd_page() closely follows the logic of migrate_vma_insert_page()
Support for splitting pages as needed for migration will follow in later patches in this series.
--- include/linux/migrate.h | 2 + mm/migrate_device.c | 456 ++++++++++++++++++++++++++++++++++------ 2 files changed, 395 insertions(+), 63 deletions(-)
include/linux/migrate.h +2 -0
@@ -134,6 +134,7 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)#define MIGRATE_PFN_VALID (1UL << 0)#define MIGRATE_PFN_MIGRATE (1UL << 1)#define MIGRATE_PFN_WRITE (1UL << 3) +#define MIGRATE_PFN_COMPOUND (1UL << 4)#define MIGRATE_PFN_SHIFT 6static inline struct page *migrate_pfn_to_page (unsigned long mpfn)
@@ -152,6 +153,7 @@ enum migrate_vma_direction {MIGRATE_VMA_SELECT_SYSTEM =1 <<0 ,MIGRATE_VMA_SELECT_DEVICE_PRIVATE =1 <<1 ,MIGRATE_VMA_SELECT_DEVICE_COHERENT =1 <<2 , + MIGRATE_VMA_SELECT_COMPOUND = 1 << 3, }; struct migrate_vma {
mm/migrate_device.c +393 -63
619 lines not on this page · raw message
Signature · 1 line
2.50.1
Re: [v4 06/15] mm/migrate_device: implement THP migration of zone device pages
On Wed, Sep 03, 2025 at 11:18:51AM +1000, Balbir Singh wrote:
[...]
mm/migrate_device.c static void __migrate_device_pages(unsigned long *src_pfns, · line 1113
Quoted 9 lines
mm/migrate_device.c +4
-1@@ -836,15 +1107,47 @@ static void __migrate_device_pages(unsigned long *src_pfns, migrate->pgmap_owner);mmu_notifier_invalidate_range_start (&range); } - migrate_vma_insert_page(migrate, addr, newpage, + +if ((src_pfns[i] &MIGRATE_PFN_COMPOUND ) && + (!(dst_pfns[i] &MIGRATE_PFN_COMPOUND ))) { + nr =HPAGE_PMD_NR ;
This one trigger BUILD_BUG() if !PGTABLE_HAS_HUGE_LEAVES.
Quoted 9 lines
mm/migrate_device.c +7
-1+ src_pfns[i] &= ~MIGRATE_PFN_COMPOUND ; + src_pfns[i] &= ~MIGRATE_PFN_MIGRATE ; + goto next; + } + +migrate_vma_insert_page (migrate, addr, &dst_pfns[i], &src_pfns[i]); -continue ; + goto next;
Signature · 2 lines
Wei Yang Help you, Help me
Re: [v4 06/15] mm/migrate_device: implement THP migration of zone device pages
In reply to Balbir SinghHi,
Quoted 104 lines · Balbir Singh
MIGRATE_VMA_SELECT_COMPOUNDwill be used to select THP pages duringmigrate_vma_setup()andMIGRATE_PFN_COMPOUNDwill make migrating device pages as compound pages during device pfn migration.
migrate_devicecode paths go through the collect, setup and finalize phases of migration.The entries in src and dst arrays passed to these functions still remain at a
PAGE_SIZEgranularity. When a compound page is passed, the first entry has the PFN along withMIGRATE_PFN_COMPOUNDand other flags set (MIGRATE_PFN_MIGRATE,MIGRATE_PFN_VALID), the remaining entries (HPAGE_PMD_NR- 1) are filled with 0's. This representation allows for the compound page to be split into smaller page sizes.
migrate_vma_collect_hole(),migrate_vma_collect_pmd()are now THP page aware. Two new helper functionsmigrate_vma_collect_huge_pmd()andmigrate_vma_insert_huge_pmd_page()have been added.
migrate_vma_collect_huge_pmd()can collect THP pages, but if for some reason this fails, there is fallback support to split the folio and migrate it.
migrate_vma_insert_huge_pmd_page()closely follows the logic ofmigrate_vma_insert_page()Support for splitting pages as needed for migration will follow in later patches in this series.
Cc Andrew Morton <[email protected]>Cc David Hildenbrand <[email protected]>Cc Zi Yan <[email protected]>Cc Joshua Hahn <[email protected]>Cc Rakie Kim <[email protected]>Cc Byungchul Park <[email protected]>Cc Gregory Price <[email protected]>Cc Ying Huang <[email protected]>Cc Alistair Popple <[email protected]>Cc Oscar Salvador <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Baolin Wang <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc Nico Pache <[email protected]>Cc Ryan Roberts <[email protected]>Cc Dev Jain <[email protected]>Cc Barry Song <[email protected]>Cc Lyude Paul <[email protected]>Cc Danilo Krummrich <[email protected]>Cc David Airlie <[email protected]>Cc Simona Vetter <[email protected]>Cc Ralph Campbell <[email protected]>Cc Mika Penttilä <[email protected]>Cc Matthew Brost <[email protected]>Cc Francois Dugast <[email protected]>Signed-off-by Balbir Singh <[email protected]>--- include/linux/migrate.h | 2 + mm/migrate_device.c | 456 ++++++++++++++++++++++++++++++++++------ 2 files changed, 395 insertions(+), 63 deletions(-)include/linux/migrate.h +2
-0@@ -134,6 +134,7 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)#define MIGRATE_PFN_VALID (1UL << 0)#define MIGRATE_PFN_MIGRATE (1UL << 1)#define MIGRATE_PFN_WRITE (1UL << 3) +#define MIGRATE_PFN_COMPOUND (1UL << 4)#define MIGRATE_PFN_SHIFT 6static inline struct page *migrate_pfn_to_page (unsigned long mpfn)@@ -152,6 +153,7 @@ enum migrate_vma_direction {MIGRATE_VMA_SELECT_SYSTEM =1 <<0 ,MIGRATE_VMA_SELECT_DEVICE_PRIVATE =1 <<1 ,MIGRATE_VMA_SELECT_DEVICE_COHERENT =1 <<2 , + MIGRATE_VMA_SELECT_COMPOUND = 1 << 3, }; struct migrate_vma {mm/migrate_device.c +8
-0@@ -14,6 +14,7 @@#include <linux/pagewalk.h> #include <linux/rmap.h> #include <linux/swapops.h> +#include <asm/pgalloc.h> #include <asm/tlbflush.h> #include "internal.h" @@ -44,6 +45,23 @@ static int migrate_vma_collect_hole(unsigned long start,if (!vma_is_anonymous (walk->vma))return migrate_vma_collect_skip (start, end, walk); +if (thp_migration_supported () && + (migrate->flags &MIGRATE_VMA_SELECT_COMPOUND ) && + (IS_ALIGNED (start,HPAGE_PMD_SIZE ) && +IS_ALIGNED (end,HPAGE_PMD_SIZE ))) { + migrate->src[migrate->npages] =MIGRATE_PFN_MIGRATE | +MIGRATE_PFN_COMPOUND ; + migrate->dst[migrate->npages] =0 ;
mm/migrate_device.c static int migrate_vma_collect_hole(unsigned long start, · line 64
Quoted 10 lines
mm/migrate_device.c +10
-0+ migrate->npages++; + migrate->cpages++; + +/* +* Collect the remaining entries as holes, in case we +* need to split later +*/ +return migrate_vma_collect_skip (start +PAGE_SIZE , end, walk); + } +
seems you have to split_huge_pmd() for the huge zero page here in case of !thp_migration_supported() afaics
mm/migrate_device.c static int migrate_vma_split_folio(struct folio *folio, · line 121
Quoted 6 lines
mm/migrate_device.c +0
-0for (addr = start; addr < end; addr +=PAGE_SIZE ) { migrate->src[migrate->npages] =MIGRATE_PFN_MIGRATE ; migrate->dst[migrate->npages] =0 ;@@ -102,57 +120,150 @@ static int migrate_vma_split_folio(struct folio *folio,return 0 ; }
--Mika
PATCH v4 7/15 mm/memory/fault: add THP fault handling for zone device private pages
In reply to Balbir SinghImplement CPU fault handling for zone device THP entries through do_huge_pmd_device_private(), enabling transparent migration of device-private large pages back to system memory on CPU access.
When the CPU accesses a zone device THP entry, the fault handler calls the device driver's migrate_to_ram() callback to migrate the entire large page back to system memory.
--- include/linux/huge_mm.h | 7 +++++++ mm/huge_memory.c | 36 ++++++++++++++++++++++++++++++++++++ mm/memory.c | 6 ++++-- 3 files changed, 47 insertions(+), 2 deletions(-)
include/linux/huge_mm.h +7 -0
@@ -474,6 +474,8 @@ static inline bool folio_test_pmd_mappable(struct folio *folio)vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf); +vm_fault_t do_huge_pmd_device_private (struct vm_fault *vmf); +extern struct folio *huge_zero_folio;extern unsigned long huge_zero_pfn;
@@ -632,6 +634,11 @@ static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)return 0 ; } +static inline vm_fault_t do_huge_pmd_device_private (struct vm_fault *vmf) +{ +return 0 ; +} +static inline bool is_huge_zero_folio (const struct folio *folio) {return false;
mm/huge_memory.c +36 -0
@@ -1267,6 +1267,42 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf) } +vm_fault_t do_huge_pmd_device_private (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +vm_fault_t ret =0 ; +spinlock_t *ptl; +swp_entry_t swp_entry; +struct page *page; + +if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { +vma_end_read (vma); +return VM_FAULT_RETRY ; + } + + ptl =pmd_lock (vma->vm_mm, vmf->pmd); +if (unlikely (!pmd_same (*vmf->pmd, vmf->orig_pmd))) { +spin_unlock (ptl); +return 0 ; + } + + swp_entry =pmd_to_swp_entry (vmf->orig_pmd); + page =pfn_swap_entry_to_page (swp_entry); + vmf->page = page; + vmf->pte =NULL ; +if (trylock_page (vmf->page)) { +get_page (page); +spin_unlock (ptl); + ret =page_pgmap (page)->ops->migrate_to_ram (vmf); +unlock_page (vmf->page); +put_page (page); + }else { +spin_unlock (ptl); + } + +return ret; +} +/* * always: directly stall for all thp allocations * defer: wake kswapd and fail if not immediately available
mm/memory.c +4 -2
@@ -6170,8 +6170,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, vmf.orig_pmd =pmdp_get_lockless (vmf.pmd);if (unlikely (is_swap_pmd (vmf.orig_pmd))) { -VM_BUG_ON (thp_migration_supported () && - !is_pmd_migration_entry (vmf.orig_pmd)); +if (is_device_private_entry ( +pmd_to_swp_entry (vmf.orig_pmd))) +return do_huge_pmd_device_private (&vmf); +if (is_pmd_migration_entry (vmf.orig_pmd))pmd_migration_entry_wait (mm, vmf.pmd);return 0 ;
Signature · 1 line
2.50.1
PATCH v4 8/15 lib/test_hmm: add zone device private THP test infrastructure
In reply to Balbir SinghEnhance the hmm test driver (lib/test_hmm) with support for THP pages.
A new pool of free_folios() has now been added to the dmirror device, which can be allocated when a request for a THP zone device private page is made.
Add compound page awareness to the allocation function during normal migration and fault based migration. These routines also copy folio_nr_pages() when moving data between system memory and device memory.
args.src and args.dst used to hold migration entries are now dynamically allocated (as they need to hold HPAGE_PMD_NR entries or more).
Split and migrate support will be added in future patches in this series.
--- include/linux/memremap.h | 12 ++ lib/test_hmm.c | 368 +++++++++++++++++++++++++++++++-------- 2 files changed, 304 insertions(+), 76 deletions(-)
include/linux/memremap.h +12 -0
@@ -169,6 +169,18 @@ static inline bool folio_is_device_private(const struct folio *folio)return is_device_private_page (&folio->page); } +static inline void *folio_zone_device_data (const struct folio *folio) +{ +VM_WARN_ON_FOLIO (!folio_is_device_private (folio), folio); +return folio->page.zone_device_data; +} + +static inline void folio_set_zone_device_data (struct folio *folio,void *data) +{ +VM_WARN_ON_FOLIO (!folio_is_device_private (folio), folio); + folio->page.zone_device_data = data; +} +static inline bool is_pci_p2pdma_page (const struct page *page) { return IS_ENABLED(CONFIG_PCI_P2PDMA) &&
lib/test_hmm.c +292 -76
650 lines not on this page · raw message
Signature · 1 line
2.50.1
PATCH v4 9/15 mm/memremap: add driver callback support for folio splitting
In reply to Balbir SinghWhen a zone device page is split (via huge pmd folio split). The driver callback for folio_split is invoked to let the device driver know that the folio size has been split into a smaller order.
Provide a default implementation for drivers that do not provide this callback that copies the pgmap and mapping fields for the split folios.
Update the HMM test driver to handle the split.
--- include/linux/memremap.h | 29 +++++++++++++++++++++++++++++ include/linux/mm.h | 1 + lib/test_hmm.c | 35 +++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 +- 4 files changed, 66 insertions(+), 1 deletion(-)
include/linux/memremap.h +29 -0
@@ -100,6 +100,13 @@ struct dev_pagemap_ops {*/ int (*memory_failure)(struct dev_pagemap *pgmap,unsigned long pfn,unsigned long nr_pages,int mf_flags); + +/* +* Used for private (un-addressable) device memory only. +* This callback is used when a folio is split into +* a smaller folio +*/ +void (*folio_split)(struct folio *head,struct folio *tail); };#define PGMAP_ALTMAP_VALID (1 << 0)
@@ -229,6 +236,23 @@ static inline void zone_device_page_init(struct page *page) zone_device_folio_init(folio, 0); } +static inline void zone_device_private_split_cb (struct folio *original_folio, +struct folio *new_folio) +{ +if (folio_is_device_private (original_folio)) { +if (!original_folio->pgmap->ops->folio_split) { +if (new_folio) { + new_folio->pgmap = original_folio->pgmap; + new_folio->page.mapping = + original_folio->page.mapping; + } + }else { + original_folio->pgmap->ops->folio_split (original_folio, + new_folio); + } + } +} +#else static inline void *devm_memremap_pages (struct device *dev,struct dev_pagemap *pgmap)
@@ -263,6 +287,11 @@ static inline unsigned long memremap_compat_align(void) {return PAGE_SIZE ; } + +static inline void zone_device_private_split_cb (struct folio *original_folio, +struct folio *new_folio) +{ +}#endif /* CONFIG_ZONE_DEVICE */ static inline void put_dev_pagemap (struct dev_pagemap *pgmap)
include/linux/mm.h +1 -0
@@ -1185,6 +1185,7 @@ static inline struct folio *virt_to_folio(const void *x)void __folio_put (struct folio *folio);void split_page (struct page *page,unsigned int order); +void prep_compound_page (struct page *page,unsigned int order);void folio_copy (struct folio *dst,struct folio *src);int folio_mc_copy (struct folio *dst,struct folio *src);
lib/test_hmm.c +35 -0
@@ -1653,9 +1653,44 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf)return ret; } +static void dmirror_devmem_folio_split (struct folio *head,struct folio *tail) +{ +struct page *rpage =BACKING_PAGE (folio_page (head,0 )); +struct page *rpage_tail; +struct folio *rfolio; +unsigned long offset =0 ; + +if (!rpage) { + tail->page.zone_device_data =NULL ; +return ; + } + + rfolio =page_folio (rpage); + +if (tail ==NULL ) { +folio_reset_order (rfolio); + rfolio->mapping =NULL ; +folio_set_count (rfolio,1 ); +return ; + } + + offset =folio_pfn (tail) -folio_pfn (head); + + rpage_tail =folio_page (rfolio, offset); + tail->page.zone_device_data = rpage_tail; + rpage_tail->zone_device_data = rpage->zone_device_data; +clear_compound_head (rpage_tail); + rpage_tail->mapping =NULL ; + +folio_page (tail,0 )->mapping =folio_page (head,0 )->mapping; + tail->pgmap = head->pgmap; +folio_set_count (page_folio (rpage_tail),1 ); +} +static const struct dev_pagemap_ops dmirror_devmem_ops = { .page_free = dmirror_devmem_free, .migrate_to_ram = dmirror_devmem_fault, + .folio_split = dmirror_devmem_folio_split, };static int dmirror_device_init (struct dmirror_device *mdevice,int id)
mm/huge_memory.c +1 -1
@@ -3915,7 +3915,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order, ret =__split_unmapped_folio (folio, new_order, split_at, &xas, mapping, uniform_split); -/* * Unfreeze after-split folios and put them back to the right * list. @folio should be kept frozon until page cache
@@ -3966,6 +3965,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,__filemap_remove_folio (new_folio,NULL );folio_put_refs (new_folio, nr_pages); } +/* * Unfreeze @folio only after all page cache entries, which * used to point to it, have been updated with new folios.
Signature · 1 line
2.50.1
PATCH v4 10/15 mm/migrate_device: add THP splitting during migration
In reply to Balbir SinghImplement migrate_vma_split_pages() to handle THP splitting during the migration process when destination cannot allocate compound pages.
This addresses the common scenario where migrate_vma_setup() succeeds with MIGRATE_PFN_COMPOUND pages, but the destination device cannot allocate large pages during the migration phase.
Key changes:
- migrate_vma_split_pages(): Split already-isolated pages during migration
- Enhanced folio_split() and __split_unmapped_folio() with isolated parameter to avoid redundant unmap/remap operations
This provides a fallback mechansim to ensure migration succeeds even when large page allocation fails at the destination.
--- include/linux/huge_mm.h | 11 +++++-- lib/test_hmm.c | 9 ++++++ mm/huge_memory.c | 45 ++++++++++++++------------ mm/migrate_device.c | 71 ++++++++++++++++++++++++++++++++++------- 4 files changed, 101 insertions(+), 35 deletions(-)
include/linux/huge_mm.h +9 -2
@@ -343,8 +343,8 @@ unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long add vm_flags_t vm_flags);bool can_split_folio (struct folio *folio,int caller_pins,int *pextra_pins); -int split_huge_page_to_list_to_order (struct page *page,struct list_head *list, -unsigned int new_order); +int __split_huge_page_to_list_to_order (struct page *page,struct list_head *list, +unsigned int new_order,bool unmapped);int min_order_for_split (struct folio *folio);int split_folio_to_list (struct folio *folio,struct list_head *list); bool uniform_split_supported(struct folio *folio, unsigned int new_order,
@@ -353,6 +353,13 @@ bool non_uniform_split_supported(struct folio *folio, unsigned int new_order, bool warns);int folio_split (struct folio *folio,unsigned int new_order,struct page *page,struct list_head *list); + +static inline int split_huge_page_to_list_to_order (struct page *page,struct list_head *list, +unsigned int new_order) +{ +return __split_huge_page_to_list_to_order (page, list, new_order, false); +} +/* * try_folio_split - try to split a @folio at @page using non uniform split. * @folio: folio to be split
lib/test_hmm.c +9 -0
@@ -1611,6 +1611,15 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) order =folio_order (page_folio (vmf->page)); nr =1 << order; +/* +* When folios are partially mapped, we can't rely on the folio +* order of vmf->page as the folio might not be fully split yet +*/ +if (vmf->pte) { + order =0 ; + nr =1 ; + } +/* * Consider a per-cpu cache of src and dst pfns, but with * large number of cpus that might not scale well.
mm/huge_memory.c +24 -21
113 lines not on this page · raw message
mm/migrate_device.c +59 -12
110 lines not on this page · raw message
Signature · 1 line
2.50.1
PATCH v4 11/15 lib/test_hmm: add large page allocation failure testing
In reply to Balbir SinghAdd HMM_DMIRROR_FLAG_FAIL_ALLOC flag to simulate large page allocation failures, enabling testing of split migration code paths.
This test flag allows validation of the fallback behavior when destination device cannot allocate compound pages. This is useful for testing the split migration functionality.
--- lib/test_hmm.c | 61 ++++++++++++++++++++++++++++++--------------- lib/test_hmm_uapi.h | 3 +++ 2 files changed, 44 insertions(+), 20 deletions(-)
lib/test_hmm.c +41 -20
@@ -92,6 +92,7 @@ struct dmirror {struct xarray pt;struct mmu_interval_notifier notifier;struct mutex mutex; +__u64 flags; };/*
@@ -699,7 +700,12 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, page_to_pfn(spage))) goto next; - dpage =dmirror_devmem_alloc_page (dmirror, is_large); +if (dmirror->flags &HMM_DMIRROR_FLAG_FAIL_ALLOC ) { + dmirror->flags &= ~HMM_DMIRROR_FLAG_FAIL_ALLOC ; + dpage =NULL ; + }else + dpage =dmirror_devmem_alloc_page (dmirror, is_large); +if (!dpage) {struct folio *folio;unsigned long i;
@@ -959,44 +965,55 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, spage =BACKING_PAGE (spage); order =folio_order (page_folio (spage)); -if (order) + *dst =MIGRATE_PFN_COMPOUND ; +if (*src &MIGRATE_PFN_WRITE ) + *dst |=MIGRATE_PFN_WRITE ; + +if (dmirror->flags &HMM_DMIRROR_FLAG_FAIL_ALLOC ) { + dmirror->flags &= ~HMM_DMIRROR_FLAG_FAIL_ALLOC ; + *dst &= ~MIGRATE_PFN_COMPOUND ; + dpage =NULL ; + }else if (order) { dpage =folio_page (vma_alloc_folio (GFP_HIGHUSER_MOVABLE , order, args->vma, addr),0 ); -else - dpage =alloc_page_vma (GFP_HIGHUSER_MOVABLE , args->vma, addr); - -/* Try with smaller pages if large allocation fails */ -if (!dpage && order) { + }else { dpage =alloc_page_vma (GFP_HIGHUSER_MOVABLE , args->vma, addr); -if (!dpage) -return VM_FAULT_OOM ; - order =0 ; } +if (!dpage && !order) +return VM_FAULT_OOM ; +pr_debug ("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n" ,page_to_pfn (spage),page_to_pfn (dpage)); -lock_page (dpage); -xa_erase (&dmirror->pt, addr >>PAGE_SHIFT ); -copy_highpage (dpage, spage); - *dst =migrate_pfn (page_to_pfn (dpage)); -if (*src &MIGRATE_PFN_WRITE ) - *dst |=MIGRATE_PFN_WRITE ; -if (order) - *dst |=MIGRATE_PFN_COMPOUND ; + +if (dpage) { +lock_page (dpage); + *dst |=migrate_pfn (page_to_pfn (dpage)); + }for (i =0 ; i < (1 << order); i++) {struct page *src_page;struct page *dst_page; +/* Try with smaller pages if large allocation fails */ +if (!dpage && order) { + dpage =alloc_page_vma (GFP_HIGHUSER_MOVABLE , args->vma, addr); +lock_page (dpage); + dst[i] =migrate_pfn (page_to_pfn (dpage)); + dst_page =pfn_to_page (page_to_pfn (dpage)); + dpage =NULL ;/* For the next iteration */ + }else { + dst_page =pfn_to_page (page_to_pfn (dpage) + i); + } + src_page =pfn_to_page (page_to_pfn (spage) + i); - dst_page =pfn_to_page (page_to_pfn (dpage) + i);xa_erase (&dmirror->pt, addr >>PAGE_SHIFT ); + addr +=PAGE_SIZE ;copy_highpage (dst_page, src_page); } next: - addr +=PAGE_SIZE << order; src +=1 << order; dst +=1 << order; }
@@ -1514,6 +1531,10 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,dmirror_device_remove_chunks (dmirror->mdevice); ret =0 ;break ; +case HMM_DMIRROR_FLAGS : + dmirror->flags = cmd.npages; + ret =0 ; +break ;default :return -EINVAL ;
lib/test_hmm_uapi.h +3 -0
@@ -37,6 +37,9 @@ struct hmm_dmirror_cmd {#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd)#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd)#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) + +#define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0)/* * Values returned in hmm_dmirror_cmd.ptr for HMM_DMIRROR_SNAPSHOT.
Signature · 1 line
2.50.1
PATCH v4 12/15 selftests/mm/hmm-tests: new tests for zone device THP migration
In reply to Balbir SinghAdd new tests for migrating anon THP pages, including anon_huge, anon_huge_zero and error cases involving forced splitting of pages during migration.
--- tools/testing/selftests/mm/hmm-tests.c | 410 +++++++++++++++++++++++++ 1 file changed, 410 insertions(+)
tools/testing/selftests/mm/hmm-tests.c +410 -0
@@ -2056,4 +2056,414 @@ TEST_F(hmm, hmm_cow_in_device)hmm_buffer_free (buffer); } + +/* +* Migrate private anonymous huge empty page. +*/ +TEST_F(hmm, migrate_anon_huge_empty) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size; +unsigned long i; +void *old_ptr; +void *map; +int *ptr; +int ret; + + size =TWOMEG ; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =2 * size; + buffer->mirror =malloc (size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF , size); + + buffer->ptr =mmap (NULL ,2 * size, +PROT_READ , +MAP_PRIVATE |MAP_ANONYMOUS , + buffer->fd,0 ); +ASSERT_NE (buffer->ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )buffer->ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + old_ptr = buffer->ptr; + buffer->ptr = map; + +/* Migrate memory to device. */ + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i],0 ); + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); +} + +/* +* Migrate private anonymous huge zero page. +*/ +TEST_F(hmm, migrate_anon_huge_zero) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size; +unsigned long i; +void *old_ptr; +void *map; +int *ptr; +int ret; +int val; + + size =TWOMEG ; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =2 * size; + buffer->mirror =malloc (size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF , size); + + buffer->ptr =mmap (NULL ,2 * size, +PROT_READ , +MAP_PRIVATE |MAP_ANONYMOUS , + buffer->fd,0 ); +ASSERT_NE (buffer->ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )buffer->ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + old_ptr = buffer->ptr; + buffer->ptr = map; + +/* Initialize a read-only zero huge page. */ + val = *(int *)buffer->ptr; +ASSERT_EQ (val,0 ); + +/* Migrate memory to device. */ + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i],0 ); + +/* Fault pages back to system memory and check them. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) { +ASSERT_EQ (ptr[i],0 ); +/* If it asserts once, it probably will 500,000 times */ +if (ptr[i] !=0 ) +break ; + } + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); +} + +/* +* Migrate private anonymous huge page and free. +*/ +TEST_F(hmm, migrate_anon_huge_free) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size; +unsigned long i; +void *old_ptr; +void *map; +int *ptr; +int ret; + + size =TWOMEG ; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =2 * size; + buffer->mirror =malloc (size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF , size); + + buffer->ptr =mmap (NULL ,2 * size, +PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS , + buffer->fd,0 ); +ASSERT_NE (buffer->ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )buffer->ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + old_ptr = buffer->ptr; + buffer->ptr = map; + +/* Initialize buffer in system memory. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) + ptr[i] = i; + +/* Migrate memory to device. */ + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); + +/* Try freeing it. */ + ret =madvise (map, size,MADV_FREE ); +ASSERT_EQ (ret,0 ); + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); +} + +/* +* Migrate private anonymous huge page and fault back to sysmem. +*/ +TEST_F(hmm, migrate_anon_huge_fault) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size; +unsigned long i; +void *old_ptr; +void *map; +int *ptr; +int ret; + + size =TWOMEG ; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =2 * size; + buffer->mirror =malloc (size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF , size); + + buffer->ptr =mmap (NULL ,2 * size, +PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS , + buffer->fd,0 ); +ASSERT_NE (buffer->ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )buffer->ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + old_ptr = buffer->ptr; + buffer->ptr = map; + +/* Initialize buffer in system memory. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) + ptr[i] = i; + +/* Migrate memory to device. */ + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); + +/* Fault pages back to system memory and check them. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); +} + +/* +* Migrate private anonymous huge page with allocation errors. +*/ +TEST_F(hmm, migrate_anon_huge_err) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size; +unsigned long i; +void *old_ptr; +void *map; +int *ptr; +int ret; + + size =TWOMEG ; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =2 * size; + buffer->mirror =malloc (2 * size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF ,2 * size); + + old_ptr =mmap (NULL ,2 * size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS , buffer->fd,0 ); +ASSERT_NE (old_ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )old_ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + buffer->ptr = map; + +/* Initialize buffer in system memory. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) + ptr[i] = i; + +/* Migrate memory to device but force a THP allocation error. */ + ret =hmm_dmirror_cmd (self->fd,HMM_DMIRROR_FLAGS , buffer, +HMM_DMIRROR_FLAG_FAIL_ALLOC ); +ASSERT_EQ (ret,0 ); + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) { +ASSERT_EQ (ptr[i], i); +if (ptr[i] != i) +break ; + } + +/* Try faulting back a single (PAGE_SIZE) page. */ + ptr = buffer->ptr; +ASSERT_EQ (ptr[2048 ],2048 ); + +/* unmap and remap the region to reset things. */ + ret =munmap (old_ptr,2 * size); +ASSERT_EQ (ret,0 ); + old_ptr =mmap (NULL ,2 * size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS , buffer->fd,0 ); +ASSERT_NE (old_ptr,MAP_FAILED ); + map = (void *)ALIGN ((uintptr_t )old_ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + buffer->ptr = map; + +/* Initialize buffer in system memory. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) + ptr[i] = i; + +/* Migrate THP to device. */ + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* +* Force an allocation error when faulting back a THP resident in the +* device. +*/ + ret =hmm_dmirror_cmd (self->fd,HMM_DMIRROR_FLAGS , buffer, +HMM_DMIRROR_FLAG_FAIL_ALLOC ); +ASSERT_EQ (ret,0 ); + + ret =hmm_migrate_dev_to_sys (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); + ptr = buffer->ptr; +ASSERT_EQ (ptr[2048 ],2048 ); + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); +} + +/* +* Migrate private anonymous huge zero page with allocation errors. +*/ +TEST_F(hmm, migrate_anon_huge_zero_err) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size; +unsigned long i; +void *old_ptr; +void *map; +int *ptr; +int ret; + + size =TWOMEG ; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =2 * size; + buffer->mirror =malloc (2 * size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF ,2 * size); + + old_ptr =mmap (NULL ,2 * size,PROT_READ , +MAP_PRIVATE |MAP_ANONYMOUS , buffer->fd,0 ); +ASSERT_NE (old_ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )old_ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + buffer->ptr = map; + +/* Migrate memory to device but force a THP allocation error. */ + ret =hmm_dmirror_cmd (self->fd,HMM_DMIRROR_FLAGS , buffer, +HMM_DMIRROR_FLAG_FAIL_ALLOC ); +ASSERT_EQ (ret,0 ); + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i],0 ); + +/* Try faulting back a single (PAGE_SIZE) page. */ + ptr = buffer->ptr; +ASSERT_EQ (ptr[2048 ],0 ); + +/* unmap and remap the region to reset things. */ + ret =munmap (old_ptr,2 * size); +ASSERT_EQ (ret,0 ); + old_ptr =mmap (NULL ,2 * size,PROT_READ , +MAP_PRIVATE |MAP_ANONYMOUS , buffer->fd,0 ); +ASSERT_NE (old_ptr,MAP_FAILED ); + map = (void *)ALIGN ((uintptr_t )old_ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + buffer->ptr = map; + +/* Initialize buffer in system memory (zero THP page). */ + ret = ptr[0 ]; +ASSERT_EQ (ret,0 ); + +/* Migrate memory to device but force a THP allocation error. */ + ret =hmm_dmirror_cmd (self->fd,HMM_DMIRROR_FLAGS , buffer, +HMM_DMIRROR_FLAG_FAIL_ALLOC ); +ASSERT_EQ (ret,0 ); + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Fault the device memory back and check it. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i],0 ); + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); +}TEST_HARNESS_MAIN
Signature · 1 line
2.50.1
PATCH v4 13/15 selftests/mm/hmm-tests: partial unmap, mremap and anon_write tests
In reply to Balbir SinghFrom: Matthew Brost <[email protected]>
Add partial unmap test case which munmaps memory while in the device.
Add tests exercising mremap on faulted-in memory (CPU and GPU) at various offsets and verify correctness.
Update anon_write_child to read device memory after fork verifying this flow works in the kernel.
Both THP and non-THP cases are updated.
--- tools/testing/selftests/mm/hmm-tests.c | 312 ++++++++++++++++++++----- 1 file changed, 252 insertions(+), 60 deletions(-)
tools/testing/selftests/mm/hmm-tests.c +252 -60
@@ -50,6 +50,8 @@ enum { HMM_COHERENCE_DEVICE_TWO, }; +#define ONEKB (1 << 10) +#define ONEMEG (1 << 20)#define TWOMEG (1 << 21)#define HMM_BUFFER_SIZE (1024 << 12)#define HMM_PATH_MAX 64
@@ -525,6 +527,8 @@ TEST_F(hmm, anon_write_prot)/* * Check that a device writing an anonymous private mapping * will copy-on-write if a child process inherits the mapping. +* +* Also verifies after fork() memory the device can be read by child. */ TEST_F(hmm, anon_write_child) {
@@ -532,72 +536,101 @@ TEST_F(hmm, anon_write_child)unsigned long npages;unsigned long size;unsigned long i; +void *old_ptr; +void *map;int *ptr;pid_t pid;int child_fd; -int ret; - - npages =ALIGN (HMM_BUFFER_SIZE , self->page_size) >> self->page_shift; -ASSERT_NE (npages,0 ); - size = npages << self->page_shift; - - buffer =malloc (sizeof (*buffer)); -ASSERT_NE (buffer,NULL ); - - buffer->fd =-1 ; - buffer->size = size; - buffer->mirror =malloc (size); -ASSERT_NE (buffer->mirror,NULL ); - - buffer->ptr =mmap (NULL , size, -PROT_READ |PROT_WRITE , -MAP_PRIVATE |MAP_ANONYMOUS , - buffer->fd,0 ); -ASSERT_NE (buffer->ptr,MAP_FAILED ); - -/* Initialize buffer->ptr so we can tell if it is written. */ -for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) - ptr[i] = i; +int ret, use_thp, migrate; + +for (migrate =0 ; migrate <2 ; ++migrate) { +for (use_thp =0 ; use_thp <2 ; ++use_thp) { + npages =ALIGN (use_thp ?TWOMEG :HMM_BUFFER_SIZE , + self->page_size) >> self->page_shift; +ASSERT_NE (npages,0 ); + size = npages << self->page_shift; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size = size *2 ; + buffer->mirror =malloc (size); +ASSERT_NE (buffer->mirror,NULL ); + + buffer->ptr =mmap (NULL , size *2 , +PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS , + buffer->fd,0 ); +ASSERT_NE (buffer->ptr,MAP_FAILED ); + + old_ptr = buffer->ptr; +if (use_thp) { + map = (void *)ALIGN ((uintptr_t )buffer->ptr, size); + ret =madvise (map, size,MADV_HUGEPAGE ); +ASSERT_EQ (ret,0 ); + buffer->ptr = map; + } + +/* Initialize buffer->ptr so we can tell if it is written. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) + ptr[i] = i; + +/* Initialize data that the device will write to buffer->ptr. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) + ptr[i] = -i; + +if (migrate) { + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + + } + + pid =fork (); +if (pid ==-1 ) +ASSERT_EQ (pid,0 ); +if (pid !=0 ) { +waitpid (pid, &ret,0 ); +ASSERT_EQ (WIFEXITED (ret),1 ); + +/* Check that the parent's buffer did not change. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); +continue ; + } + +/* Check that we see the parent's values. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); +if (!migrate) { +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], -i); + } + +/* The child process needs its own mirror to its own mm. */ + child_fd =hmm_open (0 ); +ASSERT_GE (child_fd,0 ); + +/* Simulate a device writing system memory. */ + ret =hmm_dmirror_cmd (child_fd,HMM_DMIRROR_WRITE , buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); +ASSERT_EQ (buffer->faults,1 ); -/* Initialize data that the device will write to buffer->ptr. */ -for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) - ptr[i] = -i; +/* Check what the device wrote. */ +if (!migrate) { +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], -i); + } - pid =fork (); -if (pid ==-1 ) -ASSERT_EQ (pid,0 ); -if (pid !=0 ) { -waitpid (pid, &ret,0 ); -ASSERT_EQ (WIFEXITED (ret),1 ); - -/* Check that the parent's buffer did not change. */ -for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) -ASSERT_EQ (ptr[i], i); -return ; +close (child_fd); +exit (0 ); + } } - -/* Check that we see the parent's values. */ -for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) -ASSERT_EQ (ptr[i], i); -for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) -ASSERT_EQ (ptr[i], -i); - -/* The child process needs its own mirror to its own mm. */ - child_fd =hmm_open (0 ); -ASSERT_GE (child_fd,0 ); - -/* Simulate a device writing system memory. */ - ret =hmm_dmirror_cmd (child_fd,HMM_DMIRROR_WRITE , buffer, npages); -ASSERT_EQ (ret,0 ); -ASSERT_EQ (buffer->cpages, npages); -ASSERT_EQ (buffer->faults,1 ); - -/* Check what the device wrote. */ -for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) -ASSERT_EQ (ptr[i], -i); - -close (child_fd); -exit (0 ); }/*
@@ -2290,6 +2323,165 @@ TEST_F(hmm, migrate_anon_huge_fault)hmm_buffer_free (buffer); } +/* +* Migrate memory and fault back to sysmem after partially unmapping. +*/ +TEST_F(hmm, migrate_partial_unmap_fault) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size =TWOMEG ; +unsigned long i; +void *old_ptr; +void *map; +int *ptr; +int ret, j, use_thp; +int offsets[] = {0 ,512 *ONEKB ,ONEMEG }; + +for (use_thp =0 ; use_thp <2 ; ++use_thp) { +for (j =0 ; j <ARRAY_SIZE (offsets); ++j) { + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =2 * size; + buffer->mirror =malloc (size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF , size); + + buffer->ptr =mmap (NULL ,2 * size, +PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS , + buffer->fd,0 ); +ASSERT_NE (buffer->ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )buffer->ptr, size); +if (use_thp) + ret =madvise (map, size,MADV_HUGEPAGE ); +else + ret =madvise (map, size,MADV_NOHUGEPAGE ); +ASSERT_EQ (ret,0 ); + old_ptr = buffer->ptr; + buffer->ptr = map; + +/* Initialize buffer in system memory. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) + ptr[i] = i; + +/* Migrate memory to device. */ + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); + +munmap (buffer->ptr + offsets[j],ONEMEG ); + +/* Fault pages back to system memory and check them. */ +for (i =0 , ptr = buffer->ptr; i < size /sizeof (*ptr); ++i) +if (i *sizeof (int ) < offsets[j] || + i *sizeof (int ) >= offsets[j] +ONEMEG ) +ASSERT_EQ (ptr[i], i); + + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); + } + } +} + +TEST_F(hmm, migrate_remap_fault) +{ +struct hmm_buffer *buffer; +unsigned long npages; +unsigned long size =TWOMEG ; +unsigned long i; +void *old_ptr, *new_ptr =NULL ; +void *map; +int *ptr; +int ret, j, use_thp, dont_unmap, before; +int offsets[] = {0 ,512 *ONEKB ,ONEMEG }; + +for (before =0 ; before <2 ; ++before) { +for (dont_unmap =0 ; dont_unmap <2 ; ++dont_unmap) { +for (use_thp =0 ; use_thp <2 ; ++use_thp) { +for (j =0 ; j <ARRAY_SIZE (offsets); ++j) { +int flags =MREMAP_MAYMOVE |MREMAP_FIXED ; + +if (dont_unmap) + flags |=MREMAP_DONTUNMAP ; + + buffer =malloc (sizeof (*buffer)); +ASSERT_NE (buffer,NULL ); + + buffer->fd =-1 ; + buffer->size =8 * size; + buffer->mirror =malloc (size); +ASSERT_NE (buffer->mirror,NULL ); +memset (buffer->mirror,0xFF , size); + + buffer->ptr =mmap (NULL , buffer->size, +PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS , + buffer->fd,0 ); +ASSERT_NE (buffer->ptr,MAP_FAILED ); + + npages = size >> self->page_shift; + map = (void *)ALIGN ((uintptr_t )buffer->ptr, size); +if (use_thp) + ret =madvise (map, size,MADV_HUGEPAGE ); +else + ret =madvise (map, size,MADV_NOHUGEPAGE ); +ASSERT_EQ (ret,0 ); + old_ptr = buffer->ptr; +munmap (map + size, size *2 ); + buffer->ptr = map; + +/* Initialize buffer in system memory. */ +for (i =0 , ptr = buffer->ptr; + i < size /sizeof (*ptr); ++i) + ptr[i] = i; + +if (before) { + new_ptr =mremap ((void *)map, size, size, flags, + map + size + offsets[j]); +ASSERT_NE (new_ptr,MAP_FAILED ); + buffer->ptr = new_ptr; + } + +/* Migrate memory to device. */ + ret =hmm_migrate_sys_to_dev (self->fd, buffer, npages); +ASSERT_EQ (ret,0 ); +ASSERT_EQ (buffer->cpages, npages); + +/* Check what the device read. */ +for (i =0 , ptr = buffer->mirror; + i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); + +if (!before) { + new_ptr =mremap ((void *)map, size, size, flags, + map + size + offsets[j]); +ASSERT_NE (new_ptr,MAP_FAILED ); + buffer->ptr = new_ptr; + } + +/* Fault pages back to system memory and check them. */ +for (i =0 , ptr = buffer->ptr; + i < size /sizeof (*ptr); ++i) +ASSERT_EQ (ptr[i], i); + +munmap (new_ptr, size); + buffer->ptr = old_ptr; +hmm_buffer_free (buffer); + } + } + } + } +} +/* * Migrate private anonymous huge page with allocation errors. */
Signature · 1 line
2.50.1
PATCH v4 14/15 selftests/mm/hmm-tests: new throughput tests including THP
In reply to Balbir SinghAdd new benchmark style support to test transfer bandwidth for zone device memory operations.
--- tools/testing/selftests/mm/hmm-tests.c | 197 ++++++++++++++++++++++++- 1 file changed, 196 insertions(+), 1 deletion(-)
tools/testing/selftests/mm/hmm-tests.c +196 -1
@@ -25,6 +25,7 @@#include <sys/stat.h> #include <sys/mman.h> #include <sys/ioctl.h> +#include <sys/time.h> /*
@@ -209,8 +210,10 @@ static void hmm_buffer_free(struct hmm_buffer *buffer)if (buffer ==NULL )return ; -if (buffer->ptr) +if (buffer->ptr) {munmap (buffer->ptr, buffer->size); + buffer->ptr =NULL ; + }free (buffer->mirror);free (buffer); }
@@ -2658,4 +2661,196 @@ TEST_F(hmm, migrate_anon_huge_zero_err) buffer->ptr = old_ptr;hmm_buffer_free (buffer); } + +struct benchmark_results { +double sys_to_dev_time; +double dev_to_sys_time; +double throughput_s2d; +double throughput_d2s; +}; + +static double get_time_ms (void ) +{ +struct timeval tv; + +gettimeofday (&tv,NULL ); +return (tv.tv_sec *1000.0 ) + (tv.tv_usec /1000.0 ); +} + +static inline struct hmm_buffer *hmm_buffer_alloc (unsigned long size) +{ +struct hmm_buffer *buffer; + + buffer =malloc (sizeof (*buffer)); + + buffer->fd =-1 ; + buffer->size = size; + buffer->mirror =malloc (size); +memset (buffer->mirror,0xFF , size); +return buffer; +} + +static void print_benchmark_results (const char *test_name,size_t buffer_size, +struct benchmark_results *thp, +struct benchmark_results *regular) +{ +double s2d_improvement = ((regular->sys_to_dev_time - thp->sys_to_dev_time) / + regular->sys_to_dev_time) *100.0 ; +double d2s_improvement = ((regular->dev_to_sys_time - thp->dev_to_sys_time) / + regular->dev_to_sys_time) *100.0 ; +double throughput_s2d_improvement = ((thp->throughput_s2d - regular->throughput_s2d) / + regular->throughput_s2d) *100.0 ; +double throughput_d2s_improvement = ((thp->throughput_d2s - regular->throughput_d2s) / + regular->throughput_d2s) *100.0 ; + +printf ("\n=== %s (%.1f MB) ===\n" , test_name, buffer_size / (1024.0 *1024.0 )); +printf (" | With THP | Without THP | Improvement\n" ); +printf ("---------------------------------------------------------------------\n" ); +printf ("Sys->Dev Migration | %.3f ms | %.3f ms | %.1f%%\n" , + thp->sys_to_dev_time, regular->sys_to_dev_time, s2d_improvement); +printf ("Dev->Sys Migration | %.3f ms | %.3f ms | %.1f%%\n" , + thp->dev_to_sys_time, regular->dev_to_sys_time, d2s_improvement); +printf ("S->D Throughput | %.2f GB/s | %.2f GB/s | %.1f%%\n" , + thp->throughput_s2d, regular->throughput_s2d, throughput_s2d_improvement); +printf ("D->S Throughput | %.2f GB/s | %.2f GB/s | %.1f%%\n" , + thp->throughput_d2s, regular->throughput_d2s, throughput_d2s_improvement); +} + +/* +* Run a single migration benchmark +* fd: file descriptor for hmm device +* use_thp: whether to use THP +* buffer_size: size of buffer to allocate +* iterations: number of iterations +* results: where to store results +*/ +static inline int run_migration_benchmark (int fd,int use_thp,size_t buffer_size, +int iterations,struct benchmark_results *results) +{ +struct hmm_buffer *buffer; +unsigned long npages = buffer_size /sysconf (_SC_PAGESIZE); +double start, end; +double s2d_total =0 , d2s_total =0 ; +int ret, i; +int *ptr; + + buffer =hmm_buffer_alloc (buffer_size); + +/* Map memory */ + buffer->ptr =mmap (NULL , buffer_size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); + +if (!buffer->ptr) +return -1 ; + +/* Apply THP hint if requested */ +if (use_thp) + ret =madvise (buffer->ptr, buffer_size,MADV_HUGEPAGE ); +else + ret =madvise (buffer->ptr, buffer_size,MADV_NOHUGEPAGE ); + +if (ret) +return ret; + +/* Initialize memory to make sure pages are allocated */ + ptr = (int *)buffer->ptr; +for (i =0 ; i < buffer_size /sizeof (int ); i++) + ptr[i] = i &0xFF ; + +/* Warmup iteration */ + ret =hmm_migrate_sys_to_dev (fd, buffer, npages); +if (ret) +return ret; + + ret =hmm_migrate_dev_to_sys (fd, buffer, npages); +if (ret) +return ret; + +/* Benchmark iterations */ +for (i =0 ; i < iterations; i++) { +/* System to device migration */ + start =get_time_ms (); + + ret =hmm_migrate_sys_to_dev (fd, buffer, npages); +if (ret) +return ret; + + end =get_time_ms (); + s2d_total += (end - start); + +/* Device to system migration */ + start =get_time_ms (); + + ret =hmm_migrate_dev_to_sys (fd, buffer, npages); +if (ret) +return ret; + + end =get_time_ms (); + d2s_total += (end - start); + } + +/* Calculate average times and throughput */ + results->sys_to_dev_time = s2d_total / iterations; + results->dev_to_sys_time = d2s_total / iterations; + results->throughput_s2d = (buffer_size / (1024.0 *1024.0 *1024.0 )) / + (results->sys_to_dev_time /1000.0 ); + results->throughput_d2s = (buffer_size / (1024.0 *1024.0 *1024.0 )) / + (results->dev_to_sys_time /1000.0 ); + +/* Cleanup */ +hmm_buffer_free (buffer); +return 0 ; +} + +/* +* Benchmark THP migration with different buffer sizes +*/ +TEST_F_TIMEOUT(hmm, benchmark_thp_migration, 120) +{ +struct benchmark_results thp_results, regular_results; +size_t thp_size =2 *1024 *1024 ;/* 2MB - typical THP size */ +int iterations =5 ; + +printf ("\nHMM THP Migration Benchmark\n" ); +printf ("---------------------------\n" ); +printf ("System page size: %ld bytes\n" ,sysconf (_SC_PAGESIZE)); + +/* Test different buffer sizes */ +size_t test_sizes[] = { + thp_size /4 ,/* 512KB - smaller than THP */ + thp_size /2 ,/* 1MB - half THP */ + thp_size,/* 2MB - single THP */ + thp_size *2 ,/* 4MB - two THPs */ + thp_size *4 ,/* 8MB - four THPs */ + thp_size *8 ,/* 16MB - eight THPs */ + thp_size *128 ,/* 256MB - one twenty eight THPs */ + }; + +static const char *const test_names[] = { +"Small Buffer (512KB)" , +"Half THP Size (1MB)" , +"Single THP Size (2MB)" , +"Two THP Size (4MB)" , +"Four THP Size (8MB)" , +"Eight THP Size (16MB)" , +"One twenty eight THP Size (256MB)" + }; + +int num_tests =ARRAY_SIZE (test_sizes); + +/* Run all tests */ +for (int i =0 ; i < num_tests; i++) { +/* Test with THP */ +ASSERT_EQ (run_migration_benchmark (self->fd,1 , test_sizes[i], + iterations, &thp_results),0 ); + +/* Test without THP */ +ASSERT_EQ (run_migration_benchmark (self->fd,0 , test_sizes[i], + iterations, ®ular_results),0 ); + +/* Print results */ +print_benchmark_results (test_names[i], test_sizes[i], + &thp_results, ®ular_results); + } +}TEST_HARNESS_MAIN
Signature · 1 line
2.50.1
PATCH v4 15/15 gpu/drm/nouveau: enable THP support for GPU memory migration
In reply to Balbir SinghEnable MIGRATE_VMA_SELECT_COMPOUND support in nouveau driver to take advantage of THP zone device migration capabilities.
Update migration and eviction code paths to handle compound page sizes appropriately, improving memory bandwidth utilization and reducing migration overhead for large GPU memory allocations.
--- drivers/gpu/drm/nouveau/nouveau_dmem.c | 306 ++++++++++++++++++------- drivers/gpu/drm/nouveau/nouveau_svm.c | 6 +- drivers/gpu/drm/nouveau/nouveau_svm.h | 3 +- 3 files changed, 231 insertions(+), 84 deletions(-)
drivers/gpu/drm/nouveau/nouveau_dmem.c +225 -81
@@ -48,8 +48,9 @@* bigger page size) at lowest level and have some shim layer on top that would * provide the same functionality as TTM. */ -#define DMEM_CHUNK_SIZE (2UL << 20) +#define DMEM_CHUNK_SIZE (HPAGE_PMD_SIZE)#define DMEM_CHUNK_NPAGES (DMEM_CHUNK_SIZE >> PAGE_SHIFT) +#define NR_CHUNKS (128)enum nouveau_aper {NOUVEAU_APER_VIRT ,
@@ -83,9 +84,15 @@ struct nouveau_dmem {struct list_head chunks;struct mutex mutex;struct page *free_pages; +struct folio *free_folios;spinlock_t lock; }; +struct nouveau_dmem_dma_info { +dma_addr_t dma_addr; +size_t size; +}; +static struct nouveau_dmem_chunk *nouveau_page_to_chunk (struct page *page) { return container_of(page_pgmap(page), struct nouveau_dmem_chunk,
@@ -112,10 +119,16 @@ static void nouveau_dmem_page_free(struct page *page) {struct nouveau_dmem_chunk *chunk =nouveau_page_to_chunk (page);struct nouveau_dmem *dmem = chunk->drm->dmem; +struct folio *folio =page_folio (page);spin_lock (&dmem->lock); - page->zone_device_data = dmem->free_pages; - dmem->free_pages = page; +if (folio_order (folio)) { + page->zone_device_data = dmem->free_folios; + dmem->free_folios = folio; + }else { + page->zone_device_data = dmem->free_pages; + dmem->free_pages = page; + }WARN_ON (!chunk->callocated); chunk->callocated--;
@@ -139,20 +152,28 @@ static void nouveau_dmem_fence_done(struct nouveau_fence **fence) } } -static int nouveau_dmem_copy_one (struct nouveau_drm *drm,struct page *spage, -struct page *dpage,dma_addr_t *dma_addr) +static int nouveau_dmem_copy_folio (struct nouveau_drm *drm, +struct folio *sfolio,struct folio *dfolio, +struct nouveau_dmem_dma_info *dma_info) {struct device *dev = drm->dev->dev; +struct page *dpage =folio_page (dfolio,0 ); +struct page *spage =folio_page (sfolio,0 ); -lock_page (dpage); +folio_lock (dfolio); - *dma_addr =dma_map_page (dev, dpage,0 ,PAGE_SIZE ,DMA_BIDIRECTIONAL ); -if (dma_mapping_error (dev, *dma_addr)) + dma_info->dma_addr =dma_map_page (dev, dpage,0 ,page_size (dpage), +DMA_BIDIRECTIONAL ); + dma_info->size =page_size (dpage); +if (dma_mapping_error (dev, dma_info->dma_addr))return -EIO ; -if (drm->dmem->migrate.copy_func (drm,1 ,NOUVEAU_APER_HOST , *dma_addr, -NOUVEAU_APER_VRAM ,nouveau_dmem_page_addr (spage))) { -dma_unmap_page (dev, *dma_addr,PAGE_SIZE ,DMA_BIDIRECTIONAL ); +if (drm->dmem->migrate.copy_func (drm,folio_nr_pages (sfolio), +NOUVEAU_APER_HOST , dma_info->dma_addr, +NOUVEAU_APER_VRAM , +nouveau_dmem_page_addr (spage))) { +dma_unmap_page (dev, dma_info->dma_addr,page_size (dpage), +DMA_BIDIRECTIONAL );return -EIO ; }
@@ -165,21 +186,47 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)struct nouveau_dmem *dmem = drm->dmem;struct nouveau_fence *fence;struct nouveau_svmm *svmm; -struct page *spage, *dpage; -unsigned long src =0 , dst =0 ; -dma_addr_t dma_addr =0 ; +struct page *dpage;vm_fault_t ret =0 ;struct migrate_vma args = { .vma = vmf->vma, - .start = vmf->address, - .end = vmf->address +PAGE_SIZE , - .src = &src, - .dst = &dst, .pgmap_owner = drm->dev, .fault_page = vmf->page, - .flags =MIGRATE_VMA_SELECT_DEVICE_PRIVATE , + .flags =MIGRATE_VMA_SELECT_DEVICE_PRIVATE | +MIGRATE_VMA_SELECT_COMPOUND , + .src =NULL , + .dst =NULL , }; +unsigned int order, nr; +struct folio *sfolio, *dfolio; +struct nouveau_dmem_dma_info dma_info; + + sfolio =page_folio (vmf->page); + order =folio_order (sfolio); + nr =1 << order; + +/* +* Handle partial unmap faults, where the folio is large, but +* the pmd is split. +*/ +if (vmf->pte) { + order =0 ; + nr =1 ; + } + +if (order) + args.flags |=MIGRATE_VMA_SELECT_COMPOUND ; + args.start =ALIGN_DOWN (vmf->address, (PAGE_SIZE << order)); + args.vma = vmf->vma; + args.end = args.start + (PAGE_SIZE << order); + args.src =kcalloc (nr,sizeof (*args.src),GFP_KERNEL ); + args.dst =kcalloc (nr,sizeof (*args.dst),GFP_KERNEL ); + +if (!args.src || !args.dst) { + ret =VM_FAULT_OOM ; + goto err; + }/* * FIXME what we really want is to find some heuristic to migrate more * than just one page on CPU fault. When such fault happens it is very
@@ -190,20 +237,26 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)if (!args.cpages)return 0 ; - spage =migrate_pfn_to_page (src); -if (!spage || !(src &MIGRATE_PFN_MIGRATE )) - goto done; - - dpage =alloc_page_vma (GFP_HIGHUSER | __GFP_ZERO, vmf->vma, vmf->address); -if (!dpage) +if (order) + dpage =folio_page (vma_alloc_folio (GFP_HIGHUSER | __GFP_ZERO, + order, vmf->vma, vmf->address),0 ); +else + dpage =alloc_page_vma (GFP_HIGHUSER | __GFP_ZERO, vmf->vma, + vmf->address); +if (!dpage) { + ret =VM_FAULT_OOM ; goto done; + } - dst =migrate_pfn (page_to_pfn (dpage)); + args.dst[0 ] =migrate_pfn (page_to_pfn (dpage)); +if (order) + args.dst[0 ] |=MIGRATE_PFN_COMPOUND ; + dfolio =page_folio (dpage); - svmm = spage->zone_device_data; + svmm =folio_zone_device_data (sfolio);mutex_lock (&svmm->mutex);nouveau_svmm_invalidate (svmm, args.start, args.end); - ret =nouveau_dmem_copy_one (drm, spage, dpage, &dma_addr); + ret =nouveau_dmem_copy_folio (drm, sfolio, dfolio, &dma_info);mutex_unlock (&svmm->mutex);if (ret) { ret =VM_FAULT_SIGBUS ;
@@ -213,25 +266,40 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)nouveau_fence_new (&fence, dmem->migrate.chan);migrate_vma_pages (&args);nouveau_dmem_fence_done (&fence); -dma_unmap_page (drm->dev->dev, dma_addr,PAGE_SIZE ,DMA_BIDIRECTIONAL ); +dma_unmap_page (drm->dev->dev, dma_info.dma_addr,PAGE_SIZE , +DMA_BIDIRECTIONAL ); done:migrate_vma_finalize (&args); +err: +kfree (args.src); +kfree (args.dst);return ret; } +static void nouveau_dmem_folio_split (struct folio *head,struct folio *tail) +{ +if (tail ==NULL ) +return ; + tail->pgmap = head->pgmap; + tail->mapping = head->mapping; +folio_set_zone_device_data (tail,folio_zone_device_data (head)); +} +static const struct dev_pagemap_ops nouveau_dmem_pagemap_ops = { .page_free = nouveau_dmem_page_free, .migrate_to_ram = nouveau_dmem_migrate_to_ram, + .folio_split = nouveau_dmem_folio_split, };static int -nouveau_dmem_chunk_alloc (struct nouveau_drm *drm,struct page **ppage) +nouveau_dmem_chunk_alloc (struct nouveau_drm *drm,struct page **ppage, +bool is_large) {struct nouveau_dmem_chunk *chunk;struct resource *res;struct page *page;void *ptr; -unsigned long i, pfn_first; +unsigned long i, pfn_first, pfn;int ret; chunk =kzalloc (sizeof (*chunk),GFP_KERNEL );
@@ -241,7 +309,7 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage) }/* Allocate unused physical address space for device private pages. */ - res =request_free_mem_region (&iomem_resource,DMEM_CHUNK_SIZE , + res =request_free_mem_region (&iomem_resource,DMEM_CHUNK_SIZE *NR_CHUNKS ,"nouveau_dmem" );if (IS_ERR (res)) { ret =PTR_ERR (res);
@@ -274,16 +342,40 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage) pfn_first = chunk->pagemap.range.start >>PAGE_SHIFT ; page =pfn_to_page (pfn_first);spin_lock (&drm->dmem->lock); -for (i =0 ; i <DMEM_CHUNK_NPAGES -1 ; ++i, ++page) { - page->zone_device_data = drm->dmem->free_pages; - drm->dmem->free_pages = page; + + pfn = pfn_first; +for (i =0 ; i <NR_CHUNKS ; i++) { +int j; + +if (!IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) || !is_large) { +for (j =0 ; j <DMEM_CHUNK_NPAGES -1 ; j++, pfn++) { + page =pfn_to_page (pfn); + page->zone_device_data = drm->dmem->free_pages; + drm->dmem->free_pages = page; + } + }else { + page =pfn_to_page (pfn); + page->zone_device_data = drm->dmem->free_folios; + drm->dmem->free_folios =page_folio (page); + pfn +=DMEM_CHUNK_NPAGES ; + } } - *ppage = page; + +/* Move to next page */ +if (is_large) { + *ppage = &drm->dmem->free_folios->page; + drm->dmem->free_folios = (*ppage)->zone_device_data; + }else { + *ppage = drm->dmem->free_pages; + drm->dmem->free_pages = (*ppage)->zone_device_data; + } + chunk->callocated++;spin_unlock (&drm->dmem->lock); -NV_INFO (drm,"DMEM: registered %ldMB of device memory\n" , -DMEM_CHUNK_SIZE >>20 ); +NV_INFO (drm,"DMEM: registered %ldMB of %sdevice memory %lx %lx\n" , +NR_CHUNKS *DMEM_CHUNK_SIZE >>20 , is_large ?"THP " :"" , pfn_first, +nouveau_dmem_page_addr (page));return 0 ;
@@ -298,27 +390,41 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage) }static struct page * -nouveau_dmem_page_alloc_locked (struct nouveau_drm *drm) +nouveau_dmem_page_alloc_locked (struct nouveau_drm *drm,bool is_large) {struct nouveau_dmem_chunk *chunk;struct page *page =NULL ; +struct folio *folio =NULL ;int ret; +unsigned int order =0 ;spin_lock (&drm->dmem->lock); -if (drm->dmem->free_pages) { +if (is_large && drm->dmem->free_folios) { + folio = drm->dmem->free_folios; + page = &folio->page; + drm->dmem->free_folios = page->zone_device_data; + chunk =nouveau_page_to_chunk (&folio->page); + chunk->callocated++; +spin_unlock (&drm->dmem->lock); + order =ilog2 (DMEM_CHUNK_NPAGES ); + }else if (!is_large && drm->dmem->free_pages) { page = drm->dmem->free_pages; drm->dmem->free_pages = page->zone_device_data; chunk =nouveau_page_to_chunk (page); chunk->callocated++;spin_unlock (&drm->dmem->lock); + folio =page_folio (page); }else {spin_unlock (&drm->dmem->lock); - ret =nouveau_dmem_chunk_alloc (drm, &page); + ret =nouveau_dmem_chunk_alloc (drm, &page, is_large);if (ret)return NULL ; + folio =page_folio (page); +if (is_large) + order =ilog2 (DMEM_CHUNK_NPAGES ); } -zone_device_page_init (page); +zone_device_folio_init (folio, order);return page; }
@@ -369,12 +475,12 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) {unsigned long i, npages =range_len (&chunk->pagemap.range) >>PAGE_SHIFT ;unsigned long *src_pfns, *dst_pfns; -dma_addr_t *dma_addrs; +struct nouveau_dmem_dma_info *dma_info;struct nouveau_fence *fence; src_pfns =kvcalloc (npages,sizeof (*src_pfns),GFP_KERNEL | __GFP_NOFAIL); dst_pfns =kvcalloc (npages,sizeof (*dst_pfns),GFP_KERNEL | __GFP_NOFAIL); - dma_addrs =kvcalloc (npages,sizeof (*dma_addrs),GFP_KERNEL | __GFP_NOFAIL); + dma_info =kvcalloc (npages,sizeof (*dma_info),GFP_KERNEL | __GFP_NOFAIL);migrate_device_range (src_pfns, chunk->pagemap.range.start >>PAGE_SHIFT , npages);
@@ -382,17 +488,28 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)for (i =0 ; i < npages; i++) {if (src_pfns[i] &MIGRATE_PFN_MIGRATE ) {struct page *dpage; +struct folio *folio =page_folio ( +migrate_pfn_to_page (src_pfns[i])); +unsigned int order =folio_order (folio); + +if (src_pfns[i] &MIGRATE_PFN_COMPOUND ) { + dpage =folio_page ( +folio_alloc ( +GFP_HIGHUSER_MOVABLE , order),0 ); + }else { +/* +* _GFP_NOFAIL because the GPU is going away and there +* is nothing sensible we can do if we can't copy the +* data back. +*/ + dpage =alloc_page (GFP_HIGHUSER | __GFP_NOFAIL); + } -/* -* _GFP_NOFAIL because the GPU is going away and there -* is nothing sensible we can do if we can't copy the -* data back. -*/ - dpage =alloc_page (GFP_HIGHUSER | __GFP_NOFAIL); dst_pfns[i] =migrate_pfn (page_to_pfn (dpage)); -nouveau_dmem_copy_one (chunk->drm, -migrate_pfn_to_page (src_pfns[i]), dpage, - &dma_addrs[i]); +nouveau_dmem_copy_folio (chunk->drm, +page_folio (migrate_pfn_to_page (src_pfns[i])), +page_folio (dpage), + &dma_info[i]); } }
@@ -403,8 +520,9 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)kvfree (src_pfns);kvfree (dst_pfns);for (i =0 ; i < npages; i++) -dma_unmap_page (chunk->drm->dev->dev, dma_addrs[i],PAGE_SIZE ,DMA_BIDIRECTIONAL ); -kvfree (dma_addrs); +dma_unmap_page (chunk->drm->dev->dev, dma_info[i].dma_addr, + dma_info[i].size,DMA_BIDIRECTIONAL ); +kvfree (dma_info); }void
@@ -607,31 +725,36 @@ nouveau_dmem_init(struct nouveau_drm *drm)static unsigned long nouveau_dmem_migrate_copy_one (struct nouveau_drm *drm,struct nouveau_svmm *svmm,unsigned long src, -dma_addr_t *dma_addr,u64 *pfn) +struct nouveau_dmem_dma_info *dma_info,u64 *pfn) {struct device *dev = drm->dev->dev;struct page *dpage, *spage;unsigned long paddr; +bool is_large = false; +unsigned long mpfn; spage =migrate_pfn_to_page (src);if (!(src &MIGRATE_PFN_MIGRATE )) goto out; - dpage =nouveau_dmem_page_alloc_locked (drm); + is_large = src &MIGRATE_PFN_COMPOUND ; + dpage =nouveau_dmem_page_alloc_locked (drm, is_large);if (!dpage) goto out; paddr =nouveau_dmem_page_addr (dpage);if (spage) { - *dma_addr =dma_map_page (dev, spage,0 ,page_size (spage), + dma_info->dma_addr =dma_map_page (dev, spage,0 ,page_size (spage),DMA_BIDIRECTIONAL ); -if (dma_mapping_error (dev, *dma_addr)) + dma_info->size =page_size (spage); +if (dma_mapping_error (dev, dma_info->dma_addr)) goto out_free_page; -if (drm->dmem->migrate.copy_func (drm,1 , -NOUVEAU_APER_VRAM , paddr,NOUVEAU_APER_HOST , *dma_addr)) +if (drm->dmem->migrate.copy_func (drm,folio_nr_pages (page_folio (spage)), +NOUVEAU_APER_VRAM , paddr,NOUVEAU_APER_HOST , + dma_info->dma_addr)) goto out_dma_unmap; }else { - *dma_addr =DMA_MAPPING_ERROR ; + dma_info->dma_addr =DMA_MAPPING_ERROR ;if (drm->dmem->migrate.clear_func (drm,page_size (dpage),NOUVEAU_APER_VRAM , paddr)) goto out_free_page;
@@ -642,10 +765,13 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm, ((paddr >>PAGE_SHIFT ) <<NVIF_VMM_PFNMAP_V0_ADDR_SHIFT );if (src &MIGRATE_PFN_WRITE ) *pfn |=NVIF_VMM_PFNMAP_V0_W ; -return migrate_pfn (page_to_pfn (dpage)); + mpfn =migrate_pfn (page_to_pfn (dpage)); +if (folio_order (page_folio (dpage))) + mpfn |=MIGRATE_PFN_COMPOUND ; +return mpfn; out_dma_unmap: -dma_unmap_page (dev, *dma_addr,PAGE_SIZE ,DMA_BIDIRECTIONAL ); +dma_unmap_page (dev, dma_info->dma_addr,PAGE_SIZE ,DMA_BIDIRECTIONAL ); out_free_page:nouveau_dmem_page_free_locked (drm, dpage); out:
@@ -655,27 +781,38 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,static void nouveau_dmem_migrate_chunk (struct nouveau_drm *drm,struct nouveau_svmm *svmm,struct migrate_vma *args, -dma_addr_t *dma_addrs,u64 *pfns) +struct nouveau_dmem_dma_info *dma_info,u64 *pfns) {struct nouveau_fence *fence;unsigned long addr = args->start, nr_dma =0 , i; +unsigned long order =0 ; + +for (i =0 ; addr < args->end; ) { +struct folio *folio; -for (i =0 ; addr < args->end; i++) { args->dst[i] =nouveau_dmem_migrate_copy_one (drm, svmm, - args->src[i], dma_addrs + nr_dma, pfns + i); -if (!dma_mapping_error (drm->dev->dev, dma_addrs[nr_dma])) + args->src[i], dma_info + nr_dma, pfns + i); +if (!args->dst[i]) { + i++; + addr +=PAGE_SIZE ; +continue ; + } +if (!dma_mapping_error (drm->dev->dev, dma_info[nr_dma].dma_addr)) nr_dma++; - addr +=PAGE_SIZE ; + folio =page_folio (migrate_pfn_to_page (args->dst[i])); + order =folio_order (folio); + i +=1 << order; + addr += (1 << order) *PAGE_SIZE ; }nouveau_fence_new (&fence, drm->dmem->migrate.chan);migrate_vma_pages (args);nouveau_dmem_fence_done (&fence); -nouveau_pfns_map (svmm, args->vma->vm_mm, args->start, pfns, i); +nouveau_pfns_map (svmm, args->vma->vm_mm, args->start, pfns, i, order);while (nr_dma--) { -dma_unmap_page (drm->dev->dev, dma_addrs[nr_dma],PAGE_SIZE , -DMA_BIDIRECTIONAL ); +dma_unmap_page (drm->dev->dev, dma_info[nr_dma].dma_addr, + dma_info[nr_dma].size,DMA_BIDIRECTIONAL ); }migrate_vma_finalize (args); }
@@ -688,20 +825,27 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm, unsigned long end) {unsigned long npages = (end - start) >>PAGE_SHIFT ; -unsigned long max =min (SG_MAX_SINGLE_ALLOC , npages); -dma_addr_t *dma_addrs; +unsigned long max = npages;struct migrate_vma args = { .vma = vma, .start = start, .pgmap_owner = drm->dev, - .flags =MIGRATE_VMA_SELECT_SYSTEM , + .flags =MIGRATE_VMA_SELECT_SYSTEM + |MIGRATE_VMA_SELECT_COMPOUND , };unsigned long i;u64 *pfns;int ret = -ENOMEM ; +struct nouveau_dmem_dma_info *dma_info; -if (drm->dmem ==NULL ) -return -ENODEV ; +if (drm->dmem ==NULL ) { + ret = -ENODEV ; + goto out; + } + +if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) +if (max > (unsigned long )HPAGE_PMD_NR ) + max = (unsigned long )HPAGE_PMD_NR ; args.src =kcalloc (max,sizeof (*args.src),GFP_KERNEL );if (!args.src)
@@ -710,8 +854,8 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm,if (!args.dst) goto out_free_src; - dma_addrs =kmalloc_array (max,sizeof (*dma_addrs),GFP_KERNEL ); -if (!dma_addrs) + dma_info =kmalloc_array (max,sizeof (*dma_info),GFP_KERNEL ); +if (!dma_info) goto out_free_dst; pfns =nouveau_pfns_alloc (max);
@@ -729,7 +873,7 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm, goto out_free_pfns;if (args.cpages) -nouveau_dmem_migrate_chunk (drm, svmm, &args, dma_addrs, +nouveau_dmem_migrate_chunk (drm, svmm, &args, dma_info, pfns); args.start = args.end; }
@@ -738,7 +882,7 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm, out_free_pfns:nouveau_pfns_free (pfns); out_free_dma: -kfree (dma_addrs); +kfree (dma_info); out_free_dst:kfree (args.dst); out_free_src:
drivers/gpu/drm/nouveau/nouveau_svm.c +4 -2
@@ -921,12 +921,14 @@ nouveau_pfns_free(u64 *pfns)void nouveau_pfns_map (struct nouveau_svmm *svmm,struct mm_struct *mm, -unsigned long addr,u64 *pfns,unsigned long npages) +unsigned long addr,u64 *pfns,unsigned long npages, +unsigned int page_shift) {struct nouveau_pfnmap_args *args =nouveau_pfns_to_args (pfns); args->p.addr = addr; - args->p.size = npages <<PAGE_SHIFT ; + args->p.size = npages << page_shift; + args->p.page = page_shift;mutex_lock (&svmm->mutex);
drivers/gpu/drm/nouveau/nouveau_svm.h +2 -1
@@ -33,7 +33,8 @@ void nouveau_svmm_invalidate(struct nouveau_svmm *svmm, u64 start, u64 limit);u64 *nouveau_pfns_alloc (unsigned long npages);void nouveau_pfns_free (u64 *pfns);void nouveau_pfns_map (struct nouveau_svmm *svmm,struct mm_struct *mm, -unsigned long addr,u64 *pfns,unsigned long npages); +unsigned long addr,u64 *pfns,unsigned long npages, +unsigned int page_shift);#else /* IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM) */ static inline void nouveau_svm_init (struct nouveau_drm *drm) {}static inline void nouveau_svm_fini (struct nouveau_drm *drm) {}
Signature · 1 line
2.50.1