mm: remove extra ZONE_DEVICE struct page refcount
Subsystems (17)
v2 cover mm: remove extra ZONE_DEVICE struct page refcount
Matthew Wilcox, Ira Weiny, and others have complained that ZONE_DEVICE struct page reference counting is ugly because they are "free" when the reference count is one instead of zero. This leads to explicit checks for ZONE_DEVICE pages in places like put_page(), GUP, THP splitting, and page migration which have to adjust the expected reference count when determining if the page is isolated or idle. This is my attempt to make ZONE_DEVICE pages be free when the reference count is zero and removing the special cases.
I'm only sending this out as a RFC since I'm not that familiar with the DAX, PMEM, XEN, and other uses of ZONE_DEVICE struct pages allocated with devm_memremap_pages() or memremap_pages() but my best reading of the code looks like it might be OK. I could use help testing these configurations.
I have been able to successfully run xfstests on ext4 with the memmap kernel boot option to simulate pmem.
One of the big changes in v2 is that devm_memremap_pages() and memremap_pages() now return the struct pages' reference count set to zero instead of one. Normally, get_page() will VM_BUG_ON_PAGE() if page->_refcount is zero. I didn't see any such warnings running the xfstests with dax/pmem but I'm not clear how the zero to one reference count is handled.
Other changes in v2:
Rebased to Linux-5.9.0-rc6 to include pmem fixes.
I added patch 1 to introduce a page refcount helper for ext4 and xfs as suggested by Christoph Hellwig.
I also applied Christoph Hellwig's other suggested changes for removing the devmap_managed_key, etc.
Ralph Campbell (2):
ext4/xfs: add page refcount helper
mm: remove extra ZONE_DEVICE struct page refcount
arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- fs/dax.c | 8 +-- fs/ext4/inode.c | 2 +- fs/xfs/xfs_file.c | 2 +- include/linux/dax.h | 5 ++ include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 82 ++++++-------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 46 +++------------ 15 files changed, 54 insertions(+), 208 deletions(-)
Signature · 1 line
2.20.1
PATCH v2 1/2 ext4/xfs: add page refcount helper
There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.
--- fs/dax.c | 8 ++++---- fs/ext4/inode.c | 2 +- fs/xfs/xfs_file.c | 2 +- include/linux/dax.h | 5 +++++ 4 files changed, 11 insertions(+), 6 deletions(-)
fs/dax.c +4 -4
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -WARN_ON_ONCE (trunc &&page_ref_count (page) >1 ); +WARN_ON_ONCE (trunc && !dax_layout_is_idle_page (page));WARN_ON_ONCE (page->mapping && page->mapping != mapping); page->mapping =NULL ; page->index =0 ;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -if (page_ref_count (page) >1 ) +if (!dax_layout_is_idle_page (page))return page; }return NULL ;
@@ -560,11 +560,11 @@ static void *grab_mapping_entry(struct xa_state *xas,/** * dax_layout_busy_page - find first pinned page in @mapping -* @mapping: address space to scan for a page with ref count > 1 +* @mapping: address space to scan for a page with ref count > 0 * * DAX requires ZONE_DEVICE mapped pages. These pages are never * 'onlined' to the page allocator so they are considered idle when -* page->count == 1. A filesystem uses this interface to determine if +* page->count == 0. A filesystem uses this interface to determine if * any page in the mapping is busy, i.e. for DMA, or other * get_user_pages() usages. *
fs/ext4/inode.c +1 -1
@@ -3927,7 +3927,7 @@ int ext4_break_layouts(struct inode *inode)return 0 ; error =___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) ==1 , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,ext4_wait_dax_page (ei)); }while (error ==0 );
fs/xfs/xfs_file.c +1 -1
@@ -750,7 +750,7 @@ xfs_break_dax_layouts( *retry = true;return ___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) ==1 ,TASK_INTERRUPTIBLE , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,xfs_wait_dax_page (inode)); }
include/linux/dax.h +5 -0
@@ -238,4 +238,9 @@ static inline bool dax_mapping(struct address_space *mapping) return mapping->host && IS_DAX(mapping->host); } +static inline bool dax_layout_is_idle_page (struct page *page) +{ +return page_ref_count (page) <=1 ; +} +#endif
Signature · 1 line
2.20.1
Re: [PATCH 1/2] ext4/xfs: add page refcount helper
Quoted 80 lines · Ralph Campbell
There are several places where
ZONE_DEVICEstruct pages assume a reference count = 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.Signed-off-by Ralph Campbell <[email protected]>--- fs/dax.c | 8 ++++---- fs/ext4/inode.c | 2 +- fs/xfs/xfs_file.c | 2 +- include/linux/dax.h | 5 +++++ 4 files changed, 11 insertions(+), 6 deletions(-)fs/dax.c +4
-4@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -WARN_ON_ONCE (trunc &&page_ref_count (page) >1 ); +WARN_ON_ONCE (trunc && !dax_layout_is_idle_page (page));WARN_ON_ONCE (page->mapping && page->mapping != mapping); page->mapping =NULL ; page->index =0 ;@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -if (page_ref_count (page) >1 ) +if (!dax_layout_is_idle_page (page))return page; }return NULL ;@@ -560,11 +560,11 @@ static void *grab_mapping_entry(struct xa_state *xas,/** * dax_layout_busy_page - find first pinned page in @mapping -* @mapping: address space to scan for a page with ref count > 1 +* @mapping: address space to scan for a page with ref count > 0 * * DAX requires ZONE_DEVICE mapped pages. These pages are never * 'onlined' to the page allocator so they are considered idle when -* page->count = 1. A filesystem uses this interface to determine if +* page->count = 0. A filesystem uses this interface to determine if * any page in the mapping is busy, i.e. for DMA, or other * get_user_pages() usages. * fs/ext4/inode.c +1
-1@@ -3927,7 +3927,7 @@ int ext4_break_layouts(struct inode *inode)return 0 ; error =___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) =1 , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,ext4_wait_dax_page (ei)); }while (error =0 );fs/xfs/xfs_file.c +1
-1@@ -750,7 +750,7 @@ xfs_break_dax_layouts( *retry = true;return ___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) =1 ,TASK_INTERRUPTIBLE , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,xfs_wait_dax_page (inode)); }include/linux/dax.h +0
-0
include/linux/dax.h static inline bool dax_mapping(struct address_space *mapping) · line 243
Quoted 6 lines
include/linux/dax.h +3
-0@@ -238,4 +238,9 @@ static inline bool dax_mapping(struct address_space *mapping)return mapping->host &&IS_DAX (mapping->host); } +static inline bool dax_layout_is_idle_page (struct page *page) +{ +return page_ref_count (page) <=1 ;
Why convert the check from "= 1" to "<= 1" and then back to the = operator in the next patch? A refcount < 1 in this path before your other change is a bug.
Re: [PATCH 1/2] ext4/xfs: add page refcount helper
Quoted 84 lines · Dan Williams
Quoted 83 lines · Ralph Campbell
There are several places where
ZONE_DEVICEstruct pages assume a reference count = 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.Signed-off-by Ralph Campbell <[email protected]>--- fs/dax.c | 8 ++++---- fs/ext4/inode.c | 2 +- fs/xfs/xfs_file.c | 2 +- include/linux/dax.h | 5 +++++ 4 files changed, 11 insertions(+), 6 deletions(-)fs/dax.c +4
-4@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -WARN_ON_ONCE (trunc &&page_ref_count (page) >1 ); +WARN_ON_ONCE (trunc && !dax_layout_is_idle_page (page));WARN_ON_ONCE (page->mapping && page->mapping != mapping); page->mapping =NULL ; page->index =0 ;@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -if (page_ref_count (page) >1 ) +if (!dax_layout_is_idle_page (page))return page; }return NULL ;@@ -560,11 +560,11 @@ static void *grab_mapping_entry(struct xa_state *xas,/** * dax_layout_busy_page - find first pinned page in @mapping -* @mapping: address space to scan for a page with ref count > 1 +* @mapping: address space to scan for a page with ref count > 0 * * DAX requires ZONE_DEVICE mapped pages. These pages are never * 'onlined' to the page allocator so they are considered idle when -* page->count = 1. A filesystem uses this interface to determine if +* page->count = 0. A filesystem uses this interface to determine if * any page in the mapping is busy, i.e. for DMA, or other * get_user_pages() usages. * fs/ext4/inode.c +1
-1@@ -3927,7 +3927,7 @@ int ext4_break_layouts(struct inode *inode)return 0 ; error =___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) =1 , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,ext4_wait_dax_page (ei)); }while (error =0 );fs/xfs/xfs_file.c +1
-1@@ -750,7 +750,7 @@ xfs_break_dax_layouts( *retry = true;return ___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) =1 ,TASK_INTERRUPTIBLE , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,xfs_wait_dax_page (inode)); }include/linux/dax.h +0
-0@@ -238,4 +238,9 @@ static inline bool dax_mapping(struct address_space *mapping)return mapping->host &&IS_DAX (mapping->host); }
Quoted 8 lines
include/linux/dax.h static inline bool dax_mapping(struct address_space *mapping) · line 243
Quoted 3 lines
include/linux/dax.h +3
-0+static inline bool dax_layout_is_idle_page (struct page *page) +{ +return page_ref_count (page) <=1 ;Why convert the check from "= 1" to "<= 1" and then back to the = operator in the next patch? A refcount < 1 in this path before your other change is a bug.
Mostly I was thinking > 1 was busy so <= 1 is idle. And yes, <=0 is never supposed to happen. Checking for = 1 is probably better though.
Re: [PATCH 1/2] ext4/xfs: add page refcount helper
In reply to Ralph CampbellQuoted 8 lines · Ralph Campbell
fs/ext4/inode.c +2
-1error =___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) =1 , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,ext4_wait_dax_page (ei)); +++ b/fs/xfs/xfs_file.c
fs/ext4/inode.c xfs_break_dax_layouts( · line 755
Quoted 7 lines
fs/ext4/inode.c +1
-1@@ -750,7 +750,7 @@ xfs_break_dax_layouts( *retry = true;return ___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) =1 ,TASK_INTERRUPTIBLE , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,xfs_wait_dax_page (inode)); }
I still think a litte helper macro would be nice here:
#define dax_wait_page(_inode, _page, _wait_cb) \
___wait_var_event(&(_page)->_refcount, \ atomic_read(&(_page)->_refcount) = 1, \ TASK_INTERRUPTIBLE, dax_layout_is_idle_page(_page), \ TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode));
Re: [PATCH 1/2] ext4/xfs: add page refcount helper
Quoted 19 lines · Christoph Hellwig
Quoted 8 lines · Ralph Campbell
error = ___wait_var_event(&page->_refcount,
fs/ext4/inode.c +2
-1-atomic_read (&page->_refcount) =1 , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 , ext4_wait_dax_page(ei)); +++ b/fs/xfs/xfs_file.cfs/ext4/inode.c xfs_break_dax_layouts( · line 755
Quoted 7 lines
fs/ext4/inode.c +1
-1@@ -750,7 +750,7 @@ xfs_break_dax_layouts( *retry = true;return ___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) =1 ,TASK_INTERRUPTIBLE , +dax_layout_is_idle_page (page),TASK_INTERRUPTIBLE ,0 ,0 ,xfs_wait_dax_page (inode)); }I still think a litte helper macro would be nice here:
Quoted 5 lines
#define dax_wait_page(_inode, _page, _wait_cb) \ ___wait_var_event(&(_page)->_refcount, \ atomic_read(&(_page)->_refcount) = 1, \ TASK_INTERRUPTIBLE, dax_layout_is_idle_page(_page), \ TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode));
OK, I'll add it in v3.
PATCH v2 2/2 mm: remove extra ZONE_DEVICE struct page refcount
In reply to Ralph CampbellZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- include/linux/dax.h | 2 +- include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 82 ++++++-------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 46 +++------------ 12 files changed, 44 insertions(+), 203 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) dpage =pfn_to_page (uvmem_pfn); dpage->zone_device_data = pvt; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage; out_clear:
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -1
@@ -323,7 +323,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)return NULL ; } -get_page (page); +init_page_count (page);lock_page (page);return page; }
include/linux/dax.h +1 -1
@@ -240,7 +240,7 @@ static inline bool dax_mapping(struct address_space *mapping)static inline bool dax_layout_is_idle_page (struct page *page) { -return page_ref_count (page) <=1 ; +return page_ref_count (page) ==0 ; }#endif
include/linux/memremap.h +4 -3
@@ -65,9 +65,10 @@ enum memory_type {struct dev_pagemap_ops {/* -* Called once the page refcount reaches 1. (ZONE_DEVICE pages never -* reach 0 refcount unless there is a refcount bug. This allows the -* device driver to implement its own memory management.) +* Called once the page refcount reaches 0. The reference count +* should be reset to one with init_page_count(page) before reusing +* the page. This allows the device driver to implement its own +* memory management. */ void (*page_free)(struct page *page);
include/linux/mm.h +0 -44
56 lines not on this page · raw message
lib/test_hmm.c +1 -1
@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) } dpage->zone_device_data = rpage; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage;
mm/gup.c +0 -44
56 lines not on this page · raw message
mm/internal.h +8 -0
@@ -620,4 +620,12 @@ struct migration_target_control { gfp_t gfp_mask; }; +#ifdef CONFIG_DEV_PAGEMAP_OPS +void free_zone_device_page (struct page *page); +#else +static inline void free_zone_device_page (struct page *page) +{ +} +#endif +#endif /* __MM_INTERNAL_H */
mm/memremap.c +18 -64
153 lines not on this page · raw message
mm/migrate.c +0 -5
@@ -380,11 +380,6 @@ static int expected_page_refs(struct address_space *mapping, struct page *page) {int expected_count =1 ; -/* -* Device public or private pages have an extra refcount as they are -* ZONE_DEVICE pages. -*/ - expected_count +=is_device_private_page (page);if (mapping) expected_count +=thp_nr_pages (page) +page_has_private (page);
mm/page_alloc.c +3 -0
@@ -6070,6 +6070,9 @@ void __ref memmap_init_zone_device(struct zone *zone,__init_single_page (page, pfn, zone_idx, nid); +/* ZONE_DEVICE pages start with a zero reference count. */ +set_page_count (page,0 ); +/* * Mark page reserved as it will need to wait for onlining * phase for it to be fully associated with a zone.
mm/swap.c +7 -39
75 lines not on this page · raw message
Signature · 1 line
2.20.1
Quoted 146 lines · Ralph Campbell
ZONE_DEVICEstruct pages have an extra reference count that complicates the code forput_page()and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially forZONE_DEVICE.Signed-off-by Ralph Campbell <[email protected]>--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- include/linux/dax.h | 2 +- include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 82 ++++++-------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 46 +++------------ 12 files changed, 44 insertions(+), 203 deletions(-)arch/powerpc/kvm/book3s_hv_uvmem.c +1
-1@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) dpage =pfn_to_page (uvmem_pfn); dpage->zone_device_data = pvt; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage; out_clear:drivers/gpu/drm/nouveau/nouveau_dmem.c +1
-1@@ -323,7 +323,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)return NULL ; } -get_page (page); +init_page_count (page);lock_page (page);return page; }include/linux/dax.h +1
-1@@ -240,7 +240,7 @@ static inline bool dax_mapping(struct address_space *mapping)static inline bool dax_layout_is_idle_page (struct page *page) { -return page_ref_count (page) <=1 ; +return page_ref_count (page) =0 ; }#endif include/linux/memremap.h +4
-3@@ -65,9 +65,10 @@ enum memory_type {struct dev_pagemap_ops {/* -* Called once the page refcount reaches 1. (ZONE_DEVICE pages never -* reach 0 refcount unless there is a refcount bug. This allows the -* device driver to implement its own memory management.) +* Called once the page refcount reaches 0. The reference count +* should be reset to one with init_page_count(page) before reusing +* the page. This allows the device driver to implement its own +* memory management. */ void (*page_free)(struct page *page);include/linux/mm.h +0
-44@@ -1092,39 +1092,6 @@ static inline bool is_zone_device_page(const struct page *page) }#endif -#ifdef CONFIG_DEV_PAGEMAP_OPS -void free_devmap_managed_page (struct page *page); -DECLARE_STATIC_KEY_FALSE (devmap_managed_key); - -static inline bool page_is_devmap_managed (struct page *page) -{ -if (!static_branch_unlikely (&devmap_managed_key)) -return false; -if (!is_zone_device_page (page)) -return false; -switch (page->pgmap->type) { -case MEMORY_DEVICE_PRIVATE : -case MEMORY_DEVICE_FS_DAX : -return true; -default : -break ; - } -return false; -} - -void put_devmap_managed_page (struct page *page); - -#else /* CONFIG_DEV_PAGEMAP_OPS */ -static inline bool page_is_devmap_managed (struct page *page) -{ -return false; -} - -static inline void put_devmap_managed_page (struct page *page) -{ -} -#endif /* CONFIG_DEV_PAGEMAP_OPS */ -static inline bool is_device_private_page (const struct page *page) { return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&@@ -1171,17 +1138,6 @@ static inline void put_page(struct page *page) { page =compound_head (page); -/* -* For devmap managed pages we need to catch refcount transition from -* 2 to 1, when refcount reach one it means the page is free and we -* need to inform the device driver through callback. See -* include/linux/memremap.h and HMM for details. -*/ -if (page_is_devmap_managed (page)) { -put_devmap_managed_page (page); -return ; - } -if (put_page_testzero (page))__put_page (page); }lib/test_hmm.c +0
-0
lib/test_hmm.c static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) · line 567
Quoted 8 lines
lib/test_hmm.c +1
-1@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) } dpage->zone_device_data = rpage; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage;
Doesn't test_hmm also need to reinitialize the refcount before freeing the page in hmm_dmirror_exit?
lib/test_hmm.c void *memremap_pages(struct dev_pagemap *pgmap, int nid) · line 173
Quoted 9 lines
lib/test_hmm.c +0
-1int error, is_ram; -bool need_devmap_managed = true;switch (pgmap->type) {case MEMORY_DEVICE_PRIVATE :@@ -217,11 +171,9 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) }break ;case MEMORY_DEVICE_GENERIC :
The MEMORY_DEVICE_PRIVATE cases loses the sanity check that the page_free method is set.
Otherwise this looks good.
Quoted 149 lines · Christoph Hellwig
Quoted 148 lines · Ralph Campbell
ZONE_DEVICEstruct pages have an extra reference count that complicates the code forput_page()and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially forZONE_DEVICE.Signed-off-by Ralph Campbell <[email protected]>--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- include/linux/dax.h | 2 +- include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 82 ++++++-------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 46 +++------------ 12 files changed, 44 insertions(+), 203 deletions(-)arch/powerpc/kvm/book3s_hv_uvmem.c +1
-1@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) dpage =pfn_to_page (uvmem_pfn); dpage->zone_device_data = pvt; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage; out_clear:drivers/gpu/drm/nouveau/nouveau_dmem.c +1
-1@@ -323,7 +323,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)return NULL ; } -get_page (page); +init_page_count (page);lock_page (page);return page; }include/linux/dax.h +1
-1@@ -240,7 +240,7 @@ static inline bool dax_mapping(struct address_space *mapping)static inline bool dax_layout_is_idle_page (struct page *page) { -return page_ref_count (page) <=1 ; +return page_ref_count (page) =0 ; }#endif include/linux/memremap.h +4
-3@@ -65,9 +65,10 @@ enum memory_type {struct dev_pagemap_ops {/* -* Called once the page refcount reaches 1. (ZONE_DEVICE pages never -* reach 0 refcount unless there is a refcount bug. This allows the -* device driver to implement its own memory management.) +* Called once the page refcount reaches 0. The reference count +* should be reset to one with init_page_count(page) before reusing +* the page. This allows the device driver to implement its own +* memory management. */ void (*page_free)(struct page *page);include/linux/mm.h +0
-44@@ -1092,39 +1092,6 @@ static inline bool is_zone_device_page(const struct page *page) }#endif -#ifdef CONFIG_DEV_PAGEMAP_OPS -void free_devmap_managed_page (struct page *page); -DECLARE_STATIC_KEY_FALSE (devmap_managed_key); - -static inline bool page_is_devmap_managed (struct page *page) -{ -if (!static_branch_unlikely (&devmap_managed_key)) -return false; -if (!is_zone_device_page (page)) -return false; -switch (page->pgmap->type) { -case MEMORY_DEVICE_PRIVATE : -case MEMORY_DEVICE_FS_DAX : -return true; -default : -break ; - } -return false; -} - -void put_devmap_managed_page (struct page *page); - -#else /* CONFIG_DEV_PAGEMAP_OPS */ -static inline bool page_is_devmap_managed (struct page *page) -{ -return false; -} - -static inline void put_devmap_managed_page (struct page *page) -{ -} -#endif /* CONFIG_DEV_PAGEMAP_OPS */ -static inline bool is_device_private_page (const struct page *page) { return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&@@ -1171,17 +1138,6 @@ static inline void put_page(struct page *page) { page =compound_head (page); -/* -* For devmap managed pages we need to catch refcount transition from -* 2 to 1, when refcount reach one it means the page is free and we -* need to inform the device driver through callback. See -* include/linux/memremap.h and HMM for details. -*/ -if (page_is_devmap_managed (page)) { -put_devmap_managed_page (page); -return ; - } -if (put_page_testzero (page))__put_page (page); }lib/test_hmm.c +0
-0@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) }
Quoted 9 lines
lib/test_hmm.c static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) · line 567
Quoted 6 lines
lib/test_hmm.c +1
-1dpage->zone_device_data = rpage; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage;Doesn't
test_hmmalso need to reinitialize the refcount before freeing the page inhmm_dmirror_exit?
The dmirror_zero_page is dead code, it isn't used. There is a patch queued in linux-mm which removes it. Besides, it was allocated with alloc_page() so it isn't a device private struct page.
Quoted 12 lines
lib/test_hmm.c void *memremap_pages(struct dev_pagemap *pgmap, int nid) · line 173
Quoted 9 lines
int error,
is_ram;lib/test_hmm.c +0
-1-bool need_devmap_managed = true;switch (pgmap->type) {case MEMORY_DEVICE_PRIVATE :@@ -217,11 +171,9 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) }break ;case MEMORY_DEVICE_GENERIC :The
MEMORY_DEVICE_PRIVATEcases loses the sanity check that thepage_freemethod is set.
I'll add that back into memremap_pages() with other sanity checks in v3.
Quoted 1 line
Otherwise this looks good.
Thanks!
Quoted 27 lines · Ralph Campbell
ZONE_DEVICEstruct pages have an extra reference count that complicates the code forput_page()and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially forZONE_DEVICE.Signed-off-by Ralph Campbell <[email protected]>--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- include/linux/dax.h | 2 +- include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 82 ++++++-------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 46 +++------------ 12 files changed, 44 insertions(+), 203 deletions(-)arch/powerpc/kvm/book3s_hv_uvmem.c +0
-0
arch/powerpc/kvm/book3s_hv_uvmem.c static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) · line 714
Quoted 5 lines
arch/powerpc/kvm/book3s_hv_uvmem.c +1
-1@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) dpage =pfn_to_page (uvmem_pfn); dpage->zone_device_data = pvt; -get_page (dpage); +init_page_count (dpage);
The powerpc change looks good. Passes a quick sanity test of booting/rebooting a secure guest on Power.
Regards,
Bharata.