mm: remove extra ZONE_DEVICE struct page refcount
Subsystems (17)
v3 cover mm: remove extra ZONE_DEVICE struct page refcount
This is still an RFC because after looking at the pmem/dax code some more, I realized that the ZONE_DEVICE struct pages are being inserted into the process' page tables with vmf_insert_mixed() and a zero refcount on the ZONE_DEVICE struct page. This is sort of OK because insert_pfn() increments the reference count on the pgmap which is what prevents memunmap_pages() from freeing the struct pages and it doesn't check for a non-zero struct page reference count.
But, any calls to get_page() will hit the VM_BUG_ON_PAGE() that checks for a reference count == 0.
// mmap() an ext4 file that is mounted -o dax.ext4_dax_fault()
ext4_dax_huge_fault()
dax_iomap_fault(&ext4_iomap_ops)
dax_iomap_pte_fault()
ops->iomap_begin() // ext4_iomap_begin()
ext4_map_blocks()
ext4_set_iomap()
dax_iomap_pfn()
dax_insert_entry()
vmf_insert_mixed(pfn)
__vm_insert_mixed()
if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL) &&
!pfn_t_devmap(pfn) && pfn_t_valid(pfn))
insert_page()
get_page(page) // XXX would trigger VM_BUG_ON_PAGE()
page_add_file_rmap()
set_pte_at()
else
insert_pfn()
pte_mkdevmap()
set_pte_at()Should pmem set the page reference count to one before inserting the pfn into the page tables (and decrement when removing devmap PTEs)? What about MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_PCI_P2PDMA use cases? Where should they icrement/decrement the page reference count? I don't know enough about how these are used to really know what to do at this point. If people want me to continue to work on this series, I will need some guidance.
---
Matthew Wilcox, Ira Weiny, and others have complained that ZONE_DEVICE struct page reference counting is ugly because they are "free" when the reference count is one instead of zero. This leads to explicit checks for ZONE_DEVICE pages in places like put_page(), GUP, THP splitting, and page migration which have to adjust the expected reference count when determining if the page is isolated or idle. This is my attempt to make ZONE_DEVICE pages be free when the reference count is zero and removing the special cases.
I'm only sending this out as a RFC since I'm not that familiar with the DAX, PMEM, XEN, and other uses of ZONE_DEVICE struct pages allocated with devm_memremap_pages() or memremap_pages() but my best reading of the code looks like it might be OK. I could use help testing these configurations.
I have been able to successfully run xfstests on ext4 with the memmap kernel boot option to simulate pmem.
Changes in v3:
Rebase to linux-mm 5.9.0-rc7-mm1.
Added a check for page_free() as suggested by Christoph Hellwig. Added a helper for dax_wait_page() as suggested by Christoph Hellwig.
Changes in v2:
One of the big changes in v2 is that devm_memremap_pages() and memremap_pages() now return the struct pages' reference count set to zero instead of one. Normally, get_page() will VM_BUG_ON_PAGE() if page->_refcount is zero. I didn't see any such warnings running the xfstests with dax/pmem but I'm not clear how the zero to one reference count is handled.
Other changes in v2:
Rebased to Linux-5.9.0-rc6 to include pmem fixes.
I added patch 1 to introduce a page refcount helper for ext4 and xfs as suggested by Christoph Hellwig.
I also applied Christoph Hellwig's other suggested changes for removing the devmap_managed_key, etc.
Ralph Campbell (2):
ext4/xfs: add page refcount helper
mm: remove extra ZONE_DEVICE struct page refcount
arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- fs/dax.c | 8 +-- fs/ext4/inode.c | 5 +- fs/xfs/xfs_file.c | 4 +- include/linux/dax.h | 10 +++ include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 84 +++++++------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 44 ++------------ 15 files changed, 63 insertions(+), 209 deletions(-)
Signature · 1 line
2.20.1
RFC PATCH v3 1/2 ext4/xfs: add page refcount helper
There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.
--- fs/dax.c | 4 ++-- fs/ext4/inode.c | 5 +---- fs/xfs/xfs_file.c | 4 +--- include/linux/dax.h | 10 ++++++++++ 4 files changed, 14 insertions(+), 9 deletions(-)
fs/dax.c +2 -2
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -WARN_ON_ONCE (trunc &&page_ref_count (page) >1 ); +WARN_ON_ONCE (trunc && !dax_layout_is_idle_page (page));WARN_ON_ONCE (page->mapping && page->mapping != mapping); page->mapping =NULL ; page->index =0 ;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)for_each_mapped_pfn (entry, pfn) {struct page *page =pfn_to_page (pfn); -if (page_ref_count (page) >1 ) +if (!dax_layout_is_idle_page (page))return page; }return NULL ;
fs/ext4/inode.c +1 -4
@@ -3926,10 +3926,7 @@ int ext4_break_layouts(struct inode *inode)if (!page)return 0 ; - error =___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) ==1 , -TASK_INTERRUPTIBLE ,0 ,0 , -ext4_wait_dax_page (ei)); + error =dax_wait_page (ei, page, ext4_wait_dax_page); }while (error ==0 );return error;
fs/xfs/xfs_file.c +1 -3
@@ -749,9 +749,7 @@ xfs_break_dax_layouts(return 0 ; *retry = true; -return ___wait_var_event (&page->_refcount, -atomic_read (&page->_refcount) ==1 ,TASK_INTERRUPTIBLE , -0 ,0 ,xfs_wait_dax_page (inode)); +return dax_wait_page (inode, page, xfs_wait_dax_page); }int
include/linux/dax.h +10 -0
@@ -243,6 +243,16 @@ static inline bool dax_mapping(struct address_space *mapping)return mapping->host &&IS_DAX (mapping->host); } +static inline bool dax_layout_is_idle_page (struct page *page) +{ +return page_ref_count (page) ==1 ; +} + +#define dax_wait_page (_inode, _page, _wait_cb) \ + ___wait_var_event(&(_page)->_refcount, \ + dax_layout_is_idle_page(_page), \ + TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode)) +#ifdef CONFIG_DEV_DAX_HMEM_DEVICES void hmem_register_device (int target_nid,struct resource *r);#else
Signature · 1 line
2.20.1
Re: [RFC PATCH v3 1/2] ext4/xfs: add page refcount helper
Quoted 5 lines · Ralph Campbell
There are several places where
ZONE_DEVICEstruct pages assume a reference count = 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.Signed-off-by Ralph Campbell <[email protected]>
Looks good:
Re: [RFC PATCH v3 1/2] ext4/xfs: add page refcount helper
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Ralph Campbell
There are several places where
ZONE_DEVICEstruct pages assume a reference count = 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.Signed-off-by Ralph Campbell <[email protected]>Looks good:
Reviewed-by Christoph Hellwig <[email protected]>
Thanks for the review.
I'll resend this as an independent patch.
RFC PATCH v3 2/2 mm: remove extra ZONE_DEVICE struct page refcount
In reply to Ralph CampbellZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.
--- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- fs/dax.c | 4 +- include/linux/dax.h | 2 +- include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 84 +++++++------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 44 ++------------ 13 files changed, 50 insertions(+), 201 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) dpage =pfn_to_page (uvmem_pfn); dpage->zone_device_data = pvt; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage; out_clear:
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -1
@@ -324,7 +324,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)return NULL ; } -get_page (page); +init_page_count (page);lock_page (page);return page; }
fs/dax.c +2 -2
@@ -560,14 +560,14 @@ static void *grab_mapping_entry(struct xa_state *xas,/** * dax_layout_busy_page_range - find first pinned page in @mapping -* @mapping: address space to scan for a page with ref count > 1 +* @mapping: address space to scan for a page with ref count > 0 * @start: Starting offset. Page containing 'start' is included. * @end: End offset. Page containing 'end' is included. If 'end' is LLONG_MAX, * pages from 'start' till the end of file are included. * * DAX requires ZONE_DEVICE mapped pages. These pages are never * 'onlined' to the page allocator so they are considered idle when -* page->count == 1. A filesystem uses this interface to determine if +* page->count == 0. A filesystem uses this interface to determine if * any page in the mapping is busy, i.e. for DMA, or other * get_user_pages() usages. *
include/linux/dax.h +1 -1
@@ -245,7 +245,7 @@ static inline bool dax_mapping(struct address_space *mapping)static inline bool dax_layout_is_idle_page (struct page *page) { -return page_ref_count (page) ==1 ; +return page_ref_count (page) ==0 ; }#define dax_wait_page (_inode, _page, _wait_cb) \
include/linux/memremap.h +4 -3
@@ -66,9 +66,10 @@ enum memory_type {struct dev_pagemap_ops {/* -* Called once the page refcount reaches 1. (ZONE_DEVICE pages never -* reach 0 refcount unless there is a refcount bug. This allows the -* device driver to implement its own memory management.) +* Called once the page refcount reaches 0. The reference count +* should be reset to one with init_page_count(page) before reusing +* the page. This allows the device driver to implement its own +* memory management. */ void (*page_free)(struct page *page);
include/linux/mm.h +0 -44
56 lines not on this page · raw message
lib/test_hmm.c +1 -1
@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) } dpage->zone_device_data = rpage; -get_page (dpage); +init_page_count (dpage);lock_page (dpage);return dpage;
mm/gup.c +0 -44
56 lines not on this page · raw message
mm/internal.h +8 -0
@@ -618,4 +618,12 @@ struct migration_target_control { gfp_t gfp_mask; }; +#ifdef CONFIG_DEV_PAGEMAP_OPS +void free_zone_device_page (struct page *page); +#else +static inline void free_zone_device_page (struct page *page) +{ +} +#endif +#endif /* __MM_INTERNAL_H */
mm/memremap.c +23 -61
163 lines not on this page · raw message
mm/migrate.c +0 -5
@@ -380,11 +380,6 @@ static int expected_page_refs(struct address_space *mapping, struct page *page) {int expected_count =1 ; -/* -* Device private pages have an extra refcount as they are -* ZONE_DEVICE pages. -*/ - expected_count +=is_device_private_page (page);if (mapping) expected_count +=thp_nr_pages (page) +page_has_private (page);
mm/page_alloc.c +3 -0
@@ -6154,6 +6154,9 @@ void __ref memmap_init_zone_device(struct zone *zone,__init_single_page (page, pfn, zone_idx, nid); +/* ZONE_DEVICE pages start with a zero reference count. */ +set_page_count (page,0 ); +/* * Mark page reserved as it will need to wait for onlining * phase for it to be fully associated with a zone.
mm/swap.c +6 -38
70 lines not on this page · raw message
Signature · 1 line
2.20.1
Quoted 7 lines · Ralph Campbell
ZONE_DEVICEstruct pages have an extra reference count that complicates the code forput_page()and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially forZONE_DEVICE.Signed-off-by Ralph Campbell <[email protected]>
Looks good,
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Ralph Campbell
ZONE_DEVICEstruct pages have an extra reference count that complicates the code forput_page()and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially forZONE_DEVICE.
Quoted 5 lines
Quoted 1 line
Signed-off-by Ralph Campbell <[email protected]>Looks good,
Reviewed-by Christoph Hellwig <[email protected]>
Thanks for the review.
I still have reservations about making this an official patch. Did you see the updated cover letter?
Basically, I'm concerned about ZONE_DEVICE struct pages being inserted into the process page table with a zero reference count with vmf_insert_mixed(). If it is to be a non-zero reference count, then DAX, pmem, and other uses of ZONE_DEVICE pages need to be changed (or vmf_insert_mixed()) to inc/dec in appropriate places but I don't feel I know that code well enough to make those changes.
Quoted 5 lines · Ralph Campbell
ZONE_DEVICEstruct pages have an extra reference count that complicates the code forput_page()and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially forZONE_DEVICE.
I was hoping this patch would resolve a page-reference issue that we run into at random times while migrating a page to a device page backed by secure-memory.
Unfortunately I continue to see the problem. There is a reference held on that page, which fails the migration.
FYI
RP
Quoted 12 lines · Ram Pai
Quoted 5 lines · Ralph Campbell
ZONE_DEVICEstruct pages have an extra reference count that complicates the code forput_page()and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially forZONE_DEVICE.I was hoping this patch would resolve a page-reference issue that we run into at random times while migrating a page to a device page backed by secure-memory.
Quoted 5 lines
Unfortunately I continue to see the problem. There is a reference held on that page, which fails the migration.
FYI
RP
I'm willing to look into it but I would need more information. Can you give any more details about the conditions when it happens? It would be great if you have a program that reproduces the problem.
Matthew Wilcox
reviewerQuoted 9 lines · Ralph Campbell
This is still an RFC because after looking at the pmem/dax code some more, I realized that the
ZONE_DEVICEstruct pages are being inserted into the process' page tables withvmf_insert_mixed()and a zero refcount on theZONE_DEVICEstruct page. This is sort of OK becauseinsert_pfn()increments the reference count on the pgmap which is what preventsmemunmap_pages()from freeing the struct pages and it doesn't check for a non-zero struct page reference count.
But, any calls toget_page()will hit theVM_BUG_ON_PAGE()that checks for a reference count = 0.
And presumably get_user_pages() would call get_page()? Or is that gated off somewhere?
Looking at this from a design standpoint instead of getting into the code, I would say that a struct page which cannot be referenced from anywhere should have a reference count of 0. As soon as it can be found by something, it should have a positive refcount.
For DAX, I think that means it should have its refcount raised to 1 immediately before a DAX entry for it is put in the page cache. It should have its refcount decreased when its DAX entry is removed from the page cache.
That should ensure that we can't put a 0-ref DAX page in the page tables, and thus it can't be found by get_user_pages().
Quoted 95 lines
//
mmap()an ext4 file that is mounted -o dax.ext4_dax_fault()
ext4_dax_huge_fault()dax_iomap_fault(&ext4_iomap_ops) dax_iomap_pte_fault() ops->iomap_begin() // ext4_iomap_begin() ext4_map_blocks() ext4_set_iomap() dax_iomap_pfn() dax_insert_entry() vmf_insert_mixed(pfn) __vm_insert_mixed() if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL) && !pfn_t_devmap(pfn) && pfn_t_valid(pfn)) insert_page() get_page(page) // XXX would trigger VM_BUG_ON_PAGE() page_add_file_rmap() set_pte_at() else insert_pfn() pte_mkdevmap() set_pte_at()Should pmem set the page reference count to one before inserting the pfn into the page tables (and decrement when removing devmap PTEs)? What about
MEMORY_DEVICE_GENERICandMEMORY_DEVICE_PCI_P2PDMAuse cases? Where should they icrement/decrement the page reference count? I don't know enough about how these are used to really know what to do at this point. If people want me to continue to work on this series, I will need some guidance.---
Matthew Wilcox, Ira Weiny, and others have complained that
ZONE_DEVICEstruct page reference counting is ugly because they are "free" when the reference count is one instead of zero. This leads to explicit checks forZONE_DEVICEpages in places likeput_page(), GUP, THP splitting, and page migration which have to adjust the expected reference count when determining if the page is isolated or idle. This is my attempt to makeZONE_DEVICEpages be free when the reference count is zero and removing the special cases.I'm only sending this out as a RFC since I'm not that familiar with the DAX, PMEM, XEN, and other uses of
ZONE_DEVICEstruct pages allocated withdevm_memremap_pages()ormemremap_pages()but my best reading of the code looks like it might be OK. I could use help testing these configurations.
I have been able to successfully run xfstests on ext4 with the memmap kernel boot option to simulate pmem.Changes in v3:
Rebase to linux-mm 5.9.0-rc7-mm1.
Added a check forpage_free()as suggested by Christoph Hellwig. Added a helper fordax_wait_page()as suggested by Christoph Hellwig.Changes in v2:
One of the big changes in v2 is thatdevm_memremap_pages()andmemremap_pages()now return the struct pages' reference count set to zero instead of one. Normally,get_page()willVM_BUG_ON_PAGE()ifpage->_refcountis zero. I didn't see any such warnings running the xfstests with dax/pmem but I'm not clear how the zero to one reference count is handled.Other changes in v2:
Rebased to Linux-5.9.0-rc6 to include pmem fixes.
I added patch 1 to introduce a page refcount helper for ext4 and xfs as suggested by Christoph Hellwig.
I also applied Christoph Hellwig's other suggested changes for removing thedevmap_managed_key, etc.Ralph Campbell (2):
ext4/xfs: add page refcount helper
mm: remove extraZONE_DEVICEstruct page refcountarch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- fs/dax.c | 8 +-- fs/ext4/inode.c | 5 +- fs/xfs/xfs_file.c | 4 +- include/linux/dax.h | 10 +++ include/linux/memremap.h | 7 ++- include/linux/mm.h | 44 -------------- lib/test_hmm.c | 2 +- mm/gup.c | 44 -------------- mm/internal.h | 8 +++ mm/memremap.c | 84 +++++++------------------- mm/migrate.c | 5 -- mm/page_alloc.c | 3 + mm/swap.c | 44 ++------------ 15 files changed, 63 insertions(+), 209 deletions(-)Signature · 1 line
2.20.1
Quoted 34 lines · Ralph Campbell
This is still an RFC because after looking at the pmem/dax code some more, I realized that the
ZONE_DEVICEstruct pages are being inserted into the process' page tables withvmf_insert_mixed()and a zero refcount on theZONE_DEVICEstruct page. This is sort of OK becauseinsert_pfn()increments the reference count on the pgmap which is what preventsmemunmap_pages()from freeing the struct pages and it doesn't check for a non-zero struct page reference count.
But, any calls toget_page()will hit theVM_BUG_ON_PAGE()that checks for a reference count = 0.//
mmap()an ext4 file that is mounted -o dax.ext4_dax_fault()
ext4_dax_huge_fault()dax_iomap_fault(&ext4_iomap_ops) dax_iomap_pte_fault() ops->iomap_begin() // ext4_iomap_begin() ext4_map_blocks() ext4_set_iomap() dax_iomap_pfn() dax_insert_entry() vmf_insert_mixed(pfn) __vm_insert_mixed() if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL) && !pfn_t_devmap(pfn) && pfn_t_valid(pfn)) insert_page() get_page(page) // XXX would trigger VM_BUG_ON_PAGE() page_add_file_rmap() set_pte_at() else insert_pfn() pte_mkdevmap() set_pte_at()
Quoted 7 lines
Should pmem set the page reference count to one before inserting the pfn into the page tables (and decrement when removing devmap PTEs)? What about
MEMORY_DEVICE_GENERICandMEMORY_DEVICE_PCI_P2PDMAuse cases? Where should they icrement/decrement the page reference count? I don't know enough about how these are used to really know what to do at this point. If people want me to continue to work on this series, I will need some guidance.
fs/dax could take the reference when inserting, but that would mean that ext4 and xfs would need to go back to checking for 1 to be page idle. I think that's ok because the filesystem is actually not checking for page-idle it's checking for "get_user_pages()" idle.
Matthew Wilcox
reviewerQuoted 9 lines · Ralph Campbell
This is still an RFC because after looking at the pmem/dax code some more, I realized that the
ZONE_DEVICEstruct pages are being inserted into the process' page tables withvmf_insert_mixed()and a zero refcount on theZONE_DEVICEstruct page. This is sort of OK becauseinsert_pfn()increments the reference count on the pgmap which is what preventsmemunmap_pages()from freeing the struct pages and it doesn't check for a non-zero struct page reference count.
But, any calls toget_page()will hit theVM_BUG_ON_PAGE()that checks for a reference count = 0.
This seems to have gone quiet. What needs to happen to resurrect this?
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · Ralph Campbell
This is still an RFC because after looking at the pmem/dax code some more, I realized that the
ZONE_DEVICEstruct pages are being inserted into the process' page tables withvmf_insert_mixed()and a zero refcount on theZONE_DEVICEstruct page. This is sort of OK becauseinsert_pfn()increments the reference count on the pgmap which is what preventsmemunmap_pages()from freeing the struct pages and it doesn't check for a non-zero struct page reference count.
But, any calls toget_page()will hit theVM_BUG_ON_PAGE()that checks for a reference count = 0.
Quoted 2 lines
This seems to have gone quiet. What needs to happen to resurrect this?
The main thing I need is time. I have been tied up with other commitments, there has been a lot of changes going on in FS/DAX and the page cache, and FS/DAX doesn't use the page reference count to indicate the page is "free" but rather that it is "idle" so I need a lot of time to really understand why FS/DAX isn't just any FS on top of a DAX block device.
I too wish this was easier to fix.