mm/smaps: Fixes and optimizations on shmem swap handling
cover mm/smaps: Fixes and optimizations on shmem swap handling
This series grows from the patch previously posted here:
[PATCH] mm/smaps: Use vma->vm_pgoff directly when counting partial swap
https://lore.kernel.org/lkml/[email protected]/
Vlastimil reported a bug that is even more important to fix than the cleanup, so I put it as patch 1 here. There's a test program we can use to verify the bug before/after the patch. I used the same program to test patch 2/3 because it covers walking shmem swap both in page cache and in pgtables.
Patch 2 is the original patch, though with a tiny touchup as Vlastimil suggested.
Patch 3 is a further cleanup of the shmem swap logic, hopefully make it even cleaner.
Please review, thanks.
Peter Xu (3):
mm/smaps: Fix shmem pte hole swap calculation mm/smaps: Use vma->vm_pgoff directly when counting partial swap mm/smaps: Simplify shmem handling of pte holes
fs/proc/task_mmu.c | 28 ++++++++++++++++------------ mm/shmem.c | 5 ++--- 2 files changed, 18 insertions(+), 15 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/3 mm/smaps: Fix shmem pte hole swap calculation
The shmem swap calculation on the privately writable mappings are using wrong parameters as spotted by Vlastimil. Fix them. That's introduced in commit 48131e03ca4e, when rework shmem_swap_usage to shmem_partial_swap_usage.
Test program:
==================
void main(void)
{
char *buffer, *p;
int i, fd;
fd = memfd_create("test", 0);
assert(fd > 0);
/* isize==2M*3, fill in pages, swap them out */
ftruncate(fd, SIZE_2M * 3);
buffer = mmap(NULL, SIZE_2M * 3, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
assert(buffer);
for (i = 0, p = buffer; i < SIZE_2M * 3 / 4096; i++) {
*p = 1;
p += 4096;
}
madvise(buffer, SIZE_2M * 3, MADV_PAGEOUT);
munmap(buffer, SIZE_2M * 3);
/*
* Remap with private+writtable mappings on partial of the inode (<= 2M*3),
* while the size must also be >= 2M*2 to make sure there's a none pmd so
* smaps_pte_hole will be triggered.
*/
buffer = mmap(NULL, SIZE_2M * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
printf("pid=%d, buffer=%p\n", getpid(), buffer);
/* Check /proc/$PID/smap_rollup, should see 4MB swap */
sleep(1000000);
}==================
Before the patch, smaps_rollup shows <4MB swap and the number will be random depending on the alignment of the buffer of mmap() allocated. After this patch, it'll show 4MB.
--- fs/proc/task_mmu.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fs/proc/task_mmu.c +4 -2
@@ -478,9 +478,11 @@ static int smaps_pte_hole(unsigned long addr, unsigned long end, __always_unused int depth, struct mm_walk *walk) { struct mem_size_stats *mss = walk->private; +struct vm_area_struct *vma = walk->vma; - mss->swap +=shmem_partial_swap_usage ( - walk->vma->vm_file->f_mapping, addr, end); + mss->swap +=shmem_partial_swap_usage (walk->vma->vm_file->f_mapping, +linear_page_index (vma, addr), +linear_page_index (vma, end));return 0 ; }
Signature · 1 line
2.31.1
Re: [PATCH 1/3] mm/smaps: Fix shmem pte hole swap calculation
Quoted 38 lines · Peter Xu
The shmem swap calculation on the privately writable mappings are using wrong parameters as spotted by Vlastimil. Fix them. That's introduced in commit
48131e03ca4e, when reworkshmem_swap_usagetoshmem_partial_swap_usage.Test program:
==================
void main(void)
{ char *buffer, *p; int i, fd; fd = memfd_create("test", 0); assert(fd > 0); /* isize==2M*3, fill in pages, swap them out */ ftruncate(fd, SIZE_2M * 3); buffer = mmap(NULL, SIZE_2M * 3, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); assert(buffer); for (i = 0, p = buffer; i < SIZE_2M * 3 / 4096; i++) { *p = 1; p += 4096; } madvise(buffer, SIZE_2M * 3, MADV_PAGEOUT); munmap(buffer, SIZE_2M * 3); /* * Remap with private+writtable mappings on partial of the inode (<= 2M*3), * while the size must also be >= 2M*2 to make sure there's a none pmd so * smaps_pte_hole will be triggered. */ buffer = mmap(NULL, SIZE_2M * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); printf("pid=%d, buffer=%p\n", getpid(), buffer); /* Check /proc/$PID/smap_rollup, should see 4MB swap */ sleep(1000000); }
Quoted 7 lines
==================
Before the patch,
smaps_rollupshows <4MB swap and the number will be random depending on the alignment of the buffer ofmmap()allocated. After this patch, it'll show 4MB.Fixes 48131e03ca4e ("mm, proc: reduce cost of /proc/pid/smaps for unpopulated shmem mappings")
Thanks, too bad I didn't spot it when sending that patch :)
Quoted 2 lines
Reported-by Vlastimil Babka <[email protected]>Signed-off-by Peter Xu <[email protected]>
Quoted 23 lines
--- fs/proc/task_mmu.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)fs/proc/task_mmu.c +4
-2@@ -478,9 +478,11 @@ static int smaps_pte_hole(unsigned long addr, unsigned long end, __always_unused int depth, struct mm_walk *walk) { struct mem_size_stats *mss = walk->private; +struct vm_area_struct *vma = walk->vma; - mss->swap +=shmem_partial_swap_usage ( - walk->vma->vm_file->f_mapping, addr, end); + mss->swap +=shmem_partial_swap_usage (walk->vma->vm_file->f_mapping, +linear_page_index (vma, addr), +linear_page_index (vma, end));return 0 ; }
PATCH 2/3 mm/smaps: Use vma->vm_pgoff directly when counting partial swap
In reply to Peter XuAs it's trying to cover the whole vma anyways, use direct vm_pgoff value and vma_pages() rather than linear_page_index.
--- mm/shmem.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
mm/shmem.c +2 -3
@@ -856,9 +856,8 @@ unsigned long shmem_swap_usage(struct vm_area_struct *vma)return swapped <<PAGE_SHIFT ;/* Here comes the more involved part */ -return shmem_partial_swap_usage (mapping, -linear_page_index (vma, vma->vm_start), -linear_page_index (vma, vma->vm_end)); +return shmem_partial_swap_usage (mapping, vma->vm_pgoff, + vma->vm_pgoff +vma_pages (vma)); }/*
Signature · 1 line
2.31.1
Re: [PATCH 2/3] mm/smaps: Use vma->vm_pgoff directly when counting partial swap
Quoted 6 lines · Peter Xu
As it's trying to cover the whole vma anyways, use direct
vm_pgoffvalue andvma_pages()rather thanlinear_page_index.Cc Vlastimil Babka <[email protected]>Cc Hugh Dickins <[email protected]>Signed-off-by Peter Xu <[email protected]>
Quoted 21 lines
--- mm/shmem.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)mm/shmem.c +2
-3@@ -856,9 +856,8 @@ unsigned long shmem_swap_usage(struct vm_area_struct *vma)return swapped <<PAGE_SHIFT ;/* Here comes the more involved part */ -return shmem_partial_swap_usage (mapping, -linear_page_index (vma, vma->vm_start), -linear_page_index (vma, vma->vm_end)); +return shmem_partial_swap_usage (mapping, vma->vm_pgoff, + vma->vm_pgoff +vma_pages (vma)); }/*
PATCH 3/3 mm/smaps: Simplify shmem handling of pte holes
In reply to Peter XuFirstly, check_shmem_swap variable is actually not necessary, because it's always set with pte_hole hook; checking each would work.
Meanwhile, the check within smaps_pte_entry is not easy to follow. E.g., pte_none() check is not needed as "!pte_present && !is_swap_pte" is the same. Since at it, use the pte_hole() helper rather than dup the page cache lookup.
Still keep the CONFIG_SHMEM part so the code can be optimized to nop for !SHMEM.
There will be a very slight functional change in smaps_pte_entry(), that for !SHMEM we'll return early for pte_none (before checking page==NULL), but that's even nicer.
--- fs/proc/task_mmu.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
fs/proc/task_mmu.c +12 -10
@@ -397,7 +397,6 @@ struct mem_size_stats {u64 pss_shmem;u64 pss_locked;u64 swap_pss; -bool check_shmem_swap; }; static void smaps_page_accumulate(struct mem_size_stats *mss,
@@ -490,6 +489,16 @@ static int smaps_pte_hole(unsigned long addr, unsigned long end,#define smaps_pte_hole NULL#endif /* CONFIG_SHMEM */ +static void smaps_pte_hole_lookup (unsigned long addr,struct mm_walk *walk) +{ +#ifdef CONFIG_SHMEM +if (walk->ops->pte_hole) { +/* depth is not used */ +smaps_pte_hole (addr, addr +PAGE_SIZE ,0 , walk); + } +#endif +} +static void smaps_pte_entry (pte_t *pte,unsigned long addr,struct mm_walk *walk) {
@@ -518,12 +527,8 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, } }else if (is_pfn_swap_entry (swpent)) page =pfn_swap_entry_to_page (swpent); - }else if (unlikely (IS_ENABLED (CONFIG_SHMEM ) && mss->check_shmem_swap - &&pte_none (*pte))) { - page =xa_load (&vma->vm_file->f_mapping->i_pages, -linear_page_index (vma, addr)); -if (xa_is_value (page)) - mss->swap +=PAGE_SIZE ; + }else { +smaps_pte_hole_lookup (addr, walk);return ; }
@@ -737,8 +742,6 @@ static void smap_gather_stats(struct vm_area_struct *vma,return ;#ifdef CONFIG_SHMEM -/* In case of smaps_rollup, reset the value from previous vma */ - mss->check_shmem_swap = false;if (vma->vm_file &&shmem_mapping (vma->vm_file->f_mapping)) {/* * For shared or readonly shmem mappings we know that all
@@ -756,7 +759,6 @@ static void smap_gather_stats(struct vm_area_struct *vma, !(vma->vm_flags & VM_WRITE))) { mss->swap += shmem_swapped; }else { - mss->check_shmem_swap = true; ops = &smaps_shmem_walk_ops; } }
Signature · 1 line
2.31.1
Re: [PATCH 3/3] mm/smaps: Simplify shmem handling of pte holes
Quoted 2 lines · Peter Xu
Firstly,
check_shmem_swapvariable is actually not necessary, because it's always set withpte_holehook; checking each would work.
Right...
Quoted 2 lines
Meanwhile, the check within
smaps_pte_entryis not easy to follow. E.g.,pte_none()check is not needed as "!pte_present && !is_swap_pte" is the same.
Seems to be true, indeed.
Quoted 1 line
Since at it, use the
pte_hole()helper rather than dup the page cache lookup.
pte_hole() is for checking a range and we are calling it for single page, isnt't that causing larger overhead in the end? There's xarray involved, so maybe Matthew will know best.
Quoted 5 lines
Still keep the
CONFIG_SHMEMpart so the code can be optimized to nop for !SHMEM.There will be a very slight functional change in
smaps_pte_entry(), that for !SHMEM we'll return early forpte_none(before checking page==NULL), but that's even nicer.
I don't think this is true, 'unlikely(IS_ENABLED(CONFIG_SHMEM))' will be a compile-time constant false and shortcut the rest of the 'if' evaluation thus there will be no page check? Or I misunderstood.
Quoted 70 lines
Cc Hugh Dickins <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Vlastimil Babka <[email protected]>Signed-off-by Peter Xu <[email protected]>--- fs/proc/task_mmu.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)fs/proc/task_mmu.c +12
-10@@ -397,7 +397,6 @@ struct mem_size_stats {u64 pss_shmem;u64 pss_locked;u64 swap_pss; -bool check_shmem_swap; }; static void smaps_page_accumulate(struct mem_size_stats *mss,@@ -490,6 +489,16 @@ static int smaps_pte_hole(unsigned long addr, unsigned long end,#define smaps_pte_hole NULL#endif /* CONFIG_SHMEM */ +static void smaps_pte_hole_lookup (unsigned long addr,struct mm_walk *walk) +{ +#ifdef CONFIG_SHMEM +if (walk->ops->pte_hole) { +/* depth is not used */ +smaps_pte_hole (addr, addr +PAGE_SIZE ,0 , walk); + } +#endif +} +static void smaps_pte_entry (pte_t *pte,unsigned long addr,struct mm_walk *walk) {@@ -518,12 +527,8 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, } }else if (is_pfn_swap_entry (swpent)) page =pfn_swap_entry_to_page (swpent); - }else if (unlikely (IS_ENABLED (CONFIG_SHMEM ) && mss->check_shmem_swap - &&pte_none (*pte))) { - page =xa_load (&vma->vm_file->f_mapping->i_pages, -linear_page_index (vma, addr)); -if (xa_is_value (page)) - mss->swap +=PAGE_SIZE ; + }else { +smaps_pte_hole_lookup (addr, walk);return ; }@@ -737,8 +742,6 @@ static void smap_gather_stats(struct vm_area_struct *vma,return ;#ifdef CONFIG_SHMEM -/* In case of smaps_rollup, reset the value from previous vma */ - mss->check_shmem_swap = false;if (vma->vm_file &&shmem_mapping (vma->vm_file->f_mapping)) {/* * For shared or readonly shmem mappings we know that all @@ -756,7 +759,6 @@ static void smap_gather_stats(struct vm_area_struct *vma, !(vma->vm_flags & VM_WRITE))) { mss->swap += shmem_swapped; }else { - mss->check_shmem_swap = true; ops = &smaps_shmem_walk_ops; } }
Re: [PATCH 3/3] mm/smaps: Simplify shmem handling of pte holes
Hi, Vlastimil,
Quoted 5 lines · Vlastimil Babka
Quoted 1 line
Since at it, use the
pte_hole()helper rather than dup the page cache lookup.
pte_hole()is for checking a range and we are calling it for single page, isnt't that causing larger overhead in the end? There's xarray involved, so maybe Matthew will know best.
Per my understanding, pte_hole() calls xas_load() too at last, just like the old code; it's just that the xas_for_each() of shmem_partial_swap_usage() will only run one iteration, iiuc.
Quoted 10 lines
Quoted 6 lines
Still keep the
CONFIG_SHMEMpart so the code can be optimized to nop for !SHMEM.There will be a very slight functional change in
smaps_pte_entry(), that for !SHMEM we'll return early forpte_none(before checking page==NULL), but that's even nicer.I don't think this is true, 'unlikely(IS_ENABLED(CONFIG_SHMEM))' will be a compile-time constant false and shortcut the rest of the 'if' evaluation thus there will be no page check? Or I misunderstood.
The page check I was referring is this one in smaps_pte_entry():
if (!page) return;
After the change, with !SHMEM the "else" block will be kept there (unlike the old code as you mentioned it'll be optimized), the smaps_pte_hole_lookup() will be noop so it'll be a direct "return" in that "else", then it should return a bit earlier by not checking "!page" (because in that case pte_none must have page==NULL).
Thanks,
Signature · 1 line
Peter Xu