mm, dump_page: do not crash with bad compound_mapcount()
Subsystems (6)
PATCH v2 1/1 mm, dump_page: do not crash with bad compound_mapcount()
If a compound page is being split while dump_page() is being run on that page, we can end up calling compound_mapcount() on a page that is no longer compound. This leads to a crash (already seen at least once in the field), due to the VM_BUG_ON_PAGE() assertion inside compound_mapcount().
(The above is from Matthew Wilcox's analysis of Qian Cai's bug report.)
A similar problem is possible, via compound_pincount() instead of compound_mapcount().
In order to avoid this kind of crash, make dump_page() slightly more robust, by providing a pair of simpler routines that don't contain
assertions: head_mapcount() and head_pincount().
For debug tools, we don't want to go *too* far in this direction, but this is a simple small fix, and the crash has already been seen, so it's a good trade-off.
---
Hi,
I'm assuming that a fix is not required for -stable, but let me know if others feel differently. The dump_page() code has changed a lot in that area.
Changes since v1 [1]:
1) Rebased onto mmotm
2) Used a simpler head_*count() approach.
3) Added Matthew's Suggested-by: tag
4) Support pincount as well as mapcount.
[1] https://lore.kernel.org/linux-mm/[email protected]/
thanks,
John Hubbard
include/linux/mm.h | 14 ++++++++++++-- mm/debug.c | 6 +++--- 2 files changed, 15 insertions(+), 5 deletions(-)
include/linux/mm.h +12 -2
@@ -776,6 +776,11 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)extern void kvfree (const void *addr);extern void kvfree_sensitive (const void *addr,size_t len); +static inline int head_mapcount (struct page *head) +{ +return atomic_read (compound_mapcount_ptr (head)) +1 ; +} +/* * Mapcount of compound page as a whole, does not include mapped sub-pages. *
@@ -785,7 +790,7 @@ static inline int compound_mapcount(struct page *page) {VM_BUG_ON_PAGE (!PageCompound (page), page); page =compound_head (page); -return atomic_read (compound_mapcount_ptr (page)) +1 ; +return head_mapcount (page); }/*
@@ -898,11 +903,16 @@ static inline bool hpage_pincount_available(struct page *page)return PageCompound (page) &&compound_order (page) >1 ; } +static inline int head_pincount (struct page *head) +{ +return atomic_read (compound_pincount_ptr (head)); +} +static inline int compound_pincount (struct page *page) {VM_BUG_ON_PAGE (!hpage_pincount_available (page), page); page =compound_head (page); -return atomic_read (compound_pincount_ptr (page)); +return head_pincount (page); }static inline void set_compound_order (struct page *page,unsigned int order)
mm/debug.c +3 -3
@@ -102,12 +102,12 @@ void __dump_page(struct page *page, const char *reason)if (hpage_pincount_available (page)) {pr_warn ("head:%p order:%u compound_mapcount:%d compound_pincount:%d\n" , head,compound_order (head), -compound_mapcount (head), -compound_pincount (head)); +head_mapcount (head), +head_pincount (head)); }else {pr_warn ("head:%p order:%u compound_mapcount:%d\n" , head,compound_order (head), -compound_mapcount (head)); +head_mapcount (head)); } }if (PageKsm (page))
Signature · 1 line
2.28.0
Quoted 19 lines · John Hubbard
If a compound page is being split while
dump_page()is being run on that page, we can end up callingcompound_mapcount()on a page that is no longer compound. This leads to a crash (already seen at least once in the field), due to theVM_BUG_ON_PAGE()assertion insidecompound_mapcount().(The above is from Matthew Wilcox's analysis of Qian Cai's bug report.)
A similar problem is possible, via
compound_pincount()instead ofcompound_mapcount().In order to avoid this kind of crash, make
dump_page()slightly more robust, by providing a pair of simpler routines that don't contain
assertions:head_mapcount()andhead_pincount().For debug tools, we don't want to go *too* far in this direction, but this is a simple small fix, and the crash has already been seen, so it's a good trade-off.
Quoted 5 lines
Reported-by Qian Cai <[email protected]>Suggested-by Matthew Wilcox <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Kirill A. Shutemov <[email protected]>Signed-off-by John Hubbard <[email protected]>
Quoted 6 lines
---
Hi,I'm assuming that a fix is not required for -stable, but let me know if others feel differently. The
dump_page()code has changed a lot in that area.
I'd say only if we backport all those changes, as most were to avoid similar kind of crashes?
How about this additional patch now that we have head_mapcoun()? (I wouldn't go for squashing as the goal and scope is too different).
----8<----
From 3b3d5b4639eea9c0787eed510a32acdc918d569f Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <[email protected]>
Date: Thu, 6 Aug 2020 13:33:58 +0200
Subject: [PATCH] mm, thp: use head_mapcount when we know we have a head page
Patch "mm, dump_page: do not crash with bad compound_mapcount()" has introduced head_mapcount() to split out the part of compound_mapcount() where we already know/assume we have a head page. We can use the new function in more places where we already have a head page, to avoid the overhead of compound_head() and (with DEBUG_VM) a debug check. This patch does that. There are few more applicable places, but behind DEBUG_VM so performance is not important, and the extra debug check in compound_mapcount() could be useful instead.
The bloat-o-meter difference without DEBUG_VM is the following:
add/remove: 0/0 grow/shrink: 1/4 up/down: 32/-56 (-24)
Function old new delta
__split_huge_pmd 2867 2899 +32 shrink_page_list 3860 3847 -13 reuse_swap_page 762 748 -14 page_trans_huge_mapcount 153 139 -14 total_mapcount 187 172 -15
Total: Before=8687306, After=8687282, chg -0.00%
This just shows that compiler wasn't able to prove we have a head page by itself. In __split_huge_pmd() the eliminated check probably led to different optimization decisions thus code size increased.
--- mm/huge_memory.c | 6 +++--- mm/swapfile.c | 2 +- mm/vmscan.c | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)
mm/huge_memory.c +3 -3
@@ -2125,7 +2125,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Set PG_double_map before dropping compound_mapcount to avoid * false-negative page_mapped(). */ -if (compound_mapcount (page) >1 && !TestSetPageDoubleMap (page)) { +if (head_mapcount (page) >1 && !TestSetPageDoubleMap (page)) {for (i =0 ; i <HPAGE_PMD_NR ; i++)atomic_inc (&page[i]._mapcount); }
@@ -2467,7 +2467,7 @@ int total_mapcount(struct page *page)if (likely (!PageCompound (page)))return atomic_read (&page->_mapcount) +1 ; - compound =compound_mapcount (page); + compound =head_mapcount (page);if (PageHuge (page))return compound; ret = compound;
@@ -2531,7 +2531,7 @@ int page_trans_huge_mapcount(struct page *page, int *total_mapcount) ret -=1 ; _total_mapcount -=HPAGE_PMD_NR ; } - mapcount =compound_mapcount (page); + mapcount =head_mapcount (page); ret += mapcount; _total_mapcount += mapcount;if (total_mapcount)
mm/swapfile.c +1 -1
@@ -1673,7 +1673,7 @@ static int page_trans_huge_map_swapcount(struct page *page, int *total_mapcount, map_swapcount -=1 ; _total_mapcount -=HPAGE_PMD_NR ; } - mapcount =compound_mapcount (page); + mapcount =head_mapcount (page); map_swapcount += mapcount; _total_mapcount += mapcount;if (total_mapcount)
mm/vmscan.c +1 -1
@@ -1248,7 +1248,7 @@ static unsigned int shrink_page_list(struct list_head *page_list,* away. Chances are some or all of the * tail pages can be freed without IO. */ -if (!compound_mapcount (page) && +if (!head_mapcount (page) &&split_huge_page_to_list (page, page_list)) goto activate_locked;
Signature · 1 line
2.28.0
Quoted 2 lines · Vlastimil Babka
How about this additional patch now that we have
head_mapcoun()? (I wouldn't go for squashing as the goal and scope is too different).
I like it. It bothers me that the compiler doesn't know that compound_head(compound_head(x)) == compound_head(x). I updated
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=32911 with a request to be able to tell the compiler that compound_head() is idempotent.
Quoted 10 lines
The bloat-o-meter difference without
DEBUG_VMis the following:add/remove: 0/0 grow/shrink: 1/4 up/down: 32/-56 (-24)
Function old new delta__split_huge_pmd 2867 2899 +32 shrink_page_list 3860 3847 -13 reuse_swap_page 762 748 -14 page_trans_huge_mapcount 153 139 -14 total_mapcount 187 172 -15Total: Before=8687306, After=8687282, chg -0.00%
That's great. I'm expecting improvements from my thp_head() macro when that lands (currently in Andrew's tree). I have been reluctant to replace current callers of compound_head() with thp_head(), but I suspect PF_HEAD could use thp_head() and save a few bytes on a tinyconfig build.
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 2128
Quoted 7 lines
+++
b/mm/huge_memory.cmm/huge_memory.c +1
-1@@ -2125,7 +2125,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Set PG_double_map before dropping compound_mapcount to avoid * false-negative page_mapped(). */ -if (compound_mapcount (page) >1 && !TestSetPageDoubleMap (page)) { +if (head_mapcount (page) >1 && !TestSetPageDoubleMap (page)) {
I'm a little nervous about this one. The page does actually come from pmd_page(), and today that's guaranteed to be a head page. But I'm not convinced that's going to still be true in twenty years. With the current THP patchset, I won't allocate pages larger than PMD order, but I can see there being interest in tracking pages in chunks larger than 2MB in the future. And then pmd_page() might well return a tail page. So it might be a good idea to not convert this one.
mm/huge_memory.c int total_mapcount(struct page *page) · line 2473
Quoted 9 lines
mm/huge_memory.c +1
-1@@ -2467,7 +2467,7 @@ int total_mapcount(struct page *page)if (likely (!PageCompound (page)))return atomic_read (&page->_mapcount) +1 ; - compound =compound_mapcount (page); + compound =head_mapcount (page);if (PageHuge (page))return compound; ret = compound;
Yes. This function is a little confusing because it uses PageCompound() all the way through when it really should be using PageHead ... because the opening line of the function is: VM_BUG_ON_PAGE(PageTail(page), page);
mm/huge_memory.c int page_trans_huge_mapcount(struct page *page, int *total_mapcount) · line 2537
Quoted 9 lines
mm/huge_memory.c +1
-1@@ -2531,7 +2531,7 @@ int page_trans_huge_mapcount(struct page *page, int *total_mapcount) ret -=1 ; _total_mapcount -=HPAGE_PMD_NR ; } - mapcount =compound_mapcount (page); + mapcount =head_mapcount (page); ret += mapcount; _total_mapcount += mapcount;if (total_mapcount)
Yes, we called compound_head() earlier in the function. Safe.
mm/swapfile.c static int page_trans_huge_map_swapcount(struct page *page, int *total_mapcount, · line 1679
Quoted 13 lines
mm/swapfile.c +1
-1@@ -1673,7 +1673,7 @@ static int page_trans_huge_map_swapcount(struct page *page, int *total_mapcount, map_swapcount -=1 ; _total_mapcount -=HPAGE_PMD_NR ; } - mapcount =compound_mapcount (page); + mapcount =head_mapcount (page); map_swapcount += mapcount; _total_mapcount += mapcount;if (total_mapcount)
Yes. page is a head page at this point.
mm/vmscan.c static unsigned int shrink_page_list(struct list_head *page_list, · line 1254
Quoted 13 lines
mm/vmscan.c +1
-1@@ -1248,7 +1248,7 @@ static unsigned int shrink_page_list(struct list_head *page_list,* away. Chances are some or all of the * tail pages can be freed without IO. */ -if (!compound_mapcount (page) && +if (!head_mapcount (page) &&split_huge_page_to_list (page, page_list)) goto activate_locked;
Yes. We don't put (can't put!) tail pages on the lists, so this must be a head page.
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Vlastimil Babka
How about this additional patch now that we have
head_mapcoun()? (I wouldn't go for squashing as the goal and scope is too different).I like it. It bothers me that the compiler doesn't know that compound_head(compound_head(x)) ==
compound_head(x). I updated
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=32911 with a request to be able to tell the compiler thatcompound_head()is idempotent.
Yeah it would be nice to get the benefits everywhere automatically. But I guess the compiler would have to discard the idempotence assumptions if there are multiple consecutive (perhaps hidden behind page flag access) compound_head(page) from a function, as soon as we modify the struct page somewhere.
Quoted 16 lines
Quoted 10 lines
The bloat-o-meter difference without
DEBUG_VMis the following:add/remove: 0/0 grow/shrink: 1/4 up/down: 32/-56 (-24)
Function old new delta__split_huge_pmd 2867 2899 +32 shrink_page_list 3860 3847 -13 reuse_swap_page 762 748 -14 page_trans_huge_mapcount 153 139 -14 total_mapcount 187 172 -15Total: Before=8687306, After=8687282, chg -0.00%
That's great. I'm expecting improvements from my
thp_head()macro when that lands (currently in Andrew's tree). I have been reluctant to replace current callers ofcompound_head()withthp_head(), but I suspectPF_HEADcould usethp_head()and save a few bytes on a tinyconfig build.
Quoted 15 lines
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 2128
Quoted 7 lines
+++
b/mm/huge_memory.cmm/huge_memory.c +1
-1@@ -2125,7 +2125,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Set PG_double_map before dropping compound_mapcount to avoid * false-negative page_mapped(). */ -if (compound_mapcount (page) >1 && !TestSetPageDoubleMap (page)) { +if (head_mapcount (page) >1 && !TestSetPageDoubleMap (page)) {I'm a little nervous about this one. The page does actually come from
pmd_page(), and today that's guaranteed to be a head page. But I'm not convinced that's going to still be true in twenty years. With the current THP patchset, I won't allocate pages larger than PMD order, but I can see there being interest in tracking pages in chunks larger than 2MB in the future. And thenpmd_page()might well return a tail page. So it might be a good idea to not convert this one.
Hmm the function converts the compound mapcount of the whole page to a HPAGE_PMD_NR of base pages. If suddenly the compound page was bigger than a pmd, then I guess this wouldn't work properly anymore without changes anyway? Maybe we could stick something like VM_BUG_ON(PageTransHuge(page)) there as "enforced documentation" for now?
Quoted 10 lines · Vlastimil Babka
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Vlastimil Babka
How about this additional patch now that we have
head_mapcoun()? (I wouldn't go for squashing as the goal and scope is too different).I like it. It bothers me that the compiler doesn't know that compound_head(compound_head(x)) ==
compound_head(x). I updated
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=32911 with a request to be able to tell the compiler thatcompound_head()is idempotent.
Quoted 4 lines
Yeah it would be nice to get the benefits everywhere automatically. But I guess the compiler would have to discard the idempotence assumptions if there are multiple consecutive (perhaps hidden behind page flag access)
compound_head(page)from a function, as soon as we modify the struct page somewhere.
The only place where we modify the struct page is the split code, and I think we're pretty careful to handle the pages appropriately there. The buddy allocator has its own way of combining pages.
Quoted 16 lines
Quoted 8 lines
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 2128
Quoted 7 lines
+++
b/mm/huge_memory.cmm/huge_memory.c +1
-1@@ -2125,7 +2125,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Set PG_double_map before dropping compound_mapcount to avoid * false-negative page_mapped(). */ -if (compound_mapcount (page) >1 && !TestSetPageDoubleMap (page)) { +if (head_mapcount (page) >1 && !TestSetPageDoubleMap (page)) {Quoted 7 lines
I'm a little nervous about this one. The page does actually come from
pmd_page(), and today that's guaranteed to be a head page. But I'm not convinced that's going to still be true in twenty years. With the current THP patchset, I won't allocate pages larger than PMD order, but I can see there being interest in tracking pages in chunks larger than 2MB in the future. And thenpmd_page()might well return a tail page. So it might be a good idea to not convert this one.
Quoted 5 lines
Hmm the function converts the compound mapcount of the whole page to a
HPAGE_PMD_NRof base pages. If suddenly the compound page was bigger than a pmd, then I guess this wouldn't work properly anymore without changes anyway? Maybe we could stick something like VM_BUG_ON(PageTransHuge(page)) there as "enforced documentation" for now?
I think it would work as-is. But also I may have totally misunderstood it. I'll write this declaratively and specifically for x86 (PMD order is 9) ... tell me when I've made a mistake ;-)
This function is for splitting the PMD. We're leaving the underlying page intact and just changing the page table. So if, say, we have an underlying 4MB page (and maybe the pages are mapped as PMDs in this process), we might get subpage number 512 of this order-10 page. We'd need to check the DoubleMap bit on subpage 1, and the compound_mapcount also stored in page 1, but we'd only want to spread the mapcount out over the 512 subpages from 512-1023; we wouldn't want to spread it out over 0-511 because they aren't affected by this particular PMD.
Having to reason about stuff like this is why I limited the THP code to stop at PMD order ... I don't want to make my life even more complicated than I have to!
Quoted 26 lines · Matthew Wilcox
Quoted 16 lines
Quoted 8 lines
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 2128
Quoted 7 lines
+++
b/mm/huge_memory.cmm/huge_memory.c +1
-1@@ -2125,7 +2125,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Set PG_double_map before dropping compound_mapcount to avoid * false-negative page_mapped(). */ -if (compound_mapcount (page) >1 && !TestSetPageDoubleMap (page)) { +if (head_mapcount (page) >1 && !TestSetPageDoubleMap (page)) {Quoted 7 lines
I'm a little nervous about this one. The page does actually come from
pmd_page(), and today that's guaranteed to be a head page. But I'm not convinced that's going to still be true in twenty years. With the current THP patchset, I won't allocate pages larger than PMD order, but I can see there being interest in tracking pages in chunks larger than 2MB in the future. And thenpmd_page()might well return a tail page. So it might be a good idea to not convert this one.Quoted 5 lines
Hmm the function converts the compound mapcount of the whole page to a
HPAGE_PMD_NRof base pages. If suddenly the compound page was bigger than a pmd, then I guess this wouldn't work properly anymore without changes anyway? Maybe we could stick something like VM_BUG_ON(PageTransHuge(page)) there as "enforced documentation" for now?I think it would work as-is. But also I may have totally misunderstood it. I'll write this declaratively and specifically for x86 (PMD order is 9) ... tell me when I've made a mistake ;-)
Quoted 8 lines
This function is for splitting the PMD. We're leaving the underlying page intact and just changing the page table. So if, say, we have an underlying 4MB page (and maybe the pages are mapped as PMDs in this process), we might get subpage number 512 of this order-10 page. We'd need to check the DoubleMap bit on subpage 1, and the
compound_mapcountalso stored in page 1, but we'd only want to spread the mapcount out over the 512 subpages from 512-1023; we wouldn't want to spread it out over 0-511 because they aren't affected by this particular PMD.
Yeah, and then we decrease the compound mapcount, which is a counter of "how many times is this compound page mapped as a whole". But we only removed (the second) half of the compound mapping, so imho that would be wrong?
Quoted 3 lines
Having to reason about stuff like this is why I limited the THP code to stop at PMD order ... I don't want to make my life even more complicated than I have to!
Kirill might correct me but I'd expect the THP code right now has baked in many assumptions about THP pages being exactly HPAGE_PMD_ORDER large?
Quoted 36 lines · Vlastimil Babka
Quoted 26 lines · Matthew Wilcox
Quoted 16 lines
Quoted 8 lines
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 2128
Quoted 7 lines
+++
b/mm/huge_memory.cmm/huge_memory.c +1
-1@@ -2125,7 +2125,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Set PG_double_map before dropping compound_mapcount to avoid * false-negative page_mapped(). */ -if (compound_mapcount (page) >1 && !TestSetPageDoubleMap (page)) { +if (head_mapcount (page) >1 && !TestSetPageDoubleMap (page)) {Quoted 7 lines
I'm a little nervous about this one. The page does actually come from
pmd_page(), and today that's guaranteed to be a head page. But I'm not convinced that's going to still be true in twenty years. With the current THP patchset, I won't allocate pages larger than PMD order, but I can see there being interest in tracking pages in chunks larger than 2MB in the future. And thenpmd_page()might well return a tail page. So it might be a good idea to not convert this one.Quoted 5 lines
Hmm the function converts the compound mapcount of the whole page to a
HPAGE_PMD_NRof base pages. If suddenly the compound page was bigger than a pmd, then I guess this wouldn't work properly anymore without changes anyway? Maybe we could stick something like VM_BUG_ON(PageTransHuge(page)) there as "enforced documentation" for now?I think it would work as-is. But also I may have totally misunderstood it. I'll write this declaratively and specifically for x86 (PMD order is 9) ... tell me when I've made a mistake ;-)
Quoted 8 lines
This function is for splitting the PMD. We're leaving the underlying page intact and just changing the page table. So if, say, we have an underlying 4MB page (and maybe the pages are mapped as PMDs in this process), we might get subpage number 512 of this order-10 page. We'd need to check the DoubleMap bit on subpage 1, and the
compound_mapcountalso stored in page 1, but we'd only want to spread the mapcount out over the 512 subpages from 512-1023; we wouldn't want to spread it out over 0-511 because they aren't affected by this particular PMD.
Quoted 3 lines
Yeah, and then we decrease the compound mapcount, which is a counter of "how many times is this compound page mapped as a whole". But we only removed (the second) half of the compound mapping, so imho that would be wrong?
I'd expect that count to be incremented by 1 for each PMD that it's mapped to? ie change the definition of that counter slightly.
Quoted 6 lines
Quoted 3 lines
Having to reason about stuff like this is why I limited the THP code to stop at PMD order ... I don't want to make my life even more complicated than I have to!
Kirill might correct me but I'd expect the THP code right now has baked in many assumptions about THP pages being exactly
HPAGE_PMD_ORDERlarge?
There are somewhat fewer places that make that assumption after applying the ~80 patches here ... http://git.infradead.org/users/willy/pagecache.git
I have mostly not touched the anonymous THPs (obviously some of the code paths are shared), although both Kirill & I think there's a win to be had there too.
Quoted 37 lines · Matthew Wilcox
Quoted 36 lines · Vlastimil Babka
Quoted 34 lines · Matthew Wilcox
Quoted 16 lines
Quoted 8 lines
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 2128
Quoted 7 lines
+++
b/mm/huge_memory.cmm/huge_memory.c +1
-1@@ -2125,7 +2125,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,* Set PG_double_map before dropping compound_mapcount to avoid * false-negative page_mapped(). */ -if (compound_mapcount (page) >1 && !TestSetPageDoubleMap (page)) { +if (head_mapcount (page) >1 && !TestSetPageDoubleMap (page)) {Quoted 7 lines
I'm a little nervous about this one. The page does actually come from
pmd_page(), and today that's guaranteed to be a head page. But I'm not convinced that's going to still be true in twenty years. With the current THP patchset, I won't allocate pages larger than PMD order, but I can see there being interest in tracking pages in chunks larger than 2MB in the future. And thenpmd_page()might well return a tail page. So it might be a good idea to not convert this one.Quoted 5 lines
Hmm the function converts the compound mapcount of the whole page to a
HPAGE_PMD_NRof base pages. If suddenly the compound page was bigger than a pmd, then I guess this wouldn't work properly anymore without changes anyway? Maybe we could stick something like VM_BUG_ON(PageTransHuge(page)) there as "enforced documentation" for now?I think it would work as-is. But also I may have totally misunderstood it. I'll write this declaratively and specifically for x86 (PMD order is 9) ... tell me when I've made a mistake ;-)
This function is for splitting the PMD. We're leaving the underlying page intact and just changing the page table. So if, say, we have an underlying 4MB page (and maybe the pages are mapped as PMDs in this process), we might get subpage number 512 of this order-10 page. We'd need to check the DoubleMap bit on subpage 1, and the
compound_mapcountalso stored in page 1, but we'd only want to spread the mapcount out over the 512 subpages from 512-1023; we wouldn't want to spread it out over 0-511 because they aren't affected by this particular PMD.
Quoted 13 lines
Quoted 3 lines
Yeah, and then we decrease the compound mapcount, which is a counter of "how many times is this compound page mapped as a whole". But we only removed (the second) half of the compound mapping, so imho that would be wrong?
I'd expect that count to be incremented by 1 for each PMD that it's mapped to? ie change the definition of that counter slightly.
Quoted 6 lines
Quoted 3 lines
Having to reason about stuff like this is why I limited the THP code to stop at PMD order ... I don't want to make my life even more complicated than I have to!
Kirill might correct me but I'd expect the THP code right now has baked in many assumptions about THP pages being exactly
HPAGE_PMD_ORDERlarge?
That will be true for PMD-mapped THP pages after applying Matthew's patchset.
Quoted 2 lines
There are somewhat fewer places that make that assumption after applying the ~80 patches here ... http://git.infradead.org/users/willy/pagecache.git
The patchset allows for THP to be anywhere between order-2 and order-9 (on x86-64).
Quoted 3 lines
I have mostly not touched the anonymous THPs (obviously some of the code paths are shared), although both Kirill & I think there's a win to be had there too.
Yeah. Reducing LRU handling overhead alone should be enough to justify the effort. But we still would need to have numbers.
Signature · 1 line
Kirill A. Shutemov
Quoted 6 lines · John Hubbard
If a compound page is being split while
dump_page()is being run on that page, we can end up callingcompound_mapcount()on a page that is no longer compound. This leads to a crash (already seen at least once in the field), due to theVM_BUG_ON_PAGE()assertion insidecompound_mapcount().
Quoted 8 lines
(The above is from Matthew Wilcox's analysis of Qian Cai's bug report.)
A similar problem is possible, via
compound_pincount()instead ofcompound_mapcount().In order to avoid this kind of crash, make
dump_page()slightly more robust, by providing a pair of simpler routines that don't contain
assertions:head_mapcount()andhead_pincount().
I find naming misleading. head_mapcount() and head_pincount() sounds like a mapcount/pincount of the head page, but it's not. It's mapcount and pincount of the compound page.
Maybe compound_mapcount_head() and compound_pincoun_head()? Or __compound_mapcount() and __compound_pincount().
Quoted 40 lines
For debug tools, we don't want to go *too* far in this direction, but this is a simple small fix, and the crash has already been seen, so it's a good trade-off.
Reported-by Qian Cai <[email protected]>Suggested-by Matthew Wilcox <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Kirill A. Shutemov <[email protected]>Signed-off-by John Hubbard <[email protected]>---
Hi,I'm assuming that a fix is not required for -stable, but let me know if others feel differently. The
dump_page()code has changed a lot in that area.Changes since v1 [1]:
1) Rebased onto mmotm
2) Used a simpler head_*count() approach.
3) Added Matthew's Suggested-by: tag
4) Support pincount as well as mapcount.
[1] https://lore.kernel.org/linux-mm/[email protected]/
thanks,
John Hubbardinclude/linux/mm.h | 14 ++++++++++++-- mm/debug.c | 6 +++--- 2 files changed, 15 insertions(+), 5 deletions(-)include/linux/mm.h +0
-0
include/linux/mm.h static inline void *kvcalloc(size_t n, size_t size, gfp_t flags) · line 780
Quoted 5 lines
include/linux/mm.h +2
-0@@ -776,6 +776,11 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)extern void kvfree (const void *addr);extern void kvfree_sensitive (const void *addr,size_t len); +static inline int head_mapcount (struct page *head) +{
Do we want VM_BUG_ON_PAGE(!PageHead(head), head) here?
Quoted 16 lines
include/linux/mm.h +4
-1+return atomic_read (compound_mapcount_ptr (head)) +1 ; +} +/* * Mapcount of compound page as a whole, does not include mapped sub-pages. * @@ -785,7 +790,7 @@ static inline int compound_mapcount(struct page *page) {VM_BUG_ON_PAGE (!PageCompound (page), page); page =compound_head (page); -return atomic_read (compound_mapcount_ptr (page)) +1 ; +return head_mapcount (page); }/*
include/linux/mm.h static inline bool hpage_pincount_available(struct page *page) · line 907
Quoted 5 lines
include/linux/mm.h +2
-0@@ -898,11 +903,16 @@ static inline bool hpage_pincount_available(struct page *page)return PageCompound (page) &&compound_order (page) >1 ; } +static inline int head_pincount (struct page *head) +{
Ditto.
Quoted 35 lines
include/linux/mm.h +4
-1+return atomic_read (compound_pincount_ptr (head)); +} +static inline int compound_pincount (struct page *page) {VM_BUG_ON_PAGE (!hpage_pincount_available (page), page); page =compound_head (page); -return atomic_read (compound_pincount_ptr (page)); +return head_pincount (page); }static inline void set_compound_order (struct page *page,unsigned int order)mm/debug.c +3
-3@@ -102,12 +102,12 @@ void __dump_page(struct page *page, const char *reason)if (hpage_pincount_available (page)) {pr_warn ("head:%p order:%u compound_mapcount:%d compound_pincount:%d\n" , head,compound_order (head), -compound_mapcount (head), -compound_pincount (head)); +head_mapcount (head), +head_pincount (head)); }else {pr_warn ("head:%p order:%u compound_mapcount:%d\n" , head,compound_order (head), -compound_mapcount (head)); +head_mapcount (head)); } }if (PageKsm (page))Signature · 1 line
2.28.0
Signature · 1 line
Kirill A. Shutemov
Quoted 6 lines · Kirill A. Shutemov
Quoted 5 lines · John Hubbard
If a compound page is being split while
dump_page()is being run on that page, we can end up callingcompound_mapcount()on a page that is no longer compound. This leads to a crash (already seen at least once in the field), due to theVM_BUG_ON_PAGE()assertion insidecompound_mapcount().
[...]
Quoted 4 lines
include/linux/mm.h static inline void *kvcalloc(size_t n, size_t size, gfp_t flags) · line 780
Quoted 2 lines
include/linux/mm.h +2
-0+static inline int head_mapcount (struct page *head) +{Do we want
VM_BUG_ON_PAGE(!PageHead(head), head)here?
Well, no. That was the point of the bug report -- by the time we called compound_mapcount, the page was no longer a head page.
Quoted 10 lines
Quoted 6 lines
A similar problem is possible, via
compound_pincount()instead ofcompound_mapcount().In order to avoid this kind of crash, make
dump_page()slightly more robust, by providing a pair of simpler routines that don't contain
assertions:head_mapcount()andhead_pincount().I find naming misleading.
head_mapcount()andhead_pincount()sounds like a mapcount/pincount of the head page, but it's not. It's mapcount and pincount of the compound page.
OK, point taken. I might go for head_compound_mapcount()? Or as I originally suggested, just opencoding it like we do in __page_mapcount().
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · Kirill A. Shutemov
Quoted 5 lines · John Hubbard
If a compound page is being split while
dump_page()is being run on that page, we can end up callingcompound_mapcount()on a page that is no longer compound. This leads to a crash (already seen at least once in the field), due to theVM_BUG_ON_PAGE()assertion insidecompound_mapcount().
Quoted 8 lines
[...]
Quoted 4 lines
include/linux/mm.h static inline void *kvcalloc(size_t n, size_t size, gfp_t flags) · line 780
Quoted 2 lines
include/linux/mm.h +2
-0+static inline int head_mapcount (struct page *head) +{Do we want
VM_BUG_ON_PAGE(!PageHead(head), head)here?Well, no. That was the point of the bug report -- by the time we called
compound_mapcount, the page was no longer a head page.
Right. VM_BUG_ON_PAGE(PageTail(head), head)?
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
A similar problem is possible, via
compound_pincount()instead ofcompound_mapcount().In order to avoid this kind of crash, make
dump_page()slightly more robust, by providing a pair of simpler routines that don't contain
assertions:head_mapcount()andhead_pincount().
Quoted 6 lines
Quoted 3 lines
I find naming misleading.
head_mapcount()andhead_pincount()sounds like a mapcount/pincount of the head page, but it's not. It's mapcount and pincount of the compound page.OK, point taken. I might go for
head_compound_mapcount()? Or as I originally suggested, just opencoding it like we do in__page_mapcount().
I'm fine either way.
Signature · 1 line
Kirill A. Shutemov
PATCH 1/1 mm, dump_page: rename head_mapcount() --> head_compound_mapcount()
And similarly, rename head_pincount() --> head_compound_pincount(). These names are more accurate (or less misleading) than the original ones.
---
Hi,
This is a follow-up patch to v2 of "mm, dump_page: do not crash with bad compound_mapcount()", which I see has has already been sent as part of Andrew's pull request. Otherwise I would have sent a v3.
Of course, if it's somehow not too late, then squashing this patch into that one, effectively creating a v3 with the preferred names, would be a nice touch.
thanks,
John Hubbard
include/linux/mm.h | 8 ++++---- mm/debug.c | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-)
include/linux/mm.h +4 -4
@@ -776,7 +776,7 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)extern void kvfree (const void *addr);extern void kvfree_sensitive (const void *addr,size_t len); -static inline int head_mapcount (struct page *head) +static inline int head_compound_mapcount (struct page *head) {return atomic_read (compound_mapcount_ptr (head)) +1 ; }
@@ -790,7 +790,7 @@ static inline int compound_mapcount(struct page *page) {VM_BUG_ON_PAGE (!PageCompound (page), page); page =compound_head (page); -return head_mapcount (page); +return head_compound_mapcount (page); }/*
@@ -903,7 +903,7 @@ static inline bool hpage_pincount_available(struct page *page)return PageCompound (page) &&compound_order (page) >1 ; } -static inline int head_pincount (struct page *head) +static inline int head_compound_pincount (struct page *head) {return atomic_read (compound_pincount_ptr (head)); }
@@ -912,7 +912,7 @@ static inline int compound_pincount(struct page *page) {VM_BUG_ON_PAGE (!hpage_pincount_available (page), page); page =compound_head (page); -return head_pincount (page); +return head_compound_pincount (page); }static inline void set_compound_order (struct page *page,unsigned int order)
mm/debug.c +3 -3
@@ -102,12 +102,12 @@ void __dump_page(struct page *page, const char *reason)if (hpage_pincount_available (page)) {pr_warn ("head:%p order:%u compound_mapcount:%d compound_pincount:%d\n" , head,compound_order (head), -head_mapcount (head), -head_pincount (head)); +head_compound_mapcount (head), +head_compound_pincount (head)); }else {pr_warn ("head:%p order:%u compound_mapcount:%d\n" , head,compound_order (head), -head_mapcount (head)); +head_compound_mapcount (head)); } }if (PageKsm (page))
Signature · 1 line
2.28.0
Quoted 10 lines · Kirill A. Shutemov
Quoted 8 lines
[...]
Quoted 4 lines
include/linux/mm.h static inline void *kvcalloc(size_t n, size_t size, gfp_t flags) · line 780
Quoted 2 lines
include/linux/mm.h +2
-0+static inline int head_mapcount (struct page *head) +{Do we want
VM_BUG_ON_PAGE(!PageHead(head), head)here?Well, no. That was the point of the bug report -- by the time we called
compound_mapcount, the page was no longer a head page.Right.
VM_BUG_ON_PAGE(PageTail(head), head)?
Sorry for overlooking that feedback point. Looking at it now, I'd much rather not put any assertions at all here. This supposed to be for implementing the failure case, and we've moved past assertions at this point. In other words, dump_page() is part of *implementing* an assertion, so calling assertions from within it is undesirable.
It's better to put the assertions in code that would call these inner routines. Then, if we want to use these routines outside of mm/debug.c, as per the other thread, then we should provide a wrapper that has such assertions.
thanks,
Signature · 2 lines
John Hubbard NVIDIA