mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()
Subsystems (2)
PATCH v2 1/1 mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()
Replace superfluous VM_BUG_ON() with comment about correct usage.
Technically reverts commit 1d148e218a0d0566b1c06f2f45f1436d53b049b2 ("mm: add VM_BUG_ON_PAGE() to page_mapcount()"), but context have changed.
Function isolate_migratepages_block() runs some checks out of lru_lock when choose pages for migration. After checking PageLRU() it checks extra page references by comparing page_count() and page_mapcount(). Between these two checks page could be removed from lru, freed and taken by slab.
As a result this race triggers VM_BUG_ON(PageSlab()) in page_mapcount(). Race window is tiny. For certain workload this happens around once a year.
page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0 compound_mapcount: 0
flags: 0x500000000008100(slab|head)
raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180 raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 page dumped because: VM_BUG_ON_PAGE(PageSlab(page)) ------------[ cut here ]------------ kernel BUG at ./include/linux/mm.h:628! invalid opcode: 0000 [#1] SMP NOPTI CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G W 4.19.109-27 #1 Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019 RIP: 0010:isolate_migratepages_block+0x986/0x9b0
Code in isolate_migratepages_block() was added in commit 119d6d59dcc0 ("mm, compaction: avoid isolating pinned pages") before adding VM_BUG_ON into page_mapcount().
This race has been predicted in 2015 by Vlastimil Babka (see link below).
--- include/linux/mm.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
include/linux/mm.h +12 -2
@@ -782,6 +782,11 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)extern void kvfree (const void *addr); +/* +* Mapcount of compound page as a whole, not includes mapped sub-pages. +* +* Must be called only for compound pages or any their tail sub-pages. +*/ static inline int compound_mapcount (struct page *page) {VM_BUG_ON_PAGE (!PageCompound (page), page);
@@ -801,10 +806,15 @@ static inline void page_mapcount_reset(struct page *page)int __page_mapcount (struct page *page); +/* +* Mapcount of 0-order page, for sub-page includes compound_mapcount(). +* +* Result is undefined for pages which cannot be mapped into userspace. +* For example SLAB or special types of pages. See function page_has_type(). +* They use this place in struct page differently. +*/ static inline int page_mapcount (struct page *page) { -VM_BUG_ON_PAGE (PageSlab (page), page); -if (unlikely (PageCompound (page)))return __page_mapcount (page);return atomic_read (&page->_mapcount) +1 ;
Quoted 27 lines · Konstantin Khlebnikov
Replace superfluous
VM_BUG_ON()with comment about correct usage.Technically reverts commit
1d148e218a0d0566b1c06f2f45f1436d53b049b2("mm: addVM_BUG_ON_PAGE()topage_mapcount()"), but context have changed.Function
isolate_migratepages_block()runs some checks out oflru_lockwhen choose pages for migration. After checkingPageLRU()it checks extra page references by comparingpage_count()andpage_mapcount(). Between these two checks page could be removed from lru, freed and taken by slab.As a result this race triggers VM_BUG_ON(PageSlab()) in
page_mapcount(). Race window is tiny. For certain workload this happens around once a year.page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0
compound_mapcount: 0
flags: 0x500000000008100(slab|head)raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180 raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 page dumped because: VM_BUG_ON_PAGE(PageSlab(page)) ------------[ cut here ]------------ kernel BUG at ./include/linux/mm.h:628! invalid opcode: 0000 [#1] SMP NOPTI CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G W 4.19.109-27 #1 Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019 RIP: 0010:isolate_migratepages_block+0x986/0x9b0
Quoted 5 lines
Code in
isolate_migratepages_block()was added in commit119d6d59dcc0("mm, compaction: avoid isolating pinned pages") before addingVM_BUG_ONintopage_mapcount().This race has been predicted in 2015 by Vlastimil Babka (see link below).
Ah, kudos to Vlastimil.
Quoted 2 lines
Signed-off-by Konstantin Khlebnikov <[email protected]>
Thanks,
For the BUG deletion. I got a bit confused by the comments at first:
slight rewording suggested below.
I had also intended a comment in page_mapcount(), on being used in that racy exceptional way by isolate_migratepages_block(): but now that the BUG is gone, I think you're right not to add that - it's not for the function to document its various callers, nor what's not in it. That kind of info belongs to the commit message, as you have done.
Quoted 12 lines
Fixes 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")--- include/linux/mm.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)include/linux/mm.h +0
-0
include/linux/mm.h static inline void *kvcalloc(size_t n, size_t size, gfp_t flags) · line 786
Quoted 5 lines
include/linux/mm.h +2
-0@@ -782,6 +782,11 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)extern void kvfree (const void *addr); +/* +* Mapcount of compound page as a whole, not includes mapped sub-pages.
s/not includes/does not include/
include/linux/mm.h static inline void page_mapcount_reset(struct page *page) · line 810
Quoted 12 lines
include/linux/mm.h +5
-0+* +* Must be called only for compound pages or any their tail sub-pages. +*/ static inline int compound_mapcount (struct page *page) {VM_BUG_ON_PAGE (!PageCompound (page), page);@@ -801,10 +806,15 @@ static inline void page_mapcount_reset(struct page *page)int __page_mapcount (struct page *page); +/* +* Mapcount of 0-order page, for sub-page includes compound_mapcount().
s/, for sub-page/; when compound sub-page,/
Quoted 12 lines
include/linux/mm.h +5
-2+* +* Result is undefined for pages which cannot be mapped into userspace. +* For example SLAB or special types of pages. See function page_has_type(). +* They use this place in struct page differently. +*/ static inline int page_mapcount (struct page *page) { -VM_BUG_ON_PAGE (PageSlab (page), page); -if (unlikely (PageCompound (page)))return __page_mapcount (page);return atomic_read (&page->_mapcount) +1 ;
Quoted 31 lines · Konstantin Khlebnikov
Replace superfluous
VM_BUG_ON()with comment about correct usage.Technically reverts commit
1d148e218a0d0566b1c06f2f45f1436d53b049b2("mm: addVM_BUG_ON_PAGE()topage_mapcount()"), but context have changed.Function
isolate_migratepages_block()runs some checks out oflru_lockwhen choose pages for migration. After checkingPageLRU()it checks extra page references by comparingpage_count()andpage_mapcount(). Between these two checks page could be removed from lru, freed and taken by slab.As a result this race triggers VM_BUG_ON(PageSlab()) in
page_mapcount(). Race window is tiny. For certain workload this happens around once a year.page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0
compound_mapcount: 0
flags: 0x500000000008100(slab|head)raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180 raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 page dumped because: VM_BUG_ON_PAGE(PageSlab(page)) ------------[ cut here ]------------ kernel BUG at ./include/linux/mm.h:628! invalid opcode: 0000 [#1] SMP NOPTI CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G W 4.19.109-27 #1 Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019 RIP: 0010:isolate_migratepages_block+0x986/0x9b0Code in
isolate_migratepages_block()was added in commit119d6d59dcc0("mm, compaction: avoid isolating pinned pages") before addingVM_BUG_ONintopage_mapcount().
Quoted 6 lines
This race has been predicted in 2015 by Vlastimil Babka (see link below).
Signed-off-by Konstantin Khlebnikov <[email protected]>Fixes 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")
Signature · 1 line
Kirill A. Shutemov
Quoted 27 lines · Konstantin Khlebnikov
Replace superfluous
VM_BUG_ON()with comment about correct usage.Technically reverts commit
1d148e218a0d0566b1c06f2f45f1436d53b049b2("mm: addVM_BUG_ON_PAGE()topage_mapcount()"), but context have changed.Function
isolate_migratepages_block()runs some checks out oflru_lockwhen choose pages for migration. After checkingPageLRU()it checks extra page references by comparingpage_count()andpage_mapcount(). Between these two checks page could be removed from lru, freed and taken by slab.As a result this race triggers VM_BUG_ON(PageSlab()) in
page_mapcount(). Race window is tiny. For certain workload this happens around once a year.page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0
compound_mapcount: 0
flags: 0x500000000008100(slab|head)raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180 raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 page dumped because: VM_BUG_ON_PAGE(PageSlab(page)) ------------[ cut here ]------------ kernel BUG at ./include/linux/mm.h:628! invalid opcode: 0000 [#1] SMP NOPTI CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G W 4.19.109-27 #1 Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019 RIP: 0010:isolate_migratepages_block+0x986/0x9b0
Quoted 5 lines
Code in
isolate_migratepages_block()was added in commit119d6d59dcc0("mm, compaction: avoid isolating pinned pages") before addingVM_BUG_ONintopage_mapcount().This race has been predicted in 2015 by Vlastimil Babka (see link below).
Huh, looks like I made that prediction only half year after that patch has been posted. Now if only I remembered why... I hope it was just a code inspection while chasing something else. I most likely didn't actually see the bug happen, as we don't compile with DEBUG_VM.
Quoted 4 lines
Signed-off-by Konstantin Khlebnikov <[email protected]>Fixes 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")
With Hugh's wording tweaks,
Thanks.