mm/mm_init: fix out-of-range first_deferred_pfn
PATCH 1/1 mm/mm_init: fix out-of-range first_deferred_pfn
deferred_grow_zone() initializes deferred struct pages a section at a time until the allocation that called into it can be satisfied, and records where to resume in pgdat->first_deferred_pfn. Reserve most of the top zone early (a large CMA reservation is the easy way) and a single early allocation has to walk the whole zone instead of stopping in its first free section. If that zone does not end on a section boundary, the pgdatinit kthread then dies:
kernel BUG at mm/mm_init.c:2131!
Oops: invalid opcode: 0000 [#1] SMP NOPTI CPU: 3 UID: 0 PID: 36 Comm: pgdatinit0 Not tainted 7.2.0-rc6 #1 RIP: 0010:deferred_init_memmap+0x1b8/0x1c0 RAX: 0000000000236000 R13: 0000000000238000 Call Trace: kthread+0xdf/0x120 ret_from_fork+0x187/0x250
RAX is pgdat_end_pfn(), R13 the pfn that was stored. The loop advances spfn in whole PAGES_PER_SECTION steps and only tests it before entering an iteration, so once the walk reaches the end of a zone that ends mid-section the escaping spfn is SECTION_ALIGN_UP(zone_end_pfn()). Commit 3acb913c9d5b ("mm/mm_init: use deferred_init_memmap_chunk() in deferred_grow_zone()") dropped the clamp that used to prevent that: epfn came from __next_mem_pfn_range_in_zone(), since removed, which capped it with min(zone_end_pfn(zone), epfn), so spfn could reach zone_end_pfn but never pass it. The assert is fatal either way, panicking under panic_on_oops and otherwise leaving page_alloc_init_late() waiting forever for a completion the dead kthread never reports.
Store ULONG_MAX once spfn has left the zone. Nothing is left
uninitialized: the loop covers a single gap-free interval, and because it only enters with spfn < zone_end_pfn() the escaping value is exactly SECTION_ALIGN_UP(zone_end_pfn()), which is the last_pfn that deferred_init_memmap() would have used for the same pfn range. A zone that does end section-aligned now takes this path too and loses its zero-work padata job along with that node's pr_info() and the WARN_ON() on the next zone.
To reproduce with CONFIG_DEFERRED_STRUCT_PAGE_INIT=y and CONFIG_CMA=y:
qemu-system-x86_64 -enable-kvm -m 8032M -kernel bzImage \
-append "nokaslr cma=4768M@0x100000000"Top of RAM is then 0x235ffffff, so ZONE_NORMAL ends 96 MiB into its last section, and less than a section stays free above the reservation once the early memblock allocations are done. The walk therefore runs off the end of the zone and stores 0x238000. Sweeping that free remainder from 96M to 288M in 16M steps, an unpatched kernel dies on 7 of the 13 boots and a patched one on none. On an 8 GiB cloud instance that reserves most of its top zone for a memory pool, roughly one boot in three panicked before reaching userspace.
---
Notes:
Applies unchanged to 6.18.y, 6.19.y, 7.0.y and 7.1.y (checked against v6.18.39, v6.19.14, v7.0.14 and v7.1.4); the deferred_init_memmap_chunk() signature change in cbbbf7795fc3 sits outside the hunk context, so stable needs no separate backport.
First seen on 6.18.y and 6.19-rc distribution kernels. There is no public report to link, hence no Closes: tag.
mm/mm_init.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
mm/mm_init.c +6 -3
@@ -2214,10 +2214,13 @@ bool __init deferred_grow_zone(struct zone *zone, unsigned int order) }/* -* There were no pages to initialize and free which means the zone's -* memory map is completely initialized. +* The loop only tests spfn before entering an iteration, so on exit it +* may point up to a section past the end of the zone. When it does, +* the rest of the zone has already been handed to +* deferred_init_memmap_chunk() and nothing is left to initialize. */ - pgdat->first_deferred_pfn = nr_pages ? spfn :ULONG_MAX ; + pgdat->first_deferred_pfn = + spfn <zone_end_pfn (zone) ? spfn :ULONG_MAX ;pgdat_resize_unlock (pgdat, &flags);
base-commit: 0d839570765118029aa8bf4a95444c6a11aacf85
Signature · 1 line
2.47.1
Mike Rapoport
maintainerHi Alex,
Quoted 38 lines · Alexander Graf
deferred_grow_zone()initializes deferred struct pages a section at a time until the allocation that called into it can be satisfied, and records where to resume inpgdat->first_deferred_pfn. Reserve most of the top zone early (a large CMA reservation is the easy way) and a single early allocation has to walk the whole zone instead of stopping in its first free section. If that zone does not end on a section boundary, the pgdatinit kthread then dies:kernel BUG at
mm/mm_init.c:2131!Oops: invalid opcode: 0000 [#1] SMP NOPTI CPU: 3 UID: 0 PID: 36 Comm: pgdatinit0 Not tainted 7.2.0-rc6 #1 RIP: 0010:deferred_init_memmap+0x1b8/0x1c0 RAX: 0000000000236000 R13: 0000000000238000 Call Trace: kthread+0xdf/0x120 ret_from_fork+0x187/0x250RAX is
pgdat_end_pfn(), R13 the pfn that was stored. The loop advances spfn in wholePAGES_PER_SECTIONsteps and only tests it before entering an iteration, so once the walk reaches the end of a zone that ends mid-section the escaping spfn is SECTION_ALIGN_UP(zone_end_pfn()). Commit3acb913c9d5b("mm/mm_init: usedeferred_init_memmap_chunk()indeferred_grow_zone()") dropped the clamp that used to prevent that: epfn came from__next_mem_pfn_range_in_zone(), since removed, which capped it withmin(zone_end_pfn(zone), epfn), so spfn could reachzone_end_pfnbut never pass it. The assert is fatal either way, panicking underpanic_on_oopsand otherwise leavingpage_alloc_init_late()waiting forever for a completion the dead kthread never reports.Store
ULONG_MAXonce spfn has left the zone. Nothing is left
uninitialized: the loop covers a single gap-free interval, and because it only enters with spfn <zone_end_pfn()the escaping value is exactly SECTION_ALIGN_UP(zone_end_pfn()), which is thelast_pfnthatdeferred_init_memmap()would have used for the same pfn range. A zone that does end section-aligned now takes this path too and loses its zero-work padata job along with that node'spr_info()and theWARN_ON()on the next zone.
Quoted 4 lines
To reproduce with
CONFIG_DEFERRED_STRUCT_PAGE_INIT=yandCONFIG_CMA=y:qemu-system-x86_64 -enable-kvm -m 8032M -kernel bzImage \ -append "nokaslr cma=4768M@0x100000000"
The only two paragraphs I understood is this and the BUG splat ;-P
Can we please have a lot more of human touch on the changelog?
Quoted 53 lines
Top of RAM is then
0x235ffffff, soZONE_NORMALends 96 MiB into its last section, and less than a section stays free above the reservation once the early memblock allocations are done. The walk therefore runs off the end of the zone and stores0x238000. Sweeping that free remainder from 96M to 288M in 16M steps, an unpatched kernel dies on 7 of the 13 boots and a patched one on none. On an 8 GiB cloud instance that reserves most of its top zone for a memory pool, roughly one boot in three panicked before reaching userspace.Fixes 3acb913c9d5b ("mm/mm_init: use deferred_init_memmap_chunk() in deferred_grow_zone()")Assisted-by Kiro:claude-opus-5Signed-off-by Alexander Graf <[email protected]>---
Notes:
Applies unchanged to 6.18.y, 6.19.y, 7.0.y and 7.1.y (checked against v6.18.39, v6.19.14, v7.0.14 and v7.1.4); thedeferred_init_memmap_chunk()signature change incbbbf7795fc3sits outside the hunk context, so stable needs no separate backport.First seen on 6.18.y and 6.19-rc distribution kernels. There is no public report to link, hence no Closes: tag.
mm/mm_init.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/mm_init.c +6
-3@@ -2214,10 +2214,13 @@ bool __init deferred_grow_zone(struct zone *zone, unsigned int order) }/* -* There were no pages to initialize and free which means the zone's -* memory map is completely initialized. +* The loop only tests spfn before entering an iteration, so on exit it +* may point up to a section past the end of the zone. When it does, +* the rest of the zone has already been handed to +* deferred_init_memmap_chunk() and nothing is left to initialize. */ - pgdat->first_deferred_pfn = nr_pages ? spfn :ULONG_MAX ; + pgdat->first_deferred_pfn = + spfn <zone_end_pfn (zone) ? spfn :ULONG_MAX ;pgdat_resize_unlock (pgdat, &flags);base-commit:
0d839570765118029aa8bf4a95444c6a11aacf85Signature · 1 line
2.47.1
Signature · 2 lines
Sincerely yours, Mike.
Quoted 41 lines · Mike Rapoport
Hi Alex,
Quoted 38 lines · Alexander Graf
deferred_grow_zone()initializes deferred struct pages a section at a time until the allocation that called into it can be satisfied, and records where to resume inpgdat->first_deferred_pfn. Reserve most of the top zone early (a large CMA reservation is the easy way) and a single early allocation has to walk the whole zone instead of stopping in its first free section. If that zone does not end on a section boundary, the pgdatinit kthread then dies:kernel BUG at mm/mm_init.c:2131! Oops: invalid opcode: 0000 [#1] SMP NOPTI CPU: 3 UID: 0 PID: 36 Comm: pgdatinit0 Not tainted 7.2.0-rc6 #1 RIP: 0010:deferred_init_memmap+0x1b8/0x1c0 RAX: 0000000000236000 R13: 0000000000238000 Call Trace: kthread+0xdf/0x120 ret_from_fork+0x187/0x250RAX is
pgdat_end_pfn(), R13 the pfn that was stored. The loop advances spfn in wholePAGES_PER_SECTIONsteps and only tests it before entering an iteration, so once the walk reaches the end of a zone that ends mid-section the escaping spfn is SECTION_ALIGN_UP(zone_end_pfn()). Commit3acb913c9d5b("mm/mm_init: usedeferred_init_memmap_chunk()indeferred_grow_zone()") dropped the clamp that used to prevent that: epfn came from__next_mem_pfn_range_in_zone(), since removed, which capped it withmin(zone_end_pfn(zone), epfn), so spfn could reachzone_end_pfnbut never pass it. The assert is fatal either way, panicking underpanic_on_oopsand otherwise leavingpage_alloc_init_late()waiting forever for a completion the dead kthread never reports.Store
ULONG_MAXonce spfn has left the zone. Nothing is left
uninitialized: the loop covers a single gap-free interval, and because it only enters with spfn <zone_end_pfn()the escaping value is exactly SECTION_ALIGN_UP(zone_end_pfn()), which is thelast_pfnthatdeferred_init_memmap()would have used for the same pfn range. A zone that does end section-aligned now takes this path too and loses its zero-work padata job along with that node'spr_info()and theWARN_ON()on the next zone.
Quoted 7 lines
Quoted 4 lines
To reproduce with
CONFIG_DEFERRED_STRUCT_PAGE_INIT=yandCONFIG_CMA=y:qemu-system-x86_64 -enable-kvm -m 8032M -kernel bzImage \ -append "nokaslr cma=4768M@0x100000000"The only two paragraphs I understood is this and the BUG splat ;-P
Can we please have a lot more of human touch on the changelog?
Fair callout. The patch description is truly awful :(. Please excuse the slop. I wanted to send this out quickly before it keeps collecting dust :).
The problem is that with CONFIG_DEFERRED_STRUCT_PAGE_INIT enabled, we undefer some struct page ranges early in boot in deferred_grow_zone(). With a large CMA allocation in place, the early allocation we can satisfy may not span the full desired allocation and we end up undeferring all available RAM and still not satisfy the fully targeted allocation. That is perfectly fine: The function accounts for that case and leaves it to the caller to determine whether it now has sufficient available memory.
However, the function *also* remembers where deferred allocation starts next (pgdat->first_deferred_pfn). And it does so based on PAGES_PER_SECTION (128M) chunks. If the node's RAM end is not 128M aligned and we undeferred everything, then that "next" points past end of the node's RAM end.
When later deferred_init_memmap() tries to pick up from first_deferred_pfn, it runs into a BUG_ON because it expects the pfn to be within its node's RAM range. The canary for "everything is already initialized" is ULONG_MAX.
Hence this patch modifies the check in deferred_grow_zone() to already properly detect end of RAM. Instead of assuming that there is always more deferral available, we cap it at end of zone (which is equivalent to end of this node's end of RAM).
Let me send a v2 with the description above.
Alex