kswapd0: page allocation failure (bisected to "slab: add sheaves to most caches")
Subsystems (3)
Hi,
The latest mainline kernel (v6.19-11831-ga95f71ad3e2e) has page allocation failures when doing things like compiling a kernel. I can also reproduce this with a stress test likestress-ng --vm 2 --vm-bytes 110% --verify -v
[ 104.032925] kswapd0: page allocation failure: order:0, mode:0xc0c40(GFP_NOFS|__GFP_COMP|__GFP_NOMEMALLOC), nodemask=(null),cpuset=/,mems_allowed=0
[ 104.033307] CPU: 4 UID: 0 PID: 156 Comm: kswapd0 Not tainted 6.19.0-rc5-00027-g40fd0acc45d0 #435 PREEMPT(voluntary)
[ 104.033312] Hardware name: HP HP Pavilion Aero Laptop 13-be0xxx/8916, BIOS F.17 12/18/2024
[ 104.033314] Call Trace:
[ 104.033316] <TASK>
[ 104.033319] dump_stack_lvl+0x6a/0x90
[ 104.033328] warn_alloc.cold+0x95/0x1af
[ 104.033334] ? zone_watermark_ok+0x80/0x80
[ 104.033350] __alloc_frozen_pages_noprof+0xec3/0x2470
[ 104.033353] ? __lock_acquire+0x489/0x2600
[ 104.033359] ? stack_access_ok+0x1c0/0x1c0
[ 104.033367] ? warn_alloc+0x1d0/0x1d0
[ 104.033371] ? __lock_acquire+0x489/0x2600
[ 104.033375] ? _raw_spin_unlock_irqrestore+0x48/0x60
[ 104.033379] ? _raw_spin_unlock_irqrestore+0x48/0x60
[ 104.033382] ? lockdep_hardirqs_on+0x78/0x100
[ 104.033394] allocate_slab+0x2b7/0x510
[ 104.033399] refill_objects+0x25d/0x380
[ 104.033407] __pcs_replace_empty_main+0x193/0x5f0
[ 104.033412] kmem_cache_alloc_noprof+0x5b6/0x6f0
[ 104.033415] ? alloc_extent_state+0x1b/0x210 [btrfs]
[ 104.033479] alloc_extent_state+0x1b/0x210 [btrfs]
[ 104.033527] btrfs_clear_extent_bit_changeset+0x2be/0x9c0 [btrfs]
[ 104.033575] btrfs_clear_record_extent_bits+0x10/0x20 [btrfs]
[ 104.033615] btrfs_qgroup_check_reserved_leak+0xbd/0x2b0 [btrfs]
[ 104.033659] ? lock_release+0x17b/0x2a0
[ 104.033663] ? btrfs_qgroup_convert_reserved_meta+0xe90/0xe90 [btrfs]
[ 104.033703] ? do_raw_spin_unlock+0x54/0x1e0
[ 104.033707] ? _raw_spin_unlock+0x29/0x40
[ 104.033710] ? btrfs_lookup_first_ordered_extent+0x1d4/0x370 [btrfs]
[ 104.033762] ? preempt_count_add+0x73/0x140
[ 104.033768] btrfs_destroy_inode+0x301/0x6a0 [btrfs]
[ 104.033820] ? __destroy_inode+0x194/0x570
[ 104.033826] destroy_inode+0xb9/0x190
[ 104.033830] evict+0x4d8/0x900
[ 104.033832] ? lock_release+0x17b/0x2a0
[ 104.033835] ? find_held_lock+0x2b/0x80
[ 104.033839] ? destroy_inode+0x190/0x190
[ 104.033842] ? __list_lru_walk_one+0x30d/0x440
[ 104.033849] ? _raw_spin_unlock+0x29/0x40
[ 104.033851] ? __list_lru_walk_one+0x30d/0x440
[ 104.033854] ? __wait_on_freeing_inode+0x2a0/0x2a0
[ 104.033860] dispose_list+0xf0/0x1b0
[ 104.033866] prune_icache_sb+0xde/0x150
[ 104.033869] ? list_lru_count_one+0x13f/0x270
[ 104.033873] ? dump_mapping+0x250/0x250
[ 104.033875] ? lock_release+0x17b/0x2a0
[ 104.033882] super_cache_scan+0x302/0x4d0
[ 104.033889] do_shrink_slab+0x32e/0xd30
[ 104.033898] shrink_slab+0x7b6/0xda0
[ 104.033902] ? shrink_slab+0x4b1/0xda0
[ 104.033908] ? reparent_shrinker_deferred+0x330/0x330
[ 104.033914] ? trace_event_raw_event_sched_switch+0x410/0x410
[ 104.033921] shrink_node+0xac4/0x36e0
[ 104.033933] ? lru_gen_release_memcg+0x3c0/0x3c0
[ 104.033940] ? pgdat_balanced+0x15f/0x4b0
[ 104.033943] ? __cond_resched+0x23/0x30
[ 104.033950] ? balance_pgdat+0x739/0x1530
[ 104.033952] balance_pgdat+0x739/0x1530
[ 104.033960] ? shrink_node+0x36e0/0x36e0
[ 104.033962] ? __timer_delete_sync+0x177/0x240
[ 104.033966] ? __timer_delete_sync+0x177/0x240
[ 104.033970] ? _raw_spin_unlock_irqrestore+0x48/0x60
[ 104.033975] ? __lock_acquire+0x489/0x2600
[ 104.033979] ? call_timer_fn+0x3b0/0x3b0
[ 104.033981] ? schedule+0x2ba/0x390
[ 104.033990] ? lock_is_held_type+0xd5/0x130
[ 104.033997] ? kswapd+0x364/0x7f0
[ 104.034004] kswapd+0x445/0x7f0
[ 104.034010] ? balance_pgdat+0x1530/0x1530
[ 104.034013] ? _raw_spin_unlock_irqrestore+0x48/0x60
[ 104.034016] ? finish_wait+0x280/0x280
[ 104.034022] ? __kthread_parkme+0xb4/0x200
[ 104.034027] ? balance_pgdat+0x1530/0x1530
[ 104.034029] kthread+0x3ad/0x760
[ 104.034033] ? kthread_is_per_cpu+0xb0/0xb0
[ 104.034035] ? ret_from_fork+0x70/0x850
[ 104.034039] ? ret_from_fork+0x70/0x850
[ 104.034042] ? _raw_spin_unlock_irq+0x24/0x50
[ 104.034045] ? kthread_is_per_cpu+0xb0/0xb0
[ 104.034049] ret_from_fork+0x6dc/0x850
[ 104.034053] ? exit_thread+0x70/0x70
[ 104.034057] ? __switch_to+0x36f/0xd80
[ 104.034061] ? kthread_is_per_cpu+0xb0/0xb0
[ 104.034065] ret_from_fork_asm+0x11/0x20
[ 104.034077] </TASK>
[ 104.034078] Mem-Info:
[ 104.034111] active_anon:511 inactive_anon:2355672 isolated_anon:0
active_file:77595 inactive_file:204731 isolated_file:0
unevictable:7150 dirty:925 writeback:57
slab_reclaimable:20227 slab_unreclaimable:201840
mapped:121227 shmem:10197 pagetables:9634
sec_pagetables:733 bounce:0
kernel_misc_reclaimable:0
free:36223 free_pcp:529 free_cma:0
[ 104.034119] Node 0 active_anon:2044kB inactive_anon:9422688kB active_file:310380kB inactive_file:818924kB unevictable:28600kB isolated(anon):0kB isolated(file):0kB mapped:484908kB dirty:3700kB writeback:228kB shmem:40788kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:8534016kB kernel_stack:31616kB pagetables:38536kB sec_pagetables:2932kB all_unreclaimable? no Balloon:0kB
[ 104.034126] Node 0 DMA free:13316kB boost:0kB min:84kB low:104kB high:124kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB zspages:0kB present:15996kB managed:15364kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 104.034135] lowmem_reserve[]: 0 2862 11990 11990 11990
[ 104.034147] Node 0 DMA32 free:52184kB boost:0kB min:15860kB low:19824kB high:23788kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:2871780kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB zspages:0kB present:2997084kB managed:2931416kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 104.034155] lowmem_reserve[]: 0 0 9127 9127 9127
[ 104.034166] Node 0 Normal free:79392kB boost:28672kB min:80308kB low:93216kB high:106124kB reserved_highatomic:2048KB free_highatomic:32KB active_anon:2044kB inactive_anon:6550896kB active_file:310380kB inactive_file:818924kB unevictable:28600kB writepending:4252kB zspages:0kB present:13077504kB managed:9346788kB mlocked:28600kB bounce:0kB free_pcp:2116kB local_pcp:0kB free_cma:0kB
[ 104.034174] lowmem_reserve[]: 0 0 0 0 0
[ 104.034185] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13316kB
[ 104.034308] Node 0 DMA32: 3*4kB (U) 5*8kB (UM) 5*16kB (UM) 8*32kB (UM) 11*64kB (UM) 15*128kB (UM) 11*256kB (UM) 9*512kB (UM) 9*1024kB (UM) 0*2048kB 8*4096kB (UM) = 52420kB
[ 104.034348] Node 0 Normal: 1024*4kB (UMEH) 534*8kB (UEH) 409*16kB (UME) 1301*32kB (UME) 154*64kB (UME) 39*128kB (UME) 14*256kB (UM) 1*512kB (U) 2*1024kB (UM) 1*2048kB (U) 0*4096kB = 79584kB
[ 104.034390] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 104.034393] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 104.034396] 299766 total pagecache pages
[ 104.034398] 0 pages in swap cache
[ 104.034401] Free swap = 8387580kB
[ 104.034403] Total swap = 8387580kB
[ 104.034405] 4022646 pages RAM
[ 104.034407] 0 pages HighMem/MovableOnly
[ 104.034410] 949254 pages reserved
[ 104.034412] 0 pages hwpoisonedThe page allocation failures bisect to:
e47c897a29491ade20b27612fdd3107c39a07357 slab: add sheaves to most caches
#regzbot introduced: e47c897a29491ade20b27612fdd3107c39a07357 #regzbot title: kswapd0: page allocation failure
Quoted 6 lines · Chris Bainbridge
Hi,
The latest mainline kernel (v6.19-11831-ga95f71ad3e2e) has page allocation failures when doing things like compiling a kernel. I can also reproduce this with a stress test like
stress-ng --vm 2 --vm-bytes 110% --verify -v
Hi, thanks for the report!
Quoted 15 lines
[ 104.032925] kswapd0: page allocation failure: order:0, mode:0xc0c40(GFP_NOFS|__GFP_COMP|__GFP_NOMEMALLOC), nodemask=(null),cpuset=/,mems_allowed=0 [ 104.033307] CPU: 4 UID: 0 PID: 156 Comm: kswapd0 Not tainted 6.19.0-rc5-00027-g40fd0acc45d0 #435 PREEMPT(voluntary) [ 104.033312] Hardware name: HP HP Pavilion Aero Laptop 13-be0xxx/8916, BIOS F.17 12/18/2024 [ 104.033314] Call Trace: [ 104.033316] <TASK> [ 104.033319] dump_stack_lvl+0x6a/0x90 [ 104.033328] warn_alloc.cold+0x95/0x1af [ 104.033334] ? zone_watermark_ok+0x80/0x80 [ 104.033350] __alloc_frozen_pages_noprof+0xec3/0x2470 [ 104.033353] ? __lock_acquire+0x489/0x2600 [ 104.033359] ? stack_access_ok+0x1c0/0x1c0 [ 104.033367] ? warn_alloc+0x1d0/0x1d0 [ 104.033371] ? __lock_acquire+0x489/0x2600 [ 104.033375] ? _raw_spin_unlock_irqrestore+0x48/0x60 [ 104.033379] ? _raw_spin_unlock_irqrestore+0x48/0x60
Quoted 8 lines
[ 104.033382] ? lockdep_hardirqs_on+0x78/0x100 [ 104.033394] allocate_slab+0x2b7/0x510 [ 104.033399] refill_objects+0x25d/0x380 [ 104.033407] __pcs_replace_empty_main+0x193/0x5f0 [ 104.033412] kmem_cache_alloc_noprof+0x5b6/0x6f0 [ 104.033415] ? alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033479] alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033527] btrfs_clear_extent_bit_changeset+0x2be/0x9c0 [btrfs]
Hmm while bisect points out the first bad commit is commit e47c897a2949 ("slab: add sheaves to most caches"),
I think the caller is supposed to specify __GFP_NOWARN if it doesn't care about allocation failure?
btrfs_clear_extent_bit_changeset() says:
Quoted 10 lines
if (!prealloc) {
/*
* Don't care for allocation failure here because we might end
* up not needing the pre-allocated extent state at all, which
* is the case if we only have in the tree extent states that
* cover our input range and don't cover too any other range.
* If we end up needing a new extent state we allocate it later. */
prealloc =alloc_extent_state(mask);
}
Oh wait, I see what's going on. bisection pointed out the commit because slab tries to refill sheaves with __GFP_NOMEMALLOC (and then falls back to slowpath if it fails).
Since failing to refill sheaves doesn't mean the allocation will fail, it should specify __GFP_NOWARN with __GFP_NOMEMALLOC as long as there's fallback method.
But for __prefill_sheaf_pfmemalloc(), it should specify __GPF_NOWARN on the first attempt only when gfp_pfmemalloc_allowed() returns true.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 100 lines
[ 104.033575] btrfs_clear_record_extent_bits+0x10/0x20 [btrfs] [ 104.033615] btrfs_qgroup_check_reserved_leak+0xbd/0x2b0 [btrfs] [ 104.033659] ? lock_release+0x17b/0x2a0 [ 104.033663] ? btrfs_qgroup_convert_reserved_meta+0xe90/0xe90 [btrfs] [ 104.033703] ? do_raw_spin_unlock+0x54/0x1e0 [ 104.033707] ? _raw_spin_unlock+0x29/0x40 [ 104.033710] ? btrfs_lookup_first_ordered_extent+0x1d4/0x370 [btrfs] [ 104.033762] ? preempt_count_add+0x73/0x140 [ 104.033768] btrfs_destroy_inode+0x301/0x6a0 [btrfs] [ 104.033820] ? __destroy_inode+0x194/0x570 [ 104.033826] destroy_inode+0xb9/0x190 [ 104.033830] evict+0x4d8/0x900 [ 104.033832] ? lock_release+0x17b/0x2a0 [ 104.033835] ? find_held_lock+0x2b/0x80 [ 104.033839] ? destroy_inode+0x190/0x190 [ 104.033842] ? __list_lru_walk_one+0x30d/0x440 [ 104.033849] ? _raw_spin_unlock+0x29/0x40 [ 104.033851] ? __list_lru_walk_one+0x30d/0x440 [ 104.033854] ? __wait_on_freeing_inode+0x2a0/0x2a0 [ 104.033860] dispose_list+0xf0/0x1b0 [ 104.033866] prune_icache_sb+0xde/0x150 [ 104.033869] ? list_lru_count_one+0x13f/0x270 [ 104.033873] ? dump_mapping+0x250/0x250 [ 104.033875] ? lock_release+0x17b/0x2a0 [ 104.033882] super_cache_scan+0x302/0x4d0 [ 104.033889] do_shrink_slab+0x32e/0xd30 [ 104.033898] shrink_slab+0x7b6/0xda0 [ 104.033902] ? shrink_slab+0x4b1/0xda0 [ 104.033908] ? reparent_shrinker_deferred+0x330/0x330 [ 104.033914] ? trace_event_raw_event_sched_switch+0x410/0x410 [ 104.033921] shrink_node+0xac4/0x36e0 [ 104.033933] ? lru_gen_release_memcg+0x3c0/0x3c0 [ 104.033940] ? pgdat_balanced+0x15f/0x4b0 [ 104.033943] ? __cond_resched+0x23/0x30 [ 104.033950] ? balance_pgdat+0x739/0x1530 [ 104.033952] balance_pgdat+0x739/0x1530 [ 104.033960] ? shrink_node+0x36e0/0x36e0 [ 104.033962] ? __timer_delete_sync+0x177/0x240 [ 104.033966] ? __timer_delete_sync+0x177/0x240 [ 104.033970] ? _raw_spin_unlock_irqrestore+0x48/0x60 [ 104.033975] ? __lock_acquire+0x489/0x2600 [ 104.033979] ? call_timer_fn+0x3b0/0x3b0 [ 104.033981] ? schedule+0x2ba/0x390 [ 104.033990] ? lock_is_held_type+0xd5/0x130 [ 104.033997] ? kswapd+0x364/0x7f0 [ 104.034004] kswapd+0x445/0x7f0 [ 104.034010] ? balance_pgdat+0x1530/0x1530 [ 104.034013] ? _raw_spin_unlock_irqrestore+0x48/0x60 [ 104.034016] ? finish_wait+0x280/0x280 [ 104.034022] ? __kthread_parkme+0xb4/0x200 [ 104.034027] ? balance_pgdat+0x1530/0x1530 [ 104.034029] kthread+0x3ad/0x760 [ 104.034033] ? kthread_is_per_cpu+0xb0/0xb0 [ 104.034035] ? ret_from_fork+0x70/0x850 [ 104.034039] ? ret_from_fork+0x70/0x850 [ 104.034042] ? _raw_spin_unlock_irq+0x24/0x50 [ 104.034045] ? kthread_is_per_cpu+0xb0/0xb0 [ 104.034049] ret_from_fork+0x6dc/0x850 [ 104.034053] ? exit_thread+0x70/0x70 [ 104.034057] ? __switch_to+0x36f/0xd80 [ 104.034061] ? kthread_is_per_cpu+0xb0/0xb0 [ 104.034065] ret_from_fork_asm+0x11/0x20 [ 104.034077] </TASK> [ 104.034078] Mem-Info: [ 104.034111] active_anon:511 inactive_anon:2355672 isolated_anon:0 active_file:77595 inactive_file:204731 isolated_file:0 unevictable:7150 dirty:925 writeback:57 slab_reclaimable:20227 slab_unreclaimable:201840 mapped:121227 shmem:10197 pagetables:9634 sec_pagetables:733 bounce:0 kernel_misc_reclaimable:0 free:36223 free_pcp:529 free_cma:0 [ 104.034119] Node 0 active_anon:2044kB inactive_anon:9422688kB active_file:310380kB inactive_file:818924kB unevictable:28600kB isolated(anon):0kB isolated(file):0kB mapped:484908kB dirty:3700kB writeback:228kB shmem:40788kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:8534016kB kernel_stack:31616kB pagetables:38536kB sec_pagetables:2932kB all_unreclaimable? no Balloon:0kB [ 104.034126] Node 0 DMA free:13316kB boost:0kB min:84kB low:104kB high:124kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB zspages:0kB present:15996kB managed:15364kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 104.034135] lowmem_reserve[]: 0 2862 11990 11990 11990 [ 104.034147] Node 0 DMA32 free:52184kB boost:0kB min:15860kB low:19824kB high:23788kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:2871780kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB zspages:0kB present:2997084kB managed:2931416kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 104.034155] lowmem_reserve[]: 0 0 9127 9127 9127 [ 104.034166] Node 0 Normal free:79392kB boost:28672kB min:80308kB low:93216kB high:106124kB reserved_highatomic:2048KB free_highatomic:32KB active_anon:2044kB inactive_anon:6550896kB active_file:310380kB inactive_file:818924kB unevictable:28600kB writepending:4252kB zspages:0kB present:13077504kB managed:9346788kB mlocked:28600kB bounce:0kB free_pcp:2116kB local_pcp:0kB free_cma:0kB [ 104.034174] lowmem_reserve[]: 0 0 0 0 0 [ 104.034185] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13316kB [ 104.034308] Node 0 DMA32: 3*4kB (U) 5*8kB (UM) 5*16kB (UM) 8*32kB (UM) 11*64kB (UM) 15*128kB (UM) 11*256kB (UM) 9*512kB (UM) 9*1024kB (UM) 0*2048kB 8*4096kB (UM) = 52420kB [ 104.034348] Node 0 Normal: 1024*4kB (UMEH) 534*8kB (UEH) 409*16kB (UME) 1301*32kB (UME) 154*64kB (UME) 39*128kB (UME) 14*256kB (UM) 1*512kB (U) 2*1024kB (UM) 1*2048kB (U) 0*4096kB = 79584kB [ 104.034390] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 104.034393] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 104.034396] 299766 total pagecache pages [ 104.034398] 0 pages in swap cache [ 104.034401] Free swap = 8387580kB [ 104.034403] Total swap = 8387580kB [ 104.034405] 4022646 pages RAM [ 104.034407] 0 pages HighMem/MovableOnly [ 104.034410] 949254 pages reserved [ 104.034412] 0 pages hwpoisonedThe page allocation failures bisect to:
e47c897a29491ade20b27612fdd3107c39a07357slab: add sheaves to most caches#regzbot introduced:
e47c897a29491ade20b27612fdd3107c39a07357#regzbot title: kswapd0: page allocation failure
Quoted 56 lines · Harry Yoo
Quoted 6 lines · Chris Bainbridge
Hi,
The latest mainline kernel (v6.19-11831-ga95f71ad3e2e) has page allocation failures when doing things like compiling a kernel. I can also reproduce this with a stress test like
stress-ng --vm 2 --vm-bytes 110% --verify -vHi, thanks for the report!
Quoted 15 lines
[ 104.032925] kswapd0: page allocation failure: order:0, mode:0xc0c40(GFP_NOFS|__GFP_COMP|__GFP_NOMEMALLOC), nodemask=(null),cpuset=/,mems_allowed=0 [ 104.033307] CPU: 4 UID: 0 PID: 156 Comm: kswapd0 Not tainted 6.19.0-rc5-00027-g40fd0acc45d0 #435 PREEMPT(voluntary) [ 104.033312] Hardware name: HP HP Pavilion Aero Laptop 13-be0xxx/8916, BIOS F.17 12/18/2024 [ 104.033314] Call Trace: [ 104.033316] <TASK> [ 104.033319] dump_stack_lvl+0x6a/0x90 [ 104.033328] warn_alloc.cold+0x95/0x1af [ 104.033334] ? zone_watermark_ok+0x80/0x80 [ 104.033350] __alloc_frozen_pages_noprof+0xec3/0x2470 [ 104.033353] ? __lock_acquire+0x489/0x2600 [ 104.033359] ? stack_access_ok+0x1c0/0x1c0 [ 104.033367] ? warn_alloc+0x1d0/0x1d0 [ 104.033371] ? __lock_acquire+0x489/0x2600 [ 104.033375] ? _raw_spin_unlock_irqrestore+0x48/0x60 [ 104.033379] ? _raw_spin_unlock_irqrestore+0x48/0x60Quoted 8 lines
[ 104.033382] ? lockdep_hardirqs_on+0x78/0x100 [ 104.033394] allocate_slab+0x2b7/0x510 [ 104.033399] refill_objects+0x25d/0x380 [ 104.033407] __pcs_replace_empty_main+0x193/0x5f0 [ 104.033412] kmem_cache_alloc_noprof+0x5b6/0x6f0 [ 104.033415] ? alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033479] alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033527] btrfs_clear_extent_bit_changeset+0x2be/0x9c0 [btrfs]Hmm while bisect points out the first bad commit is commit
e47c897a2949("slab: add sheaves to most caches"),I think the caller is supposed to specify
__GFP_NOWARNif it doesn't care about allocation failure?
btrfs_clear_extent_bit_changeset()says:Quoted 10 lines
if (!prealloc) {
/*
* Don't care for allocation failure here because we might end
* up not needing the pre-allocated extent state at all, which
* is the case if we only have in the tree extent states that
* cover our input range and don't cover too any other range.
* If we end up needing a new extent state we allocate it later. */
prealloc =alloc_extent_state(mask);
}Oh wait, I see what's going on. bisection pointed out the commit because slab tries to refill sheaves with
__GFP_NOMEMALLOC(and then falls back to slowpath if it fails).
Quoted 6 lines
Since failing to refill sheaves doesn't mean the allocation will fail, it should specify
__GFP_NOWARNwith__GFP_NOMEMALLOCas long as there's fallback method.But for
__prefill_sheaf_pfmemalloc(), it should specify__GPF_NOWARNon the first attempt only whengfp_pfmemalloc_allowed()returns true.
Is this fix sufficient to do the right thing? I tested it, and it does appear to prevent logging of the allocation failures for my test case.
fs/btrfs/extent-io-tree.c +1 -1
@@ -641,7 +641,7 @@ int btrfs_clear_extent_bit_changeset(struct extent_io_tree *tree, u64 start, u64* cover our input range and don't cover too any other range. * If we end up needing a new extent state we allocate it later. */ - prealloc =alloc_extent_state (mask); + prealloc =alloc_extent_state (mask | __GFP_NOWARN); }spin_lock (&tree->lock);
Quoted 61 lines · Chris Bainbridge
Quoted 60 lines · Harry Yoo
Quoted 6 lines · Chris Bainbridge
Hi,
The latest mainline kernel (v6.19-11831-ga95f71ad3e2e) has page allocation failures when doing things like compiling a kernel. I can also reproduce this with a stress test like
stress-ng --vm 2 --vm-bytes 110% --verify -vHi, thanks for the report!
Quoted 15 lines
[ 104.032925] kswapd0: page allocation failure: order:0, mode:0xc0c40(GFP_NOFS|__GFP_COMP|__GFP_NOMEMALLOC), nodemask=(null),cpuset=/,mems_allowed=0 [ 104.033307] CPU: 4 UID: 0 PID: 156 Comm: kswapd0 Not tainted 6.19.0-rc5-00027-g40fd0acc45d0 #435 PREEMPT(voluntary) [ 104.033312] Hardware name: HP HP Pavilion Aero Laptop 13-be0xxx/8916, BIOS F.17 12/18/2024 [ 104.033314] Call Trace: [ 104.033316] <TASK> [ 104.033319] dump_stack_lvl+0x6a/0x90 [ 104.033328] warn_alloc.cold+0x95/0x1af [ 104.033334] ? zone_watermark_ok+0x80/0x80 [ 104.033350] __alloc_frozen_pages_noprof+0xec3/0x2470 [ 104.033353] ? __lock_acquire+0x489/0x2600 [ 104.033359] ? stack_access_ok+0x1c0/0x1c0 [ 104.033367] ? warn_alloc+0x1d0/0x1d0 [ 104.033371] ? __lock_acquire+0x489/0x2600 [ 104.033375] ? _raw_spin_unlock_irqrestore+0x48/0x60 [ 104.033379] ? _raw_spin_unlock_irqrestore+0x48/0x60Quoted 8 lines
[ 104.033382] ? lockdep_hardirqs_on+0x78/0x100 [ 104.033394] allocate_slab+0x2b7/0x510 [ 104.033399] refill_objects+0x25d/0x380 [ 104.033407] __pcs_replace_empty_main+0x193/0x5f0 [ 104.033412] kmem_cache_alloc_noprof+0x5b6/0x6f0 [ 104.033415] ? alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033479] alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033527] btrfs_clear_extent_bit_changeset+0x2be/0x9c0 [btrfs]Hmm while bisect points out the first bad commit is commit
e47c897a2949("slab: add sheaves to most caches"),I think the caller is supposed to specify
__GFP_NOWARNif it doesn't care about allocation failure?
btrfs_clear_extent_bit_changeset()says:Quoted 10 lines
if (!prealloc) {
/*
* Don't care for allocation failure here because we might end
* up not needing the pre-allocated extent state at all, which
* is the case if we only have in the tree extent states that
* cover our input range and don't cover too any other range.
* If we end up needing a new extent state we allocate it later. */
prealloc =alloc_extent_state(mask);
}Oh wait, I see what's going on. bisection pointed out the commit because slab tries to refill sheaves with
__GFP_NOMEMALLOC(and then falls back to slowpath if it fails).Since failing to refill sheaves doesn't mean the allocation will fail, it should specify
__GFP_NOWARNwith__GFP_NOMEMALLOCas long as there's fallback method.
Quoted 5 lines
Quoted 2 lines
But for
__prefill_sheaf_pfmemalloc(), it should specify__GPF_NOWARNon the first attempt only whengfp_pfmemalloc_allowed()returns true.Is this fix sufficient to do the right thing? I tested it, and it does appear to prevent logging of the allocation failures for my test case.
I think we should do both both 1) setting __GFP_NOWARN from btrfs side and 2) making slab try to refill sheaves with __GFP_NOWARN when there's a fallback path.
I'm writing a fix for 2) and I'll send it soon.
fs/btrfs/extent-io-tree.c int btrfs_clear_extent_bit_changeset(struct extent_io_tree *tree, u64 start, u64 · line 644
Quoted 10 lines
fs/btrfs/extent-io-tree.c +1
-1@@ -641,7 +641,7 @@ int btrfs_clear_extent_bit_changeset(struct extent_io_tree *tree, u64 start, u64* cover our input range and don't cover too any other range. * If we end up needing a new extent state we allocate it later. */ - prealloc =alloc_extent_state (mask); + prealloc =alloc_extent_state (mask | __GFP_NOWARN);
This seems to be a right thing to do to me, but as I'm not familiar with btrfs, I'll let btrfs folks leave comment on it :)
Quoted 3 lines
fs/btrfs/extent-io-tree.c +0
-0}spin_lock (&tree->lock);
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 74 lines · Harry Yoo
Quoted 61 lines · Chris Bainbridge
Quoted 60 lines · Harry Yoo
Quoted 6 lines · Chris Bainbridge
Hi,
The latest mainline kernel (v6.19-11831-ga95f71ad3e2e) has page allocation failures when doing things like compiling a kernel. I can also reproduce this with a stress test like
stress-ng --vm 2 --vm-bytes 110% --verify -vHi, thanks for the report!
Quoted 15 lines
[ 104.032925] kswapd0: page allocation failure: order:0, mode:0xc0c40(GFP_NOFS|__GFP_COMP|__GFP_NOMEMALLOC), nodemask=(null),cpuset=/,mems_allowed=0 [ 104.033307] CPU: 4 UID: 0 PID: 156 Comm: kswapd0 Not tainted 6.19.0-rc5-00027-g40fd0acc45d0 #435 PREEMPT(voluntary) [ 104.033312] Hardware name: HP HP Pavilion Aero Laptop 13-be0xxx/8916, BIOS F.17 12/18/2024 [ 104.033314] Call Trace: [ 104.033316] <TASK> [ 104.033319] dump_stack_lvl+0x6a/0x90 [ 104.033328] warn_alloc.cold+0x95/0x1af [ 104.033334] ? zone_watermark_ok+0x80/0x80 [ 104.033350] __alloc_frozen_pages_noprof+0xec3/0x2470 [ 104.033353] ? __lock_acquire+0x489/0x2600 [ 104.033359] ? stack_access_ok+0x1c0/0x1c0 [ 104.033367] ? warn_alloc+0x1d0/0x1d0 [ 104.033371] ? __lock_acquire+0x489/0x2600 [ 104.033375] ? _raw_spin_unlock_irqrestore+0x48/0x60 [ 104.033379] ? _raw_spin_unlock_irqrestore+0x48/0x60Quoted 8 lines
[ 104.033382] ? lockdep_hardirqs_on+0x78/0x100 [ 104.033394] allocate_slab+0x2b7/0x510 [ 104.033399] refill_objects+0x25d/0x380 [ 104.033407] __pcs_replace_empty_main+0x193/0x5f0 [ 104.033412] kmem_cache_alloc_noprof+0x5b6/0x6f0 [ 104.033415] ? alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033479] alloc_extent_state+0x1b/0x210 [btrfs] [ 104.033527] btrfs_clear_extent_bit_changeset+0x2be/0x9c0 [btrfs]Hmm while bisect points out the first bad commit is commit
e47c897a2949("slab: add sheaves to most caches"),I think the caller is supposed to specify
__GFP_NOWARNif it doesn't care about allocation failure?
btrfs_clear_extent_bit_changeset()says:Quoted 10 lines
if (!prealloc) {
/*
* Don't care for allocation failure here because we might end
* up not needing the pre-allocated extent state at all, which
* is the case if we only have in the tree extent states that
* cover our input range and don't cover too any other range.
* If we end up needing a new extent state we allocate it later. */
prealloc =alloc_extent_state(mask);
}Oh wait, I see what's going on. bisection pointed out the commit because slab tries to refill sheaves with
__GFP_NOMEMALLOC(and then falls back to slowpath if it fails).Since failing to refill sheaves doesn't mean the allocation will fail, it should specify
__GFP_NOWARNwith__GFP_NOMEMALLOCas long as there's fallback method.Quoted 5 lines
Quoted 2 lines
But for
__prefill_sheaf_pfmemalloc(), it should specify__GPF_NOWARNon the first attempt only whengfp_pfmemalloc_allowed()returns true.Is this fix sufficient to do the right thing? I tested it, and it does appear to prevent logging of the allocation failures for my test case.
I think we should do both both 1) setting
__GFP_NOWARNfrom btrfs side and 2) making slab try to refill sheaves with__GFP_NOWARNwhen there's a fallback path.I'm writing a fix for 2) and I'll send it soon.
Quoted 13 lines
fs/btrfs/extent-io-tree.c int btrfs_clear_extent_bit_changeset(struct extent_io_tree *tree, u64 start, u64 · line 644
Quoted 10 lines
fs/btrfs/extent-io-tree.c +1
-1@@ -641,7 +641,7 @@ int btrfs_clear_extent_bit_changeset(struct extent_io_tree *tree, u64 start, u64* cover our input range and don't cover too any other range. * If we end up needing a new extent state we allocate it later. */ - prealloc =alloc_extent_state (mask); + prealloc =alloc_extent_state (mask | __GFP_NOWARN);This seems to be a right thing to do to me, but as I'm not familiar with btrfs, I'll let btrfs folks leave comment on it :)
I agree the flag should be added, as the comment explains allocation failures are not fatal at this place. There's another call to the alloc_extent_state() with GFP_ATOMIC so we cannot simply sink NOWARN there.
Mikhail Gavrilov
Tested-byI can confirm this issue on my system:
Hardware: ASUS ROG STRIX B650E-I (AMD Ryzen), 64GB RAM
Storage: btrfs with zram swap
Kernel: 7.0.0-rc1 (commit 6de23f81a5e0+)
I was seeing the same kswapd0 page allocation failures periodically under memory pressure with the identical call trace through alloc_from_pcs -> __pcs_replace_empty_main -> refill_objects -> allocate_slab.
Chris's btrfs __GFP_NOWARN patch suppresses the btrfs-originated warnings, but after ~10 hours I hit the same sheaf refill failure from a different caller -- amdgpu via kmalloc:
chrome: page allocation failure: order:0, mode:0xc0cc0(GFP_KERNEL|__GFP_COMP|__GFP_NOMEMALLOC)
allocate_slab
refill_objects
__pcs_replace_empty_main
__kmalloc_cache_noprof
drm_suballoc_new
amdgpu_sa_bo_new
This confirms the fix needs to be on the slab side as Harry suggested -- adding __GFP_NOWARN to sheaf refill when there's a fallback path -- rather than patching individual callers.
Happy to test any slab-side fix.
Quoted 18 lines · Mikhail Gavrilov
I can confirm this issue on my system:
Hardware: ASUS ROG STRIX B650E-I (AMD Ryzen), 64GB RAM
Storage: btrfs with zram swap
Kernel: 7.0.0-rc1 (commit 6de23f81a5e0+)I was seeing the same kswapd0 page allocation failures periodically under memory pressure with the identical call trace through
alloc_from_pcs->__pcs_replace_empty_main->refill_objects->allocate_slab.Chris's btrfs
__GFP_NOWARNpatch suppresses the btrfs-originated warnings, but after ~10 hours I hit the same sheaf refill failure from a different caller -- amdgpu via kmalloc:chrome: page allocation failure: order:0, mode:0xc0cc0(GFP_KERNEL|__GFP_COMP|__GFP_NOMEMALLOC)
Quoted 6 lines
allocate_slab
refill_objects
__pcs_replace_empty_main
__kmalloc_cache_noprof
drm_suballoc_new
amdgpu_sa_bo_new
Hi Mikhail,
In this case __GFP_NOWARN is not specified, so indeed we need slab side fix.
Quoted 3 lines
This confirms the fix needs to be on the slab side as Harry suggested -- adding
__GFP_NOWARNto sheaf refill when there's a fallback path -- rather than patching individual callers.
Thanks for testing and sharing the result!
That's very helpful.
Quoted 1 line
Happy to test any slab-side fix.
Slab fix is submitted here, please feel free to test:
https://lore.kernel.org/linux-mm/[email protected]
Signature · 2 lines
Cheers, Harry / Hyeonggon
Mikhail Gavrilov
Tested-byQuoted 3 lines · Harry Yoo
Slab fix is submitted here, please feel free to test:
No page allocation failures after 33 hours of uptime under normal desktop workload with memory pressure (64GB RAM, btrfs + zram swap, Chrome with many tabs, AMD GPU).
Previously without this patch, I was seeing failures from both kswapd0/btrfs and chrome/amdgpu callers within 10 hours.
Kernel: 7.0.0-rc1 (commit 7dff99b35460 + this patch)
Hardware: ASUS ROG STRIX B650E-I, AMD Ryzen, 64GB RAM
Quoted 9 lines · Mikhail Gavrilov
Quoted 3 lines · Harry Yoo
Slab fix is submitted here, please feel free to test:
No page allocation failures after 33 hours of uptime under normal desktop workload with memory pressure (64GB RAM, btrfs + zram swap, Chrome with many tabs, AMD GPU).
Quoted 7 lines
Previously without this patch, I was seeing failures from both kswapd0/btrfs and chrome/amdgpu callers within 10 hours.
Kernel: 7.0.0-rc1 (commit
7dff99b35460+ this patch)
Hardware: ASUS ROG STRIX B650E-I, AMD Ryzen, 64GB RAMTested-by Mikhail Gavrilov <[email protected]>
Thanks, added to the commit.