kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc)
Subsystems (3)
Greetings!
Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [effe5cc0] [c0784b64] dump_stack_lvl+0x80/0xac (unreliable) [effe5ce0] [c01aef80] warn_alloc+0x100/0x178 [effe5d40] [c01af34c] __alloc_pages+0x354/0x8ac [effe5e00] [c01af988] page_frag_alloc_align+0x68/0x17c [effe5e30] [c063b7b4] __netdev_alloc_skb+0xb4/0x17c [effe5e60] [be99573c] setup_rx_descbuffer+0x40/0x144 [b43legacy] [effe5e90] [be99687c] b43legacy_dma_rx+0x20c/0x278 [b43legacy] [effe5ee0] [be9897f4] b43legacy_interrupt_tasklet+0x580/0x5a4 [b43legacy] [effe5f50] [c0050b34] tasklet_action_common.isra.0+0xb0/0xe8 [effe5f80] [c07aaf5c] __do_softirq+0x1dc/0x218 [effe5ff0] [c0008820] do_softirq_own_stack+0x34/0x40
[c1c555f0] [c07ff5f0] 0xc07ff5f0
[c1c55610] [c0050384] __irq_exit_rcu+0x6c/0xbc [c1c55620] [c00507f8] irq_exit+0x10/0x20 [c1c55630] [c00045b4] HardwareInterrupt_virt+0x108/0x10c --- interrupt: 500 at BIT_flushBits+0x1c/0x58
NIP: c0458840 LR: c0458fdc CTR: 00000000 REGS: c1c55640 TRAP: 0500 Not tainted (6.8.9-gentoo-PMacG4) MSR: 0220b032 <VEC,EE,FP,ME,IR,DR,RI> CR: 88082202 XER: 00000000
GPR00: c0458f98 c1c55700 c1c2b9a0 c1c5575c 00000000 00000000 00000000 00000000
GPR08: 00000000 00000000 f2c1d0d5 c1c55710 88084202 00000000 f2c1e000 f35f8150
GPR16: 00000000 f34bf238 f34bf30c 00000000 0000001b 00000002 00000000 00000000
GPR24: f35f8150 00000080 f35f7950 f35f7d50 00000000 c07ff5f0 00000052 f35f5be0
NIP [c0458840] BIT_flushBits+0x1c/0x58
LR [c0458fdc] ZSTD_encodeSequences+0x2ac/0x308
--- interrupt: 500
[c1c55700] [c07ff5f0] 0xc07ff5f0 (unreliable)
[c1c55710] [c0458f98] ZSTD_encodeSequences+0x268/0x308
[c1c557b0] [c0452eb0] ZSTD_entropyCompressSeqStore.constprop.0+0x1c4/0x2bc
[c1c55840] [c045357c] ZSTD_compressBlock_internal+0xac/0x144
[c1c55870] [c04543c8] ZSTD_compressContinue_internal+0x734/0x7c0
[c1c55920] [c04559d4] ZSTD_compressEnd+0x2c/0x13c
[c1c55950] [c04574b0] ZSTD_compressStream2+0x1b8/0x508
[c1c559a0] [c045788c] ZSTD_compressStream2_simpleArgs+0x48/0x60
[c1c559e0] [c0457904] ZSTD_compress2+0x60/0x90
[c1c55a10] [c03d0abc] __zstd_compress+0x54/0x78
[c1c55a70] [c03c3794] scomp_acomp_comp_decomp+0xe8/0x16c
[c1c55aa0] [c01c4c80] zswap_store+0x4b4/0x668
[c1c55b20] [c01bc404] swap_writepage+0x3c/0xa8
[c1c55b40] [c0172dd4] pageout+0x11c/0x1ec
[c1c55bc0] [c0174e2c] shrink_folio_list+0x6b0/0x878
[c1c55c40] [c01767e8] evict_folios+0x9d4/0xd08
[c1c55d40] [c0176cec] try_to_shrink_lruvec+0x1d0/0x210
[c1c55da0] [c0176dcc] shrink_one+0xa0/0x134
[c1c55dd0] [c01786a4] shrink_node+0x248/0x844
[c1c55e50] [c0178f68] balance_pgdat+0x2c8/0x614
[c1c55f50] [c01794cc] kswapd+0x218/0x24c
[c1c55fc0] [c006aae4] kthread+0xe4/0xe8
[c1c55ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:247864 inactive_anon:215615 isolated_anon:0
active_file:9633 inactive_file:12323 isolated_file:0
unevictable:4 dirty:0 writeback:2
slab_reclaimable:1293 slab_unreclaimable:6904
mapped:16876 shmem:155 pagetables:899
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2224 free_pcp:596 free_cma:0
Node 0 active_anon:991456kB inactive_anon:862460kB active_file:38532kB inactive_file:49292kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:67504kB dirty:0kB writeback:8kB shmem:620kB writeback_tmp:0kB kernel_stack:1536kB pagetables:3596kB sec_pagetables:0kB all_unreclaimable? no DMA free:1680kB boost:4096kB min:7548kB low:8408kB high:9268kB reserved_highatomic:0KB active_anon:675928kB inactive_anon:5644kB active_file:124kB inactive_file:596kB unevictable:0kB writepending:0kB present:786432kB managed:746656kB mlocked:0kB bounce:0kB free_pcp:2384kB local_pcp:1004kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 18*32kB (UME) 7*64kB (UM) 0*128kB 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1824kB
35907 total pagecache pages
13813 pages in swap cache
Free swap = 8270180kB
[...]
There was no crash though, the G4 was still usable via VNC.
Full dmesg and kernel .config attached.
Regards,
Erhard
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:
Greetings!
Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":
[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c [c1c65fc0] [c006bbac] kthread+0xe4/0xe8 [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c
slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.
Full dmesg and kernel .config attached.
Regards,
Erhard
Quoted 244 lines · Erhard Furtner
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:
Greetings!
Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMacVery similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":
[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c [c1c65fc0] [c006bbac] kthread+0xe4/0xe8 [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kBall_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c
slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac]kthread+0xe4/0xe8
[c1c65ff0] [c0015304]start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kBall_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4]list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00]zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c]do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40]shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c]shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4]shrink_node+0x238/0x84c
[c1c65e50] [c017ed38]balance_pgdat+0x340/0x650
[c1c65f50] [c017f270]kswapd+0x228/0x25c
[c1c65fc0] [c006bbac]kthread+0xe4/0xe8
[c1c65ff0] [c0015304]start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kBall_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
Quoted 6 lines
I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.
Full dmesg and kernel .config attached.
Regards,
Erhard
Hi Erhard,
Thanks for the reports. I'll take a look at them and get back to you in a few days.
Quoted 250 lines · Yu Zhao
Quoted 248 lines · Erhard Furtner
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:
Greetings!
Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMacVery similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":
[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c [c1c65fc0] [c006bbac] kthread+0xe4/0xe8 [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kBall_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c
slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac]kthread+0xe4/0xe8
[c1c65ff0] [c0015304]start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kBall_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4]list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00]zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c]do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40]shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c]shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4]shrink_node+0x238/0x84c
[c1c65e50] [c017ed38]balance_pgdat+0x340/0x650
[c1c65f50] [c017f270]kswapd+0x228/0x25c
[c1c65fc0] [c006bbac]kthread+0xe4/0xe8
[c1c65ff0] [c0015304]start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kBall_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedI switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.
Full dmesg and kernel .config attached.
Quoted 7 lines
Quoted 2 lines
Regards,
ErhardHi Erhard,
Thanks for the reports. I'll take a look at them and get back to you in a few days.
Hi Erhard,
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]
The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.
Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)
Thanks!
Quoted 268 lines · Yu Zhao
Quoted 250 lines · Yu Zhao
Quoted 248 lines · Erhard Furtner
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:
Greetings!
Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMacVery similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":
[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c [c1c65fc0] [c006bbac] kthread+0xe4/0xe8 [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kBall_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154 [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4 [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304 [c1c65d20] [c0180a40] shrink_slab+0x174/0x260 [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134 [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650 [c1c65f50] [c017f270] kswapd+0x228/0x25c
slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac]kthread+0xe4/0xe8
[c1c65ff0] [c0015304]start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kBall_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedwarn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1 Hardware name: PowerMac3,6 7455 0x80010303 PowerMac Call Trace: [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable) [c1c65960] [c01b6234] warn_alloc+0x100/0x178 [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0 [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154 [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache:skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4]list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00]zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c]do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40]shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c]shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4]shrink_node+0x238/0x84c
[c1c65e50] [c017ed38]balance_pgdat+0x340/0x650
[c1c65f50] [c017f270]kswapd+0x228/0x25c
[c1c65fc0] [c006bbac]kthread+0xe4/0xe8
[c1c65ff0] [c0015304]start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kBall_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reservedI switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.
Full dmesg and kernel .config attached.
Quoted 7 lines
Quoted 2 lines
Regards,
ErhardHi Erhard,
Thanks for the reports. I'll take a look at them and get back to you in a few days.
Hi Erhard,
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
Quoted 4 lines
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]
That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...
Signature · 3 lines
Cheers, David / dhildenb
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Yu Zhao
On Wed, May 15, 2024 at 4:06 PM Yu Zhao <[email protected]> wrote:
...
Quoted 6 lines
Quoted 6 lines
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
Quoted 7 lines
Quoted 4 lines
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...
The DMA zone exists because 9739ab7eda45 ("powerpc: enable a 30-bit ZONE_DMA for 32-bit pmac") selects it.
It's 768MB (not 0.25GB) because it's clamped at max_low_pfn:
#ifdef CONFIG_ZONE_DMA max_zone_pfns[ZONE_DMA] = min(max_low_pfn, 1UL << (zone_dma_bits - PAGE_SHIFT));
#endif
Which comes eventually from CONFIG_LOWMEM_SIZE, which defaults to 768MB.
I think it's 768MB because the user:kernel split is 3G:1G, and then the kernel needs some of that 1G virtual space for vmalloc/ioremap/highmem, so it splits it 768M:256M.
Then ZONE_NORMAL is empty because it is also limited to max_low_pfn:
max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
The rest of RAM is highmem.
So I think that's all behaving as expected, but I don't know 32-bit / highmem stuff that well so I could be wrong.
cheers
Quoted 16 lines · Michael Ellerman
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Yu Zhao
On Wed, May 15, 2024 at 4:06 PM Yu Zhao <[email protected]> wrote:
...
Quoted 11 lines
Quoted 10 lines
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]
Quoted 7 lines
Quoted 2 lines
That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...
The DMA zone exists because
9739ab7eda45("powerpc: enable a 30-bitZONE_DMAfor 32-bit pmac") selects it.It's 768MB (not 0.25GB) because it's clamped at
max_low_pfn:
Right. (I meant 0.75GB.)
Quoted 6 lines
#ifdef
CONFIG_ZONE_DMA
max_zone_pfns[ZONE_DMA] = min(max_low_pfn,
1UL << (zone_dma_bits-PAGE_SHIFT));
#endifWhich comes eventually from
CONFIG_LOWMEM_SIZE, which defaults to 768MB.
I see. I grep'ed VMSPLIT which is used on x86 and arm but apparently not on powerpc.
Quoted 4 lines
I think it's 768MB because the user:kernel split is 3G:1G, and then the kernel needs some of that 1G virtual space for vmalloc/ioremap/highmem, so it splits it 768M:256M.
Quoted 8 lines
Then
ZONE_NORMALis empty because it is also limited tomax_low_pfn:max_zone_pfns[ZONE_NORMAL] = max_low_pfn;The rest of RAM is highmem.
So I think that's all behaving as expected, but I don't know 32-bit / highmem stuff that well so I could be wrong.
Yes, the three zones work as intended.
Erhard,
Since your system only has 2GB memory, I'd try the 2G:2G split, which would in theory allow both the kernel and userspace to all memory.
CONFIG_LOWMEM_SIZE_BOOL=yCONFIG_LOWMEM_SIZE=0x7000000
(Michael, please correct me if the above wouldn't work.)
Quoted 32 lines · Yu Zhao
Quoted 16 lines · Michael Ellerman
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Yu Zhao
On Wed, May 15, 2024 at 4:06 PM Yu Zhao <[email protected]> wrote:
...
Quoted 11 lines
Quoted 10 lines
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]Quoted 7 lines
Quoted 2 lines
That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...
The DMA zone exists because
9739ab7eda45("powerpc: enable a 30-bitZONE_DMAfor 32-bit pmac") selects it.It's 768MB (not 0.25GB) because it's clamped at
max_low_pfn:Right. (I meant 0.75GB.)
Quoted 5 lines
#ifdef
CONFIG_ZONE_DMA
max_zone_pfns[ZONE_DMA] = min(max_low_pfn,
1UL << (zone_dma_bits-PAGE_SHIFT));
#endif
Quoted 4 lines
Quoted 1 line
Which comes eventually from
CONFIG_LOWMEM_SIZE, which defaults to 768MB.I see. I grep'ed VMSPLIT which is used on x86 and arm but apparently not on powerpc.
Those VMSPLIT configs are nice, on powerpc it's all done manually :}
Quoted 17 lines
Quoted 4 lines
I think it's 768MB because the user:kernel split is 3G:1G, and then the kernel needs some of that 1G virtual space for vmalloc/ioremap/highmem, so it splits it 768M:256M.
Quoted 8 lines
Then
ZONE_NORMALis empty because it is also limited tomax_low_pfn:max_zone_pfns[ZONE_NORMAL] = max_low_pfn;The rest of RAM is highmem.
So I think that's all behaving as expected, but I don't know 32-bit / highmem stuff that well so I could be wrong.
Yes, the three zones work as intended.
Erhard,
Quoted 7 lines
Since your system only has 2GB memory, I'd try the 2G:2G split, which would in theory allow both the kernel and userspace to all memory.
CONFIG_LOWMEM_SIZE_BOOL=yCONFIG_LOWMEM_SIZE=0x7000000(Michael, please correct me if the above wouldn't work.)
It's a bit more complicated, in order to increase LOWMEM_SIZE you need to adjust all the other variables to make space.
To get 2G of user virtual space I think you need:
CONFIG_ADVANCED_OPTIONS=yCONFIG_LOWMEM_SIZE_BOOL=yCONFIG_LOWMEM_SIZE=0x60000000CONFIG_PAGE_OFFSET_BOOL=yCONFIG_PAGE_OFFSET=0x90000000CONFIG_KERNEL_START_BOOL=yCONFIG_KERNEL_START=0x90000000CONFIG_PHYSICAL_START=0x00000000CONFIG_TASK_SIZE_BOOL=yCONFIG_TASK_SIZE=0x80000000
Which results in 1.5GB of lowmem.
Or if you want to map all 2G of RAM directly in the kernel without highmem, but limit user virtual space to 1.5G:
CONFIG_ADVANCED_OPTIONS=yCONFIG_LOWMEM_SIZE_BOOL=yCONFIG_LOWMEM_SIZE=0x80000000CONFIG_PAGE_OFFSET_BOOL=yCONFIG_PAGE_OFFSET=0x70000000CONFIG_KERNEL_START_BOOL=yCONFIG_KERNEL_START=0x70000000CONFIG_PHYSICAL_START=0x00000000CONFIG_TASK_SIZE_BOOL=yCONFIG_TASK_SIZE=0x60000000
You can also reclaim another 256MB of virtual space if you disable CONFIG_MODULES.
Those configs do boot on qemu. But I don't have easy access to my 32-bit machine to test if they boot on actual hardware.
cheers
Quoted 16 lines · On Thu, 06 Jun 2024 22:08:40 +1000 Michael Ellerman <[email protected]> wrote:
It's a bit more complicated, in order to increase
LOWMEM_SIZEyou need to adjust all the other variables to make space.To get 2G of user virtual space I think you need:
CONFIG_ADVANCED_OPTIONS=yCONFIG_LOWMEM_SIZE_BOOL=yCONFIG_LOWMEM_SIZE=0x60000000CONFIG_PAGE_OFFSET_BOOL=yCONFIG_PAGE_OFFSET=0x90000000CONFIG_KERNEL_START_BOOL=yCONFIG_KERNEL_START=0x90000000CONFIG_PHYSICAL_START=0x00000000CONFIG_TASK_SIZE_BOOL=yCONFIG_TASK_SIZE=0x80000000
Quoted 1 line
Which results in 1.5GB of lowmem.
Booting this config on the G4 worked, but the issue showed up anyhow.
Quoted 13 lines
Or if you want to map all 2G of RAM directly in the kernel without highmem, but limit user virtual space to 1.5G:
CONFIG_ADVANCED_OPTIONS=yCONFIG_LOWMEM_SIZE_BOOL=yCONFIG_LOWMEM_SIZE=0x80000000CONFIG_PAGE_OFFSET_BOOL=yCONFIG_PAGE_OFFSET=0x70000000CONFIG_KERNEL_START_BOOL=yCONFIG_KERNEL_START=0x70000000CONFIG_PHYSICAL_START=0x00000000CONFIG_TASK_SIZE_BOOL=yCONFIG_TASK_SIZE=0x60000000
This actually did the trick!
Also I disabled highmem via HIGHMEM=n. With this config the machine did run the "stress-ng --vm 2 --vm-bytes 1930M --verify -v" load for about 2 hrs without hitting the issue.
Quoted 2 lines
You can also reclaim another 256MB of virtual space if you disable
CONFIG_MODULES.
Did not try that 'cause the 2nd config worked.
Working 2G_no-highmem .config attached and the dmesg of both configs attached.
Regards,
Erhard
Quoted 18 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:
Hi Erhard,
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.
Quoted 6 lines
Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling
CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)Thanks!
Hi Yu!
Thanks for looking into this.
The reason for this 0.25GB DMA / 1.75GB highmem split is beyond my knowledge. I can only tell this much that it's like this at least since kernel v4.14.x (dmesg of an old bugreport of mine at https://bugzilla.kernel.org/show_bug.cgi?id=201723), I guess earlier kernel versions too.
Without CONFIG_HIGHMEM the memory layout looks like this:
Total memory = 768MB; using 2048kB for hash table
[...]
Top of RAM: 0x30000000, Total RAM: 0x30000000
Memory hole size: 0MB
Zone ranges:
DMA [mem 0x0000000000000000-0x000000002fffffff]
Normal empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000002fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff]
percpu: Embedded 29 pages/cpu s28448 r8192 d82144 u118784
pcpu-alloc: s28448 r8192 d82144 u118784 alloc=29*4096
pcpu-alloc: [0] 0 [0] 1
Kernel command line: ro root=/dev/sda5 slub_debug=FZP page_poison=1 [email protected]/eth0,[email protected]/A8:A1:59:16:4F:EA debug Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Built 1 zonelists, mobility grouping on. Total pages: 194880
mem auto-init: stack:all(pattern), heap alloc:off, heap free:off
Kernel virtual memory layout:
* 0xffbdf000..0xfffff000 : fixmap * 0xff8f4000..0xffbdf000 : early ioremap * 0xf1000000..0xff8f4000 : vmalloc & ioremap * 0xb0000000..0xc0000000 : modules
Memory: 761868K/786432K available (7760K kernel code, 524K rwdata, 4528K rodata, 1100K init, 253K bss, 24564K reserved, 0K cma-reserved) [...]
With only 768 MB RAM and 2048K hashtable I get pretty much the same "kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL),nodemask=(null),cpuset=/,mems_allowed=0" as with the HIGHMEM enabled kernel at running "stress-ng --vm 2 --vm-bytes 1930M --verify -v".
I tried the workload on v6.6.32 LTS where the issue shows up too. But v6.1.92 LTS seems ok! Triple checked v6.1.92 to be sure.
Attached please find kernel v6.9.3 dmesg (without HIGHMEM) and kernel v6.1.92 .config.
Regards,
Erhard
Quoted 64 lines · Erhard Furtner
Quoted 18 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:
Hi Erhard,
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.
Quoted 6 lines
Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling
CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)Thanks!
Hi Yu!
Thanks for looking into this.
The reason for this 0.25GB DMA / 1.75GB highmem split is beyond my knowledge. I can only tell this much that it's like this at least since kernel v4.14.x (dmesg of an old bugreport of mine at https://bugzilla.kernel.org/show_bug.cgi?id=201723), I guess earlier kernel versions too.
Without
CONFIG_HIGHMEMthe memory layout looks like this:Total memory = 768MB; using 2048kB for hash table
[...]
Top of RAM:0x30000000, Total RAM:0x30000000
Memory hole size: 0MB
Zone ranges:
DMA [mem 0x0000000000000000-0x000000002fffffff]
Normal empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000002fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff]
percpu: Embedded 29 pages/cpu s28448 r8192 d82144 u118784
pcpu-alloc: s28448 r8192 d82144 u118784 alloc=29*4096
pcpu-alloc: [0] 0 [0] 1
Kernel command line: ro root=/dev/sda5 slub_debug=FZP page_poison=1 [email protected]/eth0,[email protected]/A8:A1:59:16:4F:EA debug Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Built 1 zonelists, mobility grouping on. Total pages: 194880
mem auto-init: stack:all(pattern), heap alloc:off, heap free:off
Kernel virtual memory layout:* 0xffbdf000..0xfffff000 : fixmap * 0xff8f4000..0xffbdf000 : early ioremap * 0xf1000000..0xff8f4000 : vmalloc & ioremap * 0xb0000000..0xc0000000 : modulesMemory: 761868K/786432K available (7760K kernel code, 524K rwdata, 4528K rodata, 1100K init, 253K bss, 24564K reserved, 0K cma-reserved) [...]
Quoted 6 lines
With only 768 MB RAM and 2048K hashtable I get pretty much the same "kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL),nodemask=(null),cpuset=/,mems_allowed=0" as with the HIGHMEM enabled kernel at running "stress-ng --vm 2 --vm-bytes 1930M --verify -v".
I tried the workload on v6.6.32 LTS where the issue shows up too. But v6.1.92 LTS seems ok! Triple checked v6.1.92 to be sure.
Attached please find kernel v6.9.3 dmesg (without HIGHMEM) and kernel v6.1.92 .config.
Thanks.
I compared the .config between v6.8.9 (you attached previously) and v6.1.92 -- I didn't see any major differences (both have ZONE_DMA, HIGHMEM, MGLRU and zswap/zsmalloc). Either there is something broken between v6.1.92 and v6.6.32 (as you mentioned above), or it's just a kernel allocation bloat which puts the DMA zone (0.25GB) under too heavy pressure. The latter isn't uncommon when upgrading to a newer version of the kernel.
Could you please attach the dmesg from v6.1.92? I want to compare the dmegs between the two kernel versions as well -- that might provide some hints.
Quoted 11 lines · On Sun, 2 Jun 2024 14:38:18 -0600 Yu Zhao <[email protected]> wrote:
I compared the .config between v6.8.9 (you attached previously) and v6.1.92 -- I didn't see any major differences (both have
ZONE_DMA, HIGHMEM, MGLRU and zswap/zsmalloc). Either there is something broken between v6.1.92 and v6.6.32 (as you mentioned above), or it's just a kernel allocation bloat which puts the DMA zone (0.25GB) under too heavy pressure. The latter isn't uncommon when upgrading to a newer version of the kernel.Could you please attach the dmesg from v6.1.92? I want to compare the dmegs between the two kernel versions as well -- that might provide some hints.
No problem, attached please find the v6.1.92 dmesg and also the v6.9.3 .config and dmesg.
I stripped down the v6.9.3 .config a bit (to ease 'make oldconfig' for older kernels) from my originally posted v6.8.9 .config and used that as a starting point for v6.6.32 and v6.1.92.
Also I started a git bisect. Let's see if I get something meaningful out of this...
Regards,
Erhard
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]
Quoted 8 lines
Quoted 8 lines
The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.
Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling
CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)
Ok, the bisect I did actually revealed something meaningful:
# git bisect good b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17 Author: Yosry Ahmed <[email protected]> Date: Tue Jun 20 19:46:44 2023 +0000
mm: zswap: multiple zpools support
Support using multiple zpools of the same type in zswap, for concurrency
purposes. A fixed number of 32 zpools is suggested by this commit, which
was determined empirically. It can be later changed or made into a config
option if needed.
On a setup with zswap and zsmalloc, comparing a single zpool to 32 zpools
shows improvements in the zsmalloc lock contention, especially on the swap
out path.
The following shows the perf analysis of the swapout path when 10
workloads are simultaneously reclaiming and refaulting tmpfs pages. There
are some improvements on the swap in path as well, but less significant.
1 zpool:
|--28.99%--zswap_frontswap_store
|
<snip>
|
|--8.98%--zpool_map_handle
| |
| --8.98%--zs_zpool_map
| |
| --8.95%--zs_map_object
| |
| --8.38%--_raw_spin_lock
| |
| --7.39%--queued_spin_lock_slowpath
|
|--8.82%--zpool_malloc
| |
| --8.82%--zs_zpool_malloc
| |
| --8.80%--zs_malloc
| |
| |--7.21%--_raw_spin_lock
| | |
| | --6.81%--queued_spin_lock_slowpath
<snip>
32 zpools:
|--16.73%--zswap_frontswap_store
|
<snip>
|
|--1.81%--zpool_malloc
| |
| --1.81%--zs_zpool_malloc
| |
| --1.79%--zs_malloc
| |
| --0.73%--obj_malloc
|
|--1.06%--zswap_update_total_size
|
|--0.59%--zpool_map_handle
| |
| --0.59%--zs_zpool_map
| |
| --0.57%--zs_map_object
| |
| --0.51%--_raw_spin_lock
<snip>
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Yosry Ahmed <[email protected]>
Suggested-by: Yu Zhao <[email protected]>
Acked-by: Chris Li (Google) <[email protected]>
Reviewed-by: Nhat Pham <[email protected]>
Tested-by: Nhat Pham <[email protected]>
Cc: Dan Streetman <[email protected]>
Cc: Domenico Cerasuolo <[email protected]>
Cc: Johannes Weiner <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Seth Jennings <[email protected]>
Cc: Vitaly Wool <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>mm/zswap.c | 81 +++++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 27 deletions(-)
'bad' bisects were where the "kswapd0: page allocation failure:" showed up when running the workload, 'good' bisects were the cases where I only got the kernels OOM reaper killing the workload. In the good cases the machine stayed usable via VNC, in the bad cases with the issue showing up the machine crashed and rebooted >80% of the time shortly after showing the issue in dmesg (via netconsole). I triple checked the good cases to be sure only the OOM reaper showed up and not the kswapd0: page allocation failure.
Bisect.log attached.
Regards,
Erhard
git bisect start
# Status: warte auf guten und schlechten Commit
# good: [88690811da69826fdb59d908a6e5e9d0c63b581a] Linux 6.1.92
git bisect good 88690811da69826fdb59d908a6e5e9d0c63b581a
# Status: warte auf schlechten Commit, 1 guter Commit bekannt
# bad: [a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6] Linux 6.9
git bisect bad a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
# good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
git bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476
# good: [1b580c9bb63414903cfb414571e92e4fd36d6969] Merge branch 'samples-bpf-remove-unmaintained-xdp-sample-utilities' git bisect good 1b580c9bb63414903cfb414571e92e4fd36d6969
# bad: [5ab500d6f9f50d8246865a2ead85d0e88ea30004] Merge branch 'sfc-implement-ndo_hwtstamp_-get-set'
git bisect bad 5ab500d6f9f50d8246865a2ead85d0e88ea30004
# bad: [cc33a80b816406f900a53c7f98a50f6eacdd2e31] MAINTAINERS: Maintainer change for ptp_vmw driver
git bisect bad cc33a80b816406f900a53c7f98a50f6eacdd2e31
# bad: [4debf77169ee459c46ec70e13dc503bc25efd7d2] Merge tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd
git bisect bad 4debf77169ee459c46ec70e13dc503bc25efd7d2
# bad: [1086eeac9c333b6db6c98594f02996c8261c60c5] Merge tag 'lsm-pr-20230829' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/lsm
git bisect bad 1086eeac9c333b6db6c98594f02996c8261c60c5
# good: [0fc81f376468ce67d70ecd1016dfe901a51a39f5] Merge tag 'regmap-v6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap
git bisect good 0fc81f376468ce67d70ecd1016dfe901a51a39f5
# good: [bd6c11bc43c496cddfc6cf603b5d45365606dbd5] Merge tag 'net-next-6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
git bisect good bd6c11bc43c496cddfc6cf603b5d45365606dbd5
# bad: [5553b15a4bbba8039e1f31b63642048286f540dc] m68k: implement the new page table range API
git bisect bad 5553b15a4bbba8039e1f31b63642048286f540dc
# good: [0c2e394ab23017303f676e6206a54c54bb0e3681] mm: move FAULT_FLAG_VMA_LOCK check down in handle_pte_fault()
git bisect good 0c2e394ab23017303f676e6206a54c54bb0e3681
# bad: [3f32c49ed6f15c8412a8abc93a92c4b37e6c4592] mm: memtest: convert to memtest_report_meminfo()
git bisect bad 3f32c49ed6f15c8412a8abc93a92c4b37e6c4592
# bad: [6ad243b83b5094026fdb3171711ddb25246b3d8a] mm/damon/sysfs: implement a command for updating only schemes tried total bytes git bisect bad 6ad243b83b5094026fdb3171711ddb25246b3d8a
# bad: [68af05143fd4b49d4b12eab8d63c91ffbc7c4e5e] kernel/iomem.c: remove __weak ioremap_cache helper
git bisect bad 68af05143fd4b49d4b12eab8d63c91ffbc7c4e5e
# good: [27af67f35631ac4b61b5e4455b44c9aee8d2cc4b] powerpc/book3s64/mm: enable transparent pud hugepage
git bisect good 27af67f35631ac4b61b5e4455b44c9aee8d2cc4b
# bad: [34f4c198bfbe86612c368eb122002787acecaa93] zswap: make zswap_store() take a folio
git bisect bad 34f4c198bfbe86612c368eb122002787acecaa93
# good: [601f006fddc66e369fdac7c572f981eafd159dac] powerpc/book3s64/radix: remove mmu_vmemmap_psize
git bisect good 601f006fddc66e369fdac7c572f981eafd159dac
# bad: [b8cf32dc6e8c75b712cbf638e0fd210101c22f17] mm: zswap: multiple zpools support
git bisect bad b8cf32dc6e8c75b712cbf638e0fd210101c22f17
# good: [6be3601517d90b728095d70c14f3a04b9adcb166] powerpc/book3s64/radix: add debug message to give more details of vmemmap allocation git bisect good 6be3601517d90b728095d70c14f3a04b9adcb166
# first bad commit: [b8cf32dc6e8c75b712cbf638e0fd210101c22f17] mm: zswap: multiple zpools support
Quoted 30 lines · Erhard Furtner
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]Quoted 8 lines
Quoted 8 lines
The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.
Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling
CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)Ok, the bisect I did actually revealed something meaningful:
Quoted 7 lines
# git bisect good b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17 Author: Yosry Ahmed <[email protected]> Date: Tue Jun 20 19:46:44 2023 +0000mm: zswap: multiple zpools support
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.
Do you have /proc/slabinfo from a good and a bad run by any chance?
Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
From 1ac0fa34783f66ea7334fd2e9129bc362bee51dd Mon Sep 17 00:00:00 2001
From: Yosry Ahmed <[email protected]>
Date: Sat, 13 May 2023 01:10:34 +0000
Subject: [PATCH] mm: zsmalloc: share slab caches for all zsmalloc zpools
Zswap creates 32 zpools to improve concurrency. Each zsmalloc zpool creates its own 'zs_handle' and 'zspage' slab caches. We end up with 32 slab caches of each type.
Since each slab cache holds some free objects, we end up with a lot of free objects distributed among the separate zpool caches. Slab caches are designed to handle concurrent allocations by using percpu structures, so having a single instance of each cache should be enough, and avoids wasting more memory than needed due to fragmentation.
Additionally, having more slab caches than needed unnecessarily slows down code paths that iterate slab_caches.
--- mm/zsmalloc.c | 60 ++++++++++++++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 27 deletions(-)
mm/zsmalloc.c +33 -27
@@ -289,29 +289,8 @@ static void init_deferred_free(struct zs_pool *pool) {}static void SetZsPageMovable (struct zs_pool *pool,struct zspage *zspage) {}#endif -static int create_cache (struct zs_pool *pool) -{ - pool->handle_cachep =kmem_cache_create ("zs_handle" ,ZS_HANDLE_SIZE , -0 ,0 ,NULL ); -if (!pool->handle_cachep) -return 1 ; - - pool->zspage_cachep =kmem_cache_create ("zspage" ,sizeof (struct zspage ), -0 ,0 ,NULL ); -if (!pool->zspage_cachep) { -kmem_cache_destroy (pool->handle_cachep); - pool->handle_cachep =NULL ; -return 1 ; - } - -return 0 ; -} - -static void destroy_cache (struct zs_pool *pool) -{ -kmem_cache_destroy (pool->handle_cachep); -kmem_cache_destroy (pool->zspage_cachep); -} +static struct kmem_cache *zs_handle_cache; +static struct kmem_cache *zspage_cache;static unsigned long cache_alloc_handle (struct zs_pool *pool,gfp_t gfp) {
@@ -2107,13 +2086,13 @@ struct zs_pool *zs_create_pool(const char *name)spin_lock_init (&pool->lock);atomic_set (&pool->compaction_in_progress,0 ); + pool->handle_cachep = zs_handle_cache; + pool->zspage_cachep = zspage_cache; + pool->name =kstrdup (name,GFP_KERNEL );if (!pool->name) goto err; -if (create_cache (pool)) - goto err; -/* * Iterate reversely, because, size of size_class that we want to use * for merging should be larger or equal to current size.
@@ -2234,16 +2213,41 @@ void zs_destroy_pool(struct zs_pool *pool)kfree (class); } -destroy_cache (pool);kfree (pool->name);kfree (pool); }EXPORT_SYMBOL_GPL (zs_destroy_pool); +static void zs_destroy_caches (void ) +{ +kmem_cache_destroy (zs_handle_cache); +kmem_cache_destroy (zspage_cache); + zs_handle_cache =NULL ; + zspage_cache =NULL ; +} + +static int zs_create_caches (void ) +{ + zs_handle_cache =kmem_cache_create ("zs_handle" ,ZS_HANDLE_SIZE , +0 ,0 ,NULL ); + zspage_cache =kmem_cache_create ("zspage" ,sizeof (struct zspage ), +0 ,0 ,NULL ); + +if (!zs_handle_cache || !zspage_cache) { +zs_destroy_caches (); +return -1 ; + } +return 0 ; +} +static int __init zs_init (void ) {int ret; + ret =zs_create_caches (); +if (ret) + goto out; + ret =cpuhp_setup_state (CPUHP_MM_ZS_PREPARE ,"mm/zsmalloc:prepare" , zs_cpu_prepare, zs_cpu_dead);if (ret)
@@ -2258,6 +2262,7 @@ static int __init zs_init(void)return 0 ; out: +zs_destroy_caches ();return ret; }
@@ -2269,6 +2274,7 @@ static void __exit zs_exit(void)cpuhp_remove_state (CPUHP_MM_ZS_PREPARE );zs_stat_exit (); +zs_destroy_caches (); }module_init (zs_init);
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.
Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.
The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.
Regards,
Erhard
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ext4_groupinfo_1k 48 48 168 24 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_groupinfo_4k 320 322 176 23 1 : tunables 0 0 0 : slabdata 14 14 0 btrfs_prelim_ref 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_extent_op 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_data_ref 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_tree_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_ref_head 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_defrag 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_node 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_ordered_extent 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_extent_map 3 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 bio-232 2 14 288 14 1 : tunables 0 0 0 : slabdata 1 1 0 bio-280 4 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_buffer 11 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_state 3 46 88 46 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_free_space_bitmap 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 btrfs_free_space 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_path 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_trans_handle 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 bio-296 2 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_inode 3 16 1008 16 4 : tunables 0 0 0 : slabdata 1 1 0 bio-328 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 zswap_entry 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 p9_req_t 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 ip6-frags 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0 fib6_nodes 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ip6_dst_cache 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 PINGv6 0 0 1056 15 4 : tunables 0 0 0 : slabdata 0 0 0 RAWv6 7 15 1056 15 4 : tunables 0 0 0 : slabdata 1 1 0 UDPLITEv6 0 0 1152 14 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 1 14 1152 14 4 : tunables 0 0 0 : slabdata 1 1 0 tw_sock_TCPv6 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCPv6 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 2 15 2080 15 8 : tunables 0 0 0 : slabdata 1 1 0 scsi_sense_cache 9 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 virtio_scsi_cmd 64 64 256 16 1 : tunables 0 0 0 : slabdata 4 4 0 io_kiocb 0 0 224 18 1 : tunables 0 0 0 : slabdata 0 0 0 bfq_io_cq 0 0 1032 15 4 : tunables 0 0 0 : slabdata 0 0 0 bfq_queue 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 bio-88 10 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 bio-168 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 mqueue_inode_cache 1 21 768 21 4 : tunables 0 0 0 : slabdata 1 1 0 v9fs_inode_cache 0 0 600 13 2 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 560 14 2 : tunables 0 0 0 : slabdata 0 0 0 jbd2_transaction_s 3 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_inode 23 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_handle 0 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_head 6 84 96 42 1 : tunables 0 0 0 : slabdata 2 2 0 jbd2_revoke_table_s 2 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_revoke_record_s 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_fc_dentry_update 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 1354 1360 1008 16 4 : tunables 0 0 0 : slabdata 85 85 0 ext4_free_data 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_allocation_context 0 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_prealloc_space 6 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_system_zone 89 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_io_end_vec 0 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_io_end 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 bio_post_read_ctx 128 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0 pending_reservation 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 extent_status 1083 1095 56 73 1 : tunables 0 0 0 : slabdata 15 15 0 mbcache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 kioctx 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 aio_kiocb 0 0 160 25 1 : tunables 0 0 0 : slabdata 0 0 0 userfaultfd_ctx_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_path_event 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 fsnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_struct 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 dio 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 fasync_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 pid_namespace 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 UNIX-STREAM 27 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 UNIX 13 17 928 17 4 : tunables 0 0 0 : slabdata 1 1 0 ip4-frags 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 tcp_bind2_bucket 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 tcp_bind_bucket 2 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 inet_peer_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_dst_cache 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_state 0 0 640 12 2 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_fib_alias 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_dst_cache 6 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 PING 0 0 896 18 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 5 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 UDP 19 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCP 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCP 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 TCP 3 16 1952 16 8 : tunables 0 0 0 : slabdata 1 1 0 bio-184 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 ep_head 36 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_pwq 37 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 eventpoll_epi 36 96 128 32 1 : tunables 0 0 0 : slabdata 3 3 0 inotify_inode_mark 40 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 sgpool-128 2 12 2624 12 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-64 2 12 1344 12 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-32 2 23 704 23 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 sgpool-8 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 request_queue 4 22 720 22 4 : tunables 0 0 0 : slabdata 1 1 0 blkdev_ioc 1 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 bio-136 4 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 biovec-max 48 60 3136 10 8 : tunables 0 0 0 : slabdata 6 6 0 biovec-128 0 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 biovec-64 0 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0 biovec-16 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 ksm_mm_slot 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_stable_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_rmap_item 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 user_namespace 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 uid_cache 4 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 93 108 672 12 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_ext_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 skbuff_small_head 130 136 480 17 2 : tunables 0 0 0 : slabdata 8 8 0 skbuff_fclone_cache 1 19 416 19 2 : tunables 0 0 0 : slabdata 1 1 0 skbuff_head_cache 397 414 224 18 1 : tunables 0 0 0 : slabdata 23 23 0 file_lock_cache 1 24 168 24 1 : tunables 0 0 0 : slabdata 1 1 0 file_lock_ctx 10 192 64 64 1 : tunables 0 0 0 : slabdata 3 3 0 fsnotify_mark_connector 33 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 buffer_head 5653 5694 104 39 1 : tunables 0 0 0 : slabdata 146 146 0 task_delay_info 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 taskstats 2 17 456 17 2 : tunables 0 0 0 : slabdata 1 1 0 proc_dir_entry 274 286 152 26 1 : tunables 0 0 0 : slabdata 11 11 0 pde_opener 0 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 proc_inode_cache 197 224 568 14 2 : tunables 0 0 0 : slabdata 16 16 0 seq_file 2 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 sigqueue 1 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 bdev_cache 27 39 1216 13 4 : tunables 0 0 0 : slabdata 3 3 0 shmem_inode_cache 1313 1425 632 25 4 : tunables 0 0 0 : slabdata 57 57 0 kernfs_iattrs_cache 11 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 kernfs_node_cache 17325 17388 112 36 1 : tunables 0 0 0 : slabdata 483 483 0 mnt_cache 29 36 224 18 1 : tunables 0 0 0 : slabdata 2 2 0 filp 476 512 256 16 1 : tunables 0 0 0 : slabdata 32 32 0 inode_cache 2406 2415 528 15 2 : tunables 0 0 0 : slabdata 161 161 0 dentry 6389 6394 176 23 1 : tunables 0 0 0 : slabdata 278 278 0 names_cache 0 14 4160 7 8 : tunables 0 0 0 : slabdata 2 2 0 net_namespace 0 0 2624 12 8 : tunables 0 0 0 : slabdata 0 0 0 key_jar 7 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 uts_namespace 0 0 440 18 2 : tunables 0 0 0 : slabdata 0 0 0 nsproxy 1 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 vm_area_struct 1401 1428 96 42 1 : tunables 0 0 0 : slabdata 34 34 0 fs_cache 32 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 files_cache 32 42 288 14 1 : tunables 0 0 0 : slabdata 3 3 0 signal_cache 111 126 864 18 4 : tunables 0 0 0 : slabdata 7 7 0 sighand_cache 111 115 1376 23 8 : tunables 0 0 0 : slabdata 5 5 0 task_struct 111 121 2848 11 8 : tunables 0 0 0 : slabdata 11 11 0 cred_jar 161 189 192 21 1 : tunables 0 0 0 : slabdata 9 9 0 anon_vma_chain 1055 1168 56 73 1 : tunables 0 0 0 : slabdata 16 16 0 anon_vma 761 782 88 46 1 : tunables 0 0 0 : slabdata 17 17 0 pid 118 128 128 32 1 : tunables 0 0 0 : slabdata 4 4 0 perf_event 0 0 848 19 4 : tunables 0 0 0 : slabdata 0 0 0 pool_workqueue 29 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 maple_node 118 252 768 21 4 : tunables 0 0 0 : slabdata 12 12 0 radix_tree_node 1814 1824 320 12 1 : tunables 0 0 0 : slabdata 152 152 0 task_group 31 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 mm_struct 31 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 vmap_area 137 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 pgtable-2^10 31 32 12288 2 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-cg-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-cg-4k 5 10 12288 2 8 : tunables 0 0 0 : slabdata 5 5 0 kmalloc-cg-2k 16 20 6144 5 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-1k 40 40 3072 10 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-512 15 63 1536 21 8 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-256 27 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-192 24 54 216 18 1 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-128 18 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-96 40 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-64 40 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-32 28 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-16 18 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-8 23 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 dma-kmalloc-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-96 32 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-rcl-64 132 210 192 21 1 : tunables 0 0 0 : slabdata 10 10 0 kmalloc-rcl-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8k 15 16 24576 1 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-4k 14 16 12288 2 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-2k 380 380 6144 5 8 : tunables 0 0 0 : slabdata 76 76 0 kmalloc-1k 256 260 3072 10 8 : tunables 0 0 0 : slabdata 26 26 0 kmalloc-512 612 630 1536 21 8 : tunables 0 0 0 : slabdata 30 30 0 kmalloc-256 148 168 768 21 4 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-192 4307 4320 216 18 1 : tunables 0 0 0 : slabdata 240 240 0 kmalloc-128 724 735 384 21 2 : tunables 0 0 0 : slabdata 35 35 0 kmalloc-96 721 748 120 34 1 : tunables 0 0 0 : slabdata 22 22 0 kmalloc-64 1346 1995 192 21 1 : tunables 0 0 0 : slabdata 95 95 0 kmalloc-32 1831 1890 96 42 1 : tunables 0 0 0 : slabdata 45 45 0 kmalloc-16 3264 3315 48 85 1 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-8 2864 2944 32 128 1 : tunables 0 0 0 : slabdata 23 23 0 kmem_cache_node 217 252 96 42 1 : tunables 0 0 0 : slabdata 6 6 0 kmem_cache 217 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0 slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ext4_groupinfo_1k 48 48 168 24 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_groupinfo_4k 320 322 176 23 1 : tunables 0 0 0 : slabdata 14 14 0 btrfs_prelim_ref 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_extent_op 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_data_ref 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_tree_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_ref_head 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_defrag 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_node 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_ordered_extent 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_extent_map 3 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 bio-232 2 14 288 14 1 : tunables 0 0 0 : slabdata 1 1 0 bio-280 4 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_buffer 9 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_state 3 46 88 46 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_free_space_bitmap 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 btrfs_free_space 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_path 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_trans_handle 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 bio-296 2 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_inode 2 16 1008 16 4 : tunables 0 0 0 : slabdata 1 1 0 bio-328 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 zswap_entry 248 448 64 64 1 : tunables 0 0 0 : slabdata 7 7 0 p9_req_t 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 ip6-frags 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0 fib6_nodes 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ip6_dst_cache 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 PINGv6 0 0 1056 15 4 : tunables 0 0 0 : slabdata 0 0 0 RAWv6 7 15 1056 15 4 : tunables 0 0 0 : slabdata 1 1 0 UDPLITEv6 0 0 1152 14 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 1 14 1152 14 4 : tunables 0 0 0 : slabdata 1 1 0 tw_sock_TCPv6 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCPv6 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 2 15 2080 15 8 : tunables 0 0 0 : slabdata 1 1 0 scsi_sense_cache 9 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 virtio_scsi_cmd 64 64 256 16 1 : tunables 0 0 0 : slabdata 4 4 0 io_kiocb 0 0 224 18 1 : tunables 0 0 0 : slabdata 0 0 0 bfq_io_cq 0 0 1032 15 4 : tunables 0 0 0 : slabdata 0 0 0 bfq_queue 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 bio-88 10 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 bio-168 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 mqueue_inode_cache 1 21 768 21 4 : tunables 0 0 0 : slabdata 1 1 0 v9fs_inode_cache 0 0 600 13 2 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 560 14 2 : tunables 0 0 0 : slabdata 0 0 0 jbd2_transaction_s 3 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_inode 10 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_handle 0 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_head 9 84 96 42 1 : tunables 0 0 0 : slabdata 2 2 0 jbd2_revoke_table_s 2 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_revoke_record_s 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_fc_dentry_update 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 264 800 1008 16 4 : tunables 0 0 0 : slabdata 50 50 0 ext4_free_data 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_allocation_context 0 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_prealloc_space 6 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_system_zone 89 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_io_end_vec 0 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_io_end 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 bio_post_read_ctx 128 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0 pending_reservation 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 extent_status 54 292 56 73 1 : tunables 0 0 0 : slabdata 4 4 0 mbcache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 kioctx 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 aio_kiocb 0 0 160 25 1 : tunables 0 0 0 : slabdata 0 0 0 userfaultfd_ctx_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_path_event 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 fsnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_struct 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 dio 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 fasync_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 221 448 64 64 1 : tunables 0 0 0 : slabdata 7 7 0 zs_handle 238 850 24 170 1 : tunables 0 0 0 : slabdata 5 5 0 pid_namespace 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 UNIX-STREAM 29 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 UNIX 13 17 928 17 4 : tunables 0 0 0 : slabdata 1 1 0 ip4-frags 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 tcp_bind2_bucket 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 tcp_bind_bucket 2 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 inet_peer_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_dst_cache 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_state 0 0 640 12 2 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_fib_alias 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_dst_cache 6 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 PING 0 0 896 18 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 5 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 UDP 19 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCP 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCP 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 TCP 3 16 1952 16 8 : tunables 0 0 0 : slabdata 1 1 0 bio-184 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 ep_head 40 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_pwq 42 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 eventpoll_epi 40 96 128 32 1 : tunables 0 0 0 : slabdata 3 3 0 inotify_inode_mark 40 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 sgpool-128 2 12 2624 12 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-64 2 12 1344 12 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-32 2 23 704 23 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 sgpool-8 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 request_queue 4 22 720 22 4 : tunables 0 0 0 : slabdata 1 1 0 blkdev_ioc 1 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 bio-136 4 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 biovec-max 48 60 3136 10 8 : tunables 0 0 0 : slabdata 6 6 0 biovec-128 0 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 biovec-64 0 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0 biovec-16 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 ksm_mm_slot 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_stable_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_rmap_item 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 user_namespace 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 uid_cache 4 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 95 108 672 12 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_ext_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 skbuff_small_head 165 187 480 17 2 : tunables 0 0 0 : slabdata 11 11 0 skbuff_fclone_cache 1 19 416 19 2 : tunables 0 0 0 : slabdata 1 1 0 skbuff_head_cache 419 468 224 18 1 : tunables 0 0 0 : slabdata 26 26 0 file_lock_cache 1 24 168 24 1 : tunables 0 0 0 : slabdata 1 1 0 file_lock_ctx 7 192 64 64 1 : tunables 0 0 0 : slabdata 3 3 0 fsnotify_mark_connector 33 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 buffer_head 128 1170 104 39 1 : tunables 0 0 0 : slabdata 30 30 0 task_delay_info 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 taskstats 2 17 456 17 2 : tunables 0 0 0 : slabdata 1 1 0 proc_dir_entry 274 286 152 26 1 : tunables 0 0 0 : slabdata 11 11 0 pde_opener 0 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 proc_inode_cache 531 574 568 14 2 : tunables 0 0 0 : slabdata 41 41 0 seq_file 2 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 sigqueue 1 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 bdev_cache 27 39 1216 13 4 : tunables 0 0 0 : slabdata 3 3 0 shmem_inode_cache 1313 1425 632 25 4 : tunables 0 0 0 : slabdata 57 57 0 kernfs_iattrs_cache 11 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 kernfs_node_cache 17299 17388 112 36 1 : tunables 0 0 0 : slabdata 483 483 0 mnt_cache 29 36 224 18 1 : tunables 0 0 0 : slabdata 2 2 0 filp 527 592 256 16 1 : tunables 0 0 0 : slabdata 37 37 0 inode_cache 1894 2340 528 15 2 : tunables 0 0 0 : slabdata 156 156 0 dentry 4434 6555 176 23 1 : tunables 0 0 0 : slabdata 285 285 0 names_cache 0 14 4160 7 8 : tunables 0 0 0 : slabdata 2 2 0 net_namespace 0 0 2624 12 8 : tunables 0 0 0 : slabdata 0 0 0 key_jar 7 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 uts_namespace 0 0 440 18 2 : tunables 0 0 0 : slabdata 0 0 0 nsproxy 1 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 vm_area_struct 1569 1722 96 42 1 : tunables 0 0 0 : slabdata 41 41 0 fs_cache 34 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 files_cache 34 42 288 14 1 : tunables 0 0 0 : slabdata 3 3 0 signal_cache 113 126 864 18 4 : tunables 0 0 0 : slabdata 7 7 0 sighand_cache 113 138 1376 23 8 : tunables 0 0 0 : slabdata 6 6 0 task_struct 113 121 2848 11 8 : tunables 0 0 0 : slabdata 11 11 0 cred_jar 165 189 192 21 1 : tunables 0 0 0 : slabdata 9 9 0 anon_vma_chain 1139 1387 56 73 1 : tunables 0 0 0 : slabdata 19 19 0 anon_vma 809 920 88 46 1 : tunables 0 0 0 : slabdata 20 20 0 pid 120 128 128 32 1 : tunables 0 0 0 : slabdata 4 4 0 perf_event 0 0 848 19 4 : tunables 0 0 0 : slabdata 0 0 0 pool_workqueue 29 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 maple_node 120 273 768 21 4 : tunables 0 0 0 : slabdata 13 13 0 radix_tree_node 1575 1860 320 12 1 : tunables 0 0 0 : slabdata 155 155 0 task_group 33 48 256 16 1 : tunables 0 0 0 : slabdata 3 3 0 mm_struct 33 51 928 17 4 : tunables 0 0 0 : slabdata 3 3 0 vmap_area 139 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 pgtable-2^10 33 38 12288 2 8 : tunables 0 0 0 : slabdata 19 19 0 kmalloc-cg-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-cg-4k 5 10 12288 2 8 : tunables 0 0 0 : slabdata 5 5 0 kmalloc-cg-2k 16 20 6144 5 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-1k 43 50 3072 10 8 : tunables 0 0 0 : slabdata 5 5 0 kmalloc-cg-512 16 63 1536 21 8 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-256 27 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-192 25 54 216 18 1 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-128 18 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-96 41 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-64 40 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-32 29 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-16 18 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-8 23 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 dma-kmalloc-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-96 32 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-rcl-64 122 210 192 21 1 : tunables 0 0 0 : slabdata 10 10 0 kmalloc-rcl-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8k 15 16 24576 1 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-4k 14 16 12288 2 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-2k 342 370 6144 5 8 : tunables 0 0 0 : slabdata 74 74 0 kmalloc-1k 223 270 3072 10 8 : tunables 0 0 0 : slabdata 27 27 0 kmalloc-512 621 630 1536 21 8 : tunables 0 0 0 : slabdata 30 30 0 kmalloc-256 146 168 768 21 4 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-192 4220 4338 216 18 1 : tunables 0 0 0 : slabdata 241 241 0 kmalloc-128 724 735 384 21 2 : tunables 0 0 0 : slabdata 35 35 0 kmalloc-96 733 782 120 34 1 : tunables 0 0 0 : slabdata 23 23 0 kmalloc-64 1348 1995 192 21 1 : tunables 0 0 0 : slabdata 95 95 0 kmalloc-32 1772 1890 96 42 1 : tunables 0 0 0 : slabdata 45 45 0 kmalloc-16 3254 3315 48 85 1 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-8 2870 2944 32 128 1 : tunables 0 0 0 : slabdata 23 23 0 kmem_cache_node 217 252 96 42 1 : tunables 0 0 0 : slabdata 6 6 0 kmem_cache 217 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0 slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ext4_groupinfo_1k 48 48 168 24 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_groupinfo_4k 320 322 176 23 1 : tunables 0 0 0 : slabdata 14 14 0 btrfs_prelim_ref 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_extent_op 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_data_ref 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_tree_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_ref_head 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_defrag 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_node 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_ordered_extent 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_extent_map 3 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 bio-232 2 14 288 14 1 : tunables 0 0 0 : slabdata 1 1 0 bio-280 4 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_buffer 11 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_state 3 46 88 46 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_free_space_bitmap 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 btrfs_free_space 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_path 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_trans_handle 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 bio-296 2 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_inode 3 16 1008 16 4 : tunables 0 0 0 : slabdata 1 1 0 bio-328 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zswap_entry 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 p9_req_t 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 ip6-frags 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0 fib6_nodes 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ip6_dst_cache 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 PINGv6 0 0 1056 15 4 : tunables 0 0 0 : slabdata 0 0 0 RAWv6 7 15 1056 15 4 : tunables 0 0 0 : slabdata 1 1 0 UDPLITEv6 0 0 1152 14 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 1 14 1152 14 4 : tunables 0 0 0 : slabdata 1 1 0 tw_sock_TCPv6 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCPv6 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 2 15 2080 15 8 : tunables 0 0 0 : slabdata 1 1 0 scsi_sense_cache 9 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 virtio_scsi_cmd 64 64 256 16 1 : tunables 0 0 0 : slabdata 4 4 0 io_kiocb 0 0 224 18 1 : tunables 0 0 0 : slabdata 0 0 0 bfq_io_cq 0 0 1032 15 4 : tunables 0 0 0 : slabdata 0 0 0 bfq_queue 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 bio-88 10 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 bio-168 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 mqueue_inode_cache 1 21 768 21 4 : tunables 0 0 0 : slabdata 1 1 0 v9fs_inode_cache 0 0 600 13 2 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 560 14 2 : tunables 0 0 0 : slabdata 0 0 0 jbd2_transaction_s 4 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_inode 23 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_handle 0 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_head 18 84 96 42 1 : tunables 0 0 0 : slabdata 2 2 0 jbd2_revoke_table_s 2 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_revoke_record_s 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_fc_dentry_update 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 1351 1360 1008 16 4 : tunables 0 0 0 : slabdata 85 85 0 ext4_free_data 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_allocation_context 0 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_prealloc_space 6 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_system_zone 89 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_io_end_vec 0 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_io_end 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 bio_post_read_ctx 128 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0 pending_reservation 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 extent_status 1080 1095 56 73 1 : tunables 0 0 0 : slabdata 15 15 0 mbcache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 kioctx 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 aio_kiocb 0 0 160 25 1 : tunables 0 0 0 : slabdata 0 0 0 userfaultfd_ctx_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_path_event 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 fsnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_struct 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 dio 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 fasync_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 pid_namespace 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 UNIX-STREAM 27 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 UNIX 13 17 928 17 4 : tunables 0 0 0 : slabdata 1 1 0 ip4-frags 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 tcp_bind2_bucket 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 tcp_bind_bucket 2 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 inet_peer_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_dst_cache 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_state 0 0 640 12 2 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_fib_alias 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_dst_cache 6 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 PING 0 0 896 18 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 5 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 UDP 18 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCP 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCP 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 TCP 3 16 1952 16 8 : tunables 0 0 0 : slabdata 1 1 0 bio-184 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 ep_head 36 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_pwq 37 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_epi 36 96 128 32 1 : tunables 0 0 0 : slabdata 3 3 0 inotify_inode_mark 40 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 sgpool-128 2 12 2624 12 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-64 2 12 1344 12 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-32 2 23 704 23 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 sgpool-8 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 request_queue 4 22 720 22 4 : tunables 0 0 0 : slabdata 1 1 0 blkdev_ioc 1 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 bio-136 4 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 biovec-max 48 60 3136 10 8 : tunables 0 0 0 : slabdata 6 6 0 biovec-128 0 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 biovec-64 0 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0 biovec-16 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 ksm_mm_slot 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_stable_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_rmap_item 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 user_namespace 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 uid_cache 4 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 92 108 672 12 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_ext_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 skbuff_small_head 126 136 480 17 2 : tunables 0 0 0 : slabdata 8 8 0 skbuff_fclone_cache 0 19 416 19 2 : tunables 0 0 0 : slabdata 1 1 0 skbuff_head_cache 404 414 224 18 1 : tunables 0 0 0 : slabdata 23 23 0 file_lock_cache 1 24 168 24 1 : tunables 0 0 0 : slabdata 1 1 0 file_lock_ctx 10 192 64 64 1 : tunables 0 0 0 : slabdata 3 3 0 fsnotify_mark_connector 33 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 buffer_head 5655 5694 104 39 1 : tunables 0 0 0 : slabdata 146 146 0 task_delay_info 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 taskstats 2 17 456 17 2 : tunables 0 0 0 : slabdata 1 1 0 proc_dir_entry 274 286 152 26 1 : tunables 0 0 0 : slabdata 11 11 0 pde_opener 0 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 proc_inode_cache 194 252 568 14 2 : tunables 0 0 0 : slabdata 18 18 0 seq_file 2 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 sigqueue 1 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 bdev_cache 27 39 1216 13 4 : tunables 0 0 0 : slabdata 3 3 0 shmem_inode_cache 1315 1425 632 25 4 : tunables 0 0 0 : slabdata 57 57 0 kernfs_iattrs_cache 11 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 kernfs_node_cache 17387 17460 112 36 1 : tunables 0 0 0 : slabdata 485 485 0 mnt_cache 29 36 224 18 1 : tunables 0 0 0 : slabdata 2 2 0 filp 475 512 256 16 1 : tunables 0 0 0 : slabdata 32 32 0 inode_cache 2469 2475 528 15 2 : tunables 0 0 0 : slabdata 165 165 0 dentry 6477 6486 176 23 1 : tunables 0 0 0 : slabdata 282 282 0 names_cache 0 21 4160 7 8 : tunables 0 0 0 : slabdata 3 3 0 net_namespace 0 0 2624 12 8 : tunables 0 0 0 : slabdata 0 0 0 key_jar 7 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 uts_namespace 0 0 440 18 2 : tunables 0 0 0 : slabdata 0 0 0 nsproxy 1 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 vm_area_struct 1402 1428 96 42 1 : tunables 0 0 0 : slabdata 34 34 0 fs_cache 32 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 files_cache 32 42 288 14 1 : tunables 0 0 0 : slabdata 3 3 0 signal_cache 109 126 864 18 4 : tunables 0 0 0 : slabdata 7 7 0 sighand_cache 109 115 1376 23 8 : tunables 0 0 0 : slabdata 5 5 0 task_struct 109 110 2848 11 8 : tunables 0 0 0 : slabdata 10 10 0 cred_jar 159 189 192 21 1 : tunables 0 0 0 : slabdata 9 9 0 anon_vma_chain 1058 1168 56 73 1 : tunables 0 0 0 : slabdata 16 16 0 anon_vma 764 782 88 46 1 : tunables 0 0 0 : slabdata 17 17 0 pid 116 128 128 32 1 : tunables 0 0 0 : slabdata 4 4 0 perf_event 0 0 848 19 4 : tunables 0 0 0 : slabdata 0 0 0 pool_workqueue 29 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 maple_node 119 252 768 21 4 : tunables 0 0 0 : slabdata 12 12 0 radix_tree_node 1842 1848 320 12 1 : tunables 0 0 0 : slabdata 154 154 0 task_group 31 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 mm_struct 31 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 vmap_area 139 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 pgtable-2^10 31 32 12288 2 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-cg-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-cg-4k 5 8 12288 2 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-2k 16 20 6144 5 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-1k 40 40 3072 10 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-512 15 42 1536 21 8 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-256 27 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-192 24 54 216 18 1 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-128 18 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-96 40 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-64 40 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-32 28 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-16 18 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-8 23 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 dma-kmalloc-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-96 32 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-rcl-64 132 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0 kmalloc-rcl-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8k 15 16 24576 1 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-4k 14 16 12288 2 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-2k 380 380 6144 5 8 : tunables 0 0 0 : slabdata 76 76 0 kmalloc-1k 287 290 3072 10 8 : tunables 0 0 0 : slabdata 29 29 0 kmalloc-512 611 630 1536 21 8 : tunables 0 0 0 : slabdata 30 30 0 kmalloc-256 179 189 768 21 4 : tunables 0 0 0 : slabdata 9 9 0 kmalloc-192 4306 4320 216 18 1 : tunables 0 0 0 : slabdata 240 240 0 kmalloc-128 722 735 384 21 2 : tunables 0 0 0 : slabdata 35 35 0 kmalloc-96 724 748 120 34 1 : tunables 0 0 0 : slabdata 22 22 0 kmalloc-64 1346 2079 192 21 1 : tunables 0 0 0 : slabdata 99 99 0 kmalloc-32 1831 1890 96 42 1 : tunables 0 0 0 : slabdata 45 45 0 kmalloc-16 3262 3315 48 85 1 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-8 2862 2944 32 128 1 : tunables 0 0 0 : slabdata 23 23 0 kmem_cache_node 279 294 96 42 1 : tunables 0 0 0 : slabdata 7 7 0 kmem_cache 279 294 192 21 1 : tunables 0 0 0 : slabdata 14 14 0 slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ext4_groupinfo_1k 48 48 168 24 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_groupinfo_4k 320 322 176 23 1 : tunables 0 0 0 : slabdata 14 14 0 btrfs_prelim_ref 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_extent_op 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_data_ref 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_tree_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_ref_head 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_defrag 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_node 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_ordered_extent 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_extent_map 3 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 bio-232 2 14 288 14 1 : tunables 0 0 0 : slabdata 1 1 0 bio-280 4 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_buffer 11 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_state 3 46 88 46 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_free_space_bitmap 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 btrfs_free_space 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_path 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_trans_handle 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 bio-296 2 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_inode 3 16 1008 16 4 : tunables 0 0 0 : slabdata 1 1 0 bio-328 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 zspage 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 zs_handle 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 zswap_entry 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 p9_req_t 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 ip6-frags 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0 fib6_nodes 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ip6_dst_cache 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 PINGv6 0 0 1056 15 4 : tunables 0 0 0 : slabdata 0 0 0 RAWv6 7 15 1056 15 4 : tunables 0 0 0 : slabdata 1 1 0 UDPLITEv6 0 0 1152 14 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 1 14 1152 14 4 : tunables 0 0 0 : slabdata 1 1 0 tw_sock_TCPv6 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCPv6 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 2 15 2080 15 8 : tunables 0 0 0 : slabdata 1 1 0 scsi_sense_cache 9 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 virtio_scsi_cmd 64 64 256 16 1 : tunables 0 0 0 : slabdata 4 4 0 io_kiocb 0 0 224 18 1 : tunables 0 0 0 : slabdata 0 0 0 bfq_io_cq 0 0 1032 15 4 : tunables 0 0 0 : slabdata 0 0 0 bfq_queue 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 bio-88 10 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 bio-168 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 mqueue_inode_cache 1 21 768 21 4 : tunables 0 0 0 : slabdata 1 1 0 v9fs_inode_cache 0 0 600 13 2 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 560 14 2 : tunables 0 0 0 : slabdata 0 0 0 jbd2_transaction_s 2 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_inode 23 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_handle 0 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_head 8 84 96 42 1 : tunables 0 0 0 : slabdata 2 2 0 jbd2_revoke_table_s 2 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_revoke_record_s 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_fc_dentry_update 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 1350 1360 1008 16 4 : tunables 0 0 0 : slabdata 85 85 0 ext4_free_data 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_allocation_context 0 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_prealloc_space 6 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_system_zone 89 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_io_end_vec 0 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_io_end 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 bio_post_read_ctx 128 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0 pending_reservation 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 extent_status 1079 1095 56 73 1 : tunables 0 0 0 : slabdata 15 15 0 mbcache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 kioctx 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 aio_kiocb 0 0 160 25 1 : tunables 0 0 0 : slabdata 0 0 0 userfaultfd_ctx_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_path_event 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 fsnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_struct 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 dio 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 fasync_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 pid_namespace 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 UNIX-STREAM 27 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 UNIX 13 17 928 17 4 : tunables 0 0 0 : slabdata 1 1 0 ip4-frags 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 tcp_bind2_bucket 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 tcp_bind_bucket 2 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 inet_peer_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_dst_cache 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_state 0 0 640 12 2 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_fib_alias 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_dst_cache 6 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 PING 0 0 896 18 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 5 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 UDP 18 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCP 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCP 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 TCP 3 16 1952 16 8 : tunables 0 0 0 : slabdata 1 1 0 bio-184 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 ep_head 36 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_pwq 37 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_epi 36 96 128 32 1 : tunables 0 0 0 : slabdata 3 3 0 inotify_inode_mark 40 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 sgpool-128 2 12 2624 12 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-64 2 12 1344 12 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-32 2 23 704 23 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 sgpool-8 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 request_queue 4 22 720 22 4 : tunables 0 0 0 : slabdata 1 1 0 blkdev_ioc 1 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 bio-136 4 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 biovec-max 48 60 3136 10 8 : tunables 0 0 0 : slabdata 6 6 0 biovec-128 0 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 biovec-64 0 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0 biovec-16 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 ksm_mm_slot 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_stable_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_rmap_item 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 user_namespace 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 uid_cache 4 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 92 108 672 12 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_ext_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 skbuff_small_head 111 119 480 17 2 : tunables 0 0 0 : slabdata 7 7 0 skbuff_fclone_cache 0 19 416 19 2 : tunables 0 0 0 : slabdata 1 1 0 skbuff_head_cache 352 378 224 18 1 : tunables 0 0 0 : slabdata 21 21 0 file_lock_cache 1 24 168 24 1 : tunables 0 0 0 : slabdata 1 1 0 file_lock_ctx 10 192 64 64 1 : tunables 0 0 0 : slabdata 3 3 0 fsnotify_mark_connector 33 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 buffer_head 5653 5694 104 39 1 : tunables 0 0 0 : slabdata 146 146 0 task_delay_info 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 taskstats 2 17 456 17 2 : tunables 0 0 0 : slabdata 1 1 0 proc_dir_entry 274 286 152 26 1 : tunables 0 0 0 : slabdata 11 11 0 pde_opener 0 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 proc_inode_cache 192 238 568 14 2 : tunables 0 0 0 : slabdata 17 17 0 seq_file 2 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 sigqueue 1 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 bdev_cache 27 39 1216 13 4 : tunables 0 0 0 : slabdata 3 3 0 shmem_inode_cache 1313 1425 632 25 4 : tunables 0 0 0 : slabdata 57 57 0 kernfs_iattrs_cache 11 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 kernfs_node_cache 17325 17388 112 36 1 : tunables 0 0 0 : slabdata 483 483 0 mnt_cache 29 36 224 18 1 : tunables 0 0 0 : slabdata 2 2 0 filp 475 496 256 16 1 : tunables 0 0 0 : slabdata 31 31 0 inode_cache 2406 2415 528 15 2 : tunables 0 0 0 : slabdata 161 161 0 dentry 6379 6532 176 23 1 : tunables 0 0 0 : slabdata 284 284 0 names_cache 0 28 4160 7 8 : tunables 0 0 0 : slabdata 4 4 0 net_namespace 0 0 2624 12 8 : tunables 0 0 0 : slabdata 0 0 0 key_jar 7 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 uts_namespace 0 0 440 18 2 : tunables 0 0 0 : slabdata 0 0 0 nsproxy 1 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 vm_area_struct 1400 1428 96 42 1 : tunables 0 0 0 : slabdata 34 34 0 fs_cache 32 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 files_cache 32 42 288 14 1 : tunables 0 0 0 : slabdata 3 3 0 signal_cache 108 126 864 18 4 : tunables 0 0 0 : slabdata 7 7 0 sighand_cache 108 115 1376 23 8 : tunables 0 0 0 : slabdata 5 5 0 task_struct 108 110 2848 11 8 : tunables 0 0 0 : slabdata 10 10 0 cred_jar 158 189 192 21 1 : tunables 0 0 0 : slabdata 9 9 0 anon_vma_chain 1053 1168 56 73 1 : tunables 0 0 0 : slabdata 16 16 0 anon_vma 759 782 88 46 1 : tunables 0 0 0 : slabdata 17 17 0 pid 115 128 128 32 1 : tunables 0 0 0 : slabdata 4 4 0 perf_event 0 0 848 19 4 : tunables 0 0 0 : slabdata 0 0 0 pool_workqueue 29 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 maple_node 121 273 768 21 4 : tunables 0 0 0 : slabdata 13 13 0 radix_tree_node 1814 1824 320 12 1 : tunables 0 0 0 : slabdata 152 152 0 task_group 31 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 mm_struct 31 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 vmap_area 139 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 pgtable-2^10 31 32 12288 2 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-cg-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-cg-4k 5 8 12288 2 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-2k 16 20 6144 5 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-1k 40 40 3072 10 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-512 15 42 1536 21 8 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-256 27 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-192 24 54 216 18 1 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-128 18 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-96 40 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-64 40 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-32 28 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-16 18 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-8 23 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 dma-kmalloc-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-96 32 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-rcl-64 132 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0 kmalloc-rcl-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8k 15 16 24576 1 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-4k 14 16 12288 2 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-2k 349 350 6144 5 8 : tunables 0 0 0 : slabdata 70 70 0 kmalloc-1k 255 260 3072 10 8 : tunables 0 0 0 : slabdata 26 26 0 kmalloc-512 610 630 1536 21 8 : tunables 0 0 0 : slabdata 30 30 0 kmalloc-256 149 168 768 21 4 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-192 615 630 216 18 1 : tunables 0 0 0 : slabdata 35 35 0 kmalloc-128 721 735 384 21 2 : tunables 0 0 0 : slabdata 35 35 0 kmalloc-96 722 782 120 34 1 : tunables 0 0 0 : slabdata 23 23 0 kmalloc-64 1344 1974 192 21 1 : tunables 0 0 0 : slabdata 94 94 0 kmalloc-32 1831 1890 96 42 1 : tunables 0 0 0 : slabdata 45 45 0 kmalloc-16 3261 3315 48 85 1 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-8 2769 2816 32 128 1 : tunables 0 0 0 : slabdata 22 22 0 kmem_cache_node 217 252 96 42 1 : tunables 0 0 0 : slabdata 6 6 0 kmem_cache 217 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0 slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ext4_groupinfo_1k 48 48 168 24 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_groupinfo_4k 320 322 176 23 1 : tunables 0 0 0 : slabdata 14 14 0 btrfs_prelim_ref 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_extent_op 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_data_ref 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_tree_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_ref_head 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_defrag 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_node 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_ordered_extent 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_extent_map 3 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 bio-232 2 14 288 14 1 : tunables 0 0 0 : slabdata 1 1 0 bio-280 4 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_buffer 9 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_state 3 46 88 46 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_free_space_bitmap 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 btrfs_free_space 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_path 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_trans_handle 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 bio-296 2 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_inode 2 16 1008 16 4 : tunables 0 0 0 : slabdata 1 1 0 bio-328 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 10 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 10 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 8 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 8 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 7 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 7 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 7 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 8 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 9 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 5 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 5 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 5 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 8 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 8 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 5 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 8 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 8 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 5 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 5 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 4 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 4 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 11 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 11 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 4 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 4 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 9 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 9 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 8 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 8 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 7 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 5 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 8 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 9 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 4 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 4 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 7 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 7 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 8 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 8 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 7 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 7 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 5 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 5 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 9 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 9 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zspage 6 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 zs_handle 6 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0 zswap_entry 226 512 64 64 1 : tunables 0 0 0 : slabdata 8 8 0 p9_req_t 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 ip6-frags 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0 fib6_nodes 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ip6_dst_cache 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 PINGv6 0 0 1056 15 4 : tunables 0 0 0 : slabdata 0 0 0 RAWv6 7 15 1056 15 4 : tunables 0 0 0 : slabdata 1 1 0 UDPLITEv6 0 0 1152 14 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 1 14 1152 14 4 : tunables 0 0 0 : slabdata 1 1 0 tw_sock_TCPv6 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCPv6 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 2 15 2080 15 8 : tunables 0 0 0 : slabdata 1 1 0 scsi_sense_cache 9 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 virtio_scsi_cmd 64 64 256 16 1 : tunables 0 0 0 : slabdata 4 4 0 io_kiocb 0 0 224 18 1 : tunables 0 0 0 : slabdata 0 0 0 bfq_io_cq 0 0 1032 15 4 : tunables 0 0 0 : slabdata 0 0 0 bfq_queue 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 bio-88 10 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 bio-168 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 mqueue_inode_cache 1 21 768 21 4 : tunables 0 0 0 : slabdata 1 1 0 v9fs_inode_cache 0 0 600 13 2 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 560 14 2 : tunables 0 0 0 : slabdata 0 0 0 jbd2_transaction_s 3 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_inode 16 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_handle 0 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_head 7 84 96 42 1 : tunables 0 0 0 : slabdata 2 2 0 jbd2_revoke_table_s 2 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_revoke_record_s 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_fc_dentry_update 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 564 912 1008 16 4 : tunables 0 0 0 : slabdata 57 57 0 ext4_free_data 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_allocation_context 0 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_prealloc_space 6 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_system_zone 89 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_io_end_vec 0 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_io_end 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 bio_post_read_ctx 128 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0 pending_reservation 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 extent_status 45 292 56 73 1 : tunables 0 0 0 : slabdata 4 4 0 mbcache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 kioctx 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 aio_kiocb 0 0 160 25 1 : tunables 0 0 0 : slabdata 0 0 0 userfaultfd_ctx_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_path_event 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 fsnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_struct 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 dio 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 fasync_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 pid_namespace 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 UNIX-STREAM 29 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 UNIX 13 17 928 17 4 : tunables 0 0 0 : slabdata 1 1 0 ip4-frags 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 tcp_bind2_bucket 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 tcp_bind_bucket 2 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 inet_peer_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_dst_cache 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_state 0 0 640 12 2 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_fib_alias 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_dst_cache 6 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 PING 0 0 896 18 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 5 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 UDP 18 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCP 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCP 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 TCP 3 16 1952 16 8 : tunables 0 0 0 : slabdata 1 1 0 bio-184 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 ep_head 40 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_pwq 42 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_epi 40 96 128 32 1 : tunables 0 0 0 : slabdata 3 3 0 inotify_inode_mark 40 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 sgpool-128 2 12 2624 12 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-64 2 12 1344 12 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-32 2 23 704 23 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 sgpool-8 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 request_queue 4 22 720 22 4 : tunables 0 0 0 : slabdata 1 1 0 blkdev_ioc 1 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 bio-136 4 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 biovec-max 48 60 3136 10 8 : tunables 0 0 0 : slabdata 6 6 0 biovec-128 0 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 biovec-64 0 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0 biovec-16 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 ksm_mm_slot 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_stable_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_rmap_item 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 user_namespace 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 uid_cache 4 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 94 108 672 12 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_ext_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 skbuff_small_head 158 170 480 17 2 : tunables 0 0 0 : slabdata 10 10 0 skbuff_fclone_cache 0 19 416 19 2 : tunables 0 0 0 : slabdata 1 1 0 skbuff_head_cache 418 450 224 18 1 : tunables 0 0 0 : slabdata 25 25 0 file_lock_cache 1 24 168 24 1 : tunables 0 0 0 : slabdata 1 1 0 file_lock_ctx 8 192 64 64 1 : tunables 0 0 0 : slabdata 3 3 0 fsnotify_mark_connector 33 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 buffer_head 126 1248 104 39 1 : tunables 0 0 0 : slabdata 32 32 0 task_delay_info 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 taskstats 2 17 456 17 2 : tunables 0 0 0 : slabdata 1 1 0 proc_dir_entry 274 286 152 26 1 : tunables 0 0 0 : slabdata 11 11 0 pde_opener 0 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 proc_inode_cache 834 910 568 14 2 : tunables 0 0 0 : slabdata 65 65 0 seq_file 2 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 sigqueue 1 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 bdev_cache 27 39 1216 13 4 : tunables 0 0 0 : slabdata 3 3 0 shmem_inode_cache 1315 1425 632 25 4 : tunables 0 0 0 : slabdata 57 57 0 kernfs_iattrs_cache 11 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 kernfs_node_cache 17359 17424 112 36 1 : tunables 0 0 0 : slabdata 484 484 0 mnt_cache 29 36 224 18 1 : tunables 0 0 0 : slabdata 2 2 0 filp 526 592 256 16 1 : tunables 0 0 0 : slabdata 37 37 0 inode_cache 756 1575 528 15 2 : tunables 0 0 0 : slabdata 105 105 0 dentry 3245 6348 176 23 1 : tunables 0 0 0 : slabdata 276 276 0 names_cache 0 21 4160 7 8 : tunables 0 0 0 : slabdata 3 3 0 net_namespace 0 0 2624 12 8 : tunables 0 0 0 : slabdata 0 0 0 key_jar 7 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 uts_namespace 0 0 440 18 2 : tunables 0 0 0 : slabdata 0 0 0 nsproxy 1 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 vm_area_struct 1570 1638 96 42 1 : tunables 0 0 0 : slabdata 39 39 0 fs_cache 34 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 files_cache 34 42 288 14 1 : tunables 0 0 0 : slabdata 3 3 0 signal_cache 111 126 864 18 4 : tunables 0 0 0 : slabdata 7 7 0 sighand_cache 111 138 1376 23 8 : tunables 0 0 0 : slabdata 6 6 0 task_struct 111 121 2848 11 8 : tunables 0 0 0 : slabdata 11 11 0 cred_jar 163 189 192 21 1 : tunables 0 0 0 : slabdata 9 9 0 anon_vma_chain 1142 1314 56 73 1 : tunables 0 0 0 : slabdata 18 18 0 anon_vma 812 874 88 46 1 : tunables 0 0 0 : slabdata 19 19 0 pid 118 128 128 32 1 : tunables 0 0 0 : slabdata 4 4 0 perf_event 0 0 848 19 4 : tunables 0 0 0 : slabdata 0 0 0 pool_workqueue 29 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 maple_node 130 294 768 21 4 : tunables 0 0 0 : slabdata 14 14 0 radix_tree_node 1698 1908 320 12 1 : tunables 0 0 0 : slabdata 159 159 0 task_group 33 48 256 16 1 : tunables 0 0 0 : slabdata 3 3 0 mm_struct 33 51 928 17 4 : tunables 0 0 0 : slabdata 3 3 0 vmap_area 141 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 pgtable-2^10 33 38 12288 2 8 : tunables 0 0 0 : slabdata 19 19 0 kmalloc-cg-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-cg-4k 5 8 12288 2 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-2k 16 20 6144 5 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-1k 43 50 3072 10 8 : tunables 0 0 0 : slabdata 5 5 0 kmalloc-cg-512 16 42 1536 21 8 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-256 27 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-192 25 54 216 18 1 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-128 18 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-96 41 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-64 40 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-32 29 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-16 18 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-8 23 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 dma-kmalloc-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-96 32 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-rcl-64 122 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0 kmalloc-rcl-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8k 15 16 24576 1 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-4k 14 16 12288 2 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-2k 342 370 6144 5 8 : tunables 0 0 0 : slabdata 74 74 0 kmalloc-1k 256 300 3072 10 8 : tunables 0 0 0 : slabdata 30 30 0 kmalloc-512 619 630 1536 21 8 : tunables 0 0 0 : slabdata 30 30 0 kmalloc-256 177 189 768 21 4 : tunables 0 0 0 : slabdata 9 9 0 kmalloc-192 4218 4320 216 18 1 : tunables 0 0 0 : slabdata 240 240 0 kmalloc-128 722 735 384 21 2 : tunables 0 0 0 : slabdata 35 35 0 kmalloc-96 722 782 120 34 1 : tunables 0 0 0 : slabdata 23 23 0 kmalloc-64 1325 2058 192 21 1 : tunables 0 0 0 : slabdata 98 98 0 kmalloc-32 1768 1890 96 42 1 : tunables 0 0 0 : slabdata 45 45 0 kmalloc-16 3252 3315 48 85 1 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-8 2867 2944 32 128 1 : tunables 0 0 0 : slabdata 23 23 0 kmem_cache_node 279 294 96 42 1 : tunables 0 0 0 : slabdata 7 7 0 kmem_cache 279 294 192 21 1 : tunables 0 0 0 : slabdata 14 14 0 slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ext4_groupinfo_1k 48 48 168 24 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_groupinfo_4k 320 322 176 23 1 : tunables 0 0 0 : slabdata 14 14 0 btrfs_prelim_ref 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_extent_op 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_data_ref 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_tree_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_ref_head 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_defrag 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_delayed_node 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_ordered_extent 0 0 304 13 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_extent_map 3 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 bio-232 2 14 288 14 1 : tunables 0 0 0 : slabdata 1 1 0 bio-280 4 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_buffer 9 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_extent_state 3 46 88 46 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_free_space_bitmap 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 btrfs_free_space 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_path 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 btrfs_trans_handle 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 bio-296 2 23 352 23 2 : tunables 0 0 0 : slabdata 1 1 0 btrfs_inode 3 16 1008 16 4 : tunables 0 0 0 : slabdata 1 1 0 bio-328 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 zspage 45 256 64 64 1 : tunables 0 0 0 : slabdata 4 4 0 zs_handle 131 1190 24 170 1 : tunables 0 0 0 : slabdata 7 7 0 zswap_entry 131 512 64 64 1 : tunables 0 0 0 : slabdata 8 8 0 p9_req_t 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 ip6-frags 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0 fib6_nodes 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ip6_dst_cache 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 PINGv6 0 0 1056 15 4 : tunables 0 0 0 : slabdata 0 0 0 RAWv6 7 15 1056 15 4 : tunables 0 0 0 : slabdata 1 1 0 UDPLITEv6 0 0 1152 14 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 1 14 1152 14 4 : tunables 0 0 0 : slabdata 1 1 0 tw_sock_TCPv6 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCPv6 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 2 15 2080 15 8 : tunables 0 0 0 : slabdata 1 1 0 scsi_sense_cache 9 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 virtio_scsi_cmd 64 64 256 16 1 : tunables 0 0 0 : slabdata 4 4 0 io_kiocb 0 0 224 18 1 : tunables 0 0 0 : slabdata 0 0 0 bfq_io_cq 0 0 1032 15 4 : tunables 0 0 0 : slabdata 0 0 0 bfq_queue 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 bio-88 10 25 160 25 1 : tunables 0 0 0 : slabdata 1 1 0 bio-168 4 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 mqueue_inode_cache 1 21 768 21 4 : tunables 0 0 0 : slabdata 1 1 0 v9fs_inode_cache 0 0 600 13 2 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 560 14 2 : tunables 0 0 0 : slabdata 0 0 0 jbd2_transaction_s 2 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_inode 25 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_handle 0 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_head 7 84 96 42 1 : tunables 0 0 0 : slabdata 2 2 0 jbd2_revoke_table_s 2 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_revoke_record_s 0 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_fc_dentry_update 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 1349 1376 1008 16 4 : tunables 0 0 0 : slabdata 86 86 0 ext4_free_data 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_allocation_context 0 28 144 28 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_prealloc_space 6 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_system_zone 89 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 ext4_io_end_vec 0 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_io_end 0 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 bio_post_read_ctx 128 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0 pending_reservation 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 extent_status 435 1095 56 73 1 : tunables 0 0 0 : slabdata 15 15 0 mbcache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 kioctx 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 aio_kiocb 0 0 160 25 1 : tunables 0 0 0 : slabdata 0 0 0 userfaultfd_ctx_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_path_event 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 fsnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 dnotify_struct 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 dio 0 0 416 19 2 : tunables 0 0 0 : slabdata 0 0 0 fasync_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 pid_namespace 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 UNIX-STREAM 29 34 928 17 4 : tunables 0 0 0 : slabdata 2 2 0 UNIX 13 17 928 17 4 : tunables 0 0 0 : slabdata 1 1 0 ip4-frags 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 tcp_bind2_bucket 5 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 tcp_bind_bucket 2 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 inet_peer_cache 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_dst_cache 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 xfrm_state 0 0 640 12 2 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_fib_alias 7 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 ip_dst_cache 6 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 PING 0 0 896 18 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 5 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 UDP 18 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 tw_sock_TCP 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 request_sock_TCP 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 TCP 3 16 1952 16 8 : tunables 0 0 0 : slabdata 1 1 0 bio-184 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 ep_head 40 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_pwq 42 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_epi 40 96 128 32 1 : tunables 0 0 0 : slabdata 3 3 0 inotify_inode_mark 40 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 sgpool-128 2 12 2624 12 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-64 2 12 1344 12 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-32 2 23 704 23 4 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 2 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 sgpool-8 2 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 request_queue 4 22 720 22 4 : tunables 0 0 0 : slabdata 1 1 0 blkdev_ioc 1 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 bio-136 4 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 biovec-max 48 60 3136 10 8 : tunables 0 0 0 : slabdata 6 6 0 biovec-128 0 20 1600 20 8 : tunables 0 0 0 : slabdata 1 1 0 biovec-64 0 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0 biovec-16 0 16 256 16 1 : tunables 0 0 0 : slabdata 1 1 0 ksm_mm_slot 0 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 ksm_stable_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 ksm_rmap_item 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0 user_namespace 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 uid_cache 4 21 192 21 1 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 94 108 672 12 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_ext_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 skbuff_small_head 145 153 480 17 2 : tunables 0 0 0 : slabdata 9 9 0 skbuff_fclone_cache 0 19 416 19 2 : tunables 0 0 0 : slabdata 1 1 0 skbuff_head_cache 416 432 224 18 1 : tunables 0 0 0 : slabdata 24 24 0 file_lock_cache 1 24 168 24 1 : tunables 0 0 0 : slabdata 1 1 0 file_lock_ctx 10 192 64 64 1 : tunables 0 0 0 : slabdata 3 3 0 fsnotify_mark_connector 33 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 buffer_head 214 1560 104 39 1 : tunables 0 0 0 : slabdata 40 40 0 task_delay_info 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 taskstats 2 17 456 17 2 : tunables 0 0 0 : slabdata 1 1 0 proc_dir_entry 274 286 152 26 1 : tunables 0 0 0 : slabdata 11 11 0 pde_opener 0 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 proc_inode_cache 513 560 568 14 2 : tunables 0 0 0 : slabdata 40 40 0 seq_file 2 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 sigqueue 1 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 bdev_cache 27 39 1216 13 4 : tunables 0 0 0 : slabdata 3 3 0 shmem_inode_cache 1313 1425 632 25 4 : tunables 0 0 0 : slabdata 57 57 0 kernfs_iattrs_cache 11 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0 kernfs_node_cache 17313 17388 112 36 1 : tunables 0 0 0 : slabdata 483 483 0 mnt_cache 29 36 224 18 1 : tunables 0 0 0 : slabdata 2 2 0 filp 527 576 256 16 1 : tunables 0 0 0 : slabdata 36 36 0 inode_cache 896 1620 528 15 2 : tunables 0 0 0 : slabdata 108 108 0 dentry 4324 6095 176 23 1 : tunables 0 0 0 : slabdata 265 265 0 names_cache 0 28 4160 7 8 : tunables 0 0 0 : slabdata 4 4 0 net_namespace 0 0 2624 12 8 : tunables 0 0 0 : slabdata 0 0 0 key_jar 7 18 224 18 1 : tunables 0 0 0 : slabdata 1 1 0 uts_namespace 0 0 440 18 2 : tunables 0 0 0 : slabdata 0 0 0 nsproxy 1 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 vm_area_struct 1568 1680 96 42 1 : tunables 0 0 0 : slabdata 40 40 0 fs_cache 34 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 files_cache 34 42 288 14 1 : tunables 0 0 0 : slabdata 3 3 0 signal_cache 110 126 864 18 4 : tunables 0 0 0 : slabdata 7 7 0 sighand_cache 110 115 1376 23 8 : tunables 0 0 0 : slabdata 5 5 0 task_struct 110 121 2848 11 8 : tunables 0 0 0 : slabdata 11 11 0 cred_jar 162 189 192 21 1 : tunables 0 0 0 : slabdata 9 9 0 anon_vma_chain 1137 1460 56 73 1 : tunables 0 0 0 : slabdata 20 20 0 anon_vma 807 920 88 46 1 : tunables 0 0 0 : slabdata 20 20 0 pid 117 128 128 32 1 : tunables 0 0 0 : slabdata 4 4 0 perf_event 0 0 848 19 4 : tunables 0 0 0 : slabdata 0 0 0 pool_workqueue 29 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 maple_node 129 273 768 21 4 : tunables 0 0 0 : slabdata 13 13 0 radix_tree_node 1827 1908 320 12 1 : tunables 0 0 0 : slabdata 159 159 0 task_group 33 48 256 16 1 : tunables 0 0 0 : slabdata 3 3 0 mm_struct 33 51 928 17 4 : tunables 0 0 0 : slabdata 3 3 0 vmap_area 141 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 pgtable-2^10 33 38 12288 2 8 : tunables 0 0 0 : slabdata 19 19 0 kmalloc-cg-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-cg-4k 5 8 12288 2 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-2k 16 20 6144 5 8 : tunables 0 0 0 : slabdata 4 4 0 kmalloc-cg-1k 43 50 3072 10 8 : tunables 0 0 0 : slabdata 5 5 0 kmalloc-cg-512 16 63 1536 21 8 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-256 27 42 768 21 4 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-192 25 54 216 18 1 : tunables 0 0 0 : slabdata 3 3 0 kmalloc-cg-128 18 21 384 21 2 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-96 41 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-64 40 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-cg-32 29 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-16 18 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-cg-8 23 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0 dma-kmalloc-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8k 0 0 24576 1 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-4k 0 0 12288 2 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-2k 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-1k 0 0 3072 10 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-512 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-256 0 0 768 21 4 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-192 0 0 216 18 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-128 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-96 32 68 120 34 1 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-rcl-64 128 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0 kmalloc-rcl-32 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-16 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-rcl-8 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8k 15 16 24576 1 8 : tunables 0 0 0 : slabdata 16 16 0 kmalloc-4k 14 16 12288 2 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-2k 343 355 6144 5 8 : tunables 0 0 0 : slabdata 71 71 0 kmalloc-1k 289 290 3072 10 8 : tunables 0 0 0 : slabdata 29 29 0 kmalloc-512 620 630 1536 21 8 : tunables 0 0 0 : slabdata 30 30 0 kmalloc-256 149 168 768 21 4 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-192 562 648 216 18 1 : tunables 0 0 0 : slabdata 36 36 0 kmalloc-128 721 735 384 21 2 : tunables 0 0 0 : slabdata 35 35 0 kmalloc-96 708 782 120 34 1 : tunables 0 0 0 : slabdata 23 23 0 kmalloc-64 1328 1974 192 21 1 : tunables 0 0 0 : slabdata 94 94 0 kmalloc-32 1829 1890 96 42 1 : tunables 0 0 0 : slabdata 45 45 0 kmalloc-16 3263 3315 48 85 1 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-8 2774 2816 32 128 1 : tunables 0 0 0 : slabdata 22 22 0 kmem_cache_node 217 252 96 42 1 : tunables 0 0 0 : slabdata 6 6 0 kmem_cache 217 231 192 21 1 : tunables 0 0 0 : slabdata 11 11 0
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
Quoted 5 lines
Thanks for looking into this! I got you 'cat
/proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.Good was
6be3601517d90b728095d70c14f3a04b9adcb166, bad wasb8cf32dc6e8c75b712cbf638e0fd210101c22f17which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.The patch applied to git
b8cf32dc6e8c75b712cbf638e0fd210101c22f17unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.
Thanks for trying this out. The patch reduces the amount of wasted memory due to the 'zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).
I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.
The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.
Quoted 3 lines
Regards,
Erhard
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
Quoted 5 lines
Thanks for looking into this! I got you 'cat
/proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.Good was
6be3601517d90b728095d70c14f3a04b9adcb166, bad wasb8cf32dc6e8c75b712cbf638e0fd210101c22f17which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.The patch applied to git
b8cf32dc6e8c75b712cbf638e0fd210101c22f17unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.Thanks for trying this out. The patch reduces the amount of wasted memory due to the '
zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).
Quoted 8 lines
I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.
The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.
Hi Erhard,
If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.
The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.
Thank you.
Quoted 39 lines · Yu Zhao
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
Quoted 5 lines
Thanks for looking into this! I got you 'cat
/proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.Good was
6be3601517d90b728095d70c14f3a04b9adcb166, bad wasb8cf32dc6e8c75b712cbf638e0fd210101c22f17which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.The patch applied to git
b8cf32dc6e8c75b712cbf638e0fd210101c22f17unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.Thanks for trying this out. The patch reduces the amount of wasted memory due to the '
zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).Quoted 8 lines
I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.
The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.
Quoted 8 lines
Hi Erhard,
If it's not too much trouble, could you "grep
nr_zspages/proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.
I thought about the internal fragmentation of pools, but zsmalloc should have access to highmem, and if I understand correctly the problem here is that we are running out of space in the DMA zone when making kernel allocations.
Do you suspect zsmalloc is allocating memory from the DMA zone initially, even though it has access to highmem?
Quoted 2 lines
Thank you.
Quoted 50 lines · Yosry Ahmed
Quoted 39 lines · Yu Zhao
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
Quoted 5 lines
Thanks for looking into this! I got you 'cat
/proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.Good was
6be3601517d90b728095d70c14f3a04b9adcb166, bad wasb8cf32dc6e8c75b712cbf638e0fd210101c22f17which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.The patch applied to git
b8cf32dc6e8c75b712cbf638e0fd210101c22f17unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.Thanks for trying this out. The patch reduces the amount of wasted memory due to the '
zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).Quoted 8 lines
I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.
The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.
Quoted 8 lines
Hi Erhard,
If it's not too much trouble, could you "grep
nr_zspages/proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.
Quoted 7 lines
I thought about the internal fragmentation of pools, but zsmalloc should have access to highmem, and if I understand correctly the problem here is that we are running out of space in the DMA zone when making kernel allocations.
Do you suspect zsmalloc is allocating memory from the DMA zone initially, even though it has access to highmem?
There was a lot of user memory in the DMA zone. So at a point the highmem zone was full and allocation fallback happened.
The problem with zone fallback is that recent allocations go into lower zones, meaning they are further back on the LRU list. This applies to both user memory and zsmalloc memory -- the latter has a writeback LRU. On top of this, neither the zswap shrinker nor the zsmalloc shrinker (compaction) is zone aware. So page reclaim might have trouble hitting the right target zone.
We can't really tell how zspages are distributed across zones, but the overall number might be helpful. It'd be great if someone could make nr_zspages per zone :)
Quoted 63 lines · Yu Zhao
Quoted 50 lines · Yosry Ahmed
Quoted 39 lines · Yu Zhao
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
Quoted 5 lines
Thanks for looking into this! I got you 'cat
/proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.Good was
6be3601517d90b728095d70c14f3a04b9adcb166, bad wasb8cf32dc6e8c75b712cbf638e0fd210101c22f17which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.The patch applied to git
b8cf32dc6e8c75b712cbf638e0fd210101c22f17unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.Thanks for trying this out. The patch reduces the amount of wasted memory due to the '
zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).Quoted 8 lines
I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.
The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.
Quoted 8 lines
Hi Erhard,
If it's not too much trouble, could you "grep
nr_zspages/proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.
Quoted 7 lines
I thought about the internal fragmentation of pools, but zsmalloc should have access to highmem, and if I understand correctly the problem here is that we are running out of space in the DMA zone when making kernel allocations.
Do you suspect zsmalloc is allocating memory from the DMA zone initially, even though it has access to highmem?
There was a lot of user memory in the DMA zone. So at a point the highmem zone was full and allocation fallback happened.
Quoted 6 lines
The problem with zone fallback is that recent allocations go into lower zones, meaning they are further back on the LRU list. This applies to both user memory and zsmalloc memory -- the latter has a writeback LRU. On top of this, neither the zswap shrinker nor the zsmalloc shrinker (compaction) is zone aware. So page reclaim might have trouble hitting the right target zone.
I see what you mean. In this case, yeah I think the internal fragmentation in the zsmalloc pools may be the reason behind the problem.
How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be max(nr_cpus, 32)?
Alternatively, the number of pools should scale with the memory size in some way, such that we only increase fragmentation when it's tolerable.
Quoted 4 lines
We can't really tell how zspages are distributed across zones, but the overall number might be helpful. It'd be great if someone could make
nr_zspagesper zone :)
Quoted 15 lines · Yosry Ahmed
Quoted 9 lines · Yu Zhao
There was a lot of user memory in the DMA zone. So at a point the highmem zone was full and allocation fallback happened.
The problem with zone fallback is that recent allocations go into lower zones, meaning they are further back on the LRU list. This applies to both user memory and zsmalloc memory -- the latter has a writeback LRU. On top of this, neither the zswap shrinker nor the zsmalloc shrinker (compaction) is zone aware. So page reclaim might have trouble hitting the right target zone.
I see what you mean. In this case, yeah I think the internal fragmentation in the zsmalloc pools may be the reason behind the problem.
Quoted 7 lines
How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be
max(nr_cpus, 32)?Alternatively, the number of pools should scale with the memory size in some way, such that we only increase fragmentation when it's tolerable.
Sounds like a good idea to me, maybe a combination of both. No point in trying to scale if there's no benefit and only downside of more memory consumption.
Quoted 3 lines · On Tue, 4 Jun 2024 11:01:39 -0700 Yosry Ahmed <[email protected]> wrote:
How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be
max(nr_cpus, 32)?
This PowerMac G4 DP got 2 CPUs. Not much for a desktop machine by todays standards but some SoCs have less. ;)
# lscpu Architecture: ppc CPU op-mode(s): 32-bit Byte Order: Big Endian CPU(s): 2 On-line CPU(s) list: 0,1 Model name: 7455, altivec supported Model: 3.3 (pvr 8001 0303) Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 2 BogoMIPS: 83.78 Caches (sum of all): L1d: 64 KiB (2 instances) L1i: 64 KiB (2 instances) L2: 512 KiB (2 instances) L3: 4 MiB (2 instances)
Regards,
Erhard
Quoted 28 lines · Erhard Furtner
Quoted 3 lines · On Tue, 4 Jun 2024 11:01:39 -0700 Yosry Ahmed <[email protected]> wrote:
How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be
max(nr_cpus, 32)?This PowerMac G4 DP got 2 CPUs. Not much for a desktop machine by todays standards but some SoCs have less. ;)
# lscpu Architecture: ppc CPU op-mode(s): 32-bit Byte Order: Big Endian CPU(s): 2 On-line CPU(s) list: 0,1 Model name: 7455, altivec supported Model: 3.3 (pvr 8001 0303) Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 2 BogoMIPS: 83.78 Caches (sum of all): L1d: 64 KiB (2 instances) L1i: 64 KiB (2 instances) L2: 512 KiB (2 instances) L3: 4 MiB (2 instances)
Quoted 2 lines
Regards,
Erhard
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).
From eb2baf6f3497936ec56fe17e22af5b5add624371 Mon Sep 17 00:00:00 2001
From: Yosry Ahmed <[email protected]>
Date: Wed, 5 Jun 2024 02:58:11 +0000
Subject: [PATCH] mm: zswap: do not scale the number of zpools unnecessarily
Zswap allocates several zpools per zswap pool for scalability. It currently allocates 32 zpools, an empirically determined magic number.
For small machines that has less than 32 CPUs, this doesn't make much sense. Allocating more zpools than CPUs will not significantly increase scalability, but it may increase internal fragmentation due to spreading of compressed pages among different zpools.
determined at boot as the smaller of 32 and the number of CPUs.
--- mm/zswap.c | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-)
mm/zswap.c +26 -9
@@ -123,8 +123,8 @@ static unsigned int zswap_accept_thr_percent = 90; /* of max pool size */module_param_named (accept_threshold_percent, zswap_accept_thr_percent, uint,0644 ); -/* Number of zpools in zswap_pool (empirically determined for scalability) */ -#define ZSWAP_NR_ZPOOLS 32 +/* Number of zpools in zswap_pool */ +static unsigned int nr_zswap_zpools__ro_after_init ;/* Enable/disable memory pressure-based shrinker. */ static bool zswap_shrinker_enabled = IS_ENABLED(
@@ -156,7 +156,7 @@ struct crypto_acomp_ctx {* needs to be verified that it's still valid in the tree. */ struct zswap_pool { -struct zpool *zpools[ZSWAP_NR_ZPOOLS ]; +struct zpool **zpools;struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref;struct list_head list;
@@ -268,7 +268,14 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor)if (!pool)return NULL ; -for (i =0 ; i <ZSWAP_NR_ZPOOLS ; i++) { + pool->zpools =kcalloc (nr_zswap_zpools,sizeof (pool->zpools[0 ]), +GFP_KERNEL ); +if (!pool->zpools) { +pr_err ("zpools alloc failed\n" ); + goto error; + } + +for (i =0 ; i < nr_zswap_zpools; i++) {/* unique name for each pool specifically required by zsmalloc */ snprintf (name,38 ,"zswap%x" ,atomic_inc_return (&zswap_pools_count));
@@ -312,8 +319,11 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) error:if (pool->acomp_ctx)free_percpu (pool->acomp_ctx); -while (i--) -zpool_destroy_pool (pool->zpools[i]); +if (pool->zpools) { +while (i--) +zpool_destroy_pool (pool->zpools[i]); +kfree (pool->zpools); + }kfree (pool);return NULL ; }
@@ -369,8 +379,9 @@ static void zswap_pool_destroy(struct zswap_pool *pool)cpuhp_state_remove_instance (CPUHP_MM_ZSWP_POOL_PREPARE , &pool->node);free_percpu (pool->acomp_ctx); -for (i =0 ; i <ZSWAP_NR_ZPOOLS ; i++) +for (i =0 ; i < nr_zswap_zpools; i++)zpool_destroy_pool (pool->zpools[i]); +kfree (pool->zpools);kfree (pool); }
@@ -496,7 +507,7 @@ unsigned long zswap_total_pages(void)list_for_each_entry_rcu (pool, &zswap_pools, list) {int i; -for (i =0 ; i <ZSWAP_NR_ZPOOLS ; i++) +for (i =0 ; i < nr_zswap_zpools; i++) total +=zpool_get_total_pages (pool->zpools[i]); }rcu_read_unlock ();
@@ -805,7 +816,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)static struct zpool *zswap_find_zpool (struct zswap_entry *entry) { -return entry->pool->zpools[hash_ptr (entry,ilog2 (ZSWAP_NR_ZPOOLS ))]; +return entry->pool->zpools[hash_ptr (entry,ilog2 (nr_zswap_zpools))]; }/*
@@ -1767,6 +1778,12 @@ static int zswap_setup(void)static int __init zswap_init (void ) { +/* +* Use multiple zpools for scalability, but avoid allocating too many. +* 32 is an empirically determined magic number. +*/ + nr_zswap_zpools =min (32U ,num_possible_cpus ()); +if (!zswap_enabled)return 0 ;return zswap_setup ();
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to
min(32, nr_cpus).
Thanks! The patch does not fix the issue but it helps.
Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.
Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.
The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.
Regards,
Erhard
Quoted 4 lines · Erhard Furtner
On Tue, 4 Jun 2024 20:03:27 -0700
Yosry Ahmed <[email protected]> wrote:
Quoted 8 lines
Quoted 2 lines
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to
min(32, nr_cpus).Thanks! The patch does not fix the issue but it helps.
Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.
Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).
(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).
(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).
The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
(e) A creative mix of the above.
(f) Something else (probably simpler).
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.
Quoted 5 lines
The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.
Regards,
Erhard
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to
min(32, nr_cpus).
Quoted 8 lines
Quoted 5 lines
Thanks! The patch does not fix the issue but it helps.
Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.
Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
Now I'm a little bit skeptical that the problem is due to fragmentation.
Quoted 33 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g.
SWAP_ADDRESS_SPACE_SHIFT).The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
Quoted 6 lines
(e) A creative mix of the above.
(f) Something else (probably simpler).
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,
static struct zpool *zswap_find_zpool(struct zswap_entry *entry) {
- return entry->pool->zpools[hash_ptr(entry, ilog2(ZSWAP_NR_ZPOOLS))];
+ return entry->pool->zpools[0];
}
Quoted 8 lines
In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.
Quoted 5 lines
The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.
Regards,
Erhard
Quoted 57 lines · Yu Zhao
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to
min(32, nr_cpus).Quoted 8 lines
Quoted 5 lines
Thanks! The patch does not fix the issue but it helps.
Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.
Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
Now I'm a little bit skeptical that the problem is due to fragmentation.
Quoted 35 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g.
SWAP_ADDRESS_SPACE_SHIFT).The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
(e) A creative mix of the above.
Quoted 8 lines
Quoted 4 lines
(f) Something else (probably simpler).
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,
Erhard, do you mind checking if Yu's diff below to use a single zpool fixes the problem completely? There is also an attached patch that does the same thing if this is easier to apply for you.
Quoted 15 lines
static struct zpool *zswap_find_zpool(struct
zswap_entry*entry) {
- return entry->pool->zpools[hash_ptr(entry,ilog2(ZSWAP_NR_ZPOOLS))];
+ return entry->pool->zpools[0];
}Quoted 8 lines
In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.
Quoted 5 lines
The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.
Regards,
Erhard
From c6c477dae9cb8bcdefaf1c1a0e8869efa8bfe3f9 Mon Sep 17 00:00:00 2001
From: Yosry Ahmed <[email protected]>
Date: Wed, 5 Jun 2024 23:56:15 +0000
Subject: [PATCH] mm: zswap: set ZSWAP_NR_ZPOOLS to 1
--- mm/zswap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/zswap.c +1 -1
@@ -124,7 +124,7 @@ module_param_named(accept_threshold_percent, zswap_accept_thr_percent, uint, 0644);/* Number of zpools in zswap_pool (empirically determined for scalability) */ -#define ZSWAP_NR_ZPOOLS 32 +#define ZSWAP_NR_ZPOOLS 1/* Enable/disable memory pressure-based shrinker. */ static bool zswap_shrinker_enabled = IS_ENABLED(
Signature · 1 line
2.45.1.467.gbab1589fc0-goog
Quoted 63 lines · On Wed, 5 Jun 2024 16:58:11 -0700 Yosry Ahmed <[email protected]> wrote:
Quoted 62 lines · Yu Zhao
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to
min(32, nr_cpus).Quoted 8 lines
Quoted 5 lines
Thanks! The patch does not fix the issue but it helps.
Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.
Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
Now I'm a little bit skeptical that the problem is due to fragmentation.
Quoted 39 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g.
SWAP_ADDRESS_SPACE_SHIFT).The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
(e) A creative mix of the above.
(f) Something else (probably simpler).
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
Quoted 7 lines
Quoted 3 lines
I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,
Erhard, do you mind checking if Yu's diff below to use a single zpool fixes the problem completely? There is also an attached patch that does the same thing if this is easier to apply for you.
No, setting ZSWAP_NR_ZPOOLS to 1 does not fix the problem unfortunately (that being the only patch applied on v6.10-rc2).
Trying to alter the lowmem and virtual mem limits next as Michael suggested.
Regards,
Erhard
Quoted 71 lines · Erhard Furtner
Quoted 67 lines · On Wed, 5 Jun 2024 16:58:11 -0700 Yosry Ahmed <[email protected]> wrote:
Quoted 65 lines · Yu Zhao
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to
min(32, nr_cpus).Quoted 8 lines
Quoted 5 lines
Thanks! The patch does not fix the issue but it helps.
Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.
Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
Now I'm a little bit skeptical that the problem is due to fragmentation.
Quoted 33 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g.
SWAP_ADDRESS_SPACE_SHIFT).The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
Quoted 6 lines
(e) A creative mix of the above.
(f) Something else (probably simpler).
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,
Quoted 5 lines
Quoted 3 lines
Erhard, do you mind checking if Yu's diff below to use a single zpool fixes the problem completely? There is also an attached patch that does the same thing if this is easier to apply for you.
No, setting
ZSWAP_NR_ZPOOLSto 1 does not fix the problem unfortunately (that being the only patch applied on v6.10-rc2).
This confirms Yu's theory that the zpools fragmentation is not the main reason for the problem. As Vlastimil said, the setup is already tight on memory and that commit may have just pushed it over the edge. Since setting ZSWAP_NR_ZPOOLS to 1 (which effectively reverts the commit) does not help in v6.10-rc2, then something else that came after the commit would have pushed it over the edge anyway.
Quoted 2 lines
Trying to alter the lowmem and virtual mem limits next as Michael suggested.
I saw that this worked. So it seems like we don't need to worry about the number of zpools, for now at least :)
Thanks for helping with the testing, and thanks to everyone else who helped on this thread.
Quoted 3 lines
Regards,
Erhard
Chengming Zhou
reviewerQuoted 52 lines · Yosry Ahmed
Quoted 4 lines · Erhard Furtner
On Tue, 4 Jun 2024 20:03:27 -0700
Yosry Ahmed <[email protected]> wrote:Quoted 8 lines
Quoted 2 lines
Could you check if the attached patch helps? It basically changes the number of zpools from 32 to
min(32, nr_cpus).Thanks! The patch does not fix the issue but it helps.
Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.
Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g.
SWAP_ADDRESS_SPACE_SHIFT).The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
(e) A creative mix of the above.
Quoted 8 lines
(f) Something else (probably simpler).
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.
Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:
1. First, we can change the pool global lock to per-class lock, which is more fine-grained.
2. Actually, we only need to take per-zspage lock when malloc/free, only need to take class lock when its fullness changed.
3. If this is not enough, we can have fewer fullness groups, so the need to take class lock becomes fewer. (will need some test data)
More comments are welcome. Thanks!
Sergey Senozhatsky
maintainerQuoted 38 lines · Chengming Zhou
Quoted 38 lines
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g.
SWAP_ADDRESS_SPACE_SHIFT).The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
(e) A creative mix of the above.
Quoted 7 lines
Quoted 7 lines
(f) Something else (probably simpler).
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.
Sorry, I'm sure I'm not following this discussion closely enough, has the lock contention been demonstrated/proved somehow? lock-stats?
Quoted 5 lines
Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:
1. First, we can change the pool global lock to per-class lock, which is more fine-grained.
Commit c0547d0b6a4b6 "zsmalloc: consolidate zs_pool's migrate_lock and size_class's locks" [1] claimed no significant difference between class->lock and pool->lock.
Chengming Zhou
reviewerQuoted 41 lines · Sergey Senozhatsky
Quoted 40 lines · Chengming Zhou
Quoted 40 lines
Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g.
SWAP_ADDRESS_SPACE_SHIFT).The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.
(e) A creative mix of the above.
(f) Something else (probably simpler).
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.
Sorry, I'm sure I'm not following this discussion closely enough, has the lock contention been demonstrated/proved somehow? lock-stats?
Yosry has some stats in his commit b8cf32dc6e8c ("mm: zswap: multiple zpools support"), and I have also seen some locking contention when using zram to test kernel building, since zram still has only one pool.
Quoted 10 lines
Quoted 6 lines
Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:
1. First, we can change the pool global lock to per-class lock, which is more fine-grained.
Commit
c0547d0b6a4b6"zsmalloc: consolidate zs_pool'smigrate_lockand size_class's locks" [1] claimed no significant difference betweenclass->lockandpool->lock.
Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.
Thanks.
Quoted 2 lines
Sergey Senozhatsky
maintainerQuoted 6 lines · Chengming Zhou
Quoted 6 lines
Quoted 5 lines
Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:
1. First, we can change the pool global lock to per-class lock, which is more fine-grained.
Quoted 7 lines
Quoted 3 lines
Commit
c0547d0b6a4b6"zsmalloc: consolidate zs_pool'smigrate_lockand size_class's locks" [1] claimed no significant difference betweenclass->lockandpool->lock.Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is
Well, the point that commit made was that Nhat (and Johannes?) were unable to detect any impact of pool->lock on a variety of cases. So we went on with code simplification.
Quoted 2 lines
obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.
Thanks, we'll need data on this. I'm happy to take the patch, but jumping back and forth between class->lock and pool->lock merely "for obvious reasons" is not what I'm extremely excited about.
Chengming Zhou
reviewerQuoted 11 lines · Sergey Senozhatsky
Quoted 10 lines · Chengming Zhou
Quoted 9 lines
Quoted 5 lines
Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:
1. First, we can change the pool global lock to per-class lock, which is more fine-grained.
Commit
c0547d0b6a4b6"zsmalloc: consolidate zs_pool'smigrate_lockand size_class's locks" [1] claimed no significant difference betweenclass->lockandpool->lock.
Quoted 7 lines
Quoted 3 lines
Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is
Well, the point that commit made was that Nhat (and Johannes?) were unable to detect any impact of
pool->lockon a variety of cases. So we went on with code simplification.
Right, the code is simpler.
Quoted 7 lines
Quoted 3 lines
obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.
Thanks, we'll need data on this. I'm happy to take the patch, but jumping back and forth between
class->lockandpool->lockmerely "for obvious reasons" is not what I'm extremely excited about.
Yeah, agree, we need test data.
Nhat Pham
maintainerQuoted 12 lines · Sergey Senozhatsky
Quoted 10 lines · Chengming Zhou
Quoted 9 lines
Quoted 5 lines
Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:
1. First, we can change the pool global lock to per-class lock, which is more fine-grained.
Commit
c0547d0b6a4b6"zsmalloc: consolidate zs_pool'smigrate_lockand size_class's locks" [1] claimed no significant difference betweenclass->lockandpool->lock.
Quoted 7 lines
Quoted 3 lines
Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is
Well, the point that commit made was that Nhat (and Johannes?) were unable to detect any impact of
pool->lockon a variety of cases. So we went on with code simplification.
Yeah, we benchmarked it before zsmalloc writeback was introduced (the patch to remove class lock was a prep patch of the series). We weren't able to detect any regression at the time with just using a global pool lock.
Quoted 7 lines
Quoted 3 lines
obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.
Thanks, we'll need data on this. I'm happy to take the patch, but jumping back and forth between
class->lockandpool->lockmerely "for obvious reasons" is not what I'm extremely excited about.
FWIW, I do think it'd be nice if we can make the locking more granular
- the pool lock now is essentially a global lock, and we're just getting around that by replicating the (z)pools themselves.
Personally, I'm not super convinced about class locks. We're essentially relying on the post-compression size of the data to load-balance the queries - I can imagine a scenario where a workload has a concentrated distribution of post-compression data (i.e its pages are compressed to similar-ish sizes), and we're once again contending for a (few) lock(s) again.
That said, I'll let the data tell the story :) We don't need a perfect solution, just a good enough solution for now.
Sergey Senozhatsky
maintainerQuoted 9 lines · Nhat Pham
Personally, I'm not super convinced about class locks. We're essentially relying on the post-compression size of the data to load-balance the queries - I can imagine a scenario where a workload has a concentrated distribution of post-compression data (i.e its pages are compressed to similar-ish sizes), and we're once again contending for a (few) lock(s) again.
That said, I'll let the data tell the story :) We don't need a perfect solution, just a good enough solution for now.
Speaking of size class locks:
One thing to mention is that zsmalloc merges size classes, we never have documented/claimed 256 size classe, the actual number is always much much lower. Each such "cluster" (merged size classes) holds a range of objects' sizes (e.g. 3504-3584 bytes). The wider the cluster's size range the more likely the (size class) lock contention is.
Setting CONFIG_ZSMALLOC_CHAIN_SIZE to 10 or higher makes zsmalloc pool to be configured with more size class clusters (which means that clusters hold narrower size intervals).
Quoted 4 lines · Yosry Ahmed
On Wed, Jun 5, 2024 at 4:04 PM Erhard Furtner <[email protected]> wrote:
I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?
Besides the zpool commit which might have just pushed the machine over the edge, but it was probably close to it already. I've noticed a more general problem that there are GFP_KERNEL allocations failing from kswapd. Those could probably use be __GFP_NOMEMALLOC (or scoped variant, is there one?) since it's the case of "allocating memory to free memory". Or use mempools if the progress (success will lead to freeing memory) is really guaranteed.
Another interesting data point could be to see if traditional reclaim behaves any better on this machine than MGLRU. I saw in the config:
CONFIG_LRU_GEN=yCONFIG_LRU_GEN_ENABLED=y
So disabling at least the second one would revert to the traditional reclaim and we could see if it handles such a constrained system better or not.
Quoted 9 lines
In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.
Quoted 5 lines
The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.
Regards,
Erhard
Quoted 7 lines · On Thu, 6 Jun 2024 09:24:56 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
Besides the zpool commit which might have just pushed the machine over the edge, but it was probably close to it already. I've noticed a more general problem that there are
GFP_KERNELallocations failing from kswapd. Those could probably use be__GFP_NOMEMALLOC(or scoped variant, is there one?) since it's the case of "allocating memory to free memory". Or use mempools if the progress (success will lead to freeing memory) is really guaranteed.
Quoted 8 lines
Another interesting data point could be to see if traditional reclaim behaves any better on this machine than MGLRU. I saw in the config:
CONFIG_LRU_GEN=yCONFIG_LRU_GEN_ENABLED=ySo disabling at least the second one would revert to the traditional reclaim and we could see if it handles such a constrained system better or not.
I set RANDOM_KMALLOC_CACHES=n and LRU_GEN_ENABLED=n but still hit the issue.
dmesg looks a bit different (unpatched v6.10-rc2).
Regards,
Erhard
Quoted 16 lines · Erhard Furtner
Quoted 13 lines · On Thu, 6 Jun 2024 09:24:56 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
Besides the zpool commit which might have just pushed the machine over the edge, but it was probably close to it already. I've noticed a more general problem that there are
GFP_KERNELallocations failing from kswapd. Those could probably use be__GFP_NOMEMALLOC(or scoped variant, is there one?) since it's the case of "allocating memory to free memory". Or use mempools if the progress (success will lead to freeing memory) is really guaranteed.Another interesting data point could be to see if traditional reclaim behaves any better on this machine than MGLRU. I saw in the config:
CONFIG_LRU_GEN=yCONFIG_LRU_GEN_ENABLED=y
Quoted 6 lines
Quoted 2 lines
So disabling at least the second one would revert to the traditional reclaim and we could see if it handles such a constrained system better or not.
I set RANDOM_KMALLOC_CACHES=n and LRU_GEN_ENABLED=n but still hit the issue.
dmesg looks a bit different (unpatched v6.10-rc2).
What caught my eye, but it's also in some of the previous dmesg with MGRLU, is that in one case there's:
DMA free:0kB
That means many allocations went through that are allowed to just ignore all reserves, and depleted everything. That would mean __GFP_MEMALLOC or PF_MEMALLOC, which I suggested earlier for the GFP_KERNEL failure, is being used somewhere, but not leading to the expected memory freeing.
Quoted 2 lines
Regards,
Erhard
2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:
Quoted 13 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
Quoted 7 lines
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).
I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?
Signature · 1 line
<[email protected]>
Quoted 24 lines · Takero Funaki
2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:
Quoted 13 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
Quoted 7 lines
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).
Quoted 8 lines
I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?
We use 32 zpools on machines with 100s of CPUs. Two zpools per CPU is an overkill imo.
I have further comments that I will leave on the patch, but I mainly think this should be driven by real data, not theoretical possibility of lock contention.
Quoted 4 lines
--
Quoted 35 lines · Yosry Ahmed
Quoted 24 lines · Takero Funaki
2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:
Quoted 13 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
Quoted 7 lines
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).Quoted 8 lines
I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?
Quoted 2 lines
We use 32 zpools on machines with 100s of CPUs. Two zpools per CPU is an overkill imo.
Not to choose a camp; just a friendly note on why I strongly disagree with the N zpools per CPU approach:
1. It is fundamentally flawed to assume the system is linear;
2. Nonlinear systems usually have diminishing returns.
For Google data centers, using nr_cpus as the scaling factor had long passed the acceptable ROI threshold. Per-CPU data, especially when compounded per memcg or even per process, is probably the number-one overhead in terms of DRAM efficiency.
Quoted 3 lines
I have further comments that I will leave on the patch, but I mainly think this should be driven by real data, not theoretical possibility of lock contention.
Quoted 45 lines · Yu Zhao
Quoted 35 lines · Yosry Ahmed
Quoted 24 lines · Takero Funaki
2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:
Quoted 13 lines
I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.
Quoted 7 lines
(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a
nr_zpools>nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).Quoted 8 lines
I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?Quoted 2 lines
We use 32 zpools on machines with 100s of CPUs. Two zpools per CPU is an overkill imo.
Not to choose a camp; just a friendly note on why I strongly disagree with the N zpools per CPU approach:
1. It is fundamentally flawed to assume the system is linear;
2. Nonlinear systems usually have diminishing returns.
Quoted 4 lines
For Google data centers, using
nr_cpusas the scaling factor had long passed the acceptable ROI threshold. Per-CPU data, especially when compounded per memcg or even per process, is probably the number-one overhead in terms of DRAM efficiency.
100% agreed. If you look at option (b) above, I specifically called out that scaling the number of zpools linearly with the number with CPUs have diminishing returns :)
Quoted 10 lines · On Tue, 4 Jun 2024 11:18:25 -0600 Yu Zhao <[email protected]> wrote:
Hi Erhard,
If it's not too much trouble, could you "grep
nr_zspages/proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.
Thank you.
I used watch -n1 'grep nr_zspages /proc/vmstat' to get the readings and repeated this 3 times to check whether the reported values differ much.
Bad commit was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 "mm: zswap: multiple zpools support", next commit in git log after the bad one would be 42c06a0e8ebe95b81e5fb41c6556ff22d9255b0c "mm: kill frontswap".
With this kernel I got 2440, 2337, 3245 as nr_zspages and the kswapd0: page allocation error.
Commit in git log before the bad one would be bfaa4a0ce1bbc1b2b67de7e4c2e1679495f7b905 "scsi: gvp11: Remove unused gvp11_setup() function".
With this kernel I got 25537, 11321, 16087 as nr_zspages and the OOM reaper.
Tomorrow I could also check the .config changes Vlastimil suggested (CONFIG_SLAB_MERGE_DEFAULT=y, no CONFIG_RANDOM_KMALLOC_CACHES) and report back if that's of interest.
Regards,
Erhard
Quoted 45 lines · Yosry Ahmed
Quoted 30 lines · Erhard Furtner
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]Quoted 8 lines
Quoted 8 lines
The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.
Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling
CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)Ok, the bisect I did actually revealed something meaningful:
Quoted 7 lines
# git bisect good b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17 Author: Yosry Ahmed <[email protected]> Date: Tue Jun 20 19:46:44 2023 +0000mm: zswap: multiple zpools support
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.
Quoted 5 lines
Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
As for reducing slab fragmentation/footprint, I would also recommend these changes to .config:
CONFIG_SLAB_MERGE_DEFAULT=y - this will unify the separate zpool caches as well (but the patch still makes sense), but also many others CONFIG_RANDOM_KMALLOC_CACHES=n - no 16 separate copies of kmalloc caches
although the slabinfo output doesn't seem to show CONFIG_RANDOM_KMALLOC_CACHES in action, weirdly. It was enabled in the config attached to the first mail.
Both these changes mean giving up some mitigation against potentai lvulnerabilities. But it's not perfect anyway and the memory seems really tight here.
Quoted 53 lines · Vlastimil Babka (SUSE)
Quoted 45 lines · Yosry Ahmed
Quoted 30 lines · Erhard Furtner
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:
The OOM kills on both kernel versions seem to be reasonable to me.
Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] [ 0.000000] Normal empty [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff]Quoted 8 lines
Quoted 8 lines
The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.
Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling
CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)Ok, the bisect I did actually revealed something meaningful:
Quoted 7 lines
# git bisect good b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17 Author: Yosry Ahmed <[email protected]> Date: Tue Jun 20 19:46:44 2023 +0000mm: zswap: multiple zpools support
Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and '
zs_handle', which may contribute to fragmentation in slab memory.Quoted 5 lines
Do you have
/proc/slabinfofrom a good and a bad run by any chance?Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.
Quoted 6 lines
As for reducing slab fragmentation/footprint, I would also recommend these changes to .config:
CONFIG_SLAB_MERGE_DEFAULT=y- this will unify the separate zpool caches as well (but the patch still makes sense), but also many othersCONFIG_RANDOM_KMALLOC_CACHES=n- no 16 separate copies of kmalloc caches
Yeah, I did send that patch separately, but I think the problem here is probably fragmentation in the zsmalloc pools themselves, not the slab caches used by them.
Quoted 8 lines
although the slabinfo output doesn't seem to show
CONFIG_RANDOM_KMALLOC_CACHESin action, weirdly. It was enabled in the config attached to the first mail.Both these changes mean giving up some mitigation against potentai lvulnerabilities. But it's not perfect anyway and the memory seems really tight here.
I think we may be able to fix the problem here if we address the zsmalloc fragmentation. In regards to slab caches, the patch proposed above should avoid the replication without enabling slab cache merging in general.
Thanks for chiming in!