Skip to contentSkip to conversation

kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc)

DISCUSSION Erhard Furtner 10 participants 46 messages May 8, 2024 – Jun 7, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Greetings!

Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:

[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[effe5cc0] [c0784b64] dump_stack_lvl+0x80/0xac (unreliable)
[effe5ce0] [c01aef80] warn_alloc+0x100/0x178
[effe5d40] [c01af34c] __alloc_pages+0x354/0x8ac
[effe5e00] [c01af988] page_frag_alloc_align+0x68/0x17c
[effe5e30] [c063b7b4] __netdev_alloc_skb+0xb4/0x17c
[effe5e60] [be99573c] setup_rx_descbuffer+0x40/0x144 [b43legacy]
[effe5e90] [be99687c] b43legacy_dma_rx+0x20c/0x278 [b43legacy]
[effe5ee0] [be9897f4] b43legacy_interrupt_tasklet+0x580/0x5a4 [b43legacy]
[effe5f50] [c0050b34] tasklet_action_common.isra.0+0xb0/0xe8
[effe5f80] [c07aaf5c] __do_softirq+0x1dc/0x218
[effe5ff0] [c0008820] do_softirq_own_stack+0x34/0x40

[c1c555f0] [c07ff5f0] 0xc07ff5f0
[c1c55610] [c0050384] __irq_exit_rcu+0x6c/0xbc [c1c55620] [c00507f8] irq_exit+0x10/0x20 [c1c55630] [c00045b4] HardwareInterrupt_virt+0x108/0x10c --- interrupt: 500 at BIT_flushBits+0x1c/0x58

NIP:  c0458840 LR: c0458fdc CTR: 00000000
REGS: c1c55640 TRAP: 0500   Not tainted  (6.8.9-gentoo-PMacG4)
MSR:  0220b032 <VEC,EE,FP,ME,IR,DR,RI>  CR: 88082202  XER: 00000000

GPR00: c0458f98 c1c55700 c1c2b9a0 c1c5575c 00000000 00000000 00000000 00000000
GPR08: 00000000 00000000 f2c1d0d5 c1c55710 88084202 00000000 f2c1e000 f35f8150
GPR16: 00000000 f34bf238 f34bf30c 00000000 0000001b 00000002 00000000 00000000
GPR24: f35f8150 00000080 f35f7950 f35f7d50 00000000 c07ff5f0 00000052 f35f5be0
NIP [c0458840] BIT_flushBits+0x1c/0x58
LR [c0458fdc] ZSTD_encodeSequences+0x2ac/0x308
--- interrupt: 500
[c1c55700] [c07ff5f0] 0xc07ff5f0 (unreliable)
[c1c55710] [c0458f98] ZSTD_encodeSequences+0x268/0x308
[c1c557b0] [c0452eb0] ZSTD_entropyCompressSeqStore.constprop.0+0x1c4/0x2bc
[c1c55840] [c045357c] ZSTD_compressBlock_internal+0xac/0x144
[c1c55870] [c04543c8] ZSTD_compressContinue_internal+0x734/0x7c0
[c1c55920] [c04559d4] ZSTD_compressEnd+0x2c/0x13c
[c1c55950] [c04574b0] ZSTD_compressStream2+0x1b8/0x508
[c1c559a0] [c045788c] ZSTD_compressStream2_simpleArgs+0x48/0x60
[c1c559e0] [c0457904] ZSTD_compress2+0x60/0x90
[c1c55a10] [c03d0abc] __zstd_compress+0x54/0x78
[c1c55a70] [c03c3794] scomp_acomp_comp_decomp+0xe8/0x16c
[c1c55aa0] [c01c4c80] zswap_store+0x4b4/0x668
[c1c55b20] [c01bc404] swap_writepage+0x3c/0xa8
[c1c55b40] [c0172dd4] pageout+0x11c/0x1ec
[c1c55bc0] [c0174e2c] shrink_folio_list+0x6b0/0x878
[c1c55c40] [c01767e8] evict_folios+0x9d4/0xd08
[c1c55d40] [c0176cec] try_to_shrink_lruvec+0x1d0/0x210
[c1c55da0] [c0176dcc] shrink_one+0xa0/0x134
[c1c55dd0] [c01786a4] shrink_node+0x248/0x844
[c1c55e50] [c0178f68] balance_pgdat+0x2c8/0x614
[c1c55f50] [c01794cc] kswapd+0x218/0x24c
[c1c55fc0] [c006aae4] kthread+0xe4/0xe8
[c1c55ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:247864 inactive_anon:215615 isolated_anon:0
active_file:9633 inactive_file:12323 isolated_file:0
unevictable:4 dirty:0 writeback:2
slab_reclaimable:1293 slab_unreclaimable:6904
mapped:16876 shmem:155 pagetables:899
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2224 free_pcp:596 free_cma:0
Node 0 active_anon:991456kB inactive_anon:862460kB active_file:38532kB inactive_file:49292kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:67504kB dirty:0kB writeback:8kB shmem:620kB writeback_tmp:0kB kernel_stack:1536kB pagetables:3596kB sec_pagetables:0kB all_unreclaimable? no DMA free:1680kB boost:4096kB min:7548kB low:8408kB high:9268kB reserved_highatomic:0KB active_anon:675928kB inactive_anon:5644kB active_file:124kB inactive_file:596kB unevictable:0kB writepending:0kB present:786432kB managed:746656kB mlocked:0kB bounce:0kB free_pcp:2384kB local_pcp:1004kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 18*32kB (UME) 7*64kB (UM) 0*128kB 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 1824kB
35907 total pagecache pages
13813 pages in swap cache
Free swap = 8270180kB
[...]

There was no crash though, the G4 was still usable via VNC.

Full dmesg and kernel .config attached.

Regards,
Erhard

#1 lore raw
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:

Greetings!

Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:

[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac

Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":

[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14

SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c

slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154

slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved

I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.

Full dmesg and kernel .config attached.

Regards,
Erhard

#2 lore raw
Quoted 244 lines · Erhard Furtner
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:

Greetings!

Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:

[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac

Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":

[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14

SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c

slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154

slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved

Quoted 6 lines

I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.

Full dmesg and kernel .config attached.

Regards,
Erhard

Hi Erhard,

Thanks for the reports. I'll take a look at them and get back to you in a few days.

#3 lore raw
Quoted 250 lines · Yu Zhao
Quoted 248 lines · Erhard Furtner
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:

Greetings!

Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:

[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac

Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":

[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14

SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c

slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154

slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved

I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.

Full dmesg and kernel .config attached.

Quoted 7 lines
Quoted 2 lines

Regards,
Erhard

Hi Erhard,

Thanks for the reports. I'll take a look at them and get back to you in a few days.

Hi Erhard,

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.

Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)

Thanks!

#4 lore raw
Quoted 268 lines · Yu Zhao
Quoted 250 lines · Yu Zhao
Quoted 248 lines · Erhard Furtner
Quoted 8 lines · On Wed, 8 May 2024 20:21:11 +0200 Erhard Furtner <[email protected]> wrote:

Greetings!

Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:

[...]
kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac

Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":

[...]
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14

SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 15, objs: 225, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
Mem-Info:
active_anon:340071 inactive_anon:139179 isolated_anon:0
active_file:8297 inactive_file:2506 isolated_file:0
unevictable:4 dirty:1 writeback:18
slab_reclaimable:1377 slab_unreclaimable:7426
mapped:6804 shmem:112 pagetables:946
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1141 free_pcp:7 free_cma:0
Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
63943 total pagecache pages
53024 pages in swap cache
Free swap = 8057248kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 396 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c

slab_out_of_memory: 53 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:351976 inactive_anon:123514 isolated_anon:0
active_file:4648 inactive_file:2081 isolated_file:0
unevictable:4 dirty:1 writeback:39
slab_reclaimable:918 slab_unreclaimable:7222
mapped:5359 shmem:21 pagetables:940
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:2563 free_pcp:142 free_cma:0
Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
45961 total pagecache pages
39207 pages in swap cache
Free swap = 8093096kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved
warn_alloc: 343 callbacks suppressed
kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0

CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
[c1c65960] [c01b6234] warn_alloc+0x100/0x178
[c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
[c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
[c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
[c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
[c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154

slab_out_of_memory: 59 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
node 0: slabs: 18, objs: 270, free: 0
SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
node 0: slabs: 33, objs: 165, free: 0
[c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
[c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
[c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
[c1c65d20] [c0180a40] shrink_slab+0x174/0x260
[c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
[c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
[c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
[c1c65f50] [c017f270] kswapd+0x228/0x25c
[c1c65fc0] [c006bbac] kthread+0xe4/0xe8
[c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
Mem-Info:
active_anon:235002 inactive_anon:240975 isolated_anon:0
active_file:4356 inactive_file:2551 isolated_file:0
unevictable:4 dirty:7 writeback:19
slab_reclaimable:1008 slab_unreclaimable:7218
mapped:5601 shmem:21 pagetables:939
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:1340 free_pcp:23 free_cma:0
Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
116345 total pagecache pages
109413 pages in swap cache
Free swap = 7819300kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
9947 pages reserved

I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.

Full dmesg and kernel .config attached.

Quoted 7 lines
Quoted 2 lines

Regards,
Erhard

Hi Erhard,

Thanks for the reports. I'll take a look at them and get back to you in a few days.

Hi Erhard,

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

Quoted 4 lines
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...

Signature · 3 lines
Cheers,

David / dhildenb
#5 lore raw
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Yu Zhao

On Wed, May 15, 2024 at 4:06 PM Yu Zhao <[email protected]> wrote:

...

Quoted 6 lines
Quoted 6 lines

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

Quoted 7 lines
Quoted 4 lines
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...

The DMA zone exists because 9739ab7eda45 ("powerpc: enable a 30-bit ZONE_DMA for 32-bit pmac") selects it.

It's 768MB (not 0.25GB) because it's clamped at max_low_pfn:

#ifdef CONFIG_ZONE_DMA
	max_zone_pfns[ZONE_DMA]	= min(max_low_pfn,
				      1UL << (zone_dma_bits - PAGE_SHIFT));

#endif

Which comes eventually from CONFIG_LOWMEM_SIZE, which defaults to 768MB.

I think it's 768MB because the user:kernel split is 3G:1G, and then the kernel needs some of that 1G virtual space for vmalloc/ioremap/highmem, so it splits it 768M:256M.

Then ZONE_NORMAL is empty because it is also limited to max_low_pfn:

	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;

The rest of RAM is highmem.

So I think that's all behaving as expected, but I don't know 32-bit / highmem stuff that well so I could be wrong.

cheers

#6 lore raw
Quoted 16 lines · Michael Ellerman
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Yu Zhao

On Wed, May 15, 2024 at 4:06 PM Yu Zhao <[email protected]> wrote:

...

Quoted 11 lines
Quoted 10 lines

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]
Quoted 7 lines
Quoted 2 lines

That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...

The DMA zone exists because 9739ab7eda45 ("powerpc: enable a 30-bit ZONE_DMA for 32-bit pmac") selects it.

It's 768MB (not 0.25GB) because it's clamped at max_low_pfn:

Right. (I meant 0.75GB.)

Quoted 6 lines

#ifdef CONFIG_ZONE_DMA
max_zone_pfns[ZONE_DMA] = min(max_low_pfn,
1UL << (zone_dma_bits - PAGE_SHIFT));
#endif

Which comes eventually from CONFIG_LOWMEM_SIZE, which defaults to 768MB.

I see. I grep'ed VMSPLIT which is used on x86 and arm but apparently not on powerpc.

Quoted 4 lines

I think it's 768MB because the user:kernel split is 3G:1G, and then the kernel needs some of that 1G virtual space for vmalloc/ioremap/highmem, so it splits it 768M:256M.

Quoted 8 lines

Then ZONE_NORMAL is empty because it is also limited to max_low_pfn:

        max_zone_pfns[ZONE_NORMAL] = max_low_pfn;

The rest of RAM is highmem.

So I think that's all behaving as expected, but I don't know 32-bit / highmem stuff that well so I could be wrong.

Yes, the three zones work as intended.

Erhard,

Since your system only has 2GB memory, I'd try the 2G:2G split, which would in theory allow both the kernel and userspace to all memory.

CONFIG_LOWMEM_SIZE_BOOL=y
CONFIG_LOWMEM_SIZE=0x7000000

(Michael, please correct me if the above wouldn't work.)

#7 lore raw
Quoted 32 lines · Yu Zhao
Quoted 16 lines · Michael Ellerman
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Yu Zhao

On Wed, May 15, 2024 at 4:06 PM Yu Zhao <[email protected]> wrote:

...

Quoted 11 lines
Quoted 10 lines

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]
Quoted 7 lines
Quoted 2 lines

That's really odd. But we are messing with "PowerMac3,6", so I don't really know what's right or wrong ...

The DMA zone exists because 9739ab7eda45 ("powerpc: enable a 30-bit ZONE_DMA for 32-bit pmac") selects it.

It's 768MB (not 0.25GB) because it's clamped at max_low_pfn:

Right. (I meant 0.75GB.)

Quoted 5 lines

#ifdef CONFIG_ZONE_DMA
max_zone_pfns[ZONE_DMA] = min(max_low_pfn,
1UL << (zone_dma_bits - PAGE_SHIFT));
#endif

Quoted 4 lines
Quoted 1 line

Which comes eventually from CONFIG_LOWMEM_SIZE, which defaults to 768MB.

I see. I grep'ed VMSPLIT which is used on x86 and arm but apparently not on powerpc.

Those VMSPLIT configs are nice, on powerpc it's all done manually :}

Quoted 17 lines
Quoted 4 lines

I think it's 768MB because the user:kernel split is 3G:1G, and then the kernel needs some of that 1G virtual space for vmalloc/ioremap/highmem, so it splits it 768M:256M.

Quoted 8 lines

Then ZONE_NORMAL is empty because it is also limited to max_low_pfn:

        max_zone_pfns[ZONE_NORMAL] = max_low_pfn;

The rest of RAM is highmem.

So I think that's all behaving as expected, but I don't know 32-bit / highmem stuff that well so I could be wrong.

Yes, the three zones work as intended.

Erhard,

Quoted 7 lines

Since your system only has 2GB memory, I'd try the 2G:2G split, which would in theory allow both the kernel and userspace to all memory.

CONFIG_LOWMEM_SIZE_BOOL=y
CONFIG_LOWMEM_SIZE=0x7000000

(Michael, please correct me if the above wouldn't work.)

It's a bit more complicated, in order to increase LOWMEM_SIZE you need to adjust all the other variables to make space.

To get 2G of user virtual space I think you need:

CONFIG_ADVANCED_OPTIONS=y
CONFIG_LOWMEM_SIZE_BOOL=y
CONFIG_LOWMEM_SIZE=0x60000000
CONFIG_PAGE_OFFSET_BOOL=y
CONFIG_PAGE_OFFSET=0x90000000
CONFIG_KERNEL_START_BOOL=y
CONFIG_KERNEL_START=0x90000000
CONFIG_PHYSICAL_START=0x00000000
CONFIG_TASK_SIZE_BOOL=y
CONFIG_TASK_SIZE=0x80000000

Which results in 1.5GB of lowmem.

Or if you want to map all 2G of RAM directly in the kernel without highmem, but limit user virtual space to 1.5G:

CONFIG_ADVANCED_OPTIONS=y
CONFIG_LOWMEM_SIZE_BOOL=y
CONFIG_LOWMEM_SIZE=0x80000000
CONFIG_PAGE_OFFSET_BOOL=y
CONFIG_PAGE_OFFSET=0x70000000
CONFIG_KERNEL_START_BOOL=y
CONFIG_KERNEL_START=0x70000000
CONFIG_PHYSICAL_START=0x00000000
CONFIG_TASK_SIZE_BOOL=y
CONFIG_TASK_SIZE=0x60000000

You can also reclaim another 256MB of virtual space if you disable CONFIG_MODULES.

Those configs do boot on qemu. But I don't have easy access to my 32-bit machine to test if they boot on actual hardware.

cheers

#8 lore raw
Quoted 16 lines · On Thu, 06 Jun 2024 22:08:40 +1000 Michael Ellerman <[email protected]> wrote:

It's a bit more complicated, in order to increase LOWMEM_SIZE you need to adjust all the other variables to make space.

To get 2G of user virtual space I think you need:

CONFIG_ADVANCED_OPTIONS=y
CONFIG_LOWMEM_SIZE_BOOL=y
CONFIG_LOWMEM_SIZE=0x60000000
CONFIG_PAGE_OFFSET_BOOL=y
CONFIG_PAGE_OFFSET=0x90000000
CONFIG_KERNEL_START_BOOL=y
CONFIG_KERNEL_START=0x90000000
CONFIG_PHYSICAL_START=0x00000000
CONFIG_TASK_SIZE_BOOL=y
CONFIG_TASK_SIZE=0x80000000

Quoted 1 line

Which results in 1.5GB of lowmem.

Booting this config on the G4 worked, but the issue showed up anyhow.

Quoted 13 lines

Or if you want to map all 2G of RAM directly in the kernel without highmem, but limit user virtual space to 1.5G:

CONFIG_ADVANCED_OPTIONS=y
CONFIG_LOWMEM_SIZE_BOOL=y
CONFIG_LOWMEM_SIZE=0x80000000
CONFIG_PAGE_OFFSET_BOOL=y
CONFIG_PAGE_OFFSET=0x70000000
CONFIG_KERNEL_START_BOOL=y
CONFIG_KERNEL_START=0x70000000
CONFIG_PHYSICAL_START=0x00000000
CONFIG_TASK_SIZE_BOOL=y
CONFIG_TASK_SIZE=0x60000000

This actually did the trick!

Also I disabled highmem via HIGHMEM=n. With this config the machine did run the "stress-ng --vm 2 --vm-bytes 1930M --verify -v" load for about 2 hrs without hitting the issue.

Quoted 2 lines

You can also reclaim another 256MB of virtual space if you disable CONFIG_MODULES.

Did not try that 'cause the 2nd config worked.

Working 2G_no-highmem .config attached and the dmesg of both configs attached.

Regards,
Erhard

#9 lore raw
In reply to Yu Zhao
Quoted 18 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:

Hi Erhard,

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.

Quoted 6 lines

Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)

Thanks!

Hi Yu!

Thanks for looking into this.

The reason for this 0.25GB DMA / 1.75GB highmem split is beyond my knowledge. I can only tell this much that it's like this at least since kernel v4.14.x (dmesg of an old bugreport of mine at https://bugzilla.kernel.org/show_bug.cgi?id=201723), I guess earlier kernel versions too.

Without CONFIG_HIGHMEM the memory layout looks like this:

Total memory = 768MB; using 2048kB for hash table
[...]
Top of RAM: 0x30000000, Total RAM: 0x30000000
Memory hole size: 0MB
Zone ranges:
DMA [mem 0x0000000000000000-0x000000002fffffff]
Normal empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000002fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff]
percpu: Embedded 29 pages/cpu s28448 r8192 d82144 u118784
pcpu-alloc: s28448 r8192 d82144 u118784 alloc=29*4096
pcpu-alloc: [0] 0 [0] 1
Kernel command line: ro root=/dev/sda5 slub_debug=FZP page_poison=1 [email protected]/eth0,[email protected]/A8:A1:59:16:4F:EA debug Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Built 1 zonelists, mobility grouping on. Total pages: 194880
mem auto-init: stack:all(pattern), heap alloc:off, heap free:off
Kernel virtual memory layout:

  * 0xffbdf000..0xfffff000  : fixmap
  * 0xff8f4000..0xffbdf000  : early ioremap
  * 0xf1000000..0xff8f4000  : vmalloc & ioremap
  * 0xb0000000..0xc0000000  : modules

Memory: 761868K/786432K available (7760K kernel code, 524K rwdata, 4528K rodata, 1100K init, 253K bss, 24564K reserved, 0K cma-reserved) [...]

With only 768 MB RAM and 2048K hashtable I get pretty much the same "kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL),nodemask=(null),cpuset=/,mems_allowed=0" as with the HIGHMEM enabled kernel at running "stress-ng --vm 2 --vm-bytes 1930M --verify -v".

I tried the workload on v6.6.32 LTS where the issue shows up too. But v6.1.92 LTS seems ok! Triple checked v6.1.92 to be sure.

Attached please find kernel v6.9.3 dmesg (without HIGHMEM) and kernel v6.1.92 .config.

Regards,
Erhard

#10 lore raw
Quoted 64 lines · Erhard Furtner
Quoted 18 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:

Hi Erhard,

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.

Quoted 6 lines

Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)

Thanks!

Hi Yu!

Thanks for looking into this.

The reason for this 0.25GB DMA / 1.75GB highmem split is beyond my knowledge. I can only tell this much that it's like this at least since kernel v4.14.x (dmesg of an old bugreport of mine at https://bugzilla.kernel.org/show_bug.cgi?id=201723), I guess earlier kernel versions too.

Without CONFIG_HIGHMEM the memory layout looks like this:

Total memory = 768MB; using 2048kB for hash table
[...]
Top of RAM: 0x30000000, Total RAM: 0x30000000
Memory hole size: 0MB
Zone ranges:
DMA [mem 0x0000000000000000-0x000000002fffffff]
Normal empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000002fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff]
percpu: Embedded 29 pages/cpu s28448 r8192 d82144 u118784
pcpu-alloc: s28448 r8192 d82144 u118784 alloc=29*4096
pcpu-alloc: [0] 0 [0] 1
Kernel command line: ro root=/dev/sda5 slub_debug=FZP page_poison=1 [email protected]/eth0,[email protected]/A8:A1:59:16:4F:EA debug Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Built 1 zonelists, mobility grouping on. Total pages: 194880
mem auto-init: stack:all(pattern), heap alloc:off, heap free:off
Kernel virtual memory layout:

  * 0xffbdf000..0xfffff000  : fixmap
  * 0xff8f4000..0xffbdf000  : early ioremap
  * 0xf1000000..0xff8f4000  : vmalloc & ioremap
  * 0xb0000000..0xc0000000  : modules

Memory: 761868K/786432K available (7760K kernel code, 524K rwdata, 4528K rodata, 1100K init, 253K bss, 24564K reserved, 0K cma-reserved) [...]

Quoted 6 lines

With only 768 MB RAM and 2048K hashtable I get pretty much the same "kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL),nodemask=(null),cpuset=/,mems_allowed=0" as with the HIGHMEM enabled kernel at running "stress-ng --vm 2 --vm-bytes 1930M --verify -v".

I tried the workload on v6.6.32 LTS where the issue shows up too. But v6.1.92 LTS seems ok! Triple checked v6.1.92 to be sure.

Attached please find kernel v6.9.3 dmesg (without HIGHMEM) and kernel v6.1.92 .config.

Thanks.

I compared the .config between v6.8.9 (you attached previously) and v6.1.92 -- I didn't see any major differences (both have ZONE_DMA, HIGHMEM, MGLRU and zswap/zsmalloc). Either there is something broken between v6.1.92 and v6.6.32 (as you mentioned above), or it's just a kernel allocation bloat which puts the DMA zone (0.25GB) under too heavy pressure. The latter isn't uncommon when upgrading to a newer version of the kernel.

Could you please attach the dmesg from v6.1.92? I want to compare the dmegs between the two kernel versions as well -- that might provide some hints.

#11 lore raw
Quoted 11 lines · On Sun, 2 Jun 2024 14:38:18 -0600 Yu Zhao <[email protected]> wrote:

I compared the .config between v6.8.9 (you attached previously) and v6.1.92 -- I didn't see any major differences (both have ZONE_DMA, HIGHMEM, MGLRU and zswap/zsmalloc). Either there is something broken between v6.1.92 and v6.6.32 (as you mentioned above), or it's just a kernel allocation bloat which puts the DMA zone (0.25GB) under too heavy pressure. The latter isn't uncommon when upgrading to a newer version of the kernel.

Could you please attach the dmesg from v6.1.92? I want to compare the dmegs between the two kernel versions as well -- that might provide some hints.

No problem, attached please find the v6.1.92 dmesg and also the v6.9.3 .config and dmesg.

I stripped down the v6.9.3 .config a bit (to ease 'make oldconfig' for older kernels) from my originally posted v6.8.9 .config and used that as a starting point for v6.6.32 and v6.1.92.

Also I started a git bisect. Let's see if I get something meaningful out of this...

Regards,
Erhard

#12 lore raw
In reply to Erhard Furtner
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]
Quoted 8 lines
Quoted 8 lines

The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.

Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)

Ok, the bisect I did actually revealed something meaningful:

 # git bisect good
b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit
commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17
Author: Yosry Ahmed <[email protected]>
Date:   Tue Jun 20 19:46:44 2023 +0000
    mm: zswap: multiple zpools support
    
    Support using multiple zpools of the same type in zswap, for concurrency
    purposes.  A fixed number of 32 zpools is suggested by this commit, which
    was determined empirically.  It can be later changed or made into a config
    option if needed.
    
    On a setup with zswap and zsmalloc, comparing a single zpool to 32 zpools
    shows improvements in the zsmalloc lock contention, especially on the swap
    out path.
    
    The following shows the perf analysis of the swapout path when 10
    workloads are simultaneously reclaiming and refaulting tmpfs pages.  There
    are some improvements on the swap in path as well, but less significant.
    
    1 zpool:
    
     |--28.99%--zswap_frontswap_store
           |
           <snip>
           |
           |--8.98%--zpool_map_handle
           |     |
           |      --8.98%--zs_zpool_map
           |           |
           |            --8.95%--zs_map_object
           |                 |
           |                  --8.38%--_raw_spin_lock
           |                       |
           |                        --7.39%--queued_spin_lock_slowpath
           |
           |--8.82%--zpool_malloc
           |     |
           |      --8.82%--zs_zpool_malloc
           |           |
           |            --8.80%--zs_malloc
           |                 |
           |                 |--7.21%--_raw_spin_lock
           |                 |     |
           |                 |      --6.81%--queued_spin_lock_slowpath
           <snip>
    
    32 zpools:
    
     |--16.73%--zswap_frontswap_store
           |
           <snip>
           |
           |--1.81%--zpool_malloc
           |     |
           |      --1.81%--zs_zpool_malloc
           |           |
           |            --1.79%--zs_malloc
           |                 |
           |                  --0.73%--obj_malloc
           |
           |--1.06%--zswap_update_total_size
           |
           |--0.59%--zpool_map_handle
           |     |
           |      --0.59%--zs_zpool_map
           |           |
           |            --0.57%--zs_map_object
           |                 |
           |                  --0.51%--_raw_spin_lock
           <snip>
    
    Link: https://lkml.kernel.org/r/[email protected]
    Signed-off-by: Yosry Ahmed <[email protected]>
    Suggested-by: Yu Zhao <[email protected]>
    Acked-by: Chris Li (Google) <[email protected]>
    Reviewed-by: Nhat Pham <[email protected]>
    Tested-by: Nhat Pham <[email protected]>
    Cc: Dan Streetman <[email protected]>
    Cc: Domenico Cerasuolo <[email protected]>
    Cc: Johannes Weiner <[email protected]>
    Cc: Konrad Rzeszutek Wilk <[email protected]>
    Cc: Seth Jennings <[email protected]>
    Cc: Vitaly Wool <[email protected]>
    Signed-off-by: Andrew Morton <[email protected]>
 mm/zswap.c | 81 +++++++++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 54 insertions(+), 27 deletions(-)

'bad' bisects were where the "kswapd0: page allocation failure:" showed up when running the workload, 'good' bisects were the cases where I only got the kernels OOM reaper killing the workload. In the good cases the machine stayed usable via VNC, in the bad cases with the issue showing up the machine crashed and rebooted >80% of the time shortly after showing the issue in dmesg (via netconsole). I triple checked the good cases to be sure only the OOM reaper showed up and not the kswapd0: page allocation failure.

Bisect.log attached.

Regards,
Erhard

git bisect start
# Status: warte auf guten und schlechten Commit
# good: [88690811da69826fdb59d908a6e5e9d0c63b581a] Linux 6.1.92
git bisect good 88690811da69826fdb59d908a6e5e9d0c63b581a
# Status: warte auf schlechten Commit, 1 guter Commit bekannt
# bad: [a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6] Linux 6.9
git bisect bad a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
# good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
git bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476
# good: [1b580c9bb63414903cfb414571e92e4fd36d6969] Merge branch 'samples-bpf-remove-unmaintained-xdp-sample-utilities' git bisect good 1b580c9bb63414903cfb414571e92e4fd36d6969
# bad: [5ab500d6f9f50d8246865a2ead85d0e88ea30004] Merge branch 'sfc-implement-ndo_hwtstamp_-get-set'
git bisect bad 5ab500d6f9f50d8246865a2ead85d0e88ea30004
# bad: [cc33a80b816406f900a53c7f98a50f6eacdd2e31] MAINTAINERS: Maintainer change for ptp_vmw driver
git bisect bad cc33a80b816406f900a53c7f98a50f6eacdd2e31
# bad: [4debf77169ee459c46ec70e13dc503bc25efd7d2] Merge tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd
git bisect bad 4debf77169ee459c46ec70e13dc503bc25efd7d2
# bad: [1086eeac9c333b6db6c98594f02996c8261c60c5] Merge tag 'lsm-pr-20230829' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/lsm
git bisect bad 1086eeac9c333b6db6c98594f02996c8261c60c5
# good: [0fc81f376468ce67d70ecd1016dfe901a51a39f5] Merge tag 'regmap-v6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap
git bisect good 0fc81f376468ce67d70ecd1016dfe901a51a39f5
# good: [bd6c11bc43c496cddfc6cf603b5d45365606dbd5] Merge tag 'net-next-6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
git bisect good bd6c11bc43c496cddfc6cf603b5d45365606dbd5
# bad: [5553b15a4bbba8039e1f31b63642048286f540dc] m68k: implement the new page table range API
git bisect bad 5553b15a4bbba8039e1f31b63642048286f540dc
# good: [0c2e394ab23017303f676e6206a54c54bb0e3681] mm: move FAULT_FLAG_VMA_LOCK check down in handle_pte_fault()
git bisect good 0c2e394ab23017303f676e6206a54c54bb0e3681
# bad: [3f32c49ed6f15c8412a8abc93a92c4b37e6c4592] mm: memtest: convert to memtest_report_meminfo()
git bisect bad 3f32c49ed6f15c8412a8abc93a92c4b37e6c4592
# bad: [6ad243b83b5094026fdb3171711ddb25246b3d8a] mm/damon/sysfs: implement a command for updating only schemes tried total bytes git bisect bad 6ad243b83b5094026fdb3171711ddb25246b3d8a
# bad: [68af05143fd4b49d4b12eab8d63c91ffbc7c4e5e] kernel/iomem.c: remove __weak ioremap_cache helper
git bisect bad 68af05143fd4b49d4b12eab8d63c91ffbc7c4e5e
# good: [27af67f35631ac4b61b5e4455b44c9aee8d2cc4b] powerpc/book3s64/mm: enable transparent pud hugepage
git bisect good 27af67f35631ac4b61b5e4455b44c9aee8d2cc4b
# bad: [34f4c198bfbe86612c368eb122002787acecaa93] zswap: make zswap_store() take a folio
git bisect bad 34f4c198bfbe86612c368eb122002787acecaa93
# good: [601f006fddc66e369fdac7c572f981eafd159dac] powerpc/book3s64/radix: remove mmu_vmemmap_psize
git bisect good 601f006fddc66e369fdac7c572f981eafd159dac
# bad: [b8cf32dc6e8c75b712cbf638e0fd210101c22f17] mm: zswap: multiple zpools support
git bisect bad b8cf32dc6e8c75b712cbf638e0fd210101c22f17
# good: [6be3601517d90b728095d70c14f3a04b9adcb166] powerpc/book3s64/radix: add debug message to give more details of vmemmap allocation git bisect good 6be3601517d90b728095d70c14f3a04b9adcb166

# first bad commit: [b8cf32dc6e8c75b712cbf638e0fd210101c22f17] mm: zswap: multiple zpools support
#13 lore raw
Quoted 30 lines · Erhard Furtner
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]
Quoted 8 lines
Quoted 8 lines

The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.

Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)

Ok, the bisect I did actually revealed something meaningful:

Quoted 7 lines
 # git bisect good
b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit
commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17
Author: Yosry Ahmed <[email protected]>
Date:   Tue Jun 20 19:46:44 2023 +0000

mm: zswap: multiple zpools support

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

From 1ac0fa34783f66ea7334fd2e9129bc362bee51dd Mon Sep 17 00:00:00 2001
From: Yosry Ahmed <[email protected]>
Date: Sat, 13 May 2023 01:10:34 +0000
Subject: [PATCH] mm: zsmalloc: share slab caches for all zsmalloc zpools

Zswap creates 32 zpools to improve concurrency. Each zsmalloc zpool creates its own 'zs_handle' and 'zspage' slab caches. We end up with 32 slab caches of each type.

Since each slab cache holds some free objects, we end up with a lot of free objects distributed among the separate zpool caches. Slab caches are designed to handle concurrent allocations by using percpu structures, so having a single instance of each cache should be enough, and avoids wasting more memory than needed due to fragmentation.

Additionally, having more slab caches than needed unnecessarily slows down code paths that iterate slab_caches.

---
 mm/zsmalloc.c | 60 ++++++++++++++++++++++++++++-----------------------
 1 file changed, 33 insertions(+), 27 deletions(-)
mm/zsmalloc.c +33 -27
index b42d3545ca856..c3c90618ec8b4 100644
@@ -289,29 +289,8 @@ static void init_deferred_free(struct zs_pool *pool) {}
 static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage) {}
 #endif
 
-static int create_cache(struct zs_pool *pool)
-{
-	pool->handle_cachep = kmem_cache_create("zs_handle", ZS_HANDLE_SIZE,
-					0, 0, NULL);
-	if (!pool->handle_cachep)
-		return 1;
-
-	pool->zspage_cachep = kmem_cache_create("zspage", sizeof(struct zspage),
-					0, 0, NULL);
-	if (!pool->zspage_cachep) {
-		kmem_cache_destroy(pool->handle_cachep);
-		pool->handle_cachep = NULL;
-		return 1;
-	}
-
-	return 0;
-}
-
-static void destroy_cache(struct zs_pool *pool)
-{
-	kmem_cache_destroy(pool->handle_cachep);
-	kmem_cache_destroy(pool->zspage_cachep);
-}
+static struct kmem_cache *zs_handle_cache;
+static struct kmem_cache *zspage_cache;
 
 static unsigned long cache_alloc_handle(struct zs_pool *pool, gfp_t gfp)
 {
@@ -2107,13 +2086,13 @@ struct zs_pool *zs_create_pool(const char *name)
 	spin_lock_init(&pool->lock);
 	atomic_set(&pool->compaction_in_progress, 0);
 
+	pool->handle_cachep = zs_handle_cache;
+	pool->zspage_cachep = zspage_cache;
+
 	pool->name = kstrdup(name, GFP_KERNEL);
 	if (!pool->name)
 		goto err;
 
-	if (create_cache(pool))
-		goto err;
-
 	/*
 	 * Iterate reversely, because, size of size_class that we want to use
 	 * for merging should be larger or equal to current size.
@@ -2234,16 +2213,41 @@ void zs_destroy_pool(struct zs_pool *pool)
 		kfree(class);
 	}
 
-	destroy_cache(pool);
 	kfree(pool->name);
 	kfree(pool);
 }
 EXPORT_SYMBOL_GPL(zs_destroy_pool);
 
+static void zs_destroy_caches(void)
+{
+	kmem_cache_destroy(zs_handle_cache);
+	kmem_cache_destroy(zspage_cache);
+	zs_handle_cache = NULL;
+	zspage_cache = NULL;
+}
+
+static int zs_create_caches(void)
+{
+	zs_handle_cache = kmem_cache_create("zs_handle", ZS_HANDLE_SIZE,
+					    0, 0, NULL);
+	zspage_cache = kmem_cache_create("zspage", sizeof(struct zspage),
+					 0, 0, NULL);
+
+	if (!zs_handle_cache || !zspage_cache) {
+		zs_destroy_caches();
+		return -1;
+	}
+	return 0;
+}
+
 static int __init zs_init(void)
 {
 	int ret;
 
+	ret = zs_create_caches();
+	if (ret)
+		goto out;
+
 	ret = cpuhp_setup_state(CPUHP_MM_ZS_PREPARE, "mm/zsmalloc:prepare",
 				zs_cpu_prepare, zs_cpu_dead);
 	if (ret)
@@ -2258,6 +2262,7 @@ static int __init zs_init(void)
 	return 0;
 
 out:
+	zs_destroy_caches();
 	return ret;
 }
 
@@ -2269,6 +2274,7 @@ static void __exit zs_exit(void)
 	cpuhp_remove_state(CPUHP_MM_ZS_PREPARE);
 
 	zs_stat_exit();
+	zs_destroy_caches();
 }
 
 module_init(zs_init);
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
#14 lore raw
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.

Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.

The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.

Regards,
Erhard

slabinfo - version: 2.1

# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_1k     48     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_groupinfo_4k    320    322    176   23    1 : tunables    0    0    0 : slabdata     14     14      0
btrfs_prelim_ref       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_extent_op      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_data_ref      0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_tree_ref      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_ref_head      0      0    144   28    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_inode_defrag      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_node      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_ordered_extent      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_extent_map       3     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
bio-232                2     14    288   14    1 : tunables    0    0    0 : slabdata      1      1      0
bio-280                4     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_buffer     11     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_state      3     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_free_space_bitmap      0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
btrfs_free_space       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_path             0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_trans_handle      0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
bio-296                2     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_inode            3     16   1008   16    4 : tunables    0    0    0 : slabdata      1      1      0
bio-328                2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
zswap_entry            0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
p9_req_t               0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    168   24    1 : tunables    0    0    0 : slabdata      0      0      0
fib6_nodes             5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ip6_dst_cache          2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
PINGv6                 0      0   1056   15    4 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                  7     15   1056   15    4 : tunables    0    0    0 : slabdata      1      1      0
UDPLITEv6              0      0   1152   14    4 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                  1     14   1152   14    4 : tunables    0    0    0 : slabdata      1      1      0
tw_sock_TCPv6          0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCPv6      0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                  2     15   2080   15    8 : tunables    0    0    0 : slabdata      1      1      0
scsi_sense_cache       9     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
virtio_scsi_cmd       64     64    256   16    1 : tunables    0    0    0 : slabdata      4      4      0
io_kiocb               0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0
bfq_io_cq              0      0   1032   15    4 : tunables    0    0    0 : slabdata      0      0      0
bfq_queue              0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
bio-88                10     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
bio-168                4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
mqueue_inode_cache      1     21    768   21    4 : tunables    0    0    0 : slabdata      1      1      0
v9fs_inode_cache       0      0    600   13    2 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    560   14    2 : tunables    0    0    0 : slabdata      0      0      0
jbd2_transaction_s      3     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_inode            23     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_handle      0     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_head      6     84     96   42    1 : tunables    0    0    0 : slabdata      2      2      0
jbd2_revoke_table_s      2    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_revoke_record_s      0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_fc_dentry_update      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache    1354   1360   1008   16    4 : tunables    0    0    0 : slabdata     85     85      0
ext4_free_data         0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_allocation_context      0     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_prealloc_space      6     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_system_zone      89    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_io_end_vec        0     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_io_end            0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
bio_post_read_ctx    128    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
pending_reservation      0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
extent_status       1083   1095     56   73    1 : tunables    0    0    0 : slabdata     15     15      0
mbcache                0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
kioctx                 0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
aio_kiocb              0      0    160   25    1 : tunables    0    0    0 : slabdata      0      0      0
userfaultfd_ctx_cache      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_path_event      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_fid_event      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
fsnotify_mark          0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark           0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_struct         0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
dio                    0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
fasync_cache           0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
pid_namespace          0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
posix_timers_cache      0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
UNIX-STREAM           27     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
UNIX                  13     17    928   17    4 : tunables    0    0    0 : slabdata      1      1      0
ip4-frags              0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
UDP-Lite               0      0   1024   16    4 : tunables    0    0    0 : slabdata      0      0      0
tcp_bind2_bucket       5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
tcp_bind_bucket        2     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
inet_peer_cache        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    640   12    2 : tunables    0    0    0 : slabdata      0      0      0
ip_fib_trie            7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_fib_alias           7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_dst_cache           6     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
PING                   0      0    896   18    4 : tunables    0    0    0 : slabdata      0      0      0
RAW                    5     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0
UDP                   19     32   1024   16    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP            0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCP       0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                    3     16   1952   16    8 : tunables    0    0    0 : slabdata      1      1      0
bio-184               32     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
ep_head               36    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_pwq         37    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
eventpoll_epi         36     96    128   32    1 : tunables    0    0    0 : slabdata      3      3      0
inotify_inode_mark     40     51     80   51    1 : tunables    0    0    0 : slabdata      1      1      0
sgpool-128             2     12   2624   12    8 : tunables    0    0    0 : slabdata      1      1      0
sgpool-64              2     12   1344   12    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-32              2     23    704   23    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-16              2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
sgpool-8               2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
request_queue          4     22    720   22    4 : tunables    0    0    0 : slabdata      1      1      0
blkdev_ioc             1     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
bio-136                4     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
biovec-max            48     60   3136   10    8 : tunables    0    0    0 : slabdata      6      6      0
biovec-128             0     20   1600   20    8 : tunables    0    0    0 : slabdata      1      1      0
biovec-64              0     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
biovec-16              0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
ksm_mm_slot            0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_stable_node        0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_rmap_item          0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
user_namespace         0      0    448   18    2 : tunables    0    0    0 : slabdata      0      0      0
uid_cache              4     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache      93    108    672   12    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_ext_cache       0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
skbuff_small_head    130    136    480   17    2 : tunables    0    0    0 : slabdata      8      8      0
skbuff_fclone_cache      1     19    416   19    2 : tunables    0    0    0 : slabdata      1      1      0
skbuff_head_cache    397    414    224   18    1 : tunables    0    0    0 : slabdata     23     23      0
file_lock_cache        1     24    168   24    1 : tunables    0    0    0 : slabdata      1      1      0
file_lock_ctx         10    192     64   64    1 : tunables    0    0    0 : slabdata      3      3      0
fsnotify_mark_connector     33     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
buffer_head         5653   5694    104   39    1 : tunables    0    0    0 : slabdata    146    146      0
task_delay_info        0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
taskstats              2     17    456   17    2 : tunables    0    0    0 : slabdata      1      1      0
proc_dir_entry       274    286    152   26    1 : tunables    0    0    0 : slabdata     11     11      0
pde_opener             0    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
proc_inode_cache     197    224    568   14    2 : tunables    0    0    0 : slabdata     16     16      0
seq_file               2     34    120   34    1 : tunables    0    0    0 : slabdata      1      1      0
sigqueue               1     56     72   56    1 : tunables    0    0    0 : slabdata      1      1      0
bdev_cache            27     39   1216   13    4 : tunables    0    0    0 : slabdata      3      3      0
shmem_inode_cache   1313   1425    632   25    4 : tunables    0    0    0 : slabdata     57     57      0
kernfs_iattrs_cache     11     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
kernfs_node_cache  17325  17388    112   36    1 : tunables    0    0    0 : slabdata    483    483      0
mnt_cache             29     36    224   18    1 : tunables    0    0    0 : slabdata      2      2      0
filp                 476    512    256   16    1 : tunables    0    0    0 : slabdata     32     32      0
inode_cache         2406   2415    528   15    2 : tunables    0    0    0 : slabdata    161    161      0
dentry              6389   6394    176   23    1 : tunables    0    0    0 : slabdata    278    278      0
names_cache            0     14   4160    7    8 : tunables    0    0    0 : slabdata      2      2      0
net_namespace          0      0   2624   12    8 : tunables    0    0    0 : slabdata      0      0      0
key_jar                7     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
uts_namespace          0      0    440   18    2 : tunables    0    0    0 : slabdata      0      0      0
nsproxy                1     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
vm_area_struct      1401   1428     96   42    1 : tunables    0    0    0 : slabdata     34     34      0
fs_cache              32     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
files_cache           32     42    288   14    1 : tunables    0    0    0 : slabdata      3      3      0
signal_cache         111    126    864   18    4 : tunables    0    0    0 : slabdata      7      7      0
sighand_cache        111    115   1376   23    8 : tunables    0    0    0 : slabdata      5      5      0
task_struct          111    121   2848   11    8 : tunables    0    0    0 : slabdata     11     11      0
cred_jar             161    189    192   21    1 : tunables    0    0    0 : slabdata      9      9      0
anon_vma_chain      1055   1168     56   73    1 : tunables    0    0    0 : slabdata     16     16      0
anon_vma             761    782     88   46    1 : tunables    0    0    0 : slabdata     17     17      0
pid                  118    128    128   32    1 : tunables    0    0    0 : slabdata      4      4      0
perf_event             0      0    848   19    4 : tunables    0    0    0 : slabdata      0      0      0
pool_workqueue        29     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
maple_node           118    252    768   21    4 : tunables    0    0    0 : slabdata     12     12      0
radix_tree_node     1814   1824    320   12    1 : tunables    0    0    0 : slabdata    152    152      0
task_group            31     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
mm_struct             31     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
vmap_area            137    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
pgtable-2^10          31     32  12288    2    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-cg-8k          0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-cg-4k          5     10  12288    2    8 : tunables    0    0    0 : slabdata      5      5      0
kmalloc-cg-2k         16     20   6144    5    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-1k         40     40   3072   10    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-512        15     63   1536   21    8 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-256        27     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-192        24     54    216   18    1 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-128        18     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-96         40     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-64         40     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-32         28     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-16         18     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-8          23    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
dma-kmalloc-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-96        32     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-rcl-64       132    210    192   21    1 : tunables    0    0    0 : slabdata     10     10      0
kmalloc-rcl-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k            15     16  24576    1    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-4k            14     16  12288    2    8 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-2k           380    380   6144    5    8 : tunables    0    0    0 : slabdata     76     76      0
kmalloc-1k           256    260   3072   10    8 : tunables    0    0    0 : slabdata     26     26      0
kmalloc-512          612    630   1536   21    8 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-256          148    168    768   21    4 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-192         4307   4320    216   18    1 : tunables    0    0    0 : slabdata    240    240      0
kmalloc-128          724    735    384   21    2 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-96           721    748    120   34    1 : tunables    0    0    0 : slabdata     22     22      0
kmalloc-64          1346   1995    192   21    1 : tunables    0    0    0 : slabdata     95     95      0
kmalloc-32          1831   1890     96   42    1 : tunables    0    0    0 : slabdata     45     45      0
kmalloc-16          3264   3315     48   85    1 : tunables    0    0    0 : slabdata     39     39      0
kmalloc-8           2864   2944     32  128    1 : tunables    0    0    0 : slabdata     23     23      0
kmem_cache_node      217    252     96   42    1 : tunables    0    0    0 : slabdata      6      6      0
kmem_cache           217    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_1k     48     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_groupinfo_4k    320    322    176   23    1 : tunables    0    0    0 : slabdata     14     14      0
btrfs_prelim_ref       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_extent_op      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_data_ref      0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_tree_ref      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_ref_head      0      0    144   28    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_inode_defrag      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_node      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_ordered_extent      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_extent_map       3     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
bio-232                2     14    288   14    1 : tunables    0    0    0 : slabdata      1      1      0
bio-280                4     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_buffer      9     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_state      3     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_free_space_bitmap      0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
btrfs_free_space       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_path             0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_trans_handle      0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
bio-296                2     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_inode            2     16   1008   16    4 : tunables    0    0    0 : slabdata      1      1      0
bio-328                2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
zswap_entry          248    448     64   64    1 : tunables    0    0    0 : slabdata      7      7      0
p9_req_t               0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    168   24    1 : tunables    0    0    0 : slabdata      0      0      0
fib6_nodes             5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ip6_dst_cache          2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
PINGv6                 0      0   1056   15    4 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                  7     15   1056   15    4 : tunables    0    0    0 : slabdata      1      1      0
UDPLITEv6              0      0   1152   14    4 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                  1     14   1152   14    4 : tunables    0    0    0 : slabdata      1      1      0
tw_sock_TCPv6          0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCPv6      0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                  2     15   2080   15    8 : tunables    0    0    0 : slabdata      1      1      0
scsi_sense_cache       9     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
virtio_scsi_cmd       64     64    256   16    1 : tunables    0    0    0 : slabdata      4      4      0
io_kiocb               0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0
bfq_io_cq              0      0   1032   15    4 : tunables    0    0    0 : slabdata      0      0      0
bfq_queue              0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
bio-88                10     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
bio-168                4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
mqueue_inode_cache      1     21    768   21    4 : tunables    0    0    0 : slabdata      1      1      0
v9fs_inode_cache       0      0    600   13    2 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    560   14    2 : tunables    0    0    0 : slabdata      0      0      0
jbd2_transaction_s      3     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_inode            10     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_handle      0     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_head      9     84     96   42    1 : tunables    0    0    0 : slabdata      2      2      0
jbd2_revoke_table_s      2    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_revoke_record_s      0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_fc_dentry_update      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache     264    800   1008   16    4 : tunables    0    0    0 : slabdata     50     50      0
ext4_free_data         0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_allocation_context      0     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_prealloc_space      6     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_system_zone      89    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_io_end_vec        0     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_io_end            0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
bio_post_read_ctx    128    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
pending_reservation      0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
extent_status         54    292     56   73    1 : tunables    0    0    0 : slabdata      4      4      0
mbcache                0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
kioctx                 0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
aio_kiocb              0      0    160   25    1 : tunables    0    0    0 : slabdata      0      0      0
userfaultfd_ctx_cache      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_path_event      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_fid_event      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
fsnotify_mark          0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark           0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_struct         0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
dio                    0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
fasync_cache           0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zspage               221    448     64   64    1 : tunables    0    0    0 : slabdata      7      7      0
zs_handle            238    850     24  170    1 : tunables    0    0    0 : slabdata      5      5      0
pid_namespace          0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
posix_timers_cache      0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
UNIX-STREAM           29     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
UNIX                  13     17    928   17    4 : tunables    0    0    0 : slabdata      1      1      0
ip4-frags              0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
UDP-Lite               0      0   1024   16    4 : tunables    0    0    0 : slabdata      0      0      0
tcp_bind2_bucket       5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
tcp_bind_bucket        2     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
inet_peer_cache        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    640   12    2 : tunables    0    0    0 : slabdata      0      0      0
ip_fib_trie            7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_fib_alias           7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_dst_cache           6     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
PING                   0      0    896   18    4 : tunables    0    0    0 : slabdata      0      0      0
RAW                    5     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0
UDP                   19     32   1024   16    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP            0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCP       0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                    3     16   1952   16    8 : tunables    0    0    0 : slabdata      1      1      0
bio-184               32     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
ep_head               40    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_pwq         42    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
eventpoll_epi         40     96    128   32    1 : tunables    0    0    0 : slabdata      3      3      0
inotify_inode_mark     40     51     80   51    1 : tunables    0    0    0 : slabdata      1      1      0
sgpool-128             2     12   2624   12    8 : tunables    0    0    0 : slabdata      1      1      0
sgpool-64              2     12   1344   12    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-32              2     23    704   23    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-16              2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
sgpool-8               2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
request_queue          4     22    720   22    4 : tunables    0    0    0 : slabdata      1      1      0
blkdev_ioc             1     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
bio-136                4     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
biovec-max            48     60   3136   10    8 : tunables    0    0    0 : slabdata      6      6      0
biovec-128             0     20   1600   20    8 : tunables    0    0    0 : slabdata      1      1      0
biovec-64              0     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
biovec-16              0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
ksm_mm_slot            0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_stable_node        0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_rmap_item          0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
user_namespace         0      0    448   18    2 : tunables    0    0    0 : slabdata      0      0      0
uid_cache              4     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache      95    108    672   12    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_ext_cache       0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
skbuff_small_head    165    187    480   17    2 : tunables    0    0    0 : slabdata     11     11      0
skbuff_fclone_cache      1     19    416   19    2 : tunables    0    0    0 : slabdata      1      1      0
skbuff_head_cache    419    468    224   18    1 : tunables    0    0    0 : slabdata     26     26      0
file_lock_cache        1     24    168   24    1 : tunables    0    0    0 : slabdata      1      1      0
file_lock_ctx          7    192     64   64    1 : tunables    0    0    0 : slabdata      3      3      0
fsnotify_mark_connector     33     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
buffer_head          128   1170    104   39    1 : tunables    0    0    0 : slabdata     30     30      0
task_delay_info        0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
taskstats              2     17    456   17    2 : tunables    0    0    0 : slabdata      1      1      0
proc_dir_entry       274    286    152   26    1 : tunables    0    0    0 : slabdata     11     11      0
pde_opener             0    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
proc_inode_cache     531    574    568   14    2 : tunables    0    0    0 : slabdata     41     41      0
seq_file               2     34    120   34    1 : tunables    0    0    0 : slabdata      1      1      0
sigqueue               1     56     72   56    1 : tunables    0    0    0 : slabdata      1      1      0
bdev_cache            27     39   1216   13    4 : tunables    0    0    0 : slabdata      3      3      0
shmem_inode_cache   1313   1425    632   25    4 : tunables    0    0    0 : slabdata     57     57      0
kernfs_iattrs_cache     11     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
kernfs_node_cache  17299  17388    112   36    1 : tunables    0    0    0 : slabdata    483    483      0
mnt_cache             29     36    224   18    1 : tunables    0    0    0 : slabdata      2      2      0
filp                 527    592    256   16    1 : tunables    0    0    0 : slabdata     37     37      0
inode_cache         1894   2340    528   15    2 : tunables    0    0    0 : slabdata    156    156      0
dentry              4434   6555    176   23    1 : tunables    0    0    0 : slabdata    285    285      0
names_cache            0     14   4160    7    8 : tunables    0    0    0 : slabdata      2      2      0
net_namespace          0      0   2624   12    8 : tunables    0    0    0 : slabdata      0      0      0
key_jar                7     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
uts_namespace          0      0    440   18    2 : tunables    0    0    0 : slabdata      0      0      0
nsproxy                1     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
vm_area_struct      1569   1722     96   42    1 : tunables    0    0    0 : slabdata     41     41      0
fs_cache              34     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
files_cache           34     42    288   14    1 : tunables    0    0    0 : slabdata      3      3      0
signal_cache         113    126    864   18    4 : tunables    0    0    0 : slabdata      7      7      0
sighand_cache        113    138   1376   23    8 : tunables    0    0    0 : slabdata      6      6      0
task_struct          113    121   2848   11    8 : tunables    0    0    0 : slabdata     11     11      0
cred_jar             165    189    192   21    1 : tunables    0    0    0 : slabdata      9      9      0
anon_vma_chain      1139   1387     56   73    1 : tunables    0    0    0 : slabdata     19     19      0
anon_vma             809    920     88   46    1 : tunables    0    0    0 : slabdata     20     20      0
pid                  120    128    128   32    1 : tunables    0    0    0 : slabdata      4      4      0
perf_event             0      0    848   19    4 : tunables    0    0    0 : slabdata      0      0      0
pool_workqueue        29     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
maple_node           120    273    768   21    4 : tunables    0    0    0 : slabdata     13     13      0
radix_tree_node     1575   1860    320   12    1 : tunables    0    0    0 : slabdata    155    155      0
task_group            33     48    256   16    1 : tunables    0    0    0 : slabdata      3      3      0
mm_struct             33     51    928   17    4 : tunables    0    0    0 : slabdata      3      3      0
vmap_area            139    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
pgtable-2^10          33     38  12288    2    8 : tunables    0    0    0 : slabdata     19     19      0
kmalloc-cg-8k          0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-cg-4k          5     10  12288    2    8 : tunables    0    0    0 : slabdata      5      5      0
kmalloc-cg-2k         16     20   6144    5    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-1k         43     50   3072   10    8 : tunables    0    0    0 : slabdata      5      5      0
kmalloc-cg-512        16     63   1536   21    8 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-256        27     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-192        25     54    216   18    1 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-128        18     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-96         41     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-64         40     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-32         29     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-16         18     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-8          23    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
dma-kmalloc-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-96        32     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-rcl-64       122    210    192   21    1 : tunables    0    0    0 : slabdata     10     10      0
kmalloc-rcl-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k            15     16  24576    1    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-4k            14     16  12288    2    8 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-2k           342    370   6144    5    8 : tunables    0    0    0 : slabdata     74     74      0
kmalloc-1k           223    270   3072   10    8 : tunables    0    0    0 : slabdata     27     27      0
kmalloc-512          621    630   1536   21    8 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-256          146    168    768   21    4 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-192         4220   4338    216   18    1 : tunables    0    0    0 : slabdata    241    241      0
kmalloc-128          724    735    384   21    2 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-96           733    782    120   34    1 : tunables    0    0    0 : slabdata     23     23      0
kmalloc-64          1348   1995    192   21    1 : tunables    0    0    0 : slabdata     95     95      0
kmalloc-32          1772   1890     96   42    1 : tunables    0    0    0 : slabdata     45     45      0
kmalloc-16          3254   3315     48   85    1 : tunables    0    0    0 : slabdata     39     39      0
kmalloc-8           2870   2944     32  128    1 : tunables    0    0    0 : slabdata     23     23      0
kmem_cache_node      217    252     96   42    1 : tunables    0    0    0 : slabdata      6      6      0
kmem_cache           217    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_1k     48     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_groupinfo_4k    320    322    176   23    1 : tunables    0    0    0 : slabdata     14     14      0
btrfs_prelim_ref       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_extent_op      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_data_ref      0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_tree_ref      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_ref_head      0      0    144   28    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_inode_defrag      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_node      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_ordered_extent      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_extent_map       3     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
bio-232                2     14    288   14    1 : tunables    0    0    0 : slabdata      1      1      0
bio-280                4     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_buffer     11     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_state      3     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_free_space_bitmap      0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
btrfs_free_space       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_path             0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_trans_handle      0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
bio-296                2     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_inode            3     16   1008   16    4 : tunables    0    0    0 : slabdata      1      1      0
bio-328                2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zswap_entry            0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
p9_req_t               0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    168   24    1 : tunables    0    0    0 : slabdata      0      0      0
fib6_nodes             5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ip6_dst_cache          4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
PINGv6                 0      0   1056   15    4 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                  7     15   1056   15    4 : tunables    0    0    0 : slabdata      1      1      0
UDPLITEv6              0      0   1152   14    4 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                  1     14   1152   14    4 : tunables    0    0    0 : slabdata      1      1      0
tw_sock_TCPv6          0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCPv6      0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                  2     15   2080   15    8 : tunables    0    0    0 : slabdata      1      1      0
scsi_sense_cache       9     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
virtio_scsi_cmd       64     64    256   16    1 : tunables    0    0    0 : slabdata      4      4      0
io_kiocb               0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0
bfq_io_cq              0      0   1032   15    4 : tunables    0    0    0 : slabdata      0      0      0
bfq_queue              0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
bio-88                10     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
bio-168                4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
mqueue_inode_cache      1     21    768   21    4 : tunables    0    0    0 : slabdata      1      1      0
v9fs_inode_cache       0      0    600   13    2 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    560   14    2 : tunables    0    0    0 : slabdata      0      0      0
jbd2_transaction_s      4     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_inode            23     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_handle      0     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_head     18     84     96   42    1 : tunables    0    0    0 : slabdata      2      2      0
jbd2_revoke_table_s      2    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_revoke_record_s      0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_fc_dentry_update      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache    1351   1360   1008   16    4 : tunables    0    0    0 : slabdata     85     85      0
ext4_free_data         0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_allocation_context      0     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_prealloc_space      6     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_system_zone      89    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_io_end_vec        0     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_io_end            0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
bio_post_read_ctx    128    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
pending_reservation      0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
extent_status       1080   1095     56   73    1 : tunables    0    0    0 : slabdata     15     15      0
mbcache                0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
kioctx                 0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
aio_kiocb              0      0    160   25    1 : tunables    0    0    0 : slabdata      0      0      0
userfaultfd_ctx_cache      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_path_event      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_fid_event      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
fsnotify_mark          0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark           0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_struct         0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
dio                    0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
fasync_cache           0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
pid_namespace          0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
posix_timers_cache      0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
UNIX-STREAM           27     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
UNIX                  13     17    928   17    4 : tunables    0    0    0 : slabdata      1      1      0
ip4-frags              0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
UDP-Lite               0      0   1024   16    4 : tunables    0    0    0 : slabdata      0      0      0
tcp_bind2_bucket       5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
tcp_bind_bucket        2     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
inet_peer_cache        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    640   12    2 : tunables    0    0    0 : slabdata      0      0      0
ip_fib_trie            7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_fib_alias           7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_dst_cache           6     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
PING                   0      0    896   18    4 : tunables    0    0    0 : slabdata      0      0      0
RAW                    5     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0
UDP                   18     32   1024   16    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP            0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCP       0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                    3     16   1952   16    8 : tunables    0    0    0 : slabdata      1      1      0
bio-184               32     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
ep_head               36    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_pwq         37     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_epi         36     96    128   32    1 : tunables    0    0    0 : slabdata      3      3      0
inotify_inode_mark     40     51     80   51    1 : tunables    0    0    0 : slabdata      1      1      0
sgpool-128             2     12   2624   12    8 : tunables    0    0    0 : slabdata      1      1      0
sgpool-64              2     12   1344   12    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-32              2     23    704   23    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-16              2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
sgpool-8               2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
request_queue          4     22    720   22    4 : tunables    0    0    0 : slabdata      1      1      0
blkdev_ioc             1     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
bio-136                4     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
biovec-max            48     60   3136   10    8 : tunables    0    0    0 : slabdata      6      6      0
biovec-128             0     20   1600   20    8 : tunables    0    0    0 : slabdata      1      1      0
biovec-64              0     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
biovec-16              0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
ksm_mm_slot            0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_stable_node        0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_rmap_item          0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
user_namespace         0      0    448   18    2 : tunables    0    0    0 : slabdata      0      0      0
uid_cache              4     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache      92    108    672   12    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_ext_cache       0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
skbuff_small_head    126    136    480   17    2 : tunables    0    0    0 : slabdata      8      8      0
skbuff_fclone_cache      0     19    416   19    2 : tunables    0    0    0 : slabdata      1      1      0
skbuff_head_cache    404    414    224   18    1 : tunables    0    0    0 : slabdata     23     23      0
file_lock_cache        1     24    168   24    1 : tunables    0    0    0 : slabdata      1      1      0
file_lock_ctx         10    192     64   64    1 : tunables    0    0    0 : slabdata      3      3      0
fsnotify_mark_connector     33     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
buffer_head         5655   5694    104   39    1 : tunables    0    0    0 : slabdata    146    146      0
task_delay_info        0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
taskstats              2     17    456   17    2 : tunables    0    0    0 : slabdata      1      1      0
proc_dir_entry       274    286    152   26    1 : tunables    0    0    0 : slabdata     11     11      0
pde_opener             0    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
proc_inode_cache     194    252    568   14    2 : tunables    0    0    0 : slabdata     18     18      0
seq_file               2     34    120   34    1 : tunables    0    0    0 : slabdata      1      1      0
sigqueue               1     56     72   56    1 : tunables    0    0    0 : slabdata      1      1      0
bdev_cache            27     39   1216   13    4 : tunables    0    0    0 : slabdata      3      3      0
shmem_inode_cache   1315   1425    632   25    4 : tunables    0    0    0 : slabdata     57     57      0
kernfs_iattrs_cache     11     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
kernfs_node_cache  17387  17460    112   36    1 : tunables    0    0    0 : slabdata    485    485      0
mnt_cache             29     36    224   18    1 : tunables    0    0    0 : slabdata      2      2      0
filp                 475    512    256   16    1 : tunables    0    0    0 : slabdata     32     32      0
inode_cache         2469   2475    528   15    2 : tunables    0    0    0 : slabdata    165    165      0
dentry              6477   6486    176   23    1 : tunables    0    0    0 : slabdata    282    282      0
names_cache            0     21   4160    7    8 : tunables    0    0    0 : slabdata      3      3      0
net_namespace          0      0   2624   12    8 : tunables    0    0    0 : slabdata      0      0      0
key_jar                7     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
uts_namespace          0      0    440   18    2 : tunables    0    0    0 : slabdata      0      0      0
nsproxy                1     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
vm_area_struct      1402   1428     96   42    1 : tunables    0    0    0 : slabdata     34     34      0
fs_cache              32     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
files_cache           32     42    288   14    1 : tunables    0    0    0 : slabdata      3      3      0
signal_cache         109    126    864   18    4 : tunables    0    0    0 : slabdata      7      7      0
sighand_cache        109    115   1376   23    8 : tunables    0    0    0 : slabdata      5      5      0
task_struct          109    110   2848   11    8 : tunables    0    0    0 : slabdata     10     10      0
cred_jar             159    189    192   21    1 : tunables    0    0    0 : slabdata      9      9      0
anon_vma_chain      1058   1168     56   73    1 : tunables    0    0    0 : slabdata     16     16      0
anon_vma             764    782     88   46    1 : tunables    0    0    0 : slabdata     17     17      0
pid                  116    128    128   32    1 : tunables    0    0    0 : slabdata      4      4      0
perf_event             0      0    848   19    4 : tunables    0    0    0 : slabdata      0      0      0
pool_workqueue        29     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
maple_node           119    252    768   21    4 : tunables    0    0    0 : slabdata     12     12      0
radix_tree_node     1842   1848    320   12    1 : tunables    0    0    0 : slabdata    154    154      0
task_group            31     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
mm_struct             31     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
vmap_area            139    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
pgtable-2^10          31     32  12288    2    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-cg-8k          0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-cg-4k          5      8  12288    2    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-2k         16     20   6144    5    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-1k         40     40   3072   10    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-512        15     42   1536   21    8 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-256        27     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-192        24     54    216   18    1 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-128        18     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-96         40     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-64         40     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-32         28     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-16         18     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-8          23    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
dma-kmalloc-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-96        32     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-rcl-64       132    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
kmalloc-rcl-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k            15     16  24576    1    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-4k            14     16  12288    2    8 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-2k           380    380   6144    5    8 : tunables    0    0    0 : slabdata     76     76      0
kmalloc-1k           287    290   3072   10    8 : tunables    0    0    0 : slabdata     29     29      0
kmalloc-512          611    630   1536   21    8 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-256          179    189    768   21    4 : tunables    0    0    0 : slabdata      9      9      0
kmalloc-192         4306   4320    216   18    1 : tunables    0    0    0 : slabdata    240    240      0
kmalloc-128          722    735    384   21    2 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-96           724    748    120   34    1 : tunables    0    0    0 : slabdata     22     22      0
kmalloc-64          1346   2079    192   21    1 : tunables    0    0    0 : slabdata     99     99      0
kmalloc-32          1831   1890     96   42    1 : tunables    0    0    0 : slabdata     45     45      0
kmalloc-16          3262   3315     48   85    1 : tunables    0    0    0 : slabdata     39     39      0
kmalloc-8           2862   2944     32  128    1 : tunables    0    0    0 : slabdata     23     23      0
kmem_cache_node      279    294     96   42    1 : tunables    0    0    0 : slabdata      7      7      0
kmem_cache           279    294    192   21    1 : tunables    0    0    0 : slabdata     14     14      0
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_1k     48     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_groupinfo_4k    320    322    176   23    1 : tunables    0    0    0 : slabdata     14     14      0
btrfs_prelim_ref       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_extent_op      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_data_ref      0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_tree_ref      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_ref_head      0      0    144   28    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_inode_defrag      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_node      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_ordered_extent      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_extent_map       3     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
bio-232                2     14    288   14    1 : tunables    0    0    0 : slabdata      1      1      0
bio-280                4     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_buffer     11     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_state      3     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_free_space_bitmap      0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
btrfs_free_space       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_path             0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_trans_handle      0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
bio-296                2     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_inode            3     16   1008   16    4 : tunables    0    0    0 : slabdata      1      1      0
bio-328                2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
zspage                 0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
zs_handle              0      0     24  170    1 : tunables    0    0    0 : slabdata      0      0      0
zswap_entry            0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
p9_req_t               0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    168   24    1 : tunables    0    0    0 : slabdata      0      0      0
fib6_nodes             5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ip6_dst_cache          2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
PINGv6                 0      0   1056   15    4 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                  7     15   1056   15    4 : tunables    0    0    0 : slabdata      1      1      0
UDPLITEv6              0      0   1152   14    4 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                  1     14   1152   14    4 : tunables    0    0    0 : slabdata      1      1      0
tw_sock_TCPv6          0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCPv6      0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                  2     15   2080   15    8 : tunables    0    0    0 : slabdata      1      1      0
scsi_sense_cache       9     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
virtio_scsi_cmd       64     64    256   16    1 : tunables    0    0    0 : slabdata      4      4      0
io_kiocb               0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0
bfq_io_cq              0      0   1032   15    4 : tunables    0    0    0 : slabdata      0      0      0
bfq_queue              0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
bio-88                10     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
bio-168                4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
mqueue_inode_cache      1     21    768   21    4 : tunables    0    0    0 : slabdata      1      1      0
v9fs_inode_cache       0      0    600   13    2 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    560   14    2 : tunables    0    0    0 : slabdata      0      0      0
jbd2_transaction_s      2     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_inode            23     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_handle      0     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_head      8     84     96   42    1 : tunables    0    0    0 : slabdata      2      2      0
jbd2_revoke_table_s      2    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_revoke_record_s      0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_fc_dentry_update      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache    1350   1360   1008   16    4 : tunables    0    0    0 : slabdata     85     85      0
ext4_free_data         0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_allocation_context      0     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_prealloc_space      6     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_system_zone      89    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_io_end_vec        0     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_io_end            0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
bio_post_read_ctx    128    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
pending_reservation      0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
extent_status       1079   1095     56   73    1 : tunables    0    0    0 : slabdata     15     15      0
mbcache                0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
kioctx                 0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
aio_kiocb              0      0    160   25    1 : tunables    0    0    0 : slabdata      0      0      0
userfaultfd_ctx_cache      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_path_event      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_fid_event      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
fsnotify_mark          0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark           0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_struct         0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
dio                    0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
fasync_cache           0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
pid_namespace          0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
posix_timers_cache      0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
UNIX-STREAM           27     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
UNIX                  13     17    928   17    4 : tunables    0    0    0 : slabdata      1      1      0
ip4-frags              0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
UDP-Lite               0      0   1024   16    4 : tunables    0    0    0 : slabdata      0      0      0
tcp_bind2_bucket       5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
tcp_bind_bucket        2     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
inet_peer_cache        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    640   12    2 : tunables    0    0    0 : slabdata      0      0      0
ip_fib_trie            7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_fib_alias           7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_dst_cache           6     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
PING                   0      0    896   18    4 : tunables    0    0    0 : slabdata      0      0      0
RAW                    5     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0
UDP                   18     32   1024   16    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP            0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCP       0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                    3     16   1952   16    8 : tunables    0    0    0 : slabdata      1      1      0
bio-184               32     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
ep_head               36    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_pwq         37     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_epi         36     96    128   32    1 : tunables    0    0    0 : slabdata      3      3      0
inotify_inode_mark     40     51     80   51    1 : tunables    0    0    0 : slabdata      1      1      0
sgpool-128             2     12   2624   12    8 : tunables    0    0    0 : slabdata      1      1      0
sgpool-64              2     12   1344   12    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-32              2     23    704   23    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-16              2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
sgpool-8               2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
request_queue          4     22    720   22    4 : tunables    0    0    0 : slabdata      1      1      0
blkdev_ioc             1     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
bio-136                4     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
biovec-max            48     60   3136   10    8 : tunables    0    0    0 : slabdata      6      6      0
biovec-128             0     20   1600   20    8 : tunables    0    0    0 : slabdata      1      1      0
biovec-64              0     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
biovec-16              0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
ksm_mm_slot            0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_stable_node        0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_rmap_item          0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
user_namespace         0      0    448   18    2 : tunables    0    0    0 : slabdata      0      0      0
uid_cache              4     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache      92    108    672   12    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_ext_cache       0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
skbuff_small_head    111    119    480   17    2 : tunables    0    0    0 : slabdata      7      7      0
skbuff_fclone_cache      0     19    416   19    2 : tunables    0    0    0 : slabdata      1      1      0
skbuff_head_cache    352    378    224   18    1 : tunables    0    0    0 : slabdata     21     21      0
file_lock_cache        1     24    168   24    1 : tunables    0    0    0 : slabdata      1      1      0
file_lock_ctx         10    192     64   64    1 : tunables    0    0    0 : slabdata      3      3      0
fsnotify_mark_connector     33     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
buffer_head         5653   5694    104   39    1 : tunables    0    0    0 : slabdata    146    146      0
task_delay_info        0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
taskstats              2     17    456   17    2 : tunables    0    0    0 : slabdata      1      1      0
proc_dir_entry       274    286    152   26    1 : tunables    0    0    0 : slabdata     11     11      0
pde_opener             0    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
proc_inode_cache     192    238    568   14    2 : tunables    0    0    0 : slabdata     17     17      0
seq_file               2     34    120   34    1 : tunables    0    0    0 : slabdata      1      1      0
sigqueue               1     56     72   56    1 : tunables    0    0    0 : slabdata      1      1      0
bdev_cache            27     39   1216   13    4 : tunables    0    0    0 : slabdata      3      3      0
shmem_inode_cache   1313   1425    632   25    4 : tunables    0    0    0 : slabdata     57     57      0
kernfs_iattrs_cache     11     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
kernfs_node_cache  17325  17388    112   36    1 : tunables    0    0    0 : slabdata    483    483      0
mnt_cache             29     36    224   18    1 : tunables    0    0    0 : slabdata      2      2      0
filp                 475    496    256   16    1 : tunables    0    0    0 : slabdata     31     31      0
inode_cache         2406   2415    528   15    2 : tunables    0    0    0 : slabdata    161    161      0
dentry              6379   6532    176   23    1 : tunables    0    0    0 : slabdata    284    284      0
names_cache            0     28   4160    7    8 : tunables    0    0    0 : slabdata      4      4      0
net_namespace          0      0   2624   12    8 : tunables    0    0    0 : slabdata      0      0      0
key_jar                7     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
uts_namespace          0      0    440   18    2 : tunables    0    0    0 : slabdata      0      0      0
nsproxy                1     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
vm_area_struct      1400   1428     96   42    1 : tunables    0    0    0 : slabdata     34     34      0
fs_cache              32     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
files_cache           32     42    288   14    1 : tunables    0    0    0 : slabdata      3      3      0
signal_cache         108    126    864   18    4 : tunables    0    0    0 : slabdata      7      7      0
sighand_cache        108    115   1376   23    8 : tunables    0    0    0 : slabdata      5      5      0
task_struct          108    110   2848   11    8 : tunables    0    0    0 : slabdata     10     10      0
cred_jar             158    189    192   21    1 : tunables    0    0    0 : slabdata      9      9      0
anon_vma_chain      1053   1168     56   73    1 : tunables    0    0    0 : slabdata     16     16      0
anon_vma             759    782     88   46    1 : tunables    0    0    0 : slabdata     17     17      0
pid                  115    128    128   32    1 : tunables    0    0    0 : slabdata      4      4      0
perf_event             0      0    848   19    4 : tunables    0    0    0 : slabdata      0      0      0
pool_workqueue        29     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
maple_node           121    273    768   21    4 : tunables    0    0    0 : slabdata     13     13      0
radix_tree_node     1814   1824    320   12    1 : tunables    0    0    0 : slabdata    152    152      0
task_group            31     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
mm_struct             31     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
vmap_area            139    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
pgtable-2^10          31     32  12288    2    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-cg-8k          0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-cg-4k          5      8  12288    2    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-2k         16     20   6144    5    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-1k         40     40   3072   10    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-512        15     42   1536   21    8 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-256        27     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-192        24     54    216   18    1 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-128        18     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-96         40     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-64         40     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-32         28     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-16         18     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-8          23    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
dma-kmalloc-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-96        32     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-rcl-64       132    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
kmalloc-rcl-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k            15     16  24576    1    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-4k            14     16  12288    2    8 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-2k           349    350   6144    5    8 : tunables    0    0    0 : slabdata     70     70      0
kmalloc-1k           255    260   3072   10    8 : tunables    0    0    0 : slabdata     26     26      0
kmalloc-512          610    630   1536   21    8 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-256          149    168    768   21    4 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-192          615    630    216   18    1 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-128          721    735    384   21    2 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-96           722    782    120   34    1 : tunables    0    0    0 : slabdata     23     23      0
kmalloc-64          1344   1974    192   21    1 : tunables    0    0    0 : slabdata     94     94      0
kmalloc-32          1831   1890     96   42    1 : tunables    0    0    0 : slabdata     45     45      0
kmalloc-16          3261   3315     48   85    1 : tunables    0    0    0 : slabdata     39     39      0
kmalloc-8           2769   2816     32  128    1 : tunables    0    0    0 : slabdata     22     22      0
kmem_cache_node      217    252     96   42    1 : tunables    0    0    0 : slabdata      6      6      0
kmem_cache           217    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_1k     48     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_groupinfo_4k    320    322    176   23    1 : tunables    0    0    0 : slabdata     14     14      0
btrfs_prelim_ref       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_extent_op      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_data_ref      0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_tree_ref      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_ref_head      0      0    144   28    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_inode_defrag      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_node      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_ordered_extent      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_extent_map       3     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
bio-232                2     14    288   14    1 : tunables    0    0    0 : slabdata      1      1      0
bio-280                4     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_buffer      9     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_state      3     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_free_space_bitmap      0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
btrfs_free_space       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_path             0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_trans_handle      0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
bio-296                2     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_inode            2     16   1008   16    4 : tunables    0    0    0 : slabdata      1      1      0
bio-328                2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                10     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle             10    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 8     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              8    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              7    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 7     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              7    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 8     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              9    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 5     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              5    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 5     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 8     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              8    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 5     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 8     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              8    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 5     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              5    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 4     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              4    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                11     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle             11    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 4     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              4    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 9     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              9    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 8     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              8    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              7    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 5     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 8     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              9    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 4     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              4    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 7     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              7    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 8     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              8    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 7     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              7    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 5     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              5    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 9     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              9    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zspage                 6     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
zs_handle              6    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
zswap_entry          226    512     64   64    1 : tunables    0    0    0 : slabdata      8      8      0
p9_req_t               0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    168   24    1 : tunables    0    0    0 : slabdata      0      0      0
fib6_nodes             5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ip6_dst_cache          4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
PINGv6                 0      0   1056   15    4 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                  7     15   1056   15    4 : tunables    0    0    0 : slabdata      1      1      0
UDPLITEv6              0      0   1152   14    4 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                  1     14   1152   14    4 : tunables    0    0    0 : slabdata      1      1      0
tw_sock_TCPv6          0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCPv6      0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                  2     15   2080   15    8 : tunables    0    0    0 : slabdata      1      1      0
scsi_sense_cache       9     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
virtio_scsi_cmd       64     64    256   16    1 : tunables    0    0    0 : slabdata      4      4      0
io_kiocb               0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0
bfq_io_cq              0      0   1032   15    4 : tunables    0    0    0 : slabdata      0      0      0
bfq_queue              0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
bio-88                10     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
bio-168                4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
mqueue_inode_cache      1     21    768   21    4 : tunables    0    0    0 : slabdata      1      1      0
v9fs_inode_cache       0      0    600   13    2 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    560   14    2 : tunables    0    0    0 : slabdata      0      0      0
jbd2_transaction_s      3     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_inode            16     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_handle      0     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_head      7     84     96   42    1 : tunables    0    0    0 : slabdata      2      2      0
jbd2_revoke_table_s      2    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_revoke_record_s      0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_fc_dentry_update      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache     564    912   1008   16    4 : tunables    0    0    0 : slabdata     57     57      0
ext4_free_data         0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_allocation_context      0     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_prealloc_space      6     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_system_zone      89    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_io_end_vec        0     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_io_end            0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
bio_post_read_ctx    128    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
pending_reservation      0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
extent_status         45    292     56   73    1 : tunables    0    0    0 : slabdata      4      4      0
mbcache                0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
kioctx                 0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
aio_kiocb              0      0    160   25    1 : tunables    0    0    0 : slabdata      0      0      0
userfaultfd_ctx_cache      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_path_event      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_fid_event      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
fsnotify_mark          0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark           0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_struct         0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
dio                    0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
fasync_cache           0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
pid_namespace          0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
posix_timers_cache      0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
UNIX-STREAM           29     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
UNIX                  13     17    928   17    4 : tunables    0    0    0 : slabdata      1      1      0
ip4-frags              0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
UDP-Lite               0      0   1024   16    4 : tunables    0    0    0 : slabdata      0      0      0
tcp_bind2_bucket       5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
tcp_bind_bucket        2     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
inet_peer_cache        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    640   12    2 : tunables    0    0    0 : slabdata      0      0      0
ip_fib_trie            7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_fib_alias           7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_dst_cache           6     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
PING                   0      0    896   18    4 : tunables    0    0    0 : slabdata      0      0      0
RAW                    5     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0
UDP                   18     32   1024   16    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP            0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCP       0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                    3     16   1952   16    8 : tunables    0    0    0 : slabdata      1      1      0
bio-184               32     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
ep_head               40    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_pwq         42     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_epi         40     96    128   32    1 : tunables    0    0    0 : slabdata      3      3      0
inotify_inode_mark     40     51     80   51    1 : tunables    0    0    0 : slabdata      1      1      0
sgpool-128             2     12   2624   12    8 : tunables    0    0    0 : slabdata      1      1      0
sgpool-64              2     12   1344   12    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-32              2     23    704   23    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-16              2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
sgpool-8               2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
request_queue          4     22    720   22    4 : tunables    0    0    0 : slabdata      1      1      0
blkdev_ioc             1     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
bio-136                4     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
biovec-max            48     60   3136   10    8 : tunables    0    0    0 : slabdata      6      6      0
biovec-128             0     20   1600   20    8 : tunables    0    0    0 : slabdata      1      1      0
biovec-64              0     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
biovec-16              0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
ksm_mm_slot            0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_stable_node        0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_rmap_item          0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
user_namespace         0      0    448   18    2 : tunables    0    0    0 : slabdata      0      0      0
uid_cache              4     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache      94    108    672   12    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_ext_cache       0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
skbuff_small_head    158    170    480   17    2 : tunables    0    0    0 : slabdata     10     10      0
skbuff_fclone_cache      0     19    416   19    2 : tunables    0    0    0 : slabdata      1      1      0
skbuff_head_cache    418    450    224   18    1 : tunables    0    0    0 : slabdata     25     25      0
file_lock_cache        1     24    168   24    1 : tunables    0    0    0 : slabdata      1      1      0
file_lock_ctx          8    192     64   64    1 : tunables    0    0    0 : slabdata      3      3      0
fsnotify_mark_connector     33     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
buffer_head          126   1248    104   39    1 : tunables    0    0    0 : slabdata     32     32      0
task_delay_info        0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
taskstats              2     17    456   17    2 : tunables    0    0    0 : slabdata      1      1      0
proc_dir_entry       274    286    152   26    1 : tunables    0    0    0 : slabdata     11     11      0
pde_opener             0    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
proc_inode_cache     834    910    568   14    2 : tunables    0    0    0 : slabdata     65     65      0
seq_file               2     34    120   34    1 : tunables    0    0    0 : slabdata      1      1      0
sigqueue               1     56     72   56    1 : tunables    0    0    0 : slabdata      1      1      0
bdev_cache            27     39   1216   13    4 : tunables    0    0    0 : slabdata      3      3      0
shmem_inode_cache   1315   1425    632   25    4 : tunables    0    0    0 : slabdata     57     57      0
kernfs_iattrs_cache     11     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
kernfs_node_cache  17359  17424    112   36    1 : tunables    0    0    0 : slabdata    484    484      0
mnt_cache             29     36    224   18    1 : tunables    0    0    0 : slabdata      2      2      0
filp                 526    592    256   16    1 : tunables    0    0    0 : slabdata     37     37      0
inode_cache          756   1575    528   15    2 : tunables    0    0    0 : slabdata    105    105      0
dentry              3245   6348    176   23    1 : tunables    0    0    0 : slabdata    276    276      0
names_cache            0     21   4160    7    8 : tunables    0    0    0 : slabdata      3      3      0
net_namespace          0      0   2624   12    8 : tunables    0    0    0 : slabdata      0      0      0
key_jar                7     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
uts_namespace          0      0    440   18    2 : tunables    0    0    0 : slabdata      0      0      0
nsproxy                1     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
vm_area_struct      1570   1638     96   42    1 : tunables    0    0    0 : slabdata     39     39      0
fs_cache              34     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
files_cache           34     42    288   14    1 : tunables    0    0    0 : slabdata      3      3      0
signal_cache         111    126    864   18    4 : tunables    0    0    0 : slabdata      7      7      0
sighand_cache        111    138   1376   23    8 : tunables    0    0    0 : slabdata      6      6      0
task_struct          111    121   2848   11    8 : tunables    0    0    0 : slabdata     11     11      0
cred_jar             163    189    192   21    1 : tunables    0    0    0 : slabdata      9      9      0
anon_vma_chain      1142   1314     56   73    1 : tunables    0    0    0 : slabdata     18     18      0
anon_vma             812    874     88   46    1 : tunables    0    0    0 : slabdata     19     19      0
pid                  118    128    128   32    1 : tunables    0    0    0 : slabdata      4      4      0
perf_event             0      0    848   19    4 : tunables    0    0    0 : slabdata      0      0      0
pool_workqueue        29     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
maple_node           130    294    768   21    4 : tunables    0    0    0 : slabdata     14     14      0
radix_tree_node     1698   1908    320   12    1 : tunables    0    0    0 : slabdata    159    159      0
task_group            33     48    256   16    1 : tunables    0    0    0 : slabdata      3      3      0
mm_struct             33     51    928   17    4 : tunables    0    0    0 : slabdata      3      3      0
vmap_area            141    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
pgtable-2^10          33     38  12288    2    8 : tunables    0    0    0 : slabdata     19     19      0
kmalloc-cg-8k          0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-cg-4k          5      8  12288    2    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-2k         16     20   6144    5    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-1k         43     50   3072   10    8 : tunables    0    0    0 : slabdata      5      5      0
kmalloc-cg-512        16     42   1536   21    8 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-256        27     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-192        25     54    216   18    1 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-128        18     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-96         41     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-64         40     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-32         29     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-16         18     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-8          23    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
dma-kmalloc-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-96        32     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-rcl-64       122    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
kmalloc-rcl-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k            15     16  24576    1    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-4k            14     16  12288    2    8 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-2k           342    370   6144    5    8 : tunables    0    0    0 : slabdata     74     74      0
kmalloc-1k           256    300   3072   10    8 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-512          619    630   1536   21    8 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-256          177    189    768   21    4 : tunables    0    0    0 : slabdata      9      9      0
kmalloc-192         4218   4320    216   18    1 : tunables    0    0    0 : slabdata    240    240      0
kmalloc-128          722    735    384   21    2 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-96           722    782    120   34    1 : tunables    0    0    0 : slabdata     23     23      0
kmalloc-64          1325   2058    192   21    1 : tunables    0    0    0 : slabdata     98     98      0
kmalloc-32          1768   1890     96   42    1 : tunables    0    0    0 : slabdata     45     45      0
kmalloc-16          3252   3315     48   85    1 : tunables    0    0    0 : slabdata     39     39      0
kmalloc-8           2867   2944     32  128    1 : tunables    0    0    0 : slabdata     23     23      0
kmem_cache_node      279    294     96   42    1 : tunables    0    0    0 : slabdata      7      7      0
kmem_cache           279    294    192   21    1 : tunables    0    0    0 : slabdata     14     14      0
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_1k     48     48    168   24    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_groupinfo_4k    320    322    176   23    1 : tunables    0    0    0 : slabdata     14     14      0
btrfs_prelim_ref       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_extent_op      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_data_ref      0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_tree_ref      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_ref_head      0      0    144   28    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_inode_defrag      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_delayed_node      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_ordered_extent      0      0    304   13    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_extent_map       3     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
bio-232                2     14    288   14    1 : tunables    0    0    0 : slabdata      1      1      0
bio-280                4     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_buffer      9     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_extent_state      3     46     88   46    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_free_space_bitmap      0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
btrfs_free_space       0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
btrfs_path             0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
btrfs_trans_handle      0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
bio-296                2     23    352   23    2 : tunables    0    0    0 : slabdata      1      1      0
btrfs_inode            3     16   1008   16    4 : tunables    0    0    0 : slabdata      1      1      0
bio-328                2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
zspage                45    256     64   64    1 : tunables    0    0    0 : slabdata      4      4      0
zs_handle            131   1190     24  170    1 : tunables    0    0    0 : slabdata      7      7      0
zswap_entry          131    512     64   64    1 : tunables    0    0    0 : slabdata      8      8      0
p9_req_t               0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    168   24    1 : tunables    0    0    0 : slabdata      0      0      0
fib6_nodes             5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ip6_dst_cache          2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
PINGv6                 0      0   1056   15    4 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                  7     15   1056   15    4 : tunables    0    0    0 : slabdata      1      1      0
UDPLITEv6              0      0   1152   14    4 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                  1     14   1152   14    4 : tunables    0    0    0 : slabdata      1      1      0
tw_sock_TCPv6          0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCPv6      0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                  2     15   2080   15    8 : tunables    0    0    0 : slabdata      1      1      0
scsi_sense_cache       9     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
virtio_scsi_cmd       64     64    256   16    1 : tunables    0    0    0 : slabdata      4      4      0
io_kiocb               0      0    224   18    1 : tunables    0    0    0 : slabdata      0      0      0
bfq_io_cq              0      0   1032   15    4 : tunables    0    0    0 : slabdata      0      0      0
bfq_queue              0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
bio-88                10     25    160   25    1 : tunables    0    0    0 : slabdata      1      1      0
bio-168                4     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
mqueue_inode_cache      1     21    768   21    4 : tunables    0    0    0 : slabdata      1      1      0
v9fs_inode_cache       0      0    600   13    2 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    560   14    2 : tunables    0    0    0 : slabdata      0      0      0
jbd2_transaction_s      2     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_inode            25     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_handle      0     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_journal_head      7     84     96   42    1 : tunables    0    0    0 : slabdata      2      2      0
jbd2_revoke_table_s      2    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_revoke_record_s      0     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_fc_dentry_update      0      0    112   36    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache    1349   1376   1008   16    4 : tunables    0    0    0 : slabdata     86     86      0
ext4_free_data         0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_allocation_context      0     28    144   28    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_prealloc_space      6     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_system_zone      89    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
ext4_io_end_vec        0     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_io_end            0     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
bio_post_read_ctx    128    170     48   85    1 : tunables    0    0    0 : slabdata      2      2      0
pending_reservation      0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
extent_status        435   1095     56   73    1 : tunables    0    0    0 : slabdata     15     15      0
mbcache                0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
kioctx                 0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
aio_kiocb              0      0    160   25    1 : tunables    0    0    0 : slabdata      0      0      0
userfaultfd_ctx_cache      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_path_event      0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
fanotify_fid_event      0      0     72   56    1 : tunables    0    0    0 : slabdata      0      0      0
fsnotify_mark          0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_mark           0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_struct         0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
dio                    0      0    416   19    2 : tunables    0    0    0 : slabdata      0      0      0
fasync_cache           0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
pid_namespace          0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
posix_timers_cache      0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
UNIX-STREAM           29     34    928   17    4 : tunables    0    0    0 : slabdata      2      2      0
UNIX                  13     17    928   17    4 : tunables    0    0    0 : slabdata      1      1      0
ip4-frags              0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
UDP-Lite               0      0   1024   16    4 : tunables    0    0    0 : slabdata      0      0      0
tcp_bind2_bucket       5     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
tcp_bind_bucket        2     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
inet_peer_cache        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    640   12    2 : tunables    0    0    0 : slabdata      0      0      0
ip_fib_trie            7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_fib_alias           7     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
ip_dst_cache           6     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
PING                   0      0    896   18    4 : tunables    0    0    0 : slabdata      0      0      0
RAW                    5     18    896   18    4 : tunables    0    0    0 : slabdata      1      1      0
UDP                   18     32   1024   16    4 : tunables    0    0    0 : slabdata      2      2      0
tw_sock_TCP            0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
request_sock_TCP       0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
TCP                    3     16   1952   16    8 : tunables    0    0    0 : slabdata      1      1      0
bio-184               32     32    256   16    1 : tunables    0    0    0 : slabdata      2      2      0
ep_head               40    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_pwq         42     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
eventpoll_epi         40     96    128   32    1 : tunables    0    0    0 : slabdata      3      3      0
inotify_inode_mark     40     51     80   51    1 : tunables    0    0    0 : slabdata      1      1      0
sgpool-128             2     12   2624   12    8 : tunables    0    0    0 : slabdata      1      1      0
sgpool-64              2     12   1344   12    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-32              2     23    704   23    4 : tunables    0    0    0 : slabdata      1      1      0
sgpool-16              2     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
sgpool-8               2     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
request_queue          4     22    720   22    4 : tunables    0    0    0 : slabdata      1      1      0
blkdev_ioc             1     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
bio-136                4     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
biovec-max            48     60   3136   10    8 : tunables    0    0    0 : slabdata      6      6      0
biovec-128             0     20   1600   20    8 : tunables    0    0    0 : slabdata      1      1      0
biovec-64              0     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
biovec-16              0     16    256   16    1 : tunables    0    0    0 : slabdata      1      1      0
ksm_mm_slot            0     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
ksm_stable_node        0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
ksm_rmap_item          0      0     56   73    1 : tunables    0    0    0 : slabdata      0      0      0
user_namespace         0      0    448   18    2 : tunables    0    0    0 : slabdata      0      0      0
uid_cache              4     21    192   21    1 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache      94    108    672   12    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_ext_cache       0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
skbuff_small_head    145    153    480   17    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_fclone_cache      0     19    416   19    2 : tunables    0    0    0 : slabdata      1      1      0
skbuff_head_cache    416    432    224   18    1 : tunables    0    0    0 : slabdata     24     24      0
file_lock_cache        1     24    168   24    1 : tunables    0    0    0 : slabdata      1      1      0
file_lock_ctx         10    192     64   64    1 : tunables    0    0    0 : slabdata      3      3      0
fsnotify_mark_connector     33     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
buffer_head          214   1560    104   39    1 : tunables    0    0    0 : slabdata     40     40      0
task_delay_info        0      0    176   23    1 : tunables    0    0    0 : slabdata      0      0      0
taskstats              2     17    456   17    2 : tunables    0    0    0 : slabdata      1      1      0
proc_dir_entry       274    286    152   26    1 : tunables    0    0    0 : slabdata     11     11      0
pde_opener             0    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
proc_inode_cache     513    560    568   14    2 : tunables    0    0    0 : slabdata     40     40      0
seq_file               2     34    120   34    1 : tunables    0    0    0 : slabdata      1      1      0
sigqueue               1     56     72   56    1 : tunables    0    0    0 : slabdata      1      1      0
bdev_cache            27     39   1216   13    4 : tunables    0    0    0 : slabdata      3      3      0
shmem_inode_cache   1313   1425    632   25    4 : tunables    0    0    0 : slabdata     57     57      0
kernfs_iattrs_cache     11     36    112   36    1 : tunables    0    0    0 : slabdata      1      1      0
kernfs_node_cache  17313  17388    112   36    1 : tunables    0    0    0 : slabdata    483    483      0
mnt_cache             29     36    224   18    1 : tunables    0    0    0 : slabdata      2      2      0
filp                 527    576    256   16    1 : tunables    0    0    0 : slabdata     36     36      0
inode_cache          896   1620    528   15    2 : tunables    0    0    0 : slabdata    108    108      0
dentry              4324   6095    176   23    1 : tunables    0    0    0 : slabdata    265    265      0
names_cache            0     28   4160    7    8 : tunables    0    0    0 : slabdata      4      4      0
net_namespace          0      0   2624   12    8 : tunables    0    0    0 : slabdata      0      0      0
key_jar                7     18    224   18    1 : tunables    0    0    0 : slabdata      1      1      0
uts_namespace          0      0    440   18    2 : tunables    0    0    0 : slabdata      0      0      0
nsproxy                1     73     56   73    1 : tunables    0    0    0 : slabdata      1      1      0
vm_area_struct      1568   1680     96   42    1 : tunables    0    0    0 : slabdata     40     40      0
fs_cache              34     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
files_cache           34     42    288   14    1 : tunables    0    0    0 : slabdata      3      3      0
signal_cache         110    126    864   18    4 : tunables    0    0    0 : slabdata      7      7      0
sighand_cache        110    115   1376   23    8 : tunables    0    0    0 : slabdata      5      5      0
task_struct          110    121   2848   11    8 : tunables    0    0    0 : slabdata     11     11      0
cred_jar             162    189    192   21    1 : tunables    0    0    0 : slabdata      9      9      0
anon_vma_chain      1137   1460     56   73    1 : tunables    0    0    0 : slabdata     20     20      0
anon_vma             807    920     88   46    1 : tunables    0    0    0 : slabdata     20     20      0
pid                  117    128    128   32    1 : tunables    0    0    0 : slabdata      4      4      0
perf_event             0      0    848   19    4 : tunables    0    0    0 : slabdata      0      0      0
pool_workqueue        29     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
maple_node           129    273    768   21    4 : tunables    0    0    0 : slabdata     13     13      0
radix_tree_node     1827   1908    320   12    1 : tunables    0    0    0 : slabdata    159    159      0
task_group            33     48    256   16    1 : tunables    0    0    0 : slabdata      3      3      0
mm_struct             33     51    928   17    4 : tunables    0    0    0 : slabdata      3      3      0
vmap_area            141    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
pgtable-2^10          33     38  12288    2    8 : tunables    0    0    0 : slabdata     19     19      0
kmalloc-cg-8k          0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-cg-4k          5      8  12288    2    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-2k         16     20   6144    5    8 : tunables    0    0    0 : slabdata      4      4      0
kmalloc-cg-1k         43     50   3072   10    8 : tunables    0    0    0 : slabdata      5      5      0
kmalloc-cg-512        16     63   1536   21    8 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-256        27     42    768   21    4 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-192        25     54    216   18    1 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-cg-128        18     21    384   21    2 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-96         41     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-64         40     42    192   21    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-cg-32         29     42     96   42    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-16         18     85     48   85    1 : tunables    0    0    0 : slabdata      1      1      0
kmalloc-cg-8          23    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
dma-kmalloc-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0    120   34    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0  24576    1    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0  12288    2    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   6144    5    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   3072   10    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512        0      0   1536   21    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-256        0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-192        0      0    216   18    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-128        0      0    384   21    2 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-96        32     68    120   34    1 : tunables    0    0    0 : slabdata      2      2      0
kmalloc-rcl-64       128    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
kmalloc-rcl-32         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     48   85    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k            15     16  24576    1    8 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-4k            14     16  12288    2    8 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-2k           343    355   6144    5    8 : tunables    0    0    0 : slabdata     71     71      0
kmalloc-1k           289    290   3072   10    8 : tunables    0    0    0 : slabdata     29     29      0
kmalloc-512          620    630   1536   21    8 : tunables    0    0    0 : slabdata     30     30      0
kmalloc-256          149    168    768   21    4 : tunables    0    0    0 : slabdata      8      8      0
kmalloc-192          562    648    216   18    1 : tunables    0    0    0 : slabdata     36     36      0
kmalloc-128          721    735    384   21    2 : tunables    0    0    0 : slabdata     35     35      0
kmalloc-96           708    782    120   34    1 : tunables    0    0    0 : slabdata     23     23      0
kmalloc-64          1328   1974    192   21    1 : tunables    0    0    0 : slabdata     94     94      0
kmalloc-32          1829   1890     96   42    1 : tunables    0    0    0 : slabdata     45     45      0
kmalloc-16          3263   3315     48   85    1 : tunables    0    0    0 : slabdata     39     39      0
kmalloc-8           2774   2816     32  128    1 : tunables    0    0    0 : slabdata     22     22      0
kmem_cache_node      217    252     96   42    1 : tunables    0    0    0 : slabdata      6      6      0
kmem_cache           217    231    192   21    1 : tunables    0    0    0 : slabdata     11     11      0
#15 lore raw
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

Quoted 5 lines

Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.

Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.

The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.

Thanks for trying this out. The patch reduces the amount of wasted memory due to the 'zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).

I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.

The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.

Quoted 3 lines

Regards,
Erhard

#16 lore raw
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

Quoted 5 lines

Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.

Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.

The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.

Thanks for trying this out. The patch reduces the amount of wasted memory due to the 'zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).

Quoted 8 lines

I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.

The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.

Hi Erhard,

If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.

The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.

Thank you.

#17 lore raw
Quoted 39 lines · Yu Zhao
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

Quoted 5 lines

Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.

Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.

The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.

Thanks for trying this out. The patch reduces the amount of wasted memory due to the 'zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).

Quoted 8 lines

I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.

The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.

Quoted 8 lines

Hi Erhard,

If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.

The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.

I thought about the internal fragmentation of pools, but zsmalloc should have access to highmem, and if I understand correctly the problem here is that we are running out of space in the DMA zone when making kernel allocations.

Do you suspect zsmalloc is allocating memory from the DMA zone initially, even though it has access to highmem?

Quoted 2 lines

Thank you.

#18 lore raw
Quoted 50 lines · Yosry Ahmed
Quoted 39 lines · Yu Zhao
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

Quoted 5 lines

Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.

Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.

The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.

Thanks for trying this out. The patch reduces the amount of wasted memory due to the 'zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).

Quoted 8 lines

I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.

The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.

Quoted 8 lines

Hi Erhard,

If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.

The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.

Quoted 7 lines

I thought about the internal fragmentation of pools, but zsmalloc should have access to highmem, and if I understand correctly the problem here is that we are running out of space in the DMA zone when making kernel allocations.

Do you suspect zsmalloc is allocating memory from the DMA zone initially, even though it has access to highmem?

There was a lot of user memory in the DMA zone. So at a point the highmem zone was full and allocation fallback happened.

The problem with zone fallback is that recent allocations go into lower zones, meaning they are further back on the LRU list. This applies to both user memory and zsmalloc memory -- the latter has a writeback LRU. On top of this, neither the zswap shrinker nor the zsmalloc shrinker (compaction) is zone aware. So page reclaim might have trouble hitting the right target zone.

We can't really tell how zspages are distributed across zones, but the overall number might be helpful. It'd be great if someone could make nr_zspages per zone :)

#19 lore raw
Quoted 63 lines · Yu Zhao
Quoted 50 lines · Yosry Ahmed
Quoted 39 lines · Yu Zhao
Quoted 28 lines · Yosry Ahmed
Quoted 16 lines · Erhard Furtner
Quoted 11 lines · On Mon, 3 Jun 2024 16:24:02 -0700 Yosry Ahmed <[email protected]> wrote:

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

Quoted 5 lines

Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good HEAD, from a bad HEAD and from the bad HEAD + your patch applied.

Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the slabinfo shortly after boot and a 2nd time shortly before the OOM or the kswapd0: page allocation failure happens. I terminated the workload (stress-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM exhausted and got the slabinfo then.

The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortunately didn't make a difference, I got the kswapd0: page allocation failure nevertheless.

Thanks for trying this out. The patch reduces the amount of wasted memory due to the 'zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K).

Quoted 8 lines

I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from.

The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas.

Quoted 8 lines

Hi Erhard,

If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.

The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.

Quoted 7 lines

I thought about the internal fragmentation of pools, but zsmalloc should have access to highmem, and if I understand correctly the problem here is that we are running out of space in the DMA zone when making kernel allocations.

Do you suspect zsmalloc is allocating memory from the DMA zone initially, even though it has access to highmem?

There was a lot of user memory in the DMA zone. So at a point the highmem zone was full and allocation fallback happened.

Quoted 6 lines

The problem with zone fallback is that recent allocations go into lower zones, meaning they are further back on the LRU list. This applies to both user memory and zsmalloc memory -- the latter has a writeback LRU. On top of this, neither the zswap shrinker nor the zsmalloc shrinker (compaction) is zone aware. So page reclaim might have trouble hitting the right target zone.

I see what you mean. In this case, yeah I think the internal fragmentation in the zsmalloc pools may be the reason behind the problem.

How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be max(nr_cpus, 32)?

Alternatively, the number of pools should scale with the memory size in some way, such that we only increase fragmentation when it's tolerable.

Quoted 4 lines

We can't really tell how zspages are distributed across zones, but the overall number might be helpful. It'd be great if someone could make nr_zspages per zone :)

#20 lore raw
Quoted 15 lines · Yosry Ahmed
Quoted 9 lines · Yu Zhao

There was a lot of user memory in the DMA zone. So at a point the highmem zone was full and allocation fallback happened.

The problem with zone fallback is that recent allocations go into lower zones, meaning they are further back on the LRU list. This applies to both user memory and zsmalloc memory -- the latter has a writeback LRU. On top of this, neither the zswap shrinker nor the zsmalloc shrinker (compaction) is zone aware. So page reclaim might have trouble hitting the right target zone.

I see what you mean. In this case, yeah I think the internal fragmentation in the zsmalloc pools may be the reason behind the problem.

Quoted 7 lines

How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be max(nr_cpus, 32)?

Alternatively, the number of pools should scale with the memory size in some way, such that we only increase fragmentation when it's tolerable.

Sounds like a good idea to me, maybe a combination of both. No point in trying to scale if there's no benefit and only downside of more memory consumption.

#21 lore raw
In reply to Yosry Ahmed
Quoted 3 lines · On Tue, 4 Jun 2024 11:01:39 -0700 Yosry Ahmed <[email protected]> wrote:

How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be max(nr_cpus, 32)?

This PowerMac G4 DP got 2 CPUs. Not much for a desktop machine by todays standards but some SoCs have less. ;)

 # lscpu 
Architecture:          ppc
  CPU op-mode(s):      32-bit
  Byte Order:          Big Endian
CPU(s):                2
  On-line CPU(s) list: 0,1
Model name:            7455, altivec supported
  Model:               3.3 (pvr 8001 0303)
  Thread(s) per core:  1
  Core(s) per socket:  1
  Socket(s):           2
  BogoMIPS:            83.78
Caches (sum of all):   
  L1d:                 64 KiB (2 instances)
  L1i:                 64 KiB (2 instances)
  L2:                  512 KiB (2 instances)
  L3:                  4 MiB (2 instances)

Regards,
Erhard

#22 lore raw
Quoted 28 lines · Erhard Furtner
Quoted 3 lines · On Tue, 4 Jun 2024 11:01:39 -0700 Yosry Ahmed <[email protected]> wrote:

How many CPUs does this machine have? I am wondering if 32 can be an overkill for small machines, perhaps the number of pools should be max(nr_cpus, 32)?

This PowerMac G4 DP got 2 CPUs. Not much for a desktop machine by todays standards but some SoCs have less. ;)

 # lscpu
Architecture:          ppc
  CPU op-mode(s):      32-bit
  Byte Order:          Big Endian
CPU(s):                2
  On-line CPU(s) list: 0,1
Model name:            7455, altivec supported
  Model:               3.3 (pvr 8001 0303)
  Thread(s) per core:  1
  Core(s) per socket:  1
  Socket(s):           2
  BogoMIPS:            83.78
Caches (sum of all):
  L1d:                 64 KiB (2 instances)
  L1i:                 64 KiB (2 instances)
  L2:                  512 KiB (2 instances)
  L3:                  4 MiB (2 instances)
Quoted 2 lines

Regards,
Erhard

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

From eb2baf6f3497936ec56fe17e22af5b5add624371 Mon Sep 17 00:00:00 2001
From: Yosry Ahmed <[email protected]>
Date: Wed, 5 Jun 2024 02:58:11 +0000
Subject: [PATCH] mm: zswap: do not scale the number of zpools unnecessarily

Zswap allocates several zpools per zswap pool for scalability. It currently allocates 32 zpools, an empirically determined magic number.

For small machines that has less than 32 CPUs, this doesn't make much sense. Allocating more zpools than CPUs will not significantly increase scalability, but it may increase internal fragmentation due to spreading of compressed pages among different zpools.

determined at boot as the smaller of 32 and the number of CPUs.

---
 mm/zswap.c | 35 ++++++++++++++++++++++++++---------
 1 file changed, 26 insertions(+), 9 deletions(-)
mm/zswap.c +26 -9
index a50e2986cd2fa..9c4a6d309b23c 100644
@@ -123,8 +123,8 @@ static unsigned int zswap_accept_thr_percent = 90; /* of max pool size */
 module_param_named(accept_threshold_percent, zswap_accept_thr_percent,
 		   uint, 0644);
 
-/* Number of zpools in zswap_pool (empirically determined for scalability) */
-#define ZSWAP_NR_ZPOOLS 32
+/* Number of zpools in zswap_pool */
+static unsigned int nr_zswap_zpools __ro_after_init;
 
 /* Enable/disable memory pressure-based shrinker. */
 static bool zswap_shrinker_enabled = IS_ENABLED(
@@ -156,7 +156,7 @@ struct crypto_acomp_ctx {
  * needs to be verified that it's still valid in the tree.
  */
 struct zswap_pool {
-	struct zpool *zpools[ZSWAP_NR_ZPOOLS];
+	struct zpool **zpools;
 	struct crypto_acomp_ctx __percpu *acomp_ctx;
 	struct percpu_ref ref;
 	struct list_head list;
@@ -268,7 +268,14 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor)
 	if (!pool)
 		return NULL;
 
-	for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) {
+	pool->zpools = kcalloc(nr_zswap_zpools, sizeof(pool->zpools[0]),
+			       GFP_KERNEL);
+	if (!pool->zpools) {
+		pr_err("zpools alloc failed\n");
+		goto error;
+	}
+
+	for (i = 0; i < nr_zswap_zpools; i++) {
 		/* unique name for each pool specifically required by zsmalloc */
 		snprintf(name, 38, "zswap%x",
 			 atomic_inc_return(&zswap_pools_count));
@@ -312,8 +319,11 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor)
 error:
 	if (pool->acomp_ctx)
 		free_percpu(pool->acomp_ctx);
-	while (i--)
-		zpool_destroy_pool(pool->zpools[i]);
+	if (pool->zpools) {
+		while (i--)
+			zpool_destroy_pool(pool->zpools[i]);
+		kfree(pool->zpools);
+	}
 	kfree(pool);
 	return NULL;
 }
@@ -369,8 +379,9 @@ static void zswap_pool_destroy(struct zswap_pool *pool)
 	cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node);
 	free_percpu(pool->acomp_ctx);
 
-	for (i = 0; i < ZSWAP_NR_ZPOOLS; i++)
+	for (i = 0; i < nr_zswap_zpools; i++)
 		zpool_destroy_pool(pool->zpools[i]);
+	kfree(pool->zpools);
 	kfree(pool);
 }
 
@@ -496,7 +507,7 @@ unsigned long zswap_total_pages(void)
 	list_for_each_entry_rcu(pool, &zswap_pools, list) {
 		int i;
 
-		for (i = 0; i < ZSWAP_NR_ZPOOLS; i++)
+		for (i = 0; i < nr_zswap_zpools; i++)
 			total += zpool_get_total_pages(pool->zpools[i]);
 	}
 	rcu_read_unlock();
@@ -805,7 +816,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
 
 static struct zpool *zswap_find_zpool(struct zswap_entry *entry)
 {
-	return entry->pool->zpools[hash_ptr(entry, ilog2(ZSWAP_NR_ZPOOLS))];
+	return entry->pool->zpools[hash_ptr(entry, ilog2(nr_zswap_zpools))];
 }
 
 /*
@@ -1767,6 +1778,12 @@ static int zswap_setup(void)
 
 static int __init zswap_init(void)
 {
+	/*
+	 * Use multiple zpools for scalability, but avoid allocating too many.
+	 * 32 is an empirically determined magic number.
+	 */
+	nr_zswap_zpools = min(32U, num_possible_cpus());
+
 	if (!zswap_enabled)
 		return 0;
 	return zswap_setup();
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
#23 lore raw
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

Thanks! The patch does not fix the issue but it helps.

Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.

Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.

The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.

Regards,
Erhard

#24 lore raw
Quoted 4 lines · Erhard Furtner

On Tue, 4 Jun 2024 20:03:27 -0700
Yosry Ahmed <[email protected]> wrote:

Quoted 8 lines
Quoted 2 lines

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

Thanks! The patch does not fix the issue but it helps.

Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.

Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

(e) A creative mix of the above.

(f) Something else (probably simpler).

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.

Quoted 5 lines

The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.

Regards,
Erhard

#25 lore raw
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

Quoted 8 lines
Quoted 5 lines

Thanks! The patch does not fix the issue but it helps.

Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.

Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

Now I'm a little bit skeptical that the problem is due to fragmentation.

Quoted 33 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

Quoted 6 lines

(e) A creative mix of the above.

(f) Something else (probably simpler).

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,

static struct zpool *zswap_find_zpool(struct zswap_entry *entry) {
- return entry->pool->zpools[hash_ptr(entry, ilog2(ZSWAP_NR_ZPOOLS))];
+ return entry->pool->zpools[0];
}

Quoted 8 lines

In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.

Quoted 5 lines

The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.

Regards,
Erhard

#26 lore raw
Quoted 57 lines · Yu Zhao
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

Quoted 8 lines
Quoted 5 lines

Thanks! The patch does not fix the issue but it helps.

Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.

Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

Now I'm a little bit skeptical that the problem is due to fragmentation.

Quoted 35 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

(e) A creative mix of the above.

Quoted 8 lines
Quoted 4 lines

(f) Something else (probably simpler).

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,

Erhard, do you mind checking if Yu's diff below to use a single zpool fixes the problem completely? There is also an attached patch that does the same thing if this is easier to apply for you.

Quoted 15 lines

static struct zpool *zswap_find_zpool(struct zswap_entry *entry) {
- return entry->pool->zpools[hash_ptr(entry, ilog2(ZSWAP_NR_ZPOOLS))];
+ return entry->pool->zpools[0];
}

Quoted 8 lines

In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.

Quoted 5 lines

The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.

Regards,
Erhard

From c6c477dae9cb8bcdefaf1c1a0e8869efa8bfe3f9 Mon Sep 17 00:00:00 2001
From: Yosry Ahmed <[email protected]>
Date: Wed, 5 Jun 2024 23:56:15 +0000
Subject: [PATCH] mm: zswap: set ZSWAP_NR_ZPOOLS to 1

---
 mm/zswap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/zswap.c +1 -1
index a50e2986cd2fa..2bfa91518d405 100644
@@ -124,7 +124,7 @@ module_param_named(accept_threshold_percent, zswap_accept_thr_percent,
 		   uint, 0644);
 
 /* Number of zpools in zswap_pool (empirically determined for scalability) */
-#define ZSWAP_NR_ZPOOLS 32
+#define ZSWAP_NR_ZPOOLS 1
 
 /* Enable/disable memory pressure-based shrinker. */
 static bool zswap_shrinker_enabled = IS_ENABLED(
Signature · 1 line
2.45.1.467.gbab1589fc0-goog
#27 lore raw
Quoted 63 lines · On Wed, 5 Jun 2024 16:58:11 -0700 Yosry Ahmed <[email protected]> wrote:
Quoted 62 lines · Yu Zhao
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

Quoted 8 lines
Quoted 5 lines

Thanks! The patch does not fix the issue but it helps.

Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.

Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

Now I'm a little bit skeptical that the problem is due to fragmentation.

Quoted 39 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

(e) A creative mix of the above.

(f) Something else (probably simpler).

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

Quoted 7 lines
Quoted 3 lines

I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,

Erhard, do you mind checking if Yu's diff below to use a single zpool fixes the problem completely? There is also an attached patch that does the same thing if this is easier to apply for you.

No, setting ZSWAP_NR_ZPOOLS to 1 does not fix the problem unfortunately (that being the only patch applied on v6.10-rc2).

Trying to alter the lowmem and virtual mem limits next as Michael suggested.

Regards,
Erhard

#28 lore raw
Quoted 71 lines · Erhard Furtner
Quoted 67 lines · On Wed, 5 Jun 2024 16:58:11 -0700 Yosry Ahmed <[email protected]> wrote:
Quoted 65 lines · Yu Zhao
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Erhard Furtner
Quoted 2 lines · On Tue, 4 Jun 2024 20:03:27 -0700 Yosry Ahmed <[email protected]> wrote:

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

Quoted 8 lines
Quoted 5 lines

Thanks! The patch does not fix the issue but it helps.

Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.

Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

Now I'm a little bit skeptical that the problem is due to fragmentation.

Quoted 33 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

Quoted 6 lines

(e) A creative mix of the above.

(f) Something else (probably simpler).

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

I double checked that commit and didn't find anything wrong. If we are all in the mood of getting to the bottom, can we try using only 1 zpool while there are 2 available? I.e.,

Quoted 5 lines
Quoted 3 lines

Erhard, do you mind checking if Yu's diff below to use a single zpool fixes the problem completely? There is also an attached patch that does the same thing if this is easier to apply for you.

No, setting ZSWAP_NR_ZPOOLS to 1 does not fix the problem unfortunately (that being the only patch applied on v6.10-rc2).

This confirms Yu's theory that the zpools fragmentation is not the main reason for the problem. As Vlastimil said, the setup is already tight on memory and that commit may have just pushed it over the edge. Since setting ZSWAP_NR_ZPOOLS to 1 (which effectively reverts the commit) does not help in v6.10-rc2, then something else that came after the commit would have pushed it over the edge anyway.

Quoted 2 lines

Trying to alter the lowmem and virtual mem limits next as Michael suggested.

I saw that this worked. So it seems like we don't need to worry about the number of zpools, for now at least :)

Thanks for helping with the testing, and thanks to everyone else who helped on this thread.

Quoted 3 lines

Regards,
Erhard

#29 lore raw
In reply to Yosry Ahmed
Quoted 52 lines · Yosry Ahmed
Quoted 4 lines · Erhard Furtner

On Tue, 4 Jun 2024 20:03:27 -0700
Yosry Ahmed <[email protected]> wrote:

Quoted 8 lines
Quoted 2 lines

Could you check if the attached patch helps? It basically changes the number of zpools from 32 to min(32, nr_cpus).

Thanks! The patch does not fix the issue but it helps.

Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer.

Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure.

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

(e) A creative mix of the above.

Quoted 8 lines

(f) Something else (probably simpler).

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.

Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:

1. First, we can change the pool global lock to per-class lock, which is more fine-grained.
2. Actually, we only need to take per-zspage lock when malloc/free, only need to take class lock when its fullness changed.
3. If this is not enough, we can have fewer fullness groups, so the need to take class lock becomes fewer. (will need some test data)

More comments are welcome. Thanks!

#30 lore raw
Quoted 38 lines · Chengming Zhou
Quoted 38 lines

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

(e) A creative mix of the above.

Quoted 7 lines
Quoted 7 lines

(f) Something else (probably simpler).

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.

Sorry, I'm sure I'm not following this discussion closely enough, has the lock contention been demonstrated/proved somehow? lock-stats?

Quoted 5 lines

Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:

1. First, we can change the pool global lock to per-class lock, which is more fine-grained.

Commit c0547d0b6a4b6 "zsmalloc: consolidate zs_pool's migrate_lock and size_class's locks" [1] claimed no significant difference between class->lock and pool->lock.

[1] https://lkml.kernel.org/r/[email protected]

#31 lore raw
Quoted 41 lines · Sergey Senozhatsky
Quoted 40 lines · Chengming Zhou
Quoted 40 lines

Thanks for trying this out. This is interesting, so even two zpools is too much fragmentation for your use case.

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

(d) Make the number of zpools scale linearly with memory. This makes more sense than scaling with CPUs because increasing the number of zpools increases fragmentation, so it makes sense to limit it by the available memory. This is also more consistent with other magic numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT).

The problem is that unlike zswap trees, the zswap pool is not connected to the swapfile size, so we don't have an indication for how much memory will be in the zswap pool. We can scale the number of zpools with the entire memory on the machine during boot, but this seems like it would be difficult to figure out, and will not take into consideration memory hotplugging and the zswap global limit changing.

(e) A creative mix of the above.

(f) Something else (probably simpler).

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.

Sorry, I'm sure I'm not following this discussion closely enough, has the lock contention been demonstrated/proved somehow? lock-stats?

Yosry has some stats in his commit b8cf32dc6e8c ("mm: zswap: multiple zpools support"), and I have also seen some locking contention when using zram to test kernel building, since zram still has only one pool.

Quoted 10 lines
Quoted 6 lines

Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:

1. First, we can change the pool global lock to per-class lock, which is more fine-grained.

Commit c0547d0b6a4b6 "zsmalloc: consolidate zs_pool's migrate_lock and size_class's locks" [1] claimed no significant difference between class->lock and pool->lock.

Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.

Thanks.

Quoted 2 lines

[1] https://lkml.kernel.org/r/[email protected]

#32 lore raw
Quoted 6 lines · Chengming Zhou
Quoted 6 lines
Quoted 5 lines

Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:

1. First, we can change the pool global lock to per-class lock, which is more fine-grained.

Quoted 7 lines
Quoted 3 lines

Commit c0547d0b6a4b6 "zsmalloc: consolidate zs_pool's migrate_lock and size_class's locks" [1] claimed no significant difference between class->lock and pool->lock.

Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is

Well, the point that commit made was that Nhat (and Johannes?) were unable to detect any impact of pool->lock on a variety of cases. So we went on with code simplification.

Quoted 2 lines

obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.

Thanks, we'll need data on this. I'm happy to take the patch, but jumping back and forth between class->lock and pool->lock merely "for obvious reasons" is not what I'm extremely excited about.

#33 lore raw
Quoted 11 lines · Sergey Senozhatsky
Quoted 10 lines · Chengming Zhou
Quoted 9 lines
Quoted 5 lines

Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:

1. First, we can change the pool global lock to per-class lock, which is more fine-grained.

Commit c0547d0b6a4b6 "zsmalloc: consolidate zs_pool's migrate_lock and size_class's locks" [1] claimed no significant difference between class->lock and pool->lock.

Quoted 7 lines
Quoted 3 lines

Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is

Well, the point that commit made was that Nhat (and Johannes?) were unable to detect any impact of pool->lock on a variety of cases. So we went on with code simplification.

Right, the code is simpler.

Quoted 7 lines
Quoted 3 lines

obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.

Thanks, we'll need data on this. I'm happy to take the patch, but jumping back and forth between class->lock and pool->lock merely "for obvious reasons" is not what I'm extremely excited about.

Yeah, agree, we need test data.

maintainer
#34 lore raw
In reply to Sergey Senozhatsky
Quoted 12 lines · Sergey Senozhatsky
Quoted 10 lines · Chengming Zhou
Quoted 9 lines
Quoted 5 lines

Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet:

1. First, we can change the pool global lock to per-class lock, which is more fine-grained.

Commit c0547d0b6a4b6 "zsmalloc: consolidate zs_pool's migrate_lock and size_class's locks" [1] claimed no significant difference between class->lock and pool->lock.

Quoted 7 lines
Quoted 3 lines

Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is

Well, the point that commit made was that Nhat (and Johannes?) were unable to detect any impact of pool->lock on a variety of cases. So we went on with code simplification.

Yeah, we benchmarked it before zsmalloc writeback was introduced (the patch to remove class lock was a prep patch of the series). We weren't able to detect any regression at the time with just using a global pool lock.

Quoted 7 lines
Quoted 3 lines

obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later.

Thanks, we'll need data on this. I'm happy to take the patch, but jumping back and forth between class->lock and pool->lock merely "for obvious reasons" is not what I'm extremely excited about.

FWIW, I do think it'd be nice if we can make the locking more granular
- the pool lock now is essentially a global lock, and we're just getting around that by replicating the (z)pools themselves.

Personally, I'm not super convinced about class locks. We're essentially relying on the post-compression size of the data to load-balance the queries - I can imagine a scenario where a workload has a concentrated distribution of post-compression data (i.e its pages are compressed to similar-ish sizes), and we're once again contending for a (few) lock(s) again.

That said, I'll let the data tell the story :) We don't need a perfect solution, just a good enough solution for now.

#35 lore raw
Quoted 9 lines · Nhat Pham

Personally, I'm not super convinced about class locks. We're essentially relying on the post-compression size of the data to load-balance the queries - I can imagine a scenario where a workload has a concentrated distribution of post-compression data (i.e its pages are compressed to similar-ish sizes), and we're once again contending for a (few) lock(s) again.

That said, I'll let the data tell the story :) We don't need a perfect solution, just a good enough solution for now.

Speaking of size class locks:

One thing to mention is that zsmalloc merges size classes, we never have documented/claimed 256 size classe, the actual number is always much much lower. Each such "cluster" (merged size classes) holds a range of objects' sizes (e.g. 3504-3584 bytes). The wider the cluster's size range the more likely the (size class) lock contention is.

Setting CONFIG_ZSMALLOC_CHAIN_SIZE to 10 or higher makes zsmalloc pool to be configured with more size class clusters (which means that clusters hold narrower size intervals).

#36 lore raw
In reply to Yosry Ahmed
Quoted 4 lines · Yosry Ahmed

On Wed, Jun 5, 2024 at 4:04 PM Erhard Furtner <[email protected]> wrote:

I am personally leaning toward (c), but I want to hear the opinions of other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else?

Besides the zpool commit which might have just pushed the machine over the edge, but it was probably close to it already. I've noticed a more general problem that there are GFP_KERNEL allocations failing from kswapd. Those could probably use be __GFP_NOMEMALLOC (or scoped variant, is there one?) since it's the case of "allocating memory to free memory". Or use mempools if the progress (success will lead to freeing memory) is really guaranteed.

Another interesting data point could be to see if traditional reclaim behaves any better on this machine than MGLRU. I saw in the config:

CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y

So disabling at least the second one would revert to the traditional reclaim and we could see if it handles such a constrained system better or not.

Quoted 9 lines

In the long-term, I think we may want to address the lock contention in zsmalloc itself instead of zswap spawning multiple zpools.

Quoted 5 lines

The patch did not apply cleanly on v6.9.3 so I applied it on v6.10-rc2. dmesg of the current v6.10-rc2 run attached.

Regards,
Erhard

#37 lore raw
Quoted 7 lines · On Thu, 6 Jun 2024 09:24:56 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…

Besides the zpool commit which might have just pushed the machine over the edge, but it was probably close to it already. I've noticed a more general problem that there are GFP_KERNEL allocations failing from kswapd. Those could probably use be __GFP_NOMEMALLOC (or scoped variant, is there one?) since it's the case of "allocating memory to free memory". Or use mempools if the progress (success will lead to freeing memory) is really guaranteed.

Quoted 8 lines

Another interesting data point could be to see if traditional reclaim behaves any better on this machine than MGLRU. I saw in the config:

CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y

So disabling at least the second one would revert to the traditional reclaim and we could see if it handles such a constrained system better or not.

I set RANDOM_KMALLOC_CACHES=n and LRU_GEN_ENABLED=n but still hit the issue.

dmesg looks a bit different (unpatched v6.10-rc2).

Regards,
Erhard

#38 lore raw
Quoted 16 lines · Erhard Furtner
Quoted 13 lines · On Thu, 6 Jun 2024 09:24:56 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…

Besides the zpool commit which might have just pushed the machine over the edge, but it was probably close to it already. I've noticed a more general problem that there are GFP_KERNEL allocations failing from kswapd. Those could probably use be __GFP_NOMEMALLOC (or scoped variant, is there one?) since it's the case of "allocating memory to free memory". Or use mempools if the progress (success will lead to freeing memory) is really guaranteed.

Another interesting data point could be to see if traditional reclaim behaves any better on this machine than MGLRU. I saw in the config:

CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y

Quoted 6 lines
Quoted 2 lines

So disabling at least the second one would revert to the traditional reclaim and we could see if it handles such a constrained system better or not.

I set RANDOM_KMALLOC_CACHES=n and LRU_GEN_ENABLED=n but still hit the issue.

dmesg looks a bit different (unpatched v6.10-rc2).

What caught my eye, but it's also in some of the previous dmesg with MGRLU, is that in one case there's:

DMA free:0kB

That means many allocations went through that are allowed to just ignore all reserves, and depleted everything. That would mean __GFP_MEMALLOC or PF_MEMALLOC, which I suggested earlier for the GFP_KERNEL failure, is being used somewhere, but not leading to the expected memory freeing.

Quoted 2 lines

Regards,
Erhard

#39 lore raw
In reply to Yosry Ahmed

2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:

Quoted 13 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

Quoted 7 lines

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?

Signature · 1 line
<[email protected]>
#40 lore raw
Quoted 24 lines · Takero Funaki

2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:

Quoted 13 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

Quoted 7 lines

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

Quoted 8 lines

I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?

We use 32 zpools on machines with 100s of CPUs. Two zpools per CPU is an overkill imo.

I have further comments that I will leave on the patch, but I mainly think this should be driven by real data, not theoretical possibility of lock contention.

Quoted 4 lines

--

<[email protected]>

#41 lore raw
Quoted 35 lines · Yosry Ahmed
Quoted 24 lines · Takero Funaki

2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:

Quoted 13 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

Quoted 7 lines

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

Quoted 8 lines

I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?

Quoted 2 lines

We use 32 zpools on machines with 100s of CPUs. Two zpools per CPU is an overkill imo.

Not to choose a camp; just a friendly note on why I strongly disagree with the N zpools per CPU approach:
1. It is fundamentally flawed to assume the system is linear;
2. Nonlinear systems usually have diminishing returns.

For Google data centers, using nr_cpus as the scaling factor had long passed the acceptable ROI threshold. Per-CPU data, especially when compounded per memcg or even per process, is probably the number-one overhead in terms of DRAM efficiency.

Quoted 3 lines

I have further comments that I will leave on the patch, but I mainly think this should be driven by real data, not theoretical possibility of lock contention.

#42 lore raw
Quoted 45 lines · Yu Zhao
Quoted 35 lines · Yosry Ahmed
Quoted 24 lines · Takero Funaki

2024年6月6日(木) 8:42 Yosry Ahmed <[email protected]>:

Quoted 13 lines

I think there are multiple ways to go forward here:
(a) Make the number of zpools a config option, leave the default as 32, but allow special use cases to set it to 1 or similar. This is probably not preferable because it is not clear to users how to set it, but the idea is that no one will have to set it except special use cases such as Erhard's (who will want to set it to 1 in this case).

(b) Make the number of zpools scale linearly with the number of CPUs. Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this approach is that with a large number of CPUs, too many zpools will start having diminishing returns. Fragmentation will keep increasing, while the scalability/concurrency gains will diminish.

Quoted 7 lines

(c) Make the number of zpools scale logarithmically with the number of CPUs. Maybe something like 4log2(nr_cpus). This will keep the number of zpools from increasing too much and close to the status quo. The problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) will actually give a nr_zpools > nr_cpus. So we will need to come up with a more fancy magic equation (e.g. 4log2(nr_cpus/4)).

Quoted 8 lines

I just posted a patch to limit the number of zpools, with some theoretical background explained in the code comments. I believe that 2 * CPU linearly is sufficient to reduce contention, but the scale can be reduced further. All CPUs are trying to allocate/free zswap is unlikely to happen.
How many concurrent accesses were the original 32 zpools supposed to
handle? I think it was for 16 cpu or more. or nr_cpus/4 would be enough?

Quoted 2 lines

We use 32 zpools on machines with 100s of CPUs. Two zpools per CPU is an overkill imo.

Not to choose a camp; just a friendly note on why I strongly disagree with the N zpools per CPU approach:
1. It is fundamentally flawed to assume the system is linear;
2. Nonlinear systems usually have diminishing returns.

Quoted 4 lines

For Google data centers, using nr_cpus as the scaling factor had long passed the acceptable ROI threshold. Per-CPU data, especially when compounded per memcg or even per process, is probably the number-one overhead in terms of DRAM efficiency.

100% agreed. If you look at option (b) above, I specifically called out that scaling the number of zpools linearly with the number with CPUs have diminishing returns :)

#43 lore raw
In reply to Yu Zhao
Quoted 10 lines · On Tue, 4 Jun 2024 11:18:25 -0600 Yu Zhao <[email protected]> wrote:

Hi Erhard,

If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" on kernels before and after the bad commit? It'd be great if you could run the grep command right before the OOM kills.

The overall internal fragmentation of multiple zsmalloc pools might be higher than a single one. I suspect this might be the cause.

Thank you.

I used watch -n1 'grep nr_zspages /proc/vmstat' to get the readings and repeated this 3 times to check whether the reported values differ much.

Bad commit was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 "mm: zswap: multiple zpools support", next commit in git log after the bad one would be 42c06a0e8ebe95b81e5fb41c6556ff22d9255b0c "mm: kill frontswap".

With this kernel I got 2440, 2337, 3245 as nr_zspages and the kswapd0: page allocation error.

Commit in git log before the bad one would be bfaa4a0ce1bbc1b2b67de7e4c2e1679495f7b905 "scsi: gvp11: Remove unused gvp11_setup() function".

With this kernel I got 25537, 11321, 16087 as nr_zspages and the OOM reaper.

Tomorrow I could also check the .config changes Vlastimil suggested (CONFIG_SLAB_MERGE_DEFAULT=y, no CONFIG_RANDOM_KMALLOC_CACHES) and report back if that's of interest.

Regards,
Erhard

#44 lore raw
In reply to Yosry Ahmed
Quoted 45 lines · Yosry Ahmed
Quoted 30 lines · Erhard Furtner
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]
Quoted 8 lines
Quoted 8 lines

The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.

Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)

Ok, the bisect I did actually revealed something meaningful:

Quoted 7 lines
 # git bisect good
b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit
commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17
Author: Yosry Ahmed <[email protected]>
Date:   Tue Jun 20 19:46:44 2023 +0000

mm: zswap: multiple zpools support

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Quoted 5 lines

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

As for reducing slab fragmentation/footprint, I would also recommend these changes to .config:

CONFIG_SLAB_MERGE_DEFAULT=y - this will unify the separate zpool caches as well (but the patch still makes sense), but also many others CONFIG_RANDOM_KMALLOC_CACHES=n - no 16 separate copies of kmalloc caches

although the slabinfo output doesn't seem to show CONFIG_RANDOM_KMALLOC_CACHES in action, weirdly. It was enabled in the config attached to the first mail.

Both these changes mean giving up some mitigation against potentai lvulnerabilities. But it's not perfect anyway and the memory seems really tight here.

#45 lore raw
Quoted 53 lines · Vlastimil Babka (SUSE)
Quoted 45 lines · Yosry Ahmed
Quoted 30 lines · Erhard Furtner
Quoted 15 lines · On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner <[email protected]> wrote:
Quoted 12 lines · On Sat, 1 Jun 2024 00:01:48 -0600 Yu Zhao <[email protected]> wrote:

The OOM kills on both kernel versions seem to be reasonable to me.

Your system has 2GB memory and it uses zswap with zsmalloc (which is good since it can allocate from the highmem zone) and zstd/lzo (which doesn't matter much). Somehow -- I couldn't figure out why -- it splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]
Quoted 8 lines
Quoted 8 lines

The kernel can't allocate from the highmem zone -- only userspace and zsmalloc can. OOM kills were due to the low memory conditions in the DMA zone where the kernel itself failed to allocate from.

Do you know a kernel version that doesn't have OOM kills while running the same workload? If so, could you send that .config to me? If not, could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out of ideas at the moment.)

Ok, the bisect I did actually revealed something meaningful:

Quoted 7 lines
 # git bisect good
b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit
commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17
Author: Yosry Ahmed <[email protected]>
Date:   Tue Jun 20 19:46:44 2023 +0000

mm: zswap: multiple zpools support

Thanks for bisecting. Taking a look at the thread, it seems like you have a very limited area of memory to allocate kernel memory from. One possible reason why that commit can cause an issue is because we will have multiple instances of the zsmalloc slab caches 'zspage' and 'zs_handle', which may contribute to fragmentation in slab memory.

Quoted 5 lines

Do you have /proc/slabinfo from a good and a bad run by any chance?

Also, could you check if the attached patch helps? It makes sure that even when we use multiple zsmalloc zpools, we will use a single slab cache of each type.

Quoted 6 lines

As for reducing slab fragmentation/footprint, I would also recommend these changes to .config:

CONFIG_SLAB_MERGE_DEFAULT=y - this will unify the separate zpool caches as well (but the patch still makes sense), but also many others CONFIG_RANDOM_KMALLOC_CACHES=n - no 16 separate copies of kmalloc caches

Yeah, I did send that patch separately, but I think the problem here is probably fragmentation in the zsmalloc pools themselves, not the slab caches used by them.

Quoted 8 lines

although the slabinfo output doesn't seem to show CONFIG_RANDOM_KMALLOC_CACHES in action, weirdly. It was enabled in the config attached to the first mail.

Both these changes mean giving up some mitigation against potentai lvulnerabilities. But it's not perfect anyway and the memory seems really tight here.

I think we may be able to fix the problem here if we address the zsmalloc fragmentation. In regards to slab caches, the patch proposed above should avoid the replication without enabling slab cache merging in general.

Thanks for chiming in!