Memory leak in 6.18
Hi guys,
Been using 6.18 kernel and i have noticed that there is some memory leak. Currently mapple_node takes 86GB when server does not do much. I do not see that issue on 6.17 kernel at all.
Total 1000 GB memory
ASRockRack GENOA2D24G-2L
2x AMD EPYC 9654 96-Core Processor
Running Proxmox 9
Active / Total Objects (% used) : 472110239 / 472257124 (100.0%) Active / Total Slabs (% used) : 7385489 / 7385489 (100.0%) Active / Total Caches (% used) : 164 / 231 (71.0%) Active / Total Size (% used) : 95486861.00K / 95528053.72K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.20K / 8.06K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
345907216 345897683 99% 0.25K 5404801 64 86476816K maple_node 120873408 120841337 99% 0.06K 1888647 64 7554588K dmaengine-unmap-2 224256 223324 99% 0.01K 438 512 1752K kmalloc-8 224040 224040 100% 0.13K 3734 60 29872K kernfs_node_cache 196608 196608 100% 0.01K 384 512 1536K kmalloc-cg-8 196160 166455 84% 0.50K 3065 64 98080K kmalloc-512
[image: slabtop.png]
[image: htop.png]
--
tel. 790 202 300
*Tytus Rogalewski*
Dolina Krzemowa 6A
83-010 Jagatowo
NIP: 9570976234
A message that is not in the archive (sent off-list or not yet received): aRGAUjkKe_H1xc7H@hyeyoo
Quoted 6 lines · Harry Yoo
Quoted 5 lines · Tytus Rogalewski
Hi guys,
Been using 6.18 kernel and i have noticed that there is some memory leak. Currently
mapple_nodetakes 86GB when server does not do much. I do not see that issue on 6.17 kernel at all.
Cc'ing [email protected] properly as I modified the address by mistake.
Quoted 41 lines
Hi Tytus, thanks for the report!
Could you please boot your machine with kernel boot parameter slab_debug=U [1] and run
$ cat /sys/kernel/debug/slab/maple_node/alloc_tracesand
$ cat /sys/kernel/debug/slab/maple_node/free_traces?
Quoted 11 lines
Total 1000 GB memory
ASRockRack GENOA2D24G-2L
2x AMD EPYC 9654 96-Core Processor
Running Proxmox 9Active / Total Objects (% used) : 472110239 / 472257124 (100.0%) Active / Total Slabs (% used) : 7385489 / 7385489 (100.0%) Active / Total Caches (% used) : 164 / 231 (71.0%) Active / Total Size (% used) : 95486861.00K / 95528053.72K (100.0%)Minimum / Average / Maximum Object : 0.01K / 0.20K / 8.06K
Quoted 7 lines
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
345907216 345897683 99% 0.25K 5404801 64 86476816K maple_node 120873408 120841337 99% 0.06K 1888647 64 7554588K dmaengine-unmap-2 224256 223324 99% 0.01K 438 512 1752K kmalloc-8 224040 224040 100% 0.13K 3734 60 29872K kernfs_node_cache 196608 196608 100% 0.01K 384 512 1536K kmalloc-cg-8 196160 166455 84% 0.50K 3065 64 98080K kmalloc-512Not sure if this is because of sheaves or maple tree changes. Let's see what's in the alloc & free traces.
Or it would be great if you could build the kernel and perform git bisection [2] and give us what is the first bad commit.
Thanks!
[1] https://docs.kernel.org/next/admin-guide/mm/slab.html
[2] https://git-scm.com/docs/git-bisect
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 20 lines · Harry Yoo
Quoted 6 lines · Harry Yoo
Quoted 5 lines · Tytus Rogalewski
Hi guys,
Been using 6.18 kernel and i have noticed that there is some memory leak. Currently
mapple_nodetakes 86GB when server does not do much. I do not see that issue on 6.17 kernel at all.Cc'ing [email protected] properly as I modified the address by mistake.
Quoted 10 lines
Hi Tytus, thanks for the report!
Could you please boot your machine with kernel boot parameter slab_debug=U [1] and run
$ cat /sys/kernel/debug/slab/maple_node/alloc_tracesand
$ cat /sys/kernel/debug/slab/maple_node/free_traces
Agreed. What could help in addition to above is also enabling CONFIG_SLUB_STATS and also providing the output of:
grep . /sys/kernel/slab/maple_node/*
Thanks.
Quoted 31 lines
Quoted 30 lines
?
Quoted 11 lines
Total 1000 GB memory
ASRockRack GENOA2D24G-2L
2x AMD EPYC 9654 96-Core Processor
Running Proxmox 9Active / Total Objects (% used) : 472110239 / 472257124 (100.0%) Active / Total Slabs (% used) : 7385489 / 7385489 (100.0%) Active / Total Caches (% used) : 164 / 231 (71.0%) Active / Total Size (% used) : 95486861.00K / 95528053.72K (100.0%)Minimum / Average / Maximum Object : 0.01K / 0.20K / 8.06K
Quoted 7 lines
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
345907216 345897683 99% 0.25K 5404801 64 86476816K maple_node 120873408 120841337 99% 0.06K 1888647 64 7554588K dmaengine-unmap-2 224256 223324 99% 0.01K 438 512 1752K kmalloc-8 224040 224040 100% 0.13K 3734 60 29872K kernfs_node_cache 196608 196608 100% 0.01K 384 512 1536K kmalloc-cg-8 196160 166455 84% 0.50K 3065 64 98080K kmalloc-512Not sure if this is because of sheaves or maple tree changes. Let's see what's in the alloc & free traces.
Or it would be great if you could build the kernel and perform git bisection [2] and give us what is the first bad commit.
Thanks!
[1] https://docs.kernel.org/next/admin-guide/mm/slab.html
[2] https://git-scm.com/docs/git-bisect
Quoted 20 lines · Vlastimil Babka
Quoted 19 lines · Harry Yoo
Quoted 6 lines · Harry Yoo
Quoted 5 lines · Tytus Rogalewski
Hi guys,
Been using 6.18 kernel and i have noticed that there is some memory leak. Currently
mapple_nodetakes 86GB when server does not do much. I do not see that issue on 6.17 kernel at all.Cc'ing [email protected] properly as I modified the address by mistake.
Quoted 9 lines
Hi Tytus, thanks for the report!
Could you please boot your machine with kernel boot parameter slab_debug=U [1] and run
$ cat /sys/kernel/debug/slab/maple_node/alloc_tracesand
Quoted 8 lines
Quoted 1 line
Quoted 1 line
$ cat /sys/kernel/debug/slab/maple_node/free_tracesAgreed. What could help in addition to above is also enabling
CONFIG_SLUB_STATSand also providing the output of:grep .
/sys/kernel/slab/maple_node/*Thanks.
Also the output of this please:
numactl -H
Quoted 32 lines
Quoted 31 lines
Quoted 30 lines
?
Quoted 11 lines
Total 1000 GB memory
ASRockRack GENOA2D24G-2L
2x AMD EPYC 9654 96-Core Processor
Running Proxmox 9Active / Total Objects (% used) : 472110239 / 472257124 (100.0%) Active / Total Slabs (% used) : 7385489 / 7385489 (100.0%) Active / Total Caches (% used) : 164 / 231 (71.0%) Active / Total Size (% used) : 95486861.00K / 95528053.72K (100.0%)Minimum / Average / Maximum Object : 0.01K / 0.20K / 8.06K
Quoted 7 lines
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
345907216 345897683 99% 0.25K 5404801 64 86476816K maple_node 120873408 120841337 99% 0.06K 1888647 64 7554588K dmaengine-unmap-2 224256 223324 99% 0.01K 438 512 1752K kmalloc-8 224040 224040 100% 0.13K 3734 60 29872K kernfs_node_cache 196608 196608 100% 0.01K 384 512 1536K kmalloc-cg-8 196160 166455 84% 0.50K 3065 64 98080K kmalloc-512Not sure if this is because of sheaves or maple tree changes. Let's see what's in the alloc & free traces.
Or it would be great if you could build the kernel and perform git bisection [2] and give us what is the first bad commit.
Thanks!
[1] https://docs.kernel.org/next/admin-guide/mm/slab.html
[2] https://git-scm.com/docs/git-bisect