Skip to contentSkip to conversation

Possible memory leak in 6.17.7

DISCUSSION Mal Haak 3 participants 22 messages Nov 10, 2025 – Dec 17, 2025

Subsystems (3)
lore
Share & tools
lore

Mal Haak

author
#0 lore raw

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.

Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        88.21%     36.25%     51.96% 
userspace memory              9.49%      0.15%      9.34% 
free memory                   2.30%      2.30%      0.00% 

free -h output:

       total  used   free   shared  buff/cache available 
Mem:   31Gi   3.6Gi  500Mi  4.0Mi   11Gi      27Gi 
Swap:  4.0Gi  179Mi  3.8Gi

Reverting to the previous LTS fixes the issue

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 

kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%

I am unsure of the best way to track down the memory usage. I have tried stopping the workload and unmounting the cephfs filesystem. As well as removing the ceph and network related kernel modules.

I assume some kind of tracing would be a way to find the culprit, however I am unsure of the best way to do that.

I can do a git bisect and am in the process of getting a test reproducer made for that.

But if there is an easier way to do it I would happily do that.

Just to note, slabtop looks normal and doesn't show the memory usage.

Thanks in advance

Mal

author
#1 lore raw
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.

Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        88.21%     36.25%     51.96% 
userspace memory              9.49%      0.15%      9.34% 
free memory                   2.30%      2.30%      0.00% 

free -h output:

       total  used   free   shared  buff/cache available 
Mem:   31Gi   3.6Gi  500Mi  4.0Mi   11Gi      27Gi 
Swap:  4.0Gi  179Mi  3.8Gi

Reverting to the previous LTS fixes the issue

Quoted 8 lines

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 

kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%

I have more information. The leaking of kernel memory only starts once there is a lot of data in buffers/cache. And only once it's been in that state for several hours.

Currently in my search for a reproducer I have found that downloading then seeding of multiple torrents of linux distribution ISO's will replicate the issue. But it only begins leaking at around the 6-9 hour mark.

It does not appear to be dependant on cephfs; but due to it's use of sockets I believe this is making the situation worse.

I cannot replicate it at all with the LTS kernel release but it does look like the current RC releases do have this issue.

I was looking at doing a kernel build with CONFIG_DEBUG_KMEMLEAK enabled and will if it's thought this would find the issue. However as the memory usage is still somewhat tracked and obviously marked as reclaimable it feels more like something in the reclaim logic is getting broken.

I do wonder if due to it only happening after ram is mostly consumed by cache, and even then only if it has been that way for hours, if the issue is memory fragmentation related.

Regardless, some advice on how to narrow this down faster than a git bisect as 9hrs to even confirm replication of the issue makes git bisect painfully slow.

Thanks in advance

Mal Haak

author
#2 lore raw

I have a reproducer. It's slow but it works.

I kept rsync running for 2 days by moving 5TB of files.

smem -wp

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        98.81%      1.69%     97.13% 
userspace memory              0.08%      0.05%      0.03% 
free memory                   1.11%      1.11%      0.00% 

[root@kerneltest ~]# uname -a
Linux kerneltest 6.18.0-1-mainline #1 SMP PREEMPT_DYNAMIC Tue, 11 Nov 2025 00:02:22 +0000 x86_64 GNU/Linux

The issue is in 6.18.

Quoted 74 lines · On Thu, 20 Nov 2025 12:23:51 +1000 Mal Haak <[email protected]> wrote:
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.

Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        88.21%     36.25%     51.96% 
userspace memory              9.49%      0.15%      9.34% 
free memory                   2.30%      2.30%      0.00% 

free -h output:

       total  used   free   shared  buff/cache available 
Mem:   31Gi   3.6Gi  500Mi  4.0Mi   11Gi      27Gi 
Swap:  4.0Gi  179Mi  3.8Gi

Reverting to the previous LTS fixes the issue

Quoted 8 lines

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 

kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%

I have more information. The leaking of kernel memory only starts once there is a lot of data in buffers/cache. And only once it's been in that state for several hours.

Currently in my search for a reproducer I have found that downloading then seeding of multiple torrents of linux distribution ISO's will replicate the issue. But it only begins leaking at around the 6-9 hour mark.

It does not appear to be dependant on cephfs; but due to it's use of sockets I believe this is making the situation worse.

I cannot replicate it at all with the LTS kernel release but it does look like the current RC releases do have this issue.

I was looking at doing a kernel build with CONFIG_DEBUG_KMEMLEAK enabled and will if it's thought this would find the issue. However as the memory usage is still somewhat tracked and obviously marked as reclaimable it feels more like something in the reclaim logic is getting broken.

I do wonder if due to it only happening after ram is mostly consumed by cache, and even then only if it has been that way for hours, if the issue is memory fragmentation related.

Regardless, some advice on how to narrow this down faster than a git bisect as 9hrs to even confirm replication of the issue makes git bisect painfully slow.

Thanks in advance

Mal Haak

author
#3 lore raw

So four and a half hours is how long it takes to get it to reproduce with two rsync's running in parallel.

Looks like git bisect is the only way forward as I built a kernel with kmemleak enabled and it cannot locate the leak.

Probably because the pages are marked as able to be freed.

Any other suggestions on how to trace down the leak would be appreciated.

Thanks

Mal

Quoted 96 lines · On Sat, 6 Dec 2025 08:23:36 +1000 Mal Haak <[email protected]> wrote:

I have a reproducer. It's slow but it works.

I kept rsync running for 2 days by moving 5TB of files.

smem -wp

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        98.81%      1.69%     97.13% 
userspace memory              0.08%      0.05%      0.03% 
free memory                   1.11%      1.11%      0.00% 

[root@kerneltest ~]# uname -a
Linux kerneltest 6.18.0-1-mainline #1 SMP PREEMPT_DYNAMIC Tue, 11 Nov 2025 00:02:22 +0000 x86_64 GNU/Linux

The issue is in 6.18.

Quoted 74 lines · On Thu, 20 Nov 2025 12:23:51 +1000 Mal Haak <[email protected]> wrote:
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.

Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        88.21%     36.25%     51.96% 
userspace memory              9.49%      0.15%      9.34% 
free memory                   2.30%      2.30%      0.00% 

free -h output:

       total  used   free   shared  buff/cache available 
Mem:   31Gi   3.6Gi  500Mi  4.0Mi   11Gi      27Gi 
Swap:  4.0Gi  179Mi  3.8Gi

Reverting to the previous LTS fixes the issue

Quoted 8 lines

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 

kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%

I have more information. The leaking of kernel memory only starts once there is a lot of data in buffers/cache. And only once it's been in that state for several hours.

Currently in my search for a reproducer I have found that downloading then seeding of multiple torrents of linux distribution ISO's will replicate the issue. But it only begins leaking at around the 6-9 hour mark.

It does not appear to be dependant on cephfs; but due to it's use of sockets I believe this is making the situation worse.

I cannot replicate it at all with the LTS kernel release but it does look like the current RC releases do have this issue.

I was looking at doing a kernel build with CONFIG_DEBUG_KMEMLEAK enabled and will if it's thought this would find the issue. However as the memory usage is still somewhat tracked and obviously marked as reclaimable it feels more like something in the reclaim logic is getting broken.

I do wonder if due to it only happening after ram is mostly consumed by cache, and even then only if it has been that way for hours, if the issue is memory fragmentation related.

Regardless, some advice on how to narrow this down faster than a git bisect as 9hrs to even confirm replication of the issue makes git bisect painfully slow.

Thanks in advance

Mal Haak

#4 lore raw
In reply to Mal Haak
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.

Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        88.21%     36.25%     51.96% 
userspace memory              9.49%      0.15%      9.34% 
free memory                   2.30%      2.30%      0.00% 

free -h output:

       total  used   free   shared  buff/cache available 
Mem:   31Gi   3.6Gi  500Mi  4.0Mi   11Gi      27Gi 
Swap:  4.0Gi  179Mi  3.8Gi

Reverting to the previous LTS fixes the issue

Quoted 8 lines

smem -wp output:

Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 

kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%

I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html

You would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=y
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y

And check /proc/allocinfo for the suspicious allocations which take more memory than expected.

(I once caught a nvidia driver memory leak.)

FYI
David

author
#5 lore raw
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html

You would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=y
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y

Quoted 9 lines

And check /proc/allocinfo for the suspicious allocations which take more memory than expected.

(I once caught a nvidia driver memory leak.)

FYI
David

Thank you for your suggestion. I have some results.

Ran the rsync workload for about 9 hours. It started to look like it was happening.

# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        80.46%     65.80%     14.66% 
userspace memory              0.35%      0.16%      0.19% 
free memory                  19.19%     19.19%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
         22M     5609 mm/memory.c:1190 func:folio_prealloc 
         23M     1932 fs/xfs/xfs_buf.c:226 [xfs]
func:xfs_buf_alloc_backing_mem 
         24M    24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc 
         27M     6693 mm/memory.c:1192 func:folio_prealloc 
         58M    14784 mm/page_ext.c:271 func:alloc_page_ext 
        258M      129 mm/khugepaged.c:1069 func:alloc_charge_folio 
        430M   770788 lib/xarray.c:378 func:xas_alloc 
        545M    36444 mm/slub.c:3059 func:alloc_slab_page 
        9.8G  2563617 mm/readahead.c:189 func:ractl_alloc_folio 
         20G  5164004 mm/filemap.c:2012 func:__filemap_get_folio 

So I stopped the workload and dropped caches to confirm.

# echo 3 > /proc/sys/vm/drop_caches
# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        33.45%      0.09%     33.36% 
userspace memory              0.36%      0.16%      0.19% 
free memory                  66.20%     66.20%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
         12M     2987 mm/execmem.c:41 func:execmem_vmalloc 
         12M        3 kernel/dma/pool.c:96 func:atomic_pool_expand 
         13M      751 mm/slub.c:3061 func:alloc_slab_page 
         16M        8 mm/khugepaged.c:1069 func:alloc_charge_folio 
         18M     4355 mm/memory.c:1190 func:folio_prealloc 
         24M     6119 mm/memory.c:1192 func:folio_prealloc 
         58M    14784 mm/page_ext.c:271 func:alloc_page_ext 
         61M    15448 mm/readahead.c:189 func:ractl_alloc_folio 
         79M     6726 mm/slub.c:3059 func:alloc_slab_page 
         11G  2674488 mm/filemap.c:2012 func:__filemap_get_folio

So if I'm reading this correctly something is causing folios collect and not be able to be freed?

Also it's clear that some of the folio's are counting as cache and some aren't.

Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.

I'll do it the packages way until I get closer, then solve the build issues.

Thanks,
Mal

#6 lore raw
Quoted 76 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html

You would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=y
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y

Quoted 9 lines

And check /proc/allocinfo for the suspicious allocations which take more memory than expected.

(I once caught a nvidia driver memory leak.)

FYI
David

Thank you for your suggestion. I have some results.

Ran the rsync workload for about 9 hours. It started to look like it was happening.

# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        80.46%     65.80%     14.66% 
userspace memory              0.35%      0.16%      0.19% 
free memory                  19.19%     19.19%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        22M     5609 mm/memory.c:1190 func:folio_prealloc 
        23M     1932 fs/xfs/xfs_buf.c:226 [xfs]
func:xfs_buf_alloc_backing_mem 
        24M    24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc 
        27M     6693 mm/memory.c:1192 func:folio_prealloc 
        58M    14784 mm/page_ext.c:271 func:alloc_page_ext 
       258M      129 mm/khugepaged.c:1069 func:alloc_charge_folio 
       430M   770788 lib/xarray.c:378 func:xas_alloc 
       545M    36444 mm/slub.c:3059 func:alloc_slab_page 
       9.8G  2563617 mm/readahead.c:189 func:ractl_alloc_folio 
        20G  5164004 mm/filemap.c:2012 func:__filemap_get_folio 

So I stopped the workload and dropped caches to confirm.

# echo 3 > /proc/sys/vm/drop_caches
# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        33.45%      0.09%     33.36% 
userspace memory              0.36%      0.16%      0.19% 
free memory                  66.20%     66.20%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        12M     2987 mm/execmem.c:41 func:execmem_vmalloc 
        12M        3 kernel/dma/pool.c:96 func:atomic_pool_expand 
        13M      751 mm/slub.c:3061 func:alloc_slab_page 
        16M        8 mm/khugepaged.c:1069 func:alloc_charge_folio 
        18M     4355 mm/memory.c:1190 func:folio_prealloc 
        24M     6119 mm/memory.c:1192 func:folio_prealloc 
        58M    14784 mm/page_ext.c:271 func:alloc_page_ext 
        61M    15448 mm/readahead.c:189 func:ractl_alloc_folio 
        79M     6726 mm/slub.c:3059 func:alloc_slab_page 
        11G  2674488 mm/filemap.c:2012 func:__filemap_get_folio
Quoted 2 lines

So if I'm reading this correctly something is causing folios collect and not be able to be freed?

CC cephfs, maybe someone could have an easy reading out of those folio usage

Quoted 16 lines

Also it's clear that some of the folio's are counting as cache and some aren't.

Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.

I'll do it the packages way until I get closer, then solve the build issues.

Thanks,
Mal

author
#7 lore raw
Quoted 77 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 76 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html

You would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=y
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y

Quoted 9 lines

And check /proc/allocinfo for the suspicious allocations which take more memory than expected.

(I once caught a nvidia driver memory leak.)

FYI
David

Thank you for your suggestion. I have some results.

Ran the rsync workload for about 9 hours. It started to look like it was happening.

# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        80.46%     65.80%     14.66% 
userspace memory              0.35%      0.16%      0.19% 
free memory                  19.19%     19.19%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        22M     5609 mm/memory.c:1190 func:folio_prealloc 
        23M     1932 fs/xfs/xfs_buf.c:226 [xfs]
func:xfs_buf_alloc_backing_mem 
        24M    24135 fs/xfs/xfs_icache.c:97 [xfs]
func:xfs_inode_alloc 27M     6693 mm/memory.c:1192
func:folio_prealloc 58M    14784 mm/page_ext.c:271
func:alloc_page_ext 258M      129 mm/khugepaged.c:1069
func:alloc_charge_folio 430M   770788 lib/xarray.c:378
func:xas_alloc 545M    36444 mm/slub.c:3059 func:alloc_slab_page 
       9.8G  2563617 mm/readahead.c:189 func:ractl_alloc_folio 
        20G  5164004 mm/filemap.c:2012 func:__filemap_get_folio 

So I stopped the workload and dropped caches to confirm.

# echo 3 > /proc/sys/vm/drop_caches
# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        33.45%      0.09%     33.36% 
userspace memory              0.36%      0.16%      0.19% 
free memory                  66.20%     66.20%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        12M     2987 mm/execmem.c:41 func:execmem_vmalloc 
        12M        3 kernel/dma/pool.c:96 func:atomic_pool_expand 
        13M      751 mm/slub.c:3061 func:alloc_slab_page 
        16M        8 mm/khugepaged.c:1069 func:alloc_charge_folio 
        18M     4355 mm/memory.c:1190 func:folio_prealloc 
        24M     6119 mm/memory.c:1192 func:folio_prealloc 
        58M    14784 mm/page_ext.c:271 func:alloc_page_ext 
        61M    15448 mm/readahead.c:189 func:ractl_alloc_folio 
        79M     6726 mm/slub.c:3059 func:alloc_slab_page 
        11G  2674488 mm/filemap.c:2012 func:__filemap_get_folio
Quoted 23 lines
Quoted 2 lines

So if I'm reading this correctly something is causing folios collect and not be able to be freed?

CC cephfs, maybe someone could have an easy reading out of those folio usage

Quoted 16 lines

Also it's clear that some of the folio's are counting as cache and some aren't.

Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.

I'll do it the packages way until I get closer, then solve the build issues.

Thanks,
Mal

Thanks David.

I've contacted the ceph developers as well.

There was a suggestion it was due to the change from, to quote:
folio.free() to folio.put() or something like this.

The change happened around 6.14/6.15

I've found an easier reproducer.

There has been a suggestion that perhaps the ceph team might not fix this as "you can just reboot before the machine becomes unstable" and "Nobody else has encountered this bug"

I'll leave that to other people to make a call on but I'd assume the lack of reports is due to the fact that most stable distros are still on a a far too early kernel and/or are using the fuse driver with k8s.

Anyway, thanks for your assistance.

#8 lore raw

RRe: Possible memory leak in 6.17.7

Quoted 71 lines · Mal Haak
Quoted 68 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 67 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html

You would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=y
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y

Quoted 9 lines

And check /proc/allocinfo for the suspicious allocations which take more memory than expected.

(I once caught a nvidia driver memory leak.)

FYI
David

Thank you for your suggestion. I have some results.

Ran the rsync workload for about 9 hours. It started to look like it was happening.

# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        80.46%     65.80%     14.66% 
userspace memory              0.35%      0.16%      0.19% 
free memory                  19.19%     19.19%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        22M     5609 mm/memory.c:1190 func:folio_prealloc 
        23M     1932 fs/xfs/xfs_buf.c:226 [xfs]
func:xfs_buf_alloc_backing_mem 
        24M    24135 fs/xfs/xfs_icache.c:97 [xfs]
func:xfs_inode_alloc 27M     6693 mm/memory.c:1192
func:folio_prealloc 58M    14784 mm/page_ext.c:271
func:alloc_page_ext 258M      129 mm/khugepaged.c:1069
func:alloc_charge_folio 430M   770788 lib/xarray.c:378
func:xas_alloc 545M    36444 mm/slub.c:3059 func:alloc_slab_page 
       9.8G  2563617 mm/readahead.c:189 func:ractl_alloc_folio 
        20G  5164004 mm/filemap.c:2012 func:__filemap_get_folio 

So I stopped the workload and dropped caches to confirm.

# echo 3 > /proc/sys/vm/drop_caches
# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        33.45%      0.09%     33.36% 
userspace memory              0.36%      0.16%      0.19% 
free memory                  66.20%     66.20%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        12M     2987 mm/execmem.c:41 func:execmem_vmalloc 
        12M        3 kernel/dma/pool.c:96 func:atomic_pool_expand 
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
        13M      751 mm/slub.c:3061 func:alloc_slab_page 
        16M        8 mm/khugepaged.c:1069 func:alloc_charge_folio 
        18M     4355 mm/memory.c:1190 func:folio_prealloc 
        24M     6119 mm/memory.c:1192 func:folio_prealloc 
        58M    14784 mm/page_ext.c:271 func:alloc_page_ext 
        61M    15448 mm/readahead.c:189 func:ractl_alloc_folio 
        79M     6726 mm/slub.c:3059 func:alloc_slab_page 
        11G  2674488 mm/filemap.c:2012 func:__filemap_get_folio

Maybe narrowing down the "Noncache" caller of __filemap_get_folio would help clarify things. (It could be designed that way, and needs other route than dropping-cache to release the memory, just guess....) If you want, you can modify code to split the accounting for __filemap_get_folio according to its callers.

Following is a draft patch: (based on v6.18)

include/linux/pagemap.h +5 -1
index 09b581c1d878..ba8c659a6ae3 100644
@@ -753,7 +753,11 @@ static inline fgf_t fgf_set_order(size_t size)
 }
 
 void *filemap_get_entry(struct address_space *mapping, pgoff_t index);
-struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
+
+#define __filemap_get_folio(...)			\
+	alloc_hooks(__filemap_get_folio_noprof(__VA_ARGS__))
+
+struct folio *__filemap_get_folio_noprof(struct address_space *mapping, pgoff_t index,
 		fgf_t fgp_flags, gfp_t gfp);
 struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
 		fgf_t fgp_flags, gfp_t gfp);
mm/filemap.c +3 -3
index 024b71da5224..e1c1c26d7cb3 100644
@@ -1938,7 +1938,7 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)
  *
  * Return: The found folio or an ERR_PTR() otherwise.
  */
-struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
+struct folio *__filemap_get_folio_noprof(struct address_space *mapping, pgoff_t index,
 		fgf_t fgp_flags, gfp_t gfp)
 {
 	struct folio *folio;
@@ -2009,7 +2009,7 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
 			err = -ENOMEM;
 			if (order > min_order)
 				alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN;
-			folio = filemap_alloc_folio(alloc_gfp, order);
+			folio = filemap_alloc_folio_noprof(alloc_gfp, order);
 			if (!folio)
 				continue;
 
@@ -2056,7 +2056,7 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
 		folio_clear_dropbehind(folio);
 	return folio;
 }
-EXPORT_SYMBOL(__filemap_get_folio);
+EXPORT_SYMBOL(__filemap_get_folio_noprof);
 
 static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
 		xa_mark_t mark)

FYI
David

Quoted 44 lines
Quoted 24 lines
Quoted 3 lines

So if I'm reading this correctly something is causing folios collect and not be able to be freed?

CC cephfs, maybe someone could have an easy reading out of those folio usage

Quoted 16 lines

Also it's clear that some of the folio's are counting as cache and some aren't.

Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.

I'll do it the packages way until I get closer, then solve the build issues.

Thanks,
Mal

Thanks David.

I've contacted the ceph developers as well.

There was a suggestion it was due to the change from, to quote:
folio.free() to folio.put() or something like this.

The change happened around 6.14/6.15

I've found an easier reproducer.

There has been a suggestion that perhaps the ceph team might not fix this as "you can just reboot before the machine becomes unstable" and "Nobody else has encountered this bug"

I'll leave that to other people to make a call on but I'd assume the lack of reports is due to the fact that most stable distros are still on a a far too early kernel and/or are using the fuse driver with k8s.

Anyway, thanks for your assistance.

author
#9 lore raw

Re: RRe: Possible memory leak in 6.17.7

Quoted 82 lines · On Thu, 11 Dec 2025 11:28:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 72 lines · Mal Haak
Quoted 69 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 68 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html

You would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=y
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y

Quoted 9 lines

And check /proc/allocinfo for the suspicious allocations which take more memory than expected.

(I once caught a nvidia driver memory leak.)

FYI
David

Thank you for your suggestion. I have some results.

Ran the rsync workload for about 9 hours. It started to look like it was happening.

# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        80.46%     65.80%     14.66% 
userspace memory              0.35%      0.16%      0.19% 
free memory                  19.19%     19.19%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        22M     5609 mm/memory.c:1190 func:folio_prealloc 
        23M     1932 fs/xfs/xfs_buf.c:226 [xfs]
func:xfs_buf_alloc_backing_mem 
        24M    24135 fs/xfs/xfs_icache.c:97 [xfs]
func:xfs_inode_alloc 27M     6693 mm/memory.c:1192
func:folio_prealloc 58M    14784 mm/page_ext.c:271
func:alloc_page_ext 258M      129 mm/khugepaged.c:1069
func:alloc_charge_folio 430M   770788 lib/xarray.c:378
func:xas_alloc 545M    36444 mm/slub.c:3059 func:alloc_slab_page 
       9.8G  2563617 mm/readahead.c:189 func:ractl_alloc_folio 
        20G  5164004 mm/filemap.c:2012 func:__filemap_get_folio 

So I stopped the workload and dropped caches to confirm.

# echo 3 > /proc/sys/vm/drop_caches
# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        33.45%      0.09%     33.36% 
userspace memory              0.36%      0.16%      0.19% 
free memory                  66.20%     66.20%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        12M     2987 mm/execmem.c:41 func:execmem_vmalloc 
        12M        3 kernel/dma/pool.c:96
func:atomic_pool_expand 13M      751 mm/slub.c:3061
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

func:alloc_slab_page 16M 8 mm/khugepaged.c:1069
func:alloc_charge_folio 18M 4355 mm/memory.c:1190
func:folio_prealloc 24M 6119 mm/memory.c:1192
func:folio_prealloc 58M 14784 mm/page_ext.c:271
func:alloc_page_ext 61M 15448 mm/readahead.c:189
func:ractl_alloc_folio 79M 6726 mm/slub.c:3059
func:alloc_slab_page 11G 2674488 mm/filemap.c:2012 func:__filemap_get_folio

Quoted 5 lines

Maybe narrowing down the "Noncache" caller of __filemap_get_folio would help clarify things. (It could be designed that way, and needs other route than dropping-cache to release the memory, just guess....) If you want, you can modify code to split the accounting for __filemap_get_folio according to its callers.

Thanks again, I'll add this patch in and see where I end up.

The issue is nothing will cause the memory to be freed. Dropping caches doesn't work, memory pressure doesn't work, unmounting the filesystems doesn't work. Removing the cephfs and netfs kernel modules also doesn't work.

This is why I feel it's a ref_count (or similar) issue.

I've also found it seems to be a fixed amount leaked each time, per file. Simply doing lots of IO on one large file doesn't leak as fast as lots of "small" (greater than 10MB less than 100MB seems to be a sweet spot)

Also, dropping caches while the workload is running actually amplifies the issue. So it very much feels like something is wrong in the reclaim code.

Anyway I'll get this patch applied and see where I end up.

I now have crash dumps (after enabling crash_on_oom) so I'm going to try and see if I can find these structures and see what state they are in

Thanks again.

Mal

Quoted 104 lines

Following is a draft patch: (based on v6.18)

include/linux/pagemap.h +0 -0
index 09b581c1d878..ba8c659a6ae3 100644
@@ -753,7 +753,11 @@ static inline fgf_t fgf_set_order(size_t size)
 }
 
 void *filemap_get_entry(struct address_space *mapping, pgoff_t

index); -struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, +

include/linux/pagemap.h +4 -0
+#define __filemap_get_folio(...)			\
+	alloc_hooks(__filemap_get_folio_noprof(__VA_ARGS__))
+
+struct folio *__filemap_get_folio_noprof(struct address_space

*mapping, pgoff_t index, fgf_t fgp_flags, gfp_t gfp);
struct page *pagecache_get_page(struct address_space *mapping,
pgoff_t index, fgf_t fgp_flags, gfp_t gfp);

mm/filemap.c +0 -1
index 024b71da5224..e1c1c26d7cb3 100644
@@ -1938,7 +1938,7 @@ void *filemap_get_entry(struct address_space
 *mapping, pgoff_t index) *
  * Return: The found folio or an ERR_PTR() otherwise.
  */
-struct folio *__filemap_get_folio(struct address_space *mapping,

pgoff_t index, +struct folio *__filemap_get_folio_noprof(struct address_space *mapping, pgoff_t index, fgf_t fgp_flags, gfp_t gfp)
{

mm/filemap.c +1 -1
 	struct folio *folio;
@@ -2009,7 +2009,7 @@ struct folio *__filemap_get_folio(struct
 address_space *mapping, pgoff_t index, err = -ENOMEM;
 			if (order > min_order)
 				alloc_gfp |= __GFP_NORETRY |
 __GFP_NOWARN;
-			folio = filemap_alloc_folio(alloc_gfp,
 order);
+			folio =
 filemap_alloc_folio_noprof(alloc_gfp, order); if (!folio)
mm/filemap.c +1 -1
 				continue;
@@ -2056,7 +2056,7 @@ struct folio *__filemap_get_folio(struct
 address_space *mapping, pgoff_t index, folio_clear_dropbehind(folio);
 	return folio;
 }
-EXPORT_SYMBOL(__filemap_get_folio);
+EXPORT_SYMBOL(__filemap_get_folio_noprof);
 
 static inline struct folio *find_get_entry(struct xa_state *xas,
 pgoff_t max, xa_mark_t mark)

FYI
David

Quoted 45 lines
Quoted 24 lines
Quoted 3 lines

So if I'm reading this correctly something is causing folios collect and not be able to be freed?

CC cephfs, maybe someone could have an easy reading out of those folio usage

Quoted 16 lines

Also it's clear that some of the folio's are counting as cache and some aren't.

Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.

I'll do it the packages way until I get closer, then solve the build issues.

Thanks,
Mal

Thanks David.

I've contacted the ceph developers as well.

There was a suggestion it was due to the change from, to quote:
folio.free() to folio.put() or something like this.

The change happened around 6.14/6.15

I've found an easier reproducer.

There has been a suggestion that perhaps the ceph team might not fix this as "you can just reboot before the machine becomes unstable" and "Nobody else has encountered this bug"

I'll leave that to other people to make a call on but I'd assume the lack of reports is due to the fact that most stable distros are still on a a far too early kernel and/or are using the fuse driver with k8s.

Anyway, thanks for your assistance.

#10 lore raw

RE: RRe: Possible memory leak in 6.17.7

Hi Mal,

Quoted 110 lines · Mal Haak
Quoted 82 lines · On Thu, 11 Dec 2025 11:28:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 72 lines · Mal Haak
Quoted 69 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 68 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:

Hello,

I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.

I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html

You would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=y
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y

Quoted 9 lines

And check /proc/allocinfo for the suspicious allocations which take more memory than expected.

(I once caught a nvidia driver memory leak.)

FYI
David

Thank you for your suggestion. I have some results.

Ran the rsync workload for about 9 hours. It started to look like it was happening.

# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        80.46%     65.80%     14.66% 
userspace memory              0.35%      0.16%      0.19% 
free memory                  19.19%     19.19%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        22M     5609 mm/memory.c:1190 func:folio_prealloc 
        23M     1932 fs/xfs/xfs_buf.c:226 [xfs]
func:xfs_buf_alloc_backing_mem 
        24M    24135 fs/xfs/xfs_icache.c:97 [xfs]
func:xfs_inode_alloc 27M     6693 mm/memory.c:1192
func:folio_prealloc 58M    14784 mm/page_ext.c:271
func:alloc_page_ext 258M      129 mm/khugepaged.c:1069
func:alloc_charge_folio 430M   770788 lib/xarray.c:378
func:xas_alloc 545M    36444 mm/slub.c:3059 func:alloc_slab_page 
       9.8G  2563617 mm/readahead.c:189 func:ractl_alloc_folio 
        20G  5164004 mm/filemap.c:2012 func:__filemap_get_folio 

So I stopped the workload and dropped caches to confirm.

# echo 3 > /proc/sys/vm/drop_caches
# smem -pw
Area                           Used      Cache   Noncache 
firmware/hardware             0.00%      0.00%      0.00% 
kernel image                  0.00%      0.00%      0.00% 
kernel dynamic memory        33.45%      0.09%     33.36% 
userspace memory              0.36%      0.16%      0.19% 
free memory                  66.20%     66.20%      0.00% 
# sort -g /proc/allocinfo|tail|numfmt --to=iec
        12M     2987 mm/execmem.c:41 func:execmem_vmalloc 
        12M        3 kernel/dma/pool.c:96
func:atomic_pool_expand 13M      751 mm/slub.c:3061
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

func:alloc_slab_page 16M 8 mm/khugepaged.c:1069
func:alloc_charge_folio 18M 4355 mm/memory.c:1190
func:folio_prealloc 24M 6119 mm/memory.c:1192
func:folio_prealloc 58M 14784 mm/page_ext.c:271
func:alloc_page_ext 61M 15448 mm/readahead.c:189
func:ractl_alloc_folio 79M 6726 mm/slub.c:3059
func:alloc_slab_page 11G 2674488 mm/filemap.c:2012 func:__filemap_get_folio

Quoted 5 lines

Maybe narrowing down the "Noncache" caller of __filemap_get_folio would help clarify things. (It could be designed that way, and needs other route than dropping-cache to release the memory, just guess....) If you want, you can modify code to split the accounting for __filemap_get_folio according to its callers.

Thanks again, I'll add this patch in and see where I end up.

The issue is nothing will cause the memory to be freed. Dropping caches doesn't work, memory pressure doesn't work, unmounting the filesystems doesn't work. Removing the cephfs and netfs kernel modules also doesn't work.

This is why I feel it's a ref_count (or similar) issue.

I've also found it seems to be a fixed amount leaked each time, per file. Simply doing lots of IO on one large file doesn't leak as fast as lots of "small" (greater than 10MB less than 100MB seems to be a sweet spot)

Also, dropping caches while the workload is running actually amplifies the issue. So it very much feels like something is wrong in the reclaim code.

Quoted 7 lines

Anyway I'll get this patch applied and see where I end up.

I now have crash dumps (after enabling crash_on_oom) so I'm going to try and see if I can find these structures and see what state they are in

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

author
#11 lore raw

Re: RRe: Possible memory leak in 6.17.7

Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Regards

Mal Haak

#12 lore raw

RE: RRe: Possible memory leak in 6.17.7

Quoted 85 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Quoted 4 lines

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Thanks a lot for of your efforts. I hope it will help a lot. Let me start to reproduce the issue. I'll let you know if I need additional details. I'll share my progress and potential troubles in the ticket that you've created.

Thanks,
Slava.

#13 lore raw
In reply to Mal Haak
Quoted 14 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 8 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Quoted 7 lines
Quoted 2 lines

Thanks,
Slava.

Hello,

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

Hi, just out of curiosity, have you narrowed down the caller of __filemap_get_folio causing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?

David

Quoted 70 lines

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Regards

Mal Haak

author
#14 lore raw
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Quoted 9 lines
Quoted 2 lines

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

Hi, just out of curiosity, have you narrowed down the caller of __filemap_get_folio causing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?

David

Hi David,

I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.

But actually this could speed things up considerably. I will do that now and see what I get.

Thanks

Mal

Quoted 70 lines
Quoted 70 lines

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Regards

Mal Haak

author
#15 lore raw
Quoted 37 lines · On Tue, 16 Dec 2025 17:09:18 +1000 Mal Haak <[email protected]> wrote:
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Quoted 9 lines
Quoted 2 lines

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

Hi, just out of curiosity, have you narrowed down the caller of __filemap_get_folio causing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?

David

Hi David,

I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.

Quoted 7 lines

But actually this could speed things up considerably. I will do that now and see what I get.

Thanks

Mal

I did just give it a blast.

Unfortunately it returned exactly what I expected, that is the calls are all coming from netfs.

Which makes sense for cephfs.

# sort -g /proc/allocinfo|tail|numfmt --to=iec
         10M     2541 drivers/block/zram/zram_drv.c:1597 [zram]
func:zram_meta_alloc 12M     3001 mm/execmem.c:41 func:execmem_vmalloc 
         12M     3605 kernel/fork.c:311 func:alloc_thread_stack_node 
         16M      992 mm/slub.c:3061 func:alloc_slab_page 
         20M    35544 lib/xarray.c:378 func:xas_alloc 
         31M     7704 mm/memory.c:1192 func:folio_prealloc 
         69M    17562 mm/memory.c:1190 func:folio_prealloc 
        104M     8212 mm/slub.c:3059 func:alloc_slab_page 
        124M    30075 mm/readahead.c:189 func:ractl_alloc_folio 
        2.6G   661392 fs/netfs/buffered_read.c:635 [netfs]
func:netfs_write_begin 

So, unfortunately it doesn't reveal the true source. But was worth a shot! So thanks again

Mal

Quoted 73 lines
Quoted 72 lines
Quoted 72 lines

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Regards

Mal Haak

#16 lore raw
Quoted 67 lines · Mal Haak
Quoted 37 lines · On Tue, 16 Dec 2025 17:09:18 +1000 Mal Haak <[email protected]> wrote:
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Quoted 9 lines
Quoted 2 lines

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

Hi, just out of curiosity, have you narrowed down the caller of __filemap_get_folio causing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?

David

Hi David,

I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.

Quoted 7 lines

But actually this could speed things up considerably. I will do that now and see what I get.

Thanks

Mal

I did just give it a blast.

Unfortunately it returned exactly what I expected, that is the calls are all coming from netfs.

Which makes sense for cephfs.

# sort -g /proc/allocinfo|tail|numfmt --to=iec
        10M     2541 drivers/block/zram/zram_drv.c:1597 [zram]
func:zram_meta_alloc 12M     3001 mm/execmem.c:41 func:execmem_vmalloc 
        12M     3605 kernel/fork.c:311 func:alloc_thread_stack_node 
        16M      992 mm/slub.c:3061 func:alloc_slab_page 
        20M    35544 lib/xarray.c:378 func:xas_alloc 
        31M     7704 mm/memory.c:1192 func:folio_prealloc 
        69M    17562 mm/memory.c:1190 func:folio_prealloc 
       104M     8212 mm/slub.c:3059 func:alloc_slab_page 
       124M    30075 mm/readahead.c:189 func:ractl_alloc_folio 
       2.6G   661392 fs/netfs/buffered_read.c:635 [netfs]
func:netfs_write_begin 
Quoted 2 lines

So, unfortunately it doesn't reveal the true source. But was worth a shot! So thanks again

Oh, at least cephfs could be ruled out, right?

CC netfs folks then. :)

Quoted 77 lines

Mal

Quoted 73 lines
Quoted 72 lines
Quoted 72 lines

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Regards

Mal Haak

#17 lore raw
Quoted 69 lines · David Wang
Quoted 67 lines · Mal Haak
Quoted 37 lines · On Tue, 16 Dec 2025 17:09:18 +1000 Mal Haak <[email protected]> wrote:
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Quoted 9 lines
Quoted 2 lines

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

Hi, just out of curiosity, have you narrowed down the caller of __filemap_get_folio causing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?

David

Hi David,

I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.

Quoted 7 lines

But actually this could speed things up considerably. I will do that now and see what I get.

Thanks

Mal

I did just give it a blast.

Unfortunately it returned exactly what I expected, that is the calls are all coming from netfs.

Which makes sense for cephfs.

# sort -g /proc/allocinfo|tail|numfmt --to=iec
        10M     2541 drivers/block/zram/zram_drv.c:1597 [zram]
func:zram_meta_alloc 12M     3001 mm/execmem.c:41 func:execmem_vmalloc 
        12M     3605 kernel/fork.c:311 func:alloc_thread_stack_node 
        16M      992 mm/slub.c:3061 func:alloc_slab_page 
        20M    35544 lib/xarray.c:378 func:xas_alloc 
        31M     7704 mm/memory.c:1192 func:folio_prealloc 
        69M    17562 mm/memory.c:1190 func:folio_prealloc 
       104M     8212 mm/slub.c:3059 func:alloc_slab_page 
       124M    30075 mm/readahead.c:189 func:ractl_alloc_folio 
       2.6G   661392 fs/netfs/buffered_read.c:635 [netfs]
func:netfs_write_begin 
Quoted 4 lines
Quoted 2 lines

So, unfortunately it doesn't reveal the true source. But was worth a shot! So thanks again

Oh, at least cephfs could be ruled out, right?

ehh...., I think I could be wrong about this.....

Quoted 82 lines

CC netfs folks then. :)

Quoted 77 lines

Mal

Quoted 73 lines
Quoted 72 lines
Quoted 72 lines

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Regards

Mal Haak

#18 lore raw

Hi Mal,

Quoted 3 lines · David Wang
Quoted 2 lines · David Wang

<skipped>

Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Could you please add your CephFS kernel client's mount options into the ticket [1]?

Thanks a lot,
Slava.

[1] https://tracker.ceph.com/issues/74156

author
#19 lore raw
Quoted 91 lines · On Wed, 17 Dec 2025 01:56:52 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

Quoted 3 lines · David Wang
Quoted 2 lines · David Wang

<skipped>

Quoted 76 lines
Quoted 76 lines
Quoted 76 lines
Quoted 76 lines
Quoted 76 lines
Quoted 76 lines

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Quoted 7 lines

Could you please add your CephFS kernel client's mount options into the ticket [1]?

Thanks a lot,
Slava.

[1] https://tracker.ceph.com/issues/74156

I've updated the ticket.

I am curious about the differences between your test setup and my actual setup in terms of capacity and hardware.

I can provide crash dumps if it is helpful.

Thanks

Mal

#20 lore raw
In reply to Mal Haak
Quoted 27 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

Quoted 6 lines

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Hi,
Just a suggestion, in case you run out of idea for further investigation. I think you can bisect *manually* targeting changes of fs/cephfs between 6.14 and 6.15

$ git log  --pretty='format:%h %an' v6.14..v6.15 fs/ceph
349b7d77f5a1 Linus Torvalds
b261d2222063 Eric Biggers
f452a2204614 David Howells
e63046adefc0 Linus Torvalds
59b59a943177 Matthew Wilcox (Oracle)  <-----------3
efbdd92ed9f6 Matthew Wilcox (Oracle)
d1b452673af4 Matthew Wilcox (Oracle)
ad49fe2b3d54 Matthew Wilcox (Oracle)
a55cf4fd8fae Matthew Wilcox (Oracle)
15fdaf2fd60d Matthew Wilcox (Oracle)
62171c16da60 Matthew Wilcox (Oracle)
baff9740bc8f Matthew Wilcox (Oracle)
f9707a8b5b9d Matthew Wilcox (Oracle)
88a59bda3f37 Matthew Wilcox (Oracle)
19a288110435 Matthew Wilcox (Oracle)
fd7449d937e7 Viacheslav Dubeyko  <---------2
1551ec61dc55 Viacheslav Dubeyko
ce80b76dd327 Viacheslav Dubeyko
f08068df4aa4 Viacheslav Dubeyko
3f92c7b57687 NeilBrown               <-----------1
88d5baf69082 NeilBrown

There were 3 major patch set (group by author), the suspect could be narrowed down further.

(Bisect, even over a short range of patch, is quite an unpleasant experience though....)

FYI
David

Quoted 58 lines

Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.

I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.

I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".

I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.

This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.

None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.

I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.

Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.

Regards

Mal Haak

author
#21 lore raw
Quoted 68 lines · On Wed, 17 Dec 2025 13:59:47 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 27 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…

Hi Mal,

<SNIP>

Quoted 10 lines

Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)

Thanks,
Slava.

Hello,

Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.

I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.

Quoted 6 lines

I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.

Hi,
Just a suggestion, in case you run out of idea for further investigation. I think you can bisect *manually* targeting changes of fs/cephfs between 6.14 and 6.15

$ git log  --pretty='format:%h %an' v6.14..v6.15 fs/ceph
349b7d77f5a1 Linus Torvalds
b261d2222063 Eric Biggers
f452a2204614 David Howells
e63046adefc0 Linus Torvalds
59b59a943177 Matthew Wilcox (Oracle)  <-----------3
efbdd92ed9f6 Matthew Wilcox (Oracle)
d1b452673af4 Matthew Wilcox (Oracle)
ad49fe2b3d54 Matthew Wilcox (Oracle)
a55cf4fd8fae Matthew Wilcox (Oracle)
15fdaf2fd60d Matthew Wilcox (Oracle)
62171c16da60 Matthew Wilcox (Oracle)
baff9740bc8f Matthew Wilcox (Oracle)
f9707a8b5b9d Matthew Wilcox (Oracle)
88a59bda3f37 Matthew Wilcox (Oracle)
19a288110435 Matthew Wilcox (Oracle)
fd7449d937e7 Viacheslav Dubeyko  <---------2
1551ec61dc55 Viacheslav Dubeyko
ce80b76dd327 Viacheslav Dubeyko
f08068df4aa4 Viacheslav Dubeyko
3f92c7b57687 NeilBrown               <-----------1
88d5baf69082 NeilBrown

There were 3 major patch set (group by author), the suspect could be narrowed down further.

Quoted 7 lines

(Bisect, even over a short range of patch, is quite an unpleasant experience though....)

FYI
David

Quoted 1 line

<SNIP>

Yeah, I don't think it's a small patch that is the cause of the issue.

It looks like there was a patch set that migrated cephfs off handling individual pages and onto folios to enable wider use of netfs features like local caching and encryption, as examples. I'm not sure that set can be broken up and result in a working cephfs module. Which limits the utility of a git-bisect. I'm pretty sure the issue is in addr.c somewhere and most of the changes in there are one patch. That said, after I get this crash dump I'll probably do it anyway.

What I really need to do is get a crash dump to look at what state the folios and their tracking is in. Assuming I can grok what I'm looking at. This is the bit I'm most apprehensive of. I'm hoping I can find a list of folios used by the reclaim machinery that is missing a bunch of folios. That or a bunch with inflated refcounts or something.

Something is going awry, but it's not fast. I thought I had a quick reproducer. I was wrong, I sized the DD workload incorrectly and triggered the panic_on_oom due to that, not the bug.

I'm re-running the reproducer now, on a VM with 2GB of ram and its been running for around 3hrs and I think at most its leaked possibly 100MB-150MB of ram at most. (It was averaging 190-200MB of noncache usage. It's now averaging 290-340MB).

It does accelerate. The more folios that are in the weird state, the more end up in the weird state. Which feels like fragmentation side effects, but I'm just speculating.

Anyway, one of the wonderful ceph developers is looking into it. I just hope I can do enough to help them locate the issue. They are having troubles reproducing last I heard from them but they might have been expecting a slightly faster reproducer.

I have however recreated it on a physical host not just a vm. So I feel like I can rule out being a VM as a cause.

Anyway thanks for your continued assistance!

Mal