Possible memory leak in 6.17.7
Subsystems (3)
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.
Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 88.21% 36.25% 51.96% userspace memory 9.49% 0.15% 9.34% free memory 2.30% 2.30% 0.00%
free -h output:
total used free shared buff/cache available Mem: 31Gi 3.6Gi 500Mi 4.0Mi 11Gi 27Gi Swap: 4.0Gi 179Mi 3.8Gi
Reverting to the previous LTS fixes the issue
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00%
kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%
I am unsure of the best way to track down the memory usage. I have tried stopping the workload and unmounting the cephfs filesystem. As well as removing the ceph and network related kernel modules.
I assume some kind of tracing would be a way to find the culprit, however I am unsure of the best way to do that.
I can do a git bisect and am in the process of getting a test reproducer made for that.
But if there is an easier way to do it I would happily do that.
Just to note, slabtop looks normal and doesn't show the memory usage.
Thanks in advance
Mal
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.
Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 88.21% 36.25% 51.96% userspace memory 9.49% 0.15% 9.34% free memory 2.30% 2.30% 0.00%free -h output:
total used free shared buff/cache available Mem: 31Gi 3.6Gi 500Mi 4.0Mi 11Gi 27Gi Swap: 4.0Gi 179Mi 3.8GiReverting to the previous LTS fixes the issue
Quoted 8 lines
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00%kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%
I have more information. The leaking of kernel memory only starts once there is a lot of data in buffers/cache. And only once it's been in that state for several hours.
Currently in my search for a reproducer I have found that downloading then seeding of multiple torrents of linux distribution ISO's will replicate the issue. But it only begins leaking at around the 6-9 hour mark.
It does not appear to be dependant on cephfs; but due to it's use of sockets I believe this is making the situation worse.
I cannot replicate it at all with the LTS kernel release but it does look like the current RC releases do have this issue.
I was looking at doing a kernel build with CONFIG_DEBUG_KMEMLEAK enabled and will if it's thought this would find the issue. However as the memory usage is still somewhat tracked and obviously marked as reclaimable it feels more like something in the reclaim logic is getting broken.
I do wonder if due to it only happening after ram is mostly consumed by cache, and even then only if it has been that way for hours, if the issue is memory fragmentation related.
Regardless, some advice on how to narrow this down faster than a git bisect as 9hrs to even confirm replication of the issue makes git bisect painfully slow.
Thanks in advance
Mal Haak
I have a reproducer. It's slow but it works.
I kept rsync running for 2 days by moving 5TB of files.
smem -wp
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 98.81% 1.69% 97.13% userspace memory 0.08% 0.05% 0.03% free memory 1.11% 1.11% 0.00%
[root@kerneltest ~]# uname -a
Linux kerneltest 6.18.0-1-mainline #1 SMP PREEMPT_DYNAMIC Tue, 11 Nov 2025 00:02:22 +0000 x86_64 GNU/Linux
The issue is in 6.18.
Quoted 74 lines · On Thu, 20 Nov 2025 12:23:51 +1000 Mal Haak <[email protected]> wrote:
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.
Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 88.21% 36.25% 51.96% userspace memory 9.49% 0.15% 9.34% free memory 2.30% 2.30% 0.00%free -h output:
total used free shared buff/cache available Mem: 31Gi 3.6Gi 500Mi 4.0Mi 11Gi 27Gi Swap: 4.0Gi 179Mi 3.8GiReverting to the previous LTS fixes the issue
Quoted 8 lines
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00%kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%I have more information. The leaking of kernel memory only starts once there is a lot of data in buffers/cache. And only once it's been in that state for several hours.
Currently in my search for a reproducer I have found that downloading then seeding of multiple torrents of linux distribution ISO's will replicate the issue. But it only begins leaking at around the 6-9 hour mark.
It does not appear to be dependant on cephfs; but due to it's use of sockets I believe this is making the situation worse.
I cannot replicate it at all with the LTS kernel release but it does look like the current RC releases do have this issue.
I was looking at doing a kernel build with
CONFIG_DEBUG_KMEMLEAKenabled and will if it's thought this would find the issue. However as the memory usage is still somewhat tracked and obviously marked as reclaimable it feels more like something in the reclaim logic is getting broken.I do wonder if due to it only happening after ram is mostly consumed by cache, and even then only if it has been that way for hours, if the issue is memory fragmentation related.
Regardless, some advice on how to narrow this down faster than a git bisect as 9hrs to even confirm replication of the issue makes git bisect painfully slow.
Thanks in advance
Mal Haak
So four and a half hours is how long it takes to get it to reproduce with two rsync's running in parallel.
Looks like git bisect is the only way forward as I built a kernel with kmemleak enabled and it cannot locate the leak.
Probably because the pages are marked as able to be freed.
Any other suggestions on how to trace down the leak would be appreciated.
Thanks
Mal
Quoted 96 lines · On Sat, 6 Dec 2025 08:23:36 +1000 Mal Haak <[email protected]> wrote:
I have a reproducer. It's slow but it works.
I kept rsync running for 2 days by moving 5TB of files.
smem -wp
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 98.81% 1.69% 97.13% userspace memory 0.08% 0.05% 0.03% free memory 1.11% 1.11% 0.00%[root@kerneltest ~]# uname -a
Linux kerneltest 6.18.0-1-mainline #1 SMPPREEMPT_DYNAMICTue, 11 Nov 2025 00:02:22 +0000x86_64GNU/LinuxThe issue is in 6.18.
Quoted 74 lines · On Thu, 20 Nov 2025 12:23:51 +1000 Mal Haak <[email protected]> wrote:
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.
Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 88.21% 36.25% 51.96% userspace memory 9.49% 0.15% 9.34% free memory 2.30% 2.30% 0.00%free -h output:
total used free shared buff/cache available Mem: 31Gi 3.6Gi 500Mi 4.0Mi 11Gi 27Gi Swap: 4.0Gi 179Mi 3.8GiReverting to the previous LTS fixes the issue
Quoted 8 lines
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00%kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%I have more information. The leaking of kernel memory only starts once there is a lot of data in buffers/cache. And only once it's been in that state for several hours.
Currently in my search for a reproducer I have found that downloading then seeding of multiple torrents of linux distribution ISO's will replicate the issue. But it only begins leaking at around the 6-9 hour mark.
It does not appear to be dependant on cephfs; but due to it's use of sockets I believe this is making the situation worse.
I cannot replicate it at all with the LTS kernel release but it does look like the current RC releases do have this issue.
I was looking at doing a kernel build with
CONFIG_DEBUG_KMEMLEAKenabled and will if it's thought this would find the issue. However as the memory usage is still somewhat tracked and obviously marked as reclaimable it feels more like something in the reclaim logic is getting broken.I do wonder if due to it only happening after ram is mostly consumed by cache, and even then only if it has been that way for hours, if the issue is memory fragmentation related.
Regardless, some advice on how to narrow this down faster than a git bisect as 9hrs to even confirm replication of the issue makes git bisect painfully slow.
Thanks in advance
Mal Haak
Quoted 30 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I am running a server that has a heavy read/write workload to a cephfs file system. It is a VM.
Over time it appears that the non-cache useage of kernel dynamic memory increases. The kernel seems to think the pages are reclaimable however nothing appears to trigger the reclaim. This leads to workloads getting killed via oomkiller.
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 88.21% 36.25% 51.96% userspace memory 9.49% 0.15% 9.34% free memory 2.30% 2.30% 0.00%free -h output:
total used free shared buff/cache available Mem: 31Gi 3.6Gi 500Mi 4.0Mi 11Gi 27Gi Swap: 4.0Gi 179Mi 3.8GiReverting to the previous LTS fixes the issue
Quoted 8 lines
smem -wp output:
Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00%kernel dynamic memory 80.22% 79.32% 0.90%
userspace memory 10.48% 0.20% 10.28%
free memory 9.30% 9.30% 0.00%
I think the memory allocation profiling feature can help.
https://docs.kernel.org/mm/allocation-profiling.html
You would need to build a kernel withCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y
And check /proc/allocinfo for the suspicious allocations which take more memory than expected.
(I once caught a nvidia driver memory leak.)
FYI
David
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I think the
memory allocation profilingfeature can help.
https://docs.kernel.org/mm/allocation-profiling.htmlYou would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y
Quoted 9 lines
And check
/proc/allocinfofor the suspicious allocations which take more memory than expected.(I once caught a nvidia driver memory leak.)
FYI
David
Thank you for your suggestion. I have some results.
Ran the rsync workload for about 9 hours. It started to look like it was happening.
# smem -pw
Area Used Cache Noncache
firmware/hardware 0.00% 0.00% 0.00%
kernel image 0.00% 0.00% 0.00%
kernel dynamic memory 80.46% 65.80% 14.66%
userspace memory 0.35% 0.16% 0.19%
free memory 19.19% 19.19% 0.00%
# sort -g /proc/allocinfo|tail|numfmt --to=iec
22M 5609 mm/memory.c:1190 func:folio_prealloc
23M 1932 fs/xfs/xfs_buf.c:226 [xfs]
func:xfs_buf_alloc_backing_mem
24M 24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc
27M 6693 mm/memory.c:1192 func:folio_prealloc
58M 14784 mm/page_ext.c:271 func:alloc_page_ext
258M 129 mm/khugepaged.c:1069 func:alloc_charge_folio
430M 770788 lib/xarray.c:378 func:xas_alloc
545M 36444 mm/slub.c:3059 func:alloc_slab_page
9.8G 2563617 mm/readahead.c:189 func:ractl_alloc_folio
20G 5164004 mm/filemap.c:2012 func:__filemap_get_folio So I stopped the workload and dropped caches to confirm.
# echo 3 > /proc/sys/vm/drop_caches
# smem -pw
Area Used Cache Noncache
firmware/hardware 0.00% 0.00% 0.00%
kernel image 0.00% 0.00% 0.00%
kernel dynamic memory 33.45% 0.09% 33.36%
userspace memory 0.36% 0.16% 0.19%
free memory 66.20% 66.20% 0.00%
# sort -g /proc/allocinfo|tail|numfmt --to=iec
12M 2987 mm/execmem.c:41 func:execmem_vmalloc
12M 3 kernel/dma/pool.c:96 func:atomic_pool_expand
13M 751 mm/slub.c:3061 func:alloc_slab_page
16M 8 mm/khugepaged.c:1069 func:alloc_charge_folio
18M 4355 mm/memory.c:1190 func:folio_prealloc
24M 6119 mm/memory.c:1192 func:folio_prealloc
58M 14784 mm/page_ext.c:271 func:alloc_page_ext
61M 15448 mm/readahead.c:189 func:ractl_alloc_folio
79M 6726 mm/slub.c:3059 func:alloc_slab_page
11G 2674488 mm/filemap.c:2012 func:__filemap_get_folioSo if I'm reading this correctly something is causing folios collect and not be able to be freed?
Also it's clear that some of the folio's are counting as cache and some aren't.
Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.
I'll do it the packages way until I get closer, then solve the build issues.
Thanks,
Mal
Quoted 76 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I think the
memory allocation profilingfeature can help.
https://docs.kernel.org/mm/allocation-profiling.htmlYou would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yQuoted 9 lines
And check
/proc/allocinfofor the suspicious allocations which take more memory than expected.(I once caught a nvidia driver memory leak.)
FYI
DavidThank you for your suggestion. I have some results.
Ran the rsync workload for about 9 hours. It started to look like it was happening.
# smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 80.46% 65.80% 14.66% userspace memory 0.35% 0.16% 0.19% free memory 19.19% 19.19% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 22M 5609 mm/memory.c:1190 func:folio_prealloc 23M 1932 fs/xfs/xfs_buf.c:226 [xfs] func:xfs_buf_alloc_backing_mem 24M 24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc 27M 6693 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 258M 129 mm/khugepaged.c:1069 func:alloc_charge_folio 430M 770788 lib/xarray.c:378 func:xas_alloc 545M 36444 mm/slub.c:3059 func:alloc_slab_page 9.8G 2563617 mm/readahead.c:189 func:ractl_alloc_folio 20G 5164004 mm/filemap.c:2012 func:__filemap_get_folioSo I stopped the workload and dropped caches to confirm.
# echo 3 > /proc/sys/vm/drop_caches # smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 33.45% 0.09% 33.36% userspace memory 0.36% 0.16% 0.19% free memory 66.20% 66.20% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 12M 2987 mm/execmem.c:41 func:execmem_vmalloc 12M 3 kernel/dma/pool.c:96 func:atomic_pool_expand 13M 751 mm/slub.c:3061 func:alloc_slab_page 16M 8 mm/khugepaged.c:1069 func:alloc_charge_folio 18M 4355 mm/memory.c:1190 func:folio_prealloc 24M 6119 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 61M 15448 mm/readahead.c:189 func:ractl_alloc_folio 79M 6726 mm/slub.c:3059 func:alloc_slab_page 11G 2674488 mm/filemap.c:2012 func:__filemap_get_folio
Quoted 2 lines
So if I'm reading this correctly something is causing folios collect and not be able to be freed?
CC cephfs, maybe someone could have an easy reading out of those folio usage
Quoted 16 lines
Also it's clear that some of the folio's are counting as cache and some aren't.
Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.
I'll do it the packages way until I get closer, then solve the build issues.
Thanks,
Mal
Quoted 77 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 76 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I think the
memory allocation profilingfeature can help.
https://docs.kernel.org/mm/allocation-profiling.htmlYou would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yQuoted 9 lines
And check
/proc/allocinfofor the suspicious allocations which take more memory than expected.(I once caught a nvidia driver memory leak.)
FYI
DavidThank you for your suggestion. I have some results.
Ran the rsync workload for about 9 hours. It started to look like it was happening.
# smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 80.46% 65.80% 14.66% userspace memory 0.35% 0.16% 0.19% free memory 19.19% 19.19% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 22M 5609 mm/memory.c:1190 func:folio_prealloc 23M 1932 fs/xfs/xfs_buf.c:226 [xfs] func:xfs_buf_alloc_backing_mem 24M 24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc 27M 6693 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 258M 129 mm/khugepaged.c:1069 func:alloc_charge_folio 430M 770788 lib/xarray.c:378 func:xas_alloc 545M 36444 mm/slub.c:3059 func:alloc_slab_page 9.8G 2563617 mm/readahead.c:189 func:ractl_alloc_folio 20G 5164004 mm/filemap.c:2012 func:__filemap_get_folioSo I stopped the workload and dropped caches to confirm.
# echo 3 > /proc/sys/vm/drop_caches # smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 33.45% 0.09% 33.36% userspace memory 0.36% 0.16% 0.19% free memory 66.20% 66.20% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 12M 2987 mm/execmem.c:41 func:execmem_vmalloc 12M 3 kernel/dma/pool.c:96 func:atomic_pool_expand 13M 751 mm/slub.c:3061 func:alloc_slab_page 16M 8 mm/khugepaged.c:1069 func:alloc_charge_folio 18M 4355 mm/memory.c:1190 func:folio_prealloc 24M 6119 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 61M 15448 mm/readahead.c:189 func:ractl_alloc_folio 79M 6726 mm/slub.c:3059 func:alloc_slab_page 11G 2674488 mm/filemap.c:2012 func:__filemap_get_folio
Quoted 23 lines
Quoted 2 lines
So if I'm reading this correctly something is causing folios collect and not be able to be freed?
CC cephfs, maybe someone could have an easy reading out of those folio usage
Quoted 16 lines
Also it's clear that some of the folio's are counting as cache and some aren't.
Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.
I'll do it the packages way until I get closer, then solve the build issues.
Thanks,
Mal
Thanks David.
I've contacted the ceph developers as well.
There was a suggestion it was due to the change from, to quote:folio.free() to folio.put() or something like this.
The change happened around 6.14/6.15
I've found an easier reproducer.
There has been a suggestion that perhaps the ceph team might not fix this as "you can just reboot before the machine becomes unstable" and "Nobody else has encountered this bug"
I'll leave that to other people to make a call on but I'd assume the lack of reports is due to the fact that most stable distros are still on a a far too early kernel and/or are using the fuse driver with k8s.
Anyway, thanks for your assistance.
RRe: Possible memory leak in 6.17.7
Quoted 71 lines · Mal Haak
Quoted 68 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 67 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I think the
memory allocation profilingfeature can help.
https://docs.kernel.org/mm/allocation-profiling.htmlYou would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yQuoted 9 lines
And check
/proc/allocinfofor the suspicious allocations which take more memory than expected.(I once caught a nvidia driver memory leak.)
FYI
DavidThank you for your suggestion. I have some results.
Ran the rsync workload for about 9 hours. It started to look like it was happening.
# smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 80.46% 65.80% 14.66% userspace memory 0.35% 0.16% 0.19% free memory 19.19% 19.19% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 22M 5609 mm/memory.c:1190 func:folio_prealloc 23M 1932 fs/xfs/xfs_buf.c:226 [xfs] func:xfs_buf_alloc_backing_mem 24M 24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc 27M 6693 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 258M 129 mm/khugepaged.c:1069 func:alloc_charge_folio 430M 770788 lib/xarray.c:378 func:xas_alloc 545M 36444 mm/slub.c:3059 func:alloc_slab_page 9.8G 2563617 mm/readahead.c:189 func:ractl_alloc_folio 20G 5164004 mm/filemap.c:2012 func:__filemap_get_folioSo I stopped the workload and dropped caches to confirm.
# echo 3 > /proc/sys/vm/drop_caches # smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 33.45% 0.09% 33.36% userspace memory 0.36% 0.16% 0.19% free memory 66.20% 66.20% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 12M 2987 mm/execmem.c:41 func:execmem_vmalloc 12M 3 kernel/dma/pool.c:96 func:atomic_pool_expand
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
13M 751 mm/slub.c:3061 func:alloc_slab_page 16M 8 mm/khugepaged.c:1069 func:alloc_charge_folio 18M 4355 mm/memory.c:1190 func:folio_prealloc 24M 6119 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 61M 15448 mm/readahead.c:189 func:ractl_alloc_folio 79M 6726 mm/slub.c:3059 func:alloc_slab_page 11G 2674488 mm/filemap.c:2012 func:__filemap_get_folio
Maybe narrowing down the "Noncache" caller of __filemap_get_folio would help clarify things. (It could be designed that way, and needs other route than dropping-cache to release the memory, just guess....) If you want, you can modify code to split the accounting for __filemap_get_folio according to its callers.
Following is a draft patch: (based on v6.18)
include/linux/pagemap.h +5 -1
@@ -753,7 +753,11 @@ static inline fgf_t fgf_set_order(size_t size) }void *filemap_get_entry (struct address_space *mapping,pgoff_t index); -struct folio *__filemap_get_folio (struct address_space *mapping,pgoff_t index, + +#define __filemap_get_folio (...) \ + alloc_hooks(__filemap_get_folio_noprof(__VA_ARGS__)) + +struct folio *__filemap_get_folio_noprof (struct address_space *mapping,pgoff_t index,fgf_t fgp_flags,gfp_t gfp);struct page *pagecache_get_page (struct address_space *mapping,pgoff_t index,fgf_t fgp_flags,gfp_t gfp);
mm/filemap.c +3 -3
@@ -1938,7 +1938,7 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)* * Return: The found folio or an ERR_PTR() otherwise. */ -struct folio *__filemap_get_folio (struct address_space *mapping,pgoff_t index, +struct folio *__filemap_get_folio_noprof (struct address_space *mapping,pgoff_t index,fgf_t fgp_flags,gfp_t gfp) {struct folio *folio;
@@ -2009,7 +2009,7 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, err = -ENOMEM ;if (order > min_order) alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN; - folio =filemap_alloc_folio (alloc_gfp, order); + folio =filemap_alloc_folio_noprof (alloc_gfp, order);if (!folio)continue ;
@@ -2056,7 +2056,7 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,folio_clear_dropbehind (folio);return folio; } -EXPORT_SYMBOL (__filemap_get_folio); +EXPORT_SYMBOL (__filemap_get_folio_noprof);static inline struct folio *find_get_entry (struct xa_state *xas,pgoff_t max,xa_mark_t mark)
FYI
David
Quoted 44 lines
Quoted 24 lines
Quoted 3 lines
So if I'm reading this correctly something is causing folios collect and not be able to be freed?
CC cephfs, maybe someone could have an easy reading out of those folio usage
Quoted 16 lines
Also it's clear that some of the folio's are counting as cache and some aren't.
Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.
I'll do it the packages way until I get closer, then solve the build issues.
Thanks,
MalThanks David.
I've contacted the ceph developers as well.
There was a suggestion it was due to the change from, to quote:
folio.free()tofolio.put()or something like this.The change happened around 6.14/6.15
I've found an easier reproducer.
There has been a suggestion that perhaps the ceph team might not fix this as "you can just reboot before the machine becomes unstable" and "Nobody else has encountered this bug"
I'll leave that to other people to make a call on but I'd assume the lack of reports is due to the fact that most stable distros are still on a a far too early kernel and/or are using the fuse driver with k8s.
Anyway, thanks for your assistance.
Re: RRe: Possible memory leak in 6.17.7
Quoted 82 lines · On Thu, 11 Dec 2025 11:28:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 72 lines · Mal Haak
Quoted 69 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 68 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I think the
memory allocation profilingfeature can help.
https://docs.kernel.org/mm/allocation-profiling.htmlYou would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yQuoted 9 lines
And check
/proc/allocinfofor the suspicious allocations which take more memory than expected.(I once caught a nvidia driver memory leak.)
FYI
DavidThank you for your suggestion. I have some results.
Ran the rsync workload for about 9 hours. It started to look like it was happening.
# smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 80.46% 65.80% 14.66% userspace memory 0.35% 0.16% 0.19% free memory 19.19% 19.19% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 22M 5609 mm/memory.c:1190 func:folio_prealloc 23M 1932 fs/xfs/xfs_buf.c:226 [xfs] func:xfs_buf_alloc_backing_mem 24M 24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc 27M 6693 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 258M 129 mm/khugepaged.c:1069 func:alloc_charge_folio 430M 770788 lib/xarray.c:378 func:xas_alloc 545M 36444 mm/slub.c:3059 func:alloc_slab_page 9.8G 2563617 mm/readahead.c:189 func:ractl_alloc_folio 20G 5164004 mm/filemap.c:2012 func:__filemap_get_folioSo I stopped the workload and dropped caches to confirm.
# echo 3 > /proc/sys/vm/drop_caches # smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 33.45% 0.09% 33.36% userspace memory 0.36% 0.16% 0.19% free memory 66.20% 66.20% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 12M 2987 mm/execmem.c:41 func:execmem_vmalloc 12M 3 kernel/dma/pool.c:96 func:atomic_pool_expand 13M 751 mm/slub.c:3061Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
func:alloc_slab_page 16M 8
mm/khugepaged.c:1069
func:alloc_charge_folio 18M 4355mm/memory.c:1190
func:folio_prealloc 24M 6119mm/memory.c:1192
func:folio_prealloc 58M 14784mm/page_ext.c:271
func:alloc_page_ext 61M 15448mm/readahead.c:189
func:ractl_alloc_folio 79M 6726mm/slub.c:3059
func:alloc_slab_page 11G 2674488mm/filemap.c:2012func:__filemap_get_folio
Quoted 5 lines
Maybe narrowing down the "Noncache" caller of
__filemap_get_foliowould help clarify things. (It could be designed that way, and needs other route than dropping-cache to release the memory, just guess....) If you want, you can modify code to split the accounting for__filemap_get_folioaccording to its callers.
Thanks again, I'll add this patch in and see where I end up.
The issue is nothing will cause the memory to be freed. Dropping caches doesn't work, memory pressure doesn't work, unmounting the filesystems doesn't work. Removing the cephfs and netfs kernel modules also doesn't work.
This is why I feel it's a ref_count (or similar) issue.
I've also found it seems to be a fixed amount leaked each time, per file. Simply doing lots of IO on one large file doesn't leak as fast as lots of "small" (greater than 10MB less than 100MB seems to be a sweet spot)
Also, dropping caches while the workload is running actually amplifies the issue. So it very much feels like something is wrong in the reclaim code.
Anyway I'll get this patch applied and see where I end up.
I now have crash dumps (after enabling crash_on_oom) so I'm going to try and see if I can find these structures and see what state they are in
Thanks again.
Mal
Quoted 104 lines
Following is a draft patch: (based on v6.18)
include/linux/pagemap.h +0
-0@@ -753,7 +753,11 @@ static inline fgf_t fgf_set_order(size_t size) } void *filemap_get_entry(struct address_space *mapping, pgoff_tindex); -struct folio *__filemap_get_folio(struct
address_space*mapping,pgoff_tindex, +include/linux/pagemap.h +4
-0+#define __filemap_get_folio (...) \ + alloc_hooks(__filemap_get_folio_noprof(__VA_ARGS__)) + +struct folio *__filemap_get_folio_noprof(struct address_space*mapping,
pgoff_tindex,fgf_tfgp_flags,gfp_tgfp);
struct page *pagecache_get_page(structaddress_space*mapping,pgoff_tindex,fgf_tfgp_flags,gfp_tgfp);mm/filemap.c +0
-1@@ -1938,7 +1938,7 @@ void *filemap_get_entry(struct address_space*mapping, pgoff_t index) * * Return: The found folio or an ERR_PTR() otherwise. */ -struct folio *__filemap_get_folio(struct address_space *mapping,
pgoff_tindex, +struct folio *__filemap_get_folio_noprof(structaddress_space*mapping,pgoff_tindex,fgf_tfgp_flags,gfp_tgfp)
{mm/filemap.c +1
-1struct folio *folio;@@ -2009,7 +2009,7 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, err = -ENOMEM;if (order > min_order) alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN; - folio =filemap_alloc_folio (alloc_gfp, order); + folio =filemap_alloc_folio_noprof (alloc_gfp, order);if (!folio)mm/filemap.c +1
-1continue ;@@ -2056,7 +2056,7 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, folio_clear_dropbehind(folio);return folio; } -EXPORT_SYMBOL (__filemap_get_folio); +EXPORT_SYMBOL (__filemap_get_folio_noprof);static inline struct folio *find_get_entry (struct xa_state *xas,pgoff_t max,xa_mark_t mark)FYI
DavidQuoted 45 lines
Quoted 24 lines
Quoted 3 lines
So if I'm reading this correctly something is causing folios collect and not be able to be freed?
CC cephfs, maybe someone could have an easy reading out of those folio usage
Quoted 16 lines
Also it's clear that some of the folio's are counting as cache and some aren't.
Like I said 6.17 and 6.18 both have the issue. 6.12 does not. I'm now going to manually walk through previous kernel releases and find where it first starts happening purely because I'm having issues building earlier kernels due to rust stuff and other python incompatibilities making doing a git-bisect a bit fun.
I'll do it the packages way until I get closer, then solve the build issues.
Thanks,
MalThanks David.
I've contacted the ceph developers as well.
There was a suggestion it was due to the change from, to quote:
folio.free()tofolio.put()or something like this.The change happened around 6.14/6.15
I've found an easier reproducer.
There has been a suggestion that perhaps the ceph team might not fix this as "you can just reboot before the machine becomes unstable" and "Nobody else has encountered this bug"
I'll leave that to other people to make a call on but I'd assume the lack of reports is due to the fact that most stable distros are still on a a far too early kernel and/or are using the fuse driver with k8s.
Anyway, thanks for your assistance.
RE: RRe: Possible memory leak in 6.17.7
Hi Mal,
Quoted 110 lines · Mal Haak
Quoted 82 lines · On Thu, 11 Dec 2025 11:28:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 72 lines · Mal Haak
Quoted 69 lines · On Tue, 9 Dec 2025 12:40:21 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 68 lines · Mal Haak
Quoted 16 lines · On Mon, 8 Dec 2025 19:08:29 +0800 David Wang <[email protected]> wrote:
Quoted 6 lines · On Mon, 10 Nov 2025 18:20:08 +1000 Mal Haak <[email protected]> wrote:
Hello,
I have found a memory leak in 6.17.7 but I am unsure how to track it down effectively.
I think the
memory allocation profilingfeature can help.
https://docs.kernel.org/mm/allocation-profiling.htmlYou would need to build a kernel with
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yQuoted 9 lines
And check
/proc/allocinfofor the suspicious allocations which take more memory than expected.(I once caught a nvidia driver memory leak.)
FYI
DavidThank you for your suggestion. I have some results.
Ran the rsync workload for about 9 hours. It started to look like it was happening.
# smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 80.46% 65.80% 14.66% userspace memory 0.35% 0.16% 0.19% free memory 19.19% 19.19% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 22M 5609 mm/memory.c:1190 func:folio_prealloc 23M 1932 fs/xfs/xfs_buf.c:226 [xfs] func:xfs_buf_alloc_backing_mem 24M 24135 fs/xfs/xfs_icache.c:97 [xfs] func:xfs_inode_alloc 27M 6693 mm/memory.c:1192 func:folio_prealloc 58M 14784 mm/page_ext.c:271 func:alloc_page_ext 258M 129 mm/khugepaged.c:1069 func:alloc_charge_folio 430M 770788 lib/xarray.c:378 func:xas_alloc 545M 36444 mm/slub.c:3059 func:alloc_slab_page 9.8G 2563617 mm/readahead.c:189 func:ractl_alloc_folio 20G 5164004 mm/filemap.c:2012 func:__filemap_get_folioSo I stopped the workload and dropped caches to confirm.
# echo 3 > /proc/sys/vm/drop_caches # smem -pw Area Used Cache Noncache firmware/hardware 0.00% 0.00% 0.00% kernel image 0.00% 0.00% 0.00% kernel dynamic memory 33.45% 0.09% 33.36% userspace memory 0.36% 0.16% 0.19% free memory 66.20% 66.20% 0.00% # sort -g /proc/allocinfo|tail|numfmt --to=iec 12M 2987 mm/execmem.c:41 func:execmem_vmalloc 12M 3 kernel/dma/pool.c:96 func:atomic_pool_expand 13M 751 mm/slub.c:3061Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
func:alloc_slab_page 16M 8
mm/khugepaged.c:1069
func:alloc_charge_folio 18M 4355mm/memory.c:1190
func:folio_prealloc 24M 6119mm/memory.c:1192
func:folio_prealloc 58M 14784mm/page_ext.c:271
func:alloc_page_ext 61M 15448mm/readahead.c:189
func:ractl_alloc_folio 79M 6726mm/slub.c:3059
func:alloc_slab_page 11G 2674488mm/filemap.c:2012func:__filemap_get_folioQuoted 5 lines
Maybe narrowing down the "Noncache" caller of
__filemap_get_foliowould help clarify things. (It could be designed that way, and needs other route than dropping-cache to release the memory, just guess....) If you want, you can modify code to split the accounting for__filemap_get_folioaccording to its callers.Thanks again, I'll add this patch in and see where I end up.
The issue is nothing will cause the memory to be freed. Dropping caches doesn't work, memory pressure doesn't work, unmounting the filesystems doesn't work. Removing the cephfs and netfs kernel modules also doesn't work.
This is why I feel it's a
ref_count(or similar) issue.I've also found it seems to be a fixed amount leaked each time, per file. Simply doing lots of IO on one large file doesn't leak as fast as lots of "small" (greater than 10MB less than 100MB seems to be a sweet spot)
Also, dropping caches while the workload is running actually amplifies the issue. So it very much feels like something is wrong in the reclaim code.
Quoted 7 lines
Anyway I'll get this patch applied and see where I end up.
I now have crash dumps (after enabling
crash_on_oom) so I'm going to try and see if I can find these structures and see what state they are in
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.
Re: RRe: Possible memory leak in 6.17.7
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.
Hello,
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on addr.c and didn't see any changes post 6.15 that looked like they would impact the bug behavior.
Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in ceph_check_page_before_write where the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.
I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with crash_on_oom it could potentially be used for regression testing by way of "did vm crash?".
I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Regards
Mal Haak
RE: RRe: Possible memory leak in 6.17.7
Quoted 85 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.Hello,
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Quoted 4 lines
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Thanks a lot for of your efforts. I hope it will help a lot. Let me start to reproduce the issue. I'll let you know if I need additional details. I'll share my progress and potential troubles in the ticket that you've created.
Thanks,
Slava.
Quoted 14 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 8 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Quoted 7 lines
Quoted 2 lines
Thanks,
Slava.Hello,
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
Hi, just out of curiosity, have you narrowed down the caller of __filemap_get_folio causing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?
David
Quoted 70 lines
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Regards
Mal Haak
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.Hello,
Quoted 9 lines
Quoted 2 lines
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
Hi, just out of curiosity, have you narrowed down the caller of
__filemap_get_foliocausing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?David
Hi David,
I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.
But actually this could speed things up considerably. I will do that now and see what I get.
Thanks
Mal
Quoted 70 lines
Quoted 70 lines
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Regards
Mal Haak
Quoted 37 lines · On Tue, 16 Dec 2025 17:09:18 +1000 Mal Haak <[email protected]> wrote:
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.Hello,
Quoted 9 lines
Quoted 2 lines
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
Hi, just out of curiosity, have you narrowed down the caller of
__filemap_get_foliocausing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?David
Hi David,
I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.
Quoted 7 lines
But actually this could speed things up considerably. I will do that now and see what I get.
Thanks
Mal
I did just give it a blast.
Unfortunately it returned exactly what I expected, that is the calls are all coming from netfs.
Which makes sense for cephfs.
# sort -g /proc/allocinfo|tail|numfmt --to=iec
10M 2541 drivers/block/zram/zram_drv.c:1597 [zram]
func:zram_meta_alloc 12M 3001 mm/execmem.c:41 func:execmem_vmalloc
12M 3605 kernel/fork.c:311 func:alloc_thread_stack_node
16M 992 mm/slub.c:3061 func:alloc_slab_page
20M 35544 lib/xarray.c:378 func:xas_alloc
31M 7704 mm/memory.c:1192 func:folio_prealloc
69M 17562 mm/memory.c:1190 func:folio_prealloc
104M 8212 mm/slub.c:3059 func:alloc_slab_page
124M 30075 mm/readahead.c:189 func:ractl_alloc_folio
2.6G 661392 fs/netfs/buffered_read.c:635 [netfs]
func:netfs_write_begin So, unfortunately it doesn't reveal the true source. But was worth a shot! So thanks again
Mal
Quoted 73 lines
Quoted 72 lines
Quoted 72 lines
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Regards
Mal Haak
Quoted 67 lines · Mal Haak
Quoted 37 lines · On Tue, 16 Dec 2025 17:09:18 +1000 Mal Haak <[email protected]> wrote:
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.Hello,
Quoted 9 lines
Quoted 2 lines
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
Hi, just out of curiosity, have you narrowed down the caller of
__filemap_get_foliocausing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?David
Hi David,
I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.
Quoted 7 lines
But actually this could speed things up considerably. I will do that now and see what I get.
Thanks
Mal
I did just give it a blast.
Unfortunately it returned exactly what I expected, that is the calls are all coming from netfs.
Which makes sense for cephfs.
# sort -g /proc/allocinfo|tail|numfmt --to=iec 10M 2541 drivers/block/zram/zram_drv.c:1597 [zram] func:zram_meta_alloc 12M 3001 mm/execmem.c:41 func:execmem_vmalloc 12M 3605 kernel/fork.c:311 func:alloc_thread_stack_node 16M 992 mm/slub.c:3061 func:alloc_slab_page 20M 35544 lib/xarray.c:378 func:xas_alloc 31M 7704 mm/memory.c:1192 func:folio_prealloc 69M 17562 mm/memory.c:1190 func:folio_prealloc 104M 8212 mm/slub.c:3059 func:alloc_slab_page 124M 30075 mm/readahead.c:189 func:ractl_alloc_folio 2.6G 661392 fs/netfs/buffered_read.c:635 [netfs] func:netfs_write_begin
Quoted 2 lines
So, unfortunately it doesn't reveal the true source. But was worth a shot! So thanks again
Oh, at least cephfs could be ruled out, right?
CC netfs folks then. :)
Quoted 77 lines
Mal
Quoted 73 lines
Quoted 72 lines
Quoted 72 lines
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Regards
Mal Haak
Quoted 69 lines · David Wang
Quoted 67 lines · Mal Haak
Quoted 37 lines · On Tue, 16 Dec 2025 17:09:18 +1000 Mal Haak <[email protected]> wrote:
Quoted 20 lines · On Tue, 16 Dec 2025 15:00:43 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 19 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.Hello,
Quoted 9 lines
Quoted 2 lines
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
Hi, just out of curiosity, have you narrowed down the caller of
__filemap_get_foliocausing the memory problem? Or do you have trouble applying the debug patch for memory allocation profiling?David
Hi David,
I hadn't yet as I did test XFS and NFS to see if it replicated the behaviour and it did not.
Quoted 7 lines
But actually this could speed things up considerably. I will do that now and see what I get.
Thanks
Mal
I did just give it a blast.
Unfortunately it returned exactly what I expected, that is the calls are all coming from netfs.
Which makes sense for cephfs.
# sort -g /proc/allocinfo|tail|numfmt --to=iec 10M 2541 drivers/block/zram/zram_drv.c:1597 [zram] func:zram_meta_alloc 12M 3001 mm/execmem.c:41 func:execmem_vmalloc 12M 3605 kernel/fork.c:311 func:alloc_thread_stack_node 16M 992 mm/slub.c:3061 func:alloc_slab_page 20M 35544 lib/xarray.c:378 func:xas_alloc 31M 7704 mm/memory.c:1192 func:folio_prealloc 69M 17562 mm/memory.c:1190 func:folio_prealloc 104M 8212 mm/slub.c:3059 func:alloc_slab_page 124M 30075 mm/readahead.c:189 func:ractl_alloc_folio 2.6G 661392 fs/netfs/buffered_read.c:635 [netfs] func:netfs_write_begin
Quoted 4 lines
Quoted 2 lines
So, unfortunately it doesn't reveal the true source. But was worth a shot! So thanks again
Oh, at least cephfs could be ruled out, right?
ehh...., I think I could be wrong about this.....
Quoted 82 lines
CC netfs folks then. :)
Quoted 77 lines
Mal
Quoted 73 lines
Quoted 72 lines
Quoted 72 lines
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Regards
Mal Haak
Hi Mal,
Quoted 3 lines · David Wang
Quoted 2 lines · David Wang
<skipped>
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Could you please add your CephFS kernel client's mount options into the ticket [1]?
Thanks a lot,
Slava.
Quoted 91 lines · On Wed, 17 Dec 2025 01:56:52 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
Quoted 3 lines · David Wang
Quoted 2 lines · David Wang
<skipped>
Quoted 76 lines
Quoted 76 lines
Quoted 76 lines
Quoted 76 lines
Quoted 76 lines
Quoted 76 lines
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Quoted 7 lines
Could you please add your CephFS kernel client's mount options into the ticket [1]?
Thanks a lot,
Slava.
I've updated the ticket.
I am curious about the differences between your test setup and my actual setup in terms of capacity and hardware.
I can provide crash dumps if it is helpful.
Thanks
Mal
Quoted 27 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.Hello,
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
Quoted 6 lines
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.
Hi,
Just a suggestion, in case you run out of idea for further investigation. I think you can bisect *manually* targeting changes of fs/cephfs between 6.14 and 6.15
$ git log --pretty='format:%h %an' v6.14..v6.15 fs/ceph 349b7d77f5a1 Linus Torvalds b261d2222063 Eric Biggers f452a2204614 David Howells e63046adefc0 Linus Torvalds 59b59a943177 Matthew Wilcox (Oracle) <-----------3 efbdd92ed9f6 Matthew Wilcox (Oracle) d1b452673af4 Matthew Wilcox (Oracle) ad49fe2b3d54 Matthew Wilcox (Oracle) a55cf4fd8fae Matthew Wilcox (Oracle) 15fdaf2fd60d Matthew Wilcox (Oracle) 62171c16da60 Matthew Wilcox (Oracle) baff9740bc8f Matthew Wilcox (Oracle) f9707a8b5b9d Matthew Wilcox (Oracle) 88a59bda3f37 Matthew Wilcox (Oracle) 19a288110435 Matthew Wilcox (Oracle) fd7449d937e7 Viacheslav Dubeyko <---------2 1551ec61dc55 Viacheslav Dubeyko ce80b76dd327 Viacheslav Dubeyko f08068df4aa4 Viacheslav Dubeyko 3f92c7b57687 NeilBrown <-----------1 88d5baf69082 NeilBrown
There were 3 major patch set (group by author), the suspect could be narrowed down further.
(Bisect, even over a short range of patch, is quite an unpleasant experience though....)
FYI
David
Quoted 58 lines
Again, I'm not super familiar with the CephFS code but to hazard a guess, but I think that the web download workload triggers things faster suggests that unaligned writes might make things worse. But again, I'm not 100% sure. I can't find a reproducer as fast as downloading a dataset. Rsync of lots and lots of tiny files is a tad faster than the dd case.
I did see some changes in
ceph_check_page_before_writewhere the previous code unlocked pages and then continued where as the changed folio code just returns ENODATA and doesn't unlock anything with most of the rest of the logic unchanged. This might be perfectly fine, but in my, admittedly limited, reading of the code I couldn't figure out where anything that was locked prior to this being called would get unlocked like it did prior to the change. Again, I could be miles off here and one of the bulk reclaim/unlock passes that was added might be cleaning this up correctly or some other functional change might take care of this, but it looks to be potentially in the code path I'm excising and it has had some unlock logic changed.I've spent most of my time trying to find a solid quick reproducer. Not that it takes long to start leaking folios, but I wanted something that aggressively triggered it so a small vm would oom quickly and when combined with
crash_on_oomit could potentially be used for regression testing by way of "did vm crash?".I'm not sure if it will super help, but I'll provide what details I can about the actual workload that really sets it off. It's a python based tool for downloading datasets. Datasets are split into N chunks and the tool downloads them in parallel 100 at a time until all N chunks are down. The compressed dataset is then unpacked and reassembled for use with workloads.
This is replicating a common home folder usecase in HPC. CephFS is very attractive for home folders due to it's "NFS-like" utility and performance. And many tools use a similar method for fetching large datasets. Tools are frequently written in python or go.
None of my customers have hit this yet, not have any enterprise customers as none have moved to a new enough kernel yet due to slow upgrade cycles. Even Proxmox have only just started testing on a kernel version > 6.14.
I'm more than happy to help however I can with testing. I can run instrumented kernels or test patches or whatever you need. I am sorry I haven't been able to produce a super clean, fast reproducer (my test cluster at home is all spinners and only 500TB usable). But I figured I needed to get the word out asap as distros and soon customers are going to be moving past 6.12-6.14 kernels as the 5-7 year update cycle marches on. Especially those wanting to take full advantage of CacheFS and encryption functionality.
Again thanks for looking at this and do reach out if I can help in anyway. I am in the ceph slack if it's faster to reach out that way.
Regards
Mal Haak
Quoted 68 lines · On Wed, 17 Dec 2025 13:59:47 +0800 (CST) "David Wang" <[email protected]> wrote:
Quoted 27 lines · Mal Haak
Quoted 2 lines · On Mon, 15 Dec 2025 19:42:56 +0000 Viacheslav Dubeyko <[email protected]>…
Hi Mal,
<SNIP>
Quoted 10 lines
Thanks a lot for reporting the issue. Finally, I can see the discussion in email list. :) Are you working on the patch with the fix? Should we wait for the fix or I need to start the issue reproduction and investigation? I am simply trying to avoid patches collision and, also, I have multiple other issues for the fix in CephFS kernel client. :)
Thanks,
Slava.Hello,
Unfortunately creating a patch is just outside my comfort zone, I've lived too long in Lustre land.
I've have been trying to narrow down a consistent reproducer that's as fast as my production workload. (It crashes a 32GB VM in 2hrs) And I haven't got it quite as fast. I think the dd workload is too well behaved.
Quoted 6 lines
I can confirm the issue appeared in the major patch set that was applied as part of the 6.15 kernel. So during the more complete pages to folios switch and that nothing has changed in the bug behaviour since then. I did have a look at all the diffs from 6.14 to 6.18 on
addr.cand didn't see any changes post 6.15 that looked like they would impact the bug behavior.Hi,
Just a suggestion, in case you run out of idea for further investigation. I think you can bisect *manually* targeting changes of fs/cephfs between 6.14 and 6.15$ git log --pretty='format:%h %an' v6.14..v6.15 fs/ceph 349b7d77f5a1 Linus Torvalds b261d2222063 Eric Biggers f452a2204614 David Howells e63046adefc0 Linus Torvalds 59b59a943177 Matthew Wilcox (Oracle) <-----------3 efbdd92ed9f6 Matthew Wilcox (Oracle) d1b452673af4 Matthew Wilcox (Oracle) ad49fe2b3d54 Matthew Wilcox (Oracle) a55cf4fd8fae Matthew Wilcox (Oracle) 15fdaf2fd60d Matthew Wilcox (Oracle) 62171c16da60 Matthew Wilcox (Oracle) baff9740bc8f Matthew Wilcox (Oracle) f9707a8b5b9d Matthew Wilcox (Oracle) 88a59bda3f37 Matthew Wilcox (Oracle) 19a288110435 Matthew Wilcox (Oracle) fd7449d937e7 Viacheslav Dubeyko <---------2 1551ec61dc55 Viacheslav Dubeyko ce80b76dd327 Viacheslav Dubeyko f08068df4aa4 Viacheslav Dubeyko 3f92c7b57687 NeilBrown <-----------1 88d5baf69082 NeilBrownThere were 3 major patch set (group by author), the suspect could be narrowed down further.
Quoted 7 lines
(Bisect, even over a short range of patch, is quite an unpleasant experience though....)
FYI
DavidQuoted 1 line
<SNIP>
Yeah, I don't think it's a small patch that is the cause of the issue.
It looks like there was a patch set that migrated cephfs off handling individual pages and onto folios to enable wider use of netfs features like local caching and encryption, as examples. I'm not sure that set can be broken up and result in a working cephfs module. Which limits the utility of a git-bisect. I'm pretty sure the issue is in addr.c somewhere and most of the changes in there are one patch. That said, after I get this crash dump I'll probably do it anyway.
What I really need to do is get a crash dump to look at what state the folios and their tracking is in. Assuming I can grok what I'm looking at. This is the bit I'm most apprehensive of. I'm hoping I can find a list of folios used by the reclaim machinery that is missing a bunch of folios. That or a bunch with inflated refcounts or something.
Something is going awry, but it's not fast. I thought I had a quick reproducer. I was wrong, I sized the DD workload incorrectly and triggered the panic_on_oom due to that, not the bug.
I'm re-running the reproducer now, on a VM with 2GB of ram and its been running for around 3hrs and I think at most its leaked possibly 100MB-150MB of ram at most. (It was averaging 190-200MB of noncache usage. It's now averaging 290-340MB).
It does accelerate. The more folios that are in the weird state, the more end up in the weird state. Which feels like fragmentation side effects, but I'm just speculating.
Anyway, one of the wonderful ceph developers is looking into it. I just hope I can do enough to help them locate the issue. They are having troubles reproducing last I heard from them but they might have been expecting a slightly faster reproducer.
I have however recreated it on a physical host not just a vm. So I feel like I can rule out being a VM as a cause.
Anyway thanks for your continued assistance!
Mal