mm/show_mem: Bug fix for print mem alloc info
Subsystems (2)
cover mm/show_mem: Bug fix for print mem alloc info
This patch set fixes two issues we saw in production rollout.
The first issue is that we saw all zero output of memory allocation profiling information from show_mem() if CONFIG_MEM_ALLOC_PROFILING is set and sysctl.vm.mem_profiling=0. In this case, the behaviour should be the same as when CONFIG_MEM_ALLOC_PROFILING is unset, where show_mem prints nothing about the information. This will make further parse easier as we don't have to differentiate what a all zero line actually means (Does it mean 0 bytes are allocated or simply memory allocation profiling is disabled).
The second issue is that multiple entities can call show_mem() which messed up the allocation info in dmesg. We saw outputs like this:
```
327 MiB 83635 mm/compaction.c:1880 func:compaction_alloc
48.4 GiB 12684937 mm/memory.c:1061 func:folio_prealloc
7.48 GiB 10899 mm/huge_memory.c:1159 func:vma_alloc_anon_folio_pmd
298 MiB 95216 kernel/fork.c:318 func:alloc_thread_stack_node
250 MiB 63901 mm/zsmalloc.c:987 func:alloc_zspage
1.42 GiB 372527 mm/memory.c:1063 func:folio_prealloc
1.17 GiB 95693 mm/slub.c:2424 func:alloc_slab_page
651 MiB 166732 mm/readahead.c:270 func:page_cache_ra_unbounded
419 MiB 107261 net/core/page_pool.c:572 func:__page_pool_alloc_pages_slow
404 MiB 103425 arch/x86/mm/pgtable.c:25 func:pte_alloc_one
```The above example is because one kthread invokes show_mem() from __alloc_pages_slowpath while kernel itself calls oom_kill_process()
Yueyang Pan (2):
mm/show_mem: No print when not mem_alloc_profiling_enabled() mm/show_mem: Add trylock while printing alloc info
mm/show_mem.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Signature · 1 line
2.47.3
PATCH 1/2 mm/show_mem: No print when not mem_alloc_profiling_enabled()
This patch makes print kernel memory allocation information controlled by mem_alloc_profiling_enabled() so that we won't see all zero numbers in production.
--- mm/show_mem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/show_mem.c +1 -1
@@ -419,7 +419,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif #ifdef CONFIG_MEM_ALLOC_PROFILING - { +if (mem_alloc_profiling_enabled ()) {struct codetag_bytes tags[10 ];size_t i, nr;
Signature · 1 line
2.47.3
PATCH 2/2 mm/show_mem: Add trylock while printing alloc info
In reply to Yueyang PanIn production, show_mem() can be called concurrently from two different entities, for example one from oom_kill_process() another from __alloc_pages_slowpath from another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info in show_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.
--- mm/show_mem.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/show_mem.c +4 -1
@@ -23,6 +23,8 @@ EXPORT_SYMBOL(_totalram_pages);unsigned long totalreserve_pages__read_mostly ;unsigned long totalcma_pages__read_mostly ; +static DEFINE_MUTEX (mem_alloc_profiling_mutex); +static inline void show_node (struct zone *zone) {if (IS_ENABLED (CONFIG_NUMA ))
@@ -419,7 +421,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -if (mem_alloc_profiling_enabled ()) { +if (mem_alloc_profiling_enabled () &&mutex_trylock (&mem_alloc_profiling_mutex)) {struct codetag_bytes tags[10 ];size_t i, nr;
@@ -445,6 +447,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) ct->lineno, ct->function); } } +mutex_unlock (&mem_alloc_profiling_mutex); }#endif }
Signature · 1 line
2.47.3
Quoted 6 lines · Yueyang Pan
This patch set fixes two issues we saw in production rollout.
The first issue is that we saw all zero output of memory allocation profiling information from
show_mem()ifCONFIG_MEM_ALLOC_PROFILINGis set and sysctl.vm.mem_profiling=0. In this case, the behaviour should be the same as whenCONFIG_MEM_ALLOC_PROFILINGis unset,
Did you mean to say when sysctl.vm.mem_profiling=never?
My understanding is that setting the sysctl=0 Pauses memory allocation profiling, while 1 Resumes it. When the sysctl=never should be the same as when the config is unset, but I suspect we might still want the info when set to 0.
Quoted 19 lines
where
show_memprints nothing about the information. This will make further parse easier as we don't have to differentiate what a all zero line actually means (Does it mean 0 bytes are allocated or simply memory allocation profiling is disabled).The second issue is that multiple entities can call
show_mem()which messed up the allocation info in dmesg. We saw outputs like this:``` 327 MiB 83635 mm/compaction.c:1880 func:compaction_alloc 48.4 GiB 12684937 mm/memory.c:1061 func:folio_prealloc 7.48 GiB 10899 mm/huge_memory.c:1159 func:vma_alloc_anon_folio_pmd 298 MiB 95216 kernel/fork.c:318 func:alloc_thread_stack_node 250 MiB 63901 mm/zsmalloc.c:987 func:alloc_zspage 1.42 GiB 372527 mm/memory.c:1063 func:folio_prealloc 1.17 GiB 95693 mm/slub.c:2424 func:alloc_slab_page 651 MiB 166732 mm/readahead.c:270 func:page_cache_ra_unbounded 419 MiB 107261 net/core/page_pool.c:572 func:__page_pool_alloc_pages_slow 404 MiB 103425 arch/x86/mm/pgtable.c:25 func:pte_alloc_one ```
Quoted 3 lines
The above example is because one kthread invokes
show_mem()from__alloc_pages_slowpathwhile kernel itself callsoom_kill_process()
I'm not familiar with show_mem(). Could you spell out what's wrong with the output above?
Quoted 10 lines
Yueyang Pan (2):
mm/show_mem: No print when notmem_alloc_profiling_enabled()mm/show_mem: Add trylock while printing alloc infomm/show_mem.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)Signature · 1 line
2.47.3
Andrew Morton
maintainerRe: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
In reply to Yueyang PanQuoted 7 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.
Fair enough, I guess.
mm/show_mem.c EXPORT_SYMBOL(_totalram_pages); · line 26
Quoted 7 lines
mm/show_mem.c +1
-0@@ -23,6 +23,8 @@ EXPORT_SYMBOL(_totalram_pages);unsigned long totalreserve_pages__read_mostly ;unsigned long totalcma_pages__read_mostly ; +static DEFINE_MUTEX (mem_alloc_profiling_mutex);
It would be a bit neater to make this local to __show_mem() - it didn't need file scope.
Also, mutex_unlock() isn't to be used from interrupt context, so problem.
Something like atomic cmpxchg or test_and_set_bit could be used and wouldn't involve mutex_unlock()'s wakeup logic, which isn't needed here.
Quoted 13 lines
mm/show_mem.c +1
-1static inline void show_node (struct zone *zone) {if (IS_ENABLED (CONFIG_NUMA ))@@ -419,7 +421,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -if (mem_alloc_profiling_enabled ()) { +if (mem_alloc_profiling_enabled () &&mutex_trylock (&mem_alloc_profiling_mutex)) {struct codetag_bytes tags[10 ];size_t i, nr;
mm/show_mem.c void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) · line 451
Quoted 5 lines
mm/show_mem.c +1
-0@@ -445,6 +447,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) ct->lineno, ct->function); } } +mutex_unlock (&mem_alloc_profiling_mutex); }
If we're going to suppress the usual output then how about we let people know this happened, rather than silently dropping it?
pr_notice("memory allocation output suppressed due to show_mem() contention\n")
or something like that?
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
Quoted 18 lines · Andrew Morton
Quoted 7 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.Fair enough, I guess.
Quoted 6 lines
mm/show_mem.c +0
-0@@ -23,6 +23,8 @@ EXPORT_SYMBOL(_totalram_pages);unsigned long totalreserve_pages__read_mostly ;unsigned long totalcma_pages__read_mostly ;
Quoted 4 lines
mm/show_mem.c EXPORT_SYMBOL(_totalram_pages); · line 26
Quoted 1 line
mm/show_mem.c +1
-0+static DEFINE_MUTEX (mem_alloc_profiling_mutex);It would be a bit neater to make this local to
__show_mem()- it didn't need file scope.
+1, something static to __show_mem().
Quoted 7 lines
Also,
mutex_unlock()isn't to be used from interrupt context, so problem.Something like atomic cmpxchg or
test_and_set_bitcould be used and wouldn't involve mutex_unlock()'s wakeup logic, which isn't needed here.
+1
Quoted 23 lines
Quoted 14 lines
mm/show_mem.c +1
-1static inline void show_node (struct zone *zone) {if (IS_ENABLED (CONFIG_NUMA ))@@ -419,7 +421,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -if (mem_alloc_profiling_enabled ()) { +if (mem_alloc_profiling_enabled () &&mutex_trylock (&mem_alloc_profiling_mutex)) {struct codetag_bytes tags[10 ];size_t i, nr;mm/show_mem.c void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) · line 451
Quoted 5 lines
mm/show_mem.c +1
-0@@ -445,6 +447,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) ct->lineno, ct->function); } } +mutex_unlock (&mem_alloc_profiling_mutex); }If we're going to suppress the usual output then how about we let people know this happened, rather than silently dropping it?
Quoted 3 lines
pr_notice("memory allocation output suppressed due to show_mem() contention\n")or something like that?
Personally I think this is not needed as this patch is suppressing only the memory allocation profiling output which is global, will be same for all the consumers and context does not matter. All consumers will get the memory allocation profiling data eventually.
Quoted 8 lines · Vishal Moola (Oracle)
Quoted 6 lines · Yueyang Pan
This patch set fixes two issues we saw in production rollout.
The first issue is that we saw all zero output of memory allocation profiling information from
show_mem()ifCONFIG_MEM_ALLOC_PROFILINGis set and sysctl.vm.mem_profiling=0. In this case, the behaviour should be the same as whenCONFIG_MEM_ALLOC_PROFILINGis unset,
Quoted 6 lines
Did you mean to say when sysctl.vm.mem_profiling=never?
My understanding is that setting the sysctl=0 Pauses memory allocation profiling, while 1 Resumes it. When the sysctl=never should be the same as when the config is unset, but I suspect we might still want the info when set to 0.
Thanks for your feedback Vishal. Here I mean for both =0 and =never. In both cases, now __show_mem() will print all 0s, which both is redundant and also makes differentiate hard. IMO when __show_mem() prints something the output should be useful at least.
Quoted 24 lines
Quoted 20 lines
where
show_memprints nothing about the information. This will make further parse easier as we don't have to differentiate what a all zero line actually means (Does it mean 0 bytes are allocated or simply memory allocation profiling is disabled).The second issue is that multiple entities can call
show_mem()which messed up the allocation info in dmesg. We saw outputs like this:``` 327 MiB 83635 mm/compaction.c:1880 func:compaction_alloc 48.4 GiB 12684937 mm/memory.c:1061 func:folio_prealloc 7.48 GiB 10899 mm/huge_memory.c:1159 func:vma_alloc_anon_folio_pmd 298 MiB 95216 kernel/fork.c:318 func:alloc_thread_stack_node 250 MiB 63901 mm/zsmalloc.c:987 func:alloc_zspage 1.42 GiB 372527 mm/memory.c:1063 func:folio_prealloc 1.17 GiB 95693 mm/slub.c:2424 func:alloc_slab_page 651 MiB 166732 mm/readahead.c:270 func:page_cache_ra_unbounded 419 MiB 107261 net/core/page_pool.c:572 func:__page_pool_alloc_pages_slow 404 MiB 103425 arch/x86/mm/pgtable.c:25 func:pte_alloc_one ```Quoted 3 lines
The above example is because one kthread invokes
show_mem()from__alloc_pages_slowpathwhile kernel itself callsoom_kill_process()
Quoted 2 lines
I'm not familiar with
show_mem(). Could you spell out what's wrong with the output above?
So here in the normal case, the output should be sorted by size. Here two print happen at the same time so they interleave with each other, making further parse harder (need to sort again and dedup).
Quoted 11 lines
Quoted 11 lines
Yueyang Pan (2):
mm/show_mem: No print when notmem_alloc_profiling_enabled()mm/show_mem: Add trylock while printing alloc infomm/show_mem.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)Signature · 1 line
2.47.3
Thanks,
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
In reply to Andrew MortonQuoted 18 lines · Andrew Morton
Quoted 7 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.Fair enough, I guess.
Quoted 6 lines
mm/show_mem.c +0
-0@@ -23,6 +23,8 @@ EXPORT_SYMBOL(_totalram_pages);unsigned long totalreserve_pages__read_mostly ;unsigned long totalcma_pages__read_mostly ;
Quoted 4 lines
mm/show_mem.c EXPORT_SYMBOL(_totalram_pages); · line 26
Quoted 1 line
mm/show_mem.c +1
-0+static DEFINE_MUTEX (mem_alloc_profiling_mutex);It would be a bit neater to make this local to
__show_mem()- it didn't need file scope.
Thanks for your feedback, Andrew. I will move it the next version.
Quoted 7 lines
Also,
mutex_unlock()isn't to be used from interrupt context, so problem.Something like atomic cmpxchg or
test_and_set_bitcould be used and wouldn't involve mutex_unlock()'s wakeup logic, which isn't needed here.
I was not aware of interrupt context before. I will change to test-and-set lock in the next version.
Quoted 23 lines
Quoted 14 lines
mm/show_mem.c +1
-1static inline void show_node (struct zone *zone) {if (IS_ENABLED (CONFIG_NUMA ))@@ -419,7 +421,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -if (mem_alloc_profiling_enabled ()) { +if (mem_alloc_profiling_enabled () &&mutex_trylock (&mem_alloc_profiling_mutex)) {struct codetag_bytes tags[10 ];size_t i, nr;mm/show_mem.c void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) · line 451
Quoted 5 lines
mm/show_mem.c +1
-0@@ -445,6 +447,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) ct->lineno, ct->function); } } +mutex_unlock (&mem_alloc_profiling_mutex); }If we're going to suppress the usual output then how about we let people know this happened, rather than silently dropping it?
Quoted 3 lines
pr_notice("memory allocation output suppressed due to show_mem() contention\n")or something like that?
For this point, I am sort of on Shakeel's side. Probably I won't call it suppressed as two concurrent printers is actually sharing this global information.
Thanks,
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
In reply to Shakeel ButtQuoted 19 lines · Shakeel Butt
Quoted 18 lines · Andrew Morton
Quoted 7 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.Fair enough, I guess.
Quoted 6 lines
mm/show_mem.c +0
-0@@ -23,6 +23,8 @@ EXPORT_SYMBOL(_totalram_pages);unsigned long totalreserve_pages__read_mostly ;unsigned long totalcma_pages__read_mostly ;
Quoted 6 lines
Quoted 4 lines
mm/show_mem.c EXPORT_SYMBOL(_totalram_pages); · line 26
Quoted 1 line
mm/show_mem.c +1
-0+static DEFINE_MUTEX (mem_alloc_profiling_mutex);It would be a bit neater to make this local to
__show_mem()- it didn't need file scope.+1, something static to
__show_mem().
Thanks for your feedback, Shakeel. See my reply to Andrew for this.
Quoted 10 lines
Quoted 8 lines
Also,
mutex_unlock()isn't to be used from interrupt context, so problem.Something like atomic cmpxchg or
test_and_set_bitcould be used and wouldn't involve mutex_unlock()'s wakeup logic, which isn't needed here.+1
Again, see my reply to Andrew.
Quoted 24 lines
Quoted 24 lines
Quoted 15 lines
mm/show_mem.c +1
-1static inline void show_node (struct zone *zone) {if (IS_ENABLED (CONFIG_NUMA ))@@ -419,7 +421,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)printk ("%lu pages hwpoisoned\n" ,atomic_long_read (&num_poisoned_pages));#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -if (mem_alloc_profiling_enabled ()) { +if (mem_alloc_profiling_enabled () &&mutex_trylock (&mem_alloc_profiling_mutex)) {struct codetag_bytes tags[10 ];size_t i, nr;mm/show_mem.c void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) · line 451
Quoted 5 lines
mm/show_mem.c +1
-0@@ -445,6 +447,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) ct->lineno, ct->function); } } +mutex_unlock (&mem_alloc_profiling_mutex); }If we're going to suppress the usual output then how about we let people know this happened, rather than silently dropping it?
Quoted 8 lines
Quoted 3 lines
pr_notice("memory allocation output suppressed due to show_mem() contention\n")or something like that?
Personally I think this is not needed as this patch is suppressing only the memory allocation profiling output which is global, will be same for all the consumers and context does not matter. All consumers will get the memory allocation profiling data eventually.
For this point, I sort of agree with you. Wait for others' opinions?
Thanks
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
In reply to Yueyang PanQuoted 9 lines · Yueyang Pan
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.
What about the rest of the information printed by show_mem() being interleaved?
Quoted 21 lines
Quoted 10 lines
Quoted 1 line
Fair enough, I guess.
Quoted 6 lines
mm/show_mem.c +0
-0@@ -23,6 +23,8 @@ EXPORT_SYMBOL(_totalram_pages);unsigned long totalreserve_pages__read_mostly ;unsigned long totalcma_pages__read_mostly ;Quoted 4 lines
mm/show_mem.c EXPORT_SYMBOL(_totalram_pages); · line 26
Quoted 1 line
mm/show_mem.c +1
-0+static DEFINE_MUTEX (mem_alloc_profiling_mutex);It would be a bit neater to make this local to
__show_mem()- it didn't need file scope.Thanks for your feedback, Andrew. I will move it the next version.
Quoted 4 lines
Also,
mutex_unlock()isn't to be used from interrupt context, so problem.
Quoted 6 lines
Quoted 3 lines
Something like atomic cmpxchg or
test_and_set_bitcould be used and wouldn't involve mutex_unlock()'s wakeup logic, which isn't needed here.I was not aware of interrupt context before. I will change to test-and-set lock in the next version.
Perhaps simply spinlock_t with spin_trylock()?
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
Quoted 4 lines · Vlastimil Babka
Quoted 3 lines · Yueyang Pan
Quoted 2 lines · Andrew Morton
On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.What about the rest of the information printed by
show_mem()being interleaved?
Thanks for your feedback, Vlastimil. We cannot use trylock for the rest part as node filter can be different.
Do you think we need a lock to prevent the whole show_mem() from being interleaved and to acquire it at the very beginning? Will it be too heavy?
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 2 lines
Fair enough, I guess.
Quoted 6 lines
mm/show_mem.c +0
-0@@ -23,6 +23,8 @@ EXPORT_SYMBOL(_totalram_pages);unsigned long totalreserve_pages__read_mostly ;unsigned long totalcma_pages__read_mostly ;Quoted 4 lines
mm/show_mem.c EXPORT_SYMBOL(_totalram_pages); · line 26
Quoted 1 line
mm/show_mem.c +1
-0+static DEFINE_MUTEX (mem_alloc_profiling_mutex);It would be a bit neater to make this local to
__show_mem()- it didn't need file scope.Thanks for your feedback, Andrew. I will move it the next version.
Quoted 4 lines
Also,
mutex_unlock()isn't to be used from interrupt context, so problem.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Something like atomic cmpxchg or
test_and_set_bitcould be used and wouldn't involve mutex_unlock()'s wakeup logic, which isn't needed here.I was not aware of interrupt context before. I will change to test-and-set lock in the next version.
Perhaps simply
spinlock_twithspin_trylock()?
Agreed.
Thanks
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
Quoted 12 lines · Yueyang Pan
Quoted 11 lines · Vlastimil Babka
Quoted 9 lines · Yueyang Pan
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.
Quoted 4 lines
Quoted 1 line
What about the rest of the information printed by
show_mem()being interleaved?Thanks for your feedback, Vlastimil. We cannot use trylock for the rest part as node filter can be different.
Right.
Quoted 3 lines
Do you think we need a lock to prevent the whole
show_mem()from being interleaved and to acquire it at the very beginning? Will it be too heavy?
It might be risky so perhaps let's not. Guess we can disentangle by dmesg showing the thread id prefix.
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
Quoted 18 lines · Vlastimil Babka
Quoted 12 lines · Yueyang Pan
Quoted 11 lines · Vlastimil Babka
Quoted 9 lines · Yueyang Pan
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.Quoted 4 lines
Quoted 1 line
What about the rest of the information printed by
show_mem()being interleaved?Thanks for your feedback, Vlastimil. We cannot use trylock for the rest part as node filter can be different.
Quoted 8 lines
Right.
Quoted 3 lines
Do you think we need a lock to prevent the whole
show_mem()from being interleaved and to acquire it at the very beginning? Will it be too heavy?It might be risky so perhaps let's not. Guess we can disentangle by dmesg showing the thread id prefix.
I have thought about this. Since each line can interleave with another, we would end up adding tid to each line. Not sure if this is acceptable.
Thanks
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
Quoted 25 lines · Yueyang Pan
Quoted 24 lines · Vlastimil Babka
Quoted 12 lines · Yueyang Pan
Quoted 11 lines · Vlastimil Babka
Quoted 9 lines · Yueyang Pan
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.Quoted 4 lines
Quoted 1 line
What about the rest of the information printed by
show_mem()being interleaved?Thanks for your feedback, Vlastimil. We cannot use trylock for the rest part as node filter can be different.
Right.
Quoted 3 lines
Do you think we need a lock to prevent the whole
show_mem()from being interleaved and to acquire it at the very beginning? Will it be too heavy?
Quoted 5 lines
Quoted 2 lines
It might be risky so perhaps let's not. Guess we can disentangle by dmesg showing the thread id prefix.
I have thought about this. Since each line can interleave with another, we would end up adding tid to each line. Not sure if this is acceptable.
I meant that printk/dmesg already does that so it's fine.
Quoted 2 lines
Thanks
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
In reply to Vlastimil BabkaQuoted 6 lines · Vlastimil Babka
Quoted 3 lines
I was not aware of interrupt context before. I will change to test-and-set lock in the next version.
Perhaps simply
spinlock_twithspin_trylock()?
Will lockdep complain that this spinlock is taken in non-irq and irq context without disabling irqs?
Quoted 16 lines · Yueyang Pan
Quoted 8 lines · Vishal Moola (Oracle)
Quoted 6 lines · Yueyang Pan
This patch set fixes two issues we saw in production rollout.
The first issue is that we saw all zero output of memory allocation profiling information from
show_mem()ifCONFIG_MEM_ALLOC_PROFILINGis set and sysctl.vm.mem_profiling=0. In this case, the behaviour should be the same as whenCONFIG_MEM_ALLOC_PROFILINGis unset,Quoted 6 lines
Did you mean to say when sysctl.vm.mem_profiling=never?
My understanding is that setting the sysctl=0 Pauses memory allocation profiling, while 1 Resumes it. When the sysctl=never should be the same as when the config is unset, but I suspect we might still want the info when set to 0.
Quoted 4 lines
Thanks for your feedback Vishal. Here I mean for both =0 and =never. In both cases, now
__show_mem()will print all 0s, which both is redundant and also makes differentiate hard. IMO when__show_mem()prints something the output should be useful at least.
If differentiating between 0 allocations vs disabled is the primary concern, I think prefacing the dump with the status of the tool is better than treating =0 and =never as the same.
The way I see it, the {0,1,never} tristate offers a level of versatility that I'm not sure we need to eliminate.
I'm thinking about cases where we may temporarily set =1 to track some allocations, then back to =0 'pause' on that exact period of time. Memory allocation profiling still has those allocations tracked while set to =0 (we can still see them in /proc/allocinfo at least). If a user decided to do that just before an oom, could they see something useful from show_mem() even when =0?
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
where
show_memprints nothing about the information. This will make further parse easier as we don't have to differentiate what a all zero line actually means (Does it mean 0 bytes are allocated or simply memory allocation profiling is disabled).The second issue is that multiple entities can call
show_mem()which messed up the allocation info in dmesg. We saw outputs like this:``` 327 MiB 83635 mm/compaction.c:1880 func:compaction_alloc 48.4 GiB 12684937 mm/memory.c:1061 func:folio_prealloc 7.48 GiB 10899 mm/huge_memory.c:1159 func:vma_alloc_anon_folio_pmd 298 MiB 95216 kernel/fork.c:318 func:alloc_thread_stack_node 250 MiB 63901 mm/zsmalloc.c:987 func:alloc_zspage 1.42 GiB 372527 mm/memory.c:1063 func:folio_prealloc 1.17 GiB 95693 mm/slub.c:2424 func:alloc_slab_page 651 MiB 166732 mm/readahead.c:270 func:page_cache_ra_unbounded 419 MiB 107261 net/core/page_pool.c:572 func:__page_pool_alloc_pages_slow 404 MiB 103425 arch/x86/mm/pgtable.c:25 func:pte_alloc_one ```The above example is because one kthread invokes
show_mem()from__alloc_pages_slowpathwhile kernel itself callsoom_kill_process()
Quoted 6 lines
Quoted 2 lines
I'm not familiar with
show_mem(). Could you spell out what's wrong with the output above?So here in the normal case, the output should be sorted by size. Here two print happen at the same time so they interleave with each other, making further parse harder (need to sort again and dedup).
Gotcha.
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
Yueyang Pan (2):
mm/show_mem: No print when notmem_alloc_profiling_enabled()mm/show_mem: Add trylock while printing alloc infomm/show_mem.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)Signature · 1 line
2.47.3Thanks,
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
In reply to Shakeel ButtQuoted 10 lines · Shakeel Butt
Quoted 6 lines · Vlastimil Babka
Quoted 3 lines
I was not aware of interrupt context before. I will change to test-and-set lock in the next version.
Perhaps simply
spinlock_twithspin_trylock()?Will lockdep complain that this spinlock is taken in non-irq and irq context without disabling irqs?
If we only use spin_trylock then it won't.
Quoted 29 lines · Vishal Moola (Oracle)
Quoted 16 lines · Yueyang Pan
Quoted 8 lines · Vishal Moola (Oracle)
Quoted 6 lines · Yueyang Pan
This patch set fixes two issues we saw in production rollout.
The first issue is that we saw all zero output of memory allocation profiling information from
show_mem()ifCONFIG_MEM_ALLOC_PROFILINGis set and sysctl.vm.mem_profiling=0. In this case, the behaviour should be the same as whenCONFIG_MEM_ALLOC_PROFILINGis unset,Quoted 6 lines
Did you mean to say when sysctl.vm.mem_profiling=never?
My understanding is that setting the sysctl=0 Pauses memory allocation profiling, while 1 Resumes it. When the sysctl=never should be the same as when the config is unset, but I suspect we might still want the info when set to 0.
Quoted 4 lines
Thanks for your feedback Vishal. Here I mean for both =0 and =never. In both cases, now
__show_mem()will print all 0s, which both is redundant and also makes differentiate hard. IMO when__show_mem()prints something the output should be useful at least.If differentiating between 0 allocations vs disabled is the primary concern, I think prefacing the dump with the status of the tool is better than treating =0 and =never as the same.
The way I see it, the {0,1,never} tristate offers a level of versatility that I'm not sure we need to eliminate.
Quoted 6 lines
I'm thinking about cases where we may temporarily set =1 to track some allocations, then back to =0 'pause' on that exact period of time. Memory allocation profiling still has those allocations tracked while set to =0 (we can still see them in
/proc/allocinfoat least). If a user decided to do that just before an oom, could they see something useful fromshow_mem()even when =0?
This is a good point. I agree with your suggestion about adding the state to print. I am still unsure about if we want to print it when =0. The first reason is that memory allocation profiler does not support runtime enabling now. We have to set it via boot cmdline. It will make more sense if we have this feature. Second is because memory allocation profiling is quite light-weighted, I would assume user really don't need this feature when they set =0.
The original reason why I tried to disable this is because in our production table we see a lot of 0Bs coming from the machines where the changes in boot cmdline have not been pushed to. If we have state info, we could possibly filter this info out before sending it to the table. So I agree upon adding the state to print.
Maybe others also have thoughts about this?
Quoted 48 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
where
show_memprints nothing about the information. This will make further parse easier as we don't have to differentiate what a all zero line actually means (Does it mean 0 bytes are allocated or simply memory allocation profiling is disabled).The second issue is that multiple entities can call
show_mem()which messed up the allocation info in dmesg. We saw outputs like this:``` 327 MiB 83635 mm/compaction.c:1880 func:compaction_alloc 48.4 GiB 12684937 mm/memory.c:1061 func:folio_prealloc 7.48 GiB 10899 mm/huge_memory.c:1159 func:vma_alloc_anon_folio_pmd 298 MiB 95216 kernel/fork.c:318 func:alloc_thread_stack_node 250 MiB 63901 mm/zsmalloc.c:987 func:alloc_zspage 1.42 GiB 372527 mm/memory.c:1063 func:folio_prealloc 1.17 GiB 95693 mm/slub.c:2424 func:alloc_slab_page 651 MiB 166732 mm/readahead.c:270 func:page_cache_ra_unbounded 419 MiB 107261 net/core/page_pool.c:572 func:__page_pool_alloc_pages_slow 404 MiB 103425 arch/x86/mm/pgtable.c:25 func:pte_alloc_one ```The above example is because one kthread invokes
show_mem()from__alloc_pages_slowpathwhile kernel itself callsoom_kill_process()Quoted 6 lines
Quoted 2 lines
I'm not familiar with
show_mem(). Could you spell out what's wrong with the output above?So here in the normal case, the output should be sorted by size. Here two print happen at the same time so they interleave with each other, making further parse harder (need to sort again and dedup).
Gotcha.
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
Yueyang Pan (2):
mm/show_mem: No print when notmem_alloc_profiling_enabled()mm/show_mem: Add trylock while printing alloc infomm/show_mem.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)Signature · 1 line
2.47.3Thanks,
Pan
Re: [PATCH v1 2/2] mm/show_mem: Add trylock while printing alloc info
In reply to Vlastimil BabkaQuoted 26 lines · Vlastimil Babka
Quoted 25 lines · Yueyang Pan
Quoted 24 lines · Vlastimil Babka
Quoted 12 lines · Yueyang Pan
Quoted 11 lines · Vlastimil Babka
Quoted 9 lines · Yueyang Pan
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Wed, 27 Aug 2025 11:34:23 -0700 Yueyang Pan <[email protected]> wrote:
In production,
show_mem()can be called concurrently from two different entities, for example one fromoom_kill_process()another from__alloc_pages_slowpathfrom another kthread. This patch adds a mutex and invokes trylock before printing out the kernel alloc info inshow_mem(). This way two alloc info won't interleave with each other, which then makes parsing easier.Quoted 4 lines
Quoted 1 line
What about the rest of the information printed by
show_mem()being interleaved?Thanks for your feedback, Vlastimil. We cannot use trylock for the rest part as node filter can be different.
Right.
Quoted 3 lines
Do you think we need a lock to prevent the whole
show_mem()from being interleaved and to acquire it at the very beginning? Will it be too heavy?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
It might be risky so perhaps let's not. Guess we can disentangle by dmesg showing the thread id prefix.
I have thought about this. Since each line can interleave with another, we would end up adding tid to each line. Not sure if this is acceptable.
I meant that printk/dmesg already does that so it's fine.
Cool. Then I will do this for the previous part before memory allocation info.
Quoted 4 lines
Quoted 3 lines
Thanks
Pan