Try to add memory allocation info for cgroup oom kill
Subsystems (2)
cover Try to add memory allocation info for cgroup oom kill
Right now in the oom_kill_process if the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!
Yueyang Pan (1):
Add memory allocation info for cgroup oom
mm/oom_kill.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Signature · 1 line
2.47.3
RFC 1/1 Add memory allocation info for cgroup oom
Enable show_mem for the cgroup oom case. We will have memory allocation information in such case for the machine.
--- mm/oom_kill.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/oom_kill.c +3 -1
@@ -465,8 +465,10 @@ static void dump_header(struct oom_control *oc)pr_warn ("COMPACTION is disabled!!!\n" );dump_stack (); -if (is_memcg_oom (oc)) +if (is_memcg_oom (oc)) {mem_cgroup_print_oom_meminfo (oc->memcg); +show_mem (); + }else {__show_mem (SHOW_MEM_FILTER_NODES , oc->nodemask,gfp_zone (oc->gfp_mask));if (should_dump_unreclaim_slab ())
Signature · 1 line
2.47.3
Re: [RFC 1/1] Add memory allocation info for cgroup oom
Quoted 2 lines · On Thu, 14 Aug 2025 10:11:57 -0700 Yueyang Pan <[email protected]> wrote:
Enable
show_memfor the cgroup oom case. We will have memory allocation information in such case for the machine.
Hi Pan,
Thank you for your patch! This makes sense to me. As for your concerns from the cover letter on whether this is too much information: personally I don't think so, but perhaps other developers will have different opinions?
I just have a few comments / nits.
Quoted 10 lines
Signed-off-by Yueyang Pan <[email protected]>--- mm/oom_kill.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/oom_kill.c +0
-0
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 470
Quoted 7 lines
mm/oom_kill.c +2
-1@@ -465,8 +465,10 @@ static void dump_header(struct oom_control *oc)pr_warn ("COMPACTION is disabled!!!\n" );dump_stack (); -if (is_memcg_oom (oc)) +if (is_memcg_oom (oc)) {mem_cgroup_print_oom_meminfo (oc->memcg); +show_mem ();
Below, there is a direct call to __show_mem, which limits node and zone filtering. I am wondering whether it would make sense to also call __show_mem with the same arguments? show_mem() is just a wrapper around __show_mem with default parameters (i.e. not filtering out nodes, not filtering out zones).
If you think this makes sense, we can even take it out of the if-else statement and call it unconditionally. But this is just my opinion, please feel free to keep the unfiltered call if you believe that fits better in here.
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 471
Quoted 1 line
mm/oom_kill.c +1
-0+ }
NIT: Should this closing brace be on the same line as the following else statement, as per the kernel style guide [1]
Quoted 5 lines
mm/oom_kill.c +0
-0else {__show_mem (SHOW_MEM_FILTER_NODES , oc->nodemask,gfp_zone (oc->gfp_mask));if (should_dump_unreclaim_slab ())Signature · 1 line
2.47.3
Thanks again Pan, I hope you have a great day!
Joshua
[1] https://docs.kernel.org/process/coding-style.html
Sent using hkml (https://github.com/sjp38/hackermail)
Re: [RFC 1/1] Add memory allocation info for cgroup oom
Quoted 5 lines · Joshua Hahn
Quoted 2 lines · On Thu, 14 Aug 2025 10:11:57 -0700 Yueyang Pan <[email protected]> wrote:
Enable
show_memfor the cgroup oom case. We will have memory allocation information in such case for the machine.
Quoted 7 lines
Hi Pan,
Thank you for your patch! This makes sense to me. As for your concerns from the cover letter on whether this is too much information: personally I don't think so, but perhaps other developers will have different opinions?
I just have a few comments / nits.
Thanks for your comment, Joshua.
Quoted 13 lines
Quoted 13 lines
Signed-off-by Yueyang Pan <[email protected]>--- mm/oom_kill.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/oom_kill.c +0
-0@@ -465,8 +465,10 @@ static void dump_header(struct oom_control *oc)pr_warn ("COMPACTION is disabled!!!\n" );
Quoted 11 lines
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 470
Quoted 5 lines
mm/oom_kill.c +2
-1dump_stack (); -if (is_memcg_oom (oc)) +if (is_memcg_oom (oc)) {mem_cgroup_print_oom_meminfo (oc->memcg); +show_mem ();Below, there is a direct call to
__show_mem, which limits node and zone filtering. I am wondering whether it would make sense to also call__show_memwith the same arguments?show_mem()is just a wrapper around__show_memwith default parameters (i.e. not filtering out nodes, not filtering out zones).
The reason why I call show_mem here directly is because cgroup is not bound to a specific zone or node (correctly me if I am wrong). Thus I simply invoke show_mem to show system-wide memory info.
Quoted 9 lines
If you think this makes sense, we can even take it out of the if-else statement and call it unconditionally. But this is just my opinion, please feel free to keep the unfiltered call if you believe that fits better in here.
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 471
Quoted 1 line
mm/oom_kill.c +1
-0+ }NIT: Should this closing brace be on the same line as the following else statement, as per the kernel style guide [1]
Sorry for this. I will run checkpatch for my formal patch definitely
Quoted 7 lines
Quoted 6 lines
mm/oom_kill.c +0
-0else {__show_mem (SHOW_MEM_FILTER_NODES , oc->nodemask,gfp_zone (oc->gfp_mask));if (should_dump_unreclaim_slab ())Signature · 1 line
2.47.3
Quoted 6 lines
Thanks again Pan, I hope you have a great day!
Joshua[1] https://docs.kernel.org/process/coding-style.html
Sent using hkml (https://github.com/sjp38/hackermail)
Sorry that I forgot to cc some maintainers so I added them in this reply. Pan
Re: [RFC 1/1] Add memory allocation info for cgroup oom
Quoted 6 lines · Yueyang Pan
Quoted 4 lines · Joshua Hahn
Quoted 2 lines · On Thu, 14 Aug 2025 10:11:57 -0700 Yueyang Pan <[email protected]> wrote:
Enable
show_memfor the cgroup oom case. We will have memory allocation information in such case for the machine.
Memory allocations are only a part of show_mem(), so I would not call this change memory allocation profiling specific. The title and the changelog should be corrected to reflect exactly what is being done here - logging system in addition to cgroup memory state during cgroup oom-kill.
As for whether it makes sense to report system memory during cgroup oom-kill... I'm not too sure. Maybe people who use memcgs more extensively than what I've seen (in Android) can chime in?
Quoted 68 lines
Quoted 8 lines
Hi Pan,
Thank you for your patch! This makes sense to me. As for your concerns from the cover letter on whether this is too much information: personally I don't think so, but perhaps other developers will have different opinions?
I just have a few comments / nits.
Thanks for your comment, Joshua.
Quoted 13 lines
Quoted 13 lines
Signed-off-by Yueyang Pan <[email protected]>--- mm/oom_kill.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/oom_kill.c +0
-0@@ -465,8 +465,10 @@ static void dump_header(struct oom_control *oc)pr_warn ("COMPACTION is disabled!!!\n" );Quoted 11 lines
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 470
Quoted 5 lines
mm/oom_kill.c +2
-1dump_stack (); -if (is_memcg_oom (oc)) +if (is_memcg_oom (oc)) {mem_cgroup_print_oom_meminfo (oc->memcg); +show_mem ();Below, there is a direct call to
__show_mem, which limits node and zone filtering. I am wondering whether it would make sense to also call__show_memwith the same arguments?show_mem()is just a wrapper around__show_memwith default parameters (i.e. not filtering out nodes, not filtering out zones).The reason why I call
show_memhere directly is because cgroup is not bound to a specific zone or node (correctly me if I am wrong). Thus I simply invokeshow_memto show system-wide memory info.Quoted 9 lines
If you think this makes sense, we can even take it out of the if-else statement and call it unconditionally. But this is just my opinion, please feel free to keep the unfiltered call if you believe that fits better in here.
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 471
Quoted 1 line
mm/oom_kill.c +1
-0+ }NIT: Should this closing brace be on the same line as the following else statement, as per the kernel style guide [1]
Sorry for this. I will run checkpatch for my formal patch definitely
Quoted 7 lines
Quoted 6 lines
mm/oom_kill.c +0
-1else {__show_mem (SHOW_MEM_FILTER_NODES , oc->nodemask,gfp_zone (oc->gfp_mask));if (should_dump_unreclaim_slab ()) --2.47.3
Quoted 6 lines
Thanks again Pan, I hope you have a great day!
Joshua[1] https://docs.kernel.org/process/coding-style.html
Sent using hkml (https://github.com/sjp38/hackermail)
Sorry that I forgot to cc some maintainers so I added them in this reply. Pan
Shakeel Butt
reviewerQuoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Re: [RFC 1/1] Add memory allocation info for cgroup oom
In reply to Suren BaghdasaryanQuoted 5 lines · Suren Baghdasaryan
Quoted 4 lines · Yueyang Pan
Quoted 2 lines · Joshua Hahn
On Thu, 14 Aug 2025 10:11:57 -0700 Yueyang Pan <[email protected]> wrote:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Enable
show_memfor the cgroup oom case. We will have memory allocation information in such case for the machine.Memory allocations are only a part of
show_mem(), so I would not call this change memory allocation profiling specific. The title and the changelog should be corrected to reflect exactly what is being done here - logging system in addition to cgroup memory state during cgroup oom-kill.
Thanks for your feedback Suren! I will change the title to be precise in the next version.
Quoted 4 lines
As for whether it makes sense to report system memory during cgroup oom-kill... I'm not too sure. Maybe people who use memcgs more extensively than what I've seen (in Android) can chime in?
In my opinion, the show_free_areas and memory allocation profiling data can provide an entrypoint to understand what happens with cgroup oom. We can also compare them with historical data to see if some memory usage has a spike.
Feel free to critize me if I am not making sense.
Quoted 63 lines
Quoted 63 lines
Quoted 9 lines
Hi Pan,
Thank you for your patch! This makes sense to me. As for your concerns from the cover letter on whether this is too much information: personally I don't think so, but perhaps other developers will have different opinions?
I just have a few comments / nits.
Thanks for your comment, Joshua.
Quoted 13 lines
Quoted 13 lines
Signed-off-by Yueyang Pan <[email protected]>--- mm/oom_kill.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/oom_kill.c +0
-0@@ -465,8 +465,10 @@ static void dump_header(struct oom_control *oc)pr_warn ("COMPACTION is disabled!!!\n" );Quoted 11 lines
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 470
Quoted 5 lines
mm/oom_kill.c +2
-1dump_stack (); -if (is_memcg_oom (oc)) +if (is_memcg_oom (oc)) {mem_cgroup_print_oom_meminfo (oc->memcg); +show_mem ();Below, there is a direct call to
__show_mem, which limits node and zone filtering. I am wondering whether it would make sense to also call__show_memwith the same arguments?show_mem()is just a wrapper around__show_memwith default parameters (i.e. not filtering out nodes, not filtering out zones).The reason why I call
show_memhere directly is because cgroup is not bound to a specific zone or node (correctly me if I am wrong). Thus I simply invokeshow_memto show system-wide memory info.Quoted 9 lines
If you think this makes sense, we can even take it out of the if-else statement and call it unconditionally. But this is just my opinion, please feel free to keep the unfiltered call if you believe that fits better in here.
mm/oom_kill.c static void dump_header(struct oom_control *oc) · line 471
Quoted 1 line
mm/oom_kill.c +1
-0+ }NIT: Should this closing brace be on the same line as the following else statement, as per the kernel style guide [1]
Sorry for this. I will run checkpatch for my formal patch definitely
Quoted 10 lines
Quoted 6 lines
mm/oom_kill.c +0
-1else {__show_mem (SHOW_MEM_FILTER_NODES , oc->nodemask,gfp_zone (oc->gfp_mask));if (should_dump_unreclaim_slab ()) --2.47.3
Thanks again Pan, I hope you have a great day!
Joshua
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
[1] https://docs.kernel.org/process/coding-style.html
Sent using hkml (https://github.com/sjp38/hackermail)
Sorry that I forgot to cc some maintainers so I added them in this reply. Pan
Thanks,
Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!
Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
As in my reply to Suren, I was thinking the system-wide memory usage info provided by show_free_pages and memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?
Thanks,
Pan
Shakeel Butt
reviewerQuoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?
I am not really sure about show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?
Quoted 30 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?
Quoted 5 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?
No, memory allocation profiling is not cgroup-aware. It tracks allocations and their code locations but no other context.
Shakeel Butt
reviewerQuoted 31 lines · Suren Baghdasaryan
Quoted 30 lines · Shakeel Butt
Quoted 27 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?
Quoted 8 lines
Quoted 5 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?No, memory allocation profiling is not cgroup-aware. It tracks allocations and their code locations but no other context.
Thanks for the info. Pan, will having memcg info along with allocation profile help your use-case? (Though adding that might not be easy or cheaper)
Quoted 38 lines · Shakeel Butt
Quoted 37 lines · Suren Baghdasaryan
Quoted 35 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?
Quoted 6 lines
Quoted 2 lines
No, memory allocation profiling is not cgroup-aware. It tracks allocations and their code locations but no other context.
Thanks for the info. Pan, will having memcg info along with allocation profile help your use-case? (Though adding that might not be easy or cheaper)
Yeah I have been thinking about it with eBPF hooks but it is going to be a long term effort as we need to measure the overhead. Now the way memory profiling is implemented incur almost "zero" overhead.
Quoted 29 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?
Quoted 3 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is
Sorry for my late reply. I have been trying hard to think about a use case. One specific case I can think about is when there is no workload stacking, when one job is running solely on the machine. For example, memory allocation profiling can tell the memory usage of the network driver, which can make cg allocates memory harder and eventually leads to cgoom. Without this information, it would be hard to reason about what is happening in the kernel given increased oom number.
show_free_areas() will give a summary of different types of memory which can possibably lead to increased cgoom in my previous case. Then one looks deeper via the memory allocation profiling as an entrypoint to debug.
Does this make sense to you?
Quoted 2 lines
it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?
Thanks
Pan
Quoted 30 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?
Quoted 5 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?
Actually I was thinking about making memory profiling memcg-aware but it would be quite costly both from memory and performance points of view. Currently we have a per-cpu counter for each allocation in the kernel codebase. To make it work for each memcg we would have to add memcg dimension to the counters, so each counter becomes per-cpu plus per-memcg. I'll be thinking about possible optimizations since many of these counters will stay at 0 but any such optimization would come at a performance cost, which we tried to keep at the absolute minimum.
I'm CC'ing Sourav and Pasha since they were also interested in making memory allocation profiling memcg-aware. Would Meta folks (Usama, Shakeel, Johannes) be interested in such enhancement as well? Would it be preferable to have such accounting for a specific memcg which we pre-select (less memory and performance overhead) or we need that for all memcgs as a generic feature? We have some options here but I want to understand what would be sufficient and add as little overhead as possible.
Thanks,
Suren.
Quoted 43 lines · Yueyang Pan
Quoted 29 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?Quoted 3 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, isSorry for my late reply. I have been trying hard to think about a use case. One specific case I can think about is when there is no workload stacking, when one job is running solely on the machine. For example, memory allocation profiling can tell the memory usage of the network driver, which can make cg allocates memory harder and eventually leads to cgoom. Without this information, it would be hard to reason about what is happening in the kernel given increased oom number.
Quoted 5 lines
show_free_areas()will give a summary of different types of memory which can possibably lead to increased cgoom in my previous case. Then one looks deeper via the memory allocation profiling as an entrypoint to debug.Does this make sense to you?
I think if we had per-memcg memory profiling that would make sense. Counters would reflect only allocations made by the processes from that memcg and you could easily identify the allocation that caused memcg to oom. But dumping system-wide profiling information at memcg-oom time I think would not help you with this task. It will be polluted with allocations from other memcgs, so likely won't help much (unless there is some obvious leak or you know that a specific allocation is done only by a process from your memcg and no other process).
Quoted 6 lines
Quoted 3 lines
it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?
Thanks
Pan
Quoted 46 lines · Suren Baghdasaryan
Quoted 30 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?Quoted 5 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?Actually I was thinking about making memory profiling memcg-aware but it would be quite costly both from memory and performance points of view. Currently we have a per-cpu counter for each allocation in the kernel codebase. To make it work for each memcg we would have to add memcg dimension to the counters, so each counter becomes per-cpu plus per-memcg. I'll be thinking about possible optimizations since many of these counters will stay at 0 but any such optimization would come at a performance cost, which we tried to keep at the absolute minimum.
Quoted 8 lines
I'm CC'ing Sourav and Pasha since they were also interested in making memory allocation profiling memcg-aware. Would Meta folks (Usama, Shakeel, Johannes) be interested in such enhancement as well? Would it be preferable to have such accounting for a specific memcg which we pre-select (less memory and performance overhead) or we need that for all memcgs as a generic feature? We have some options here but I want to understand what would be sufficient and add as little overhead as possible.
Yes, having per memcg counters is going to be extremely useful (we were thinking of having this as a future project to work on). For meta fleet in particular, we might have almost 100 memcgs running, but the number of memcgs running workloads is particularly small (usually less than 10). In the rest, you might have services that are responsible for telemetry, monitoring, security, etc (for which we arent really interested in the memory allocation profile). So yes, it would be ideal to have the profile for just pre-select memcgs, especially if it leads to lower memory and performance overhead.
Having memory allocation profile at memcg level is especially needed when we have multiple workloads stacked on the same host. Having it at host level in such a case makes the data less useful when we have OOMs and for workload analysis as you dont know which workload is contributing how much.
Quoted 2 lines
Thanks,
Suren.
Shakeel Butt
reviewerQuoted 46 lines · Suren Baghdasaryan
Quoted 30 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?Quoted 5 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?Actually I was thinking about making memory profiling memcg-aware but it would be quite costly both from memory and performance points of view. Currently we have a per-cpu counter for each allocation in the kernel codebase. To make it work for each memcg we would have to add memcg dimension to the counters, so each counter becomes per-cpu plus per-memcg. I'll be thinking about possible optimizations since many of these counters will stay at 0 but any such optimization would come at a performance cost, which we tried to keep at the absolute minimum.
Quoted 8 lines
I'm CC'ing Sourav and Pasha since they were also interested in making memory allocation profiling memcg-aware. Would Meta folks (Usama, Shakeel, Johannes) be interested in such enhancement as well? Would it be preferable to have such accounting for a specific memcg which we pre-select (less memory and performance overhead) or we need that for all memcgs as a generic feature? We have some options here but I want to understand what would be sufficient and add as little overhead as possible.
Thanks Suren, yes, as already mentioned by Usama, Meta will be interested in memcg aware allocation profiling. I would say start simple and as little overhead as possible. More functionality can be added later when the need arises. Maybe the first useful addition is just adding how many allocations for a specific allocation site are memcg charged.
Michal Hocko
maintainerQuoted 50 lines · Suren Baghdasaryan
Quoted 43 lines · Yueyang Pan
Quoted 29 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?Quoted 3 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, isSorry for my late reply. I have been trying hard to think about a use case. One specific case I can think about is when there is no workload stacking, when one job is running solely on the machine. For example, memory allocation profiling can tell the memory usage of the network driver, which can make cg allocates memory harder and eventually leads to cgoom. Without this information, it would be hard to reason about what is happening in the kernel given increased oom number.
Quoted 5 lines
show_free_areas()will give a summary of different types of memory which can possibably lead to increased cgoom in my previous case. Then one looks deeper via the memory allocation profiling as an entrypoint to debug.Does this make sense to you?
Quoted 9 lines
I think if we had per-memcg memory profiling that would make sense. Counters would reflect only allocations made by the processes from that memcg and you could easily identify the allocation that caused memcg to oom. But dumping system-wide profiling information at memcg-oom time I think would not help you with this task. It will be polluted with allocations from other memcgs, so likely won't help much (unless there is some obvious leak or you know that a specific allocation is done only by a process from your memcg and no other process).
I agree with Suren. It makes very little sense and in many cases it could be actively misleading to print global memory state on memcg OOMs. Not to mention that those events, unlike global OOMs, could happen much more often.
If you are interested in a more information on memcg oom occurance you can detext OOM events and print whatever information you need.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 57 lines · Shakeel Butt
Quoted 46 lines · Suren Baghdasaryan
Quoted 30 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?Quoted 5 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, is it possible to filter for the given memcg? Do we save memcg information in the memory allocation profiling?Actually I was thinking about making memory profiling memcg-aware but it would be quite costly both from memory and performance points of view. Currently we have a per-cpu counter for each allocation in the kernel codebase. To make it work for each memcg we would have to add memcg dimension to the counters, so each counter becomes per-cpu plus per-memcg. I'll be thinking about possible optimizations since many of these counters will stay at 0 but any such optimization would come at a performance cost, which we tried to keep at the absolute minimum.
Quoted 8 lines
I'm CC'ing Sourav and Pasha since they were also interested in making memory allocation profiling memcg-aware. Would Meta folks (Usama, Shakeel, Johannes) be interested in such enhancement as well? Would it be preferable to have such accounting for a specific memcg which we pre-select (less memory and performance overhead) or we need that for all memcgs as a generic feature? We have some options here but I want to understand what would be sufficient and add as little overhead as possible.
Quoted 6 lines
Thanks Suren, yes, as already mentioned by Usama, Meta will be interested in memcg aware allocation profiling. I would say start simple and as little overhead as possible. More functionality can be added later when the need arises. Maybe the first useful addition is just adding how many allocations for a specific allocation site are memcg charged.
Adding back Sourav, Pasha and Johannes who got accidentally dropped in the replies.
I looked a bit into adding memcg-awareness into memory allocation profiling and it's more complicated than I first thought (as usual). The main complication is that we need to add memcg_id or some other memcg identifier into codetag_ref. That's needed so that we can unaccount the correct memcg when we free an allocation - that's the usual function of the codetag_ref. Now, extending codetag_ref is not a problem by itself but when we use mem_profiling_compressed mode, we store an index of the codetag instead of codetag_ref in the unused page flag bits. This is useful optimization to avoid using page_ext and overhead associated with it. So, full blown memcg support seems problematic.
What I'm thinking is easily doable is a filtering interface where we could select a specific memcg to be profiled, IOW we profile only allocations from a chosen memcg. Filtering can be done using ioctl interface on /proc/allocinfo, which can be used for other things as well, like filtering non-zero allocations, returning per-NUMA node information, etc. I see that Damon uses similar memcg filtering (see damos_filter.memcg_id), so I can reuse some of that code for implementing this facility. From high-level, userspace will be able to select one memcg at a time to be profiled. At some later time profiling information is gathered and another memcg can be selected or filtering can be reset to profile all allocations from all memcgs. I expect overhead for this kind of memcg filtering to be quite low. WDYT folks, would this be helpful and cover your usecases?
Quoted 1 line
Quoted 61 lines · Michal Hocko
Quoted 50 lines · Suren Baghdasaryan
Quoted 43 lines · Yueyang Pan
Quoted 29 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?Quoted 3 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, isSorry for my late reply. I have been trying hard to think about a use case. One specific case I can think about is when there is no workload stacking, when one job is running solely on the machine. For example, memory allocation profiling can tell the memory usage of the network driver, which can make cg allocates memory harder and eventually leads to cgoom. Without this information, it would be hard to reason about what is happening in the kernel given increased oom number.
Quoted 5 lines
show_free_areas()will give a summary of different types of memory which can possibably lead to increased cgoom in my previous case. Then one looks deeper via the memory allocation profiling as an entrypoint to debug.Does this make sense to you?
Quoted 9 lines
I think if we had per-memcg memory profiling that would make sense. Counters would reflect only allocations made by the processes from that memcg and you could easily identify the allocation that caused memcg to oom. But dumping system-wide profiling information at memcg-oom time I think would not help you with this task. It will be polluted with allocations from other memcgs, so likely won't help much (unless there is some obvious leak or you know that a specific allocation is done only by a process from your memcg and no other process).
Quoted 6 lines
I agree with Suren. It makes very little sense and in many cases it could be actively misleading to print global memory state on memcg OOMs. Not to mention that those events, unlike global OOMs, could happen much more often.
If you are interested in a more information on memcg oom occurance you can detext OOM events and print whatever information you need.
"Misleading" is a concern; the show_mem report would want to print very explicitly which information is specifically for the memcg and which is global, and we don't do that now.
I don't think that means we shouldn't print it at all though, because it can happen that we're in an OOM because one specific codepath is allocating way more memory than we should be; even if the memory allocation profiling info isn't correct for the memcg it'll be useful information in a situation like that, it just needs to very clearly state what it's reporting on.
I'm not sure we do that very well at all now, I'm looking at __show_mem() ad it's not even passed a memcg. !?
Also, if anyone's thinking about "what if memory allocation profiling was memcg aware" - the thing we saw when doing performance testing is that memcg accounting was much higher overhead than memory allocation profiling - hence, most kernel memory allocations don't even get memcg accounting.
I think that got the memcg people looking at ways to make the accounting cheaper, but I'm not sure if anything landed from that.
Quoted 84 lines · Kent Overstreet
Quoted 61 lines · Michal Hocko
Quoted 50 lines · Suren Baghdasaryan
Quoted 43 lines · Yueyang Pan
Quoted 29 lines · Shakeel Butt
Quoted 22 lines · Yueyang Pan
Quoted 10 lines · Shakeel Butt
Quoted 8 lines · Yueyang Pan
Right now in the
oom_kill_processif the oom is because of the cgroup limit, we won't get memory allocation infomation. In some cases, we can have a large cgroup workload running which dominates the machine. The reason using cgroup is to leave some resource for system. When this cgroup is killed, we would also like to have some memory allocation information for the whole server as well. This is reason behind this mini change. Is it an acceptable thing to do? Will it be too much information for people? I am happy with any suggestions!Quoted 2 lines
For a single patch, it is better to have all the context in the patch and there is no need for cover letter.
Thanks for your suggestion Shakeel! I will change this in the next version.
Quoted 5 lines
What exact information you want on the memcg oom that will be helpful for the users in general? You mentioned memory allocation information, can you please elaborate a bit more.
Quoted 5 lines
As in my reply to Suren, I was thinking the system-wide memory usage info provided by
show_free_pagesand memory allocation profiling info can help us debug cgoom by comparing them with historical data. What is your take on this?Quoted 3 lines
I am not really sure about
show_free_areas(). More specifically how the historical data diff will be useful for a memcg oom. If you have a concrete example, please give one. For memory allocation profiling, isSorry for my late reply. I have been trying hard to think about a use case. One specific case I can think about is when there is no workload stacking, when one job is running solely on the machine. For example, memory allocation profiling can tell the memory usage of the network driver, which can make cg allocates memory harder and eventually leads to cgoom. Without this information, it would be hard to reason about what is happening in the kernel given increased oom number.
Quoted 5 lines
show_free_areas()will give a summary of different types of memory which can possibably lead to increased cgoom in my previous case. Then one looks deeper via the memory allocation profiling as an entrypoint to debug.Does this make sense to you?
Quoted 9 lines
I think if we had per-memcg memory profiling that would make sense. Counters would reflect only allocations made by the processes from that memcg and you could easily identify the allocation that caused memcg to oom. But dumping system-wide profiling information at memcg-oom time I think would not help you with this task. It will be polluted with allocations from other memcgs, so likely won't help much (unless there is some obvious leak or you know that a specific allocation is done only by a process from your memcg and no other process).
Quoted 6 lines
I agree with Suren. It makes very little sense and in many cases it could be actively misleading to print global memory state on memcg OOMs. Not to mention that those events, unlike global OOMs, could happen much more often.
If you are interested in a more information on memcg oom occurance you can detext OOM events and print whatever information you need."Misleading" is a concern; the
show_memreport would want to print very explicitly which information is specifically for the memcg and which is global, and we don't do that now.I don't think that means we shouldn't print it at all though, because it can happen that we're in an OOM because one specific codepath is allocating way more memory than we should be; even if the memory allocation profiling info isn't correct for the memcg it'll be useful information in a situation like that, it just needs to very clearly state what it's reporting on.
I'm not sure we do that very well at all now, I'm looking at
__show_mem()ad it's not even passed a memcg. !?
Quoted 8 lines
Also, if anyone's thinking about "what if memory allocation profiling was memcg aware" - the thing we saw when doing performance testing is that memcg accounting was much higher overhead than memory allocation profiling - hence, most kernel memory allocations don't even get memcg accounting.
I think that got the memcg people looking at ways to make the accounting cheaper, but I'm not sure if anything landed from that.
Yes, Roman landed a series of changes reducing the memcg accounting overhead.
Quoted 7 lines · Suren Baghdasaryan
Quoted 3 lines · Kent Overstreet
I think that got the memcg people looking at ways to make the accounting cheaper, but I'm not sure if anything landed from that.
Yes, Roman landed a series of changes reducing the memcg accounting overhead.
Do you know offhand how big that was?
Quoted 11 lines · Kent Overstreet
Quoted 7 lines · Suren Baghdasaryan
Quoted 3 lines · Kent Overstreet
I think that got the memcg people looking at ways to make the accounting cheaper, but I'm not sure if anything landed from that.
Yes, Roman landed a series of changes reducing the memcg accounting overhead.
Do you know offhand how big that was?
I'll need to dig it up but it was still much higher than memory profiling.
Shakeel Butt
reviewerQuoted 7 lines · Suren Baghdasaryan
Quoted 5 lines · Kent Overstreet
Quoted 3 lines · Suren Baghdasaryan
Quoted 1 line · Kent Overstreet
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
I think that got the memcg people looking at ways to make the accounting cheaper, but I'm not sure if anything landed from that.
Yes, Roman landed a series of changes reducing the memcg accounting overhead.
Do you know offhand how big that was?
I'll need to dig it up but it was still much higher than memory profiling.
What benchmark/workload was used to compare memcg accounting and memory profiling?
Quoted 12 lines · Shakeel Butt
Quoted 10 lines · Suren Baghdasaryan
Quoted 8 lines · Kent Overstreet
Quoted 6 lines · Suren Baghdasaryan
Quoted 3 lines · Kent Overstreet
I think that got the memcg people looking at ways to make the accounting cheaper, but I'm not sure if anything landed from that.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yes, Roman landed a series of changes reducing the memcg accounting overhead.
Do you know offhand how big that was?
I'll need to dig it up but it was still much higher than memory profiling.
What benchmark/workload was used to compare memcg accounting and memory profiling?
It was an in-kernel allocation stress test. Not very realistic but good for comparing the overhead.
Michal Hocko
maintainerOn Mon 08-09-25 13:34:17, Kent Overstreet wrote:
[...]
Quoted 2 lines
I'm not sure we do that very well at all now, I'm looking at
__show_mem()ad it's not even passed a memcg. !?
__show_mem is not called from memcg oom path. There is mem_cgroup_print_oom_meminfo for that purpose as memcg stats are sufficiently different to have their own thing.
Now, what mem_cgroup_print_oom_meminfo prints is missing more detailed slab memory usage break down. This might be useful in some situations. If we had memcg aware memory profiling then it could be added to that path as well.
Signature · 2 lines
Michal Hocko SUSE Labs