doc: memcontrol: add description for oom_kill
Subsystems (3)
PATCH 1/1 doc: memcontrol: add description for oom_kill
When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.
The cgroup v2 documents it, but the description is missed for cgroup v1.
--- Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++ 1 file changed, 3 insertions(+)
Documentation/admin-guide/cgroup-v1/memory.rst +3 -0
@@ -851,6 +851,9 @@ At reading, current status of OOM is shown. (if 1, oom-killer is disabled) - under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.) + - oom_kill integer counter + The number of processes belonging to this cgroup killed by any + kind of OOM killer. 11. Memory Pressure ===================
Signature · 1 line
2.26.2
Quoted 4 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.
Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally confusing because in many cases there would be no victim counted for the above mentioned memcg ooms.
Quoted 3 lines
The cgroup v2 documents it, but the description is missed for cgroup v1.
Signed-off-by Yang Shi <[email protected]>
Quoted 21 lines
--- Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++ 1 file changed, 3 insertions(+)Documentation/admin-guide/cgroup-v1/memory.rst +3
-0@@ -851,6 +851,9 @@ At reading, current status of OOM is shown. (if 1, oom-killer is disabled) - under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.) + - oom_kill integer counter + The number of processes belonging to this cgroup killed by any + kind of OOM killer. 11. Memory Pressure ===================Signature · 1 line
2.26.2
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally
Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.
Quoted 34 lines
confusing because in many cases there would be no victim counted for the above mentioned memcg ooms.
Quoted 3 lines
The cgroup v2 documents it, but the description is missed for cgroup v1.
Signed-off-by Yang Shi <[email protected]>Acked-by Michal Hocko <[email protected]>Quoted 19 lines
--- Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++ 1 file changed, 3 insertions(+)Documentation/admin-guide/cgroup-v1/memory.rst +3
-1@@ -851,6 +851,9 @@ At reading, current status of OOM is shown. (if 1, oom-killer is disabled) - under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.) + - oom_kill integer counter + The number of processes belonging to this cgroup killed by any + kind of OOM killer. 11. Memory Pressure =================== --Quoted 2 lines
2.26.2
--
Michal Hocko
SUSE Labs
Quoted 14 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally
Quoted 4 lines
Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.
BTW, having the counter recorded hierarchically may help out one of our usecases. We want to monitor the oom_kill for some services, but systemd would wipe out the cgroup if the service is oom killed then restart the service from scratch (it means create a brand new cgroup with the same name). So this systemd behavior makes the counter useless if it is not recorded hierarchically.
Quoted 35 lines
Quoted 35 lines
confusing because in many cases there would be no victim counted for the above mentioned memcg ooms.
Quoted 3 lines
The cgroup v2 documents it, but the description is missed for cgroup v1.
Signed-off-by Yang Shi <[email protected]>Acked-by Michal Hocko <[email protected]>Quoted 19 lines
--- Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++ 1 file changed, 3 insertions(+)Documentation/admin-guide/cgroup-v1/memory.rst +3
-1@@ -851,6 +851,9 @@ At reading, current status of OOM is shown. (if 1, oom-killer is disabled) - under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.) + - oom_kill integer counter + The number of processes belonging to this cgroup killed by any + kind of OOM killer. 11. Memory Pressure =================== --Quoted 2 lines
2.26.2
--
Michal Hocko
SUSE Labs
Quoted 20 lines · Yang Shi
Quoted 14 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally
Quoted 4 lines
Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.
Quoted 6 lines
BTW, having the counter recorded hierarchically may help out one of our usecases. We want to monitor the
oom_killfor some services, but systemd would wipe out the cgroup if the service is oom killed then restart the service from scratch (it means create a brand new cgroup with the same name). So this systemd behavior makes the counter useless if it is not recorded hierarchically.
Just to make sure I understand correctly. You have a setup where memcg for a service has a hard limit configured and it is destroyed when oom happens inside that memcg. A new instance is created at the same place of the hierarchy with a new memcg. Your problem is that the oom killed memcg will not be recorded in its parent oom event and the information will get lost with the torn down memcg. Correct?
If yes then how do you tell which of the child cgroup was killed from the parent counter? Or is there only a single child?
Anyway, cgroup v2 will offer the hierarchical behavior. Do you have any strong reasons that you cannot use v2?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 29 lines · Michal Hocko
Quoted 20 lines · Yang Shi
Quoted 14 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally
Quoted 4 lines
Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.
Quoted 6 lines
BTW, having the counter recorded hierarchically may help out one of our usecases. We want to monitor the
oom_killfor some services, but systemd would wipe out the cgroup if the service is oom killed then restart the service from scratch (it means create a brand new cgroup with the same name). So this systemd behavior makes the counter useless if it is not recorded hierarchically.
Quoted 6 lines
Just to make sure I understand correctly. You have a setup where memcg for a service has a hard limit configured and it is destroyed when oom happens inside that memcg. A new instance is created at the same place of the hierarchy with a new memcg. Your problem is that the oom killed memcg will not be recorded in its parent oom event and the information will get lost with the torn down memcg. Correct?
Yes. But global oom instead of memcg oom.
Quoted 3 lines
If yes then how do you tell which of the child cgroup was killed from the parent counter? Or is there only a single child?
Not only a single child, but our case is that oom-killed child consumes 90% memory, then global oom would kill it. This definitely doesn't prevent from accounting oom from other children, but we don't have to have a very accurate counter and in our case we can tell 99% oom kill happens with that specific memcg.
Quoted 3 lines
Anyway, cgroup v2 will offer the hierarchical behavior. Do you have any strong reasons that you cannot use v2?
I do prefer to migrate to cgroup v2 personally. But it incurs significant work for orchestration tools, infrastructure configuration, monitoring tools, etc which are out of my control.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Quoted 13 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally
Quoted 4 lines
Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.
I believe the reson is that v1 can have tasks in the intermediate (non-leaf) memcgs. So you wouldn't have a way to tell whether the oom kill has happened in such a memcg or somewhere down the hierarchy.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 15 lines · Michal Hocko
Quoted 13 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally
Quoted 8 lines
Quoted 4 lines
Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.
I believe the reson is that v1 can have tasks in the intermediate (non-leaf) memcgs. So you wouldn't have a way to tell whether the oom kill has happened in such a memcg or somewhere down the hierarchy.
Aha, I forgot it, that's bad. Although we don't have tasks in intermediate nodes in practice, I do understand it is not forbidden as cgroup v2.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Quoted 9 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.The cgroup v2 documents it, but the description is missed for cgroup v1.
Signed-off-by Yang Shi <[email protected]>
Quoted 8 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.The cgroup v2 documents it, but the description is missed for cgroup v1.
Signed-off-by Yang Shi <[email protected]>
Thanks.
Quoted 22 lines
---
Documentation/admin-guide/cgroup-v1/memory.rst| 3 +++1 file changed, 3 insertions(+)Documentation/admin-guide/cgroup-v1/memory.rst +3
-0@@ -851,6 +851,9 @@ At reading, current status of OOM is shown. (if 1, oom-killer is disabled) - under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.) + - oom_kill integer counter + The number of processes belonging to this cgroup killed by any + kind of OOM killer.11. Memory Pressure
===================Signature · 1 line
2.26.2
Quoted 17 lines · Yang Shi
When debugging an oom issue, I found the
oom_killcounter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.The cgroup v2 documents it, but the description is missed for cgroup v1.
Signed-off-by Yang Shi <[email protected]>--- Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++ 1 file changed, 3 insertions(+)Documentation/admin-guide/cgroup-v1/memory.rst +0
-0
Documentation/admin-guide/cgroup-v1/memory.rst At reading, current status of OOM is shown. · line 856
Quoted 6 lines
Documentation/admin-guide/cgroup-v1/memory.rst +3
-0@@ -851,6 +851,9 @@ At reading, current status of OOM is shown. (if 1, oom-killer is disabled) - under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.) + - oom_kill integer counter + The number of processes belonging to this cgroup killed by any + kind of OOM killer.
Applied, thanks.
jon