Skip to contentSkip to conversation

doc: memcontrol: add description for oom_kill

PATCH Applied Yang Shi 5 participants 11 messages Feb 26, 2021 – Mar 1, 2021

Subsystems (3)
lore
Share & tools
lore

Yang Shi

author
#0 lore raw

PATCH 1/1 doc: memcontrol: add description for oom_kill

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

The cgroup v2 documents it, but the description is missed for cgroup v1.

---
 Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++
 1 file changed, 3 insertions(+)
Documentation/admin-guide/cgroup-v1/memory.rst +3 -0
index 0936412e044e..44d5429636e2 100644
@@ -851,6 +851,9 @@ At reading, current status of OOM is shown.
 	  (if 1, oom-killer is disabled)
 	- under_oom	   0 or 1
 	  (if 1, the memory cgroup is under OOM, tasks may be stopped.)
+        - oom_kill         integer counter
+          The number of processes belonging to this cgroup killed by any
+          kind of OOM killer.
 
 11. Memory Pressure
 ===================
Signature · 1 line
2.26.2
Acked-by on v1 1/1
#1 lore raw
Quoted 4 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally confusing because in many cases there would be no victim counted for the above mentioned memcg ooms.

Quoted 3 lines

The cgroup v2 documents it, but the description is missed for cgroup v1.

Quoted 21 lines
---
 Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++
 1 file changed, 3 insertions(+)
Documentation/admin-guide/cgroup-v1/memory.rst +3 -0
index 0936412e044e..44d5429636e2 100644
@@ -851,6 +851,9 @@ At reading, current status of OOM is shown.
 	  (if 1, oom-killer is disabled)
 	- under_oom	   0 or 1
 	  (if 1, the memory cgroup is under OOM, tasks may be stopped.)
+        - oom_kill         integer counter
+          The number of processes belonging to this cgroup killed by any
+          kind of OOM killer.
 
 11. Memory Pressure
 ===================
Signature · 1 line
2.26.2
Signature · 2 lines
Michal Hocko
SUSE Labs
author
#2 lore raw
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally

Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.

Quoted 34 lines

confusing because in many cases there would be no victim counted for the above mentioned memcg ooms.

Quoted 3 lines

The cgroup v2 documents it, but the description is missed for cgroup v1.

Quoted 19 lines
---
 Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++
 1 file changed, 3 insertions(+)
Documentation/admin-guide/cgroup-v1/memory.rst +3 -1
index 0936412e044e..44d5429636e2 100644
@@ -851,6 +851,9 @@ At reading, current status of OOM is shown.
        (if 1, oom-killer is disabled)
      - under_oom        0 or 1
        (if 1, the memory cgroup is under OOM, tasks may be stopped.)
+        - oom_kill         integer counter
+          The number of processes belonging to this cgroup killed by any
+          kind of OOM killer.
 
 11. Memory Pressure
 ===================
--
Quoted 2 lines

2.26.2

--
Michal Hocko
SUSE Labs

author
#3 lore raw
Quoted 14 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally

Quoted 4 lines

Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.

BTW, having the counter recorded hierarchically may help out one of our usecases. We want to monitor the oom_kill for some services, but systemd would wipe out the cgroup if the service is oom killed then restart the service from scratch (it means create a brand new cgroup with the same name). So this systemd behavior makes the counter useless if it is not recorded hierarchically.

Quoted 35 lines
Quoted 35 lines

confusing because in many cases there would be no victim counted for the above mentioned memcg ooms.

Quoted 3 lines

The cgroup v2 documents it, but the description is missed for cgroup v1.

Quoted 19 lines
---
 Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++
 1 file changed, 3 insertions(+)
Documentation/admin-guide/cgroup-v1/memory.rst +3 -1
index 0936412e044e..44d5429636e2 100644
@@ -851,6 +851,9 @@ At reading, current status of OOM is shown.
        (if 1, oom-killer is disabled)
      - under_oom        0 or 1
        (if 1, the memory cgroup is under OOM, tasks may be stopped.)
+        - oom_kill         integer counter
+          The number of processes belonging to this cgroup killed by any
+          kind of OOM killer.
 
 11. Memory Pressure
 ===================
--
Quoted 2 lines

2.26.2

--
Michal Hocko
SUSE Labs

#4 lore raw
Quoted 20 lines · Yang Shi
Quoted 14 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally

Quoted 4 lines

Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.

Quoted 6 lines

BTW, having the counter recorded hierarchically may help out one of our usecases. We want to monitor the oom_kill for some services, but systemd would wipe out the cgroup if the service is oom killed then restart the service from scratch (it means create a brand new cgroup with the same name). So this systemd behavior makes the counter useless if it is not recorded hierarchically.

Just to make sure I understand correctly. You have a setup where memcg for a service has a hard limit configured and it is destroyed when oom happens inside that memcg. A new instance is created at the same place of the hierarchy with a new memcg. Your problem is that the oom killed memcg will not be recorded in its parent oom event and the information will get lost with the torn down memcg. Correct?

If yes then how do you tell which of the child cgroup was killed from the parent counter? Or is there only a single child?

Anyway, cgroup v2 will offer the hierarchical behavior. Do you have any strong reasons that you cannot use v2?

Signature · 2 lines
Michal Hocko
SUSE Labs
author
#5 lore raw
Quoted 29 lines · Michal Hocko
Quoted 20 lines · Yang Shi
Quoted 14 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally

Quoted 4 lines

Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.

Quoted 6 lines

BTW, having the counter recorded hierarchically may help out one of our usecases. We want to monitor the oom_kill for some services, but systemd would wipe out the cgroup if the service is oom killed then restart the service from scratch (it means create a brand new cgroup with the same name). So this systemd behavior makes the counter useless if it is not recorded hierarchically.

Quoted 6 lines

Just to make sure I understand correctly. You have a setup where memcg for a service has a hard limit configured and it is destroyed when oom happens inside that memcg. A new instance is created at the same place of the hierarchy with a new memcg. Your problem is that the oom killed memcg will not be recorded in its parent oom event and the information will get lost with the torn down memcg. Correct?

Yes. But global oom instead of memcg oom.

Quoted 3 lines

If yes then how do you tell which of the child cgroup was killed from the parent counter? Or is there only a single child?

Not only a single child, but our case is that oom-killed child consumes 90% memory, then global oom would kill it. This definitely doesn't prevent from accounting oom from other children, but we don't have to have a very accurate counter and in our case we can tell 99% oom kill happens with that specific memcg.

Quoted 3 lines

Anyway, cgroup v2 will offer the hierarchical behavior. Do you have any strong reasons that you cannot use v2?

I do prefer to migrate to cgroup v2 personally. But it incurs significant work for orchestration tools, infrastructure configuration, monitoring tools, etc which are out of my control.

Quoted 3 lines

--
Michal Hocko
SUSE Labs

#6 lore raw
In reply to Yang Shi
Quoted 13 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally

Quoted 4 lines

Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.

I believe the reson is that v1 can have tasks in the intermediate (non-leaf) memcgs. So you wouldn't have a way to tell whether the oom kill has happened in such a memcg or somewhere down the hierarchy.

Signature · 2 lines
Michal Hocko
SUSE Labs
author
#7 lore raw
Quoted 15 lines · Michal Hocko
Quoted 13 lines · Yang Shi
Quoted 11 lines · Michal Hocko
Quoted 4 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

Yes, this is the case indeed. The point of the counter was to count oom victims from the memcg rather than matching that to the source of the oom. Rememeber that this could have been a memcg oom up in the hierarchy as well. Counting victims on the oom origin could be equally

Quoted 8 lines
Quoted 4 lines

Yes, it is updated hierarchically on v2, but not on v1. I'm supposed this is because v1 may work in non-hierarchcal mode? If this is the only reason we may be able to remove this to get aligned with v2 since non-hierarchal mode is no longer supported.

I believe the reson is that v1 can have tasks in the intermediate (non-leaf) memcgs. So you wouldn't have a way to tell whether the oom kill has happened in such a memcg or somewhere down the hierarchy.

Aha, I forgot it, that's bad. Although we don't have tasks in intermediate nodes in practice, I do understand it is not forbidden as cgroup v2.

Quoted 3 lines

--
Michal Hocko
SUSE Labs

Acked-by on v1 1/1
#9 lore raw
In reply to Yang Shi
Quoted 8 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

The cgroup v2 documents it, but the description is missed for cgroup v1.

Thanks.

Quoted 22 lines

---
Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++

1 file changed, 3 insertions(+)
Documentation/admin-guide/cgroup-v1/memory.rst +3 -0
index 0936412e044e..44d5429636e2 100644
@@ -851,6 +851,9 @@ At reading, current status of OOM is shown.
 	  (if 1, oom-killer is disabled)
 	- under_oom	   0 or 1
 	  (if 1, the memory cgroup is under OOM, tasks may be stopped.)
+        - oom_kill         integer counter
+          The number of processes belonging to this cgroup killed by any
+          kind of OOM killer.
 

11. Memory Pressure
===================

Signature · 1 line
2.26.2
maintainer Applied
#10 lore raw
In reply to Yang Shi
Quoted 17 lines · Yang Shi

When debugging an oom issue, I found the oom_kill counter of memcg is confusing. At the first glance without checking document, I thought it just counts for memcg oom, but it turns out it counts both global and memcg oom.

The cgroup v2 documents it, but the description is missed for cgroup v1.

---
 Documentation/admin-guide/cgroup-v1/memory.rst | 3 +++
 1 file changed, 3 insertions(+)
Documentation/admin-guide/cgroup-v1/memory.rst +0 -0
index 0936412e044e..44d5429636e2 100644

Documentation/admin-guide/cgroup-v1/memory.rst At reading, current status of OOM is shown. · line 856

Quoted 6 lines
Documentation/admin-guide/cgroup-v1/memory.rst +3 -0
@@ -851,6 +851,9 @@ At reading, current status of OOM is shown.
 	  (if 1, oom-killer is disabled)
 	- under_oom	   0 or 1
 	  (if 1, the memory cgroup is under OOM, tasks may be stopped.)
+        - oom_kill         integer counter
+          The number of processes belonging to this cgroup killed by any
+          kind of OOM killer.

Applied, thanks.

jon