Skip to contentSkip to conversation

mm: introduce oom_kill_disable sysctl knob

PATCH In review Minchan Kim 3 participants 7 messages Nov 6, 2020 – Nov 9, 2020

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: introduce oom_kill_disable sysctl knob

It's hard to have some tests to be supposed to work under heavy memory pressure(e.g., injecting some memory hogger) because out-of-memory killer easily kicks out one of processes so system is broken or system loses the memory pressure state since it has plenty of free memory soon so.
Even though we could mark existing process's oom_adj to -1000, it couldn't cover upcoming processes to be forked for the job.

This knob is handy to keep system memory pressure.

---
 Documentation/admin-guide/sysctl/vm.rst | 14 ++++++++++++++
 include/linux/mm.h                      |  2 ++
 include/linux/oom.h                     |  1 +
 kernel/sysctl.c                         |  9 +++++++++
 mm/oom_kill.c                           | 24 ++++++++++++++++++++++++
 5 files changed, 50 insertions(+)
Documentation/admin-guide/sysctl/vm.rst +14 -0
index f455fa00c00f..49dcedfaf0c0 100644
@@ -694,6 +694,20 @@ is used in oom_kill_allocating_task.
 
 The default value is 0.
 
+oom_kill_disable
+================
+
+This disables or enables OOM killing in out-of-memory situations.
+
+If this is set to one, the OOM killer is disabled so OOM kill never
+hapens in out-of-memory situation. It could cause system dangerous
+state due to memory allocation failure so user should be careful to
+use it.
+
+If this is set to zero, the OOM killer is enabled so OOM kill happens
+in out-of-memory situations.
+
+The default value is 0.
 
 overcommit_kbytes
 =================
include/linux/mm.h +2 -0
index db6ae4d3fb4e..a98400cee341 100644
@@ -215,6 +215,8 @@ int overcommit_kbytes_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);
 int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);
+int oom_kill_disable_handler(struct ctl_table *, int, void *, size_t *,
+		loff_t *);
 
 #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n))
 
include/linux/oom.h +1 -0
index 2db9a1432511..0f378498e6aa 100644
@@ -126,5 +126,6 @@ extern struct task_struct *find_lock_task_mm(struct task_struct *p);
 /* sysctls */
 extern int sysctl_oom_dump_tasks;
 extern int sysctl_oom_kill_allocating_task;
+extern int sysctl_oom_kill_disable;
 extern int sysctl_panic_on_oom;
 #endif /* _INCLUDE_LINUX_OOM_H */
kernel/sysctl.c +9 -0
index afad085960b8..1fe872fe1c05 100644
@@ -2690,6 +2690,15 @@ static struct ctl_table vm_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec,
 	},
+	{
+		.procname	= "oom_kill_disable",
+		.data		= &sysctl_oom_kill_disable,
+		.maxlen		= sizeof(sysctl_oom_kill_disable),
+		.mode		= 0644,
+		.proc_handler	= oom_kill_disable_handler,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE,
+	},
 	{
 		.procname	= "oom_dump_tasks",
 		.data		= &sysctl_oom_dump_tasks,
mm/oom_kill.c +24 -0
index 8b84661a6410..0f48cdeeb1e7 100644
@@ -53,6 +53,7 @@
 
 int sysctl_panic_on_oom;
 int sysctl_oom_kill_allocating_task;
+int sysctl_oom_kill_disable;
 int sysctl_oom_dump_tasks = 1;
 
 /*
@@ -72,6 +73,29 @@ static inline bool is_memcg_oom(struct oom_control *oc)
 	return oc->memcg != NULL;
 }
 
+int oom_kill_disable_handler(struct ctl_table *table, int write, void *buffer,
+		size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+	if (ret || !write)
+		goto out;
+	if (sysctl_oom_kill_disable == 1) {
+		if (!oom_killer_disable(HZ))
+			ret = -EBUSY;
+	} else {
+		if (mutex_lock_killable(&oom_lock)) {
+			ret = -EBUSY;
+			goto out;
+		}
+		oom_killer_enable();
+		mutex_unlock(&oom_lock);
+	}
+out:
+	return ret;
+}
+
 #ifdef CONFIG_NUMA
 /**
  * oom_cpuset_eligible() - check task eligiblity for kill
Signature · 1 line
2.29.1.341.ge80a0c044ae-goog
reviewer
#1 lore raw

Hi,

Fix a few typos:

Quoted 13 lines · Minchan Kim
---
 Documentation/admin-guide/sysctl/vm.rst | 14 ++++++++++++++
 include/linux/mm.h                      |  2 ++
 include/linux/oom.h                     |  1 +
 kernel/sysctl.c                         |  9 +++++++++
 mm/oom_kill.c                           | 24 ++++++++++++++++++++++++
 5 files changed, 50 insertions(+)
Documentation/admin-guide/sysctl/vm.rst +0 -0
index f455fa00c00f..49dcedfaf0c0 100644

Documentation/admin-guide/sysctl/vm.rst is used in oom_kill_allocating_task. · line 703

Quoted 10 lines
Documentation/admin-guide/sysctl/vm.rst +7 -0
@@ -694,6 +694,20 @@ is used in oom_kill_allocating_task.
 
 The default value is 0.
 
+oom_kill_disable
+================
+
+This disables or enables OOM killing in out-of-memory situations.
+
+If this is set to one, the OOM killer is disabled so OOM kill never
+hapens in out-of-memory situation. It could cause system dangerous

happens It could cause a dangerous system

Documentation/admin-guide/sysctl/vm.rst is used in oom_kill_allocating_task. · line 704

Quoted 1 line
Documentation/admin-guide/sysctl/vm.rst +1 -0
+state due to memory allocation failure so user should be careful to

careful when

Documentation/admin-guide/sysctl/vm.rst is used in oom_kill_allocating_task. · line 705

Quoted 1 line
Documentation/admin-guide/sysctl/vm.rst +1 -0
+use it.

using it.

Quoted 13 lines
Documentation/admin-guide/sysctl/vm.rst +5 -0
+
+If this is set to zero, the OOM killer is enabled so OOM kill happens
+in out-of-memory situations.
+
+The default value is 0.
 
 overcommit_kbytes
 =================
mm/oom_kill.c +0 -0
index 8b84661a6410..0f48cdeeb1e7 100644

mm/oom_kill.c static inline bool is_memcg_oom(struct oom_control *oc) · line 101

Quoted 4 lines
mm/oom_kill.c +0 -0
 
 #ifdef CONFIG_NUMA
 /**
  * oom_cpuset_eligible() - check task eligiblity for kill

eligibility

but that's not in your patch, so don't bother with it. :)

Signature · 1 line
~Randy
#2 lore raw
Quoted 52 lines · Randy Dunlap

Hi,

Fix a few typos:

Quoted 13 lines · Minchan Kim
---
 Documentation/admin-guide/sysctl/vm.rst | 14 ++++++++++++++
 include/linux/mm.h                      |  2 ++
 include/linux/oom.h                     |  1 +
 kernel/sysctl.c                         |  9 +++++++++
 mm/oom_kill.c                           | 24 ++++++++++++++++++++++++
 5 files changed, 50 insertions(+)
Documentation/admin-guide/sysctl/vm.rst +0 -0
index f455fa00c00f..49dcedfaf0c0 100644

Documentation/admin-guide/sysctl/vm.rst is used in oom_kill_allocating_task. · line 703

Quoted 10 lines
Documentation/admin-guide/sysctl/vm.rst +7 -0
@@ -694,6 +694,20 @@ is used in oom_kill_allocating_task.
 
 The default value is 0.
 
+oom_kill_disable
+================
+
+This disables or enables OOM killing in out-of-memory situations.
+
+If this is set to one, the OOM killer is disabled so OOM kill never
+hapens in out-of-memory situation. It could cause system dangerous

happens It could cause a dangerous system

Documentation/admin-guide/sysctl/vm.rst is used in oom_kill_allocating_task. · line 704

Quoted 1 line
Documentation/admin-guide/sysctl/vm.rst +1 -0
+state due to memory allocation failure so user should be careful to

careful when

Documentation/admin-guide/sysctl/vm.rst is used in oom_kill_allocating_task. · line 705

Quoted 1 line
Documentation/admin-guide/sysctl/vm.rst +1 -0
+use it.

using it.

Quoted 13 lines
Documentation/admin-guide/sysctl/vm.rst +5 -0
+
+If this is set to zero, the OOM killer is enabled so OOM kill happens
+in out-of-memory situations.
+
+The default value is 0.
 
 overcommit_kbytes
 =================
mm/oom_kill.c +0 -0
index 8b84661a6410..0f48cdeeb1e7 100644
Quoted 7 lines
Quoted 3 lines

#ifdef CONFIG_NUMA
/**
* oom_cpuset_eligible() - check task eligiblity for kill

eligibility

but that's not in your patch, so don't bother with it. :)

Sure, I will resend it with fixing after waiting some feedback.

Thanks for the review, Randy.

maintainer
#3 lore raw
In reply to Minchan Kim
Quoted 5 lines · Minchan Kim

It's hard to have some tests to be supposed to work under heavy memory pressure(e.g., injecting some memory hogger) because out-of-memory killer easily kicks out one of processes so system is broken or system loses the memory pressure state since it has plenty of free memory soon so.

I do not follow the reasoning here. So you want to test for a close to no memory available situation and the oom killer stands in the way because it puts a relief?

Quoted 2 lines

Even though we could mark existing process's oom_adj to -1000, it couldn't cover upcoming processes to be forked for the job.

Why?

Quoted 1 line

This knob is handy to keep system memory pressure.

This sounds like a very dubious reason to introduce a knob to cripple the system.

I can see some reason to control the oom handling policy because the effect of the oom killer is really disruptive but a global on/off switch sounds like a too coarse interface. Really what kind of production environment would ever go with oom killer disabled completely?

Signature · 2 lines
Michal Hocko
SUSE Labs
#4 lore raw
Quoted 10 lines · Michal Hocko
Quoted 5 lines · Minchan Kim

It's hard to have some tests to be supposed to work under heavy memory pressure(e.g., injecting some memory hogger) because out-of-memory killer easily kicks out one of processes so system is broken or system loses the memory pressure state since it has plenty of free memory soon so.

I do not follow the reasoning here. So you want to test for a close to no memory available situation and the oom killer stands in the way because it puts a relief?

Yub, technically, I'd like to have consistent memory pressure to cause direct reclaims on proesses on the system and swapping in/out.

Quoted 5 lines
Quoted 3 lines

Even though we could mark existing process's oom_adj to -1000, it couldn't cover upcoming processes to be forked for the job.

Why?

Thing is the system has out-of-control processes created on demand. so only option to prevent OOM is echo -1000 > pidof the process since they are forked. However, I have no idea when they are forked so should race with OOM with /proc polling and OOM is frequently ahead of me.

Quoted 10 lines
Quoted 2 lines

This knob is handy to keep system memory pressure.

This sounds like a very dubious reason to introduce a knob to cripple the system.

I can see some reason to control the oom handling policy because the effect of the oom killer is really disruptive but a global on/off switch sounds like a too coarse interface. Really what kind of production environment would ever go with oom killer disabled completely?

I don't think shipping production system will use it. It would be just testing only option.
My intention uses such heavy memory load to see various system behaviors before the production launching because it usually happens in real workload once we shipped but hard to generate such a corner case without artificial memory pressure.

Any suggestion?

maintainer
#5 lore raw
Quoted 19 lines · Minchan Kim
Quoted 10 lines · Michal Hocko
Quoted 5 lines · Minchan Kim

It's hard to have some tests to be supposed to work under heavy memory pressure(e.g., injecting some memory hogger) because out-of-memory killer easily kicks out one of processes so system is broken or system loses the memory pressure state since it has plenty of free memory soon so.

I do not follow the reasoning here. So you want to test for a close to no memory available situation and the oom killer stands in the way because it puts a relief?

Yub, technically, I'd like to have consistent memory pressure to cause direct reclaims on proesses on the system and swapping in/out.

Quoted 4 lines
Quoted 3 lines

Even though we could mark existing process's oom_adj to -1000, it couldn't cover upcoming processes to be forked for the job.

Quoted 7 lines
Quoted 1 line

Why?

Thing is the system has out-of-control processes created on demand. so only option to prevent OOM is echo -1000 > pidof the process since they are forked. However, I have no idea when they are forked so should race with OOM with /proc polling and OOM is frequently ahead of me.

I am still confused. Why would you want all/most processes to be hidden from the oom killer?

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

This knob is handy to keep system memory pressure.

This sounds like a very dubious reason to introduce a knob to cripple the system.

Quoted 7 lines
Quoted 4 lines

I can see some reason to control the oom handling policy because the effect of the oom killer is really disruptive but a global on/off switch sounds like a too coarse interface. Really what kind of production environment would ever go with oom killer disabled completely?

I don't think shipping production system will use it. It would be just testing only option.

Then it doesn't really belong to the kernel IMHO.

Quoted 4 lines

My intention uses such heavy memory load to see various system behaviors before the production launching because it usually happens in real workload once we shipped but hard to generate such a corner case without artificial memory pressure.

But changing the oom behavior will result in a completely different system behavior. So you would be testing something that doesn't really happen in any production system.

Quoted 1 line

Any suggestion?

Not really because I still do not understand your objective. You can generate memory pressure and tune it up for specific testing scenario. Sure there will be a some interference from the background noise (kernel subsystems reacting to external events, processes created etc.) but why that is a problem? This is normal to any running system.

Signature · 2 lines
Michal Hocko
SUSE Labs
#6 lore raw
Quoted 22 lines · Michal Hocko
Quoted 21 lines · Minchan Kim
Quoted 10 lines · Michal Hocko
Quoted 5 lines · Minchan Kim

It's hard to have some tests to be supposed to work under heavy memory pressure(e.g., injecting some memory hogger) because out-of-memory killer easily kicks out one of processes so system is broken or system loses the memory pressure state since it has plenty of free memory soon so.

I do not follow the reasoning here. So you want to test for a close to no memory available situation and the oom killer stands in the way because it puts a relief?

Yub, technically, I'd like to have consistent memory pressure to cause direct reclaims on proesses on the system and swapping in/out.

Quoted 5 lines
Quoted 3 lines

Even though we could mark existing process's oom_adj to -1000, it couldn't cover upcoming processes to be forked for the job.

Why?

Quoted 8 lines
Quoted 5 lines

Thing is the system has out-of-control processes created on demand. so only option to prevent OOM is echo -1000 > pidof the process since they are forked. However, I have no idea when they are forked so should race with OOM with /proc polling and OOM is frequently ahead of me.

I am still confused. Why would you want all/most processes to be hidden from the oom killer?

If one of processes in the system is killed, the memory pressure disappear.

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

This knob is handy to keep system memory pressure.

This sounds like a very dubious reason to introduce a knob to cripple the system.

Quoted 7 lines
Quoted 4 lines

I can see some reason to control the oom handling policy because the effect of the oom killer is really disruptive but a global on/off switch sounds like a too coarse interface. Really what kind of production environment would ever go with oom killer disabled completely?

I don't think shipping production system will use it. It would be just testing only option.

Then it doesn't really belong to the kernel IMHO.

Quoted 8 lines
Quoted 4 lines

My intention uses such heavy memory load to see various system behaviors before the production launching because it usually happens in real workload once we shipped but hard to generate such a corner case without artificial memory pressure.

But changing the oom behavior will result in a completely different system behavior. So you would be testing something that doesn't really happen in any production system.

Since OOM is not instantly reacted, it still provides a good chance how the system reacts on such memory pressure until someeone releases the memory. For example, Android already gives lots of system processes to -1000 which could effectively disable the OOM at certain point.

Quoted 8 lines
Quoted 2 lines

Any suggestion?

Not really because I still do not understand your objective. You can generate memory pressure and tune it up for specific testing scenario. Sure there will be a some interference from the background noise (kernel subsystems reacting to external events, processes created etc.) but why that is a problem? This is normal to any running system.

Putting more pressure from background processes is okay for the goal but not okay for relieving th memory pressure since we lost the testing environment.