Memory overcommit

/proc/meminfo

MemTotal:        3542532 kB
MemFree:          892972 kB
Buffers:            2664 kB
Cached:           130940 kB

...that there is almost 900MB free memory. But OK, I can live with it.

So, my question is: why today overcommit isn't turned off *by default*? I have it turned off for a few years now and only side effect is that I don't get processes killed randomly anymore, I don't loose valuable time and data.

Quoted 6 lines · On Mon, 12 Oct 2009 13:51:07 +0200 Vedran Furač <[email protected]> wrote:

/proc/meminfo MemTotal: 3542532 kB MemFree: 892972
kB Buffers: 2664 kB Cached: 130940 kB

...that there is almost 900MB free memory. But OK, I can live with it.

Quoted 5 lines

So, my question is: why today overcommit isn't turned off *by default*? I have it turned off for a few years now and only side effect is that I don't get processes killed randomly anymore, I don't loose valuable time and data.

"isn't turned off" means "vm.overcommit_memory==2" ?

And...what's version your kernel is ?

oom-killer still finds "definitely-not-guilty" ones ?

I guess the reason of default value is that the kernel assume processes will not always use all mmaped range. There will be unused range in process's virtual memory and it can be big.

For example, typical case in a server, when you run multi-thread program (like java VM),

- stack per thread - malloc() arena per thread

can makes difference among size-of-mapped-range v.s. used-pages bigger. I saw Gigabytes of unused range on ia64 host,...statck size was big.

And if strict check(vm.ovecommit_memory=2) is used, mmap() return -ENOMEM whenever it hits limit.

Against random-kill, you may have 2 choices.

1. use /proc/<pid>/oom_adj 2. use memory cgroup.

Something more easy-to-use method may be appriciated. We have above 2 now.

Quoted 4 lines · KAMEZAWA Hiroyuki

I guess the reason of default value is that the kernel assume processes will not always use all mmaped range. There will be unused range in process's virtual memory and it can be big.

Quoted 8 lines

For example, typical case in a server, when you run multi-thread program (like java VM),

- stack per thread - malloc() arena per thread

can makes difference among size-of-mapped-range v.s. used-pages bigger. I saw Gigabytes of unused range on ia64 host,...statck size was big.

Yes, I noticed that JVM allocates gigabytes but then uses less than 10% of that and, as a consequence, eclipse sometimes fails to start although there's plenty of free memory. So overcommiting is some kind of a workaround for broken software that allocate not what they need but what they might need in some rare occurrences. I would rather like fixing this userland software than risking OOM situations and random killing of innocent processes.

Quoted 2 lines

And if strict check(vm.ovecommit_memory=2) is used, mmap() return -ENOMEM whenever it hits limit.

% strace -f -e mmap java -version
[...]
mmap(NULL, 996147200, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)

And that should be fine.

Quoted 6 lines

Against random-kill, you may have 2 choices.

1. use /proc/<pid>/oom_adj 2. use memory cgroup.

Something more easy-to-use method may be appriciated. We have above 2 now.

These are just bad workarounds for bad OOM algorithm. I tested this little program on multiple systems (including windows) without any tweaking and linux behavior is, unfortunately *the worst*. :/

Quoted 4 lines · On Tue, 13 Oct 2009 19:13:34 +0200 Vedran FuraA? <[email protected]>…

Quoted 4 lines

Against random-kill, you may have 2 choices.

1. use /proc/<pid>/oom_adj 2. use memory cgroup.

Quoted 6 lines

Quoted 2 lines

Something more easy-to-use method may be appriciated. We have above 2 now.

These are just bad workarounds for bad OOM algorithm. I tested this little program on multiple systems (including windows) without any tweaking and linux behavior is, unfortunately *the worst*. :/

Yes, they are workaround. You can use /etc/sysctl.conf. But if making it default _now_, many threaded programs will not work.

But I agree, OOM killer should be sophisticated.
Please give us a sample program/test case which causes problem. [email protected] may be a better place. lkml has too much traffic.

Hi and sorry for delay. Also, please CC me.

Quoted 3 lines

But I agree, OOM killer should be sophisticated.
Please give us a sample program/test case which causes problem. [email protected] may be a better place. lkml has too much traffic.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>

int main()
{
char *buf;

  while(1) {
    buf = malloc (1024*1024*100);
    if ( buf == NULL ) {
      perror("malloc");
      getchar();
      exit(EXIT_FAILURE);
    }
    sleep(1);
    memset(buf, 1, 1024*1024*100);

}
return 0;
}

After running this on a typical desktop with gnome or kde, OOM killer will kill 5-10 innocent processes before killing this one. Tested multiple times on multiple installations.

Regards,

Vedran

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Can I make more questions ?

- What's cpu ?

- How much memory ?
- Do you have swap ?

- What's the latest kernel version you tested?

- Could you show me /var/log/dmesg and /var/log/messages at OOM ?

Quoted 1 line

- Could you show me /var/log/dmesg and /var/log/messages at OOM ?

It was catastrophe. :) X crashed (or killed) with all the programs, but my little program was alive for 20 minutes (see timestamps). And for that time computer was completely unusable. Couldn't even get the console via ssh. Rally embarrassing for a modern OS to get destroyed by a 5 lines of C run as an ordinary user. Luckily screen was still alive, oomk usually kills it also. See for yourself:

dmesg: http://pastebin.com/f3f83738a
messages: http://pastebin.com/f2091110a

(CCing to lklm again... I just want people to see the logs.)

Quoted 9 lines · On Mon, 26 Oct 2009 17:16:14 +0100 Vedran Furač <[email protected]> wrote:

Quoted 1 line

 - Could you show me /var/log/dmesg and /var/log/messages at OOM ?

It was catastrophe. :) X crashed (or killed) with all the programs, but my little program was alive for 20 minutes (see timestamps). And for that time computer was completely unusable. Couldn't even get the console via ssh. Rally embarrassing for a modern OS to get destroyed by a 5 lines of C run as an ordinary user. Luckily screen was still alive, oomk usually kills it also. See for yourself:

Quoted 5 lines

dmesg: http://pastebin.com/f3f83738a
messages: http://pastebin.com/f2091110a

(CCing to lklm again... I just want people to see the logs.)

Thank you for reporting and your patience. It seems something strange that your KDE programs are killed. I agree.

I attached a scirpt for checking oom_score of all exisiting process. (oom_score is a value used for selecting "bad" processs.") please run if you have time.

This is a result of my own desktop(on virtual machine.)
In this environ (Total memory is 1.6GBytes), mmap(1G) program is running.

%check_badness.pl | sort -n | tail
--
89924   3938    mixer_applet2
90210   3942    tomboy
94753   3936    clock-applet
101994  3919    pulseaudio
113525  4028    gnome-terminal
127340  1       init
128177  3871    nautilus
151003  11515   bash

256944  11653   mmap
425561  3829    gnome-session
--
Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?
(CCed kosaki)

2009/10/27 KAMEZAWA Hiroyuki <[email protected]>:

Quoted 36 lines

Quoted 9 lines · On Mon, 26 Oct 2009 17:16:14 +0100 Vedran Furač <[email protected]> wrote:

Quoted 1 line

 - Could you show me /var/log/dmesg and /var/log/messages at OOM ?

It was catastrophe. :) X crashed (or killed) with all the programs, but my little program was alive for 20 minutes (see timestamps). And for that time computer was completely unusable. Couldn't even get the console via ssh. Rally embarrassing for a modern OS to get destroyed by a 5 lines of C run as an ordinary user. Luckily screen was still alive, oomk usually kills it also. See for yourself:

Quoted 5 lines

dmesg: http://pastebin.com/f3f83738a
messages: http://pastebin.com/f2091110a

(CCing to lklm again... I just want people to see the logs.)

Thank you for reporting and your patience. It seems something strange that your KDE programs are killed. I agree.

I attached a scirpt for checking oom_score of all exisiting process. (oom_score is a value used for selecting "bad" processs.") please run if you have time.

This is a result of my own desktop(on virtual machine.)
In this environ (Total memory is 1.6GBytes), mmap(1G) program is running.

%check_badness.pl | sort -n | tail
--
89924   3938    mixer_applet2
90210   3942    tomboy
94753   3936    clock-applet
101994  3919    pulseaudio
113525  4028    gnome-terminal
127340  1       init
128177  3871    nautilus
151003  11515   bash

Quoted 8 lines

256944  11653   mmap
425561  3829    gnome-session
--
Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?
(CCed kosaki)

Following output address the issue.
The fact is, modern desktop application linked pretty many library. it makes bloat VSS size and increase
OOM score.

Ideally, We shouldn't account evictable file-backed mappings for oom_score.

Quoted 51 lines · On Tue, 27 Oct 2009 15:10:52 +0900 KOSAKI Motohiro…

2009/10/27 KAMEZAWA Hiroyuki <[email protected]>:

Quoted 36 lines

Quoted 9 lines · On Mon, 26 Oct 2009 17:16:14 +0100 Vedran Furač <[email protected]> wrote:

Quoted 1 line

 - Could you show me /var/log/dmesg and /var/log/messages at OOM ?

It was catastrophe. :) X crashed (or killed) with all the programs, but my little program was alive for 20 minutes (see timestamps). And for that time computer was completely unusable. Couldn't even get the console via ssh. Rally embarrassing for a modern OS to get destroyed by a 5 lines of C run as an ordinary user. Luckily screen was still alive, oomk usually kills it also. See for yourself:

Quoted 5 lines

dmesg: http://pastebin.com/f3f83738a
messages: http://pastebin.com/f2091110a

(CCing to lklm again... I just want people to see the logs.)

Thank you for reporting and your patience. It seems something strange that your KDE programs are killed. I agree.

I attached a scirpt for checking oom_score of all exisiting process. (oom_score is a value used for selecting "bad" processs.") please run if you have time.

This is a result of my own desktop(on virtual machine.)
In this environ (Total memory is 1.6GBytes), mmap(1G) program is running.

%check_badness.pl | sort -n | tail
--
89924   3938    mixer_applet2
90210   3942    tomboy
94753   3936    clock-applet
101994  3919    pulseaudio
113525  4028    gnome-terminal
127340  1       init
128177  3871    nautilus
151003  11515   bash

Quoted 8 lines

256944  11653   mmap
425561  3829    gnome-session
--
Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?
(CCed kosaki)

Following output address the issue.
The fact is, modern desktop application linked pretty many library. it makes bloat VSS size and increase
OOM score.

Quoted 2 lines

Ideally, We shouldn't account evictable file-backed mappings for oom_score.

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Quoted 54 lines · On Tue, 27 Oct 2009 15:34:29 +0900 Minchan Kim <[email protected]> wrote:

Quoted 51 lines · On Tue, 27 Oct 2009 15:10:52 +0900 KOSAKI Motohiro…

2009/10/27 KAMEZAWA Hiroyuki <[email protected]>:

Quoted 36 lines

Quoted 9 lines · On Mon, 26 Oct 2009 17:16:14 +0100 Vedran Furač <[email protected]> wrote:

Quoted 1 line

 - Could you show me /var/log/dmesg and /var/log/messages at OOM ?

It was catastrophe. :) X crashed (or killed) with all the programs, but my little program was alive for 20 minutes (see timestamps). And for that time computer was completely unusable. Couldn't even get the console via ssh. Rally embarrassing for a modern OS to get destroyed by a 5 lines of C run as an ordinary user. Luckily screen was still alive, oomk usually kills it also. See for yourself:

Quoted 5 lines

dmesg: http://pastebin.com/f3f83738a
messages: http://pastebin.com/f2091110a

(CCing to lklm again... I just want people to see the logs.)

Thank you for reporting and your patience. It seems something strange that your KDE programs are killed. I agree.

I attached a scirpt for checking oom_score of all exisiting process. (oom_score is a value used for selecting "bad" processs.") please run if you have time.

This is a result of my own desktop(on virtual machine.)
In this environ (Total memory is 1.6GBytes), mmap(1G) program is running.

%check_badness.pl | sort -n | tail
--
89924   3938    mixer_applet2
90210   3942    tomboy
94753   3936    clock-applet
101994  3919    pulseaudio
113525  4028    gnome-terminal
127340  1       init
128177  3871    nautilus
151003  11515   bash

Quoted 8 lines

256944  11653   mmap
425561  3829    gnome-session
--
Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?
(CCed kosaki)

Following output address the issue.
The fact is, modern desktop application linked pretty many library. it makes bloat VSS size and increase
OOM score.

Quoted 6 lines

Quoted 2 lines

Ideally, We shouldn't account evictable file-backed mappings for oom_score.

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Maybe the current code assumes "Tons of swap have been generated, already" if oom-kill is invoked. Then, just using mm->anon_rss will not be correct.

Hm, should we count # of swap entries reference from mm ?....

Regards,
-Kame

Quoted 5 lines

Quoted 4 lines

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Quoted 4 lines

Maybe the current code assumes "Tons of swap have been generated, already" if oom-kill is invoked. Then, just using mm->anon_rss will not be correct.

Hm, should we count # of swap entries reference from mm ?....

In Vedran case, he didn't use swap. So, Only considering vm is the problem. I think it would be better to consider both RSS + # of swap entries as Kosaki mentioned.

Quoted 5 lines · On Tue, 27 Oct 2009 15:55:26 +0900 Minchan Kim <[email protected]> wrote:

Quoted 5 lines

Quoted 4 lines

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Quoted 9 lines

Quoted 4 lines

Maybe the current code assumes "Tons of swap have been generated, already" if oom-kill is invoked. Then, just using mm->anon_rss will not be correct.

Hm, should we count # of swap entries reference from mm ?....

In Vedran case, he didn't use swap. So, Only considering vm is the problem. I think it would be better to consider both RSS + # of swap entries as Kosaki mentioned.

Then, maybe this kind of patch is necessary.
This is on 2.6.31...then I may have to rebase this to mmotom. Added more CCs.

Vedran, I'm glad if you can test this patch.

==
Now, oom-killer's score uses mm->total_vm as its base value. But, in these days, applications like GUI program tend to use much shared libraries and total_vm grows too high even when pages are not fully mapped.

For example, running a program "mmap" which allocates 1 GBbytes of anonymous memory, oom_score top 10 on system will be..

score PID name
89924 3938 mixer_applet2
90210 3942 tomboy
94753 3936 clock-applet
101994 3919 pulseaudio
113525 4028 gnome-terminal
127340 1 init
128177 3871 nautilus
151003 11515 bash
256944 11653 mmap <-----------------use 1G of anon
425561 3829 gnome-session

No one believes gnome-session is more guilty than "mmap".

Instead of total_vm, we should use anon/file/swap usage of a process, I think. This patch adds mm->swap_usage and calculate oom_score based on
anon_rss + file_rss + swap_usage.
Considering usual applications, this will be much better information than total_vm. After this patch, the score on my desktop is

score PID name
4033 3176 gnome-panel
4077 3113 xinit
4526 3190 python
4820 3161 gnome-settings-
4989 3289 gnome-terminal
7105 3271 tomboy
8427 3177 nautilus
17549 3140 gnome-session
128501 3299 bash
256106 3383 mmap

This order is not bad, I think.

Note: This adss new counter...then new cost is added.

Signed-off-by KAMEZAWA Hiroyuki <[email protected]>

Thanks for making the patch.
Let's hear other's opinion. :)

Quoted 2 lines · Minchan Kim

Thanks for making the patch.
Let's hear other's opinion. :)

total_vm is nearly meaningless, especially on 64bit that reduces the mmap load on libs, I tried to change it to something "physical" (rss, didn't add swap too) some time ago too, not sure why I didn't manage to get it in. Trying again surely sounds good. Accounting swap isn't necessarily good, we may be killing a task that isn't accessing memory at all. So yes, we free swap but if the task is the "bloater" it's unlikely to be all in swap as it did all recent activity that lead to the oom. So I'm unsure if swap is good to account here, but surely I ack to replace virtual with rss. I would include the whole rss, as the file one may also be rendered unswappable if it is accessed in a loop refreshing the young bit all the time.

Hmm, maybe
anon_rss + file_rss/2 + swap_usage/4 + kosaki's time accounting change
can give us some better value. I'll consider what number is logical and technically correct, again.

mm/oom_kill.c +0 -0

 	 * After this unlock we can no longer dereference local variable `mm'

@@ -92,8 +93,13 @@ unsigned long badness(struct task_struct
 	 */
 	list_for_each_entry(child, &p->children, sibling) {
 		task_lock(child);

On Tue, 27 Oct 2009 16:45:26 +0900
KAMEZAWA Hiroyuki <[email protected]> wrote:
/*

Quoted 5 lines

mm/oom_kill.c +0 -0

 	 * After this unlock we can no longer dereference local variable `mm'

@@ -92,8 +93,13 @@ unsigned long badness(struct task_struct
 	 */
 	list_for_each_entry(child, &p->children, sibling) {
 		task_lock(child);

mm/oom_kill.c unsigned long badness(struct task_struct · line 103

Quoted 10 lines

mm/oom_kill.c +7 -2

-		if (child->mm != mm && child->mm)
-			points += child->mm->total_vm/2 + 1;
+		if (child->mm != mm && child->mm) {
+			unsigned long cpoint;
+			/* At considering child, we don't count swap */
+			cpoint = get_mm_counter(child->mm, anon_rss) +
+				 get_mm_counter(child->mm, file_rss);
+			points += cpoint/2 + 1;
+		}
 		task_unlock(child);

BTW, I'd like to get rid of this code.

Can't we use other techniques for detecting fork-bomb ?

This check can't catch following type, anyway.

   fork()
     -> fork()
          -> fork()
               -> fork()
                    ....

but I have no good idea.
What is the difference with task-launcher and fork bomb()...

Thanks,
-Kame

Quoted 25 lines · On Tue, 27 Oct 2009 16:56:28 +0900 KAMEZAWA Hiroyuki…

On Tue, 27 Oct 2009 16:45:26 +0900
KAMEZAWA Hiroyuki <[email protected]> wrote:
/*

Quoted 5 lines

mm/oom_kill.c +0 -0

 	 * After this unlock we can no longer dereference local variable `mm'

@@ -92,8 +93,13 @@ unsigned long badness(struct task_struct
 	 */
 	list_for_each_entry(child, &p->children, sibling) {
 		task_lock(child);

mm/oom_kill.c unsigned long badness(struct task_struct · line 103

Quoted 10 lines

mm/oom_kill.c +7 -2

-		if (child->mm != mm && child->mm)
-			points += child->mm->total_vm/2 + 1;
+		if (child->mm != mm && child->mm) {
+			unsigned long cpoint;
+			/* At considering child, we don't count swap */
+			cpoint = get_mm_counter(child->mm, anon_rss) +
+				 get_mm_counter(child->mm, file_rss);
+			points += cpoint/2 + 1;
+		}
 		task_unlock(child);

BTW, I'd like to get rid of this code.

Can't we use other techniques for detecting fork-bomb ?

This check can't catch following type, anyway.

Quoted 9 lines

   fork()
     -> fork()
          -> fork()
               -> fork()
                    ....

but I have no good idea.
What is the difference with task-launcher and fork bomb()...

I think it's good as-is.
Kernel is hard to know it by effiecient method. It depends on applications. so Doesnt's task-launcher like gnome-session have to control his oom_score?

Welcome to any ideas if kernel can do it well.

Quoted 38 lines · On Tue, 27 Oct 2009 17:14:41 +0900 Minchan Kim <[email protected]> wrote:

Quoted 34 lines · On Tue, 27 Oct 2009 16:56:28 +0900 KAMEZAWA Hiroyuki…

On Tue, 27 Oct 2009 16:45:26 +0900
KAMEZAWA Hiroyuki <[email protected]> wrote:
/*

Quoted 5 lines

mm/oom_kill.c +0 -0

 	 * After this unlock we can no longer dereference local variable `mm'

@@ -92,8 +93,13 @@ unsigned long badness(struct task_struct
 	 */
 	list_for_each_entry(child, &p->children, sibling) {
 		task_lock(child);

mm/oom_kill.c unsigned long badness(struct task_struct · line 103

Quoted 10 lines

mm/oom_kill.c +7 -2

-		if (child->mm != mm && child->mm)
-			points += child->mm->total_vm/2 + 1;
+		if (child->mm != mm && child->mm) {
+			unsigned long cpoint;
+			/* At considering child, we don't count swap */
+			cpoint = get_mm_counter(child->mm, anon_rss) +
+				 get_mm_counter(child->mm, file_rss);
+			points += cpoint/2 + 1;
+		}
 		task_unlock(child);

BTW, I'd like to get rid of this code.

Can't we use other techniques for detecting fork-bomb ?

This check can't catch following type, anyway.

   fork()
     -> fork()
          -> fork()
               -> fork()
                    ....

but I have no good idea.
What is the difference with task-launcher and fork bomb()...

Quoted 7 lines

I think it's good as-is.
Kernel is hard to know it by effiecient method. It depends on applications. so Doesnt's task-launcher like gnome-session have to control his oom_score?

Welcome to any ideas if kernel can do it well.

Hmmm, check system-wide fork/sec and fork-depth ? Maybe not difficult to calculate..

Regards,
-Kame

Quoted 46 lines · On Tue, 27 Oct 2009 17:33:08 +0900 KAMEZAWA Hiroyuki…

Quoted 43 lines · On Tue, 27 Oct 2009 17:14:41 +0900 Minchan Kim <[email protected]> wrote:

Quoted 25 lines · On Tue, 27 Oct 2009 16:56:28 +0900 KAMEZAWA Hiroyuki…

On Tue, 27 Oct 2009 16:45:26 +0900
KAMEZAWA Hiroyuki <[email protected]> wrote:
/*

Quoted 5 lines

mm/oom_kill.c +0 -0

 	 * After this unlock we can no longer dereference local variable `mm'

@@ -92,8 +93,13 @@ unsigned long badness(struct task_struct
 	 */
 	list_for_each_entry(child, &p->children, sibling) {
 		task_lock(child);

mm/oom_kill.c unsigned long badness(struct task_struct · line 103

Quoted 10 lines

mm/oom_kill.c +7 -2

-		if (child->mm != mm && child->mm)
-			points += child->mm->total_vm/2 + 1;
+		if (child->mm != mm && child->mm) {
+			unsigned long cpoint;
+			/* At considering child, we don't count swap */
+			cpoint = get_mm_counter(child->mm, anon_rss) +
+				 get_mm_counter(child->mm, file_rss);
+			points += cpoint/2 + 1;
+		}
 		task_unlock(child);

BTW, I'd like to get rid of this code.

Can't we use other techniques for detecting fork-bomb ?

This check can't catch following type, anyway.

Quoted 9 lines

   fork()
     -> fork()
          -> fork()
               -> fork()
                    ....

but I have no good idea.
What is the difference with task-launcher and fork bomb()...

I think it's good as-is.
Kernel is hard to know it by effiecient method. It depends on applications. so Doesnt's task-launcher like gnome-session have to control his oom_score?

Quoted 3 lines

Quoted 2 lines

Welcome to any ideas if kernel can do it well.

Hmmm, check system-wide fork/sec and fork-depth ? Maybe not difficult to calculate..

Yes. We can do anything to achieve the goal in kernel. Maybe check the time or fork-depth counting.
What I have a concern is how we can do it nicely if it is a serious problem in kernel. ;)

I think most of program which have many child are victims of OOM killing. It make sense to me. There is some cases to not make sense like task-launcher. So I think if task-launcher which is very rare and special program can change oom_adj by itself, it's good than thing that add new heuristic in kernel.

It's just my opinon. :)

Quoted 4 lines · On Tue, 27 Oct 2009 15:55:26 +0900 Minchan Kim <[email protected]> wrote:

Quoted 4 lines

Quoted 4 lines

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Quoted 8 lines

Quoted 4 lines

Maybe the current code assumes "Tons of swap have been generated, already" if oom-kill is invoked. Then, just using mm->anon_rss will not be correct.

Hm, should we count # of swap entries reference from mm ?....

In Vedran case, he didn't use swap. So, Only considering vm is the problem. I think it would be better to consider both RSS + # of swap entries as Kosaki mentioned.

Then, maybe this kind of patch is necessary.
This is on 2.6.31...then I may have to rebase this to mmotom. Added more CCs.

Vedran, I'm glad if you can test this patch.

Instead of total_vm, we should use anon/file/swap usage of a process, I think. This patch adds mm->swap_usage and calculate oom_score based on
anon_rss + file_rss + swap_usage.

Considering usual applications, this will be much better information than total_vm.

score PID name
4033 3176 gnome-panel
4077 3113 xinit
4526 3190 python
4820 3161 gnome-settings-
4989 3289 gnome-terminal
7105 3271 tomboy
8427 3177 nautilus
17549 3140 gnome-session
128501 3299 bash
256106 3383 mmap

This order is not bad, I think.

Quoted 15 lines · KAMEZAWA Hiroyuki

Quoted 4 lines · On Tue, 27 Oct 2009 15:55:26 +0900 Minchan Kim <[email protected]> wrote:

Quoted 4 lines

Quoted 4 lines

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Quoted 8 lines

Quoted 4 lines

Maybe the current code assumes "Tons of swap have been generated, already" if oom-kill is invoked. Then, just using mm->anon_rss will not be correct.

Hm, should we count # of swap entries reference from mm ?....

In Vedran case, he didn't use swap. So, Only considering vm is the problem. I think it would be better to consider both RSS + # of swap entries as Kosaki mentioned.

Quoted 5 lines

Then, maybe this kind of patch is necessary.
This is on 2.6.31...then I may have to rebase this to mmotom. Added more CCs.

Vedran, I'm glad if you can test this patch.

Thanks for the patch! I'll test it during this week a report after that.

Quoted 3 lines

Instead of total_vm, we should use anon/file/swap usage of a process, I think. This patch adds mm->swap_usage and calculate oom_score based on
anon_rss + file_rss + swap_usage.

Isn't file_rss shared between processes? Sorry, I'm newbie. :)

% pmap $(pidof test)
29049: ./test

0000000000400000      4K r-x--  /home/vedranf/dev/tmp/test
0000000000600000      4K rw---  /home/vedranf/dev/tmp/test
00002ba362a80000    116K r-x--  /lib/ld-2.10.1.so
00002ba362a9d000     12K rw---    [ anon ]
00002ba362c9c000      4K r----  /lib/ld-2.10.1.so
00002ba362c9d000      4K rw---  /lib/ld-2.10.1.so
00002ba362c9e000   1320K r-x--  /lib/libc-2.10.1.so
00002ba362de8000   2044K -----  /lib/libc-2.10.1.so
00002ba362fe7000     16K r----  /lib/libc-2.10.1.so
00002ba362feb000      4K rw---  /lib/libc-2.10.1.so
00002ba362fec000 1024028K rw---    [ anon ] // <-- This
00007ffff4618000     84K rw---    [ stack ]
00007ffff47b7000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]

total 1027648K

I would just look at anon if that's OK (or possible).

Quoted 2 lines

Considering usual applications, this will be much better information than total_vm.

Agreed.

Quoted 12 lines

score PID name
4033 3176 gnome-panel
4077 3113 xinit
4526 3190 python
4820 3161 gnome-settings-
4989 3289 gnome-terminal
7105 3271 tomboy
8427 3177 nautilus
17549 3140 gnome-session
128501 3299 bash
256106 3383 mmap

Quoted 1 line

This order is not bad, I think.

Yes, this looks much better now. Bash is only having somewhat strangely high score.

Regards,

Vedran

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Now, oom-killer's score uses mm->total_vm as its base value. But, in these days, applications like GUI program tend to use much shared libraries and total_vm grows too high even when pages are not fully mapped.

For example, running a program "mmap" which allocates 1 GBbytes of anonymous memory, oom_score top 10 on system will be..

score PID name
89924 3938 mixer_applet2
90210 3942 tomboy
94753 3936 clock-applet
101994 3919 pulseaudio
113525 4028 gnome-terminal
127340 1 init
128177 3871 nautilus
151003 11515 bash
256944 11653 mmap <-----------------use 1G of anon
425561 3829 gnome-session

No one believes gnome-session is more guilty than "mmap".

Instead of total_vm, we should use anon/file/swap usage of a process, I think. This patch adds mm->swap_usage and calculate oom_score based on
anon_rss + file_rss + swap_usage.
Considering usual applications, this will be much better information than total_vm. After this patch, the score on my desktop is

score PID name
4033 3176 gnome-panel
4077 3113 xinit
4526 3190 python
4820 3161 gnome-settings-
4989 3289 gnome-terminal
7105 3271 tomboy
8427 3177 nautilus
17549 3140 gnome-session
128501 3299 bash
256106 3383 mmap

This order is not bad, I think.

Note: This adss new counter...then new cost is added.

OOM (physical memory) decisions on total_vm (virtual memory) has seemed weird, so it's well worth trying this approach. Whether swap

should be included along with rss isn't quite clear to me: I'm not saying you're wrong, not at all, just that it's not quite obvious.

I've several observations to make about bad OOM kill decisions, but it's probably better that I make them in the original "Memory overcommit" thread, rather than divert this thread.

Quoted 2 lines

should be included along with rss isn't quite clear to me: I'm not saying you're wrong, not at all, just that it's not quite obvious.

Agreed it's not obvious. Intuitively I think only including RSS and no swap is best, but clearly I can't be entirely against including swap too as there may be scenarios where including swap provides for a better choice.

My argument for not including swap is that we kill tasks to free RAM (we don't really care to free swap, system needs RAM at oom time). Freeing swap won't immediately help because no RAM is freed when swap is released (sure other tasks that sits huge in RAM can be moved to swap after swap isn't full but if we immediately killed those tasks that were huge in RAM in the first place we'd be better off).

Agreed it's not obvious. Intuitively I think only including RSS and no swap is best, but clearly I can't be entirely against including swap too as there may be scenarios where including swap provides for a better choice.

My argument for not including swap is that we kill tasks to free RAM (we don't really care to free swap, system needs RAM at oom time).

Quoted 40 lines · KAMEZAWA Hiroyuki

Now, oom-killer's score uses mm->total_vm as its base value. But, in these days, applications like GUI program tend to use much shared libraries and total_vm grows too high even when pages are not fully mapped.

For example, running a program "mmap" which allocates 1 GBbytes of anonymous memory, oom_score top 10 on system will be..

score PID name
89924 3938 mixer_applet2
90210 3942 tomboy
94753 3936 clock-applet
101994 3919 pulseaudio
113525 4028 gnome-terminal
127340 1 init
128177 3871 nautilus
151003 11515 bash
256944 11653 mmap <-----------------use 1G of anon
425561 3829 gnome-session

No one believes gnome-session is more guilty than "mmap".

Instead of total_vm, we should use anon/file/swap usage of a process, I think. This patch adds mm->swap_usage and calculate oom_score based on
anon_rss + file_rss + swap_usage.
Considering usual applications, this will be much better information than total_vm. After this patch, the score on my desktop is

score PID name
4033 3176 gnome-panel
4077 3113 xinit
4526 3190 python
4820 3161 gnome-settings-
4989 3289 gnome-terminal
7105 3271 tomboy
8427 3177 nautilus
17549 3140 gnome-session
128501 3299 bash
256106 3383 mmap

Quoted 3 lines

This order is not bad, I think.

Note: This adss new counter...then new cost is added.

I've often thought we ought to supply such a swap_usage statistic; and show it in /proc/pid/statsomething, presumably VmSwap in /proc/pid/status, even an additional field on the end of statm.

A slight new cost, yes: doesn't matter at the swapping end, but would slightly impact fork and exit - I do hope we can afford it, because I think it should have been available all along.

I've not checked your patch in detail; but I do agree that basing OOM (physical memory) decisions on total_vm (virtual memory) has seemed weird, so it's well worth trying this approach. Whether swap should be included along with rss isn't quite clear to me: I'm not saying you're wrong, not at all, just that it's not quite obvious.

I've several observations to make about bad OOM kill decisions, but it's probably better that I make them in the original "Memory overcommit" thread, rather than divert this thread.

Quoted 24 lines

Quoted 10 lines

%check_badness.pl | sort -n | tail
--
89924   3938    mixer_applet2
90210   3942    tomboy
94753   3936    clock-applet
101994  3919    pulseaudio
113525  4028    gnome-terminal
127340  1       init
128177  3871    nautilus
151003  11515   bash

Quoted 8 lines

256944  11653   mmap
425561  3829    gnome-session
--
Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?
(CCed kosaki)

Following output address the issue.
The fact is, modern desktop application linked pretty many library. it makes bloat VSS size and increase
OOM score.

Quoted 2 lines

Ideally, We shouldn't account evictable file-backed mappings for oom_score.

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Quoted 26 lines

Quoted 26 lines

Quoted 10 lines

%check_badness.pl | sort -n | tail
--
89924   3938    mixer_applet2
90210   3942    tomboy
94753   3936    clock-applet
101994  3919    pulseaudio
113525  4028    gnome-terminal
127340  1       init
128177  3871    nautilus
151003  11515   bash

Quoted 8 lines

256944  11653   mmap
425561  3829    gnome-session
--
Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?
(CCed kosaki)

Following output address the issue.
The fact is, modern desktop application linked pretty many library. it makes bloat VSS size and increase
OOM score.

Ideally, We shouldn't account evictable file-backed mappings for oom_score.

Quoted 3 lines

Hmm.
I wonder why we consider VM size for OOM kiling.
How about RSS size?

Because, swap out-ed bad body (e.g. fork bomb process) still should be killed by oom.
RSS + swap-entries is acceptable to me.

Quoted 8 lines · On Mon, 26 Oct 2009 17:16:14 +0100 Vedran Furač <[email protected]> wrote:

Quoted 1 line

- Could you show me /var/log/dmesg and /var/log/messages at OOM ?

It was catastrophe. :) X crashed (or killed) with all the programs, but my little program was alive for 20 minutes (see timestamps). And for that time computer was completely unusable. Couldn't even get the console via ssh. Rally embarrassing for a modern OS to get destroyed by a 5 lines of C run as an ordinary user. Luckily screen was still alive, oomk usually kills it also. See for yourself:

Quoted 5 lines

dmesg: http://pastebin.com/f3f83738a
messages: http://pastebin.com/f2091110a

(CCing to lklm again... I just want people to see the logs.)

Thank you for reporting and your patience. It seems something strange that your KDE programs are killed. I agree.

I attached a scirpt for checking oom_score of all exisiting process. (oom_score is a value used for selecting "bad" processs.") please run if you have time.

Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to

Quoted 3 lines

I attached a scirpt for checking oom_score of all exisiting process. (oom_score is a value used for selecting "bad" processs.") please run if you have time.

96890   21463   VirtualBox // OK
118615  11144   kded4 // WRONG
127455  11158   knotify4 // WRONG
132198  1       init // WRONG
133940  11151   ksmserver // WRONG
134109  11224   audacious2 // Audio player, maybe 145476  21503   VirtualBox // OK
174939  11322   icedove-bin // thunderbird, maybe 178015  11223   akregator // rss reader, maybe
201043  22672   krusader  // WRONG
212609  11187   krunner // WRONG
256911  24252   test // culprit, malloced 1GB
1750371 11318   run-mozilla.sh // tiny, parent of firefox threads 2044902 11141   kdeinit4 // tiny, parent of most KDE apps

Quoted 17 lines

Quoted 3 lines

I attached a scirpt for checking oom_score of all exisiting process. (oom_score is a value used for selecting "bad" processs.") please run if you have time.

96890   21463   VirtualBox // OK
118615  11144   kded4 // WRONG
127455  11158   knotify4 // WRONG
132198  1       init // WRONG
133940  11151   ksmserver // WRONG
134109  11224   audacious2 // Audio player, maybe
145476  21503   VirtualBox // OK
174939  11322   icedove-bin // thunderbird, maybe
178015  11223   akregator // rss reader, maybe
201043  22672   krusader  // WRONG
212609  11187   krunner // WRONG
256911  24252   test // culprit, malloced 1GB
1750371 11318   run-mozilla.sh // tiny, parent of firefox threads
2044902 11141   kdeinit4 // tiny, parent of most KDE apps

Verdran, I made alternative improvement idea. Can you please mesure badness score
on your system?
Maybe your culprit process take biggest badness value.

Note: this patch change time related thing. So, please drink a cup of coffee before mesurement.
small rest time makes correct test result.

Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?

When preparing KSM unmerge to handle OOM, I looked at how the precedent was handled by running a little program which mmaps an anonymous region of the same size as physical memory, then tries to mlock it. The program was such an obvious candidate to be killed, I was shocked by the poor decisions the OOM killer made. Usually I ran it with mem=512M, with gnome and firefox active. Often the OOM killer killed it right the first time, but went wrong when I tried it a second time (I think that's because of what's already swapped out the first time).

1. select_bad_process() tries to avoid killing another process while there's still a TIF_MEMDIE, but its loop starts by skipping !p->mm processes. However, p->mm is set to NULL well before p reaches exit_mmap() to actually free the memory, and there may be significant delays in between (I think exit_robust_list() gave me a hang at one stage). So in practice, even when the OOM killer selects the right process to kill, there can be lots of collateral damage from it not waiting long enough for that process to give up its memory.

I tried to deal with that by moving the TIF_MEMDIE test up before the p->mm test, but adding in a check on p->exit_state:

		if (test_tsk_thread_flag(p, TIF_MEMDIE) &&
		    !p->exit_state)
			return ERR_PTR(-1UL);

But this is then liable to hang the system if there's some reason why the selected process cannot proceed to free its memory (e.g. the current KSM unmerge case). It needs to wait "a while", but give up if no progress is made, instead of hanging: originally I thought that setting PF_MEMALLOC more widely in page_alloc.c, and giving up on the TIF_MEMDIE if it was waiting in PF_MEMALLOC, would deal with that; but we cannot be sure that waiting of memory is the only reason for a holdup there (in the KSM unmerge case it's waiting for an mmap_sem, and there may well be other such cases).

2. I started out running my mlock test program as root (later switched to use "ulimit -l unlimited" first). But badness() reckons CAP_SYS_ADMIN or CAP_SYS_RESOURCE is a reason to quarter your points; and CAP_SYS_RAWIO another reason to quarter your points: so running as root makes you sixteen times less likely to be killed. Quartering is anyway debatable, but sixteenthing seems utterly excessive to me.

I moved the CAP_SYS_RAWIO test in with the others, so it does no more than quartering; but is quartering appropriate anyway? I did wonder if I was right to be "subverting" the fine-grained CAPs in this way, but have since seen unrelated mail from one who knows better, implying they're something of a fantasy, that su and sudo are indeed what's used in the real world. Maybe this patch was okay.

4. In some cases those children are sharing exactly the same mm, yet its total_vm is being added again and again to the points:
I had a nasty inner loop searching back to see if we'd already counted this mm (but then, what if the different tasks sharing the mm deserved different adjustments to the total_vm?).

There's advantages to either approach, but it depends on the contextual goal of the oom killer when it's called: kill a rogue task that is allocating more memory than expected,

or kill a task that will free the most memory.

But it is wrong at counting allocated memory!
Come on, it kills /usr/lib/icedove/run-mozilla.sh. Parent, a shell script, instead of its child(s) which allocated memory. Look, "test" allocates some (0.1GB) memory, and you have:

% cat test.sh

#!/bin/sh
./test&
./test&
./test&
./test

% perl check_badness.pl|sort -n|g test

26511   7884    test
26511   7885    test
26511   7886    test
26511   7887    test
53994   7883    test.sh

// great, so test.sh "is" the bad ass, ok, emulate OOMK:

% kill -9 7883

// did we kill "a rogue task"

% perl check_badness.pl|sort -n|g test

26511   7884    test
26511   7885    test
26511   7886    test
26511   7887    test

// nooo, they are still alive and eating our memory!

This is wrong; it doesn't "emulate oom" since oom_kill_process() always kills a child of the selected process instead if they do not share the same memory. The chosen task in that case is untouched.

Quoted 3 lines

This is wrong; it doesn't "emulate oom" since oom_kill_process() always kills a child of the selected process instead if they do not share the same memory. The chosen task in that case is untouched.

OK, I stand corrected then. Thanks! But, while testing this I lost X once again and "test" survived for some time (check the timestamps):

http://pastebin.com/d5c9d026e

- It started by killing gkrellm(!!!)
- Then I lost X (kdeinit4 I guess)
- Then 103 seconds after the killing started, it killed "test" - the real culprit.

I mean... how?!

Quoted 7 lines · Vedran Furac

Quoted 3 lines

This is wrong; it doesn't "emulate oom" since oom_kill_process() always kills a child of the selected process instead if they do not share the same memory. The chosen task in that case is untouched.

OK, I stand corrected then. Thanks! But, while testing this I lost X once again and "test" survived for some time (check the timestamps):

Quoted 9 lines

http://pastebin.com/d5c9d026e

- It started by killing gkrellm(!!!)
- Then I lost X (kdeinit4 I guess)
- Then 103 seconds after the killing started, it killed "test" - the real culprit.

I mean... how?!

Here are the five oom kills that occurred in your log, and notice that the first four times it kills a child and not the actual task as I explained:

[97137.724971] Out of memory: kill process 21485 (VBoxSVC) score 1564940 or a child
[97137.725017] Killed process 21503 (VirtualBox)
[97137.864622] Out of memory: kill process 11141 (kdeinit4) score 1196178 or a child
[97137.864656] Killed process 11142 (klauncher)
[97137.888146] Out of memory: kill process 11141 (kdeinit4) score 1184308 or a child
[97137.888180] Killed process 11151 (ksmserver)
[97137.972875] Out of memory: kill process 11141 (kdeinit4) score 1146255 or a child
[97137.972888] Killed process 11224 (audacious2)

Those are practically happening simultaneously with very little memory being available between each oom kill. Only later is "test" killed:

[97240.203228] Out of memory: kill process 5005 (test) score 256912 or a child
[97240.206832] Killed process 5005 (test)

Notice how the badness score is less than 1/4th of the others. So while you may find it to be hogging a lot of memory, there were others that consumed much more.

not related to child-parent problem.

Seeing this number more.
==

[97137.709272] Active_anon:671487 active_file:82 inactive_anon:132316
[97137.709273]  inactive_file:82 unevictable:50 dirty:0 writeback:0 unstable:0
[97137.709273]  free:6122 slab:17179 mapped:30661 pagetables:8052 bounce:0

==

acitve_file + inactive_file is very low. Almost all pages are for anon. But "mapped(NR_FILE_MAPPED)" is a little high. This implies remaining file caches are mapped by many processes OR some mega bytes of shmem is used.

# of pagetables is 8052, this means
  8052x4096/8*4k bytes = 16Gbytes of mapped area.

Total available memory is near to be active/inactive + slab 671487+82+132316+82+50+6122+17179+8052=835370x4k= 3.2Gbytes ? (this system is swapless)

Then, considering the pmap kosaki shows,
I guess killed ones had big total_vm but has not much real rss, and no helps for oom.

Yep:

[97137.724965] 917504 pages RAM
[97137.724967] 69721 pages reserved

(917504 - 69721) * 4K = ~3.23G

Quoted 4 lines

Then, considering the pmap kosaki shows,
I guess killed ones had big total_vm but has not much real rss, and no helps for oom.

echo 1 > /proc/sys/vm/oom_dump_tasks can confirm that.

The bigger issue is making the distinction between killing a rogue task that is using much more memory than expected (the supposed current behavior, influenced from userspace by /proc/pid/oom_adj), and killing the task with the highest rss.

The latter is definitely desired if we are
allocating tons of memory but reduces the ability of the user to influence the badness score.

All kernel engineers know "than expected or not" can be never known to the kernel. So, oom_adj workaround is used now. (by some special users.)
OOM Killer itself is also a workaround, too.
"No kill" is the best thing but we know there are tend to be memory-leaker on bad systems and all systems in this world are not perfect.

Yes, some more trustable values other than vmsize/rss/time are appriciated. I wonder recent memory consumption speed can be an another key value.

Anyway, current bahavior of "killing X" is a bad thing. We need some fixes.

Quoted 6 lines · KAMEZAWA Hiroyuki

All kernel engineers know "than expected or not" can be never known to the kernel. So, oom_adj workaround is used now. (by some special users.)
OOM Killer itself is also a workaround, too.
"No kill" is the best thing but we know there are tend to be memory-leaker on bad systems and all systems in this world are not perfect.

Right, and historically that has been addressed by considering total_vm and adjusting it with oom_adj so that we can identify memory leaking tasks through user-defined criteria.

Quoted 3 lines

Yes, some more trustable values other than vmsize/rss/time are appriciated. I wonder recent memory consumption speed can be an another key value.

Sounds very logical.

Quoted 3 lines

Anyway, current bahavior of "killing X" is a bad thing. We need some fixes.

You can easily protect X with OOM_DISABLE, as you know. I don't think we need any X-specific heuristics added to the kernel, it looks like the special cases have already polluted badness() enough.

It's _not_ special to X.

Almost all applications which uses many dynamica libraries can be affected by this, total_vm. And, as I explained to Vedran, multi-threaded program like Java can easily increase total_vm without using many anon_rss.
And it's the reason I hate overcommit_memory. size of VM doesn't tell anything.

Right, because in Vedran's latest oom log it shows that Xorg is preferred more than any other thread other than the memory hogging test program with your patch than without. I pointed out a clear distinction in the killing order using both total_vm and rss in that log and in my opinion killing Xorg as opposed to krunner would be undesireable.

But then you should rename OOM killer to TRIPK:
Totally Random Innocent Process Killer

If you have OOM situation and Xorg is the first, that means it's leaking memory badly and the system is probably already frozen/FUBAR. Killing krunner in that situation wouldn't do any good. From a user perspective, nothing changes, system is still FUBAR and (s)he would probably reboot cursing linux in the process.

Quoted 6 lines

If you have OOM situation and Xorg is the first, that means it's leaking memory badly and the system is probably already frozen/FUBAR. Killing krunner in that situation wouldn't do any good. From a user perspective, nothing changes, system is still FUBAR and (s)he would probably reboot cursing linux in the process.

It depends on what you're running, we need to be able to have the option of protecting very large tasks on production servers. Imagine if "test" here is actually a critical application that we need to protect, its not solely mlocked anonymous memory, but still kill if it is leaking memory beyond your approximate 2.5GB. How do you do that when using rss as the baseline?

As I wrote repeatedly,

- OOM-Killer itselfs is bad thing, bad situation.

- The kernel can't know the program is bad or not. just guess it.

- Then, there is no "correct" OOM-Killer other than fork-bomb killer.

- User has a knob as oom_adj. This is very strong.

Then, there is only "reasonable" or "easy-to-understand" OOM-Kill. "Current biggest memory eater is killed" sounds reasonable, easy to understand. And if total_vm works well, overcommit_guess should catch it. Please improve overcommit_guess if you want to stay on total_vm.

Quoted 1 line

- The kernel can't know the program is bad or not. just guess it.

Totally irrelevant, given your fourth point about /proc/pid/oom_adj. We can tell the kernel what we'd like the oom killer behavior should be if the situation arises.

Quoted 1 line

- Then, there is no "correct" OOM-Killer other than fork-bomb killer.

Well of course there is, you're seeing this is a WAY too simplistic manner. If we are oom, we want to be able to influence how the oom killer behaves and respond to that situation. You are proposing that we change the baseline for how the oom killer selects tasks which we use CONSTANTLY as part of our normal production environment. I'd appreciate it if you'd take it a little more seriously.

Quoted 2 lines

- User has a knob as oom_adj. This is very strong.

Agreed.

Quoted 5 lines

Then, there is only "reasonable" or "easy-to-understand" OOM-Kill. "Current biggest memory eater is killed" sounds reasonable, easy to understand. And if total_vm works well, overcommit_guess should catch it. Please improve overcommit_guess if you want to stay on total_vm.

I don't necessarily want to stay on total_vm, but I also don't want to move to rss as a baseline, as you would probably agree.

We disagree about a very fundamental principle: you are coming from a perspective of always wanting to kill the biggest resident memory eater even for a single order-0 allocation that fails and I'm coming from a perspective of wanting to ensure that our machines know how the oom killer will react when it is used.

Moving to rss reduces the ability of the user to specify an expected oom priority other than polarizing it by either
disabling it completely with an oom_adj value of -17 or choosing the definite next victim with +15. That's my objection to it: the user cannot possibly be expected to predict what proportion of each application's memory will be resident at the time of oom.

I understand you want to totally rewrite the oom killer for whatever reason, but I think you need to spend a lot more time understanding the needs that the Linux community has for its behavior instead of insisting on your point of view.

On Fri, 30 Oct 2009 02:10:37 -0700 (PDT)

Quoted 5 lines · David Rientjes

Quoted 1 line

- The kernel can't know the program is bad or not. just guess it.

Totally irrelevant, given your fourth point about /proc/pid/oom_adj. We can tell the kernel what we'd like the oom killer behavior should be if the situation arises.

My point is that the server cannot distinguish memory leak from
intentional memory usage. No other than that.

Quoted 9 lines

Quoted 2 lines

- Then, there is no "correct" OOM-Killer other than fork-bomb
killer.

Well of course there is, you're seeing this is a WAY too simplistic manner. If we are oom, we want to be able to influence how the oom killer behaves and respond to that situation. You are proposing that we change the baseline for how the oom killer selects tasks which we use CONSTANTLY as part of our normal production environment. I'd appreciate it if you'd take it a little more seriously.

Yes, I'm serious.

In this summer, at lunch with a daily linux user, I was said "you, enterprise guys, don't consider desktop or laptop problem at all." yes, I use only servers. My customer uses server, too. My first priority is always on server users.
But, for this time, I wrote reply to Vedran and try to fix desktop
problem. Even if current logic works well for servers, "KDE/GNOME is killed" problem seems to be serious. And this may be a problem for EMBEDED people, I guess.

Quoted 6 lines

Quoted 1 line

- The kernel can't know the program is bad or not. just guess it.

Totally irrelevant, given your fourth point about /proc/pid/oom_adj. We can tell the kernel what we'd like the oom killer behavior should be if the situation arises.

My point is that the server cannot distinguish memory leak from intentional memory usage. No other than that.

In this summer, at lunch with a daily linux user, I was said "you, enterprise guys, don't consider desktop or laptop problem at all." yes, I use only servers. My customer uses server, too. My first priority is always on server users.
But, for this time, I wrote reply to Vedran and try to fix desktop problem. Even if current logic works well for servers, "KDE/GNOME is killed" problem seems to be serious. And this may be a problem for EMBEDED people, I guess.

I can say the same thing to total_vm size. total_vm size doesn't include any good information for oom situation. And tweaking based on that not-useful parameter will make things worse.

Quoted 10 lines · KAMEZAWA Hiroyuki

Quoted 6 lines

Quoted 1 line

- The kernel can't know the program is bad or not. just guess it.

Totally irrelevant, given your fourth point about /proc/pid/oom_adj. We can tell the kernel what we'd like the oom killer behavior should be if the situation arises.

My point is that the server cannot distinguish memory leak from intentional memory usage. No other than that.

That's a different point. Today, we can influence the badness score of any user thread to prioritize oom killing from userspace and that can be done regardless of whether there's a memory leaker, a fork bomber, etc. The priority based oom killing is important to production scenarios and cannot be replaced by a heuristic that works everytime if it cannot be influenced by userspace.

A spike in memory consumption when a process is initially forked would be defined as a memory leaker in your quiet_time model.

Quoted 8 lines

In this summer, at lunch with a daily linux user, I was said "you, enterprise guys, don't consider desktop or laptop problem at all." yes, I use only servers. My customer uses server, too. My first priority is always on server users.
But, for this time, I wrote reply to Vedran and try to fix desktop problem. Even if current logic works well for servers, "KDE/GNOME is killed" problem seems to be serious. And this may be a problem for EMBEDED people, I guess.

You argued before that the problem wasn't specific to X (after I said you could protect it very trivially with /proc/pid/oom_adj set to OOM_DISABLE), but that's now your reasoning for rewriting the oom killer heuristics?

Quoted 4 lines

I can say the same thing to total_vm size. total_vm size doesn't include any good information for oom situation. And tweaking based on that not-useful parameter will make things worse.

Tweaking on the heuristic will probably make it more convoluted and overall worse, I agree. But it's a more stable baseline than rss from which we can set oom killing priorities from userspace.

Quoted 7 lines

That's a different point. Today, we can influence the badness score of any user thread to prioritize oom killing from userspace and that can be done regardless of whether there's a memory leaker, a fork bomber, etc. The priority based oom killing is important to production scenarios and cannot be replaced by a heuristic that works everytime if it cannot be influenced by userspace.

I don't removed oom_adj...

Quoted 3 lines

Tweaking on the heuristic will probably make it more convoluted and overall worse, I agree. But it's a more stable baseline than rss from which we can set oom killing priorities from userspace.

- "rss < total_vm_size" always.

- oom_adj culculation is quite strong.
- total_vm of processes which maps hugetlb is very big ....but killing them is no help for usual oom.

I recommend you to add "stable baseline" knob for user space, as I wrote. My patch 6 adds stable baseline bonus as 50% of vm size if run_time is enough large.

Quoted 9 lines · KAMEZAWA Hiroyuki

Quoted 7 lines

That's a different point. Today, we can influence the badness score of any user thread to prioritize oom killing from userspace and that can be done regardless of whether there's a memory leaker, a fork bomber, etc. The priority based oom killing is important to production scenarios and cannot be replaced by a heuristic that works everytime if it cannot be influenced by userspace.

I don't removed oom_adj...

Right, but we must ensure that we have the same ability to influence a priority based oom killing scheme from userspace as we currently do with a relatively static total_vm. total_vm may not be the optimal baseline, but it does allow users to tune oom_adj specifically to identify tasks that are using more memory than expected and to be static enough to not depend on rss, for example, that is really hard to predict at the time of oom.

That's actually my main goal in this discussion: to avoid losing any ability of userspace to influence to priority of tasks being oom killed (if you haven't noticed :).

Quoted 5 lines

Quoted 3 lines

Tweaking on the heuristic will probably make it more convoluted and overall worse, I agree. But it's a more stable baseline than rss from which we can set oom killing priorities from userspace.

- "rss < total_vm_size" always.

But rss is much more dynamic than total_vm, that's my point.

Quoted 8 lines

- oom_adj culculation is quite strong.
- total_vm of processes which maps hugetlb is very big ....but killing them is no help for usual oom.

I recommend you to add "stable baseline" knob for user space, as I wrote. My patch 6 adds stable baseline bonus as 50% of vm size if run_time is enough large.

There's no clear relationship between VM size and runtime. The forkbomb heuristic itself could easily return a badness of ULONG_MAX if one is detected using runtime and number of children, as I earlier proposed, but that doesn't seem helpful to factor into the scoring.

My point and your point are differnt.

1. All my concern is "baseline for heuristics"
2. All your concern is "baseline for knob, as oom_adj"

ok ? For selecting victim by the kernel, dynamic value is much more useful. Current behavior of "Random kill" and "Kill multiple processes" are too bad. Considering oom-killer is for what, I think "1" is more important.

But I know what you want, so, I offers new knob which is not affected by RSS as I wrote in previous mail.

Off-topic:
As memcg is growing better, using OOM-Killer for resource control should be ended, I think. Maybe Fake-NUMA+cpuset is working well for google system, but plz consider to use memcg.

Old processes are important, younger are not. But as I wrote, I'll drop most of patch "6". So, plz forget about this part.

I'm interested in fork-bomb killer rather than crazy badness calculation, now.

Quoted 9 lines · KAMEZAWA Hiroyuki

My point and your point are differnt.

1. All my concern is "baseline for heuristics"
2. All your concern is "baseline for knob, as oom_adj"

ok ? For selecting victim by the kernel, dynamic value is much more useful. Current behavior of "Random kill" and "Kill multiple processes" are too bad. Considering oom-killer is for what, I think "1" is more important.

Quoted 8 lines

But I know what you want, so, I offers new knob which is not affected by RSS as I wrote in previous mail.

Off-topic:
As memcg is growing better, using OOM-Killer for resource control should be ended, I think. Maybe Fake-NUMA+cpuset is working well for google system, but plz consider to use memcg.

I understand what you're trying to do, and I agree with it for most desktop systems. However, I think that admins should have a very strong influence in what tasks the oom killer kills. It doesn't really matter if it's via oom_adj or not, and its debatable whether an adjustment on a static heuristic score is in our best interest in the first place. But we must have an alternative so that our control over oom killing isn't lost.

I'd also like to open another topic for discussion if you're proposing such sweeping changes: at what point do we allow ~__GFP_NOFAIL allocations to fail even if order < PAGE_ALLOC_COSTLY_ORDER and defer killing anything? We both agreed that it's not always in the best interest to kill a task so that an allocation can succeed, so we need to define some criteria to simply fail the allocation instead.

Quoted 3 lines · Vedran Furac

But then you should rename OOM killer to TRIPK:
Totally Random Innocent Process Killer

The randomness here is the order of the child list when the oom killer selects a task, based on the badness score, and then tries to kill a child with a different mm before the parent.

The problem you identified in http://pastebin.com/f3f9674a0, however, is a forkbomb issue where the badness score should never have been so high for kdeinit4 compared to "test". That's directly proportional to adding the scores of all disjoint child total_vm values into the badness score for the parent and then killing the children instead.

That's the problem, not using total_vm as a baseline. Replacing that with rss is not going to solve the issue and reducing the user's ability to specify a rough oom priority from userspace is simply not an option.

Quoted 5 lines

The problem you identified in http://pastebin.com/f3f9674a0, however, is a forkbomb issue where the badness score should never have been so high for kdeinit4 compared to "test". That's directly proportional to adding the scores of all disjoint child total_vm values into the badness score for the parent and then killing the children instead.

Could you explain me why ntpd invoked oom killer? Its parent is init. Or syslog-ng?

Quoted 3 lines

That's the problem, not using total_vm as a baseline. Replacing that with rss is not going to solve the issue and reducing the user's ability to specify a rough oom priority from userspace is simply not an option.

OK then, if you have a solution, I would be glad to test your patch. I won't care much if you don't change total_vm as a baseline. Just make random killing history.

Quoted 8 lines · Vedran Furac

Quoted 5 lines

The problem you identified in http://pastebin.com/f3f9674a0, however, is a forkbomb issue where the badness score should never have been so high for kdeinit4 compared to "test". That's directly proportional to adding the scores of all disjoint child total_vm values into the badness score for the parent and then killing the children instead.

Could you explain me why ntpd invoked oom killer? Its parent is init. Or syslog-ng?

Because it attempted an order-0 GFP_USER allocation and direct reclaim could not free any pages.

The task that invoked the oom killer is simply the unlucky task that tried an allocation that couldn't be satisified through direct reclaim. It's usually unrelated to the task chosen for kill unless /proc/sys/vm/oom_kill_allocating_task is enabled (which SGI requested to avoid excessively long tasklist scans).

Quoted 3 lines

OK then, if you have a solution, I would be glad to test your patch. I won't care much if you don't change total_vm as a baseline. Just make random killing history.

The only randomness is in selecting a task that has a different mm from the parent in the order of its child list. Yes, that can be addressed by doing a smarter iteration through the children before killing one of them.

Keep in mind that a heuristic as simple as this:

- kill the task that was started most recently by the same uid, or

- kill the task that was started most recently on the system if a root task calls the oom killer,

would have yielded perfect results for your testcase but isn't necessarily something that we'd ever want to see.

Quoted 13 lines · Vedran Furac

Quoted 3 lines

This is wrong; it doesn't "emulate oom" since oom_kill_process() always kills a child of the selected process instead if they do not share the same memory. The chosen task in that case is untouched.

OK, I stand corrected then. Thanks! But, while testing this I lost X once again and "test" survived for some time (check the timestamps):

http://pastebin.com/d5c9d026e

- It started by killing gkrellm(!!!)
- Then I lost X (kdeinit4 I guess)
- Then 103 seconds after the killing started, it killed "test" - the real culprit.

Quoted 2 lines

I mean... how?!

Here are the five oom kills that occurred in your log, and notice that the first four times it kills a child and not the actual task as I explained:

Those are practically happening simultaneously with very little memory being available between each oom kill. Only later is "test" killed:

[97240.203228] Out of memory: kill process 5005 (test) score 256912 or a child
[97240.206832] Killed process 5005 (test)

Notice how the badness score is less than 1/4th of the others. So while you may find it to be hogging a lot of memory, there were others that consumed much more.

You can get a more detailed understanding of this by doing

echo 1 > /proc/sys/vm/oom_dump_tasks

before trying your testcase; it will show various information like the total_vm

and oom_adj value for each task at the time of oom (and the actual badness score is exported per-task via /proc/pid/oom_score in real-time). This will also include the rss and show what the end result would be in using that value as part of the heuristic on this particular workload compared to the current implementation.

Quoted 9 lines

Those are practically happening simultaneously with very little memory being available between each oom kill. Only later is "test" killed:

[97240.203228] Out of memory: kill process 5005 (test) score 256912 or a child
[97240.206832] Killed process 5005 (test)

Notice how the badness score is less than 1/4th of the others. So while you may find it to be hogging a lot of memory, there were others that consumed much more.

^^^^^^^^^^^^^^^^^^^^^

This is just wrong. I have 3.5GB of RAM, free says that 2GB are empty (ignoring cache). Culprit then allocates all free memory (2GB). That means it is using *more* than all other processes *together*. There cannot be any other "that consumed much more".

Thanks, I'll try that... but I guess that using rss would yield better results.

We would know if you posted the data.

Quoted 1 line

We would know if you posted the data.

I need to find some free time to destroy a session on a computer which I use for work. You could easily test it yourself also as this doesn't happen only to me.

Anyways, here it is... this time it started with ntpd:

http://pastebin.com/f3f9674a0

From your log at the link above:

Quoted 5 lines · Vedran Furac

Quoted 1 line

We would know if you posted the data.

I need to find some free time to destroy a session on a computer which I use for work. You could easily test it yourself also as this doesn't happen only to me.

Quoted 4 lines

Anyways, here it is... this time it started with ntpd:

http://pastebin.com/f3f9674a0

That oom log shows 12 ooms but no tasks actually appear to be getting killed (there're no "Killed process 1234 (task)" found). Do you have any idea why?

Can you explain why Xorg is preferred as a baseline to kill rather than krunner in your example?

[ 1493.064458] Out of memory: kill process 6304 (kdeinit4) score 1190231
or a child
[ 1493.064467] Killed process 6409 (konqueror)
[ 1493.261149] knotify4 invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1493.261166]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1493.276528] Out of memory: kill process 6304 (kdeinit4) score 1161265
or a child
[ 1493.276538] Killed process 6411 (krusader)
[ 1499.221160] akregator invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1499.221178]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1499.236431] Out of memory: kill process 6304 (kdeinit4) score 1067593
or a child
[ 1499.236441] Killed process 6412 (irexec)
[ 1499.370192] firefox-bin invoked oom-killer: gfp_mask=0x201da,
order=0, oomkilladj=0
[ 1499.370209]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1499.385417] Out of memory: kill process 6304 (kdeinit4) score 1066861
or a child
[ 1499.385427] Killed process 6420 (xchm)
[ 1499.458304] kio_file invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1499.458333]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1499.458367]  [<ffffffff81120900>] ? d_kill+0x5c/0x7c
[ 1499.473573] Out of memory: kill process 6304 (kdeinit4) score 1043690
or a child
[ 1499.473582] Killed process 6425 (kio_file)
[ 1500.250746] korgac invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1500.250765]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1500.266186] Out of memory: kill process 6304 (kdeinit4) score 1020350
or a child
[ 1500.266196] Killed process 6464 (icedove)
[ 1500.349355] syslog-ng invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1500.349371]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1500.364689] Out of memory: kill process 6304 (kdeinit4) score 1019864
or a child
[ 1500.364699] Killed process 6477 (kio_http)
[ 1500.452151] kded4 invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1500.452167]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1500.452196]  [<ffffffff81120900>] ? d_kill+0x5c/0x7c
[ 1500.467307] Out of memory: kill process 6304 (kdeinit4) score 993142
or a child
[ 1500.467316] Killed process 6478 (kio_http)
[ 1500.780222] akregator invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1500.780239]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1500.796280] Out of memory: kill process 6304 (kdeinit4) score 966331
or a child
[ 1500.796290] Killed process 6484 (kio_http)
[ 1501.065374] syslog-ng invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1501.065390]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1501.080579] Out of memory: kill process 6304 (kdeinit4) score 939434
or a child
[ 1501.080587] Killed process 6486 (kio_http)
[ 1501.381188] knotify4 invoked oom-killer: gfp_mask=0x201da, order=0,
oomkilladj=0
[ 1501.381204]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1501.396338] Out of memory: kill process 6304 (kdeinit4) score 912691

or a child

[ 1501.396346] Killed process 6487 (firefox-bin)
[ 1502.661294] icedove-bin invoked oom-killer: gfp_mask=0x201da,
order=0, oomkilladj=0
[ 1502.661311]  [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
[ 1502.676563] Out of memory: kill process 7580 (test) score 708945 or a
child
[ 1502.676575] Killed process 7580 (test)

Quoted 2 lines

Can you explain why Xorg is preferred as a baseline to kill rather than krunner in your example?

Krunner is a small app for running other apps and do similar things. It shouldn't use a lot of memory. OTOH, Xorg has to hold all the pixmaps and so on. That was expected result. Fist Xorg, then firefox and thunderbird.

Ok, so this is the forkbomb problem by adding half of each child's total_vm into the badness score of the parent. We should address this completely seperately by addressing that specific part of the heuristic, not changing what we consider to be a baseline.
thunderbird.

You're making all these claims and assertions based _solely_ on the theory that killing the application with the most resident RAM is always the optimal solution. That's just not true, especially if we're just allocating small numbers of order-0 memory.

Much better is to allow the user to decide at what point, regardless of swap usage, their application is using much more memory than expected or required. They can do that right now pretty well with /proc/pid/oom_adj without this outlandish claim that they should be expected to know the rss of their applications at the time of oom to effectively tune oom_adj.

What would you suggest? A script that sits in a loop checking each task's current rss from /proc/pid/stat or their current oom priority though /proc/pid/oom_score and adjusting oom_adj preemptively just in case the oom killer is invoked in the next second?

Quoted 10 lines · David Rientjes

Ok, so this is the forkbomb problem by adding half of each child's total_vm into the badness score of the parent. We should address this completely seperately by addressing that specific part of the heuristic, not changing what we consider to be a baseline. thunderbird.

You're making all these claims and assertions based _solely_ on the theory that killing the application with the most resident RAM is always the optimal solution. That's just not true, especially if we're just allocating small numbers of order-0 memory.

Well, you are kernel hacker, not me. You know how linux mm works much more than I do. I just reported a, what I think is a big problem, which needs to be solved ASAP (2.6.33). I'm afraid that we'll just talk much and nothing will be done with solution/fix postponed indefinitely. Not sure if you are interested, but I tested this on windowsxp also, and nothing bad happens there, system continues to function properly.

For 2-3 years I had memory overcommit turn off. I didn't get any OOM, but sometimes Java didn't work and it seems that because of some kernel weirdness (or misunderstanding on my part) I couldn't use all the available memory:

# echo 2 > /proc/sys/vm/overcommit_memory

# echo 95 > /proc/sys/vm/overcommit_ratio
% ./test  /* malloc in loop as before */
malloc: Cannot allocate memory /* Great, no OOM, but: */

% free -m

          total       used       free     shared    buffers     cached
Mem:      3458        3429         29          0        102       1119
-/+ buffers/cache:    2207       1251

There's plenty of memory available. Shouldn't cache be automatically dropped (this question was in my original mail, hence the subject)?

All this frustrated not only me, but a great number of users on our local Croatian linux usenet newsgroup with some of them pointing that as the reason they use solaris. And so on...

Quoted 6 lines

Much better is to allow the user to decide at what point, regardless of swap usage, their application is using much more memory than expected or required. They can do that right now pretty well with /proc/pid/oom_adj without this outlandish claim that they should be expected to know the rss of their applications at the time of oom to effectively tune oom_adj.

Believe me, barely a few developers use oom_adj for their applications, and probably almost none of the end users. What should they do, every time they start an application, go to console and set the oom_adj. You cannot expect them to do that.

Quoted 5 lines

What would you suggest? A script that sits in a loop checking each task's current rss from /proc/pid/stat or their current oom priority though /proc/pid/oom_score and adjusting oom_adj preemptively just in case the oom killer is invoked in the next second?

:)

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 6 lines

malloc: Cannot allocate memory /* Great, no OOM, but: */

% free -m total       used       free     shared    buffers cached
Mem:      3458        3429         29          0        102 1119
-/+ buffers/cache:    2207       1251

Quoted 4 lines

There's plenty of memory available. Shouldn't cache be automatically dropped (this question was in my original mail, hence
the subject)?

I think this is the MOST serious issue related to the oom killer. For some reason it refuses to drop pages before trying to kill. When it should drop cache, THEN kill if needed.

% free -m

          total       used       free     shared    buffers     cached
Mem:      3458        3429         29          0        102       1119
-/+ buffers/cache:    2207       1251

There's plenty of memory available. Shouldn't cache be automatically dropped (this question was in my original mail, hence the subject)?

Quoted 5 lines · Vedran Furač

% free -m

          total       used       free     shared    buffers     cached
Mem:      3458        3429         29          0        102       1119
-/+ buffers/cache:    2207       1251

Quoted 2 lines

There's plenty of memory available. Shouldn't cache be automatically dropped (this question was in my original mail, hence the subject)?

This is not about cache, cache amount is physical, this about virtual amount that can only go in ram or swap (at any later time, current time is irrelevant) vs "ram + swap".

In short add more swap if
you don't like overcommit and check grep Commit /proc/meminfo in case this is accounting bug...

Oh... so this is because apps "reserve" (Committed_AS?) more then they currently need.

A the time of "malloc: Cannot allocate memory":

CommitLimit:     3364440 kB
Committed_AS:    3240200 kB

So probably everything is ok (and free is misleading). Overcommit is unfortunately necessary if I want to be able to use all my memory.

Btw. http://www.redhat.com/advice/tips/meminfo.html says Committed_AS is a (gu)estimate. Hope it is a good (not to high) guesstimate. :)

It is a guess in the sense to guarantee no ENOMEM it has to take into account the worst possible case, that is all shared lib MAP_PRIVATE mappings are cowed, which is very far from reality.

Quoted 2 lines · Vedran Furač

Oh... so this is because apps "reserve" (Committed_AS?) more then they currently need.

They don't actually reserve, they end up "reserving" if overcommit is set to 2 (OVERCOMMIT_NEVER)... Apps aren't reserving, more likely they simply avoid a flood of mmap when a single one is enough to map an huge MAP_PRIVATE region like shared libs that you may only execute partially (this is why total_vm is usually much bigger than real ram mapped by pagetables represented in rss). But those shared libs are 99% pageable and they don't need to stay in swap or ram, so overcommit-as greatly overstimates the actual needs even if shared lib loading wouldn't be 64bit optimized (i.e. large and a single one).

Quoted 7 lines

A the time of "malloc: Cannot allocate memory":

CommitLimit:     3364440 kB
Committed_AS:    3240200 kB

So probably everything is ok (and free is misleading). Overcommit is unfortunately necessary if I want to be able to use all my memory.

Add more swap.

Well, you are kernel hacker, not me. You know how linux mm works much more than I do. I just reported a, what I think is a big problem, which needs to be solved ASAP (2.6.33).

I'm afraid that we'll just talk much
and nothing will be done with solution/fix postponed indefinitely. Not sure if you are interested, but I tested this on windowsxp also, and nothing bad happens there, system continues to function properly.

Quoted 5 lines

Much better is to allow the user to decide at what point, regardless of swap usage, their application is using much more memory than expected or required. They can do that right now pretty well with /proc/pid/oom_adj without this outlandish claim that they should be expected to know the rss of their applications at the time of oom to effectively tune oom_adj.

Believe me, barely a few developers use oom_adj for their applications, and probably almost none of the end users. What should they do, every time they start an application, go to console and set the oom_adj. You cannot expect them to do that.

Quoted 3 lines · Vedran Furac

Well, you are kernel hacker, not me. You know how linux mm works much more than I do. I just reported a, what I think is a big problem, which needs to be solved ASAP (2.6.33).

The oom killer heuristics have not been changed recently, why is this suddenly a problem that needs to be immediately addressed? The heuristics you've been referring to have been used for at least three years.

However, I don't think we can simply change the baseline (like the rss change which has been added to -mm (??)) and consider it a major improvement when it severely impacts how system administrators are able to tune the badness heuristic from userspace via /proc/pid/oom_adj. I'm sure you'd agree that user input is important in this matter and so that we should maximize that ability rather than make it more difficult. That's my main criticism of the suggestions thus far (and, sorry, but I have to look out for production server interests here: you can't take away our ability to influence oom badness scoring just because other simple heuristics may be more understandable).

What would be better, and what I think we'll end up with, is a root selectable heuristic so that production servers and desktop machines can use different heuristics to make oom kill selections. We already have /proc/sys/vm/oom_kill_allocating_task which I added 1-2 years ago to address concerns specifically of SGI and their enormously long tasklist scans. This would be variation on that idea and would include different simplistic behaviors (such as always killing the most memory hogging task, killing the most recently started task by the same uid, etc), and leave the default heuristic much the same as currently.

Quoted 4 lines · KAMEZAWA Hiroyuki

Sigh, gnome-session has twice value of mmap(1G).
Of course, gnome-session only uses 6M bytes of anon.
I wonder this is because gnome-session has many children..but need to dig more. Does anyone has idea ?

When preparing KSM unmerge to handle OOM, I looked at how the precedent was handled by running a little program which mmaps an anonymous region of the same size as physical memory, then tries to mlock it. The program was such an obvious candidate to be killed, I was shocked by the poor decisions the OOM killer made. Usually I ran it with mem=512M, with gnome and firefox active. Often the OOM killer killed it right the first time, but went wrong when I tried it a second time (I think that's because of what's already swapped out the first time).

I built up a patchset of fixes, but once I came to split them up for submission, not one of them seemed entirely satisfactory; and Andrea's fix to the KSM/mlock deadlock forced me to abandon even the first of the patches (we've since then fixed the way munlocking behaves, so in theory could revisit that; but Andrea disliked what I was trying to do there in KSM for other reasons, so I've not touched it since). I had to get on with KSM, so I set it all aside: none of the issues was a recent regression.

I did briefly wonder about the reliance on total_vm which you're now looking into, but didn't touch that at all. Let me describe those issues which I did try but fail to fix - I've no more time to deal with them now than then, but ought at least to mention them to you.

1. select_bad_process() tries to avoid killing another process while there's still a TIF_MEMDIE, but its loop starts by skipping !p->mm processes. However, p->mm is set to NULL well before p reaches exit_mmap() to actually free the memory, and there may be significant delays in between (I think exit_robust_list() gave me a hang at one stage). So in practice, even when the OOM killer selects the right process to kill, there can be lots of collateral damage from it not waiting long enough for that process to give up its memory.

I tried to deal with that by moving the TIF_MEMDIE test up before the p->mm test, but adding in a check on p->exit_state:

		if (test_tsk_thread_flag(p, TIF_MEMDIE) &&
		    !p->exit_state)
			return ERR_PTR(-1UL);

But this is then liable to hang the system if there's some reason why the selected process cannot proceed to free its memory (e.g. the current KSM unmerge case). It needs to wait "a while", but give up if no progress is made, instead of hanging: originally I thought that setting PF_MEMALLOC more widely in page_alloc.c, and giving up on the TIF_MEMDIE if it was waiting in PF_MEMALLOC, would deal with that; but we cannot be sure that waiting of memory is the only reason for a holdup there (in the KSM unmerge case it's waiting for an mmap_sem, and there may well be other such cases).

2. I started out running my mlock test program as root (later switched to use "ulimit -l unlimited" first). But badness() reckons CAP_SYS_ADMIN or CAP_SYS_RESOURCE is a reason to quarter your points; and CAP_SYS_RAWIO another reason to quarter your points: so running as root makes you sixteen times less likely to be killed. Quartering is anyway debatable, but sixteenthing seems utterly excessive to me.

I moved the CAP_SYS_RAWIO test in with the others, so it does no more than quartering; but is quartering appropriate anyway? I did wonder if I was right to be "subverting" the fine-grained CAPs in this way, but have since seen unrelated mail from one who knows better, implying they're something of a fantasy, that su and sudo are indeed what's used in the real world. Maybe this patch was okay.

3. badness() has a comment above it which says:
* 5) we try to kill the process the user expects us to kill, this
* algorithm has been meticulously tuned to meet the principle
* of least surprise ... (be careful when you change it)
But Andrea's 2.6.11 86a4c6d9e2e43796bb362debd3f73c0e3b198efa (later refined by Kurt's 2.6.16 9827b781f20828e5ceb911b879f268f78fe90815) adds plenty of surprise there, by trying to factor children into the calculation. Intended to deal with forkbombs, but any reasonable process whose purpose is to fork children (e.g. gnome-session) becomes very vulnerable. And whereas badness() itself goes on to refine the total_vm points by various adjustments peculiar to the process in question, those refinements have been ignored when adding the child's total_vm/2. (Andrea does remark that he'd rather have rewritten badness() from scratch.)

I tried to fix this by moving the PF_OOM_ORIGIN (was PF_SWAPOFF) part of the calculation up to select_bad_process(), making a solo_badness() function which makes all those adjustments to total_vm, then badness() itself a simple function adding half the children's solo_badness()es to the process' own solo_badness(). But probably lots more needs doing - Andrea's rewrite?

4. In some cases those children are sharing exactly the same mm, yet its total_vm is being added again and again to the points:
I had a nasty inner loop searching back to see if we'd already counted this mm (but then, what if the different tasks sharing the mm deserved different adjustments to the total_vm?).

I hope these notes help someone towards a better solution (and be prepared to discover more on the way). I agree with Vedran that the present behaviour is pretty unimpressive, and I'm puzzled as to how people can have been tinkering with oom_kill.c down the years without seeing any of this.

2. I started out running my mlock test program as root (later switched to use "ulimit -l unlimited" first). But badness() reckons CAP_SYS_ADMIN or CAP_SYS_RESOURCE is a reason to quarter your points; and CAP_SYS_RAWIO another reason to quarter your points: so running as root makes you sixteen times less likely to be killed. Quartering is anyway debatable, but sixteenthing seems utterly excessive to me.

I moved the CAP_SYS_RAWIO test in with the others, so it does no more than quartering; but is quartering appropriate anyway? I did wonder if I was right to be "subverting" the fine-grained CAPs in this way, but have since seen unrelated mail from one who knows better, implying they're something of a fantasy, that su and sudo are indeed what's used in the real world. Maybe this patch was okay.

From 27331555366c908a93c2cdd780b77e421869c5af Mon Sep 17 00:00:00 2001

Quoted 7 lines

2. I started out running my mlock test program as root (later switched to use "ulimit -l unlimited" first). But badness() reckons CAP_SYS_ADMIN or CAP_SYS_RESOURCE is a reason to quarter your points; and CAP_SYS_RAWIO another reason to quarter your points: so running as root makes you sixteen times less likely to be killed. Quartering is anyway debatable, but sixteenthing seems utterly excessive to me.

Quoted 6 lines

I moved the CAP_SYS_RAWIO test in with the others, so it does no more than quartering; but is quartering appropriate anyway? I did wonder if I was right to be "subverting" the fine-grained CAPs in this way, but have since seen unrelated mail from one who knows better, implying they're something of a fantasy, that su and sudo are indeed what's used in the real world. Maybe this patch was okay.

I agree quartering is debatable.
At least, killing quartering is worth for any user, and it can be push into -stable.

From 27331555366c908a93c2cdd780b77e421869c5af Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Wed, 28 Oct 2009 11:28:39 +0900
Subject: [PATCH] oom: Mitigate suer-user's bonus of oom-score

Currently, badness calculation code of oom contemplate following bonus.
- Super-user have quartering oom-score
- CAP_SYS_RAWIO process (e.g. database) also have quartering oom-score

The problem is, Super-users have CAP_SYS_RAWIO too. Then, they have sixteenthing bonus. it's obviously too excessive and meaningless.

This patch fixes it.

Signed-off-by KOSAKI Motohiro <[email protected]>

---
 mm/oom_kill.c |   13 +++++--------
 1 files changed, 5 insertions(+), 8 deletions(-)

mm/oom_kill.c +5 -8

index ea2147d..40d323d 100644
@@ -152,18 +152,15 @@ unsigned long badness(struct task_struct *p, unsigned long uptime)
 	/*
 	 * Superuser processes are usually more important, so we make it
 	 * less likely that we kill those.
-	 */
-	if (has_capability_noaudit(p, CAP_SYS_ADMIN) ||
-	    has_capability_noaudit(p, CAP_SYS_RESOURCE))
-		points /= 4;
-
-	/*
-	 * We don't want to kill a process with direct hardware access.
+	 *
+	 * Plus, We don't want to kill a process with direct hardware access.
 	 * Not only could that mess up the hardware, but usually users
 	 * tend to only have this flag set on applications they think
 	 * of as important.
 	 */
-	if (has_capability_noaudit(p, CAP_SYS_RAWIO))
+	if (has_capability_noaudit(p, CAP_SYS_ADMIN) ||
+	    has_capability_noaudit(p, CAP_SYS_RESOURCE) ||
+	    has_capability_noaudit(p, CAP_SYS_RAWIO))
 		points /= 4;
 
 	/*

Signature · 1 line

1.6.2.5

I agree quartering is debatable.
At least, killing quartering is worth for any user, and it can be push into -stable.

mm/oom_kill.c +2 -7

index ea2147d..40d323d 100644
@@ -152,18 +152,15 @@ unsigned long badness(struct task_struct *p, unsigned long uptime)
 	/*
 	 * Superuser processes are usually more important, so we make it
 	 * less likely that we kill those.
-	 */
-	if (has_capability_noaudit(p, CAP_SYS_ADMIN) ||
-	    has_capability_noaudit(p, CAP_SYS_RESOURCE))
-		points /= 4;
-
-	/*
-	 * We don't want to kill a process with direct hardware access.
+	 *
+	 * Plus, We don't want to kill a process with direct hardware access.
 	 * Not only could that mess up the hardware, but usually users

Not sure where the -stable reference came from, I don't think this is a candidate.

mm/oom_kill.c unsigned long badness(struct task_struct *p, unsigned long uptime) · line 163

Quoted 3 lines

mm/oom_kill.c +3 -0

+	if (has_capability_noaudit(p, CAP_SYS_ADMIN) ||
+	    has_capability_noaudit(p, CAP_SYS_RESOURCE) ||
+	    has_capability_noaudit(p, CAP_SYS_RAWIO))

Acked-by David Rientjes <[email protected]>