mm: oom_kill: revert 3% system memory bonus for privileged tasks
Subsystems (3)
PATCH 1/1 mm: oom_kill: revert 3% system memory bonus for privileged tasks
With a63d83f427fb ("oom: badness heuristic rewrite"), the OOM killer tries to avoid killing privileged tasks by subtracting 3% of overall memory (system or cgroup) from their per-task consumption. But as a result, all root tasks that consume less than 3% of overall memory are considered equal, and so it only takes 33+ privileged tasks pushing the system out of memory for the OOM killer to do something stupid and kill sshd or dhclient. For example, on a 32G machine it can't tell the difference between the 1M agetty and the 10G fork bomb member.
The changelog describes this 3% boost as the equivalent to the global overcommit limit being 3% higher for privileged tasks, but this is not the same as discounting 3% of overall memory from _every privileged task individually_ during OOM selection.
Revert back to the old priority boost of pretending root tasks are only a quarter of their actual size.
--- mm/oom_kill.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/oom_kill.c +3 -3
@@ -166,11 +166,11 @@ unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg,task_unlock (p);/* -* Root processes get 3% bonus, just like the __vm_enough_memory() -* implementation used by LSMs. +* Memory consumption being equal, prefer killing an +* unprivileged task over a root task. */ if (has_capability_noaudit (p,CAP_SYS_ADMIN )) - adj -=30 ; + points /=4 ;/* Normalize to oom_score_adj units */ adj *= totalpages /1000 ;
Signature · 7 lines
1.8.4.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 9 lines · Johannes Weiner
With
a63d83f427fb("oom: badness heuristic rewrite"), the OOM killer tries to avoid killing privileged tasks by subtracting 3% of overall memory (system or cgroup) from their per-task consumption. But as a result, all root tasks that consume less than 3% of overall memory are considered equal, and so it only takes 33+ privileged tasks pushing the system out of memory for the OOM killer to do something stupid and kill sshd or dhclient. For example, on a 32G machine it can't tell the difference between the 1M agetty and the 10G fork bomb member.
Quoted 8 lines
The changelog describes this 3% boost as the equivalent to the global overcommit limit being 3% higher for privileged tasks, but this is not the same as discounting 3% of overall memory from
_everyprivileged taskindividually_during OOM selection.Revert back to the old priority boost of pretending root tasks are only a quarter of their actual size.
Unfortunately, I think this could potentially be too much of a bonus. On your same 32GB machine, if a root process is using 18GB and a user process is using 14GB, the user process ends up getting selected while the current discount of 3% still selects the root process.
I do like the idea of scaling this bonus depending on points, however. I think it would be better if we could scale the discount but also limit it to some sane value.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 20 lines · David Rientjes
Quoted 9 lines · Johannes Weiner
With
a63d83f427fb("oom: badness heuristic rewrite"), the OOM killer tries to avoid killing privileged tasks by subtracting 3% of overall memory (system or cgroup) from their per-task consumption. But as a result, all root tasks that consume less than 3% of overall memory are considered equal, and so it only takes 33+ privileged tasks pushing the system out of memory for the OOM killer to do something stupid and kill sshd or dhclient. For example, on a 32G machine it can't tell the difference between the 1M agetty and the 10G fork bomb member.Quoted 8 lines
The changelog describes this 3% boost as the equivalent to the global overcommit limit being 3% higher for privileged tasks, but this is not the same as discounting 3% of overall memory from
_everyprivileged taskindividually_during OOM selection.Revert back to the old priority boost of pretending root tasks are only a quarter of their actual size.
Quoted 8 lines
Unfortunately, I think this could potentially be too much of a bonus. On your same 32GB machine, if a root process is using 18GB and a user process is using 14GB, the user process ends up getting selected while the current discount of 3% still selects the root process.
I do like the idea of scaling this bonus depending on points, however. I think it would be better if we could scale the discount but also limit it to some sane value.
I just reverted to the /= 4 because we had that for a long time and it seemed to work. I don't really mind either way as long as we get rid of that -3%. Do you have a suggestion?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 5 lines · Johannes Weiner
Quoted 5 lines
Unfortunately, I think this could potentially be too much of a bonus. On your same 32GB machine, if a root process is using 18GB and a user process is using 14GB, the user process ends up getting selected while the current discount of 3% still selects the root process.
Quoted 8 lines
Quoted 3 lines
I do like the idea of scaling this bonus depending on points, however. I think it would be better if we could scale the discount but also limit it to some sane value.
I just reverted to the /= 4 because we had that for a long time and it seemed to work. I don't really mind either way as long as we get rid of that -3%. Do you have a suggestion?
How about simply using 3% of the root process's points so that root processes get some bonus compared to non-root processes with the same memory usage and it's scaled to the usage rather than amount of available memory?
So rather than points /= 4, we do
if (has_capability_noaudit(p, CAP_SYS_ADMIN)) points -= (points * 3) / 100;
instead. Sound good?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 21 lines · David Rientjes
Quoted 5 lines · Johannes Weiner
Quoted 5 lines
Unfortunately, I think this could potentially be too much of a bonus. On your same 32GB machine, if a root process is using 18GB and a user process is using 14GB, the user process ends up getting selected while the current discount of 3% still selects the root process.
Quoted 8 lines
Quoted 3 lines
I do like the idea of scaling this bonus depending on points, however. I think it would be better if we could scale the discount but also limit it to some sane value.
I just reverted to the /= 4 because we had that for a long time and it seemed to work. I don't really mind either way as long as we get rid of that -3%. Do you have a suggestion?
How about simply using 3% of the root process's points so that root processes get some bonus compared to non-root processes with the same memory usage and it's scaled to the usage rather than amount of available memory?
Quoted 6 lines
So rather than points /= 4, we do
if (has_capability_noaudit(p, CAP_SYS_ADMIN)) points -= (points * 3) / 100;instead. Sound good?
Yes, should be okay.
Do you want to send a patch? Want me to update mine?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerPATCH 1/1 mm, oom: base root bonus on current usage
A 3% of system memory bonus is sometimes too excessive in comparison to other processes and can yield poor results when all processes on the system are root and none of them use over 3% of memory.
Replace the 3% of system memory bonus with a 3% of current memory usage bonus.
--- Documentation/filesystems/proc.txt | 4 ++-- mm/oom_kill.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
Documentation/filesystems/proc.txt +2 -2
@@ -1386,8 +1386,8 @@ may allocate from based on an estimation of its current memory and swap use. For example, if a task is using all allowed memory, its badness score will be 1000. If it is using half of its allowed memory, its score will be 500. -There is an additional factor included in the badness score: root -processes are given 3% extra memory over other tasks. +There is an additional factor included in the badness score: the current memory +and swap usage is discounted by 3% for root processes. The amount of "allowed" memory depends on the context in which the oom killer was called. If it is due to the memory assigned to the allocating task's cpuset
mm/oom_kill.c +1 -1
@@ -178,7 +178,7 @@ unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg,* implementation used by LSMs. */ if (has_capability_noaudit (p,CAP_SYS_ADMIN )) - adj -=30 ; + points -= (points *3 ) /100 ;/* Normalize to oom_score_adj units */ adj *= totalpages /1000 ;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch] mm, oom: base root bonus on current usage
Quoted 9 lines · David Rientjes
A 3% of system memory bonus is sometimes too excessive in comparison to other processes and can yield poor results when all processes on the system are root and none of them use over 3% of memory.
Replace the 3% of system memory bonus with a 3% of current memory usage bonus.
Reported-by Johannes Weiner <[email protected]>Signed-off-by David Rientjes <[email protected]>
Looks good, thanks a lot!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch] mm, oom: base root bonus on current usage
In reply to David RientjesQuoted 6 lines · On Sat, 25 Jan 2014 19:48:32 -0800 (PST) David Rientjes <[email protected]>…
A 3% of system memory bonus is sometimes too excessive in comparison to other processes and can yield poor results when all processes on the system are root and none of them use over 3% of memory.
Replace the 3% of system memory bonus with a 3% of current memory usage bonus.
This changelog has deteriorated :( We should provide sufficient info so that people will be able to determine whether this patch will fix a problem they or their customers are observing. And so that people who maintain -stable and its derivatives can decide whether to backport it.
I went back and stole some text from the v1 patch. Please review the result. The changelog would be even better if it were to describe the new behaviour under the problematic workloads.
We don't think -stable needs this?
From: David Rientjes <[email protected]>
Subject: mm, oom: base root bonus on current usage
A 3% of system memory bonus is sometimes too excessive in comparison to other processes.
With a63d83f427fb ("oom: badness heuristic rewrite"), the OOM killer tries to avoid killing privileged tasks by subtracting 3% of overall memory (system or cgroup) from their per-task consumption. But as a result, all root tasks that consume less than 3% of overall memory are considered equal, and so it only takes 33+ privileged tasks pushing the system out of memory for the OOM killer to do something stupid and kill sshd or dhclient. For example, on a 32G machine it can't tell the difference between the 1M agetty and the 10G fork bomb member.
The changelog describes this 3% boost as the equivalent to the global overcommit limit being 3% higher for privileged tasks, but this is not the same as discounting 3% of overall memory from _every privileged task individually_ during OOM selection.
Replace the 3% of system memory bonus with a 3% of current memory usage bonus.
---
Documentation/filesystems/proc.txt | 4 ++-- mm/oom_kill.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
Documentation/filesystems/proc.txt renamed from Documentation/filesystems/proc.txt~mm-oom-base-root-bonus-on-current-usage +2 -2
@@ -1386,8 +1386,8 @@ may allocate from based on an estimation For example, if a task is using all allowed memory, its badness score will be 1000. If it is using half of its allowed memory, its score will be 500. -There is an additional factor included in the badness score: root -processes are given 3% extra memory over other tasks. +There is an additional factor included in the badness score: the current memory +and swap usage is discounted by 3% for root processes. The amount of "allowed" memory depends on the context in which the oom killer was called. If it is due to the memory assigned to the allocating task's cpuset
mm/oom_kill.c renamed from mm/oom_kill.c~mm-oom-base-root-bonus-on-current-usage +1 -1
@@ -178,7 +178,7 @@ unsigned long oom_badness(struct task_st* implementation used by LSMs. */ if (has_capability_noaudit (p,CAP_SYS_ADMIN )) - adj -=30 ; + points -= (points *3 ) /100 ;/* Normalize to oom_score_adj units */ adj *= totalpages /1000 ;
_
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [patch] mm, oom: base root bonus on current usage
Quoted 9 lines · Andrew Morton
This changelog has deteriorated :( We should provide sufficient info so that people will be able to determine whether this patch will fix a problem they or their customers are observing. And so that people who maintain -stable and its derivatives can decide whether to backport it.
I went back and stole some text from the v1 patch. Please review the result. The changelog would be even better if it were to describe the new behaviour under the problematic workloads.
The new changelog looks fine with the exception of the mention of sshd which typically sets itself to be disabled from oom killing altogether.
Quoted 2 lines
We don't think -stable needs this?
Nobody has reported it in over three years as causing an issue, probably because people typically have enough memory that oom kills don't come from a ton of small processes allocating memory that can't be reclaimed, there's usually at least one large process to kill.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch] mm, oom: base root bonus on current usage
In reply to Andrew MortonQuoted 9 lines · Andrew Morton
Quoted 6 lines · On Sat, 25 Jan 2014 19:48:32 -0800 (PST) David Rientjes <[email protected]>…
A 3% of system memory bonus is sometimes too excessive in comparison to other processes and can yield poor results when all processes on the system are root and none of them use over 3% of memory.
Replace the 3% of system memory bonus with a 3% of current memory usage bonus.
Quoted 8 lines
This changelog has deteriorated :( We should provide sufficient info so that people will be able to determine whether this patch will fix a problem they or their customers are observing. And so that people who maintain -stable and its derivatives can decide whether to backport it.
I went back and stole some text from the v1 patch. Please review the result. The changelog would be even better if it were to describe the new behaviour under the problematic workloads.
Looks good to me, thanks. How about the below?
Quoted 1 line
We don't think -stable needs this?
That's actually a good idea, we're putting it into RHEL too.
Quoted 15 lines
From: David Rientjes <[email protected]>
Subject: mm, oom: base root bonus on current usageA 3% of system memory bonus is sometimes too excessive in comparison to other processes.
With
a63d83f427fb("oom: badness heuristic rewrite"), the OOM killer tries to avoid killing privileged tasks by subtracting 3% of overall memory (system or cgroup) from their per-task consumption. But as a result, all root tasks that consume less than 3% of overall memory are considered equal, and so it only takes 33+ privileged tasks pushing the system out of memory for the OOM killer to do something stupid and kill sshd or dhclient. For example, on a 32G machine it can't tell the difference between the 1M agetty and the 10G fork bomb member.
Quoted 7 lines
The changelog describes this 3% boost as the equivalent to the global overcommit limit being 3% higher for privileged tasks, but this is not the same as discounting 3% of overall memory from
_everyprivileged taskindividually_during OOM selection.Replace the 3% of system memory bonus with a 3% of current memory usage bonus.
By giving root tasks a bonus that is proportional to their actual size, they remain comparable even when relatively small. In the example above, the OOM killer will discount the 1M agetty's 256 badness points down to 179, and the 10G fork bomb's 262144 points down to 183500 points and make the right choice, instead of discounting both to 0 and killing agetty because it's first in the task list.
Quoted 4 lines
Signed-off-by David Rientjes <[email protected]>Reported-by Johannes Weiner <[email protected]>Acked-by Johannes Weiner <[email protected]>Signed-off-by Andrew Morton <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>