OOM killer not nearly agressive enough?
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Do we agree that OOM killer should have reacted way sooner?
Is there something I can tweak to make it behave more reasonably?
Best regards,
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 10 lines · Pavel Machek
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Does sysrq+f help?
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Quoted 1 line
Is there something I can tweak to make it behave more reasonably?
PSI based early OOM killing might help. See https://github.com/facebookincubator/oomd
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 6 lines · Michal Hocko
Quoted 5 lines · Pavel Machek
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Quoted 7 lines
Quoted 5 lines
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Does sysrq+f help?
May try that next time.
Quoted 6 lines
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Swap was full, so "completely out of memory", I guess. Chromium does that fairly often :-(.
Quoted 3 lines
Quoted 1 line
Is there something I can tweak to make it behave more reasonably?
PSI based early OOM killing might help. See https://github.com/facebookincubator/oomd
Um. Before doing that... is there some knob somewhere saying "hey oomkiller, one hour to recover machine is a bit too much, can you please react sooner"? PSI is completely different system, but I guess I should attempt to tweak the existing one first...
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 12 lines · Michal Hocko
Quoted 10 lines · Pavel Machek
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Quoted 8 lines
Does sysrq+f help?
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
What statistics are best to collect? Would the memory lines from top do the trick? I normally have gkrellm running, but I found its results hard to interpret.
Best regards,
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 19 lines · Pavel Machek
Quoted 6 lines · Michal Hocko
Quoted 5 lines · Pavel Machek
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Quoted 7 lines
Quoted 5 lines
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Does sysrq+f help?
May try that next time.
Quoted 2 lines
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
Quoted 7 lines
Quoted 4 lines
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Swap was full, so "completely out of memory", I guess. Chromium does that fairly often :-(.
The oom heuristic is based on the reclaim failure. If the reclaim makes some progress then the oom killer is not hit. Have a look at should_reclaim_retry for more details.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Is there something I can tweak to make it behave more reasonably?
PSI based early OOM killing might help. See https://github.com/facebookincubator/oomd
Um. Before doing that... is there some knob somewhere saying "hey oomkiller, one hour to recover machine is a bit too much, can you please react sooner"?
No, there is nothing like that.
Quoted 2 lines
PSI is completely different system, but I guess I should attempt to tweak the existing one first...
PSI is measuring the cost of the allocation (among other things) and that can give you some idea on how much time is spent to get memory. Userspace can implement a policy based on that and act. The kernel oom killer is the last resort when there is really no memory to allocate.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 17 lines · Pavel Machek
Quoted 16 lines · Michal Hocko
Quoted 10 lines · Pavel Machek
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Does sysrq+f help?
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
Quoted 8 lines
Quoted 4 lines
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
What statistics are best to collect? Would the memory lines from top do the trick? I normally have gkrellm running, but I found its results hard to interpret.
/proc/vmstat (and collecting it periodically) gives the most comprehensive picture about the state of MM. Interpreting numbers is far from trivial though. It requires to analyze multiple snapshots usually to see how the situation evolves.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 19 lines · Pavel Machek
Quoted 6 lines · Michal Hocko
Quoted 5 lines · Pavel Machek
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Quoted 7 lines
Quoted 5 lines
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Does sysrq+f help?
May try that next time.
Quoted 2 lines
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
Quoted 8 lines
Quoted 4 lines
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Swap was full, so "completely out of memory", I guess. Chromium does that fairly often :-(.
Have you considered restricting its memory limits a la ulimit -m?
I've taken to running browsers in nspawn containers for general isolation improvements, but this also makes it easy to set cgroup resource limits like memcg. i.e. --property MemoryMax=2G
This prevents the browser from bogging down the entire system, but it doesn't prevent thrashing before FF OOMs within its control group.
I do feel there's a problem with the kernel's reclaim algorithm, it seems far too willing to evict file-backed pages that are recently in use. But at least with memcg this behavior is isolated to the cgroup, though it still generates a crapload of disk reads from all the thrashing.
Regards,
Vito Caputo
Quoted 25 lines · Vito Caputo
Quoted 24 lines · Pavel Machek
Quoted 6 lines · Michal Hocko
Quoted 5 lines · Pavel Machek
Hi!
I updated my userspace to x86-64, and now chromium likes to eat all the memory and bring the system to standstill.
Quoted 7 lines
Quoted 5 lines
Unfortunately, OOM killer does not react:
I'm now running "ps aux", and it prints one line every 20 seconds or more. Do we agree that is "unusable" system? I attempted to do kill from other session.
Does sysrq+f help?
May try that next time.
Quoted 6 lines
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Quoted 5 lines
Quoted 3 lines
Swap was full, so "completely out of memory", I guess. Chromium does that fairly often :-(.
Have you considered restricting its memory limits a la
ulimit -m?
The kernel ignores RLIMIT_RSS. Unless the browser takes it into consideration then I do not see how that would help.
Quoted 3 lines
I've taken to running browsers in nspawn containers for general isolation improvements, but this also makes it easy to set cgroup resource limits like memcg. i.e. --property MemoryMax=2G
Yes, this should help to isolate the problem.
Quoted 6 lines
This prevents the browser from bogging down the entire system, but it doesn't prevent thrashing before FF OOMs within its control group.
I do feel there's a problem with the kernel's reclaim algorithm, it seems far too willing to evict file-backed pages that are recently in use.
It is true that the memory reclaim is quite page cache reclaim biased unless there is very small amount of the page cache. Page cache refault is considered during the reclaim but I am afraid that there are still corner cases where the workload might end up threshing. Be it on the page cache or the anonymous memory depending on the workload. Anyway getting data from real workloads is always good so that we can think on improving existing heuristics.
Signature · 2 lines
Michal Hocko SUSE Labs
Hi!
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Quoted 6 lines
Quoted 2 lines
Swap was full, so "completely out of memory", I guess. Chromium does that fairly often :-(.
The oom heuristic is based on the reclaim failure. If the reclaim makes some progress then the oom killer is not hit. Have a look at
should_reclaim_retryfor more details.
Thanks for pointer.
I guess setting MAX_RECLAIM_RETRIES to 1 is not something you'd recommend? :-).
Quoted 8 lines
Quoted 2 lines
PSI is completely different system, but I guess I should attempt to tweak the existing one first...
PSI is measuring the cost of the allocation (among other things) and that can give you some idea on how much time is spent to get memory. Userspace can implement a policy based on that and act. The kernel oom killer is the last resort when there is really no memory to allocate.
So what I'm seeing is system that is unresponsive, easily for an hour.
Sometimes, I'm able to log in. When I could do that, system was absurdly slow, like ps printing at more than 10 seconds per line. ps on my system takes 300msec, estimate in the slow case would be 2000 seconds, that is slowdown by factor of 6000x. That would be X terminal opening in like two hours... that's not really usable.
DRAM is in 100nsec range, disk is in 10msec range; so worst case slowdown is somewhere in 100000x range. (Actually, in the worst case userland will do no progress at all, since you can need at 4+ pages in single CPU instruction, right?)
But kernel is happy; system is unusable and will stay unusable for hour or more, and there's not much user can do. (Besides sysrq, thanks for the hint).
Can we do better? This is equivalent of system crash, and it is _way_ too easy to trigger. Should we do better by default?
Dunno. If user moved the mouse, and cursor did not move for 10 seconds, perhaps it is time for oom kill?
Or should I add more swap? Is it terrible to place swap on SSD?
Best regards,
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 48 lines · Pavel Machek
Hi!
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Quoted 6 lines
Quoted 2 lines
Swap was full, so "completely out of memory", I guess. Chromium does that fairly often :-(.
The oom heuristic is based on the reclaim failure. If the reclaim makes some progress then the oom killer is not hit. Have a look at
should_reclaim_retryfor more details.Thanks for pointer.
I guess setting
MAX_RECLAIM_RETRIESto 1 is not something you'd recommend? :-).Quoted 8 lines
Quoted 2 lines
PSI is completely different system, but I guess I should attempt to tweak the existing one first...
PSI is measuring the cost of the allocation (among other things) and that can give you some idea on how much time is spent to get memory. Userspace can implement a policy based on that and act. The kernel oom killer is the last resort when there is really no memory to allocate.
So what I'm seeing is system that is unresponsive, easily for an hour.
Sometimes, I'm able to log in. When I could do that, system was absurdly slow, like ps printing at more than 10 seconds per line. ps on my system takes 300msec, estimate in the slow case would be 2000 seconds, that is slowdown by factor of 6000x. That would be X terminal opening in like two hours... that's not really usable.
DRAM is in 100nsec range, disk is in 10msec range; so worst case slowdown is somewhere in 100000x range. (Actually, in the worst case userland will do no progress at all, since you can need at 4+ pages in single CPU instruction, right?)
But kernel is happy; system is unusable and will stay unusable for hour or more, and there's not much user can do. (Besides sysrq, thanks for the hint).
Quoted 8 lines
Can we do better? This is equivalent of system crash, and it is _way_ too easy to trigger. Should we do better by default?
Dunno. If user moved the mouse, and cursor did not move for 10 seconds, perhaps it is time for oom kill?
Or should I add more swap? Is it terrible to place swap on SSD?
What's the kernel version? How much memory is anon and file pages? What's your swap to DRAM ratio? Are you using in-memory compression based swap? Have you tried to disable swap completely?
Shakeel
Quoted 12 lines · Pavel Machek
Hi!
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Do we agree that OOM killer should have reacted way sooner?
This is impossible to answer without knowing what was going on at the time. Was the system threshing over page cache/swap? In other words, is the system completely out of memory or refaulting the working set all the time because it doesn't fit into memory?
Swap was full, so "completely out of memory", I guess. Chromium does that fairly often :-(.
Quoted 8 lines
Quoted 3 lines
The oom heuristic is based on the reclaim failure. If the reclaim makes some progress then the oom killer is not hit. Have a look at
should_reclaim_retryfor more details.Thanks for pointer.
I guess setting
MAX_RECLAIM_RETRIESto 1 is not something you'd recommend? :-).
You can certainly play with that. I am not overly optimistic that would help though because symptoms of a threshing system is that we actually do not even reach this point. Pages are simply recycled but they evict other part of the hot working set. But I am only guessing what is the problem in your case. Anyway MAX_RECLAIM_RETRIES would tend to be more timing sensitive in general. If the reclaim progress cannot be made because of IO latencies or other resource depletion then the OOM be declared too early. The current MAX_RECLAIM_RETRIES is not something we have tuned for in any sense. I remember it didn't make much difference to change it unless the number would be really high which would be signal that the reclaim is not throttled very well.
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
PSI is completely different system, but I guess I should attempt to tweak the existing one first...
PSI is measuring the cost of the allocation (among other things) and that can give you some idea on how much time is spent to get memory. Userspace can implement a policy based on that and act. The kernel oom killer is the last resort when there is really no memory to allocate.
Quoted 7 lines
So what I'm seeing is system that is unresponsive, easily for an hour.
Sometimes, I'm able to log in. When I could do that, system was absurdly slow, like ps printing at more than 10 seconds per line. ps on my system takes 300msec, estimate in the slow case would be 2000 seconds, that is slowdown by factor of 6000x. That would be X terminal opening in like two hours... that's not really usable.
It would be great to find out what is the bottle neck. Is the allocator stuck in the memory reclaim? Waiting on some lock? Reclaiming pages which are stolen by other contending processes?
Signature · 2 lines
Michal Hocko SUSE Labs