Exhausting memory makes the system unresponsive but doesn't invoke OOM killer
Hi,
In 2010 I noticed that viewing many GIFs in a row using gpicview renders my Linux unresponsive. The problem still exists. There is very little I can do in such a situation. Rarely after some minutes the OOM killer kicks in and saves the day. Nevertheless, usually I end up using Alt+SysRq+B.
What happens is gpicview exhausting whole available memory in such a pattern that userspace becomes unresponsive. My application (crash.c) allocates memory in a very similar way using GDK to replicate the problem.
I keep the updated description of the problem and the source code here:
https://github.com/wodny/crasher
I've originally posted to linux-kernel:
http://marc.info/?t=145070009500007&r=1&w=2
but got no response.
I'm using:
3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
## Symptoms
The unresponsiveness goes with high CPU load and a lot of IO (read) operations on the root file system and its block device.
If I start the application from a text terminal (TTY) I can switch between them, but I gain nothing because shells in other terminals are unresponsive. Additionally I cannot perform any new logins (fork fails). If I stay at the same terminal I can kill the process almost immediately using a keystroke (e.g. ^C or ^\\). So apparently the kernel doesn't go into a deadlock.
When running the application under Xorg I cannot switch from X to a text terminal. Probably because Xorg uses VT_PROCESS to control terminal switching. Because the system is very busy Xorg doesn't get scheduled for running so it doesn't have time to acknowledge the switch request. Using SysRq+Alt+R doesn't help.
## OOM killer not triggered
At first I thought that the OOM killer needs so much time to find and kill the process. But further experiments using just text terminals showed that the real problem is that the kernel doesn't notice it should use OOM killer to kill the naughty application. The experiment:
0. switch to a text terminal, e.g. TTY2,
0. run the application (make test) and stay at the TTY,
0. wait until the system becomes unresponsive,
0. wait a lot...
0. either the kernel finally starts suspecting something and the OOM killer terminates the application or you just press ^C or ^\ and the system comes back almost immediately.
Killing the application with a signal sent via the TTY doesn't leave any suggestions in dmesg that anything bad happened. The only symptom is that for a moment the system behaves like after dropping caches.
## IO activity
top (or htop) and iostat are very useful in approximating the time left to the magic moment.
I suppose that in such a situation the OS starts to oscillate between freeing memory, cleaning caches and buffers, and loading some new data (see iostat logs).
I can observe the most impressive effects on my physical machine (logs/ph-*). On a VM (logs/vm-*) usually the OOM killer kills the process after a short time (5-120 seconds).
Logs from the VM have been gathered by piping top and iostat output to netcat. Logs from the physical machine have been gathered using primitive scripts and an Android phone connected over wifi.
Notice how the hang prevents scripts from reporting at stable 2-second intervals.
## Factors
Possible factors differentiating cases of recovering in seconds from recoveries after minutes (or never):
- another memory-consuming process running (e.g. Firefox),
- physical machine vs a VM (see dmesg logs),
- chipset and associated kernel functions (see dmesg logs) but see the remark on the i915 module,
- I'm ncursed.
Things that seem irrelevant (after testing):
- running the application in Xorg or a TTY,
- LUKS encryption of the root filesystem,
- vm.oom_kill_allocating_task setting,
- increasing vm.admin_reserve_kbytes,
- using swap space,
- disabling the i915 module caused there are no i915-specific functions in dmesg traces, but the system still blocks the same way,
- running the application with nice.
Any suggestions?
Signature · 9 lines
Marcin Szewczyk http://wodny.org mailto:[email protected] <- remove b / usuA? b xmpp:[email protected] xmpp:[email protected] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Marcin,
Quoted 7 lines · Marcin Szewczyk
Hi,
In 2010 I noticed that viewing many GIFs in a row using gpicview renders my Linux unresponsive. The problem still exists. There is very little I can do in such a situation. Rarely after some minutes the OOM killer kicks in and saves the day. Nevertheless, usually I end up using Alt+SysRq+B.
Have you tried kicking the OOM killer manually with sysrq+f?
Quoted 12 lines
What happens is gpicview exhausting whole available memory in such a pattern that userspace becomes unresponsive. My application (
crash.c) allocates memory in a very similar way using GDK to replicate the problem.I keep the updated description of the problem and the source code here:
https://github.com/wodny/crasherI've originally posted to linux-kernel:
http://marc.info/?t=145070009500007&r=1&w=2
but got no response.
Quoted 7 lines
I'm using:
3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09)x86_64GNU/Linux## Symptoms
The unresponsiveness goes with high CPU load and a lot of IO (read) operations on the root file system and its block device.
There is a semi-known issue of heavily thrashing page cache. Your crash program sucks up most memory and leaves very little for the executables and libraries to be cached, which results in multiple threads experiencing cache misses in their executable code, followed by fighting over the few remaining page cache slots, which are not enough to meet the demand at any given point in time. These threads than end up spending a lot of time a) searching for reusable cache slots and taking away slots that were only recently populated by another thread, possibly before that other thread has returned to userspace, and then b) waiting for disk to repopulate the cache slot which will be stolen by another thread soon, possibly before this thread had a chance to return to userspace as well.
That being said, there is no real solution to thrashing page cache as of this day. We have most infrastructure in place to detect it, but it isn't hooked up to the OOM killer yet. The only answer until then is try to keep free+buffer+cache at at least 10-15% of overall memory.
Since you can reproduce it easily, is there any chance you could grab backtraces (sysrq+t) of the tasks while the machine is in that state? That should confirm that most tasks are either waiting for IO or are inside page reclaim.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Johannes Weiner
Hi Marcin,
Hi,
Quoted 8 lines · Marcin Szewczyk
Quoted 5 lines
In 2010 I noticed that viewing many GIFs in a row using gpicview renders my Linux unresponsive. The problem still exists. There is very little I can do in such a situation. Rarely after some minutes the OOM killer kicks in and saves the day. Nevertheless, usually I end up using Alt+SysRq+B.
Have you tried kicking the OOM killer manually with sysrq+f?
I completely forgot about that option. It works both at TTY and under Xorg. Thank you very much.
Quoted 9 lines
Quoted 2 lines
The unresponsiveness goes with high CPU load and a lot of IO (read) operations on the root file system and its block device.
There is a semi-known issue of heavily thrashing page cache. Your crash program sucks up most memory and leaves very little for the executables and libraries to be cached, which results in multiple threads experiencing cache misses in their executable code, followed by fighting over the few remaining page cache slots, which are not enough to meet the demand at any given point in time. [...]
Thank you for the explanation.
Quoted 4 lines
That being said, there is no real solution to thrashing page cache as of this day. We have most infrastructure in place to detect it, but it isn't hooked up to the OOM killer yet. The only answer until then is try to keep free+buffer+cache at at least 10-15% of overall memory.
OK. Is there a good source of information I could subscribe to so I don't miss the moment when the integration code enters the kernel? Do you think LWN would mention it or should I just follow "oom" messages on linux-kernel and linux-mm?
Quoted 4 lines
Since you can reproduce it easily, is there any chance you could grab backtraces (sysrq+t) of the tasks while the machine is in that state? That should confirm that most tasks are either waiting for IO or are inside page reclaim.
I've updated the repository. I will later add this thread to the README.
Dump is available here:
https://github.com/wodny/crasher/blob/master/logs/kern.log
I didn't want to post 200kB to everybody so I didn't attach it to this email.
Signature · 9 lines
Marcin Szewczyk http://wodny.org mailto:[email protected] <- remove b / usuA? b xmpp:[email protected] xmpp:[email protected] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>