2.5.74-mm3 OOM killer fubared ?
I was running 2.5.74 on our newsfeeder box for a day without problems (and 2.5.72-mm2 for weeks before that).
Now with 2.5.74-mm3 (booted 11 hours ago) it keeps killing processes for no apparent reason:
Jul 10 11:59:01 quantum kernel: Out of Memory: Killed process 9952 (innfeed). Jul 10 12:25:48 quantum kernel: Out of Memory: Killed process 10498 (innfeed). Jul 10 12:25:48 quantum kernel: Fixed up OOM kill of mm-less task Jul 10 12:45:41 quantum kernel: Out of Memory: Killed process 11894 (innfeed). Jul 10 12:47:14 quantum kernel: Out of Memory: Killed process 13128 (innfeed). Jul 10 12:53:09 quantum kernel: Out of Memory: Killed process 13221 (innfeed). Jul 10 12:55:12 quantum kernel: Out of Memory: Killed process 13649 (innfeed).
I check seconds before the last kill:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND news 13649 0.2 0.1 60368 1108 ? SN 12:53 0:00 innfeed -y
# free
total used free shared buffers cached
Mem: 1035212 1030104 5108 0 548208 307148
-/+ buffers/cache: 174748 860464
Swap: 996020 41708 954312Enough memory free, no problems at all .. yet every few minutes the OOM killer kills one of my innfeed processes.
I notice that in -mm3 this was deleted relative to -vanilla:
-
- /*
- * Enough swap space left? Not OOM.
- */
- if (nr_swap_pages > 0)
- return;
.. is that what causes this ? In any case, that should't vene matter - there's plenty of memory in this box, all buffers and cached, but that should be easily freed ..
Related mm question - this box is a news server, which does a lot of streaming I/O, and also keeps a history database open. I have the idea that the streaming I/O evicts the history database hash and index file caches from memory, which I do not want. Any chance of a control on a filedescriptor that tells it how persistant to be in caching file data ? E.g. a sort of "nice" for the cache, so that I could say that streaming data may be flushed from buffers/cache earlier than other data (where the other data would be the database files) ?
Mike.
Quoted 10 lines · Miquel van Smoorenburg
Enough memory free, no problems at all .. yet every few minutes the OOM killer kills one of my innfeed processes.
I notice that in -mm3 this was deleted relative to -vanilla:(quoted diff) +0
-6- - /* - * Enough swap space left? Not OOM. - */ - if (nr_swap_pages > 0) - return;
Quoted 3 lines
.. is that what causes this ? In any case, that should't vene matter - there's plenty of memory in this box, all buffers and cached, but that should be easily freed ..
This means we're calling into it more often than we should be. Basically, we hit __alloc_pages() with __GFP_WAIT set, find nothing we're allowed to touch, dive into try_to_free_pages(), fall through scanning there, sleep in blk_congestion_wait(), wake up again, try to shrink_slab(), find nothing there either, repeat that 11 more times, and then fall through to out_of_memory()... and this happens at at least 10Hz.
since = now - lastkill;
if (since < HZ*5)
goto out_unlock;try s/goto out_unlock/goto reset/ and let me know how it goes.
-- wli
In article <[email protected]>,
Quoted 23 lines · William Lee Irwin III
Quoted 10 lines · Miquel van Smoorenburg
Enough memory free, no problems at all .. yet every few minutes the OOM killer kills one of my innfeed processes.
I notice that in -mm3 this was deleted relative to -vanilla:(quoted diff) +0
-6- - /* - * Enough swap space left? Not OOM. - */ - if (nr_swap_pages > 0) - return;Quoted 3 lines
.. is that what causes this ? In any case, that should't vene matter - there's plenty of memory in this box, all buffers and cached, but that should be easily freed ..
This means we're calling into it more often than we should be. Basically, we hit
__alloc_pages()with__GFP_WAITset, find nothing we're allowed to touch, dive intotry_to_free_pages(), fall through scanning there, sleep inblk_congestion_wait(), wake up again, try toshrink_slab(), find nothing there either, repeat that 11 more times, and then fall through toout_of_memory()... and this happens at at least 10Hz.
Quoted 5 lines
since = now - lastkill; if (since < HZ*5) goto out_unlock;try s/goto out_unlock/goto reset/ and let me know how it goes.
But that will only change the rate at which processes are killed, not the fact that they are killed in the first place, right ?
As I said I've got plenty memory free ... perhaps I need to tune /proc/sys/vm because I've got so much streaming I/O ? Possibly, there are too many dirty pages so cleaning them out faster might help (and let pflushd do it instead of my single-threaded app)
# cd /proc/sys/vm # echo 200 > dirty_writeback_centisecs # echo 60 > dirty_ratio # echo 500 > dirty_expire_centisecs
I'll let it run like this for a while first.
If this helps, perhaps it means that the VM doesn't write out dirty pages fast enough when low on memory ?
Mike.
Quoted 31 lines · Miquel van Smoorenburg
In article <[email protected]>,
Quoted 23 lines · William Lee Irwin III
Quoted 10 lines · Miquel van Smoorenburg
Enough memory free, no problems at all .. yet every few minutes the OOM killer kills one of my innfeed processes.
I notice that in -mm3 this was deleted relative to -vanilla:(quoted diff) +0
-6- - /* - * Enough swap space left? Not OOM. - */ - if (nr_swap_pages > 0) - return;Quoted 3 lines
.. is that what causes this ? In any case, that should't vene matter - there's plenty of memory in this box, all buffers and cached, but that should be easily freed ..
This means we're calling into it more often than we should be. Basically, we hit
__alloc_pages()with__GFP_WAITset, find nothing we're allowed to touch, dive intotry_to_free_pages(), fall through scanning there, sleep inblk_congestion_wait(), wake up again, try toshrink_slab(), find nothing there either, repeat that 11 more times, and then fall through toout_of_memory()... and this happens at at least 10Hz.Quoted 5 lines
since = now - lastkill; if (since < HZ*5) goto out_unlock;try s/goto out_unlock/goto reset/ and let me know how it goes.
Quoted 8 lines
But that will only change the rate at which processes are killed, not the fact that they are killed in the first place, right ?
As I said I've got plenty memory free ... perhaps I need to tune
/proc/sys/vmbecause I've got so much streaming I/O ? Possibly, there are too many dirty pages so cleaning them out faster might help (and let pflushd do it instead of my single-threaded app)
The problem, as I see it, is that you can dirty pages 10-15 times faster than they can be written to disk. So, you will always have the possibility of an OOM situation as long as you are I/O bound. FYI, you can read/write RAM at 1,000+ megabytes/second, but you can only write to disk at 80 megabytes/second with the fastest SCSI around, 40 megabytes/second with ATA, 20 megabytes/second with IDE/DMA, 10 megabytes/second with PIOW, etc. There just aren't any disks around that will run at RAM speeds so buffered I/O will always result in full buffers if the I/O is sustained. To completely solve the OOM situation requires throttling the generation of data.
It is only when the data generation rate is less than or equal to the data storage rate that you can generate data forever.
A possibility may be to not return control to the writing process (including swap), until the write completes if RAM gets low. In other words, stop buffering data in RAM in tight memory situations. This forces all the tasks to wait and, therefore slows down the dirty-page and data generation rate to match the RAM available.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips). Why is the government concerned about the lunatic fringe? Think about it.
In article <Pine.LNX.4.53.0307100918410.203@chaos>,
Quoted 7 lines · Richard B. Johnson
Quoted 5 lines · Miquel van Smoorenburg
As I said I've got plenty memory free ... perhaps I need to tune
/proc/sys/vmbecause I've got so much streaming I/O ? Possibly, there are too many dirty pages so cleaning them out faster might help (and let pflushd do it instead of my single-threaded app)
I did the tuning now, but it did not help much. Alas.
Quoted 10 lines
The problem, as I see it, is that you can dirty pages 10-15 times faster than they can be written to disk. So, you will always have the possibility of an OOM situation as long as you are I/O bound. FYI, you can read/write RAM at 1,000+ megabytes/second, but you can only write to disk at 80 megabytes/second with the fastest SCSI around, 40 megabytes/second with ATA, 20 megabytes/second with IDE/DMA, 10 megabytes/second with PIOW, etc. There just aren't any disks around that will run at RAM speeds so buffered I/O will always result in full buffers if the I/O is sustained. To completely solve the OOM situation requires throttling the generation of data.
My disks are fast enough - under 2.5.74-vanilla, no problem.
Quoted 5 lines
It is only when the data generation rate is less than or equal to the data storage rate that you can generate data forever.
A possibility may be to not return control to the writing process (including swap), until the write completes if RAM gets low.
That's what can be tuned with /proc/sys/vm/dirty_ratio , right ? If I understand Documentation/filesystems/proc.txt correctly.
Mike.
Quoted 4 lines · Richard B. Johnson
The problem, as I see it, is that you can dirty pages 10-15 times faster than they can be written to disk. So, you will always have the possibility of an OOM situation as long as you are I/O bound.
That's not a problem at all. I think the VM never starts pageout IO on a page that doesn't look like it'll become freeable after the IO is done, so we simply shouldn't go into the OOM killer as long as there are pages waiting on pageout IO to finish.
Once we really are OOM we shouldn't have pages in pageout IO.
This is what I am doing in current 2.4-rmap and it seems to do the right thing in both the "heavy IO" and the "out of memory" corner cases.
Signature · 1 line
Great minds drink alike.
Quoted 5 lines
Quoted 4 lines
The problem, as I see it, is that you can dirty pages 10-15 times faster than they can be written to disk. So, you will always have the possibility of an OOM situation as long as you are I/O bound.
Quoted 8 lines
That's not a problem at all. I think the VM never starts pageout IO on a page that doesn't look like it'll become freeable after the IO is done, so we simply shouldn't go into the OOM killer as long as there are pages waiting on pageout IO to finish.
Once we really are OOM we shouldn't have pages in pageout IO.
What piece of code does prevent that?
As I see, OOM is triggered if no pages were freed in few loops. It doesn't care about pages that are already being written or pages for which write operation was started.
The only thing that prevents total oom when writing a lot of dirty pages is blk_congestion_wait, but it's pretty unreliable because
1) it uses timeout
2) not all page writes go through block devices (NFS, NBD etc.) blk_congestion_wait may be used for improving performance, but not for ensuring stability.
Quoted 3 lines
This is what I am doing in current 2.4-rmap and it seems to do the right thing in both the "heavy IO" and the "out of memory" corner cases.
I remember there was (and probably still is, only with smaller probability) a similar bug in 2.2.
Mikulas
Quoted 4 lines · William Lee Irwin III
Quoted 4 lines
since = now - lastkill; if (since < HZ*5) goto out_unlock;try s/goto out_unlock/goto reset/ and let me know how it goes.
Quoted 6 lines · Miquel van Smoorenburg
But that will only change the rate at which processes are killed, not the fact that they are killed in the first place, right ? As I said I've got plenty memory free ... perhaps I need to tune
/proc/sys/vmbecause I've got so much streaming I/O ? Possibly, there are too many dirty pages so cleaning them out faster might help (and let pflushd do it instead of my single-threaded app)
That's not what it's supposed to do. The thought behind it is that since out_of_memory()'s count is not reset unless it's been 5s since the last time this was ever invoked, it will happen on a regular basis after the first kill if it is invoked regularly. It's actually a bit too late, since something's already been killed, but it should make a larger difference than merely altering the rate.
-- wli
In article <[email protected]>,
Quoted 15 lines · William Lee Irwin III
In article <[email protected]>, William Lee Irwin
Quoted 4 lines · III
Quoted 4 lines
since = now - lastkill; if (since < HZ*5) goto out_unlock;try s/goto out_unlock/goto reset/ and let me know how it goes.
Quoted 6 lines · Miquel van Smoorenburg
But that will only change the rate at which processes are killed, not the fact that they are killed in the first place, right ? As I said I've got plenty memory free ... perhaps I need to tune
/proc/sys/vmbecause I've got so much streaming I/O ? Possibly, there are too many dirty pages so cleaning them out faster might help (and let pflushd do it instead of my single-threaded app)
Quoted 6 lines
That's not what it's supposed to do. The thought behind it is that since out_of_memory()'s count is not reset unless it's been 5s since the last time this was ever invoked, it will happen on a regular basis after the first kill if it is invoked regularly. It's actually a bit too late, since something's already been killed, but it should make a larger difference than merely altering the rate.
Well, that won't help in my case, as my problem is not that many processes are killed - it's just that every few minutes (sometimes 3 minutes, sometimes 30, sometimes an hour) an innocent process gets killed (just one) with 2.5.74-mm3. And that did not happen with 2.5.74 or 2.5.72-mm2
Mike.
Quoted 6 lines · William Lee Irwin III
Quoted 6 lines
That's not what it's supposed to do. The thought behind it is that since out_of_memory()'s count is not reset unless it's been 5s since the last time this was ever invoked, it will happen on a regular basis after the first kill if it is invoked regularly. It's actually a bit too late, since something's already been killed, but it should make a larger difference than merely altering the rate.
Quoted 5 lines · Miquel van Smoorenburg
Well, that won't help in my case, as my problem is not that many processes are killed - it's just that every few minutes (sometimes 3 minutes, sometimes 30, sometimes an hour) an innocent process gets killed (just one) with 2.5.74-mm3. And that did not happen with 2.5.74 or 2.5.72-mm2
Okay, it won't help your case, then. I've had this improvement to its heuristics on the back burner for a while but haven't gone so far as to dig up a case it directly benefits. It's small enough I'll think about it later.
-- wli
Quoted 3 lines · Miquel van Smoorenburg
I was running 2.5.74 on our newsfeeder box for a day without problems (and 2.5.72-mm2 for weeks before that).
And how was 2.5.72-mm2 performing, generally?
Quoted 12 lines
Now with 2.5.74-mm3 (booted 11 hours ago) it keeps killing processes for no apparent reason:
...
I notice that in -mm3 this was deleted relative to -vanilla:(quoted diff) +0
-6- - /* - * Enough swap space left? Not OOM. - */ - if (nr_swap_pages > 0) - return;
Quoted 1 line
.. is that what causes this?
Yes. That was a "hmm, I wonder what happens if I do this" patch. It's interesting that we're going down that code path.
Is your INND configured to use the strange mmap(MAP_SHARED) of a blockdev
thing? That could explain the scanning hysteria.
Quoted 9 lines
Related mm question - this box is a news server, which does a lot of streaming I/O, and also keeps a history database open. I have the idea that the streaming I/O evicts the history database hash and index file caches from memory, which I do not want. Any chance of a control on a filedescriptor that tells it how persistant to be in caching file data ? E.g. a sort of "nice" for the cache, so that I could say that streaming data may be flushed from buffers/cache earlier than other data (where the other data would be the database files) ?
Makes sense. We can use posix_fadvise(POSIX_FADV_NOREUSE) as a hint to tell the VM/VFS to throw away old pages. I'll take a look at that.
In article <[email protected]>,
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Miquel van Smoorenburg
I was running 2.5.74 on our newsfeeder box for a day without problems (and 2.5.72-mm2 for weeks before that).
And how was 2.5.72-mm2 performing, generally?
Quite allright, but it had (known) ext3 problems so I went to 2.5.74. Linus fixed the issue with siimage yesterday, so that was stable. No fun in that so I went to -mm3 ;)
The VM appears to run better with 2.5 than with 2.4, but I do not experience a real difference between 2.5.74-vanilla and -mm3.
Difference between 2.4.21 and 2.5.7x is that interactive response under load is much better, and that the VM is better. With 2.4.21, the nightly expire process (rebuilds the history database) was running for 3 days without making progress if the news server (INN) was running at the same time, because its working set kept getting evicted to swap. 2.5 behaves better - expire finished in a few hours. Ofcourse now I stop the INN process before expire and restart it afterwards and the expire process runs in 5 minutes ... which simply means I need more memory.
Quoted 12 lines
Quoted 12 lines
Now with 2.5.74-mm3 (booted 11 hours ago) it keeps killing processes for no apparent reason:
...
I notice that in -mm3 this was deleted relative to -vanilla:(quoted diff) +0
-6- - /* - * Enough swap space left? Not OOM. - */ - if (nr_swap_pages > 0) - return;
Quoted 4 lines
Quoted 1 line
.. is that what causes this?
Yes. That was a "hmm, I wonder what happens if I do this" patch. It's interesting that we're going down that code path.
Yes, and it's really not happening all that often. For example:
Jul 10 12:25:48 quantum kernel: Fixed up OOM kill of mm-less task Jul 10 12:45:41 quantum kernel: Out of Memory: Killed process 11894 (innfeed). Jul 10 12:47:14 quantum kernel: Out of Memory: Killed process 13128 (innfeed). Jul 10 12:53:09 quantum kernel: Out of Memory: Killed process 13221 (innfeed). Jul 10 12:55:12 quantum kernel: Out of Memory: Killed process 13649 (innfeed). Jul 10 15:18:24 quantum kernel: Out of Memory: Killed process 13754 (innfeed). Jul 10 15:35:54 quantum kernel: Out of Memory: Killed process 22667 (innfeed). Jul 10 16:38:05 quantum kernel: Out of Memory: Killed process 23781 (innfeed). Jul 10 17:04:32 quantum kernel: Out of Memory: Killed process 27640 (innfeed). Jul 10 20:26:12 quantum kernel: Out of Memory: Killed process 29298 (innfeed). Jul 10 22:05:43 quantum kernel: Out of Memory: Killed process 9333 (innfeed). Jul 10 22:24:00 quantum kernel: Out of Memory: Killed process 15476 (innfeed). Jul 10 22:24:00 quantum kernel: Fixed up OOM kill of mm-less task Jul 10 22:44:47 quantum kernel: Out of Memory: Killed process 16616 (innfeed). Jul 11 01:23:32 quantum kernel: Out of Memory: Killed process 17896 (innfeed). Jul 11 01:24:50 quantum kernel: Out of Memory: Killed process 28108 (innfeed).
Sometimes it happens every 5 minutes though.
Quoted 2 lines
Is your INND configured to use the strange
mmap(MAP_SHARED)of a blockdev
thing? That could explain the scanning hysteria.
Yes. In fact, INND itself just mmaps small parts of the blockdev (just a few 100K of bitmap at the beginning), but the outgoing processes called "innfeed" actually mmaps whole articles from the blockdevs, which could be several megs. Not hundeds of megs, though. Maybe 20.
Quoted 10 lines
Quoted 9 lines
Related mm question - this box is a news server, which does a lot of streaming I/O, and also keeps a history database open. I have the idea that the streaming I/O evicts the history database hash and index file caches from memory, which I do not want. Any chance of a control on a filedescriptor that tells it how persistant to be in caching file data ? E.g. a sort of "nice" for the cache, so that I could say that streaming data may be flushed from buffers/cache earlier than other data (where the other data would be the database files) ?
Quoted 2 lines
Makes sense. We can use
posix_fadvise(POSIX_FADV_NOREUSE)as a hint to tell the VM/VFS to throw away old pages. I'll take a look at that.
Well, throw away earlier than other pages. If nothing else needs those pages, I'd like to keep them around anyway. What comes in streaming, goes out streaming - but usually within seconds.
Mike.
Quoted 12 lines
Enough memory free, no problems at all .. yet every few minutes the OOM killer kills one of my innfeed processes.
I notice that in -mm3 this was deleted relative to -vanilla:
(quoted diff) +0
-6- - /* - * Enough swap space left? Not OOM. - */ - if (nr_swap_pages > 0) - return;
Quoted 3 lines
.. is that what causes this ? In any case, that should't vene matter - there's plenty of memory in this box, all buffers and cached, but that should be easily freed ..
This is not cause, it only makes the bug more obvious.
With that code it is wrong too, it would trigger exactly same unnecesary oom kills if you didn't have swap or if you had swap full and lot of free memory as dirty filesystem cache.
Mikulas