VM in 2.4.7-pre hurts...
Oh this is a fun one :)
When building gcc-2.96 RPM using gcc-2.96 under kernel 2.4.7 on alpha, the system goes --deeply-- into swap. Not pretty at all. The system will be 200MB+ into swap, with 200MB+ in cache! I presume this affects 2.4.7-release also.
System has 256MB of swap, and 384MB of RAM.
Only patches applied are Rik's recent OOM killer friendliness patch, and Andrea's ksoftirq patch.
I ran "vmstat 3" throughout the build, and that output is attached. I also manually ran "ps wwwaux >> ps.txt" periodically. This second output is not overly helpful, because the system was swapping and unuseable for the times when the 'ps' output would be most useful.
Both outputs are attached, as they compressed pretty nicely.
Signature · 3 lines
Jeff Garzik | A recent study has shown that too much soup Building 1024 | can cause malaise in laboratory mice. MandrakeSoft |
Quoted 10 lines · Jeff Garzik
Oh this is a fun one :)
When building gcc-2.96 RPM using gcc-2.96 under kernel 2.4.7 on alpha, the system goes --deeply-- into swap. Not pretty at all. The system will be 200MB+ into swap, with 200MB+ in cache! I presume this affects 2.4.7-release also.
System has 256MB of swap, and 384MB of RAM.
Quoted 7 lines
Only patches applied are Rik's recent OOM killer friendliness patch, and Andrea's ksoftirq patch.
I ran "vmstat 3" throughout the build, and that output is attached. I also manually ran "ps wwwaux >> ps.txt" periodically. This second output is not overly helpful, because the system was swapping and unuseable for the times when the 'ps' output would be most useful.
Sorry, I forgot to mention that OOM killer kicked in twice. You can probably pick out the points where it kicked in, in the vmstat output.
Signature · 3 lines
Jeff Garzik | A recent study has shown that too much soup Building 1024 | can cause malaise in laboratory mice. MandrakeSoft |
Quoted 5 lines · Jeff Garzik
When building gcc-2.96 RPM using gcc-2.96 under kernel 2.4.7 on alpha, the system goes --deeply-- into swap. Not pretty at all. The system will be 200MB+ into swap, with 200MB+ in cache! I presume this affects 2.4.7-release also.
Note that "200MB+ into swap, with 200MB+ in cache" is NOT bad in itself.
It only means that we have scanned the VM, and allocated swap-space for 200MB worth of VM space. It does NOT necessarily mean that any actual swapping has been taking place: you should realize that the "cache" is likely to be not at least partly the _swap_ cache that hasn't been written out.
This is an accounting problem, nothing more. It looks strange, but it's normal.
Now, the fact that the system appears unusable does obviously mean that something is wrong. But you're barking up the wrong tree.
Although it might be the "right tree" in the sense that we might want to remove the swap cache from the "cached" output in /proc/meminfo. It might be more useful to separate out "Cached" and "SwapCache": add a new line to /proc/meminfo that is "swapper_space.nr_pages", and make the current code that does
atomic_read(&page_cache_size)
do
(atomic_read(&page_cache_size) - swapper_space.nrpages)
instead. That way the vmstat output might be more useful, although vmstat obviously won't know about the new "SwapCache:" field..
Can you try that, and see if something else stands out once the misleading accounting is taken care of?
Linus
Quoted 16 lines · Linus Torvalds
Quoted 5 lines · Jeff Garzik
When building gcc-2.96 RPM using gcc-2.96 under kernel 2.4.7 on alpha, the system goes --deeply-- into swap. Not pretty at all. The system will be 200MB+ into swap, with 200MB+ in cache! I presume this affects 2.4.7-release also.
Note that "200MB+ into swap, with 200MB+ in cache" is NOT bad in itself.
It only means that we have scanned the VM, and allocated swap-space for 200MB worth of VM space. It does NOT necessarily mean that any actual swapping has been taking place: you should realize that the "cache" is likely to be not at least partly the _swap_ cache that hasn't been written out.
Quoted 5 lines
This is an accounting problem, nothing more. It looks strange, but it's normal.
Now, the fact that the system appears unusable does obviously mean that something is wrong. But you're barking up the wrong tree.
Two more additional data points,
1) partially kernel-unrelated. MDK's "make" macro didn't support alpha's /proc/cpuinfo output, "make -j$numprocs" became "make -j" and fun ensued.
2) I agree that 200MB into swap and 200MB into cache isn't bad per se, but when it triggers the OOM killer it is bad.
Alan suggested that I insert the following into the OOM killer code, as the last test before returning 1.
cnt++; if ((cnt % 5000) != 0) return 0;
I did this, and while watching "vmstat 3", the cache was indeed being trimmed, whereas it was not before.
So, the OOM killer appears to be getting triggered early, but the rest of the report was my screwup not the kernel.
Signature · 3 lines
Jeff Garzik | A recent study has shown that too much soup Building 1024 | can cause malaise in laboratory mice. MandrakeSoft |
Quoted 10 lines · Jeff Garzik
Quoted 3 lines · Linus Torvalds
Now, the fact that the system appears unusable does obviously mean that something is wrong. But you're barking up the wrong tree.
Two more additional data points,
1) partially kernel-unrelated. MDK's "make" macro didn't support alpha's
/proc/cpuinfooutput, "make -j$numprocs" became "make -j" and fun ensued.
Ahh, well..
The kernel source code is set up to scale quite well, so yes a "make -j" will parallellise a bit teoo well for most machines, and you'll certainly run out of memory on just about anything (I routinely get load averages of 30+, and yes, you need at least half a GB of RAM for it to not be unpleasant - and probably more like a full gigabyte on an alpha).
So I definitely think the kernel likely did the right thing. It's not even clear that the OOM killer might not have been right - due to the 2.4.x swap space allocation, 256MB of swap-space is a bit tight on a 384MB machine that actually wants to use a lot of memory.
Quoted 2 lines
2) I agree that 200MB into swap and 200MB into cache isn't bad per se, but when it triggers the OOM killer it is bad.
Note that it might easily have been 256MB into swap (ie it had eaten _all_ of your swap) at some stage - and you just didn't see it in the vmstat output because obviously at that point the machine was a bit loaded.
But neutering the OOM killer like Alan suggested may be a rather valid approach anyway. Delaying the killing sounds valid: if we're truly livelocked on the VM, we'll be calling down to the OOM killer so much that it's probably quite valid to say "only return 1 after X iterations".
Linus
Quoted 13 lines · Linus Torvalds
Quoted 10 lines · Jeff Garzik
Quoted 3 lines · Linus Torvalds
Now, the fact that the system appears unusable does obviously mean that something is wrong. But you're barking up the wrong tree.
Two more additional data points,
1) partially kernel-unrelated. MDK's "make" macro didn't support alpha's
/proc/cpuinfooutput, "make -j$numprocs" became "make -j" and fun ensued.
Quoted 7 lines
Ahh, well..
The kernel source code is set up to scale quite well, so yes a "make -j" will parallellise a bit teoo well for most machines, and you'll certainly run out of memory on just about anything (I routinely get load averages of 30+, and yes, you need at least half a GB of RAM for it to not be unpleasant - and probably more like a full gigabyte on an alpha).
"make -j" is a lot of fun on a dual athlon w/ 512mb :)
Quoted 4 lines
So I definitely think the kernel likely did the right thing. It's not even clear that the OOM killer might not have been right - due to the 2.4.x swap space allocation, 256MB of swap-space is a bit tight on a 384MB machine that actually wants to use a lot of memory.
Sigh. since I am a VM ignoramus I doubt my opinion matters much at all here... but it would be nice if oddball configurations like 384MB with 50MB swap could be supported. I don't ask that it perform optimally at all, but at least the machine should behave predictably...
This type of swap configuration makes sense for, "my working set is pretty much always in RAM, including i/dcache, but let's have some swap just-in-case"
Quoted 6 lines
Quoted 2 lines
2) I agree that 200MB into swap and 200MB into cache isn't bad per se, but when it triggers the OOM killer it is bad.
Note that it might easily have been 256MB into swap (ie it had eaten _all_ of your swap) at some stage - and you just didn't see it in the vmstat output because obviously at that point the machine was a bit loaded.
I'm pretty sure swap was 100% full. I should have sysrq'd and checked but I forgot.
Quoted 4 lines
But neutering the OOM killer like Alan suggested may be a rather valid approach anyway. Delaying the killing sounds valid: if we're truly livelocked on the VM, we'll be calling down to the OOM killer so much that it's probably quite valid to say "only return 1 after X iterations".
cnt % 5000 may have been a bit extreme but it was fun to see it thrash. sysrq was pretty much the only talking point into the system.
Signature · 3 lines
Jeff Garzik | A recent study has shown that too much soup Building 1024 | can cause malaise in laboratory mice. MandrakeSoft |
Quoted 3 lines · Jeff Garzik
Sigh. since I am a VM ignoramus I doubt my opinion matters much at all here... but it would be nice if oddball configurations like 384MB with 50MB swap could be supported.
It would be fun if we had 48 hours in a day, too ;)
This particular thing has been on the TODO list of the VM developers for a while, but we just haven't gotten around to it.
regards,
Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"
http://www.surriel.com/
Quoted 4 lines
But neutering the OOM killer like Alan suggested may be a rather valid approach anyway. Delaying the killing sounds valid: if we're truly livelocked on the VM, we'll be calling down to the OOM killer so much that it's probably quite valid to say "only return 1 after X iterations".
Its hiding the real accounting screw up with a 'goes bang at random less often' - nice hack, but IMHO bad long term approach. We need to get the maths right. We had similar 2.2 problems the other way (with nasty deadlocks) until Andrea fixed that
Quoted 2 lines · Jeff Garzik
2) I agree that 200MB into swap and 200MB into cache isn't bad per se, but when it triggers the OOM killer it is bad.
Please read my patch for the OOM killer. It substracts the swap cache from the cache figure you quote and ONLY goes into oom_kill() if the page & buffer cache together take less than 4% of memory (see /proc/sys/vm/{buffermem,pagecache}).
regards,
Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"
http://www.surriel.com/
Quoted 5 lines
instead. That way the vmstat output might be more useful, although vmstat obviously won't know about the new "SwapCache:" field..
Can you try that, and see if something else stands out once the misleading accounting is taken care of?
Its certainly misleading. I got Jeff to try making oom return 4999 out of 5000 times regardless.
Quoted 8 lines · Alan Cox
Quoted 5 lines
instead. That way the vmstat output might be more useful, although vmstat obviously won't know about the new "SwapCache:" field..
Can you try that, and see if something else stands out once the misleading accounting is taken care of?
Its certainly misleading. I got Jeff to try making oom return 4999 out of 5000 times regardless.
In that case, he _is_ OOM. ;)
1) (almost) no free memory
2) no free swap
3) very little pagecache + buffer cache
regards,
Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"
http://www.surriel.com/
Quoted 9 lines · Rik van Riel
Quoted 6 lines · Alan Cox
Quoted 5 lines
instead. That way the vmstat output might be more useful, although vmstat obviously won't know about the new "SwapCache:" field..
Can you try that, and see if something else stands out once the misleading accounting is taken care of?
Quoted 8 lines
Quoted 2 lines
Its certainly misleading. I got Jeff to try making oom return 4999 out of 5000 times regardless.
In that case, he _is_ OOM. ;)
1) (almost) no free memory
2) no free swap
3) very little pagecache + buffer cache
It got -considerably- farther after Alan's suggested hack to the OOM killer; so at least in this instance, OOM killer appeared to me to be killing too early...
Signature · 3 lines
Jeff Garzik | A recent study has shown that too much soup Building 1024 | can cause malaise in laboratory mice. MandrakeSoft |
Quoted 4 lines
Quoted 2 lines
Its certainly misleading. I got Jeff to try making oom return 4999 out of 5000 times regardless.
In that case, he _is_ OOM. ;)
Hardly
Quoted 3 lines
1) (almost) no free memory
2) no free swap
3) very little pagecache + buffer cache
Large amounts of cache, which went away when the OOM code was neutered
Quoted 5 lines · Alan Cox
Quoted 4 lines
Quoted 2 lines
Its certainly misleading. I got Jeff to try making oom return 4999 out of 5000 times regardless.
In that case, he _is_ OOM. ;)
Quoted 7 lines
Hardly
Quoted 3 lines
1) (almost) no free memory
2) no free swap
3) very little pagecache + buffer cacheLarge amounts of cache, which went away when the OOM code was neutered
So Jeff backed out my patch before testing yours? ;)
Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"
http://www.surriel.com/