Swapping in 2.6.10 and 2.6.11.11 on a desktop system
Hi,
[1] With swappiness = 60 I either get swap hell (2.6.10) or the oom killer kicks in (2.6.11.11)
[2] I upgraded to 2.6.11.11 from 2.6.8.1 yesterday and tried to compile something. After some time I come back and the compile has aborted because the oom killer killed the compiler process. There is no additional use of swap space (although some applications that were also running could have been swapped out). There was a similar bugreport with this behaviour some time ago for 2.6.11.8 but that one included a swappiness value of 0, i got 60.
Then I tried 2.6.10. The oom bugs were gone, the compile finished over night. About an hour ago I started a very RAM consuming application and at 2/3 RAM usage I get swap hell. The system constantly swaps, yet ram usage _and_ swap usage stay about the same. Even setting swappiness to 0 from an ssh login (switching to a console does not work) did not help, only killing the consuming application.
I also applied the ck hard-swappiness Patch to my 2.6.10 and tested a bit further. Depending on the swappiness I get hellish swapping behaviour at different levels of RAM usage and even managed to get the oom killer to step in (at swappiness=20). I had about 250MB of RAM used and boom -> oom killer shoots down some bigger java processes. At swappiness=80 I get massive swapping with 250MB of RAM usage. When reaching that level, the fs cache goes up to 300MB while the applications that need the ram are left with about 120 megs. Rest was buffers.
I'm really out of ideas. 2.6.8.1 with swappiness=0 is by far the best I've managed to get by now. With that I can use my system until RAM and swap are completely full. No swap hell and no oom killer yet. Maybe I just have to search the lkml a bit more to find something usable. If someone else already has, please tell me (and tell me why it's not in the mainline kernel already :))
I have 512MB of RAM and another 512MB of swap.
Please cc me in replies as I'm not on the list.
Alex
Quoted 13 lines · Alexander Gretencord
Hi,
[1] With swappiness = 60 I either get swap hell (2.6.10) or the oom killer kicks in (2.6.11.11)
[2] I upgraded to 2.6.11.11 from 2.6.8.1 yesterday and tried to compile something. After some time I come back and the compile has aborted because the oom killer killed the compiler process. There is no additional use of swap space (although some applications that were also running could have been swapped out). There was a similar bugreport with this behaviour some time ago for 2.6.11.8 but that one included a swappiness value of 0, i got 60.
Quoted 1 line
I have 512MB of RAM and another 512MB of swap.
Try the mapped watermark patch from -ck on 2.6.11*
http://ck.kolivas.org/patches/2.6/2.6.11/2.6.11-ck10/patches/mapped_watermark3.diff
Cheers,
Con
Quoted 1 line · Con Kolivas
Try the mapped watermark patch from -ck on 2.6.11*
Unfortunately this patch does not help either. The patch buys me time but then I get swapping at the 300MB mark. 2.6.8.1 with swappiness=0 swaps later than this...
These are 'free -m' statistics a minute after reaching the 300MB mark:
total used free shared buffers cached
Mem: 503 498 5 0 36 376
-/+ buffers/cache: 85 418
Swap: 494 227 266
Why does the kernel think that I need 400MB of disk cache when I start some memory hungry apps? Can't we have a hard limit on disk cache, like: "Don't use more than 100MB of disk cache".
The problem seems to be that instead of using a big disk cache when theres plenty of ram and reducing disk cache when applications need the ram, the disk cache shrinks until a magic watermark and then grows and grows until theres no ram left for the applications. At least thats the behaviour I am seeing.
Alex
P.S.: Please cc me as I'm not on the list.
Quoted 6 lines · Alexander Gretencord
Quoted 1 line · Con Kolivas
Try the mapped watermark patch from -ck on 2.6.11*
Unfortunately this patch does not help either. The patch buys me time but then I get swapping at the 300MB mark. 2.6.8.1 with swappiness=0 swaps later than this...
Try tweaking it with this:
echo 100 > /proc/sys/vm/mapped
If this tries so hard to avoid swap that you get an out-of-memory condition you may also have to disable the hard maplimit with this:
echo 0 > /proc/sys/vm/hardmaplimit
Cheers,
Con
Quoted 1 line · you
echo 100 >
/proc/sys/vm/mapped
Doesn't work either.
What I do to test this is the following:
Start X with KDE, start a Konqueror instance, a firefox, eclipse and vmware. This increases memory load to about 250MB. Then I start os inside the vmware virtual machine. This increases memory allocation because the guest operating system has to exist somewhere and generates IOs while the virtual disk is accessed. With mapped=100 it takes longer until the system begins to swap but once it has begun swapping I get this:
total used free shared buffers cached
Mem: 503 498 5 0 10 412
-/+ buffers/cache: 75 428
Swap: 494 230 264
It does not matter which patch I am using, once the magical "begin swapping"-mark is reached, real ram usage drops to a bare minimum and cache usage goes up. The IO cache is probably not even used as there are virtually no reusable parts.
The problem is the kernel thinking in the wrong direction (freeing ram for disk cache). Maybe that's ok for a streaming server which uses 10MB of RAM for code and internal data and is constantly accessing the same 400MB of streaming video data. Or a webserver serving a big amount of static pages etc. but it is not ok for my desktop workload where I have about 400MB of data belonging to applications and IOs are not repeating very often (disk cache efficiency is probably very low).
Any idea why the kernel behaves like this and how I can get expected behaviour? Expected behaviour for a desktop would be:
Use as much disk cache as there is free RAM. Once applications start using all the ram, only keep a certain percentage/bare minimum of disk cache. What is absolutely undesirable is a growing and growing disk cache when there is not enough ram to keep applications and/or their data in ram.
Quoted 3 lines
If this tries so hard to avoid swap that you get an out-of-memory condition you may also have to disable the hard maplimit with this:
echo 0 >/proc/sys/vm/hardmaplimit
Yes I get oom conditions with mapped=100 but they are not my real problem, the problem is the disk cache usage pattern. I it would help, I still have some dmesg output from the oom killer.
Any ideas? Am I just doing something wrong? I don't use any special /proc-settings other than the swappiness/mapped test values.
Alex