OOM kills if swappiness set to 0, swap storms otherwise
I am simply trying to run a Gnome desktop (Gnome 2.14, Evolution, Firefox, gtk-gnutella usually open) without swapping or getting OOM killed.
I have to set the swappiness to 0 or else I get swap storms when simply browsing the web and reading my mail. I think this is insane as I have 512MB of RAM. It seems as if the kernel will OOM kill firefox rather than shrink the file cache!
What is the problem here? Is the modern Linux desktop really too bloated to run in half a gig of RAM, or is the kernel overzealous with its OOM killing?
Lee
oom-killer: gfp_mask=0x280d2, order=0
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu: empty
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:23
cpu 0 cold: high 62, batch 15 used:46
HighMem per-cpu: empty
Free pages: 4476kB (0kB HighMem)
Active:84715 inactive:9418 dirty:0 writeback:0 unstable:0 free:1119 slab:4615 mapped:93948 pagetables:662
DMA free:1252kB min:96kB low:120kB high:144kB active:0kB inactive:0kB present:16384kB pages_scanned:10238 all_unreclaimable? yes lowmem_reserve[]: 0 0 431 431
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 431 431
Normal free:3224kB min:2608kB low:3260kB high:3912kB active:338860kB inactive:37672kB present:442304kB pages_scanned:61895 all_unreclaim able? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
DMA: 3*4kB 3*8kB 2*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1252kB
DMA32: empty
Normal: 174*4kB 28*8kB 0*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3224kB
HighMem: empty
Swap cache: add 635780, delete 617326, find 164667/241301, race 0+0 Free swap = 315164kB
Total swap = 499928kB
Free swap: 315164kB
114672 pages of RAM
0 pages of HIGHMEM
5150 reserved pages
41086 pages shared
18438 pages swap cached
0 pages dirty
0 pages writeback
93765 pages mapped
4615 pages slab
662 pages pagetables
Out of Memory: Killed process 32663 (firefox-bin).
rlrevell@mindpipe:~$ vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 248992 5612 4072 75268 3 2 41 49 12 33 82 4 14 1 0 0 248992 5364 4084 75300 0 0 0 128 1273 640 34 3 63 0 0 0 248992 5356 4084 75464 0 0 0 0 1257 611 23 4 73 0 0 0 248992 5356 4084 75496 0 0 0 0 1292 635 21 2 77 0 0 0 248992 5356 4084 75532 0 0 0 0 1277 599 22 3 75 0 0 0 248992 5356 4084 75532 0 0 0 0 1270 610 23 2 75 0 1 0 248992 5232 4092 75564 0 0 0 200 1274 596 21 2 77 0 0 0 248992 5232 4092 75564 0 0 0 0 1280 648 22 2 76 0 0 0 248992 5108 4092 75696 0 0 0 0 1296 621 21 4 75 0
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND rlrevell 1792 4.1 37.9 320340 166724 ? Sl Mar21 353:33 evolution --component=mail rlrevell 2298 9.2 18.9 200556 83112 ? Sl 15:16 30:55 /usr/lib/firefox/firefox-bin -a firefox rlrevell 1794 0.0 2.1 92704 9344 ? Sl Mar21 0:47 /usr/lib/evolution/evolution-data-server-1.6 --oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_InterfaceCheck --oaf-ior-fd=41 pdnsd 10545 0.0 0.1 75676 600 ? Sl 18:55 0:00 /usr/sbin/pdnsd --daemon -p /var/run/pdnsd.pid rlrevell 1805 0.0 0.6 65236 2740 ? Sl Mar21 0:06 /usr/lib/evolution/2.6/evolution-alarm-notify --oaf-activate-iid=OAFIID:GNOME_Evolution_Calendar_AlarmNotify_Factory:2.6 --oaf-ior-fd=43 root 2295 70.3 4.5 58160 19908 tty7 Ss+ Mar18 9207:34 /usr/bin/X :0 -br -audit 0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt7 rlrevell 2759 0.0 1.0 57716 4832 ? Ssl Mar18 2:48 nautilus --sm-config-prefix /nautilus-kFkhxG/ --sm-client-id 106281f446000113283283500000043730002 --screen 0 --no-default-window rlrevell 2749 0.0 0.5 33720 2404 ? Sl Mar18 3:09 /usr/lib/control-center/gnome-settings-daemon --oaf-activate-iid=OAFIID:GNOME_SettingsDaemon --oaf-ior-fd=25 rlrevell 11452 17.2 3.9 31828 17348 ? S 20:15 5:49 gtk-gnutella rlrevell 2934 0.1 1.7 28736 7548 ? Sl Mar18 20:09 /usr/lib/gnome-panel/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oaf-ior-fd=33 rlrevell 11751 0.0 0.1 27180 524 pts/0 S+ 20:49 0:00 sort -k5 -rn rlrevell 2757 0.0 1.8 24220 8148 ? Ssl Mar18 5:54 gnome-panel --sm-config-prefix /gnome-panel-daqxPK/ --sm-client-id 106281f446000113283283400000043730001 --screen 0 rlrevell 2785 0.0 0.7 20784 3400 ? S Mar18 1:10 /usr/lib/gnome-panel/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf-ior-fd=35 rlrevell 2669 0.0 0.6 17212 2704 ? Ss Mar18 0:22 x-session-manager rlrevell 2783 0.0 0.5 15288 2328 ? S Mar18 0:17 /usr/lib/gnome-panel/notification-area-applet --oaf-activate-iid=OAFIID:GNOME_NotificationAreaApplet_Factory --oaf-ior-fd=34 rlrevell 11453 0.0 0.1 14088 480 ? S 20:15 0:00 DNS helper for gtk-gnutella rlrevell 2860 0.0 0.5 13732 2512 ? Ss Mar18 2:29 gnome-screensaver rlrevell 2755 0.1 1.1 13484 5216 ? Ss Mar18 13:17 metacity --sm-save-file 1132974963-4486-1014928945.ms root 2291 0.0 0.0 10148 388 ? S Mar18 0:04 /usr/sbin/gdm root 2286 0.0 0.0 9668 256 ? Ss Mar18 0:00 /usr/sbin/gdm
Quoted 9 lines · Lee Revell
I am simply trying to run a Gnome desktop (Gnome 2.14, Evolution, Firefox, gtk-gnutella usually open) without swapping or getting OOM killed.
I have to set the swappiness to 0 or else I get swap storms when simply browsing the web and reading my mail. I think this is insane as I have 512MB of RAM. It seems as if the kernel will OOM kill firefox rather than shrink the file cache!
No, there is little pagecache left.
Quoted 3 lines
What is the problem here? Is the modern Linux desktop really too bloated to run in half a gig of RAM, or is the kernel overzealous with its OOM killing?
Looks like it.
Quoted 14 lines
...
rlrevell@mindpipe:~$ vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 248992 5612 4072 75268 3 2 41 49 12 33 82 4 14 1 0 0 248992 5364 4084 75300 0 0 0 128 1273 640 34 3 63 0 0 0 248992 5356 4084 75464 0 0 0 0 1257 611 23 4 73 0 0 0 248992 5356 4084 75496 0 0 0 0 1292 635 21 2 77 0 0 0 248992 5356 4084 75532 0 0 0 0 1277 599 22 3 75 0 0 0 248992 5356 4084 75532 0 0 0 0 1270 610 23 2 75 0 1 0 248992 5232 4092 75564 0 0 0 200 1274 596 21 2 77 0 0 0 248992 5232 4092 75564 0 0 0 0 1280 648 22 2 76 0 0 0 248992 5108 4092 75696 0 0 0 0 1296 621 21 4 75 0
70-odd MB in pagecache. Probably a lot of this is mmapped into process pagetables and by setting swappiness==0 you've basically told the kernel that this memory is to be preserved.
Quoted 7 lines
rlrevell 1792 4.1 37.9 320340 166724 ? Sl Mar21 353:33 evolution --component=mail rlrevell 2298 9.2 18.9 200556 83112 ? Sl 15:16 30:55 /usr/lib/firefox/firefox-bin -a firefox rlrevell 1794 0.0 2.1 92704 9344 ? Sl Mar21 0:47 /usr/lib/evolution/evolution-data-server-1.6 --oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_InterfaceCheck --oaf-ior-fd=41 rlrevell 1805 0.0 0.6 65236 2740 ? Sl Mar21 0:06 /usr/lib/evolution/2.6/evolution-alarm-notify --oaf-activate-iid=OAFIID:GNOME_Evolution_Calendar_AlarmNotify_Factory:2.6 --oaf-ior-fd=43 root 2295 70.3 4.5 58160 19908 tty7 Ss+ Mar18 9207:34 /usr/bin/X :0 -br -audit 0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt7 rlrevell 2759 0.0 1.0 57716 4832 ? Ssl Mar18 2:48 nautilus --sm-config-prefix /nautilus-kFkhxG/ --sm-client-id 106281f446000113283283500000043730002 --screen 0 --no-default-window rlrevell 2749 0.0 0.5 33720 2404 ? Sl Mar18 3:09 /usr/lib/control-center/gnome-settings-daemon --oaf-activate-iid=OAFIID:GNOME_SettingsDaemon --oaf-ior-fd=25
Much porkiness.
/proc/meminfo is very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.
Quoted 6 lines · Andrew Morton
Much porkiness.
/proc/meminfois very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.
Thanks. It seems the problem is Evo (or possibly firefox) has a slow memory leak. I overlooked the fact that Evo had been running for more than a week.
I guess the only possible kernel issue is that Evo never gets OOM-killed, always Firefox (or OpenOffice if it's running), although it's the biggest hog. I'll have to investigate more.
Lee
Quoted 2 lines · Andrew Morton
I'd tentatively say that your options are to put up with the swapping or find a new mail client.
I tried mem=512m , swapoff -a and swappiness=0 and all works fine for me on x86. I use Evolution and Firefox's KDE counterparts - in addition to that Amarok, Kopete and Akregator fit in too.
Lee - Perhaps you are running fat 64 bit code?
Parag
Quoted 1 line · Parag Warudkar
Lee - Perhaps you are running fat 64 bit code?
Nope, 32 bit system. I think the problem is just a boring old memory leak.
Lee
Quoted 5 lines · Andrew Morton
Much porkiness.
/proc/meminfois very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.
I use mutt for my email, and I have the same issue on a 1GB system.
I really wish we could put an upper limit on what file cache can use.
I understand the original poster was running a lot of pork, but you don't have to and still see a problem with swapping. Even running KDE my total application memory most of the time is 300MB or less on a machine with 1GB of memory.
I shouldn't be suffering from swap storms.
For example, my normal working set of programs eats about 250MB of memory. If I also start a job running to something like tag some mp3s, copy a CD, or just process a lot of files, it only takes a few minutes before performance becomes unacceptable.
If you are doing some work where you switch among several applications frequently, the pigginess of file cache becomes a serious problem.
Isn't that bad behavior by any measure?
Signature · 2 lines
shannon "AT" widomaker.com -- ["It's a damn poor mind that can only think of one way to spell a word." -- Andrew Jackson]
Quoted 10 lines · Charles Shannon Hendrix
Quoted 5 lines · Andrew Morton
Much porkiness.
/proc/meminfois very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.I use mutt for my email, and I have the same issue on a 1GB system.
Quoted 8 lines
I really wish we could put an upper limit on what file cache can use.
I understand the original poster was running a lot of pork, but you don't have to and still see a problem with swapping. Even running KDE my total application memory most of the time is 300MB or less on a machine with 1GB of memory.
I shouldn't be suffering from swap storms.
Agreed, does meminfo show that you are? The reason I ask is that I have noted that large memory machines and CD/DVD image writing suffer from some interesting disk write patterns. The image being built gets cached but not written, then the file is closed. At some point the kernel notices several GB of old unwritten data and decides to write it. This makes everything pretty slow for a while, even if you have 100MB/s disk system.
Quoted 5 lines
For example, my normal working set of programs eats about 250MB of memory. If I also start a job running to something like tag some mp3s, copy a CD, or just process a lot of files, it only takes a few minutes before performance becomes unacceptable.
In theory you should be able to tune this, but in practice I see what you do. On small memory machines it's less noticable, oddly.
Quoted 5 lines
If you are doing some work where you switch among several applications frequently, the pigginess of file cache becomes a serious problem.
Isn't that bad behavior by any measure?
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Wed, 05 Apr 2006 @ 16:47 -0400, Bill Davidsen said:
Sorry for the late reply, and if this is a duplicate.
Quoted 3 lines
Quoted 1 line
I shouldn't be suffering from swap storms.
Agreed, does meminfo show that you are?
meminfo?
procinfo and other tools show a lot swapping, if that's what you mean, and you can see it in disk I/O to the swap drives as well.
Quoted 6 lines
The reason I ask is that I have noted that large memory machines and CD/DVD image writing suffer from some interesting disk write patterns. The image being built gets cached but not written, then the file is closed. At some point the kernel notices several GB of old unwritten data and decides to write it. This makes everything pretty slow for a while, even if you have 100MB/s disk system.
I see that kind of behavior quite a lot. Not just for DVD/CD images either. Basically any large data processing fills memory with cached file data at the expense of other programs and data.
Quoted 2 lines
In theory you should be able to tune this, but in practice I see what you do. On small memory machines it's less noticable, oddly.
I tried putting swapiness down to 30. It helped, most of the time, but still I saw way too much useless file data being cached.
I would personally rather just limit how much file data can be cached. I don't mind agressive swapping, I just hate seeing a ton of file data being cached that isn't going to be used again.
I'm also trying the ck kernels just to see how they run. So far they work better.
Signature · 2 lines
shannon "AT" widomaker.com -- ["All of us get lost in the darkness, dreamers turn to look at the stars" -- Rush ]
Quoted 5 lines · Charles Shannon Hendrix
Quoted 2 lines · Andrew Morton
Much porkiness.
Quoted 9 lines
Quoted 4 lines
/proc/meminfois very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.I use mutt for my email, and I have the same issue on a 1GB system.
I really wish we could put an upper limit on what file cache can use.
----
Hmmm, not to be contrary, but I have a 1GB system that refuses to swap
during large file i/o operations. For the first time in a *long* time, I read someone's suggestion to increase swappiness -- I did, to 75 or 80, (I've booted since then, so it's back to 60 and no swap usage) and some of the programs that rarely run actually swapped. It was great! I finally had more memory for file i/o operations.
Maybe you are telling the system to "feel free" to use swap by having a
large swap file? I have a 256Mb swap partition near the front of the disk where I/O is fastest, but it usually remains untouched, but that also means I can only overcommit used memory by 25% (1Gphys, 256M swap). I don't know what the OOM "looks like" when it runs. I had a buggy perlapp, once, that tried to consume infinite memory, but I think it segfaulted before the OOM could act.
Maybe if you don't want linux to swap as much, it might swap less if
you give
it less to swap with? I know it sounds simplistic, but I've run with smallish swap files ever since I got up to 1GB of main memory. The only reason I have one at all is I presumed that infrequently used functions, like dhcp (only used when friends are over) would be shuffled out -- but it isn't unless I increase swappiness.
Quoted 2 lines
I shouldn't be suffering from swap storms.
I agree. Try getting rid of your swap file entirely -- your system will still run unless you are overloading memory, but you have a Gig. How much do you need to keep in memory? Sure, if/when I get a 4-way CPU (I have a 2-cpu setup now), I might go up to 4G, but I might be running multiple virtual machines too!
Quoted 5 lines
For example, my normal working set of programs eats about 250MB of memory. If I also start a job running to something like tag some mp3s, copy a CD, or just process a lot of files, it only takes a few minutes before performance becomes unacceptable.
---
You might try the "cfq" block i/o algorithm. Then you can
ionice down the disk priority of background processes (though you need to be root to reduce ionice levels at this point, unlike cpu nice).
Quoted 3 lines
If you are doing some work where you switch among several applications frequently, the pigginess of file cache becomes a serious problem.
---
Never a problem. I don't allocate enough swap for it to be a
problem I'm
guessing. Programs stay in memory and blocks get forced out to disk more frequently.
Linda
Quoted 6 lines · Linda Walsh
Hmmm, not to be contrary, but I have a 1GB system that refuses to swap
during large file i/o operations. For the first time in a *long* time, I read someone's suggestion to increase swappiness -- I did, to 75 or 80, (I've booted since then, so it's back to 60 and no swap usage) and some of the programs that rarely run actually swapped. It was great! I finally had more memory for file i/o operations.
It's great if you actually need the file data that gets stored.
Quoted 2 lines
Maybe you are telling the system to "feel free" to use swap by having a
large swap file?
I don't believe that matters, and certainly doesn't seem to affect my own system.
If I use a smaller swap file, I just run out faster.
Is your experience different?
Quoted 5 lines
I agree. Try getting rid of your swap file entirely -- your system will still run unless you are overloading memory, but you have a Gig. How much do you need to keep in memory? Sure, if/when I get a 4-way CPU (I have a 2-cpu setup now), I might go up to 4G, but I might be running multiple virtual machines too!
Sure it will run, but I *want* swap to be used to remove unused programs.
My current problem is that *useful* program code is being swapped out and being replaced by *useless* cached file data.
Quoted 3 lines
You might try the "cfq" block i/o algorithm. Then you can
ionice down the disk priority of background processes (though you need to be root to reduce ionice levels at this point, unlike cpu nice).
I've not seen ionice.
Signature · 2 lines
shannon "AT" widomaker.com -- ["All of us get lost in the darkness, dreamers turn to look at the stars" -- Rush ]
Quoted 30 lines · On Tue, 2 May 2006 00:21:08 -0400 Charles Shannon Hendrix wrote:
Quoted 6 lines · Linda Walsh
Hmmm, not to be contrary, but I have a 1GB system that refuses to swap
during large file i/o operations. For the first time in a *long* time, I read someone's suggestion to increase swappiness -- I did, to 75 or 80, (I've booted since then, so it's back to 60 and no swap usage) and some of the programs that rarely run actually swapped. It was great! I finally had more memory for file i/o operations.It's great if you actually need the file data that gets stored.
Quoted 2 lines
Maybe you are telling the system to "feel free" to use swap by having a
large swap file?I don't believe that matters, and certainly doesn't seem to affect my own system.
If I use a smaller swap file, I just run out faster.
Is your experience different?
Quoted 5 lines
I agree. Try getting rid of your swap file entirely -- your system will still run unless you are overloading memory, but you have a Gig. How much do you need to keep in memory? Sure, if/when I get a 4-way CPU (I have a 2-cpu setup now), I might go up to 4G, but I might be running multiple virtual machines too!
Sure it will run, but I *want* swap to be used to remove unused programs.
Quoted 8 lines
My current problem is that *useful* program code is being swapped out and being replaced by *useless* cached file data.
Quoted 3 lines
You might try the "cfq" block i/o algorithm. Then you can
ionice down the disk priority of background processes (though you need to be root to reduce ionice levels at this point, unlike cpu nice).I've not seen ionice.
It's hidden^w embedded in Documentation/block/ioprio.txt ...
---
~Randy
Quoted 3 lines · Lee Revell
What is the problem here? Is the modern Linux desktop really too bloated to run in half a gig of RAM, or is the kernel overzealous with its OOM killing?
Evolution will regularly try and eat over 1GByte of RAM. In addition it is possible for an attacker to send you a plain text email that makes it do this.
If you want to run gnome & firefox & evolution you need a lot more RAM. Alternatively run something sane and sensible.
Alan
(whose todo list does indeed have 'new mailer' on it)