Skip to contentSkip to conversation

OOM kills if swappiness set to 0, swap storms otherwise

DISCUSSION Lee Revell 8 participants 12 messages Mar 28, 2006 – May 2, 2006

lore
Share & tools
lore

Lee Revell

author
#0 lore raw

I am simply trying to run a Gnome desktop (Gnome 2.14, Evolution, Firefox, gtk-gnutella usually open) without swapping or getting OOM killed.

I have to set the swappiness to 0 or else I get swap storms when simply browsing the web and reading my mail. I think this is insane as I have 512MB of RAM. It seems as if the kernel will OOM kill firefox rather than shrink the file cache!

What is the problem here? Is the modern Linux desktop really too bloated to run in half a gig of RAM, or is the kernel overzealous with its OOM killing?

Lee

oom-killer: gfp_mask=0x280d2, order=0
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu: empty
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:23
cpu 0 cold: high 62, batch 15 used:46
HighMem per-cpu: empty
Free pages: 4476kB (0kB HighMem)
Active:84715 inactive:9418 dirty:0 writeback:0 unstable:0 free:1119 slab:4615 mapped:93948 pagetables:662
DMA free:1252kB min:96kB low:120kB high:144kB active:0kB inactive:0kB present:16384kB pages_scanned:10238 all_unreclaimable? yes lowmem_reserve[]: 0 0 431 431
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 431 431
Normal free:3224kB min:2608kB low:3260kB high:3912kB active:338860kB inactive:37672kB present:442304kB pages_scanned:61895 all_unreclaim able? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
DMA: 3*4kB 3*8kB 2*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1252kB
DMA32: empty
Normal: 174*4kB 28*8kB 0*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3224kB
HighMem: empty
Swap cache: add 635780, delete 617326, find 164667/241301, race 0+0 Free swap = 315164kB
Total swap = 499928kB
Free swap: 315164kB
114672 pages of RAM
0 pages of HIGHMEM
5150 reserved pages
41086 pages shared
18438 pages swap cached
0 pages dirty
0 pages writeback
93765 pages mapped
4615 pages slab
662 pages pagetables
Out of Memory: Killed process 32663 (firefox-bin).

rlrevell@mindpipe:~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0 248992   5612   4072  75268    3    2    41    49   12    33 82  4 14  1
 0  0 248992   5364   4084  75300    0    0     0   128 1273   640 34  3 63  0
 0  0 248992   5356   4084  75464    0    0     0     0 1257   611 23  4 73  0
 0  0 248992   5356   4084  75496    0    0     0     0 1292   635 21  2 77  0
 0  0 248992   5356   4084  75532    0    0     0     0 1277   599 22  3 75  0
 0  0 248992   5356   4084  75532    0    0     0     0 1270   610 23  2 75  0
 1  0 248992   5232   4092  75564    0    0     0   200 1274   596 21  2 77  0
 0  0 248992   5232   4092  75564    0    0     0     0 1280   648 22  2 76  0
 0  0 248992   5108   4092  75696    0    0     0     0 1296   621 21  4 75  0
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
rlrevell  1792  4.1 37.9 320340 166724 ?       Sl   Mar21 353:33 evolution --component=mail
rlrevell  2298  9.2 18.9 200556 83112 ?        Sl   15:16  30:55 /usr/lib/firefox/firefox-bin -a firefox
rlrevell  1794  0.0  2.1  92704  9344 ?        Sl   Mar21   0:47 /usr/lib/evolution/evolution-data-server-1.6 --oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_InterfaceCheck --oaf-ior-fd=41
pdnsd    10545  0.0  0.1  75676   600 ?        Sl   18:55   0:00 /usr/sbin/pdnsd --daemon -p /var/run/pdnsd.pid
rlrevell  1805  0.0  0.6  65236  2740 ?        Sl   Mar21   0:06 /usr/lib/evolution/2.6/evolution-alarm-notify --oaf-activate-iid=OAFIID:GNOME_Evolution_Calendar_AlarmNotify_Factory:2.6 --oaf-ior-fd=43
root      2295 70.3  4.5  58160 19908 tty7     Ss+  Mar18 9207:34 /usr/bin/X :0 -br -audit 0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt7
rlrevell  2759  0.0  1.0  57716  4832 ?        Ssl  Mar18   2:48 nautilus --sm-config-prefix /nautilus-kFkhxG/ --sm-client-id 106281f446000113283283500000043730002 --screen 0 --no-default-window
rlrevell  2749  0.0  0.5  33720  2404 ?        Sl   Mar18   3:09 /usr/lib/control-center/gnome-settings-daemon --oaf-activate-iid=OAFIID:GNOME_SettingsDaemon --oaf-ior-fd=25
rlrevell 11452 17.2  3.9  31828 17348 ?        S    20:15   5:49 gtk-gnutella
rlrevell  2934  0.1  1.7  28736  7548 ?        Sl   Mar18  20:09 /usr/lib/gnome-panel/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oaf-ior-fd=33
rlrevell 11751  0.0  0.1  27180   524 pts/0    S+   20:49   0:00 sort -k5 -rn
rlrevell  2757  0.0  1.8  24220  8148 ?        Ssl  Mar18   5:54 gnome-panel --sm-config-prefix /gnome-panel-daqxPK/ --sm-client-id 106281f446000113283283400000043730001 --screen 0
rlrevell  2785  0.0  0.7  20784  3400 ?        S    Mar18   1:10 /usr/lib/gnome-panel/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf-ior-fd=35
rlrevell  2669  0.0  0.6  17212  2704 ?        Ss   Mar18   0:22 x-session-manager
rlrevell  2783  0.0  0.5  15288  2328 ?        S    Mar18   0:17 /usr/lib/gnome-panel/notification-area-applet --oaf-activate-iid=OAFIID:GNOME_NotificationAreaApplet_Factory --oaf-ior-fd=34
rlrevell 11453  0.0  0.1  14088   480 ?        S    20:15   0:00 DNS helper for gtk-gnutella
rlrevell  2860  0.0  0.5  13732  2512 ?        Ss   Mar18   2:29 gnome-screensaver
rlrevell  2755  0.1  1.1  13484  5216 ?        Ss   Mar18  13:17 metacity --sm-save-file 1132974963-4486-1014928945.ms
root      2291  0.0  0.0  10148   388 ?        S    Mar18   0:04 /usr/sbin/gdm
root      2286  0.0  0.0   9668   256 ?        Ss   Mar18   0:00 /usr/sbin/gdm
#1 lore raw
Quoted 9 lines · Lee Revell

I am simply trying to run a Gnome desktop (Gnome 2.14, Evolution, Firefox, gtk-gnutella usually open) without swapping or getting OOM killed.

I have to set the swappiness to 0 or else I get swap storms when simply browsing the web and reading my mail. I think this is insane as I have 512MB of RAM. It seems as if the kernel will OOM kill firefox rather than shrink the file cache!

No, there is little pagecache left.

Quoted 3 lines

What is the problem here? Is the modern Linux desktop really too bloated to run in half a gig of RAM, or is the kernel overzealous with its OOM killing?

Looks like it.

Quoted 14 lines

...

rlrevell@mindpipe:~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0 248992   5612   4072  75268    3    2    41    49   12    33 82  4 14  1
 0  0 248992   5364   4084  75300    0    0     0   128 1273   640 34  3 63  0
 0  0 248992   5356   4084  75464    0    0     0     0 1257   611 23  4 73  0
 0  0 248992   5356   4084  75496    0    0     0     0 1292   635 21  2 77  0
 0  0 248992   5356   4084  75532    0    0     0     0 1277   599 22  3 75  0
 0  0 248992   5356   4084  75532    0    0     0     0 1270   610 23  2 75  0
 1  0 248992   5232   4092  75564    0    0     0   200 1274   596 21  2 77  0
 0  0 248992   5232   4092  75564    0    0     0     0 1280   648 22  2 76  0
 0  0 248992   5108   4092  75696    0    0     0     0 1296   621 21  4 75  0

70-odd MB in pagecache. Probably a lot of this is mmapped into process pagetables and by setting swappiness==0 you've basically told the kernel that this memory is to be preserved.

Quoted 7 lines
rlrevell  1792  4.1 37.9 320340 166724 ?       Sl   Mar21 353:33 evolution --component=mail
rlrevell  2298  9.2 18.9 200556 83112 ?        Sl   15:16  30:55 /usr/lib/firefox/firefox-bin -a firefox
rlrevell  1794  0.0  2.1  92704  9344 ?        Sl   Mar21   0:47 /usr/lib/evolution/evolution-data-server-1.6 --oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_InterfaceCheck --oaf-ior-fd=41
rlrevell  1805  0.0  0.6  65236  2740 ?        Sl   Mar21   0:06 /usr/lib/evolution/2.6/evolution-alarm-notify --oaf-activate-iid=OAFIID:GNOME_Evolution_Calendar_AlarmNotify_Factory:2.6 --oaf-ior-fd=43
root      2295 70.3  4.5  58160 19908 tty7     Ss+  Mar18 9207:34 /usr/bin/X :0 -br -audit 0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt7
rlrevell  2759  0.0  1.0  57716  4832 ?        Ssl  Mar18   2:48 nautilus --sm-config-prefix /nautilus-kFkhxG/ --sm-client-id 106281f446000113283283500000043730002 --screen 0 --no-default-window
rlrevell  2749  0.0  0.5  33720  2404 ?        Sl   Mar18   3:09 /usr/lib/control-center/gnome-settings-daemon --oaf-activate-iid=OAFIID:GNOME_SettingsDaemon --oaf-ior-fd=25

Much porkiness.

/proc/meminfo is very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.

author
#2 lore raw
Quoted 6 lines · Andrew Morton

Much porkiness.

/proc/meminfo is very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.

Thanks. It seems the problem is Evo (or possibly firefox) has a slow memory leak. I overlooked the fact that Evo had been running for more than a week.

I guess the only possible kernel issue is that Evo never gets OOM-killed, always Firefox (or OpenOffice if it's running), although it's the biggest hog. I'll have to investigate more.

Lee

#3 lore raw
In reply to Andrew Morton
Quoted 2 lines · Andrew Morton

I'd tentatively say that your options are to put up with the swapping or find a new mail client.

I tried mem=512m , swapoff -a and swappiness=0 and all works fine for me on x86. I use Evolution and Firefox's KDE counterparts - in addition to that Amarok, Kopete and Akregator fit in too.

Lee - Perhaps you are running fat 64 bit code?

Parag

author
#4 lore raw
Quoted 1 line · Parag Warudkar

Lee - Perhaps you are running fat 64 bit code?

Nope, 32 bit system. I think the problem is just a boring old memory leak.

Lee

#11 lore raw
In reply to Lee Revell
Quoted 3 lines · Lee Revell

What is the problem here? Is the modern Linux desktop really too bloated to run in half a gig of RAM, or is the kernel overzealous with its OOM killing?

Evolution will regularly try and eat over 1GByte of RAM. In addition it is possible for an attacker to send you a plain text email that makes it do this.

If you want to run gnome & firefox & evolution you need a lot more RAM. Alternatively run something sane and sensible.

Alan
(whose todo list does indeed have 'new mailer' on it)

#5 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton

Much porkiness.

/proc/meminfo is very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.

I use mutt for my email, and I have the same issue on a 1GB system.

I really wish we could put an upper limit on what file cache can use.

I understand the original poster was running a lot of pork, but you don't have to and still see a problem with swapping. Even running KDE my total application memory most of the time is 300MB or less on a machine with 1GB of memory.

I shouldn't be suffering from swap storms.

For example, my normal working set of programs eats about 250MB of memory. If I also start a job running to something like tag some mp3s, copy a CD, or just process a lot of files, it only takes a few minutes before performance becomes unacceptable.

If you are doing some work where you switch among several applications frequently, the pigginess of file cache becomes a serious problem.

Isn't that bad behavior by any measure?

Signature · 2 lines
shannon "AT" widomaker.com -- ["It's a damn poor mind that can only think
of one way to spell a word." -- Andrew Jackson]
#6 lore raw
Quoted 10 lines · Charles Shannon Hendrix
Quoted 5 lines · Andrew Morton

Much porkiness.

/proc/meminfo is very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.

I use mutt for my email, and I have the same issue on a 1GB system.

Quoted 8 lines

I really wish we could put an upper limit on what file cache can use.

I understand the original poster was running a lot of pork, but you don't have to and still see a problem with swapping. Even running KDE my total application memory most of the time is 300MB or less on a machine with 1GB of memory.

I shouldn't be suffering from swap storms.

Agreed, does meminfo show that you are? The reason I ask is that I have noted that large memory machines and CD/DVD image writing suffer from some interesting disk write patterns. The image being built gets cached but not written, then the file is closed. At some point the kernel notices several GB of old unwritten data and decides to write it. This makes everything pretty slow for a while, even if you have 100MB/s disk system.

Quoted 5 lines

For example, my normal working set of programs eats about 250MB of memory. If I also start a job running to something like tag some mp3s, copy a CD, or just process a lot of files, it only takes a few minutes before performance becomes unacceptable.

In theory you should be able to tune this, but in practice I see what you do. On small memory machines it's less noticable, oddly.

Quoted 5 lines

If you are doing some work where you switch among several applications frequently, the pigginess of file cache becomes a serious problem.

Isn't that bad behavior by any measure?

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#8 lore raw
In reply to Charles Shannon Hendrix
Quoted 5 lines · Charles Shannon Hendrix
Quoted 2 lines · Andrew Morton

Much porkiness.

Quoted 9 lines
Quoted 4 lines

/proc/meminfo is very useful for obtaining a top-level view of where all the memory's gone to. I'd tentatively say that your options are to put up with the swapping or find a new mail client.

I use mutt for my email, and I have the same issue on a 1GB system.

I really wish we could put an upper limit on what file cache can use.

----
Hmmm, not to be contrary, but I have a 1GB system that refuses to swap
during large file i/o operations. For the first time in a *long* time, I read someone's suggestion to increase swappiness -- I did, to 75 or 80, (I've booted since then, so it's back to 60 and no swap usage) and some of the programs that rarely run actually swapped. It was great! I finally had more memory for file i/o operations.

Maybe you are telling the system to "feel free" to use swap by having a
large swap file? I have a 256Mb swap partition near the front of the disk where I/O is fastest, but it usually remains untouched, but that also means I can only overcommit used memory by 25% (1Gphys, 256M swap). I don't know what the OOM "looks like" when it runs. I had a buggy perlapp, once, that tried to consume infinite memory, but I think it segfaulted before the OOM could act.

Maybe if you don't want linux to swap as much, it might swap less if
you give
it less to swap with? I know it sounds simplistic, but I've run with smallish swap files ever since I got up to 1GB of main memory. The only reason I have one at all is I presumed that infrequently used functions, like dhcp (only used when friends are over) would be shuffled out -- but it isn't unless I increase swappiness.

Quoted 2 lines

I shouldn't be suffering from swap storms.

I agree. Try getting rid of your swap file entirely -- your system will still run unless you are overloading memory, but you have a Gig. How much do you need to keep in memory? Sure, if/when I get a 4-way CPU (I have a 2-cpu setup now), I might go up to 4G, but I might be running multiple virtual machines too!

Quoted 5 lines

For example, my normal working set of programs eats about 250MB of memory. If I also start a job running to something like tag some mp3s, copy a CD, or just process a lot of files, it only takes a few minutes before performance becomes unacceptable.

---
You might try the "cfq" block i/o algorithm. Then you can
ionice down the disk priority of background processes (though you need to be root to reduce ionice levels at this point, unlike cpu nice).

Quoted 3 lines

If you are doing some work where you switch among several applications frequently, the pigginess of file cache becomes a serious problem.

---
Never a problem. I don't allocate enough swap for it to be a
problem I'm
guessing. Programs stay in memory and blocks get forced out to disk more frequently.

Linda

#7 lore raw
In reply to Bill Davidsen

Wed, 05 Apr 2006 @ 16:47 -0400, Bill Davidsen said:

Sorry for the late reply, and if this is a duplicate.

Quoted 3 lines
Quoted 1 line

I shouldn't be suffering from swap storms.

Agreed, does meminfo show that you are?

meminfo?

procinfo and other tools show a lot swapping, if that's what you mean, and you can see it in disk I/O to the swap drives as well.

Quoted 6 lines

The reason I ask is that I have noted that large memory machines and CD/DVD image writing suffer from some interesting disk write patterns. The image being built gets cached but not written, then the file is closed. At some point the kernel notices several GB of old unwritten data and decides to write it. This makes everything pretty slow for a while, even if you have 100MB/s disk system.

I see that kind of behavior quite a lot. Not just for DVD/CD images either. Basically any large data processing fills memory with cached file data at the expense of other programs and data.

Quoted 2 lines

In theory you should be able to tune this, but in practice I see what you do. On small memory machines it's less noticable, oddly.

I tried putting swapiness down to 30. It helped, most of the time, but still I saw way too much useless file data being cached.

I would personally rather just limit how much file data can be cached. I don't mind agressive swapping, I just hate seeing a ton of file data being cached that isn't going to be used again.

I'm also trying the ck kernels just to see how they run. So far they work better.

Signature · 2 lines
shannon "AT" widomaker.com -- ["All of us get lost in the darkness,
dreamers turn to look at the stars" -- Rush ]
#9 lore raw
In reply to Linda Walsh
Quoted 6 lines · Linda Walsh

Hmmm, not to be contrary, but I have a 1GB system that refuses to swap
during large file i/o operations. For the first time in a *long* time, I read someone's suggestion to increase swappiness -- I did, to 75 or 80, (I've booted since then, so it's back to 60 and no swap usage) and some of the programs that rarely run actually swapped. It was great! I finally had more memory for file i/o operations.

It's great if you actually need the file data that gets stored.

Quoted 2 lines

Maybe you are telling the system to "feel free" to use swap by having a
large swap file?

I don't believe that matters, and certainly doesn't seem to affect my own system.

If I use a smaller swap file, I just run out faster.

Is your experience different?

Quoted 5 lines

I agree. Try getting rid of your swap file entirely -- your system will still run unless you are overloading memory, but you have a Gig. How much do you need to keep in memory? Sure, if/when I get a 4-way CPU (I have a 2-cpu setup now), I might go up to 4G, but I might be running multiple virtual machines too!

Sure it will run, but I *want* swap to be used to remove unused programs.

My current problem is that *useful* program code is being swapped out and being replaced by *useless* cached file data.

Quoted 3 lines

You might try the "cfq" block i/o algorithm. Then you can
ionice down the disk priority of background processes (though you need to be root to reduce ionice levels at this point, unlike cpu nice).

I've not seen ionice.

Signature · 2 lines
shannon "AT" widomaker.com -- ["All of us get lost in the darkness,
dreamers turn to look at the stars" -- Rush ]
#10 lore raw
Quoted 30 lines · On Tue, 2 May 2006 00:21:08 -0400 Charles Shannon Hendrix wrote:
Quoted 6 lines · Linda Walsh

Hmmm, not to be contrary, but I have a 1GB system that refuses to swap
during large file i/o operations. For the first time in a *long* time, I read someone's suggestion to increase swappiness -- I did, to 75 or 80, (I've booted since then, so it's back to 60 and no swap usage) and some of the programs that rarely run actually swapped. It was great! I finally had more memory for file i/o operations.

It's great if you actually need the file data that gets stored.

Quoted 2 lines

Maybe you are telling the system to "feel free" to use swap by having a
large swap file?

I don't believe that matters, and certainly doesn't seem to affect my own system.

If I use a smaller swap file, I just run out faster.

Is your experience different?

Quoted 5 lines

I agree. Try getting rid of your swap file entirely -- your system will still run unless you are overloading memory, but you have a Gig. How much do you need to keep in memory? Sure, if/when I get a 4-way CPU (I have a 2-cpu setup now), I might go up to 4G, but I might be running multiple virtual machines too!

Sure it will run, but I *want* swap to be used to remove unused programs.

Quoted 8 lines

My current problem is that *useful* program code is being swapped out and being replaced by *useless* cached file data.

Quoted 3 lines

You might try the "cfq" block i/o algorithm. Then you can
ionice down the disk priority of background processes (though you need to be root to reduce ionice levels at this point, unlike cpu nice).

I've not seen ionice.

It's hidden^w embedded in Documentation/block/ioprio.txt ...

---
~Randy