Skip to contentSkip to conversation

oom with 2.6.11

DISCUSSION Christian Kujau 5 participants 21 messages Mar 8, 2005 – Mar 20, 2005

lore
Share & tools
lore
#0 lore raw

hallo list,

today my machine went out out memory and noticing it several hours after the first OOM message in the log, i wonder
1) why this happened at all and
2) why almost every service was killed despite the clever algorithms documented in mm/oom_kill.c.

the first oom message went to the syslog at 01:27, i was away and no heavy tasks were scheduled:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.txt

mysqld got killed by the oom killer, so i have to suspect mysql for being the reason for oom here, even that i know that mysqld is running all day long. several other tasks got killed, but "Free swap" stays at 0kB and the oom killer kills almost every other tasks, with no success in freeing ram.

the log stops at 03:21, perhaps syslog-ng got killed.
at around 07:31 i noticed the mess, did SYSRQ-E and now i was able to login again. i pressed SYSRQ-M/T/P too, they are all in the log. at this time loadavg was at 249 ;)

i went to runlevel 2, then up again to 3 and all services are up and running again.

some 2.6.11-rc3 BK snapshot was running pretty stable (no OOM) for ~30 days before i switched to 2.6.11 (vanilla) a few days ago. i have to (not) reproduce the problem the next night, i wonder if it will happen again.

do you vm-gurus have any idea to the points asked above?

more infos about the box here: http://nerdbynature.de/bits/sheep/2.6.11/oom/

thank you for your comments,
Christian.

Signature · 3 lines
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server.
#1 lore raw

Hi Christian,

Could you check the mm/oom_kill.c for your kernel 2.6.11-rc3?

During the 2.6.11-rc development, the oom killer was changed by Andrea Arcangeli. I do not remember exactly which was the version that this modification was included, perhaps in kernel 2.6.11-rc4.

Now this oom killer modification is part of 2.6.11 vanilla kernel.

Send the mm/oom_kill.c of 2.6.11-rc3 to me, please. Let me confirm my doubt.

BR,

Mauricio Lin.

Quoted 46 lines · Christian Kujau

hallo list,

today my machine went out out memory and noticing it several hours after the first OOM message in the log, i wonder
1) why this happened at all and
2) why almost every service was killed despite the clever algorithms documented in mm/oom_kill.c.

the first oom message went to the syslog at 01:27, i was away and no heavy tasks were scheduled:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.txt

mysqld got killed by the oom killer, so i have to suspect mysql for being the reason for oom here, even that i know that mysqld is running all day long. several other tasks got killed, but "Free swap" stays at 0kB and the oom killer kills almost every other tasks, with no success in freeing ram.

the log stops at 03:21, perhaps syslog-ng got killed.
at around 07:31 i noticed the mess, did SYSRQ-E and now i was able to login again. i pressed SYSRQ-M/T/P too, they are all in the log. at this time loadavg was at 249 ;)

i went to runlevel 2, then up again to 3 and all services are up and running again.

some 2.6.11-rc3 BK snapshot was running pretty stable (no OOM) for ~30 days before i switched to 2.6.11 (vanilla) a few days ago. i have to (not) reproduce the problem the next night, i wonder if it will happen again.

do you vm-gurus have any idea to the points asked above?

more infos about the box here: http://nerdbynature.de/bits/sheep/2.6.11/oom/

thank you for your comments,
Christian.
--
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw

Hi Christian,

I found the 2.6.11-rc3 patch. The oom killer modification from Arcangeli was included in 2.6.11-rc3. Right? So this is correct, so the problem is not related to Arcangeli modification.

Does anyone have idea?

BR,

Mauricio Lin.

Quoted 64 lines · Mauricio Lin

Hi Christian,

Could you check the mm/oom_kill.c for your kernel 2.6.11-rc3?

During the 2.6.11-rc development, the oom killer was changed by Andrea Arcangeli. I do not remember exactly which was the version that this modification was included, perhaps in kernel 2.6.11-rc4.

Now this oom killer modification is part of 2.6.11 vanilla kernel.

Send the mm/oom_kill.c of 2.6.11-rc3 to me, please. Let me confirm my doubt.

BR,

Mauricio Lin.

Quoted 46 lines · Christian Kujau

hallo list,

today my machine went out out memory and noticing it several hours after the first OOM message in the log, i wonder
1) why this happened at all and
2) why almost every service was killed despite the clever algorithms documented in mm/oom_kill.c.

the first oom message went to the syslog at 01:27, i was away and no heavy tasks were scheduled:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.txt

mysqld got killed by the oom killer, so i have to suspect mysql for being the reason for oom here, even that i know that mysqld is running all day long. several other tasks got killed, but "Free swap" stays at 0kB and the oom killer kills almost every other tasks, with no success in freeing ram.

the log stops at 03:21, perhaps syslog-ng got killed.
at around 07:31 i noticed the mess, did SYSRQ-E and now i was able to login again. i pressed SYSRQ-M/T/P too, they are all in the log. at this time loadavg was at 249 ;)

i went to runlevel 2, then up again to 3 and all services are up and running again.

some 2.6.11-rc3 BK snapshot was running pretty stable (no OOM) for ~30 days before i switched to 2.6.11 (vanilla) a few days ago. i have to (not) reproduce the problem the next night, i wonder if it will happen again.

do you vm-gurus have any idea to the points asked above?

more infos about the box here: http://nerdbynature.de/bits/sheep/2.6.11/oom/

thank you for your comments,
Christian.
--
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 7 lines · Mauricio Lin

Hi Christian,

I found the 2.6.11-rc3 patch. The oom killer modification from Arcangeli was included in 2.6.11-rc3. Right? So this is correct, so the problem is not related to Arcangeli modification.

Does anyone have idea?

hi Mauricio,

thank you for your answers, but i'm unable to investigate right now, will do so in the evening.

thanks,
Christian.

Signature · 4 lines
BOFH excuse #240:

Too many little pins on CPU confusing it, bend back and forth until 10-20%
are neatly removed. Do _not_ leave metal bits visible!
#4 lore raw

ok,

as "promised", it the OOM happened again with the same plain 2.6.11, details here.

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11_2.txt

the following is a quite long, but please read on
(if anyone is reading at all :))

this time it happened at 08:01, and i could image some heavy cron jobs were going on. but as i said: "it did not happen before". there are also output of SYSRQ-T/M/P. i did SYSRQ-E to recover the machine, but then decided to reboot back to 2.6.11-rc5-bk2.

i had a look at the changelogs too and noticed that ChangeLog-2.6.11 contains 7 occurrences of "OOM" in the patch desctiption:

[PATCH] mm: overcommit updates, 2005-01-03
[PATCH] vmscan: count writeback pages in nr_scanned, 2005-01-08 [PATCH] possible rq starvation on oom, 2005-01-13
[PATCH] mm: adjust dirty threshold for lowmem-only mappings, 2005-01-25 [PATCH] mm: oom-killer tunable, 2005-02-02
[PATCH] mm: fix several oom killer bugs, 2005-02-02
[PATCH] Fix oops in alloc_zeroed_user_highpage() when [...],2005-02-09

release dates:

2.6.11-rc5-bk1  26-Feb-2005
2.6.11-rc5-bk2  27-Feb-2005  <
2.6.11-rc5-bk3  28-Feb-2005
2.6.11-rc5-bk4  01-Mar-2005
2.6.11          02-Mar-2005

so i really don't see any patches that *could* have something to do with the issue here.

now comes the weird part:

i was going to compile 2.6.11-rc5-bk4, to sort out the "bad" kernel. compiling went fine. ok, finished some email, ok, suddenly my swap was used up again, and no memory left - uh oh! OOM again, with 2.6.11-rc5-bk2!

to summarize it:
i've run 2.6.11-rc2-bk10 during whole february, then switched to 2.6.11-rc5-bk2 on 28.02.2005, then to 2.6.11 on 05.03.2005 - and only noticed with 2.6.11 first, now with 2.6.11-rc5-bk2 too.

there is an interesting part in the logfiles:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.txt
http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11_2.txt
http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11-rc5-bk2.txt

every last message before the "OOM" messages is something with pppd:

Mar 10 13:45:55 sheep pppd[1567]: Starting link
Mar 10 14:12:29 sheep kernel: oom-killer: gfp_mask=0x1d2
Mar  8 00:59:58 sheep pppd[418]: Starting link
Mar  8 01:27:33 sheep kernel: oom-killer: gfp_mask=0xd0
Mar  9 07:33:49 sheep pppd[30937]: Starting link
Mar  9 08:01:35 sheep kernel: oom-killer: gfp_mask=0x1d2

and 30min later OOM kicks in. normally, pppd (pppoe) gives messages like this:

Mar 10 14:23:38 sheep pppd[26365]: Starting link
Mar 10 14:23:38 sheep pppd[26365]: Serial connection established.
Mar 10 14:23:38 sheep pppd[26365]: Connect: ppp0 <--> /dev/pts/0
Mar 10 14:23:38 sheep pppoe[26383]: PADS: Service-Name: ''
Mar 10 14:23:38 sheep pppoe[26383]: PPP session is 6804
Mar 10 14:23:39 sheep pppd[26365]: CHAP authentication succeeded
Mar 10 14:23:40 sheep pppd[26365]: Local IP address changed to

[...]

is this strange? or not?

i hope someone has a hint for me, because "going back to the stable kernel" would mean "being bound to 2.6.11-rc2-bk10" :(

thank you for any hints,
Christian.

PS: Steven, i've cc'ed you because you have trouble with new 2.6.11 kernels and pppd too. maybe unrelated, maybe not.

Signature · 3 lines
BOFH excuse #185:

system consumed all the paper for paging
#5 lore raw
Quoted 4 lines · Christian Kujau

i was going to compile 2.6.11-rc5-bk4, to sort out the "bad" kernel. compiling went fine. ok, finished some email, ok, suddenly my swap was used up again, and no memory left - uh oh! OOM again, with 2.6.11-rc5-bk2!

Well if you ran out of swap then yes, the oom-killer will visit you.

Why did you run out of swapspace?

#6 lore raw
Quoted 10 lines · Andrew Morton
Quoted 3 lines · Christian Kujau

i was going to compile 2.6.11-rc5-bk4, to sort out the "bad" kernel. compiling went fine. ok, finished some email, ok, suddenly my swap was used up again, and no memory left - uh oh! OOM again, with 2.6.11-rc5-bk2!

Well if you ran out of swap then yes, the oom-killer will visit you.

Why did you run out of swapspace?

hm, if i only knew. i don't know how long it took the other night to go from "normal" to "OOM". but today, with 2.6.11-rc5-bk2 (well, yesterday actually) i was working normally, and all of a sudden swap goes from 170MB used swap (normal) to OOM. i think it took a minute or so, but i just can't tell which application went nuts. today the first process that got killed was "ssh-agent", the other day it was mysqld. but even after this, it should've released some memory, right? but the oom-killer goes on and on and kills the next task.

i'll monitor memory usage tonight and see what it gives. these "pppd" messages are suspicious though.

thank you,
Christian.

Signature · 3 lines
BOFH excuse #135:

You put the disk in upside down.
#7 lore raw
Quoted 21 lines · Christian Kujau
Quoted 10 lines · Andrew Morton
Quoted 3 lines · Christian Kujau

i was going to compile 2.6.11-rc5-bk4, to sort out the "bad" kernel. compiling went fine. ok, finished some email, ok, suddenly my swap was used up again, and no memory left - uh oh! OOM again, with 2.6.11-rc5-bk2!

Well if you ran out of swap then yes, the oom-killer will visit you.

Why did you run out of swapspace?

hm, if i only knew. i don't know how long it took the other night to go from "normal" to "OOM". but today, with 2.6.11-rc5-bk2 (well, yesterday actually) i was working normally, and all of a sudden swap goes from 170MB used swap (normal) to OOM. i think it took a minute or so, but i just can't tell which application went nuts. today the first process that got killed was "ssh-agent", the other day it was mysqld. but even after this, it should've released some memory, right? but the oom-killer goes on and on and kills the next task.

Quoted 3 lines

i'll monitor memory usage tonight and see what it gives. these "pppd" messages are suspicious though.

I've also seen this 3 times now.. I'm running Xen 2.0 and the 2.6.10-xen0 kernel (with -as5 patch) goes OOM after a couple of days operation.. and then the box reboots.

The "virtual machines" are OK, it's only the dom0 kernel that goes OOM.. And I'm not running anything special on dom0, only xen control stuff (which is written in python..), ntp, nfs server, ssh, lvm2 and software raid.

Now I'm running a script which logs the cpu/memory/swap usage every 1 minutes.. trying to see if I can find the cause for the OOM.

-- Pasi Kärkkäinen

                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.
#8 lore raw
In reply to Christian Kujau

Hi Christian,

I would like to know what are the kernel versions this problem happened.

Did this problem start from 2.6.11-rc2-bk10?

BR,

Mauricio Lin.

Quoted 90 lines · Christian Kujau

ok,

as "promised", it the OOM happened again with the same plain 2.6.11, details here.

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11_2.txt

the following is a quite long, but please read on
(if anyone is reading at all :))

this time it happened at 08:01, and i could image some heavy cron jobs were going on. but as i said: "it did not happen before". there are also output of SYSRQ-T/M/P. i did SYSRQ-E to recover the machine, but then decided to reboot back to 2.6.11-rc5-bk2.

i had a look at the changelogs too and noticed that ChangeLog-2.6.11 contains 7 occurrences of "OOM" in the patch desctiption:

[PATCH] mm: overcommit updates, 2005-01-03
[PATCH] vmscan: count writeback pages in nr_scanned, 2005-01-08 [PATCH] possible rq starvation on oom, 2005-01-13
[PATCH] mm: adjust dirty threshold for lowmem-only mappings, 2005-01-25 [PATCH] mm: oom-killer tunable, 2005-02-02
[PATCH] mm: fix several oom killer bugs, 2005-02-02
[PATCH] Fix oops in alloc_zeroed_user_highpage() when [...],2005-02-09

release dates:

2.6.11-rc5-bk1  26-Feb-2005
2.6.11-rc5-bk2  27-Feb-2005  <
2.6.11-rc5-bk3  28-Feb-2005
2.6.11-rc5-bk4  01-Mar-2005
2.6.11          02-Mar-2005

so i really don't see any patches that *could* have something to do with the issue here.

now comes the weird part:

i was going to compile 2.6.11-rc5-bk4, to sort out the "bad" kernel. compiling went fine. ok, finished some email, ok, suddenly my swap was used up again, and no memory left - uh oh! OOM again, with 2.6.11-rc5-bk2!

to summarize it:
i've run 2.6.11-rc2-bk10 during whole february, then switched to 2.6.11-rc5-bk2 on 28.02.2005, then to 2.6.11 on 05.03.2005 - and only noticed with 2.6.11 first, now with 2.6.11-rc5-bk2 too.

there is an interesting part in the logfiles:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.txt
http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11_2.txt
http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11-rc5-bk2.txt

every last message before the "OOM" messages is something with pppd:

Mar 10 13:45:55 sheep pppd[1567]: Starting link
Mar 10 14:12:29 sheep kernel: oom-killer: gfp_mask=0x1d2
Mar  8 00:59:58 sheep pppd[418]: Starting link
Mar  8 01:27:33 sheep kernel: oom-killer: gfp_mask=0xd0
Mar  9 07:33:49 sheep pppd[30937]: Starting link
Mar  9 08:01:35 sheep kernel: oom-killer: gfp_mask=0x1d2

and 30min later OOM kicks in. normally, pppd (pppoe) gives messages like this:

Mar 10 14:23:38 sheep pppd[26365]: Starting link
Mar 10 14:23:38 sheep pppd[26365]: Serial connection established.
Mar 10 14:23:38 sheep pppd[26365]: Connect: ppp0 <--> /dev/pts/0
Mar 10 14:23:38 sheep pppoe[26383]: PADS: Service-Name: ''
Mar 10 14:23:38 sheep pppoe[26383]: PPP session is 6804
Mar 10 14:23:39 sheep pppd[26365]: CHAP authentication succeeded
Mar 10 14:23:40 sheep pppd[26365]: Local IP address changed to

[...]

is this strange? or not?

i hope someone has a hint for me, because "going back to the stable kernel" would mean "being bound to 2.6.11-rc2-bk10" :(

thank you for any hints,
Christian.

PS: Steven, i've cc'ed you because you have trouble with new 2.6.11 kernels and pppd too. maybe unrelated, maybe not.
--
BOFH excuse #185:

system consumed all the paper for paging

#9 lore raw
Quoted 5 lines · Mauricio Lin

Hi Christian,

I would like to know what are the kernel versions this problem happened.

Did this problem start from 2.6.11-rc2-bk10?

i noticed it first at 2.6.11, then again with 2.6.11-rc5-bk2. suspecting pppd to be the culprit to chew up all RAM after being terminated by my ISP once a day - i just have to wait (must be around 2a.m.).

thanks,
Christian.

Signature · 3 lines
BOFH excuse #134:

because of network lag due to too many people playing deathmatch
#10 lore raw

Hi Christian,

Quoted 10 lines · Christian Kujau
Quoted 5 lines · Mauricio Lin

Hi Christian,

I would like to know what are the kernel versions this problem happened.

Did this problem start from 2.6.11-rc2-bk10?

i noticed it first at 2.6.11, then again with 2.6.11-rc5-bk2. suspecting pppd to be the culprit to chew up all RAM after being terminated by my ISP once a day - i just have to wait (must be around 2a.m.).

Have you tried with 2.6.10 in order to check this problem?

BR,

Mauricio Lin.

#11 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 8 lines · Mauricio Lin
Quoted 6 lines
Quoted 2 lines

Did this problem start from 2.6.11-rc2-bk10?

i noticed it first at 2.6.11, then again with 2.6.11-rc5-bk2. suspecting pppd to be the culprit to chew up all RAM after being terminated by my ISP once a day - i just have to wait (must be around 2a.m.).

Have you tried with 2.6.10 in order to check this problem?

i don't think i've run plain 2.6.10 at all, syslog does not say so. i've run several versions of 2.6.9, then 2.6.10-rc1, 2.6.10-rc1-bk19, 2.6.10-rc2-bk9, 2.6.10-rc3-bk11, 2.6.11-rc2-bk10, 2.6.11-rc5-bk2.

as stated earlier, the problem kicked in when i switched to 2.6.11, i switched back to 2.6.11-rc5-bk2 and it happened again. by looking at the syslog i am (still) suspecting pppd to be the source of OOM, but it did not show up as a memory pig in "ps". i am running 2.6.11.3 now and the machine survived almost 2 passes of pppd's daily hangups. i'll keep an eye on it.

thank you for your concern,
Christian.
- --
BOFH excuse #357:

I'd love to help you -- it's just that the Boss won't let me near the computer.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCNu1e+A7rjkF8z0wRAmSPAJ49erb21bO6inHPn/zoUH2gNHJk8QCguYDA HyzXS9whrJnb9eXVz9IP8zc=
=zhEx
-----END PGP SIGNATURE-----

#12 lore raw

for the record: the "oom bug" turned out to be user generated. a *lot* of small scripts were started, triggering oom again and again, user error.

the source of the problem is still pppd and the discussion continues as a debian bugreport:

    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=299875

thank you for all your help,
Christian.

Signature · 3 lines
BOFH excuse #139:

UBNC (user brain not connected)
#13 lore raw
In reply to Christian Kujau

In file mm/oom_kill.c, uncomment line 24: /* #define DEBUG */. And next time when oom happens again, we'll see the badness.

--coywolf

Quoted 46 lines · Christian Kujau

hallo list,

today my machine went out out memory and noticing it several hours after the first OOM message in the log, i wonder
1) why this happened at all and
2) why almost every service was killed despite the clever algorithms documented in mm/oom_kill.c.

the first oom message went to the syslog at 01:27, i was away and no heavy tasks were scheduled:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.txt

mysqld got killed by the oom killer, so i have to suspect mysql for being the reason for oom here, even that i know that mysqld is running all day long. several other tasks got killed, but "Free swap" stays at 0kB and the oom killer kills almost every other tasks, with no success in freeing ram.

the log stops at 03:21, perhaps syslog-ng got killed.
at around 07:31 i noticed the mess, did SYSRQ-E and now i was able to login again. i pressed SYSRQ-M/T/P too, they are all in the log. at this time loadavg was at 249 ;)

i went to runlevel 2, then up again to 3 and all services are up and running again.

some 2.6.11-rc3 BK snapshot was running pretty stable (no OOM) for ~30 days before i switched to 2.6.11 (vanilla) a few days ago. i have to (not) reproduce the problem the next night, i wonder if it will happen again.

do you vm-gurus have any idea to the points asked above?

more infos about the box here: http://nerdbynature.de/bits/sheep/2.6.11/oom/

thank you for your comments,
Christian.
--
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Coywolf Qi Hunt
Homepage http://sosdg.org/~coywolf/
#14 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 2 lines · Coywolf Qi Hunt

In file mm/oom_kill.c, uncomment line 24: /* #define DEBUG */. And next time when oom happens again, we'll see the badness.

oh, good hint. will do this before the next reboot (in a few hours i guess)

thanks,
Christian.

- --
BOFH excuse #134:

because of network lag due to too many people playing deathmatch -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCMbTj+A7rjkF8z0wRAmdAAJ4sCIaUqZaKn7gGtkpN7Wb47acFiACgrOSe DpT3tE1/4zfPIDueDwBMhDU=
=o6Lr
-----END PGP SIGNATURE-----

Signature · 3 lines
BOFH excuse #37:

heavy gravity fluctuation, move computer to floor rapidly
#15 lore raw

hi again,

i had to wait for my pppoe session to be terminated by the remote peer [1], and now it happened again with 2.6.11-rc5-bk2:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11-rc5-bk2_2.txt
http://nerdbynature.de/bits/sheep/2.6.11/oom/lsmod_2.6.11-rc5-bk2
http://nerdbynature.de/bits/sheep/2.6.11/oom/config-2.6.11-rc5-bk2.gz

i wanted to see the application chewing up the most RAM so i was running "ps aux --sort=-vsz,-rss | head -n1" every 5sec and wrote it's output to a file:

http://nerdbynature.de/bits/sheep/2.6.11/oom/daily_stats-2.6.11-rc5-bk2.log.gz

but even when almost all swap is used up, mysqld ist still no#1 with 131928 KB VSZ, as always. still suspecting something wrong with pppd, the binary istself does *not* show up on the top-ten-memory-pigs.

real memory is almost always used up, but that's ok.
i stripped it down to show only the timestamp and the "Swap: .. used" numbers:

http://nerdbynature.de/bits/sheep/2.6.11/oom/daily_stats-2.6.11-rc5-bk2_stripped.log.gz

as you can see, until "Sat Mar 12 01:05:51" 122MB swap was used, from then on swap-usage goes up until "Sat Mar 12 01:49:45", when all swap is in use. the system is unable to keep up with writing memory usage every 5 seconds, the next entry is from 12 "01:56:24" - 6 minutes after the first OOM message (i'm sure someone could feed this to gnuplot and make nice bars out of it - i can't)

where else could i look to actually *see* where the memory goes to?

somehow lost in kernel versions,
Christian.

[1] "remote peer" aka "ISP". is there a way to "simulate" a "LCP terminated by peer" locally? otherwise i really have to wait 24h to trigger the bug.

Signature · 3 lines
BOFH excuse #373:

Suspicious pointer corrupted virtual machine
#16 lore raw

hello again,

unfortunately i've hit OOM again, this time with "#define DEBUG" enabled in mm/oom_kill.c:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.3.txt

by "Mar 16 18:32" pppd died again and OOM kicked in 30min later. (there are a *lot* messages of a shell script named "check-route.sh". it's a little script which runs every minute or so to check if my default route is still ok and if ping to the outside world are possible. definitely not a memory hog, but noisy)

since tracking the "most memory consuming applications" did not reveal any hints [1], i have monitored /proc/slabinfo and /proc/meminfo this time:

http://nerdbynature.de/bits/sheep/2.6.11/oom/daily_stats-2.6.11.3.gz

as stated before, i was suspecting pppd to be the bad guy here, and yes: i downgraded pppd to an earlier version and pppd (and the system) survived 2 terminations of my dial-up ISP. yesterday i've upgraded back again to current pppd (debian/unstable) and the OOM problem returned. yes, i'll bug the debian people now (hello!), but grepping for "ppp" in daily_stats-2.6.11.3.gz gives no hits. so "pppd" does not get *any* points from mm/oom_kill.c and thus no attempts are made to kill it (it is always only kill'able with "-9"). furthermore, i thought /proc/slabinfo coud give me some hints about *where* all the memory went in. scrolling down this file to the bottom, where "SwapFree" shows "0 kB" i don't see any alarming numbers in the "slabinfo" right above "meminfo".

could someone give me a hint, please?

thanks,
Christian.

[1] http://lkml.org/lkml/2005/3/12/88

more info for this recent OOM issue:
http://nerdbynature.de/bits/sheep/2.6.11/oom/dmesg.2.6.11.3
http://nerdbynature.de/bits/sheep/2.6.11/oom/lsmod_2.6.11.3
http://nerdbynature.de/bits/sheep/2.6.11/oom/config-2.6.11.3.gz

Signature · 3 lines
BOFH excuse #62:

need to wrap system in aluminum foil to fix problem
#17 lore raw
Quoted 17 lines · Christian Kujau

unfortunately i've hit OOM again, this time with "#define DEBUG" enabled
in mm/oom_kill.c:

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.3.txt

by "Mar 16 18:32" pppd died again and OOM kicked in 30min later. (there are a *lot* messages of a shell script named "check-route.sh". it's a little script which runs every minute or so to check if my default route is still ok and if ping to the outside world are possible. definitely not a memory hog, but noisy)

since tracking the "most memory consuming applications" did not reveal any hints [1], i have monitored /proc/slabinfo and /proc/meminfo this time:

http://nerdbynature.de/bits/sheep/2.6.11/oom/daily_stats-2.6.11.3.gz

Quoted 8 lines

as stated before, i was suspecting pppd to be the bad guy here, and yes: i downgraded pppd to an earlier version and pppd (and the system) survived 2 terminations of my dial-up ISP. yesterday i've upgraded back again to current pppd (debian/unstable) and the OOM problem returned. yes, i'll bug the debian people now (hello!), but grepping for "ppp" in daily_stats-2.6.11.3.gz gives no hits. so "pppd" does not get *any* points from mm/oom_kill.c and thus no attempts are made to kill it (it is always only kill'able with "-9").

The oom-killer tries to be nicer to processes which are running as root.

Quoted 4 lines

furthermore, i thought /proc/slabinfo coud give
me some hints about *where* all the memory went in. scrolling down this file to the bottom, where "SwapFree" shows "0 kB" i don't see any alarming numbers in the "slabinfo" right above "meminfo".

MemTotal:       256372 kB
MemFree:          3280 kB
Buffers:           608 kB
Cached:           3256 kB
SwapCached:        664 kB
Active:         105020 kB
Inactive:        20364 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       256372 kB
LowFree:          3280 kB
SwapTotal:      784468 kB
SwapFree:            0 kB
Dirty:              12 kB
Writeback:           0 kB
Mapped:         130332 kB
Slab:            61424 kB
CommitLimit:    912652 kB
Committed_AS:  1323548 kB
PageTables:      51668 kB
VmallocTotal:   778184 kB
VmallocUsed:      3464 kB
VmallocChunk:   774492 kB

Some application went berzerk, used up all the swap and then oomed the box.

You could perhaps run `top -d1' then hit M so the output is sorted by bloatiness, then try to catch the culprit.

But it would be better to have some app which prints the N most memory-hungry processes every second and simply scrolls that up the screen. I'm not aware of such a thing, but it could be cooked up via /proc/N/cmdline and /proc/N/statm.

#18 lore raw
Quoted 5 lines · Andrew Morton

Some application went berzerk, used up all the swap and then oomed the box.

You could perhaps run `top -d1' then hit M so the output is sorted by bloatiness, then try to catch the culprit.

i've already done that. as OOM happens when i am not around, i did that with "ps":

http://lkml.org/lkml/2005/3/12/88
http://nerdbynature.de/bits/sheep/2.6.11/oom/daily_stats-2.6.11-rc5-bk2.log.gz

Quoted 4 lines

But it would be better to have some app which prints the N most memory-hungry processes every second and simply scrolls that up the screen. I'm not aware of such a thing, but it could be cooked up via /proc/N/cmdline and /proc/N/statm.

i hope the link above does reveal this information.

i just wrote a bug report for the (debian), ppp package, but to know *where* the memory goes to would really help, i think.

thank you,
Christian.

Signature · 3 lines
BOFH excuse #72:

Satan did it
#19 lore raw
In reply to Christian Kujau
Quoted 5 lines · Christian Kujau

hello again,

unfortunately i've hit OOM again, this time with "#define DEBUG" enabled in mm/oom_kill.c:

Quoted 7 lines

http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.3.txt

by "Mar 16 18:32" pppd died again and OOM kicked in 30min later. (there are a *lot* messages of a shell script named "check-route.sh". it's a little script which runs every minute or so to check if my default route is still ok and if ping to the outside world are possible. definitely not a memory hog, but noisy)

I do "grep check-route.sh oom_2.6.11.3.txt | wc" and it shows 4365 lines, which means
there're 4365 that script processes running, from pid 4260 to12747, mostly with pretty low points, 123. Based on this points, suppose each script consumes 100k, that'll be 100k*4k=400M roughly. And your box's is merely 256M MemTotal.

The currently oom algorithm fails to find out such kinds of memory hog. And the kernel kills other innocent processes because the its points is much lower than most others.

Check this script and disable it; see what will happen.

Signature · 2 lines
Coywolf Qi Hunt
http://sosdg.org/~coywolf/
#20 lore raw
Quoted 1 line · Coywolf Qi Hunt

I do "grep check-route.sh oom_2.6.11.3.txt | wc" and it shows 4365

duh, good catch! really!

Quoted 4 lines

lines, which means there're 4365 that script processes running, from pid 4260 to12747, mostly with pretty low points, 123. Based on this points, suppose each script consumes 100k, that'll be 100k*4k=400M roughly. And your box's is merely 256M MemTotal.

yes, i just checked, the script is looping and crond is starting a new one, and another....and the oom-killer does not catch it, because it's too small and of course don't know where it is coming from (crond).

Quoted 1 line

Check this script and disable it; see what will happen.

yes, will do that. on a (not so unimportant) side-note: i was told the whole thing should be fixed with 2.6.11.4:

[PATCH] CAN-2005-0384: Remote Linux DoS on ppp servers

after all it seems to be PEBKAC and bad luck...what a week.

thank you for your help,
Christian.

Signature · 3 lines
BOFH excuse #416:

We're out of slots on the server