Skip to contentSkip to conversation

Looking for better 2.2-based VM (do_try_to_free_pages fails, machine hangs)

DISCUSSION Yann Dirson 8 participants 16 messages Nov 1, 2000 – Nov 7, 2000

lore
Share & tools
lore
#0 lore raw

Hi,

Using a 2.2.17 kernel I often experience problems where I get messages like "VM: do_try_to_free_pages failed for <some process>", and the machine hangs until the VM can recover, which sometimes takes too long for me to wait. I suppose that the problem is similar sometimes when I get a frozen system under X, but can't see the kernel messages then.

Yesterday I could reproduce this at will, with a "make -j50" on 2.2.17 sources (as unpriviledged user). In less than half an our syslogd stopped to log anything (at 00:38), and this morning I could only see those messages trying to free pages for (or from ?) wwwoffled. Last load see by "top" on another VC was ~74.

Have some work been done for 2.2.18 that could help me ? Are there some 2.2-based VM patches that could help (I found the VM-global patch from Andrea but have no info about what it is, and could not find 2.2-based patches on Rik's pages) ?

Also I can't be sure for now I don't run into a hw problem...

I'm willing to investigate, but clearly lack experience in the VM...

Regards,

Signature · 8 lines
Yann Dirson    <[email protected]> |    Why make M$-Bill richer & richer ?
debian-email:   <[email protected]> |   Support Debian GNU/Linux:
                                    | Cheaper, more Powerful, more Stable !
http://ydirson.free.fr/             | Check <http://www.debian.org/>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#1 lore raw

In article <[email protected]>,

Quoted 5 lines · Yann Dirson

Using a 2.2.17 kernel I often experience problems where I get messages like "VM: do_try_to_free_pages failed for <some process>", and the machine hangs until the VM can recover, which sometimes takes too long for me to wait. I suppose that the problem is similar sometimes when I get a frozen system under X, but can't see the kernel messages then.

I'm seeing the same on some machines. Running several instances of bonnie on a dual SMP Intel with a DAC 1164 raid controller would kill the machine in a few hours. However it has been running several bonnies now without a hitch for 2 days, on 2.2.18pre18

Mike.

Signature · 5 lines
People get the operating system they deserve.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#2 lore raw
In reply to Yann Dirson
Quoted 14 lines · Yann Dirson

Hi,

Using a 2.2.17 kernel I often experience problems where I get messages like "VM: do_try_to_free_pages failed for <some process>", and the machine hangs until the VM can recover, which sometimes takes too long for me to wait. I suppose that the problem is similar sometimes when I get a frozen system under X, but can't see the kernel messages then.

Yesterday I could reproduce this at will, with a "make -j50" on 2.2.17 sources (as unpriviledged user). In less than half an our syslogd stopped to log anything (at 00:38), and this morning I could only see those messages trying to free pages for (or from ?) wwwoffled. Last load see by "top" on another VC was ~74.

Quoted 3 lines

Have some work been done for 2.2.18 that could help me ? Are there some 2.2-based VM patches that could help (I found the VM-global patch from Andrea but have no info about what it is

VM-global It should fix your problem.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
In reply to Yann Dirson
Quoted 1 line · Marcelo Tosatti

VM-global It should fix your problem.

Thanks for the hint, it works great indeed, I couldn't freeze the machine any more - at least the make process understood Ctrl-C.

However, the OOM killer behaves in strange ways, it seems. In the 2 "make -j50" runs I had (one with the working fs mounted 'sync', the other 'async'), it mostly killed non-root processes, but once it killed 'cron', which was run as root:

[sync run]

Nov  1 15:51:05 bylbo kernel: VM: killing process cpp
Nov  1 15:56:38 bylbo kernel: VM: killing process apache
Nov  1 16:02:54 bylbo kernel: VM: killing process cc1
Nov  1 16:08:51 bylbo kernel: VM: killing process wwwoffled

[async run]

Nov  1 17:13:08 bylbo kernel: VM: killing process apache
Nov  1 17:14:01 bylbo kernel: VM: killing process cron

apache was running as "www-data" (uid 33), wwwoffled as "proxy" (uid 13).

An idea that came upon me was whether it would be possible to add to the OOMK some sort of preference for processes owned by "system users", to be defined by a "uid limit". For example, on Debian systems, where "real user" uids start at 1000, it would be great if the OOMK would leave those processes as far as possible off its kill list. Opinions ?

Best regards,

Signature · 8 lines
Yann Dirson    <[email protected]> |    Why make M$-Bill richer & richer ?
debian-email:   <[email protected]> |   Support Debian GNU/Linux:
                                    | Cheaper, more Powerful, more Stable !
http://ydirson.free.fr/             | Check <http://www.debian.org/>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#4 lore raw
In reply to Yann Dirson
Quoted 1 line · Yann Dirson

However, the OOM killer behaves in strange ways, it seems. In the 2 "make

Fair enough as there isn't an oom killer in the kernel you're running :). So it can kill unlucky tasks as well.

Since nobody cares to implement it, for 2.4.x on my TODO list there's an alternative oom killer based on the task fault rate.

(btw, make sure you're using the -7 revision of the VM-global patch, as it includes the same MM corruption bugfix that is been included into 18pre18)

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
Quoted 5 lines · Andrea Arcangeli
Quoted 1 line · Yann Dirson

However, the OOM killer behaves in strange ways, it seems. In the 2 "make

Fair enough as there isn't an oom killer in the kernel you're running :). So it can kill unlucky tasks as well.

There's a (slightly outdated?) patch available on my home page, though...

Quoted 2 lines

Since nobody cares to implement it, for 2.4.x on my TODO list there's an alternative oom killer based on the task fault rate.

Cool. It will be interesting to see how this compares to my OOM killer (and to the other approaches that will undoubtedly surface over the next few months).

I'm definately looking forward to an "OOM killer showdown" where we can compare how the different OOM tactics work. Not because I think it matters all that much on most systems (good admins put in enough memory&swap), but simply because it appears there has been amazingly little research on this subject and it's completely unknown which approach will work "best" ... or even, what kind of behaviour is considered to be best by the users...

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#6 lore raw
In reply to Andrea Arcangeli

On Wed, Nov 01, 2000 at 02:59:01PM -0200, Rik van Riel wrote:

Quoted 2 lines · Andrea

(btw, make sure you're using the -7 revision of the VM-global patch, as it includes the same MM corruption bugfix that is been included into 18pre18)

Damn, I was using -6. http://www.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.18pre9/ does not have -7. Neither does your e-mind repository hlinked from linux-mm.

I'm currently running -6 :(

Quoted 4 lines · Rik van Riel

it appears there has been amazingly little research on this subject and it's completely unknown which approach will work "best" ... or even, what kind of behaviour is considered to be best by the users...

Sounds to me like a good point to favour a config-time selection of OOM killers.

Quoted 6 lines · Andrea Arcangeli
Quoted 2 lines

Fair enough as there isn't an oom killer in the kernel you're running :). So it can kill unlucky tasks as well.

There's a (slightly outdated?) patch available on my home page, though...

Found http://www.surriel.com/patches/2.2.17pre8-oomkill. Will take a look, thanks.

Quoted 2 lines

Not because I think it matters all that much on most systems (good admins put in enough memory&swap), but simply because

Ah, I'll have to reconsider how much I rate my skills :)

Best regards,

Signature · 8 lines
Yann Dirson    <[email protected]> |    Why make M$-Bill richer & richer ?
debian-email:   <[email protected]> |   Support Debian GNU/Linux:
                                    | Cheaper, more Powerful, more Stable !
http://ydirson.free.fr/             | Check <http://www.debian.org/>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#7 lore raw
In reply to Andrea Arcangeli
Quoted 10 lines · Yann Dirson

On Wed, Nov 01, 2000 at 02:59:01PM -0200, Rik van Riel wrote:

Quoted 2 lines · Andrea

(btw, make sure you're using the -7 revision of the VM-global patch, as it includes the same MM corruption bugfix that is been included into 18pre18)

Damn, I was using -6. http://www.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.18pre9/ does not have -7. Neither does your e-mind repository hlinked from linux-mm.

I'm currently running -6 :(

A -7 to apply to 2.2.18pre17 and _previous_ releases is in the directory patches/v2.2/2.2.18pre17/. An equivalent one against 2.2.18pre18 (and probably future releases) is in patches/v2.2/2.2.18pre18/.

After applying the patch you should make sure there are no rejects with a find -name \*.rej. If there aren't rejects all gone right.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#8 lore raw
In reply to Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli

A -7 to apply to 2.2.18pre17 and _previous_ releases is in the directory

Argh !! Damned sort order - I thought pre9 was the latest :(

Thanks,

Signature · 8 lines
Yann Dirson    <[email protected]> |    Why make M$-Bill richer & richer ?
debian-email:   <[email protected]> |   Support Debian GNU/Linux:
                                    | Cheaper, more Powerful, more Stable !
http://ydirson.free.fr/             | Check <http://www.debian.org/>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#9 lore raw
In reply to Andrea Arcangeli
Quoted 8 lines · Yann Dirson
Quoted 4 lines · Rik van Riel

it appears there has been amazingly little research on this subject and it's completely unknown which approach will work "best" ... or even, what kind of behaviour is considered to be best by the users...

Sounds to me like a good point to favour a config-time selection of OOM killers.

Better yet: Apply my OOM-Killer-API-Patch[1] and build your own OOM-Killer!

Just lock your module into memory, supply an function to install_oom_killer(), save the old one (you get it as return if installing it went ok) and be happy.

And now have fun bringing your machine into OOM situations.

Want to change it back? No problem. Just get signaled somehow[2], reinstall the old one, unlock your module and wait to be cleaned up.

I never tried it above Riks 2.2.x-OOM-Killer-Patch, but it should work on top of it, because oom_kill.c isn't all that different.

Regards

Ingo Oeser

[1] http://www.tu-chemnitz.de/~ioe/oom_kill_api.patch
[2] if you don't know that much about the kernel, you shouldn't play with oom-handlers anyway ;-)

Signature · 6 lines
Feel the power of the penguin - run [email protected]
<esc>:x
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#10 lore raw
Quoted 17 lines · Ingo Oeser
Quoted 8 lines · Yann Dirson
Quoted 4 lines · Rik van Riel

it appears there has been amazingly little research on this subject and it's completely unknown which approach will work "best" ... or even, what kind of behaviour is considered to be best by the users...

Sounds to me like a good point to favour a config-time selection of OOM killers.

Better yet: Apply my OOM-Killer-API-Patch[1] and build your own OOM-Killer!

Just lock your module into memory, supply an function to install_oom_killer(), save the old one (you get it as return if installing it went ok) and be happy.

Quoted 8 lines

And now have fun bringing your machine into OOM situations.

Want to change it back? No problem. Just get signaled somehow[2], reinstall the old one, unlock your module and wait to be cleaned up.

I never tried it above Riks 2.2.x-OOM-Killer-Patch, but it should work on top of it, because oom_kill.c isn't all that different.

Ingo,

Maybe you could change Rik's oom killer to be a "module" of your OOM killer API?

With this in place, its easier for people to compare Rik's OOM killer with other possible new algorithms.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#11 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · Yann Dirson

On Wed, Nov 01, 2000 at 02:59:01PM -0200, Rik van Riel wrote:

Quoted 2 lines · Andrea

(btw, make sure you're using the -7 revision of the VM-global patch, as it includes the same MM corruption bugfix that is been included into 18pre18)

Quoted 8 lines
Quoted 4 lines

Damn, I was using -6. http://www.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.18pre9/ does not have -7. Neither does your e-mind repository hlinked from linux-mm.

I'm currently running -6 :(

A -7 to apply to 2.2.18pre17 and _previous_ releases is in the directory patches/v2.2/2.2.18pre17/. An equivalent one against 2.2.18pre18 (and probably future releases) is in patches/v2.2/2.2.18pre18/.

Now running 2.2.18pre17 + -7 since a few days

The problems I had appear to be gone. However with this kernel I have bursts of error messages from nscd (from glibc 2.1.95) like this one taken from daemon.log:

Nov  5 22:36:17 bylbo nscd: 925: while accepting connection: Cannot allocate memory

They usually appear at cron.daily time, although it looks like I kinda can reproduce them. I'm still investigating and narrowing - they seem to avoid me unfortunately :( Will launch a tracking job for the night, hopefully I'll narrow to the single cron job this time.

Anyone seen that ?

Quoted 2 lines

After applying the patch you should make sure there are no rejects with a find -name \*.rej. If there aren't rejects all gone right.

I prefer running "patch --dry-run", I hate it when I have to wait for bzip2 and tar :}

Best regards,

Signature · 8 lines
Yann Dirson    <[email protected]> |    Why make M$-Bill richer & richer ?
debian-email:   <[email protected]> |   Support Debian GNU/Linux:
                                    | Cheaper, more Powerful, more Stable !
http://ydirson.free.fr/             | Check <http://www.debian.org/>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#12 lore raw
In reply to Andrea Arcangeli
Quoted 9 lines · Yann Dirson
Nov  5 22:36:17 bylbo nscd: 925: while accepting connection: Cannot allocate memory

They usually appear at cron.daily time, although it looks like I kinda can reproduce them. I'm still investigating and narrowing - they seem to avoid me unfortunately :( Will launch a tracking job for the night, hopefully I'll narrow to the single cron job this time.

Anyone seen that ?

I see it with sendmail all the time when the fs gets really busy, and memory gets low in
box.

Thanks

Jeff

Quoted 3 lines
Quoted 3 lines

After applying the patch you should make sure there are no rejects with a find -name \*.rej. If there aren't rejects all gone right.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#13 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Jeff V. Merkey
Quoted 6 lines · Yann Dirson
Nov  5 22:36:17 bylbo nscd: 925: while accepting connection: Cannot allocate memory

They usually appear at cron.daily time, although it looks like I kinda can reproduce them. I'm still investigating and narrowing - they seem to avoid me unfortunately :( Will launch a tracking job for the night, hopefully I'll narrow to the single cron job this time.

Hm... 12h non-stop looping on the cron jobs and nothing in the logs.

Heisenbug :}

Quoted 5 lines
Quoted 1 line

Anyone seen that ?

I see it with sendmail all the time when the fs gets really busy, and memory gets low in
box.

But if your memory gets low it seems at least less anormal... In my case it occured at night when only the cron job was running.

Signature · 8 lines
Yann Dirson    <[email protected]> |    Why make M$-Bill richer & richer ?
debian-email:   <[email protected]> |   Support Debian GNU/Linux:
                                    | Cheaper, more Powerful, more Stable !
http://ydirson.free.fr/             | Check <http://www.debian.org/>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#14 lore raw
In reply to Yann Dirson
Quoted 5 lines · On Wed, 01 Nov 2000 17:48:16 Andrea Arcangeli wrote:

(btw, make sure you're using the -7 revision of the VM-global patch, as it includes the same MM corruption bugfix that is been included into 18pre18)

Andrea

"Includes" means that the full patch is not included in pre18 ?. So, will the VM-pre17 work with pre18 ?.

Signature · 6 lines
Juan Antonio Magallon Lacarta                          mailto:[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#15 lore raw
In reply to Yann Dirson
Quoted 1 line · J . A . Magallon

"Includes" means that the full patch is not included in pre18 ?.

Only the strict bugfix broadcasted to l-k is been included in pre18.

Quoted 1 line

So, will the VM-pre17 work with pre18 ?.

It will generate a trivial reject but I just uploaded a new VM-global against pre18 that generates exactly the same source code of the previous one against pre17.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/