lot of VM problem with 2.4.23
Subsystems (2)
Hi,
Since we use 2.4.23 we have lot of crash. I have no kernel panic. All I can report is this kind of syslog's message:__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process rateup
Mysql doesn't like 2.4.23 either.
SQL Error : 1 Can't create/write to file '/tmp/#sql2ec_1acd2_2.MYI' (Errcode: 30)
It arrives on servers which need swap.
No problem with 2.4.22.
Thanks
Octave
Quoted 8 lines · Octave
Hi,
Since we use 2.4.23 we have lot of crash. I have no kernel panic. All I can report is this kind of syslog's message:__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process rateupMysql doesn't like 2.4.23 either.
SQL Error : 1 Can't create/write to file '/tmp/#sql2ec_1acd2_2.MYI' (Errcode: 30)
Octave,
This looks like for some reason your /tmp file-system is read only or for any other reason kernel returns this error code.
Do not you have any more error messages dmesg or logs ?
Signature · 4 lines
Peter Zaitsev, Full-Time Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Quoted 2 lines · Octave
Hi,
Since we use 2.4.23 we have lot of crash. I have no kernel panic.
Now might be a great time to try 2.6!
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
Quoted 6 lines · Peter Zaitsev
Quoted 5 lines · Octave
Hi,
Since we use 2.4.23 we have lot of crash. I have no kernel panic. All I can report is this kind of syslog's message:__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process rateup
How much swap do you have in your system?
This is happening because the system is unable to free memory (you probably ran out of swap for some reason).
Quoted 11 lines · Marcelo Tosatti
Quoted 6 lines · Peter Zaitsev
Quoted 5 lines · Octave
Hi,
Since we use 2.4.23 we have lot of crash. I have no kernel panic. All I can report is this kind of syslog's message:__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process rateup
Quoted 4 lines
How much swap do you have in your system?
This is happening because the system is unable to free memory (you probably ran out of swap for some reason).
Marcelo,
For exemple this server: Piv 2.4GHz with 1Go RAM I think it arrives when all memory is used (ram + swap).
# free
total used free shared buffers cached
Mem: 1032592 1016492 16100 0 36184 759668
-/+ buffers/cache: 220640 811952
Swap: 265064 122784 142280# ps auxw | grep -v "0 0.0" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND mysql 26958 0.0 14.0 275000 144972 ? S Dec19 2:09 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- mysql 27004 0.0 14.0 275000 144972 ? S Dec19 2:28 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- mysql 27005 0.0 14.0 275000 144972 ? S Dec19 2:03 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- root 9621 0.0 0.1 6316 1860 ? S 01:02 0:00 /usr/sbin/sshd root 9631 0.0 0.1 2500 1352 pts/0 S 01:02 0:00 -bash root 9683 0.0 0.1 6276 1844 ? S 13:47 0:00 /usr/sbin/sshd root 9707 0.0 0.1 2504 1356 pts/2 S 13:47 0:00 -bash postfix 29728 0.0 0.1 3508 1184 ? S 15:45 0:00 pickup -l -t fifo -u mysql 7341 0.0 14.0 275000 144972 ? S 15:59 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- root 7347 0.0 0.1 2504 1356 pts/2 R 15:59 0:00 -bash
There is nothing to take 1Go of RAM.
I switched to 2.4.22 some servers and there is no more problem with. So I think it's 2.4.23's problem only.
Octave
Quoted 13 lines · Octave
# ps auxw | grep -v "0 0.0" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND mysql 26958 0.0 14.0 275000 144972 ? S Dec19 2:09 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- mysql 27004 0.0 14.0 275000 144972 ? S Dec19 2:28 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- mysql 27005 0.0 14.0 275000 144972 ? S Dec19 2:03 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- root 9621 0.0 0.1 6316 1860 ? S 01:02 0:00 /usr/sbin/sshd root 9631 0.0 0.1 2500 1352 pts/0 S 01:02 0:00 -bash root 9683 0.0 0.1 6276 1844 ? S 13:47 0:00 /usr/sbin/sshd root 9707 0.0 0.1 2504 1356 pts/2 S 13:47 0:00 -bash postfix 29728 0.0 0.1 3508 1184 ? S 15:45 0:00 pickup -l -t fifo -u mysql 7341 0.0 14.0 275000 144972 ? S 15:59 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql -- root 7347 0.0 0.1 2504 1356 pts/2 R 15:59 0:00 -bash
Quoted 1 line
There is nothing to take 1Go of RAM.
Octave,
one of my collegues had a server which occasionally crashed at night with mysql taking all the memory. I think it was with an old 2.4.18 kernel. He finally reinstalled all the machine and it never happened anymore. So eventhough it works for you with 2.4.22, perhaps 2.4.23 triggers a mysql bug which is fixed in more recent releases ?
Cheers,
Willy
Quoted 5 lines
one of my collegues had a server which occasionally crashed at night with mysql taking all the memory. I think it was with an old 2.4.18 kernel. He finally reinstalled all the machine and it never happened anymore. So eventhough it works for you with 2.4.22, perhaps 2.4.23 triggers a mysql bug which is fixed in more recent releases ?
Willy,
Hmm ... could be, but I don't think so. I use last mysql3 version and I have this problem without mysql too.
For example on this box there is no mysql (Piv 2.4GHz/512Mo):__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process watchdog__alloc_pages: 1-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process watchdog
# free
total used free shared buffers cached
Mem: 514468 508416 6052 0 11608 205464
-/+ buffers/cache: 291344 223124
Swap: 265032 77524 187508When I swithed more that 700 servers 10-15 days ago to 2.4.23, I saw that servers swaped less that with 2.4.22. So I believe VM was modified. Cool. Great job. Now servers begin to crash :/
Octave
Quoted 4 lines
How much swap do you have in your system?
This is happening because the system is unable to free memory (you probably ran out of swap for some reason).
Marcelo,
You can run this easy script. 2.4.19 takes about 30 minutes to kill all process. 2.4.23 takes about 60 minutes.
I think, server crashs when VM kills a process like watchdog (why ?).
Octave
# head -n 100000 /dev/urandom > file # cat > full.pl #!/usr/bin/perl
open (F,"file");@F=<F>;close(F);
for (;;) { push @F,@F; }
# chmod 755 full.pl # for i in `seq 1 100`; do ./full.pl & done [1] 767 [2] 768 [3] 769 [4] 770 [5] 771 [...]
# tail -f /var/log/messages Dec 21 18:55:32 stock kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 21 18:55:32 stock kernel: VM: killing process full.pl Dec 21 18:55:37 stock kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 21 18:55:37 stock kernel: VM: killing process full.pl
Quoted 23 lines · Octave
Quoted 5 lines
one of my collegues had a server which occasionally crashed at night with mysql taking all the memory. I think it was with an old 2.4.18 kernel. He finally reinstalled all the machine and it never happened anymore. So eventhough it works for you with 2.4.22, perhaps 2.4.23 triggers a mysql bug which is fixed in more recent releases ?
Willy,
Hmm ... could be, but I don't think so. I use last mysql3 version and I have this problem without mysql too.For example on this box there is no mysql (Piv 2.4GHz/512Mo):
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process watchdog__alloc_pages: 1-order allocation failed (gfp=0x1f0/0)__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process watchdog# free total used free shared buffers cached Mem: 514468 508416 6052 0 11608 205464 -/+ buffers/cache: 291344 223124 Swap: 265032 77524 187508
Quoted 3 lines
When I swithed more that 700 servers 10-15 days ago to 2.4.23, I saw that servers swaped less that with 2.4.22. So I believe VM was modified. Cool. Great job. Now servers begin to crash :/
Octave,
Can you please "echo 1 > /proc/sys/vm_gfp_debug" (to turn VM debugging on) and rerun the tests which crash the box.
Also run "vmstat 5" in the background and save that to a file.
Quoted 2 lines · On Dec-21 2003, Sun, 19:47 +0100 Octave <[email protected]> wrote:
You can run this easy script. 2.4.19 takes about 30 minutes to kill all process. 2.4.23 takes about 60 minutes.
Can you also try 2.4.24-pre1 with the OOM killer enabled?
Signature · 1 line
Tomas Szepe <[email protected]>
Quoted 7 lines
Octave,
Can you please "echo 1 >
/proc/sys/vm_gfp_debug" (to turn VM debugging on) and rerun the tests which crash the box.Also run "vmstat 5" in the background and save that to a file.
Marcelo,
I got this kernel panic:
Dec 21 20:04:34 stock kernel: e9f01e50 c012e1d8 000001d2 08451094 00000001 e7493ea0 0000040e 00000000 Dec 21 20:04:34 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:04:34 stock kernel: 00104025 c01225a3 e7493ea0 08451094 00000001 e9efec60 f7ec5380 c01226bf Dec 21 20:04:34 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c01225a3>] [<c01226bf>] [<c01228e8>] Dec 21 20:04:34 stock kernel: [<c0111707>] [<c011157c>] [<c0123db0>] [<c0122d85>] [<c0106fa0>] Dec 21 20:04:52 stock kernel: f7313e18 c012e1d8 000001d2 f770c184 00000020 f7f0ffe0 0000040e 00000000 Dec 21 20:04:52 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:04:52 stock kernel: 00000012 c0124e80 00000007 00000020 0000004a f717e920 f717e920 c0124efe Dec 21 20:04:52 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c0124e80>] [<c0124efe>] [<c0126793>] Dec 21 20:04:52 stock kernel: [<c01226e5>] [<c01228e8>] [<c0111707>] [<c011157c>] [<c019047c>] [<c0135190>] Dec 21 20:04:52 stock kernel: [<c0106fa0>] Dec 21 20:05:01 stock kernel: dba23e50 c012e1d8 000001d2 094de03c 00000001 e6461500 0000040e 00000000 Dec 21 20:05:01 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:01 stock kernel: 00104025 c01225a3 e6461500 094de03c 00000001 c23eea80 c10e6f90 c01226bf Dec 21 20:05:01 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c01225a3>] [<c01226bf>] [<c01228e8>] Dec 21 20:05:01 stock kernel: [<c0111707>] [<c011157c>] [<c0123db0>] [<c0122d85>] [<c0106fa0>] Dec 21 20:05:42 stock kernel: c8525e50 c012e1d8 000001d2 08cb90f4 00000001 e14a2f20 0000040e 00000000 Dec 21 20:05:42 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:42 stock kernel: 00104025 c01225a3 e14a2f20 08cb90f4 00000001 df9fc420 c138dbc0 c01226bf Dec 21 20:05:42 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c01225a3>] [<c01226bf>] [<c01228e8>] Dec 21 20:05:42 stock kernel: [<c0111707>] [<c011157c>] [<c0123db0>] [<c0122d85>] [<c0106fa0>] Dec 21 20:05:42 stock kernel: d70ebedc c012e1d8 000001d2 00000000 00000000 f71aa544 00000414 00000000 Dec 21 20:05:42 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:42 stock kernel: 00000000 c012595a 00000000 ffffffea f71b0ba0 00000000 f7fb29cc 00001000 Dec 21 20:05:42 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c012595a>] [<c0125f1a>] [<c0125d9c>] Dec 21 20:05:42 stock kernel: [<c0135126>] [<c0106eb7>] Dec 21 20:05:42 stock kernel: e3e97edc c012e1d8 000001d2 00000000 00000000 f71aa544 00000414 00000000 Dec 21 20:05:42 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:42 stock kernel: 00000000 c012595a 00000000 ffffffea f6ee63c0 00000000 f7fb31f4 00001000 Dec 21 20:05:42 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c012595a>] [<c0125f1a>] [<c0125d9c>] Dec 21 20:05:42 stock kernel: [<c0135126>] [<c0106eb7>]
Warning (Oops_read): Code line not seen, dumping what data is available
Trace; c012e1d8 <__alloc_pages+2a4/2b0>
Trace; c012dd80 <_alloc_pages+18/1c>
Trace; c01225a3 <do_anonymous_page+3b/124>
Trace; c01226bf <do_no_page+33/200>
Trace; c01228e8 <handle_mm_fault+5c/b8>
Trace; c0111707 <do_page_fault+18b/4cd>
Trace; c011157c <do_page_fault+0/4cd>
Trace; c0123db0 <do_brk+130/230>
Trace; c0122d85 <sys_brk+c1/f0>
Trace; c0106fa0 <error_code+34/3c>
Trace; c012e1d8 <__alloc_pages+2a4/2b0>
Trace; c012dd80 <_alloc_pages+18/1c>
Trace; c0124e80 <page_cache_read+7c/d0>
Trace; c0124efe <read_cluster_nonblocking+2a/40>
Trace; c0126793 <filemap_nopage+13f/23c>
Trace; c01226e5 <do_no_page+59/200>
Trace; c01228e8 <handle_mm_fault+5c/b8>
Trace; c0111707 <do_page_fault+18b/4cd>
Trace; c011157c <do_page_fault+0/4cd>
Trace; c019047c <tty_read+dc/124>
Trace; c0135190 <sys_read+100/108>
Trace; c0106fa0 <error_code+34/3c>
Trace; c012e1d8 <__alloc_pages+2a4/2b0>
Trace; c012dd80 <_alloc_pages+18/1c>
Trace; c01225a3 <do_anonymous_page+3b/124>
Trace; c01226bf <do_no_page+33/200>
Trace; c01228e8 <handle_mm_fault+5c/b8>
Trace; c0111707 <do_page_fault+18b/4cd>
Trace; c011157c <do_page_fault+0/4cd>
Trace; c0123db0 <do_brk+130/230>
Trace; c0122d85 <sys_brk+c1/f0>
Trace; c0106fa0 <error_code+34/3c>
Trace; c012e1d8 <__alloc_pages+2a4/2b0>
Trace; c012dd80 <_alloc_pages+18/1c>
Trace; c01225a3 <do_anonymous_page+3b/124>
Trace; c01226bf <do_no_page+33/200>
Trace; c01228e8 <handle_mm_fault+5c/b8>
Trace; c0111707 <do_page_fault+18b/4cd>
Trace; c011157c <do_page_fault+0/4cd>
Trace; c0123db0 <do_brk+130/230>
Trace; c0122d85 <sys_brk+c1/f0>
Trace; c0106fa0 <error_code+34/3c>
Trace; c012e1d8 <__alloc_pages+2a4/2b0>
Trace; c012dd80 <_alloc_pages+18/1c>
Trace; c012595a <do_generic_file_read+356/488>
Trace; c0125f1a <generic_file_read+96/198>
Trace; c0125d9c <file_read_actor+0/e8>
Trace; c0135126 <sys_read+96/108>
Trace; c0106eb7 <system_call+2f/34>
Trace; c012e1d8 <__alloc_pages+2a4/2b0>
Trace; c012dd80 <_alloc_pages+18/1c>
Trace; c012595a <do_generic_file_read+356/488>
Trace; c0125f1a <generic_file_read+96/198>
Trace; c0125d9c <file_read_actor+0/e8>
Trace; c0135126 <sys_read+96/108>
Trace; c0106eb7 <system_call+2f/34>
Quoted 43 lines · Octave
Quoted 7 lines
Octave,
Can you please "echo 1 >
/proc/sys/vm_gfp_debug" (to turn VM debugging on) and rerun the tests which crash the box.Also run "vmstat 5" in the background and save that to a file.
Marcelo,
I got this kernel panic:
Dec 21 20:04:34 stock kernel: e9f01e50 c012e1d8 000001d2 08451094 00000001 e7493ea0 0000040e 00000000 Dec 21 20:04:34 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:04:34 stock kernel: 00104025 c01225a3 e7493ea0 08451094 00000001 e9efec60 f7ec5380 c01226bf Dec 21 20:04:34 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c01225a3>] [<c01226bf>] [<c01228e8>] Dec 21 20:04:34 stock kernel: [<c0111707>] [<c011157c>] [<c0123db0>] [<c0122d85>] [<c0106fa0>] Dec 21 20:04:52 stock kernel: f7313e18 c012e1d8 000001d2 f770c184 00000020 f7f0ffe0 0000040e 00000000 Dec 21 20:04:52 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:04:52 stock kernel: 00000012 c0124e80 00000007 00000020 0000004a f717e920 f717e920 c0124efe Dec 21 20:04:52 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c0124e80>] [<c0124efe>] [<c0126793>] Dec 21 20:04:52 stock kernel: [<c01226e5>] [<c01228e8>] [<c0111707>] [<c011157c>] [<c019047c>] [<c0135190>] Dec 21 20:04:52 stock kernel: [<c0106fa0>] Dec 21 20:05:01 stock kernel: dba23e50 c012e1d8 000001d2 094de03c 00000001 e6461500 0000040e 00000000 Dec 21 20:05:01 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:01 stock kernel: 00104025 c01225a3 e6461500 094de03c 00000001 c23eea80 c10e6f90 c01226bf Dec 21 20:05:01 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c01225a3>] [<c01226bf>] [<c01228e8>] Dec 21 20:05:01 stock kernel: [<c0111707>] [<c011157c>] [<c0123db0>] [<c0122d85>] [<c0106fa0>] Dec 21 20:05:42 stock kernel: c8525e50 c012e1d8 000001d2 08cb90f4 00000001 e14a2f20 0000040e 00000000 Dec 21 20:05:42 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:42 stock kernel: 00104025 c01225a3 e14a2f20 08cb90f4 00000001 df9fc420 c138dbc0 c01226bf Dec 21 20:05:42 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c01225a3>] [<c01226bf>] [<c01228e8>] Dec 21 20:05:42 stock kernel: [<c0111707>] [<c011157c>] [<c0123db0>] [<c0122d85>] [<c0106fa0>] Dec 21 20:05:42 stock kernel: d70ebedc c012e1d8 000001d2 00000000 00000000 f71aa544 00000414 00000000 Dec 21 20:05:42 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:42 stock kernel: 00000000 c012595a 00000000 ffffffea f71b0ba0 00000000 f7fb29cc 00001000 Dec 21 20:05:42 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c012595a>] [<c0125f1a>] [<c0125d9c>] Dec 21 20:05:42 stock kernel: [<c0135126>] [<c0106eb7>] Dec 21 20:05:42 stock kernel: e3e97edc c012e1d8 000001d2 00000000 00000000 f71aa544 00000414 00000000 Dec 21 20:05:42 stock kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 21 20:05:42 stock kernel: 00000000 c012595a 00000000 ffffffea f6ee63c0 00000000 f7fb31f4 00001000 Dec 21 20:05:42 stock kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c012595a>] [<c0125f1a>] [<c0125d9c>] Dec 21 20:05:42 stock kernel: [<c0135126>] [<c0106eb7>]
Quoted 1 line
Warning (
Oops_read): Code line not seen, dumping what data is available
Octave,
This is not a kernel panic, its the VM debugging output.
Can you please apply the attached patch on top of 2.4.23 and rerun the test with "echo 1 > /proc/sys/vm_gfp_debug" ?
It will printout the number of available swap pages when processes get killed.
Thanks
Quoted 48 lines
Trace;
c012e1d8<__alloc_pages+2a4/2b0>
Trace;c012dd80<_alloc_pages+18/1c>
Trace;c01225a3<do_anonymous_page+3b/124>
Trace;c01226bf<do_no_page+33/200>
Trace;c01228e8<handle_mm_fault+5c/b8>
Trace;c0111707<do_page_fault+18b/4cd>
Trace;c011157c<do_page_fault+0/4cd>
Trace;c0123db0<do_brk+130/230>
Trace;c0122d85<sys_brk+c1/f0>
Trace;c0106fa0<error_code+34/3c>
Trace;c012e1d8<__alloc_pages+2a4/2b0>
Trace;c012dd80<_alloc_pages+18/1c>
Trace;c0124e80<page_cache_read+7c/d0>
Trace;c0124efe<read_cluster_nonblocking+2a/40>
Trace;c0126793<filemap_nopage+13f/23c>
Trace;c01226e5<do_no_page+59/200>
Trace;c01228e8<handle_mm_fault+5c/b8>
Trace;c0111707<do_page_fault+18b/4cd>
Trace;c011157c<do_page_fault+0/4cd>
Trace;c019047c<tty_read+dc/124>
Trace;c0135190<sys_read+100/108>
Trace;c0106fa0<error_code+34/3c>
Trace;c012e1d8<__alloc_pages+2a4/2b0>
Trace;c012dd80<_alloc_pages+18/1c>
Trace;c01225a3<do_anonymous_page+3b/124>
Trace;c01226bf<do_no_page+33/200>
Trace;c01228e8<handle_mm_fault+5c/b8>
Trace;c0111707<do_page_fault+18b/4cd>
Trace;c011157c<do_page_fault+0/4cd>
Trace;c0123db0<do_brk+130/230>
Trace;c0122d85<sys_brk+c1/f0>
Trace;c0106fa0<error_code+34/3c>
Trace;c012e1d8<__alloc_pages+2a4/2b0>
Trace;c012dd80<_alloc_pages+18/1c>
Trace;c01225a3<do_anonymous_page+3b/124>
Trace;c01226bf<do_no_page+33/200>
Trace;c01228e8<handle_mm_fault+5c/b8>
Trace;c0111707<do_page_fault+18b/4cd>
Trace;c011157c<do_page_fault+0/4cd>
Trace;c0123db0<do_brk+130/230>
Trace;c0122d85<sys_brk+c1/f0>
Trace;c0106fa0<error_code+34/3c>
Trace;c012e1d8<__alloc_pages+2a4/2b0>
Trace;c012dd80<_alloc_pages+18/1c>
Trace;c012595a<do_generic_file_read+356/488>
Trace;c0125f1a<generic_file_read+96/198>
Trace;c0125d9c<file_read_actor+0/e8>
Trace;c0135126<sys_read+96/108>
Quoted 10 lines
Trace;
c0106eb7<system_call+2f/34>
Trace;c012e1d8<__alloc_pages+2a4/2b0>
Trace;c012dd80<_alloc_pages+18/1c>
Trace;c012595a<do_generic_file_read+356/488>
Trace;c0125f1a<generic_file_read+96/198>
Trace;c0125d9c<file_read_actor+0/e8>
Trace;c0135126<sys_read+96/108>
Trace;c0106eb7<system_call+2f/34>
mm/page_alloc.c renamed from mm/page_alloc.c.orig +3 -1
@@ -436,8 +436,10 @@ out:printk (KERN_NOTICE "__alloc_pages: %u-order allocation failed (gfp=0x%x/%i)\n" , order, gfp_mask, !!(current->flags &PF_MEMALLOC )); -if (unlikely (vm_gfp_debug)) +if (unlikely (vm_gfp_debug)) { +printk (KERN_ERR "OOM: nr_swap_pages=%d" , nr_swap_pages);dump_stack (); + }return NULL ; }
Quoted 7 lines
This is not a kernel panic, its the VM debugging output.
Can you please apply the attached patch on top of 2.4.23 and rerun the test with "echo 1 >
/proc/sys/vm_gfp_debug" ?It will printout the number of available swap pages when processes get killed.
Marcelo,
How about this ?
Dec 21 22:08:44 stock kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 21 22:08:44 stock kernel: OOM: nr_swap_pages=0cd865e6c c012e1e8 c0262e3c 00000000 000001d2 00000000 00000001 cd863c00 Dec 21 22:08:44 stock kernel: 0000040e 00000000 00000018 00000002 c029e9d8 c029ead4 00000000 000001d2 Dec 21 22:08:44 stock kernel: 00000000 c012dd80 c11d3db0 c0121fe8 e0b12220 bffffb20 00000001 cd863c00 Dec 21 22:08:44 stock kernel: Call Trace: [__get_free_pages+4/24] [_alloc_pages+24/28] [do_wp_page+168/736] [handle_mm_fault+135/184] [do_page_fault+395/1229] Dec 21 22:08:44 stock kernel: Call Trace: [<c012e1e8>] [<c012dd80>] [<c0121fe8>] [<c0122913>] [<c0111707>] Dec 21 22:08:44 stock kernel: [do_page_fault+0/1229] [error_code+52/60] Dec 21 22:08:44 stock kernel: [<c011157c>] [<c0106fa0>] Dec 21 22:08:44 stock kernel: VM: killing process watchdog Dec 21 22:08:44 stock kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 21 22:08:44 stock kernel: OOM: nr_swap_pages=0f7767e6c c012e1e8 c0262e3c 00000000 000001d2 00000000 00000001 f79bf6e0 Dec 21 22:08:44 stock kernel: 0000040e 00000000 00000018 00000002 c029e9d8 c029ead4 00000000 000001d2 Dec 21 22:08:44 stock kernel: 00000000 c012dd80 c11d3db0 c0121fe8 f7be36c0 bffffb20 00000001 f79bf6e0 Dec 21 22:08:44 stock kernel: Call Trace: [__get_free_pages+4/24] [_alloc_pages+24/28] [do_wp_page+168/736] [_alloc_pages+24/28] [handle_mm_fault+135/184] Dec 21 22:08:44 stock kernel: Call Trace: [<c012e1e8>] [<c012dd80>] [<c0121fe8>] [<c012dd80>] [<c0122913>] Dec 21 22:08:45 stock kernel: [do_page_fault+395/1229] [do_page_fault+0/1229] [do_fork+1719/2028] [sys_fork+20/28] [error_code+52/60] Dec 21 22:08:45 stock kernel: [<c0111707>] [<c011157c>] [<c01150eb>] [<c0105a44>] [<c0106fa0>] Dec 21 22:08:45 stock kernel: VM: killing process watchdog
Octave
Quoted 8 lines · Octave
Quoted 7 lines
This is not a kernel panic, its the VM debugging output.
Can you please apply the attached patch on top of 2.4.23 and rerun the test with "echo 1 >
/proc/sys/vm_gfp_debug" ?It will printout the number of available swap pages when processes get killed.
Quoted 6 lines
Marcelo,
How about this ?
Dec 21 22:08:44 stock kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 21 22:08:44 stock kernel: OOM: nr_swap_pages=0cd865e6c c012e1e8 c0262e3c 00000000 000001d2 00000000 00000001 cd863c00
OK, so there's no available swap anymore (nr_swap_pages=0, Marcelo forgot the '\n' in the patch). I simply think that with other kernels, you're very short of memory, but it runs, while with this one, all memory gets consumed, and since there's no smart oom killer, one process has to get killed.
Cheers,
Willy
Quoted 7 lines · Tomas Szepe
Quoted 2 lines · On Dec-21 2003, Sun, 19:47 +0100 Octave <[email protected]> wrote:
You can run this easy script. 2.4.19 takes about 30 minutes to kill all process. 2.4.23 takes about 60 minutes.
Can you also try 2.4.24-pre1 with the OOM killer enabled?
I complied 2.4.24-pre1 with OOM killer. After 2 minutes of test, server is down.
Octave
Quoted 8 lines · you [Willy Tarreau]
Quoted 3 lines
Dec 21 22:08:44 stock kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 21 22:08:44 stock kernel: OOM: nr_swap_pages=0cd865e6c c012e1e8 c0262e3c 00000000 000001d2 00000000 00000001 cd863c00OK, so there's no available swap anymore (nr_swap_pages=0, Marcelo forgot the '\n' in the patch). I simply think that with other kernels, you're very short of memory, but it runs, while with this one, all memory gets consumed, and since there's no smart oom killer, one process has to get killed.
BTW, I have a box with 128MB ram and 512MB swap running 2.4.21-jam1 (it has the -aa vm). I can't shut it down cleanly, because trying it goes into endless loop trying to free memory when turning off swap. Nothing but alt-sysrq-b seems to work.
I don't know if there is a kernel memory leak, since all user level processes should be killed at that point, right? Unfortunately I didn't have time to dig deeper, as the box is in (sort of) production.
-- v --
Quoted 11 lines · Octave
Quoted 7 lines · Tomas Szepe
Quoted 2 lines · On Dec-21 2003, Sun, 19:47 +0100 Octave <[email protected]> wrote:
You can run this easy script. 2.4.19 takes about 30 minutes to kill all process. 2.4.23 takes about 60 minutes.
Can you also try 2.4.24-pre1 with the OOM killer enabled?
I complied 2.4.24-pre1 with OOM killer. After 2 minutes of test, server is down.
Hi Octave,
What do you mean with "server is down" ? The OOM killer killed an application ? What were the messages?
Under out of memory, 2.4.22 should also kill a process, but you say it doesnt.
Quoted 7 lines
Hi Octave,
What do you mean with "server is down" ? The OOM killer killed an application ? What were the messages?
Under out of memory, 2.4.22 should also kill a process, but you say it doesnt.
Marcelo,
All I have with
- 2.4.24-pre1 is
# echo 1 > /proc/sys/vm/vm_gfp_debug # for i in `seq 1 100`; do ./full.pl & done [1] 849 [2] 850 [...] # tail -f /var/log/messages [...]
SOFTDOG: Initiating system reboot.
LILO:
- 2.4.23
Dec 22 13:16:30 u8668 kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 22 13:16:30 u8668 kernel: f7465e74 c012e1d8 000001d2 00000000 00000001 c493e120 000003ef 00000000 Dec 22 13:16:30 u8668 kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 22 13:16:30 u8668 kernel: c1ab6bf0 c0121fe8 f5759d40 00125eac 00000001 c493e120 00000000 0003923f Dec 22 13:16:30 u8668 kernel: Call Trace: [raw_devices+7128/8192] [raw_devices+6016/8192] [buf.0+40/1024] [log_buf+275/16384] [device_list+2031/2032] Dec 22 13:16:30 u8668 kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c0121fe8>] [<c0122913>] [<c0111707>] Dec 22 13:16:30 u8668 kernel: [device_list+1636/2032] [dfont_unitable+352/608] Dec 22 13:16:30 u8668 kernel: [<c011157c>] [<c0106fa0>] Dec 22 13:16:30 u8668 kernel: VM: killing process full.pl Dec 22 13:16:30 u8668 kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 22 13:16:30 u8668 kernel: f7465d40 c012e1d8 000001d2 00000000 00000000 f79d79a4 000003ef 00000000 Dec 22 13:16:30 u8668 kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 22 13:16:34 u8668 kernel: 00000000 c012595a f7464000 c0000000 f7465eec 00000000 f7ddb788 00001000 Dec 22 13:16:34 u8668 kernel: Call Trace: [raw_devices+7128/8192] [raw_devices+6016/8192] [log_buf+12634/16384] [log_buf+14106/16384] [log_buf+13724/16384] Dec 22 13:16:34 u8668 kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c012595a>] [<c0125f1a>] [<c0125d9c>] Dec 22 13:16:34 u8668 kernel: [read_ahead+669/1020] [blk_dev+29672/35712] [blk_dev+31175/35712] [blk_dev+32524/35712] [devpts_root_inode_operations+67/80] [df ont_unitable+119/608] Dec 22 13:16:35 u8668 kernel: [<c0135dbd>] [<c013d328>] [<c013d907>] [<c013de4c>] [<c0105ac3>] [<c0106eb7>] Dec 22 13:16:35 u8668 kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 22 13:16:35 u8668 kernel: f729fe14 c012e1d8 000001d2 00128000 00000001 f77ef8e0 000003ef 00000000 Dec 22 13:16:35 u8668 kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 22 13:16:35 u8668 kernel: 00104025 c01225a3 f77ef8e0 00128000 00000001 f7317aa0 f7ce4800 c01226bf Dec 22 13:16:36 u8668 kernel: Call Trace: [raw_devices+7128/8192] [raw_devices+6016/8192] [printk_buf.1+451/1024] [printk_buf.1+735/1024] [log_buf+232/16384] Dec 22 13:16:36 u8668 kernel: Call Trace: [<c012e1d8>] [<c012dd80>] [<c01225a3>] [<c01226bf>] [<c01228e8>] Dec 22 13:16:36 u8668 kernel: [pidhash+2049/4096] [log_buf+704/16384] [log_buf+3230/16384] [DAC960_MessageLevelMap+30/32] [dfont_unitable+119/608] Dec 22 13:16:36 u8668 kernel: [<c0121781>] [<c0122ac0>] [<c012349e>] [<c010c6be>] [<c0106eb7>] Dec 22 13:16:36 u8668 kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 22 13:16:36 u8668 kernel: f729fe14 c012e1d8 000001d2 00128000 00000001 f77ef8e0 000003ef 00000000 Dec 22 13:16:36 u8668 kernel: 00000018 00000002 c029e998 c029ea94 00000000 000001d2 00000000 c012dd80 Dec 22 13:16:36 u8668 kernel: 00104025 c01225a3 f77ef8e0 00128000 00000001 f7317aa0 f7ce4800 c01226bf Dec 22 13:16:36 u8668 kernel: Call Trace: [raw_devices+7128/8192] [raw_devices+6016/8192] [printk_buf.1+451/1024] [printk_buf.1+735/1024] [log_buf+232/16384]
[...]
then nothing more.
- 2.4.22
# echo 1 > /proc/sys/vm/vm_gfp_debug bash: /proc/sys/vm/vm_gfp_debug: No such file or directory # echo 1 > /proc/sys/vm/ bdflush max-readahead min-readahead page-cluster kswapd max_map_count overcommit_memory pagetable_cache # tail -f /var/log/messages Dec 22 13:28:34 u8668 kernel: Out of Memory: Killed process 441 (named). Dec 22 13:28:34 u8668 kernel: Out of Memory: Killed process 443 (named). Dec 22 13:28:34 u8668 kernel: Out of Memory: Killed process 444 (named). Dec 22 13:28:34 u8668 kernel: Out of Memory: Killed process 445 (named). Dec 22 13:28:34 u8668 kernel: Out of Memory: Killed process 446 (named). Dec 22 13:28:34 u8668 kernel: Out of Memory: Killed process 447 (named). Dec 22 13:28:42 u8668 kernel: Out of Memory: Killed process 750 (mysqld). Dec 22 13:28:42 u8668 kernel: Out of Memory: Killed process 760 (mysqld). Dec 22 13:28:42 u8668 kernel: Out of Memory: Killed process 761 (mysqld). Dec 22 13:28:48 u8668 kernel: Out of Memory: Killed process 636 (httpd). Dec 22 13:28:57 u8668 kernel: Out of Memory: Killed process 637 (httpd). Dec 22 13:29:03 u8668 kernel: Out of Memory: Killed process 638 (httpd). Dec 22 13:29:14 u8668 kernel: Out of Memory: Killed process 639 (httpd). SOFTDOG: Initiating system reboot.
Thanks for help
Octave
Quoted 20 lines · Octave
Quoted 7 lines
Hi Octave,
What do you mean with "server is down" ? The OOM killer killed an application ? What were the messages?
Under out of memory, 2.4.22 should also kill a process, but you say it doesnt.
Marcelo,
All I have with
- 2.4.24-pre1 is# echo 1 > /proc/sys/vm/vm_gfp_debug # for i in `seq 1 100`; do ./full.pl & done [1] 849 [2] 850 [...] # tail -f /var/log/messages [...]
Quoted 1 line
SOFTDOG: Initiating system reboot.
your softdog is too strict for the workload you're running. You can't pretend a low latency scheduling behaviour with hundres oom. what you see is perfectly normal.
Quoted 8 lines · Ville Herva
BTW, I have a box with 128MB ram and 512MB swap running 2.4.21-jam1 (it has the -aa vm). I can't shut it down cleanly, because trying it goes into endless loop trying to free memory when turning off swap. Nothing but alt-sysrq-b seems to work.
I don't know if there is a kernel memory leak, since all user level processes should be killed at that point, right? Unfortunately I didn't have time to dig deeper, as the box is in (sort of) production.
Maybe, it depends on your init scripts. Does your distribution do a kill -9 of all processes before turning off swap?
Quoted 10 lines · Ville Herva
Quoted 8 lines · you [Willy Tarreau]
Quoted 3 lines
Dec 21 22:08:44 stock kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 21 22:08:44 stock kernel: OOM: nr_swap_pages=0cd865e6c c012e1e8 c0262e3c 00000000 000001d2 00000000 00000001 cd863c00OK, so there's no available swap anymore (nr_swap_pages=0, Marcelo forgot the '\n' in the patch). I simply think that with other kernels, you're very short of memory, but it runs, while with this one, all memory gets consumed, and since there's no smart oom killer, one process has to get killed.
Quoted 8 lines
BTW, I have a box with 128MB ram and 512MB swap running 2.4.21-jam1 (it has the -aa vm). I can't shut it down cleanly, because trying it goes into endless loop trying to free memory when turning off swap. Nothing but alt-sysrq-b seems to work.
I don't know if there is a kernel memory leak, since all user level processes should be killed at that point, right? Unfortunately I didn't have time to dig deeper, as the box is in (sort of) production.
if this is a leak, I doubt it has been introduced recently, the only swap accounting related change was in the shm layer, and it was supposed to be a race fix. You may want to check if you've some shm allocated in /dev/shm or ipcs, while the machine reboots.
Quoted 7 lines · you [Mike Fedyk]
Quoted 4 lines
I don't know if there is a kernel memory leak, since all user level processes should be killed at that point, right? Unfortunately I didn't have time to dig deeper, as the box is in (sort of) production.
Maybe, it depends on your init scripts. Does your distribution do a kill -9 of all processes before turning off swap?
(It's a 7.0 Red Hat).
It does
runcmd "Sending all processes the KILL signal..." /sbin/killall5 -9
before
[ -n "$SWAPS" ] && runcmd "Turning off swap: " swapoff $SWAPS
in /etc/rc6.d/S01reboot and I've seen the "Sending all processes the KILL signal..." message appear before the memory freeing loop starts rolling.
-- v --
Quoted 7 lines · you [Andrea Arcangeli]
Quoted 6 lines · Ville Herva
BTW, I have a box with 128MB ram and 512MB swap running 2.4.21-jam1 (it has the -aa vm). I can't shut it down cleanly, because trying it goes into endless loop trying to free memory when turning off swap. Nothing but alt-sysrq-b seems to work.
Quoted 8 lines
Quoted 3 lines
I don't know if there is a kernel memory leak, since all user level processes should be killed at that point, right? Unfortunately I didn't have time to dig deeper, as the box is in (sort of) production.
if this is a leak, I doubt it has been introduced recently, the only swap accounting related change was in the shm layer, and it was supposed to be a race fix. You may want to check if you've some shm allocated in
/dev/shmor ipcs, while the machine reboots.
I'll check that the next time I reboot.
Unfortunately the box is in (semi-)production so I can't reboot it at will.
-- v --
Quoted 11 lines · Ville Herva
Quoted 7 lines · you [Mike Fedyk]
Quoted 4 lines
I don't know if there is a kernel memory leak, since all user level processes should be killed at that point, right? Unfortunately I didn't have time to dig deeper, as the box is in (sort of) production.
Maybe, it depends on your init scripts. Does your distribution do a kill -9 of all processes before turning off swap?
(It's a 7.0 Red Hat).
Quoted 7 lines
It does
runcmd "Sending all processes the KILL signal..."/sbin/killall5
-9 before
[ -n "$SWAPS" ] && runcmd "Turning off swap: " swapoff $SWAPS
in/etc/rc6.d/S01rebootand I've seen the "Sending all processes the KILL signal..." message appear before the memory freeing loop starts rolling.
If its a pristine rh7.0 install, that version of bind has a notorious rootkit hole. So I wonder if the machine has been kitted by some script kiddie whose good at covering his tracks but not the rest of the housekeeping. Do a google search for "chkrootkit", and install it to get a better view of that possibility.
An OS upgrade does seem to be in order, lots has happened since 7.0. 7.3 with an updated kernel is my firewall, uptime is about 95 days now. It was shut down while I was out of state for a couple of months last fall. RH8.0 on this machine, with the real KDE-3.1.1a, but since RH is gonna force a switch, debian may be the next thing installed here. Or maybe Mepis since he's only 50 miles up the interstate from me. :)
Quoted 8 lines
-- v --
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 7 lines
Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.22% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attornies please note, additions to this message by Gene Heskett are: Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
Quoted 11 lines · you [Gene Heskett]
Quoted 10 lines
(It's a 7.0 Red Hat).
It does
runcmd "Sending all processes the KILL signal..."/sbin/killall5
-9 before
[ -n "$SWAPS" ] && runcmd "Turning off swap: " swapoff $SWAPS
in/etc/rc6.d/S01rebootand I've seen the "Sending all processes the KILL signal..." message appear before the memory freeing loop starts rolling.
Quoted 2 lines
If its a pristine rh7.0 install, that version of bind has a notorious rootkit hole.
Pristine - hell no. None of my install are pristine :).
Bind - hell no. I may be^W^Wam stupid, but does *anyone* put boxes in production without disabling services first?
Quoted 3 lines
So I wonder if the machine has been kitted by some script kiddie whose good at covering his tracks but not the rest of the housekeeping.
Uh, it may be rooted allright, but I seriously doubt that is cause of this symptom.
Quoted 1 line
An OS upgrade does seem to be in order, lots has happened since 7.0.
I know that, but I'm not exactly lacking interesting admin things to do so that I would run around upgrading all the boxes there are lying around on the corners each time a distro upgrade comes up. The box works for what it is supposed to do (a miracle, that, considering it's a 200MHz/64MB screamer), it's behind a fw, most services are disabled. And it's actually being phased out of production anyway.
Sure, if I had endlessly time on my hands, I'd upgrade the distro, but right now, I'll have to settle with just keeping the crucial services up-to-date.
Quoted 2 lines
7.3 with an updated kernel is my firewall, uptime is about 95 days now.
No problem with getting good uptimes with this kernel (2.4.21-jam1). I only had to boot it to install the do_brk patch.
Quoted 2 lines
It was shut down while I was out of state for a couple of months last fall.
Longest uptimes I've got are with 2.0 and 2.2 so far. And I have a number of RH 6.2 distros running still. I see no fundamental problem with an old distro (as long as you know the down sides (keep the services up-to-date, no
Quoted 1 line
2GB file support with all applications etc etc).
-- v --
Quoted 3 lines
your softdog is too strict for the workload you're running. You can't pretend a low latency scheduling behaviour with hundres oom. what you see is perfectly normal.
You are right. I can see oom killer's work without watchdog. It takes 1 hour before all bad process are killed, since oom kills first the process with shared memory (named, httpd, mysql etc), then the bad process. I think it's the right way for general used.
Thanks
Octave