2.6.12.2 dies after 24 hours
As background, we've been using a relatively old kernel (2.6.4-mm2) on some IBM x235 machines with 6G of RAM, umem cards, and serveraid storage. These machines are under continuous heavy-ish load, load avg between about 1 and 5, with between 2500-3500 procs at all times, with several largish ReiserFS partitions and have been running *really* well with >250 days uptime on one machine.
We recently tried upgrading one of the machines to the latest kernel (2.6.12.2) and it's died after about 24 hours. It seemed to end up in some weird state where we could ssh into it, and some commands worked (eg uptime) but process list related commands (ps) would just freeze up into an unkillable state and we'd have to close the seesion and ssh in again.
I did manage to get a full sysrq-t dump. I've placed it, a kernel config dump, and a dmesg boot dump here:
http://robm.fastmail.fm/kernel/t7/
Hope this provides some useful data to track down the problem.
Rob
PS. Yes, I know this is a non-PAE kernel on a 6G machine so 2G was unused. This was a mistake in this case, but it still doesn't explain the crash...
Quoted 7 lines · Rob Mueller
As background, we've been using a relatively old kernel (2.6.4-mm2) on some IBM x235 machines with 6G of RAM, umem cards, and serveraid storage. These machines are under continuous heavy-ish load, load avg between about 1 and 5, with between 2500-3500 procs at all times, with several largish ReiserFS partitions and have been running *really* well with >250 days uptime on one machine.
Quoted 5 lines
We recently tried upgrading one of the machines to the latest kernel (2.6.12.2) and it's died after about 24 hours. It seemed to end up in some weird state where we could ssh into it, and some commands worked (eg uptime) but process list related commands (ps) would just freeze up into an unkillable state and we'd have to close the seesion and ssh in again.
I experienced the exact same thing on a IBM 335 - in my case I had messed up with the ACPI setup. Could you paste the output from /proc/interupts also is your kernel running with IRQ balancing ?.
Regards.
Lars Roland
Quoted 10 lines
Quoted 6 lines
We recently tried upgrading one of the machines to the latest kernel (2.6.12.2) and it's died after about 24 hours. It seemed to end up in some weird state where we could ssh into it, and some commands worked (eg uptime) but process list related commands (ps) would just freeze up into an unkillable state and we'd have to close the seesion and ssh in again.
I experienced the exact same thing on a IBM 335 - in my case I had messed up with the ACPI setup. Could you paste the output from
/proc/interuptsalso is your kernel running with IRQ balancing ?.
Here's the /proc/interrupts dump:
CPU0 CPU1 CPU2 CPU3
0: 11524000 0 0 0 IO-APIC-edge timer 1: 8 0 0 0 IO-APIC-edge i8042 5: 0 0 0 0 IO-APIC-level acpi 14: 13 0 0 0 IO-APIC-edge ide0 16: 2 0 0 0 IO-APIC-level ibmasm0 20: 2978604 0 2338027 0 IO-APIC-level eth0 22: 1321957 0 0 0 IO-APIC-level ips 24: 581291 0 0 0 IO-APIC-level pci-umem 29: 257154 0 0 0 IO-APIC-level eth1 NMI: 0 0 0 0 LOC: 11524185 11524201 11524194 11524121 ERR: 0 MIS: 0
I'm not sure about IRQ balancing sorry. How do I tell? The entire boot process output is here:
http://robm.fastmail.fm/kernel/t7/bootdmesg.txt
And the config is here:
http://robm.fastmail.fm/kernel/t7/config.txt
Does that help?
Our boot doesn't pass any special parameters, just choosing the deadline elevator...
image=/boot/bzImage-2.6.12.2
label=linux-2.6.12.2
append="elevator=deadline"
read-only
root=/dev/sda2
Thanks for your help!
Rob
Quoted 16 lines · Rob Mueller
Here's the
/proc/interruptsdump:CPU0 CPU1 CPU2 CPU3
0: 11524000 0 0 0 IO-APIC-edge timer 1: 8 0 0 0 IO-APIC-edge i8042 5: 0 0 0 0 IO-APIC-level acpi 14: 13 0 0 0 IO-APIC-edge ide0 16: 2 0 0 0 IO-APIC-level ibmasm0 20: 2978604 0 2338027 0 IO-APIC-level eth0 22: 1321957 0 0 0 IO-APIC-level ips 24: 581291 0 0 0 IO-APIC-level pci-umem 29: 257154 0 0 0 IO-APIC-level eth1 NMI: 0 0 0 0 LOC: 11524185 11524201 11524194 11524121 ERR: 0 MIS: 0
Looks fine to me
Quoted 9 lines
I'm not sure about IRQ balancing sorry. How do I tell? The entire boot process output is here:
http://robm.fastmail.fm/kernel/t7/bootdmesg.txt
And the config is here:
You have irq balancing, the line
CONFIG_IRQBALANCE=y
in your config file confirms it - I am not completely sure that it is the root of the problem but when I experienced the problem I changed two things: my acpi code and irq balancing and one of then made the difference, I am just to lazy to check which one it is (also it is production servers so I cannot do whatever I want).
Quoted 8 lines
Our boot doesn't pass any special parameters, just choosing the deadline elevator...
image=/boot/bzImage-2.6.12.2
label=linux-2.6.12.2
append="elevator=deadline"
read-only
root=/dev/sda2
I use the same io scheduler so that should not be a problem. I have uploaded my config file - it works on ibm 335/336 servers, and a quick look at your boot msg seams to indicate that your server have some of the same hardware - note however that I load ide/scsi/filesystem stuff as modules so you will need to build a initrd to use my config.
the config is here
http://randompage.org/static/kernel.conf
--
Lars
Quoted 9 lines · Lars Roland
You have irq balancing, the line
CONFIG_IRQBALANCE=yin your config file confirms it - I am not completely sure that it is the root of the problem but when I experienced the problem I changed two things: my acpi code and irq balancing and one of then made the difference, I am just to lazy to check which one it is (also it is production servers so I cannot do whatever I want).
Our ACPI looked very similar to yours, so I've disabled IRQBALANCE.
We'll be rebooting the server during a less busy time to try the new kernel, so not for about 12 hours or so.
Quoted 11 lines
Quoted 1 line
append="elevator=deadline"
I use the same io scheduler so that should not be a problem. I have uploaded my config file - it works on ibm 335/336 servers, and a quick look at your boot msg seams to indicate that your server have some of the same hardware - note however that I load ide/scsi/filesystem stuff as modules so you will need to build a initrd to use my config.
the config is here
Great, thanks for that. I've had a look through and I think the IRQBALANCE issue is the most likely cause.
We're also applying the attached patch. There's a bug in reiserfs that gets tickled by our huge MMAP usage (it's amazing what really busy Cyrus daemons can do to a server, ouch). It's fixed in generic_write, so we take the few percent performance hit for something that doesn't break!
Bron.
Signature · 2 lines
Bron Gondwana [email protected]
Quoted 5 lines · Bron Gondwana
We're also applying the attached patch. There's a bug in reiserfs that gets tickled by our huge MMAP usage (it's amazing what really busy Cyrus daemons can do to a server, ouch). It's fixed in
generic_write, so we take the few percent performance hit for something that doesn't break!
Interesting - When I got the problem it was on mail servers under high load (handling 60.000 emails pr. hour) with reiserfs as file system. I have seen this problem on 5 different servers so I am confident that it is not hardware failure.
Sometimes the server load just rises and then the server dies other times the load rises but the kernel manages to get it back alive filling up syslog with messages like this
---------
June 29 14:06:59 dkcphmx12 kernel: oom-killer: gfp_mask=0xd2 June 29 14:07:15 dkcphmx12 kernel: DMA per-cpu:
June 29 14:07:24 dkcphmx12 kernel: cpu 0 hot: low 2, high 6, batch 1 June 29 14:07:24 dkcphmx12 kernel: cpu 0 cold: low 0, high 2, batch 1 June 29 14:07:26 dkcphmx12 logger[17427]: *** SYSTEM UPDATE STARTED *** June 29 14:07:26 dkcphmx12 kernel: cpu 1 hot: low 2, high 6, batch 1 June 29 14:07:26 dkcphmx12 kernel: cpu 1 cold: low 0, high 2, batch 1 June 29 14:07:26 dkcphmx12 kernel: cpu 2 hot: low 2, high 6, batch 1 June 29 14:07:26 dkcphmx12 kernel: cpu 2 cold: low 0, high 2, batch 1 June 29 14:07:26 dkcphmx12 kernel: cpu 3 hot: low 2, high 6, batch 1 June 29 14:07:26 dkcphmx12 kernel: cpu 3 cold: low 0, high 2, batch 1 June 29 14:07:26 dkcphmx12 kernel: Normal per-cpu:
June 29 14:07:26 dkcphmx12 kernel: cpu 0 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 0 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 1 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 1 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 2 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 2 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 3 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 3 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel: HighMem per-cpu:
June 29 14:07:26 dkcphmx12 kernel: cpu 0 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 0 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 1 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 1 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 2 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 2 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 3 hot: low 62, high 186, batch 31 June 29 14:07:26 dkcphmx12 kernel: cpu 3 cold: low 0, high 62, batch 31 June 29 14:07:26 dkcphmx12 kernel:
June 29 14:07:26 dkcphmx12 kernel: Free pages: 49196kB (496kB HighMem)
June 29 14:07:26 dkcphmx12 kernel: Active:246580 inactive:244789 dirty:0 writeback:0 unstable:0 free:12299 slab:4271 mapped:494975 pagetables:2332
June 29 14:07:26 dkcphmx12 kernel: DMA free:8192kB min:68kB low:84kB high:100kB active:2644kB inactive:2108kB present:16384kB pages_scanned:35654 all_unreclaimable? yes
June 29 14:07:26 dkcphmx12 kernel: lowmem_reserve[]: 0 880 2031 June 29 14:07:26 dkcphmx12 kernel: Normal free:40508kB min:3756kB low:4692kB high:5632kB active:400096kB inactive:396232kB present:901120kB pages_scanned:1933718 all_unreclaimable? yes
June 29 14:07:26 dkcphmx12 kernel: lowmem_reserve[]: 0 0 9214 June 29 14:07:26 dkcphmx12 kernel: HighMem free:496kB min:512kB low:640kB high:768kB active:583452kB inactive:580816kB present:1179504kB pages_scanned:8651915 all_unreclaimable? yes
June 29 14:07:26 dkcphmx12 kernel: lowmem_reserve[]: 0 0 0 June 29 14:07:26 dkcphmx12 kernel: DMA: 0*4kB 2*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8192kB
June 29 14:07:26 dkcphmx12 kernel: Normal: 251*4kB 38*8kB 6*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 9*4096kB = 40508kB
June 29 14:07:26 dkcphmx12 kernel: HighMem: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 496kB
June 29 14:07:26 dkcphmx12 kernel: Swap cache: add 958067, delete 958073, find 182890/223964, race 0+1251
June 29 14:07:26 dkcphmx12 kernel: Free swap = 0kB June 29 14:07:26 dkcphmx12 kernel: Total swap = 2097136kB
Quoted 7 lines
Quoted 6 lines
We're also applying the attached patch. There's a bug in reiserfs that gets tickled by our huge MMAP usage (it's amazing what really busy Cyrus daemons can do to a server, ouch). It's fixed in
generic_write, so we take the few percent performance hit for something that doesn't break!
Quoted 8 lines
Interesting - When I got the problem it was on mail servers under high load (handling 60.000 emails pr. hour) with reiserfs as file system. I have seen this problem on 5 different servers so I am confident that it is not hardware failure.
Sometimes the server load just rises and then the server dies other times the load rises but the kernel manages to get it back alive filling up syslog with messages like this
Sounds like a different issue. The patch Bron included before fixes (or at least reduces to the point where it fixes it for us) a problem where processes get stuck in D state and are unkillable. A reboot is required to remove them. Apparently this is a known bug in ReiserFS (see messages below). As noted, the same bug exists in ext3. There appears to have been some patches to try and fix it for both reiserfs and ext3, but I'm not sure if they're in the mainline kernel yet.
http://www.ussg.iu.edu/hypermail/linux/kernel/0409.0/2056.html
http://hulllug.principalhosting.net/archive/index.php/t-22774.html
Rob
----- Original Message ----- From: "Vladimir Saveliev" <[email protected]>
To: "Jeremy Howard" <[email protected]>
Cc: "Hans Reiser" <[email protected]>; <[email protected]>
Sent: Friday, October 08, 2004 4:57 PM
Subject: Re: URGENT: Need fix for problem with copy_from_user inside a transaction
Quoted 61 lines
Hello
Quoted 7 lines · Jeremy Howard
Quoted 1 line · Hans Reiser
Chris, can you comment please?
Also... can you guys suggest any ways to minimise the problem, e.g. external vs internal journal? metadata vs full journalling? changing the elevator? ...
Would you please check whether the attached patch for
./fs/reiserfs/file.cfixes the problem?Quick benchmark shows that using
generic_file_writedoes not hurt reiserfs performance too much comparing to using originalreiserfs_file_write.---Sequential Output (nosync)--- ---Sequential
Input-- --Rnd Seek--Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)- MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU generic_file_write 256 16690 99.7 28332 29.9 11528 16.8 12411 77.2 28888 17.3 237.4 2.1 reiserfs_file_write 256 15647 86.8 30875 22.1 10822 14.1 11631 72.1 29184 16.1 250.0 2.0Quoted 19 lines
Quoted 19 lines
Quoted 16 lines · Vladimir Saveliev
Hello
Quoted 4 lines · Hans Reiser
vs, this is not enough of an answer to share your understanding of the problem. Please say much more.
Reiserfs_write_filelocks set of pages, and then tries to copy data to them. If it is to copy data from one of pages which are locked and if that page is not uptodate, pagefault requires to lock that page, but as it is locked already - process deadlocks with itself.Quoted 8 lines
Quoted 8 lines
Is this when copying from one file in a formatted node to another file in that node?
Quoted 5 lines
As Chris said - fix is not trivial. Also, it is known that he did already something about it, so, I thought that it would make sence to find first what is his state at this problem.
Content-Disposition: attachment; filename=file.c.diff2
file.c 2004-10-08 10:03:03.001561661 +0400 line 1144
Quoted 8 lines
file.c 2004-10-08 10:03:03.001561661 +0400 renamed from file.c~ 2004-10-02 12:29:33.223660850 +0400 +2
-0@@ -1137,6 +1137,8 @@ return result; } + return generic_file_write(file, buf, count, ppos); + if ( unlikely((ssize_t) count < 0 )) return -EINVAL;
----- Original Message ----- From: "Chris Mason" <[email protected]>
To: "Hans Reiser" <[email protected]>
Cc: "Vladimir Saveliev" <[email protected]>; "Oleg Drokin" <[email protected]>; "Jeremy Howard" <[email protected]>; <[email protected]>
Sent: Saturday, October 09, 2004 1:05 AM
Subject: Re: URGENT: Need fix for problem with copy_from_user inside atransaction
Quoted 40 lines · Hans Reiser
Quoted 8 lines
No, this is not the right answer.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
file.c 2004-10-08 10:03:03.001561661 +0400 renamed from file.c~ 2004-10-02 12:29:33.223660850 +0400 +0
-0@@ -1137,6 +1137,8 @@ return result; }Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
file.c 2004-10-08 10:03:03.001561661 +0400 line 1143
Quoted 4 lines
file.c 2004-10-08 10:03:03.001561661 +0400 renamed from file.c~ 2004-10-02 12:29:33.223660850 +0400 +2
-0+ return generic_file_write(file, buf, count, ppos); + if ( unlikely((ssize_t) count < 0 )) return -EINVAL;It's not right because ext3 has exactly the same bug. The only real solution is to change the order in which things happen during
file_write.Right now, we do this:
prepare_write()allocates space, starts a transactioncopy_from_user()can deadlock here because a transaction is runningcommit_write()end the transactionThis is the same in
generic_file_writeandreiserfs_file_write, althoughreiserfs_file_writedoesn't callreiserfs_prepare_write, it has the same basic structure in terms of when the transaction starts and ends.The solution is to move most of the work from
reiserfs_prepare_writeintoreiserfs_commit_write. I've made a number of different patches for this, all have had problems, but I'm working through it and will have a real tested fix.Jeremy the best thing you can do right now is to mount your filesystems with -o nolargeio=1, and use the temporary workarounds I sent you before. The -o nolargeio=1 will reduce the amount of work we try to do with each pass in
reiserfs_file_write(note that Vladimir's patch above will have very similar effects).-chris
Quoted 23 lines · Rob Mueller
Quoted 6 lines
Quoted 5 lines
We're also applying the attached patch. There's a bug in reiserfs that gets tickled by our huge MMAP usage (it's amazing what really busy Cyrus daemons can do to a server, ouch). It's fixed in
generic_write, so we take the few percent performance hit for something that doesn't break!Quoted 8 lines
Interesting - When I got the problem it was on mail servers under high load (handling 60.000 emails pr. hour) with reiserfs as file system. I have seen this problem on 5 different servers so I am confident that it is not hardware failure.
Sometimes the server load just rises and then the server dies other times the load rises but the kernel manages to get it back alive filling up syslog with messages like this
Sounds like a different issue. The patch Bron included before fixes (or at least reduces to the point where it fixes it for us) a problem where processes get stuck in D state and are unkillable. A reboot is required to remove them. Apparently this is a known bug in ReiserFS (see messages below). As noted, the same bug exists in ext3. There appears to have been some patches to try and fix it for both reiserfs and ext3, but I'm not sure if they're in the mainline kernel yet.
Quoted 3 lines
http://www.ussg.iu.edu/hypermail/linux/kernel/0409.0/2056.html
http://hulllug.principalhosting.net/archive/index.php/t-22774.html
There is a much less complex solution that I've just recently gotten working in the SUSE kernel. If reiser3/ext3 don't log the inode during atime updates, the problem goes away.
You can solve this now by mounting with -o noatime (although that might not play well with cyrus, not sure). My current patch works around this in ugly ways, what I plan on doing during OLS is finding out why ext3 is still logging the inode all the time.
For reiser3, this was to avoid kswapd having to log a bunch of inodes in response to memory pressure, but that was back in 2.4 when things were different. We shouldn't need to do it anymore...
-chris
Quoted 9 lines
There is a much less complex solution that I've just recently gotten working in the SUSE kernel. If reiser3/ext3 don't log the inode during atime updates, the problem goes away.
You can solve this now by mounting with -o noatime (although that might not play well with cyrus, not sure). My current patch works around this in ugly ways, what I plan on doing during OLS is finding out why ext3 is still logging the inode all the time.
Well we have always mounted our cyrus filesystems with:
noatime,nodiratime,notail
And the problem was occuring all the time with these mount options. We've since also added on your suggestion nolargeio=1 and used the patch Vladimir created. I'm not sure which of those fixed the problem, but it definitely has not occured since we did those last 2 things.
Are you saying that if you mount with noatime *and* use your new patch it will fix the problem?
What about the 2 threads linked to. Did those end up getting anywhere?
Quoted 2 lines
http://www.ussg.iu.edu/hypermail/linux/kernel/0409.0/2056.html
http://hulllug.principalhosting.net/archive/index.php/t-22774.html
Rob
Quoted 8 lines · Chris Mason
Quoted 8 lines
Sounds like a different issue. The patch Bron included before fixes (or at least reduces to the point where it fixes it for us) a problem where processes get stuck in D state and are unkillable. A reboot is required to remove them. Apparently this is a known bug in ReiserFS (see messages below). As noted, the same bug exists in ext3. There appears to have been some patches to try and fix it for both reiserfs and ext3, but I'm not sure if they're in the mainline kernel yet.
Quoted 6 lines
Quoted 2 lines
http://www.ussg.iu.edu/hypermail/linux/kernel/0409.0/2056.html
http://hulllug.principalhosting.net/archive/index.php/t-22774.htmlThere is a much less complex solution that I've just recently gotten working in the SUSE kernel. If reiser3/ext3 don't log the inode during atime updates, the problem goes away.
The sysrq is huge, and I haven't yet found the person holding the transaction open. But, here's another place that starts a transaction with the mmap sem held, and I would guess the transaction writer is waiting on something for that mmap sem.
atime updates alone won't fix this one....
-chris
imapd D F3159530 0 32412 2292 32413 32411 (NOTLB)e1cdfdfc 00000082 00000008 f3159530 00000202 00000000 c1e5b0a0 c013b1a0
00000034 00000202 c301b520 00000001 00004609 beca3f0b 00005566 c301b520
c315b530 f3159530 f3159654 00000000 00000001 0000000e d9309dac 00000000
Call Trace:
[<c013b1a0>] free_hot_cold_page+0x20/0xd0
[<c01ac8dd>] queue_log_writer+0x5d/0x80
[<c0114b10>] default_wake_function+0x0/0x20
[<c01acb8a>] do_journal_begin_r+0x1ca/0x2b0
[<c01409e0>] truncate_inode_pages+0x290/0x2b0
[<c01ace9e>] journal_begin+0x8e/0xe0
[<c0191061>] reiserfs_delete_inode+0x51/0xc0
[<c01447fa>] unmap_vmas+0x14a/0x260
[<c0191010>] reiserfs_delete_inode+0x0/0xc0
[<c016c97d>] generic_delete_inode+0x7d/0xe0
[<c016cb83>] iput+0x63/0x70
[<c0169db6>] dput+0x176/0x1b0
[<c01547cb>] __fput+0xcb/0x100
[<c01470ff>] remove_vm_struct+0x5f/0x80
[<c014873a>] unmap_vma_list+0x1a/0x30
[<c0148a9f>] do_munmap+0xdf/0xf0
[<c0148aff>] sys_munmap+0x4f/0x70
[<c0102a15>] syscall_call+0x7/0xbQuoted 5 lines · Rob Mueller
Are you saying that if you mount with noatime *and* use your new patch it will fix the problem?
What about the 2 threads linked to. Did those end up getting anywhere?
Sorry for the confusion, you're hitting the other mmap_sem -> transaction lock problem. This one should be solvable with an iget so we make sure not to do the final unlink until after the mmap sem is dropped.
Lets see what I can do...
-chris
Quoted 6 lines
Sorry for the confusion, you're hitting the other
mmap_sem-> transaction lock problem. This one should be solvable with an iget so we make sure not to do the final unlink until after the mmap sem is dropped.Lets see what I can do...
Oh dang.
I thought this last crash after upgrading to 2.6.12.2 was due to the IRQ BALANCE issue Lars suggested, but you're saying that it's actually a whole different bug, though similar to the previous "prepare_write ... copy_from_user ... commit_write" lock problem?
Is this something new between 2.6.4-mm2 and 2.6.12.2? Or would it have always been present, just for some reason we're hitting it in 2.6.12 now but weren't hitting it in 2.6.4?
I feel we're moving backwards... :(
Rob