kernel BUG at include/linux/list.h:164!
Subsystems (2)
Hi Folks :)
I have a server at my hosting ISP, which got a new Kernel with devmapper patches in it to use LVM2 with 2.6.x.
I test these patches at home, no error, but my Server crashes after a couple of days, second time now.
May 1 12:03:02 kermit spamd[3861]: checking message <[email protected]> for nobody:1005. May 1 12:03:07 kermit kernel: ------------[ cut here ]------------ May 1 12:03:07 kermit kernel: kernel BUG at include/linux/list.h:164! May 1 12:03:07 kermit kernel: invalid operand: 0000 [#1] May 1 12:03:07 kermit kernel: PREEMPT May 1 12:03:07 kermit kernel: CPU: 0 May 1 12:03:07 kermit kernel: EIP: 0060:[exit_rmap+237/272] Not tainted VLI May 1 12:03:07 kermit kernel: EFLAGS: 00010283 (2.6.6-rc2-mm1) May 1 12:03:07 kermit kernel: EIP is at exit_rmap+0xed/0x110 May 1 12:03:07 kermit kernel: eax: c600c001 ebx: ccb35e8c ecx: ccb35e0c edx: ccb35e80 May 1 12:03:07 kermit kernel: esi: c13bd2e0 edi: c19c5700 ebp: c2fdf8c0 esp: c600dc54 May 1 12:03:07 kermit kernel: ds: 007b es: 007b ss: 0068 May 1 12:03:07 kermit kernel: Process spamd (pid: 3862, threadinfo=c600c000 task=c0ad6030) May 1 12:03:07 kermit kernel: Stack: 000015c4 c2fdf8c0 c2fdf8c0 c2fdf8c0 c0116d15 c2fdf8c0 c03b818c c0ad6030 May 1 12:03:07 kermit kernel: c015791b c2fdf8c0 c2fdf8c0 00000000 c600c000 c0b1d900 c600de44 c0157aea May 1 12:03:07 kermit kernel: c19c5700 cffe4740 c600dcbc c600c000 c0000000 c02d5295 c600de44 c0157850 May 1 12:03:07 kermit kernel: Call Trace: May 1 12:03:07 kermit kernel: [mmput+117/192] mmput+0x75/0xc0 May 1 12:03:07 kermit kernel: [exec_mmap+187/320] exec_mmap+0xbb/0x140 May 1 12:03:07 kermit kernel: [flush_old_exec+330/2288] flush_old_exec+0x14a/0x8f0 May 1 12:03:07 kermit kernel: [kernel_read+80/96] kernel_read+0x50/0x60 May 1 12:03:07 kermit kernel: [load_elf_binary+835/3296] load_elf_binary+0x343/0xce0 May 1 12:03:07 kermit kernel: [load_elf_binary+0/3296] load_elf_binary+0x0/0xce0 May 1 12:03:07 kermit kernel: [search_binary_handler+396/704] search_binary_handler+0x18c/0x2c0 May 1 12:03:07 kermit kernel: [do_execve+482/640] do_execve+0x1e2/0x280 May 1 12:03:07 kermit kernel: [sys_rt_sigaction+168/192] sys_rt_sigaction+0xa8/0xc0 May 1 12:03:07 kermit kernel: [sys_execve+66/128] sys_execve+0x42/0x80 May 1 12:03:07 kermit kernel: [syscall_call+7/11] syscall_call+0x7/0xb May 1 12:03:07 kermit kernel: May 1 12:03:07 kermit kernel: Code: ff eb d8 0f 0b 78 00 9b 3f 2d c0 eb e3 0f 0b 77 00 9b 3f 2d c0 eb d1 e8 12 5a 17 00 eb b8 0f 0b a5 00 11 02 2d c0 e9 65 ff ff ff <0f> 0b a4 00 11 02 2d c0 e9 4c ff ff ff 0f 0b 6f 00 9b 3f 2d c0 May 1 12:03:07 kermit kernel: <6>note: spamd[3862] exited with preempt_count 1 May 1 16:32:35 kermit syslogd 1.4.1#14: restart.
2.6.6-rc2-mm1 with devmapper udm1 patch
Is this dump due to udma1 or not and the error is fixed now?
Regards, Konsti
Signature · 5 lines
2.6.5-mm1 Konstantin Kletschke <[email protected]>, <[email protected]> GPG KeyID EF62FCEF Fingerprint: 13C9 B16B 9844 EC15 CC2E A080 1E69 3FDA EF62 FCEF keulator.homelinux.org up 4:01, 15 users
Quoted 7 lines · Konstantin Kletschke
May 1 12:03:07 kermit kernel: kernel BUG at include/linux/list.h:164! May 1 12:03:07 kermit kernel: invalid operand: 0000 [#1] May 1 12:03:07 kermit kernel: PREEMPT May 1 12:03:07 kermit kernel: CPU: 0 May 1 12:03:07 kermit kernel: EIP: 0060:[exit_rmap+237/272] Not tainted VLI May 1 12:03:07 kermit kernel: EFLAGS: 00010283 (2.6.6-rc2-mm1)
....
Quoted 3 lines
2.6.6-rc2-mm1 with devmapper udm1 patch
Is this dump due to udma1 or not and the error is fixed now?
Probably not - I don't know this udma1 patch, but the error above definitely occurs in my rmap area. Could be due to slab corruption from somewhere else, but that would be a poor assumption to make - though neither my testing nor anyone else has seen the same (yet).
I imagine you're limited in the experiments you can do on that server, and wouldn't want to run it with CONFIG_SLAB_DEBUG=y, which might show up unrelated problems - interesting for us, but troublesome for you.
You're UP but PREEMPT on. Hmm, there's a suspicious atomic_read in exit_rmap: once upon a time I convinced myself it was good, but it looks unsafe to me now - could cause a double free of an anonmm, which would lead to your BUG if reallocated in between.
Not so likely for me to feel this is definitely the answer, but likely enough to be well worth trying. Please try patch below - simple enough not to make your case worse anyway - thanks.
Hugh
mm/rmap.c +3 -1
@@ -103,6 +103,7 @@ void exit_rmap(struct mm_struct *mm) {struct anonmm *anonmm = mm->anonmm;struct anonmm *anonhd = anonmm->head; +int anonhd_count; mm->anonmm =NULL ;spin_lock (&anonhd->lock);
@@ -114,8 +115,9 @@ void exit_rmap(struct mm_struct *mm)if (atomic_dec_and_test (&anonhd->count))BUG (); } + anonhd_count =atomic_read (&anonhd->count);spin_unlock (&anonhd->lock); -if (atomic_read (&anonhd->count) ==1 ) { +if (anonhd_count ==1 ) {BUG_ON (anonhd->mm);BUG_ON (!list_empty (&anonhd->list));kmem_cache_free (anonmm_cachep, anonhd);
Am 2004-05-02 09:22 +0100 schrieb Hugh Dickins:
Quoted 8 lines · Konstantin Kletschke
Quoted 7 lines
May 1 12:03:07 kermit kernel: kernel BUG at include/linux/list.h:164! May 1 12:03:07 kermit kernel: invalid operand: 0000 [#1] May 1 12:03:07 kermit kernel: PREEMPT May 1 12:03:07 kermit kernel: CPU: 0 May 1 12:03:07 kermit kernel: EIP: 0060:[exit_rmap+237/272] Not tainted VLI May 1 12:03:07 kermit kernel: EFLAGS: 00010283 (2.6.6-rc2-mm1)
Quoted 6 lines
....
Quoted 3 lines
2.6.6-rc2-mm1 with devmapper udm1 patch
Is this dump due to udma1 or not and the error is fixed now?
Probably not - I don't know this udma1 patch, but the error above
It provides snapshot and pvmove functionality for LVM2 and 2.6.x kernels and is in the early creation if I understood right...
Quoted 1 line
definitely occurs in my rmap area.
HA! :)
Quoted 1 line
though neither my testing nor anyone else has seen the same (yet).
Ok, lets see...
Quoted 3 lines
I imagine you're limited in the experiments you can do on that server, and wouldn't want to run it with
CONFIG_SLAB_DEBUG=y, which might show up unrelated problems - interesting for us, but troublesome for you.
Yes, sadly :(
Quoted 8 lines
You're UP but PREEMPT on. Hmm, there's a suspicious
atomic_readinexit_rmap: once upon a time I convinced myself it was good, but it looks unsafe to me now - could cause a double free of an anonmm, which would lead to your BUG if reallocated in between.Not so likely for me to feel this is definitely the answer, but likely enough to be well worth trying. Please try patch below - simple enough not to make your case worse anyway - thanks.
Ok, I will try that out and incorporate the patch tomorrow. Works on test machine fine so far (and compiles of course :P). Then we will see, if the uptime lasts longer...
reHugh
Signature · 5 lines
2.6.5-mm1 Konstantin Kletschke <[email protected]>, <[email protected]> GPG KeyID EF62FCEF Fingerprint: 13C9 B16B 9844 EC15 CC2E A080 1E69 3FDA EF62 FCEF keulator.homelinux.org up 5:23, 18 users