2.6.21-rc5-mm3
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines
BUG: NMI Watchdog detected LOCKUP on CPU0, eip c014ce9c, registers:
I suspect it is just because his console is too slow and then unwinding took too long and it happened to hit the unwinder.
You did use a slow console, right?
I suppose it just needs a strategic touch_nmi_watchdog. Will add that.
Quoted 28 lines
Modules linked in: ide_cd cdrom rtc unix CPU: 0 EIP: 0060:[<c014ce9c>] Not tainted VLI EFLAGS: 00000093 (2.6.21-rc5-mm3 #10)EIP is at
read_pointer+0x49/0x2d8
eax:c7a8dd04ebx: 00000000 ecx: 00000000 edx:c043d19c
esi:c043d184edi:c043d19cebp:c7a8dbf4esp:c7a8dbbc
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process udevd (pid: 864, ti=c7a8c000 task=c97cd4d0 task.ti=c7a8c000)Stack: 0000000b c7a8dc70 c7a8dbf4 c014d6d0 c7a8dd04 c043e454 c7a8dbf4 c011fdd1 c043e404 c043e454 c043d190 0005cd94 c043d184 c7a8dd04 c7a8dd14 c014dd3a 00000000 00000000 00000000 00000000 c7a8df44 c7a8dd74 c741eefb 00000008 Call Trace: [<c014dd3a>] unwind+0x414/0xfa2 [<c010510d>] dump_trace_unwind+0xb4/0xe5 [<c014ce4d>] unwind_init_running+0x25/0x2b [<c01051a1>] dump_trace+0x63/0x1eb [<c010ad39>] save_stack_trace+0x23/0x42 [<c01393c9>] update_cpu_base_expires_next+0x56/0x5a [<c013a475>] hrtimer_interrupt+0x17c/0x1b8 [<c0115e8e>] smp_apic_timer_interrupt+0x72/0x85 [<c0104bef>] apic_timer_interrupt+0x33/0x38 [<c014320c>] lock_release+0x1d2/0x1da [<c013a7ef>] up_read+0x19/0x2e [<c011b800>] do_page_fault+0x28f/0x55b [<c034d191>] error_code+0x79/0x80 [<c020c23e>] __put_user_4+0x12/0x18 DWARF2 unwinder stuck at __put_user_4+0x12/0x18
Quoted 2 lines
Leftover inexact backtrace:
[<c01040d6>]ret_from_fork+0x6/0x1c
Hmpf. I saw it once in child_rip here too. Then I wanted to reproduce it to report properly and couldn't again. I had a few other backtraces that were all non stuck with child_rip then on essentially the same kernel. Something weird is going on.
Quoted 6 lines
[<c013a475>] hrtimer_interrupt+0x17c/0x1b8 [<c0115e8e>] smp_apic_timer_interrupt+0x72/0x85 [<c0104bef>] apic_timer_interrupt+0x33/0x38 [<c020c184>] __get_user_4+0x14/0x17 DWARF2 unwinder stuck at __get_user_4+0x14/0x17Leftover inexact backtrace:
Now that is weird. Never seen before. Jan, any ideas?
What is your gcc/compiler, Michal?
-Andi
Were there any strange binutils in use, Michal?
Quoted 3 lines
[<c018b42f>] do_execve+0xdd/0x210 [<c0102497>] sys_execve+0x3f/0x62 [<c01041c2>] sysenter_past_esp+0x5f/0x99
Quoted 6 lines · Andi Kleen
Quoted 1 line
[<
c01040d6>]ret_from_fork+0x6/0x1cHmpf. I saw it once in
child_riphere too. Then I wanted to reproduce it to report properly and couldn't again. I had a few other backtraces that were all non stuck withchild_ripthen on essentially the same kernel. Something weird is going on.
find below a colorful unwinder crash, on an i386 UNWIND_STACK + FRAME_POINTERS kernel. It crashed on the context-> dereference:
/* Should be after the line below, but somewhere
in early boot context comes out corrupted and we
can't reference it -AK */
if (ops->stack(data, "IRQ") < 0)
break;
stack = (unsigned long*)context->previous_esp;
if (!stack)
break;the comment suggests that such a crash isnt without precedence, but my crash wasnt during early bootup, it was on a working system.
Ingo
-------------->
[<c0106442>] dump_trace+0x78/0x210 [<c010660f>] show_trace_log_lvl+0x35/0x54 [<c0106a77>] show_trace+0x2c/0x2e [<c0106aa2>] dump_stack+0x29/0x2b [<c01637fa>] check_critical_timing+0x26a/0x37e [<c0163e7f>] time_hardirqs_on+0xac/0xc2 [<c014d385>] trace_hardirqs_on+0x16b/0x172 [<c010533a>] restore_nocheck+0x12/0x15 [<c02a007b>] acpi_rs_get_address_common+0x63/0x71 [<c06bc000>] init_thread_union+0x0/0x1000 DWARF2 unwinder stuck at init_thread_union+0x0/0x1000 Leftover inexact backtrace: BUG: unable to handle kernel paging request at virtual address 70252034 printing eip: c0106592 *pde = 00000000 stopped custom tracer. Oops: 0000 [#1] PREEMPT SMP Modules linked in: CPU: 0 EIP: 0060:[<c0106592>] Not tainted VLI EFLAGS: 00010046 (2.6.21-rc5-rt6 #83)
EIP is at dump_trace+0x1c8/0x210
eax: 00000000 ebx: c06bce10 ecx: fffd85a4 edx: c064728c
esi: 70252000 edi: 70252070 ebp: c06bce10 esp: c06bcda4
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 preempt:00000002
Process swapper (pid: 0, ti=c06bc000 task=c0645280 task.ti=c06bc000)
Stack: c055faad c059b2f4 c06bc000 00000000 c02ae350 00000000 00000000 00000000
00000000 c4b83528 0000007b 0000007b 00000000 00000000 c06bc000 00000060
00000000 c0645288 00000068 c0645280 0835b643 c064728c c055faad c011c8f4
Call Trace:
[<c010660f>] show_trace_log_lvl+0x35/0x54
[<c0106a77>] show_trace+0x2c/0x2e
[<c0106aa2>] dump_stack+0x29/0x2b
[<c01637fa>] check_critical_timing+0x26a/0x37e
[<c0163e7f>] time_hardirqs_on+0xac/0xc2
[<c014d385>] trace_hardirqs_on+0x16b/0x172
[<c010533a>] restore_nocheck+0x12/0x15
[<c02a007b>] acpi_rs_get_address_common+0x63/0x71
[<c06bc000>] init_thread_union+0x0/0x1000
DWARF2 unwinder stuck at init_thread_union+0x0/0x1000
Leftover inexact backtrace:
BUG: unable to handle kernel paging request at virtual address 70252034
printing eip:
c0106592
*pde = 00000000
Oops: 0000 [#2]
PREEMPT SMP
Modules linked in:
CPU: 0
EIP: 0060:[<c0106592>] Not tainted VLI
EFLAGS: 00010046 (2.6.21-rc5-rt6 #83)EIP is at dump_trace+0x1c8/0x210
eax: 00000000 ebx: c06bcc34 ecx: fffd85b7 edx: c064728c
esi: 70252000 edi: 70252070 ebp: c06bcc34 esp: c06bcbc8
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 preempt:00000002
Process swapper (pid: 0, ti=c06bc000 task=c0645280 task.ti=c06bc000)
Stack: c0550554 c059b2f4 c06bc000 00000000 c02ae350 00000000 00000000 00000000
00000000 c4b83528 c064007b fffd007b c06400d8 ffffffff c06bc000 00000060
00010046 c0645288 00000068 c0645280 00000019 c064728c c0550554 c011c8f4
Call Trace:
[<c010660f>] show_trace_log_lvl+0x35/0x54
[<c01066db>] show_stack_log_lvl+0xad/0xc5
[<c010691a>] show_registers+0x227/0x31d
[<c0106d37>] die+0x137/0x21d
[<c0487550>] do_page_fault+0x4c1/0x5a8
[<c048585c>] error_code+0x7c/0x84
[<c0106592>] dump_trace+0x1c8/0x210
[<c010660f>] show_trace_log_lvl+0x35/0x54
[<c0106a77>] show_trace+0x2c/0x2e
[<c0106aa2>] dump_stack+0x29/0x2b
[<c01637fa>] check_critical_timing+0x26a/0x37e
[<c0163e7f>] time_hardirqs_on+0xac/0xc2
[<c014d385>] trace_hardirqs_on+0x16b/0x172
[<c010533a>] restore_nocheck+0x12/0x15
[<c02a007b>] acpi_rs_get_address_common+0x63/0x71
[<c06bc000>] init_thread_union+0x0/0x1000
DWARF2 unwinder stuck at init_thread_union+0x0/0x1000Andi Kleen napisał(a):
Quoted 6 lines
Quoted 1 line
BUG: NMI Watchdog detected LOCKUP on CPU0, eip c014ce9c, registers:I suspect it is just because his console is too slow and then unwinding took too long and it happened to hit the unwinder.
You did use a slow console, right?
console=ttyS0,115200n8
Quoted 45 lines
I suppose it just needs a strategic
touch_nmi_watchdog. Will add that.Quoted 28 lines
Modules linked in: ide_cd cdrom rtc unix CPU: 0 EIP: 0060:[<c014ce9c>] Not tainted VLI EFLAGS: 00000093 (2.6.21-rc5-mm3 #10)EIP is at
read_pointer+0x49/0x2d8
eax:c7a8dd04ebx: 00000000 ecx: 00000000 edx:c043d19c
esi:c043d184edi:c043d19cebp:c7a8dbf4esp:c7a8dbbc
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process udevd (pid: 864, ti=c7a8c000 task=c97cd4d0 task.ti=c7a8c000)Stack: 0000000b c7a8dc70 c7a8dbf4 c014d6d0 c7a8dd04 c043e454 c7a8dbf4 c011fdd1 c043e404 c043e454 c043d190 0005cd94 c043d184 c7a8dd04 c7a8dd14 c014dd3a 00000000 00000000 00000000 00000000 c7a8df44 c7a8dd74 c741eefb 00000008 Call Trace: [<c014dd3a>] unwind+0x414/0xfa2 [<c010510d>] dump_trace_unwind+0xb4/0xe5 [<c014ce4d>] unwind_init_running+0x25/0x2b [<c01051a1>] dump_trace+0x63/0x1eb [<c010ad39>] save_stack_trace+0x23/0x42 [<c01393c9>] update_cpu_base_expires_next+0x56/0x5a [<c013a475>] hrtimer_interrupt+0x17c/0x1b8 [<c0115e8e>] smp_apic_timer_interrupt+0x72/0x85 [<c0104bef>] apic_timer_interrupt+0x33/0x38 [<c014320c>] lock_release+0x1d2/0x1da [<c013a7ef>] up_read+0x19/0x2e [<c011b800>] do_page_fault+0x28f/0x55b [<c034d191>] error_code+0x79/0x80 [<c020c23e>] __put_user_4+0x12/0x18 DWARF2 unwinder stuck at __put_user_4+0x12/0x18Quoted 2 lines
Leftover inexact backtrace:
[<c01040d6>]ret_from_fork+0x6/0x1cHmpf. I saw it once in
child_riphere too. Then I wanted to reproduce it to report properly and couldn't again. I had a few other backtraces that were all non stuck withchild_ripthen on essentially the same kernel. Something weird is going on.Quoted 6 lines
[<c013a475>] hrtimer_interrupt+0x17c/0x1b8 [<c0115e8e>] smp_apic_timer_interrupt+0x72/0x85 [<c0104bef>] apic_timer_interrupt+0x33/0x38 [<c020c184>] __get_user_4+0x14/0x17 DWARF2 unwinder stuck at __get_user_4+0x14/0x17Leftover inexact backtrace:
Quoted 3 lines
Now that is weird. Never seen before. Jan, any ideas?
What is your gcc/compiler, Michal?
gcc --version
gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-51)
Quoted 5 lines
-Andi
Were there any strange binutils in use, Michal?
I don't remember when I had problems with this compiler.
Quoted 5 lines
Quoted 4 lines
[<c018b42f>] do_execve+0xdd/0x210 [<c0102497>] sys_execve+0x3f/0x62 [<c01041c2>] sysenter_past_esp+0x5f/0x99
Regards,
Michal
Signature · 5 lines
Michal K. K. Piotrowski LTG - Linux Testers Group (PL) (http://www.stardust.webpages.pl/ltg/) LTG - Linux Testers Group (EN) (http://www.stardust.webpages.pl/linux_testers_group_en/)
Quoted 3 lines · Michal Piotrowski
Quoted 1 line
Were there any strange binutils in use, Michal?
I don't remember when I had problems with this compiler.
^^^
(sorry, to many bottles of beer)
ld --version
GNU ld version 2.17.50.0.6-2.fc6 20061020
Regards,
Michal
Signature · 5 lines
Michal K. K. Piotrowski LTG - Linux Testers Group (PL) (http://www.stardust.webpages.pl/ltg/) LTG - Linux Testers Group (EN) (http://www.stardust.webpages.pl/linux_testers_group_en/)
Quoted 9 lines
Quoted 4 lines
[<c020c23e>] __put_user_4+0x12/0x18 DWARF2 unwinder stuck at __put_user_4+0x12/0x18Leftover inexact backtrace:
[<c01040d6>]ret_from_fork+0x6/0x1cHmpf. I saw it once in
child_riphere too. Then I wanted to reproduce it to report properly and couldn't again. I had a few other backtraces that were all non stuck withchild_ripthen on essentially the same kernel. Something weird is going on.
Quoted 8 lines
Quoted 6 lines
[<c013a475>] hrtimer_interrupt+0x17c/0x1b8 [<c0115e8e>] smp_apic_timer_interrupt+0x72/0x85 [<c0104bef>] apic_timer_interrupt+0x33/0x38 [<c020c184>] __get_user_4+0x14/0x17 DWARF2 unwinder stuck at __get_user_4+0x14/0x17Leftover inexact backtrace:
Now that is weird. Never seen before. Jan, any ideas?
Not weird at all - these functions simply aren't annotated (i.e. I assume you forgot to apply the respective patch when you re-added the unwinder). As I don't have this as a standalone patch anymore, I'll include the full diff for arch/i386/lib/ below - this is likely to apply.
Jan
arch/i386/lib/checksum.S +60 -9
@@ -25,6 +25,8 @@ * 2 of the License, or (at your option) any later version. */ +#include <linux/linkage.h> +#include <asm/dwarf2.h> #include <asm/errno.h> /*
@@ -36,8 +38,6 @@ unsigned int csum_partial(const unsigned */ .text -.align 4 -.globl csum_partial #ifndef CONFIG_X86_USE_PPRO_CHECKSUM
@@ -48,9 +48,14 @@ unsigned int csum_partial(const unsigned * Fortunately, it is easy to convert 2-byte alignment to 4-byte * alignment for the unrolled loop. */ -csum_partial: +ENTRY(csum_partial) + CFI_STARTPROC pushl %esi + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET esi, 0 pushl %ebx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET ebx, 0 movl 20(%esp),%eax # Function arg: unsigned int sum movl 16(%esp),%ecx # Function arg: int len movl 12(%esp),%esi # Function arg: unsigned char *buff
@@ -128,16 +133,27 @@ csum_partial: roll $8, %eax 8: popl %ebx + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE ebx popl %esi + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE esi ret + CFI_ENDPROC +ENDPROC(csum_partial) #else /* Version for PentiumII/PPro */ -csum_partial: +ENTRY(csum_partial) + CFI_STARTPROC pushl %esi + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET esi, 0 pushl %ebx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET ebx, 0 movl 20(%esp),%eax # Function arg: unsigned int sum movl 16(%esp),%ecx # Function arg: int len movl 12(%esp),%esi # Function arg: const unsigned char *buf
@@ -245,8 +261,14 @@ csum_partial: roll $8, %eax 90: popl %ebx + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE ebx popl %esi + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE esi ret + CFI_ENDPROC +ENDPROC(csum_partial) #endif
@@ -278,19 +300,24 @@ unsigned int csum_partial_copy_generic ( .long 9999b, 6002f ; \ .previous -.align 4 -.globl csum_partial_copy_generic - #ifndef CONFIG_X86_USE_PPRO_CHECKSUM #define ARGBASE 16 #define FP 12 -csum_partial_copy_generic: +ENTRY(csum_partial_copy_generic) + CFI_STARTPROC subl $4,%esp + CFI_ADJUST_CFA_OFFSET 4 pushl %edi + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET edi, 0 pushl %esi + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET esi, 0 pushl %ebx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET ebx, 0 movl ARGBASE+16(%esp),%eax # sum movl ARGBASE+12(%esp),%ecx # len movl ARGBASE+4(%esp),%esi # src
@@ -400,10 +427,19 @@ DST( movb %cl, (%edi) ) .previous popl %ebx + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE ebx popl %esi + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE esi popl %edi + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE edi popl %ecx # equivalent to addl $4,%esp + CFI_ADJUST_CFA_OFFSET -4 ret + CFI_ENDPROC +ENDPROC(csum_partial_copy_generic) #else
@@ -421,10 +457,17 @@ DST( movb %cl, (%edi) ) #define ARGBASE 12 -csum_partial_copy_generic: +ENTRY(csum_partial_copy_generic) + CFI_STARTPROC pushl %ebx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET ebx, 0 pushl %edi + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET edi, 0 pushl %esi + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET esi, 0 movl ARGBASE+4(%esp),%esi #src movl ARGBASE+8(%esp),%edi #dst movl ARGBASE+12(%esp),%ecx #len
@@ -485,9 +528,17 @@ DST( movb %dl, (%edi) ) .previous popl %esi + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE esi popl %edi + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE edi popl %ebx + CFI_ADJUST_CFA_OFFSET -4 + CFI_RESTORE ebx ret + CFI_ENDPROC +ENDPROC(csum_partial_copy_generic) #undef ROUND #undef ROUND1
arch/i386/lib/getuser.S +17 -9
@@ -8,6 +8,8 @@ * return an error value in addition to the "real" * return value. */ +#include <linux/linkage.h> +#include <asm/dwarf2.h> #include <asm/thread_info.h>
@@ -24,19 +26,19 @@ */ .text -.align 4 -.globl __get_user_1 -__get_user_1: +ENTRY(__get_user_1) + CFI_STARTPROC GET_THREAD_INFO(%edx) cmpl TI_addr_limit(%edx),%eax jae bad_get_user 1: movzbl (%eax),%edx xorl %eax,%eax ret + CFI_ENDPROC +ENDPROC(__get_user_1) -.align 4 -.globl __get_user_2 -__get_user_2: +ENTRY(__get_user_2) + CFI_STARTPROC addl $1,%eax jc bad_get_user GET_THREAD_INFO(%edx)
@@ -45,10 +47,11 @@ __get_user_2: 2: movzwl -1(%eax),%edx xorl %eax,%eax ret + CFI_ENDPROC +ENDPROC(__get_user_2) -.align 4 -.globl __get_user_4 -__get_user_4: +ENTRY(__get_user_4) + CFI_STARTPROC addl $3,%eax jc bad_get_user GET_THREAD_INFO(%edx)
@@ -57,11 +60,16 @@ __get_user_4: 3: movl -3(%eax),%edx xorl %eax,%eax ret + CFI_ENDPROC +ENDPROC(__get_user_4) bad_get_user: + CFI_STARTPROC xorl %edx,%edx movl $-14,%eax ret + CFI_ENDPROC +END(bad_get_user) .section __ex_table,"a" .long 1b,bad_get_user
arch/i386/lib/putuser.S +25 -14
@@ -8,6 +8,8 @@ * return an error value in addition to the "real" * return value. */ +#include <linux/linkage.h> +#include <asm/dwarf2.h> #include <asm/thread_info.h>
@@ -23,23 +25,28 @@ * as they get called from within inline assembly. */ -#define ENTER pushl %ebx ; GET_THREAD_INFO(%ebx) -#define EXIT popl %ebx ; ret +#define ENTER CFI_STARTPROC ; \ + pushl %ebx ; \ + CFI_ADJUST_CFA_OFFSET 4 ; \ + CFI_REL_OFFSET ebx, 0 ; \ + GET_THREAD_INFO(%ebx) +#define EXIT popl %ebx ; \ + CFI_ADJUST_CFA_OFFSET -4 ; \ + CFI_RESTORE ebx ; \ + ret ; \ + CFI_ENDPROC .text -.align 4 -.globl __put_user_1 -__put_user_1: +ENTRY(__put_user_1) ENTER cmpl TI_addr_limit(%ebx),%ecx jae bad_put_user 1: movb %al,(%ecx) xorl %eax,%eax EXIT +ENDPROC(__put_user_1) -.align 4 -.globl __put_user_2 -__put_user_2: +ENTRY(__put_user_2) ENTER movl TI_addr_limit(%ebx),%ebx subl $1,%ebx
@@ -48,10 +55,9 @@ __put_user_2: 2: movw %ax,(%ecx) xorl %eax,%eax EXIT +ENDPROC(__put_user_2) -.align 4 -.globl __put_user_4 -__put_user_4: +ENTRY(__put_user_4) ENTER movl TI_addr_limit(%ebx),%ebx subl $3,%ebx
@@ -60,10 +66,9 @@ __put_user_4: 3: movl %eax,(%ecx) xorl %eax,%eax EXIT +ENDPROC(__put_user_4) -.align 4 -.globl __put_user_8 -__put_user_8: +ENTRY(__put_user_8) ENTER movl TI_addr_limit(%ebx),%ebx subl $7,%ebx
@@ -73,10 +78,16 @@ __put_user_8: 5: movl %edx,4(%ecx) xorl %eax,%eax EXIT +ENDPROC(__put_user_8) bad_put_user: + CFI_STARTPROC simple + CFI_DEF_CFA esp, 2*4 + CFI_OFFSET eip, -1*4 + CFI_OFFSET ebx, -2*4 movl $-14,%eax EXIT +END(bad_put_user) .section __ex_table,"a" .long 1b,bad_put_user