Skip to contentSkip to conversation

2.6.26 kernel crash

PATCH In review Helge Deller 5 participants 9 messages Jul 20, 2008 – Aug 8, 2008 PA-RISC architecture

lore
Share & tools
lore

Helge Deller

author maintainer
#0 lore raw

While debugging some user-space stuff I just faced this 32bit kernel crash (2.6.26):
Maybe someone has an idea?

Kernel Fault: Code=15 regs=5f578640 (Addr=000127ec)

YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111110000001111 Not tainted

r00-03  0004fc0f 10644f10 1011ee68 5f578288
r04-07  00000014 8c1fe860 8c1fe828 8eb7d030
r08-11  105ee890 00000040 0001264a 000d3b60
r12-15  00000000 000d3b5c 000dde68 000b0000
r16-19  8eb7d250 000b0000 ffffffff 000127ec
r20-23  ffffe040 00000000 101205d4 00000000
r24-27  00012800 101205cc 5f578288 1062ff10
r28-31  00000000 000002ee 5f578640 10120630

sr00-03 00000000 000013a7 00000000 000013a7 sr04-07 00000000 00000000 00000000 00000000

IASQ: 00000000 00000000 IAOQ: 1011ed04 1011ed08
IIR: 0e601093 ISR: 00000000 IOR: 000127ec
CPU: 0 CR30: 5f578000 CR31: ffffffff
ORIG_R28: 40400000
IAOQ[0]: unwind_frame_regs+0x214/0x244 IAOQ[1]: unwind_frame_regs+0x218/0x244 RP(r2): unwind_once+0x10/0x48

Backtrace:
 [<1011ee68>] unwind_once+0x10/0x48
 [<10112bbc>] do_show_stack+0x28/0xac
 [<10112cd0>] parisc_show_stack+0x90/0xa8
 [<10111258>] do_page_fault+0x174/0x294
 [<10113824>] handle_interruption+0x570/0x58c
 [<10117078>] intr_check_sig+0x0/0x34

Kernel panic - not syncing: Kernel Fault
author maintainer
#1 lore raw
Quoted 2 lines · Helge Deller

While debugging some user-space stuff I just faced this 32bit kernel crash (2.6.26):
Maybe someone has an idea?

This is reproduceable.
Testcase and docu is here: http://gsyprf10.external.hp.com/~deller/crash.tgz

Just run:
LD_LIBRARY_PATH=. ./a.out

Output is:
making contexts
swapping contexts
start f2
&on_stack=0x12d88
start f1(a0=1,a1=2,a2=3,a3=fffffffc)
<crash>

Info:
- test program is from glibc:
glibc-2.7/stdlib/tst-setcontext.c
- the compiled libc.so.6 includes some changes for *context() functions from me.

Quoted 32 lines

Kernel Fault: Code=15 regs=5f578640 (Addr=000127ec)

YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111110000001111 Not tainted

r00-03  0004fc0f 10644f10 1011ee68 5f578288
r04-07  00000014 8c1fe860 8c1fe828 8eb7d030
r08-11  105ee890 00000040 0001264a 000d3b60
r12-15  00000000 000d3b5c 000dde68 000b0000
r16-19  8eb7d250 000b0000 ffffffff 000127ec
r20-23  ffffe040 00000000 101205d4 00000000
r24-27  00012800 101205cc 5f578288 1062ff10
r28-31  00000000 000002ee 5f578640 10120630

sr00-03 00000000 000013a7 00000000 000013a7 sr04-07 00000000 00000000 00000000 00000000

IASQ: 00000000 00000000 IAOQ: 1011ed04 1011ed08
IIR: 0e601093 ISR: 00000000 IOR: 000127ec
CPU: 0 CR30: 5f578000 CR31: ffffffff
ORIG_R28: 40400000
IAOQ[0]: unwind_frame_regs+0x214/0x244 IAOQ[1]: unwind_frame_regs+0x218/0x244 RP(r2): unwind_once+0x10/0x48

Backtrace:
 [<1011ee68>] unwind_once+0x10/0x48
 [<10112bbc>] do_show_stack+0x28/0xac
 [<10112cd0>] parisc_show_stack+0x90/0xa8
 [<10111258>] do_page_fault+0x174/0x294
 [<10113824>] handle_interruption+0x570/0x58c
 [<10117078>] intr_check_sig+0x0/0x34

Kernel panic - not syncing: Kernel Fault
#2 lore raw
Quoted 9 lines
Quoted 9 lines

Kernel Fault: Code=15 regs=5f578640 (Addr=000127ec)

YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111110000001111 Not tainted

r00-03  0004fc0f 10644f10 1011ee68 5f578288
r04-07  00000014 8c1fe860 8c1fe828 8eb7d030
r08-11  105ee890 00000040 0001264a 000d3b60
r12-15  00000000 000d3b5c 000dde68 000b0000
r16-19  8eb7d250 000b0000 ffffffff 000127ec

FWIW, this is the instruction:

dave@hiauly6:~/opt/gnu/bin$ disasm 0x0e601093
   0:	0e 60 10 93 	ldw 0(r19),r19

Dave

Signature · 2 lines
J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
author maintainer
#3 lore raw

PATCH 1/1 fix unwind crash - was: Re: 2.6.26 kernel crash

In reply to Helge Deller

I narrowed down to the problematic codepath, and I assume the attached patch might fix it. Problem is, that I can't test without my testcase which is on gsyprf10 (which is unreachable right now), so this patch here is currently a RFC...

Helge

Signed-off-by Helge Deller <[email protected]>
Quoted 18 lines · Helge Deller
Quoted 2 lines · Helge Deller

While debugging some user-space stuff I just faced this 32bit kernel crash (2.6.26):
Maybe someone has an idea?

This is reproduceable.
Testcase and docu is here: http://gsyprf10.external.hp.com/~deller/crash.tgz

Just run:
LD_LIBRARY_PATH=. ./a.out

Output is:
making contexts
swapping contexts
start f2
&on_stack=0x12d88
start f1(a0=1,a1=2,a2=3,a3=fffffffc)
<crash>

Quoted 37 lines

Info:
- test program is from glibc:
glibc-2.7/stdlib/tst-setcontext.c
- the compiled libc.so.6 includes some changes for *context() functions from me.

Quoted 31 lines

Kernel Fault: Code=15 regs=5f578640 (Addr=000127ec)

YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111110000001111 Not tainted

r00-03  0004fc0f 10644f10 1011ee68 5f578288
r04-07  00000014 8c1fe860 8c1fe828 8eb7d030
r08-11  105ee890 00000040 0001264a 000d3b60
r12-15  00000000 000d3b5c 000dde68 000b0000
r16-19  8eb7d250 000b0000 ffffffff 000127ec
r20-23  ffffe040 00000000 101205d4 00000000
r24-27  00012800 101205cc 5f578288 1062ff10
r28-31  00000000 000002ee 5f578640 10120630

sr00-03 00000000 000013a7 00000000 000013a7 sr04-07 00000000 00000000 00000000 00000000

IASQ: 00000000 00000000 IAOQ: 1011ed04 1011ed08
IIR: 0e601093 ISR: 00000000 IOR: 000127ec
CPU: 0 CR30: 5f578000 CR31: ffffffff
ORIG_R28: 40400000
IAOQ[0]: unwind_frame_regs+0x214/0x244 IAOQ[1]: unwind_frame_regs+0x218/0x244 RP(r2): unwind_once+0x10/0x48

Backtrace:
 [<1011ee68>] unwind_once+0x10/0x48
 [<10112bbc>] do_show_stack+0x28/0xac
 [<10112cd0>] parisc_show_stack+0x90/0xa8
 [<10111258>] do_page_fault+0x174/0x294
 [<10113824>] handle_interruption+0x570/0x58c
 [<10117078>] intr_check_sig+0x0/0x34

Kernel panic - not syncing: Kernel Fault
arch/parisc/kernel/unwind.c +9 -2
index 701b2d2..8b4708d 100644
@@ -341,8 +341,15 @@ static void unwind_frame_regs(struct unwind_frame_info *info)
 			info->prev_sp = info->sp - frame_size;
 			if (e->Millicode)
 				info->rp = info->r31;
-			else if (rpoffset)
-				info->rp = *(unsigned long *)(info->prev_sp - rpoffset);
+			else if (rpoffset) {
+				unsigned long tmp;
+
+				if (get_user(tmp,
+				       (unsigned long *)(info->prev_sp - rpoffset)))
+					info->rp = 0;
+				else
+					info->rp = tmp;
+			}
 			info->prev_ip = info->rp;
 			info->rp = 0;
 		}
author maintainer
#4 lore raw

Re: [PATCH] fix unwind crash - was: Re: 2.6.26 kernel crash

Hi Kyle,

I verified, that my attached patch fixes the kernel panic. Testcase is here: http://gsyprf10.external.hp.com/~deller/crash.tgz
Could you please apply the patch?

Signed-off-by Helge Deller <[email protected]>

Thanks,
Helge

PS:
arch/parisc/kernel/unwind.c, line 225 looks kinda fishy as well:
225: info->prev_ip = *(unsigned long *)(info->prev_sp - RP_OFFSET);

PPS:
Instead of a kernel panic (which is really annoying since you need to reboot the machine) I now get as expected an user fault:

do_page_fault() pid=1846 command='a.out' type=6 address=0x87802043 vm_start = 0x407ff000, vm_end = 0x40802000

YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111111100001111 Not tainted

r00-03  0004ff0f 407c9f6b 406fbfe3 00012b00
r04-07  fb4ec308 000125b8 407fd534 000e6ba8
r08-11  fb4ec014 00000001 0001264a 000d3b60
r12-15  00000000 000d3b5c 000db4c8 000b0000
r16-19  000d06a0 000b0000 ffffffff 23882000
r20-23  406fc15f 406fc138 87802042 00012d80
r24-27  407fd534 000125b8 407fd534 000125b8
r28-31  00000000 00000000 00012e00 406fbfe3

sr00-03 000007f6 000007f6 00000000 000007f6
sr04-07 000007f6 000007f6 000007f6 000007f6

IASQ: 000007f6 000007f6 IAOQ: 87802043 87802047
IIR: 43ffff80 ISR: 000007f6 IOR: 407d9adc
CPU: 0 CR30: 8dbd0000 CR31: d2b345e2
ORIG_R28: 000107c7
IAOQ[0]: 0x87802040
IAOQ[1]: 0x87802044
RP(r2): 0x406fbfe0

Backtrace:
 [<10120650>] outb+0x64/0x70
Quoted 64 lines · Helge Deller

I narrowed down to the problematic codepath, and I assume the attached patch might fix it. Problem is, that I can't test without my testcase which is on gsyprf10 (which is unreachable right now), so this patch here is currently a RFC...

Helge

Signed-off-by Helge Deller <[email protected]>
Quoted 56 lines · Helge Deller
Quoted 2 lines · Helge Deller

While debugging some user-space stuff I just faced this 32bit kernel crash (2.6.26):
Maybe someone has an idea?

This is reproduceable.
Testcase and docu is here: http://gsyprf10.external.hp.com/~deller/crash.tgz

Just run:
LD_LIBRARY_PATH=. ./a.out

Output is:
making contexts
swapping contexts
start f2
&on_stack=0x12d88
start f1(a0=1,a1=2,a2=3,a3=fffffffc)
<crash>

Info:
- test program is from glibc:
glibc-2.7/stdlib/tst-setcontext.c
- the compiled libc.so.6 includes some changes for *context() functions from me.

Quoted 31 lines

Kernel Fault: Code=15 regs=5f578640 (Addr=000127ec)

YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111110000001111 Not tainted

r00-03  0004fc0f 10644f10 1011ee68 5f578288
r04-07  00000014 8c1fe860 8c1fe828 8eb7d030
r08-11  105ee890 00000040 0001264a 000d3b60
r12-15  00000000 000d3b5c 000dde68 000b0000
r16-19  8eb7d250 000b0000 ffffffff 000127ec
r20-23  ffffe040 00000000 101205d4 00000000
r24-27  00012800 101205cc 5f578288 1062ff10
r28-31  00000000 000002ee 5f578640 10120630

sr00-03 00000000 000013a7 00000000 000013a7 sr04-07 00000000 00000000 00000000 00000000

IASQ: 00000000 00000000 IAOQ: 1011ed04 1011ed08
IIR: 0e601093 ISR: 00000000 IOR: 000127ec
CPU: 0 CR30: 5f578000 CR31: ffffffff
ORIG_R28: 40400000
IAOQ[0]: unwind_frame_regs+0x214/0x244 IAOQ[1]: unwind_frame_regs+0x218/0x244 RP(r2): unwind_once+0x10/0x48

Backtrace:
 [<1011ee68>] unwind_once+0x10/0x48
 [<10112bbc>] do_show_stack+0x28/0xac
 [<10112cd0>] parisc_show_stack+0x90/0xa8
 [<10111258>] do_page_fault+0x174/0x294
 [<10113824>] handle_interruption+0x570/0x58c
 [<10117078>] intr_check_sig+0x0/0x34

Kernel panic - not syncing: Kernel Fault
#5 lore raw

Re: [PATCH] fix unwind crash - was: Re: 2.6.26 kernel crash

In reply to Helge Deller

(Apologize for my earlier html mail - resent in a more proper format)

Helge, your patch doesn't look quite right.

The kernel unwinder is only supposed to be called for kernel addresses. Kyle says he thinks he knows what is wrong so he's going to poke at it.

Maybe we can make the kernel unwinder more robust against invalid addresses passed to it though.

thanks,
randolph

Quoted 91 lines · Helge Deller

I narrowed down to the problematic codepath, and I assume the attached patch might fix it. Problem is, that I can't test without my testcase which is on gsyprf10 (which is unreachable right now), so this patch here is currently a RFC...

Helge

Signed-off-by Helge Deller <[email protected]>
Quoted 58 lines · Helge Deller
Quoted 3 lines · Helge Deller

While debugging some user-space stuff I just faced this 32bit kernel crash (2.6.26):
Maybe someone has an idea?

This is reproduceable.
Testcase and docu is here: http://gsyprf10.external.hp.com/~deller/crash.tgz

Just run:
LD_LIBRARY_PATH=. ./a.out

Output is:
making contexts
swapping contexts
start f2
&on_stack=0x12d88
start f1(a0=1,a1=2,a2=3,a3=fffffffc)
<crash>

Info:
- test program is from glibc:
glibc-2.7/stdlib/tst-setcontext.c
- the compiled libc.so.6 includes some changes for *context() functions from me.

Quoted 32 lines

Kernel Fault: Code=15 regs=5f578640 (Addr=000127ec)

YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111110000001111 Not tainted

r00-03  0004fc0f 10644f10 1011ee68 5f578288
r04-07  00000014 8c1fe860 8c1fe828 8eb7d030
r08-11  105ee890 00000040 0001264a 000d3b60
r12-15  00000000 000d3b5c 000dde68 000b0000
r16-19  8eb7d250 000b0000 ffffffff 000127ec
r20-23  ffffe040 00000000 101205d4 00000000
r24-27  00012800 101205cc 5f578288 1062ff10
r28-31  00000000 000002ee 5f578640 10120630

sr00-03 00000000 000013a7 00000000 000013a7 sr04-07 00000000 00000000 00000000 00000000

IASQ: 00000000 00000000 IAOQ: 1011ed04 1011ed08
IIR: 0e601093 ISR: 00000000 IOR: 000127ec
CPU: 0 CR30: 5f578000 CR31: ffffffff
ORIG_R28: 40400000
IAOQ[0]: unwind_frame_regs+0x214/0x244 IAOQ[1]: unwind_frame_regs+0x218/0x244 RP(r2): unwind_once+0x10/0x48

Backtrace:
 [<1011ee68>] unwind_once+0x10/0x48
 [<10112bbc>] do_show_stack+0x28/0xac
 [<10112cd0>] parisc_show_stack+0x90/0xa8
 [<10111258>] do_page_fault+0x174/0x294
 [<10113824>] handle_interruption+0x570/0x58c
 [<10117078>] intr_check_sig+0x0/0x34

Kernel panic - not syncing: Kernel Fault
arch/parisc/kernel/unwind.c +9 -2
index 701b2d2..8b4708d 100644
@@ -341,8 +341,15 @@ static void unwind_frame_regs(struct unwind_frame_info *info)
                       info->prev_sp = info->sp - frame_size;
                       if (e->Millicode)
                               info->rp = info->r31;
-                       else if (rpoffset)
-                               info->rp = *(unsigned long *)(info->prev_sp - rpoffset);
+                       else if (rpoffset) {
+                               unsigned long tmp;
+
+                               if (get_user(tmp,
+                                      (unsigned long *)(info->prev_sp - rpoffset)))
+                                       info->rp = 0;
+                               else
+                                       info->rp = tmp;
+                       }
                       info->prev_ip = info->rp;
                       info->rp = 0;
               }
#6 lore raw

Re: [PATCH] fix unwind crash - was: Re: 2.6.26 kernel crash

Quoted 11 lines · Randolph Chung

(Apologize for my earlier html mail - resent in a more proper format)

Helge, your patch doesn't look quite right.

The kernel unwinder is only supposed to be called for kernel addresses. Kyle says he thinks he knows what is wrong so he's going to poke at it.

Maybe we can make the kernel unwinder more robust against invalid addresses passed to it though.

I think this will fix the problem, though I should audit show_stack callers to make sure.

arch/parisc/kernel/traps.c +5 -3
index 675f1d0..721d281 100644
@@ -121,11 +121,12 @@ static void print_fr(char *level, struct pt_regs *regs)
 
 void show_regs(struct pt_regs *regs)
 {
-	int i;
+	int i, user;
 	char *level;
 	unsigned long cr30, cr31;
 
-	level = user_mode(regs) ? KERN_DEBUG : KERN_CRIT;
+	user = !!user_mode(regs);
+	level = user ? KERN_DEBUG : KERN_CRIT;
 
 	print_gr(level, regs);
 

@@ -152,7 +153,8 @@ void show_regs(struct pt_regs *regs)
 	printk(level);
 	print_symbol(" RP(r2): %s\n", regs->gr[2]);
 
-	parisc_show_stack(current, NULL, regs);
+	if (!user)
+		parisc_show_stack(current, NULL, regs);
 }
#7 lore raw

Re: [PATCH] fix unwind crash - was: Re: 2.6.26 kernel crash

Quoted 1 line · Kyle McMartin

+ user = !!user_mode(regs);

Exqueeze me? Baking powder? Why can't the compiler just mush all of this up and get back the same thing you had before?

Cheers,
Carlos.

#8 lore raw

Re: [PATCH] fix unwind crash - was: Re: 2.6.26 kernel crash

In reply to Kyle McMartin

arch/parisc/kernel/traps.c void show_regs(struct pt_regs *regs) · line 157

Quoted 3 lines · Kyle McMartin
arch/parisc/kernel/traps.c +2 -1
-       parisc_show_stack(current, NULL, regs);
+       if (!user)
+               parisc_show_stack(current, NULL, regs);

or

if (level == KERN_CRIT)
parisc_show_stack (current, NULL, regs);

I think this bit of code is the only thing that is required e.g. Only call parisc_show_stack if it's not a usermode set of registers?

Cheers,
Carlos.