Skip to contentSkip to conversation

x86: don't destroy %rbp on kernel-mode faults

RFC PATCH Superseded Vegard Nossum 3 participants 5 messages May 19, 2008 – May 22, 2008 x86

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 x86: don't destroy %rbp on kernel-mode faults

Hi,

The RFC part of this patch is: Does anybody see why touching %rcx would be bad? It certainly looks like %ecx is free. This fixes the stacktrace problem I was seeing, and Pekka tested a bootup to userspace. (Pekka also did half of the debugging. When will git allow multiple authors for a patch? :-))

Vegard

Quoted 1 line

From b1cbf24fcd05aa5ed2e610c80c06bc519d3188f7 Mon Sep 17 00:00:00 2001

From: Vegard Nossum <[email protected]>
Date: Mon, 19 May 2008 21:39:44 +0200
Subject: [PATCH] x86: don't destroy %rbp on kernel-mode faults

Quoted 1 line

From the code:

B stepping K8s sometimes report an truncated RIP for IRET exceptions returning to compat mode. Check for these here too.

The code then proceeds to truncate the upper 32 bits of %rbp. This means that when do_page_fault() is finally called, its prologue,

do_page_fault:
push %rbp
movl %rsp, %rbp

will put the truncated base pointer on the stack. This means that the stack tracer will not be able to follow the base-pointer changes and will see all subsequent stack frames as unreliable.

This patch changes the code to use a different register (%rcx) for the checking and leaves %rbp untouched.

Cc Andi Kleen <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arjan van de Ven <[email protected]>
---
 arch/x86/kernel/entry_64.S |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/entry_64.S +4 -4
index 1edd9ac..ff53692 100644
@@ -926,11 +926,11 @@ error_kernelspace:
 	   iret run with kernel gs again, so don't set the user space flag.
 	   B stepping K8s sometimes report an truncated RIP for IRET 
 	   exceptions returning to compat mode. Check for these here too. */
-	leaq irq_return(%rip),%rbp
-	cmpq %rbp,RIP(%rsp) 
+	leaq irq_return(%rip),%rcx
+	cmpq %rcx,RIP(%rsp) 
 	je   error_swapgs
-	movl %ebp,%ebp	/* zero extend */
-	cmpq %rbp,RIP(%rsp) 
+	movl %ecx,%ecx	/* zero extend */
+	cmpq %rcx,RIP(%rsp) 
 	je   error_swapgs
 	cmpq $gs_change,RIP(%rsp)
         je   error_swapgs
Signature · 1 line
1.5.4.1
Acked-by on v1 1/1
#1 lore raw
Quoted 33 lines · Vegard Nossum

Hi,

The RFC part of this patch is: Does anybody see why touching %rcx would be bad? It certainly looks like %ecx is free. This fixes the stacktrace problem I was seeing, and Pekka tested a bootup to userspace. (Pekka also did half of the debugging. When will git allow multiple authors for a patch? :-))

Vegard

From b1cbf24fcd05aa5ed2e610c80c06bc519d3188f7 Mon Sep 17 00:00:00 2001
From: Vegard Nossum <[email protected]>
Date: Mon, 19 May 2008 21:39:44 +0200
Subject: [PATCH] x86: don't destroy %rbp on kernel-mode faults

From the code:

B stepping K8s sometimes report an truncated RIP for IRET exceptions returning to compat mode. Check for these here too.

The code then proceeds to truncate the upper 32 bits of %rbp. This means that when do_page_fault() is finally called, its prologue,

do_page_fault:
push %rbp
movl %rsp, %rbp

will put the truncated base pointer on the stack. This means that the stack tracer will not be able to follow the base-pointer changes and will see all subsequent stack frames as unreliable.

Quoted 8 lines

This patch changes the code to use a different register (%rcx) for the checking and leaves %rbp untouched.

Cc Andi Kleen <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arjan van de Ven <[email protected]>

looks good to me; good debugging!

#2 lore raw
In reply to Vegard Nossum
Quoted 7 lines · Vegard Nossum

Hi,

The RFC part of this patch is: Does anybody see why touching %rcx would be bad? It certainly looks like %ecx is free. This fixes the stacktrace problem I was seeing, and Pekka tested a bootup to userspace. (Pekka also did half of the debugging. When will git allow multiple authors for a patch? :-))

The patch is ok, but I'm sure there's lots of other assembler code that destroys %rbp when it was saved elsewhere.

When I wrote all the assembler the assumption was always that a real unwinder would be used for backtraces, not frame pointer.

-Andi

#3 lore raw
Quoted 11 lines · Andi Kleen
Quoted 7 lines · Vegard Nossum

Hi,

The RFC part of this patch is: Does anybody see why touching %rcx would be bad? It certainly looks like %ecx is free. This fixes the stacktrace problem I was seeing, and Pekka tested a bootup to userspace. (Pekka also did half of the debugging. When will git allow multiple authors for a patch? :-))

The patch is ok, but I'm sure there's lots of other assembler code that destroys %rbp when it was saved elsewhere.

Thanks, The real intention of this code (you might have guessed it) was to fix kmemcheck on 64-bit, and it did, so I'm happy. If we (or others) hit another similar case, I'm sure we'll be able to fix those too.

The problem seems to be that %rbp was never restored before it was used again, and that's what I consider the real error in this case. I changed it to use a different register for the temporary computation, but restoring %rbp from wherever it was stored would also have been a valid, albeit less efficient, solution.

Quoted 2 lines

When I wrote all the assembler the assumption was always that a real unwinder would be used for backtraces, not frame pointer.

Hm, I am not sure exactly what a "real unwinder" would be. But I do think it's fair to say that it is the assembly code in this case that is violating the binary interface, and not the stack tracer code.

Vegard

Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036
#4 lore raw
Quoted 1 line

Hm, I am not sure exactly what a "real unwinder" would be. But I do

A dwarf2 unwinder that doesn't require pipe line stalls on many CPUs on each function entry point for setting up a frame.

Instead of letting all code
maintain a frame at runtime the stack frames are described by an external unwind table that is then walked by the unwinder.

The unwinder was in for a short time, but
Linus unfortunately removed it again because it took some time to debug it in tree and he lost patience. I believe an updated and stable version is available in the SUSE kernels.

Quoted 2 lines

think it's fair to say that it is the assembly code in this case that is violating the binary interface, and not the stack tracer code.

There is a no binary interface for page faults (or other exceptions) except that "all registers must be restored in the end". They certainly don't follow the normal ABI.

Still the fix is good, just pointing out that you'll likely need to change a lot more code to get the frame pointer fully supported everywhere because it was all written under the explicit "no frame pointer" assumption.

-Andi (who still prefers the unwinder)