Double syscall exit traces on x86_64

We are seeing double ptrace notifications of system call returns on recent x86_64 kernels. This breaks UML and at least one other app.

The patch below appears to fix the problem. The bug is caused by both syscall_trace and int_very_careful both calling syscall_trace_leave, and the system call tracing path going through int_very_careful.

I would have liked to get rid of one or the other call to syscall_trace_leave. However, the syscall_trace path looks like it can exit to userspace without going through int_very_careful, and int_very_careful does things other than system call tracing.

So, instead, I took _TIF_SYSCALL_TRACE and _TIF_SYSCALL_AUDIT out of the flags test on the grounds that they had already been checked in syscall_trace. There is possibly a preemption and call to schedule between syscall_trace and int_very_careful, so if it can be attached at that point, then the first return will be missed. However, I think that ptrace attachment requires a stopped child, not just one that has been preempted.

I don't see signal delivery between syscall_trace and int_very_careful, so I don't see that there can be a ptrace attach followed by int_very_careful missing the first return.

This is an RFC - if it turns out to be actually correct, some comments need fixing before this goes anywhere.

UML works with this applied, and it doesn't seem to break singlestepping, either on normal instructions or across system calls, which looks like the next most vulnerable thing.

Jeff

Index: linux-2.6.16.x86_64/arch/x86_64/kernel/entry.S
===================================================================
@@ -345,7 +345,7 @@ int_very_careful:
 	sti
 	SAVE_REST
 	/* Check for syscall exit trace */	
-	testl $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP),%edx
+	testl $(_TIF_SINGLESTEP),%edx
 	jz int_signal
 	pushq %rdi
 	CFI_ADJUST_CFA_OFFSET 8

@@ -353,7 +353,7 @@ int_very_careful:
 	call syscall_trace_leave
 	popq %rdi
 	CFI_ADJUST_CFA_OFFSET -8
-	andl $~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP),%edi
+	andl $~(_TIF_SINGLESTEP),%edi
 	cli
 	jmp int_restore_rest
 	

I believe this patch is the correct fix. Can you confirm it works for you?

-Andi

Don't do syscall exit tracing twice

int_ret_from_syscall already does syscall exit tracing, so no need to do it again in the caller.

This caused problems for UML and some other special programs doing syscall interception.

Index: linux/arch/x86_64/kernel/entry.S
===================================================================
@@ -282,12 +282,7 @@ tracesys:			 
 	ja  1f
 	movq %r10,%rcx	/* fixup for C */
 	call *sys_call_table(,%rax,8)
-	movq %rax,RAX-ARGOFFSET(%rsp)
-1:	SAVE_REST
-	movq %rsp,%rdi
-	call syscall_trace_leave
-	RESTORE_TOP_OF_STACK %rbx
-	RESTORE_REST
+1:	movq %rax,RAX-ARGOFFSET(%rsp)
 	/* Use IRET because user could have changed frame */
 	jmp int_ret_from_sys_call
 	CFI_ENDPROC

Looks good, thanks.

Jeff

-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Sorry for the question, but has this been sent to -stable (since it's a -stable regression, it should be)? To 2.6.17 -git?

And have you tested it (somebody should have, but it's not sure)? Sorry for not helping myself, I'll be back at work ASAP.

Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

Chiacchiera con i tuoi amici in tempo reale! 
 http://it.yahoo.com/mail_it/foot/*http://it.messenger.yahoo.com 

It's in current git.

I'm having a hard time telling when the bug was introduced. The git web interface seems to be telling me that the double notification was around since last year, which I don't believe, since I've run much more recent x86_64 kernels. If the bug existed before 2.6.16, then it's fine -stable fodder.

Yes, it's been continuously and happily running UML since Andi sent me his patch.

Jeff

_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

I know that the bug is NOT present in 2.6.15.7.

The patch is likely ok for -stable, but below I express some doubts for inclusion in kernel tree - it probably isn't sufficient and maybe it's not the better fix.

Likely you'll looking into the wrong place - int_ret_from_syscall (if this is the name) wasn't used till recently, ret_from_syscall was used. And it never does syscall exit tracing - it expects the caller to have switched away to the slow path.

There's a path in which it calls int_with_check, but it does "movl $_TIF_NEED_RESCHED,%edi", so that int_with_check only tests _TIF_NEED_RESCHED and no other flag.

However, there's another potential problem in current code, it seems. ret_from_fork. It can jump to rff_trace and then again go to int_ret_from_sys_call, if _TIF_IA32 is set. The check for tracing/auditing should be moved to when we decide to jump to ret_from_sys_call.

I've tried doing that with the attached untested patch, but it's likely I've got something wrong - in particular the following pass of the patch wasn't easy to get right (this change is buried in the middle of other things).

+ testl $3,CS(%rsp) # from kernel_thread?
RESTORE_REST
- testl $3,CS-ARGOFFSET(%rsp) # from kernel_thread?

However, I have a bigger doubt and a proposal for a simpler change. Do we need at all to test for syscall tracing in int_ret_from_sys_call? And why is there a difference with ret_from_sys_call?

Instead of removing the test from tracesys, could we remove it from int_ret_from_sys_call (and no, calling int_with_check with a custom %edi doesn't work)?

Can syscall tracing be needed on exit and not on entry?

I've the suspect that either:
a) if some other process kicks in, we'll make him trace from next syscall start onwards (and int_ret_from_sys_call is buggy) - this makes much more sense and works almost always.
b) or if we want tracer to see syscall exit, it only works in int_ret_from_sys_call (rare cases).

I've thought to how a process can kick in and answers were:

*) schedule() to a process which does a ptrace attach *) the syscall is ptrace(PTRACE_TRACEME)

in both cases the ptracer must do PTRACE_SYSCALL, and in both cases we need behaviour a).

For singlestep on syscall exit I'm totally unsure. But I've not the time to look more this code - I didn't know it and it's very intricate.

Can you handle sending it to [email protected]?

First introduced in 2.6.16.5 as first diagnosed by Chuck Ebbert:

commit 6b12095a4a0e1f21bbf83f95f13299ca99d758fe
tree 5d2a3d96f7b99a3a225c0f7a110c6631848524b0
parent 59b2832a31ae2f3279bb5b16ae9b1c4e38e40dea
author Andi Kleen <[email protected]> Wed, 12 Apr 2006 08:19:29 +0200 committer Greg Kroah-Hartman <[email protected]> Wed, 12 Apr 2006 13:06:54 -0700

[PATCH] x86_64: When user could have changed RIP always force IRET
(CVE-2006-0744)

Intel EM64T CPUs handle uncanonical return addresses differently from AMD CPUs.

The exception is reported in the SYSRET, not the next instruction. Thgis leads to the kernel exception handler running on the user stack with the wrong GS because the kernel didn't expect exceptions on this instruction.

This version of the patch has the teething problems that plagued an earlier version fixed.

This is CVE-2006-0744

Thanks to Ernie Petrides and Asit B. Mallick for analysis and initial patches.

Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

Index: linux-2.6.16/arch/x86_64/kernel/entry.S
===================================================================
@@ -137,15 +137,17 @@
 ENTRY(ret_from_fork)
 	CFI_DEFAULT_STACK
 	call schedule_tail
+
+	testl $3,CS(%rsp)	# from kernel_thread?
+	je   rff_int_path
 	GET_THREAD_INFO(%rcx)
+	testl $_TIF_IA32,threadinfo_flags(%rcx)
+	jnz  rff_int_path
+
 	testl $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT),threadinfo_flags(%rcx)
 	jnz rff_trace
-rff_action:	
+rff_action:
 	RESTORE_REST
-	testl $3,CS-ARGOFFSET(%rsp)	# from kernel_thread?
-	je   int_ret_from_sys_call
-	testl $_TIF_IA32,threadinfo_flags(%rcx)
-	jnz  int_ret_from_sys_call
 	RESTORE_TOP_OF_STACK %rdi,ARGOFFSET
 	jmp ret_from_sys_call
 rff_trace:

@@ -154,6 +156,9 @@ rff_trace:
 	GET_THREAD_INFO(%rcx)	
 	jmp rff_action
 	CFI_ENDPROC
+rff_int_path:
+	RESTORE_REST
+	jmp  int_ret_from_sys_call
 
 /*
  * System call entry. Upto 6 arguments in registers are supported.

@@ -211,6 +216,7 @@ ENTRY(system_call)
 /*
  * Syscall return path ending with SYSRET (fast path)
  * Has incomplete stack frame and undefined top of stack. 
+ * Will not test _TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SECCOMP
  */		
 	.globl ret_from_sys_call
 ret_from_sys_call:

@@ -316,6 +322,7 @@ ENTRY(int_ret_from_sys_call)
 	testl $3,CS-ARGOFFSET(%rsp)
 	je retint_restore_args
 	movl $_TIF_ALLWORK_MASK,%edi
+
 	/* edi:	mask to check */
 int_with_check:
 	GET_THREAD_INFO(%rcx)