Skip to contentSkip to conversation

32bit binaries on x86_64/Xen segfaults in syscall-vdso

DISCUSSION Bastian Blank 2 participants 10 messages Aug 30, 2009 – Sep 4, 2009

lore
Share & tools
lore
#0 lore raw

Hi folks

I upgraded one of my 32bit chroots on a x86-64 machine runing under Xen lately. All binaries started to segfault. Some extensive checks later show the vdso as the culprit. Later I found <[email protected]> with the same problem. The full story can be found in the Debian bug 544145[1].

It happens with Linux 2.6.30 and 2.6.31-rc8 on Xen 3.2 and 3.4.

For the tests I set the vdso to compat mode to have it loaded on a fixed location.

The following program is a minimal test case for the vdso in compat mode, it can be compiled against dietlibc to minimize other effects.

| int main() {
|   unsigned int resultvar;
|   asm volatile (
|     "movl %1, %%eax\n\t"
|     "call 0xffffe420\n\t"
|     : "=a" (resultvar) : "i" (0) : "memory", "cc");
| }
| (gdb) run
| Starting program: /test 
| 
| Program received signal SIGSEGV, Segmentation fault.
| 0xffffe42f in ?? ()
| (gdb) bt
| #0  0xffffe42f in ?? ()
| #1  0xf7eb17a5 in __libc_start_main (main=0x8048394 <main>, argc=1, ubp_av=0xffffd884, init=0x80483d0 <__libc_csu_init>, 
|     fini=0x80483c0 <__libc_csu_fini>, rtld_fini=0xf7fee6e0 <_dl_fini>, stack_end=0xffffd87c) at libc-start.c:222
| #2  0x08048301 in _start () at ../sysdeps/i386/elf/start.S:119
| (gdb) disassemble 0xffffe420 0xffffe430
| Dump of assembler code from 0xffffe420 to 0xffffe430:
| 0xffffe420:     push   %ebp
| 0xffffe421:     mov    %ecx,%ebp
| 0xffffe423:     syscall 
| 0xffffe425:     mov    $0x2b,%ecx
| 0xffffe42a:     mov    %ecx,%ss
| 0xffffe42c:     mov    %ebp,%ecx
| 0xffffe42e:     pop    %ebp
| 0xffffe42f:     ret    
| End of assembler dump.

It segfaults on the ret opcode, in some variants also directly after the ret. If I single-step over the syscall opcode it works. The register contents slightly differ in this case.

Break on last opcode, state at the last opcode:

| (gdb) b *0xffffe42f
| Breakpoint 7 at 0xffffe42f
| (gdb) run
| Starting program: /test
| 
| Breakpoint 7, 0xffffe42f in ?? ()
| (gdb) info registers
| eax            0xfffffffc       -4
| ecx            0xffffd800       -10240
| edx            0xffffd820       -10208
| ebx            0xf7fd7ff4       -134381580
| esp            0xffffd7d0       0xffffd7d0
| ebp            0xffffd7e8       0xffffd7e8
| esi            0x80483d0        134513616
| edi            0x80482e0        134513376
| eip            0xffffe42f       0xffffe42f
| eflags         0x282    [ SF IF ]
| cs             0xe033   57395
| ss             0x2b     43
| ds             0x2b     43
| es             0x2b     43
| fs             0x0      0
| gs             0x63     99

Break on first opcode, state at the last opcode:

| (gdb) b *0xffffe420
| Breakpoint 8 at 0xffffe420
| (gdb) run
| Starting program: /test
| 
| Breakpoint 8, 0xffffe420 in ?? ()
| (gdb) stepi
| 0xffffe421 in ?? ()
[...]
| 0xffffe42f in ?? ()
| (gdb) info registers
| eax            0xfffffffc       -4
| ecx            0xffffd800       -10240
| edx            0xffffd820       -10208
| ebx            0xf7fd7ff4       -134381580
| esp            0xffffd7cc       0xffffd7cc
| ebp            0xffffd7e8       0xffffd7e8
| esi            0x80483d0        134513616
| edi            0x80482e0        134513376
| eip            0xffffe42f       0xffffe42f
| eflags         0x282    [ SF IF ]
| cs             0x23     35
| ss             0x2b     43
| ds             0x2b     43
| es             0x2b     43
| fs             0x0      0
| gs             0x63     99

The stack pointer and code segment are different in this two cases.

I think I found the problem. In the normal codeflow, sysret is used to return as expected. In the compat codeflow, iret is used.

Bastian

[1]: http://bugs.debian.org/544145

Signature · 2 lines
Wait!  You have not been prepared!
		-- Mr. Atoz, "Tomorrow is Yesterday", stardate 3113.2
#1 lore raw
Quoted 8 lines · Bastian Blank

Hi folks

I upgraded one of my 32bit chroots on a x86-64 machine runing under Xen lately. All binaries started to segfault. Some extensive checks later show the vdso as the culprit. Later I found <[email protected]> with the same problem. The full story can be found in the Debian bug 544145[1].

Quoted 8 lines

It happens with Linux 2.6.30 and 2.6.31-rc8 on Xen 3.2 and 3.4.

For the tests I set the vdso to compat mode to have it loaded on a fixed location.

The following program is a minimal test case for the vdso in compat mode, it can be compiled against dietlibc to minimize other effects.

Is this an AMD machine? Does booting with vdso32=0 on the kernel command line work around the problem?

J

#2 lore raw
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 8 lines · Bastian Blank

I upgraded one of my 32bit chroots on a x86-64 machine runing under Xen lately. All binaries started to segfault. Some extensive checks later show the vdso as the culprit. Later I found <[email protected]> with the same problem. The full story can be found in the Debian bug 544145[1].

It happens with Linux 2.6.30 and 2.6.31-rc8 on Xen 3.2 and 3.4.

Quoted 8 lines
Quoted 5 lines

For the tests I set the vdso to compat mode to have it loaded on a fixed location.

The following program is a minimal test case for the vdso in compat mode, it can be compiled against dietlibc to minimize other effects.

Is this an AMD machine? Does booting with vdso32=0 on the kernel command line work around the problem?

AFAIK only AMD support the syscall instruction, so yes it is an AMD machine. And yes, disabling the only thing that make the glibc call this instruction works around it.

Bastian

Signature · 2 lines
Conquest is easy. Control is not.
		-- Kirk, "Mirror, Mirror", stardate unknown
#3 lore raw
Quoted 4 lines · Bastian Blank

AFAIK only AMD support the syscall instruction, so yes it is an AMD machine. And yes, disabling the only thing that make the glibc call this instruction works around it.

The bug actually appears to be in xen_sysret32, ie the crash happens on the way out of the kernel.

J

#4 lore raw
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 3 lines · Bastian Blank

AFAIK only AMD support the syscall instruction, so yes it is an AMD machine. And yes, disabling the only thing that make the glibc call this instruction works around it.

The bug actually appears to be in xen_sysret32, ie the crash happens on the way out of the kernel.

This function looks weird. It tries to restores the user code segment. But the documentation from AMD explicitely stat that the CS and SS are restored from the STAR register.

Bastian

Signature · 2 lines
Killing is stupid; useless!
		-- McCoy, "A Private Little War", stardate 4211.8
#5 lore raw
Quoted 3 lines · Bastian Blank

This function looks weird. It tries to restores the user code segment. But the documentation from AMD explicitely stat that the CS and SS are restored from the STAR register.

And STAR is always set with:

    wrmsrl(MSR_STAR,  ((u64)__USER32_CS)<<48  | ((u64)__KERNEL_CS)<<32);

so when using sysret to return to 32-bit, it:

The CS selector value is set to MSR IA32_STAR[63:48]. The SS is set to IA32_STAR[63:48] + 8.

so CS is __USER32_CS and SS is __USER32_DS.

The code for xen_sysret32 is:

ENTRY(xen_sysret32)

	/*
	 * We're already on the usermode stack at this point, but
	 * still with the kernel gs, so we can easily switch back
	 */
	movq %rsp, PER_CPU_VAR(old_rsp)
	movq PER_CPU_VAR(kernel_stack), %rsp

	pushq $__USER32_DS
	pushq PER_CPU_VAR(old_rsp)
	pushq %r11
	pushq $__USER32_CS
	pushq %rcx

	pushq $VGCF_in_syscall

1: jmp hypercall_iret

The iret frame is:

ss
rsp
rflags
cs
rip <-- rsp

so this constructs a frame of:

	__USER32_DS
	user_esp
	user_eflags
	__USER32_CS
	user_eip	<-- kernel rsp

and then it does the iret hypercall.

But for some reason that's triggering a failsafe callback, which invokes a GP.

J

#6 lore raw
Quoted 2 lines · Jeremy Fitzhardinge

But for some reason that's triggering a failsafe callback, which invokes a GP.

Hmm, not in my tests. It always returned to userspace correctly and died some operations later, usually the "ret". This then produced either a segfault (unreadable address), sigill (if it managed to reach the ELF header of the ld.so) or a GPF.

Bastian

Signature · 2 lines
You!  What PLANET is this!
		-- McCoy, "The City on the Edge of Forever", stardate 3134.0
#7 lore raw
Quoted 9 lines · Bastian Blank
Quoted 3 lines · Jeremy Fitzhardinge

But for some reason that's triggering a failsafe callback, which invokes a GP.

Hmm, not in my tests. It always returned to userspace correctly and died some operations later, usually the "ret". This then produced either a segfault (unreadable address), sigill (if it managed to reach the ELF header of the ld.so) or a GPF.

Hm, I may have misdiagnosed it then. Your symptoms are odd; either its landing back in userspace in the right place but then stumbles on for a while before crashing (wrong processor mode?) or the eip is wrong and its just landing in the wrong place and crashing immediately.

How non-deterministic is it? Does it differ every time, or from boot to boot, build to build?

J

#8 lore raw
In reply to Jeremy Fitzhardinge
Quoted 7 lines · Jeremy Fitzhardinge
Quoted 3 lines · Bastian Blank

This function looks weird. It tries to restores the user code segment. But the documentation from AMD explicitely stat that the CS and SS are restored from the STAR register.

And STAR is always set with:

    wrmsrl(MSR_STAR,  ((u64)__USER32_CS)<<48  | ((u64)__KERNEL_CS)<<32);

No. This is the normal kernel setup. But the Xen setup (the relevant one) looks different:

| #define FLAT_RING3_CS32 0xe023
| wrmsr(MSR_STAR, 0, (FLAT_RING3_CS32<<16) | __HYPERVISOR_CS);

But this does not match my observation either.

And even the native Linux kernel uses "iret" to jump out of a compat (32bit) syscall. No, I don't want to understand this, but it looks highly broken.

Bastian

Signature · 1 line
Captain's Log, star date 21:34.5...
#9 lore raw
Quoted 8 lines · Bastian Blank
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 3 lines · Bastian Blank

This function looks weird. It tries to restores the user code segment. But the documentation from AMD explicitely stat that the CS and SS are restored from the STAR register.

| #define FLAT_RING3_CS32 0xe023
| wrmsr(MSR_STAR, 0, (FLAT_RING3_CS32<<16) | __HYPERVISOR_CS); But this does not match my observation either.

Well, it does. The values for a long-mode program within Xen:

| cs             0xe033   57395
| ss             0xe02b   57387

Values on the bare hardware:

| cs             0x33     51
| ss             0x2b     43

Values for a compatibility-mode program on the bare hardware:

| cs             0x23     35
| ss             0x2b     43

So Xen adds 0xe000 (no idea what that means), but the Linux kernel expects the value without. Long mode is not affected.

Okay, I tried the test program again and yes, it jumps back into long-mode. (See the 0x10 in the restored CS[1].)

| cs             0xe033   57395
| ss             0xe02b   57387

Bastian

[1]: http://amd.com/us-en/assets/content_type/white_papers_and_tech_docs/24593.pdf,

     page 151
Signature · 2 lines
Where there's no emotion, there's no motive for violence.
		-- Spock, "Dagger of the Mind", stardate 2715.1