Skip to contentSkip to conversation

ESP corruption bug - what CPUs are affected?

DISCUSSION Stas Sergeev 3 participants 10 messages Sep 16, 2004 – Sep 22, 2004 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hello.

My question is not directly linux-related, but I think people here can give me some clue on an issue.

There is a "semi-official" bug in Intel CPUs,
which is described here:
http://www.intel.com/design/intarch/specupdt/27287402.PDF
chapter "Specification Clarifications"
section 4: "Use Of ESP In 16-Bit Code With 32-Bit Interrupt Handlers".

A short quote:
"ISSUE: When a 32-bit IRET is used to return to another privilege level,
and the old level uses a 4G stack (D/B bit in the segment register = 1),
while the new level uses a 64k stack (D/B bit = 0), then only the lower word of ESP is updated."

Which means that the higher word of ESP gets trashed. This bug beats dosemu rather badly, but there seem to be not much info about that bug available on the net.
What I want to find out, is what CPUs are affected. I wrote a small test-case. It is attached. I tried it on Intel Pentium and on AMD Athlon - bug is present on both. But I'd like to know if it is also present on a Transmeta Crusoe, Cyrix and other CPUs. Can someone please run the test-case on some of that CPUs and see how it goes?
Note: specifying "32" as an arg to the test-case, will make it to use the 32bit segment for the stack, and it will not detect the bug. It is there to prove that it detects the Right Thing. Run it without that argument.

I'd also like to know any thoughts on whether it is possible to work around the bug, probably in a kernel? Well, I am not hoping on such a possibility, but who knows...
Anyway, I'd be glad to get any info on that bug. Why it was not fixed for so many years, looks also like an interesting question, as for me.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <signal.h>
#include <linux/unistd.h>
#include <asm/ldt.h>
#include <asm/ucontext.h>

unsigned char stack[65536];

void my_segv(int signum, siginfo_t *info, void *c)
{
struct sigcontext *ctx = &((struct ucontext *)c)->uc_mcontext; printf("In sighandler: esp=%lx\n", ctx->esp);
/* Skip HLT */
ctx->eip++;
}

_syscall3(int, modify_ldt, int, func, void *, ptr, unsigned long, bytecount);

static int set_ldt_entry(int entry, unsigned long base, unsigned int limit,
int seg_32bit_flag, int contents, int read_only_flag, int limit_in_pages_flag, int seg_not_present, int useable)
{
struct modify_ldt_ldt_s ldt_info;
ldt_info.entry_number = entry;
ldt_info.base_addr = base;
ldt_info.limit = limit;
ldt_info.seg_32bit = seg_32bit_flag;
ldt_info.contents = contents;
ldt_info.read_exec_only = read_only_flag; ldt_info.limit_in_pages = limit_in_pages_flag; ldt_info.seg_not_present = seg_not_present;
ldt_info.useable = useable;

return modify_ldt(1, &ldt_info, sizeof(ldt_info));
}

int main(int argc, char *argv[])
{
unsigned short _ss, new_ss;
unsigned long _esp, new_esp;
int is32 = 0;
stack_t sig_stack;
struct sigaction sa;

  if (argc > 1 && !strncmp(argv[1], "32", 3))
    is32 = 1;

  sig_stack.ss_sp = stack;
  sig_stack.ss_flags = 0;
  sig_stack.ss_size = sizeof(stack);
  if (sigaltstack(&sig_stack, NULL)) {
    perror("sigaltstack()");
    return 1;
  }

  sa.sa_sigaction = my_segv;
  sigemptyset(&sa.sa_mask);
  sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
  sigaction(SIGSEGV, &sa, NULL);

  /* Get SS, ESP */
  asm volatile(
    "movw %%ss, %0\n"
    "movl %%esp, %1\n"
    :"=m"(_ss), "=m"(_esp)
  );
  printf("sp=%#lx, ss=%#hx\n", _esp, _ss);

  /* Force to LDT */
  new_ss = _ss | 4;
  /* Create the LDT entry */
  set_ldt_entry(new_ss >> 3, 0, 0xbffff, is32, MODIFY_LDT_CONTENTS_DATA, 0, 1, 0, 0);

/* Do the trick... Switch stack and then switch context. */ asm volatile(

    "movw %3, %%ss\n"		/* Load our LDT selector to SS */
    "hlt\n"			/* Force the context switch */
    "movw %1, %%ss\n"		/* Restore SS ASAP */
    "movl %%esp, %0\n"		/* Get new ESP */
    "movl %2, %%esp\n"		/* Restore ESP */
    :"=m"(new_esp)
    : "m"(_ss), "m"(_esp), "m"(new_ss)

);
printf("Now sp=%#lx, ss=%#hx\n", new_esp, new_ss);

/* See what we've got... */

  if (new_esp > 0xc0000000) {
    printf("BUG!\n");
  } else if (new_esp != _esp) {
    printf("Esp changed, strange...\n");
  } else {
    printf("No bug here! What CPU is this?\n");
    if (!is32)
      system("cat /proc/cpuinfo");

}
return 0;
}

#1 lore raw
Quoted 18 lines · Stas Sergeev

Hello.

My question is not directly linux-related,
but I think people here can give me some clue
on an issue.

There is a "semi-official" bug in Intel CPUs,
which is described here:
http://www.intel.com/design/intarch/specupdt/27287402.PDF
chapter "Specification Clarifications"
section 4: "Use Of ESP In 16-Bit Code With 32-Bit Interrupt Handlers".

A short quote:
"ISSUE: When a 32-bit IRET is used to return to another privilege level, and the old level uses a 4G stack (D/B bit in the segment register = 1), while the new level uses a 64k stack (D/B bit = 0), then only the lower word of ESP is updated."

Quoted 4 lines

Which means that the higher word of ESP gets
trashed. This bug beats dosemu rather badly,
but there seem to be not much info about that
bug available on the net.

Well. Strictly speaking it's not a bug.
Code executes as it should. IRET returns to the correct location. Upper 16 bits of ESP do not affect execution in this case.

You want CPU to preserve upper bits,
but this will waste a handful of transistors practically for no gain.

I think dosemu should just work around this. Is there some subtle problem with that in dosemu?

Quoted 14 lines

What I want to find out, is what CPUs are
affected. I wrote a small test-case. It is
attached. I tried it on Intel Pentium and
on AMD Athlon - bug is present on both. But
I'd like to know if it is also present on a
Transmeta Crusoe, Cyrix and other CPUs.
Can someone please run the test-case on some
of that CPUs and see how it goes?
Note: specifying "32" as an arg to the
test-case, will make it to use the 32bit
segment for the stack, and it will not detect
the bug. It is there to prove that it detects
the Right Thing. Run it without that argument.

Quoted 7 lines

I'd also like to know any thoughts on whether
it is possible to work around the bug, probably
in a kernel? Well, I am not hoping on such a
possibility, but who knows...
Anyway, I'd be glad to get any info on that bug.
Why it was not fixed for so many years, looks
also like an interesting question, as for me.

Because it does not matter for 99.9999% of users...

# gcc -O2 stk.c
# ./a.out
sp=0xbffffb30, ss=0x7b
In sighandler: esp=bffffb10
Now sp=0xccf1fb10, ss=0x7f
Esp changed, strange...
# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 8
model name      : Unknown CPU Typ
stepping        : 1
cpu MHz         : 1743.632
cache size      : 64 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow
--
vda
#2 lore raw

Hello.

Quoted 8 lines · Denis Vlasenko
Quoted 4 lines

Which means that the higher word of ESP gets trashed. This bug beats dosemu rather badly, but there seem to be not much info about that bug available on the net.

Well. Strictly speaking it's not a bug.
Code executes as it should. IRET returns to the correct location. Upper 16 bits of ESP do not affect execution in this case.

Unfortunately this is not true. Mainly because the code is still 32bit, so when is operates with the stack pointer directly, it gets ESP, not SP. Here are a few examples from the real DOS progs:
---
frstor [esp] <--crash
---

push    ebp
mov     ebp,esp
mov     edx,[ebp+0x8]  <--crash

---
and there are much more. "mov ebp,esp" is just a standard thing, and then the ebp is being used to access the locals and function arguments. Why do they use the 16bit stack in that case, is another question. Unfortunately some of them do.

Quoted 3 lines

You want CPU to preserve upper bits,
but this will waste a handful of transistors
practically for no gain.

I don't think so. After all, if only the B bit is set, this is being done. So I guess all the transistors are in place (well, of course this is just a wild guess).

Quoted 1 line

I think dosemu should just work around this.

I see no way to work around this in user-space. Esp. since dosemu doesn't control the execution completely, it acts as a monitor, only getting a control when exceptions occur. For dosemu code there are no problems, but the DOS program that is running, crashes.

Quoted 1 line

Is there some subtle problem with that in dosemu?

Nothing subtle I think. The problem is real and quite hurts. Fortunately it happened that using the dos32a extender instead of dos4gw "fixes" it for the dos4gw-based apps. But users wants everything out of the box, and never read docs or even the error messages in a log, so this doesn't really help:) And there are some not dos4gw-based progs that are affected either.

Quoted 3 lines
Quoted 2 lines

Why it was not fixed for so many years, looks also like an interesting question, as for me.

Because it does not matter for 99.9999% of users...

Aparently you are underestimating the amount of the dosemu users out there:) AFAIK it is also a problem for Win95 DOS box sometimes.

Quoted 6 lines
# gcc -O2 stk.c
# ./a.out
sp=0xbffffb30, ss=0x7b
In sighandler: esp=bffffb10
Now sp=0xccf1fb10, ss=0x7f
Esp changed, strange...

That's strange indeed! Apparently your ESP value, 0xccf1fb10, is greater than 0xc0000000, in which case my program should just write "BUG!", but for you it doesn't. Haven't you altered the code somehow? Why I compare it with 0xc0000000, is because ESP has the higher word of a kernel's stack address, so it should be above the TASK_SIZE when corrupted.

#3 lore raw
Quoted 20 lines · Stas Sergeev

Hello.

Quoted 9 lines · Denis Vlasenko
Quoted 4 lines

Which means that the higher word of ESP gets trashed. This bug beats dosemu rather badly, but there seem to be not much info about that bug available on the net.

Well. Strictly speaking it's not a bug.
Code executes as it should. IRET returns to the correct location. Upper 16 bits of ESP do not affect execution in this case.

Unfortunately this is not true. Mainly because the code is still 32bit, so when is operates with the stack pointer directly, it gets ESP, not SP. Here are a few examples from the real DOS progs:
---
frstor [esp] <--crash
---

Quoted 4 lines
push    ebp
mov     ebp,esp
mov     edx,[ebp+0x8]  <--crash

---

Hmmm. Let me dust off my i386 knwoledge.

This is what happens: CPU is doing IRET in 32bit mode. Stack layout:

+20 (empty) <--- SS:ESP
+16 old_CS
+12 old_EIP
+8 old_EFLAGS
+4 old_SS
+0 old_ESP

If old_SS references 'small' data segment (16-bit one), processor does not restore ESP from old_ESP.
It restores lower 16 bits only. Upper 16 bits are filled with garbage (or just not modified at all?).
This works fine because processor uses SP, not ESP, for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.

Quoted 5 lines

and there are much more. "mov ebp,esp" is just a standard thing, and then the ebp is being used to access the locals and function arguments. Why do they use the 16bit stack in that case, is another question. Unfortunately some of them do.

Hmm. I think you need to *emulate* this IRET.
Is this IRET happens in kernel or in dosemu code?

Quoted 7 lines
Quoted 6 lines
# gcc -O2 stk.c
# ./a.out
sp=0xbffffb30, ss=0x7b
In sighandler: esp=bffffb10
Now sp=0xccf1fb10, ss=0x7f
Esp changed, strange...
Quoted 5 lines

That's strange indeed! Apparently your ESP value, 0xccf1fb10, is greater than 0xc0000000, in which case my program should just write "BUG!", but for you it doesn't. Haven't you altered the code somehow?

No, I didn't alter anything. Strange indeed.

Quoted 4 lines

Why I compare it with 0xc0000000, is because ESP has the higher word of a kernel's stack address, so it should be above the TASK_SIZE when corrupted.

Even more strange! Now I did modify the code (attached), and this is what I've got:

# gcc -o stk2 -O2 stk2.c
# ./stk2
sp=0xbffffb30, ss=0x7b
In sighandler: esp=bffffb10
Now sp=cd79fb10   <=========
Now sp=bffffb50   <========= !?
Now sp=bffffb50, ss=007f
Now sp=bffffb50
Now sp=bffffb50
Now sp=bffffb50
Esp changed, strange...

I am thoroughly confused now. Two back-to-back
'printf("Now sp=%08lx\n", new_esp);'
gave different result. How this can happen??

Corresponding gcc -S code. new_esp is at -180(%ebp):

#APP

        movw -182(%ebp), %ss

hlt
movw -170(%ebp), %ss
movl %esp, -180(%ebp)
movl -176(%ebp), %esp

#NO_APP

        addl    $40, %esp
        pushl   -180(%ebp)
        pushl   $.LC4
        call    printf
        popl    %eax
        popl    %edx
        pushl   -180(%ebp)
        pushl   $.LC4
        call    printf
        addl    $12, %esp
        movzwl  -182(%ebp), %eax
        pushl   %eax
        pushl   -180(%ebp)
        pushl   $.LC5
        call    printf
        popl    %ecx
        popl    %ebx
        pushl   -180(%ebp)
        pushl   $.LC4
        call    printf
        popl    %eax
        popl    %edx

--
vda

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <signal.h>
#include <linux/unistd.h>
#include <asm/ldt.h>
#include <asm/ucontext.h>

unsigned char stack[65536];

void my_segv(int signum, siginfo_t *info, void *c)
{
struct sigcontext *ctx = &((struct ucontext *)c)->uc_mcontext; printf("In sighandler: esp=%lx\n", ctx->esp);
/* Skip HLT */
ctx->eip++;
}

_syscall3(int, modify_ldt, int, func, void *, ptr, unsigned long, bytecount);

static int set_ldt_entry(int entry, unsigned long base, unsigned int limit,
int seg_32bit_flag, int contents, int read_only_flag, int limit_in_pages_flag, int seg_not_present, int useable)
{
struct modify_ldt_ldt_s ldt_info;
ldt_info.entry_number = entry;
ldt_info.base_addr = base;
ldt_info.limit = limit;
ldt_info.seg_32bit = seg_32bit_flag;
ldt_info.contents = contents;
ldt_info.read_exec_only = read_only_flag; ldt_info.limit_in_pages = limit_in_pages_flag; ldt_info.seg_not_present = seg_not_present;
ldt_info.useable = useable;

return modify_ldt(1, &ldt_info, sizeof(ldt_info));
}

int main(int argc, char *argv[])
{
unsigned short _ss, new_ss;
unsigned long _esp, new_esp;
int is32 = 0;
stack_t sig_stack;
struct sigaction sa;

  if (argc > 1 && !strncmp(argv[1], "32", 3))
    is32 = 1;

  sig_stack.ss_sp = stack;
  sig_stack.ss_flags = 0;
  sig_stack.ss_size = sizeof(stack);
  if (sigaltstack(&sig_stack, NULL)) {
    perror("sigaltstack()");
    return 1;
  }

  sa.sa_sigaction = my_segv;
  sigemptyset(&sa.sa_mask);
  sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
  sigaction(SIGSEGV, &sa, NULL);

  /* Get SS, ESP */
  asm volatile(
    "movw %%ss, %0\n"
    "movl %%esp, %1\n"
    :"=m"(_ss), "=m"(_esp)
  );
  printf("sp=%#lx, ss=%#hx\n", _esp, _ss);

  /* Force to LDT */
  new_ss = _ss | 4;
  /* Create the LDT entry */
  set_ldt_entry(new_ss >> 3, 0, 0xbffff, is32, MODIFY_LDT_CONTENTS_DATA, 0, 1, 0, 0);

  /* Do the trick... Switch stack and then switch context. */
  asm volatile(
    "movw %3, %%ss\n"		/* Load our LDT selector to SS */
    "hlt\n"			/* Force the context switch */
    "movw %1, %%ss\n"		/* Restore SS ASAP */
    "movl %%esp, %0\n"		/* Get new ESP */
    "movl %2, %%esp\n"		/* Restore ESP */
    :"=m"(new_esp)
    : "m"(_ss), "m"(_esp), "m"(new_ss)
  );
  printf("Now sp=%08lx\n", new_esp);
  printf("Now sp=%08lx\n", new_esp);
  printf("Now sp=%08lx, ss=%04hx\n", new_esp, new_ss);
  printf("Now sp=%08lx\n", new_esp);
  printf("Now sp=%08lx\n", new_esp);

/* See what we've got... */

  if (new_esp > 0xc0000000) {
    printf("Now sp=%08lx\n", new_esp);
    printf("BUG!\n");
  } else if (new_esp != _esp) {
    printf("Now sp=%08lx\n", new_esp);
    printf("Esp changed, strange...\n");
  } else {
    printf("Now sp=%08lx\n", new_esp);
    printf("No bug here! What CPU is this?\n");
    if (!is32)
      system("cat /proc/cpuinfo");

}
return 0;
}

#4 lore raw

Hello.

Quoted 8 lines · Denis Vlasenko
Quoted 2 lines

This is what happens: CPU is doing IRET in 32bit mode. Stack layout:

+20 (empty) <--- SS:ESP
+16 old_CS
+12 old_EIP
+8 old_EFLAGS
+4 old_SS
+0 old_ESP

JFYI, looks like you swapped CS<-->EIP and SS<-->ESP on your stack frame layout.

Quoted 4 lines

If old_SS references 'small' data segment (16-bit one), processor does not restore ESP from old_ESP.
It restores lower 16 bits only. Upper 16 bits are filled with garbage (or just not modified at all?).

Not modified at all, yes. That's why it is always greater than TASK_SIZE (0xc0000000) - it is still in a kernel space.

Quoted 4 lines

This works fine because processor uses SP, not ESP, for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.

That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero... My test-case does "hlt" to force the context switch, but in fact it is not necessary. I tried an empty loop instead of HLT. If the loop is long enough and some IRQ being handled by the kernel in a mean time, the ESP is corrupted. So the user-space never knows when exactly it gets corrupted. And, as I mentioned already, since the code is 32bit, it just does the things like "mov ebp,esp" and uses ebp to access the function params and locals, which crashes, or uses ESP directly to access something, etc.

Quoted 2 lines

Hmm. I think you need to *emulate* this IRET.
Is this IRET happens in kernel or in dosemu code?

The problem happens only when iret is used to switch to another priv level. So it is the kernel's iret of course, something to the effect of entry.S:128 I think. The control is transferred from kernel code to the DOS code directly. dosemu code is completely bypassed. dosemu have no chances to intercept that. The DOS program is running on its own (as long as it doesn't trigger an exception, or SIGALRM comes), and the CPU is corrupting its ESP in *any* place, since it happens when an external IRQ arrives.

Quoted 6 lines
Quoted 5 lines

That's strange indeed! Apparently your ESP value, 0xccf1fb10, is greater than 0xc0000000, in which case my program should just write "BUG!", but for you it doesn't. Haven't you altered the code somehow?

No, I didn't alter anything. Strange indeed.

So that proves that writing the bug-free code, esp. when the asm is involved, is not always as easy as I assume...

Quoted 3 lines

I am thoroughly confused now. Two back-to-back
'printf("Now sp=%08lx\n", new_esp);'
gave different result. How this can happen??

My code did the assumption that the ESP should not change within the execution of main(). This is true only if you don't enable the optimization (as I did), or use -fno-defer-pop option of gcc. Fixed code is attached (just for the overall completeness:) Should survive the optimization now.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <signal.h>
#include <linux/unistd.h>
#include <asm/ldt.h>
#include <asm/ucontext.h>

unsigned char stack[65536];

void my_segv(int signum, siginfo_t *info, void *c)
{
struct sigcontext *ctx = &((struct ucontext *)c)->uc_mcontext; printf("In sighandler: esp=%lx\n", ctx->esp);
/* Skip HLT */
ctx->eip++;
}

_syscall3(int, modify_ldt, int, func, void *, ptr, unsigned long, bytecount)

static int set_ldt_entry(int entry, unsigned long base, unsigned int limit,
int seg_32bit_flag, int contents, int read_only_flag, int limit_in_pages_flag, int seg_not_present, int useable)
{
struct modify_ldt_ldt_s ldt_info;
ldt_info.entry_number = entry;
ldt_info.base_addr = base;
ldt_info.limit = limit;
ldt_info.seg_32bit = seg_32bit_flag;
ldt_info.contents = contents;
ldt_info.read_exec_only = read_only_flag; ldt_info.limit_in_pages = limit_in_pages_flag; ldt_info.seg_not_present = seg_not_present;
ldt_info.useable = useable;

return modify_ldt(1, &ldt_info, sizeof(ldt_info));
}

int main(int argc, char *argv[])
{
unsigned short _ss, new_ss;
unsigned long _esp, new_esp;
int is32 = 0;
stack_t sig_stack;
struct sigaction sa;

  if (argc > 1 && !strncmp(argv[1], "32", 3))
    is32 = 1;

  sig_stack.ss_sp = stack;
  sig_stack.ss_flags = 0;
  sig_stack.ss_size = sizeof(stack);
  if (sigaltstack(&sig_stack, NULL)) {
    perror("sigaltstack()");
    return 1;
  }

  sa.sa_sigaction = my_segv;
  sigemptyset(&sa.sa_mask);
  sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
  sigaction(SIGSEGV, &sa, NULL);

/* Get SS */
asm volatile(

    "movw %%ss, %0\n"
    :"=m"(_ss)

);

  /* Force to LDT */
  new_ss = _ss | 4;
  /* Create the LDT entry */
  set_ldt_entry(new_ss >> 3, 0, 0xbffff, is32, MODIFY_LDT_CONTENTS_DATA, 0, 1, 0, 0);
  printf("old_ss=%#hx new_ss=%#hx\n", _ss, new_ss);

/* Do the trick... Switch stack and then switch context. */ asm volatile(

    "movl %%esp, %1\n"		/* Save ESP */
    "movw %2, %%ss\n"		/* Load our LDT selector to SS */
    "hlt\n"			/* Force the context switch */
    "movw %3, %%ss\n"		/* Restore SS ASAP */
    "movl %%esp, %0\n"		/* Get new ESP */
    "movl %1, %%esp\n"		/* Restore ESP */
    :"=m"(new_esp), "=a"(_esp)
    :"m"(new_ss), "m"(_ss)

);
printf("old_esp=%#lx new_esp=%#lx\n", _esp, new_esp);

/* See what we've got... */

  if (new_esp > 0xc0000000) {
    printf("BUG!\n");
  } else if (new_esp != _esp) {
    printf("Esp changed, strange...\n");
  } else {
    printf("No bug here! What CPU is this?\n");
    if (!is32)
      system("cat /proc/cpuinfo");

}
return 0;
}

#5 lore raw
Quoted 13 lines · Stas Sergeev

Hello.

Quoted 9 lines · Denis Vlasenko
Quoted 2 lines

This is what happens: CPU is doing IRET in 32bit mode. Stack layout:

+20 (empty) <--- SS:ESP
+16 old_CS
+12 old_EIP
+8 old_EFLAGS
+4 old_SS
+0 old_ESP

Quoted 2 lines

JFYI, looks like you swapped CS<-->EIP and SS<-->ESP on your stack frame layout.

Hm. More than that. My figure is totally incorrect. Corrected one:

+16 old_SS
+12 old_ESP
+8 old_EFLAGS
+4 old_CS
+0 old_EIP <--- SS:ESP

Addresses grow from the bottom in this fig. Note that offsets are always stored in lower address relative to where selectors are stored.

Quoted 33 lines
Quoted 4 lines

If old_SS references 'small' data segment (16-bit one), processor does not restore ESP from old_ESP.
It restores lower 16 bits only. Upper 16 bits are filled with garbage (or just not modified at all?).

Not modified at all, yes. That's why it is always greater than TASK_SIZE (0xc0000000) - it is still in a kernel space.

Quoted 4 lines

This works fine because processor uses SP, not ESP, for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.

That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero... My test-case does "hlt" to force the context switch, but in fact it is not necessary. I tried an empty loop instead of HLT. If the loop is long enough and some IRQ being handled by the kernel in a mean time, the ESP is corrupted. So the user-space never knows when exactly it gets corrupted. And, as I mentioned already, since the code is 32bit, it just does the things like "mov ebp,esp" and uses ebp to access the function params and locals, which crashes, or uses ESP directly to access something, etc.

Quoted 2 lines

Hmm. I think you need to *emulate* this IRET.
Is this IRET happens in kernel or in dosemu code?

Quoted 10 lines

The problem happens only when iret is used to switch to another priv level. So it is the kernel's iret of course, something to the effect of entry.S:128 I think. The control is transferred from kernel code to the DOS code directly. dosemu code is completely bypassed. dosemu have no chances to intercept that. The DOS program is running on its own (as long as it doesn't trigger an exception, or SIGALRM comes), and the CPU is corrupting its ESP in *any* place, since it happens when an external IRQ arrives.

Aha. The only way to sanely handle it is to
hack on entry.S I'm afraid. Something like rewriting CS:EIP so that it returns to a small ring-3 trampoline which clears upper 16 bits of ESP and jumps to original CS:EIP.

You may use ring-3 user stack as a place to save original CS:EIP, and then just do RETF. This way, trampoline boils down to

	movzxl	%sp,%esp
	retf

which does not touch EFLAGS and you don't need to bother with thinking about preserving it. IRET will restore it, and trampoline wouldn't touch it anymore.

Now, how to detect when to use this? Hmm.... the simplest thing is to check that

(old_ESP <= 0xffff) && !(old_EFLAGS & VM_MASK) && (descr_old_SS is 16bit one)

This will cost us only one comparison in the normal path, because typically ESP of Linus executables is greater than 0xffff.

P.S.

# gcc -o stk1 -O2 stk1.c
# ./stk1
old_ss=0x7b new_ss=0x7f
In sighandler: esp=bffffb30
old_esp=0xbffffb30 new_esp=0xc618fb30
BUG!

--
vda

#6 lore raw

Hi Denis.

Quoted 4 lines · Denis Vlasenko

Aha. The only way to sanely handle it is to
hack on entry.S I'm afraid. Something like rewriting CS:EIP so that it returns to a small ring-3 trampoline which clears upper 16 bits of ESP and jumps to original CS:EIP.

Well, I don't really want to clear the higher word of ESP (even if I want to do that in most cases), but I'd really like to just restore it properly. Of course (I think) ring-3 trampoline will not work for many reasons. The most obvious one is that it itself can be interrupted in any place. Another problem with it is that the return frame will have to be pushed to the stack of a DOS prog. This is not the good thing to do. dosemu avoids ever touching the stack of a DOS prog by setting up the sigaltstack.
What *will* work, however, is a ring-1 trampoline, as Petr Vandrovec suggested. It can be executed with interrupts disabled (I think) so will not be interrupted, and it can (isn't it?) use the kernel stack, if I understand that correctly.

Quoted 3 lines

Now, how to detect when to use this? Hmm.... the simplest thing is to check that
(old_ESP <= 0xffff) && !(old_EFLAGS & VM_MASK) && (descr_old_SS is 16bit one)

Yes, that's an excellent idea (except for the ESP<=0xffff check - I don't think this one is necessary).

Quoted 2 lines

This will cost us only one comparison in the normal path,

Yes, so I think I should just try to implement that. Not as a ring-3 trampoline, but as a ring-1 one.

Quoted 2 lines

because typically ESP of Linus executables
is greater than 0xffff.

I'd rather say, because the stack segment is 32bit for them always.
And this all should work, as it seems to me. Thanks for the hint about the checking.

#9 lore raw
In reply to Stas Sergeev

Hi!

Quoted 7 lines
Quoted 3 lines

for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.

That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero...

Hmm, perhaps this can also be viewed as a "information leak"? Program running under dosemu is not expected to know high bits of kernel %esp...
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#10 lore raw
Quoted 6 lines · Pavel Machek

Hi!

Quoted 4 lines
Quoted 3 lines

for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.

Quoted 8 lines
Quoted 4 lines

That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero...

Hmm, perhaps this can also be viewed as a "information leak"? Program running under dosemu is not expected to know high bits of kernel %esp...

Yes.

I must admit that Stas was right and I was wrong.
This is a 386 CPU bug.

Since then it seems to be codified in i86 architecture and duplicated in all successors. Incredible... :(
--
vda

#8 lore raw
In reply to a message not in the archive

Hi,

Quoted 1 line · Denis Vlasenko

Maybe. This would be a complicated thing.

I bet it was! :)

Quoted 4 lines

Well. Not okay. Maybe this?
1. We build IRET frame on ring1 stack for step 4 (see below), modify IRET frame on ring0 stack so that intrs are disabled and CS:EIP and SS:ESP point to values of ring1 code/stack.

Much simpler: IRET frame to return to user is already there. Just push another one to return to ring1 first.

Quoted 3 lines

2. IRET returns to ring1 code with dedicated *16-bit* ring1 stack Upper word of ESP is wrong now, but we can safely fiddle with it.
3. trampoline code fixes upper word of ESP (how?)

popl %esp (as per Petr Vandrovec's suggestion) The value on stack is carefully prepared on ring0.

Quoted 2 lines

4. trampoline IRETs to user code.
May work.

Works!

Quoted 1 line

ring1 stacks must be per-CPU.

I allocate it on a ring0 stack. Noone seem to suggest that. Is this flawed for some reasons?

Quoted 5 lines
Quoted 2 lines

ESP<=0xffff check - I don't think this one is
necessary).

Any program which runs with 16bit stack and yet with ESP > 0xffff is doing something *terminally* weird. I think it is acceptable to leave this case unfixed.

For what? We can have that fixed so why not?

Quoted 2 lines

You cannot check 16bitness of SS descriptor in two insns.

I do actually:
larl OLDSS(%esp), %eax
testl $0x00400000, %eax
which is exactly two insns.

Since I've forgot to CC the patch to you, I uploaded it here:
http://www.dosemu.org/stas/linux-2.6.8-stacks2.diff
so that you (or anyone interested) can make a review.