ESP corruption bug - what CPUs are affected?
Hello.
My question is not directly linux-related, but I think people here can give me some clue on an issue.
There is a "semi-official" bug in Intel CPUs,
which is described here:
http://www.intel.com/design/intarch/specupdt/27287402.PDF
chapter "Specification Clarifications"
section 4: "Use Of ESP In 16-Bit Code With 32-Bit Interrupt Handlers".
A short quote:
"ISSUE: When a 32-bit IRET is used to return to another privilege level,
and the old level uses a 4G stack (D/B bit in the segment register = 1),
while the new level uses a 64k stack (D/B bit = 0), then only the lower word of ESP is updated."
Which means that the higher word of ESP gets trashed. This bug beats dosemu rather badly, but there seem to be not much info about that bug available on the net.
What I want to find out, is what CPUs are affected. I wrote a small test-case. It is attached. I tried it on Intel Pentium and on AMD Athlon - bug is present on both. But I'd like to know if it is also present on a Transmeta Crusoe, Cyrix and other CPUs. Can someone please run the test-case on some of that CPUs and see how it goes?
Note: specifying "32" as an arg to the test-case, will make it to use the 32bit segment for the stack, and it will not detect the bug. It is there to prove that it detects the Right Thing. Run it without that argument.
I'd also like to know any thoughts on whether it is possible to work around the bug, probably in a kernel? Well, I am not hoping on such a possibility, but who knows...
Anyway, I'd be glad to get any info on that bug. Why it was not fixed for so many years, looks also like an interesting question, as for me.
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <signal.h>
#include <linux/unistd.h>
#include <asm/ldt.h>
#include <asm/ucontext.h>
unsigned char stack[65536];
void my_segv(int signum, siginfo_t *info, void *c)
{
struct sigcontext *ctx = &((struct ucontext *)c)->uc_mcontext; printf("In sighandler: esp=%lx\n", ctx->esp);
/* Skip HLT */
ctx->eip++;
}
_syscall3(int, modify_ldt, int, func, void *, ptr, unsigned long, bytecount);
static int set_ldt_entry(int entry, unsigned long base, unsigned int limit,
int seg_32bit_flag, int contents, int read_only_flag, int limit_in_pages_flag, int seg_not_present, int useable)
{
struct modify_ldt_ldt_s ldt_info;
ldt_info.entry_number = entry;
ldt_info.base_addr = base;
ldt_info.limit = limit;
ldt_info.seg_32bit = seg_32bit_flag;
ldt_info.contents = contents;
ldt_info.read_exec_only = read_only_flag; ldt_info.limit_in_pages = limit_in_pages_flag; ldt_info.seg_not_present = seg_not_present;
ldt_info.useable = useable;
return modify_ldt(1, &ldt_info, sizeof(ldt_info));
}
int main(int argc, char *argv[])
{
unsigned short _ss, new_ss;
unsigned long _esp, new_esp;
int is32 = 0;
stack_t sig_stack;
struct sigaction sa;
if (argc > 1 && !strncmp(argv[1], "32", 3))
is32 = 1;
sig_stack.ss_sp = stack;
sig_stack.ss_flags = 0;
sig_stack.ss_size = sizeof(stack);
if (sigaltstack(&sig_stack, NULL)) {
perror("sigaltstack()");
return 1;
}
sa.sa_sigaction = my_segv;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
sigaction(SIGSEGV, &sa, NULL);
/* Get SS, ESP */
asm volatile(
"movw %%ss, %0\n"
"movl %%esp, %1\n"
:"=m"(_ss), "=m"(_esp)
);
printf("sp=%#lx, ss=%#hx\n", _esp, _ss);
/* Force to LDT */
new_ss = _ss | 4;
/* Create the LDT entry */
set_ldt_entry(new_ss >> 3, 0, 0xbffff, is32, MODIFY_LDT_CONTENTS_DATA, 0, 1, 0, 0);/* Do the trick... Switch stack and then switch context. */ asm volatile(
"movw %3, %%ss\n" /* Load our LDT selector to SS */
"hlt\n" /* Force the context switch */
"movw %1, %%ss\n" /* Restore SS ASAP */
"movl %%esp, %0\n" /* Get new ESP */
"movl %2, %%esp\n" /* Restore ESP */
:"=m"(new_esp)
: "m"(_ss), "m"(_esp), "m"(new_ss) );
printf("Now sp=%#lx, ss=%#hx\n", new_esp, new_ss);
/* See what we've got... */
if (new_esp > 0xc0000000) {
printf("BUG!\n");
} else if (new_esp != _esp) {
printf("Esp changed, strange...\n");
} else {
printf("No bug here! What CPU is this?\n");
if (!is32)
system("cat /proc/cpuinfo"); }
return 0;
}
Quoted 18 lines · Stas Sergeev
Hello.
My question is not directly linux-related,
but I think people here can give me some clue
on an issue.There is a "semi-official" bug in Intel CPUs,
which is described here:
http://www.intel.com/design/intarch/specupdt/27287402.PDF
chapter "Specification Clarifications"
section 4: "Use Of ESP In 16-Bit Code With 32-Bit Interrupt Handlers".A short quote:
"ISSUE: When a 32-bit IRET is used to return to another privilege level, and the old level uses a 4G stack (D/B bit in the segment register = 1), while the new level uses a 64k stack (D/B bit = 0), then only the lower word of ESP is updated."
Quoted 4 lines
Which means that the higher word of ESP gets
trashed. This bug beats dosemu rather badly,
but there seem to be not much info about that
bug available on the net.
Well. Strictly speaking it's not a bug.
Code executes as it should. IRET returns to the correct location. Upper 16 bits of ESP do not affect execution in this case.
You want CPU to preserve upper bits,
but this will waste a handful of transistors practically for no gain.
I think dosemu should just work around this. Is there some subtle problem with that in dosemu?
Quoted 14 lines
What I want to find out, is what CPUs are
affected. I wrote a small test-case. It is
attached. I tried it on Intel Pentium and
on AMD Athlon - bug is present on both. But
I'd like to know if it is also present on a
Transmeta Crusoe, Cyrix and other CPUs.
Can someone please run the test-case on some
of that CPUs and see how it goes?
Note: specifying "32" as an arg to the
test-case, will make it to use the 32bit
segment for the stack, and it will not detect
the bug. It is there to prove that it detects
the Right Thing. Run it without that argument.
Quoted 7 lines
I'd also like to know any thoughts on whether
it is possible to work around the bug, probably
in a kernel? Well, I am not hoping on such a
possibility, but who knows...
Anyway, I'd be glad to get any info on that bug.
Why it was not fixed for so many years, looks
also like an interesting question, as for me.
Because it does not matter for 99.9999% of users...
# gcc -O2 stk.c # ./a.out sp=0xbffffb30, ss=0x7b In sighandler: esp=bffffb10 Now sp=0xccf1fb10, ss=0x7f Esp changed, strange... # cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 8 model name : Unknown CPU Typ stepping : 1 cpu MHz : 1743.632 cache size : 64 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow -- vda
Hello.
Quoted 8 lines · Denis Vlasenko
Quoted 4 lines
Which means that the higher word of ESP gets trashed. This bug beats dosemu rather badly, but there seem to be not much info about that bug available on the net.
Well. Strictly speaking it's not a bug.
Code executes as it should. IRET returns to the correct location. Upper 16 bits of ESP do not affect execution in this case.
Unfortunately this is not true. Mainly because the code is still 32bit, so when is operates with the stack pointer directly, it gets ESP, not SP. Here are a few examples from the real DOS progs:
---
frstor [esp] <--crash
---
push ebp mov ebp,esp mov edx,[ebp+0x8] <--crash
---
and there are much more. "mov ebp,esp" is just a standard thing, and then the ebp is being used to access the locals and function arguments. Why do they use the 16bit stack in that case, is another question. Unfortunately some of them do.
Quoted 3 lines
You want CPU to preserve upper bits,
but this will waste a handful of transistors
practically for no gain.
I don't think so. After all, if only the B bit is set, this is being done. So I guess all the transistors are in place (well, of course this is just a wild guess).
Quoted 1 line
I think dosemu should just work around this.
I see no way to work around this in user-space. Esp. since dosemu doesn't control the execution completely, it acts as a monitor, only getting a control when exceptions occur. For dosemu code there are no problems, but the DOS program that is running, crashes.
Quoted 1 line
Is there some subtle problem with that in dosemu?
Nothing subtle I think. The problem is real and quite hurts. Fortunately it happened that using the dos32a extender instead of dos4gw "fixes" it for the dos4gw-based apps. But users wants everything out of the box, and never read docs or even the error messages in a log, so this doesn't really help:) And there are some not dos4gw-based progs that are affected either.
Quoted 3 lines
Quoted 2 lines
Why it was not fixed for so many years, looks also like an interesting question, as for me.
Because it does not matter for 99.9999% of users...
Aparently you are underestimating the amount of the dosemu users out there:) AFAIK it is also a problem for Win95 DOS box sometimes.
Quoted 6 lines
# gcc -O2 stk.c # ./a.out sp=0xbffffb30, ss=0x7b In sighandler: esp=bffffb10 Now sp=0xccf1fb10, ss=0x7f Esp changed, strange...
That's strange indeed! Apparently your ESP value, 0xccf1fb10, is greater than 0xc0000000, in which case my program should just write "BUG!", but for you it doesn't. Haven't you altered the code somehow? Why I compare it with 0xc0000000, is because ESP has the higher word of a kernel's stack address, so it should be above the TASK_SIZE when corrupted.
Quoted 20 lines · Stas Sergeev
Hello.
Quoted 9 lines · Denis Vlasenko
Quoted 4 lines
Which means that the higher word of ESP gets trashed. This bug beats dosemu rather badly, but there seem to be not much info about that bug available on the net.
Well. Strictly speaking it's not a bug.
Code executes as it should. IRET returns to the correct location. Upper 16 bits of ESP do not affect execution in this case.Unfortunately this is not true. Mainly because the code is still 32bit, so when is operates with the stack pointer directly, it gets ESP, not SP. Here are a few examples from the real DOS progs:
---
frstor [esp] <--crash
---
Quoted 4 lines
push ebp mov ebp,esp mov edx,[ebp+0x8] <--crash---
Hmmm. Let me dust off my i386 knwoledge.
This is what happens: CPU is doing IRET in 32bit mode. Stack layout:
+20 (empty) <--- SS:ESP
+16 old_CS
+12 old_EIP
+8 old_EFLAGS
+4 old_SS
+0 old_ESP
If old_SS references 'small' data segment (16-bit one), processor does not restore ESP from old_ESP.
It restores lower 16 bits only. Upper 16 bits are filled with garbage (or just not modified at all?).
This works fine because processor uses SP, not ESP, for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.
Quoted 5 lines
and there are much more. "mov ebp,esp" is just a standard thing, and then the ebp is being used to access the locals and function arguments. Why do they use the 16bit stack in that case, is another question. Unfortunately some of them do.
Hmm. I think you need to *emulate* this IRET.
Is this IRET happens in kernel or in dosemu code?
Quoted 7 lines
Quoted 6 lines
# gcc -O2 stk.c # ./a.out sp=0xbffffb30, ss=0x7b In sighandler: esp=bffffb10 Now sp=0xccf1fb10, ss=0x7f Esp changed, strange...
Quoted 5 lines
That's strange indeed! Apparently your ESP value,
0xccf1fb10, is greater than0xc0000000, in which case my program should just write "BUG!", but for you it doesn't. Haven't you altered the code somehow?
No, I didn't alter anything. Strange indeed.
Quoted 4 lines
Why I compare it with
0xc0000000, is because ESP has the higher word of a kernel's stack address, so it should be above theTASK_SIZEwhen corrupted.
Even more strange! Now I did modify the code (attached), and this is what I've got:
# gcc -o stk2 -O2 stk2.c # ./stk2 sp=0xbffffb30, ss=0x7b In sighandler: esp=bffffb10 Now sp=cd79fb10 <========= Now sp=bffffb50 <========= !? Now sp=bffffb50, ss=007f Now sp=bffffb50 Now sp=bffffb50 Now sp=bffffb50 Esp changed, strange...
I am thoroughly confused now. Two back-to-back
'printf("Now sp=%08lx\n", new_esp);'
gave different result. How this can happen??
Corresponding gcc -S code. new_esp is at -180(%ebp):
#APP
movw -182(%ebp), %ss
hlt
movw -170(%ebp), %ss
movl %esp, -180(%ebp)
movl -176(%ebp), %esp
#NO_APP
addl $40, %esp
pushl -180(%ebp)
pushl $.LC4
call printf
popl %eax
popl %edx
pushl -180(%ebp)
pushl $.LC4
call printf
addl $12, %esp
movzwl -182(%ebp), %eax
pushl %eax
pushl -180(%ebp)
pushl $.LC5
call printf
popl %ecx
popl %ebx
pushl -180(%ebp)
pushl $.LC4
call printf
popl %eax
popl %edx--
vda
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <signal.h>
#include <linux/unistd.h>
#include <asm/ldt.h>
#include <asm/ucontext.h>
unsigned char stack[65536];
void my_segv(int signum, siginfo_t *info, void *c)
{
struct sigcontext *ctx = &((struct ucontext *)c)->uc_mcontext; printf("In sighandler: esp=%lx\n", ctx->esp);
/* Skip HLT */
ctx->eip++;
}
_syscall3(int, modify_ldt, int, func, void *, ptr, unsigned long, bytecount);
static int set_ldt_entry(int entry, unsigned long base, unsigned int limit,
int seg_32bit_flag, int contents, int read_only_flag, int limit_in_pages_flag, int seg_not_present, int useable)
{
struct modify_ldt_ldt_s ldt_info;
ldt_info.entry_number = entry;
ldt_info.base_addr = base;
ldt_info.limit = limit;
ldt_info.seg_32bit = seg_32bit_flag;
ldt_info.contents = contents;
ldt_info.read_exec_only = read_only_flag; ldt_info.limit_in_pages = limit_in_pages_flag; ldt_info.seg_not_present = seg_not_present;
ldt_info.useable = useable;
return modify_ldt(1, &ldt_info, sizeof(ldt_info));
}
int main(int argc, char *argv[])
{
unsigned short _ss, new_ss;
unsigned long _esp, new_esp;
int is32 = 0;
stack_t sig_stack;
struct sigaction sa;
if (argc > 1 && !strncmp(argv[1], "32", 3))
is32 = 1;
sig_stack.ss_sp = stack;
sig_stack.ss_flags = 0;
sig_stack.ss_size = sizeof(stack);
if (sigaltstack(&sig_stack, NULL)) {
perror("sigaltstack()");
return 1;
}
sa.sa_sigaction = my_segv;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
sigaction(SIGSEGV, &sa, NULL);
/* Get SS, ESP */
asm volatile(
"movw %%ss, %0\n"
"movl %%esp, %1\n"
:"=m"(_ss), "=m"(_esp)
);
printf("sp=%#lx, ss=%#hx\n", _esp, _ss);
/* Force to LDT */
new_ss = _ss | 4;
/* Create the LDT entry */
set_ldt_entry(new_ss >> 3, 0, 0xbffff, is32, MODIFY_LDT_CONTENTS_DATA, 0, 1, 0, 0);
/* Do the trick... Switch stack and then switch context. */
asm volatile(
"movw %3, %%ss\n" /* Load our LDT selector to SS */
"hlt\n" /* Force the context switch */
"movw %1, %%ss\n" /* Restore SS ASAP */
"movl %%esp, %0\n" /* Get new ESP */
"movl %2, %%esp\n" /* Restore ESP */
:"=m"(new_esp)
: "m"(_ss), "m"(_esp), "m"(new_ss)
);
printf("Now sp=%08lx\n", new_esp);
printf("Now sp=%08lx\n", new_esp);
printf("Now sp=%08lx, ss=%04hx\n", new_esp, new_ss);
printf("Now sp=%08lx\n", new_esp);
printf("Now sp=%08lx\n", new_esp);/* See what we've got... */
if (new_esp > 0xc0000000) {
printf("Now sp=%08lx\n", new_esp);
printf("BUG!\n");
} else if (new_esp != _esp) {
printf("Now sp=%08lx\n", new_esp);
printf("Esp changed, strange...\n");
} else {
printf("Now sp=%08lx\n", new_esp);
printf("No bug here! What CPU is this?\n");
if (!is32)
system("cat /proc/cpuinfo"); }
return 0;
}
Hello.
Quoted 8 lines · Denis Vlasenko
Quoted 2 lines
This is what happens: CPU is doing IRET in 32bit mode. Stack layout:
+20 (empty) <--- SS:ESP
+16old_CS
+12old_EIP
+8old_EFLAGS
+4old_SS
+0old_ESP
JFYI, looks like you swapped CS<-->EIP and SS<-->ESP on your stack frame layout.
Quoted 4 lines
If
old_SSreferences 'small' data segment (16-bit one), processor does not restore ESP fromold_ESP.
It restores lower 16 bits only. Upper 16 bits are filled with garbage (or just not modified at all?).
Not modified at all, yes. That's why it is always greater than TASK_SIZE (0xc0000000) - it is still in a kernel space.
Quoted 4 lines
This works fine because processor uses SP, not ESP, for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.
That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero... My test-case does "hlt" to force the context switch, but in fact it is not necessary. I tried an empty loop instead of HLT. If the loop is long enough and some IRQ being handled by the kernel in a mean time, the ESP is corrupted. So the user-space never knows when exactly it gets corrupted. And, as I mentioned already, since the code is 32bit, it just does the things like "mov ebp,esp" and uses ebp to access the function params and locals, which crashes, or uses ESP directly to access something, etc.
Quoted 2 lines
Hmm. I think you need to *emulate* this IRET.
Is this IRET happens in kernel or in dosemu code?
The problem happens only when iret is used to switch to another priv level. So it is the kernel's iret of course, something to the effect of entry.S:128 I think. The control is transferred from kernel code to the DOS code directly. dosemu code is completely bypassed. dosemu have no chances to intercept that. The DOS program is running on its own (as long as it doesn't trigger an exception, or SIGALRM comes), and the CPU is corrupting its ESP in *any* place, since it happens when an external IRQ arrives.
Quoted 6 lines
Quoted 5 lines
That's strange indeed! Apparently your ESP value,
0xccf1fb10, is greater than0xc0000000, in which case my program should just write "BUG!", but for you it doesn't. Haven't you altered the code somehow?No, I didn't alter anything. Strange indeed.
So that proves that writing the bug-free code, esp. when the asm is involved, is not always as easy as I assume...
Quoted 3 lines
I am thoroughly confused now. Two back-to-back
'printf("Now sp=%08lx\n", new_esp);'
gave different result. How this can happen??
My code did the assumption that the ESP should not change within the execution of main(). This is true only if you don't enable the optimization (as I did), or use -fno-defer-pop option of gcc. Fixed code is attached (just for the overall completeness:) Should survive the optimization now.
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <signal.h>
#include <linux/unistd.h>
#include <asm/ldt.h>
#include <asm/ucontext.h>
unsigned char stack[65536];
void my_segv(int signum, siginfo_t *info, void *c)
{
struct sigcontext *ctx = &((struct ucontext *)c)->uc_mcontext; printf("In sighandler: esp=%lx\n", ctx->esp);
/* Skip HLT */
ctx->eip++;
}
_syscall3(int, modify_ldt, int, func, void *, ptr, unsigned long, bytecount)
static int set_ldt_entry(int entry, unsigned long base, unsigned int limit,
int seg_32bit_flag, int contents, int read_only_flag, int limit_in_pages_flag, int seg_not_present, int useable)
{
struct modify_ldt_ldt_s ldt_info;
ldt_info.entry_number = entry;
ldt_info.base_addr = base;
ldt_info.limit = limit;
ldt_info.seg_32bit = seg_32bit_flag;
ldt_info.contents = contents;
ldt_info.read_exec_only = read_only_flag; ldt_info.limit_in_pages = limit_in_pages_flag; ldt_info.seg_not_present = seg_not_present;
ldt_info.useable = useable;
return modify_ldt(1, &ldt_info, sizeof(ldt_info));
}
int main(int argc, char *argv[])
{
unsigned short _ss, new_ss;
unsigned long _esp, new_esp;
int is32 = 0;
stack_t sig_stack;
struct sigaction sa;
if (argc > 1 && !strncmp(argv[1], "32", 3))
is32 = 1;
sig_stack.ss_sp = stack;
sig_stack.ss_flags = 0;
sig_stack.ss_size = sizeof(stack);
if (sigaltstack(&sig_stack, NULL)) {
perror("sigaltstack()");
return 1;
}
sa.sa_sigaction = my_segv;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
sigaction(SIGSEGV, &sa, NULL); /* Get SS */
asm volatile(
"movw %%ss, %0\n"
:"=m"(_ss));
/* Force to LDT */
new_ss = _ss | 4;
/* Create the LDT entry */
set_ldt_entry(new_ss >> 3, 0, 0xbffff, is32, MODIFY_LDT_CONTENTS_DATA, 0, 1, 0, 0);
printf("old_ss=%#hx new_ss=%#hx\n", _ss, new_ss);/* Do the trick... Switch stack and then switch context. */ asm volatile(
"movl %%esp, %1\n" /* Save ESP */
"movw %2, %%ss\n" /* Load our LDT selector to SS */
"hlt\n" /* Force the context switch */
"movw %3, %%ss\n" /* Restore SS ASAP */
"movl %%esp, %0\n" /* Get new ESP */
"movl %1, %%esp\n" /* Restore ESP */
:"=m"(new_esp), "=a"(_esp)
:"m"(new_ss), "m"(_ss) );
printf("old_esp=%#lx new_esp=%#lx\n", _esp, new_esp);
/* See what we've got... */
if (new_esp > 0xc0000000) {
printf("BUG!\n");
} else if (new_esp != _esp) {
printf("Esp changed, strange...\n");
} else {
printf("No bug here! What CPU is this?\n");
if (!is32)
system("cat /proc/cpuinfo"); }
return 0;
}
Quoted 13 lines · Stas Sergeev
Hello.
Quoted 9 lines · Denis Vlasenko
Quoted 2 lines
This is what happens: CPU is doing IRET in 32bit mode. Stack layout:
+20 (empty) <--- SS:ESP
+16old_CS
+12old_EIP
+8old_EFLAGS
+4old_SS
+0old_ESP
Quoted 2 lines
JFYI, looks like you swapped CS<-->EIP and SS<-->ESP on your stack frame layout.
Hm. More than that. My figure is totally incorrect. Corrected one:
+16 old_SS
+12 old_ESP
+8 old_EFLAGS
+4 old_CS
+0 old_EIP <--- SS:ESP
Addresses grow from the bottom in this fig. Note that offsets are always stored in lower address relative to where selectors are stored.
Quoted 33 lines
Quoted 4 lines
If
old_SSreferences 'small' data segment (16-bit one), processor does not restore ESP fromold_ESP.
It restores lower 16 bits only. Upper 16 bits are filled with garbage (or just not modified at all?).Not modified at all, yes. That's why it is always greater than
TASK_SIZE(0xc0000000) - it is still in a kernel space.Quoted 4 lines
This works fine because processor uses SP, not ESP, for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero... My test-case does "hlt" to force the context switch, but in fact it is not necessary. I tried an empty loop instead of HLT. If the loop is long enough and some IRQ being handled by the kernel in a mean time, the ESP is corrupted. So the user-space never knows when exactly it gets corrupted. And, as I mentioned already, since the code is 32bit, it just does the things like "mov ebp,esp" and uses ebp to access the function params and locals, which crashes, or uses ESP directly to access something, etc.
Quoted 2 lines
Hmm. I think you need to *emulate* this IRET.
Is this IRET happens in kernel or in dosemu code?
Quoted 10 lines
The problem happens only when iret is used to switch to another priv level. So it is the kernel's iret of course, something to the effect of entry.S:128 I think. The control is transferred from kernel code to the DOS code directly. dosemu code is completely bypassed. dosemu have no chances to intercept that. The DOS program is running on its own (as long as it doesn't trigger an exception, or SIGALRM comes), and the CPU is corrupting its ESP in *any* place, since it happens when an external IRQ arrives.
Aha. The only way to sanely handle it is to
hack on entry.S I'm afraid. Something like rewriting CS:EIP so that it returns to a small ring-3 trampoline which clears upper 16 bits of ESP and jumps to original CS:EIP.
You may use ring-3 user stack as a place to save original CS:EIP, and then just do RETF. This way, trampoline boils down to
movzxl %sp,%esp retf
which does not touch EFLAGS and you don't need to bother with thinking about preserving it. IRET will restore it, and trampoline wouldn't touch it anymore.
Now, how to detect when to use this? Hmm.... the simplest thing is to check that
(old_ESP <= 0xffff) && !(old_EFLAGS & VM_MASK) && (descr_old_SS is 16bit one)
This will cost us only one comparison in the normal path, because typically ESP of Linus executables is greater than 0xffff.
P.S.
# gcc -o stk1 -O2 stk1.c # ./stk1 old_ss=0x7b new_ss=0x7f In sighandler: esp=bffffb30 old_esp=0xbffffb30 new_esp=0xc618fb30 BUG!
--
vda
Hi Denis.
Quoted 4 lines · Denis Vlasenko
Aha. The only way to sanely handle it is to
hack onentry.SI'm afraid. Something like rewriting CS:EIP so that it returns to a small ring-3 trampoline which clears upper 16 bits of ESP and jumps to original CS:EIP.
Well, I don't really want to clear the higher word of ESP (even if I want to do that in most cases), but I'd really like to just restore it properly. Of course (I think) ring-3 trampoline will not work for many reasons. The most obvious one is that it itself can be interrupted in any place. Another problem with it is that the return frame will have to be pushed to the stack of a DOS prog. This is not the good thing to do. dosemu avoids ever touching the stack of a DOS prog by setting up the sigaltstack.
What *will* work, however, is a ring-1 trampoline, as Petr Vandrovec suggested. It can be executed with interrupts disabled (I think) so will not be interrupted, and it can (isn't it?) use the kernel stack, if I understand that correctly.
Quoted 3 lines
Now, how to detect when to use this? Hmm.... the simplest thing is to check that
(old_ESP<=0xffff) && !(old_EFLAGS &VM_MASK) && (descr_old_SSis 16bit one)
Yes, that's an excellent idea (except for the ESP<=0xffff check - I don't think this one is necessary).
Quoted 2 lines
This will cost us only one comparison in the normal path,
Yes, so I think I should just try to implement that. Not as a ring-3 trampoline, but as a ring-1 one.
Quoted 2 lines
because typically ESP of Linus executables
is greater than0xffff.
I'd rather say, because the stack segment is 32bit for them always.
And this all should work, as it seems to me. Thanks for the hint about the checking.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 1 line · Denis Vlasenko
Maybe. This would be a complicated thing.
I bet it was! :)
Quoted 4 lines
Well. Not okay. Maybe this?
1. We build IRET frame on ring1 stack for step 4 (see below), modify IRET frame on ring0 stack so that intrs are disabled and CS:EIP and SS:ESP point to values of ring1 code/stack.
Much simpler: IRET frame to return to user is already there. Just push another one to return to ring1 first.
Quoted 3 lines
2. IRET returns to ring1 code with dedicated *16-bit* ring1 stack Upper word of ESP is wrong now, but we can safely fiddle with it.
3. trampoline code fixes upper word of ESP (how?)
popl %esp (as per Petr Vandrovec's suggestion) The value on stack is carefully prepared on ring0.
Quoted 2 lines
4. trampoline IRETs to user code.
May work.
Works!
Quoted 1 line
ring1 stacks must be per-CPU.
I allocate it on a ring0 stack. Noone seem to suggest that. Is this flawed for some reasons?
Quoted 5 lines
Quoted 2 lines
ESP<=0xffff check - I don't think this one is
necessary).Any program which runs with 16bit stack and yet with ESP >
0xffffis doing something *terminally* weird. I think it is acceptable to leave this case unfixed.
For what? We can have that fixed so why not?
Quoted 2 lines
You cannot check 16bitness of SS descriptor in two insns.
I do actually:
larl OLDSS(%esp), %eax
testl $0x00400000, %eax
which is exactly two insns.
Since I've forgot to CC the patch to you, I uploaded it here:
http://www.dosemu.org/stas/linux-2.6.8-stacks2.diff
so that you (or anyone interested) can make a review.
Hi!
Quoted 7 lines
Quoted 3 lines
for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero...
Hmm, perhaps this can also be viewed as a "information leak"? Program running under dosemu is not expected to know high bits of kernel %esp...
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 6 lines · Pavel Machek
Hi!
Quoted 4 lines
Quoted 3 lines
for subsequent push/pop/call/ret operations.
But if code uses full ESP, thinking that upper 16 bits are zero, it will crash badly. Correct me if I'm wrong.
Quoted 8 lines
Quoted 4 lines
That's correct. But you should note that the program not just "thinks" that the upper 16 bits are zero. It writes zero there itself, and a few instructions later - oops, it is no longer zero...
Hmm, perhaps this can also be viewed as a "information leak"? Program running under dosemu is not expected to know high bits of kernel %esp...
Yes.
I must admit that Stas was right and I was wrong.
This is a 386 CPU bug.
Since then it seems to be codified in i86 architecture and duplicated in all successors. Incredible... :(
--
vda