Vmware crashes if compress/misc.c scrolls?
I just got the following message on the syslinux mailing list:
Quoted 5 lines
2. On some platforms (vmware for example :), READING from the video memory in the 32bit mode is impossible (causes an exeption). Taking in to account that the scroll function in
ilinux/arch/i386/boot/compressed/misc.cworks using a memcpy of the video memory, when the linux bootsector is given control and the cursor is at the end of the screen (-1-2 lines) an exception is
Quoted 8 lines
arisen and in the presence of it's handler in the bootloader the system halts. Both lilo and grub handle this situation - the cursor is always somewhere at the middle of the screen when the control is given to the linux bootsector. When using pxelinux taking in to account that the PXE bios prints out the presence of the second initramfs which causes the screen to scroll and the bug to appear :)
A very simple solution - the screen is cleared before giving control to the linux bootsector (in the patch).
First of all, is this actually true? This seems like a monumental screwup.
Second, assuming it is true, what should be done about it? The notion that the bootloader should erase the screen is ridiculous since we spend a lot of effort maintaining the screen context from 16-bit code.
I see a couple of options:
a. Do nothing. It's a Vmware bug, let the users bug Vmware and get an
updated version.
b. Detect Vmware versions that are affected (how?) and suppress output. c. Don't attempt to scroll in this code, and instead wrap around the
screen (presumably clearing the rest of the line.)
d. Blindly assume hardware scrolling works, instead of doing
copy-to-scroll. Although I have seen craptops on which hardware scrolling is broken, it's hardly critical messages we're dealing with here. A bug for a bug...
-hpa
Quoted 18 lines · H. Peter Anvin
I just got the following message on the syslinux mailing list:
Quoted 5 lines
2. On some platforms (vmware for example :), READING from the video memory in the 32bit mode is impossible (causes an exeption). Taking in to account that the scroll function in
ilinux/arch/i386/boot/compressed/misc.cworks using a memcpy of the video memory, when the linux bootsector is given control and the cursor is at the end of the screen (-1-2 lines) an exception isQuoted 9 lines
arisen and in the presence of it's handler in the bootloader the system halts. Both lilo and grub handle this situation - the cursor is always somewhere at the middle of the screen when the control is given to the linux bootsector. When using pxelinux taking in to account that the PXE bios prints out the presence of the second initramfs which causes the screen to scroll and the bug to appear :)
A very simple solution - the screen is cleared before giving control to the linux bootsector (in the patch).
Quoted 2 lines
First of all, is this actually true? This seems like a monumental screwup.
Yes. Is this a report of observed failure? And how would VMWare even go about implementing this kind of misfeature? Maybe it fails with particular instructions reading the framebuffer?
Quoted 6 lines
Second, assuming it is true, what should be done about it? The notion that the bootloader should erase the screen is ridiculous since we spend a lot of effort maintaining the screen context from 16-bit code.
I see a couple of options:
Maintain a shadow framebuffer, and copy it to the real framebuffer on update...
J
Quoted 7 lines · Jeremy Fitzhardinge
Quoted 3 lines
First of all, is this actually true? This seems like a monumental screwup.
Yes. Is this a report of observed failure? And how would VMWare even go about implementing this kind of misfeature? Maybe it fails with particular instructions reading the framebuffer?
It supposedly is.
Quoted 9 lines
Quoted 6 lines
Second, assuming it is true, what should be done about it? The notion that the bootloader should erase the screen is ridiculous since we spend a lot of effort maintaining the screen context from 16-bit code.
I see a couple of options:
Maintain a shadow framebuffer, and copy it to the real framebuffer on update...
How would you populate the shadow framebuffer? Pass it in from real mode?
-hpa
Quoted 9 lines · H. Peter Anvin
Quoted 4 lines · Jeremy Fitzhardinge
Yes. Is this a report of observed failure? And how would VMWare even go about implementing this kind of misfeature? Maybe it fails with particular instructions reading the framebuffer?
It supposedly is.
You mean "is a real failure"? Or "is triggered by particular instructions"? It seems profoundly bogus (as in, surely DOS or something reads the framebuffer).
Quoted 6 lines
Quoted 3 lines
Maintain a shadow framebuffer, and copy it to the real framebuffer on update...
How would you populate the shadow framebuffer? Pass it in from real mode?
Erm. BIOS always says basically the same stuff, so just compile a screenshot into the setup code?
J
Quoted 5 lines · Jeremy Fitzhardinge
You mean "is a real failure"? Or "is triggered by particular instructions"? It seems profoundly bogus (as in, surely DOS or something reads the framebuffer).
A real failure (although only triggered from some variant of protected mode -- paging off?)
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 5 lines · Jeremy Fitzhardinge
You mean "is a real failure"? Or "is triggered by particular instructions"? It seems profoundly bogus (as in, surely DOS or something reads the framebuffer).
A real failure (although only triggered from some variant of protected mode -- paging off?)
Oh, it only happens in protected mode? So the answer to "how does the shadow framebuffer get populated?" is "real mode does it before switching to protected mode".
J
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 8 lines · H. Peter Anvin
Quoted 5 lines · Jeremy Fitzhardinge
You mean "is a real failure"? Or "is triggered by particular instructions"? It seems profoundly bogus (as in, surely DOS or something reads the framebuffer).
Quoted 8 lines
Quoted 3 lines
A real failure (although only triggered from some variant of protected mode -- paging off?)
Oh, it only happens in protected mode? So the answer to "how does the shadow framebuffer get populated?" is "real mode does it before switching to protected mode".
Sure, but that means frobbing the boot parameter structure, with associated problems. I guess I'm not inherently opposed to it, but I'd like to understand if there is a cleaner workaround first.
-hpa
Quoted 23 lines · H. Peter Anvin
Quoted 11 lines · Jeremy Fitzhardinge
Quoted 9 lines · H. Peter Anvin
Quoted 6 lines · Jeremy Fitzhardinge
You mean "is a real failure"? Or "is triggered by particular instructions"? It seems profoundly bogus (as in, surely DOS or something reads the framebuffer).
Quoted 9 lines
Quoted 4 lines
A real failure (although only triggered from some variant of protected mode -- paging off?)
Oh, it only happens in protected mode? So the answer to "how does the shadow framebuffer get populated?" is "real mode does it before switching to protected mode".
Quoted 4 lines
Sure, but that means frobbing the boot parameter structure, with associated problems. I guess I'm not inherently opposed to it, but I'd like to understand if there is a cleaner workaround first.
I'm failing to understand exactly what the failure here is. Can you provide sample code that generates the problem? Surely, it should be possible to read the framebuffer.
Zach
Quoted 5 lines · Zachary Amsden
I'm failing to understand exactly what the failure here is. Can you provide sample code that generates the problem? Surely, it should be possible to read the framebuffer.
The supposed test case was leaving the cursor at the bottommost line on the screen before running the Linux kernel.
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 5 lines · Zachary Amsden
I'm failing to understand exactly what the failure here is. Can you provide sample code that generates the problem? Surely, it should be possible to read the framebuffer.
The supposed test case was leaving the cursor at the bottommost line on the screen before running the Linux kernel.
Do you have a sample bootsector that does this? I would very much like to try booting it rather than writing a mock-up which may or may not exhibit the same behavior. .iso, floppy images greatly would help.
The very good news is that I should be able to tell you exactly why the exception occurs. My first thought was perhaps a GDT foul-up resulted in a read-only code segment, and mov %cs:(a000) was faulting or something like that, but this is a total stab in the dark.
Thanks,
Zach
Quoted 3 lines · Zachary Amsden
Do you have a sample bootsector that does this? I would very much like to try booting it rather than writing a mock-up which may or may not exhibit the same behavior. .iso, floppy images greatly would help.
The very good news is that I should be able to tell you exactly why the exception occurs. My first thought was perhaps a GDT foul-up resulted in a read-only code segment, and mov %cs:(a000) was faulting or something like that, but this is a total stab in the dark.
The reported bug is that *reads* fail, while writes succeed. I can't think of any way you could manage that easily.
J
Quoted 26 lines · H. Peter Anvin
I just got the following message on the syslinux mailing list:
Quoted 5 lines
2. On some platforms (vmware for example :), READING from the video memory in the 32bit mode is impossible (causes an exeption). Taking in to account that the scroll function in
ilinux/arch/i386/boot/compressed/misc.cworks using a memcpy of the video memory, when the linux bootsector is given control and the cursor is at the end of the screen (-1-2 lines) an exception isQuoted 9 lines
arisen and in the presence of it's handler in the bootloader the system halts. Both lilo and grub handle this situation - the cursor is always somewhere at the middle of the screen when the control is given to the linux bootsector. When using pxelinux taking in to account that the PXE bios prints out the presence of the second initramfs which causes the screen to scroll and the bug to appear :)
A very simple solution - the screen is cleared before giving control to the linux bootsector (in the patch).First of all, is this actually true? This seems like a monumental screwup.
Second, assuming it is true, what should be done about it? The notion that the bootloader should erase the screen is ridiculous since we spend a lot of effort maintaining the screen context from 16-bit code.
I see a couple of options:
Quoted 10 lines
a. Do nothing. It's a Vmware bug, let the users bug Vmware and get an
updated version.
b. Detect Vmware versions that are affected (how?) and suppress output. c. Don't attempt to scroll in this code, and instead wrap around the
screen (presumably clearing the rest of the line.)
d. Blindly assume hardware scrolling works, instead of doing
copy-to-scroll. Although I have seen craptops on which hardware scrolling is broken, it's hardly critical messages we're dealing with here. A bug for a bug...
Sorry it took me so long to follow up on this.
It's not a VMware bug. Video memory operates just fine under VMware. This led me to put the bug on the back-burner.
Today it came back to the front, and I did some investigation. It's also not a syslinux bug, although it is triggered by syslinux.
Depending on how much information is on the screen, you can trigger a Linux bug caused by a compiler problem.
In the boot decompressor for the kernel in the image Iouri provided, I find the following code present in memory before the first kernel instruction executes:
Here is Linux function scroll, which calls memcpy to scroll vga memory:
0x104131 : mov %edx,0x4(%esp,1) 0x104135 : mov %ecx,(%esp,1) 0x104138 : mov %eax,0x8(%esp,1) 0x10413c : call 0x1040d0
Here is Linux function memcpy:
(gdb) x/40i 0x1040d0
0x1040d0 : push %ebp 0x1040d1 : test %ecx,%ecx 0x1040d3 : mov %esp,%ebp 0x1040d5 : push %edi 0x1040d6 : mov %eax,%edi 0x1040d8 : push %esi 0x1040d9 : mov %edx,%esi 0x1040db : push %ebx 0x1040dc : mov %ecx,%ebx 0x1040de : je 0x1040f4 0x1040e0 : xor %ecx,%ecx 0x1040e2 : xor %edx,%edx 0x1040e4 : movzbl (%esi,%edx,1),%eax 0x1040e8 : add $0x1,%ecx 0x1040eb : cmp %ebx,%ecx 0x1040ed : mov %al,(%edi,%edx,1) 0x1040f0 : mov %ecx,%edx 0x1040f2 : jne 0x1040e4 0x1040f4 : mov %edi,%eax 0x1040f6 : pop %ebx 0x1040f7 : pop %esi 0x1040f8 : pop %edi 0x1040f9 : pop %ebp 0x1040fa : ret
As you can plainly see, the call to memcpy (which is redefined in boot/compressed/misc.c) is made using stack calling convention. Unfortunately, the compiler generated the memcpy function itself using regparm(3) convention. I am guessing this happened because a leftover prototype declaring memcpy as regparm got pulled in from somewhere, but gcc ignored it in the callsite because the function was declared static in the same file.
The net result of this is the parameters to memcpy get swapped, causing a memcpy from 0xb80a0 to destination (in register edx, 80x24 * 2) 0xa00, of size 0xb8000.
This corrupts all usable memory below 640k, including the GDT which is currently loaded.
But since the compressed kernel is running in high memory, above 1Mb, the kernel is fine. The compress loader head.S then proceeds to copy the move routine to address 0x11000, and executes the following instruction:
ljmp $(__KERNEL_CS), $0x1000 # and jump to the move routine
This reloads CS descriptor from the GDT, which is now garbage, and contains an invalid segment. The ljmp then generates a #GP. Since there is no IDT setup to handle the GDT (and even if there were, the destination CS would again be reloaded, causing a fault), a double fault is generated. Since the double-fault also hits the same problem, the machine then triple faults and proceeds to reboot.
Since the SCREEN_INFO passed to the kernel in the bad case has orig_y is set to 22, when we print 3 newlines:
putstr(".\nDecompressing Linux...");
gunzip();
putstr("done.\nBooting the kernel.\n");We cause the VGA to scroll, and trigger the bug.
Does anyone recall compiler problems (I noticed this VM was booting a 64-bit kernel, but the decompressor was 32-bit code, so perhaps at some time the 64-bit makefile or toolchain for Linux had some kind of compiler bug).
Zach
Quoted 2 lines
In the boot decompressor for the kernel in the image Iouri provided, I
32bit or 64bit image?
Quoted 4 lines
As you can plainly see, the call to memcpy (which is redefined in
boot/compressed/misc.c) is made using stack calling convention. Unfortunately, the compiler generated the memcpy function itself using regparm(3) convention. I am guessing this happened because a leftover
I can't find any here grepping .i files and includes. None of the memcpy prototypes in include have a __fastcall
Are you sure this still happens in mainline?
When I look at my scroll it also looks correct:
c0: 0f af f8 imul %eax,%edi
c3: 01 c0 add %eax,%eax
c5: 89 c2 mov %eax,%edx
c7: 01 f2 add %esi,%edx
c9: 89 45 f0 mov %eax,0xfffffff0(%ebp)
cc: 89 f0 mov %esi,%eax
ce: 89 f9 mov %edi,%ecx
d0: e8 7b ff ff ff call 50 <memcpy>Quoted 2 lines
Does anyone recall compiler problems (I noticed this VM was booting a 64-bit kernel,
Ok 64bit, that was 32bit above.
Since some time the decompressor is 64bit and 64bit always uses regparms. 64bit Scroll is ok too:
92: 0f af eb imul %ebx,%ebp
95: 01 db add %ebx,%ebx
97: 48 63 f3 movslq %ebx,%rsi
9a: 4c 01 ee add %r13,%rsi
9d: 89 ea mov %ebp,%edx
9f: e8 9c ff ff ff callq 40 <memcpy>
Quoted 1 line
but the decompressor was 32-bit code,
That must be a old kernel.
Can you people please verify this all still happens with a mainline or 2.6.22 kernel?
Quoted 3 lines
so perhaps at some
time the 64-bit makefile or toolchain for Linux had some kind of compiler bug).
I'm not aware of any such bugs.
-Andi
Quoted 47 lines · Andi Kleen
Quoted 2 lines
In the boot decompressor for the kernel in the image Iouri provided, I
32bit or 64bit image?
Quoted 5 lines
As you can plainly see, the call to memcpy (which is redefined in
boot/compressed/misc.c) is made using stack calling convention. Unfortunately, the compiler generated the memcpy function itself using regparm(3) convention. I am guessing this happened because a leftoverI can't find any here grepping .i files and includes. None of the memcpy prototypes in include have a
__fastcallAre you sure this still happens in mainline?
When I look at my scroll it also looks correct:
c0: 0f af f8 imul %eax,%edi c3: 01 c0 add %eax,%eax c5: 89 c2 mov %eax,%edx c7: 01 f2 add %esi,%edx c9: 89 45 f0 mov %eax,0xfffffff0(%ebp) cc: 89 f0 mov %esi,%eax ce: 89 f9 mov %edi,%ecx d0: e8 7b ff ff ff call 50 <memcpy>Quoted 3 lines
Does anyone recall compiler problems (I noticed this VM was booting a 64-bit kernel,
Ok 64bit, that was 32bit above.
Since some time the decompressor is 64bit and 64bit always uses regparms. 64bit Scroll is ok too:
92: 0f af eb imul %ebx,%ebp
95: 01 db add %ebx,%ebx
97: 48 63 f3 movslq %ebx,%rsi
9a: 4c 01 ee add %r13,%rsi
9d: 89 ea mov %ebp,%edx
9f: e8 9c ff ff ff callq 40 <memcpy>
Quoted 8 lines
Quoted 2 lines
but the decompressor was 32-bit code,
That must be a old kernel.
Can you people please verify this all still happens with a mainline or 2.6.22 kernel?
It doesn't.
Quoted 8 lines
Quoted 4 lines
so perhaps at some time the 64-bit makefile or toolchain for Linux had some kind of compiler bug).
I'm not aware of any such bugs.
Me neither, nor do any of my current 32-bit or 64-bit kernels have this problem. I was just wondering if it got fixed deliberately, or if some old gcc bug out there could still cause such problems with specific toolchains.
I'll look up what version the kernel was; Iouri, can you provide us with the binutils / gcc versions used to build your kernel?
Zach
Quoted 56 lines · Andi Kleen
Quoted 2 lines
In the boot decompressor for the kernel in the image Iouri provided, I
32bit or 64bit image?
Quoted 5 lines
As you can plainly see, the call to memcpy (which is redefined in
boot/compressed/misc.c) is made using stack calling convention. Unfortunately, the compiler generated the memcpy function itself using regparm(3) convention. I am guessing this happened because a leftoverI can't find any here grepping .i files and includes. None of the memcpy prototypes in include have a
__fastcallAre you sure this still happens in mainline?
When I look at my scroll it also looks correct:
c0: 0f af f8 imul %eax,%edi c3: 01 c0 add %eax,%eax c5: 89 c2 mov %eax,%edx c7: 01 f2 add %esi,%edx c9: 89 45 f0 mov %eax,0xfffffff0(%ebp) cc: 89 f0 mov %esi,%eax ce: 89 f9 mov %edi,%ecx d0: e8 7b ff ff ff call 50 <memcpy>Quoted 3 lines
Does anyone recall compiler problems (I noticed this VM was booting a 64-bit kernel,
Ok 64bit, that was 32bit above.
Since some time the decompressor is 64bit and 64bit always uses regparms. 64bit Scroll is ok too:
92: 0f af eb imul %ebx,%ebp
95: 01 db add %ebx,%ebx
97: 48 63 f3 movslq %ebx,%rsi
9a: 4c 01 ee add %r13,%rsi
9d: 89 ea mov %ebp,%edx
9f: e8 9c ff ff ff callq 40 <memcpy>Quoted 2 lines
but the decompressor was 32-bit code,
That must be a old kernel.
Can you people please verify this all still happens with a mainline or 2.6.22 kernel?
Quoted 6 lines
Quoted 3 lines
so perhaps at some time the 64-bit makefile or toolchain for Linux had some kind of compiler bug).
I'm not aware of any such bugs.
It was 2.6.21.4, so it is fairly recent, but with 32-bit decompress stage. GCC was 4.2.0.
Sounds like a gcc bug, to me.
Zach
Quoted 6 lines · Zachary Amsden
It was 2.6.21.4, so it is fairly recent, but with 32-bit decompress stage. GCC was 4.2.0.
Sounds like a gcc bug, to me.
I would argue that anything that causes a function and its callsite to diverge, when they're part of the same file, is a gcc bug almost by definition.
That doesn't mean it's not a problem and that there is not a solution -- if nothing else there should be a patch for -stable.
-hpa