2.6.26 floppy boot failure with kernel packed using 'upx'
For the Debian installer we've been tracing a problem with our installation boot floppy. If booted with 'expert' at the syslinux prompt, it only shows:
<snip>
SYSLINUX 3.70 <etc.>
boot:expert
Loading linux......................
Loading initrd.gz....ready.
Probing EDD (edd=off to disable)... ok
</snip>
And then the emulator crashes (both VirtualBox and qemu), or on real hardware the system reboots. The qemu crash is included at the bottom of this mail.
The problem was initially seen with the 2.6.25 Debian kernel and traced to a set of Xen-related patches backported from upstream 2.6.26. Floppies using Debian 2.6.24 or pristine 2.6.24/2.6.25 don't show the problem.
Bisection has shown the culprit to be this very early 2.6.26 commit:
$ git bisect bad 099e1377269a47ed30a00ee131001988e5bcaa9c is first bad commit commit 099e1377269a47ed30a00ee131001988e5bcaa9c Author: Ian Campbell <[email protected]> Date: Wed Feb 13 20:54:58 2008 +0000
x86: use ELF format in compressed images.
Important factor here is that we "pack" the kernel using upx [1] (in order to fit everything on a floppy). The original (unpacked) kernel after this commit boots fine, only a packed version fails.
We have tried upx versions 2.01, 3.01 and 3.03, all with same result.
Both "good" (before commit) and "bad" (after commit) images are available
at: http://people.debian.org/~fjp/tmp/d-i/floppy/upx/
Included are the boot floppy image, the raw kernel and the packed kernel.
The issue can also be reproduced using qemu without booting the floppy itself. For the "bad" image:
# Boots correctly (but fails when mounting root fs):
$ qemu -kernel vmlinuz -hda /dev/zero # Fails: $ qemu -kernel vmlinuz.upx -hda /dev/zero
So, the primairy question here is:
- is this a kernel regression because whatever changed is no longer valid conform "kernel format specs", or
- is this a latent issue in upx that somehow creates an invalid image, or
- does this change effectively create a new "type" of image that upx just doesn't yet know how to handle correctly?
And a follow-up question in the last two cases: how likely is it that this change could/will cause similar issues in other comparable scenarios?
Note that we've been using this same compression technique for ages in the Debian installer without any problems.
Cheers,
FJP
[1] http://upx.sourceforge.net/
upx command used for compression and output
-------------------------------------------
$ upx -f -9 vmlinuz
Ultimate Packer for eXecutables
Copyright (C) 1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007
UPX 3.01 Markus Oberhumer, Laszlo Molnar & John Reiser Jul 31st 2007File size Ratio Format Name -------------------- ------ ----------- ----------- 1312304 -> 1245723 94.93% bvmlinuz/386 vmlinuz
qemu crash output
-----------------
$ qemu -fda boot.img qemu: fatal: triple fault EAX=00000018 EBX=00000000 ECX=00000000 EDX=003646f6 ESI=000333a1 EDI=00000000 EBP=ffffffb0 ESP=0003c367 EIP=00101015 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 00000000 00000000 CS =0010 00000000 ffffffff 00cf9b00 SS =0018 00000000 ffffffff 00cf9300 DS =0000 00000000 00000000 00000000 FS =0018 00000000 ffffffff 00cf9300 GS =0018 00000000 ffffffff 00cf9300 LDT=0000 00000000 00000000 00008000 TR =0020 00001000 00000067 00008900 GDT= 656e6900 0000646e IDT= 00000000 00000000 CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000 CCS=ffeb7200 CCD=00000000 CCO=LOGICB FCW=037f FSW=4000 [ST=0] FTW=00 MXCSR=00001f80 FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000 XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000 Aborted
Quoted 6 lines · Frans Pop
The issue can also be reproduced using qemu without booting the floppy itself. For the "bad" image:
# Boots correctly (but fails when mounting root fs):$ qemu -kernel vmlinuz -hda /dev/zero # Fails: $ qemu -kernel vmlinuz.upx -hda /dev/zero
I can repro this.
Quoted 6 lines
So, the primairy question here is:
- is this a kernel regression because whatever changed is no longer valid conform "kernel format specs", or
- is this a latent issue in upx that somehow creates an invalid image, or
- does this change effectively create a new "type" of image that upx just doesn't yet know how to handle correctly?
This is the first time I've looked at UPX but from glancing through the code it certainly appears to make a lot of assumptions about the structure of the bzImage (to the point that it looks for specific code sequences within the binary).
It seems that the way UPX works is that it extracts the compressed image from the bzImage, recompresses it and rebuilds a new bzImage replacing the decompression stage (and possibly some of the other 16 bit startup, I'm not quite sure yet) with its own. The issue is that its new decompressor does not understand the ELF format and expects a raw binary.
I think that UPX probably has gone a bit beyond the documented interfaces, but it's not unreasonable way. In fact the changeset which you referenced (or one of the ones around it) actually adds further documentation (in Documentation/x86/i386/boot.txt) and header fields to aid in doing the sort of extraction UPX wants to do and documents more explicitly the formats which can be expected to be found there. Also around the same time a checksum field was defined which is invalidated by the repacking.
All in all I'd say it should be treated as a new subtype/variant which UPX should support. I'd say adding support for bzImage v2.08 to UPX would be pretty easy for someone who knows the code base (I couldn't even find the decompressor code, but then it's pre-coffee time here..).
Ian.
Signature · 5 lines
Ian Campbell Fame lost its appeal for me when I went into a public restroom and an autograph seeker handed me a pen and paper under the stall door. -- Marlo Thomas
Quoted 8 lines · Ian Campbell
Quoted 6 lines · Frans Pop
The issue can also be reproduced using qemu without booting the floppy itself. For the "bad" image:
# Boots correctly (but fails when mounting root fs):$ qemu -kernel vmlinuz -hda /dev/zero # Fails: $ qemu -kernel vmlinuz.upx -hda /dev/zero
Quoted 6 lines
I can repro this.
All in all I'd say it should be treated as a new subtype/variant which UPX should support. I'd say adding support for bzImage v2.08 to UPX would be pretty easy for someone who knows the code base (I couldn't even find the decompressor code, but then it's pre-coffee time here..).
Thanks for the quick and complete reply Ian. I already expected this but thought it would at least be useful to inform people here of the issue (and get it indexed for search engines).
I have no problems with it not being treated as a bug/regression.
I've now filed a bug report against upx:
http://sourceforge.net/tracker/?func=detail&aid=2014835&group_id=2331&atid=102331
Cheers,
FJP
Quoted 8 lines · Frans Pop
So, the primairy question here is:
- is this a kernel regression because whatever changed is no longer valid conform "kernel format specs", or
- is this a latent issue in upx that somehow creates an invalid image, or
- does this change effectively create a new "type" of image that upx just doesn't yet know how to handle correctly?
Sounds to me like UPX makes assumptions about the kernel image format that it really doesn't have standing to assume.
-hpa
Quoted 5 lines · Frans Pop
Important factor here is that we "pack" the kernel using upx [1] (in order to fit everything on a floppy). The original (unpacked) kernel after this commit boots fine, only a packed version fails.
We have tried upx versions 2.01, 3.01 and 3.03, all with same result.
Ok, I do not consider this to be a regression.
Clearly UPX is expecting a very specific kernel format, and clearly UPX just needs to be updated for the changes.
That said, if some UPX person can explain what it is that UPX needs, maybe we can add some format markers into the kernel (and perhaps some extra header info) so that future format changes are transparent or at least break in obvious ways.
Linus
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Frans Pop
Important factor here is that we "pack" the kernel using upx [1] (in order to fit everything on a floppy). The original (unpacked) kernel after this commit boots fine, only a packed version fails.
We have tried upx versions 2.01, 3.01 and 3.03, all with same result.
Quoted 6 lines
Ok, I do not consider this to be a regression.
Clearly UPX is expecting a very specific kernel format, and clearly UPX just needs to be updated for the changes.
That said, if some UPX person can explain what it is that UPX needs, maybe we can add some format markers into the kernel (and perhaps some extra header info) so that future format changes are transparent or at least break in obvious ways.
The issue with UPX is that UPX actually wants to replace the kernel decompressor, which also does other tasks (relocation and ELF parsing) in recent kernels.
-hpa
Quoted 3 lines · H. Peter Anvin
The issue with UPX is that UPX actually wants to replace the kernel decompressor, which also does other tasks (relocation and ELF parsing) in recent kernels.
It does save us about 5% though, which is quite substantial when you're talking about fitting a kernel on a boot floppy. For the past few kernel versions it has made the difference between being able to base installation floppies on the standard Debian kernel (generic 486 flavor) or having to come up with radical new solutions.
Quoted 1 line
From my tests for this issue it looks like .26 has grown so much that
we've now reached that point anyway though :-(
Cheers,
FJP