CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed
% uname -mx86_64
% git log -1 | head
commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
% grep XZ .configCONFIG_HAVE_KERNEL_XZ=yCONFIG_KERNEL_XZ=yCONFIG_RD_XZ=yCONFIG_SQUASHFS_XZ=yCONFIG_XZ_DEC=yCONFIG_XZ_DEC_X86=yCONFIG_XZ_DEC_POWERPC=yCONFIG_XZ_DEC_IA64=yCONFIG_XZ_DEC_ARM=yCONFIG_XZ_DEC_ARMTHUMB=yCONFIG_XZ_DEC_SPARC=yCONFIG_XZ_DEC_BCJ=yCONFIG_XZ_DEC_TEST=yCONFIG_DECOMPRESS_XZ=y
% make -j4
...
CC arch/x86/boot/tty.o CC arch/x86/oprofile/oprofile.mod.o XZKERN arch/x86/boot/compressed/vmlinux.bin.xz exec: 23: xz: not found CC arch/x86/boot/video.o CC crypto/crc32c.mod.o CC crypto/pcrypt.mod.o CC arch/x86/boot/video-mode.o CC drivers/acpi/acpi_memhotplug.mod.o
MKPIGGY arch/x86/boot/compressed/piggy.Sarch/x86/boot/compressed/vmlinux.bin.xz: No such file or directory
CC arch/x86/boot/version.o
LD arch/x86/boot/compressed/vmlinux
... [hundreds of lines of output snipped]
LD [M] net/sched/act_csum.ko IHEX firmware/bnx2/bnx2-mips-09-6.2.1.fw IHEX firmware/bnx2/bnx2-mips-06-6.2.1.fw
% echo $?
2
% ls -l arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41 arch/x86/boot/bzImage
The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible failure is after reboot when I get the lovely
Decompressing Linux...
Input is not in the XZ format (wrong magic bytes)
-- System halted
Sometimes I run "make -j4 && make install" which would have caught this, but the error should be more prominent and we shouldn't create bzImage if the dependencies are not present.
-andy
09.02.2011 22:52, Andy Isaacson пишет:
Quoted 49 lines
% uname -m
x86_64
% git log -1 | head
commit100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
% grep XZ .configCONFIG_HAVE_KERNEL_XZ=yCONFIG_KERNEL_XZ=yCONFIG_RD_XZ=yCONFIG_SQUASHFS_XZ=yCONFIG_XZ_DEC=yCONFIG_XZ_DEC_X86=yCONFIG_XZ_DEC_POWERPC=yCONFIG_XZ_DEC_IA64=yCONFIG_XZ_DEC_ARM=yCONFIG_XZ_DEC_ARMTHUMB=yCONFIG_XZ_DEC_SPARC=yCONFIG_XZ_DEC_BCJ=yCONFIG_XZ_DEC_TEST=yCONFIG_DECOMPRESS_XZ=y
% make -j4
...CC arch/x86/boot/tty.o CC arch/x86/oprofile/oprofile.mod.o XZKERN arch/x86/boot/compressed/vmlinux.bin.xz exec: 23: xz: not found CC arch/x86/boot/video.o CC crypto/crc32c.mod.o CC crypto/pcrypt.mod.o CC arch/x86/boot/video-mode.o CC drivers/acpi/acpi_memhotplug.mod.oMKPIGGY
arch/x86/boot/compressed/piggy.Sarch/x86/boot/compressed/vmlinux.bin.xz: No such file or directory
CCarch/x86/boot/version.o
LDarch/x86/boot/compressed/vmlinux
... [hundreds of lines of output snipped]LD [M] net/sched/act_csum.ko IHEX firmware/bnx2/bnx2-mips-09-6.2.1.fw IHEX firmware/bnx2/bnx2-mips-06-6.2.1.fw% echo $?
2
% ls -larch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41arch/x86/boot/bzImageThe failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible failure is after reboot when I get the lovely
Decompressing Linux...
Quoted 7 lines
Input is not in the XZ format (wrong magic bytes)
-- System halted
Sometimes I run "make -j4 && make install" which would have caught this, but the error should be more prominent and we shouldn't create bzImage if the dependencies are not present.
:)
---
scripts/xz_wrap.sh +1 -1
@@ -20,4 +20,4 @@ case $ARCH in sparc) BCJ=--sparc ;; esac -exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB +exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1
Signature · 1 line
Pavel.
Quoted 2 lines · Pavel Vasilyev
Quoted 2 lines
The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible
[snip]
Quoted 10 lines
:)
Signed-off-by Pavel Vasilyev <[email protected]>---
scripts/xz_wrap.sh +0
-0
scripts/xz_wrap.sh case $ARCH in · line 23
Quoted 5 lines
scripts/xz_wrap.sh +1
-1@@ -20,4 +20,4 @@ case $ARCH in sparc) BCJ=--sparc ;; esac -exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB +exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1
I get the same behavior with and without your patch, tested with both bash and dash. (FWIW, /bin/sh on my systems is a symlink to /bin/dash.)
% printf '#!/bin/bash\nexec /no/such/file\n' > foo.sh
% chmod +x foo.sh
% ./foo.sh; echo $?
./foo.sh: line 2: /no/such/file: No such file or directory ./foo.sh: line 2: exec: /no/such/file: cannot execute: No such file or directory 126
% printf '#!/bin/bash\nexec /no/such/file\n' > foo.sh
% ./foo.sh; echo $?
./foo.sh: line 2: /no/such/file: No such file or directory ./foo.sh: line 2: exec: /no/such/file: cannot execute: No such file or directory
126
% printf '#!/bin/dash\nexec /no/such/file\n' > foo.sh % ./foo.sh; echo $?
exec: 2: /no/such/file: not found
2
% printf '#!/bin/dash\nexec /no/such/file || exit 1\n' > foo.sh % ./foo.sh; echo $?
exec: 2: /no/such/file: not found
2
-andy
09.02.2011 23:38, Andy Isaacson пишет:
Quoted 16 lines · Pavel Vasilyev
Quoted 2 lines
Quoted 2 lines
The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible
[snip]
Quoted 12 lines
:)
Signed-off-by Pavel Vasilyev <[email protected]>---
scripts/xz_wrap.sh +0
-0@@ -20,4 +20,4 @@ case $ARCH in sparc) BCJ=--sparc ;; esac
Quoted 7 lines
scripts/xz_wrap.sh case $ARCH in · line 23
Quoted 3 lines
scripts/xz_wrap.sh +1
-1-exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB +exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1I get the same behavior with and without your patch, tested with both bash and dash. (FWIW,
/bin/shon my systems is a symlink to/bin/dash.)
# mv /usr/bin/xz / # rm arch/x86/boot/compressed/vmlinux.bin* # rm arch/x86/boot/bzImage # make # make CHK include/linux/version.h CHK include/generated/utsrelease.h CALL scripts/checksyscalls.sh CHK include/generated/compile.h CHK include/linux/version.h make[2]: `scripts/unifdef' is up to date. XZKERN arch/x86/boot/compressed/vmlinux.bin.xz /media/kernel/linux-2.6/scripts/xz_wrap.sh: line 23: exec: xz: not found make[2]: *** [arch/x86/boot/compressed/vmlinux.bin.xz] Error 1 make[1]: *** [arch/x86/boot/compressed/vmlinux] Error 2 make: *** [bzImage] Error 2
# make install
sh /media/kernel/linux-2.6/arch/x86/boot/install.sh 2.6.38-rc4+
arch/x86/boot/bzImage \
System.map "/boot" *** Missing file: arch/x86/boot/bzImage *** You need to run "make" before "make install".
make[1]: *** [install] Error 1
make: *** [install] Error 2
All work, without and with patch. :-/
# sh --version GNU bash, version 4.1.9(1)-release (x86_64-suse-linux-gnu)
Signature · 1 line
Pavel.
Quoted 7 lines · Andy Isaacson
The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible failure is after reboot when I get the lovely
Decompressing Linux...
Quoted 8 lines
Input is not in the XZ format (wrong magic bytes)
-- System halted
Sometimes I run "make -j4 && make install" which would have caught this, but the error should be more prominent and we shouldn't create bzImage if the dependencies are not present.
Agreed that the bzImage should not have been created. That should make the error prominent enough (your complaint that it gets obscured with make -j is "just how things work, sorry.")
-hpa
Lasse Collin
maintainerQuoted 6 lines · On 2011-02-09 Andy Isaacson wrote:
% make -j4
...CC arch/x86/boot/tty.o CC arch/x86/oprofile/oprofile.mod.o XZKERN arch/x86/boot/compressed/vmlinux.bin.xzexec: 23: xz: not found
This may have something to do with the shell that "make" uses to run the commands. The compression command isn't seen as failed with dash for some reason.
It doesn't matter what shell is used to run xz_wrap.sh. The same problem exists with other compressors if they are missing.
Quoted 1 line
commit
100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
[...]
Quoted 2 lines
% ls -l
arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41arch/x86/boot/bzImage
"Feb 1" was about a week ago. The file is a few days older than the commit ID, assuming that your clock is set correctly.
Signature · 1 line
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
Quoted 15 lines · Lasse Collin
Quoted 6 lines · On 2011-02-09 Andy Isaacson wrote:
% make -j4
...CC arch/x86/boot/tty.o CC arch/x86/oprofile/oprofile.mod.o XZKERN arch/x86/boot/compressed/vmlinux.bin.xzexec: 23: xz: not found
This may have something to do with the shell that "make" uses to run the commands. The compression command isn't seen as failed with dash for some reason.
It doesn't matter what shell is used to run
xz_wrap.sh. The same problem exists with other compressors if they are missing.
Quoted 7 lines
Quoted 1 line
commit
100b33c8bd8a3235fd0b7948338d6cbb3db3c63d[...]
Quoted 2 lines
% ls -l
arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41arch/x86/boot/bzImage"Feb 1" was about a week ago. The file is a few days older than the commit ID, assuming that your clock is set correctly.
Ah, yes, this is my regular build tree; I routinely pull and build there.
So the build failed, but the output file remains because it failed on an intermediate step. (This testing was from me reproducing it on a different system, with 100b33).
Rewinding to the original system which failed to boot...
How did "make install" create a broken /boot/vmlinuz-2.6.38-rc3-0312-gcb5520f that caused the boot-time error message?
If I re-run "make -j4" in the tree that caused the problem, make fails with exit code 2.
The following seems to reproduce the silent error:
rm .config
make defconfig
make -j4
edit .config and set CONFIG_KERNEL_XZ=y, CONFIG_KERNEL_GZIP=n
% make -j4
...
Kernel: arch/x86/boot/bzImage is ready (#64)
% echo $?
0
The resulting bzImage fails to boot with the aforementioned "Input is not in the XZ format" error.
FWIW, the machine that's showing the problem is an Ubuntu Karmic x86_64 desktop install with all updates through mid-January installed.
-andy
Lasse Collin
maintainerQuoted 31 lines · On 2011-02-10 Andy Isaacson wrote:
Quoted 6 lines · Lasse Collin
Quoted 2 lines · On 2011-02-09 Andy Isaacson wrote:
% ls -l
arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41arch/x86/boot/bzImage"Feb 1" was about a week ago. The file is a few days older than the commit ID, assuming that your clock is set correctly.
Ah, yes, this is my regular build tree; I routinely pull and build there.
So the build failed, but the output file remains because it failed on an intermediate step. (This testing was from me reproducing it on a different system, with 100b33).
Rewinding to the original system which failed to boot...
How did "make install" create a broken
/boot/vmlinuz-2.6.38-rc3-0312-gcb5520fthat caused the boot-time error message?If I re-run "make -j4" in the tree that caused the problem, make fails with exit code 2.
The following seems to reproduce the silent error:
rm .config
make defconfig
make -j4
edit .config and setCONFIG_KERNEL_XZ=y,CONFIG_KERNEL_GZIP=n
Quoted 5 lines
% make -j4
...
Kernel:arch/x86/boot/bzImageis ready (#64)
% echo $?
0
It looks a like a bug in dash. The following with dash prints foo bar and returns with exit status 0, but other shells print only foo and return with non-zero exit status:
sh -c 'set -e ; echo foo ; ( false ) ; echo bar'
That is, "set -e" has no effect on a subshell returning a non-zero exit status. Without a subshell it works correctly:
sh -c 'set -e ; echo foo ; { false ; } ; echo bar'
Kernel compression and mkpiggy use command lines that hit this bug. Here is a quick hack to avoid the unneeded subshell for these command lines, which also works around the dash bug:
sed -i 's|( *\( *rm -f $@ ; false\) *)|{ \1 ; }|' \
scripts/Makefile.lib arch/x86/boot/compressed/MakefileThere are similar "set -e" + subshell constructs in other makefiles and scripts. I don't know if it makes sense to accommodate a buggy but common /bin/sh by modifying makefiles and scripts to avoid the problematic construct. Maybe it would be better to have a test to catch the buggy shell and tell the user to use e.g. "make SHELL=/bin/bash".
Signature · 1 line
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
Lasse Collin
maintainerQuoted 5 lines · On 2011-02-10 Lasse Collin wrote:
It looks a like a bug in dash. The following with dash prints foo bar and returns with exit status 0, but other shells print only foo and return with non-zero exit status:
sh -c 'set -e ; echo foo ; ( false ) ; echo bar'
This dash bug has been fixed on 2010-07-06.
http://git.kernel.org/?p=utils/dash/dash.git;a=commitdiff;h=a92255d6fa8b8efb0f9fb093b77ac601fe3e9ede
Signature · 1 line
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode