Skip to contentSkip to conversation

CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

DISCUSSION Andy Isaacson 4 participants 9 messages Feb 9, 2011 – Feb 10, 2011 Xz embedded

lore
Share & tools
lore
#0 lore raw

% uname -m
x86_64
% git log -1 | head
commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
% grep XZ .config
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_KERNEL_XZ=y
CONFIG_RD_XZ=y
CONFIG_SQUASHFS_XZ=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_XZ_DEC_TEST=y
CONFIG_DECOMPRESS_XZ=y
% make -j4
...

  CC      arch/x86/boot/tty.o
  CC      arch/x86/oprofile/oprofile.mod.o
  XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
exec: 23: xz: not found
  CC      arch/x86/boot/video.o
  CC      crypto/crc32c.mod.o
  CC      crypto/pcrypt.mod.o
  CC      arch/x86/boot/video-mode.o
  CC      drivers/acpi/acpi_memhotplug.mod.o

MKPIGGY arch/x86/boot/compressed/piggy.S
arch/x86/boot/compressed/vmlinux.bin.xz: No such file or directory
CC arch/x86/boot/version.o
LD arch/x86/boot/compressed/vmlinux
... [hundreds of lines of output snipped]

  LD [M]  net/sched/act_csum.ko
  IHEX    firmware/bnx2/bnx2-mips-09-6.2.1.fw
  IHEX    firmware/bnx2/bnx2-mips-06-6.2.1.fw

% echo $?
2
% ls -l arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41 arch/x86/boot/bzImage

The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible failure is after reboot when I get the lovely

Decompressing Linux...

Input is not in the XZ format (wrong magic bytes)

-- System halted

Sometimes I run "make -j4 && make install" which would have caught this, but the error should be more prominent and we shouldn't create bzImage if the dependencies are not present.

-andy

#1 lore raw

09.02.2011 22:52, Andy Isaacson пишет:

Quoted 49 lines

% uname -m
x86_64
% git log -1 | head
commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
% grep XZ .config
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_KERNEL_XZ=y
CONFIG_RD_XZ=y
CONFIG_SQUASHFS_XZ=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_XZ_DEC_TEST=y
CONFIG_DECOMPRESS_XZ=y
% make -j4
...

  CC      arch/x86/boot/tty.o
  CC      arch/x86/oprofile/oprofile.mod.o
  XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
exec: 23: xz: not found
  CC      arch/x86/boot/video.o
  CC      crypto/crc32c.mod.o
  CC      crypto/pcrypt.mod.o
  CC      arch/x86/boot/video-mode.o
  CC      drivers/acpi/acpi_memhotplug.mod.o

MKPIGGY arch/x86/boot/compressed/piggy.S
arch/x86/boot/compressed/vmlinux.bin.xz: No such file or directory
CC arch/x86/boot/version.o
LD arch/x86/boot/compressed/vmlinux
... [hundreds of lines of output snipped]

  LD [M]  net/sched/act_csum.ko
  IHEX    firmware/bnx2/bnx2-mips-09-6.2.1.fw
  IHEX    firmware/bnx2/bnx2-mips-06-6.2.1.fw

% echo $?
2
% ls -l arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41 arch/x86/boot/bzImage

The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible failure is after reboot when I get the lovely

Decompressing Linux...

Quoted 7 lines

Input is not in the XZ format (wrong magic bytes)

-- System halted

Sometimes I run "make -j4 && make install" which would have caught this, but the error should be more prominent and we shouldn't create bzImage if the dependencies are not present.

:)

---

scripts/xz_wrap.sh +1 -1
index 17a5798..5ec9d36 100644
@@ -20,4 +20,4 @@ case $ARCH in
        sparc)          BCJ=--sparc ;;
 esac
 
-exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
+exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1
Signature · 1 line
                                                         Pavel.
#2 lore raw
Quoted 2 lines · Pavel Vasilyev
Quoted 2 lines

The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible

[snip]

Quoted 10 lines

:)

---

scripts/xz_wrap.sh +0 -0
index 17a5798..5ec9d36 100644

scripts/xz_wrap.sh case $ARCH in · line 23

Quoted 5 lines
scripts/xz_wrap.sh +1 -1
@@ -20,4 +20,4 @@ case $ARCH in
        sparc)          BCJ=--sparc ;;
 esac
 
-exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
+exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1

I get the same behavior with and without your patch, tested with both bash and dash. (FWIW, /bin/sh on my systems is a symlink to /bin/dash.)

% printf '#!/bin/bash\nexec /no/such/file\n' > foo.sh
% chmod +x foo.sh
% ./foo.sh; echo $?
./foo.sh: line 2: /no/such/file: No such file or directory ./foo.sh: line 2: exec: /no/such/file: cannot execute: No such file or directory 126
% printf '#!/bin/bash\nexec /no/such/file\n' > foo.sh
% ./foo.sh; echo $?
./foo.sh: line 2: /no/such/file: No such file or directory ./foo.sh: line 2: exec: /no/such/file: cannot execute: No such file or directory
126

% printf '#!/bin/dash\nexec /no/such/file\n' > foo.sh % ./foo.sh; echo $?
exec: 2: /no/such/file: not found
2
% printf '#!/bin/dash\nexec /no/such/file || exit 1\n' > foo.sh % ./foo.sh; echo $?
exec: 2: /no/such/file: not found
2

-andy

#3 lore raw

09.02.2011 23:38, Andy Isaacson пишет:

Quoted 16 lines · Pavel Vasilyev
Quoted 2 lines
Quoted 2 lines

The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible

[snip]

Quoted 12 lines

:)

---

scripts/xz_wrap.sh +0 -0
index 17a5798..5ec9d36 100644
@@ -20,4 +20,4 @@ case $ARCH in
        sparc)          BCJ=--sparc ;;
 esac
 
Quoted 7 lines

scripts/xz_wrap.sh case $ARCH in · line 23

Quoted 3 lines
scripts/xz_wrap.sh +1 -1
-exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
+exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1

I get the same behavior with and without your patch, tested with both bash and dash. (FWIW, /bin/sh on my systems is a symlink to /bin/dash.)

# mv /usr/bin/xz /
# rm arch/x86/boot/compressed/vmlinux.bin*
# rm arch/x86/boot/bzImage
# make
# make
  CHK     include/linux/version.h
  CHK     include/generated/utsrelease.h
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     include/linux/version.h
make[2]: `scripts/unifdef' is up to date.
  XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
/media/kernel/linux-2.6/scripts/xz_wrap.sh: line 23: exec: xz: not found
make[2]: *** [arch/x86/boot/compressed/vmlinux.bin.xz] Error 1
make[1]: *** [arch/x86/boot/compressed/vmlinux] Error 2
make: *** [bzImage] Error 2
# make install
sh /media/kernel/linux-2.6/arch/x86/boot/install.sh 2.6.38-rc4+
arch/x86/boot/bzImage \
        System.map "/boot"

*** Missing file: arch/x86/boot/bzImage *** You need to run "make" before "make install".

make[1]: *** [install] Error 1
make: *** [install] Error 2

All work, without and with patch. :-/

# sh --version
GNU bash, version 4.1.9(1)-release (x86_64-suse-linux-gnu)
Signature · 1 line
                                                         Pavel.
#4 lore raw
In reply to Andy Isaacson
Quoted 7 lines · Andy Isaacson

The failure is obscured by trailing make output, and since a broken bzImage is created, "make install" succeeds, and the first visible failure is after reboot when I get the lovely

Decompressing Linux...

Quoted 8 lines

Input is not in the XZ format (wrong magic bytes)

-- System halted

Sometimes I run "make -j4 && make install" which would have caught this, but the error should be more prominent and we shouldn't create bzImage if the dependencies are not present.

Agreed that the bzImage should not have been created. That should make the error prominent enough (your complaint that it gets obscured with make -j is "just how things work, sorry.")

-hpa

maintainer
#5 lore raw
In reply to Andy Isaacson
Quoted 6 lines · On 2011-02-09 Andy Isaacson wrote:

% make -j4
...

  CC      arch/x86/boot/tty.o
  CC      arch/x86/oprofile/oprofile.mod.o
  XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz

exec: 23: xz: not found

This may have something to do with the shell that "make" uses to run the commands. The compression command isn't seen as failed with dash for some reason.

It doesn't matter what shell is used to run xz_wrap.sh. The same problem exists with other compressors if they are missing.

Quoted 1 line

commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d

[...]

Quoted 2 lines

% ls -l arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41 arch/x86/boot/bzImage

"Feb 1" was about a week ago. The file is a few days older than the commit ID, assuming that your clock is set correctly.

Signature · 1 line
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode
#6 lore raw
Quoted 15 lines · Lasse Collin
Quoted 6 lines · On 2011-02-09 Andy Isaacson wrote:

% make -j4
...

  CC      arch/x86/boot/tty.o
  CC      arch/x86/oprofile/oprofile.mod.o
  XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz

exec: 23: xz: not found

This may have something to do with the shell that "make" uses to run the commands. The compression command isn't seen as failed with dash for some reason.

It doesn't matter what shell is used to run xz_wrap.sh. The same problem exists with other compressors if they are missing.

Quoted 7 lines
Quoted 1 line

commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d

[...]

Quoted 2 lines

% ls -l arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41 arch/x86/boot/bzImage

"Feb 1" was about a week ago. The file is a few days older than the commit ID, assuming that your clock is set correctly.

Ah, yes, this is my regular build tree; I routinely pull and build there.

So the build failed, but the output file remains because it failed on an intermediate step. (This testing was from me reproducing it on a different system, with 100b33).

Rewinding to the original system which failed to boot...

How did "make install" create a broken /boot/vmlinuz-2.6.38-rc3-0312-gcb5520f that caused the boot-time error message?

If I re-run "make -j4" in the tree that caused the problem, make fails with exit code 2.

The following seems to reproduce the silent error:

rm .config
make defconfig
make -j4
edit .config and set CONFIG_KERNEL_XZ=y, CONFIG_KERNEL_GZIP=n

% make -j4
...
Kernel: arch/x86/boot/bzImage is ready (#64)
% echo $?
0

The resulting bzImage fails to boot with the aforementioned "Input is not in the XZ format" error.

FWIW, the machine that's showing the problem is an Ubuntu Karmic x86_64 desktop install with all updates through mid-January installed.

-andy

maintainer
#7 lore raw
Quoted 31 lines · On 2011-02-10 Andy Isaacson wrote:
Quoted 6 lines · Lasse Collin
Quoted 2 lines · On 2011-02-09 Andy Isaacson wrote:

% ls -l arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb 1 18:41 arch/x86/boot/bzImage

"Feb 1" was about a week ago. The file is a few days older than the commit ID, assuming that your clock is set correctly.

Ah, yes, this is my regular build tree; I routinely pull and build there.

So the build failed, but the output file remains because it failed on an intermediate step. (This testing was from me reproducing it on a different system, with 100b33).

Rewinding to the original system which failed to boot...

How did "make install" create a broken /boot/vmlinuz-2.6.38-rc3-0312-gcb5520f that caused the boot-time error message?

If I re-run "make -j4" in the tree that caused the problem, make fails with exit code 2.

The following seems to reproduce the silent error:

rm .config
make defconfig
make -j4
edit .config and set CONFIG_KERNEL_XZ=y, CONFIG_KERNEL_GZIP=n

Quoted 5 lines

% make -j4
...
Kernel: arch/x86/boot/bzImage is ready (#64)
% echo $?
0

It looks a like a bug in dash. The following with dash prints foo bar and returns with exit status 0, but other shells print only foo and return with non-zero exit status:

sh -c 'set -e ; echo foo ; ( false ) ; echo bar'

That is, "set -e" has no effect on a subshell returning a non-zero exit status. Without a subshell it works correctly:

sh -c 'set -e ; echo foo ; { false ; } ; echo bar'

Kernel compression and mkpiggy use command lines that hit this bug. Here is a quick hack to avoid the unneeded subshell for these command lines, which also works around the dash bug:

    sed -i 's|( *\( *rm -f $@ ; false\) *)|{ \1 ; }|' \
        scripts/Makefile.lib arch/x86/boot/compressed/Makefile

There are similar "set -e" + subshell constructs in other makefiles and scripts. I don't know if it makes sense to accommodate a buggy but common /bin/sh by modifying makefiles and scripts to avoid the problematic construct. Maybe it would be better to have a test to catch the buggy shell and tell the user to use e.g. "make SHELL=/bin/bash".

Signature · 1 line
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode
maintainer
#8 lore raw
Quoted 5 lines · On 2011-02-10 Lasse Collin wrote:

It looks a like a bug in dash. The following with dash prints foo bar and returns with exit status 0, but other shells print only foo and return with non-zero exit status:

sh -c 'set -e ; echo foo ; ( false ) ; echo bar'

This dash bug has been fixed on 2010-07-06.

http://git.kernel.org/?p=utils/dash/dash.git;a=commitdiff;h=a92255d6fa8b8efb0f9fb093b77ac601fe3e9ede

Signature · 1 line
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode