ARM: support XZ compressed kernels

PATCH Superseded Imre Kaloz 2 participants 13 messages Jul 12, 2011 – Jul 22, 2011

Subsystems (2)

Arm (32-bit) Xz embedded

lore

Share & tools

lore

Imre Kaloz

author

2011-07-12 12:56 UTC #0 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Wire up support for the XZ decompressor

Signed-off-by Imre Kaloz <[email protected]>

---
Change since v1: added missing .gitignore modification

 arch/arm/Kconfig                        |    1 +
 arch/arm/boot/compressed/.gitignore     |    1 +
 arch/arm/boot/compressed/Makefile       |   11 +++++++++--
 arch/arm/boot/compressed/decompress.c   |    4 ++++
 arch/arm/boot/compressed/piggy.xzkern.S |    6 ++++++
 lib/xz/xz_dec_stream.c                  |    1 +
 6 files changed, 22 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/boot/compressed/piggy.xzkern.S

arch/arm/Kconfig +1 -0

index 9adc278..286873e 100644
@@ -20,6 +20,7 @@ config ARM
 	select HAVE_KERNEL_GZIP
 	select HAVE_KERNEL_LZO
 	select HAVE_KERNEL_LZMA
+	select HAVE_KERNEL_XZ
 	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select PERF_USE_VMALLOC

arch/arm/boot/compressed/.gitignore +1 -0

index c602896..593a712 100644
@@ -3,5 +3,6 @@ lib1funcs.S
 piggy.gzip
 piggy.lzo
 piggy.lzma
+piggy.xzkern
 vmlinux
 vmlinux.lds

arch/arm/boot/compressed/Makefile +9 -2

index 23aad07..e5db34e 100644
@@ -82,13 +82,14 @@ SEDFLAGS	= s/TEXT_START/$(ZTEXTADDR)/;s/BSS_START/$(ZBSSADDR)/
 suffix_$(CONFIG_KERNEL_GZIP) = gzip
 suffix_$(CONFIG_KERNEL_LZO)  = lzo
 suffix_$(CONFIG_KERNEL_LZMA) = lzma
+suffix_$(CONFIG_KERNEL_XZ)   = xzkern
 
 targets       := vmlinux vmlinux.lds \
 		 piggy.$(suffix_y) piggy.$(suffix_y).o \
 		 font.o font.c head.o misc.o $(OBJS)
 
 # Make sure files are removed during clean
-extra-y       += piggy.gzip piggy.lzo piggy.lzma lib1funcs.S
+extra-y       += piggy.gzip piggy.lzo piggy.lzma piggy.xzkern lib1funcs.S ashldi3.S
 
 ifeq ($(CONFIG_FUNCTION_TRACER),y)
 ORIG_CFLAGS := $(KBUILD_CFLAGS)

@@ -133,8 +134,14 @@ bad_syms=$$($(CROSS_COMPILE)nm $@ | sed -n 's/^.\{8\} [bc] \(.*\)/\1/p') && \
   ( echo "following symbols must have non local/private scope:" >&2; \
     echo "$$bad_syms" >&2; rm -f $@; false )
 
+# For __aeabi_llsl
+ashldi3 = $(obj)/ashldi3.o
+
+$(obj)/ashldi3.S: $(srctree)/arch/$(SRCARCH)/lib/ashldi3.S FORCE
+	$(call cmd,shipped)
+
 $(obj)/vmlinux: $(obj)/vmlinux.lds $(obj)/$(HEAD) $(obj)/piggy.$(suffix_y).o \
-	 	$(addprefix $(obj)/, $(OBJS)) $(lib1funcs) FORCE
+	 	$(addprefix $(obj)/, $(OBJS)) $(lib1funcs) $(ashldi3) FORCE
 	$(call if_changed,ld)
 	@$(check_for_bad_syms)

arch/arm/boot/compressed/decompress.c +4 -0

index 07be5a2..0ecd8b4 100644
@@ -44,6 +44,10 @@ extern void error(char *);
 #include "../../../../lib/decompress_unlzma.c"
 #endif
 
+#ifdef CONFIG_KERNEL_XZ
+#include "../../../../lib/decompress_unxz.c"
+#endif
+
 int do_decompress(u8 *input, int len, u8 *output, void (*error)(char *x))
 {
 	return decompress(input, len, NULL, NULL, output, NULL, error);

arch/arm/boot/compressed/piggy.xzkern.S new file +6 -0

new file mode 100644
index 0000000..5703f30
@@ -0,0 +1,6 @@
+	.section .piggydata,#alloc
+	.globl	input_data
+input_data:
+	.incbin	"arch/arm/boot/compressed/piggy.xzkern"
+	.globl	input_data_end
+input_data_end:

lib/xz/xz_dec_stream.c +1 -0

index ac809b1..9a60cc2 100644
@@ -9,6 +9,7 @@
 
 #include "xz_private.h"
 #include "xz_stream.h"
+#include <linux/kernel.h>
 
 /* Hash used to validate the Index field */
 struct xz_dec_hash {

Signature · 1 line

1.7.1

Russell King - ARM Linux

2011-07-15 16:25 UTC #1 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 1 line · Imre Kaloz

Wire up support for the XZ decompressor

Ok.

Imre Kaloz

author

2011-07-16 14:07 UTC #2 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 4 lines · Russell King - ARM Linux

Quoted 1 line · Imre Kaloz
Wire up support for the XZ decompressor
Ok.

Thanks, added to the patch system.

Imre

Russell King - ARM Linux

2011-07-19 11:02 UTC #3 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 8 lines · Imre Kaloz

Quoted 4 lines · Russell King - ARM Linux
Quoted 1 line · Imre Kaloz
Wire up support for the XZ decompressor
Ok.
Thanks, added to the patch system.

FYI:

Patching 7001/1...
<stdin>:56: space before tab in indent.

                $(addprefix $(obj)/, $(OBJS)) $(lib1funcs) $(ashldi3) FORCE

warning: 1 line applied after fixing whitespace errors.

Russell King - ARM Linux

2011-07-19 11:07 UTC #4 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 8 lines · Russell King - ARM Linux

Quoted 7 lines · Imre Kaloz
Quoted 4 lines · Russell King - ARM Linux
Quoted 1 line · Imre Kaloz
Wire up support for the XZ decompressor
Ok.

Quoted 8 lines

Quoted 1 line
Thanks, added to the patch system.
FYI:
Patching 7001/1...
<stdin>:56: space before tab in indent.
                $(addprefix $(obj)/, $(OBJS)) $(lib1funcs) $(ashldi3) FORCE
warning: 1 line applied after fixing whitespace errors.

BTW, has anyone outside of the ARM community seen the patch to lib/xz/xz_dec_stream.c adding the linux/kernel.h include? How well tested is that change on other architectures?

Russell King - ARM Linux

2011-07-22 09:16 UTC #5 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 13 lines · Russell King - ARM Linux

Quoted 12 lines · Russell King - ARM Linux
Quoted 8 lines · Imre Kaloz
Quoted 4 lines · Russell King - ARM Linux
Quoted 1 line · Imre Kaloz
Wire up support for the XZ decompressor
Ok.
Thanks, added to the patch system.
FYI:

Quoted 8 lines

Quoted 4 lines
Patching 7001/1...
<stdin>:56: space before tab in indent.
                $(addprefix $(obj)/, $(OBJS)) $(lib1funcs) $(ashldi3) FORCE
warning: 1 line applied after fixing whitespace errors.
BTW, has anyone outside of the ARM community seen the patch to lib/xz/xz_dec_stream.c adding the linux/kernel.h include? How well tested is that change on other architectures?

Ok, no reply to this, so I'm dropping your patch for the merge window.

Imre Kaloz

author

2011-07-22 09:52 UTC #6 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 19 lines · Russell King - ARM Linux

Quoted 18 lines · Russell King - ARM Linux
Quoted 16 lines · Russell King - ARM Linux
Quoted 8 lines · Imre Kaloz
Quoted 4 lines · Russell King - ARM Linux
Quoted 1 line · Imre Kaloz
Wire up support for the XZ decompressor
Ok.
Thanks, added to the patch system.
FYI:
Patching 7001/1...
<stdin>:56: space before tab in indent.
                $(addprefix $(obj)/, $(OBJS)) $(lib1funcs) $(ashldi3) FORCE
warning: 1 line applied after fixing whitespace errors.

Quoted 6 lines

Quoted 3 lines
BTW, has anyone outside of the ARM community seen the patch to lib/xz/xz_dec_stream.c adding the linux/kernel.h include? How well tested is that change on other architectures?
Ok, no reply to this, so I'm dropping your patch for the merge window.

Guessed I'm not the one outside the ARM community, so.. I can say it didn't break neither ppc or mips for me, but have no idea who's feedback are you waiting for.

Imre

Russell King - ARM Linux

2011-07-22 09:57 UTC #7 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 25 lines · Imre Kaloz

Quoted 23 lines · Russell King - ARM Linux
Quoted 21 lines · Russell King - ARM Linux
Quoted 8 lines · Russell King - ARM Linux
Quoted 7 lines · Imre Kaloz
Quoted 4 lines · Russell King - ARM Linux
Quoted 1 line · Imre Kaloz
Wire up support for the XZ decompressor
Ok.
Quoted 8 lines
Quoted 1 line
Thanks, added to the patch system.
FYI:
Patching 7001/1...
<stdin>:56: space before tab in indent.
                $(addprefix $(obj)/, $(OBJS)) $(lib1funcs) $(ashldi3) FORCE
warning: 1 line applied after fixing whitespace errors.
BTW, has anyone outside of the ARM community seen the patch to lib/xz/xz_dec_stream.c adding the linux/kernel.h include? How well tested is that change on other architectures?

Quoted 6 lines

Quoted 2 lines
Ok, no reply to this, so I'm dropping your patch for the merge window.
Guessed I'm not the one outside the ARM community, so.. I can say it didn't break neither ppc or mips for me, but have no idea who's feedback are you waiting for.

What I'm concerned about is the apparant lack of exposure of the patch to people outside of ARM, especially as it touches lib/xz/xz_dec_stream.c.

I have no idea whether that breaks some architecture or not.

Is Lasse Collin (the original contributer of that file) aware of this change?

What is the additional include fixing on ARM? Why isn't this include needed on other architectures? And why isn't any of this mentioned in the patch commit log?

Imre Kaloz

author

2011-07-22 10:27 UTC #8 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 9 lines · Russell King - ARM Linux

Quoted 3 lines · Imre Kaloz
Guessed I'm not the one outside the ARM community, so.. I can say it didn't break neither ppc or mips for me, but have no idea who's feedback are you waiting for.
What I'm concerned about is the apparant lack of exposure of the patch to people outside of ARM, especially as it touches lib/xz/xz_dec_stream.c.

Quoted 9 lines

I have no idea whether that breaks some architecture or not.
Is Lasse Collin (the original contributer of that file) aware of this change?
What is the additional include fixing on ARM? Why isn't this include needed on other architectures? And why isn't any of this mentioned in the patch commit log?

It's needed because of missing min_t - but just looked a bit further and it seems decompress.c should use kernel.h - so I send v3 right away with the following change if you agree:

arch/arm/boot/compressed/decompress.c +1 -1

@@ -1,7 +1,7 @@
  #define _LINUX_STRING_H_
 
  #include <linux/compiler.h>	/* for inline */
-#include <linux/types.h>	/* for size_t */
+#include <linux/kernel.h>	/* for min_t */
  #include <linux/stddef.h>	/* for NULL */
  #include <linux/linkage.h>
  #include <asm/string.h>

Russell King - ARM Linux

2011-07-22 12:50 UTC #9 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

In reply to Russell King - ARM Linux

Quoted 7 lines · Russell King - ARM Linux

What I'm concerned about is the apparant lack of exposure of the patch to people outside of ARM, especially as it touches lib/xz/xz_dec_stream.c.
I have no idea whether that breaks some architecture or not.
Is Lasse Collin (the original contributer of that file) aware of this change?

I forwarded it to Lasse, and got some comments back... essentially putting includes in xz_dec_stream.c could be problematical. Lasse would - as I did - like to have the full information on why the include was necessary.

Lasse is - again like me - concerned about dragging stuff into the pre-boot environment which can cause it to fail unexpectedly, especially as linux/kernel.h includes a whole raft of other includes.

What I don't understand is if this code requires min_t, where does it get this for everyone else - that's something to be discussed in the wider discussion over this. Does it build outside of the decompressor on ARM already?

So... given the increasing number of questions over this, I think you need to put something together which describes the problem, how its caused, why it seems to only exhibit on ARM, and discuss solutions with all those involved. I know Lasse is very interested to find out more information about your patch...

But in the mean time, this isn't a patch for 3.1.

Imre Kaloz

author

2011-07-22 19:57 UTC #10 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 23 lines · Russell King - ARM Linux

Quoted 7 lines · Russell King - ARM Linux
What I'm concerned about is the apparant lack of exposure of the patch to people outside of ARM, especially as it touches lib/xz/xz_dec_stream.c.
I have no idea whether that breaks some architecture or not.
Is Lasse Collin (the original contributer of that file) aware of this change?
I forwarded it to Lasse, and got some comments back... essentially putting includes in xz_dec_stream.c could be problematical. Lasse would - as I did - like to have the full information on why the include was necessary.
Lasse is - again like me - concerned about dragging stuff into the pre-boot environment which can cause it to fail unexpectedly, especially as linux/kernel.h includes a whole raft of other includes.
What I don't understand is if this code requires min_t, where does it get this for everyone else - that's something to be discussed in the wider discussion over this. Does it build outside of the decompressor on ARM already?

Quoted 8 lines

So... given the increasing number of questions over this, I think you need to put something together which describes the problem, how its caused, why it seems to only exhibit on ARM, and discuss solutions with all those involved. I know Lasse is very interested to find out more information about your patch...
But in the mean time, this isn't a patch for 3.1.

I've already proposed the right way to fix it in http://lists.arm.linux.org.uk/lurker/message/20110722.102718.6e9fe66c.en.html

Imre

Russell King - ARM Linux

2011-07-22 20:07 UTC #11 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

Quoted 1 line · Imre Kaloz

I've already proposed the right way to fix it in http://lists.arm.linux.org.uk/lurker/message/20110722.102718.6e9fe66c.en.html

I think you're missing the point.

So, as I've already said I've dropped your original patch, and I'm now talking directly to Lasse about how best to resolve this (which IMHO is what should've already happened.) If Lasse doesn't want to fix it in his code then we'll see about modifying ARM specific stuff.

As you've already pointed out, the code uses min_t, but doesn't contain an include of linux/kernel.h - so maybe its also broken on other arches.

Until it's discussed, we won't know.

Imre Kaloz

author

2011-07-19 11:09 UTC #12 lore raw

PATCH v2 1/1 ARM: support XZ compressed kernels

In reply to Russell King - ARM Linux

Quoted 8 lines · Russell King - ARM Linux

Quoted 7 lines · Imre Kaloz
Quoted 4 lines · Russell King - ARM Linux
Quoted 1 line · Imre Kaloz
Wire up support for the XZ decompressor
Ok.

Quoted 9 lines

Quoted 1 line
Thanks, added to the patch system.
FYI:
Patching 7001/1...
<stdin>:56: space before tab in indent.
                $(addprefix $(obj)/, $(OBJS)) $(lib1funcs) $(ashldi3) FORCE
warning: 1 line applied after fixing whitespace errors.

That space is there since there's git history for the file ;)

Imre