Skip to contentSkip to conversation

Clean up ARM compressed loader

PATCH New hector at marcansoft.com 3 participants 11 messages Feb 23, 2010 – Feb 25, 2010 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Clean up ARM compressed loader

From: Hector Martin <[email protected]>

The -Dstatic= stuff in the Makefile is a hack and subtly breaks the current inflate implementation, because the fixed inflate tables are included into a function and removing static places them in the stack, which bloats the stack and breaks references after the function returns. So get rid of the hack.

Instead, clean up the stub loader so PIC vs ROM mode is handled sanely now. You can build two notably different versions now:

CONFIG_ZBOOT_ROM=y
- Non-PIC: loader will halt if it finds itself at the wrong address.
- Separate ROM and RAM bases (CONFIG_ZBOOT_ROM_TEXT and CONFIG_ZBOOT_ROM_BSS, the latter being a slight misnomer now).
- Proper initialized data handling. Initialized data is copied from ROM to RAM on boot (VMA != LMA in linker script).
- Assumes that the decompressed kernel will not overlap with the RAM used by the decompressor itself (CONFIG_ZBOOT_ROM_BSS should be somewhere after the loaded kernel address, e.g. near the end of RAM). No post-decompression relocation is done.

CONFIG_ZBOOT_ROM=n
- PIC code: you can load this anywhere
- Single blob, data follows code.
- The loaded kernel may overlap with the loader, in which case it is relocated after compression as usual.

As far as I can tell, ROM PIC would've never worked with the existing code (because, among other things, the GOT ends up in ROM), so this shouldn't break anything.

This is untested on a real ROM platform, but has been tested using RAM as fake ROM.

Cc Segher Boessenkool <[email protected]>
---
 arch/arm/Kconfig                        |   10 ++-
 arch/arm/boot/compressed/Makefile       |   25 +++----
 arch/arm/boot/compressed/head.S         |  129 ++++++++++++++++--------------
 arch/arm/boot/compressed/misc.c         |    3 +-
 arch/arm/boot/compressed/vmlinux.lds.in |   29 ++++++--
 5 files changed, 110 insertions(+), 86 deletions(-)
arch/arm/Kconfig +7 -3
index 184a6bd..deb9eef 100644
@@ -1278,8 +1278,8 @@ config ZBOOT_ROM_BSS
 	help
 	  The base address of an area of read/write memory in the target
 	  for the ROM-able zImage which must be available while the
-	  decompressor is running. It must be large enough to hold the
-	  entire decompressed kernel plus an additional 128 KiB.
+	  decompressor is running. It must be able to hold at least 128 KiB
+	  of data and it must not overlap with the loaded kernel.
 	  Platforms which normally make use of ROM-able zImage formats
 	  normally set this to a suitable value in their defconfig file.
 
@@ -1290,7 +1290,11 @@ config ZBOOT_ROM
 	depends on ZBOOT_ROM_TEXT != ZBOOT_ROM_BSS
 	help
 	  Say Y here if you intend to execute your compressed kernel image
-	  (zImage) directly from ROM or flash.  If unsure, say N.
+	  (zImage) directly from ROM or flash. If you say Y, then the zImage
+	  will only run from the ZBOOT_ROM_TEXT address (it will not be
+	  position-independent).
+
+	  If unsure, say N.
 
 config CMDLINE
 	string "Default kernel command string"
arch/arm/boot/compressed/Makefile +10 -15
index 2d4d88b..acfd324 100644
@@ -48,20 +48,22 @@ else
 endif
 endif
 
-#
-# We now have a PIC decompressor implementation.  Decompressors running
-# from RAM should not define ZTEXTADDR.  Decompressors running directly
-# from ROM or Flash must define ZTEXTADDR (preferably via the config)
-# FIXME: Previous assignment to ztextaddr-y is lost here. See SHARK
+# RAM builds should disable CONFIG_ZBOOT_ROM. The decompressor will be
+# built as position independent code and can be loaded at any address.
+# ROM builds should enable CONFIG_ZBOOT_ROM and set CONFIG_ZBOOT_ROM_TEXT
+# and CONFIG_ZBOOT_ROM_BSS. They will be built as position dependent code
+# and will only run when loaded at CONFIG_ZBOOT_ROM_TEXT.
 ifeq ($(CONFIG_ZBOOT_ROM),y)
 ZTEXTADDR	:= $(CONFIG_ZBOOT_ROM_TEXT)
-ZBSSADDR	:= $(CONFIG_ZBOOT_ROM_BSS)
+ZDATAADDR	:= $(CONFIG_ZBOOT_ROM_BSS)
+EXTRA_CFLAGS	:= -fno-builtin
 else
 ZTEXTADDR	:= 0
-ZBSSADDR	:= ALIGN(4)
+ZDATAADDR	:= ALIGN(4)
+EXTRA_CFLAGS	:= -fpic -fno-builtin
 endif
 
-SEDFLAGS	= s/TEXT_START/$(ZTEXTADDR)/;s/BSS_START/$(ZBSSADDR)/
+SEDFLAGS	= s/TEXT_START/$(ZTEXTADDR)/;s/DATA_START/$(ZDATAADDR)/
 
 suffix_$(CONFIG_KERNEL_GZIP) = gzip
 suffix_$(CONFIG_KERNEL_LZO)  = lzo
@@ -75,7 +77,6 @@ ORIG_CFLAGS := $(KBUILD_CFLAGS)
 KBUILD_CFLAGS = $(subst -pg, , $(ORIG_CFLAGS))
 endif
 
-EXTRA_CFLAGS  := -fpic -fno-builtin
 EXTRA_AFLAGS  := -Wa,-march=all
 
 # Supply ZRELADDR, INITRD_PHYS and PARAMS_PHYS to the decompressor via
@@ -106,10 +107,6 @@ lib1funcs = $(obj)/lib1funcs.o
 $(obj)/lib1funcs.S: $(srctree)/arch/$(SRCARCH)/lib/lib1funcs.S FORCE
 	$(call cmd,shipped)
 
-# Don't allow any static data in misc.o, which
-# would otherwise mess up our GOT table
-CFLAGS_misc.o := -Dstatic=
-
 $(obj)/vmlinux: $(obj)/vmlinux.lds $(obj)/$(HEAD) $(obj)/piggy.$(suffix_y).o \
 	 	$(addprefix $(obj)/, $(OBJS)) $(lib1funcs) FORCE
 	$(call if_changed,ld)
@@ -120,8 +117,6 @@ $(obj)/piggy.$(suffix_y): $(obj)/../Image FORCE
 
 $(obj)/piggy.$(suffix_y).o:  $(obj)/piggy.$(suffix_y) FORCE
 
-CFLAGS_font.o := -Dstatic=
-
 $(obj)/font.c: $(FONTC)
 	$(call cmd,shipped)
 
arch/arm/boot/compressed/head.S +69 -60

208 lines not on this page · raw message

arch/arm/boot/compressed/misc.c +1 -2
index 56a0d11..97018f8 100644
@@ -23,7 +23,6 @@ unsigned int __machine_arch_type;
 #include <linux/compiler.h>	/* for inline */
 #include <linux/types.h>	/* for size_t */
 #include <linux/stddef.h>	/* for NULL */
-#include <asm/string.h>
 #include <linux/linkage.h>
 
 #include <asm/unaligned.h>
@@ -166,7 +165,7 @@ void __memzero (__ptr_t s, size_t n)
 		*u.ucp++ = 0;
 }
 
-static inline __ptr_t memcpy(__ptr_t __dest, __const __ptr_t __src,
+static __ptr_t memcpy(__ptr_t __dest, __const __ptr_t __src,
 			    size_t __n)
 {
 	int i = 0;
arch/arm/boot/compressed/vmlinux.lds.in +23 -6
index a5924b9..46efdf4 100644
@@ -36,19 +36,36 @@ SECTIONS
 
   _etext = .;
 
+  /* The GOT is only for PIC builds, so it should not exist in the ROM build */
   _got_start = .;
   .got			: { *(.got) }
   _got_end = .;
   .got.plt		: { *(.got.plt) }
-  .data			: { *(.data) }
-  _edata = .;
 
-  . = BSS_START;
+  . = ALIGN(16);
+
+  __data_lma = .;
+
+  . = DATA_START;
+
+  __data_start = .;
+  .data : AT(__data_lma) {
+    *(.data)
+    . = ALIGN(16);
+  }
+  __data_end = .;
+  _edata = __data_lma + __data_end - __data_start;
+
   __bss_start = .;
-  .bss			: { *(.bss) }
-  _end = .;
+  .bss : { *(.bss) }
+  . = ALIGN(16);
+  __bss_end = .;
 
-  .stack (NOLOAD)	: { *(.stack) }
+  .stack : {
+    __stack_start = .;
+    . += 4096;
+    __stack_end = .;
+  }
 
   .stab 0		: { *(.stab) }
   .stabstr 0		: { *(.stabstr) }
Signature · 1 line
1.6.4.4
#1 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 3 lines · hector at marcansoft.com

As far as I can tell, ROM PIC would've never worked with the existing code (because, among other things, the GOT ends up in ROM), so this shouldn't break anything.

Clearly then you don't understand the implementation.

With ROM mode, you build it for the address you want to run it from, and if you place it into ROM, you *must* run it from that address because the GOT table can't be updated.

However, the intention is that you can still run the image from RAM and have it work correctly - and it does, because that's a situation I've tested when this was initially written.

Maybe the recent switch to the new inflate code broke this - in which case that's a problem with the new inflate code which should be fixed. Putting static data in functions is not a particularly nice thing to do.

#2 lore raw

PATCH 1/1 Clean up ARM compressed loader

In reply to hector at marcansoft.com

Hello

[expanded Cc:]

Quoted 5 lines · hector at marcansoft.com

From: Hector Martin <[email protected]>

The -Dstatic= stuff in the Makefile is a hack and subtly breaks the current inflate implementation, because the fixed inflate tables are included into a function and removing static places them in the stack,

Maybe this is the problem that people see after e7db7b4270? Is this the final hint that we should revert that one?

Quoted 2 lines

which bloats the stack and breaks references after the function returns. So get rid of the hack.

Best regards
Uwe

Signature · 2 lines
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
#3 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 4 lines · Uwe Kleine-K?nig

Hello

[expanded Cc:]

Quoted 8 lines
Quoted 5 lines · hector at marcansoft.com

From: Hector Martin <[email protected]>

The -Dstatic= stuff in the Makefile is a hack and subtly breaks the current inflate implementation, because the fixed inflate tables are included into a function and removing static places them in the stack,

Maybe this is the problem that people see after e7db7b4270? Is this the final hint that we should revert that one?

Quite likely it is this same issue. The switch to the other inflate implementation is probably what introduced the breakage. However, e7db7b4270 itself is perfectly fine. The problem has been lurking all along in the -Dstatic= thing, and that would've caused issues with several perfectly valid C constructs in the kernel lib/inflate stuff.

I can also 'fix' the issue with a two-line patch to the new deflate code, but that's not a proper solution, just an ugly workaround. I don't think restricting the inflate code to a weird subset of C due to ARM peculiarities is a good idea.

This can also break with GCC updates - the current code depends on unspecified behavior of the compiler. Essentially, the split relocation is something that is not allowed nor expected by the compiler, and -Dstatic= is a crude hack to attempt to modify code so it doesn't trigger the cases where the hack breaks down. But it doesn't work, and breaks legitimate code. Even worse, the bugs it introduces can be subtle heisenbugs, as was the case here - I spent two afternoons tracking it down through the bowels of inflate and comparing the output vs. an x86 build. In a worst-case scenario, the bug could've caused corrupted kernel output (not just a flat-out hang or crash while uncompressing), which would've been so much fun to debug (*cough*).

Signature · 2 lines
Hector Martin (hector at marcansoft.com)
Public Key: http://www.marcansoft.com/marcan.asc
#4 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 1 line · Hector Martin

But it doesn't work, and breaks legitimate code.

It does work with the previous version of the decompressor.

I noticed that you did not reply to my previous email on this subject, are you intentionally ignoring my responses?

#5 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 1 line · Russell King - ARM Linux

It does work with the previous version of the decompressor.

Sure, at this point in time. The broken code is still the loader, not the new decompressor. If a GCC update had been the culprit, it wouldn't be GCC's fault. The fact remains that the behavior is undefined and (with current GCC versions) requires carefully crafting the resulting C code in order to get reasonable behavior. I don't think you can reasonably require that a generic descompressor be maintained by others with care for compatibility with this hack; someone would have to police changes for potential issues, and you still have a decent chance of getting breakage if GCC decides to change its behavior some day. I'd say fixing the loader to not require this undefined behavior is a considerably better long-term solution.

Quoted 2 lines

I noticed that you did not reply to my previous email on this subject, are you intentionally ignoring my responses?

You had a valid point which I hadn't considered (running ROM images from RAM), so I figured I'd give addressing that a shot before going into discussion mode.

Signature · 2 lines
Hector Martin (hector at marcansoft.com)
Public Key: http://www.marcansoft.com/marcan.asc
#6 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 15 lines · Hector Martin
Quoted 1 line · Russell King - ARM Linux

It does work with the previous version of the decompressor.

Sure, at this point in time. The broken code is still the loader, not the new decompressor. If a GCC update had been the culprit, it wouldn't be GCC's fault. The fact remains that the behavior is undefined and (with current GCC versions) requires carefully crafting the resulting C code in order to get reasonable behavior. I don't think you can reasonably require that a generic descompressor be maintained by others with care for compatibility with this hack; someone would have to police changes for potential issues, and you still have a decent chance of getting breakage if GCC decides to change its behavior some day. I'd say fixing the loader to not require this undefined behavior is a considerably better long-term solution.

Quoted 6 lines
Quoted 2 lines

I noticed that you did not reply to my previous email on this subject, are you intentionally ignoring my responses?

You had a valid point which I hadn't considered (running ROM images from RAM), so I figured I'd give addressing that a shot before going into discussion mode.

Independant of the outcome of this discussion I suggest reverting the switch to the new decompressor for .33. Hectors patch definitely isn't .33 material and broken or not the old decompressor was more reliable.

In general I would welcome to get rid of -Dstatic=, but it needs careful consideration.

Best regards
Uwe

Signature · 2 lines
Pengutronix e.K.                           | Uwe Kleine-K?nig            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
#7 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 3 lines · Uwe Kleine-K?nig

Independant of the outcome of this discussion I suggest reverting the switch to the new decompressor for .33. Hectors patch definitely isn't .33 material and broken or not the old decompressor was more reliable.

That's reasonable, as far as .33 goes. Obviously I'm not expecting my patch to make it to this release.

Signature · 2 lines
Hector Martin (hector at marcansoft.com)
Public Key: http://www.marcansoft.com/marcan.asc
#8 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 7 lines · Hector Martin
Quoted 3 lines · Uwe Kleine-K?nig

Independant of the outcome of this discussion I suggest reverting the switch to the new decompressor for .33. Hectors patch definitely isn't .33 material and broken or not the old decompressor was more reliable.

That's reasonable, as far as .33 goes. Obviously I'm not expecting my patch to make it to this release.

Could you put together a description for the commit log for this revert please?

#9 lore raw

PATCH 1/1 Clean up ARM compressed loader

Quoted 10 lines · Russell King - ARM Linux
Quoted 6 lines · Hector Martin
Quoted 3 lines · Uwe Kleine-K?nig

Independant of the outcome of this discussion I suggest reverting the switch to the new decompressor for .33. Hectors patch definitely isn't .33 material and broken or not the old decompressor was more reliable.

That's reasonable, as far as .33 goes. Obviously I'm not expecting my patch to make it to this release.

Could you put together a description for the commit log for this revert please?

Something like this?

Revert "arm: add support for LZO-compressed kernels"

This commit switches to a different inflate implementation. This implementation is subtly broken when used within the current ARM compressed loader due to some build hacks. Revert the switch until those hacks can be removed.

Signature · 2 lines
Hector Martin (hector at marcansoft.com)
Public Key: http://www.marcansoft.com/marcan.asc
#10 lore raw

PATCH 1/1 Clean up ARM compressed loader

In reply to Hector Martin
Quoted 14 lines · Hector Martin
Quoted 1 line · Russell King - ARM Linux

It does work with the previous version of the decompressor.

Sure, at this point in time. The broken code is still the loader, not the new decompressor. If a GCC update had been the culprit, it wouldn't be GCC's fault. The fact remains that the behavior is undefined and (with current GCC versions) requires carefully crafting the resulting C code in order to get reasonable behavior. I don't think you can reasonably require that a generic descompressor be maintained by others with care for compatibility with this hack; someone would have to police changes for potential issues, and you still have a decent chance of getting breakage if GCC decides to change its behavior some day. I'd say fixing the loader to not require this undefined behavior is a considerably better long-term solution.

For starters, do you actually understand how this stuff works, or are you just viewing the "-Dstatic=" as a hack you don't like?

Let's examine what's going on. If you build a file with -fpic, then you're asking the compiler to generate position independent code - eg for shared libraries.

How this is achieved depends on the architecture. On ARM, it is achieved as follows:

1. references to global data are indirected via the GOT since they can be placed anywhere - in the executable or in another shared library. This is well defined and the user environment relies upon this behaviour.

2. references to local 'static' data are private to the shared library, and are addressed relative to the position of the GOT using the address of the GOT and an offset from the GOT written into the code. This is well defined and the user environment relies upon this behaviour.

3. function references to local 'static' functions are made using standard branch instructions since these are already pc relative

4. function references to global functions are via the PLT.

Now, with the kernel decompressor, we don't care about (3) or (4) - indeed, all functions could be static.

However, we do care about (1) vs (2). For read-only static data, (2) is entirely acceptable - static read-only data can be placed in the .rodata section and move with the text segment.

However, for writable local data, even if we tell the linker to locate the data segment in RAM, the relative offset between the data and text is fixed at built time. What this means is that if you try and relocate the image, any read-write data using (2) will most probably end up outside RAM.

In order to be able to relocate the data independently of the text segment, we need the read-write data to be built with global visibility, thereby causing (1) to be used.

We could investigate whether there's a better solution to "-Dstatic=" to ensure that we end up with (1) for all read/write data.

So, to sum up, contrary to your belief, the behaviour we're invoking from the toolchain is not some spurious toolchain behaviour, but well-defined behaviour which, if it changes, results in userspace breakage.

What is the hack is defining 'static' as a way to get rid of all local data, thereby avoiding writable data being addressed relative to the GOT.