Skip to contentSkip to conversation

envcrc: extract default environment from target ELF files

PATCH New Mike Frysinger 2 participants 5 messages Jun 16, 2009 – Jul 11, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 envcrc: extract default environment from target ELF files

Rather than rely on dirty hacks to compile the environment on the host and extract the CRC from that, have envcrc extract the environment straight from the ELF object that will be linked into u-boot itself. This makes the envcrc code a bit more complicated, but it simplifies the build process and host requirements because we don't have to try and recreate the environment that the target will be seeing on the host. This avoids some issues that crop up from time to time where the preprocessor defines on the host don't expand in the same way as for the target -- in case the target uses those defines to customize the environment, or the host defines conflicts with some of the target values.

---
 common/Makefile       |    4 +-
 common/env_embedded.c |   30 +---------
 tools/Makefile        |    3 +-
 tools/envcrc.c        |  156 ++++++++++++++++++++++++++++++++++++++++++-------
common/Makefile +2 -2
index c8e5d26..02bc71b 100644
@@ -173,9 +173,9 @@ all:	$(LIB) $(AOBJS)
 $(LIB): $(obj).depend $(OBJS)
 	$(AR) $(ARFLAGS) $@ $(OBJS)
 
-$(obj)env_embedded.o: $(src)env_embedded.c $(obj)../tools/envcrc
+$(obj)env_embedded.o: $(src)env_embedded.c $(obj)env_common.o $(obj)../tools/envcrc
 	$(CC) $(AFLAGS) -Wa,--no-warn \
-		-DENV_CRC=$(shell $(obj)../tools/envcrc) \
+		-DENV_CRC=$(shell $(obj)../tools/envcrc $(obj)env_common.o) \
 		-c -o $@ $(src)env_embedded.c
 
 $(obj)../tools/envcrc:
common/env_embedded.c +2 -28
index ae6cac4..60736c7 100644
@@ -21,26 +21,10 @@
  * MA 02111-1307 USA
  */
 
-#ifndef __ASSEMBLY__
-#define	__ASSEMBLY__			/* Dirty trick to get only #defines	*/
-#endif
-#define	__ASM_STUB_PROCESSOR_H__	/* don't include asm/processor.		*/
 #include <config.h>
-#undef	__ASSEMBLY__
 #include <environment.h>
 
 /*
- * Handle HOSTS that have prepended
- * crap on symbol names, not TARGETS.
- */
-#if defined(__APPLE__)
-/* Leading underscore on symbols */
-#  define SYM_CHAR "_"
-#else /* No leading character on symbols */
-#  define SYM_CHAR
-#endif
-
-/*
  * Generate embedded environment table
  * inside U-Boot image, if needed.
  */
@@ -75,7 +59,7 @@
 #else
 # define GEN_SET_VALUE(name, value) asm (GEN_SYMNAME(name) " = " GEN_VALUE(value))
 #endif
-#define GEN_SYMNAME(str) SYM_CHAR #str
+#define GEN_SYMNAME(str) #str
 #define GEN_VALUE(str) #str
 #define GEN_ABS(name, value) \
 		asm (".globl " GEN_SYMNAME(name)); \
@@ -197,17 +181,7 @@ env_t redundand_environment __PPCENV__ = {
 #endif	/* CONFIG_ENV_ADDR_REDUND */
 
 /*
- * These will end up in the .text section
- * if the environment strings are embedded
- * in the image.  When this is used for
- * tools/envcrc, they are placed in the
- * .data/.sdata section.
- *
- */
-unsigned long env_size __PPCTEXT__ = sizeof(env_t);
-
-/*
- * Add in absolutes.
+ * Add in absolutes.  This symbol is only referenced from linker scripts.
  */
 GEN_ABS(env_offset, CONFIG_ENV_OFFSET);
 
tools/Makefile +1 -2
index 43c284c..982d65a 100644
@@ -80,7 +80,6 @@ BIN_FILES-$(CONFIG_INCA_IP) += inca-swap-bytes$(SFX)
 BIN_FILES-$(CONFIG_NETCONSOLE) += ncb$(SFX)
 
 # Source files which exist outside the tools directory
-EXT_OBJ_FILES-y += common/env_embedded.o
 EXT_OBJ_FILES-y += lib_generic/crc32.o
 EXT_OBJ_FILES-y += lib_generic/md5.o
 EXT_OBJ_FILES-y += lib_generic/sha1.o
@@ -156,7 +155,7 @@ MAKEDEPEND = makedepend
 
 all:	$(obj).depend $(BINS) $(LOGO-y) subdirs
 
-$(obj)envcrc$(SFX):	$(obj)envcrc.o $(obj)crc32.o $(obj)env_embedded.o $(obj)sha1.o
+$(obj)envcrc$(SFX):	$(obj)envcrc.o $(obj)crc32.o $(obj)sha1.o
 	$(CC) $(CFLAGS) -o $@ $^
 
 $(obj)ubsha1$(SFX):	$(obj)ubsha1.o $(obj)sha1.o $(obj)os_support.o
tools/envcrc.c +134 -22

193 lines not on this page · raw message

Signature · 1 line
1.6.3.1
#1 lore raw

PATCH 1/1 envcrc: extract default environment from target ELF files

Dear Mike Frysinger,

Quoted 10 lines · In message <[email protected]> you wrote:

Rather than rely on dirty hacks to compile the environment on the host and extract the CRC from that, have envcrc extract the environment straight from the ELF object that will be linked into u-boot itself. This makes the envcrc code a bit more complicated, but it simplifies the build process and host requirements because we don't have to try and recreate the environment that the target will be seeing on the host. This avoids some issues that crop up from time to time where the preprocessor defines on the host don't expand in the same way as for the target -- in case the target uses those defines to customize the environment, or the host defines conflicts with some of the target values.

Can you please be a bit more specific about which sort of problems you are talking here? We've been using this code for many years now, but I cannot remember any problems with it.

Quoted 6 lines
---
 common/Makefile       |    4 +-
 common/env_embedded.c |   30 +---------
 tools/Makefile        |    3 +-
 tools/envcrc.c        |  156 ++++++++++++++++++++++++++++++++++++++++++-------

The diffstat indicates that the new solution is more complicated than the old one, so I'd like to understand why this is needed (or if).

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
If it has syntax, it isn't user friendly.
#2 lore raw

PATCH 1/1 envcrc: extract default environment from target ELF files

Quoted 13 lines · On Tuesday 16 June 2009 14:02:16 Wolfgang Denk wrote:
Quoted 11 lines · Mike Frysinger

Rather than rely on dirty hacks to compile the environment on the host and extract the CRC from that, have envcrc extract the environment straight from the ELF object that will be linked into u-boot itself. This makes the envcrc code a bit more complicated, but it simplifies the build process and host requirements because we don't have to try and recreate the environment that the target will be seeing on the host. This avoids some issues that crop up from time to time where the preprocessor defines on the host don't expand in the same way as for the target -- in case the target uses those defines to customize the environment, or the host defines conflicts with some of the target values.

Quoted 2 lines

Can you please be a bit more specific about which sort of problems you are talking here?

sure ... i'll be verbose so hopefully you can see what i see

Quoted 2 lines

We've been using this code for many years now, but I cannot remember any problems with it.

that's because the code paths have slowly acquired platform-specific hacks so that it kept working :)

at any rate, here's a longer problem description and surrounding issues ...

if the default environment utilizes defines that come from the toolchain in any level of indirection, the environment that is built into the target u-boot will not match the environment that is compiled on the host for crc generation.

a real world example is the unified ADI config header. it generates a default environment based on functionality actually available. the BF537 family has networking hardware in some parts, but not all (BF534 does not). so we key off of the variant to determine whether to enable networking by default. or we determine that some on-chip rom functions are available for certain variants and enable only those functions.

so the CPP dependency chain looks something like:
- Blackfin toolchain sets up variant defines
- ADI board headers set up capabilities based on variant defines
- common ADI board header enables some commands based on variant defines
- common ADI board header sets up default environment based on capabilities and commands available

since the host toolchain does not set up the same CPP dependencies as the Blackfin toolchain, this tree falls apart and the resulting environment string that envcrc operates on differs, so the CRCs differ.

i can understand your position of "setting up board configs this way is wrong", but i've structured it this way because it greatly reduces my maintenance burden while increasing regression capabilities. for example, i have a Blackfin-specific patch to the MAKEALL script that allows me to regression test not only the board-specific configuration, but also uncommon or otherwise untested cpu and boot mode configurations. for example, the bf537-stamp defaults to the BF537 cpu and BYPASS boot mode. but now my regression builds can automatically verify BF534/BF536 cpu code paths as well as PARALLEL/SPI/NAND/UART code paths. previously, some changes would accidentally break these uncommon edge cases and wouldnt be noticed until someone else happened to build. to say that these code paths were horribly broken most of the time is an understatement :).

i have also seen a case or two where the host toolchain inadvertently expanded things that ended up in the environment because they were not declared as strings. for example, many defines are not:
#define CONFIG_FOO "foo"
but rather they are:
#define CONFIG_FOO foo
just grep common/env_*.c for MKSTR() to see just how many things are affected in this way. having to worry about values here which may be arbitrarily expanded across host toolchains is a bad design practice imo.

if the environment is only ever compiled by one toolchain, the target one, then it significantly reduces the chance for unexpected and unwelcomed surprises. after all, the only way to notice something has gone wrong is to build the source with a specific host toolchain, burn the image, reset, and see the dreaded "CRC mismatch, using default environment" error from u-boot. unless you're familiar with this problem and/or have seen/root caused this before, attempting to track down the source of this problem can be very time consuming (as it was the first time i hit this problem).

Quoted 8 lines
Quoted 5 lines
 common/Makefile       |    4 +-
 common/env_embedded.c |   30 +---------
 tools/Makefile        |    3 +-
 tools/envcrc.c        |  156

++++++++++++++++++++++++++++++++++++++++++-------

The diffstat indicates that the new solution is more complicated than the old one, so I'd like to understand why this is needed (or if).

if LoC are the metric, then yes, but conceptually this change is simpler. the current behavior requires mixing target u-boot environment headers with the host toolchain, and a whole bunch of hacks to make sure things actually compile, produce the same end environment with two vastly different build environments, and end up with the same crc value. my proposed change means we only compile 1 environment object -- the one that is going into the target u- boot binary. we then extract the environment directly from the object which is actually going into u-boot (so we are absolutely assured that the environment we are working with on the host matches the target).

hope this clears things up
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20090619/1e3a4b95/attachment-0001.pgp

#3 lore raw

PATCH 1/1 envcrc: extract default environment from target ELF files

Dear Mike Frysinger,

Quoted 3 lines · In message <[email protected]> you wrote:
Quoted 3 lines

We've been using this code for many years now, but I cannot remember any problems with it.

...

Quoted 4 lines

if the default environment utilizes defines that come from the toolchain in any level of indirection, the environment that is built into the target u-boot will not match the environment that is compiled on the host for crc generation.

Hm... Defines that influence the envrionment should never come from the tool chain, but only from U-Boot source files.

Quoted 10 lines

so the CPP dependency chain looks something like:
- Blackfin toolchain sets up variant defines
- ADI board headers set up capabilities based on variant defines
- common ADI board header enables some commands based on variant defines
- common ADI board header sets up default environment based on capabilities and commands available

since the host toolchain does not set up the same CPP dependencies as the Blackfin toolchain, this tree falls apart and the resulting environment string that envcrc operates on differs, so the CRCs differ.

I see what your problem is.

Quoted 3 lines

i can understand your position of "setting up board configs this way is wrong", but i've structured it this way because it greatly reduces my maintenance burden while increasing regression capabilities. for example, i

I see. I understand what you have in mind, but I don't think that such a specific hack for a single user justifies such a change.

Instead of setting such defines in  the  tool  chain  (whatever  this
means  exactly)  it  would  be  probably  more  appropriate  to  pass
arguments or environment variables to make, I think -  and  not  more

difficult, too.

Quoted 9 lines

i have also seen a case or two where the host toolchain inadvertently expanded things that ended up in the environment because they were not declared as strings. for example, many defines are not:
#define CONFIG_FOO "foo"
but rather they are:
#define CONFIG_FOO foo
just grep common/env_*.c for MKSTR() to see just how many things are affected in this way. having to worry about values here which may be arbitrarily expanded across host toolchains is a bad design practice imo.

Patches for such problems are welcome.

Quoted 5 lines

if the environment is only ever compiled by one toolchain, the target one, then it significantly reduces the chance for unexpected and unwelcomed surprises. after all, the only way to notice something has gone wrong is to build the source with a specific host toolchain, burn the image, reset, and see the dreaded "CRC mismatch, using default environment" error from u-boot.

Ummm.. but this is not a real problem, because a simple "saveenv" will fix it, right? All boards that don't embed the environment work like this.

Quoted 1 line

hope this clears things up

I understand your arguments, but I don't come to the same conclusions. Sorry.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"Who is the oldest inhabitant of this village?"
"We haven't got one; we had one, but he died three weeks ago."
#4 lore raw

PATCH 1/1 envcrc: extract default environment from target ELF files

Quoted 8 lines · On Friday 10 July 2009 18:07:52 Wolfgang Denk wrote:
Quoted 4 lines · Mike Frysinger

i can understand your position of "setting up board configs this way is wrong", but i've structured it this way because it greatly reduces my maintenance burden while increasing regression capabilities. for example, i

I see. I understand what you have in mind, but I don't think that such a specific hack for a single user justifies such a change.

this isnt a hack. the code is thought out and designed to do one thing properly. if you want to see hacks, look at the nested mess that exists today in the code that is compiled on the host to produce the embedded environment. as for "single user", yes i might be the only arch so far *to complain aloud*, but that doesnt directly imply i'm the only one who would use it.

the current process design for creating the embedded env is fundamentally flawed. i'm not saying it doesnt work as is on most development systems we care about today, but working doesnt translate directly to "designed correctly".

Quoted 4 lines
Instead of setting such defines in  the  tool  chain  (whatever  this
means  exactly)  it  would  be  probably  more  appropriate  to  pass
arguments or environment variables to make, I think -  and  not  more

difficult, too.

i guess i could create a header that enumerates all the possible toolchain defines and translates them to CONFIG_xxx such that they get transferred between build environments, but that sounds pretty fragile and would require constant updates as new processors are supported. i dont think make env vars would work because these are dynamic CPP defines.

Quoted 11 lines
Quoted 9 lines

i have also seen a case or two where the host toolchain inadvertently expanded things that ended up in the environment because they were not declared as strings. for example, many defines are not:
#define CONFIG_FOO "foo"
but rather they are:
#define CONFIG_FOO foo
just grep common/env_*.c for MKSTR() to see just how many things are affected in this way. having to worry about values here which may be arbitrarily expanded across host toolchains is a bad design practice imo.

Patches for such problems are welcome.

i'm not interested in fighting a battle that, by design, can never be won

Quoted 10 lines
Quoted 6 lines

if the environment is only ever compiled by one toolchain, the target one, then it significantly reduces the chance for unexpected and unwelcomed surprises. after all, the only way to notice something has gone wrong is to build the source with a specific host toolchain, burn the image, reset, and see the dreaded "CRC mismatch, using default environment" error from u-boot.

Ummm.. but this is not a real problem, because a simple "saveenv" will fix it, right? All boards that don't embed the environment work like this.

yes, a saveenv will reset the crc, but again, this is a workaround for a design flaw. i'm interested in fixing these flaws, not telling people to "go run saveenv and pretend there isnt a problem". after all, it makes it pretty hard to take a u-boot binary image, send it to a factory for programming, and then additionally have the board run the native code merely to rebuild the environment crc that should have been correct in the first place.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20090710/31347006/attachment.pgp