binutils woes

Hi guys,

On ARM, we appear to have somewhat of a problem with binutils. At least the following binutils suffer from a problem whereby it is possible to create programs which contain undefined symbols:

GNU assembler 2.13.90.0.18 20030206
GNU assembler 2.14 20030612
GNU assembler 2.14.90 20031229
GNU ld version 2.14.90.0.7 20031029 Debian GNU/Linux Assembleur GNU 2.15.90.0.1 20040303

We discovered this back in April, and at the time we thought that adding --no-undefined to the linker command line would ensure that we caught this bug.

However, this appears not to be the case. I recently committed this change:

http://linux.bkbits.net:8080/linux-2.5/[email protected]??nav=index.html|ChangeSet@-4d

and tested it - it appeared to build fine, and nothing untoward. However, as you can see from this patch:

http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1950/1

It appears that we are successfully build a kernel without the correct value of SIZEOF_MACHINE_DESC - in fact, this constant appears to be undefined in arch/arm/kernel/head.S. Upon checking the build logs and object files here, I have been able to confirm that this is exactly what has happened.

Here is an example of the build output around the affected files:

...

  AS      arch/arm/kernel/debug.o
  LD      arch/arm/kernel/built-in.o
  AS      arch/arm/kernel/head.o
  CC      arch/arm/kernel/init_task.o
...
  CC      arch/arm/lib/udivdi3.o
  AR      arch/arm/lib/lib.a
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
  KSYM    .tmp_kallsyms1.S
  AS      .tmp_kallsyms1.o
  LD      .tmp_vmlinux2
  KSYM    .tmp_kallsyms2.S
  AS      .tmp_kallsyms2.o
  LD      vmlinux

OBJCOPY arch/arm/boot/Image
Kernel: arch/arm/boot/Image is ready

As you can see - there is no indication of failure what so ever. However, if we inspect the symbol information for vmlinux, we find:

[rmk@dyn-67 versatile]$ arm-linux-nm vmlinux | grep SIZEOF -2 c0008000 T _sinittext
c0060634 T si_swapinfo
U SIZEOF_MACHINE_DESC
c013cb7c T sk_alloc
c014cc34 T sk_attach_filter

and further investigation of the disassembly indicates that the assembler subsituted a value of zero for this constant.

Since ld --no-undefined does not catch this error, and this error affects a range of binutils versions from 2.13 through to 2.15, I don't think we can ignore the problem.

I think the only way we can ensure kernel correctness is to add a subsequent stage to kbuild such that whenever we generate a final program, we grep the 'nm' output for undefined symbols.

Comments?

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

[snip]

Is there a version of binutils that really does get things right? If so, can't you Just Say No to older versions and force people to upgrade (with a simple testcase done upfront) ?

Tom Rini
http://gate.crashing.org/~trini/

I've just tested:

GNU assembler 2.15.90 20040409
Copyright 2002 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of the GNU General Public License. This program has absolutely no warranty. This assembler was configured for a target of `arm-linux'.

and it also is aflicted with the same problem. I guess the answer to your question is that currently there are no recent versions which work correctly.

Looking at the timescale of 2.13 through to present day, we're talking potentially of 2 years of binutils versions with this problem, and even if it was fixed tomorrow, I don't think it's acceptable to say "use the latest binutils version."

Also, I have just received another report that somewhere in the kernel is an undefined reference to another symbol - I_BIT, but where we have no idea at present...

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

You might try (which should be on any kernel mirror):

GNU assembler 2.15.91.0.1 20040527
Copyright 2002 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of the GNU General Public License. This program has absolutely no warranty. This assembler was configured for a target of `i686-pc-linux-gnu'.

Martin Schlemmer

I don't think there's any point in trying - I just received a mail from Nick Clifton who has found the problem in the ARM backend of GAS and is apparantly in the process of committing it to CVS.

So, all ARM binutils versions from 2002 to date are affected by the bug.

Whether ld --no-undefined successfully producing binaries with undefined symbols is a problem or not remains unclear, however.

Even so, we still can not check whether we are actually using a late enough version of binutils and force kernel build failure on that basis. About the best we can do is as I originally suggested which is to add a postprocessing step after we link a binary object to ensure there are no undefined symbols remaining.

The alternative is we just say "Sorry, we're not prepared to provide any form of support for problems reported for ARM builds until the next stable release of binutils." Realistically, I don't think that's an acceptable position to take.

Therefore, unless anyone has any objections, I shall be cooking up a patch which adds an extra pass to any final object link for the kernel build system.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Ok, I decided that touching the rules used for the boot loader(s) was probably going to be too fraught, so here's one which just checks the final link for the main kernel image.

Essentially, we run 'nm' against the object, and look for any line which matches the pattern '^ *U '. With this, a failing output looks like:

LD init/built-in.o
LD .tmp_vmlinux1
ldchk: .tmp_vmlinux1: final image has undefined symbols:
SIZEOF_MACHINE_DESC
make[1]: *** [.tmp_vmlinux1] Error 1
make: *** [vmlinux] Error 2

and successful output:

  LD      init/built-in.o
  LD      .tmp_vmlinux1
  KSYM    .tmp_kallsyms1.S
  AS      .tmp_kallsyms1.o
  LD      .tmp_vmlinux2
  KSYM    .tmp_kallsyms2.S
  AS      .tmp_kallsyms2.o
  LD      vmlinux

===== Makefile 1.500 vs edited =====

@@ -533,6 +533,8 @@
 endef
 
 #	set -e makes the rule exit immediately on error
+#	Note: Ensure that there are no undefined symbols in the final
+#	linked image.  Not doing this can lead to silent link failures.
 
 define rule_vmlinux__
 	+set -e;							\

@@ -545,6 +547,12 @@
 	$(if $($(quiet)cmd_vmlinux__),					\
 	  echo '  $($(quiet)cmd_vmlinux__)' &&) 			\
 	$(cmd_vmlinux__);						\
+	if $(NM) $@ | grep -q '^ *U '; then				\
+		echo 'ldchk: $@: final image has undefined symbols:';	\
+		$(NM) $@ | sed 's/^ *U \(.*\)/  \1/p;d';		\
+		$(RM) -f $@;						\
+		exit 1;							\
+	fi;								\
 	echo 'cmd_$@ := $(cmd_vmlinux__)' > $(@D)/.$(@F).cmd
 endef
 

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

What is the user supposed to do when this happens?

Report it as an error to the most appropriate forum. Then it becomes a hunt for why the symbol is undefined.

Sorry, afaics it's not possible to do this any better.

To illustrate the severity of the problem better, the following program:

.globl _start
_start:
add r0, r0, #UNDEFINED_SYMBOL
mov pc, lr

will successfully assemble and link on ARM, and is in fact runnable despite having a symbol table of the following:

$ arm-linux-nm t
00008074 t $a
0001007c A __bss_end__
0001007c A _bss_end__
0001007c A __bss_start
0001007c A __bss_start__
0001007c D __data_start
0001007c A _edata
0001007c A _end
0001007c A __end__
00008074 T _start
         U UNDEFINED_SYMBOL

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoting Russell King <[email protected]>:

Is there a chance that this screws up binaries for x86, too? I'm having various segfault problems at the moment. Maybe I should update.

- --
Regards Michael Buesch [ http://www.tuxsoft.de.vu ]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA5T9QFGK1OIvVOP4RAnraAJ0eODTOfojvwKfwISmbK4ZSRlD2NgCgj7o7 QV61gaI87O0K7xBbzcLf5Sc=
=bKrH
-----END PGP SIGNATURE-----

...

BTW, binutils 2.15 on ARM has other issues, which I've reported to Nick Clifton/binutils-bug. The assembler seems to produce extra symbolic information in the form of symbols starting with '$', eg:

c0067860 T __fput
c0067860 t $a
c0067954 t $d
c0067958 T fget
c0067958 t $a

Not only do these symbols interfere with ld's error reporting, but they also interfere with kallsyms - the backtrace information from such a kernel is not very useful when it reports function names of '$a':

drivers/built-in.o(.text+0x6042c): In function `$a':
: relocation truncated to fit: R_ARM_PC24 .exit.text

Obviously not useful to anyone.

This seems to only affect binutils 2.15, so I think we should hang fire on any work-arounds until the binutils folk have commented.

Russell King
 ... who dearly wishes that the kernel build was still compatible
  with the good old stable binutils 2.11 versions.

Ok, here's the latest version of the patch. It shouldn't complain on Sparc64, and neither should it complain while linking the .tmp_vmlinux* objects - the original regexp on the objdump --syms output was catching the weak kallsyms symbols.

The only issue with this is that, when a problem is detected, the reported symbols will also include the Sparc64 register symbols.

===== Makefile 1.500 vs edited =====

@@ -534,6 +534,8 @@
 
 #	set -e makes the rule exit immediately on error
 
+#	Note: Ensure that there are no undefined symbols in the final
+#	linked image.  Not doing this can lead to silent link failures.
 define rule_vmlinux__
 	+set -e;							\
 	$(if $(filter .tmp_kallsyms%,$^),,				\

@@ -545,6 +547,12 @@
 	$(if $($(quiet)cmd_vmlinux__),					\
 	  echo '  $($(quiet)cmd_vmlinux__)' &&) 			\
 	$(cmd_vmlinux__);						\
+	if $(OBJDUMP) --syms $@ | egrep -q '^([^R]|R[^E]|RE[^G])[^w]*\*UND\*'; then	\
+		echo 'ldchk: $@: final image has undefined symbols:';	\
+		$(NM) $@ | sed 's/^ *U \(.*\)/  \1/p;d';		\
+		$(RM) -f $@;						\
+		exit 1;							\
+	fi;								\
 	echo 'cmd_$@ := $(cmd_vmlinux__)' > $(@D)/.$(@F).cmd
 endef
 

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core