Non-bootable kernel problems

Current cvs kernel source doesn't boot. Looks like a linker problem or bits of the kernel loaded in the wrong place to me...

I added the following to the end of net/ipv4/devinet.c:

void check_devinet(void)

{
        printk("**  devinet_sysctl.devinet_root_dir = %p\n", devinet_sysctl.devinet_root_dir);
        printk("**  devinet_sysctl.devinet_root_dir[0].procname = %p\n", devinet_sysctl.devinet_root_dir[0].procname);
        printk("**  devinet_sysctl.devinet_root_dir[0].procname = %s\n", devinet_sysctl.devinet_root_dir[0].procname);
}

and then added a call to check_devinet() to init/main.c() just after printk(linux_banner). It should report procname as "net" from the table in net/ipv4/devinet.c, but:

Linux version 2.3.99-pre8 ([email protected]) (gcc version 2.96.0614) #660 Thu Jul 13 00:21:59 BST 2000

**  devinet_sysctl.devinet_root_dir = c0279814
**  devinet_sysctl.devinet_root_dir[0].procname = c0143228
**  devinet_sysctl.devinet_root_dir[0].procname = Multiply assigned logical unit

pagetable_init

Someone else can pick it up now while I sleep ;-)

Richard

More info on this one -- and yes it's the old problem come back to haunt us.

Witness two functions:

    static void * a2(void) {
	return (void *)&devinet_sysctl.devinet_root_dir;
    }
    static void * a3(void) {
	return (void *)&devinet_sysctl;
    }

And their return values:

    a3 c028b22c a2 c0289574

a2 should be c028b574 -- it's off by 0x2000 as Richard noted earlier. a3() is OK and a2() is broken.

Here's disassembly from vmlinux via objdump:

    c021b27c <a2>:
    c021b27c:       2b 6f 00 00     addil 1e000,dp,%r1
    c021b280:       e8 40 c0 00     bv r0(rp)
    c021b284:       34 3c 2a e9     ldo -a8c(r1),ret0

    c021b288 <a3>:
    c021b288:       2b 6f 00 00     addil 1e000,dp,%r1
    c021b28c:       e8 40 c0 00     bv r0(rp)
    c021b290:       34 3c 24 58     ldo 122c(r1),ret0

The two ldo instructions should be showing a difference of 0x348 -- the ofset of the member within the struct which I calculated separately. As shown, the offsets differ by 0x2348. This code is hosed.

The object file says:

    00000000 <a2>:
       0:   2b 60 00 00     addil 0,dp,%r1
			    0: R_PARISC_DPREL21L    .data+0x78
       4:   e8 40 c0 00     bv r0(rp)
       8:   34 3c 00 00     ldo 0(r1),ret0
			    8: R_PARISC_DPREL14R    .data+0x3c0
    Disassembly of section .text.a3:

    00000000 <a3>:
       0:   2b 60 00 00     addil 0,dp,%r1
			    0: R_PARISC_DPREL21L    .data+0x78
       4:   e8 40 c0 00     bv r0(rp)
       8:   34 3c 00 00     ldo 0(r1),ret0
			    8: R_PARISC_DPREL14R    .data+0x78

I think this code is right, because 0x3c0-0x78 is 0x348. I think this makes this a linker problem. The off-by-0x2000 is suspicious since the ldo offset is 14 bits and 2^14 = 0x2000.

For completeness here's what came out of the compiler, which looks cool to me:

    a1:
	    .PROC
	    .CALLINFO FRAME=0,NO_CALLS
	    .ENTRY
	    addil LR'devinet_sysctl-$global$,%r27
	    bv %r0(%r2)
	    ldw RR'devinet_sysctl-$global$+844(%r1),%r28
	    .EXIT
	    .PROCEND

    a2:
	    .PROC
	    .CALLINFO FRAME=0,NO_CALLS
	    .ENTRY
	    addil LR'devinet_sysctl-$global$,%r27
	    bv %r0(%r2)
	    ldo RR'devinet_sysctl-$global$+840(%r1),%r28

It looks like binutils is incorrectly treating those fields as unsigned, or has an off by one error, or something similar. The reason why the version with the extra check and panic() makes it work is because this makes gcc cleverly load the offset in the struct in two steps. note:

devinet.c.1.3:

#warning Take this test out when GCC is fixed

	if ((unsigned)&devinet_sysctl.devinet_root_dir <= (unsigned)&devinet_sysctl.devinet_vars) {
		panic(__FILE__ ": probable gcc bug");
	}
	devinet_sysctl.sysctl_header =
		register_sysctl_table(devinet_sysctl.devinet_root_dir, 0);

devinet.s.1.3:

	addil LR'devinet_sysctl-$global$,%r27
	ldo RR'devinet_sysctl-$global$+4(%r1),%r3  ; &devinet_sysctl.devinet_vars (offset 4)
	ldo 836(%r3),%r19                          ; &devinet_sysctl.devinet_root_dir (offset 840)
	comb,>> %r19,%r3,.L2613
	copy %r19,%r26                             ; arg 0 of register_sysctl_table
	ldil LR'.LC44,%r26
	.CALL ARGW0=GR
	bl panic,%r2
	ldo RR'.LC44(%r26),%r26
.L2613:
	.CALL ARGW0=GR,ARGW1=GR
	bl register_sysctl_table,%r2
	nop

relevant bits of hppa-linux-objdump -S vmlinux for 1.3:

c02ae538:       2b 6f 00 00     addil 1e000,dp,%r1
c02ae53c:       34 23 24 e0     ldo 1270(r1),r3
c02ae540:       34 73 06 88     ldo 344(r3),r19
c02ae544:       88 73 a0 18     cmpb,>> r19,r3,c02ae558 <.L2613>
c02ae548:       08 13 02 5a     copy r19,r26

Note:

$ printf "%x\n" $((0x1270+0x344))
15b4

devinet.c.1.4:

	devinet_sysctl.sysctl_header =
		register_sysctl_table(devinet_sysctl.devinet_root_dir, 0);

devinet.s.1.4:

	addil LR'devinet_sysctl-$global$,%r27
	ldi 0,%r25
	ldo RR'devinet_sysctl-$global$+840(%r1),%r4
	.CALL ARGW0=GR,ARGW1=GR
	bl register_sysctl_table,%r2
	copy %r4,%r26

relevant bits of hppa-linux-objdump -S vmlinux for 1.4:

c02ae534:       2b 6f 00 00     addil 1e000,dp,%r1
c02ae538:       34 19 00 00     ldi 0,r25
c02ae53c:       34 24 2b 69     ldo -a4c(r1),r4
c02ae540:       e8 5f 08 15     b,l c02ad950 <.Lfe1+0x118>,rp
c02ae544:       08 04 02 5a     copy r4,r26

Note:

$ printf "%x\n" -0xa4c
fffff5b4

(and of course, the low 13 bits of that are 0x15b4)

I'm looking at the relevant binutils code right now.

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

No. My mistake. binutils is doing the right thing, the problem is that, due to the way the LR' and RR' field selectors work, there is a bad interaction between cases in which the DPREL21L and DPREL14R (or any 21L and 14R relocations actually) are "split" like this, and ld -r, which tends to increase the addends to a point where LR' and RR' have different effects (LR' expects RR' to be positive, but it isn't). Then during final relocation, the wrong thing happens.

So it would appear that this is a problem with GCC, but I'm not really sure.

I'll have a better explanation once I work through the failure modes and come up with a good short testcase.

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Okay, this is my final word on this for today, I promise that if I send any more mail on it, it will include a patch :-)

Here is a small program that calculates RR' and LR' fields based on a RELA according to the definitions in the PA-RISC ELF supplement:

#!/usr/bin/perl -w
use strict;

sub RND($) {
my $x = shift;
return (($x + 0x1000) & ~0x1fff);
}

sub R($) {
my $x = shift;
return $x & 0x7ff;
}

sub L($) {
my $x = shift;
return $x & ~0x7ff;
}

sub LR($$) {
my ($x, $a) = @_;
return L($x + RND($a));
}

sub RR($$) {
my ($x, $a) = @_;
return R($x + RND($a)) + ($a - RND($a));
}

while (<>) {
my ($value, $addend) = map hex, split;
my $lr = LR $value, $addend;
my $rr = RR $value, $addend;
printf "LR(%x, %x) = %x ; RR(%x, %x) = %x ; LR + RR = %x\n",

    $value, $addend, $lr, $value, $addend, $rr, $lr + $rr;
}

__END__

Now, we see that in net/ipv4/devinet.o we have:

24: 2b 60 00 00 addil 0,dp,%r1

			24: R_PARISC_DPREL21L	.data+0x78
  28:	34 19 00 00 	ldi 0,r25
  2c:	34 24 00 00 	ldo 0(r1),r4
			2c: R_PARISC_DPREL14R	.data+0x3c0

Then, in net/ipv4/ipv4.o (which is produced by running ld -r on the objects in that directory) we see that the addends have been increased:

     b9c:	2b 60 00 00 	addil 0,dp,%r1
			b9c: R_PARISC_DPREL21L	.data+0xd5c
     ba0:	34 19 00 00 	ldi 0,r25
     ba4:	34 24 00 00 	ldo 0(r1),r4
			ba4: R_PARISC_DPREL14R	.data+0x10a4

So, let's feed these addends to that script:

[dhd@tarwebok] ~/src/parisc/binutils-2.10$ perl test-broken-relocs.pl 0 78
LR(0, 78) = 0 ; RR(0, 78) = 78 ; LR + RR = 78
0 3c0
LR(0, 3c0) = 0 ; RR(0, 3c0) = 3c0 ; LR + RR = 3c0

In this case, you can see that everything is fine, because the LR fields will contain the same values for both relocations.

But now, let's see what happens after we relink it:

[dhd@tarwebok] ~/src/parisc/binutils-2.10$ perl test-broken-relocs.pl 0 d5c
LR(0, d5c) = 0 ; RR(0, d5c) = d5c ; LR + RR = d5c
0 10a4
LR(0, 10a4) = 2000 ; RR(0, 10a4) = fffff0a4 ; LR + RR = 10a4

As you can see, now the LR fields are out of sync, and we get a negative offset in the LDO instruction even though the addend + value is still well within the range of a 14-bit signed field.

As far as I can tell, the ABI and possibly the instruction set are broken by design by using an addition instead of a logical OR to fill in the low bits of an immediate value [1]. But we have to cope with this somehow.

As far as I can tell, we can either fix this in GCC or in the linker.

I don't know if the linker is able to associate paired 21L and 14R relocations, but if it were, I assume that it would be fairly easy to make sure that they stay "in sync" with each other when linking with -r.

However, it could be argued that the real problem is that GCC always omits the addend when outputting addil instructions for referring to global data. For example, this C code:

struct foo {
int a[192];
int b;
};

struct foo f = {
{}, 42
}, g = {
{}, 666
};

int a(void* foo)
{
}

int main()
{
a(&f.a[145]);
a(&f.b);
a(&g.b);
return;
}
Generates the following assembly code (edited for length and content):

main:

	.PROC
	.CALLINFO FRAME=64,CALLS,SAVE_RP
	.ENTRY
	stw %r2,-20(%r30)
	ldo 64(%r30),%r30
	addil LR'f-$global$,%r27                ; look ma, no addend
	.CALL ARGW0=GR
	bl a,%r2
	ldo RR'f-$global$+580(%r1),%r26         ; but we have one here!
	addil LR'f-$global$,%r27                ; no addend
	.CALL ARGW0=GR
	bl a,%r2
	ldo RR'f-$global$+768(%r1),%r26         ; addend
	addil LR'g-$global$,%r27                ; no addend
	.CALL ARGW0=GR
	bl a,%r2
	ldo RR'g-$global$+768(%r1),%r26         ; addend
	ldw -84(%r30),%r2
	bv %r0(%r2)
	ldo -64(%r30),%r30
	.EXIT
	.PROCEND

I don't see any good reason why GCC cannot put the addends on the ADDIL instructions as well. Manually munging the assembly code and assembling it certainly produces correct output, and in fact doing the same to net/ipv4/devinet.s in the kernel:

@@ -2806,7 +2806,7 @@
 	.CALL ARGW0=GR
 	bl register_netdevice_notifier,%r2
 	ldo RR'ip_netdev_notifier-$global$(%r1),%r26
-	addil LR'devinet_sysctl-$global$,%r27
+	addil LR'devinet_sysctl-$global$+840,%r27
 	ldi 0,%r25
 	ldo RR'devinet_sysctl-$global$+840(%r1),%r4
 	.CALL ARGW0=GR,ARGW1=GR

Fixes the non-booting problems :P

Of course, it would be a nice optimization if the above code could reuse %r1 for different parts of the same structure, if it were of an appropriate size. However GCC does not appear to do anything of the sort, and reading the RTL dumps indicates that the addends are actually present in the insns all the way up until final output (this is the RTX for the ADDIL in the .27.dbr dump from the above program):

(insn 8 35 45 (set (reg:SI 1 %r1)

        (high:SI (const:SI (plus:SI (symbol_ref:SI ("f"))
                    (const_int 580 [0x244]))))) 84 {post_std+6} (nil)
    (expr_list:REG_EQUIV (high:SI (const:SI (plus:SI (symbol_ref:SI ("f"))
                    (const_int 580 [0x244]))))
        (nil)))

Implementing the fix is left as an exercise to the writer (argh).

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Nice bit of debugging, David. Have a look in
gcc/config/pa/pa.c:output_global_address

/* How bogus. The compiler is apparently responsible for
rounding the constant if it uses an LR field selector.

The linker and/or assembler seem a better place since they have to do this kind of thing already.

If we fail to do this, HP's optimizing linker may eliminate an addil, but not update the ldw/stw/ldo instruction that uses the result of the addil. */
if (round_constant)
offset = ((offset + 0x1000) & ~0x1fff);

Zap these two lines, and I think the problem will go away.

Alan Modra

Linuxcare.  Support for the Revolution.

In message <[email protected]>y
ou write:

But that's totally the wrong thing to do as it will break hpux.

The linker is the problem.

jeff

I've conditionalized it on the existence of GNU ld, is that okay?

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

In message <[email protected]>you write:

No, because you're working around a bug in GNU-ld. GCC is following the defined behavior for the PA32 ABI.
jeff

In message <[email protected]>you write:

How/why is ld -r changing the addends? That seems wrong at first glance.

You might look at how the HP SOM tools handle addends during ld -r; I suspect they don't change.

To the best of my knowledge GCC is using LR/RR in the prescribed way.

jeff

The relocations in question are specified as <.data+0xf00>, and thus, as far as I can tell, it has to change the addends since there is no symbol value within .data that it can update.

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

In message <[email protected]>you write:

Either we need to avoid merging the section symbols to avoid twiddling the addend, or we need to avoid reducing relocs to section symbols to begin with.

jeff

We only reduce relocations to section symbols for static data. It seems that every other platform I've investigated (Sparc, PowerPC, Alpha) does the same thing, so, while I think it would be possible to not reduce them on PA, I'm not sure what the unintended side effects would be.

As for not merging section symbols ... I'm not sure how that would be done, but if it's the only option I guess we can figure it out.

Alan: any thoughts on this?

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Wait. I'm wrong. PowerPC doesn't:

static struct foo {
int a;
int b;
} a = {
a: 42,
b: 69
};

int main()
{
gar(&a.b);
...
}

Compiles to:

4c: 3d 20 00 00 lis r9,0

			4e: R_PPC_ADDR16_HA	a+0x4
  50:	38 69 00 04 	addi	r3,r9,4
			52: R_PPC_ADDR16_LO	a+0x4

Okay, I'll investigate this approach.

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Done. PA-RISC porters, update your binutils-2.10 and GCC. (again :-)

Here's the patch, which I've committed to puffin CVS (I've reverted the incorrect GCC changes):

2000-07-14 David Huggins-Daines <[email protected]>

* config/tc-hppa.c (hppa_fix_adjustable): Reject reductions of sym-sym2 expressions as well as LR% and RR% fields on ELF targets as well.

Index: tc-hppa.c
===================================================================
RCS file: /home/cvs/parisc/binutils-2.10/gas/config/tc-hppa.c,v retrieving revision 1.12
retrieving revision 1.13

diff -u -r1.12 -r1.13
@@ -8369,10 +8369,25 @@
   /* Reject reductions of symbols in 32bit relocs.  */
   if (fixp->fx_r_type == R_HPPA && hppa_fix->fx_r_format == 32)
     return 0;
+#endif
 
   /* Reject reductions of symbols in sym1-sym2 expressions when
      the fixup will occur in a CODE subspace.
 
+     NOTE: We want to reject these for ELF targets as well, because
+     otherwise we will lose in the case of 21L and 14R
+     relocations.  (FIXME: will this break ELF64 targets?)
+
+     This is because, since the symbols are reduced, the linker has no
+     choice but to manipulate the addends when coalescing input .data
+     sections.  Because the 32-bit ABI requires the compiler to
+     pre-round LR% field selectors, this means we end up with
+     different addends for the left and right relocations.  Because of
+     the way LR% and RR% field selectors work, it's possible that the
+     RR% field will turn negative without the LR% field being rounded
+     accordingly, and your Linux kernel crashes mysteriously.
+     - [email protected], 2000-07-14
+
      XXX FIXME: Long term we probably want to reject all of these;
      for example reducing in the debug section would lose if we ever
      supported using the optimizing hp linker.  */

@@ -8386,12 +8401,14 @@
     }
 
   /* We can't adjust any relocs that use LR% and RR% field selectors.
-     That confuses the HP linker.  */
+     That confuses the HP linker.
+
+     FIXME: do we actually need to do this for ELF32/GNU ld as well?
+     What about ELF64? */
   if (hppa_fix->fx_r_field == e_lrsel
       || hppa_fix->fx_r_field == e_rrsel
       || hppa_fix->fx_r_field == e_nlrsel)
     return 0;
-#endif
 
   /* Reject reductions of symbols in DLT relative relocs,
      relocations with plabels.  */

[email protected], http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Just a note on the patch. This particular bug existed in the elf hppa assembler long before anyone tried porting linux to parisc - it's not specific to the kernel. David's fix may just happen to cure some userland problems too.

Linuxcare.  Support for the Revolution.

My first thoughts when I read Jeff's email about my proposed removal of the gcc hack for the hpux linker were along the lines of "two wrongs don't make a right, so it's not surprising that ld -r is tripped up". That was late last night just before I went to bed, and I'm glad I didn't reply then, as Jeff is quite correct. The way those LR and RR field selectors work means that a reloc addend cannot be modified under any circumstances. Even if the gcc hack was removed, the linker would still do the wrong thing with certain cases of one instruction with LR selector combined with multiple instructions using RR selectors. Gross.

I'm not sure why the assembler is specifying the relocs against a sections symbols - it's something I've noticed on x86 for a long time, and never bothered to figure out why, as it didn't cause any problem. I'll have a look.

Regards, Alan Modra

Linuxcare.  Support for the Revolution.