gcc-4.8+ and R10000+

I've been banging my head on the desk over gcc PR61538 [1] the last few months, and talking to the gcc people, I went looking through the R10000 manual again to try and see if some kind of errata sticks out. I found this bit:

"""
Load Linked and Store Conditional instructions (LL, LLD, SC, and SCD) do not implicitly perform SYNC operations in the R10000. Any of the following events that occur between a Load Linked and a Store Conditional will cause the Store Conditional to fail: an exception; execution of an ERET, a load, a store, a SYNC, a CacheOp, a prefetch, or an external intervention/invalidation on the block containing the linked address. Instruction cache misses do not cause the Store Conditional to fail.
"""

The regression happens inside glibc's __lll_lock_wait_private routine:

void
__lll_lock_wait_private (int *futex)
{
if (*futex == 2)

    lll_futex_wait (futex, 2, LLL_PRIVATE);

while (atomic_exchange_acq (futex, 2) != 0)

    lll_futex_wait (futex, 2, LLL_PRIVATE);
}

It appears to hang forever on the "atomic_exchange_acq" function call.

Disassembling a statically-built copy of the "sln" binary generated by glibc's compile phase, there are slight differences in how gcc-4.7 and gcc-4.8 are compiling the __lll_lock_wait_private function. The key differences in the output asm are
this:

gcc-4.7:
x+4 <START>

         ...
   x+24  bne     v1,v0,<x+56>
         ...
   x+32  0x7c03e83b /* rdhwr */
   x+36  li      a2,2
   x+40  lw      a1,-29832(v1)
   x+44  move    a3,zero
   x+48  li      v0,4238
   x+52  syscall
*  x+56  li      v0,2
*  x+60  ll      v1,0(s0)
*  x+64  move    a0,v0
*  x+68  sc      a0,0(s0)
   x+72  beqzl   a0,<x+56>
   x+76  nop
   x+80  sync
   x+84  bnez    v1,<x+32>

gcc-4.8:
x+4 <START>

         ...
   x+24  bne     v1,v0,<x+56>
         ...
   x+32  0x7c03e83b /* rdhwr */
   x+36  li      a2,2
   x+40  lw      a1,-29832(v1)
   x+44  move    a3,zero
   x+48  li      v0,4238
   x+52  syscall
*  x+56  ll      v0,0(s0)
*  x+60  li      at,2
*  x+64  sc      at,0
   x+68  beqzl   at,<x+56>
   x+72  nop
   x+76  sync
   x+80  bnez    v0,<x+32>

Using gdb, if I step through 'sln', the gcc-4.7 copy never calls __lll_lock_wait_private, so I have no idea how the insns are being executed.
But the 4.8 copy does get into this function, and stepping each instruction
at a time yields this execution path:

x+4 <START>

         ...
   x+24  bne     v1,v0,<x+56>
   x+56  ll      v0,0(s0)
   x+68  beqzl   at,<x+56> /* beqzl check fails -> x+76 */
   x+76  sync
   x+80  bnez    v0,<x+32>
   x+32  0x7c03e83b /* rdhwr */
   x+36  li      a2,2
   x+40  lw      a1,-29832(v1)
   x+44  move    a3,zero
   x+48  li      v0,4238
   x+52  syscall
   x+56  ll      v0,0(s0)

<HANG>

Executing the 'bnez' insn puts us at the rdhwr insn (x+32), then stepping through, the 'syscall' (x+56) returns and leaves us at the 'll' a second time, where the program just hangs.

I am guessing at a few things here:

- Because ll/sc are atomic, gdb doesn't let you step through them, which is why the instruction pointer jumps over the 'li' and 'sc' insns.

- The 'li' after 'll' triggers the 'sc' to fail on R10K.

Does this look correct for an R10000, given the above statement from the manual? I'm not sure how or why this would cause the program to hang, but it seems to directly correlate.

Anyone from Debian able to test building gcc-4.8 (or greater) and glibc-2.19 on an R10K system and see if it hangs at the end of glibc's compile phase using the 'sln' binary to generate symlinks? I've ran into this on R12000 and R14000 systems. I am assuming it'll happen on an R10000 system as well.

1: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61538

Joshua Kinard
Gentoo/MIPS
[email protected]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

[...]

Note how the sc address is no longer 0(s0). Since the address does not match the address used in the ll instruction, sc will always fail on the R10k.

Miod

That would be a typo on my part. I typed that out by hand and just missed it. It should read:

gcc-4.8:
x+4 <START>

         ...
   x+24  bne     v1,v0,<x+56>
         ...
   x+32  0x7c03e83b /* rdhwr */
   x+36  li      a2,2
   x+40  lw      a1,-29832(v1)
   x+44  move    a3,zero
   x+48  li      v0,4238
   x+52  syscall
*  x+56  ll      v0,0(s0)
*  x+60  li      at,2
*  x+64  sc      at,0(s0)
   x+68  beqzl   at,<x+56>
   x+72  nop
   x+76  sync
   x+80  bnez    v0,<x+32>

Thanks!,

Joshua Kinard
Gentoo/MIPS
[email protected]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

I did some more tracing. It seems the issue with glibc itself stems from the addition of __atomic_* builtins added generally in gcc-4.7 and MIPS-specifically in gcc-4.8:

Joshua,

I can't help you with the problem, but I can confirm one of your guesses:

-- this is exactly the case, GDB tries to be smart enough and when it sees an LL or LLD instruction it examines code that follows to find a matching SC or SCD instruction and any other exit points from the atomic section and sets internal breakpoints correctly to let the code fragment run at the full speed even if single stepping. Otherwise the exception taken at each single step would cause the conditional store instruction to always fail -- which might not be a big issue if you were knowingly stepping code e.g. with `stepi', but would cause big harm in implicit stepping through unknown or unrelated code such as when a software watchpoint is active.

See `deal_with_atomic_sequence' in gdb/mips-tdep.c if curious about the
details.

Maciej

Ah ha, that does explain it! Though I don't think it's an issue with ll/sc in the R10000. It's something with gcc's builtin __atomic_* functions I think, though I still haven't ruled the kernel out yet, either. I have no way to step through the kernel syscall to make that determination, though, so I'm focusing more on gcc as time permits. Hopefully, the gcc maintainers will find time to look into PR61538 some more soon.

Thanks!

Joshua Kinard
Gentoo/MIPS
[email protected]
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic