r19 (aka pic-register akak ltp) not restored on entry back to libc from libpthread?

jda,

Perhaps you can help here with my problem, again it's an r19 related issue.

make[2]: *** [/glibc-cvs/build-hppa/posix/tst-regex.out] Error 139

Breakpoint 1, fixup (l=0xfaf00d68, reloc_offset=1248) at dl-runtime.c:72
72 alloca (sizeof (int));
(gdb) c 22

I exit the loader, and I have ~2700 insn before I get to here...

(gdb) si 2713

I've traced it down to:

0000000040024000-000000004016d000 r-xp 0000000000000000 08:04 1308771 /mnt/flaire/src/glibc-cvs/build-hppa/libc.so

Looks like "public_mALLOc" from libc/malloc/malloc.c trying to unlock the memory arena mutex.

00081ea0 <__libc_malloc>:
0x400a5f50: copy r4,r19
0x400a5f54: cmpib,= 0,ret0,0x400a5f8c
...

r19 = 40181d50 (All is good, all is quiet for 2700 insn)

(gdb) x/4 0x40181d50-0x1800+0x400
0x40180950: 0x400a6234 0x40181d50 0x40094320 0x40181d50

(Stub)

0x400abf0c:     b,l 0x400abf14,r1
0x400abf10:     addil 9f000,r1,%r1
0x400abf14:     be,n 71c(sr4,r1)
(Load r19 and target address from PLABEL)
0x4014b630:     bb,>=,n r22,1e,0x4014b640
0x4014b634:     depwi 0,31,2,r22
0x4014b638:     ldw 4(sr0,r22),r19
0x4014b63c:     ldw 0(sr0,r22),r22
0x4014b640:     bv r0(r22)
0x4014b644:     stw rp,-18(sr0,sp)

r19 = 401a57a8 (Good for libpthread ...)

(gdb) x /4 0x401a57a8-0x1800+0x400
0x401a43a8: 0x00000008 0x0000b6ac 0x0000b6e8 0x08000000

            ^^^^^^^^^^ Soon to be fatal return address.

000072e0 <__pthread_mutex_unlock>:

0x4018d2e0:     stw rp,-14(sr0,sp)
0x4018d2e4:     stw,ma r4,40(sr0,sp)
0x4018d2e8:     stw r19,-20(sr0,sp)		# 2719

...

0000000040186000-0000000040195000 r-xp 0000000000000000 08:04 655453 /mnt/flaire/src/glibc-cvs/build-hppa/linuxthreads/libpthread.so

0x4019063c: stw r19,-20(sr0,sp) # 2742
...
0x40190860: stw r19,-20(sr0,sp)
...
0x401908f8: bv r0(rp) # 2774
0x401908fc: ldo -80(sp),sp
...
0x40190770: bv r0(rp)
0x40190774: ldo -80(sp),sp
...
000072e0 <__pthread_mutex_unlock>
0x4018d334: bv r0(rp) # 2806
0x4018d338: ldw,mb -40(sr0,sp),r4
...
00081ea0 <__libc_malloc>
0x400a5f84: b,l 0x400a5edc,r0
0x400a5f88: copy r5,ret0
...

Hold your horses here, we made it back into libc but our ltp is still that which we loaded upon entry to libpthread? :(

0x400a5eec: bv r0(rp)
0x400a5ef0: ldw,mb -40(sr0,sp),r6
...
0x401028e4: cmpib,<> 0,r20,0x40102918
0x401028e8: copy r3,r25
...
0x40102918: b,l 0x40116658,rp # 2839
0x4010291c: copy r6,r26

First use of libpthread's r19 is fatal.

0x40116658:     addil -1800,r19,%r1		# 2840
0x4011665c:     ldw 400(sr0,r1),r21  <--- *BOOM* r21=0x8
0x40116660:     bv r0(r21)
0x40116664:     ldw 404(sr0,r1),r19 

Any thoughts? Did I miss something?

Cheers,
Carlos.

That's ok. It's the responsibility of the libc code to restore ltp after a call or exception. However, as discussed previously, there is no restoration after a syscall. That should be the system's job, although I believe you were going to introduce a hack/workaround to fix the syscalls that clobber r19.

You need to step through the libc code from the return point in libc to see why ltp isn't being restored. Possibly, libpthread is being called by assembly code that doesn't restore ltp. Normally, r19 is restored quite soon after a call.

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

For all syscalls whose wrappers are pure assembly or __asm(...) I have placed restorations to follow ABI. This fixed all of the major failures. It's a bit drastic, but it fixes the issue until I know exactly which syscall numbers clobber r19. It looks like just fork and it's variants that stick a return address into PT_REGS's r19. That's a side issue about optimization though.

There should be no assembly code glue between the calls. What about the gcc optimization where we don't restore r19 if it's not used between the last call and the return?

mutex_unlock becomes:
__libc_maybe_call (__pthread_mutex_unlock, (m), (*(int *)(m) = 0))

Which is a big monstrosity that looks like it might be casting things incorrectly for hppa...

===
#if defined _LIBC && defined IS_IN_libpthread

# define __libc_maybe_call(FUNC, ARGS, ELSE) FUNC ARGS
#else
# if defined __PIC__ || (defined _LIBC && defined SHARED)
#  define __libc_maybe_call(FUNC, ARGS, ELSE) \
  (__extension__ ({ __typeof (FUNC) *_fn = (FUNC); \
                    _fn != NULL ? (*_fn) ARGS : ELSE; }))
# else
#  define __libc_maybe_call(FUNC, ARGS, ELSE) \
  (FUNC != NULL ? FUNC ARGS : ELSE)
# endif
#endif
#if defined _LIBC && !defined NOT_IN_libc && defined SHARED
# define __libc_maybe_call2(FUNC, ARGS, ELSE) \
  ({__builtin_expect (__libc_pthread_functions.ptr_##FUNC != NULL, 0) \
    ? __libc_pthread_functions.ptr_##FUNC ARGS : ELSE; })
#else
# define __libc_maybe_call2(FUNC, ARGS, ELSE) __libc_maybe_call
# (__##FUNC, ARGS, ELSE)
#endif
===

I have class, then I'll be back to -E some of the malloc build to see what this evaluates to and then try to determine if gcc did the right thing.

c.

This seems all to work:

A test file

main() {
int t1 = 4;
__libc_maybe_call(test1, (t1), (t1=3));
}

compiles (PIC) to:

00000000 <main>:
[...]
20: 2a 60 00 00 addil 0,r19,%r1

                        20: R_PARISC_DLTIND21L  .LC0
  24:   48 21 00 00     ldw 0(r1),r1
                        24: R_PARISC_DLTIND14R  .LC0
  28:   0c 20 10 94     ldw 0(,r1),r20
  2c:   0c 74 12 98     stw r20,c(,r3)
  30:   0c 78 10 94     ldw c(,r3),r20
  34:   86 80 20 3a     cmpib,=,n 0,r20,58 <main+0x58>
  38:   0c 78 10 94     ldw c(,r3),r20
  3c:   0c 70 10 9a     ldw 8(,r3),r26
  40:   08 14 02 56     copy r20,r22
  44:   08 13 02 44     copy r19,r4
  48:   eb e0 00 00     b,l 50 <main+0x50>,r31
                        48: R_PARISC_PCREL17F   $$dyncall
  4c:   08 1f 02 42     copy r31,rp
  50:   08 04 02 53     copy r4,r19

[...]
00000000 <.LC0>:
0: 00 00 00 00 break 0,0
0: R_PARISC_PLABEL32 test1
[...]
00010578 <$$dyncall>:
10578: c7 d6 c0 12 bb,>=,n r22,1e,10588 <$$dyncall+0x10>
1057c: d6 c0 1c 1e depwi 0,31,2,r22
10580: 0e c8 10 93 ldw 4(,r22),r19
10584: 0e c0 10 96 ldw 0(,r22),r22
10588: ea c0 c0 00 bv r0(r22)
1058c: 6b c2 3f d1 stw rp,-18(sp)

The $$dyncall is where we indirect through r20 (which contains the function pointer). Note the copy restoring r19 around this.

James

Yah, the normal code that gcc generates to restore the ltp is very extensively tested. Shared libraries would break almost instantly if there were major problems.

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

If it were major we would have fixed it :)
I'm trying to find the "missed a restore" point.

c.

0x4019063c: stw r19,-20(sr0,sp) # 2742

Just a note, GCC saves r19 in the frame marker in the prologue of non-leaf functions but we never attempt to restore r19 from the frame marker. This is the mandated ABI behavior. At the moment, GCC copies r19 to r4 for the save.

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

I would like to start this off with:
"JDA said don't put r19 into clobber list"

I didn't listen. I thought it should all still work.
James reports that removing r19 from the clobber list works, but I still haven't rebuilt my tree, so I'll see.

Example (Assembly trace provided at the end):

getcwd: (Start of function)

	stw r19,-20(sr0,sp)
	...
syscall:
	(Syscall with save/load r19 wrapper)
	stw r19,-20(sr0,sp)
	be,l 100(sr2,r0),%sr0,%r31
	ldi 6e,r20
	ldw -20(sr0,sp),r19
	...
	(Many insn later)
	(stub) (dyncall) -> libpthread.so
	-> libc.so
	(r19 not restored)
	(Jump to syscall:)
	...

o GCC is confused by the r19 asm(...) clobber?

Notes:

I generated insn traces using gdb scripts.

__pthread_mutex_unlock: 0x7730 <0x4018c730> (In libpthread)
__libc_malloc: 0x7f4a0 <0x400a34a0> (In libc)

(return stub)
0x400a34a0: b,l 0x400a33f8,r0
0x400a34a4: copy r5,ret0

(__libc_malloc returning)

0x400a33f8:     ldw -54(sr0,sp),rp
0x400a33fc:     ldw -3c(sr0,sp),r5		(No need to restore r19)
0x400a3400:     ldw -38(sr0,sp),r4
0x400a3404:     ldw -34(sr0,sp),r3
0x400a3408:     bv r0(rp)			(Back to getcwd)
0x400a340c:     ldw,mb -40(sr0,sp),r6

getcwd: 0xdb128 <0x400ff128>

0x400ff128:     ldi 0,r21
0x400ff12c:     cmpib,<> 0,ret0,0x400ff034	(Jump back to do syscall)
0x400ff130:     copy ret0,r6

0x400ff034:     copy r3,r25
0x400ff038:     copy r6,r26
0x400ff03c:     stw r19,-20(sr0,sp)
0x400ff040:     be,l 100(sr2,r0),%sr0,%r31
0x400ff044:     ldi 6e,r20
0x400ff048:     ldw -20(sr0,sp),r19 (si gdb artifact, lost insn inside syscall)
0x400ff048:     ldw -20(sr0,sp),r19 
0x400ff04c:     ldi -1000,r20
0x400ff050:     cmpb,>>= r20,ret0,0x400ff070
0x400ff054:     copy ret0,r3

0x400ff070:     cmpib,>,n 0,r3,0x400ff0cc
0x400ff074:     cmpiclr,<> 0,r7,r21
0x400ff078:     ldi 1,r21
0x400ff07c:     cmpiclr,<> 0,r5,r20
0x400ff080:     ldi 1,r20
0x400ff084:     and r20,r21,r20
0x400ff088:     cmpib,<> 0,r20,0x400ff0bc
0x400ff08c:     copy r3,r25			(No r19 restore yet!!!)

(call stub)
0x400ff0bc: b,l 0x40114e2c,rp
0x400ff0c0: copy r6,r26

(stub)

0x40114e2c:     addil -1800,r19,%r1
0x40114e30:     ldw 428(sr0,r1),r21
0x40114e34:     bv r0(r21)
0x40114e38:     ldw 42c(sr0,r1),r19

getcwd: 0xdb0c0 <0x400ff0c0>
(stub) *BOOM*

Is there any way we can make this work?

c.

It would be better to use a general register for the save/restore. If you are going to use the slot in the frame marker, it probably isn't necessary to save r19 before every syscall. GCC saves r19 in the slot in the prologue of all pic functions. However, we don't currently copy the value when when the function does a dynamic stack allocation. That's easily fixed.

The above is an indirect call. r19 should be restored after the call if it is used after the call.

It is likely that clobbering r19 in the syscall causes the restore of r19 to be deleted. Because of the clobber, GCC believes that r19 is dead and the restore insn isn't needed. I think you should be able to see this by looking at the rtl for the routine. The save and restore of r19 in a call are split out after the GCC reload pass.

I'm still somewhat confused. Isn't the syscall going to clobber r19? Don't you need to save and restore r19 here, rather than in the wrapper? There seems to be a call (i.e., you die in a linker call stub) before you get back to the wrapper to restore r19.

The management of r19 is very tricky. It can't be exposed before reload as all uses of r19 are not known until that time.

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

So use a caller saves register, place it in the clobbers, and let gcc work around the usage (e.g. r4).

It is, but only after the asm(...) that lists r19 in the clobber.

The presence of r19 in an asm(...) clobber seems to confuse GCC into deleting an r19 restore after a return from an interlibrary call. We die in the linker call stub because it uses libpthread's r19, because GCC deleted the scheduled restore. Why it deleted the restore is unknown to me. The syscall wrapper makes sure the syscall doesn't trash r19, it doesn't assure that the incoming r19 isn't already wrong :)

Agreed, tricky to track down too.

c.

Sorry "It is" should read "It is used" :)
Things get tricky in email with complex issues.

c.

Well that's why the restore is deleted. So, the clobber must go and r19 must be preserved across the asm(...). As I said, you can't clobber r19 when generating pic code. It's treated as fixed register, so clobbering it can lead to undefined behavior.

If the syscall used r19, it could include a use of r19 in the asm. This would ensure the restore occurs before the syscall.

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

I'm going to stop writing r19 to the stack and tell gcc I'm using r4, and then do a copy/copy to get r19 back.

c.

Yes.

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)