glibc-2.5 test suite hangs/crashes the machine

DISCUSSION Fabio Massimo Di Nitto 4 participants 15 messages Oct 27, 2006 – Nov 1, 2006 PowerPC

lore

Share & tools

lore

Fabio Massimo Di Nitto

author

2006-10-27 06:13 UTC #0 lore raw

Hi everybody,

i am in the process of bootstrapping the new toolchain for ubuntu and I am hitting a problem building glibc-2.5 on ppc.

This behaviour has been reproduced on 2.6.15/2.6.17 and 2.6.19-rc2 (where the machine crashes) and with ppc32 and ppc64 kernels.
A hard reboot of the machine is required to get rid of the Zl processes hanging around that keep spinning the CPU at 100%.

I did place sources here: http://people.ubuntu.com/~fabbione/benh/

but i start to believe it is a kernel bug we are exploiting only now.

Any hint or help for what to look for would be extremely appreciated.

Thanks
Fabio

Signature · 1 line

I'm going to make him an offer he can't refuse.

Jeff Bailey

2006-10-27 16:22 UTC #1 lore raw

Le vendredi 27 octobre 2006 =C3=A0 07:56 +0200, Fabio Massimo Di Nitto a =C3=A9crit :

Quoted 3 lines

Hi everybody,
=20
i am in the process of bootstrapping the new toolchain for ubuntu and I a=

Quoted 3 lines

hitting a problem building glibc-2.5 on ppc.
=20
This behaviour has been reproduced on 2.6.15/2.6.17 and 2.6.19-rc2 (where=

the

Quoted 2 lines

machine crashes) and with ppc32 and ppc64 kernels.
A hard reboot of the machine is required to get rid of the Zl processes h=

anging

Quoted 7 lines

around that keep spinning the CPU at 100%.
=20
I did place sources here: http://people.ubuntu.com/~fabbione/benh/
=20
but i start to believe it is a kernel bug we are exploiting only now. =20
Any hint or help for what to look for would be extremely appreciated.

Heya Fabio, just an update, it looks like the tests that are zombie'ing are the nptl tst-robust[1-8] tests. According to /proc/##/wchan, the tasks are cheerfully spinning in do_exit.

If there's other info I can get you, lemme know.

--=20
Jeff Bailey
Manager, Technical Support
Canonical Ltd. - Sales, Service, and Support.
+1 514 940-8910
http://www.ubuntu.com/

Benjamin Herrenschmidt

2006-10-30 01:47 UTC #2 lore raw

Quoted 14 lines · Jeff Bailey

Le vendredi 27 octobre 2006 à 07:56 +0200, Fabio Massimo Di Nitto a écrit :
Quoted 12 lines
Hi everybody,
i am in the process of bootstrapping the new toolchain for ubuntu and I am hitting a problem building glibc-2.5 on ppc.
This behaviour has been reproduced on 2.6.15/2.6.17 and 2.6.19-rc2 (where the machine crashes) and with ppc32 and ppc64 kernels.
A hard reboot of the machine is required to get rid of the Zl processes hanging around that keep spinning the CPU at 100%.
I did place sources here: http://people.ubuntu.com/~fabbione/benh/

Quoted 7 lines

Quoted 3 lines
but i start to believe it is a kernel bug we are exploiting only now.
Any hint or help for what to look for would be extremely appreciated.
Heya Fabio, just an update, it looks like the tests that are zombie'ing are the nptl tst-robust[1-8] tests. According to /proc/##/wchan, the tasks are cheerfully spinning in do_exit.

So I've built that glibc with debian 2.6.16 kernel headers (since Fabio says the problem doesn't happen with glibc built with 2.6.19 headers) and have ran that with 2.6.19-rc3-git-du-jour.

The machine didn't crash, nor did I see any zombie with those tst-robust[1-8], however, I did get as SIGBUS with tst-robustpi1. I've tracked it down to being an alignment exception. It looks like glibc is doing a lwarx on a non-aligned value, though I can't say precisely what's up here. I don't know how I can get a backtrace when running those test-cases... the test harness seems to catch signals, I suppose it could be modified to spit one out.

At this point, it would be useful to have somebody who knows glibc to tell us:

- what are those tst-robust all about ? (what do they do "special" that
might trigger bad reactions with older kernels)
- how can glibc ever do atomic operations on a non-aligned value ?

Ben.

Steve Munroe

2006-11-01 22:18 UTC #3 lore raw

Benjamin Herrenschmidt <benh@kernel=2Ecrashing=2Eorg> wrote on 10/29/20= 06
07:47:05 PM:

Quoted 2 lines · Jeff Bailey

Quoted 1 line
Le vendredi 27 octobre 2006 =E0 07:56 +0200, Fabio Massimo Di Nitto=

Quoted 4 lines

Quoted 4 lines
=E9crit :
Quoted 3 lines
Hi everybody,
i am in the process of bootstrapping the new toolchain for ubuntu=

and
I am

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
hitting a problem building glibc-2=2E5 on ppc=2E
This behaviour has been reproduced on 2=2E6=2E15/2=2E6=2E17 and 2=

=2E6=2E19-

Quoted 7 lines

rc2 (where the
Quoted 2 lines
Quoted 2 lines
machine crashes) and with ppc32 and ppc64 kernels=2E A hard reboot of the machine is required to get rid of the Zl
processes hanging
Quoted 3 lines
Quoted 3 lines
around that keep spinning the CPU at 100%=2E
I did place sources here: http://people=2Eubuntu=2Ecom/~fabbione/=

benh/

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
but i start to believe it is a kernel bug we are exploiting only =

now=2E

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Any hint or help for what to look for would be extremely apprecia=

ted=2E

Quoted 2 lines

Quoted 2 lines
Heya Fabio, just an update, it looks like the tests that are zombie=

'ing

Quoted 1 line

Quoted 1 line
are the nptl tst-robust[1-8] tests=2E According to /proc/##/wchan,=

the

Quoted 3 lines

Quoted 1 line
tasks are cheerfully spinning in do_exit=2E
So I've built that glibc with debian 2=2E6=2E16 kernel headers (since=

Fabio

Quoted 1 line

says the problem doesn't happen with glibc built with 2=2E6=2E19 head=

ers)

Quoted 4 lines

and have ran that with 2=2E6=2E19-rc3-git-du-jour=2E
The machine didn't crash, nor did I see any zombie with those tst-robust[1-8], however, I did get as SIGBUS with tst-robustpi1=2E I=

've

Quoted 1 line

tracked it down to being an alignment exception=2E It looks like glib=

c is

Quoted 4 lines

doing a lwarx on a non-aligned value, though I can't say precisely what's up here=2E I don't know how I can get a backtrace when running=
those test-cases=2E=2E=2E the test harness seems to catch signals, I =

suppose

Quoted 7 lines

it could be modified to spit one out=2E
At this point, it would be useful to have somebody who knows glibc to=
tell us:
- what are those tst-robust all about ? (what do they do "special" t=

hat

Quoted 5 lines

might trigger bad reactions with older kernels)
- how can glibc ever do atomic operations on a non-aligned value ?
Ben=2E

The tst-robustpi# test are exercising the new PTHREAD_MUXTEX_ROBUST api= ,
with PTHREAD_PRIO_INHERIT attribute=2E

The fuxtex word seems to include the waiters TID, I don't know if the kernel cares about this or not=2E

Steven J=2E Munroe
Linux on Power Toolchain Architect
IBM Corporation, Linux Technology Center=

Benjamin Herrenschmidt

2006-11-01 22:37 UTC #4 lore raw

Quoted 6 lines

The tst-robustpi# test are exercising the new PTHREAD_MUXTEX_ROBUST api, with PTHREAD_PRIO_INHERIT attribute.
The fuxtex word seems to include the waiters TID, I don't know if the kernel cares about this or not.

Ok, well, we have seen a few issues so far with these. 2 are kernel issues, but one might not be:

- kernels 2.6.15 .. .17 at least it seems wire the robust futex
syscalls on powerpc without properly implementing the support, which can cause hangs in process exit. Do you have any way to "blacklist" kernels in glibc ?

- kernel 2.6.18 and current git until yesterday (fix got in today) has
a bug if you manage to pass a wrong futex with a non-aligned atomic value, it will possibly oops the kernel. With the fix, it will return an error.

Now what seems to be a glibc issue:

- I've had the tst-robustpi# tests (in fact the very first one, I
haven't tested the others) die on me with a SIGBUS caused by glibc trying to do a lwarx/starx. on an odd address.

I do not know for sure yet if the crash reported by Fabio with 2.6.19 (before my fix above) was related to the same kind of misaligned futex managing to reach the kernel and triggering the oops I've talked about, but it's very possible.

In my case, glibc was built against 2.6.16 headers, in fabio case, I think it was built against 2.6.15 or .17 headers. It -seems- that fabio cannot reproduce the problem when building glibc against 2.6.19 headers, though at this point I can't explain why and I haven't reproduced here yet.

Do you have any insight in what might be happening or should we just dig more ?

Cheers,
Ben.

Steve Munroe

2006-11-01 22:57 UTC #5 lore raw

Benjamin Herrenschmidt <[email protected]> wrote on 11/01/2006 04:35:52 PM:

Quoted 2 lines

Quoted 2 lines
The tst-robustpi# test are exercising the new PTHREAD_MUXTEX_ROBUST

api,

Quoted 8 lines

Quoted 4 lines
with PTHREAD_PRIO_INHERIT attribute.
The fuxtex word seems to include the waiters TID, I don't know if the kernel cares about this or not.
Ok, well, we have seen a few issues so far with these. 2 are kernel issues, but one might not be:

Quoted 6 lines

- kernels 2.6.15 .. .17 at least it seems wire the robust futex
syscalls on powerpc without properly implementing the support, which can cause hangs in process exit. Do you have any way to "blacklist" kernels in glibc ?
From libc/sysdeps/unix/sysv/linux/kernel-features.h

/* Support for inter-process robust mutexes was added in 2.6.17. */ #if __LINUX_KERNEL_VERSION >= 0x020611

# define __ASSUME_SET_ROBUST_LIST       1
#endif

/* Support for PI futexes was added in 2.6.18. */ #if __LINUX_KERNEL_VERSION >= 0x020612

# define __ASSUME_FUTEX_LOCK_PI 1
#endif

So I need to delay __ASSUME_SET_ROBUST_LIST to 2.6.18 for __powerpc__ ?

What about __ASSUME_FUTEX_LOCK_PI ?

Quoted 11 lines

- kernel 2.6.18 and current git until yesterday (fix got in today) has
a bug if you manage to pass a wrong futex with a non-aligned atomic value, it will possibly oops the kernel. With the fix, it will return an error.
Now what seems to be a glibc issue:
- I've had the tst-robustpi# tests (in fact the very first one, I
haven't tested the others) die on me with a SIGBUS caused by glibc trying to do a lwarx/starx. on an odd address.

Rayn reminded me of a bug where the new robust code did not account for the fact that the TID was not at the same place as i386. I think Ryan has a patch.

Steven J. Munroe
Linux on Power Toolchain Architect
IBM Corporation, Linux Technology Center

Benjamin Herrenschmidt

2006-11-01 23:23 UTC #6 lore raw

Quoted 2 lines

So I need to delay __ASSUME_SET_ROBUST_LIST to 2.6.18 for __powerpc__ ?

The problem is not so much the headers you build with than the runtime checks... I suppose you shouldn't bother as we are trickling down the fixes to distros & stable series anyway.

Quoted 1 line

What about __ASSUME_FUTEX_LOCK_PI ?

I don't know, I'm very unfamiliar with that futex stuff. I need to look more closely.

Quoted 7 lines

Quoted 7 lines
- kernel 2.6.18 and current git until yesterday (fix got in today) has
a bug if you manage to pass a wrong futex with a non-aligned atomic value, it will possibly oops the kernel. With the fix, it will return an error.
Now what seems to be a glibc issue:

Quoted 7 lines

Quoted 4 lines
- I've had the tst-robustpi# tests (in fact the very first one, I
haven't tested the others) die on me with a SIGBUS caused by glibc trying to do a lwarx/starx. on an odd address.
Rayn reminded me of a bug where the new robust code did not account for the fact that the TID was not at the same place as i386. I think Ryan has a patch.

Ah ok. Can you give me the details as soon as you get them ? (In addition to submitting the fix upstream of course :) ubuntu at least is starting to build their next distro on top of glibc-2.5 so it would be nice if they had the fix asap.

Thanks for looking at this !

Cheers,
Ben.

Benjamin Herrenschmidt

2006-10-30 03:02 UTC #7 lore raw

In reply to Fabio Massimo Di Nitto

Does that patch fixes it ?

arch/powerpc/kernel/traps.c +6 ~~-10~~

Index: linux-work/arch/powerpc/kernel/traps.c
===================================================================
@@ -843,7 +843,7 @@ void __kprobes program_check_exception(s
 
 void alignment_exception(struct pt_regs *regs)
 {
-	int fixed = 0;
+	int sig, fixed = 0;
 
 	/* we don't implement logging of alignment exceptions */
 	if (!(current->thread.align_ctl & PR_UNALIGN_SIGBUS))

@@ -856,15 +856,11 @@ void alignment_exception(struct pt_regs 
 	}
 
 	/* Operand address was bad */
-	if (fixed == -EFAULT) {
-		if (user_mode(regs))
-			_exception(SIGSEGV, regs, SEGV_ACCERR, regs->dar);
-		else
-			/* Search exception table */
-			bad_page_fault(regs, regs->dar, SIGSEGV);
-		return;
-	}
-	_exception(SIGBUS, regs, BUS_ADRALN, regs->dar);
+	sig = fixed == -EFAULT ? SIGSEGV : SIGBUS;
+	if (user_mode(regs))
+		_exception(sig, regs, SEGV_ACCERR, regs->dar);
+	else
+		bad_page_fault(regs, regs->dar, sig);
 }
 
 void StackOverflow(struct pt_regs *regs)

Fabio Massimo Di Nitto

author

2006-10-30 11:35 UTC #8 lore raw

Quoted 2 lines · Benjamin Herrenschmidt

Does that patch fixes it ?

tested with kernel .17 and headers from .19 and the build hangs. Still tons of Zl processes around.

On the note building with kernel .19 and .19 headers it all goes smooth.

Fabio

Signature · 1 line

I'm going to make him an offer he can't refuse.

Benjamin Herrenschmidt

2006-10-30 20:36 UTC #9 lore raw

Quoted 8 lines · Fabio Massimo Di Nitto

Quoted 2 lines · Benjamin Herrenschmidt
Does that patch fixes it ?
tested with kernel .17 and headers from .19 and the build hangs. Still tons of Zl processes around.
On the note building with kernel .19 and .19 headers it all goes smooth.

Ok, so there's a different issue from what I've found. You haven't by chance noted what those processes are (which test case typically) ? Also, there's a sysrq to get a backtrace of all pending processes, though I don't remember which one off the top of my mind, might be useful to have a look though.

Ben.

Fabio Massimo Di Nitto

author

2006-10-31 06:37 UTC #10 lore raw

Quoted 11 lines · Benjamin Herrenschmidt

Quoted 7 lines · Fabio Massimo Di Nitto
Quoted 2 lines · Benjamin Herrenschmidt
Does that patch fixes it ?
tested with kernel .17 and headers from .19 and the build hangs. Still tons of Zl processes around.
On the note building with kernel .19 and .19 headers it all goes smooth.
Ok, so there's a different issue from what I've found. You haven't by chance noted what those processes are (which test case typically) ?

I uploaded a build log and a ps ax output to
http://people.ubuntu.com/~fabbione/benh/

Quoted 3 lines

Also, there's a sysrq to get a backtrace of all pending processes, though I don't remember which one off the top of my mind, might be useful to have a look though.

I don't remember them either. Anybody?

Fabio

Signature · 1 line

I'm going to make him an offer he can't refuse.

Fabio Massimo Di Nitto

author

2006-10-31 06:51 UTC #11 lore raw

In reply to Benjamin Herrenschmidt

Quoted 9 lines · Benjamin Herrenschmidt

Quoted 7 lines · Fabio Massimo Di Nitto
Quoted 2 lines · Benjamin Herrenschmidt
Does that patch fixes it ?
tested with kernel .17 and headers from .19 and the build hangs. Still tons of Zl processes around.
On the note building with kernel .19 and .19 headers it all goes smooth.

Quoted 8 lines

Ok, so there's a different issue from what I've found. You haven't by chance noted what those processes are (which test case typically) ? Also, there's a sysrq to get a backtrace of all pending processes, though I don't remember which one off the top of my mind, might be useful to have a look though.
Ben.

Added also dmesg+sysrq.txt at the same url.

Fabio

Signature · 1 line

I'm going to make him an offer he can't refuse.

Fabio Massimo Di Nitto

author

2006-10-31 09:47 UTC #12 lore raw

In reply to Benjamin Herrenschmidt

Quoted 9 lines · Benjamin Herrenschmidt

Quoted 7 lines · Fabio Massimo Di Nitto
Quoted 2 lines · Benjamin Herrenschmidt
Does that patch fixes it ?
tested with kernel .17 and headers from .19 and the build hangs. Still tons of Zl processes around.
On the note building with kernel .19 and .19 headers it all goes smooth.

Quoted 8 lines

Ok, so there's a different issue from what I've found. You haven't by chance noted what those processes are (which test case typically) ? Also, there's a sysrq to get a backtrace of all pending processes, though I don't remember which one off the top of my mind, might be useful to have a look though.
Ben.

After discussing with Ben on IRC i applied 69588298188b40ed7f75c98a6fd328d82f23ca21 to kernel .17 and glibc does build without zombie processes and no hang whatsoever.

I suggest to push this patch back to the stable kernel trees.

Fabio

Signature · 1 line

I'm going to make him an offer he can't refuse.

Benjamin Herrenschmidt

2006-10-31 20:30 UTC #13 lore raw

Quoted 16 lines · Fabio Massimo Di Nitto

Quoted 15 lines · Benjamin Herrenschmidt
Quoted 7 lines · Fabio Massimo Di Nitto
Quoted 2 lines · Benjamin Herrenschmidt
Does that patch fixes it ?
tested with kernel .17 and headers from .19 and the build hangs. Still tons of Zl processes around.
On the note building with kernel .19 and .19 headers it all goes smooth.
Ok, so there's a different issue from what I've found. You haven't by chance noted what those processes are (which test case typically) ? Also, there's a sysrq to get a backtrace of all pending processes, though I don't remember which one off the top of my mind, might be useful to have a look though.

Quoted 7 lines

Quoted 2 lines
Ben.
After discussing with Ben on IRC i applied 69588298188b40ed7f75c98a6fd328d82f23ca21 to kernel .17 and glibc does build without zombie processes and no hang whatsoever.
I suggest to push this patch back to the stable kernel trees.

It also need the alignment bits I did though.

Ben.

Fabio Massimo Di Nitto

author

2006-10-31 20:44 UTC #14 lore raw

Quoted 18 lines · Benjamin Herrenschmidt

Quoted 17 lines · Fabio Massimo Di Nitto
Quoted 16 lines · Benjamin Herrenschmidt
Quoted 7 lines · Fabio Massimo Di Nitto
Quoted 2 lines · Benjamin Herrenschmidt
Does that patch fixes it ?
tested with kernel .17 and headers from .19 and the build hangs. Still tons of Zl processes around.
On the note building with kernel .19 and .19 headers it all goes smooth.
Ok, so there's a different issue from what I've found. You haven't by chance noted what those processes are (which test case typically) ? Also, there's a sysrq to get a backtrace of all pending processes, though I don't remember which one off the top of my mind, might be useful to have a look though.
Ben.

Quoted 9 lines

Quoted 4 lines
After discussing with Ben on IRC i applied 69588298188b40ed7f75c98a6fd328d82f23ca21 to kernel .17 and glibc does build without zombie processes and no hang whatsoever.
I suggest to push this patch back to the stable kernel trees.
It also need the alignment bits I did though.
Ben.

I did apply that one too to .17 as we agreed for the test.

Fabio

Signature · 1 line

I'm going to make him an offer he can't refuse.