Multithreaded core dumps

Hi,

First post (although lurking on and off since 0.99pl14 or so), so please be gentle.

We recently convinced my company to move from VxWorks to (embedded) Linux. For better or worse, our application is heavily multithreaded. It seems that current versions of Linux dump single-threaded core. I've done a bit of research and it seems that:

- GDB is more than happy to load multiple register sets per core file.

- There are patches to the kernel dump multiple core files, one per LWP. This is not really helpful in my case, since we're on an embedded platform with little Flash to store cores. Besides, loading up gdb on 10-20 core files looking for the one that had the problem doesn't sound very fun, as opposed to saying "info threads".

Because I am (not so) young and (very) foolish, it doesn't sound that hard, at first (and uneducated) glance, to dump register sets for all related LWPs to a single core file, even in a portable way across x86 and PPC architectures. Under SMP, it might be a bit trickier, but Solaris manages to do it, so it can't be impossible, and capturing any state for each thread would seem better than nothing at all, since all the LWPs are about to die anyways. I'm considering using some of my (copious) spare time and trying to patch the kernel to do this. I have a few questions:

0. Am I wrong or confused about the state of postmortem multithreaded debugging under Linux?

1. Is anyone else working on this? If not, why not? Am I missing something?

2. If this is simply something that nobody is working on because other things are more interesting, can anybody give me a few pointers on where to start?

Best regards,

-elan
Aspiring Kernel Hacker

At least as of mid-2.2 series this was certainly my experience. It was very frustrating that the thread/process that dumped core was not the one that dereferenced a bad pointer/failed an assert but the process group leader.

I am inclined to believe that that is the case. Unfortunately, I have no advice to give - but I am writing because I think that it would be neat if you have the time and the inclination for you to document your findings as you progress and put them on the web.

-Kip

The 2.4-ac tree supports dumping core.$pid for the threads that actually died

... and these dumps are not reliable. Living thread may modify the contents of dump as it's being written out. I.e. you are getting false alarms - inconsistent data that was never there.

Think of a linked list protected by a mutex. Half of its entries are already written out. Surviving thread removes an element. It updates the in-core structures correctly. The problem being, in the dump we get part of memory from before that change and part - after. If you notice that when you are looking at the dump - welcome to a nice chase after the bug that never existed.

That is mathematically insoluble. Think about an SMP system, you cannot stop the other thread in instantaneously small time

Alan

From: Alan Cox <[email protected]>
Date: Thu, 30 Aug 2001 10:12:49 +0100 (BST)

That is mathematically insoluble. Think about an SMP system, you cannot stop the other thread in instantaneously small time

If you mean "at the time the user process trapped", pretty much this is true.

But it _IS_ possible to capture the other threads with an SMP cross call at the time the bad condition leading to the dump is detected.

	thread_dont_schedule(t);
	smp_reschedule();
	t->dump_core(t);
	thread_re_schedule(t);

Later,
David S. Miller
[email protected]

That's a separate problem - we can't catch the exact state of process at the moment of death, indeed, but in theory we could make sure that no changes happen while we write the thing out. As it is, coredump is not only taken at some point after death, it doesn't correspond to state of the thing at _any_ moment.

Annyoing isn't it....

I just browsed thru the code, got curious myself since Linux has a very flexible way of specifying the degree of sharing. If two procs share VM, it is not necessarily said that both shall exit if one do.

THere is a CLONE_THREAD flag to clone() that sets up a linked list thru all the procs (shared VM or not) that in 2.4.3 that I currently run don't seem to be ready for general use, managed to get this:

te       31504 31504  0 10:03 pts/0    00:00:00 [clone2 <defunct>]
te       31505 31504  0 10:03 pts/0    00:00:00 [clone2 <defunct>]

Where a zombie is waiting for the parent to receive it's SIGCHLD, but it's its own parent. Kinda cute, guess Its time to reboot....

I don't know what kind of behavour it's supposed to have, but if it's intended to to mark all the clones to die with the coredumping proc, the data structures are properly in place, and all do_coredump need to do is to follow the linked list (thread_group), prempth if SMP, and read of the regs. (plus all the pitfall I don't see of course).

A COMPLETELY different matter is weither the elf format support multiple registers sets.... look it up, (I'm not...)

TJ

Den 30 Aug 2001 00:21:06 -0500, skrev Elan Feingold:

_________________________________________________________________________

Terje Eggestad                  [email protected]
Scali Scalable Linux Systems    http://www.scali.com

Olaf Helsets Vei 6              tel:    +47 22 62 89 61 (OFFICE)
P.O.Box 70 Bogerud                      +47 975 31 574  (MOBILE)
N-0621 Oslo                     fax:    +47 22 62 89 51
NORWAY            
_________________________________________________________________________

I'm pretty sure Linus had a patch for this in 2.4.7 or so, which reparented the thread to init, so it would be reaped on exit.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert