Skip to contentSkip to conversation

Process with many NPTL threads terminates slowly on core dump signal

DISCUSSION Oleg Nesterov 2 participants 3 messages Oct 12, 2005 – Oct 13, 2005

lore
Share & tools
lore
#0 lore raw
Quoted 7 lines · Michael Kerrisk

Following up (belatedly) from my earlier message, I took Daniel Jacobowitz's suggestion to investigate the result from booting with "profile=2". When running my program (shwon below) on 2.6.14-rc4 to create 100 threads, and sending a core dump signal, the program takes 90 seconds to terminate, and readprofile shows the following:

I think the coredumping code in __group_complete_signal() is bogus and what happens is:

group_send_sig_info(P, SIGQUIT):

	adds SIGQUIT to ->shared_pending

	__group_complete_signal:

		->signal->group_exit_task = P;

		for_each_thread(t) {
			P->signal->group_stop_count ++;
			// sets TIF_SIGPENDING
			signal_wake_up(t)
		}

Now, P receives the signal:

get_signal_to_deliver:

	if (->signal->group_stop_count > 0) // YES
		handle_group_stop():
			if (->signal->group_exit_task == current) { // YES
				->signal->group_exit_task = NULL
				return 0;
			}

	signr = dequeue_signal(); // SIGQUIT

	do_coredump:

		->signal->flags = SIGNAL_GROUP_EXIT;

		coredump_wait:

			yield();

Now all other threads do:

get_signal_to_deliver:

	if (->signal->group_stop_count > 0) // YES
		handle_group_stop();
			if (->signal->flags & SIGNAL_GROUP_EXIT) // YES
				return 0;

	signr = dequeue_signal(); // no pending signals
		// recalc_sigpending_tsk() DOES NOT clear TIF_SIGPENDING,
		// because it sees ->group_stop_count != 0.
	
	return 0;

TIF_SIGPENDING is not cleared, so get_signal_to_deliver() will be called again on return to userspace. When all threads will eat their ->time_slice, P will return from yield() and kill all threads.

Could you try this patch (added to mm tree):

	http://marc.theaimsgroup.com/?l=linux-kernel&m=112887453531139

? It does not solve the whole problem, but may help.

Please report the result, if possible.

Oleg.

#1 lore raw
Quoted 14 lines

Von: Oleg Nesterov <[email protected]>
Datum: Wed, 12 Oct 2005 21:33:46 +0400

Quoted 7 lines · Michael Kerrisk

Following up (belatedly) from my earlier message, I took Daniel Jacobowitz's suggestion to investigate the result from booting with "profile=2". When running my program (shwon below) on 2.6.14-rc4 to create 100 threads, and sending a core dump signal, the program takes 90 seconds to terminate, and readprofile shows the following:

I think the coredumping code in __group_complete_signal() is bogus and what happens is:

[...]

Quoted 3 lines

TIF_SIGPENDING is not cleared, so get_signal_to_deliver() will be called again on return to userspace. When all threads will eat their ->time_slice, P will return from yield() and kill all threads.

Thanks for investiagting this further.

Quoted 5 lines

Could you try this patch (added to mm tree):

	http://marc.theaimsgroup.com/?l=linux-kernel&m=112887453531139

? It does not solve the whole problem, but may help.

Please report the result, if possible.

Thanks. I've applied it to 2.6.14-rc4: this patch does fix the specific behaviour that my program demonstrates.

What remains to be solved?

Cheers,

Michael

Signature · 2 lines
10 GB Mailbox, 100 FreeSMS/Monat http://www.gmx.net/de/go/topmail
+++ GMX - die erste Adresse für Mail, Message, More +++
#2 lore raw
Quoted 5 lines · Michael Kerrisk

Thanks. I've applied it to 2.6.14-rc4: this patch does fix the specific behaviour that my program demonstrates.

What remains to be solved?

The problem remains that all threads spin with TIF_SIGPENDING flag after the coredumping thread sets SIGNAL_GROUP_EXIT in do_coredump() and before it kills them in zap_threads(). If one of these threads is SCHED_FIFO we have a deadlock. Also, the coredumping thread could be preempted, placed in ->expired array ...

Oleg.