Skip to contentSkip to conversation

Zombie process when ptracing

DISCUSSION Nick Piggin 2 participants 4 messages Nov 19, 2009 – Nov 23, 2009

lore
Share & tools
lore
#0 lore raw

Hi,

Running recent git kernel, I have a process stuck in Z state

bash ? 0000000000000000 0 3188 3187 0x00000000
ffff88012e24fec8 0000000000000046 0000000000000000 0000000000000012 ffff88012e24fec8 ffff88012e24e000 ffff88012e24ffd8 ffff88012e24e000 000000000000efc8 ffff88012e24e000 ffff88012ea82090 ffff88012ff78640

Call Trace:
 [<ffffffff8124baee>] ? proc_clear_tty+0x5e/0x70
 [<ffffffff810587a8>] ? exit_ptrace+0xb8/0x140
 [<ffffffff8105126a>] do_exit+0x58a/0x7c0
 [<ffffffff810514dd>] do_group_exit+0x3d/0xb0
 [<ffffffff81051562>] sys_exit_group+0x12/0x20
 [<ffffffff8100b3eb>] system_call_fastpath+0x16/0x1b

This was after stracing a few test programs.

It also seems to have lost job control (^C) at the same time.

Hmm, and the kernel just paniced with an nmi lockup while I was trying to get more info.

Call Trace:
 <IRQ>
 [<ffffffff811e5aa3>] __const_udelay+0x43/0x50
 [<ffffffff810261bc>] arch_trigger_all_cpu_backtrace+0x4c/0x70
 [<ffffffff81264f79>] sysrq_handle_showallcpus+0x9/0x10
 [<ffffffff81264d10>] __handle_sysrq+0x120/0x180
 [<ffffffff81264de6>] handle_sysrq+0x26/0x30
 [<ffffffff81275af0>] serial8250_handle_port+0x210/0x2f0
 [<ffffffff81275c58>] serial8250_interrupt+0x88/0x120
 [<ffffffff810872e7>] handle_IRQ_event+0xa7/0x1e0
 [<ffffffff810891cc>] handle_edge_irq+0xbc/0x150
 [<ffffffff8100e2df>] handle_irq+0x1f/0x30
 [<ffffffff8100d86a>] do_IRQ+0x6a/0xe0
 [<ffffffff8100bc93>] ret_from_intr+0x0/0xa
 <EOI>
 [<ffffffff81013842>] ? default_idle+0xa2/0xc0
 [<ffffffff8106f8d1>] ? __atomic_notifier_call_chain+0x31/0x60
 [<ffffffff81013b1a>] ? c1e_idle+0x3a/0x100
 [<ffffffff8106f911>] ? atomic_notifier_call_chain+0x11/0x20
 [<ffffffff8100a66b>] ? cpu_idle+0x6b/0xc0
 [<ffffffff81439dc6>] ? start_secondary+0x17c/0x1d6

I'll update this space if I can repeat it again. Any ideas?

#1 lore raw

Hi,

Quoted 15 lines · Nick Piggin

Running recent git kernel, I have a process stuck in Z state

bash ? 0000000000000000 0 3188 3187 0x00000000
ffff88012e24fec8 0000000000000046 0000000000000000 0000000000000012 ffff88012e24fec8 ffff88012e24e000 ffff88012e24ffd8 ffff88012e24e000 000000000000efc8 ffff88012e24e000 ffff88012ea82090 ffff88012ff78640

Call Trace:
 [<ffffffff8124baee>] ? proc_clear_tty+0x5e/0x70
 [<ffffffff810587a8>] ? exit_ptrace+0xb8/0x140
 [<ffffffff8105126a>] do_exit+0x58a/0x7c0
 [<ffffffff810514dd>] do_group_exit+0x3d/0xb0
 [<ffffffff81051562>] sys_exit_group+0x12/0x20
 [<ffffffff8100b3eb>] system_call_fastpath+0x16/0x1b
Quoted 3 lines

This was after stracing a few test programs.

It also seems to have lost job control (^C) at the same time.

This can happen if the tracer (strace) itself hangs, zombies should go away once the tracer is killed. Or its ->real_parent is stopped or hangs...

(I assume you didn't strace /sbin/init)

But,

Quoted 2 lines

Hmm, and the kernel just paniced with an nmi lockup while I was trying to get more info.

this probably means we have a kernel bug ;)

If you see a zombie again, could you look at its /ptoc/pid/status?

And of course, which programs did you trace and how? It would be great if we can reproduce the problem.

Oleg.

#2 lore raw
Quoted 20 lines · Oleg Nesterov

Hi,

Quoted 17 lines · Nick Piggin

Running recent git kernel, I have a process stuck in Z state

bash ? 0000000000000000 0 3188 3187 0x00000000
ffff88012e24fec8 0000000000000046 0000000000000000 0000000000000012 ffff88012e24fec8 ffff88012e24e000 ffff88012e24ffd8 ffff88012e24e000 000000000000efc8 ffff88012e24e000 ffff88012ea82090 ffff88012ff78640

Call Trace:
 [<ffffffff8124baee>] ? proc_clear_tty+0x5e/0x70
 [<ffffffff810587a8>] ? exit_ptrace+0xb8/0x140
 [<ffffffff8105126a>] do_exit+0x58a/0x7c0
 [<ffffffff810514dd>] do_group_exit+0x3d/0xb0
 [<ffffffff81051562>] sys_exit_group+0x12/0x20
 [<ffffffff8100b3eb>] system_call_fastpath+0x16/0x1b

This was after stracing a few test programs.

Quoted 7 lines
Quoted 1 line

It also seems to have lost job control (^C) at the same time.

This can happen if the tracer (strace) itself hangs, zombies should go away once the tracer is killed. Or its ->real_parent is stopped or hangs...

(I assume you didn't strace /sbin/init)

No, I straced something else, and all straces seemed to be killed but bash remained. I was running a script that in turn launched another process, so I ran it via strace -ff bash ./script.sh

Quoted 6 lines

But,

Quoted 2 lines

Hmm, and the kernel just paniced with an nmi lockup while I was trying to get more info.

this probably means we have a kernel bug ;)

Hmm sorry that seemed like it _may_ have been an unrelated issue (with the ssh connection).

Quoted 1 line

If you see a zombie again, could you look at its /ptoc/pid/status?

OK, any other hints if I see it again?

Quoted 2 lines

And of course, which programs did you trace and how? It would be great if we can reproduce the problem.

At this stage I have not reproduced it, and I can't share the program which was being straced. If it does happen again and I cannot distil a simple test case, I will ask permission to distribute it.

Thanks,
Nick

#3 lore raw
Quoted 28 lines · Nick Piggin
Quoted 26 lines · Oleg Nesterov

Hi,

Quoted 15 lines · Nick Piggin

Running recent git kernel, I have a process stuck in Z state

bash ? 0000000000000000 0 3188 3187 0x00000000
ffff88012e24fec8 0000000000000046 0000000000000000 0000000000000012 ffff88012e24fec8 ffff88012e24e000 ffff88012e24ffd8 ffff88012e24e000 000000000000efc8 ffff88012e24e000 ffff88012ea82090 ffff88012ff78640

Call Trace:
 [<ffffffff8124baee>] ? proc_clear_tty+0x5e/0x70
 [<ffffffff810587a8>] ? exit_ptrace+0xb8/0x140
 [<ffffffff8105126a>] do_exit+0x58a/0x7c0
 [<ffffffff810514dd>] do_group_exit+0x3d/0xb0
 [<ffffffff81051562>] sys_exit_group+0x12/0x20
 [<ffffffff8100b3eb>] system_call_fastpath+0x16/0x1b
Quoted 3 lines

This was after stracing a few test programs.

It also seems to have lost job control (^C) at the same time.

This can happen if the tracer (strace) itself hangs, zombies should go away once the tracer is killed. Or its ->real_parent is stopped or hangs...

Quoted 6 lines
Quoted 1 line

(I assume you didn't strace /sbin/init)

No, I straced something else, and all straces seemed to be killed but bash remained. I was running a script that in turn launched another process, so I ran it via strace -ff bash ./script.sh

OK, thanks.

Hmm. Just noticed the state above == '?'. Looks like sched_show_task() is buggy, it should check ->exit_state for "ZX" from TASK_STATE_TO_CHAR_STR. But this is off-topic.

Quoted 3 lines
Quoted 1 line

If you see a zombie again, could you look at its /ptoc/pid/status?

OK, any other hints if I see it again?

Well, also the contents of /proc/PPid/status and /proc/TracerPid/status may help. And sysrq-t ouput. Otherwise, currently I have no idea where to start.

Oleg.