ptrace.2: BUGS (missing WIFEXITED notification)

[resend with Cc: linux-man]

Hi again :-)

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the "BUGS" section:

"If a thread group leader is traced and exits by calling _exit(2), a PTRACE_EVENT_EXIT stop will happen for it (if requested), but the subsequent WIFEXITED notification will not be delivered until all other threads exit. As explained above, if one of other threads calls execve(2), the death of the thread group leader will never be reported. If the execed thread is not traced by this tracer, the tracer will never know that execve(2) happened. One possible workaround is to PTRACE_DETACH the thread group leader instead of restarting it in this case. Last confirmed on 2.6.38.6."

I wanted to write that we've also noticed the same thing not only for _exit() but also for terminating signals, however we also came across this bit in the manual source:

.\" Note from Denys Vlasenko:
.\" Here "exits" means any kind of death - _exit, exit_group, .\" signal death. Signal death and exit_group cases are trivial, .\" though: since signal death and exit_group kill all other threads .\" too, "until all other threads exit" thing happens rather soon .\" in these cases. Therefore, only _exit presents observably .\" puzzling behavior to ptrace users: thread leader _exit's, .\" but WIFEXITED isn't reported! We are trying to explain here .\" why it is so.

There is a difference, however -- this behaviour can also be observed for the other types of death if you are currently tracing the other threads too!

In other words, when multiple threads are being traced and the group leader exits, waitpid() on this group leader will hang indefinitely (because the other threads won't exit until we wait for and CONT/DETACH them, and we don't receive the exit notification for the group leader until the other threads have really exited).

To me, this means that not only _exit() but also other types of death present "observably puzzling behavior to ptrace users".

I'd propose the following changes:

1) include some (if not all) of Denys's explanation in the actual text:

(quoted diff) +1 -1

-If a thread group leader is traced and exits by calling _exit(2)...
+If a thread group leader is traced and exits for any reason (_exit,

exit_group, signal death, etc.), ...

2) include the bits about tracing other threads:

+If the other threads in the thread group are being traced, they will not exit until they have been either waited for and restarted or detached, thereby blocking the exit notification (WIFEXITED) of the group leader to wait()/waitpid().

3) there's a typo in the original text:

(quoted diff) +1 -1

-one of other threads
+one of the other threads

Feel free to rephrase any of the above.

Thoughts? We can also provide more details, including a reproducer, or clarification if needed.

(PS: Please also credit Quentin Casasnovas with the report as we've both spent more than a few hours tracking this down!)

Vegard

Hi Denys,

Do you have any thoughts on the below?

Quoted 5 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the "BUGS" section:

"If a thread group leader is traced and exits by calling _exit(2), a

Quoted 13 lines

PTRACE_EVENT_EXIT stop will happen for it (if requested), but the subsequent WIFEXITED notification will not be delivered until all other threads exit. As explained above, if one of other threads calls execve(2), the death of the thread group leader will never be reported. If the execed thread is not traced by this tracer, the tracer will never know that execve(2) happened. One possible workaround is to PTRACE_DETACH the thread group leader instead of restarting it in this case. Last confirmed on 2.6.38.6."

I wanted to write that we've also noticed the same thing not only for _exit() but also for terminating signals, however we also came across this bit in the manual source:

Quoted 9 lines

.\" Note from Denys Vlasenko:
.\" Here "exits" means any kind of death - _exit, exit_group, .\" signal death. Signal death and exit_group cases are trivial, .\" though: since signal death and exit_group kill all other threads .\" too, "until all other threads exit" thing happens rather soon .\" in these cases. Therefore, only _exit presents observably .\" puzzling behavior to ptrace users: thread leader _exit's, .\" but WIFEXITED isn't reported! We are trying to explain here .\" why it is so.

Quoted 3 lines · Michael Kerrisk (man-pages)

Hi Denys,

Do you have any thoughts on the below?

Yes, the poster is right: this part needs fixing, the behavior is the same on any kind of process termination.

Quoted 1 line · Vegard Nossum

Quoted 5 lines

Quoted 5 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the "BUGS" section:

"If a thread group leader is traced and exits by calling _exit(2), a

I think a possible fix is just to replace "exits by calling _exit(2)" part of the above text with "terminates".

Quoted 14 lines · Denys Vlasenko

Quoted 3 lines · Michael Kerrisk (man-pages)

Hi Denys,

Do you have any thoughts on the below?

Yes, the poster is right: this part needs fixing, the behavior is the same on any kind of process termination.

Quoted 5 lines · Vegard Nossum

Quoted 4 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the "BUGS" section:

Quoted 5 lines

Quoted 1 line

Quoted 1 line

"If a thread group leader is traced and exits by calling _exit(2), a

I think a possible fix is just to replace "exits by calling _exit(2)" part of the above text with "terminates".

Should we also add a little paragraph detailing that waitpid() would hang indefinitely if one thread terminates while the others are in ptrace-stop?

The only way for us to work-around this behaviour was to have PTRACE_O_TRACEEXIT and detach from the threads when we were notified of a thread termination, without PTRACE_O_TRACEEXIT I don't think one can be notified in way, leading to hangs almost certainly when tracing multiple threads and some are in ptrace-stop.

Or maybe that's obvious from the following sentence:

"If a thread group leader is traced and exits by calling _exit(2), a PTRACE_EVENT_EXIT stop will happen for it (if requested), but the subsequent WIFEXITED notification will not be delivered until all other threads exit."

Quoted 2 lines · Quentin Casasnovas

Should we also add a little paragraph detailing that waitpid() would hang indefinitely if one thread terminates while the others are in ptrace-stop?

It implies this by saying "but the subsequent WIFEXITED notification will not be delivered until all other threads exit".

If another thread is in ptrace-stop, it did not exit yet. Therefore, WIFEXITED notification to the thread group leader will not be delivered. Therefore, waitpid() on it would hang.

Quoted 5 lines

The only way for us to work-around this behaviour was to have PTRACE_O_TRACEEXIT and detach from the threads when we were notified of a thread termination, without PTRACE_O_TRACEEXIT I don't think one can be notified in way, leading to hangs almost certainly when tracing multiple threads and some are in ptrace-stop.

Why do you do that waitpid on thread group leader?

strace is the user of ptrace, and it has no problem with this behavior. It simply waits for all tracees via waitpid(-1). Whichever sends the next notification is processed.

Quoted 6 lines

Quoted 5 lines

The only way for us to work-around this behaviour was to have PTRACE_O_TRACEEXIT and detach from the threads when we were notified of a thread termination, without PTRACE_O_TRACEEXIT I don't think one can be notified in way, leading to hangs almost certainly when tracing multiple threads and some are in ptrace-stop.

Quoted 6 lines

Why do you do that waitpid on thread group leader?

strace is the user of ptrace, and it has no problem with this behavior. It simply waits for all tracees via waitpid(-1). Whichever sends the next notification is processed.

That's because strace has all threads running; in our use-case, we attach to all threads of a group and restart a single thread to do some work. In this particular case, waitpid(-1, ...) does not help because to exit, the other threads need to be PTRACE_CONT'ed.

Quoted 13 lines · Quentin Casasnovas

Quoted 6 lines

Quoted 5 lines

The only way for us to work-around this behaviour was to have PTRACE_O_TRACEEXIT and detach from the threads when we were notified of a thread termination, without PTRACE_O_TRACEEXIT I don't think one can be notified in way, leading to hangs almost certainly when tracing multiple threads and some are in ptrace-stop.

Quoted 6 lines

Why do you do that waitpid on thread group leader?

strace is the user of ptrace, and it has no problem with this behavior. It simply waits for all tracees via waitpid(-1). Whichever sends the next notification is processed.

Quoted 4 lines

That's because strace has all threads running; in our use-case, we attach to all threads of a group and restart a single thread to do some work. In this particular case, waitpid(-1, ...) does not help because to exit, the other threads need to be PTRACE_CONT'ed.

Can you be more specific?
IF you have thread(s) which "do some work", then waitpid(-1) would work for you - presumably, you wait for them to finish doing work, or otherwise terminate, get a signal, whatever?

Quoted 15 lines · Quentin Casasnovas

Quoted 14 lines · Denys Vlasenko

Quoted 3 lines · Michael Kerrisk (man-pages)

Hi Denys,

Do you have any thoughts on the below?

Yes, the poster is right: this part needs fixing, the behavior is the same on any kind of process termination.

Quoted 5 lines · Vegard Nossum

Quoted 4 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the "BUGS" section:

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

Quoted 1 line

"If a thread group leader is traced and exits by calling _exit(2), a

I think a possible fix is just to replace "exits by calling _exit(2)" part of the above text with "terminates".

Should we also add a little paragraph detailing that waitpid() would hang indefinitely if one thread terminates while the others are in ptrace-stop?

It implies this by saying "but the subsequent WIFEXITED notification will not be delivered until all other threads exit".

If another thread is in ptrace-stop, it did not exit yet. Therefore, WIFEXITED notification to the thread group leader will not be delivered. Therefore, waitpid() on it would hang.

Quoted 36 lines · Denys Vlasenko

Quoted 24 lines · Quentin Casasnovas

Quoted 22 lines · Denys Vlasenko

Quoted 4 lines · Michael Kerrisk (man-pages)

Hi Denys,

Do you have any thoughts on the below?

Yes, the poster is right: this part needs fixing, the behavior is the same on any kind of process termination.

Quoted 8 lines · Vegard Nossum

Quoted 7 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the
"BUGS" section:

"If a thread group leader is traced and exits by calling _exit(2), a

Quoted 8 lines

Quoted 4 lines

I think a possible fix is just to replace "exits by calling _exit(2)" part
of the above text with "terminates".

Should we also add a little paragraph detailing that waitpid() would hang indefinitely if one thread terminates while the others are in ptrace-stop?

Quoted 6 lines

It implies this by saying "but the subsequent WIFEXITED notification will not be delivered until all other threads exit".

If another thread is in ptrace-stop, it did not exit yet. Therefore, WIFEXITED notification to the thread group leader will not be delivered. Therefore, waitpid() on it would hang.

While I agree that the information in the current man page is strictly speaking sufficient, I personally still think it would be an improvement to mention it explicitly (i.e. my proposed change #2 in the original e-mail). Just because I think it's a sort of non-obvious pitfall; out of hand, you don't expect a call to waitpid() on a process that has exited to hang. That's just my opinion, though.

Hi Vegard,

Quoted 45 lines · Vegard Nossum

Quoted 36 lines · Denys Vlasenko

Quoted 24 lines · Quentin Casasnovas

Quoted 22 lines · Denys Vlasenko

Quoted 4 lines · Michael Kerrisk (man-pages)

Hi Denys,

Do you have any thoughts on the below?

Yes, the poster is right: this part needs fixing, the behavior is the same on any kind of process termination.

Quoted 8 lines · Vegard Nossum

Quoted 7 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the
"BUGS" section:

"If a thread group leader is traced and exits by calling _exit(2), a

Quoted 8 lines

Quoted 4 lines

I think a possible fix is just to replace "exits by calling _exit(2)" part
of the above text with "terminates".

Should we also add a little paragraph detailing that waitpid() would hang indefinitely if one thread terminates while the others are in ptrace-stop?

Quoted 6 lines

It implies this by saying "but the subsequent WIFEXITED notification will not be delivered until all other threads exit".

If another thread is in ptrace-stop, it did not exit yet. Therefore, WIFEXITED notification to the thread group leader will not be delivered. Therefore, waitpid() on it would hang.

Quoted 6 lines

While I agree that the information in the current man page is strictly speaking sufficient, I personally still think it would be an improvement to mention it explicitly (i.e. my proposed change #2 in the original e-mail). Just because I think it's a sort of non-obvious pitfall; out of hand, you don't expect a call to waitpid() on a process that has exited to hang. That's just my opinion, though.

That sounds okay to me. Would you and/or Quentin be willing to put together a patch to the man page?

Thanks,

Michael

Vegard (and Quentin): Ping!

Quoted 47 lines · Michael Kerrisk (man-pages)

Quoted 36 lines · Vegard Nossum

Quoted 28 lines · Denys Vlasenko

Quoted 19 lines · Quentin Casasnovas

Quoted 18 lines · Denys Vlasenko

Quoted 4 lines · Michael Kerrisk (man-pages)

Hi Denys,

Do you have any thoughts on the below?

Yes, the poster is right: this part needs fixing, the behavior is the same on any kind of process termination.

Quoted 8 lines · Vegard Nossum

Quoted 7 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the
"BUGS" section:

"If a thread group leader is traced and exits by calling _exit(2), a

Quoted 7 lines

Quoted 3 lines

I think a possible fix is just to replace "exits by calling _exit(2)" part
of the above text with "terminates".

Should we also add a little paragraph detailing that waitpid() would hang indefinitely if one thread terminates while the others are in ptrace-stop?

Quoted 6 lines

It implies this by saying "but the subsequent WIFEXITED notification will not be delivered until all other threads exit".

If another thread is in ptrace-stop, it did not exit yet. Therefore, WIFEXITED notification to the thread group leader will not be delivered. Therefore, waitpid() on it would hang.

Quoted 6 lines

While I agree that the information in the current man page is strictly speaking sufficient, I personally still think it would be an improvement to mention it explicitly (i.e. my proposed change #2 in the original e-mail). Just because I think it's a sort of non-obvious pitfall; out of hand, you don't expect a call to waitpid() on a process that has exited to hang. That's just my opinion, though.

That sounds okay to me. Would you and/or Quentin be willing to put together a patch to the man page?

From 91798cbc674787cf31b66e8cf52557495e659665 Mon Sep 17 00:00:00 2001

(including context for old thread)

Quoted 47 lines · Michael Kerrisk (man-pages)

Vegard (and Quentin): Ping!

Quoted 44 lines · Michael Kerrisk (man-pages)

Quoted 42 lines · Vegard Nossum

Quoted 28 lines · Denys Vlasenko

Quoted 19 lines · Quentin Casasnovas

Quoted 18 lines · Denys Vlasenko

Quoted 4 lines · Michael Kerrisk (man-pages)

Hi Denys,

Do you have any thoughts on the below?

Yes, the poster is right: this part needs fixing, the behavior is the same on any kind of process termination.

Quoted 8 lines · Vegard Nossum

Quoted 7 lines

We hit another edge case in the ptrace() interface and after several hours of chasing it down, we found that it was already described in the
"BUGS" section:

"If a thread group leader is traced and exits by calling _exit(2), a

Quoted 7 lines

Quoted 3 lines

I think a possible fix is just to replace "exits by calling _exit(2)" part
of the above text with "terminates".

Should we also add a little paragraph detailing that waitpid() would hang indefinitely if one thread terminates while the others are in ptrace-stop?

Quoted 6 lines

It implies this by saying "but the subsequent WIFEXITED notification will not be delivered until all other threads exit".

If another thread is in ptrace-stop, it did not exit yet. Therefore, WIFEXITED notification to the thread group leader will not be delivered. Therefore, waitpid() on it would hang.

While I agree that the information in the current man page is strictly speaking sufficient, I personally still think it would be an improvement to mention it explicitly (i.e. my proposed change #2 in the original e-mail). Just because I think it's a sort of non-obvious pitfall; out of hand, you don't expect a call to waitpid() on a process that has exited to hang. That's just my opinion, though.

Quoted 3 lines

Quoted 3 lines

That sounds okay to me. Would you and/or Quentin be willing to put together a patch to the man page?

Hi,

Apologies for the delay. Here's a new patch, feel free to munge the wording if you think it can be improved.

On 08/12/2016 04:50 PM, Vegard Nossum wrote:
stop will happen for it (if requested), but the subsequent WIFEXITED notification will not be delivered until all other threads exit.

ptrace.2 +4 -0

+(As a corollary, if the other threads in the thread group are being
+traced, they will not exit until they have been either waited for
+and restarted or detached, thereby blocking the exit notification
+of the group leader to wait(2) and waitpid(2).)

Are you trying to prevent others falling into the same trap you fell into:
thinking that if you see PTRACE_EVENT_EXIT on a thread, it has exited?

Quoted 11 lines · Denys Vlasenko

On 08/12/2016 04:50 PM, Vegard Nossum wrote:
stop will happen for it (if requested), but the subsequent WIFEXITED notification will not be delivered until all other threads exit.

ptrace.2 +4 -0

+(As a corollary, if the other threads in the thread group are being
+traced, they will not exit until they have been either waited for
+and restarted or detached, thereby blocking the exit notification
+of the group leader to wait(2) and waitpid(2).)

Are you trying to prevent others falling into the same trap you fell into:
thinking that if you see PTRACE_EVENT_EXIT on a thread, it has exited?

Yes, or not exactly; more precisely, the trap we fell into was thinking that if we see PTRACE_EVENT_EXIT, that means waitpid() will return immediately. I know, the distinction is kind of subtle...

Review

Quoted 12 lines

I think your wording is not good at that. How about this?

stop will happen for it (if requested), but the subsequent WIFEXITED notification will not be delivered until all other threads exit.

(quoted diff) +7 -0

+(Note that other threads, if they are traced and are in
+PTRACE_EVENT_EXIT ptrace-stop, they did not exit yet.
+If you assume otherwise and wait on a thread leader to exit now,
+this may hang.
+You need to restart or detach them for them to exit. Generally,
+only seeing WIFEXITED or WIFSIGNALED status from the thread is
+a definite sign that it exited).

Alright, another iteration on your version gives the following:

"""
Keep in mind that other threads in the thread group which are in a PTRACE_EVENT_EXIT ptrace-stop have not actually exited yet. This means that attempting to wait for the group leader with waitpid() will hang indefinitely. In order prevent this, you must make sure that the other threads exit properly by either restarting them or detaching from them. In general, only WIFEXITED and WIFSIGNALED indicate that the child has definitely exited.
"""