Questions about ptrace on a dying process

ptrace maintainers (and interested parties)...

I'm working on a crash handler for Linux, which uses ptrace to retrieve information about a process during it's coredump. Specifically, from within a core handler program (started within do_coredump() as a user_mode_helper), I would like to make ptrace calls against the dying process.

My problem is that the process state is not entering into TASK_TRACED, when I do an PTRACE_ATTACH against it. I have worked around the problem with the hack below, and am now trying to find a more correct solution to my problem:

@@ -122,6 +122,8 @@ int ptrace_check_attach(struct task_struct *child, int kill)
                WARN_ON_ONCE(task_is_stopped(child));
                if (task_is_traced(child) || kill)
                        ret = 0;
+               // FIXTHIS - force tracing support for crash handler
+               ret = 0;
                spin_unlock_irq(&child->sighand->siglock);
        }
        read_unlock(&tasklist_lock);

I'm trying to decipher the code, but I'm not sure if I understand it correctly.

Here's the problem I have found:

The code in ptrace_check_attach() tests whether the process state has __TASK_TRACED. (the above 'task_is_trace(child)' conditional). This appears to be the gateway for allowing ptrace operations on the process.

A process usually sets it's state to TASK_TRACED (which includes __TASK_TRACED) via the function ptrace_stop(), which is usually called (I believe), when a task processes a STOP signal. In the case of a dying process, however, it appears this is never called, since the process never actually returns through the signal-handling code on it's way back to user-space, since user-space never runs again. At least, I tried to send a signal from my crash_handler program to the dying process, and the dying process never processes the signal.

In ptrace_attach(), a stop signal *is* submitted to the process, but via a call to send_sig_info(SIGSTOP...), not by calling ptrace_stop(). This ends up adding the STOP signal to the pending bit array, but not converting the process to TASK_TRACED at that time.

The code in these codes paths looks quite tricky, and I am loathe to make any changes to the generic state machine for my case.

However, what would you think of having special case code in ptrace_attach() or ptrace_check_attach() code, for the in-coredump case?

Have you heard of other uses of ptrace on dying processes? Have other people gotten thts working successfully? That is, is there something simple I'm missing, in terms of manipulating the process state to allow for ptrace operation in this situation?

Thanks for any help or ideas you can provide.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation Senior Staff Engineer, Sony Network Entertainment
=============================

Which calls? just curious.

Yes, it can never do ptrace_stop() in do_coredump() paths.

Perhaps you can use PTRACE_O_TRACEEXIT. PTRACE_EVENT_EXIT will be reported after the coredumping. I think the core handler should close the pipe first, otherwise the dumping tracee will wait for the handler forever.

However. You need PTRACE_SEIZE, not PTRACE_ATTACH. And this can only work with the recent patch from Denys which allows to pass PTRACE_O_TRACEEXIT with PTRACE_SEIZE (currently in -mm tree).

Just in case, it could have other threads sleeping in TASK_UNINTERRUPTIBLE until do_coredump() completes. But these threads have already passed ptrace_event(PTRACE_EVENT_EXIT).

Oleg.

Right now, I'm using:
PTRACE_ATTACH, PTRACE_GETREGS,
PTRAGE_PEEKTEXT and PTRACE_GETSIGINFO

That's what I figured.

I'll look at these and see if they might work. I've determined that there are some funny races for accessing /proc, using ptrace, and reading the coredump from stdin (in the core pipe handler) associated with the technique I'm currently using. Maybe some of these other ptrace requests or options would help out.

Thanks very much for the suggestions.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation Senior Staff Engineer, Sony Network Entertainment
=============================

The standard approach is to define a core pipe handler and parse the elf memory dump.

-Andi

[email protected] -- Speaking for myself only

Yeah - I may be doing something new here. Android uses ptrace in debuggerd, which is their crash reporting tool, but they wake it up with signals before the dying program goes into coredump.

I'm taking a different approach and trying to do initiated by the coredump feature in Linux. This makes it so that a process does not need to be persistently running to capture these events.

This is on embedded systems, where the dump is not saved. The dump is available via stdin to the core pipe handler, but it would be kind of a pain to wrapper that for random access, which is needed for stuff like stack unwinding.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation Senior Staff Engineer, Sony Network Entertainment
=============================

I think ptrace API does not provide guarantees that it is possible to attach to the process when it coredumps.

It might work in current kernels, but might break in new ones.

Stack unwinding only requires the stack data and knowledge
of the mapped binary and library files. You can parse coredump's ELF header, and skip all sizable data segments which you won't need anyway.

I estimate that usually you will need to save only ~150k of data in order to produce a stacktrace, and even then,
only because Linux pre-allocates ridiculously large stack for every new process - 132k. It can easily be reduced to something saner with one-line patch.

vda

My budget for each crash report is about 8k. I have to do the unwind at the time of the crash (on target) (and without symbols - these are added later on a host). Given the other stuff I want to save, saving the whole stack is usually not an option, and saving a coredump is out of the question.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation Senior Staff Engineer, Sony Network Entertainment
=============================

How about this algorithm?

Read coredump sequentially. First come ELF header and program headers. Read headers and remember their virtual address ranges, sizes, and flags. The rest you don't need.

Then comes NOTE segment. Parse it and fetch the value of stack pointer register. Don't save anything else.

Then read LOAD segments and discard their data until you reach a stack segment (one which stack pointer points to).

Read all words from stack starting from stack pointer up to the top of the stack. If a word, when interpreted as a pointer, points into an executable segment, then add it to stack trace. Else, ignore the word.

After you finished reading the stack, read and discard the rest (there usually is nothing after stack).

Then you can sanitize stack trace: if some addresses there point to a *beginning* of a function, not in the middle of it,
then it's likely a false positive (someone was passing function pointer parameter, or had it in on-stack auto variable).
To do this sanitization, you don't need (and can't use, even if you'd have it) coredump - you need to examine binary and libraries instead.

But I suspect even without sanitizing it, resulting stack trace will often be rather good: it will contain all real return addresses, and often will have no false positives at all.

vda

If it's not too much trouble, it would be nice to continue
this behaviour. Admittedly, my style of crash handling appears
to be new, and I don't want to unnecessarily burden the code, but so far it works really well, and currently requires very minimal change to the existing ptrace code. One thing that's nice about what I'm doing, is that I don't rely on the whole signal state machine of the process to interact with it (since a dying process can't respond correctly).

So hopefully, continuing to support ptrace for a dying process won't interfere or burden the existing (rather complex) state processing in the current code.

Just for reference, below is the patch I settled on for my own kernel. I'm planning taking a look at the PTRACE_SEIZE code to see if it accomplishes what I need, but haven't done that yet.

I do have a question, though - how will a tracer know that it can use PTRACE_SEIZE? Is there some introspection API? My code will be running mainly against a patched 3.0 for some time (a few years), but if I could make it interoperate with a kernel that supports PTRACE_SEIZE (when it is mainlined), that would be great.

-- Tim

commit dd54b901759428e60ed57b2d6cb77d25a8db767f

Author: tbird <[email protected]>
Date:   Tue Feb 28 14:13:08 2012 -0800

Support ptrace_attach with no signal side-effects.

In the normal case, a ptrace_attach operation will convert a process to TASK_TRACED by sending it a SIGSTOP signal, after setting task->ptrace. This won't work on a dying process because during do_coredump(), the dying process won't process the STOP signal and change state.

Modify ptrace_attach() so that the tracee task state is modified directly. This allows subsequent ptrace_check_attach() calls to work correctly, and avoids having a pending SIGSTOP signal on the tree process (which interferes with waiting for the core pipe handler).

Note that a more full-featured implementation of this is in the works (as of March, 2012) by Tejun Heo, called PTRACE_SEIZE. Once that gets mainlined, this patch may not be needed, or might need to be reworked.

index 26147d1..9c7bf8e 100644
@@ -223,7 +223,16 @@ static int ptrace_attach(struct task_struct *task)
 		task->ptrace |= PT_PTRACE_CAP;
 
 	__ptrace_link(task, current);
-	send_sig_info(SIGSTOP, SEND_SIG_FORCED, task);
+	
+	/*
+ 	 * If doing coredump, just convert directly to TASK_TRACED.
+ 	 * A dying process doesn't process signals normally.
+ 	 */
+	if (unlikely(task->mm->core_state)) {
+		set_task_state(task, TASK_TRACED);
+	} else {
+		send_sig_info(SIGSTOP, SEND_SIG_FORCED, task);
+	}
 
 	spin_lock(&task->sighand->siglock);
 

=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation Senior Staff Engineer, Sony Network Entertainment
=============================

You added yet another quirk to ptrace API - and this API already has no shortage of quirks.
Of course you can maintain it for your kernel, but adding it in mainline is a bizarre proposition.

I'm not even sure that _if_ PTRACE_SEIZE with PTRACE_O_TRACEEXIT option works on semi-dead core-dumping process today without patching, it is guaranteed to do so in the future.

By looking at kernel version.

vda