Skip to contentSkip to conversation

proposed fix for ptrace() SMP race

PATCH In review David Mosberger 3 participants 11 messages Sep 6, 2001 – Sep 10, 2001 ptrace support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 proposed fix for ptrace() SMP race

There is currently a nasty race condition in ptrace(). The effect of this varies from one platform to another but, for example, on ia64, it could have the effect of corrupting the state of register f32-f127. The problem is that ptrace() uses the expression (child->state == TASK_STOPPED) to determine whether or not a task has stopped execution. On SMP, this is not sufficient because the task may still be executing while child->has_cpu is true. This is easy to fix, but there is a related concern: what would happen if a ptrace'd task was woken up by another thread while ptrace() is running? You could argue that this is just a bug in the user-level program, but I'm worried that the resulting race conditions could corrupt kernel data structures and therefore could provide a means to attack the integrity of the kernel. The patch below is an attempt to address this issue by clearing child->cpus_allowed while ptrace() is running. This should have the desired effect and as far as I know, it has no negative impact. However, it doesn't strike me as the most elegant solution either and it could well be there there are some problems with it that I'm overlooking. Anyhow, I'd appreciate it if the maintainers of the other platforms could take a look at the patch below and share their thoughts. The patch below fixes only the ia64 tree, but if the patch is agreeable, I can make the same update for the other platforms as well.

	--david
kernel/ptrace.c +34 -0
@@ -60,6 +60,40 @@
 	return -EPERM;
 }
 
+long ptrace_freeze_child(struct task_struct *child, long request, unsigned long *freeze_info)
+{
+	unsigned long old_cpus_allowed;
+
+	if (request == PTRACE_KILL)
+		return 0;
+
+	if (child->state != TASK_STOPPED)
+		return -ESRCH;
+
+#ifdef CONFIG_SMP
+	while (child->has_cpu) {
+		if (child->state != TASK_STOPPED)
+			return -ESRCH;
+		barrier();
+	}
+#endif
+
+	old_cpus_allowed = xchg(&child->cpus_allowed, 0);
+
+	if (child->state != TASK_STOPPED) {
+		xchg(&child->cpus_allowed, old_cpus_allowed);
+		return -ESRCH;
+	}
+
+	*freeze_info = old_cpus_allowed;
+	return 0;
+}
+
+void ptrace_thaw_child(struct task_struct *child, long request, unsigned long freeze_info)
+{
+	if (request != PTRACE_KILL)
+		xchg(&child->cpus_allowed, freeze_info);
+}
 
 /*
  * Access another process' address space, one page at a time.
arch/ia64/kernel/ptrace.c +26 -20

142 lines not on this page · raw message

#1 lore raw
Quoted 1 line · David Mosberger

There is currently a nasty race condition in ptrace(). The effect of

Last time I checked for such race it was looking ok (on x86). Do you have a testcase to demonstrate it?

Quoted 6 lines

this varies from one platform to another but, for example, on ia64, it could have the effect of corrupting the state of register f32-f127. The problem is that ptrace() uses the expression (child->state == TASK_STOPPED) to determine whether or not a task has stopped execution. On SMP, this is not sufficient because the task may still be executing while child->has_cpu is true. This is easy to fix, but

but it cannot be running in "userspace" any longer once it is set to TASK_STOPPED which should be the _only_ thing we care in ptrace. If it's still running it's in its way to schedule() a few lines after the setting of tsk->state to TASK_STOPPED and that's ok for ptrace.

Quoted 1 line

clearing child->cpus_allowed while ptrace() is running. This should

abusing cpus_allowed to forbid scheduling is racy so quite unacceptable, we want to preserve the cpus_allowed field for the administrator (he could as well set it during the ptrace).

If on ia64 you really need to have switched the task away completly (not only out of userspace) you could do this instead of messing with cpus_allowed:

if (not task_stopped)
return

#ifdef CONFIG_SMP
	rmb(); /* read child->has_cpu after child->state */
	while (child->has_cpu);
	mb(); /* allowed to work on the task only when the task is been descheduled */

#endif

in ia64/kernel/ptrace.c

Actually one scary thing I can see in ptrace is the PTRACE_KILL case that goes ahead doing the get_stack_long and put_stack_long even if the task isn't out of userspace. I'd feel better with something like this:

arch/i386/kernel/ptrace.c renamed from arch/i386/kernel/ptrace.c.~1~ +2 -4
@@ -171,10 +171,8 @@
 	ret = -ESRCH;
 	if (!(child->ptrace & PT_PTRACED))
 		goto out_tsk;
-	if (child->state != TASK_STOPPED) {
-		if (request != PTRACE_KILL)
-			goto out_tsk;
-	}
+	if (child->state != TASK_STOPPED || child->state != TASK_ZOMBIE)
+		goto out_tsk;
 	if (child->p_pptr != current)
 		goto out_tsk;
 	switch (request) {

but OTOH I've no idea who is using PTRACE_KILL (but still it looks saner or otherwise it means PTRACE_KILL implementation is at least partly wrong anyways).

Andrea

#2 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andrea Arcangeli

Andrea> Last time I checked for such race it was looking ok (on Andrea> x86). Do you have a testcase to demonstrate it?

For ia64, yes. I didn't look at x86. Other platforms may or may not be affected, that is true, but the races are so subtle, I'd rather make sure this problem cannot occur on any platform.

Andrea> but it cannot be running in "userspace" any longer once it Andrea> is set to TASK_STOPPED which should be the _only_ thing we Andrea> care in ptrace. If it's still running it's in its way to Andrea> schedule() a few lines after the setting of tsk->state to Andrea> TASK_STOPPED and that's ok for ptrace.

Yes, the code running in schedule() is sufficient. On ia64, what happened is that switch_to() may end up saving the floating-point registers f32-f127. After doing so, it will set a bit in the thread flags and since this is done non-atomically (for good reason), it creates a race with the code in ptrace() which also updates the thread flags.

Andrea> abusing cpus_allowed to forbid scheduling is racy so quite Andrea> unacceptable, we want to preserve the cpus_allowed field for Andrea> the administrator (he could as well set it during the Andrea> ptrace).

As long as the CPU manipulates cpus_allowed in an atomic fashion (xchg or cmpxchg) this will be fine. I'd argue it has to do this anyhow (unless a task is changing its own cpus_allowed field).

If you don't like the cpus_allowed approach, please propose another solution that ensures that the task does not get woken up while ptrace is running. I don't really care very much what the solution is, but I do want to make sure we can guarantee that the task does not start running while ptrace() is doing it's job. Having to prove that race conditions in ptrace() are harmless is far too painful (and bound to be violated as code changes in the future...).

	--david
#3 lore raw
In reply to David Mosberger
Quoted 8 lines · David Mosberger

Andrea> abusing cpus_allowed to forbid scheduling is racy so quite Andrea> unacceptable, we want to preserve the cpus_allowed field for Andrea> the administrator (he could as well set it during the Andrea> ptrace).

As long as the CPU manipulates cpus_allowed in an atomic fashion (xchg or cmpxchg) this will be fine. I'd argue it has to do this anyhow (unless a task is changing its own cpus_allowed field).

atomic updates of cpus_allowed it's not the point I was making, it's still racy:

ptrace admin via /proc
-------------- ---------------
save and clear
set cpus_allowed to something
restore cpus_allowed <destroy modification>

the modification of the user is been destroyed if he sets cpus_allowed inside ptrace, this is the race condition I was thinking about.

Quoted 2 lines

If you don't like the cpus_allowed approach, please propose another solution that ensures that the task does not get woken up while ptrace

For making sure the task isn't wakenup while it's under ptrace we should just do that in kernel/signal.c::ignored_signal() as far I can tell.

To ensure the task just sleeps I suggest the one I mentioned in the previous email. here a patch (possibly breaks PTRACE_KILL, I didn't backed out the PTRACE_KILL change yet):

arch/i386/kernel/ptrace.c renamed from arch/i386/kernel/ptrace.c.~1~ +7 -4
@@ -171,12 +171,15 @@
 	ret = -ESRCH;
 	if (!(child->ptrace & PT_PTRACED))
 		goto out_tsk;
-	if (child->state != TASK_STOPPED) {
-		if (request != PTRACE_KILL)
-			goto out_tsk;
-	}
+	if (child->state != TASK_STOPPED && child->state != TASK_ZOMBIE)
+		goto out_tsk;
 	if (child->p_pptr != current)
 		goto out_tsk;
+#ifdef CONFIG_SMP
+	rmb(); /* read child->has_cpu after child->state */
+	while (child->has_cpu);
+	mb(); /* allowed to work on the task only when the task is been descheduled */
+#endif
 	switch (request) {
 	/* when I and D space are separate, these will need to be fixed. */
 	case PTRACE_PEEKTEXT: /* read word at location addr. */ 

Andrea

#4 lore raw

arch/i386/kernel/ptrace.c line 180

Quoted 3 lines
arch/i386/kernel/ptrace.c renamed from arch/i386/kernel/ptrace.c.~1~ +3 -0
+#ifdef CONFIG_SMP
+	rmb(); /* read child->has_cpu after child->state */
+	while (child->has_cpu);
		rep_nop();

otherwise your PIV will overheat

#5 lore raw
In reply to Andrea Arcangeli
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andrea Arcangeli

Andrea> For making sure the task isn't wakenup while it's under Andrea> ptrace we should just do that in
Andrea> kernel/signal.c::ignored_signal() as far I can tell.

This doesn't make sense: ignored_signal() is too late as handle_stop_signal() will already have woken up the task in response to a SIGCONT. Also, if you're suggesting to ignore SIGCONT while a PT_PTRACED is set, that certainly wouldn't be right. We only want to *delay* the wakeup while the ptrace() system call is running (which is much shorter than the period of time PT_PTRACED is set). So, as far as I can tell, you'd have to add more locking to the signal path, which doesn't look attractive to me. Also, if pursuing approach, we'd have to prove that we cover all possible paths that could wake up the task. E.g., PTRACE_SYSCALL and PTRACE_CONT are other ways the task could be woken up. These particular cases should be fine, because they'll already be serialized by the BKL acquired during ptrace(), but I'm not so sure there aren't any other cases.

So, I still think cpus_allowed is a safer and better approach at least for 2.4. Yes, we'd have to add locking for writing cpus_allowed, but I'd say that makes sense anyhow given that it is being manipulated by other tasks.

Hmmh, looking at ptrace() more closely, the entire locking situation seems to be a bit confused. For example, what's stopping wait4() from releasing the task structure just after ptrace() released the tasklist_lock and before it checked child->state?

	--david
#6 lore raw
In reply to Andrea Arcangeli
Quoted 6 lines · Alan Cox

arch/i386/kernel/ptrace.c line 180

Quoted 3 lines
arch/i386/kernel/ptrace.c renamed from arch/i386/kernel/ptrace.c.~1~ +3 -0
+#ifdef CONFIG_SMP
+	rmb(); /* read child->has_cpu after child->state */
+	while (child->has_cpu);
		rep_nop();

otherwise your PIV will overheat

indeed correct (OTOH it's a so small window [not like a contended spinlock] that I wonder if it will make a difference in real life ;)

Andrea

#7 lore raw
In reply to David Mosberger
Quoted 10 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andrea Arcangeli

Andrea> For making sure the task isn't wakenup while it's under Andrea> ptrace we should just do that in
Andrea> kernel/signal.c::ignored_signal() as far I can tell.

This doesn't make sense: ignored_signal() is too late as handle_stop_signal() will already have woken up the task in response to a SIGCONT. Also, if you're suggesting to ignore SIGCONT while a PT_PTRACED is set, that certainly wouldn't be right. We only want to

correct, I suggest to ignore SIGCONT as well while PT_PTRACED is set.

Quoted 3 lines

*delay* the wakeup while the ptrace() system call is running (which is much shorter than the period of time PT_PTRACED is set). So, as far as I can tell, you'd have to add more locking to the signal path,

Not more locking, just an additional check:

kernel/signal.c renamed from kernel/signal.c.~1~ +1 -1
@@ -382,7 +382,7 @@
 	switch (sig) {
 	case SIGKILL: case SIGCONT:
 		/* Wake up the process if stopped.  */
-		if (t->state == TASK_STOPPED)
+		if (t->state == TASK_STOPPED && !(t->ptrace & PT_PTRACED))
 			wake_up_process(t);
 		t->exit_code = 0;
 		rm_sig_from_queue(SIGSTOP, t);
Quoted 1 line

So, I still think cpus_allowed is a safer and better approach at least

forget that. Also when you restore the cpus_allowed you won't effectively wakup the task, it will just keep floating in the runqueue but we won't try to reschedule the other idle cpus it so it's broken.

Quoted 4 lines

Hmmh, looking at ptrace() more closely, the entire locking situation seems to be a bit confused. For example, what's stopping wait4() from releasing the task structure just after ptrace() released the tasklist_lock and before it checked child->state?

the get_task_struct()

Andrea

#8 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andrea Arcangeli

Andrea> correct, I suggest to ignore SIGCONT as well while Andrea> PT_PTRACED is set.

Do you really think it's acceptable? With your patch, you couldn't SIGKILL or SIGCONT a task that happens to be ptraced. I certainly would expect to be able to do this for a task that is being strace'd, for example.

Also, other signals will still wake up the task. Yes, it won't get very far as do_signal() will notify the parent instead, but still, the task will run and that could be enough to create some race condition.

What about the other wakeup paths that I had mentioned? E.g., what if the ptraced task is ptracing another task? Couldn't notify_parent() end up waking up the task as well?

Andrea> Also when you restore the cpus_allowed you won't effectively Andrea> wakup the task, it will just keep floating in the runqueue Andrea> but we won't try to reschedule the other idle cpus it so Andrea> it's broken.

Ah, that's a good point, thanks. Nothing that can't be fixed, though.

Quoted 5 lines
Quoted 5 lines

Hmmh, looking at ptrace() more closely, the entire locking situation seems to be a bit confused. For example, what's stopping wait4() from releasing the task structure just after ptrace() released the tasklist_lock and before it checked child->state?

Andrea> the get_task_struct()

Yes, I missed that.

	--david
#9 lore raw
In reply to David Mosberger
Quoted 3 lines · David Mosberger

Also, other signals will still wake up the task. Yes, it won't get very far as do_signal() will notify the parent instead, but still, the task will run and that could be enough to create some race condition.

this is the real issue, agreed.

However still I don't like the cpus_allowed racy approch. I either prefer to force the deschedule with a new ptrace bitflag with new hooks in the scheduler or with a blocker (delayer) to the signals again with a new ptrace bitflag but in this case with hooks in the signal code. I think putting the hooks in the signal code is better.

BTW, checking this stuff I found two bugs, one is the check for cpus_allowed before calling reschedule_idle, such check has to be removed, then it also seems the signals seems to wakeup the task two times unless I've overlooked something.

You may want to make a new patch at the light of those considerations otherwise I'll put this in my todo list once more important things are solved.

Andrea

#10 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andrea Arcangeli

Andrea> On Fri, Sep 07, 2001 at 08:35:31AM -0700, David Mosberger

Quoted 4 lines · Andrea> wrote:
Quoted 4 lines

Also, other signals will still wake up the task. Yes, it won't get very far as do_signal() will notify the parent instead, but still, the task will run and that could be enough to create some race condition.

Andrea> this is the real issue, agreed.

Good.

Andrea> However still I don't like the cpus_allowed racy approch. I Andrea> either prefer to force the deschedule with a new ptrace Andrea> bitflag with new hooks in the scheduler or with a blocker Andrea> (delayer) to the signals again with a new ptrace bitflag but Andrea> in this case with hooks in the signal code. I think putting Andrea> the hooks in the signal code is better.

Yes, though I don't really see how you could do this without any change to the scheduler.

Andrea> BTW, checking this stuff I found two bugs, one is the check Andrea> for cpus_allowed before calling reschedule_idle, such check Andrea> has to be removed, then it also seems the signals seems to Andrea> wakeup the task two times unless I've overlooked something.

Andrea> You may want to make a new patch at the light of those Andrea> considerations otherwise I'll put this in my todo list once Andrea> more important things are solved.

Why don't you keep it on your todo list. I too have a couple of other things I need to finish first so it will be a while before I'd get to this (not before November, I'd guess). But I'll keep it in mind as well.

Thanks,

	--david