Skip to contentSkip to conversation

wait_task_inactive: don't use the dummy version when !SMP && PREEMPT

PATCH In review Oleg Nesterov 4 participants 6 messages Jul 30, 2008 – Jul 31, 2008 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 3/3 wait_task_inactive: don't use the dummy version when !SMP && PREEMPT

wait_task_inactive() must return success only when/if the task leaves the runqueue, it doesn't work correctly when !SMP && PREEMPT because in that case we use the "dummy" version which doesn't check .on_rq.

Change the "#if" around the full-blown version from SMP to SMP || PREEMPT.

The patch looks monstrous because it moves the (unchanged) definition of wait_task_inactive() outside of "#ifdef CONFIG_SMP", but it is quite trivial.

 include/linux/sched.h |    2
 kernel/sched.c        |  210 +++++++++++++++++++++++++-------------------------
 2 files changed, 107 insertions(+), 105 deletions(-)
include/linux/sched.h renamed from include/linux/sched.h~3_WTI_PREEMPT +1 -1
@@ -1874,7 +1874,7 @@ struct task_struct *fork_idle(int);
 extern void set_task_comm(struct task_struct *tsk, char *from);
 extern char *get_task_comm(char *to, struct task_struct *tsk);
 
-#ifdef CONFIG_SMP
+#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT)
 extern unsigned long wait_task_inactive(struct task_struct *, long match_state);
 #else
 static inline unsigned long wait_task_inactive(struct task_struct *p,
kernel/sched.c renamed from kernel/sched.c~3_WTI_PREEMPT +106 -104

222 lines not on this page · raw message

#1 lore raw
Quoted 4 lines · Oleg Nesterov

The patch looks monstrous because it moves the (unchanged) definition of wait_task_inactive() outside of "#ifdef CONFIG_SMP", but it is quite trivial.

Hmm. Doesn't this just deadlock in UP (PREEMPT) if wait_task_interactive() is ever called from a no-preempt context?

And if that's never the case, the comment should be updated to reflect that (right now it says that it's only invalid to call it with interrupts disabled to avoid cross-IPI deadlocks).

Oh, and shouldn't it do a "yield()" instead of a cpu_relax() on UP?

Inquiring minds want to know. That function was very much expressly designed for SMP, not for preemption, and I want to understand why it's ok (_if_ it's ok).

Linus

#2 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Oleg Nesterov

The patch looks monstrous because it moves the (unchanged) definition of wait_task_inactive() outside of "#ifdef CONFIG_SMP", but it is quite trivial.

Hmm. Doesn't this just deadlock in UP (PREEMPT) if wait_task_interactive() is ever called from a no-preempt context?

Given that it calls schedule_timeout_uninterruptible(), it can't be used from the no-preempt context,

Quoted 3 lines

And if that's never the case, the comment should be updated to reflect that (right now it says that it's only invalid to call it with interrupts disabled to avoid cross-IPI deadlocks).

Yes, I think this function is might_sleep(),

Quoted 1 line

Oh, and shouldn't it do a "yield()" instead of a cpu_relax() on UP?

I _think_ that rq->curr must be == current without CONFIG_SMP, but

Quoted 2 lines

and I want to understand why it's
ok (_if_ it's ok).

me too.

Hopefully Ingo can ack/nack.

Oleg.

#3 lore raw
In reply to Linus Torvalds

2008/7/30 Linus Torvalds <[email protected]>:

Quoted 8 lines · Oleg Nesterov
Quoted 4 lines

The patch looks monstrous because it moves the (unchanged) definition of wait_task_inactive() outside of "#ifdef CONFIG_SMP", but it is quite trivial.

Quoted 8 lines

Hmm. Doesn't this just deadlock in UP (PREEMPT) if wait_task_interactive() is ever called from a no-preempt context?

And if that's never the case, the comment should be updated to reflect that (right now it says that it's only invalid to call it with interrupts disabled to avoid cross-IPI deadlocks).

Oh, and shouldn't it do a "yield()" instead of a cpu_relax() on UP?

This part could have been skipped for UP. task_running(rq, p) just can't give 'true' for UP (otherwise it's a bug). The only relevant part is "on_rq = p->se.on_rq".

Quoted 3 lines

Inquiring minds want to know. That function was very much expressly designed for SMP,

It looks so. Otherwise it's behavior is not symmetric and I think, either [1] it shouldn't be a "nop" for !SMP or [2] there shouldn't be a version for !SMP at all --> so no one can make false assumptions.

(if [1], then I think a separate function for PREEMPT would look better, i.e. without parts with task_running())

e.g. consider this code from kthread_bind():

        /* Must have done schedule() in kthread() before we set_task_cpu */
        wait_task_inactive(k, 0);

        set_task_cpu(k, cpu);
        k->cpus_allowed = cpumask_of_cpu(cpu);
        k->rt.nr_cpus_allowed = 1;
        k->flags |= PF_THREAD_BOUND;

set_task_cpu(k, cpu) is safe _only_ if 'k' is not on the run-queue (and can't be placed onto it behind our back -- heh, a bit subtle).

Now, for !SMP + PREEMPT it's not a case. set_task_cpu() may be called while 'k' is still on the run-queue (more precisely, preempted in kthread() between complete(&create->started); and schedule();).

Yes, set_task_cpu() is a "nop" for UP so that's ok in this particular case. But let's suppose, another use-case would be introduced with 'false' assumptions causing troubles for !SMP.

Quoted 4 lines

not for preemption, and I want to understand why it's ok (_if_ it's ok).

Linus

Signature · 2 lines
Best regards,
Dmitry Adamushko
#4 lore raw
Quoted 8 lines · Dmitry Adamushko

2008/7/30 Linus Torvalds <[email protected]>:

Quoted 2 lines

Oh, and shouldn't it do a "yield()" instead of a cpu_relax() on UP?

This part could have been skipped for UP. task_running(rq, p) just can't give 'true' for UP (otherwise it's a bug). The only relevant part is "on_rq = p->se.on_rq".

That was my understanding, thanks for your confirmation.

Quoted 2 lines

(if [1], then I think a separate function for PREEMPT would look better, i.e. without parts with task_running())

in that case we can also eliminate task_rq_lock() afaics.

Quoted 13 lines

e.g. consider this code from kthread_bind():

        /* Must have done schedule() in kthread() before we set_task_cpu */
        wait_task_inactive(k, 0);

        set_task_cpu(k, cpu);
        k->cpus_allowed = cpumask_of_cpu(cpu);
        k->rt.nr_cpus_allowed = 1;
        k->flags |= PF_THREAD_BOUND;

set_task_cpu(k, cpu) is safe _only_ if 'k' is not on the run-queue (and can't be placed onto it behind our back -- heh, a bit subtle).

Quoted 7 lines

Now, for !SMP + PREEMPT it's not a case. set_task_cpu() may be called while 'k' is still on the run-queue (more precisely, preempted in kthread() between complete(&create->started); and schedule();).

Yes, set_task_cpu() is a "nop" for UP so that's ok in this particular case. But let's suppose, another use-case would be introduced with 'false' assumptions causing troubles for !SMP.

Completely agreed.

Currently the only user which can suffer on UP && PREEMPT is task_current_syscall(). As Roland suggested we can fix it if we change the !SMP version to return ->nivcsw + ->nvcsw. But personally I don't like the fact that we will have the subltle difference in behaviour depending on CONFIG_SMP. We have to wait for .on_rq == 0 on SMP, imho it is better to do the same on UP even if none of the current callers needs this.

That said, I think this problem is minor and I don't (and can't) have the strong opinion on how to fix it. I'd better wait for the authoritative verdict.

Oleg.

#5 lore raw
In reply to Oleg Nesterov

I'm not convinced we need this at all (as per prior thread "Q: wait_task_inactive() and !CONFIG_SMP && CONFIG_PREEMPT").

Let's not get hung up on what the explanation of the function's semantics supposedly is. We can change the comments to match the reality. We can rename the function to better reflect its true guarantee, if you like.

There are two uses of wait_task_inactive. One is kthread_bind. We have established that a nop is fine for !SMP there.

The other uses are for tracing. Presently that is the traditional use in ptrace, and task_current_syscall.

I've described that the requirement of the traditional ptrace call is quite weak. A nop is fine for !SMP. Preemption is not an issue because all that matters is that no access to pt_regs/thread_struct can race with the actual context switch (by definition has to be on another CPU to be simultaneous).

For task_current_syscall, it also doesn't matter that it's left the run queue. It just matters that the call before looking at pt_regs and the call after indicate whether it stayed switched out for the duration. As per the prior thread on this, I think that's covered by using nivcsw+nvcsw as I did originally rather than nvcsw alone. So a short inline is still fine.

The other uses I anticipate (e.g. utrace) follow the same model as task_current_syscall (check before and after) and have only the same requirements.

Thanks,
Roland