Cpu-Hotplug and Real-Time
Hi,
While running a cpu-hotplug test involving a high priority process (SCHED_RR, prio=94) trying to periodically offline and online cpu1 on a 2-processor machine, I noticed that the system was becoming unresponsive after a few iterations.
However, when the same test was repeated with processors greater than 2, it worked fine.
Also, if the hotplugging process, was not of rt-prio, it worked fine on a 2-processor machine.
After some debugging, I saw that the hang occured because the high prio process was stuck in a loop doing yield() inside wait_task_inactive(). Description follows:
Say a high-prio task (A) does a kthread_create(B), followed by a kthread_bind(B, cpu1). At this moment, only cpu0 is online.
Now, immediately after being created, B would
do acomplete(&create->started) [kernel/kthread.c: kthread()], before scheduling itself out.
This complete() will wake up kthreadd, which had spawned B. It is possible that during the wakeup, kthreadd might preempt B. Thus, B is still on the runqueue, and not yet called schedule().
kthreadd, will inturn do acomplete(&create->done); [kernel/kthread.c: create_kthread()] which will wake up the thread which had called kthread_create(). In our case it's task A, which will run immediately, since its priority is higher.
A will now call kthread_bind(B, cpu1). kthread_bind(), calls wait_task_inactive(B), to ensures that B has scheduled itself out.
B is still on the runqueue, so A calls yield() in wait_task_inactive(). But since A is the task with the highest prio, scheduler schedules it back again.
Thus B never gets to run to schedule itself out.
A loops waiting for B to schedule out leading to system hang.
In my case,
A was the high priority process trying to bring up cpu1, and thus doing a kthread_create/kthread_bind in migration_call(): CPU_UP_PREPARE.
B was the migration thread for cpu1.
And the above problem occurs when only one cpu is online.
Possible solutions to this problem:
a) Let the newly spawned kernel threads inherit
their parent's prio and policy.
b) Instead of using yield() in wait_task_inactive(), we could use something like a yield_to(p):
yield_to(struct task_struct p)
{
int old_prio = p->prio;
/* Temporarily boost p's priority atleast to that of current task */
if (current->prio > old_prio)
set_prio(p, current->prio);
yield();
/* Reset priority back to the original value */
set_prio(p, old_prio);
}Thoughts?
Thanks and Regards
gautham.
Signature · 5 lines
Gautham R Shenoy Linux Technology Center IBM India. "Freedom comes with a price tag of responsibility, which is still a bargain, because Freedom is priceless!"
Quoted 28 lines · Gautham R Shenoy
After some debugging, I saw that the hang occured because the high prio process was stuck in a loop doing
yield()insidewait_task_inactive(). Description follows:Say a high-prio task (A) does a
kthread_create(B), followed by akthread_bind(B, cpu1). At this moment, only cpu0 is online.Now, immediately after being created, B would
do acomplete(&create->started)[kernel/kthread.c:kthread()], before scheduling itself out.This
complete()will wake up kthreadd, which had spawned B. It is possible that during the wakeup, kthreadd might preempt B. Thus, B is still on the runqueue, and not yet calledschedule().kthreadd, will inturn do a
complete(&create->done); [kernel/kthread.c:create_kthread()] which will wake up the thread which had calledkthread_create(). In our case it's task A, which will run immediately, since its priority is higher.A will now call
kthread_bind(B, cpu1).kthread_bind(), callswait_task_inactive(B), to ensures that B has scheduled itself out.
Quoted 6 lines
B is still on the runqueue, so A calls
yield()inwait_task_inactive(). But since A is the task with the highest prio, scheduler schedules it back again.Thus B never gets to run to schedule itself out.
A loops waiting for B to schedule out leading to system hang.
As for kthread_bind(), I think wait_task_inactive+set_task_cpu is just an optimization, and easy to "fix":
proc/self/fd/0 renamed from kthread.c +1 -4
@@ -166,10 +166,7 @@ void kthread_bind(struct task_struct *k, WARN_ON(1); return; } - /* Must have done schedule() in kthread() before we set_task_cpu */ - wait_task_inactive(k); - set_task_cpu(k, cpu); - k->cpus_allowed = cpumask_of_cpu(cpu); + set_cpus_allowed(current, cpumask_of_cpu(cpu)); } EXPORT_SYMBOL(kthread_bind);
But I think we have another case. An RT ptracer can share the same CPU with ptracee. The latter sets TASK_STOPPED, unlocks ->siglock, and takes a preemption. Ptracer does ptrace_check_attach(), sees TASK_STOPPED, and yields in wait_task_inactive.
Oleg.
Quoted 43 lines · Oleg Nesterov
Quoted 28 lines · Gautham R Shenoy
After some debugging, I saw that the hang occured because the high prio process was stuck in a loop doing
yield()insidewait_task_inactive(). Description follows:Say a high-prio task (A) does a
kthread_create(B), followed by akthread_bind(B, cpu1). At this moment, only cpu0 is online.Now, immediately after being created, B would
do acomplete(&create->started)[kernel/kthread.c:kthread()], before scheduling itself out.This
complete()will wake up kthreadd, which had spawned B. It is possible that during the wakeup, kthreadd might preempt B. Thus, B is still on the runqueue, and not yet calledschedule().kthreadd, will inturn do a
complete(&create->done); [kernel/kthread.c:create_kthread()] which will wake up the thread which had calledkthread_create(). In our case it's task A, which will run immediately, since its priority is higher.A will now call
kthread_bind(B, cpu1).kthread_bind(), callswait_task_inactive(B), to ensures that B has scheduled itself out.Quoted 6 lines
B is still on the runqueue, so A calls
yield()inwait_task_inactive(). But since A is the task with the highest prio, scheduler schedules it back again.Thus B never gets to run to schedule itself out.
A loops waiting for B to schedule out leading to system hang.As for
kthread_bind(), I think wait_task_inactive+set_task_cpu is just an optimization, and easy to "fix":proc/self/fd/0 renamed from kthread.c +0
-0@@ -166,10 +166,7 @@ void kthread_bind(struct task_struct *k, WARN_ON(1);
proc/self/fd/0 void kthread_bind(struct task_struct *k, · line 172
Quoted 10 lines
proc/self/fd/0 renamed from kthread.c +1
-4return; } - /* Must have done schedule() in kthread() before we set_task_cpu */ - wait_task_inactive(k); - set_task_cpu(k, cpu); - k->cpus_allowed = cpumask_of_cpu(cpu); + set_cpus_allowed(current, cpumask_of_cpu(cpu)); } EXPORT_SYMBOL(kthread_bind);
Not sure whether set_cpus_allowed() will work here. Looks like, it needs the CPU to be online during the call and in kthread_bind() case CPU may be offline.
Thanks,
Venki
Quoted 15 lines · Venki Pallipadi
Quoted 13 lines · Oleg Nesterov
As for
kthread_bind(), I think wait_task_inactive+set_task_cpu is just an optimization, and easy to "fix":proc/self/fd/0 renamed from kthread.c +0
-3@@ -166,10 +166,7 @@ void kthread_bind(struct task_struct *k, WARN_ON(1); return; } - /* Must have done schedule() in kthread() before we set_task_cpu */ - wait_task_inactive(k); - set_task_cpu(k, cpu);
Quoted 8 lines
proc/self/fd/0 void kthread_bind(struct task_struct *k, · line 172
Quoted 5 lines
proc/self/fd/0 renamed from kthread.c +1
-1- k->cpus_allowed = cpumask_of_cpu(cpu); + set_cpus_allowed(current, cpumask_of_cpu(cpu)); } EXPORT_SYMBOL(kthread_bind);Not sure whether
set_cpus_allowed()will work here. Looks like, it needs the CPU to be online during the call and inkthread_bind()case CPU may be offline.
Aah, you are right, of course.
Thanks,
Oleg.
rt ptracer can monopolize CPU (was: Cpu-Hotplug and Real-Time)
In reply to Oleg NesterovQuoted 11 lines · Oleg Nesterov
Quoted 9 lines · Gautham R Shenoy
A will now call
kthread_bind(B, cpu1).kthread_bind(), callswait_task_inactive(B), to ensures that B has scheduled itself out.B is still on the runqueue, so A calls
yield()inwait_task_inactive(). But since A is the task with the highest prio, scheduler schedules it back again.
Quoted 7 lines
Quoted 2 lines
Thus B never gets to run to schedule itself out.
A loops waiting for B to schedule out leading to system hang.But I think we have another case. An RT ptracer can share the same CPU with ptracee. The latter sets
TASK_STOPPED, unlocks ->siglock, and takes a preemption. Ptracer doesptrace_check_attach(), seesTASK_STOPPED, and yields inwait_task_inactive.
Even simpler.
#include <stdio.h>
#include <signal.h>
#include <unistd.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#define __USE_GNU
#include <sched.h>
void die(const char *msg)
{
printf("ERR!! %s: %m\n", msg);
kill(0, SIGKILL);
}void set_cpu(int cpu)
{
unsigned cpuval = 1 << cpu;
if (sched_setaffinity(0, 4, (void*)&cpuval) < 0)
die("setaffinity");
}// __wake_up_parent() does SYNC wake up, we need a handler to provoke // signal_wake_up().
// otherwise ptrace_stop() is not preempted after read_unlock(tasklist). static void sigchld(int sig)
{
}
int main(void)
{
set_cpu(0);
int pid = fork();
if (!pid)
for (;;)
;
struct sched_param sp = { 99 };
if (sched_setscheduler(0, SCHED_FIFO, &sp))
die("setscheduler");
signal(SIGCHLD, sigchld);
if (ptrace(PTRACE_ATTACH, pid, NULL, NULL))
die("attach");
wait(NULL);
if (ptrace(PTRACE_DETACH, pid, NULL, NULL))
die("detach");
kill(pid, SIGKILL);
return 0;
}Locks CPU 0. Not a security problem, needs CAP_SYS_NICE and the task could be reniced and killed, but still not good.
ptracee does ptrace_stop()->do_notify_parent_cldstop(), ptracer preempts the child before it calls schedule(), ptrace(PTRACE_DETACH) goes to wait_task_inactive() and yields forever.
Can we just replace yield() with schedule_timeout_uninterruptible(1) ? wait_task_inactive() has no time-critical callers, and as it currently used "on_rq" case is really unlikely.
Oleg.
Re: rt ptracer can monopolize CPU (was: Cpu-Hotplug and Real-Time)
Quoted 81 lines · Oleg Nesterov
Quoted 11 lines · Oleg Nesterov
Quoted 9 lines · Gautham R Shenoy
A will now call
kthread_bind(B, cpu1).kthread_bind(), callswait_task_inactive(B), to ensures that B has scheduled itself out.B is still on the runqueue, so A calls
yield()inwait_task_inactive(). But since A is the task with the highest prio, scheduler schedules it back again.Quoted 7 lines
Quoted 2 lines
Thus B never gets to run to schedule itself out.
A loops waiting for B to schedule out leading to system hang.But I think we have another case. An RT ptracer can share the same CPU with ptracee. The latter sets
TASK_STOPPED, unlocks ->siglock, and takes a preemption. Ptracer doesptrace_check_attach(), seesTASK_STOPPED, and yields inwait_task_inactive.Even simpler.
#include <
stdio.h>
#include <signal.h>
#include <unistd.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#define__USE_GNU
#include <sched.h>void
die(const char *msg){ printf("ERR!! %s: %m\n", msg); kill(0, SIGKILL); }void
set_cpu(int cpu){ unsigned cpuval = 1 << cpu; if (sched_setaffinity(0, 4, (void*)&cpuval) < 0) die("setaffinity"); }//
__wake_up_parent()does SYNC wake up, we need a handler to provoke //signal_wake_up().
// otherwiseptrace_stop()is not preempted afterread_unlock(tasklist). static void sigchld(int sig)
{
}int main(void)
{ set_cpu(0); int pid = fork(); if (!pid) for (;;) ; struct sched_param sp = { 99 }; if (sched_setscheduler(0, SCHED_FIFO, &sp)) die("setscheduler"); signal(SIGCHLD, sigchld); if (ptrace(PTRACE_ATTACH, pid, NULL, NULL)) die("attach"); wait(NULL); if (ptrace(PTRACE_DETACH, pid, NULL, NULL)) die("detach"); kill(pid, SIGKILL); return 0; }Locks CPU 0. Not a security problem, needs
CAP_SYS_NICEand the task could be reniced and killed, but still not good.
Quoted 7 lines
ptracee does ptrace_stop()->do_notify_parent_cldstop(), ptracer preempts the child before it calls
schedule(),ptrace(PTRACE_DETACH)goes towait_task_inactive()and yields forever.Can we just replace
yield()withschedule_timeout_uninterruptible(1)?wait_task_inactive()has no time-critical callers, and as it currently used "on_rq" case is really unlikely.
schedule_timeout_uninterruptible(1) works fine, in my case. It makes sense to have it there instead of yield. Like you pointed out, it gets called only in "unlikely" case.
patch below.
Thanks and Regards
gautham.
-->yield() in wait_task_inactive(), can cause a high priority thread to be scheduled back in, and there by loop forever while it is waiting for some lower priority thread which is unfortunately still on the runqueue.
Use schedule_timeout_uninterruptible(1) instead.
Credit: Oleg Nesterov <[email protected]>
--- kernel/sched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched.c +1 -1
@@ -1106,7 +1106,7 @@ repeat:* yield - it could be a while. */ if (unlikely (on_rq)) { -yield (); +schedule_timeout_uninterruptible (1 ); goto repeat; }
Signature · 5 lines
Gautham R Shenoy Linux Technology Center IBM India. "Freedom comes with a price tag of responsibility, which is still a bargain, because Freedom is priceless!"