sched/deadline: support dl task migrate during cpu hotplug
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 1414747229.8574.99.camel@tkhai
Hi Kirill,
Quoted 60 lines · Kirill Tkhai
В Пт, 31/10/2014 в 15:28 +0800, Wanpeng Li пишет:
Quoted 59 lines
Hi all,
I observe that dl task can't be migrated to other cpus during cpu hotplug, in addition, task may/may not be running again if cpu is added back. The root cause which I found is that dl task will be throtted and removed from dl rq after comsuming all budget, which leads to stop task can't pick it up from dl rq and migrate to other cpus during hotplug.
So I try two methods.
- add throttled dl
sched_entityto athrottled_list, the list will be traversed during cpu hotplug, and the dlsched_entitywill be picked and enqueue, then stop task will pick and migrate it. However, dlsched_entityis throttled again before stop task running since the below path. This path will setrq->online0 which lead toset_rq_offline()won't be called in functionmigration_call().Call Trace: [...] rq_offline_dl+0x44/0x66 [...] set_rq_offline+0x29/0x54 [...] rq_attach_root+0x3f/0xb7 [...] cpu_attach_domain+0x1c7/0x354 [...] build_sched_domains+0x295/0x304 [...] partition_sched_domains+0x26a/0x2e6 [...] ? emulator_write_gpr+0x27/0x27 [kvm] [...] cpuset_update_active_cpus+0x12/0x2c [...] cpuset_cpu_inactive+0x1b/0x38 [...] notifier_call_chain+0x32/0x5e [...] __raw_notifier_call_chain+0x9/0xb [...] __cpu_notify+0x1b/0x2d [...] _cpu_down+0x81/0x22a [...] cpu_down+0x28/0x35 [...] cpu_subsys_offline+0xf/0x11 [...] device_offline+0x78/0xa8 [...] online_store+0x48/0x69 [...] ? kernfs_fop_write+0x61/0x129 [...] dev_attr_store+0x1b/0x1d [...] sysfs_kf_write+0x37/0x39 [...] kernfs_fop_write+0xe9/0x129 [...] vfs_write+0xc6/0x19e [...] SyS_write+0x4b/0x8f [...] system_call_fastpath+0x16/0x1b- The difference of the method two is that dl
sched_entitywon't be throtted if rq is offline, the dlsched_entitywill be replenished inupdate_curr_dl(). However, the echo 0 >/sys/devices/system/cpu/cpuN/onlinehung.Juri, your proposal is a great welcome. ;-)
Note: This patch is just a proposal and still can't successfully migrate dl task during cpu hotplug.
Signed-off-by Wanpeng Li <[email protected]>--- include/linux/sched.h | 2 ++ kernel/sched/deadline.c | 22 +++++++++++++++++++++- kernel/sched/sched.h | 3 +++ 3 files changed, 26 insertions(+), 1 deletion(-)
Quoted 13 lines
include/linux/sched.h struct sched_dl_entity { · line 1257
Quoted 10 lines
include/linux/sched.h +2
-0@@ -1253,6 +1253,8 @@ struct sched_dl_entity {* own bandwidth to be enforced, thus we need one timer per task. */ struct hrtimer dl_timer; +struct list_head throttled_node; +int on_list;Get rig of
on_list. It's better to check forlist_empty(&dl->throttled_node)instead. Of course, you should changelist_del()onlist_del_init()for this.
Agreed.
Quoted 27 lines
Quoted 27 lines
include/linux/sched.h +0
-0};union rcu_special {kernel/sched/deadline.c +5
-0@@ -80,6 +80,7 @@ void init_dl_rq(struct dl_rq *dl_rq, struct rq *rq) dl_rq->dl_nr_migratory =0 ; dl_rq->overloaded =0 ; dl_rq->pushable_dl_tasks_root =RB_ROOT ; +INIT_LIST_HEAD (&dl_rq->throttled_list);#else init_dl_bw (&dl_rq->dl_bw);#endif @@ -538,6 +539,10 @@ again:update_rq_clock (rq); dl_se->dl_throttled =0 ; dl_se->dl_yielded =0 ; +if (dl_se->on_list) { +list_del (&dl_se->throttled_node); + dl_se->on_list =0 ; + }if (task_on_rq_queued (p)) {enqueue_task_dl (rq, p,ENQUEUE_REPLENISH );if (dl_task (rq->curr))
Quoted 8 lines
kernel/sched/deadline.c static void update_curr_dl(struct rq *rq) · line 644
Quoted 6 lines
kernel/sched/deadline.c +1
-1@@ -636,8 +641,12 @@ static void update_curr_dl(struct rq *rq) dl_se->runtime -= delta_exec;if (dl_runtime_exceeded (rq, dl_se)) {__dequeue_task_dl (rq, curr,0 ); -if (likely (start_dl_timer (dl_se, curr->dl.dl_boosted))) +if (rq->online &&likely (start_dl_timer (dl_se, curr->dl.dl_boosted))) {Why is this check for
rq->onlinenecessary?
I will remove it.
Quoted 7 lines
kernel/sched/deadline.c static void update_curr_dl(struct rq *rq) · line 648
Quoted 5 lines
kernel/sched/deadline.c +3
-0dl_se->dl_throttled =1 ; + dl_se->on_list =1 ; +list_add (&dl_se->throttled_node, + &rq->dl.throttled_list);Alignment is wrong.
Agreed.
Quoted 14 lines
Quoted 14 lines
kernel/sched/deadline.c +3
-0+ }else enqueue_task_dl (rq, curr,ENQUEUE_REPLENISH );@@ -1593,9 +1602,20 @@ static void rq_online_dl(struct rq *rq)/* Assumes rq->lock is held */ static void rq_offline_dl (struct rq *rq) { +struct task_struct *p, *n; +if (rq->dl.overloaded)dl_clear_overload (rq);
Quoted 7 lines
kernel/sched/deadline.c static void rq_online_dl(struct rq *rq) · line 1613
Quoted 4 lines
kernel/sched/deadline.c +4
-0+/* Make sched_dl_entity available for pick_next_task() */ +list_for_each_entry_safe (p, n, &rq->dl.throttled_list, dl.throttled_node) { + p->dl.dl_throttled =0 ; +hrtimer_cancel (&p->dl.dl_timer);Deadlock is possible here. You're holding
rq->lockand want to cancel timer handler, which is waiting for yourrq->lock.
So what's your idea to handle this?
Quoted 9 lines
Quoted 9 lines
kernel/sched/deadline.c +5
-0+ p->dl.dl_runtime = p->dl.dl_runtime; +if (task_on_rq_queued (p)) +enqueue_task_dl (rq, p,ENQUEUE_REPLENISH ); + } +cpudl_set (&rq->rd->cpudl, rq->cpu,0 ,0 ); }
Quoted 16 lines
Quoted 5 lines
kernel/sched/sched.h +0
-0kernel/sched/sched.h struct dl_rq { · line 490
Quoted 9 lines
kernel/sched/sched.h +3
-0@@ -482,6 +482,9 @@ struct dl_rq {*/ struct rb_root pushable_dl_tasks_root;struct rb_node *pushable_dl_tasks_leftmost; + +struct list_head throttled_list; +#else struct dl_bw dl_bw;#endif What about the situations when task changes its
sched_class?
I still not consider this currently, your proposal is a great welcome. ;-)
Regards,
Wanpeng Li
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Juri,
Quoted 10 lines · Juri Lelli
Hi,
Quoted 4 lines · Wanpeng Li
Hi all,
I observe that dl task can't be migrated to other cpus during cpu hotplug, in addition, task may/may not be running again if cpu is added back. The root cause
Can you share more information about this? Do you have a test I can run on my box?
schedtool -E -t 50000:100000 -e ./test
Actually test is just a simple for loop. Then observe which cpu the test task is on.
echo 0 > /sys/devices/system/cpu/cpuN/online
Task may/may not be running again if cpu is added back in my observation, in addition, task won't be migrated to other cpus after logical remove cpu.
Regards,
Wanpeng Li
Quoted 146 lines
Thanks,
- Juri
Quoted 141 lines
which I found is that dl task will be throtted and removed from dl rq after comsuming all budget, which leads to stop task can't pick it up from dl rq and migrate to other cpus during hotplug.
So I try two methods.
- add throttled dl
sched_entityto athrottled_list, the list will be traversed during cpu hotplug, and the dlsched_entitywill be picked and enqueue, then stop task will pick and migrate it. However, dlsched_entityis throttled again before stop task running since the below path. This path will setrq->online0 which lead toset_rq_offline()won't be called in functionmigration_call().Call Trace: [...] rq_offline_dl+0x44/0x66 [...] set_rq_offline+0x29/0x54 [...] rq_attach_root+0x3f/0xb7 [...] cpu_attach_domain+0x1c7/0x354 [...] build_sched_domains+0x295/0x304 [...] partition_sched_domains+0x26a/0x2e6 [...] ? emulator_write_gpr+0x27/0x27 [kvm] [...] cpuset_update_active_cpus+0x12/0x2c [...] cpuset_cpu_inactive+0x1b/0x38 [...] notifier_call_chain+0x32/0x5e [...] __raw_notifier_call_chain+0x9/0xb [...] __cpu_notify+0x1b/0x2d [...] _cpu_down+0x81/0x22a [...] cpu_down+0x28/0x35 [...] cpu_subsys_offline+0xf/0x11 [...] device_offline+0x78/0xa8 [...] online_store+0x48/0x69 [...] ? kernfs_fop_write+0x61/0x129 [...] dev_attr_store+0x1b/0x1d [...] sysfs_kf_write+0x37/0x39 [...] kernfs_fop_write+0xe9/0x129 [...] vfs_write+0xc6/0x19e [...] SyS_write+0x4b/0x8f [...] system_call_fastpath+0x16/0x1b- The difference of the method two is that dl
sched_entitywon't be throtted if rq is offline, the dlsched_entitywill be replenished inupdate_curr_dl(). However, the echo 0 >/sys/devices/system/cpu/cpuN/onlinehung.Juri, your proposal is a great welcome. ;-)
Note: This patch is just a proposal and still can't successfully migrate dl task during cpu hotplug.
Signed-off-by Wanpeng Li <[email protected]>--- include/linux/sched.h | 2 ++ kernel/sched/deadline.c | 22 +++++++++++++++++++++- kernel/sched/sched.h | 3 +++ 3 files changed, 26 insertions(+), 1 deletion(-)include/linux/sched.h +2
-0@@ -1253,6 +1253,8 @@ struct sched_dl_entity {* own bandwidth to be enforced, thus we need one timer per task. */ struct hrtimer dl_timer; +struct list_head throttled_node; +int on_list; };union rcu_special {kernel/sched/deadline.c +21
-1@@ -80,6 +80,7 @@ void init_dl_rq(struct dl_rq *dl_rq, struct rq *rq) dl_rq->dl_nr_migratory =0 ; dl_rq->overloaded =0 ; dl_rq->pushable_dl_tasks_root =RB_ROOT ; +INIT_LIST_HEAD (&dl_rq->throttled_list);#else init_dl_bw (&dl_rq->dl_bw);#endif @@ -538,6 +539,10 @@ again:update_rq_clock (rq); dl_se->dl_throttled =0 ; dl_se->dl_yielded =0 ; +if (dl_se->on_list) { +list_del (&dl_se->throttled_node); + dl_se->on_list =0 ; + }if (task_on_rq_queued (p)) {enqueue_task_dl (rq, p,ENQUEUE_REPLENISH );if (dl_task (rq->curr))@@ -636,8 +641,12 @@ static void update_curr_dl(struct rq *rq) dl_se->runtime -= delta_exec;if (dl_runtime_exceeded (rq, dl_se)) {__dequeue_task_dl (rq, curr,0 ); -if (likely (start_dl_timer (dl_se, curr->dl.dl_boosted))) +if (rq->online &&likely (start_dl_timer (dl_se, curr->dl.dl_boosted))) { dl_se->dl_throttled =1 ; + dl_se->on_list =1 ; +list_add (&dl_se->throttled_node, + &rq->dl.throttled_list); + }else enqueue_task_dl (rq, curr,ENQUEUE_REPLENISH );@@ -1593,9 +1602,20 @@ static void rq_online_dl(struct rq *rq)/* Assumes rq->lock is held */ static void rq_offline_dl (struct rq *rq) { +struct task_struct *p, *n; +if (rq->dl.overloaded)dl_clear_overload (rq); +/* Make sched_dl_entity available for pick_next_task() */ +list_for_each_entry_safe (p, n, &rq->dl.throttled_list, dl.throttled_node) { + p->dl.dl_throttled =0 ; +hrtimer_cancel (&p->dl.dl_timer); + p->dl.dl_runtime = p->dl.dl_runtime; +if (task_on_rq_queued (p)) +enqueue_task_dl (rq, p,ENQUEUE_REPLENISH ); + } +cpudl_set (&rq->rd->cpudl, rq->cpu,0 ,0 ); }kernel/sched/sched.h +3
-0@@ -482,6 +482,9 @@ struct dl_rq {*/ struct rb_root pushable_dl_tasks_root;struct rb_node *pushable_dl_tasks_leftmost; + +struct list_head throttled_list; +#else struct dl_bw dl_bw;#endif
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Peter,
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Wanpeng Li
Hi all,
I observe that dl task can't be migrated to other cpus during cpu hotplug, in addition, task may/may not be running again if cpu is added back. The root cause which I found is that dl task will be throtted and removed from dl rq after comsuming all budget, which leads to stop task can't pick it up from dl rq and migrate to other cpus during hotplug.
Quoted 8 lines
So I try two methods.
- add throttled dl
sched_entityto athrottled_list, the list will be traversed during cpu hotplug, and the dlsched_entitywill be picked and enqueue, then stop task will pick and migrate it. However, dlsched_entityis throttled again before stop task running since the below path. This path will setrq->online0 which lead toset_rq_offline()won't be called in functionmigration_call().
Quoted 2 lines
This seems wrong to me; this screws around with the CBS by replenishing too soon.
Agreed.
Quoted 20 lines
Quoted 20 lines
(quoted diff) +11
-0@@ -1593,9 +1602,20 @@ static void rq_online_dl(struct rq *rq) /* Assumes rq->lock is held */ static void rq_offline_dl(struct rq *rq) { + struct task_struct *p, *n; + if (rq->dl.overloaded) dl_clear_overload(rq); + /* Make sched_dl_entity available for pick_next_task() */ + list_for_each_entry_safe(p, n, &rq->dl.throttled_list, dl.throttled_node) { + p->dl.dl_throttled = 0; + hrtimer_cancel(&p->dl.dl_timer); + p->dl.dl_runtime = p->dl.dl_runtime; + if (task_on_rq_queued(p)) + enqueue_task_dl(rq, p, ENQUEUE_REPLENISH); + } + cpudl_set(&rq->rd->cpudl, rq->cpu, 0, 0);
Quoted 8 lines
Review static void rq_online_dl(struct rq *rq) · line 1620
Quoted 1 line
(quoted diff) +0
-0}So what is wrong with making
dl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).
Do you mean what I need to do is push the task to another cpu in dl_task_timer() if rq is offline? In addition, what will happen if dl task can't preempt on another cpu?
Regards,
Wanpeng Li
Peter Zijlstra
maintainerQuoted 11 lines · Wanpeng Li
Quoted 6 lines · Peter Zijlstra
On Fri, Oct 31, 2014 at 03:28:17PM +0800, Wanpeng Li wrote:
So what is wrong with makingdl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).Do you mean what I need to do is push the task to another cpu in
dl_task_timer()if rq is offline?
That does indeed appear to be the sensible fix to me.
Quoted 2 lines
In addition, what will happen if dl task can't preempt on another cpu?
So if we find that the rq the task was on is no longer available, we need to select a new rq, the 'right' rq would be the one running the latest deadline.
If it cannot preempt the latest (running) deadline, it was not eligible for running in the first place so no worries, right?
Hi Peter,
Quoted 18 lines · Peter Zijlstra
Quoted 11 lines · Wanpeng Li
Quoted 6 lines · Peter Zijlstra
On Fri, Oct 31, 2014 at 03:28:17PM +0800, Wanpeng Li wrote:
So what is wrong with makingdl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).Do you mean what I need to do is push the task to another cpu in
dl_task_timer()if rq is offline?That does indeed appear to be the sensible fix to me.
Quoted 2 lines
In addition, what will happen if dl task can't preempt on another cpu?
Quoted 6 lines
So if we find that the rq the task was on is no longer available, we need to select a new rq, the 'right' rq would be the one running the latest deadline.
If it cannot preempt the latest (running) deadline, it was not eligible for running in the first place so no worries, right?
I think this will lead to this deadline task cannot running on any rqs any more. If my understanding is not right, when it will be picked?
Regards,
Wanpeng Li
Hi,
Quoted 24 lines · Wanpeng Li
Hi Peter,
Quoted 22 lines · Peter Zijlstra
Quoted 11 lines · Wanpeng Li
Quoted 6 lines · Peter Zijlstra
On Fri, Oct 31, 2014 at 03:28:17PM +0800, Wanpeng Li wrote:
So what is wrong with makingdl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).Do you mean what I need to do is push the task to another cpu in
dl_task_timer()if rq is offline?That does indeed appear to be the sensible fix to me.
Quoted 2 lines
In addition, what will happen if dl task can't preempt on another cpu?
So if we find that the rq the task was on is no longer available, we need to select a new rq, the 'right' rq would be the one running the latest deadline.
Quoted 6 lines
Quoted 2 lines
If it cannot preempt the latest (running) deadline, it was not eligible for running in the first place so no worries, right?
I think this will lead to this deadline task cannot running on any rqs any more. If my understanding is not right, when it will be picked?
I think you can just pick one random rq and move the task there. The push/pull mechanism should then move it around properly.
Thanks,
- Juri
Peter Zijlstra
maintainerQuoted 23 lines · Wanpeng Li
Quoted 22 lines · Peter Zijlstra
Quoted 11 lines · Wanpeng Li
Quoted 6 lines · Peter Zijlstra
On Fri, Oct 31, 2014 at 03:28:17PM +0800, Wanpeng Li wrote:
So what is wrong with makingdl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).Do you mean what I need to do is push the task to another cpu in
dl_task_timer()if rq is offline?That does indeed appear to be the sensible fix to me.
Quoted 2 lines
In addition, what will happen if dl task can't preempt on another cpu?
So if we find that the rq the task was on is no longer available, we need to select a new rq, the 'right' rq would be the one running the latest deadline.
Quoted 5 lines
Quoted 2 lines
If it cannot preempt the latest (running) deadline, it was not eligible for running in the first place so no worries, right?
I think this will lead to this deadline task cannot running on any rqs any more. If my understanding is not right, when it will be picked?
So you unconditionally place it on the rq with the latest deadline. If it cannot preempt, at least its on an online cpu. It will get scheduled whenever its deadline is one of the N earliest, with N the number of online CPUs.
Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Wanpeng Li
Quoted 18 lines · Peter Zijlstra
Quoted 9 lines · Wanpeng Li
Quoted 6 lines · Peter Zijlstra
On Fri, Oct 31, 2014 at 03:28:17PM +0800, Wanpeng Li wrote:
So what is wrong with makingdl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).Do you mean what I need to do is push the task to another cpu in
dl_task_timer()if rq is offline?That does indeed appear to be the sensible fix to me.
Quoted 2 lines
In addition, what will happen if dl task can't preempt on another cpu?
So if we find that the rq the task was on is no longer available, we need to select a new rq, the 'right' rq would be the one running the latest deadline.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
If it cannot preempt the latest (running) deadline, it was not eligible for running in the first place so no worries, right?
I think this will lead to this deadline task cannot running on any rqs any more. If my understanding is not right, when it will be picked?
So you unconditionally place it on the rq with the latest deadline. If it cannot preempt, at least its on an online cpu. It will get scheduled whenever its deadline is one of the N earliest, with N the number of online CPUs.
Got it. Thanks for everybody's reply. ;-) I will make a patch tomorrow.
Regards,
Wanpeng Li
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi Peter,
Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Wanpeng Li
Quoted 18 lines · Peter Zijlstra
Quoted 9 lines · Wanpeng Li
Quoted 6 lines · Peter Zijlstra
On Fri, Oct 31, 2014 at 03:28:17PM +0800, Wanpeng Li wrote:
So what is wrong with makingdl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).Do you mean what I need to do is push the task to another cpu in
dl_task_timer()if rq is offline?That does indeed appear to be the sensible fix to me.
Quoted 2 lines
In addition, what will happen if dl task can't preempt on another cpu?
So if we find that the rq the task was on is no longer available, we need to select a new rq, the 'right' rq would be the one running the latest deadline.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
If it cannot preempt the latest (running) deadline, it was not eligible for running in the first place so no worries, right?
I think this will lead to this deadline task cannot running on any rqs any more. If my understanding is not right, when it will be picked?
So you unconditionally place it on the rq with the latest deadline. If it cannot preempt, at least its on an online cpu. It will get scheduled whenever its deadline is one of the N earliest, with N the number of online CPUs.
If something like this make sense?
(Will test it tomorrow and send out a formal one)
kernel/sched/deadline.c +0 -0
@@ -553,6 +553,41 @@ again:
push_dl_task(rq);
#endif
}
+
+ /*
+ * So if we find that the rq the task was on is no longer
+ * available, we need to select a new rq, the 'right' rq
+ * would be the one running the latest deadline.
+ *
+ * If it cannot preempt, at least it's on an online cpu. It
+ * will get scheduled whenever its deadline is one of the N
+ * earliest, with N the number of online CPUs.
+ */
+
+ if (!rq->online) {
+ struct rq *latest_rq = NULL;
+ int cpu;
+ u64 dmin = LONG_MAX;
+
+ for_each_cpu(cpu, &p->cpus_allowed)
+ if (cpu_online(cpu) &&
+ cpu_rq(cpu)->dl.earliest_dl.curr < dmin) {
+ latest_rq = cpu_rq(cpu);
+ dmin = latest_rq->dl.earliest_dl.curr;
+ }
+
+ if (!latest_rq)
+ goto unlock;
+
+ raw_spin_lock(&latest_rq->lock);
+
+ deactivate_task(rq, p, 0);
+ set_task_cpu(p, latest_rq->cpu);
+ activate_task(latest_rq, p, 0);
+
+ raw_spin_unlock(&latest_rq->lock);
+ }
+
unlock:raw_spin_unlock(&rq->lock);
Regards,
Wanpeng Li
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 77 lines · Wanpeng Li
Hi Peter,
Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Wanpeng Li
Quoted 18 lines · Peter Zijlstra
Quoted 9 lines · Wanpeng Li
Quoted 6 lines · Peter Zijlstra
On Fri, Oct 31, 2014 at 03:28:17PM +0800, Wanpeng Li wrote:
So what is wrong with makingdl_task_timer()deal with it? The timer will still fire on the correct time, canceling it and or otherwise messing with the CBS is wrong. Once it fires, all we need to do is migrate it to another cpu (preferably one that is still online of course :-).Do you mean what I need to do is push the task to another cpu in
dl_task_timer()if rq is offline?That does indeed appear to be the sensible fix to me.
Quoted 2 lines
In addition, what will happen if dl task can't preempt on another cpu?
So if we find that the rq the task was on is no longer available, we need to select a new rq, the 'right' rq would be the one running the latest deadline.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
If it cannot preempt the latest (running) deadline, it was not eligible for running in the first place so no worries, right?
I think this will lead to this deadline task cannot running on any rqs any more. If my understanding is not right, when it will be picked?
So you unconditionally place it on the rq with the latest deadline. If it cannot preempt, at least its on an online cpu. It will get scheduled whenever its deadline is one of the N earliest, with N the number of online CPUs.
If something like this make sense?
(Will test it tomorrow and send out a formal one)
kernel/sched/deadline.c +35
-0@@ -553,6 +553,41 @@ again:push_dl_task (rq);#endif } + +/* +* So if we find that the rq the task was on is no longer +* available, we need to select a new rq, the 'right' rq +* would be the one running the latest deadline. +* +* If it cannot preempt, at least it's on an online cpu. It +* will get scheduled whenever its deadline is one of the N +* earliest, with N the number of online CPUs. +*/ + +if (!rq->online) { +struct rq *latest_rq =NULL ; +int cpu; +u64 dmin =LONG_MAX ; + +for_each_cpu (cpu, &p->cpus_allowed) +if (cpu_online (cpu) && +cpu_rq (cpu)->dl.earliest_dl.curr < dmin) { + latest_rq =cpu_rq (cpu); + dmin = latest_rq->dl.earliest_dl.curr; + } + +if (!latest_rq) + goto unlock; + +raw_spin_lock (&latest_rq->lock); + +deactivate_task (rq, p,0 ); +set_task_cpu (p, latest_rq->cpu); +activate_task (latest_rq, p,0 ); + +raw_spin_unlock (&latest_rq->lock); + } +
Quoted 2 lines
unlock:
raw_spin_unlock(&rq->lock);
wow, sorry for mess format.
Quoted 10 lines
Regards,
Wanpeng LiQuoted 5 lines
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Peter Zijlstra
maintainerQuoted 11 lines · Wanpeng Li
+ if (!rq->online) {
+ struct rq*latest_rq= NULL;
+ int cpu;
+ u64 dmin =LONG_MAX;
+
+for_each_cpu(cpu, &p->cpus_allowed)
+ if (cpu_online(cpu)&&
+ cpu_rq(cpu)->dl.earliest_dl.curr < dmin) {
+latest_rq=cpu_rq(cpu);
+ dmin =latest_rq->dl.earliest_dl.curr;
+ }
I would have expected something using find_later_rq(), but I might be mistaken, I'll let Juri suggest something.
Quoted 15 lines
(quoted diff) +13
-0+ + if (!latest_rq) + goto unlock; + + raw_spin_lock(&latest_rq->lock); + + deactivate_task(rq, p, 0); + set_task_cpu(p, latest_rq->cpu); + activate_task(latest_rq, p, 0); + + raw_spin_unlock(&latest_rq->lock); + } +unlock:
raw_spin_unlock(&rq->lock);
Hi,
Quoted 15 lines · Peter Zijlstra
Quoted 11 lines · Wanpeng Li
+ if (!rq->online) {
+ struct rq*latest_rq= NULL;
+ int cpu;
+ u64 dmin =LONG_MAX;
+
+for_each_cpu(cpu, &p->cpus_allowed)
+ if (cpu_online(cpu)&&
+ cpu_rq(cpu)->dl.earliest_dl.curr < dmin) {
+latest_rq=cpu_rq(cpu);
+ dmin =latest_rq->dl.earliest_dl.curr;
+ }
Quoted 3 lines
I would have expected something using
find_later_rq(), but I might be mistaken, I'll let Juri suggest something.
Yeah, we should be able to reuse something that we already have. I'm actually sorry that I'm not responsive on this, but I'm really busy on other things this week. I hope to be able to find some time soon to test this all.
Thanks,
- Juri
Quoted 16 lines
Quoted 15 lines
(quoted diff) +13
-0+ + if (!latest_rq) + goto unlock; + + raw_spin_lock(&latest_rq->lock); + + deactivate_task(rq, p, 0); + set_task_cpu(p, latest_rq->cpu); + activate_task(latest_rq, p, 0); + + raw_spin_unlock(&latest_rq->lock); + } +unlock:
raw_spin_unlock(&rq->lock);
Hi Juri,
Quoted 17 lines · Juri Lelli
Hi,
Quoted 14 lines · Peter Zijlstra
Quoted 11 lines · Wanpeng Li
+ if (!rq->online) {
+ struct rq*latest_rq= NULL;
+ int cpu;
+ u64 dmin =LONG_MAX;
+
+for_each_cpu(cpu, &p->cpus_allowed)
+ if (cpu_online(cpu)&&
+ cpu_rq(cpu)->dl.earliest_dl.curr < dmin) {
+latest_rq=cpu_rq(cpu);
+ dmin =latest_rq->dl.earliest_dl.curr;
+ }
Quoted 7 lines
Quoted 3 lines
I would have expected something using
find_later_rq(), but I might be mistaken, I'll let Juri suggest something.Yeah, we should be able to reuse something that we already have. I'm actually sorry that I'm not responsive on this, but I'm really busy on other things this week. I hope to be able to find some time soon to test this all.
I have time to do it, your proposal is a great welcome. ;-)
Regards,
Wanpeng Li
Quoted 20 lines
Thanks,
- Juri
Quoted 15 lines
Quoted 15 lines
(quoted diff) +13
-0+ + if (!latest_rq) + goto unlock; + + raw_spin_lock(&latest_rq->lock); + + deactivate_task(rq, p, 0); + set_task_cpu(p, latest_rq->cpu); + activate_task(latest_rq, p, 0); + + raw_spin_unlock(&latest_rq->lock); + } +unlock:
raw_spin_unlock(&rq->lock);