sched/fair: Fix that tasks are not constrained by cfs_b->quota on hotplug core, when hotplug core is offline and then online.
Quoted 84 lines
Quoted 60 lines · Peter Zijlstra
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.
(also, why send it twice?)
Quoted 6 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.Quoted 7 lines
The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.
(also, why send it twice?)
Quoted 31 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()registerscfs_b->quotaoncfs_rq->runtime_enabledof all leafcfs_rqon runqueue. However, because this is beforeenqueue_entity()adds&cfs_rq->leaf_cfs_rq_liston &rq_of(cfs_rq)->leaf_cfs_rq_list,cfs->quotais not register oncfs_rq->runtime_enabled.To resolve this problem, this patch makes
update_runtime_enabled()registerscfs_b->quotaby usingwalk_tg_tree_from().(quoted diff) +6
-0+static int __maybe_unused __update_runtime_enabled(struct task_group *tg, void *data) { + struct rq *rq = data; + struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; + struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth; + raw_spin_lock(&cfs_b->lock); + raw_spin_unlock(&cfs_b->lock);Quoted 15 lines
Review
Quoted 12 lines
(quoted diff) +11
-0+ return 0; +} + +static void __maybe_unused update_runtime_enabled(struct rq *rq) +{ + struct cfs_rq *cfs_rq = &rq->cfs; + + /* register cfs_b->quota on the whole tg tree */ + rcu_read_lock(); + walk_tg_tree_from(cfs_rq->tg, __update_runtime_enabled, tg_nop, (void *)rq); + rcu_read_unlock(); }Looks ok, performance on hotplug doesn't really matter. Ben, you happy with this?
I'm not 100% sure about the exact timings and mechanics of hotplug, but cfs-bandwidth wise this is ok. We may still have
runtime_remaining= 1, or we may have < 0 and yet be unthrottled, but either case is ok, even if hotplug allows tasks to have migrated here already (I'm not sure, looking at the code).
Quoted 3 lines
Now that I check again you can just loop over the list of tgs rather than the hierarchical
walk_tg_tree_from, but there's certainly no harm in it.
Ben,
Is there additional revision which I have to do?
If so, could you let me know about that?
Regards,
Jeehong Kim
Quoted 88 lines · Jeehong Kim
Quoted 84 lines
Quoted 60 lines · Peter Zijlstra
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.
(also, why send it twice?)
Quoted 6 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.Quoted 7 lines
The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.
(also, why send it twice?)
Quoted 31 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()registerscfs_b->quotaoncfs_rq->runtime_enabledof all leafcfs_rqon runqueue. However, because this is beforeenqueue_entity()adds&cfs_rq->leaf_cfs_rq_liston &rq_of(cfs_rq)->leaf_cfs_rq_list,cfs->quotais not register oncfs_rq->runtime_enabled.To resolve this problem, this patch makes
update_runtime_enabled()registerscfs_b->quotaby usingwalk_tg_tree_from().(quoted diff) +6
-0+static int __maybe_unused __update_runtime_enabled(struct task_group *tg, void *data) { + struct rq *rq = data; + struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; + struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth; + raw_spin_lock(&cfs_b->lock); + raw_spin_unlock(&cfs_b->lock);Quoted 15 lines
Review
Quoted 12 lines
(quoted diff) +11
-0+ return 0; +} + +static void __maybe_unused update_runtime_enabled(struct rq *rq) +{ + struct cfs_rq *cfs_rq = &rq->cfs; + + /* register cfs_b->quota on the whole tg tree */ + rcu_read_lock(); + walk_tg_tree_from(cfs_rq->tg, __update_runtime_enabled, tg_nop, (void *)rq); + rcu_read_unlock(); }Looks ok, performance on hotplug doesn't really matter. Ben, you happy with this?
I'm not 100% sure about the exact timings and mechanics of hotplug, but cfs-bandwidth wise this is ok. We may still have
runtime_remaining= 1, or we may have < 0 and yet be unthrottled, but either case is ok, even if hotplug allows tasks to have migrated here already (I'm not sure, looking at the code).Quoted 3 lines
Now that I check again you can just loop over the list of tgs rather than the hierarchical
walk_tg_tree_from, but there's certainly no harm in it.
Quoted 7 lines
Ben,
Is there additional revision which I have to do?
If so, could you let me know about that?Regards,
Jeehong Kim
Oh, no, this is fine by me.
Quoted 85 lines · On 2016년 09월 23일 01:53, [email protected] wrote:
Quoted 83 lines · Jeehong Kim
Quoted 83 lines
Quoted 58 lines · Peter Zijlstra
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.
(also, why send it twice?)
Quoted 6 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.Quoted 7 lines
The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.(also, why send it twice?)
Quoted 30 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()registerscfs_b->quotaoncfs_rq->runtime_enabledof all leafcfs_rqon runqueue. However, because this is beforeenqueue_entity()adds&cfs_rq->leaf_cfs_rq_liston &rq_of(cfs_rq)->leaf_cfs_rq_list,cfs->quotais not register oncfs_rq->runtime_enabled.To resolve this problem, this patch makes
update_runtime_enabled()registerscfs_b->quotaby usingwalk_tg_tree_from().(quoted diff) +6
-0+static int __maybe_unused __update_runtime_enabled(struct task_group *tg, void *data) { + struct rq *rq = data; + struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; + struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth; + raw_spin_lock(&cfs_b->lock); + raw_spin_unlock(&cfs_b->lock);Quoted 14 lines
Review
Quoted 12 lines
(quoted diff) +11
-0+ return 0; +} + +static void __maybe_unused update_runtime_enabled(struct rq *rq) +{ + struct cfs_rq *cfs_rq = &rq->cfs; + + /* register cfs_b->quota on the whole tg tree */ + rcu_read_lock(); + walk_tg_tree_from(cfs_rq->tg, __update_runtime_enabled, tg_nop, (void *)rq); + rcu_read_unlock(); }Looks ok, performance on hotplug doesn't really matter. Ben, you happy with this?
I'm not 100% sure about the exact timings and mechanics of hotplug, but cfs-bandwidth wise this is ok. We may still have
runtime_remaining= 1, or we may have < 0 and yet be unthrottled, but either case is ok, even if hotplug allows tasks to have migrated here already (I'm not sure, looking at the code).Now that I check again you can just loop over the list of tgs rather than the hierarchical
walk_tg_tree_from, but there's certainly no harm in it.
Quoted 11 lines
Quoted 7 lines
Ben,
Is there additional revision which I have to do?
If so, could you let me know about that?Regards,
Jeehong KimOh, no, this is fine by me.
Ben,
If this is fine to you, could you sign off on this patch?
Regards,
Jeehong Kim.
[email protected]
Reviewed-byQuoted 98 lines · Jeehong Kim
Quoted 85 lines · On 2016년 09월 23일 01:53, [email protected] wrote:
Quoted 83 lines · Jeehong Kim
Quoted 83 lines
Quoted 58 lines · Peter Zijlstra
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.
(also, why send it twice?)
Quoted 6 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.Quoted 7 lines
The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
You forgot to Cc Ben, who gave you feedback on v1, which is rather poor style. Also, I don't see how kernel-janitors is relevant to this patch. This is very much not a janitorial thing.(also, why send it twice?)
Quoted 30 lines · Jeehong Kim
In case that
CONFIG_HOTPLUG_CPUandCONFIG_CFS_BANDWIDTHis turned on and tasks in bandwidth controlled task group run on hotplug core, the tasks are not controlled bycfs_b->quotawhen hotplug core is offline and then online. The remaining tasks in task group consume all ofcfs_b->quotaon other cores.The cause of this problem is described as below:
1. When hotplug core is offline while tasks in task group run on hotplug core,
unregister_fair_sched_group()deletesleaf_cfs_rq_listof tg->cfs_rq[cpu] from &rq_of(cfs_rq)->leaf_cfs_rq_list.2. Then, when hotplug core is online,
update_runtime_enabled()registerscfs_b->quotaoncfs_rq->runtime_enabledof all leafcfs_rqon runqueue. However, because this is beforeenqueue_entity()adds&cfs_rq->leaf_cfs_rq_liston &rq_of(cfs_rq)->leaf_cfs_rq_list,cfs->quotais not register oncfs_rq->runtime_enabled.To resolve this problem, this patch makes
update_runtime_enabled()registerscfs_b->quotaby usingwalk_tg_tree_from().(quoted diff) +6
-0+static int __maybe_unused __update_runtime_enabled(struct task_group *tg, void *data) { + struct rq *rq = data; + struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; + struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth; + raw_spin_lock(&cfs_b->lock); + raw_spin_unlock(&cfs_b->lock);Quoted 14 lines
Review
Quoted 12 lines
(quoted diff) +11
-0+ return 0; +} + +static void __maybe_unused update_runtime_enabled(struct rq *rq) +{ + struct cfs_rq *cfs_rq = &rq->cfs; + + /* register cfs_b->quota on the whole tg tree */ + rcu_read_lock(); + walk_tg_tree_from(cfs_rq->tg, __update_runtime_enabled, tg_nop, (void *)rq); + rcu_read_unlock(); }Looks ok, performance on hotplug doesn't really matter. Ben, you happy with this?
I'm not 100% sure about the exact timings and mechanics of hotplug, but cfs-bandwidth wise this is ok. We may still have
runtime_remaining= 1, or we may have < 0 and yet be unthrottled, but either case is ok, even if hotplug allows tasks to have migrated here already (I'm not sure, looking at the code).Now that I check again you can just loop over the list of tgs rather than the hierarchical
walk_tg_tree_from, but there's certainly no harm in it.Quoted 11 lines
Quoted 7 lines
Ben,
Is there additional revision which I have to do?
If so, could you let me know about that?Regards,
Jeehong KimOh, no, this is fine by me.
Quoted 6 lines
Ben,
If this is fine to you, could you sign off on this patch?
Regards,
Jeehong Kim.
(My understanding of signed-off-by is that I don't do that in this case)