sched/fair: add comments for group_type and balancing at SD_NUMA level
Vincent Guittot
maintainerPATCH 1/1 sched/fair: add comments for group_type and balancing at SD_NUMA level
Add comments to describe each state of goup_type and to add some details about the load balance at NUMA level.
--- kernel/sched/fair.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
kernel/sched/fair.c +28 -1
@@ -6986,11 +6986,34 @@ enum fbq_type { regular, remote, all };* group. see update_sd_pick_busiest(). */ enum group_type { +/* +* The group has spare capacity that can be used to process more work. +*/ group_has_spare =0 , +/* +* The group is fully used and the tasks don't compete for more CPU +* cycles. Nevetheless, some tasks might wait before running. +*/ group_fully_busy, +/* +* One task doesn't fit with CPU's capacity and must be migrated on a +* more powerful CPU. +*/ group_misfit_task, +/* +* One local CPU with higher capacity is available and task should be +* migrated on it instead on current CPU. +*/ group_asym_packing, +/* +* The tasks affinity prevents the scheduler to balance the load across +* the system. +*/ group_imbalanced, +/* +* The CPU is overloaded and can't provide expected CPU cycles to all +* tasks. +*/ group_overloaded };
@@ -8608,7 +8631,11 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s/* * Try to use spare capacity of local group without overloading it or -* emptying busiest +* emptying busiest. +* XXX Spreading tasks across numa nodes is not always the best policy +* and special cares should be taken for SD_NUMA domain level before +* spreading the tasks. For now, load_balance() fully relies on +* NUMA_BALANCING and fbq_classify_group/rq to overide the decision. */ if (local->group_type == group_has_spare) {if (busiest->group_type > group_fully_busy) {
Signature · 1 line
2.7.4
Hi Vincent,
Quoted 4 lines · Vincent Guittot
Add comments to describe each state of
goup_typeand to add some details about the load balance at NUMA level.Signed-off-by Vincent Guittot <[email protected]>
Suggestions/nits below. There's a bit of duplication with existing comments (e.g. the nice blob atop sg_imbalanced()), but I think it can't hurt to have the few extra lines you're introducing.
---
kernel/sched/fair.c +8 -10
@@ -6955,28 +6955,26 @@ enum fbq_type { regular, remote, all };* group. see update_sd_pick_busiest(). */ enum group_type { -/* -* The group has spare capacity that can be used to process more work. -*/ +/* The group isn't significantly pressured and can be used to process more work */ group_has_spare =0 ,/* * The group is fully used and the tasks don't compete for more CPU -* cycles. Nevetheless, some tasks might wait before running. +* cycles. Nevertheless, some tasks might wait before running. */ group_fully_busy,/* -* One task doesn't fit with CPU's capacity and must be migrated on a -* more powerful CPU. +* (SD_ASYM_CPUCAPACITY only) One task doesn't fit on its CPU's +* capacity and must be migrated to a CPU of higher capacity. */ group_misfit_task,/* -* One local CPU with higher capacity is available and task should be -* migrated on it instead on current CPU. +* (SD_ASYM_PACKING only) One local CPU with higher capacity is +* available and task should be migrated to it. */ group_asym_packing,/* -* The tasks affinity prevents the scheduler to balance the load across -* the system. +* The tasks affinity previously prevented the scheduler from balancing +* load across the system. */ group_imbalanced,/*
PATCH v2 1/1 sched/fair: add comments for group_type and balancing at SD_NUMA level
Quoted 43 lines · Valentin Schneider
Hi Vincent,
Quoted 4 lines · Vincent Guittot
Add comments to describe each state of
goup_typeand to add some details about the load balance at NUMA level.Signed-off-by Vincent Guittot <[email protected]>Suggestions/nits below. There's a bit of duplication with existing comments (e.g. the nice blob atop
sg_imbalanced()), but I think it can't hurt to have the few extra lines you're introducing.---
kernel/sched/fair.c +5
-8@@ -6955,28 +6955,26 @@ enum fbq_type { regular, remote, all };* group. see update_sd_pick_busiest(). */ enum group_type { -/* -* The group has spare capacity that can be used to process more work. -*/ +/* The group isn't significantly pressured and can be used to process more work */ group_has_spare =0 ,/* * The group is fully used and the tasks don't compete for more CPU -* cycles. Nevetheless, some tasks might wait before running. +* cycles. Nevertheless, some tasks might wait before running. */ group_fully_busy,/* -* One task doesn't fit with CPU's capacity and must be migrated on a -* more powerful CPU. +* (SD_ASYM_CPUCAPACITY only) One task doesn't fit on its CPU's +* capacity and must be migrated to a CPU of higher capacity. */ group_misfit_task,/* -* One local CPU with higher capacity is available and task should be -* migrated on it instead on current CPU. +* (SD_ASYM_PACKING only) One local CPU with higher capacity is
kernel/sched/fair.c enum fbq_type { regular, remote, all }; · line 6979
Quoted 10 lines
kernel/sched/fair.c +3
-2+* available and task should be migrated to it. */ group_asym_packing,/* -* The tasks affinity prevents the scheduler to balance the load across -* the system. +* The tasks affinity previously prevented the scheduler from balancing +* load across the system. */ group_imbalanced,
Thanks - I did a few more fixes and updates to the comments, this is how it ended up looking like (full patch below):
/*
* 'group_type' describes the group of CPUs at the moment of load balancing. *
* The enum is ordered by pulling priority, with the group with lowest priority
* first so the group_type can simply be compared when selecting the busiest
* group. See update_sd_pick_busiest().
*/
enum group_type {
/* The group has spare capacity that can be used to run more tasks. */ group_has_spare = 0,
/*
* The group is fully used and the tasks don't compete for more CPU
* cycles. Nevertheless, some tasks might wait before running. */
group_fully_busy,
/*
* SD_ASYM_CPUCAPACITY only: One task doesn't fit with CPU's capacity
* and must be migrated to a more powerful CPU.
*/
group_misfit_task,
/*
* SD_ASYM_PACKING only: One local CPU with higher capacity is available,
* and the task should be migrated to it instead of running on the
* current CPU.
*/
group_asym_packing,
/*
* The tasks' affinity constraints previously prevented the scheduler
* from balancing the load across the system.
*/
group_imbalanced,
/*
* The CPU is overloaded and can't provide expected CPU cycles to all
* tasks.
*/
group_overloaded
};
I also added your Acked-by, which I think was implicit? :)
Thanks,
Ingo
=====>
From: Vincent Guittot <[email protected]>
Date: Tue, 12 Nov 2019 15:50:43 +0100
Subject: [PATCH] sched/fair: Add comments for group_type and balancing at SD_NUMA level
Add comments to describe each state of goup_type and to add some details about the load balance at NUMA level.
[ Valentin Schneider: Updates to the comments. ]
[ mingo: Other updates to the comments. ]
--- kernel/sched/fair.c | 35 +++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-)
kernel/sched/fair.c +31 -4
@@ -6980,17 +6980,40 @@ static unsigned long __read_mostly max_load_balance_interval = HZ/10;enum fbq_type { regular, remote, all };/* -* group_type describes the group of CPUs at the moment of the load balance. +* 'group_type' describes the group of CPUs at the moment of load balancing. +* * The enum is ordered by pulling priority, with the group with lowest priority -* first so the groupe_type can be simply compared when selecting the busiest -* group. see update_sd_pick_busiest(). +* first so the group_type can simply be compared when selecting the busiest +* group. See update_sd_pick_busiest(). */ enum group_type { +/* The group has spare capacity that can be used to run more tasks. */ group_has_spare =0 , +/* +* The group is fully used and the tasks don't compete for more CPU +* cycles. Nevertheless, some tasks might wait before running. +*/ group_fully_busy, +/* +* SD_ASYM_CPUCAPACITY only: One task doesn't fit with CPU's capacity +* and must be migrated to a more powerful CPU. +*/ group_misfit_task, +/* +* SD_ASYM_PACKING only: One local CPU with higher capacity is available, +* and the task should be migrated to it instead of running on the +* current CPU. +*/ group_asym_packing, +/* +* The tasks' affinity constraints previously prevented the scheduler +* from balancing the load across the system. +*/ group_imbalanced, +/* +* The CPU is overloaded and can't provide expected CPU cycles to all +* tasks. +*/ group_overloaded };
@@ -8589,7 +8612,11 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s/* * Try to use spare capacity of local group without overloading it or -* emptying busiest +* emptying busiest. +* XXX Spreading tasks across NUMA nodes is not always the best policy +* and special care should be taken for SD_NUMA domain level before +* spreading the tasks. For now, load_balance() fully relies on +* NUMA_BALANCING and fbq_classify_group/rq to override the decision. */ if (local->group_type == group_has_spare) {if (busiest->group_type > group_fully_busy) {
Quoted 3 lines · Ingo Molnar
Thanks - I did a few more fixes and updates to the comments, this is how it ended up looking like (full patch below):
[...]
LGTM, thanks!
Quoted 2 lines
I also added your Acked-by, which I think was implicit? :)
Hah, I'm not used to handing those out, but sure!
[tip: sched/core] sched/fair: Add comments for group_type and balancing at SD_NUMA level
In reply to Vincent GuittotThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: a9723389cc759c891d481de271ac73eeaa123bcb Gitweb: https://git.kernel.org/tip/a9723389cc759c891d481de271ac73eeaa123bcb Author: Vincent Guittot <[email protected]> AuthorDate: Tue, 12 Nov 2019 15:50:43 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 18 Nov 2019 14:33:12 +01:00
sched/fair: Add comments for group_type and balancing at SD_NUMA level
Add comments to describe each state of goup_type and to add some details about the load balance at NUMA level.
[ Valentin Schneider: Updates to the comments. ]
[ mingo: Other updates to the comments. ]
--- kernel/sched/fair.c | 35 +++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-)
kernel/sched/fair.c +31 -4
@@ -6980,17 +6980,40 @@ static unsigned long __read_mostly max_load_balance_interval = HZ/10;enum fbq_type { regular, remote, all };/* -* group_type describes the group of CPUs at the moment of the load balance. +* 'group_type' describes the group of CPUs at the moment of load balancing. +* * The enum is ordered by pulling priority, with the group with lowest priority -* first so the groupe_type can be simply compared when selecting the busiest -* group. see update_sd_pick_busiest(). +* first so the group_type can simply be compared when selecting the busiest +* group. See update_sd_pick_busiest(). */ enum group_type { +/* The group has spare capacity that can be used to run more tasks. */ group_has_spare =0 , +/* +* The group is fully used and the tasks don't compete for more CPU +* cycles. Nevertheless, some tasks might wait before running. +*/ group_fully_busy, +/* +* SD_ASYM_CPUCAPACITY only: One task doesn't fit with CPU's capacity +* and must be migrated to a more powerful CPU. +*/ group_misfit_task, +/* +* SD_ASYM_PACKING only: One local CPU with higher capacity is available, +* and the task should be migrated to it instead of running on the +* current CPU. +*/ group_asym_packing, +/* +* The tasks' affinity constraints previously prevented the scheduler +* from balancing the load across the system. +*/ group_imbalanced, +/* +* The CPU is overloaded and can't provide expected CPU cycles to all +* tasks. +*/ group_overloaded };
@@ -8589,7 +8612,11 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s/* * Try to use spare capacity of local group without overloading it or -* emptying busiest +* emptying busiest. +* XXX Spreading tasks across NUMA nodes is not always the best policy +* and special care should be taken for SD_NUMA domain level before +* spreading the tasks. For now, load_balance() fully relies on +* NUMA_BALANCING and fbq_classify_group/rq to override the decision. */ if (local->group_type == group_has_spare) {if (busiest->group_type > group_fully_busy) {