SCHED_IDLE extensions
v3 cover SCHED_IDLE extensions
This patch series contains improvements/extensions for SCHED_IDLE.
The first patch of the series is the previously mailed patch to add cgroup support for SCHED_IDLE.
The second patch adds some additional idle accounting.
The third and fourth patches change some idle interactions.
Josh Don (4):
sched: cgroup SCHED_IDLE support
sched: account number of SCHED_IDLE entities on each cfs_rq
sched: reduce sched slice for SCHED_IDLE entities
sched: adjust sleeper credit for SCHED_IDLE entities
kernel/sched/core.c | 25 +++++ kernel/sched/debug.c | 7 ++ kernel/sched/fair.c | 256 +++++++++++++++++++++++++++++++++++++------ kernel/sched/sched.h | 10 ++ 4 files changed, 267 insertions(+), 31 deletions(-)
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
PATCH v3 1/4 sched: cgroup SCHED_IDLE support
This extends SCHED_IDLE to cgroups.
Interface: cgroup/cpu.idle.
0: default behavior
1: SCHED_IDLE
Extending SCHED_IDLE to cgroups means that we incorporate the existing aspects of SCHED_IDLE; a SCHED_IDLE cgroup will count all of its descendant threads towards the idle_h_nr_running count of all of its ancestor cgroups. Thus, sched_idle_rq() will work properly. Additionally, SCHED_IDLE cgroups are configured with minimum weight.
There are two key differences between the per-task and per-cgroup SCHED_IDLE interface:
- The cgroup interface allows tasks within a SCHED_IDLE hierarchy to maintain their relative weights. The entity that is "idle" is the cgroup, not the tasks themselves.
- Since the idle entity is the cgroup, our SCHED_IDLE wakeup preemption decision is not made by comparing the current task with the woken task, but rather by comparing their matching sched_entity.
A typical use-case for this is a user that creates an idle and a non-idle subtree. The non-idle subtree will dominate competition vs the idle subtree, but the idle subtree will still be high priority vs other users on the system. The latter is accomplished via comparing matching sched_entity in the waken preemption path (this could also be improved by making the sched_idle_rq() decision dependent on the perspective of a specific task).
For now, we maintain the existing SCHED_IDLE semantics. Future patches may make improvements that extend how we treat SCHED_IDLE entities.
The per-task_group idle field is an integer that currently only holds either a 0 or a 1. This is explicitly typed as an integer to allow for further extensions to this API. For example, a negative value may indicate a highly latency-sensitive cgroup that should be preferred for preemption/placement/etc.
---
v3:
- no change from v2
v2:
- Use WEIGHT_IDLEPRIO for the idle cgroup weight
- Add cgroup-v1 support
kernel/sched/core.c | 25 ++++++ kernel/sched/debug.c | 3 + kernel/sched/fair.c | 197 +++++++++++++++++++++++++++++++++++++------ kernel/sched/sched.h | 8 ++ 4 files changed, 208 insertions(+), 25 deletions(-)
kernel/sched/core.c +25 -0
@@ -10195,6 +10195,20 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, }#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_FAIR_GROUP_SCHED +static s64 cpu_idle_read_s64 (struct cgroup_subsys_state *css, +struct cftype *cft) +{ +return css_tg (css)->idle; +} + +static int cpu_idle_write_s64 (struct cgroup_subsys_state *css, +struct cftype *cft,s64 idle) +{ +return sched_group_set_idle (css_tg (css), idle); +} +#endif + static struct cftype cpu_legacy_files[] = {#ifdef CONFIG_FAIR_GROUP_SCHED {
@@ -10202,6 +10216,11 @@ static struct cftype cpu_legacy_files[] = { .read_u64 = cpu_shares_read_u64, .write_u64 = cpu_shares_write_u64, }, + { + .name = "idle", + .read_s64 = cpu_idle_read_s64, + .write_s64 = cpu_idle_write_s64, + },#endif #ifdef CONFIG_CFS_BANDWIDTH {
@@ -10409,6 +10428,12 @@ static struct cftype cpu_files[] = { .read_s64 = cpu_weight_nice_read_s64, .write_s64 = cpu_weight_nice_write_s64, }, + { + .name = "idle", + .flags = CFTYPE_NOT_ON_ROOT, + .read_s64 = cpu_idle_read_s64, + .write_s64 = cpu_idle_write_s64, + },#endif #ifdef CONFIG_CFS_BANDWIDTH {
kernel/sched/debug.c +3 -0
@@ -607,6 +607,9 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %d\n" ,"nr_spread_over" , cfs_rq->nr_spread_over);SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"idle_h_nr_running" , + cfs_rq->idle_h_nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);#ifdef CONFIG_SMP SEQ_printf(m, " .%-30s: %lu\n", "load_avg",
kernel/sched/fair.c +172 -25
335 lines not on this page · raw message
kernel/sched/sched.h +8 -0
@@ -396,6 +396,9 @@ struct task_group {struct cfs_rq **cfs_rq;unsigned long shares; +/* A positive value indicates that this is a SCHED_IDLE group. */ +int idle; +#ifdef CONFIG_SMP /* * load_avg can be heavily contended at clock tick time, so put
@@ -505,6 +508,8 @@ extern void sched_move_task(struct task_struct *tsk);#ifdef CONFIG_FAIR_GROUP_SCHED extern int sched_group_set_shares (struct task_group *tg,unsigned long shares); +extern int sched_group_set_idle (struct task_group *tg,long idle); +#ifdef CONFIG_SMP extern void set_task_rq_fair (struct sched_entity *se,struct cfs_rq *prev,struct cfs_rq *next);
@@ -601,6 +606,9 @@ struct cfs_rq {struct list_head leaf_cfs_rq_list;struct task_group *tg;/* group that "owns" this runqueue */ +/* Locally cached copy of our task_group's idle value */ +int idle; +#ifdef CONFIG_CFS_BANDWIDTH int runtime_enabled;s64 runtime_remaining;
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH v3 1/4] sched: cgroup SCHED_IDLE support
Hi Josh,
Quoted 63 lines · Josh Don
This extends
SCHED_IDLEto cgroups.Interface: cgroup/cpu.idle.
0: default behavior
1:SCHED_IDLEExtending
SCHED_IDLEto cgroups means that we incorporate the existing aspects ofSCHED_IDLE; aSCHED_IDLEcgroup will count all of its descendant threads towards theidle_h_nr_runningcount of all of its ancestor cgroups. Thus,sched_idle_rq()will work properly. Additionally,SCHED_IDLEcgroups are configured with minimum weight.There are two key differences between the per-task and per-cgroup
SCHED_IDLEinterface:- The cgroup interface allows tasks within a
SCHED_IDLEhierarchy to maintain their relative weights. The entity that is "idle" is the cgroup, not the tasks themselves.- Since the idle entity is the cgroup, our
SCHED_IDLEwakeup preemption decision is not made by comparing the current task with the woken task, but rather by comparing their matchingsched_entity.A typical use-case for this is a user that creates an idle and a non-idle subtree. The non-idle subtree will dominate competition vs the idle subtree, but the idle subtree will still be high priority vs other users on the system. The latter is accomplished via comparing matching
sched_entityin the waken preemption path (this could also be improved by making thesched_idle_rq()decision dependent on the perspective of a specific task).For now, we maintain the existing
SCHED_IDLEsemantics. Future patches may make improvements that extend how we treatSCHED_IDLEentities.The per-task_group idle field is an integer that currently only holds either a 0 or a 1. This is explicitly typed as an integer to allow for further extensions to this API. For example, a negative value may indicate a highly latency-sensitive cgroup that should be preferred for preemption/placement/etc.
Signed-off-by Josh Don <[email protected]>---
v3:
- no change from v2
v2:
- UseWEIGHT_IDLEPRIOfor the idle cgroup weight
- Add cgroup-v1 supportkernel/sched/core.c | 25 ++++++ kernel/sched/debug.c | 3 + kernel/sched/fair.c | 197 +++++++++++++++++++++++++++++++++++++------ kernel/sched/sched.h | 8 ++ 4 files changed, 208 insertions(+), 25 deletions(-)kernel/sched/core.c +1
-0@@ -10195,6 +10195,20 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, }#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_FAIR_GROUP_SCHED
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 10208
Quoted 10 lines
kernel/sched/core.c +10
-0+static s64 cpu_idle_read_s64 (struct cgroup_subsys_state *css, +struct cftype *cft) +{ +return css_tg (css)->idle; +} + +static int cpu_idle_write_s64 (struct cgroup_subsys_state *css, +struct cftype *cft,s64 idle) +{ +return sched_group_set_idle (css_tg (css), idle);
return sched_group_set_idle(css_tg(css), (int)idle);
Quoted 323 lines
kernel/sched/core.c +14
-0+} +#endif + static struct cftype cpu_legacy_files[] = {#ifdef CONFIG_FAIR_GROUP_SCHED {@@ -10202,6 +10216,11 @@ static struct cftype cpu_legacy_files[] = { .read_u64 = cpu_shares_read_u64, .write_u64 = cpu_shares_write_u64, }, + { + .name = "idle", + .read_s64 = cpu_idle_read_s64, + .write_s64 = cpu_idle_write_s64, + },#endif #ifdef CONFIG_CFS_BANDWIDTH {@@ -10409,6 +10428,12 @@ static struct cftype cpu_files[] = { .read_s64 = cpu_weight_nice_read_s64, .write_s64 = cpu_weight_nice_write_s64, }, + { + .name = "idle", + .flags = CFTYPE_NOT_ON_ROOT, + .read_s64 = cpu_idle_read_s64, + .write_s64 = cpu_idle_write_s64, + },#endif #ifdef CONFIG_CFS_BANDWIDTH {kernel/sched/debug.c +3
-0@@ -607,6 +607,9 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %d\n" ,"nr_spread_over" , cfs_rq->nr_spread_over);SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"idle_h_nr_running" , + cfs_rq->idle_h_nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);#ifdef CONFIG_SMP SEQ_printf(m, " .%-30s: %lu\n", "load_avg",kernel/sched/fair.c +101
-25@@ -431,6 +431,23 @@ find_matching_se(struct sched_entity **se, struct sched_entity **pse) } } +static int tg_is_idle (struct task_group *tg) +{ +return tg->idle >0 ; +} + +static int cfs_rq_is_idle (struct cfs_rq *cfs_rq) +{ +return cfs_rq->idle >0 ; +} + +static int se_is_idle (struct sched_entity *se) +{ +if (entity_is_task (se)) +return task_has_idle_policy (task_of (se)); +return cfs_rq_is_idle (group_cfs_rq (se)); +} +#else /* !CONFIG_FAIR_GROUP_SCHED */ #define for_each_sched_entity (se) \@@ -468,6 +485,21 @@ find_matching_se(struct sched_entity **se, struct sched_entity **pse) { } +static int tg_is_idle (struct task_group *tg) +{ +return 0 ; +} + +static int cfs_rq_is_idle (struct cfs_rq *cfs_rq) +{ +return 0 ; +} + +static int se_is_idle (struct sched_entity *se) +{ +return 0 ; +} +#endif /* CONFIG_FAIR_GROUP_SCHED */ static __always_inline@@ -4841,6 +4873,9 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)dequeue_entity (qcfs_rq, se,DEQUEUE_SLEEP ); +if (cfs_rq_is_idle (group_cfs_rq (se))) + idle_task_delta = cfs_rq->h_nr_running; + qcfs_rq->h_nr_running -= task_delta; qcfs_rq->idle_h_nr_running -= idle_task_delta;@@ -4860,6 +4895,9 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)update_load_avg (qcfs_rq, se,0 );se_update_runnable (se); +if (cfs_rq_is_idle (group_cfs_rq (se))) + idle_task_delta = cfs_rq->h_nr_running; + qcfs_rq->h_nr_running -= task_delta; qcfs_rq->idle_h_nr_running -= idle_task_delta; }@@ -4904,39 +4942,45 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) task_delta = cfs_rq->h_nr_running; idle_task_delta = cfs_rq->idle_h_nr_running;for_each_sched_entity (se) { +struct cfs_rq *qcfs_rq =cfs_rq_of (se); +if (se->on_rq)break ; - cfs_rq =cfs_rq_of (se); -enqueue_entity (cfs_rq, se,ENQUEUE_WAKEUP ); +enqueue_entity (qcfs_rq, se,ENQUEUE_WAKEUP ); + +if (cfs_rq_is_idle (group_cfs_rq (se))) + idle_task_delta = cfs_rq->h_nr_running; - cfs_rq->h_nr_running += task_delta; - cfs_rq->idle_h_nr_running += idle_task_delta; + qcfs_rq->h_nr_running += task_delta; + qcfs_rq->idle_h_nr_running += idle_task_delta;/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (cfs_rq)) +if (cfs_rq_throttled (qcfs_rq)) goto unthrottle_throttle; }for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +struct cfs_rq *qcfs_rq =cfs_rq_of (se); -update_load_avg (cfs_rq, se,UPDATE_TG ); +update_load_avg (qcfs_rq, se,UPDATE_TG );se_update_runnable (se); - cfs_rq->h_nr_running += task_delta; - cfs_rq->idle_h_nr_running += idle_task_delta; +if (cfs_rq_is_idle (group_cfs_rq (se))) + idle_task_delta = cfs_rq->h_nr_running; + qcfs_rq->h_nr_running += task_delta; + qcfs_rq->idle_h_nr_running += idle_task_delta;/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (cfs_rq)) +if (cfs_rq_throttled (qcfs_rq)) goto unthrottle_throttle;/* * One parent has been throttled and cfs_rq removed from the * list. Add it back to not break the leaf list. */ -if (throttled_hierarchy (cfs_rq)) -list_add_leaf_cfs_rq (cfs_rq); +if (throttled_hierarchy (qcfs_rq)) +list_add_leaf_cfs_rq (qcfs_rq); }/* At this point se is NULL and we are at root level*/ @@ -4949,9 +4993,9 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)* assertion below. */ for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +struct cfs_rq *qcfs_rq =cfs_rq_of (se); -if (list_add_leaf_cfs_rq (cfs_rq)) +if (list_add_leaf_cfs_rq (qcfs_rq))break ; }@@ -5574,6 +5618,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq->h_nr_running++; cfs_rq->idle_h_nr_running += idle_h_nr_running; +if (cfs_rq_is_idle (cfs_rq)) + idle_h_nr_running =1 ; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) goto enqueue_throttle;@@ -5591,6 +5638,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq->h_nr_running++; cfs_rq->idle_h_nr_running += idle_h_nr_running; +if (cfs_rq_is_idle (cfs_rq)) + idle_h_nr_running =1 ; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) goto enqueue_throttle;@@ -5668,6 +5718,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq->h_nr_running--; cfs_rq->idle_h_nr_running -= idle_h_nr_running; +if (cfs_rq_is_idle (cfs_rq)) + idle_h_nr_running =1 ; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) goto dequeue_throttle;@@ -5697,6 +5750,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq->h_nr_running--; cfs_rq->idle_h_nr_running -= idle_h_nr_running; +if (cfs_rq_is_idle (cfs_rq)) + idle_h_nr_running =1 ; +/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) goto dequeue_throttle;@@ -7039,24 +7095,22 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)static void set_last_buddy (struct sched_entity *se) { -if (entity_is_task (se) &&unlikely (task_has_idle_policy (task_of (se)))) -return ; -for_each_sched_entity (se) {if (SCHED_WARN_ON (!se->on_rq))return ; +if (se_is_idle (se)) +return ;cfs_rq_of (se)->last = se; } }static void set_next_buddy (struct sched_entity *se) { -if (entity_is_task (se) &&unlikely (task_has_idle_policy (task_of (se)))) -return ; -for_each_sched_entity (se) {if (SCHED_WARN_ON (!se->on_rq))return ; +if (se_is_idle (se)) +return ;cfs_rq_of (se)->next = se; } }@@ -7077,6 +7131,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_struct cfs_rq *cfs_rq =task_cfs_rq (curr);int scale = cfs_rq->nr_running >= sched_nr_latency;int next_buddy_marked =0 ; +int cse_is_idle, pse_is_idle;if (unlikely (se == pse))return ;@@ -7121,8 +7176,21 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_return ;find_matching_se (&se, &pse); -update_curr (cfs_rq_of (se));BUG_ON (!pse); + + cse_is_idle =se_is_idle (se); + pse_is_idle =se_is_idle (pse); + +/* +* Preempt an idle group in favor of a non-idle group (and don't preempt +* in the inverse case). +*/ +if (cse_is_idle && !pse_is_idle) + goto preempt; +if (cse_is_idle != pse_is_idle) +return ; + +update_curr (cfs_rq_of (se));if (wakeup_preempt_entity (se, pse) ==1 ) {/* * Bias pick_next to pick the sched entity that is @@ -11416,10 +11484,12 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,static DEFINE_MUTEX (shares_mutex); -int sched_group_set_shares (struct task_group *tg,unsigned long shares) +static int __sched_group_set_shares (struct task_group *tg,unsigned long shares) {int i; +lockdep_assert_held (&shares_mutex); +/* * We can't change the weight of the root cgroup. */ @@ -11428,9 +11498,8 @@ int sched_group_set_shares(struct task_group *tg, unsigned long shares) shares =clamp (shares,scale_load (MIN_SHARES ),scale_load (MAX_SHARES )); -mutex_lock (&shares_mutex);if (tg->shares == shares) - goto done; +return 0 ; tg->shares = shares;for_each_possible_cpu (i) {@@ -11448,10 +11517,88 @@ int sched_group_set_shares(struct task_group *tg, unsigned long shares)rq_unlock_irqrestore (rq, &rf); } -done: +return 0 ; +} + +int sched_group_set_shares (struct task_group *tg,unsigned long shares) +{ +int ret; + +mutex_lock (&shares_mutex);
kernel/sched/fair.c int sched_group_set_shares(struct task_group *tg, unsigned long shares) · line 11537
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (tg_is_idle (tg)) + ret = -EINVAL ; +else + ret =__sched_group_set_shares (tg, shares); +mutex_unlock (&shares_mutex); + +return ret; +} + +int sched_group_set_idle (struct task_group *tg,long idle)
int sched_group_set_idle(struct task_group *tg, int idle)
Quoted 82 lines
kernel/sched/fair.c +61
-0+{ +int i; + +if (tg == &root_task_group) +return -EINVAL ; + +if (idle <0 || idle >1 ) +return -EINVAL ; + +mutex_lock (&shares_mutex); + +if (tg->idle == idle) { +mutex_unlock (&shares_mutex); +return 0 ; + } + + tg->idle = idle; + +for_each_possible_cpu (i) { +struct rq *rq =cpu_rq (i); +struct sched_entity *se = tg->se[i]; +struct cfs_rq *grp_cfs_rq = tg->cfs_rq[i]; +bool was_idle =cfs_rq_is_idle (grp_cfs_rq); +long idle_task_delta; +struct rq_flags rf; + +rq_lock_irqsave (rq, &rf); + + grp_cfs_rq->idle = idle; +if (WARN_ON_ONCE (was_idle ==cfs_rq_is_idle (grp_cfs_rq))) + goto next_cpu; + + idle_task_delta = grp_cfs_rq->h_nr_running - + grp_cfs_rq->idle_h_nr_running; +if (!cfs_rq_is_idle (grp_cfs_rq)) + idle_task_delta *=-1 ; + +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +if (!se->on_rq) +break ; + + cfs_rq->idle_h_nr_running += idle_task_delta; + +/* Already accounted at parent level and above. */ +if (cfs_rq_is_idle (cfs_rq)) +break ; + } + +next_cpu: +rq_unlock_irqrestore (rq, &rf); + } + +/* Idle groups have minimum weight. */ +if (tg_is_idle (tg)) +__sched_group_set_shares (tg,scale_load (WEIGHT_IDLEPRIO )); +else +__sched_group_set_shares (tg,NICE_0_LOAD ); +mutex_unlock (&shares_mutex);return 0 ; } +#else /* CONFIG_FAIR_GROUP_SCHED */ void free_fair_sched_group (struct task_group *tg) { }kernel/sched/sched.h +3
-0@@ -396,6 +396,9 @@ struct task_group {struct cfs_rq **cfs_rq;unsigned long shares; +/* A positive value indicates that this is a SCHED_IDLE group. */ +int idle; +#ifdef CONFIG_SMP /* * load_avg can be heavily contended at clock tick time, so put
kernel/sched/sched.h extern void sched_move_task(struct task_struct *tsk); · line 511
Quoted 4 lines
kernel/sched/sched.h +1
-0@@ -505,6 +508,8 @@ extern void sched_move_task(struct task_struct *tsk);#ifdef CONFIG_FAIR_GROUP_SCHED extern int sched_group_set_shares (struct task_group *tg,unsigned long shares); +extern int sched_group_set_idle (struct task_group *tg,long idle);
extern int sched_group_set_idle(struct task_group *tg, int idle);
Quoted 17 lines
kernel/sched/sched.h +4
-0+#ifdef CONFIG_SMP extern void set_task_rq_fair (struct sched_entity *se,struct cfs_rq *prev,struct cfs_rq *next);@@ -601,6 +606,9 @@ struct cfs_rq {struct list_head leaf_cfs_rq_list;struct task_group *tg;/* group that "owns" this runqueue */ +/* Locally cached copy of our task_group's idle value */ +int idle; +#ifdef CONFIG_CFS_BANDWIDTH int runtime_enabled;s64 runtime_remaining;Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Cfs_rq and tg define @idle with int type.
In sched_group_set_idle(..., long idle), @idle is long type. Use int instead.
But, you filter idle value:
if (idle < 0 || idle > 1)
return -EINVAL;So, no effect here.. Just @idle can use 4 bytes.
Thanks,
Tao
Re: [PATCH v3 1/4] sched: cgroup SCHED_IDLE support
Hi Tao,
On Fri, Aug 20, 2021 at 1:38 AM Tao Zhou <[email protected]> wrote:
[snip]
Quoted 26 lines
Quoted 14 lines
#ifdef
CONFIG_SMP
extern void set_task_rq_fair(structsched_entity*se,struct cfs_rq *prev, struct cfs_rq *next);kernel/sched/sched.h +3
-1@@ -601,6 +606,9 @@ struct cfs_rq {struct list_head leaf_cfs_rq_list;struct task_group *tg;/* group that "owns" this runqueue */ +/* Locally cached copy of our task_group's idle value */ +int idle; +#ifdef CONFIG_CFS_BANDWIDTH int runtime_enabled;s64 runtime_remaining; --Quoted 2 lines
2.33.0.rc2.250.ged5fa647cd-goog
Cfs_rqand tg define @idle with int type.
Insched_group_set_idle(..., long idle), @idle is long type. Use int instead.But, you filter idle value:
if (idle < 0 || idle > 1) return -EINVAL;
Quoted 6 lines
So, no effect here.. Just @idle can use 4 bytes.
Thanks,
Tao
The use of 'long' there is because the input from the cgroup interface is a 64 bit value.
- Josh
Re: [PATCH v3 1/4] sched: cgroup SCHED_IDLE support
Hi Josh,
Quoted 34 lines · Josh Don
Hi Tao,
On Fri, Aug 20, 2021 at 1:38 AM Tao Zhou <[email protected]> wrote:
[snip]Quoted 30 lines
Quoted 14 lines
#ifdef
CONFIG_SMP
extern void set_task_rq_fair(structsched_entity*se,struct cfs_rq *prev, struct cfs_rq *next);kernel/sched/sched.h +3
-1@@ -601,6 +606,9 @@ struct cfs_rq {struct list_head leaf_cfs_rq_list;struct task_group *tg;/* group that "owns" this runqueue */ +/* Locally cached copy of our task_group's idle value */ +int idle; +#ifdef CONFIG_CFS_BANDWIDTH int runtime_enabled;s64 runtime_remaining; --Quoted 2 lines
2.33.0.rc2.250.ged5fa647cd-goog
Cfs_rqand tg define @idle with int type.
Insched_group_set_idle(..., long idle), @idle is long type. Use int instead.But, you filter idle value:
if (idle < 0 || idle > 1) return -EINVAL;So, no effect here.. Just @idle can use 4 bytes.
Quoted 5 lines
Quoted 2 lines
Thanks,
TaoThe use of 'long' there is because the input from the cgroup interface is a 64 bit value.
Yes. If the compile align the stack to 8 or other, this will have no effect(I've not check this, and have not much about gcc compile align). I just presume that if the stack can save 4 bytes. But, that may not right though.
Quoted 1 line
- Josh
Thanks,
Tao
Re: [PATCH v3 1/4] sched: cgroup SCHED_IDLE support
In reply to Josh Donkernel/sched/fair.c find_matching_se(struct sched_entity **se, struct sched_entity **pse) · line 501
Quoted 4 lines · Josh Don
kernel/sched/fair.c +4
-0+static int se_is_idle (struct sched_entity *se) +{ +return 0 ; +}
I'm thinking !FAIR_GROUP_SCHED is a rare thing to behold? So not a big deal, but I think this wants to be
return task_has_idle_policy(task_of(se));
so buddies aren't set for SCHED_IDLE.
Quoted 27 lines
static void
set_last_buddy(struct sched_entity *se){kernel/sched/fair.c +4
-6-if (entity_is_task (se) &&unlikely (task_has_idle_policy (task_of (se)))) -return ; -for_each_sched_entity (se) {if (SCHED_WARN_ON (!se->on_rq))return ; +if (se_is_idle (se)) +return ;cfs_rq_of (se)->last = se; } }static void set_next_buddy (struct sched_entity *se) { -if (entity_is_task (se) &&unlikely (task_has_idle_policy (task_of (se)))) -return ; -for_each_sched_entity (se) {if (SCHED_WARN_ON (!se->on_rq))return ; +if (se_is_idle (se)) +return ;cfs_rq_of (se)->next = se; } }
Re: [PATCH v3 1/4] sched: cgroup SCHED_IDLE support
Quoted 10 lines · Daniel Jordan
kernel/sched/fair.c find_matching_se(struct sched_entity **se, struct sched_entity **pse) · line 501
Quoted 4 lines · Josh Don
kernel/sched/fair.c +4
-0+static int se_is_idle (struct sched_entity *se) +{ +return 0 ; +}I'm thinking !FAIR_GROUP_SCHED is a rare thing to behold? So not a big deal, but I think this wants to be
Quoted 3 lines
return task_has_idle_policy(task_of(se));so buddies aren't set for
SCHED_IDLE.
Good point. Peter has merged this one already, do you want to send a patch?
Re: [PATCH v3 1/4] sched: cgroup SCHED_IDLE support
Quoted 11 lines · Josh Don
Quoted 10 lines · Daniel Jordan
kernel/sched/fair.c find_matching_se(struct sched_entity **se, struct sched_entity **pse) · line 501
Quoted 4 lines · Josh Don
kernel/sched/fair.c +4
-0+static int se_is_idle (struct sched_entity *se) +{ +return 0 ; +}I'm thinking !FAIR_GROUP_SCHED is a rare thing to behold? So not a big deal, but I think this wants to be
Quoted 5 lines
Quoted 3 lines
return task_has_idle_policy(task_of(se));so buddies aren't set for
SCHED_IDLE.Good point. Peter has merged this one already, do you want to send a patch?
Sure, here it comes.
Re: [PATCH v3 1/4] sched: cgroup SCHED_IDLE support
In reply to Josh DonQuoted 12 lines · On Mon, Sep 6, 2021 at 5:47 AM <[email protected]> wrote:
...
Quoted 8 lines
For now, we maintain the existing
SCHED_IDLEsemantics. Future patches may make improvements that extend how we treatSCHED_IDLEentities.The per-task_group idle field is an integer that currently only holds either a 0 or a 1. This is explicitly typed as an integer to allow for further extensions to this API. For example, a negative value may indicate a highly latency-sensitive cgroup that should be preferred for preemption/placement/etc.
Quoted 8 lines
Hi Josh,
Sounds there is a ready solutions for colocation problem, isn't there? I'd like to evaluate its effects if it could be sent out.
Thanks
AlexQuoted 1 line
Hi Alex,
I don't have a patch written just yet. The basic idea is to utilize the (negative) idle values to make relative decisions about task preemption and wakeup vruntime placement.
Could you describe your use case?
Best,
Josh
PATCH v3 2/4 sched: account number of SCHED_IDLE entities on each cfs_rq
In reply to Josh DonAdds cfs_rq->idle_nr_running, which accounts the number of idle entities directly enqueued on the cfs_rq.
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 25 ++++++++++++++++++++++++- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 1 deletion(-)
kernel/sched/debug.c +2 -0
@@ -608,6 +608,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) cfs_rq->nr_spread_over);SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"idle_nr_running" , + cfs_rq->idle_nr_running);SEQ_printf (m," .%-30s: %d\n" ,"idle_h_nr_running" , cfs_rq->idle_h_nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);
kernel/sched/fair.c +24 -1
@@ -2995,6 +2995,8 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running++; +if (se_is_idle (se)) + cfs_rq->idle_nr_running++; } static void
@@ -3008,6 +3010,8 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running--; +if (se_is_idle (se)) + cfs_rq->idle_nr_running--; }/*
@@ -5573,6 +5577,17 @@ static int sched_idle_rq(struct rq *rq) rq->nr_running); } +/* +* Returns true if cfs_rq only has SCHED_IDLE entities enqueued. Note the use +* of idle_nr_running, which does not consider idle descendants of normal +* entities. +*/ +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq) +{ +return cfs_rq->nr_running && + cfs_rq->nr_running == cfs_rq->idle_nr_running; +} +#ifdef CONFIG_SMP static int sched_idle_cpu (int cpu) {
@@ -11556,7 +11571,7 @@ int sched_group_set_idle(struct task_group *tg, long idle)for_each_possible_cpu (i) {struct rq *rq =cpu_rq (i);struct sched_entity *se = tg->se[i]; -struct cfs_rq *grp_cfs_rq = tg->cfs_rq[i]; +struct cfs_rq *parent_cfs_rq, *grp_cfs_rq = tg->cfs_rq[i];bool was_idle =cfs_rq_is_idle (grp_cfs_rq);long idle_task_delta;struct rq_flags rf;
@@ -11567,6 +11582,14 @@ int sched_group_set_idle(struct task_group *tg, long idle)if (WARN_ON_ONCE (was_idle ==cfs_rq_is_idle (grp_cfs_rq))) goto next_cpu; +if (se->on_rq) { + parent_cfs_rq =cfs_rq_of (se); +if (cfs_rq_is_idle (grp_cfs_rq)) + parent_cfs_rq->idle_nr_running++; +else + parent_cfs_rq->idle_nr_running--; + } + idle_task_delta = grp_cfs_rq->h_nr_running - grp_cfs_rq->idle_h_nr_running;if (!cfs_rq_is_idle (grp_cfs_rq))
kernel/sched/sched.h +1 -0
@@ -530,6 +530,7 @@ struct cfs_rq {struct load_weight load;unsigned int nr_running;unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ +unsigned int idle_nr_running;/* SCHED_IDLE */ unsigned int idle_h_nr_running;/* SCHED_IDLE */ u64 exec_clock;
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH v3 2/4] sched: account number of SCHED_IDLE entities on each cfs_rq
Quoted 5 lines · Josh Don
Adds
cfs_rq->idle_nr_running, which accounts the number of idle entities directly enqueued on thecfs_rq.Signed-off-by Josh Don <[email protected]>
Quoted 98 lines
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 25 ++++++++++++++++++++++++- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 1 deletion(-)kernel/sched/debug.c +2
-0@@ -608,6 +608,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) cfs_rq->nr_spread_over);SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"idle_nr_running" , + cfs_rq->idle_nr_running);SEQ_printf (m," .%-30s: %d\n" ,"idle_h_nr_running" , cfs_rq->idle_h_nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);kernel/sched/fair.c +24
-1@@ -2995,6 +2995,8 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running++; +if (se_is_idle (se)) + cfs_rq->idle_nr_running++; } static void@@ -3008,6 +3010,8 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running--; +if (se_is_idle (se)) + cfs_rq->idle_nr_running--; }/* @@ -5573,6 +5577,17 @@ static int sched_idle_rq(struct rq *rq) rq->nr_running); } +/* +* Returns true if cfs_rq only has SCHED_IDLE entities enqueued. Note the use +* of idle_nr_running, which does not consider idle descendants of normal +* entities. +*/ +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq) +{ +return cfs_rq->nr_running && + cfs_rq->nr_running == cfs_rq->idle_nr_running; +} +#ifdef CONFIG_SMP static int sched_idle_cpu (int cpu) {@@ -11556,7 +11571,7 @@ int sched_group_set_idle(struct task_group *tg, long idle)for_each_possible_cpu (i) {struct rq *rq =cpu_rq (i);struct sched_entity *se = tg->se[i]; -struct cfs_rq *grp_cfs_rq = tg->cfs_rq[i]; +struct cfs_rq *parent_cfs_rq, *grp_cfs_rq = tg->cfs_rq[i];bool was_idle =cfs_rq_is_idle (grp_cfs_rq);long idle_task_delta;struct rq_flags rf;@@ -11567,6 +11582,14 @@ int sched_group_set_idle(struct task_group *tg, long idle)if (WARN_ON_ONCE (was_idle ==cfs_rq_is_idle (grp_cfs_rq))) goto next_cpu; +if (se->on_rq) { + parent_cfs_rq =cfs_rq_of (se); +if (cfs_rq_is_idle (grp_cfs_rq)) + parent_cfs_rq->idle_nr_running++; +else + parent_cfs_rq->idle_nr_running--; + } + idle_task_delta = grp_cfs_rq->h_nr_running - grp_cfs_rq->idle_h_nr_running;if (!cfs_rq_is_idle (grp_cfs_rq))kernel/sched/sched.h +1
-1@@ -530,6 +530,7 @@ struct cfs_rq {struct load_weight load;unsigned int nr_running;unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ +unsigned int idle_nr_running;/* SCHED_IDLE */ unsigned int idle_h_nr_running;/* SCHED_IDLE */ u64 exec_clock; --2.33.0.rc2.250.ged5fa647cd-goog
[tip: sched/core] sched: Account number of SCHED_IDLE entities on each cfs_rq
In reply to Josh DonThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4b1e9afe8af5becd9b33640a7c996ccfce4ea310 Gitweb: https://git.kernel.org/tip/4b1e9afe8af5becd9b33640a7c996ccfce4ea310 Author: Josh Don <[email protected]> AuthorDate: Thu, 19 Aug 2021 18:04:01 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 09 Sep 2021 11:27:31 +02:00
sched: Account number of SCHED_IDLE entities on each cfs_rq
Adds cfs_rq->idle_nr_running, which accounts the number of idle entities directly enqueued on the cfs_rq.
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 25 ++++++++++++++++++++++++- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 1 deletion(-)
kernel/sched/debug.c +2 -0
@@ -608,6 +608,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) cfs_rq->nr_spread_over);SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"idle_nr_running" , + cfs_rq->idle_nr_running);SEQ_printf (m," .%-30s: %d\n" ,"idle_h_nr_running" , cfs_rq->idle_h_nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);
kernel/sched/fair.c +24 -1
@@ -2995,6 +2995,8 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running++; +if (se_is_idle (se)) + cfs_rq->idle_nr_running++; } static void
@@ -3008,6 +3010,8 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running--; +if (se_is_idle (se)) + cfs_rq->idle_nr_running--; }/*
@@ -5544,6 +5548,17 @@ static int sched_idle_rq(struct rq *rq) rq->nr_running); } +/* +* Returns true if cfs_rq only has SCHED_IDLE entities enqueued. Note the use +* of idle_nr_running, which does not consider idle descendants of normal +* entities. +*/ +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq) +{ +return cfs_rq->nr_running && + cfs_rq->nr_running == cfs_rq->idle_nr_running; +} +#ifdef CONFIG_SMP static int sched_idle_cpu (int cpu) {
@@ -11542,7 +11557,7 @@ int sched_group_set_idle(struct task_group *tg, long idle)for_each_possible_cpu (i) {struct rq *rq =cpu_rq (i);struct sched_entity *se = tg->se[i]; -struct cfs_rq *grp_cfs_rq = tg->cfs_rq[i]; +struct cfs_rq *parent_cfs_rq, *grp_cfs_rq = tg->cfs_rq[i];bool was_idle =cfs_rq_is_idle (grp_cfs_rq);long idle_task_delta;struct rq_flags rf;
@@ -11553,6 +11568,14 @@ int sched_group_set_idle(struct task_group *tg, long idle)if (WARN_ON_ONCE (was_idle ==cfs_rq_is_idle (grp_cfs_rq))) goto next_cpu; +if (se->on_rq) { + parent_cfs_rq =cfs_rq_of (se); +if (cfs_rq_is_idle (grp_cfs_rq)) + parent_cfs_rq->idle_nr_running++; +else + parent_cfs_rq->idle_nr_running--; + } + idle_task_delta = grp_cfs_rq->h_nr_running - grp_cfs_rq->idle_h_nr_running;if (!cfs_rq_is_idle (grp_cfs_rq))
kernel/sched/sched.h +1 -0
@@ -530,6 +530,7 @@ struct cfs_rq {struct load_weight load;unsigned int nr_running;unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ +unsigned int idle_nr_running;/* SCHED_IDLE */ unsigned int idle_h_nr_running;/* SCHED_IDLE */ u64 exec_clock;
[tip: sched/core] sched: Account number of SCHED_IDLE entities on each cfs_rq
In reply to Josh DonThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: a480addecc0d89c200ec0b41da62ae8ceddca8d7 Gitweb: https://git.kernel.org/tip/a480addecc0d89c200ec0b41da62ae8ceddca8d7 Author: Josh Don <[email protected]> AuthorDate: Thu, 19 Aug 2021 18:04:01 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 05 Oct 2021 15:51:36 +02:00
sched: Account number of SCHED_IDLE entities on each cfs_rq
Adds cfs_rq->idle_nr_running, which accounts the number of idle entities directly enqueued on the cfs_rq.
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 25 ++++++++++++++++++++++++- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 1 deletion(-)
kernel/sched/debug.c +2 -0
@@ -614,6 +614,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) cfs_rq->nr_spread_over);SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"idle_nr_running" , + cfs_rq->idle_nr_running);SEQ_printf (m," .%-30s: %d\n" ,"idle_h_nr_running" , cfs_rq->idle_h_nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);
kernel/sched/fair.c +24 -1
@@ -2995,6 +2995,8 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running++; +if (se_is_idle (se)) + cfs_rq->idle_nr_running++; } static void
@@ -3008,6 +3010,8 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) }#endif cfs_rq->nr_running--; +if (se_is_idle (se)) + cfs_rq->idle_nr_running--; }/*
@@ -5577,6 +5581,17 @@ static int sched_idle_rq(struct rq *rq) rq->nr_running); } +/* +* Returns true if cfs_rq only has SCHED_IDLE entities enqueued. Note the use +* of idle_nr_running, which does not consider idle descendants of normal +* entities. +*/ +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq) +{ +return cfs_rq->nr_running && + cfs_rq->nr_running == cfs_rq->idle_nr_running; +} +#ifdef CONFIG_SMP static int sched_idle_cpu (int cpu) {
@@ -11575,7 +11590,7 @@ int sched_group_set_idle(struct task_group *tg, long idle)for_each_possible_cpu (i) {struct rq *rq =cpu_rq (i);struct sched_entity *se = tg->se[i]; -struct cfs_rq *grp_cfs_rq = tg->cfs_rq[i]; +struct cfs_rq *parent_cfs_rq, *grp_cfs_rq = tg->cfs_rq[i];bool was_idle =cfs_rq_is_idle (grp_cfs_rq);long idle_task_delta;struct rq_flags rf;
@@ -11586,6 +11601,14 @@ int sched_group_set_idle(struct task_group *tg, long idle)if (WARN_ON_ONCE (was_idle ==cfs_rq_is_idle (grp_cfs_rq))) goto next_cpu; +if (se->on_rq) { + parent_cfs_rq =cfs_rq_of (se); +if (cfs_rq_is_idle (grp_cfs_rq)) + parent_cfs_rq->idle_nr_running++; +else + parent_cfs_rq->idle_nr_running--; + } + idle_task_delta = grp_cfs_rq->h_nr_running - grp_cfs_rq->idle_h_nr_running;if (!cfs_rq_is_idle (grp_cfs_rq))
kernel/sched/sched.h +1 -0
@@ -530,6 +530,7 @@ struct cfs_rq {struct load_weight load;unsigned int nr_running;unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ +unsigned int idle_nr_running;/* SCHED_IDLE */ unsigned int idle_h_nr_running;/* SCHED_IDLE */ u64 exec_clock;
PATCH v3 3/4 sched: reduce sched slice for SCHED_IDLE entities
In reply to Josh DonUse a small, non-scaled min granularity for SCHED_IDLE entities, when competing with normal entities. This reduces the latency of getting a normal entity back on cpu, at the expense of increased context switch frequency of SCHED_IDLE entities.
The benefit of this change is to reduce the round-robin latency for normal entities when competing with a SCHED_IDLE entity.
Example: on a machine with HZ=1000, spawned two threads, one of which is SCHED_IDLE, and affined to one cpu. Without this patch, the SCHED_IDLE thread runs for 4ms then waits for 1.4s. With this patch, it runs for 1ms and waits 340ms (as it round-robins with the other thread).
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 29 ++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 5 deletions(-)
kernel/sched/debug.c +2 -0
@@ -305,6 +305,7 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("latency_ns" ,0644 , debugfs_sched, &sysctl_sched_latency);debugfs_create_u32 ("min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_min_granularity); +debugfs_create_u32 ("idle_min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_idle_min_granularity);debugfs_create_u32 ("wakeup_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_wakeup_granularity);debugfs_create_u32 ("latency_warn_ms" ,0644 , debugfs_sched, &sysctl_resched_latency_warn_ms);
@@ -806,6 +807,7 @@ static void sched_debug_header(struct seq_file *m) SEQ_printf(m, " .%-40s: %Ld.%06ld\n", #x, SPLIT_NS(x))PN (sysctl_sched_latency);PN (sysctl_sched_min_granularity); +PN (sysctl_sched_idle_min_granularity);PN (sysctl_sched_wakeup_granularity);P (sysctl_sched_child_runs_first);P (sysctl_sched_features);
kernel/sched/fair.c +24 -5
@@ -59,6 +59,14 @@ unsigned int sysctl_sched_tunable_scaling = SCHED_TUNABLESCALING_LOG;unsigned int sysctl_sched_min_granularity =750000ULL ;static unsigned int normalized_sysctl_sched_min_granularity =750000ULL ; +/* +* Minimal preemption granularity for CPU-bound SCHED_IDLE tasks. +* Applies only when SCHED_IDLE tasks compete with normal tasks. +* +* (default: 0.75 msec) +*/ +unsigned int sysctl_sched_idle_min_granularity =750000ULL ; +/* * This value is kept at sysctl_sched_latency/sysctl_sched_min_granularity */
@@ -665,6 +673,8 @@ static u64 __sched_period(unsigned long nr_running)return sysctl_sched_latency; } +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq); +/* * We calculate the wall-time slice from the period by taking a part * proportional to the weight.
@@ -674,6 +684,8 @@ static u64 __sched_period(unsigned long nr_running)static u64 sched_slice (struct cfs_rq *cfs_rq,struct sched_entity *se) {unsigned int nr_running = cfs_rq->nr_running; +struct sched_entity *init_se = se; +unsigned int min_gran;u64 slice;if (sched_feat (ALT_PERIOD ))
@@ -684,12 +696,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)for_each_sched_entity (se) {struct load_weight *load;struct load_weight lw; +struct cfs_rq *qcfs_rq; - cfs_rq =cfs_rq_of (se); - load = &cfs_rq->load; + qcfs_rq =cfs_rq_of (se); + load = &qcfs_rq->load;if (unlikely (!se->on_rq)) { - lw = cfs_rq->load; + lw = qcfs_rq->load;update_load_add (&lw, se->load.weight); load = &lw;
@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); } -if (sched_feat (BASE_SLICE )) - slice =max (slice, (u64 )sysctl_sched_min_granularity); +if (sched_feat (BASE_SLICE )) { +if (se_is_idle (init_se) && !sched_idle_cfs_rq (cfs_rq)) + min_gran = sysctl_sched_idle_min_granularity; +else + min_gran = sysctl_sched_min_granularity; + + slice =max_t (u64, slice, min_gran); + }return slice; }
kernel/sched/sched.h +1 -0
@@ -2399,6 +2399,7 @@ extern const_debug unsigned int sysctl_sched_migration_cost;#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency;extern unsigned int sysctl_sched_min_granularity; +extern unsigned int sysctl_sched_idle_min_granularity;extern unsigned int sysctl_sched_wakeup_granularity;extern int sysctl_resched_latency_warn_ms;extern int sysctl_resched_latency_warn_once;
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Vincent Guittot
maintainerRe: [PATCH v3 3/4] sched: reduce sched slice for SCHED_IDLE entities
Quoted 96 lines · Josh Don
Use a small, non-scaled min granularity for
SCHED_IDLEentities, when competing with normal entities. This reduces the latency of getting a normal entity back on cpu, at the expense of increased context switch frequency ofSCHED_IDLEentities.The benefit of this change is to reduce the round-robin latency for normal entities when competing with a
SCHED_IDLEentity.Example: on a machine with HZ=1000, spawned two threads, one of which is
SCHED_IDLE, and affined to one cpu. Without this patch, theSCHED_IDLEthread runs for 4ms then waits for 1.4s. With this patch, it runs for 1ms and waits 340ms (as it round-robins with the other thread).Signed-off-by Josh Don <[email protected]>--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 29 ++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 5 deletions(-)kernel/sched/debug.c +2
-0@@ -305,6 +305,7 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("latency_ns" ,0644 , debugfs_sched, &sysctl_sched_latency);debugfs_create_u32 ("min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_min_granularity); +debugfs_create_u32 ("idle_min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_idle_min_granularity);debugfs_create_u32 ("wakeup_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_wakeup_granularity);debugfs_create_u32 ("latency_warn_ms" ,0644 , debugfs_sched, &sysctl_resched_latency_warn_ms);@@ -806,6 +807,7 @@ static void sched_debug_header(struct seq_file *m) SEQ_printf(m, " .%-40s: %Ld.%06ld\n", #x, SPLIT_NS(x))PN (sysctl_sched_latency);PN (sysctl_sched_min_granularity); +PN (sysctl_sched_idle_min_granularity);PN (sysctl_sched_wakeup_granularity);P (sysctl_sched_child_runs_first);P (sysctl_sched_features);kernel/sched/fair.c +16
-3@@ -59,6 +59,14 @@ unsigned int sysctl_sched_tunable_scaling = SCHED_TUNABLESCALING_LOG;unsigned int sysctl_sched_min_granularity =750000ULL ;static unsigned int normalized_sysctl_sched_min_granularity =750000ULL ; +/* +* Minimal preemption granularity for CPU-bound SCHED_IDLE tasks. +* Applies only when SCHED_IDLE tasks compete with normal tasks. +* +* (default: 0.75 msec) +*/ +unsigned int sysctl_sched_idle_min_granularity =750000ULL ; +/* * This value is kept at sysctl_sched_latency/sysctl_sched_min_granularity */ @@ -665,6 +673,8 @@ static u64 __sched_period(unsigned long nr_running)return sysctl_sched_latency; } +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq); +/* * We calculate the wall-time slice from the period by taking a part * proportional to the weight. @@ -674,6 +684,8 @@ static u64 __sched_period(unsigned long nr_running)static u64 sched_slice (struct cfs_rq *cfs_rq,struct sched_entity *se) {unsigned int nr_running = cfs_rq->nr_running; +struct sched_entity *init_se = se; +unsigned int min_gran;u64 slice;if (sched_feat (ALT_PERIOD ))@@ -684,12 +696,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)for_each_sched_entity (se) {struct load_weight *load;struct load_weight lw; +struct cfs_rq *qcfs_rq; - cfs_rq =cfs_rq_of (se); - load = &cfs_rq->load; + qcfs_rq =cfs_rq_of (se); + load = &qcfs_rq->load;if (unlikely (!se->on_rq)) { - lw = cfs_rq->load; + lw = qcfs_rq->load;update_load_add (&lw, se->load.weight); load = &lw;
kernel/sched/fair.c static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 714
Quoted 7 lines
kernel/sched/fair.c +2
-2@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); } -if (sched_feat (BASE_SLICE )) - slice =max (slice, (u64 )sysctl_sched_min_granularity); +if (sched_feat (BASE_SLICE )) { +if (se_is_idle (init_se) && !sched_idle_cfs_rq (cfs_rq))
Like for place_entity, we should probably not dynamically switch between the 2 values below depending on the presence or not of non sched idle tasks and always use sysctl_sched_idle_min_granularity
Quoted 24 lines
kernel/sched/fair.c +6
-0+ min_gran = sysctl_sched_idle_min_granularity; + else + min_gran = sysctl_sched_min_granularity; + + slice =max_t (u64, slice, min_gran); + }return slice; }kernel/sched/sched.h +1
-1@@ -2399,6 +2399,7 @@ extern const_debug unsigned int sysctl_sched_migration_cost;#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency;extern unsigned int sysctl_sched_min_granularity; +extern unsigned int sysctl_sched_idle_min_granularity;extern unsigned int sysctl_sched_wakeup_granularity;extern int sysctl_resched_latency_warn_ms;extern int sysctl_resched_latency_warn_once; --2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH v3 3/4] sched: reduce sched slice for SCHED_IDLE entities
Quoted 24 lines · Vincent Guittot
Quoted 22 lines · Josh Don
kernel/sched/fair.c +4
-3@@ -684,12 +696,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)for_each_sched_entity (se) {struct load_weight *load;struct load_weight lw; +struct cfs_rq *qcfs_rq; - cfs_rq =cfs_rq_of (se); - load = &cfs_rq->load; + qcfs_rq =cfs_rq_of (se); + load = &qcfs_rq->load;if (unlikely (!se->on_rq)) { - lw = cfs_rq->load; + lw = qcfs_rq->load;update_load_add (&lw, se->load.weight); load = &lw;@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); }
Quoted 8 lines
kernel/sched/fair.c static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 714
Quoted 4 lines
kernel/sched/fair.c +2
-2-if (sched_feat (BASE_SLICE )) - slice =max (slice, (u64 )sysctl_sched_min_granularity); +if (sched_feat (BASE_SLICE )) { +if (se_is_idle (init_se) && !sched_idle_cfs_rq (cfs_rq))Like for
place_entity, we should probably not dynamically switch between the 2 values below depending on the presence or not of non sched idle tasks and always usesysctl_sched_idle_min_granularity
My reasoning here is that sched_slice is something we reasonably expect to change as tasks enqueue/dequeue, and unlike place_entity() it does not create fairness issues by messing with vruntime. Additionally, it would be preferable to use the larger min granularity on a cpu running only idle tasks.
Re: [PATCH v3 3/4] sched: reduce sched slice for SCHED_IDLE entities
Quoted 36 lines · Josh Don
Quoted 24 lines · Vincent Guittot
Quoted 22 lines · Josh Don
kernel/sched/fair.c +4
-3@@ -684,12 +696,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)for_each_sched_entity (se) {struct load_weight *load;struct load_weight lw; +struct cfs_rq *qcfs_rq; - cfs_rq =cfs_rq_of (se); - load = &cfs_rq->load; + qcfs_rq =cfs_rq_of (se); + load = &qcfs_rq->load;if (unlikely (!se->on_rq)) { - lw = cfs_rq->load; + lw = qcfs_rq->load;update_load_add (&lw, se->load.weight); load = &lw;@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); }Quoted 8 lines
kernel/sched/fair.c static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 714
Quoted 4 lines
kernel/sched/fair.c +2
-2-if (sched_feat (BASE_SLICE )) - slice =max (slice, (u64 )sysctl_sched_min_granularity); +if (sched_feat (BASE_SLICE )) { +if (se_is_idle (init_se) && !sched_idle_cfs_rq (cfs_rq))Like for
place_entity, we should probably not dynamically switch between the 2 values below depending on the presence or not of non sched idle tasks and always usesysctl_sched_idle_min_granularity
Quoted 5 lines
My reasoning here is that
sched_sliceis something we reasonably expect to change as tasks enqueue/dequeue, and unlikeplace_entity()it does not create fairness issues by messing with vruntime. Additionally, it would be preferable to use the larger min granularity on a cpu running only idle tasks.
Fair enough
Re: [PATCH v3 3/4] sched: reduce sched slice for SCHED_IDLE entities
In reply to Josh DonHi,
Quoted 8 lines · Josh Don
Use a small, non-scaled min granularity for
SCHED_IDLEentities, when competing with normal entities. This reduces the latency of getting a normal entity back on cpu, at the expense of increased context switch frequency ofSCHED_IDLEentities.The benefit of this change is to reduce the round-robin latency for normal entities when competing with a
SCHED_IDLEentity.
Why not just ignore min granularity when normal entities compete with a SCHED_IDLE entity? something like this,
@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
slice = __calc_delta(slice, se->load.weight, load);
} - if (sched_feat(BASE_SLICE))
- slice = max(slice, (u64)sysctl_sched_min_granularity);
+ if (sched_feat(BASE_SLICE)
+ && (!se_is_idle(init_se) || sched_idle_cfs_rq(cfs_rq)))
+ slice = max(slice, (u64)sysctl_sched_min_granularity);
return slice;
}
If so, there seems no need to introduce sysctl_sched_idle_min_granularity? :)
Quoted 5 lines
Example: on a machine with HZ=1000, spawned two threads, one of which is
SCHED_IDLE, and affined to one cpu. Without this patch, theSCHED_IDLEthread runs for 4ms then waits for 1.4s. With this patch, it runs for 1ms and waits 340ms (as it round-robins with the other thread).
In that way, the SCHED_IDLE task could be preempted more likely by the normal task, because the ideal_runtime should be less than 750us(non-scaled sysctl_sched_idle_min_granularity) in this case. And scaled sysctl_sched_min_granularity could be guaranteed normally between SCHED_IDLE tasks when only SCHED_IDLE tasks compete with each other.
Quoted 114 lines
Signed-off-by Josh Don <[email protected]>--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 29 ++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 5 deletions(-)kernel/sched/debug.c +2
-0@@ -305,6 +305,7 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("latency_ns" ,0644 , debugfs_sched, &sysctl_sched_latency);debugfs_create_u32 ("min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_min_granularity); +debugfs_create_u32 ("idle_min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_idle_min_granularity);debugfs_create_u32 ("wakeup_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_wakeup_granularity);debugfs_create_u32 ("latency_warn_ms" ,0644 , debugfs_sched, &sysctl_resched_latency_warn_ms);@@ -806,6 +807,7 @@ static void sched_debug_header(struct seq_file *m) SEQ_printf(m, " .%-40s: %Ld.%06ld\n", #x, SPLIT_NS(x))PN (sysctl_sched_latency);PN (sysctl_sched_min_granularity); +PN (sysctl_sched_idle_min_granularity);PN (sysctl_sched_wakeup_granularity);P (sysctl_sched_child_runs_first);P (sysctl_sched_features);kernel/sched/fair.c +24
-5@@ -59,6 +59,14 @@ unsigned int sysctl_sched_tunable_scaling = SCHED_TUNABLESCALING_LOG;unsigned int sysctl_sched_min_granularity =750000ULL ;static unsigned int normalized_sysctl_sched_min_granularity =750000ULL ; +/* +* Minimal preemption granularity for CPU-bound SCHED_IDLE tasks. +* Applies only when SCHED_IDLE tasks compete with normal tasks. +* +* (default: 0.75 msec) +*/ +unsigned int sysctl_sched_idle_min_granularity =750000ULL ; +/* * This value is kept at sysctl_sched_latency/sysctl_sched_min_granularity */ @@ -665,6 +673,8 @@ static u64 __sched_period(unsigned long nr_running)return sysctl_sched_latency; } +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq); +/* * We calculate the wall-time slice from the period by taking a part * proportional to the weight. @@ -674,6 +684,8 @@ static u64 __sched_period(unsigned long nr_running)static u64 sched_slice (struct cfs_rq *cfs_rq,struct sched_entity *se) {unsigned int nr_running = cfs_rq->nr_running; +struct sched_entity *init_se = se; +unsigned int min_gran;u64 slice;if (sched_feat (ALT_PERIOD ))@@ -684,12 +696,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)for_each_sched_entity (se) {struct load_weight *load;struct load_weight lw; +struct cfs_rq *qcfs_rq; - cfs_rq =cfs_rq_of (se); - load = &cfs_rq->load; + qcfs_rq =cfs_rq_of (se); + load = &qcfs_rq->load;if (unlikely (!se->on_rq)) { - lw = cfs_rq->load; + lw = qcfs_rq->load;update_load_add (&lw, se->load.weight); load = &lw;@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); } -if (sched_feat (BASE_SLICE )) - slice =max (slice, (u64 )sysctl_sched_min_granularity); +if (sched_feat (BASE_SLICE )) { +if (se_is_idle (init_se) && !sched_idle_cfs_rq (cfs_rq)) + min_gran = sysctl_sched_idle_min_granularity; +else + min_gran = sysctl_sched_min_granularity; + + slice =max_t (u64, slice, min_gran); + }return slice; }kernel/sched/sched.h +1
-1@@ -2399,6 +2399,7 @@ extern const_debug unsigned int sysctl_sched_migration_cost;#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency;extern unsigned int sysctl_sched_min_granularity; +extern unsigned int sysctl_sched_idle_min_granularity;extern unsigned int sysctl_sched_wakeup_granularity;extern int sysctl_resched_latency_warn_ms;extern int sysctl_resched_latency_warn_once; --2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH v3 3/4] sched: reduce sched slice for SCHED_IDLE entities
Hi Jiang,
Quoted 17 lines · Jiang Biao
Why not just ignore min granularity when normal entities compete with a
SCHED_IDLEentity? something like this,kernel/sched/fair.c +0
-0@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); } - if (sched_feat(BASE_SLICE)) - slice =max (slice, (u64 )sysctl_sched_min_granularity); + if (sched_feat(BASE_SLICE) + && (!se_is_idle (init_se) ||sched_idle_cfs_rq (cfs_rq))) + slice =max (slice, (u64 )sysctl_sched_min_granularity);return slice; }
Quoted 1 line
If so, there seems no need to introduce
sysctl_sched_idle_min_granularity? :)
Ignoring min_gran entirely could lead to some really tiny slices; see discussion at https://lkml.org/lkml/2021/8/12/651.
Re: [PATCH v3 3/4] sched: reduce sched slice for SCHED_IDLE entities
Quoted 19 lines · Josh Don
Hi Jiang,
Quoted 15 lines · Jiang Biao
Why not just ignore min granularity when normal entities compete with a
SCHED_IDLEentity? something like this,(quoted diff) +0
-0@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice = __calc_delta(slice, se->load.weight, load); } - if (sched_feat(BASE_SLICE)) - slice = max(slice, (u64)sysctl_sched_min_granularity); + if (sched_feat(BASE_SLICE) + && (!se_is_idle(init_se) || sched_idle_cfs_rq(cfs_rq))) + slice = max(slice, (u64)sysctl_sched_min_granularity);
Quoted 6 lines
Quoted 3 lines
return slice;}
If so, there seems no need to introducesysctl_sched_idle_min_granularity? :)Ignoring
min_granentirely could lead to some really tiny slices; see discussion at https://lkml.org/lkml/2021/8/12/651.
Got it, tiny slices could be a problem in SCHED_HRTICK case. But the sysctl_sched_idle_min_granularity used in sched_slice() and sysctl_sched_min_granularity used in check_preempt_tick would have different semantics for SCHED_IDLE task, which could be functional ok but a little confusing.
Regards,
Jiang
[tip: sched/core] sched: reduce sched slice for SCHED_IDLE entities
In reply to Josh DonThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 7e2ce158699bb7b6a489c7c1d89c0dde2d4ceef5 Gitweb: https://git.kernel.org/tip/7e2ce158699bb7b6a489c7c1d89c0dde2d4ceef5 Author: Josh Don <[email protected]> AuthorDate: Thu, 19 Aug 2021 18:04:02 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 09 Sep 2021 11:27:31 +02:00
sched: reduce sched slice for SCHED_IDLE entities
Use a small, non-scaled min granularity for SCHED_IDLE entities, when competing with normal entities. This reduces the latency of getting a normal entity back on cpu, at the expense of increased context switch frequency of SCHED_IDLE entities.
The benefit of this change is to reduce the round-robin latency for normal entities when competing with a SCHED_IDLE entity.
Example: on a machine with HZ=1000, spawned two threads, one of which is SCHED_IDLE, and affined to one cpu. Without this patch, the SCHED_IDLE thread runs for 4ms then waits for 1.4s. With this patch, it runs for 1ms and waits 340ms (as it round-robins with the other thread).
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 29 ++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 5 deletions(-)
kernel/sched/debug.c +2 -0
@@ -305,6 +305,7 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("latency_ns" ,0644 , debugfs_sched, &sysctl_sched_latency);debugfs_create_u32 ("min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_min_granularity); +debugfs_create_u32 ("idle_min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_idle_min_granularity);debugfs_create_u32 ("wakeup_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_wakeup_granularity);debugfs_create_u32 ("latency_warn_ms" ,0644 , debugfs_sched, &sysctl_resched_latency_warn_ms);
@@ -806,6 +807,7 @@ static void sched_debug_header(struct seq_file *m) SEQ_printf(m, " .%-40s: %Ld.%06ld\n", #x, SPLIT_NS(x))PN (sysctl_sched_latency);PN (sysctl_sched_min_granularity); +PN (sysctl_sched_idle_min_granularity);PN (sysctl_sched_wakeup_granularity);P (sysctl_sched_child_runs_first);P (sysctl_sched_features);
kernel/sched/fair.c +24 -5
@@ -60,6 +60,14 @@ unsigned int sysctl_sched_min_granularity = 750000ULL;static unsigned int normalized_sysctl_sched_min_granularity =750000ULL ;/* +* Minimal preemption granularity for CPU-bound SCHED_IDLE tasks. +* Applies only when SCHED_IDLE tasks compete with normal tasks. +* +* (default: 0.75 msec) +*/ +unsigned int sysctl_sched_idle_min_granularity =750000ULL ; + +/* * This value is kept at sysctl_sched_latency/sysctl_sched_min_granularity */ static unsigned int sched_nr_latency =8 ;
@@ -665,6 +673,8 @@ static u64 __sched_period(unsigned long nr_running)return sysctl_sched_latency; } +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq); +/* * We calculate the wall-time slice from the period by taking a part * proportional to the weight.
@@ -674,6 +684,8 @@ static u64 __sched_period(unsigned long nr_running)static u64 sched_slice (struct cfs_rq *cfs_rq,struct sched_entity *se) {unsigned int nr_running = cfs_rq->nr_running; +struct sched_entity *init_se = se; +unsigned int min_gran;u64 slice;if (sched_feat (ALT_PERIOD ))
@@ -684,12 +696,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)for_each_sched_entity (se) {struct load_weight *load;struct load_weight lw; +struct cfs_rq *qcfs_rq; - cfs_rq =cfs_rq_of (se); - load = &cfs_rq->load; + qcfs_rq =cfs_rq_of (se); + load = &qcfs_rq->load;if (unlikely (!se->on_rq)) { - lw = cfs_rq->load; + lw = qcfs_rq->load;update_load_add (&lw, se->load.weight); load = &lw;
@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); } -if (sched_feat (BASE_SLICE )) - slice =max (slice, (u64 )sysctl_sched_min_granularity); +if (sched_feat (BASE_SLICE )) { +if (se_is_idle (init_se) && !sched_idle_cfs_rq (cfs_rq)) + min_gran = sysctl_sched_idle_min_granularity; +else + min_gran = sysctl_sched_min_granularity; + + slice =max_t (u64, slice, min_gran); + }return slice; }
kernel/sched/sched.h +1 -0
@@ -2400,6 +2400,7 @@ extern const_debug unsigned int sysctl_sched_migration_cost;#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency;extern unsigned int sysctl_sched_min_granularity; +extern unsigned int sysctl_sched_idle_min_granularity;extern unsigned int sysctl_sched_wakeup_granularity;extern int sysctl_resched_latency_warn_ms;extern int sysctl_resched_latency_warn_once;
[tip: sched/core] sched: reduce sched slice for SCHED_IDLE entities
In reply to Josh DonThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 51ce83ed523b00d58f2937ec014b12daaad55185 Gitweb: https://git.kernel.org/tip/51ce83ed523b00d58f2937ec014b12daaad55185 Author: Josh Don <[email protected]> AuthorDate: Thu, 19 Aug 2021 18:04:02 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 05 Oct 2021 15:51:37 +02:00
sched: reduce sched slice for SCHED_IDLE entities
Use a small, non-scaled min granularity for SCHED_IDLE entities, when competing with normal entities. This reduces the latency of getting a normal entity back on cpu, at the expense of increased context switch frequency of SCHED_IDLE entities.
The benefit of this change is to reduce the round-robin latency for normal entities when competing with a SCHED_IDLE entity.
Example: on a machine with HZ=1000, spawned two threads, one of which is SCHED_IDLE, and affined to one cpu. Without this patch, the SCHED_IDLE thread runs for 4ms then waits for 1.4s. With this patch, it runs for 1ms and waits 340ms (as it round-robins with the other thread).
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 29 ++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 3 files changed, 27 insertions(+), 5 deletions(-)
kernel/sched/debug.c +2 -0
@@ -311,6 +311,7 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("latency_ns" ,0644 , debugfs_sched, &sysctl_sched_latency);debugfs_create_u32 ("min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_min_granularity); +debugfs_create_u32 ("idle_min_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_idle_min_granularity);debugfs_create_u32 ("wakeup_granularity_ns" ,0644 , debugfs_sched, &sysctl_sched_wakeup_granularity);debugfs_create_u32 ("latency_warn_ms" ,0644 , debugfs_sched, &sysctl_resched_latency_warn_ms);
@@ -812,6 +813,7 @@ static void sched_debug_header(struct seq_file *m) SEQ_printf(m, " .%-40s: %Ld.%06ld\n", #x, SPLIT_NS(x))PN (sysctl_sched_latency);PN (sysctl_sched_min_granularity); +PN (sysctl_sched_idle_min_granularity);PN (sysctl_sched_wakeup_granularity);P (sysctl_sched_child_runs_first);P (sysctl_sched_features);
kernel/sched/fair.c +24 -5
@@ -60,6 +60,14 @@ unsigned int sysctl_sched_min_granularity = 750000ULL;static unsigned int normalized_sysctl_sched_min_granularity =750000ULL ;/* +* Minimal preemption granularity for CPU-bound SCHED_IDLE tasks. +* Applies only when SCHED_IDLE tasks compete with normal tasks. +* +* (default: 0.75 msec) +*/ +unsigned int sysctl_sched_idle_min_granularity =750000ULL ; + +/* * This value is kept at sysctl_sched_latency/sysctl_sched_min_granularity */ static unsigned int sched_nr_latency =8 ;
@@ -665,6 +673,8 @@ static u64 __sched_period(unsigned long nr_running)return sysctl_sched_latency; } +static bool sched_idle_cfs_rq (struct cfs_rq *cfs_rq); +/* * We calculate the wall-time slice from the period by taking a part * proportional to the weight.
@@ -674,6 +684,8 @@ static u64 __sched_period(unsigned long nr_running)static u64 sched_slice (struct cfs_rq *cfs_rq,struct sched_entity *se) {unsigned int nr_running = cfs_rq->nr_running; +struct sched_entity *init_se = se; +unsigned int min_gran;u64 slice;if (sched_feat (ALT_PERIOD ))
@@ -684,12 +696,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)for_each_sched_entity (se) {struct load_weight *load;struct load_weight lw; +struct cfs_rq *qcfs_rq; - cfs_rq =cfs_rq_of (se); - load = &cfs_rq->load; + qcfs_rq =cfs_rq_of (se); + load = &qcfs_rq->load;if (unlikely (!se->on_rq)) { - lw = cfs_rq->load; + lw = qcfs_rq->load;update_load_add (&lw, se->load.weight); load = &lw;
@@ -697,8 +710,14 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) slice =__calc_delta (slice, se->load.weight, load); } -if (sched_feat (BASE_SLICE )) - slice =max (slice, (u64 )sysctl_sched_min_granularity); +if (sched_feat (BASE_SLICE )) { +if (se_is_idle (init_se) && !sched_idle_cfs_rq (cfs_rq)) + min_gran = sysctl_sched_idle_min_granularity; +else + min_gran = sysctl_sched_min_granularity; + + slice =max_t (u64, slice, min_gran); + }return slice; }
kernel/sched/sched.h +1 -0
@@ -2403,6 +2403,7 @@ extern const_debug unsigned int sysctl_sched_migration_cost;#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency;extern unsigned int sysctl_sched_min_granularity; +extern unsigned int sysctl_sched_idle_min_granularity;extern unsigned int sysctl_sched_wakeup_granularity;extern int sysctl_resched_latency_warn_ms;extern int sysctl_resched_latency_warn_once;
PATCH v3 4/4 sched: adjust sleeper credit for SCHED_IDLE entities
In reply to Josh DonGive reduced sleeper credit to SCHED_IDLE entities. As a result, woken SCHED_IDLE entities will take longer to preempt normal entities.
The benefit of this change is to make it less likely that a newly woken SCHED_IDLE entity will preempt a short-running normal entity before it blocks.
We still give a small sleeper credit to SCHED_IDLE entities, so that idle<->idle competition retains some fairness.
Example: With HZ=1000, spawned four threads affined to one cpu, one of which was set to SCHED_IDLE. Without this patch, wakeup latency for the SCHED_IDLE thread was ~1-2ms, with the patch the wakeup latency was ~5ms.
--- kernel/sched/fair.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/fair.c +6 -1
@@ -4230,7 +4230,12 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)/* sleeps up to a single latency don't count. */ if (!initial) { -unsigned long thresh = sysctl_sched_latency; +unsigned long thresh; + +if (se_is_idle (se)) + thresh = sysctl_sched_min_granularity; +else + thresh = sysctl_sched_latency;/* * Halve their sleep time's effect, to allow
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH v3 4/4] sched: adjust sleeper credit for SCHED_IDLE entities
Quoted 11 lines · Josh Don
Give reduced sleeper credit to
SCHED_IDLEentities. As a result, wokenSCHED_IDLEentities will take longer to preempt normal entities.The benefit of this change is to make it less likely that a newly woken
SCHED_IDLEentity will preempt a short-running normal entity before it blocks.We still give a small sleeper credit to
SCHED_IDLEentities, so that idle<->idle competition retains some fairness.
Quoted 6 lines
Example: With HZ=1000, spawned four threads affined to one cpu, one of which was set to
SCHED_IDLE. Without this patch, wakeup latency for theSCHED_IDLEthread was ~1-2ms, with the patch the wakeup latency was ~5ms.Signed-off-by Josh Don <[email protected]>
Quoted 25 lines
--- kernel/sched/fair.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)kernel/sched/fair.c +6
-2@@ -4230,7 +4230,12 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)/* sleeps up to a single latency don't count. */ if (!initial) { -unsigned long thresh = sysctl_sched_latency; +unsigned long thresh; + +if (se_is_idle (se)) + thresh = sysctl_sched_min_granularity; +else + thresh = sysctl_sched_latency;/* * Halve their sleep time's effect, to allow -- 2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH v3 4/4] sched: adjust sleeper credit for SCHED_IDLE entities
In reply to Josh DonHi,
Quoted 11 lines · Josh Don
Give reduced sleeper credit to
SCHED_IDLEentities. As a result, wokenSCHED_IDLEentities will take longer to preempt normal entities.The benefit of this change is to make it less likely that a newly woken
SCHED_IDLEentity will preempt a short-running normal entity before it blocks.We still give a small sleeper credit to
SCHED_IDLEentities, so that idle<->idle competition retains some fairness.
Quoted 6 lines
Example: With HZ=1000, spawned four threads affined to one cpu, one of which was set to
SCHED_IDLE. Without this patch, wakeup latency for theSCHED_IDLEthread was ~1-2ms, with the patch the wakeup latency was ~5ms.Signed-off-by Josh Don <[email protected]>
Tried to push a similar patch before, but failed. :)
https://lkml.org/lkml/2020/8/20/1773
Please pick my Reviewed-by if you don't mind,
Quoted 25 lines
--- kernel/sched/fair.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)kernel/sched/fair.c +6
-2@@ -4230,7 +4230,12 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)/* sleeps up to a single latency don't count. */ if (!initial) { -unsigned long thresh = sysctl_sched_latency; +unsigned long thresh; + +if (se_is_idle (se)) + thresh = sysctl_sched_min_granularity; +else + thresh = sysctl_sched_latency;/* * Halve their sleep time's effect, to allow -- 2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH v3 4/4] sched: adjust sleeper credit for SCHED_IDLE entities
Quoted 20 lines · Jiang Biao
Hi,
Quoted 16 lines · Josh Don
Give reduced sleeper credit to
SCHED_IDLEentities. As a result, wokenSCHED_IDLEentities will take longer to preempt normal entities.The benefit of this change is to make it less likely that a newly woken
SCHED_IDLEentity will preempt a short-running normal entity before it blocks.We still give a small sleeper credit to
SCHED_IDLEentities, so that idle<->idle competition retains some fairness.Example: With HZ=1000, spawned four threads affined to one cpu, one of which was set to
SCHED_IDLE. Without this patch, wakeup latency for theSCHED_IDLEthread was ~1-2ms, with the patch the wakeup latency was ~5ms.
Quoted 5 lines
Quoted 1 line
Signed-off-by Josh Don <[email protected]>Tried to push a similar patch before, but failed. :)
https://lkml.org/lkml/2020/8/20/1773
Please pick my Reviewed-by if you don't mind,Reviewed-by Jiang Biao <[email protected]>
Done, thanks :)
[tip: sched/core] sched: adjust sleeper credit for SCHED_IDLE entities
In reply to Josh DonThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: bb1fc3bc521782d018902143c8301ab4a5e53557 Gitweb: https://git.kernel.org/tip/bb1fc3bc521782d018902143c8301ab4a5e53557 Author: Josh Don <[email protected]> AuthorDate: Thu, 19 Aug 2021 18:04:03 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 09 Sep 2021 11:27:31 +02:00
sched: adjust sleeper credit for SCHED_IDLE entities
Give reduced sleeper credit to SCHED_IDLE entities. As a result, woken SCHED_IDLE entities will take longer to preempt normal entities.
The benefit of this change is to make it less likely that a newly woken SCHED_IDLE entity will preempt a short-running normal entity before it blocks.
We still give a small sleeper credit to SCHED_IDLE entities, so that idle<->idle competition retains some fairness.
Example: With HZ=1000, spawned four threads affined to one cpu, one of which was set to SCHED_IDLE. Without this patch, wakeup latency for the SCHED_IDLE thread was ~1-2ms, with the patch the wakeup latency was ~5ms.
--- kernel/sched/fair.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/fair.c +6 -1
@@ -4201,7 +4201,12 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)/* sleeps up to a single latency don't count. */ if (!initial) { -unsigned long thresh = sysctl_sched_latency; +unsigned long thresh; + +if (se_is_idle (se)) + thresh = sysctl_sched_min_granularity; +else + thresh = sysctl_sched_latency;/* * Halve their sleep time's effect, to allow
[tip: sched/core] sched: adjust sleeper credit for SCHED_IDLE entities
In reply to Josh DonThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2cae3948edd488ebdef4deaf1d1043f92f47e665 Gitweb: https://git.kernel.org/tip/2cae3948edd488ebdef4deaf1d1043f92f47e665 Author: Josh Don <[email protected]> AuthorDate: Thu, 19 Aug 2021 18:04:03 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 05 Oct 2021 15:51:39 +02:00
sched: adjust sleeper credit for SCHED_IDLE entities
Give reduced sleeper credit to SCHED_IDLE entities. As a result, woken SCHED_IDLE entities will take longer to preempt normal entities.
The benefit of this change is to make it less likely that a newly woken SCHED_IDLE entity will preempt a short-running normal entity before it blocks.
We still give a small sleeper credit to SCHED_IDLE entities, so that idle<->idle competition retains some fairness.
Example: With HZ=1000, spawned four threads affined to one cpu, one of which was set to SCHED_IDLE. Without this patch, wakeup latency for the SCHED_IDLE thread was ~1-2ms, with the patch the wakeup latency was ~5ms.
--- kernel/sched/fair.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/fair.c +6 -1
@@ -4230,7 +4230,12 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)/* sleeps up to a single latency don't count. */ if (!initial) { -unsigned long thresh = sysctl_sched_latency; +unsigned long thresh; + +if (se_is_idle (se)) + thresh = sysctl_sched_min_granularity; +else + thresh = sysctl_sched_latency;/* * Halve their sleep time's effect, to allow