Skip to contentSkip to conversation

kernel/sched/topology: Clarify root domain(s) debug string

PATCH In review Juri Lelli 4 participants 5 messages May 24, 2018 – May 28, 2018 Scheduler

lore
Share & tools
lore

Juri Lelli

author maintainer
#0 lore raw

PATCH 1/1 kernel/sched/topology: Clarify root domain(s) debug string

When scheduler debug is enabled, building scheduling domains outputs information about how the domains are laid out and to which root domain each CPU (or sets of CPUs) belongs, e.g.:

CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

[...]

span: 0-5 (max cpu_capacity = 1024)

The fact that latest line refers to CPUs 0-5 root domain doesn't however look immediately obvious to me: one might wonder why span 0-5 is reported "again".

Make it more clear by adding "root domain" to it, as to end with the following.

CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

[...]

root domain span: 0-5 (max cpu_capacity = 1024)

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Patrick Bellasi <[email protected]>
---
 kernel/sched/topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/topology.c +1 -1
index 64cc564f5255..61a1125c1ae4 100644
@@ -1708,7 +1708,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	rcu_read_unlock();
 
 	if (rq && sched_debug_enabled) {
-		pr_info("span: %*pbl (max cpu_capacity = %lu)\n",
+		pr_info("root domain span: %*pbl (max cpu_capacity = %lu)\n",
 			cpumask_pr_args(cpu_map), rq->rd->max_cpu_capacity);
 	}
 
Signature · 1 line
2.14.3
#1 lore raw

[tip:sched/core] sched/topology: Clarify root domain(s) debug string

sched/topology: Clarify root domain(s) debug string
Commit-ID:  bf5015a50f1fdb248b48405b67cae24dc02605d6
Gitweb:     https://git.kernel.org/tip/bf5015a50f1fdb248b48405b67cae24dc02605d6
Author:     Juri Lelli <[email protected]>
AuthorDate: Thu, 24 May 2018 17:29:36 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 25 May 2018 08:03:38 +0200

sched/topology: Clarify root domain(s) debug string

When scheduler debug is enabled, building scheduling domains outputs information about how the domains are laid out and to which root domain each CPU (or sets of CPUs) belongs, e.g.:

CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

[...]

span: 0-5 (max cpu_capacity = 1024)

The fact that latest line refers to CPUs 0-5 root domain doesn't however look immediately obvious to me: one might wonder why span 0-5 is reported "again".

Make it more clear by adding "root domain" to it, as to end with the following:

CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

[...]

root domain span: 0-5 (max cpu_capacity = 1024)

---
 kernel/sched/topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/topology.c +1 -1
index 64cc564f5255..61a1125c1ae4 100644
@@ -1708,7 +1708,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
 	rcu_read_unlock();
 
 	if (rq && sched_debug_enabled) {
-		pr_info("span: %*pbl (max cpu_capacity = %lu)\n",
+		pr_info("root domain span: %*pbl (max cpu_capacity = %lu)\n",
 			cpumask_pr_args(cpu_map), rq->rd->max_cpu_capacity);
 	}
 
#2 lore raw
In reply to Juri Lelli
Quoted 48 lines · Juri Lelli

When scheduler debug is enabled, building scheduling domains outputs information about how the domains are laid out and to which root domain each CPU (or sets of CPUs) belongs, e.g.:

CPU0 attaching sched-domain(s):

   domain-0: span=0-5 level=MC
    groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
  CPU1 attaching sched-domain(s):
   domain-0: span=0-5 level=MC
    groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

[...]

span: 0-5 (max cpu_capacity = 1024)

The fact that latest line refers to CPUs 0-5 root domain doesn't however look immediately obvious to me: one might wonder why span 0-5 is reported "again".

Make it more clear by adding "root domain" to it, as to end with the following.

CPU0 attaching sched-domain(s):

   domain-0: span=0-5 level=MC
    groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
  CPU1 attaching sched-domain(s):
   domain-0: span=0-5 level=MC
    groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

[...]

root domain span: 0-5 (max cpu_capacity = 1024)

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Patrick Bellasi <[email protected]>
---
  kernel/sched/topology.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/topology.c +0 -0
index 64cc564f5255..61a1125c1ae4 100644

kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 1714

Quoted 8 lines
kernel/sched/topology.c +1 -1
@@ -1708,7 +1708,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
  	rcu_read_unlock();
 
  	if (rq && sched_debug_enabled) {
-		pr_info("span: %*pbl (max cpu_capacity = %lu)\n",
+		pr_info("root domain span: %*pbl (max cpu_capacity = %lu)\n",
  			cpumask_pr_args(cpu_map), rq->rd->max_cpu_capacity);
  	}
 

Looks good to me. Probably especially helpful when setting up exclusive cpusets.

Juno with big and little exclusive cpuset:

...

[  124.231333] CPU1 attaching sched-domain(s):
[  124.235482]  domain-0: span=1-2 level=MC
[  124.239382]   groups: 1:{ span=1 }, 2:{ span=2 }
[  124.243969] CPU2 attaching sched-domain(s):
[  124.248112]  domain-0: span=1-2 level=MC
[  124.251998]   groups: 2:{ span=2 }, 1:{ span=1 }
[  124.256585] root domain span: 1-2 (max cpu_capacity = 1024)
[  124.262150] CPU0 attaching sched-domain(s):
[  124.266307]  domain-0: span=0,3-5 level=MC
[  124.270366]   groups: 0:{ span=0 cap=446 }, 3:{ span=3 cap=446 }, 4:{ span=4 cap=446 }, 5:{ span=5 cap=446 }
[  124.280131] CPU3 attaching sched-domain(s):
[  124.284273]  domain-0: span=0,3-5 level=MC
[  124.288334]   groups: 3:{ span=3 cap=446 }, 4:{ span=4 cap=446 }, 5:{ span=5 cap=446 }, 0:{ span=0 cap=446 }
[  124.298096] CPU4 attaching sched-domain(s):
[  124.302239]  domain-0: span=0,3-5 level=MC
[  124.306298]   groups: 4:{ span=4 cap=446 }, 5:{ span=5 cap=446 }, 0:{ span=0 cap=446 }, 3:{ span=3 cap=446 }
[  124.316063] CPU5 attaching sched-domain(s):
[  124.320205]  domain-0: span=0,3-5 level=MC
[  124.324265]   groups: 5:{ span=5 cap=446 }, 0:{ span=0 cap=446 }, 3:{ span=3 cap=446 }, 4:{ span=4 cap=446 }
[  124.334031] root domain span: 0,3-5 (max cpu_capacity = 446)
author maintainer
#3 lore raw

On 25/05/18 13:35, Dietmar Eggemann wrote:

[...]

Quoted 23 lines

Looks good to me. Probably especially helpful when setting up exclusive cpusets.

Juno with big and little exclusive cpuset:

...

[  124.231333] CPU1 attaching sched-domain(s):
[  124.235482]  domain-0: span=1-2 level=MC
[  124.239382]   groups: 1:{ span=1 }, 2:{ span=2 }
[  124.243969] CPU2 attaching sched-domain(s):
[  124.248112]  domain-0: span=1-2 level=MC
[  124.251998]   groups: 2:{ span=2 }, 1:{ span=1 }
[  124.256585] root domain span: 1-2 (max cpu_capacity = 1024)
[  124.262150] CPU0 attaching sched-domain(s):
[  124.266307]  domain-0: span=0,3-5 level=MC
[  124.270366]   groups: 0:{ span=0 cap=446 }, 3:{ span=3 cap=446 }, 4:{
span=4 cap=446 }, 5:{ span=5 cap=446 }
[  124.280131] CPU3 attaching sched-domain(s):
[  124.284273]  domain-0: span=0,3-5 level=MC
[  124.288334]   groups: 3:{ span=3 cap=446 }, 4:{ span=4 cap=446 }, 5:{
span=5 cap=446 }, 0:{ span=0 cap=446 }
[  124.298096] CPU4 attaching sched-domain(s):
Quoted 8 lines
[  124.302239]  domain-0: span=0,3-5 level=MC
[  124.306298]   groups: 4:{ span=4 cap=446 }, 5:{ span=5 cap=446 }, 0:{
span=0 cap=446 }, 3:{ span=3 cap=446 }
[  124.316063] CPU5 attaching sched-domain(s):
[  124.320205]  domain-0: span=0,3-5 level=MC
[  124.324265]   groups: 5:{ span=5 cap=446 }, 0:{ span=0 cap=446 }, 3:{
span=3 cap=446 }, 4:{ span=4 cap=446 }
[  124.334031] root domain span: 0,3-5 (max cpu_capacity = 446)

Yep, that's similar to how I noticed.

Thanks for testing.

Best,

- Juri

Reviewed-by on v1 1/1
#4 lore raw
In reply to Juri Lelli
Quoted 11 lines · Juri Lelli

When scheduler debug is enabled, building scheduling domains outputs information about how the domains are laid out and to which root domain each CPU (or sets of CPUs) belongs, e.g.:

CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

Quoted 5 lines

[...]

span: 0-5 (max cpu_capacity = 1024)

The fact that latest line refers to CPUs 0-5 root domain doesn't however look

last line?

Quoted 12 lines

immediately obvious to me: one might wonder why span 0-5 is reported "again".

Make it more clear by adding "root domain" to it, as to end with the following.

CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }

Quoted 5 lines

[...]

root domain span: 0-5 (max cpu_capacity = 1024)

I played the sched_load_balance flag to trigger this and it makes sense to improve the print with 'root domain'.

One thing I believe is a bit weird is sched_load_balance also can affect the wake-up path, because a NULL sd is attached to the rq if sched_load_balance is set to 0.

This could turn off the "for_each_domain(cpu, tmp)" loop in select_task_rq_fair and hence we would always end up in the select_idle_sibling path for those CPUs.

It also means that "XXX always" can/should be removed because sd can very well be NULL for other sd_flag types as well, not just sd_flag == SD_BALANCE_WAKE. I'll send a patch to remove that comment as I just tested this is true.

thanks,

- Joel