Skip to contentSkip to conversation

powerpc/perf: Use cpumask_last() to determine the designated cpu for nest/core units.

PATCH Applied Anju T Sudhakar 3 participants 4 messages Jun 10, 2019 – Jun 30, 2019 PowerPC

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 powerpc/perf: Use cpumask_last() to determine the designated cpu for nest/core units.

Nest and core imc(In-memory Collection counters) assigns a particular cpu as the designated target for counter data collection. During system boot, the first online cpu in a chip gets assigned as the designated cpu for that chip(for nest-imc) and the first online cpu in a core gets assigned as the designated cpu for that core(for core-imc).

If the designated cpu goes offline, the next online cpu from the same chip(for nest-imc)/core(for core-imc) is assigned as the next target, and the event context is migrated to the target cpu. Currently, cpumask_any_but() function is used to find the target cpu. Though this function is expected to return a random cpu, this always returns the next online cpu.

If all cpus in a chip/core is offlined in a sequential manner, starting from the first cpu, the event migration has to happen for all the cpus which goes offline. Since the migration process involves a grace period, the total time taken to offline all the cpus will be significantly high.

Example:
In a system which has 2 sockets, with

NUMA node0 CPU(s):     0-87
NUMA node8 CPU(s):     88-175

Time taken to offline cpu 88-175:

real    2m56.099s
user    0m0.191s
sys     0m0.000s

Use cpumask_last() to choose the target cpu, when the designated cpu goes online, so the migration will happen only when the last_cpu in the mask goes offline. This way the time taken to offline all cpus in a chip/core can be reduced.

With the patch,

Time taken to offline cpu 88-175:

real    0m12.207s
user    0m0.171s
sys     0m0.000s

Offlining all cpus in reverse order is also taken care because, cpumask_any_but() is used to find the designated cpu if the last cpu in the mask goes offline. Since cpumask_any_but() always return the first cpu in the mask, that becomes the designated cpu and migration will happen only when the first_cpu in the mask goes offline.

Example:
With the patch,

Time taken to offline cpu from 175-88:

real    0m9.330s
user    0m0.110s
sys     0m0.000s

---

Changes from v1:
Modified the commit log with more info.
---

 arch/powerpc/perf/imc-pmu.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
arch/powerpc/perf/imc-pmu.c +12 -2
index 31fa753..fbfd6e7 100644
@@ -366,7 +366,14 @@ static int ppc_nest_imc_cpu_offline(unsigned int cpu)
 	 */
 	nid = cpu_to_node(cpu);
 	l_cpumask = cpumask_of_node(nid);
-	target = cpumask_any_but(l_cpumask, cpu);
+	target = cpumask_last(l_cpumask);
+
+	/*
+	 * If this(target) is the last cpu in the cpumask for this chip,
+	 * check for any possible online cpu in the chip.
+	 */
+	if (unlikely(target == cpu))
+		target = cpumask_any_but(l_cpumask, cpu);
 
 	/*
 	 * Update the cpumask with the target cpu and
@@ -671,7 +678,10 @@ static int ppc_core_imc_cpu_offline(unsigned int cpu)
 		return 0;
 
 	/* Find any online cpu in that core except the current "cpu" */
-	ncpu = cpumask_any_but(cpu_sibling_mask(cpu), cpu);
+	ncpu = cpumask_last(cpu_sibling_mask(cpu));
+
+	if (unlikely(ncpu == cpu))
+		ncpu = cpumask_any_but(cpu_sibling_mask(cpu), cpu);
 
 	if (ncpu >= 0 && ncpu < nr_cpu_ids) {
 		cpumask_set_cpu(ncpu, &core_imc_cpumask);
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 47 lines · Anju T Sudhakar

Nest and core imc(In-memory Collection counters) assigns a particular cpu as the designated target for counter data collection. During system boot, the first online cpu in a chip gets assigned as the designated cpu for that chip(for nest-imc) and the first online cpu in a core gets assigned as the designated cpu for that core(for core-imc).

If the designated cpu goes offline, the next online cpu from the same chip(for nest-imc)/core(for core-imc) is assigned as the next target, and the event context is migrated to the target cpu. Currently, cpumask_any_but() function is used to find the target cpu. Though this function is expected to return a random cpu, this always returns the next online cpu.

If all cpus in a chip/core is offlined in a sequential manner, starting from the first cpu, the event migration has to happen for all the cpus which goes offline. Since the migration process involves a grace period, the total time taken to offline all the cpus will be significantly high.

Example:
In a system which has 2 sockets, with

NUMA node0 CPU(s):     0-87
NUMA node8 CPU(s):     88-175

Time taken to offline cpu 88-175:

real    2m56.099s
user    0m0.191s
sys     0m0.000s

Use cpumask_last() to choose the target cpu, when the designated cpu goes online, so the migration will happen only when the last_cpu in the mask goes offline. This way the time taken to offline all cpus in a chip/core can be reduced.

With the patch,

Time taken to offline cpu 88-175:

real    0m12.207s
user    0m0.171s
sys     0m0.000s

Offlining all cpus in reverse order is also taken care because, cpumask_any_but() is used to find the designated cpu if the last cpu in the mask goes offline. Since cpumask_any_but() always return the first cpu in the mask, that becomes the designated cpu and migration will happen only when the first_cpu in the mask goes offline.

Quoted 7 lines

Example:
With the patch,

Time taken to offline cpu from 175-88:

real    0m9.330s
user    0m0.110s
sys     0m0.000s

Seems like a very interesting work.
Out of curiosity, have you used 'chcpu -d' to create your benchmark?

Quoted 44 lines

---

Changes from v1:
Modified the commit log with more info.
---

 arch/powerpc/perf/imc-pmu.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
arch/powerpc/perf/imc-pmu.c +12 -2
index 31fa753..fbfd6e7 100644
@@ -366,7 +366,14 @@ static int ppc_nest_imc_cpu_offline(unsigned int cpu)
 	 */
 	nid = cpu_to_node(cpu);
 	l_cpumask = cpumask_of_node(nid);
-	target = cpumask_any_but(l_cpumask, cpu);
+	target = cpumask_last(l_cpumask);
+
+	/*
+	 * If this(target) is the last cpu in the cpumask for this chip,
+	 * check for any possible online cpu in the chip.
+	 */
+	if (unlikely(target == cpu))
+		target = cpumask_any_but(l_cpumask, cpu);
 
 	/*
 	 * Update the cpumask with the target cpu and
@@ -671,7 +678,10 @@ static int ppc_core_imc_cpu_offline(unsigned int cpu)
 		return 0;
 
 	/* Find any online cpu in that core except the current "cpu" */
-	ncpu = cpumask_any_but(cpu_sibling_mask(cpu), cpu);
+	ncpu = cpumask_last(cpu_sibling_mask(cpu));
+
+	if (unlikely(ncpu == cpu))
+		ncpu = cpumask_any_but(cpu_sibling_mask(cpu), cpu);
 
 	if (ncpu >= 0 && ncpu < nr_cpu_ids) {
 		cpumask_set_cpu(ncpu, &core_imc_cpumask);
#2 lore raw

Hi Leonardo,

Quoted 14 lines · Leonardo Bras
Quoted 13 lines · Anju T Sudhakar

Nest and core imc(In-memory Collection counters) assigns a particular cpu as the designated target for counter data collection. During system boot, the first online cpu in a chip gets assigned as the designated cpu for that chip(for nest-imc) and the first online cpu in a core gets assigned as the designated cpu for that core(for core-imc).

If the designated cpu goes offline, the next online cpu from the same chip(for nest-imc)/core(for core-imc) is assigned as the next target, and the event context is migrated to the target cpu. Currently, cpumask_any_but() function is used to find the target cpu. Though this function is expected to return a random cpu, this always returns the next online cpu.

Quoted 6 lines
Quoted 4 lines

If all cpus in a chip/core is offlined in a sequential manner, starting from the first cpu, the event migration has to happen for all the cpus which goes offline. Since the migration process involves a grace period, the total time taken to offline all the cpus will be significantly high.

Seems like a very interesting work.
Out of curiosity, have you used 'chcpu -d' to create your benchmark?

Here I did not use chcpu to disable the cpu.

I used a script which will offline cpus 88-175 by echoing  0 to

/sys/devices/system/cpu/cpu*/online.

Regards,

Anju

#3 lore raw
In reply to Anju T Sudhakar
Quoted 50 lines · Anju T Sudhakar

Nest and core imc(In-memory Collection counters) assigns a particular cpu as the designated target for counter data collection. During system boot, the first online cpu in a chip gets assigned as the designated cpu for that chip(for nest-imc) and the first online cpu in a core gets assigned as the designated cpu for that core(for core-imc).

If the designated cpu goes offline, the next online cpu from the same chip(for nest-imc)/core(for core-imc) is assigned as the next target, and the event context is migrated to the target cpu. Currently, cpumask_any_but() function is used to find the target cpu. Though this function is expected to return a random cpu, this always returns the next online cpu.

If all cpus in a chip/core is offlined in a sequential manner, starting from the first cpu, the event migration has to happen for all the cpus which goes offline. Since the migration process involves a grace period, the total time taken to offline all the cpus will be significantly high.

Example:
In a system which has 2 sockets, with

NUMA node0 CPU(s):     0-87
NUMA node8 CPU(s):     88-175

Time taken to offline cpu 88-175:

real    2m56.099s
user    0m0.191s
sys     0m0.000s

Use cpumask_last() to choose the target cpu, when the designated cpu goes online, so the migration will happen only when the last_cpu in the mask goes offline. This way the time taken to offline all cpus in a chip/core can be reduced.

With the patch,

Time taken to offline cpu 88-175:

real    0m12.207s
user    0m0.171s
sys     0m0.000s

Offlining all cpus in reverse order is also taken care because, cpumask_any_but() is used to find the designated cpu if the last cpu in the mask goes offline. Since cpumask_any_but() always return the first cpu in the mask, that becomes the designated cpu and migration will happen only when the first_cpu in the mask goes offline.

Example:
With the patch,

Quoted 7 lines

Time taken to offline cpu from 175-88:

real    0m9.330s
user    0m0.110s
sys     0m0.000s

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/9c9f8fb71feed144973a70455e0a4ee3da57ed2a

cheers