Optimization to improve CPU online/offline on Powerpc
v2 cover Optimization to improve CPU online/offline on Powerpc
Here are some optimizations and fixes to make CPU online/offline faster and hence result in faster bootup.
Its based on top of my v5 coregroup support patchset.
https://lore.kernel.org/linuxppc-dev/[email protected]/t/#u
Anton reported that his 4096 cpu (1024 cores in a socket) was taking too long to boot. He also analyzed that most of the time was being spent on updating cpu_core_mask.
The first two patches should solve Anton's immediate problem. On the unofficial patches, Anton reported that the boot time came from 30 mins to 6 seconds. (Basically a high core count in a single socket configuration). Satheesh also reported similar numbers.
The rest are cleanups/optimizations.
Since cpu_core_mask is an exported symbol for a long duration, lets retain as a snapshot of cpumask_of_node.
$ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 1024 On-line CPU(s) list: 0-1023 Thread(s) per core: 8 Core(s) per socket: 8 Socket(s): 16 NUMA node(s): 16 Model: 2.0 (pvr 004d 0200) Model name: POWER8 (architected), altivec supported Hypervisor vendor: pHyp Virtualization type: para L1d cache: 64K L1i cache: 32K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0-63 NUMA node1 CPU(s): 64-127 NUMA node2 CPU(s): 128-191 NUMA node3 CPU(s): 192-255 NUMA node4 CPU(s): 256-319 NUMA node5 CPU(s): 320-383 NUMA node6 CPU(s): 384-447 NUMA node7 CPU(s): 448-511 NUMA node8 CPU(s): 512-575 NUMA node9 CPU(s): 576-639 NUMA node10 CPU(s): 640-703 NUMA node11 CPU(s): 704-767 NUMA node12 CPU(s): 768-831 NUMA node13 CPU(s): 832-895 NUMA node14 CPU(s): 896-959 NUMA node15 CPU(s): 960-1023
$ dmesg -k | grep -i -e Bringing -e Brought -e sysrq -e bug With powerp/next [ 0.000000] printk: debug: ignoring loglevel setting. [ 0.354971] smp: Bringing up secondary CPUs ... [ 233.354676] smp: Brought up 16 nodes, 1024 CPUs [ 330.023073] sysrq: Changing Loglevel [ 330.023101] sysrq: Loglevel set to 9
With +patchset
[ 0.000000] printk: debug: ignoring loglevel setting. [ 0.351703] smp: Bringing up secondary CPUs ... [ 4.059859] smp: Brought up 16 nodes, 1024 CPUs [ 98.309015] sysrq: Changing Loglevel [ 98.309044] sysrq: Loglevel set to 9
Observations:
CPU bringup time reduced to 4 seconds from 233 seconds on this 1024 CPU system. This resulted in System boot up time reducing to 98 seconds from 330 seconds. The actual improvement would depend on your system topology.
Topology verification post patchset on a 2 node Power9 PowerVM LPAR
powerpc/next +patchset
------------ ---------
$ lscpu Architecture: ppc64le Architecture: ppc64le Byte Order: Little Endian Byte Order: Little Endian CPU(s): 128 CPU(s): 128 On-line CPU(s) list: 0-127 On-line CPU(s) list: 0-127 Thread(s) per core: 8 Thread(s) per core: 8 Core(s) per socket: 8 Core(s) per socket: 8 Socket(s): 2 Socket(s): 2 NUMA node(s): 2 NUMA node(s): 2 Model: 2.2 (pvr 004e 0202) Model: 2.2 (pvr 004e 0202) Model name: POWER9 (architected), altivec supported Model name: POWER9 (architected), altivec supported Hypervisor vendor: pHyp Hypervisor vendor: pHyp Virtualization type: para Virtualization type: para L1d cache: 32K L1d cache: 32K L1i cache: 32K L1i cache: 32K L2 cache: 512K L2 cache: 512K L3 cache: 10240K L3 cache: 10240K NUMA node0 CPU(s): 0-63 NUMA node0 CPU(s): 0-63 NUMA node1 CPU(s): 64-127 NUMA node1 CPU(s): 64-127
$ tail -f /proc/cpuinfo processor : 127 processor : 127 cpu : POWER9 (architected), altivec supported cpu : POWER9 (architected), altivec supported clock : 3000.000000MHz clock : 3000.000000MHz revision : 2.2 (pvr 004e 0202) revision : 2.2 (pvr 004e 0202)
timebase : 512000000 timebase : 512000000
platform : pSeries platform : pSeries
model : IBM,9008-22L model : IBM,9008-22L
machine : CHRP IBM,9008-22L machine : CHRP IBM,9008-22L
MMU : Radix MMU : Radix
$ grep . /proc/sys/kernel/sched_domain/cpu0/domain*/name -------------------------------------------------------- /proc/sys/kernel/sched_domain/cpu0/domain0/name:SMT /proc/sys/kernel/sched_domain/cpu0/domain0/name:SMT /proc/sys/kernel/sched_domain/cpu0/domain1/name:CACHE /proc/sys/kernel/sched_domain/cpu0/domain1/name:CACHE /proc/sys/kernel/sched_domain/cpu0/domain2/name:DIE /proc/sys/kernel/sched_domain/cpu0/domain2/name:DIE /proc/sys/kernel/sched_domain/cpu0/domain3/name:NUMA /proc/sys/kernel/sched_domain/cpu0/domain3/name:NUMA
$ grep . /proc/sys/kernel/sched_domain/cpu0/domain*/flags --------------------------------------------------------- /proc/sys/kernel/sched_domain/cpu0/domain0/flags:2391 /proc/sys/kernel/sched_domain/cpu0/domain0/flags:2391 /proc/sys/kernel/sched_domain/cpu0/domain1/flags:2327 /proc/sys/kernel/sched_domain/cpu0/domain1/flags:2327 /proc/sys/kernel/sched_domain/cpu0/domain2/flags:2071 /proc/sys/kernel/sched_domain/cpu0/domain2/flags:2071 /proc/sys/kernel/sched_domain/cpu0/domain3/flags:12801 /proc/sys/kernel/sched_domain/cpu0/domain3/flags:12801
Post ppc64_cpu --smt=1
$ tail -f /proc/cpuinfo processor : 120 processor : 120 cpu : POWER9 (architected), altivec supported cpu : POWER9 (architected), altivec supported clock : 3000.000000MHz clock : 3000.000000MHz revision : 2.2 (pvr 004e 0202) revision : 2.2 (pvr 004e 0202)
timebase : 512000000 timebase : 512000000
platform : pSeries platform : pSeries
model : IBM,9008-22L model : IBM,9008-22L
machine : CHRP IBM,9008-22L machine : CHRP IBM,9008-22L
MMU : Radix MMU : Radix
$ grep . /proc/sys/kernel/sched_domain/cpu0/domain*/name -------------------------------------------------------- /proc/sys/kernel/sched_domain/cpu0/domain0/name:DIE /proc/sys/kernel/sched_domain/cpu0/domain0/name:DIE /proc/sys/kernel/sched_domain/cpu0/domain1/name:NUMA /proc/sys/kernel/sched_domain/cpu0/domain1/name:NUMA
$ grep . /proc/sys/kernel/sched_domain/cpu0/domain*/flags --------------------------------------------------------- /proc/sys/kernel/sched_domain/cpu0/domain0/flags:2071 /proc/sys/kernel/sched_domain/cpu0/domain0/flags:2071 /proc/sys/kernel/sched_domain/cpu0/domain1/flags:12801 /proc/sys/kernel/sched_domain/cpu0/domain1/flags:12801
Performance impact post +patchset
---------------------------------
100 iterations of ebizzy
Units: Records/second : higher is better
-----------------------------------------
kernel N Min Max Median Avg Stddev powerpc/next 100 753917 870520 819054 817636.56 22649.7 +patchset 100 746258 874984 816681 813876.74 26424.351
100 iterations of perf bench sched pipe -l 10000000 (aka Hackbench)
units: usec/ops: lesser is better
--------------------------------
kernel N Min Max Median Avg Stddev powerpc/next 100 13.845834 14.569539 14.06263 14.086167 0.17512607 +patchset 100 13.637611 18.097744 13.862656 13.9257 0.43872453
schbench Latency percentiles (usec)
units: usec : lesser is better
-----------------------------------
powerpc/next +patchset 50.0000th: 48 50.0000th: 49 75.0000th: 65 75.0000th: 66 90.0000th: 77 90.0000th: 79 95.0000th: 84 95.0000th: 85 *99.0000th: 101 *99.0000th: 99 99.5000th: 113 99.5000th: 104 99.9000th: 159 99.9000th: 129 min=0, max=15221 min=0, max=7666
100 interations of ppc64_cpu --smt=1 / ppc64_cpu --smt=8
Units: seconds : lesser is better
---------------------------------ppc64_cpu --smt=1
kernel N Min Max Median Avg Stddev powerpc/next 100 13.39 17.55 14.71 14.7658 0.69184745 +patchset 100 13.3 16.27 14.33 14.4179 0.5427433
ppc64_cpu --smt=8
kernel N Min Max Median Avg Stddev powerpc/next 100 21.65 26.17 23.71 23.7111 0.8589786 +patchset 100 21.88 25.79 23.16 23.2945 0.86394839
Observations:
Performance of ebizzy/ perf_sched_bench / schbench remain the same with and without the patchset.
Changelog v1->v2:
v1 link: https://lore.kernel.org/linuxppc-dev/[email protected]/t/#u
Added five more patches on top of Seven.
Rebased to 19th Sept 2020 powerpc/next (based on v5.9-rc2)
Srikar Dronamraju (11):
powerpc/topology: Update topology_core_cpumask powerpc/smp: Stop updating cpu_core_mask powerpc/smp: Remove get_physical_package_id powerpc/smp: Optimize remove_cpu_from_masks powerpc/smp: Limit CPUs traversed to within a node. powerpc/smp: Stop passing mask to update_mask_by_l2 powerpc/smp: Depend on cpu_l1_cache_map when adding CPUs powerpc/smp: Check for duplicate topologies and consolidate powerpc/smp: Optimize update_mask_by_l2 powerpc/smp: Move coregroup mask updation to a new function powerpc/smp: Optimize update_coregroup_mask
arch/powerpc/include/asm/smp.h | 5 - arch/powerpc/include/asm/topology.h | 7 +- arch/powerpc/kernel/smp.c | 186 ++++++++++++++++++---------- 3 files changed, 120 insertions(+), 78 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/11 powerpc/topology: Update topology_core_cpumask
On Power, cpu_core_mask and cpu_cpu_mask refer to the same set of CPUs. cpu_cpu_mask is needed by scheduler, hence look at deprecating cpu_core_mask. Before deleting the cpu_core_mask, ensure its only user is moved to cpu_cpu_mask.
--- arch/powerpc/include/asm/topology.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/include/asm/topology.h +1 -1
@@ -122,7 +122,7 @@ int get_physical_package_id(int cpu);#endif #define topology_sibling_cpumask (cpu) (per_cpu(cpu_sibling_map, cpu)) -#define topology_core_cpumask (cpu) (per_cpu(cpu_core_map, cpu)) +#define topology_core_cpumask (cpu) (cpu_cpu_mask(cpu))#define topology_core_id (cpu) (cpu_to_core_id(cpu))#endif
Signature · 1 line
2.17.1
PATCH v2 2/11 powerpc/smp: Stop updating cpu_core_mask
In reply to Srikar DronamrajuAnton Blanchard reported that his 4096 vcpu KVM guest took around 30 minutes to boot. He also analyzed it to the time taken to iterate while setting the cpu_core_mask.
Further analysis shows that cpu_core_mask and cpu_cpu_mask for any CPU would be equal on Power. However updating cpu_core_mask took forever to update as its a per cpu cpumask variable. Instead cpu_cpu_mask was a per NODE /per DIE cpumask that was shared by all the respective CPUs.
Also cpu_cpu_mask is needed from a scheduler perspective. However cpu_core_map is an exported symbol. Hence stop updating cpu_core_map and make it point to cpu_cpu_mask.
--- arch/powerpc/include/asm/smp.h | 5 ----- arch/powerpc/kernel/smp.c | 33 +++++++-------------------------- 2 files changed, 7 insertions(+), 31 deletions(-)
arch/powerpc/include/asm/smp.h +0 -5
@@ -119,11 +119,6 @@ static inline struct cpumask *cpu_sibling_mask(int cpu)return per_cpu (cpu_sibling_map, cpu); } -static inline struct cpumask *cpu_core_mask (int cpu) -{ -return per_cpu (cpu_core_map, cpu); -} -static inline struct cpumask *cpu_l2_cache_mask (int cpu) {return per_cpu (cpu_l2_cache_map, cpu);
arch/powerpc/kernel/smp.c +7 -26
@@ -953,12 +953,17 @@ void __init smp_prepare_cpus(unsigned int max_cpus) local_memory_node(numa_cpu_lookup_table[cpu])); }#endif +/* +* cpu_core_map is now more updated and exists only since +* its been exported for long. It only will have a snapshot +* of cpu_cpu_mask. +*/ +cpumask_copy (per_cpu (cpu_core_map, cpu),cpu_cpu_mask (cpu)); }/* Init the cpumasks so the boot CPU is related to itself */ cpumask_set_cpu (boot_cpuid,cpu_sibling_mask (boot_cpuid));cpumask_set_cpu (boot_cpuid,cpu_l2_cache_mask (boot_cpuid)); -cpumask_set_cpu (boot_cpuid,cpu_core_mask (boot_cpuid));if (has_coregroup_support ())cpumask_set_cpu (boot_cpuid,cpu_coregroup_mask (boot_cpuid));
@@ -1260,9 +1265,7 @@ static void remove_cpu_from_masks(int cpu) {int i; -/* NB: cpu_core_mask is a superset of the others */ -for_each_cpu (i,cpu_core_mask (cpu)) { -set_cpus_unrelated (cpu, i, cpu_core_mask); +for_each_cpu (i,cpu_cpu_mask (cpu)) {set_cpus_unrelated (cpu, i, cpu_l2_cache_mask);set_cpus_unrelated (cpu, i, cpu_sibling_mask);if (has_big_cores)
@@ -1312,7 +1315,6 @@ EXPORT_SYMBOL_GPL(get_physical_package_id);static void add_cpu_to_masks (int cpu) {int first_thread =cpu_first_thread_sibling (cpu); -int pkg_id =get_physical_package_id (cpu);int i;/*
@@ -1320,7 +1322,6 @@ static void add_cpu_to_masks(int cpu)* add it to it's own thread sibling mask. */ cpumask_set_cpu (cpu,cpu_sibling_mask (cpu)); -cpumask_set_cpu (cpu,cpu_core_mask (cpu));for (i = first_thread; i < first_thread + threads_per_core; i++)if (cpu_online (i))
@@ -1342,26 +1343,6 @@ static void add_cpu_to_masks(int cpu)set_cpus_related (cpu, i, cpu_coregroup_mask); } } - -if (pkg_id ==-1 ) { -struct cpumask *(*mask)(int ) = cpu_sibling_mask; - -/* -* Copy the sibling mask into core sibling mask and -* mark any CPUs on the same chip as this CPU. -*/ -if (shared_caches) - mask = cpu_l2_cache_mask; - -for_each_cpu (i,mask (cpu)) -set_cpus_related (cpu, i, cpu_core_mask); - -return ; - } - -for_each_cpu (i, cpu_online_mask) -if (get_physical_package_id (i) == pkg_id) -set_cpus_related (cpu, i, cpu_core_mask); }/* Activate a secondary processor. */
Signature · 1 line
2.17.1
PATCH v2 3/11 powerpc/smp: Remove get_physical_package_id
In reply to Srikar DronamrajuNow that cpu_core_mask has been removed and topology_core_cpumask has been updated to use cpu_cpu_mask, we no more need get_physical_package_id.
--- arch/powerpc/include/asm/topology.h | 5 ----- arch/powerpc/kernel/smp.c | 20 -------------------- 2 files changed, 25 deletions(-)
arch/powerpc/include/asm/topology.h +0 -5
@@ -114,12 +114,7 @@ static inline int cpu_to_coregroup_id(int cpu)#ifdef CONFIG_PPC64 #include <asm/smp.h> -#ifdef CONFIG_PPC_SPLPAR -int get_physical_package_id (int cpu); -#define topology_physical_package_id (cpu) (get_physical_package_id(cpu)) -#else #define topology_physical_package_id (cpu) (cpu_to_chip_id(cpu)) -#endif #define topology_sibling_cpumask (cpu) (per_cpu(cpu_sibling_map, cpu))#define topology_core_cpumask (cpu) (cpu_cpu_mask(cpu))
arch/powerpc/kernel/smp.c +0 -20
@@ -1292,26 +1292,6 @@ static inline void add_cpu_to_smallcore_masks(int cpu) } } -int get_physical_package_id (int cpu) -{ -int pkg_id =cpu_to_chip_id (cpu); - -/* -* If the platform is PowerNV or Guest on KVM, ibm,chip-id is -* defined. Hence we would return the chip-id as the result of -* get_physical_package_id. -*/ -if (pkg_id ==-1 &&firmware_has_feature (FW_FEATURE_LPAR ) && -IS_ENABLED (CONFIG_PPC_SPLPAR )) { -struct device_node *np =of_get_cpu_node (cpu,NULL ); - pkg_id =of_node_to_nid (np); -of_node_put (np); - } - -return pkg_id; -} -EXPORT_SYMBOL_GPL (get_physical_package_id); -static void add_cpu_to_masks (int cpu) {int first_thread =cpu_first_thread_sibling (cpu);
Signature · 1 line
2.17.1
PATCH v2 4/11 powerpc/smp: Optimize remove_cpu_from_masks
In reply to Srikar DronamrajuWhile offlining a CPU, system currently iterate through all the CPUs in the DIE to clear sibling, l2_cache and smallcore maps. However if there are more cores in a DIE, system can end up spending more time iterating through CPUs which are completely unrelated.
Optimize this by only iterating through smaller but relevant cpumap. If shared_cache is set, cpu_l2_cache_map should be relevant else cpu_sibling_map would be relevant.
--- arch/powerpc/kernel/smp.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/powerpc/kernel/smp.c +9 -2
@@ -1263,14 +1263,21 @@ static bool update_mask_by_l2(int cpu, struct cpumask *(*mask_fn)(int))#ifdef CONFIG_HOTPLUG_CPU static void remove_cpu_from_masks (int cpu) { +struct cpumask *(*mask_fn)(int ) = cpu_sibling_mask;int i; -for_each_cpu (i,cpu_cpu_mask (cpu)) { +if (shared_caches) + mask_fn = cpu_l2_cache_mask; + +for_each_cpu (i,mask_fn (cpu)) {set_cpus_unrelated (cpu, i, cpu_l2_cache_mask);set_cpus_unrelated (cpu, i, cpu_sibling_mask);if (has_big_cores)set_cpus_unrelated (cpu, i, cpu_smallcore_mask); -if (has_coregroup_support ()) + } + +if (has_coregroup_support ()) { +for_each_cpu (i,cpu_coregroup_mask (cpu))set_cpus_unrelated (cpu, i, cpu_coregroup_mask); } }
Signature · 1 line
2.17.1
PATCH v2 5/11 powerpc/smp: Limit CPUs traversed to within a node.
In reply to Srikar DronamrajuAll the arch specific topology cpumasks are within a node/DIE. However when setting these per CPU cpumasks, system traverses through all the online CPUs. This is redundant.
Reduce the traversal to only CPUs that are online in the node to which the CPU belongs to.
--- arch/powerpc/kernel/smp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/kernel/smp.c +1 -1
@@ -1241,7 +1241,7 @@ static bool update_mask_by_l2(int cpu, struct cpumask *(*mask_fn)(int)) }cpumask_set_cpu (cpu,mask_fn (cpu)); -for_each_cpu (i, cpu_online_mask) { +for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) {/* * when updating the marks the current CPU has not been marked * online, but we need to update the cache masks
Signature · 1 line
2.17.1
PATCH v2 6/11 powerpc/smp: Stop passing mask to update_mask_by_l2
In reply to Srikar Dronamrajuupdate_mask_by_l2 is called only once. But it passes cpu_l2_cache_mask as parameter. Instead of passing cpu_l2_cache_mask, use it directly in update_mask_by_l2.
--- arch/powerpc/kernel/smp.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/powerpc/kernel/smp.c +4 -4
@@ -1218,7 +1218,7 @@ static struct device_node *cpu_to_l2cache(int cpu)return cache; } -static bool update_mask_by_l2 (int cpu,struct cpumask *(*mask_fn)(int )) +static bool update_mask_by_l2 (int cpu) {struct device_node *l2_cache, *np;int i;
@@ -1240,7 +1240,7 @@ static bool update_mask_by_l2(int cpu, struct cpumask *(*mask_fn)(int))return false; } -cpumask_set_cpu (cpu,mask_fn (cpu)); +cpumask_set_cpu (cpu,cpu_l2_cache_mask (cpu));for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) {/* * when updating the marks the current CPU has not been marked
@@ -1251,7 +1251,7 @@ static bool update_mask_by_l2(int cpu, struct cpumask *(*mask_fn)(int))continue ;if (np == l2_cache) -set_cpus_related (cpu, i, mask_fn); +set_cpus_related (cpu, i, cpu_l2_cache_mask);of_node_put (np); }
@@ -1315,7 +1315,7 @@ static void add_cpu_to_masks(int cpu)set_cpus_related (i, cpu, cpu_sibling_mask);add_cpu_to_smallcore_masks (cpu); -update_mask_by_l2 (cpu, cpu_l2_cache_mask); +update_mask_by_l2 (cpu);if (has_coregroup_support ()) {int coregroup_id =cpu_to_coregroup_id (cpu);
Signature · 1 line
2.17.1
PATCH v2 7/11 powerpc/smp: Depend on cpu_l1_cache_map when adding CPUs
In reply to Srikar DronamrajuCurrently on hotplug/hotunplug, CPU iterates through all the CPUs in its core to find threads in its thread group. However this info is already captured in cpu_l1_cache_map. Hence reduce iterations and cleanup add_cpu_to_smallcore_masks function.
--- arch/powerpc/kernel/smp.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/powerpc/kernel/smp.c +3 -4
@@ -1285,16 +1285,15 @@ static void remove_cpu_from_masks(int cpu)static inline void add_cpu_to_smallcore_masks (int cpu) { -struct cpumask *this_l1_cache_map =per_cpu (cpu_l1_cache_map, cpu); -int i, first_thread =cpu_first_thread_sibling (cpu); +int i;if (!has_big_cores)return ;cpumask_set_cpu (cpu,cpu_smallcore_mask (cpu)); -for (i = first_thread; i < first_thread + threads_per_core; i++) { -if (cpu_online (i) &&cpumask_test_cpu (i, this_l1_cache_map)) +for_each_cpu (i,per_cpu (cpu_l1_cache_map, cpu)) { +if (cpu_online (i))set_cpus_related (i, cpu, cpu_smallcore_mask); } }
Signature · 1 line
2.17.1
PATCH v2 8/11 powerpc/smp: Check for duplicate topologies and consolidate
In reply to Srikar DronamrajuCACHE and COREGROUP domains are now part of default topology. However on systems that don't support CACHE or COREGROUP, these domains will eventually be degenerated. The degeneration happens per CPU. Do note the current fixup_topology() logic ensures that mask of a domain that is not supported on the current platform is set to the previous domain.
Instead of waiting for the scheduler to degenerated try to consolidate based on their masks and sd_flags. This is done just before setting the scheduler topology.
--- arch/powerpc/kernel/smp.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
arch/powerpc/kernel/smp.c +26 -0
@@ -1401,6 +1401,8 @@ int setup_profiling_timer(unsigned int multiplier)static void fixup_topology (void ) { +int i; +#ifdef CONFIG_SCHED_SMT if (has_big_cores) {pr_info ("Big cores detected but using small core scheduling\n" );
@@ -1410,6 +1412,30 @@ static void fixup_topology(void)if (!has_coregroup_support ()) powerpc_topology[mc_idx].mask = powerpc_topology[cache_idx].mask; + +/* +* Try to consolidate topology levels here instead of +* allowing scheduler to degenerate. +* - Dont consolidate if masks are different. +* - Dont consolidate if sd_flags exists and are different. +*/ +for (i =1 ; i <= die_idx; i++) { +if (powerpc_topology[i].mask != powerpc_topology[i -1 ].mask) +continue ; + +if (powerpc_topology[i].sd_flags && powerpc_topology[i -1 ].sd_flags && + powerpc_topology[i].sd_flags != powerpc_topology[i -1 ].sd_flags) +continue ; + +if (!powerpc_topology[i -1 ].sd_flags) + powerpc_topology[i -1 ].sd_flags = powerpc_topology[i].sd_flags; + + powerpc_topology[i].mask = powerpc_topology[i +1 ].mask; + powerpc_topology[i].sd_flags = powerpc_topology[i +1 ].sd_flags; +#ifdef CONFIG_SCHED_DEBUG + powerpc_topology[i].name = powerpc_topology[i +1 ].name; +#endif + } }void __init smp_cpus_done (unsigned int max_cpus)
Signature · 1 line
2.17.1
PATCH v2 9/11 powerpc/smp: Optimize update_mask_by_l2
In reply to Srikar DronamrajuAll threads of a SMT4 core can either be part of this CPU's l2-cache mask or not related to this CPU l2-cache mask. Use this relation to reduce the number of iterations needed to find all the CPUs that share the same l2-cache.
Use a temporary mask to iterate through the CPUs that may share l2_cache mask. Also instead of setting one CPU at a time into cpu_l2_cache_mask, copy the SMT4/sub mask at one shot.
--- arch/powerpc/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++----- 1 file changed, 45 insertions(+), 6 deletions(-)
arch/powerpc/kernel/smp.c +45 -6
@@ -670,6 +670,28 @@ static void set_cpus_unrelated(int i, int j, }#endif +/* +* Extends set_cpus_related. Instead of setting one CPU at a time in +* dstmask, set srcmask at oneshot. dstmask should be super set of srcmask. +*/ +static void or_cpumasks_related (int i,int j,struct cpumask *(*srcmask)(int ), +struct cpumask *(*dstmask)(int )) +{ +struct cpumask *mask; +int k; + + mask =srcmask (j); +for_each_cpu (k,srcmask (i)) +cpumask_or (dstmask (k),dstmask (k), mask); + +if (i == j) +return ; + + mask =srcmask (i); +for_each_cpu (k,srcmask (j)) +cpumask_or (dstmask (k),dstmask (k), mask); +} +/* * parse_thread_groups: Parses the "ibm,thread-groups" device tree * property for the CPU device node @dn and stores
@@ -1220,7 +1242,9 @@ static struct device_node *cpu_to_l2cache(int cpu)static bool update_mask_by_l2 (int cpu) { +struct cpumask *(*submask_fn)(int ) = cpu_sibling_mask;struct device_node *l2_cache, *np; +cpumask_var_t mask;int i; l2_cache =cpu_to_l2cache (cpu);
@@ -1240,22 +1264,37 @@ static bool update_mask_by_l2(int cpu)return false; } -cpumask_set_cpu (cpu,cpu_l2_cache_mask (cpu)); -for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) { +alloc_cpumask_var_node (&mask,GFP_KERNEL ,cpu_to_node (cpu)); +cpumask_and (mask, cpu_online_mask,cpu_cpu_mask (cpu)); + +if (has_big_cores) + submask_fn = cpu_smallcore_mask; + +/* Update l2-cache mask with all the CPUs that are part of submask */ +or_cpumasks_related (cpu, cpu, submask_fn, cpu_l2_cache_mask); + +/* Skip all CPUs already part of current CPU l2-cache mask */ +cpumask_andnot (mask, mask,cpu_l2_cache_mask (cpu)); + +for_each_cpu (i, mask) {/* * when updating the marks the current CPU has not been marked * online, but we need to update the cache masks */ np =cpu_to_l2cache (i); -if (!np) -continue ; -if (np == l2_cache) -set_cpus_related (cpu, i, cpu_l2_cache_mask); +/* Skip all CPUs already part of current CPU l2-cache */ +if (np == l2_cache) { +or_cpumasks_related (cpu, i, submask_fn, cpu_l2_cache_mask); +cpumask_andnot (mask, mask,submask_fn (i)); + }else { +cpumask_andnot (mask, mask,cpu_l2_cache_mask (i)); + }of_node_put (np); }of_node_put (l2_cache); +free_cpumask_var (mask);return true; }
Signature · 1 line
2.17.1
Re: [PATCH v2 09/11] powerpc/smp: Optimize update_mask_by_l2
Quoted 9 lines · Srikar Dronamraju
All threads of a SMT4 core can either be part of this CPU's l2-cache mask or not related to this CPU l2-cache mask. Use this relation to reduce the number of iterations needed to find all the CPUs that share the same l2-cache.
Use a temporary mask to iterate through the CPUs that may share
l2_cachemask. Also instead of setting one CPU at a time intocpu_l2_cache_mask, copy the SMT4/sub mask at one shot.
...
Quoted 9 lines
static bool
update_mask_by_l2(int cpu)
{arch/powerpc/kernel/smp.c +2
-0+struct cpumask *(*submask_fn)(int ) = cpu_sibling_mask;struct device_node *l2_cache, *np; +cpumask_var_t mask;int i; l2_cache =cpu_to_l2cache (cpu);
arch/powerpc/kernel/smp.c static bool update_mask_by_l2(int cpu) · line 1267
Quoted 6 lines
arch/powerpc/kernel/smp.c +1
-2@@ -1240,22 +1264,37 @@ static bool update_mask_by_l2(int cpu)return false; } -cpumask_set_cpu (cpu,cpu_l2_cache_mask (cpu)); -for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) { +alloc_cpumask_var_node (&mask,GFP_KERNEL ,cpu_to_node (cpu));
Shouldn't this be GFP_ATOMIC? Otherwise, during the CPU hotplugging, we have,
(irqs were disabled in do_idle())
[ 335.420001][ T0] BUG: sleeping function called from invalid context at mm/slab.h:494 [ 335.420003][ T0] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/88 [ 335.420005][ T0] no locks held by swapper/88/0. [ 335.420007][ T0] irq event stamp: 18074448 [ 335.420015][ T0] hardirqs last enabled at (18074447): [<c0000000001a2a7c>] tick_nohz_idle_enter+0x9c/0x110 [ 335.420019][ T0] hardirqs last disabled at (18074448): [<c000000000106798>] do_idle+0x138/0x3b0 do_idle at kernel/sched/idle.c:253 (discriminator 1) [ 335.420023][ T0] softirqs last enabled at (18074440): [<c0000000000bbec4>] irq_enter_rcu+0x94/0xa0 [ 335.420026][ T0] softirqs last disabled at (18074439): [<c0000000000bbea0>] irq_enter_rcu+0x70/0xa0 [ 335.420030][ T0] CPU: 88 PID: 0 Comm: swapper/88 Tainted: G W 5.9.0-rc8-next-20201007 #1 [ 335.420032][ T0] Call Trace: [ 335.420037][ T0] [c00020000a4bfcf0] [c000000000649e98] dump_stack+0xec/0x144 (unreliable) [ 335.420043][ T0] [c00020000a4bfd30] [c0000000000f6c34] ___might_sleep+0x2f4/0x310 [ 335.420048][ T0] [c00020000a4bfdb0] [c000000000354f94] slab_pre_alloc_hook.constprop.82+0x124/0x190 [ 335.420051][ T0] [c00020000a4bfe00] [c00000000035e9e8] __kmalloc_node+0x88/0x3a0 slab_alloc_node at mm/slub.c:2817 (inlined by) __kmalloc_node at mm/slub.c:4013 [ 335.420054][ T0] [c00020000a4bfe80] [c0000000006494d8] alloc_cpumask_var_node+0x38/0x80 kmalloc_node at include/linux/slab.h:577 (inlined by) alloc_cpumask_var_node at lib/cpumask.c:116 [ 335.420060][ T0] [c00020000a4bfef0] [c00000000003eedc] start_secondary+0x27c/0x800 update_mask_by_l2 at arch/powerpc/kernel/smp.c:1267 (inlined by) add_cpu_to_masks at arch/powerpc/kernel/smp.c:1387 (inlined by) start_secondary at arch/powerpc/kernel/smp.c:1420 [ 335.420063][ T0] [c00020000a4bff90] [c00000000000c468] start_secondary_resume+0x10/0x14
Quoted 38 lines
arch/powerpc/kernel/smp.c +20
-4+cpumask_and (mask, cpu_online_mask,cpu_cpu_mask (cpu)); + +if (has_big_cores) + submask_fn = cpu_smallcore_mask; + +/* Update l2-cache mask with all the CPUs that are part of submask */ +or_cpumasks_related (cpu, cpu, submask_fn, cpu_l2_cache_mask); + +/* Skip all CPUs already part of current CPU l2-cache mask */ +cpumask_andnot (mask, mask,cpu_l2_cache_mask (cpu)); + +for_each_cpu (i, mask) {/* * when updating the marks the current CPU has not been marked * online, but we need to update the cache masks */ np =cpu_to_l2cache (i); -if (!np) -continue ; -if (np == l2_cache) -set_cpus_related (cpu, i, cpu_l2_cache_mask); +/* Skip all CPUs already part of current CPU l2-cache */ +if (np == l2_cache) { +or_cpumasks_related (cpu, i, submask_fn, cpu_l2_cache_mask); +cpumask_andnot (mask, mask,submask_fn (i)); + }else { +cpumask_andnot (mask, mask,cpu_l2_cache_mask (i)); + }of_node_put (np); }of_node_put (l2_cache); +free_cpumask_var (mask);return true;}
Re: [PATCH v2 09/11] powerpc/smp: Optimize update_mask_by_l2
* Qian Cai <[email protected]> [2020-10-07 09:05:42]:
Hi Qian,
Thanks for testing and reporting the failure.
Quoted 23 lines · Srikar Dronamraju
Quoted 9 lines
All threads of a SMT4 core can either be part of this CPU's l2-cache mask or not related to this CPU l2-cache mask. Use this relation to reduce the number of iterations needed to find all the CPUs that share the same l2-cache.
Use a temporary mask to iterate through the CPUs that may share
l2_cachemask. Also instead of setting one CPU at a time intocpu_l2_cache_mask, copy the SMT4/sub mask at one shot....
Quoted 12 lines
static bool
update_mask_by_l2(int cpu)
{arch/powerpc/kernel/smp.c +2
-0+struct cpumask *(*submask_fn)(int ) = cpu_sibling_mask;struct device_node *l2_cache, *np; +cpumask_var_t mask;int i; l2_cache =cpu_to_l2cache (cpu);@@ -1240,22 +1264,37 @@ static bool update_mask_by_l2(int cpu)return false; }
Quoted 5 lines
arch/powerpc/kernel/smp.c static bool update_mask_by_l2(int cpu) · line 1267
Quoted 3 lines
arch/powerpc/kernel/smp.c +1
-2-cpumask_set_cpu (cpu,cpu_l2_cache_mask (cpu)); -for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) { +alloc_cpumask_var_node (&mask,GFP_KERNEL ,cpu_to_node (cpu));Shouldn't this be
GFP_ATOMIC? Otherwise, during the CPU hotplugging, we have,
Can you confirm if CONFIG_CPUMASK_OFFSTACK is enabled in your config? Because if !CONFIG_CPUMASK_OFFSTACK, then alloc_cpumask_var_node would do nothing but return true.
Regarding CONFIG_CPUMASK_OFFSTACK, not sure how much powerpc was tested with that config enabled.
Please refer to
http://lore.kernel.org/lkml/[email protected]/t/#u
And we do have an issue to track the same.
https://github.com/linuxppc/issues/issues/321 for enabling/ testing / verifying if CONFIG_CPUMASK_OFFSTACK works. I also dont see any powerpc kconfig enabling this.
I do agree with your suggestion that we could substitute GFP_ATOMIC/GFP_KERNEL.
Quoted 68 lines
(irqs were disabled in
do_idle())[ 335.420001][ T0] BUG: sleeping function called from invalid context at mm/slab.h:494 [ 335.420003][ T0] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/88 [ 335.420005][ T0] no locks held by swapper/88/0. [ 335.420007][ T0] irq event stamp: 18074448 [ 335.420015][ T0] hardirqs last enabled at (18074447): [<c0000000001a2a7c>] tick_nohz_idle_enter+0x9c/0x110 [ 335.420019][ T0] hardirqs last disabled at (18074448): [<c000000000106798>] do_idle+0x138/0x3b0 do_idle at kernel/sched/idle.c:253 (discriminator 1) [ 335.420023][ T0] softirqs last enabled at (18074440): [<c0000000000bbec4>] irq_enter_rcu+0x94/0xa0 [ 335.420026][ T0] softirqs last disabled at (18074439): [<c0000000000bbea0>] irq_enter_rcu+0x70/0xa0 [ 335.420030][ T0] CPU: 88 PID: 0 Comm: swapper/88 Tainted: G W 5.9.0-rc8-next-20201007 #1 [ 335.420032][ T0] Call Trace: [ 335.420037][ T0] [c00020000a4bfcf0] [c000000000649e98] dump_stack+0xec/0x144 (unreliable) [ 335.420043][ T0] [c00020000a4bfd30] [c0000000000f6c34] ___might_sleep+0x2f4/0x310 [ 335.420048][ T0] [c00020000a4bfdb0] [c000000000354f94] slab_pre_alloc_hook.constprop.82+0x124/0x190 [ 335.420051][ T0] [c00020000a4bfe00] [c00000000035e9e8] __kmalloc_node+0x88/0x3a0 slab_alloc_node at mm/slub.c:2817 (inlined by) __kmalloc_node at mm/slub.c:4013 [ 335.420054][ T0] [c00020000a4bfe80] [c0000000006494d8] alloc_cpumask_var_node+0x38/0x80 kmalloc_node at include/linux/slab.h:577 (inlined by) alloc_cpumask_var_node at lib/cpumask.c:116 [ 335.420060][ T0] [c00020000a4bfef0] [c00000000003eedc] start_secondary+0x27c/0x800 update_mask_by_l2 at arch/powerpc/kernel/smp.c:1267 (inlined by) add_cpu_to_masks at arch/powerpc/kernel/smp.c:1387 (inlined by) start_secondary at arch/powerpc/kernel/smp.c:1420 [ 335.420063][ T0] [c00020000a4bff90] [c00000000000c468] start_secondary_resume+0x10/0x14Quoted 38 lines
arch/powerpc/kernel/smp.c +20
-4+cpumask_and (mask, cpu_online_mask,cpu_cpu_mask (cpu)); + +if (has_big_cores) + submask_fn = cpu_smallcore_mask; + +/* Update l2-cache mask with all the CPUs that are part of submask */ +or_cpumasks_related (cpu, cpu, submask_fn, cpu_l2_cache_mask); + +/* Skip all CPUs already part of current CPU l2-cache mask */ +cpumask_andnot (mask, mask,cpu_l2_cache_mask (cpu)); + +for_each_cpu (i, mask) {/* * when updating the marks the current CPU has not been marked * online, but we need to update the cache masks */ np =cpu_to_l2cache (i); -if (!np) -continue ; -if (np == l2_cache) -set_cpus_related (cpu, i, cpu_l2_cache_mask); +/* Skip all CPUs already part of current CPU l2-cache */ +if (np == l2_cache) { +or_cpumasks_related (cpu, i, submask_fn, cpu_l2_cache_mask); +cpumask_andnot (mask, mask,submask_fn (i)); + }else { +cpumask_andnot (mask, mask,cpu_l2_cache_mask (i)); + }of_node_put (np); }of_node_put (l2_cache); +free_cpumask_var (mask);return true;}
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Re: [PATCH v2 09/11] powerpc/smp: Optimize update_mask_by_l2
Quoted 1 line · Srikar Dronamraju
Can you confirm if
CONFIG_CPUMASK_OFFSTACKis enabled in your config?
Yes, https://gitlab.com/cailca/linux-mm/-/blob/master/powerpc.config
We tested here almost daily on linux-next.
PATCH v2 10/11 powerpc/smp: Move coregroup mask updation to a new function
In reply to Srikar DronamrajuMove the logic for updating the coregroup mask of a CPU to its own function. This will help in reworking the updation of coregroup mask in subsequent patch.
--- arch/powerpc/kernel/smp.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-)
arch/powerpc/kernel/smp.c +19 -13
@@ -1337,6 +1337,23 @@ static inline void add_cpu_to_smallcore_masks(int cpu) } } +static void update_coregroup_mask (int cpu) +{ +int first_thread =cpu_first_thread_sibling (cpu); +int coregroup_id =cpu_to_coregroup_id (cpu); +int i; + +cpumask_set_cpu (cpu,cpu_coregroup_mask (cpu)); +for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) { +int fcpu =cpu_first_thread_sibling (i); + +if (fcpu == first_thread) +set_cpus_related (cpu, i, cpu_coregroup_mask); +else if (coregroup_id ==cpu_to_coregroup_id (i)) +set_cpus_related (cpu, i, cpu_coregroup_mask); + } +} +static void add_cpu_to_masks (int cpu) {int first_thread =cpu_first_thread_sibling (cpu);
@@ -1355,19 +1372,8 @@ static void add_cpu_to_masks(int cpu)add_cpu_to_smallcore_masks (cpu);update_mask_by_l2 (cpu); -if (has_coregroup_support ()) { -int coregroup_id =cpu_to_coregroup_id (cpu); - -cpumask_set_cpu (cpu,cpu_coregroup_mask (cpu)); -for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) { -int fcpu =cpu_first_thread_sibling (i); - -if (fcpu == first_thread) -set_cpus_related (cpu, i, cpu_coregroup_mask); -else if (coregroup_id ==cpu_to_coregroup_id (i)) -set_cpus_related (cpu, i, cpu_coregroup_mask); - } - } +if (has_coregroup_support ()) +update_coregroup_mask (cpu); }/* Activate a secondary processor. */
Signature · 1 line
2.17.1
PATCH v2 11/11 powerpc/smp: Optimize update_coregroup_mask
In reply to Srikar DronamrajuAll threads of a SMT4/SMT8 core can either be part of CPU's coregroup mask or outside the coregroup. Use this relation to reduce the number of iterations needed to find all the CPUs that share the same coregroup
Use a temporary mask to iterate through the CPUs that may share coregroup mask. Also instead of setting one CPU at a time into cpu_coregroup_mask, copy the SMT4/SMT8/submask at one shot.
--- arch/powerpc/kernel/smp.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
arch/powerpc/kernel/smp.c +22 -8
@@ -1339,19 +1339,33 @@ static inline void add_cpu_to_smallcore_masks(int cpu)static void update_coregroup_mask (int cpu) { -int first_thread =cpu_first_thread_sibling (cpu); +struct cpumask *(*submask_fn)(int ) = cpu_sibling_mask; +cpumask_var_t mask;int coregroup_id =cpu_to_coregroup_id (cpu);int i; -cpumask_set_cpu (cpu,cpu_coregroup_mask (cpu)); -for_each_cpu_and (i, cpu_online_mask,cpu_cpu_mask (cpu)) { -int fcpu =cpu_first_thread_sibling (i); +alloc_cpumask_var_node (&mask,GFP_KERNEL ,cpu_to_node (cpu)); +cpumask_and (mask, cpu_online_mask,cpu_cpu_mask (cpu)); + +if (shared_caches) + submask_fn = cpu_l2_cache_mask; + +/* Update coregroup mask with all the CPUs that are part of submask */ +or_cpumasks_related (cpu, cpu, submask_fn, cpu_coregroup_mask); + +/* Skip all CPUs already part of coregroup mask */ +cpumask_andnot (mask, mask,cpu_coregroup_mask (cpu)); -if (fcpu == first_thread) -set_cpus_related (cpu, i, cpu_coregroup_mask); -else if (coregroup_id ==cpu_to_coregroup_id (i)) -set_cpus_related (cpu, i, cpu_coregroup_mask); +for_each_cpu (i, mask) { +/* Skip all CPUs not part of this coregroup */ +if (coregroup_id ==cpu_to_coregroup_id (i)) { +or_cpumasks_related (cpu, i, submask_fn, cpu_coregroup_mask); +cpumask_andnot (mask, mask,submask_fn (i)); + }else { +cpumask_andnot (mask, mask,cpu_coregroup_mask (i)); + } } +free_cpumask_var (mask); }static void add_cpu_to_masks (int cpu)
Signature · 1 line
2.17.1
Michael Ellerman
AppliedQuoted 11 lines · Srikar Dronamraju
Here are some optimizations and fixes to make CPU online/offline faster and hence result in faster bootup.
Its based on top of my v5 coregroup support patchset.
https://lore.kernel.org/linuxppc-dev/[email protected]/t/#uAnton reported that his 4096 cpu (1024 cores in a socket) was taking too long to boot. He also analyzed that most of the time was being spent on updating
cpu_core_mask.[...]
Applied to powerpc/next.
[01/11] powerpc/topology: Update topology_core_cpumask
https://git.kernel.org/powerpc/c/4bce545903fa0290e011cf118997717f0c4f4d20 [02/11] powerpc/smp: Stop updating cpu_core_mask https://git.kernel.org/powerpc/c/4ca234a9cbd7c3a656b34dd98c8b156f70ed7849 [03/11] powerpc/smp: Remove get_physical_package_id https://git.kernel.org/powerpc/c/e29e9ed665eeb6f98cd88672994ecf4aaefdb943 [04/11] powerpc/smp: Optimize remove_cpu_from_masks https://git.kernel.org/powerpc/c/70edd4a7c753ba18e3e4bb9e97b6d85156cea738 [05/11] powerpc/smp: Limit CPUs traversed to within a node. https://git.kernel.org/powerpc/c/53516d4abacfab1faaa075c1f79957abc3da358c [06/11] powerpc/smp: Stop passing mask to update_mask_by_l2 https://git.kernel.org/powerpc/c/1f3a4181042107e32e44047e9dde990aced845b5 [07/11] powerpc/smp: Depend on cpu_l1_cache_map when adding CPUs https://git.kernel.org/powerpc/c/661e3d42f99193b7fdd71467a87e48f6e597c285 [08/11] powerpc/smp: Check for duplicate topologies and consolidate https://git.kernel.org/powerpc/c/375370a10d061d5c75c6bc5b09c5db4cc0b0fcfe [09/11] powerpc/smp: Optimize update_mask_by_l2 https://git.kernel.org/powerpc/c/3ab33d6dc3e98e83b55732049e1d1d488207bb6d [10/11] powerpc/smp: Move coregroup mask updation to a new function https://git.kernel.org/powerpc/c/b8a97cb4599cda28bd3b3bc13042f5803b42ad65 [11/11] powerpc/smp: Optimize update_coregroup_mask https://git.kernel.org/powerpc/c/70a94089d7f7fa91bc1795622426b3ed017ec71a
cheers