cpuhp: Expedite synchronize_rcu during CPU hotplug operations
Subsystems (2)
PATCH 1/1 cpuhp: Expedite synchronize_rcu during CPU hotplug operations
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().
Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.
Performance data on a PPC64 system with 400 CPUs:
+ ppc64_cpu --smt=1 (SMT8 to SMT1)
Before: real 1m14.792s After: real 0m03.205s # ~23x improvement
+ ppc64_cpu --smt=8 (SMT1 to SMT8)
Before: real 2m27.695s After: real 0m02.510s # ~58x improvement
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
--- include/linux/rcupdate.h | 3 +++ kernel/cpu.c | 2 ++ 2 files changed, 5 insertions(+)
include/linux/rcupdate.h +3 -0
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)extern int rcu_expedited;extern int rcu_normal; +extern void rcu_expedite_gp (void ); +extern void rcu_unexpedite_gp (void ); + DEFINE_LOCK_GUARD_0(rcu, do { rcu_read_lock();
kernel/cpu.c +2 -0
@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);void cpus_write_lock (void ) { +rcu_expedite_gp ();percpu_down_write (&cpu_hotplug_lock); }void cpus_write_unlock (void ) {percpu_up_write (&cpu_hotplug_lock); +rcu_unexpedite_gp (); }void lockdep_assert_cpus_held (void )
Signature · 1 line
2.52.0
Uladzislau Rezki
maintainerQuoted 17 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement
Quoted 8 lines
+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementAbove numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.
--
Uladzislau Rezki
Hi Uladzislau,
Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Vishal Chourasia
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvement
Quoted 7 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.
# echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp
After setting,
# time ppc64_cpu --smt=1; real 1m10.726s # Run 1 real 1m12.530s # Run 2
# time ppc64_cpu --smt=8 real 0m36.661s # Run 1 real 0m41.401s # Run 2
Regards,
vishalc
Uladzislau Rezki
maintainerHello, Vishalc!
Quoted 26 lines
Hi Uladzislau,
Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Vishal Chourasia
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 7 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.# echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gpAfter setting,
Quoted 8 lines
# time ppc64_cpu --smt=1; real 1m10.726s # Run 1 real 1m12.530s # Run 2# time ppc64_cpu --smt=8 real 0m36.661s # Run 1 real 0m41.401s # Run 2
Thanks.
"ppc64_cpu --smt=1" is the same, i assume it is offlining. "ppc64_cpu --smt=8", whereas, onlining, sees the differences(~5x).
But your real "0m02.510s" is hard to beat event by activating the "rcu_normal_wake_from_gp" option.
--
Uladzislau Rezki
Hi Vishal. Thanks for the patch.
Quoted 22 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvement
Quoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.
Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.
- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.
Uladzislau Rezki
maintainerHello, Shrikanth!
Quoted 41 lines · Uladzislau Rezki
Quoted 22 lines · Vishal Chourasia
Quoted 21 lines
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.
Sounds good to me. I agree it is better to bypass parameters.
--
Uladzislau Rezki
Joel Fernandes
maintainerQuoted 47 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 8 lines
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
thanks,
- Joel
Quoted 3 lines
--
Uladzislau Rezki
Paul E. McKenney
maintainerQuoted 51 lines · Joel Fernandes
Quoted 51 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 45 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Thanx, Paul
Uladzislau Rezki
maintainerQuoted 55 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
I tested this before. I noticed that after 64K of simultaneous synchronize_rcu() calls the scalability is required. Everything less was faster with a new approach.
I can retest. Should i? :)
--
Uladzsislau Rezki
Hello Joel, Paul, Uladzislau,
Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Paul E. McKenney
Quoted 9 lines · Joel Fernandes
Quoted 6 lines · Uladzislau Rezki
Quoted 2 lines
Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
I tested this before. I noticed that after 64K of simultaneous
synchronize_rcu()calls the scalability is required. Everything less was faster with a new approach.
It is worth noting that bulk CPU hotplug represents a different stress pattern than the "simultaneous call" scenario mentioned above.
In a large-scale hotplug event (like a SMT mode switch), we aren't necessarily seeing thousands of simultaneous synchronize_rcu() calls. Instead, because CPU hotplug operations are serialized, we see a "conveyor belt" of sequential calls. One synchronize_rcu() blocks, the hotplug state machine waits, it unblocks, and then the next call is triggered shortly after.
The bottleneck here isn't RCU scalability under concurrent load, but rather the accumulated latency of hundreds of sequential Grace Periods.
For example, on pSeries, onlining 350 out of 400 CPUs triggers exactly 350 calls at three different points in the hotplug state machine. Even though they happen one at a time, the sheer volume makes the total operation time prohibitive.
Following callstack was collected during SMT mode switch where 350 out of 400 CPUs were onlined,
@[
synchronize_rcu+12
cpuidle_pause_and_lock+120
pseries_cpuidle_cpu_online+88
cpuhp_invoke_callback+500
cpuhp_thread_fun+316
smpboot_thread_fn+512
kthread+308
start_kernel_thread+20
]: 350
@[
synchronize_rcu+12
rcu_sync_enter+260
percpu_down_write+76
_cpu_up+140
cpu_up+440
cpu_subsys_online+128
device_online+176
online_store+220
dev_attr_store+52
sysfs_kf_write+120
kernfs_fop_write_iter+456
vfs_write+952
ksys_write+132
system_call_exception+292
system_call_vectored_common+348
]: 350
@[
synchronize_rcu+12
rcu_sync_enter+260
percpu_down_write+76
try_online_node+64
cpu_up+120
cpu_subsys_online+128
device_online+176
online_store+220
dev_attr_store+52
sysfs_kf_write+120
kernfs_fop_write_iter+456
vfs_write+952
ksys_write+132
system_call_exception+292
system_call_vectored_common+348]: 350
Following callstack was collected during SMT mode switch where 350 out of 400 CPUs where offlined,
@[
synchronize_rcu+12
rcu_sync_enter+260
percpu_down_write+76
_cpu_down+188
__cpu_down_maps_locked+44
work_for_cpu_fn+56
process_one_work+508
worker_thread+840
kthread+308
start_kernel_thread+20
]: 1
@[
synchronize_rcu+12
sched_cpu_deactivate+244
cpuhp_invoke_callback+500
cpuhp_thread_fun+316
smpboot_thread_fn+512
kthread+308
start_kernel_thread+20
]: 350
@[
synchronize_rcu+12
cpuidle_pause_and_lock+120
pseries_cpuidle_cpu_dead+88
cpuhp_invoke_callback+500
__cpuhp_invoke_callback_range+200
_cpu_down+412
__cpu_down_maps_locked+44
work_for_cpu_fn+56
process_one_work+508
worker_thread+840
kthread+308
start_kernel_thread+20]: 350
- vishalc
Paul E. McKenney
maintainerQuoted 126 lines · Vishal Chourasia
Hello Joel, Paul, Uladzislau,
Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Paul E. McKenney
Quoted 9 lines · Joel Fernandes
Quoted 6 lines · Uladzislau Rezki
Quoted 2 lines
Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
I tested this before. I noticed that after 64K of simultaneous
synchronize_rcu()calls the scalability is required. Everything less was faster with a new approach.It is worth noting that bulk CPU hotplug represents a different stress pattern than the "simultaneous call" scenario mentioned above.
In a large-scale hotplug event (like a SMT mode switch), we aren't necessarily seeing thousands of simultaneous
synchronize_rcu()calls. Instead, because CPU hotplug operations are serialized, we see a "conveyor belt" of sequential calls. Onesynchronize_rcu()blocks, the hotplug state machine waits, it unblocks, and then the next call is triggered shortly after.The bottleneck here isn't RCU scalability under concurrent load, but rather the accumulated latency of hundreds of sequential Grace Periods.
For example, on pSeries, onlining 350 out of 400 CPUs triggers exactly 350 calls at three different points in the hotplug state machine. Even though they happen one at a time, the sheer volume makes the total operation time prohibitive.
Following callstack was collected during SMT mode switch where 350 out of 400 CPUs were onlined,
@[
synchronize_rcu+12 cpuidle_pause_and_lock+120 pseries_cpuidle_cpu_online+88 cpuhp_invoke_callback+500 cpuhp_thread_fun+316 smpboot_thread_fn+512 kthread+308 start_kernel_thread+20 ]: 350 @[ synchronize_rcu+12 rcu_sync_enter+260 percpu_down_write+76 _cpu_up+140 cpu_up+440 cpu_subsys_online+128 device_online+176 online_store+220 dev_attr_store+52 sysfs_kf_write+120 kernfs_fop_write_iter+456 vfs_write+952 ksys_write+132 system_call_exception+292 system_call_vectored_common+348 ]: 350 @[ synchronize_rcu+12 rcu_sync_enter+260 percpu_down_write+76 try_online_node+64 cpu_up+120 cpu_subsys_online+128 device_online+176 online_store+220 dev_attr_store+52 sysfs_kf_write+120 kernfs_fop_write_iter+456 vfs_write+952 ksys_write+132 system_call_exception+292 system_call_vectored_common+348]: 350
Following callstack was collected during SMT mode switch where 350 out of 400 CPUs where offlined,
@[
synchronize_rcu+12 rcu_sync_enter+260 percpu_down_write+76 _cpu_down+188 __cpu_down_maps_locked+44 work_for_cpu_fn+56 process_one_work+508 worker_thread+840 kthread+308 start_kernel_thread+20 ]: 1 @[ synchronize_rcu+12 sched_cpu_deactivate+244 cpuhp_invoke_callback+500 cpuhp_thread_fun+316 smpboot_thread_fn+512 kthread+308 start_kernel_thread+20 ]: 350 @[ synchronize_rcu+12 cpuidle_pause_and_lock+120 pseries_cpuidle_cpu_dead+88 cpuhp_invoke_callback+500 __cpuhp_invoke_callback_range+200 _cpu_down+412 __cpu_down_maps_locked+44 work_for_cpu_fn+56 process_one_work+508 worker_thread+840 kthread+308 start_kernel_thread+20
Quoted 1 line
]: 350
I still suggest that you test on a big system. There are other sources of synchronize_rcu() calls than just CPU hotplug. ;-)
Thanx, Paul
Joel Fernandes
maintainerQuoted 55 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.
Thoughts?
thanks,
- Joel
Paul E. McKenney
maintainerQuoted 70 lines · Joel Fernandes
Quoted 55 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
Quoted 8 lines
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Thanx, Paul
Joel Fernandes
maintainerQuoted 82 lines · Paul E. McKenney
Quoted 77 lines · Joel Fernandes
Quoted 58 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.
Quoted 4 lines
Quoted 1 line
Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off.
That would address the conveyor belt pattern Vishal expressed.
thanks,
- Joel
Quoted 2 lines
Thanx, Paul
Hi.
Quoted 26 lines · Joel Fernandes
Quoted 21 lines
Quoted 21 lines
Quoted 11 lines
Quoted 8 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.Quoted 4 lines
Quoted 1 line
Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Quoted 8 lines
Makes sense. We could also just have a single list but a much smaller threshold for switching
synchronize_rcu_normaloff.That would address the conveyor belt pattern Vishal expressed.
thanks,
- Joel
Wouldn't that make most of the sync_rcu calls on large system with synchronize_rcu_normal off?
Whats the cost of doing this?
(Me not knowing much about rcu internals)
Joel Fernandes
maintainerQuoted 30 lines · Shrikanth Hegde
Hi.
Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
Quoted 8 lines
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Quoted 7 lines
Quoted 4 lines
Makes sense. We could also just have a single list but a much smaller threshold for switching
synchronize_rcu_normaloff. That would address the conveyor belt pattern Vishal expressed.
thanks,
- JoelWouldn't that make most of the
sync_rcucalls on large system withsynchronize_rcu_normaloff?
It would and that is expected.
Quoted 2 lines
Whats the cost of doing this?
There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.
- Joel
Quoted 2 lines
(Me not knowing much about rcu internals)
Paul E. McKenney
maintainerQuoted 38 lines · Joel Fernandes
Quoted 30 lines · Shrikanth Hegde
Hi.
Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
Quoted 8 lines
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Quoted 7 lines
Quoted 4 lines
Makes sense. We could also just have a single list but a much smaller threshold for switching
synchronize_rcu_normaloff. That would address the conveyor belt pattern Vishal expressed.
thanks,
- JoelWouldn't that make most of the
sync_rcucalls on large system withsynchronize_rcu_normaloff?
Quoted 7 lines
It would and that is expected.
Quoted 2 lines
Whats the cost of doing this?
There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.
Also the contention involved in the list management, if there is still only the one list.
Thanx, Paul
Joel Fernandes
maintainerQuoted 42 lines · Paul E. McKenney
Quoted 41 lines · Joel Fernandes
Quoted 30 lines · Shrikanth Hegde
Hi.
Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
Quoted 8 lines
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Quoted 7 lines
Quoted 4 lines
Makes sense. We could also just have a single list but a much smaller threshold for switching
synchronize_rcu_normaloff. That would address the conveyor belt pattern Vishal expressed.
thanks,
- JoelWouldn't that make most of the
sync_rcucalls on large system withsynchronize_rcu_normaloff?It would and that is expected.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Whats the cost of doing this?
There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.
Also the contention involved in the list management, if there is still only the one list.
Even if the number of synchronize_rcu() in flight is a small number? like < 10. To clarify, I meant keeping the threshold that small in favor of the list contention issue you're raising.
Thanks!
- Joel
Paul E. McKenney
maintainerQuoted 50 lines · Joel Fernandes
Quoted 47 lines · Paul E. McKenney
Quoted 45 lines · Joel Fernandes
Quoted 30 lines · Shrikanth Hegde
Hi.
Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
Quoted 8 lines
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Quoted 7 lines
Quoted 4 lines
Makes sense. We could also just have a single list but a much smaller threshold for switching
synchronize_rcu_normaloff. That would address the conveyor belt pattern Vishal expressed.
thanks,
- JoelWouldn't that make most of the
sync_rcucalls on large system withsynchronize_rcu_normaloff?It would and that is expected.
Quoted 2 lines
Whats the cost of doing this?
There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.
Quoted 6 lines
Quoted 3 lines
Also the contention involved in the list management, if there is still only the one list.
Even if the number of
synchronize_rcu()in flight is a small number? like < 10. To clarify, I meant keeping the threshold that small in favor of the list contention issue you're raising.
Maybe?
One remaining concern is the reads of the counter. Another remaining concern is the possibility of large numbers of CPUs reading the counter, seeing that it is small, then all piling on the list. But it looks like you might get some testing on a large system, so nothing quite like finding out.
If it breaks, you guys get to fix it on whatever schedule is indicated at that time. ;-)
Thanx, Paul
Paul E. McKenney
maintainerQuoted 82 lines · Joel Fernandes
Quoted 82 lines · Paul E. McKenney
Quoted 77 lines · Joel Fernandes
Quoted 58 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)
Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?
I was suggesting on the thread, a more dynamic approach where using
synchronize_rcu_normal()until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default tosynchronize_rcu_normal()without checking for the 16 CPU count.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Thoughts?
Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.
Makes sense. We could also just have a single list but a much smaller threshold for switching
synchronize_rcu_normaloff.That would address the conveyor belt pattern Vishal expressed.
On a system with more than 1,000 CPUs, any single list needs to be handled
extremely carefully to avoid contention of one sort or another. At that
many CPUs, the default rule is instead "never have just one of anything". Unless that "just one" is protected by some contention-avoidance scheme, for example, like the rcu_node tree protects the root rcu_node structure and the rcu_state structure from contention.
Thanx, Paul
Uladzislau Rezki
maintainerQuoted 51 lines · Joel Fernandes
Quoted 51 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 45 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
--
Uladzislau Rezki
Joel Fernandes
maintainerQuoted 61 lines · Uladzislau Rezki
Quoted 56 lines · Joel Fernandes
Quoted 55 lines
Quoted 3 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Quoted 8 lines
Quoted 4 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)
Thanks.
Quoted 3 lines
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 63 lines · Joel Fernandes
Quoted 63 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 8 lines
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless
synchronize_rcuload is high, default to thesr_normalwake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)
Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.
How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.
Note i do not say that it should not be scaled.
--
Uladzislau Rezki
Joel Fernandes
maintainerQuoted 77 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 8 lines
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless
synchronize_rcuload is high, default to thesr_normalwake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)
Quoted 7 lines
Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.
How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.
Note i do not say that it should not be scaled.
But you did not test that on large system with 1000s of CPUs right?
So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).
Let me know if I missed something. Thanks.
Quoted 3 lines
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 80 lines · Joel Fernandes
Quoted 80 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 8 lines
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless
synchronize_rcuload is high, default to thesr_normalwake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.
Quoted 7 lines
Quoted 4 lines
How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.
Note i do not say that it should not be scaled.
But you did not test that on large system with 1000s of CPUs right?
No, no. I do not have access to such systems.
Quoted 5 lines
So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).
You mean introduce threshold and count how many nodes are in queue? To me it sounds not optimal and looks like a temporary solution.
Long term wise, it is better to split it, i mean to scale.
Do you know who can test it on ~1000 CPUs system? So we have some figures.
What i have is 256 CPUs system i can test on.
--
Uladzislau Rezki
Joel Fernandes
maintainerQuoted 88 lines · Uladzislau Rezki
Quoted 80 lines · Joel Fernandes
Quoted 80 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 8 lines
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless
synchronize_rcuload is high, default to thesr_normalwake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.
Quoted 7 lines
Quoted 4 lines
How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.
Note i do not say that it should not be scaled.
But you did not test that on large system with 1000s of CPUs right?
Quoted 8 lines
No, no. I do not have access to such systems.
Quoted 5 lines
So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).
You mean introduce threshold and count how many nodes are in queue?
Yes.
Quoted 1 line
To me it sounds not optimal and looks like a temporary solution.
Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.
Quoted 2 lines
Long term wise, it is better to split it, i mean to scale.
But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.
Quoted 2 lines
Do you know who can test it on ~1000 CPUs system? So we have some figures.
I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.
Quoted 2 lines
What i have is 256 CPUs system i can test on.
Same boat. ;-)
thanks,
- Joel
Hi.
Quoted 46 lines · Joel Fernandes
Quoted 24 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless
synchronize_rcuload is high, default to thesr_normalwake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.
Quoted 7 lines
Quoted 4 lines
How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.
Note i do not say that it should not be scaled.
But you did not test that on large system with 1000s of CPUs right?
Quoted 8 lines
No, no. I do not have access to such systems.
Quoted 5 lines
So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).
You mean introduce threshold and count how many nodes are in queue?
Yes.
Quoted 1 line
To me it sounds not optimal and looks like a temporary solution.
Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.
Quoted 2 lines
Long term wise, it is better to split it, i mean to scale.
But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.
Quoted 1 line
Quoted 6 lines
Quoted 1 line
Do you know who can test it on ~1000 CPUs system? So we have some figures.
I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.
Do you have a patch to try out? We can test it on these systems.
Note: Might take a while to test it, as those systems are bit tricky to get.
Quoted 8 lines
Quoted 2 lines
What i have is 256 CPUs system i can test on.
Same boat. ;-)
thanks,
- Joel
Uladzislau Rezki
maintainerQuoted 56 lines · Shrikanth Hegde
Hi.
Quoted 46 lines · Joel Fernandes
Quoted 24 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless
synchronize_rcuload is high, default to thesr_normalwake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.
Quoted 7 lines
Quoted 4 lines
How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.
Note i do not say that it should not be scaled.
But you did not test that on large system with 1000s of CPUs right?
Quoted 8 lines
No, no. I do not have access to such systems.
Quoted 5 lines
So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).
You mean introduce threshold and count how many nodes are in queue?
Yes.
Quoted 1 line
To me it sounds not optimal and looks like a temporary solution.
Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.
Quoted 2 lines
Long term wise, it is better to split it, i mean to scale.
But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.
Quoted 1 line
Quoted 6 lines
Quoted 1 line
Do you know who can test it on ~1000 CPUs system? So we have some figures.
I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.
Quoted 6 lines
Do you have a patch to try out? We can test it on these systems.
Note: Might take a while to test it, as those systems are bit tricky to get.
Let me prepare something. I will come back.
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 98 lines · Joel Fernandes
Quoted 95 lines · Uladzislau Rezki
Quoted 80 lines · Joel Fernandes
Quoted 80 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki
Hello, Shrikanth!
Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementQuoted 8 lines
Quoted 4 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Also you can try: echo 1 >
/sys/module/rcutree/parameters/rcu_normal_wake_from_gpto speedup regularsynchronize_rcu()call. But i am not saying that it would beat your "expedited switch" improvement.Hi Uladzislau.
Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.
- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.
Quoted 8 lines
Quoted 7 lines
- userspace tools are spread across such as chcpu,
ppc64_cpuetc. though internally most do "0/1 >/sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.- There is already
/sys/kernel/rcu_expeditedwhich is better if at all we need to fallback to userspace.Sounds good to me. I agree it is better to bypass parameters.
Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.
IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)
Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless
synchronize_rcuload is high, default to thesr_normalwake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.
Quoted 7 lines
Quoted 4 lines
How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.
Note i do not say that it should not be scaled.
But you did not test that on large system with 1000s of CPUs right?
No, no. I do not have access to such systems.
Quoted 5 lines
So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).
Quoted 8 lines
Quoted 1 line
You mean introduce threshold and count how many nodes are in queue?
Yes.
Quoted 1 line
To me it sounds not optimal and looks like a temporary solution.
Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.
It was trial testing :) Agree we should do something with it.
Quoted 6 lines
Quoted 2 lines
Long term wise, it is better to split it, i mean to scale.
But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.
Fair point.
Quoted 11 lines
Quoted 2 lines
Do you know who can test it on ~1000 CPUs system? So we have some figures.
I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.
Quoted 2 lines
What i have is 256 CPUs system i can test on.
Same boat. ;-)
:)
--
Uladzislau Rezki
Quoted 14 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)
Quoted 9 lines
Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementAbove numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
Hi Vishal,
I tried on tip/master at 315f416d3e26.
It fails to apply. is rcu tree updated?
Quoted 13 lines
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Signed-off-by Vishal Chourasia <[email protected]>--- include/linux/rcupdate.h | 3 +++ kernel/cpu.c | 2 ++ 2 files changed, 5 insertions(+)include/linux/rcupdate.h +0
-0
include/linux/rcupdate.h rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f) · line 1197
Quoted 6 lines
include/linux/rcupdate.h +3
-0@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)extern int rcu_expedited;extern int rcu_normal; +extern void rcu_expedite_gp (void ); +extern void rcu_unexpedite_gp (void ); +
Why extern is needed? All it needs is declarations no?
Quoted 12 lines
include/linux/rcupdate.h +0
-0DEFINE_LOCK_GUARD_0(rcu, do { rcu_read_lock();kernel/cpu.c +1
-0@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);void cpus_write_lock (void ) { +rcu_expedite_gp ();
kernel/cpu.c EXPORT_SYMBOL_GPL(cpus_read_unlock); · line 519
Quoted 10 lines
kernel/cpu.c +1
-0percpu_down_write (&cpu_hotplug_lock); }void cpus_write_unlock (void ) {percpu_up_write (&cpu_hotplug_lock); +rcu_unexpedite_gp (); }void lockdep_assert_cpus_held (void )
Have you tested kexec path or suspend/resume path? Seems like the counter can nest, but would be good to verify.
Quoted 24 lines · Shrikanth Hegde
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)
Before: real 1m14.792s
After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)
Before: real 2m27.695s
After: real 0m02.510s # ~58x improvement
Quoted 7 lines
Quoted 2 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
Hi Vishal,
I tried on tip/master at
315f416d3e26.
It fails to apply. is rcu tree updated?
I'm currently working off the GitHub mirror (|github.com/torvalds/linux)|
Quoted 10 lines
Quoted 10 lines
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Signed-off-by Vishal Chourasia <[email protected]>---
include/linux/rcupdate.h| 3 +++
kernel/cpu.c | 2 ++2 files changed, 5 insertions(+)
Quoted 13 lines
include/linux/rcupdate.h rcu_head_after_call_rcu(struct rcu_head *rhp, · line 1197
Quoted 11 lines
include/linux/rcupdate.h +2
-0@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f) extern int rcu_expedited; extern int rcu_normal; +extern void rcu_expedite_gp(void); +extern void rcu_unexpedite_gp (void ); +Why extern is needed? All it needs is declarations no?
Already declared in kernel/rcu/rcu.h
kernel/cpu.c already includes linux/rcupdate.h, therefore added an extern.
Quoted 15 lines
Quoted 15 lines
include/linux/rcupdate.h +0
-0DEFINE_LOCK_GUARD_0(rcu, do {
rcu_read_lock();kernel/cpu.c +1
-0@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock); void cpus_write_lock(void) { + rcu_expedite_gp(); percpu_down_write(&cpu_hotplug_lock); }
Quoted 8 lines
kernel/cpu.c EXPORT_SYMBOL_GPL(cpus_read_unlock); · line 519
Quoted 6 lines
kernel/cpu.c +1
-0void cpus_write_unlock(void) { percpu_up_write(&cpu_hotplug_lock); + rcu_unexpedite_gp(); } void lockdep_assert_cpus_held(void)Have you tested kexec path or suspend/resume path?
I did test kexec patch by booting into another kernel via kexec path. But, I didn't test suspend/resume.
Quoted 1 line
Seems like the counter can nest, but would be good to verify.
Joel Fernandes
maintainerQuoted 11 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().
Quoted 3 lines
Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.
When does the user initiate this in your system?
Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.
Quoted 10 lines
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvement
This does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?
Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.
thanks,
- Joel
Quoted 47 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Signed-off-by Vishal Chourasia <[email protected]>---
include/linux/rcupdate.h| 3 +++kernel/cpu.c| 2 ++2 files changed, 5 insertions(+)include/linux/rcupdate.h +3
-0@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)extern int rcu_expedited;extern int rcu_normal; +extern void rcu_expedite_gp (void ); +extern void rcu_unexpedite_gp (void ); + DEFINE_LOCK_GUARD_0(rcu, do { rcu_read_lock();kernel/cpu.c +2
-1@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);void cpus_write_lock (void ) { +rcu_expedite_gp ();percpu_down_write (&cpu_hotplug_lock); }void cpus_write_unlock (void ) {percpu_up_write (&cpu_hotplug_lock); +rcu_unexpedite_gp (); }void lockdep_assert_cpus_held (void ) --2.52.0
Joel Fernandes
maintainerQuoted 31 lines · Joel Fernandes
Quoted 9 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Quoted 3 lines
Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.When does the user initiate this in your system?
Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.
Quoted 7 lines
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement
Quoted 7 lines
Quoted 3 lines
+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementThis does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?
Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.
Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.
- Joel
Quoted 52 lines
thanks,
- Joel
Quoted 47 lines
Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Signed-off-by Vishal Chourasia <[email protected]>---
include/linux/rcupdate.h| 3 +++kernel/cpu.c| 2 ++2 files changed, 5 insertions(+)include/linux/rcupdate.h +3
-0@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)extern int rcu_expedited;extern int rcu_normal; +extern void rcu_expedite_gp (void ); +extern void rcu_unexpedite_gp (void ); + DEFINE_LOCK_GUARD_0(rcu, do { rcu_read_lock();kernel/cpu.c +2
-1@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);void cpus_write_lock (void ) { +rcu_expedite_gp ();percpu_down_write (&cpu_hotplug_lock); }void cpus_write_unlock (void ) {percpu_up_write (&cpu_hotplug_lock); +rcu_unexpedite_gp (); }void lockdep_assert_cpus_held (void ) --2.52.0
Peter Zijlstra
maintainerQuoted 37 lines · Joel Fernandes
Quoted 37 lines · Joel Fernandes
Quoted 9 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Quoted 3 lines
Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.When does the user initiate this in your system?
Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.
Quoted 10 lines
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementThis does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?
Quoted 7 lines
Quoted 1 line
Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.
Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.
Because hotplug is not a fast path; there is no expectation of performance here.
Joel Fernandes
maintainerQuoted 44 lines · Peter Zijlstra
Quoted 39 lines · Joel Fernandes
Quoted 38 lines
Quoted 3 lines · Joel Fernandes
Quoted 9 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Quoted 3 lines
Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.When does the user initiate this in your system?
Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.
Quoted 10 lines
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementThis does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?
Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.
Quoted 8 lines
Quoted 5 lines
Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.
Because hotplug is not a fast path; there is no expectation of performance here.
Agreed, I was just wondering if it was incredibly slow or something. Looking forward to more justification from Vishal on usecase,
- Joel
Quoted 1 line
Hello Joel, Peter
Quoted 20 lines · Joel Fernandes
Quoted 20 lines · Peter Zijlstra
Quoted 15 lines · Joel Fernandes
Quoted 15 lines
Quoted 3 lines · Joel Fernandes
Quoted 9 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.When does the user initiate this in your system?
Workloads exhibit varying sensitivity to SMT levels. Users dynamically adjust SMT modes to optimize performance.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.
Yes, it doesn't happen too often, but when it does, on machines with (>= 1920 CPUs) it takes more than 20 mins to finish.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementThis does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?
The ppc64_cpu utility generates a list of target CPUs based on the requested SMT state and writes to their corresponding sysfs online entries.
Sorry, I didn't get your second question about the performance change with cycle count.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.
Sure, I will get back with the numbers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.
I initially attempted to replace synchronize_rcu() with synchronize_rcu_expedited() at specific callsites. However, the primary bottlenecks are within percpu_down_write(), called via _cpu_up() and try_online_node(). Please refer to the callstack shared below. Since percpu_down_write() is used throughout the kernel, modifying it directly would force expedited grace periods on unrelated subsystems.
@[
synchronize_rcu+12
rcu_sync_enter+260
percpu_down_write+76
_cpu_up+140
cpu_up+440
cpu_subsys_online+128
device_online+176
online_store+220
dev_attr_store+52
sysfs_kf_write+120
kernfs_fop_write_iter+456
vfs_write+952
ksys_write+132
system_call_exception+292
system_call_vectored_common+348
]: 350
@[
synchronize_rcu+12
rcu_sync_enter+260
percpu_down_write+76
try_online_node+64
cpu_up+120
cpu_subsys_online+128
device_online+176
online_store+220
dev_attr_store+52
sysfs_kf_write+120
kernfs_fop_write_iter+456
vfs_write+952
ksys_write+132
system_call_exception+292
system_call_vectored_common+348]: 350
Quoted 3 lines
Quoted 3 lines
Because hotplug is not a fast path; there is no expectation of performance here.
True.
Quoted 7 lines
Agreed, I was just wondering if it was incredibly slow or something. Looking forward to more justification from Vishal on usecase,
- Joel
Quoted 1 line
- vishalc
Peter Zijlstra
maintainerQuoted 13 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
Quoted 8 lines
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvement
But who cares? Its not like you'd *ever* do this, right?
Hello Peter,
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement
Quoted 6 lines
Quoted 4 lines
+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementBut who cares? Its not like you'd *ever* do this, right?
Users dynamically adjust SMT modes to optimize performance of the workload being run. And, yes it doesn't happen too often, but when it does, on machines with (>= 1920 CPUs) it takes more than 20 mins to finish.
- vishal
Peter Zijlstra
maintainerQuoted 28 lines · Vishal Chourasia
Hello Peter,
Quoted 23 lines · Peter Zijlstra
Quoted 21 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvement
Quoted 5 lines
Quoted 1 line
But who cares? Its not like you'd *ever* do this, right?
Users dynamically adjust SMT modes to optimize performance of the workload being run. And, yes it doesn't happen too often, but when it does, on machines with (>= 1920 CPUs) it takes more than 20 mins to finish.
Users cannot change this, it is root only.
Having to change SMT mode per workload seems quite insane; but whatever.
If you do have to put RCU hooks anywhere; I'd much rather see them in cpuhp_smt_{en,dis}able(), such that they only affect the batch hotplug case, rather than everything using cpus_write_lock().
Also note that there is a case to be made to optimize this batch hotplug case; for one it makes no sense to take cpus_write_lock() over and over and over again; if you can pull that out, just like it already lifted cpu_maps_update_begin(), this would help.
And Joel has a point, in that it might make sense for RCU to behave 'better' under these conditions.
PATCH 1/1 cpuhp: Expedite synchronize_rcu during SMT switch
Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().
SMT switch is a user-initiated administrative task, it should complete as quickly as possible.
Performance data on a PPC64 system with 2048 CPUs:
+ ppc64_cpu --smt=1 (SMT8 to SMT1)
Before: real 30m53.194s After: real 6m4.678s # ~5x improvement
+ ppc64_cpu --smt=8 (SMT1 to SMT8)
Before: real 49m5.920s After: real 36m47.798s # ~1.3x improvement
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
--- include/linux/rcupdate.h | 3 +++ kernel/cpu.c | 4 ++++ 2 files changed, 7 insertions(+)
include/linux/rcupdate.h +3 -0
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)extern int rcu_expedited;extern int rcu_normal; +extern void rcu_expedite_gp (void ); +extern void rcu_unexpedite_gp (void ); + DEFINE_LOCK_GUARD_0(rcu, do { rcu_read_lock();
kernel/cpu.c +4 -0
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_expedite_gp ();for_each_online_cpu (cpu) {if (topology_is_primary_thread (cpu))continue ;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) }if (!ret) cpu_smt_control = ctrlval; +rcu_unexpedite_gp ();cpu_maps_update_done ();return ret; }
@@ -2716,6 +2718,7 @@ int cpuhp_smt_enable(void)cpu_maps_update_begin (); cpu_smt_control =CPU_SMT_ENABLED ; +rcu_expedite_gp ();for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */ if (cpu_online (cpu) || !node_online (cpu_to_node (cpu)))
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +rcu_unexpedite_gp ();cpu_maps_update_done ();return ret; }
Signature · 1 line
2.52.0
Peter Zijlstra
maintainerRe: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
Quoted 10 lines · Vishal Chourasia
Expedite
synchronize_rcu()during the cpuhp_smt_[enable|disable] path to accelerate the operation.Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().
You seem to have left out all the useful bits from your changelog again :/
Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.
Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.
kernel/cpu.c +4 -0
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss);for_each_online_cpu (cpu) {if (topology_is_primary_thread (cpu))continue ;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) }if (!ret) cpu_smt_control = ctrlval; +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }
@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss); cpu_smt_control =CPU_SMT_ENABLED ;for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }
Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
Hi Peter.
Quoted 50 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia
Expedite
synchronize_rcu()during the cpuhp_smt_[enable|disable] path to accelerate the operation.Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().You seem to have left out all the useful bits from your changelog again :/
Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.
Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.
kernel/cpu.c +3
-0@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss);for_each_online_cpu (cpu) {if (topology_is_primary_thread (cpu))continue ;@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) }if (!ret) cpu_smt_control = ctrlval; +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss); cpu_smt_control =CPU_SMT_ENABLED ;for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */
kernel/cpu.c int cpuhp_smt_enable(void) · line 2737
Quoted 7 lines
kernel/cpu.c +1
-0@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }
Currently, cpuhp_smt_[enable/disable] calls _cpu_up/_cpu_down which does the same in cpus_write_lock/unlock. though it is per cpu enable/disable one after another.
How hoisting this up will help?
Peter Zijlstra
maintainerRe: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
Quoted 64 lines · Shrikanth Hegde
Hi Peter.
Quoted 48 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia
Expedite
synchronize_rcu()during the cpuhp_smt_[enable|disable] path to accelerate the operation.Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().You seem to have left out all the useful bits from your changelog again :/
Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.
Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.
kernel/cpu.c +3
-0@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss);for_each_online_cpu (cpu) {if (topology_is_primary_thread (cpu))continue ;@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) }if (!ret) cpu_smt_control = ctrlval; +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss); cpu_smt_control =CPU_SMT_ENABLED ;for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */ kernel/cpu.c int cpuhp_smt_enable(void) · line 2737
Quoted 7 lines
kernel/cpu.c +1
-0@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }Currently, cpuhp_smt_[enable/disable] calls _cpu_up/_cpu_down which does the same in cpus_write_lock/unlock. though it is per cpu enable/disable one after another.
Quoted 1 line
How hoisting this up will help?
By holding an extra rcu_sync reference, the percpu rwsem is kept into the the slow path, avoiding the rcu-sync on down_write(), which was very prevalent per this:
Joel Fernandes
maintainerRe: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
Quoted 5 lines · Peter Zijlstra
By holding an extra
rcu_syncreference, the percpu rwsem is kept into the the slow path, avoiding the rcu-sync ondown_write(), which was very prevalent per this:
Makes sense, though I wonder if we should have a separate percpu-rwsem API for this rather than directly accessing the lock-internal rcu_sync state? Other future percpu_rw_semaphore users may benefit as well.
- Joel
Peter Zijlstra
maintainerRe: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
Quoted 10 lines · Joel Fernandes
Quoted 5 lines · Peter Zijlstra
By holding an extra
rcu_syncreference, the percpu rwsem is kept into the the slow path, avoiding the rcu-sync ondown_write(), which was very prevalent per this:Makes sense, though I wonder if we should have a separate percpu-rwsem API for this rather than directly accessing the lock-internal
rcu_syncstate? Other futurepercpu_rw_semaphoreusers may benefit as well.
Yeah, perhaps. There is one other user, the above makes two.
kernel/cgroup/cgroup.c: rcu_sync_enter(&cgroup_threadgroup_rwsem.rss);
Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
In reply to Peter ZijlstraQuoted 49 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia
Expedite
synchronize_rcu()during the cpuhp_smt_[enable|disable] path to accelerate the operation.Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().You seem to have left out all the useful bits from your changelog again :/
Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.
Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.
kernel/cpu.c +3
-0@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss);for_each_online_cpu (cpu) {if (topology_is_primary_thread (cpu))continue ;@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) }if (!ret) cpu_smt_control = ctrlval; +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_sync_enter (&cpu_hotplug_lock.rss); cpu_smt_control =CPU_SMT_ENABLED ;for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */
kernel/cpu.c int cpuhp_smt_enable(void) · line 2737
Quoted 7 lines
kernel/cpu.c +1
-0@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +rcu_sync_exit (&cpu_hotplug_lock.rss);cpu_maps_update_done ();return ret; }
Hi,
I verified this patch using the configuration described below. Configuration:
• Kernel version: 6.19.0-rc6
• Number of CPUs: 1536
Earlier verification of an older version of this patch was performed on a system with *2048 CPUs*. Due to system unavailability, the current verification was carried out on a *different system.*
Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.
SMT=ON -> sys 50m42.805s
SMT=OFF -> sys 0m0.064s
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
SMT=off | 20m 32.210s | 5m 30.898s | +73.15% | SMT=on | 62m 46.549s | 55m 45.671s | +11.18% |
Please add below tag: Tested-by: Samir M <[email protected]>
Regards,
Samir
Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
Quoted 79 lines · Samir M
Quoted 49 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia
Expedite
synchronize_rcu()during the cpuhp_smt_[enable|disable] path to accelerate the operation.Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().You seem to have left out all the useful bits from your changelog again :/
Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.
Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.
kernel/cpu.c +2
-0@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) int cpu, ret = 0; cpu_maps_update_begin(); + rcu_sync_enter(&cpu_hotplug_lock.rss); for_each_online_cpu(cpu) { if (topology_is_primary_thread(cpu)) continue;@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) } if (!ret) cpu_smt_control = ctrlval; + rcu_sync_exit(&cpu_hotplug_lock.rss); cpu_maps_update_done(); return ret;}
kernel/cpu.c +1
-0@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void) int cpu, ret = 0; cpu_maps_update_begin(); + rcu_sync_enter(&cpu_hotplug_lock.rss); cpu_smt_control = CPU_SMT_ENABLED; for_each_present_cpu(cpu) { /* Skip online CPUs and CPUs on offline nodes */kernel/cpu.c int cpuhp_smt_enable(void) · line 2737
Quoted 7 lines
kernel/cpu.c +1
-0@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void) /* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device(cpu); } + rcu_sync_exit(&cpu_hotplug_lock.rss); cpu_maps_update_done(); return ret; }Hi,
I verified this patch using the configuration described below. Configuration:
• Kernel version: 6.19.0-rc6
• Number of CPUs: 1536Earlier verification of an older version of this patch was performed on a system with *2048 CPUs*. Due to system unavailability, the current verification was carried out on a *different system.*
Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.
SMT=ON -> sys 50m42.805s
SMT=OFF -> sys 0m0.064sSMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
Quoted 9 lines
SMT=off | 20m 32.210s | 5m 30.898s | +73.15% | SMT=on | 62m 46.549s | 55m 45.671s | +11.18% |Please add below tag: Tested-by: Samir M <[email protected]>
Regards,
Samir
Hi All,
For reference, I am updating the results for the Vishal and Rezki patches.
Configuration:
Kernel version: 6.19.0-rc6
Number of CPUs: 1536
Earlier verification of an older revision of this patch was performed on a system with 2048 CPUs*.* Due to system unavailability, the current verification was carried out on a different**system. For reference, I am updating the results corresponding to the patches verified using the above configuration.
Patch: https://lore.kernel.org/all/[email protected]/
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
SMT=off | 20m 32.210s | 5m 31.807s | +73.09% | SMT=on | 62m 46.549s | 55m 48.801s | +11.08% |
SMT=ON -> sys 50m44.105s
SMT=OFF -> sys 0m0.035s
patch: https://lore.kernel.org/all/[email protected]/
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
SMT=off | 20m 32.210s | 18m 2.012s | +12.19% | SMT=on | 62m 46.549s | 62m 35.076s | +0.30% |
SMT=ON -> sys 50m43.806s
SMT=OFF -> sys 0m0.109s
Regards,
Samir
Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch
In reply to Samir MQuoted 79 lines · Samir M
Quoted 49 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia
Expedite
synchronize_rcu()during the cpuhp_smt_[enable|disable] path to accelerate the operation.Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().You seem to have left out all the useful bits from your changelog again :/
Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.
Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.
kernel/cpu.c +2
-0@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) int cpu, ret = 0; cpu_maps_update_begin(); + rcu_sync_enter(&cpu_hotplug_lock.rss); for_each_online_cpu(cpu) { if (topology_is_primary_thread(cpu)) continue;@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) } if (!ret) cpu_smt_control = ctrlval; + rcu_sync_exit(&cpu_hotplug_lock.rss); cpu_maps_update_done(); return ret;}
kernel/cpu.c +1
-0@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void) int cpu, ret = 0; cpu_maps_update_begin(); + rcu_sync_enter(&cpu_hotplug_lock.rss); cpu_smt_control = CPU_SMT_ENABLED; for_each_present_cpu(cpu) { /* Skip online CPUs and CPUs on offline nodes */kernel/cpu.c int cpuhp_smt_enable(void) · line 2737
Quoted 7 lines
kernel/cpu.c +1
-0@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void) /* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device(cpu); } + rcu_sync_exit(&cpu_hotplug_lock.rss); cpu_maps_update_done(); return ret; }Hi,
I verified this patch using the configuration described below. Configuration:
• Kernel version: 6.19.0-rc6
• Number of CPUs: 1536Earlier verification of an older version of this patch was performed on a system with *2048 CPUs*. Due to system unavailability, the current verification was carried out on a *different system.*
Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.
SMT=ON -> sys 50m42.805s
SMT=OFF -> sys 0m0.064sSMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
Quoted 9 lines
SMT=off | 20m 32.210s | 5m 30.898s | +73.15% | SMT=on | 62m 46.549s | 55m 45.671s | +11.18% |Please add below tag: Tested-by: Samir M <[email protected]>
Regards,
Samir
Hi All,
Apologies for the confusion in the previous report.
In the previous report, I used the b4 am command to apply the patch. As a result, all patches in the mail thread were applied, rather than only the intended one. Consequently, the results that were posted earlier included changes from multiple patches, and therefore cannot be considered valid for evaluating this patch in isolation. We have since tested Peter’s patch separately, applied in isolation. Based on this testing, we did not observe any improvement in the SMT timing details.
Configuration:
• Kernel version: 6.19.0-rc6
• Number of CPUs: 1536
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
SMT=off | 20m 32.210s | 20m22.441s | +0.79% | SMT=on | 62m 46.549s | 63m0.532s | -0.37% |
Regards,
Samir
PATCH RESEND 1/1 cpuhp: Expedite synchronize_rcu during SMT switch
In reply to Peter Zijlstra[Edits] Added Suggested-by tag
Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().
SMT switch is a user-initiated administrative task, it should complete as quickly as possible.
Performance data on a PPC64 system with 2048 CPUs:
+ ppc64_cpu --smt=1 (SMT8 to SMT1)
Before: real 30m53.194s After: real 6m4.678s # ~5x improvement
+ ppc64_cpu --smt=8 (SMT1 to SMT8)
Before: real 49m5.920s After: real 36m47.798s # ~1.3x improvement
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
[2] https://lore.kernel.org/all/[email protected]/
--- include/linux/rcupdate.h | 3 +++ kernel/cpu.c | 4 ++++ 2 files changed, 7 insertions(+)
include/linux/rcupdate.h +3 -0
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)extern int rcu_expedited;extern int rcu_normal; +extern void rcu_expedite_gp (void ); +extern void rcu_unexpedite_gp (void ); + DEFINE_LOCK_GUARD_0(rcu, do { rcu_read_lock();
kernel/cpu.c +4 -0
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)int cpu, ret =0 ;cpu_maps_update_begin (); +rcu_expedite_gp ();for_each_online_cpu (cpu) {if (topology_is_primary_thread (cpu))continue ;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) }if (!ret) cpu_smt_control = ctrlval; +rcu_unexpedite_gp ();cpu_maps_update_done ();return ret; }
@@ -2716,6 +2718,7 @@ int cpuhp_smt_enable(void)cpu_maps_update_begin (); cpu_smt_control =CPU_SMT_ENABLED ; +rcu_expedite_gp ();for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */ if (cpu_online (cpu) || !node_online (cpu_to_node (cpu)))
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +rcu_unexpedite_gp ();cpu_maps_update_done ();return ret; }
Signature · 1 line
2.52.0
Quoted 54 lines · Vishal Chourasia
Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementAbove numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
Signed-off-by Vishal Chourasia <[email protected]>--- include/linux/rcupdate.h | 3 +++ kernel/cpu.c | 2 ++ 2 files changed, 5 insertions(+)include/linux/rcupdate.h +3
-0@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)extern int rcu_expedited;extern int rcu_normal; +extern void rcu_expedite_gp (void ); +extern void rcu_unexpedite_gp (void ); + DEFINE_LOCK_GUARD_0(rcu, do { rcu_read_lock();kernel/cpu.c +1
-0@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);void cpus_write_lock (void ) { +rcu_expedite_gp ();
kernel/cpu.c EXPORT_SYMBOL_GPL(cpus_read_unlock); · line 519
Quoted 10 lines
kernel/cpu.c +1
-0percpu_down_write (&cpu_hotplug_lock); }void cpus_write_unlock (void ) {percpu_up_write (&cpu_hotplug_lock); +rcu_unexpedite_gp (); }void lockdep_assert_cpus_held (void )
Hi Vishal,
I verified this patch using the configuration described below. Configuration:
• Kernel version: 6.19.0-rc5
• Number of CPUs: 2048
Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.
SMT=ON -> sys 31m18.849s
SMT=OFF -> sys 0m0.087s
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
SMT=off | 30m 53.194s | 6m 4.250s | +80.40% | SMT=on | 49m 5.920s | 36m 50.386s | +25.01% |
Please add below tag: Tested-by: Samir M <[email protected]>
Regards,
Samir
Joel Fernandes
maintainerQuoted 19 lines · Samir M
Quoted 10 lines · Vishal Chourasia
Bulk CPU hotplug operations--such as switching SMT modes across all cores--require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Hi Vishal,
I verified this patch using the configuration described below. Configuration:
* Kernel version: 6.19.0-rc5
* Number of CPUs: 2048
Quoted 4 lines
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------SMT=off | 30m 53.194s | 6m 4.250s | +80.40% | SMT=on | 49m 5.920s | 36m 50.386s | +25.01% |
Hi Vishal, Samir,
Thanks for the testing on your large CPU count system.
Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.
git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
tag: cpuhp-bulk-optimize-rfc-v1
I tested it lightly on rcutorture hotplug test and it passes. Please share any performance results, thanks.
Also I'd like to use expediting of RCU as a last resort TBH, we should optimize the outer operations that require RCU in the first place such as Peter's suggestion since that will improve the overall efficiency of the code. And if/when expediting RCU, Peter's other suggestion to not do it in cpus_write_lock() and instead do it from cpuhp_smt_enable() also makes sense to me.
---8<-----------------------
From: Joel Fernandes <[email protected]>
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiring
Bulk CPU hotplug operations such as enabling SMT across all cores require hotplugging multiple CPUs. The current implementation takes cpus_write_lock() for each individual CPU causing multiple slow grace period requests.
Therefore introduce cpu_up_locked() that assumes the caller already holds cpus_write_lock(). The cpuhp_smt_enable() function is updated to hold the lock once around the entire loop rather than for each CPU.
--- kernel/cpu.c | 40 +++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-)
kernel/cpu.c +25 -15
@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state)complete_ap_thread (st, true); } -/* Requires cpu_add_remove_lock to be held */ -static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */ +static int cpu_up_locked (unsigned int cpu,int tasks_frozen, +enum cpuhp_state target) {struct cpuhp_cpu_state *st =per_cpu_ptr (&cpuhp_state, cpu);struct task_struct *idle;int ret =0 ; -cpus_write_lock (); +lockdep_assert_cpus_held (); -if (!cpu_present (cpu)) { - ret = -EINVAL ; - goto out; - } +if (!cpu_present (cpu)) +return -EINVAL ;/* * The caller of cpu_up() might have raced with another * caller. Nothing to do. */ if (st->state >= target) - goto out; +return 0 ;if (st->state ==CPUHP_OFFLINE ) {/* Let it fail before we try to bring the cpu up */ idle =idle_thread_get (cpu); -if (IS_ERR (idle)) { - ret =PTR_ERR (idle); - goto out; - } +if (IS_ERR (idle)) +return PTR_ERR (idle);/* * Reset stale stack state from the last time this CPU was online.
@@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)* return the error code.. */ if (ret) - goto out; +return ret; }/*
@@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)*/ target =min ((int )target,CPUHP_BRINGUP_CPU ); ret =cpuhp_up_callbacks (cpu, st, target); -out: +return ret; +} + +/* Requires cpu_add_remove_lock to be held */ +static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +{ +int ret; + +cpus_write_lock (); + ret =cpu_up_locked (cpu, tasks_frozen, target);cpus_write_unlock ();arch_smt_update ();return ret;
@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void)int cpu, ret =0 ;cpu_maps_update_begin (); +/* Hold cpus_write_lock() for entire batch operation. */ +cpus_write_lock (); cpu_smt_control =CPU_SMT_ENABLED ;for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */
@@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void)continue ;if (!cpu_smt_thread_allowed (cpu) || !topology_is_core_online (cpu))continue ; - ret =_cpu_up (cpu,0 ,CPUHP_ONLINE ); + ret =cpu_up_locked (cpu,0 ,CPUHP_ONLINE );if (ret)break ;/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +cpus_write_unlock (); +arch_smt_update ();cpu_maps_update_done ();return ret; }
Signature · 1 line
2.34.1
Quoted 171 lines · Joel Fernandes
Quoted 19 lines · Samir M
Quoted 10 lines · Vishal Chourasia
Bulk CPU hotplug operations--such as switching SMT modes across all cores--require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Hi Vishal,
I verified this patch using the configuration described below. Configuration:
* Kernel version: 6.19.0-rc5
* Number of CPUs: 2048Quoted 4 lines
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------SMT=off | 30m 53.194s | 6m 4.250s | +80.40% | SMT=on | 49m 5.920s | 36m 50.386s | +25.01% |Hi Vishal, Samir,
Thanks for the testing on your large CPU count system.
Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.
git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
tag: cpuhp-bulk-optimize-rfc-v1I tested it lightly on rcutorture hotplug test and it passes. Please share any performance results, thanks.
Also I'd like to use expediting of RCU as a last resort TBH, we should optimize the outer operations that require RCU in the first place such as Peter's suggestion since that will improve the overall efficiency of the code. And if/when expediting RCU, Peter's other suggestion to not do it in
cpus_write_lock()and instead do it fromcpuhp_smt_enable()also makes sense to me.---8<-----------------------
From: Joel Fernandes <[email protected]>
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiringBulk CPU hotplug operations such as enabling SMT across all cores require hotplugging multiple CPUs. The current implementation takes
cpus_write_lock()for each individual CPU causing multiple slow grace period requests.Therefore introduce
cpu_up_locked()that assumes the caller already holdscpus_write_lock(). Thecpuhp_smt_enable()function is updated to hold the lock once around the entire loop rather than for each CPU.Link https://lore.kernel.org/ all/[email protected]/Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- kernel/cpu.c | 40 +++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-)kernel/cpu.c +9
-12@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state) complete_ap_thread(st, true); } -/* Requires cpu_add_remove_lock to be held */ -static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */ +static int cpu_up_locked(unsigned int cpu, int tasks_frozen, + enum cpuhp_state target) { struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu); struct task_struct *idle; int ret = 0; - cpus_write_lock(); + lockdep_assert_cpus_held(); - if (!cpu_present(cpu)) { - ret = -EINVAL; - goto out; - } + if (!cpu_present(cpu)) + return -EINVAL; /* * The caller of cpu_up() might have raced with another * caller. Nothing to do. */ if (st->state >= target) - goto out; + return 0; if (st->state == CPUHP_OFFLINE) { /* Let it fail before we try to bring the cpu up */ idle = idle_thread_get(cpu); - if (IS_ERR(idle)) { - ret = PTR_ERR(idle); - goto out; - } + if (IS_ERR(idle)) + return PTR_ERR(idle); /** Reset stale stack state from the last time this CPU was
online.kernel/cpu.c +1
-1@@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, inttasks_frozen, enum cpuhp_state target) * return the error code.. */ if (ret) - goto out; + return ret; }/*
kernel/cpu.c +10
-1@@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, inttasks_frozen, enum cpuhp_state target) */ target = min((int)target, CPUHP_BRINGUP_CPU); ret = cpuhp_up_callbacks(cpu, st, target); -out: + return ret; +} + +/* Requires cpu_add_remove_lock to be held */ +static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +{ + int ret; + + cpus_write_lock(); + ret = cpu_up_locked(cpu, tasks_frozen, target); cpus_write_unlock();
arch_smt_update();
return ret;kernel/cpu.c +5
-1@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void) int cpu, ret = 0; cpu_maps_update_begin(); + /* Hold cpus_write_lock() for entire batch operation. */ + cpus_write_lock(); cpu_smt_control = CPU_SMT_ENABLED; for_each_present_cpu(cpu) { /* Skip online CPUs and CPUs on offline nodes */@@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void) continue; if (!cpu_smt_thread_allowed(cpu) || !topology_is_core_online (cpu)) continue; - ret = _cpu_up(cpu, 0, CPUHP_ONLINE); + ret = cpu_up_locked(cpu, 0, CPUHP_ONLINE); if (ret) break; /* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device(cpu); } + cpus_write_unlock(); + arch_smt_update(); cpu_maps_update_done(); return ret;
Quoted 1 line
}
What about cpuhp_smt_disable?
[email protected]
maintainerQuoted 147 lines · Shrikanth Hegde
Quoted 141 lines · Joel Fernandes
Quoted 20 lines · Samir M
Quoted 11 lines · Vishal Chourasia
Bulk CPU hotplug operations--such as switching SMT modes across all cores--require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for
synchronize_rcu().Expedite
synchronize_rcu()during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.Hi Vishal,
I verified this patch using the configuration described below. Configuration:
* Kernel version: 6.19.0-rc5
* Number of CPUs: 2048Quoted 4 lines
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------SMT=off | 30m 53.194s | 6m 4.250s | +80.40% | SMT=on | 49m 5.920s | 36m 50.386s | +25.01% |Hi Vishal, Samir,
Thanks for the testing on your large CPU count system.
Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.
git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
tag: cpuhp-bulk-optimize-rfc-v1
I tested it lightly on rcutorture hotplug test and it passes. Please share
any performance results, thanks.
Also I'd like to use expediting of RCU as a last resort TBH, we should
optimize the outer operations that require RCU in the first place such as
Peter's suggestion since that will improve the overall efficiency of the
code. And if/when expediting RCU, Peter's other suggestion to not do it incpus_write_lock()and instead do it fromcpuhp_smt_enable()also makes sense
to me.
---8<-----------------------
From: Joel Fernandes
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiring
Bulk CPU hotplug operations such as enabling SMT across all cores
require hotplugging multiple CPUs. The current implementation takescpus_write_lock()for each individual CPU causing multiple slow grace
period requests.
Therefore introducecpu_up_locked()that assumes the caller already
holdscpus_write_lock(). Thecpuhp_smt_enable()function is updated to
hold the lock once around the entire loop rather than for each CPU.Link https://lore.kernel.org/ all/[email protected]/Suggested-by Peter ZijlstraSigned-off-by Joel Fernandes--- kernel/cpu.c | 40 +++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-)kernel/cpu.c +23
-15@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state)complete_ap_thread (st, true); } -/* Requires cpu_add_remove_lock to be held */ -static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */ +static int cpu_up_locked (unsigned int cpu,int tasks_frozen, +enum cpuhp_state target) {struct cpuhp_cpu_state *st =per_cpu_ptr (&cpuhp_state, cpu);struct task_struct *idle;int ret =0 ; -cpus_write_lock (); +lockdep_assert_cpus_held (); -if (!cpu_present (cpu)) { - ret = -EINVAL ; - goto out; - } +if (!cpu_present (cpu)) +return -EINVAL ;/* * The caller of cpu_up() might have raced with another * caller. Nothing to do. */ if (st->state >= target) - goto out; +return 0 ;if (st->state ==CPUHP_OFFLINE ) {/* Let it fail before we try to bring the cpu up */ idle =idle_thread_get (cpu); -if (IS_ERR (idle)) { - ret =PTR_ERR (idle); - goto out; - } +if (IS_ERR (idle)) +return PTR_ERR (idle);/* * Reset stale stack state from the last time this CPU was online. @@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)* return the error code.. */ if (ret) - goto out; +return ret; }/* @@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)*/ target =min ((int )target,CPUHP_BRINGUP_CPU ); ret =cpuhp_up_callbacks (cpu, st, target); -out: +return ret; +} + +/* Requires cpu_add_remove_lock to be held */ +static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +{ +int ret; + +cpus_write_lock (); + ret =cpu_up_locked (cpu, tasks_frozen, target);cpus_write_unlock ();arch_smt_update ();return ret;@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void)int cpu, ret =0 ;cpu_maps_update_begin (); +/* Hold cpus_write_lock() for entire batch operation. */ +cpus_write_lock (); cpu_smt_control =CPU_SMT_ENABLED ;for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */ @@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void)continue ;if (!cpu_smt_thread_allowed (cpu) || !topology_is_core_online (cpu))continue ; - ret =_cpu_up (cpu,0 ,CPUHP_ONLINE ); + ret =cpu_up_locked (cpu,0 ,CPUHP_ONLINE );if (ret)break ;/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu);
Quoted 8 lines
kernel/cpu.c int cpuhp_smt_enable(void) · line 2743
Quoted 6 lines
kernel/cpu.c +2
-0} +cpus_write_unlock (); +arch_smt_update ();cpu_maps_update_done ();return ret; }What about
cpuhp_smt_disable?
Doing this is not that easy on disable path, AFAICS. Considering that the enable path in the performance tests was much worse, I wanted to contain it to that.
This does not have to be the only fix though, but one of the cures to get there.
Thanks.
Quoted 11 lines · Vishal Chourasia
Performance data on a PPC64 system with 400 CPUs:
+
ppc64_cpu--smt=1 (SMT8 to SMT1)Before: real 1m14.792s After: real 0m03.205s # ~23x improvement+
ppc64_cpu--smt=8 (SMT1 to SMT8)Before: real 2m27.695s After: real 0m02.510s # ~58x improvementAbove numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b
Quoted 137 lines · Joel Fernandes
Hi Vishal, Samir,
Thanks for the testing on your large CPU count system.
Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.
git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
tag: cpuhp-bulk-optimize-rfc-v1I tested it lightly on rcutorture hotplug test and it passes. Please share any performance results, thanks.
Also I'd like to use expediting of RCU as a last resort TBH, we should optimize the outer operations that require RCU in the first place such as Peter's suggestion since that will improve the overall efficiency of the code. And if/when expediting RCU, Peter's other suggestion to not do it in
cpus_write_lock()and instead do it fromcpuhp_smt_enable()also makes sense to me.---8<-----------------------
From: Joel Fernandes <[email protected]>
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiringBulk CPU hotplug operations such as enabling SMT across all cores require hotplugging multiple CPUs. The current implementation takes
cpus_write_lock()for each individual CPU causing multiple slow grace period requests.Therefore introduce
cpu_up_locked()that assumes the caller already holdscpus_write_lock(). Thecpuhp_smt_enable()function is updated to hold the lock once around the entire loop rather than for each CPU.Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- kernel/cpu.c | 40 +++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-)kernel/cpu.c +25
-15@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state)complete_ap_thread (st, true); } -/* Requires cpu_add_remove_lock to be held */ -static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */ +static int cpu_up_locked (unsigned int cpu,int tasks_frozen, +enum cpuhp_state target) {struct cpuhp_cpu_state *st =per_cpu_ptr (&cpuhp_state, cpu);struct task_struct *idle;int ret =0 ; -cpus_write_lock (); +lockdep_assert_cpus_held (); -if (!cpu_present (cpu)) { - ret = -EINVAL ; - goto out; - } +if (!cpu_present (cpu)) +return -EINVAL ;/* * The caller of cpu_up() might have raced with another * caller. Nothing to do. */ if (st->state >= target) - goto out; +return 0 ;if (st->state ==CPUHP_OFFLINE ) {/* Let it fail before we try to bring the cpu up */ idle =idle_thread_get (cpu); -if (IS_ERR (idle)) { - ret =PTR_ERR (idle); - goto out; - } +if (IS_ERR (idle)) +return PTR_ERR (idle);/* * Reset stale stack state from the last time this CPU was online. @@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)* return the error code.. */ if (ret) - goto out; +return ret; }/* @@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)*/ target =min ((int )target,CPUHP_BRINGUP_CPU ); ret =cpuhp_up_callbacks (cpu, st, target); -out: +return ret; +} + +/* Requires cpu_add_remove_lock to be held */ +static int _cpu_up (unsigned int cpu,int tasks_frozen,enum cpuhp_state target) +{ +int ret; + +cpus_write_lock (); + ret =cpu_up_locked (cpu, tasks_frozen, target);cpus_write_unlock ();arch_smt_update ();return ret;@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void)int cpu, ret =0 ;cpu_maps_update_begin (); +/* Hold cpus_write_lock() for entire batch operation. */ +cpus_write_lock (); cpu_smt_control =CPU_SMT_ENABLED ;for_each_present_cpu (cpu) {/* Skip online CPUs and CPUs on offline nodes */ @@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void)continue ;if (!cpu_smt_thread_allowed (cpu) || !topology_is_core_online (cpu))continue ; - ret =_cpu_up (cpu,0 ,CPUHP_ONLINE ); + ret =cpu_up_locked (cpu,0 ,CPUHP_ONLINE );if (ret)break ;/* See comment in cpuhp_smt_disable() */ cpuhp_online_cpu_device (cpu); } +cpus_write_unlock (); +arch_smt_update ();cpu_maps_update_done ();return ret; }Signature · 1 line
2.34.1
Hi Joel,
I tested above patch on 400 CPU machine that I had originally posted the numbers for.
# time echo 1 > /sys/devices/system/cpu/smt/control real 1m27.133s # Base real 1m25.859s # With patch
# time echo 8 > /sys/devices/system/cpu/smt/control real 1m0.682s # Base real 1m3.423s # With patch