Skip to contentSkip to conversation

cpuhp: Expedite synchronize_rcu during CPU hotplug operations

PATCH In review Vishal Chourasia 7 participants 54 messages Jan 12 – Feb 3

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpuhp: Expedite synchronize_rcu during CPU hotplug operations

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

---
 include/linux/rcupdate.h | 3 +++
 kernel/cpu.c             | 2 ++
 2 files changed, 5 insertions(+)
include/linux/rcupdate.h +3 -0
index c5b30054cd01..03c06cfb2b6d 100644
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
 extern int rcu_expedited;
 extern int rcu_normal;
 
+extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+
 DEFINE_LOCK_GUARD_0(rcu,
 	do {
 		rcu_read_lock();
kernel/cpu.c +2 -0
index 8df2d773fe3b..6b0d491d73f4 100644
@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);
 
 void cpus_write_lock(void)
 {
+	rcu_expedite_gp();
 	percpu_down_write(&cpu_hotplug_lock);
 }
 
 void cpus_write_unlock(void)
 {
 	percpu_up_write(&cpu_hotplug_lock);
+	rcu_unexpedite_gp();
 }
 
 void lockdep_assert_cpus_held(void)
Signature · 1 line
2.52.0
maintainer
#1 lore raw
Quoted 17 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement
Quoted 8 lines

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

--
Uladzislau Rezki

#2 lore raw

Hi Uladzislau,

Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Vishal Chourasia

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 7 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

# echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp

After setting,

# time ppc64_cpu --smt=1;
real    1m10.726s          # Run 1
real    1m12.530s          # Run 2
# time ppc64_cpu --smt=8
real    0m36.661s          # Run 1
real    0m41.401s          # Run 2

Regards,
vishalc

maintainer
#3 lore raw

Hello, Vishalc!

Quoted 26 lines

Hi Uladzislau,

Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Vishal Chourasia

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 7 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

# echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp

After setting,

Quoted 8 lines
# time ppc64_cpu --smt=1;
real    1m10.726s          # Run 1
real    1m12.530s          # Run 2
# time ppc64_cpu --smt=8
real    0m36.661s          # Run 1
real    0m41.401s          # Run 2

Thanks.

"ppc64_cpu --smt=1" is the same, i assume it is offlining. "ppc64_cpu --smt=8", whereas, onlining, sees the differences(~5x).

But your real "0m02.510s" is hard to beat event by activating the "rcu_normal_wake_from_gp" option.

--
Uladzislau Rezki

#4 lore raw
In reply to Uladzislau Rezki

Hi Vishal. Thanks for the patch.

Quoted 22 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

maintainer
#5 lore raw

Hello, Shrikanth!

Quoted 41 lines · Uladzislau Rezki
Quoted 22 lines · Vishal Chourasia
Quoted 21 lines

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

--
Uladzislau Rezki

maintainer
#6 lore raw
Quoted 47 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 8 lines
Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

thanks,

- Joel

Quoted 3 lines

--
Uladzislau Rezki

maintainer
#7 lore raw
Quoted 51 lines · Joel Fernandes
Quoted 51 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 45 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Thanx, Paul

maintainer
#8 lore raw
Quoted 55 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

I tested this before. I noticed that after 64K of simultaneous synchronize_rcu() calls the scalability is required. Everything less was faster with a new approach.

I can retest. Should i? :)

--
Uladzsislau Rezki

#9 lore raw

Hello Joel, Paul, Uladzislau,

Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Paul E. McKenney
Quoted 9 lines · Joel Fernandes
Quoted 6 lines · Uladzislau Rezki
Quoted 2 lines

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

I tested this before. I noticed that after 64K of simultaneous synchronize_rcu() calls the scalability is required. Everything less was faster with a new approach.

It is worth noting that bulk CPU hotplug represents a different stress pattern than the "simultaneous call" scenario mentioned above.

In a large-scale hotplug event (like a SMT mode switch), we aren't necessarily seeing thousands of simultaneous synchronize_rcu() calls. Instead, because CPU hotplug operations are serialized, we see a "conveyor belt" of sequential calls. One synchronize_rcu() blocks, the hotplug state machine waits, it unblocks, and then the next call is triggered shortly after.

The bottleneck here isn't RCU scalability under concurrent load, but rather the accumulated latency of hundreds of sequential Grace Periods.

For example, on pSeries, onlining 350 out of 400 CPUs triggers exactly 350 calls at three different points in the hotplug state machine. Even though they happen one at a time, the sheer volume makes the total operation time prohibitive.

Following callstack was collected during SMT mode switch where 350 out of 400 CPUs were onlined,

@[

    synchronize_rcu+12
    cpuidle_pause_and_lock+120
    pseries_cpuidle_cpu_online+88
    cpuhp_invoke_callback+500
    cpuhp_thread_fun+316
    smpboot_thread_fn+512
    kthread+308
    start_kernel_thread+20
]: 350
@[
    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    _cpu_up+140
    cpu_up+440
    cpu_subsys_online+128
    device_online+176
    online_store+220
    dev_attr_store+52
    sysfs_kf_write+120
    kernfs_fop_write_iter+456
    vfs_write+952
    ksys_write+132
    system_call_exception+292
    system_call_vectored_common+348
]: 350
@[
    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    try_online_node+64
    cpu_up+120
    cpu_subsys_online+128
    device_online+176
    online_store+220
    dev_attr_store+52
    sysfs_kf_write+120
    kernfs_fop_write_iter+456
    vfs_write+952
    ksys_write+132
    system_call_exception+292
    system_call_vectored_common+348

]: 350

Following callstack was collected during SMT mode switch where 350 out of 400 CPUs where offlined,

@[

    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    _cpu_down+188
    __cpu_down_maps_locked+44
    work_for_cpu_fn+56
    process_one_work+508
    worker_thread+840
    kthread+308
    start_kernel_thread+20
]: 1
@[
    synchronize_rcu+12
    sched_cpu_deactivate+244
    cpuhp_invoke_callback+500
    cpuhp_thread_fun+316
    smpboot_thread_fn+512
    kthread+308
    start_kernel_thread+20
]: 350
@[
    synchronize_rcu+12
    cpuidle_pause_and_lock+120
    pseries_cpuidle_cpu_dead+88
    cpuhp_invoke_callback+500
    __cpuhp_invoke_callback_range+200
    _cpu_down+412
    __cpu_down_maps_locked+44
    work_for_cpu_fn+56
    process_one_work+508
    worker_thread+840
    kthread+308
    start_kernel_thread+20

]: 350

- vishalc

maintainer
#10 lore raw
Quoted 126 lines · Vishal Chourasia

Hello Joel, Paul, Uladzislau,

Quoted 11 lines · Uladzislau Rezki
Quoted 10 lines · Paul E. McKenney
Quoted 9 lines · Joel Fernandes
Quoted 6 lines · Uladzislau Rezki
Quoted 2 lines

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

I tested this before. I noticed that after 64K of simultaneous synchronize_rcu() calls the scalability is required. Everything less was faster with a new approach.

It is worth noting that bulk CPU hotplug represents a different stress pattern than the "simultaneous call" scenario mentioned above.

In a large-scale hotplug event (like a SMT mode switch), we aren't necessarily seeing thousands of simultaneous synchronize_rcu() calls. Instead, because CPU hotplug operations are serialized, we see a "conveyor belt" of sequential calls. One synchronize_rcu() blocks, the hotplug state machine waits, it unblocks, and then the next call is triggered shortly after.

The bottleneck here isn't RCU scalability under concurrent load, but rather the accumulated latency of hundreds of sequential Grace Periods.

For example, on pSeries, onlining 350 out of 400 CPUs triggers exactly 350 calls at three different points in the hotplug state machine. Even though they happen one at a time, the sheer volume makes the total operation time prohibitive.

Following callstack was collected during SMT mode switch where 350 out of 400 CPUs were onlined,

@[

    synchronize_rcu+12
    cpuidle_pause_and_lock+120
    pseries_cpuidle_cpu_online+88
    cpuhp_invoke_callback+500
    cpuhp_thread_fun+316
    smpboot_thread_fn+512
    kthread+308
    start_kernel_thread+20
]: 350
@[
    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    _cpu_up+140
    cpu_up+440
    cpu_subsys_online+128
    device_online+176
    online_store+220
    dev_attr_store+52
    sysfs_kf_write+120
    kernfs_fop_write_iter+456
    vfs_write+952
    ksys_write+132
    system_call_exception+292
    system_call_vectored_common+348
]: 350
@[
    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    try_online_node+64
    cpu_up+120
    cpu_subsys_online+128
    device_online+176
    online_store+220
    dev_attr_store+52
    sysfs_kf_write+120
    kernfs_fop_write_iter+456
    vfs_write+952
    ksys_write+132
    system_call_exception+292
    system_call_vectored_common+348

]: 350

Following callstack was collected during SMT mode switch where 350 out of 400 CPUs where offlined,

@[

    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    _cpu_down+188
    __cpu_down_maps_locked+44
    work_for_cpu_fn+56
    process_one_work+508
    worker_thread+840
    kthread+308
    start_kernel_thread+20
]: 1
@[
    synchronize_rcu+12
    sched_cpu_deactivate+244
    cpuhp_invoke_callback+500
    cpuhp_thread_fun+316
    smpboot_thread_fn+512
    kthread+308
    start_kernel_thread+20
]: 350
@[
    synchronize_rcu+12
    cpuidle_pause_and_lock+120
    pseries_cpuidle_cpu_dead+88
    cpuhp_invoke_callback+500
    __cpuhp_invoke_callback_range+200
    _cpu_down+412
    __cpu_down_maps_locked+44
    work_for_cpu_fn+56
    process_one_work+508
    worker_thread+840
    kthread+308
    start_kernel_thread+20
Quoted 1 line

]: 350

I still suggest that you test on a big system. There are other sources of synchronize_rcu() calls than just CPU hotplug. ;-)

Thanx, Paul

maintainer
#11 lore raw
In reply to Paul E. McKenney
Quoted 55 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Quoted 9 lines
Quoted 6 lines
Quoted 1 line

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Thoughts?

thanks,

- Joel

maintainer
#12 lore raw
Quoted 70 lines · Joel Fernandes
Quoted 55 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Quoted 9 lines
Quoted 6 lines
Quoted 1 line

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

Quoted 8 lines

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Thanx, Paul

maintainer
#13 lore raw
Quoted 82 lines · Paul E. McKenney
Quoted 77 lines · Joel Fernandes
Quoted 58 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Quoted 9 lines
Quoted 6 lines
Quoted 1 line

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Quoted 4 lines
Quoted 1 line

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off.

That would address the conveyor belt pattern Vishal expressed.

thanks,

- Joel

Quoted 2 lines

Thanx, Paul

#14 lore raw

Hi.

Quoted 26 lines · Joel Fernandes
Quoted 21 lines
Quoted 21 lines
Quoted 11 lines
Quoted 8 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Quoted 4 lines
Quoted 1 line

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Quoted 8 lines

Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off.

That would address the conveyor belt pattern Vishal expressed.

thanks,

- Joel

Wouldn't that make most of the sync_rcu calls on large system with synchronize_rcu_normal off?

Whats the cost of doing this?

(Me not knowing much about rcu internals)

maintainer
#15 lore raw
Quoted 30 lines · Shrikanth Hegde

Hi.

Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

Quoted 8 lines

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Quoted 7 lines
Quoted 4 lines

Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off. That would address the conveyor belt pattern Vishal expressed.
thanks,
- Joel

Wouldn't that make most of the sync_rcu calls on large system with synchronize_rcu_normal off?

It would and that is expected.

Quoted 2 lines

Whats the cost of doing this?

There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.

- Joel

Quoted 2 lines

(Me not knowing much about rcu internals)

maintainer
#16 lore raw
Quoted 38 lines · Joel Fernandes
Quoted 30 lines · Shrikanth Hegde

Hi.

Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

Quoted 8 lines

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Quoted 7 lines
Quoted 4 lines

Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off. That would address the conveyor belt pattern Vishal expressed.
thanks,
- Joel

Wouldn't that make most of the sync_rcu calls on large system with synchronize_rcu_normal off?

Quoted 7 lines

It would and that is expected.

Quoted 2 lines

Whats the cost of doing this?

There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.

Also the contention involved in the list management, if there is still only the one list.

Thanx, Paul

maintainer
#17 lore raw
Quoted 42 lines · Paul E. McKenney
Quoted 41 lines · Joel Fernandes
Quoted 30 lines · Shrikanth Hegde

Hi.

Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

Quoted 8 lines

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Quoted 7 lines
Quoted 4 lines

Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off. That would address the conveyor belt pattern Vishal expressed.
thanks,
- Joel

Wouldn't that make most of the sync_rcu calls on large system with synchronize_rcu_normal off?

It would and that is expected.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Whats the cost of doing this?

There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.

Also the contention involved in the list management, if there is still only the one list.

Even if the number of synchronize_rcu() in flight is a small number? like < 10. To clarify, I meant keeping the threshold that small in favor of the list contention issue you're raising.

Thanks!

- Joel

maintainer
#18 lore raw
Quoted 50 lines · Joel Fernandes
Quoted 47 lines · Paul E. McKenney
Quoted 45 lines · Joel Fernandes
Quoted 30 lines · Shrikanth Hegde

Hi.

Quoted 21 lines · Joel Fernandes
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

Quoted 8 lines

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Quoted 7 lines
Quoted 4 lines

Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off. That would address the conveyor belt pattern Vishal expressed.
thanks,
- Joel

Wouldn't that make most of the sync_rcu calls on large system with synchronize_rcu_normal off?

It would and that is expected.

Quoted 2 lines

Whats the cost of doing this?

There is no cost, that is the point right. The scalability issue Paul is referring to is the large number of wake ups. You wont have that if the number of synchronous callers is small.

Quoted 6 lines
Quoted 3 lines

Also the contention involved in the list management, if there is still only the one list.

Even if the number of synchronize_rcu() in flight is a small number? like < 10. To clarify, I meant keeping the threshold that small in favor of the list contention issue you're raising.

Maybe?

One remaining concern is the reads of the counter. Another remaining concern is the possibility of large numbers of CPUs reading the counter, seeing that it is small, then all piling on the list. But it looks like you might get some testing on a large system, so nothing quite like finding out.

If it breaks, you guys get to fix it on whatever schedule is indicated at that time. ;-)

Thanx, Paul

maintainer
#19 lore raw
In reply to Joel Fernandes
Quoted 82 lines · Joel Fernandes
Quoted 82 lines · Paul E. McKenney
Quoted 77 lines · Joel Fernandes
Quoted 58 lines · Paul E. McKenney
Quoted 54 lines · Joel Fernandes
Quoted 54 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 48 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Quoted 9 lines
Quoted 6 lines
Quoted 1 line

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.>>
I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.>

This would require increasing the scalability of this optimization, right? Or am I thinking of the wrong optimization? ;-)

Yes I think you are considering the correct one, the concern you have is regarding large number of wake ups initiated from the GP thread, correct?

I was suggesting on the thread, a more dynamic approach where using synchronize_rcu_normal() until it gets overloaded with requests. One approach might be to measure the length of the rcu_state.srs_next to detect an overload condition, similar to qhimark? Or perhaps qhimark itself can be used. And under lightly loaded conditions, default to synchronize_rcu_normal() without checking for the 16 CPU count.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Thoughts?

Or maintain multiple lists. Systems with 1000+ CPUs can be a bit unforgiving of pretty much any form of contention.

Makes sense. We could also just have a single list but a much smaller threshold for switching synchronize_rcu_normal off.

That would address the conveyor belt pattern Vishal expressed.

On a system with more than 1,000 CPUs, any single list needs to be handled
extremely carefully to avoid contention of one sort or another. At that
many CPUs, the default rule is instead "never have just one of anything". Unless that "just one" is protected by some contention-avoidance scheme, for example, like the rcu_node tree protects the root rcu_node structure and the rcu_state structure from contention.

Thanx, Paul

maintainer
#20 lore raw
In reply to Joel Fernandes
Quoted 51 lines · Joel Fernandes
Quoted 51 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 45 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

--
Uladzislau Rezki

maintainer
#21 lore raw
Quoted 61 lines · Uladzislau Rezki
Quoted 56 lines · Joel Fernandes
Quoted 55 lines
Quoted 3 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Quoted 8 lines
Quoted 4 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Thanks.

Quoted 3 lines

--
Uladzislau Rezki

maintainer
#22 lore raw
Quoted 63 lines · Joel Fernandes
Quoted 63 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 8 lines
Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.

How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.

Note i do not say that it should not be scaled.

--
Uladzislau Rezki

maintainer
#23 lore raw
Quoted 77 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 8 lines
Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Quoted 7 lines

Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.

How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.

Note i do not say that it should not be scaled.

But you did not test that on large system with 1000s of CPUs right?

So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).

Let me know if I missed something. Thanks.

Quoted 3 lines

--
Uladzislau Rezki

maintainer
#24 lore raw
Quoted 80 lines · Joel Fernandes
Quoted 80 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 8 lines
Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.

Quoted 7 lines
Quoted 4 lines

How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.

Note i do not say that it should not be scaled.

But you did not test that on large system with 1000s of CPUs right?

No, no. I do not have access to such systems.

Quoted 5 lines

So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).

You mean introduce threshold and count how many nodes are in queue? To me it sounds not optimal and looks like a temporary solution.

Long term wise, it is better to split it, i mean to scale.

Do you know who can test it on ~1000 CPUs system? So we have some figures.

What i have is 256 CPUs system i can test on.

--
Uladzislau Rezki

maintainer
#25 lore raw
Quoted 88 lines · Uladzislau Rezki
Quoted 80 lines · Joel Fernandes
Quoted 80 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 8 lines
Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.

Quoted 7 lines
Quoted 4 lines

How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.

Note i do not say that it should not be scaled.

But you did not test that on large system with 1000s of CPUs right?

Quoted 8 lines

No, no. I do not have access to such systems.

Quoted 5 lines

So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).

You mean introduce threshold and count how many nodes are in queue?

Yes.

Quoted 1 line

To me it sounds not optimal and looks like a temporary solution.

Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.

Quoted 2 lines

Long term wise, it is better to split it, i mean to scale.

But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.

Quoted 2 lines

Do you know who can test it on ~1000 CPUs system? So we have some figures.

I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.

Quoted 2 lines

What i have is 256 CPUs system i can test on.

Same boat. ;-)

thanks,

- Joel

#26 lore raw

Hi.

Quoted 46 lines · Joel Fernandes
Quoted 24 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.

Quoted 7 lines
Quoted 4 lines

How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.

Note i do not say that it should not be scaled.

But you did not test that on large system with 1000s of CPUs right?

Quoted 8 lines

No, no. I do not have access to such systems.

Quoted 5 lines

So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).

You mean introduce threshold and count how many nodes are in queue?

Yes.

Quoted 1 line

To me it sounds not optimal and looks like a temporary solution.

Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.

Quoted 2 lines

Long term wise, it is better to split it, i mean to scale.

But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

Do you know who can test it on ~1000 CPUs system? So we have some figures.

I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.

Do you have a patch to try out? We can test it on these systems.

Note: Might take a while to test it, as those systems are bit tricky to get.

Quoted 8 lines
Quoted 2 lines

What i have is 256 CPUs system i can test on.

Same boat. ;-)

thanks,

- Joel

maintainer
#27 lore raw
Quoted 56 lines · Shrikanth Hegde

Hi.

Quoted 46 lines · Joel Fernandes
Quoted 24 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.

Quoted 7 lines
Quoted 4 lines

How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.

Note i do not say that it should not be scaled.

But you did not test that on large system with 1000s of CPUs right?

Quoted 8 lines

No, no. I do not have access to such systems.

Quoted 5 lines

So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).

You mean introduce threshold and count how many nodes are in queue?

Yes.

Quoted 1 line

To me it sounds not optimal and looks like a temporary solution.

Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.

Quoted 2 lines

Long term wise, it is better to split it, i mean to scale.

But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

Do you know who can test it on ~1000 CPUs system? So we have some figures.

I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.

Quoted 6 lines

Do you have a patch to try out? We can test it on these systems.

Note: Might take a while to test it, as those systems are bit tricky to get.

Let me prepare something. I will come back.

--
Uladzislau Rezki

maintainer
#28 lore raw
In reply to Joel Fernandes
Quoted 98 lines · Joel Fernandes
Quoted 95 lines · Uladzislau Rezki
Quoted 80 lines · Joel Fernandes
Quoted 80 lines · Uladzislau Rezki
Quoted 63 lines · Joel Fernandes
Quoted 63 lines
Quoted 3 lines · Uladzislau Rezki
Quoted 58 lines · Joel Fernandes
Quoted 47 lines
Quoted 3 lines · Uladzislau Rezki

Hello, Shrikanth!

Quoted 41 lines
Quoted 24 lines · Uladzislau Rezki
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 8 lines
Quoted 4 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp to speedup regular synchronize_rcu() call. But i am not saying that it would beat your "expedited switch" improvement.

Hi Uladzislau.

Had a discussion on this at LPC, having in kernel solution is likely better than having it in userspace.

- Having it in kernel would make it work across all archs. Why should any user wait when one initiates the hotplug.

Quoted 8 lines
Quoted 7 lines

- userspace tools are spread across such as chcpu, ppc64_cpu etc. though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". We will have to repeat the same in each tool.

- There is already /sys/kernel/rcu_expedited which is better if at all we need to fallback to userspace.

Sounds good to me. I agree it is better to bypass parameters.

Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now.

IMO, we can increase that threshold. 512/1024 is not a problem at all. But as Paul mentioned, we should consider scalability enhancement. From the other hand it is also probably worth to get into the state when we really see them :)

Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-)

Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call.

Quoted 7 lines
Quoted 4 lines

How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K.

Note i do not say that it should not be scaled.

But you did not test that on large system with 1000s of CPUs right?

No, no. I do not have access to such systems.

Quoted 5 lines

So the options I see are: either default to always using the optimization, not just for less than 17 CPUs (what you are saying above). Or, do what I said above (safer for system with 1000s of CPUs and less risky).

Quoted 8 lines
Quoted 1 line

You mean introduce threshold and count how many nodes are in queue?

Yes.

Quoted 1 line

To me it sounds not optimal and looks like a temporary solution.

Not more sub-optimal than the existing 16 CPU hard-coded solution I suppose.

It was trial testing :) Agree we should do something with it.

Quoted 6 lines
Quoted 2 lines

Long term wise, it is better to split it, i mean to scale.

But the scalable solution is already there: the !synchronize_rcu_normal path, right? And splitting the list won't help this use case anyway.

Fair point.

Quoted 11 lines
Quoted 2 lines

Do you know who can test it on ~1000 CPUs system? So we have some figures.

I don't have such systems either. The most I can go is ~200+ CPUs. Perhaps the folks on this thread have such systems as they mentioned 1900+ CPU systems. They should be happy to test.

Quoted 2 lines

What i have is 256 CPUs system i can test on.

Same boat. ;-)

:)

--
Uladzislau Rezki

#29 lore raw
In reply to Vishal Chourasia
Quoted 14 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Quoted 9 lines
Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

Hi Vishal,

I tried on tip/master at 315f416d3e26.
It fails to apply. is rcu tree updated?

Quoted 13 lines

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

---
  include/linux/rcupdate.h | 3 +++
  kernel/cpu.c             | 2 ++
  2 files changed, 5 insertions(+)
include/linux/rcupdate.h +0 -0
index c5b30054cd01..03c06cfb2b6d 100644

include/linux/rcupdate.h rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f) · line 1197

Quoted 6 lines
include/linux/rcupdate.h +3 -0
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
  extern int rcu_expedited;
  extern int rcu_normal;
 
+extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+

Why extern is needed? All it needs is declarations no?

Quoted 12 lines
include/linux/rcupdate.h +0 -0
  DEFINE_LOCK_GUARD_0(rcu,
  	do {
  		rcu_read_lock();
kernel/cpu.c +1 -0
index 8df2d773fe3b..6b0d491d73f4 100644
@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);
 
  void cpus_write_lock(void)
  {
+	rcu_expedite_gp();

kernel/cpu.c EXPORT_SYMBOL_GPL(cpus_read_unlock); · line 519

Quoted 10 lines
kernel/cpu.c +1 -0
  	percpu_down_write(&cpu_hotplug_lock);
  }
 
  void cpus_write_unlock(void)
  {
  	percpu_up_write(&cpu_hotplug_lock);
+	rcu_unexpedite_gp();
  }
 
  void lockdep_assert_cpus_held(void)

Have you tested kexec path or suspend/resume path? Seems like the counter can nest, but would be good to verify.

#30 lore raw
Quoted 24 lines · Shrikanth Hegde
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)
Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)
Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

Quoted 7 lines
Quoted 2 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

Hi Vishal,

I tried on tip/master at 315f416d3e26.
It fails to apply. is rcu tree updated?

I'm currently working off the GitHub mirror (|github.com/torvalds/linux)|

Quoted 10 lines
Quoted 10 lines

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

---
  include/linux/rcupdate.h | 3 +++
  kernel/cpu.c             | 2 ++

  2 files changed, 5 insertions(+)
Quoted 13 lines

include/linux/rcupdate.h rcu_head_after_call_rcu(struct rcu_head *rhp, · line 1197

Quoted 11 lines
include/linux/rcupdate.h +2 -0
index c5b30054cd01..03c06cfb2b6d 100644
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, 
 rcu_callback_t f)
   extern int rcu_expedited;
   extern int rcu_normal;
   +extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+

Why extern is needed? All it needs is declarations no?

Already declared in kernel/rcu/rcu.h

kernel/cpu.c already includes linux/rcupdate.h, therefore added an extern.

Quoted 15 lines
Quoted 15 lines
include/linux/rcupdate.h +0 -0
   DEFINE_LOCK_GUARD_0(rcu,
       do {

          rcu_read_lock();

kernel/cpu.c +1 -0
index 8df2d773fe3b..6b0d491d73f4 100644
@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);
     void cpus_write_lock(void)
   {
+    rcu_expedite_gp();
       percpu_down_write(&cpu_hotplug_lock);
   }
Quoted 8 lines

kernel/cpu.c EXPORT_SYMBOL_GPL(cpus_read_unlock); · line 519

Quoted 6 lines
kernel/cpu.c +1 -0
     void cpus_write_unlock(void)
   {
       percpu_up_write(&cpu_hotplug_lock);
+    rcu_unexpedite_gp();
   }
     void lockdep_assert_cpus_held(void)

Have you tested kexec path or suspend/resume path?

I did test kexec patch by booting into another kernel via kexec path. But, I didn't test suspend/resume.

Quoted 1 line

Seems like the counter can nest, but would be good to verify.

maintainer
#31 lore raw
In reply to Vishal Chourasia
Quoted 11 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Quoted 3 lines

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

When does the user initiate this in your system?

Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.

Quoted 10 lines

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

This does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?

Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.

thanks,

- Joel

Quoted 47 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

---
include/linux/rcupdate.h | 3 +++
kernel/cpu.c | 2 ++

2 files changed, 5 insertions(+)
include/linux/rcupdate.h +3 -0
index c5b30054cd01..03c06cfb2b6d 100644
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
 extern int rcu_expedited;
 extern int rcu_normal;
 
+extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+
 DEFINE_LOCK_GUARD_0(rcu,
   do {
       rcu_read_lock();
kernel/cpu.c +2 -1
index 8df2d773fe3b..6b0d491d73f4 100644
@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);
 
 void cpus_write_lock(void)
 {
+    rcu_expedite_gp();
   percpu_down_write(&cpu_hotplug_lock);
 }
 
 void cpus_write_unlock(void)
 {
   percpu_up_write(&cpu_hotplug_lock);
+    rcu_unexpedite_gp();
 }
 
 void lockdep_assert_cpus_held(void)
--

2.52.0

maintainer
#32 lore raw
Quoted 31 lines · Joel Fernandes



Quoted 9 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Quoted 3 lines

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

When does the user initiate this in your system?

Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.

Quoted 7 lines

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement
Quoted 7 lines
Quoted 3 lines

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

This does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?

Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.

Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.

- Joel

Quoted 52 lines

thanks,

- Joel

Quoted 47 lines

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

---
include/linux/rcupdate.h | 3 +++
kernel/cpu.c | 2 ++

2 files changed, 5 insertions(+)
include/linux/rcupdate.h +3 -0
index c5b30054cd01..03c06cfb2b6d 100644
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
 extern int rcu_expedited;
 extern int rcu_normal;
 
+extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+
 DEFINE_LOCK_GUARD_0(rcu,
  do {
      rcu_read_lock();
kernel/cpu.c +2 -1
index 8df2d773fe3b..6b0d491d73f4 100644
@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);
 
 void cpus_write_lock(void)
 {
+    rcu_expedite_gp();
  percpu_down_write(&cpu_hotplug_lock);
 }
 
 void cpus_write_unlock(void)
 {
  percpu_up_write(&cpu_hotplug_lock);
+    rcu_unexpedite_gp();
 }
 
 void lockdep_assert_cpus_held(void)
--

2.52.0

maintainer
#33 lore raw
Quoted 37 lines · Joel Fernandes
Quoted 37 lines · Joel Fernandes



Quoted 9 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Quoted 3 lines

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

When does the user initiate this in your system?

Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.

Quoted 10 lines

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

This does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?

Quoted 7 lines
Quoted 1 line

Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.

Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.

Because hotplug is not a fast path; there is no expectation of performance here.

maintainer
#34 lore raw
Quoted 44 lines · Peter Zijlstra
Quoted 39 lines · Joel Fernandes
Quoted 38 lines
Quoted 3 lines · Joel Fernandes



Quoted 9 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Quoted 3 lines

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

When does the user initiate this in your system?

Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.

Quoted 10 lines

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

This does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?

Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.

Quoted 8 lines
Quoted 5 lines

Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.

Because hotplug is not a fast path; there is no expectation of performance here.

Agreed, I was just wondering if it was incredibly slow or something. Looking forward to more justification from Vishal on usecase,

- Joel

Quoted 1 line
#35 lore raw

Hello Joel, Peter

Quoted 20 lines · Joel Fernandes
Quoted 20 lines · Peter Zijlstra
Quoted 15 lines · Joel Fernandes
Quoted 15 lines
Quoted 3 lines · Joel Fernandes



Quoted 9 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

When does the user initiate this in your system?

Workloads exhibit varying sensitivity to SMT levels. Users dynamically adjust SMT modes to optimize performance.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Hotplug should not be happening that often to begin with, it is a slow path that depends on the disruptive stop-machine mechanism.

Yes, it doesn't happen too often, but when it does, on machines with (>= 1920 CPUs) it takes more than 20 mins to finish.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

This does look compelling but, Could you provide more information about how this was tested - what does the ppc binary do (how many hot plugs , how does the performance change with cycle count etc)?

The ppc64_cpu utility generates a list of target CPUs based on the requested SMT state and writes to their corresponding sysfs online entries.

Sorry, I didn't get your second question about the performance change with cycle count.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Can you also run rcutorture testing? Some of the scenarios like TREE03 stress hotplug.

Sure, I will get back with the numbers.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Also, why not just use the expedite api at the callsite that is slow than blanket expediting everything between hotplug lock and unlock. That is more specific fix than this fix which applies more broadly to all operations. It appears the report you provided does provide the culprit callsite.

I initially attempted to replace synchronize_rcu() with synchronize_rcu_expedited() at specific callsites. However, the primary bottlenecks are within percpu_down_write(), called via _cpu_up() and try_online_node(). Please refer to the callstack shared below. Since percpu_down_write() is used throughout the kernel, modifying it directly would force expedited grace periods on unrelated subsystems.

@[

    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    _cpu_up+140
    cpu_up+440
    cpu_subsys_online+128
    device_online+176
    online_store+220
    dev_attr_store+52
    sysfs_kf_write+120
    kernfs_fop_write_iter+456
    vfs_write+952
    ksys_write+132
    system_call_exception+292
    system_call_vectored_common+348
]: 350
@[
    synchronize_rcu+12
    rcu_sync_enter+260
    percpu_down_write+76
    try_online_node+64
    cpu_up+120
    cpu_subsys_online+128
    device_online+176
    online_store+220
    dev_attr_store+52
    sysfs_kf_write+120
    kernfs_fop_write_iter+456
    vfs_write+952
    ksys_write+132
    system_call_exception+292
    system_call_vectored_common+348

]: 350

Quoted 3 lines
Quoted 3 lines

Because hotplug is not a fast path; there is no expectation of performance here.

True.

Quoted 7 lines

Agreed, I was just wondering if it was incredibly slow or something. Looking forward to more justification from Vishal on usecase,

- Joel

Quoted 1 line

- vishalc

maintainer
#36 lore raw
In reply to Vishal Chourasia
Quoted 13 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

Quoted 8 lines

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

But who cares? Its not like you'd *ever* do this, right?

#37 lore raw

Hello Peter,

Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement
Quoted 6 lines
Quoted 4 lines

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

But who cares? Its not like you'd *ever* do this, right?

Users dynamically adjust SMT modes to optimize performance of the workload being run. And, yes it doesn't happen too often, but when it does, on machines with (>= 1920 CPUs) it takes more than 20 mins to finish.

- vishal

maintainer
#38 lore raw
Quoted 28 lines · Vishal Chourasia

Hello Peter,

Quoted 23 lines · Peter Zijlstra
Quoted 21 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement
Quoted 5 lines
Quoted 1 line

But who cares? Its not like you'd *ever* do this, right?

Users dynamically adjust SMT modes to optimize performance of the workload being run. And, yes it doesn't happen too often, but when it does, on machines with (>= 1920 CPUs) it takes more than 20 mins to finish.

Users cannot change this, it is root only.

Having to change SMT mode per workload seems quite insane; but whatever.

If you do have to put RCU hooks anywhere; I'd much rather see them in cpuhp_smt_{en,dis}able(), such that they only affect the batch hotplug case, rather than everything using cpus_write_lock().

Also note that there is a case to be made to optimize this batch hotplug case; for one it makes no sense to take cpus_write_lock() over and over and over again; if you can pull that out, just like it already lifted cpu_maps_update_begin(), this would help.

And Joel has a point, in that it might make sense for RCU to behave 'better' under these conditions.

#39 lore raw

PATCH 1/1 cpuhp: Expedite synchronize_rcu during SMT switch

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

SMT switch is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 2048 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 30m53.194s
After:  real 6m4.678s  # ~5x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 49m5.920s
After:  real 36m47.798s  # ~1.3x improvement

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

---
 include/linux/rcupdate.h | 3 +++
 kernel/cpu.c             | 4 ++++
 2 files changed, 7 insertions(+)
include/linux/rcupdate.h +3 -0
index c5b30054cd01..03c06cfb2b6d 100644
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
 extern int rcu_expedited;
 extern int rcu_normal;
 
+extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+
 DEFINE_LOCK_GUARD_0(rcu,
 	do {
 		rcu_read_lock();
kernel/cpu.c +4 -0
index 8df2d773fe3b..a264d7170842 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	int cpu, ret = 0;
 
 	cpu_maps_update_begin();
+	rcu_expedite_gp();
 	for_each_online_cpu(cpu) {
 		if (topology_is_primary_thread(cpu))
 			continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	}
 	if (!ret)
 		cpu_smt_control = ctrlval;
+	rcu_unexpedite_gp();
 	cpu_maps_update_done();
 	return ret;
 }
@@ -2716,6 +2718,7 @@ int cpuhp_smt_enable(void)
 
 	cpu_maps_update_begin();
 	cpu_smt_control = CPU_SMT_ENABLED;
+	rcu_expedite_gp();
 	for_each_present_cpu(cpu) {
 		/* Skip online CPUs and CPUs on offline nodes */
 		if (cpu_online(cpu) || !node_online(cpu_to_node(cpu)))
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
 		/* See comment in cpuhp_smt_disable() */
 		cpuhp_online_cpu_device(cpu);
 	}
+	rcu_unexpedite_gp();
 	cpu_maps_update_done();
 	return ret;
 }
Signature · 1 line
2.52.0
maintainer
#40 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

Quoted 10 lines · Vishal Chourasia

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

You seem to have left out all the useful bits from your changelog again :/

Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.

Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.

kernel/cpu.c +4 -0
index 8df2d773fe3b..1365c19444b2 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	int cpu, ret = 0;
 
 	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
 	for_each_online_cpu(cpu) {
 		if (topology_is_primary_thread(cpu))
 			continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	}
 	if (!ret)
 		cpu_smt_control = ctrlval;
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
 	cpu_maps_update_done();
 	return ret;
 }
@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)
 	int cpu, ret = 0;
 
 	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
 	cpu_smt_control = CPU_SMT_ENABLED;
 	for_each_present_cpu(cpu) {
 		/* Skip online CPUs and CPUs on offline nodes */
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
 		/* See comment in cpuhp_smt_disable() */
 		cpuhp_online_cpu_device(cpu);
 	}
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
 	cpu_maps_update_done();
 	return ret;
 }
#41 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

Hi Peter.

Quoted 50 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

You seem to have left out all the useful bits from your changelog again :/

Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.

Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.

kernel/cpu.c +3 -0
index 8df2d773fe3b..1365c19444b2 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
  	int cpu, ret = 0;
 
  	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
  	for_each_online_cpu(cpu) {
  		if (topology_is_primary_thread(cpu))
  			continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
  	}
  	if (!ret)
  		cpu_smt_control = ctrlval;
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
  	cpu_maps_update_done();
  	return ret;
  }
@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)
  	int cpu, ret = 0;
 
  	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
  	cpu_smt_control = CPU_SMT_ENABLED;
  	for_each_present_cpu(cpu) {
  		/* Skip online CPUs and CPUs on offline nodes */

kernel/cpu.c int cpuhp_smt_enable(void) · line 2737

Quoted 7 lines
kernel/cpu.c +1 -0
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
  		/* See comment in cpuhp_smt_disable() */
  		cpuhp_online_cpu_device(cpu);
  	}
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
  	cpu_maps_update_done();
  	return ret;
  }

Currently, cpuhp_smt_[enable/disable] calls _cpu_up/_cpu_down which does the same in cpus_write_lock/unlock. though it is per cpu enable/disable one after another.

How hoisting this up will help?

maintainer
#42 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

Quoted 64 lines · Shrikanth Hegde

Hi Peter.

Quoted 48 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

You seem to have left out all the useful bits from your changelog again :/

Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.

Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.

kernel/cpu.c +3 -0
index 8df2d773fe3b..1365c19444b2 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
  	int cpu, ret = 0;
  	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
  	for_each_online_cpu(cpu) {
  		if (topology_is_primary_thread(cpu))
  			continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
  	}
  	if (!ret)
  		cpu_smt_control = ctrlval;
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
  	cpu_maps_update_done();
  	return ret;
  }
@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)
  	int cpu, ret = 0;
  	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
  	cpu_smt_control = CPU_SMT_ENABLED;
  	for_each_present_cpu(cpu) {
  		/* Skip online CPUs and CPUs on offline nodes */

kernel/cpu.c int cpuhp_smt_enable(void) · line 2737

Quoted 7 lines
kernel/cpu.c +1 -0
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
  		/* See comment in cpuhp_smt_disable() */
  		cpuhp_online_cpu_device(cpu);
  	}
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
  	cpu_maps_update_done();
  	return ret;
  }

Currently, cpuhp_smt_[enable/disable] calls _cpu_up/_cpu_down which does the same in cpus_write_lock/unlock. though it is per cpu enable/disable one after another.

Quoted 1 line

How hoisting this up will help?

By holding an extra rcu_sync reference, the percpu rwsem is kept into the the slow path, avoiding the rcu-sync on down_write(), which was very prevalent per this:

https://lkml.kernel.org/r/[email protected]

maintainer
#43 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

Quoted 5 lines · Peter Zijlstra

By holding an extra rcu_sync reference, the percpu rwsem is kept into the the slow path, avoiding the rcu-sync on down_write(), which was very prevalent per this:

https://lkml.kernel.org/r/[email protected]

Makes sense, though I wonder if we should have a separate percpu-rwsem API for this rather than directly accessing the lock-internal rcu_sync state? Other future percpu_rw_semaphore users may benefit as well.

- Joel

maintainer
#44 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

Quoted 10 lines · Joel Fernandes
Quoted 5 lines · Peter Zijlstra

By holding an extra rcu_sync reference, the percpu rwsem is kept into the the slow path, avoiding the rcu-sync on down_write(), which was very prevalent per this:

https://lkml.kernel.org/r/[email protected]

Makes sense, though I wonder if we should have a separate percpu-rwsem API for this rather than directly accessing the lock-internal rcu_sync state? Other future percpu_rw_semaphore users may benefit as well.

Yeah, perhaps. There is one other user, the above makes two.

kernel/cgroup/cgroup.c: rcu_sync_enter(&cgroup_threadgroup_rwsem.rss);

#45 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

In reply to Peter Zijlstra
Quoted 49 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

You seem to have left out all the useful bits from your changelog again :/

Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.

Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.

kernel/cpu.c +3 -0
index 8df2d773fe3b..1365c19444b2 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
  	int cpu, ret = 0;
 
  	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
  	for_each_online_cpu(cpu) {
  		if (topology_is_primary_thread(cpu))
  			continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
  	}
  	if (!ret)
  		cpu_smt_control = ctrlval;
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
  	cpu_maps_update_done();
  	return ret;
  }
@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)
  	int cpu, ret = 0;
 
  	cpu_maps_update_begin();
+	rcu_sync_enter(&cpu_hotplug_lock.rss);
  	cpu_smt_control = CPU_SMT_ENABLED;
  	for_each_present_cpu(cpu) {
  		/* Skip online CPUs and CPUs on offline nodes */

kernel/cpu.c int cpuhp_smt_enable(void) · line 2737

Quoted 7 lines
kernel/cpu.c +1 -0
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
  		/* See comment in cpuhp_smt_disable() */
  		cpuhp_online_cpu_device(cpu);
  	}
+	rcu_sync_exit(&cpu_hotplug_lock.rss);
  	cpu_maps_update_done();
  	return ret;
  }

Hi,

I verified this patch using the configuration described below. Configuration:
    •    Kernel version: 6.19.0-rc6
    •    Number of CPUs: 1536

Earlier verification of an older version of this patch was performed on a system with *2048 CPUs*. Due to system unavailability, the current verification was carried out on a *different system.*

Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.

SMT=ON  -> sys 50m42.805s
SMT=OFF -> sys 0m0.064s

SMT Mode    | Without Patch    | With Patch   | % Improvement   |
------------------------------------------------------------------

SMT=off     | 20m 32.210s      |  5m 30.898s  | +73.15%         |
SMT=on      | 62m 46.549s      | 55m 45.671s  | +11.18%         |

Please add below tag: 
Tested-by: Samir M <[email protected]>

Regards,
Samir

#46 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

Quoted 79 lines · Samir M
Quoted 49 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

You seem to have left out all the useful bits from your changelog again :/

Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.

Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.

kernel/cpu.c +2 -0
index 8df2d773fe3b..1365c19444b2 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control 
 ctrlval)
       int cpu, ret = 0;
         cpu_maps_update_begin();
+    rcu_sync_enter(&cpu_hotplug_lock.rss);
       for_each_online_cpu(cpu) {
           if (topology_is_primary_thread(cpu))
               continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control 
 ctrlval)
       }
       if (!ret)
           cpu_smt_control = ctrlval;
+    rcu_sync_exit(&cpu_hotplug_lock.rss);
       cpu_maps_update_done();
       return ret;

  }

kernel/cpu.c +1 -0
@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)
       int cpu, ret = 0;
         cpu_maps_update_begin();
+    rcu_sync_enter(&cpu_hotplug_lock.rss);
       cpu_smt_control = CPU_SMT_ENABLED;
       for_each_present_cpu(cpu) {
           /* Skip online CPUs and CPUs on offline nodes */

kernel/cpu.c int cpuhp_smt_enable(void) · line 2737

Quoted 7 lines
kernel/cpu.c +1 -0
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
           /* See comment in cpuhp_smt_disable() */
           cpuhp_online_cpu_device(cpu);
       }
+    rcu_sync_exit(&cpu_hotplug_lock.rss);
       cpu_maps_update_done();
       return ret;
   }

Hi,

I verified this patch using the configuration described below. Configuration:
    •    Kernel version: 6.19.0-rc6
    •    Number of CPUs: 1536

Earlier verification of an older version of this patch was performed on a system with *2048 CPUs*. Due to system unavailability, the current verification was carried out on a *different system.*

Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.

SMT=ON  -> sys 50m42.805s
SMT=OFF -> sys 0m0.064s

SMT Mode    | Without Patch    | With Patch   | % Improvement   |
------------------------------------------------------------------

Quoted 9 lines
SMT=off     | 20m 32.210s      |  5m 30.898s  | +73.15%         |
SMT=on      | 62m 46.549s      | 55m 45.671s  | +11.18%         |

Please add below tag: 
Tested-by: Samir M <[email protected]>

Regards,
Samir

Hi All,

For reference, I am updating the results for the Vishal and Rezki patches.

Configuration:
Kernel version: 6.19.0-rc6
Number of CPUs: 1536

Earlier verification of an older revision of this patch was performed on a system with 2048 CPUs*.* Due to system unavailability, the current verification was carried out on a different**system. For reference, I am updating the results corresponding to the patches verified using the above configuration.

Patch: https://lore.kernel.org/all/[email protected]/
SMT Mode    | Without Patch    | With Patch  | % Improvement   |
------------------------------------------------------------------

SMT=off     | 20m 32.210s      |  5m 31.807s | +73.09%        |
SMT=on      | 62m 46.549s      | 55m 48.801s  | +11.08%        |

SMT=ON  -> sys 50m44.105s
SMT=OFF -> sys 0m0.035s

patch: https://lore.kernel.org/all/[email protected]/
SMT Mode    | Without Patch    | With Patch  | % Improvement   |
------------------------------------------------------------------

SMT=off     | 20m 32.210s      | 18m 2.012s  | +12.19%         |
SMT=on      | 62m 46.549s      | 62m 35.076s | +0.30%         |

SMT=ON  -> sys 50m43.806s
SMT=OFF -> sys 0m0.109s

Regards,
Samir

#47 lore raw

Re: [PATCH] cpuhp: Expedite synchronize_rcu during SMT switch

In reply to Samir M
Quoted 79 lines · Samir M
Quoted 49 lines · Peter Zijlstra
Quoted 10 lines · Vishal Chourasia

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

You seem to have left out all the useful bits from your changelog again :/

Anyway, ISTR Joel posted a patch hoisting a lock; it was a icky, but not something we can't live with either.

Also, memory got jogged and I think something like the below will remove 2/3 of your rcu woes as well.

kernel/cpu.c +2 -0
index 8df2d773fe3b..1365c19444b2 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control 
 ctrlval)
       int cpu, ret = 0;
         cpu_maps_update_begin();
+    rcu_sync_enter(&cpu_hotplug_lock.rss);
       for_each_online_cpu(cpu) {
           if (topology_is_primary_thread(cpu))
               continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control 
 ctrlval)
       }
       if (!ret)
           cpu_smt_control = ctrlval;
+    rcu_sync_exit(&cpu_hotplug_lock.rss);
       cpu_maps_update_done();
       return ret;

  }

kernel/cpu.c +1 -0
@@ -2715,6 +2717,7 @@ int cpuhp_smt_enable(void)
       int cpu, ret = 0;
         cpu_maps_update_begin();
+    rcu_sync_enter(&cpu_hotplug_lock.rss);
       cpu_smt_control = CPU_SMT_ENABLED;
       for_each_present_cpu(cpu) {
           /* Skip online CPUs and CPUs on offline nodes */

kernel/cpu.c int cpuhp_smt_enable(void) · line 2737

Quoted 7 lines
kernel/cpu.c +1 -0
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
           /* See comment in cpuhp_smt_disable() */
           cpuhp_online_cpu_device(cpu);
       }
+    rcu_sync_exit(&cpu_hotplug_lock.rss);
       cpu_maps_update_done();
       return ret;
   }

Hi,

I verified this patch using the configuration described below. Configuration:
    •    Kernel version: 6.19.0-rc6
    •    Number of CPUs: 1536

Earlier verification of an older version of this patch was performed on a system with *2048 CPUs*. Due to system unavailability, the current verification was carried out on a *different system.*

Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.

SMT=ON  -> sys 50m42.805s
SMT=OFF -> sys 0m0.064s

SMT Mode    | Without Patch    | With Patch   | % Improvement   |
------------------------------------------------------------------

Quoted 9 lines
SMT=off     | 20m 32.210s      |  5m 30.898s  | +73.15%         |
SMT=on      | 62m 46.549s      | 55m 45.671s  | +11.18%         |

Please add below tag: 
Tested-by: Samir M <[email protected]>

Regards,
Samir

Hi All,

Apologies for the confusion in the previous report.
In the previous report, I used the b4 am command to apply the patch. As a result, all patches in the mail thread were applied, rather than only the intended one. Consequently, the results that were posted earlier included changes from multiple patches, and therefore cannot be considered valid for evaluating this patch in isolation. We have since tested Peter’s patch separately, applied in isolation. Based on this testing, we did not observe any improvement in the SMT timing details.

Configuration:
    •    Kernel version: 6.19.0-rc6
    •    Number of CPUs: 1536

SMT Mode    | Without Patch    | With Patch  | % Improvement  |
------------------------------------------------------------------

SMT=off     | 20m 32.210s      |  20m22.441s  | +0.79%         |
SMT=on      | 62m 46.549s      |  63m0.532s  | -0.37%         |

Regards,
Samir

#48 lore raw

PATCH RESEND 1/1 cpuhp: Expedite synchronize_rcu during SMT switch

In reply to Peter Zijlstra

[Edits] Added Suggested-by tag

Expedite synchronize_rcu() during the cpuhp_smt_[enable|disable] path to accelerate the operation.

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs to hotplug during SMT switch grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

SMT switch is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 2048 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 30m53.194s
After:  real 6m4.678s  # ~5x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 49m5.920s
After:  real 36m47.798s  # ~1.3x improvement

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com
[2] https://lore.kernel.org/all/[email protected]/

---
 include/linux/rcupdate.h | 3 +++
 kernel/cpu.c             | 4 ++++
 2 files changed, 7 insertions(+)
include/linux/rcupdate.h +3 -0
index c5b30054cd01..03c06cfb2b6d 100644
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
 extern int rcu_expedited;
 extern int rcu_normal;
 
+extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+
 DEFINE_LOCK_GUARD_0(rcu,
 	do {
 		rcu_read_lock();
kernel/cpu.c +4 -0
index 8df2d773fe3b..a264d7170842 100644
@@ -2669,6 +2669,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	int cpu, ret = 0;
 
 	cpu_maps_update_begin();
+	rcu_expedite_gp();
 	for_each_online_cpu(cpu) {
 		if (topology_is_primary_thread(cpu))
 			continue;
@@ -2698,6 +2699,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	}
 	if (!ret)
 		cpu_smt_control = ctrlval;
+	rcu_unexpedite_gp();
 	cpu_maps_update_done();
 	return ret;
 }
@@ -2716,6 +2718,7 @@ int cpuhp_smt_enable(void)
 
 	cpu_maps_update_begin();
 	cpu_smt_control = CPU_SMT_ENABLED;
+	rcu_expedite_gp();
 	for_each_present_cpu(cpu) {
 		/* Skip online CPUs and CPUs on offline nodes */
 		if (cpu_online(cpu) || !node_online(cpu_to_node(cpu)))
@@ -2728,6 +2731,7 @@ int cpuhp_smt_enable(void)
 		/* See comment in cpuhp_smt_disable() */
 		cpuhp_online_cpu_device(cpu);
 	}
+	rcu_unexpedite_gp();
 	cpu_maps_update_done();
 	return ret;
 }
Signature · 1 line
2.52.0
#49 lore raw
In reply to Vishal Chourasia
Quoted 54 lines · Vishal Chourasia

Bulk CPU hotplug operations—such as switching SMT modes across all cores—require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

[1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com

---
  include/linux/rcupdate.h | 3 +++
  kernel/cpu.c             | 2 ++
  2 files changed, 5 insertions(+)
include/linux/rcupdate.h +3 -0
index c5b30054cd01..03c06cfb2b6d 100644
@@ -1192,6 +1192,9 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
  extern int rcu_expedited;
  extern int rcu_normal;
 
+extern void rcu_expedite_gp(void);
+extern void rcu_unexpedite_gp(void);
+
  DEFINE_LOCK_GUARD_0(rcu,
  	do {
  		rcu_read_lock();
kernel/cpu.c +1 -0
index 8df2d773fe3b..6b0d491d73f4 100644
@@ -506,12 +506,14 @@ EXPORT_SYMBOL_GPL(cpus_read_unlock);
 
  void cpus_write_lock(void)
  {
+	rcu_expedite_gp();

kernel/cpu.c EXPORT_SYMBOL_GPL(cpus_read_unlock); · line 519

Quoted 10 lines
kernel/cpu.c +1 -0
  	percpu_down_write(&cpu_hotplug_lock);
  }
 
  void cpus_write_unlock(void)
  {
  	percpu_up_write(&cpu_hotplug_lock);
+	rcu_unexpedite_gp();
  }
 
  void lockdep_assert_cpus_held(void)

Hi Vishal,

I verified this patch using the configuration described below. Configuration:
    •    Kernel version: 6.19.0-rc5
    •    Number of CPUs: 2048

Using this setup, I evaluated the patch with both SMT enabled and SMT disabled. patch shows a significant improvement in the SMT=off case and a measurable improvement in the SMT=on case. The results indicate that when SMT is enabled, the system time is noticeably higher. In contrast, with SMT disabled, no significant increase in system time is observed.

SMT=ON  -> sys 31m18.849s
SMT=OFF -> sys 0m0.087s

SMT Mode    | Without Patch    | With Patch | % Improvement    |
------------------------------------------------------------------
SMT=off     | 30m 53.194s      |  6m 4.250s  | +80.40%          | SMT=on      | 49m 5.920s       | 36m 50.386s | +25.01%          |

Please add below tag: 
Tested-by: Samir M <[email protected]>

Regards,
Samir

maintainer
#50 lore raw
Quoted 19 lines · Samir M
Quoted 10 lines · Vishal Chourasia

Bulk CPU hotplug operations--such as switching SMT modes across all cores--require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Hi Vishal,

I verified this patch using the configuration described below. Configuration:
* Kernel version: 6.19.0-rc5
* Number of CPUs: 2048

Quoted 4 lines

SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------

SMT=off     | 30m 53.194s      |  6m 4.250s  | +80.40%          |
SMT=on      | 49m 5.920s       | 36m 50.386s | +25.01%          |

Hi Vishal, Samir,

Thanks for the testing on your large CPU count system.

Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.

git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
tag: cpuhp-bulk-optimize-rfc-v1

I tested it lightly on rcutorture hotplug test and it passes. Please share any performance results, thanks.

Also I'd like to use expediting of RCU as a last resort TBH, we should optimize the outer operations that require RCU in the first place such as Peter's suggestion since that will improve the overall efficiency of the code. And if/when expediting RCU, Peter's other suggestion to not do it in cpus_write_lock() and instead do it from cpuhp_smt_enable() also makes sense to me.

---8<-----------------------

From: Joel Fernandes <[email protected]>
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiring

Bulk CPU hotplug operations such as enabling SMT across all cores require hotplugging multiple CPUs. The current implementation takes cpus_write_lock() for each individual CPU causing multiple slow grace period requests.

Therefore introduce cpu_up_locked() that assumes the caller already holds cpus_write_lock(). The cpuhp_smt_enable() function is updated to hold the lock once around the entire loop rather than for each CPU.

---
 kernel/cpu.c | 40 +++++++++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 15 deletions(-)
kernel/cpu.c +25 -15
index 8df2d773fe3b..4ce7deb236d7 100644
@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state)
 	complete_ap_thread(st, true);
 }
 
-/* Requires cpu_add_remove_lock to be held */
-static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
+/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */
+static int cpu_up_locked(unsigned int cpu, int tasks_frozen,
+			 enum cpuhp_state target)
 {
 	struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu);
 	struct task_struct *idle;
 	int ret = 0;
 
-	cpus_write_lock();
+	lockdep_assert_cpus_held();
 
-	if (!cpu_present(cpu)) {
-		ret = -EINVAL;
-		goto out;
-	}
+	if (!cpu_present(cpu))
+		return -EINVAL;
 
 	/*
 	 * The caller of cpu_up() might have raced with another
 	 * caller. Nothing to do.
 	 */
 	if (st->state >= target)
-		goto out;
+		return 0;
 
 	if (st->state == CPUHP_OFFLINE) {
 		/* Let it fail before we try to bring the cpu up */
 		idle = idle_thread_get(cpu);
-		if (IS_ERR(idle)) {
-			ret = PTR_ERR(idle);
-			goto out;
-		}
+		if (IS_ERR(idle))
+			return PTR_ERR(idle);
 
 		/*
 		 * Reset stale stack state from the last time this CPU was online.
@@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
 		 * return the error code..
 		 */
 		if (ret)
-			goto out;
+			return ret;
 	}
 
 	/*
@@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
 	 */
 	target = min((int)target, CPUHP_BRINGUP_CPU);
 	ret = cpuhp_up_callbacks(cpu, st, target);
-out:
+	return ret;
+}
+
+/* Requires cpu_add_remove_lock to be held */
+static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
+{
+	int ret;
+
+	cpus_write_lock();
+	ret = cpu_up_locked(cpu, tasks_frozen, target);
 	cpus_write_unlock();
 	arch_smt_update();
 	return ret;
@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void)
 	int cpu, ret = 0;
 
 	cpu_maps_update_begin();
+	/* Hold cpus_write_lock() for entire batch operation. */
+	cpus_write_lock();
 	cpu_smt_control = CPU_SMT_ENABLED;
 	for_each_present_cpu(cpu) {
 		/* Skip online CPUs and CPUs on offline nodes */
@@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void)
 			continue;
 		if (!cpu_smt_thread_allowed(cpu) || !topology_is_core_online(cpu))
 			continue;
-		ret = _cpu_up(cpu, 0, CPUHP_ONLINE);
+		ret = cpu_up_locked(cpu, 0, CPUHP_ONLINE);
 		if (ret)
 			break;
 		/* See comment in cpuhp_smt_disable() */
 		cpuhp_online_cpu_device(cpu);
 	}
+	cpus_write_unlock();
+	arch_smt_update();
 	cpu_maps_update_done();
 	return ret;
 }
Signature · 1 line
2.34.1
#51 lore raw
Quoted 171 lines · Joel Fernandes
Quoted 19 lines · Samir M
Quoted 10 lines · Vishal Chourasia

Bulk CPU hotplug operations--such as switching SMT modes across all cores--require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Hi Vishal,

I verified this patch using the configuration described below. Configuration:
     *    Kernel version: 6.19.0-rc5
     *    Number of CPUs: 2048

Quoted 4 lines

SMT Mode    | Without Patch    | With Patch | % Improvement    |
------------------------------------------------------------------

SMT=off     | 30m 53.194s      |  6m 4.250s  | +80.40%          |
SMT=on      | 49m 5.920s       | 36m 50.386s | +25.01%          |

Hi Vishal, Samir,

Thanks for the testing on your large CPU count system.

Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.

  git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
  tag: cpuhp-bulk-optimize-rfc-v1

I tested it lightly on rcutorture hotplug test and it passes. Please share any performance results, thanks.

Also I'd like to use expediting of RCU as a last resort TBH, we should optimize the outer operations that require RCU in the first place such as Peter's suggestion since that will improve the overall efficiency of the code. And if/when expediting RCU, Peter's other suggestion to not do it in cpus_write_lock() and instead do it from cpuhp_smt_enable() also makes sense to me.

---8<-----------------------

From: Joel Fernandes <[email protected]>
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiring

Bulk CPU hotplug operations such as enabling SMT across all cores require hotplugging multiple CPUs. The current implementation takes cpus_write_lock() for each individual CPU causing multiple slow grace period requests.

Therefore introduce cpu_up_locked() that assumes the caller already holds cpus_write_lock(). The cpuhp_smt_enable() function is updated to hold the lock once around the entire loop rather than for each CPU.

---
  kernel/cpu.c | 40 +++++++++++++++++++++++++---------------
  1 file changed, 25 insertions(+), 15 deletions(-)
kernel/cpu.c +9 -12
index 8df2d773fe3b..4ce7deb236d7 100644
@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state)
      complete_ap_thread(st, true);
  }
 
-/* Requires cpu_add_remove_lock to be held */
-static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state 
 target)
+/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */
+static int cpu_up_locked(unsigned int cpu, int tasks_frozen,
+             enum cpuhp_state target)
  {
      struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu);
      struct task_struct *idle;
      int ret = 0;
 
-    cpus_write_lock();
+    lockdep_assert_cpus_held();
 
-    if (!cpu_present(cpu)) {
-        ret = -EINVAL;
-        goto out;
-    }
+    if (!cpu_present(cpu))
+        return -EINVAL;
 
      /*
       * The caller of cpu_up() might have raced with another
       * caller. Nothing to do.
       */
      if (st->state >= target)
-        goto out;
+        return 0;
 
      if (st->state == CPUHP_OFFLINE) {
          /* Let it fail before we try to bring the cpu up */
          idle = idle_thread_get(cpu);
-        if (IS_ERR(idle)) {
-            ret = PTR_ERR(idle);
-            goto out;
-        }
+        if (IS_ERR(idle))
+            return PTR_ERR(idle);
 
          /*

          * Reset stale stack state from the last time this CPU was
online.

kernel/cpu.c +1 -1
@@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, int 
 tasks_frozen, enum cpuhp_state target)
           * return the error code..
           */
          if (ret)
-            goto out;
+            return ret;
      }
 

     /*

kernel/cpu.c +10 -1
@@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, int 
 tasks_frozen, enum cpuhp_state target)
       */
      target = min((int)target, CPUHP_BRINGUP_CPU);
      ret = cpuhp_up_callbacks(cpu, st, target);
-out:
+    return ret;
+}
+
+/* Requires cpu_add_remove_lock to be held */
+static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state 
 target)
+{
+    int ret;
+
+    cpus_write_lock();
+    ret = cpu_up_locked(cpu, tasks_frozen, target);
      cpus_write_unlock();

     arch_smt_update();
     return ret;

kernel/cpu.c +5 -1
@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void)
      int cpu, ret = 0;
 
      cpu_maps_update_begin();
+    /* Hold cpus_write_lock() for entire batch operation. */
+    cpus_write_lock();
      cpu_smt_control = CPU_SMT_ENABLED;
      for_each_present_cpu(cpu) {
          /* Skip online CPUs and CPUs on offline nodes */
@@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void)
              continue;
          if (!cpu_smt_thread_allowed(cpu) || ! 
 topology_is_core_online(cpu))
              continue;
-        ret = _cpu_up(cpu, 0, CPUHP_ONLINE);
+        ret = cpu_up_locked(cpu, 0, CPUHP_ONLINE);
          if (ret)
              break;
          /* See comment in cpuhp_smt_disable() */
          cpuhp_online_cpu_device(cpu);
      }
+    cpus_write_unlock();
+    arch_smt_update();
      cpu_maps_update_done();
      return ret;
Quoted 1 line

 }

What about cpuhp_smt_disable?

#52 lore raw
Quoted 147 lines · Shrikanth Hegde



Quoted 141 lines · Joel Fernandes
Quoted 20 lines · Samir M
Quoted 11 lines · Vishal Chourasia

Bulk CPU hotplug operations--such as switching SMT modes across all cores--require hotplugging multiple CPUs in rapid succession. On large systems, this process takes significant time, increasing as the number of CPUs grows, leading to substantial delays on high-core-count machines. Analysis [1] reveals that the majority of this time is spent waiting for synchronize_rcu().

Expedite synchronize_rcu() during the hotplug path to accelerate the operation. Since CPU hotplug is a user-initiated administrative task, it should complete as quickly as possible.

Hi Vishal,

I verified this patch using the configuration described below. Configuration:
* Kernel version: 6.19.0-rc5
* Number of CPUs: 2048

Quoted 4 lines

SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------

SMT=off     | 30m 53.194s      |  6m 4.250s  | +80.40%          |
SMT=on      | 49m 5.920s       | 36m 50.386s | +25.01%          |

Hi Vishal, Samir,
Thanks for the testing on your large CPU count system.
Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.
git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
tag: cpuhp-bulk-optimize-rfc-v1
I tested it lightly on rcutorture hotplug test and it passes. Please share
any performance results, thanks.
Also I'd like to use expediting of RCU as a last resort TBH, we should
optimize the outer operations that require RCU in the first place such as
Peter's suggestion since that will improve the overall efficiency of the
code. And if/when expediting RCU, Peter's other suggestion to not do it in
cpus_write_lock() and instead do it from cpuhp_smt_enable() also makes sense
to me.
---8<-----------------------
From: Joel Fernandes
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiring
Bulk CPU hotplug operations such as enabling SMT across all cores
require hotplugging multiple CPUs. The current implementation takes
cpus_write_lock() for each individual CPU causing multiple slow grace
period requests.
Therefore introduce cpu_up_locked() that assumes the caller already
holds cpus_write_lock(). The cpuhp_smt_enable() function is updated to
hold the lock once around the entire loop rather than for each CPU.

Suggested-by Peter Zijlstra
Signed-off-by Joel Fernandes
---
 kernel/cpu.c | 40 +++++++++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 15 deletions(-)
kernel/cpu.c +23 -15
index 8df2d773fe3b..4ce7deb236d7 100644
@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state)
     complete_ap_thread(st, true);
 }
-/* Requires cpu_add_remove_lock to be held */
-static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
+/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */
+static int cpu_up_locked(unsigned int cpu, int tasks_frozen,
+             enum cpuhp_state target)
 {
     struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu);
     struct task_struct *idle;
     int ret = 0;
-    cpus_write_lock();
+    lockdep_assert_cpus_held();
-    if (!cpu_present(cpu)) {
-        ret = -EINVAL;
-        goto out;
-    }
+    if (!cpu_present(cpu))
+        return -EINVAL;
     /*
      * The caller of cpu_up() might have raced with another
      * caller. Nothing to do.
      */
     if (st->state >= target)
-        goto out;
+        return 0;
     if (st->state == CPUHP_OFFLINE) {
         /* Let it fail before we try to bring the cpu up */
         idle = idle_thread_get(cpu);
-        if (IS_ERR(idle)) {
-            ret = PTR_ERR(idle);
-            goto out;
-        }
+        if (IS_ERR(idle))
+            return PTR_ERR(idle);
         /*
          * Reset stale stack state from the last time this CPU was online.
@@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
          * return the error code..
          */
         if (ret)
-            goto out;
+            return ret;
     }
     /*
@@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
      */
     target = min((int)target, CPUHP_BRINGUP_CPU);
     ret = cpuhp_up_callbacks(cpu, st, target);
-out:
+    return ret;
+}
+
+/* Requires cpu_add_remove_lock to be held */
+static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
+{
+    int ret;
+
+    cpus_write_lock();
+    ret = cpu_up_locked(cpu, tasks_frozen, target);
     cpus_write_unlock();
     arch_smt_update();
     return ret;
@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void)
     int cpu, ret = 0;
     cpu_maps_update_begin();
+    /* Hold cpus_write_lock() for entire batch operation. */
+    cpus_write_lock();
     cpu_smt_control = CPU_SMT_ENABLED;
     for_each_present_cpu(cpu) {
         /* Skip online CPUs and CPUs on offline nodes */
@@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void)
             continue;
         if (!cpu_smt_thread_allowed(cpu) || ! topology_is_core_online(cpu))
             continue;
-        ret = _cpu_up(cpu, 0, CPUHP_ONLINE);
+        ret = cpu_up_locked(cpu, 0, CPUHP_ONLINE);
         if (ret)
             break;
         /* See comment in cpuhp_smt_disable() */
         cpuhp_online_cpu_device(cpu);
Quoted 8 lines

kernel/cpu.c int cpuhp_smt_enable(void) · line 2743

Quoted 6 lines
kernel/cpu.c +2 -0
     }
+    cpus_write_unlock();
+    arch_smt_update();
     cpu_maps_update_done();
     return ret;
 }

What about cpuhp_smt_disable?

Doing this is not that easy on disable path, AFAICS. Considering that the enable path in the performance tests was much worse, I wanted to contain it to that.

This does not have to be the only fix though, but one of the cures to get there.

Thanks.

#53 lore raw
In reply to Joel Fernandes
Quoted 11 lines · Vishal Chourasia

Performance data on a PPC64 system with 400 CPUs:

+ ppc64_cpu --smt=1 (SMT8 to SMT1)

Before: real 1m14.792s
After:  real 0m03.205s  # ~23x improvement

+ ppc64_cpu --smt=8 (SMT1 to SMT8)

Before: real 2m27.695s
After:  real 0m02.510s  # ~58x improvement

Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b

Quoted 137 lines · Joel Fernandes

Hi Vishal, Samir,

Thanks for the testing on your large CPU count system.

Considering the SMT=on performance is still terrible, before we expedite RCU, could we try the approach Peter suggested (avoiding repeated lock/unlock)? I wrote a patch below.

git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git
tag: cpuhp-bulk-optimize-rfc-v1

I tested it lightly on rcutorture hotplug test and it passes. Please share any performance results, thanks.

Also I'd like to use expediting of RCU as a last resort TBH, we should optimize the outer operations that require RCU in the first place such as Peter's suggestion since that will improve the overall efficiency of the code. And if/when expediting RCU, Peter's other suggestion to not do it in cpus_write_lock() and instead do it from cpuhp_smt_enable() also makes sense to me.

---8<-----------------------

From: Joel Fernandes <[email protected]>
Subject: [PATCH] cpuhp: Optimize batch SMT enable by reducing lock acquiring

Bulk CPU hotplug operations such as enabling SMT across all cores require hotplugging multiple CPUs. The current implementation takes cpus_write_lock() for each individual CPU causing multiple slow grace period requests.

Therefore introduce cpu_up_locked() that assumes the caller already holds cpus_write_lock(). The cpuhp_smt_enable() function is updated to hold the lock once around the entire loop rather than for each CPU.

---
 kernel/cpu.c | 40 +++++++++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 15 deletions(-)
kernel/cpu.c +25 -15
index 8df2d773fe3b..4ce7deb236d7 100644
@@ -1623,34 +1623,31 @@ void cpuhp_online_idle(enum cpuhp_state state)
 	complete_ap_thread(st, true);
 }
-/* Requires cpu_add_remove_lock to be held */
-static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
+/* Requires cpu_add_remove_lock and cpus_write_lock to be held. */
+static int cpu_up_locked(unsigned int cpu, int tasks_frozen,
+			 enum cpuhp_state target)
 {
 	struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu);
 	struct task_struct *idle;
 	int ret = 0;
-	cpus_write_lock();
+	lockdep_assert_cpus_held();
-	if (!cpu_present(cpu)) {
-		ret = -EINVAL;
-		goto out;
-	}
+	if (!cpu_present(cpu))
+		return -EINVAL;
 	/*
 	 * The caller of cpu_up() might have raced with another
 	 * caller. Nothing to do.
 	 */
 	if (st->state >= target)
-		goto out;
+		return 0;
 	if (st->state == CPUHP_OFFLINE) {
 		/* Let it fail before we try to bring the cpu up */
 		idle = idle_thread_get(cpu);
-		if (IS_ERR(idle)) {
-			ret = PTR_ERR(idle);
-			goto out;
-		}
+		if (IS_ERR(idle))
+			return PTR_ERR(idle);
 		/*
 		 * Reset stale stack state from the last time this CPU was online.
@@ -1673,7 +1670,7 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
 		 * return the error code..
 		 */
 		if (ret)
-			goto out;
+			return ret;
 	}
 	/*
@@ -1683,7 +1680,16 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
 	 */
 	target = min((int)target, CPUHP_BRINGUP_CPU);
 	ret = cpuhp_up_callbacks(cpu, st, target);
-out:
+	return ret;
+}
+
+/* Requires cpu_add_remove_lock to be held */
+static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target)
+{
+	int ret;
+
+	cpus_write_lock();
+	ret = cpu_up_locked(cpu, tasks_frozen, target);
 	cpus_write_unlock();
 	arch_smt_update();
 	return ret;
@@ -2715,6 +2721,8 @@ int cpuhp_smt_enable(void)
 	int cpu, ret = 0;
 	cpu_maps_update_begin();
+	/* Hold cpus_write_lock() for entire batch operation. */
+	cpus_write_lock();
 	cpu_smt_control = CPU_SMT_ENABLED;
 	for_each_present_cpu(cpu) {
 		/* Skip online CPUs and CPUs on offline nodes */
@@ -2722,12 +2730,14 @@ int cpuhp_smt_enable(void)
 			continue;
 		if (!cpu_smt_thread_allowed(cpu) || !topology_is_core_online(cpu))
 			continue;
-		ret = _cpu_up(cpu, 0, CPUHP_ONLINE);
+		ret = cpu_up_locked(cpu, 0, CPUHP_ONLINE);
 		if (ret)
 			break;
 		/* See comment in cpuhp_smt_disable() */
 		cpuhp_online_cpu_device(cpu);
 	}
+	cpus_write_unlock();
+	arch_smt_update();
 	cpu_maps_update_done();
 	return ret;
 }
Signature · 1 line
2.34.1

Hi Joel,

I tested above patch on 400 CPU machine that I had originally posted the numbers for.

# time echo 1 > /sys/devices/system/cpu/smt/control
real    1m27.133s # Base 
real    1m25.859s # With patch
# time echo 8 > /sys/devices/system/cpu/smt/control
real    1m0.682s  # Base           
real    1m3.423s  # With patch