srcu: switch work func to allow concurrent gp
Subsystems (2)
PATCH 1/1 srcu: switch work func to allow concurrent gp
ssp->srcu_cb_mutex is introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the same work_struct is serialized and can not run concurrently in fact.
Quoting from Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================
Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"
To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when srcu_gp_end() releases srcu_gp_mutex, a new state machine can begin.
--- kernel/rcu/srcutree.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
kernel/rcu/srcutree.c +19 -0
@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */
@@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))
@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)
Signature · 1 line
2.31.1
Quoted 48 lines · Pingfan Liu
ssp->srcu_cb_mutexis introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the samework_structis serialized and can not run concurrently in fact.Quoting from
Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when
srcu_gp_end()releasessrcu_gp_mutex, a new state machine can begin.Signed-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Zhang, Qiang1" <[email protected]>--- kernel/rcu/srcutree.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)kernel/rcu/srcutree.c +1
-0@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */
kernel/rcu/srcutree.c static void srcu_gp_end(struct srcu_struct *ssp) · line 771
Quoted 8 lines
kernel/rcu/srcutree.c +5
-0@@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;
This looks really hacky ;-) It would be good that workqueue has an API to allow "resetting" a work.
Do you have any number of the potential performance improvement?
Regards,
Boqun
Quoted 26 lines
kernel/rcu/srcutree.c +13
-0rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)Signature · 1 line
2.31.1
Quoted 65 lines · Boqun Feng
Quoted 48 lines · Pingfan Liu
ssp->srcu_cb_mutexis introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the samework_structis serialized and can not run concurrently in fact.Quoting from
Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when
srcu_gp_end()releasessrcu_gp_mutex, a new state machine can begin.Signed-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Zhang, Qiang1" <[email protected]>To: [email protected]
---kernel/rcu/srcutree.c| 19 +++++++++++++++++++1 file changed, 19 insertions(+)kernel/rcu/srcutree.c +1
-0@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */ kernel/rcu/srcutree.c static void srcu_gp_end(struct srcu_struct *ssp) · line 771
Quoted 8 lines
kernel/rcu/srcutree.c +5
-0@@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;This looks really hacky ;-) It would be good that workqueue has an API to allow "resetting" a work.
Quoted 1 line
Do you have any number of the potential performance improvement?
Agreed. Another question - have you verified the workqueue behavior after this change (say using tracing), regardless of what the workqueue documentation says?
Thanks,
- Joel
Quoted 30 lines
Regards,
BoqunQuoted 26 lines
kernel/rcu/srcutree.c +13
-0rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)Signature · 1 line
2.31.1
Quoted 65 lines · Joel Fernandes
Quoted 65 lines · Boqun Feng
Quoted 48 lines · Pingfan Liu
ssp->srcu_cb_mutexis introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the samework_structis serialized and can not run concurrently in fact.Quoting from
Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when
srcu_gp_end()releasessrcu_gp_mutex, a new state machine can begin.Signed-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Zhang, Qiang1" <[email protected]>To: [email protected]
---kernel/rcu/srcutree.c| 19 +++++++++++++++++++1 file changed, 19 insertions(+)kernel/rcu/srcutree.c +1
-0@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */ kernel/rcu/srcutree.c static void srcu_gp_end(struct srcu_struct *ssp) · line 771
Quoted 8 lines
kernel/rcu/srcutree.c +5
-0@@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;This looks really hacky ;-) It would be good that workqueue has an API to allow "resetting" a work.
Quoted 5 lines
Quoted 1 line
Do you have any number of the potential performance improvement?
Agreed. Another question - have you verified the workqueue behavior after this change (say using tracing), regardless of what the workqueue documentation says?
I will try to answer this question after figuring out a test case.
Thanks,
Pingfan
Quoted 36 lines
Thanks,
- Joel
Quoted 30 lines
Regards,
BoqunQuoted 26 lines
kernel/rcu/srcutree.c +13
-0rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)Signature · 1 line
2.31.1
Quoted 51 lines · Boqun Feng
Quoted 50 lines · Pingfan Liu
ssp->srcu_cb_mutexis introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the samework_structis serialized and can not run concurrently in fact.Quoting from
Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when
srcu_gp_end()releasessrcu_gp_mutex, a new state machine can begin.Signed-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Zhang, Qiang1" <[email protected]>--- kernel/rcu/srcutree.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)kernel/rcu/srcutree.c +1
-0@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */ @@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;
Quoted 10 lines
kernel/rcu/srcutree.c static void srcu_gp_end(struct srcu_struct *ssp) · line 771
Quoted 6 lines
kernel/rcu/srcutree.c +5
-0WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;This looks really hacky ;-) It would be good that workqueue has an API to allow "resetting" a work.
Indeed, it is hacky and it can be done by using alternative work_struct so that from the workqueue API, it is intact.
Quoted 2 lines
Do you have any number of the potential performance improvement?
I will try to bring out one. But the result may heavily depend on the test case.
Thanks,
Pingfan
Quoted 29 lines
Regards,
BoqunQuoted 26 lines
kernel/rcu/srcutree.c +13
-0rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)Signature · 1 line
2.31.1
Quoted 63 lines · Pingfan Liu
Quoted 51 lines · Boqun Feng
Quoted 50 lines · Pingfan Liu
ssp->srcu_cb_mutexis introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the samework_structis serialized and can not run concurrently in fact.Quoting from
Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when
srcu_gp_end()releasessrcu_gp_mutex, a new state machine can begin.Signed-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Zhang, Qiang1" <[email protected]>--- kernel/rcu/srcutree.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)kernel/rcu/srcutree.c +1
-0@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */ @@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;Quoted 10 lines
kernel/rcu/srcutree.c static void srcu_gp_end(struct srcu_struct *ssp) · line 771
Quoted 6 lines
kernel/rcu/srcutree.c +5
-0WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;This looks really hacky ;-) It would be good that workqueue has an API to allow "resetting" a work.
Quoted 9 lines
Indeed, it is hacky and it can be done by using alternative
work_structso that from the workqueue API, it is intact.Quoted 2 lines
Do you have any number of the potential performance improvement?
I will try to bring out one. But the result may heavily depend on the test case.
Sorry to update late, I was interrupted by something else.
I used two machine to test
-1. on 144 cpus hpe-dl560gen10
modprobe rcutorture torture_type=srcud fwd_progress=144 fwd_progress_holdoff=1 shutdown_secs=36000 stat_interval=60 verbose=1
# no patch [37587.632155] srcud: End-test grace-period state: g28287244 f0x0 total-gps=28287244 #my patch [36443.468017] srcud: End-test grace-period state: g29026056 f0x0 total-gps=29026056
-2. on 256 cpus amd-milan
modprobe rcutorture torture_type=srcud fwd_progress=256 fwd_progress_holdoff=1 shutdown_secs=36000 stat_interval=60 verbose=1
# no patch [36093.605732] srcud: End-test grace-period state: g10850284 f0x0 total-gps=10850284 #my patch [36093.856713] srcud: End-test grace-period state: g10672632 f0x0 total-gps=10672632
The first test shows that it has about 2.6% improvement, while the second test shows it has no significant effect.
I wonder if any test case can be in flavour of my patch. Any suggestion?
Thanks,
Pingfan
Quoted 33 lines
Thanks,
Pingfan
Quoted 29 lines
Regards,
BoqunQuoted 26 lines
kernel/rcu/srcutree.c +13
-0rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)Signature · 1 line
2.31.1
Quoted 67 lines · Pingfan Liu
Quoted 66 lines · Pingfan Liu
Quoted 51 lines · Boqun Feng
Quoted 50 lines · Pingfan Liu
ssp->srcu_cb_mutexis introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the samework_structis serialized and can not run concurrently in fact.Quoting from
Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when
srcu_gp_end()releasessrcu_gp_mutex, a new state machine can begin.Signed-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Zhang, Qiang1" <[email protected]>--- kernel/rcu/srcutree.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)kernel/rcu/srcutree.c +1
-0@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */ @@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;Quoted 10 lines
kernel/rcu/srcutree.c static void srcu_gp_end(struct srcu_struct *ssp) · line 771
Quoted 6 lines
kernel/rcu/srcutree.c +5
-0WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;This looks really hacky ;-) It would be good that workqueue has an API to allow "resetting" a work.
Indeed, it is hacky and it can be done by using alternative
work_structso that from the workqueue API, it is intact.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Do you have any number of the potential performance improvement?
I will try to bring out one. But the result may heavily depend on the test case.
Sorry to update late, I was interrupted by something else.
No worries, thanks for looking into it.
Quoted 22 lines
I used two machine to test
-1. on 144 cpus hpe-dl560gen10
modprobe rcutorture torture_type=srcud fwd_progress=144 fwd_progress_holdoff=1 shutdown_secs=36000 stat_interval=60 verbose=1
# no patch [37587.632155] srcud: End-test grace-period state: g28287244 f0x0 total-gps=28287244 #my patch [36443.468017] srcud: End-test grace-period state: g29026056 f0x0 total-gps=29026056-2. on 256 cpus amd-milan
modprobe rcutorture torture_type=srcud fwd_progress=256 fwd_progress_holdoff=1 shutdown_secs=36000 stat_interval=60 verbose=1# no patch [36093.605732] srcud: End-test grace-period state: g10850284 f0x0 total-gps=10850284 #my patch [36093.856713] srcud: End-test grace-period state: g10672632 f0x0 total-gps=10672632
Quoted 6 lines
The first test shows that it has about 2.6% improvement, while the second test shows it has no significant effect.
I wonder if any test case can be in flavour of my patch. Any suggestion?
I actually wait you to tell me that ;-) Your trick clearly can improve the degree of parallel a bit, but IIUC you need that srcu_end_gp() runs at the same time with the srcu work function (i.e. pending bit is cleared) to archieve that. And that's really a small time window ;-)
And there's always the Amdahl's law: do we know whether the state machine processing of SRCU is the bottleneck in any workload?
And are we willing to give CPU time to SRCU state machine processing instead of anything else?
My suggestion is to keep this patch somewhere, and whenever you see a related problem, try the patch and see whether it helps ;-)
Regards,
Boqun
Quoted 38 lines
Thanks,
Pingfan
Quoted 33 lines
Thanks,
Pingfan
Quoted 29 lines
Regards,
BoqunQuoted 26 lines
kernel/rcu/srcutree.c +13
-0rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)Signature · 1 line
2.31.1
Quoted 105 lines · Boqun Feng
Quoted 67 lines · Pingfan Liu
Quoted 66 lines · Pingfan Liu
Quoted 51 lines · Boqun Feng
Quoted 50 lines · Pingfan Liu
ssp->srcu_cb_mutexis introduced to allow the other srcu state machine to advance as soon as possible. But according to the implement of workqueue, the samework_structis serialized and can not run concurrently in fact.Quoting from
Documentation/core-api/workqueue.rst
"
Non-reentrance Conditions
=========================Workqueue guarantees that a work item cannot be re-entrant if the following conditions hold after a work item gets queued:
1. The work function hasn't been changed.
2. No one queues the work item to another workqueue.
3. The work item hasn't been reinitiated.
"To allow the concurrence to some extent, it can be achieved by changing the work function to break the conditions. As a result, when
srcu_gp_end()releasessrcu_gp_mutex, a new state machine can begin.Signed-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Zhang, Qiang1" <[email protected]>--- kernel/rcu/srcutree.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)kernel/rcu/srcutree.c +1
-0@@ -75,6 +75,7 @@ static bool __read_mostly srcu_init_done;static void srcu_invoke_callbacks (struct work_struct *work);static void srcu_reschedule (struct srcu_struct *ssp,unsigned long delay);static void process_srcu (struct work_struct *work); +static void process_srcu_wrap (struct work_struct *work);static void srcu_delay_timer (struct timer_list *t);/* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node(). */ @@ -763,6 +764,11 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbdelay =0 ;Quoted 10 lines
kernel/rcu/srcutree.c static void srcu_gp_end(struct srcu_struct *ssp) · line 771
Quoted 6 lines
kernel/rcu/srcutree.c +5
-0WRITE_ONCE (ssp->srcu_last_gp_end,ktime_get_mono_fast_ns ()); +/* Change work func so work can be concurrent */ +if (ssp->work.work.func == process_srcu_wrap) + ssp->work.work.func = process_srcu; +else + ssp->work.work.func = process_srcu_wrap;This looks really hacky ;-) It would be good that workqueue has an API to allow "resetting" a work.
Indeed, it is hacky and it can be done by using alternative
work_structso that from the workqueue API, it is intact.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Do you have any number of the potential performance improvement?
I will try to bring out one. But the result may heavily depend on the test case.
Sorry to update late, I was interrupted by something else.
No worries, thanks for looking into it.
Quoted 26 lines
I used two machine to test
-1. on 144 cpus hpe-dl560gen10
modprobe rcutorture torture_type=srcud fwd_progress=144 fwd_progress_holdoff=1 shutdown_secs=36000 stat_interval=60 verbose=1
# no patch [37587.632155] srcud: End-test grace-period state: g28287244 f0x0 total-gps=28287244 #my patch [36443.468017] srcud: End-test grace-period state: g29026056 f0x0 total-gps=29026056-2. on 256 cpus amd-milan
modprobe rcutorture torture_type=srcud fwd_progress=256 fwd_progress_holdoff=1 shutdown_secs=36000 stat_interval=60 verbose=1# no patch [36093.605732] srcud: End-test grace-period state: g10850284 f0x0 total-gps=10850284 #my patch [36093.856713] srcud: End-test grace-period state: g10672632 f0x0 total-gps=10672632The first test shows that it has about 2.6% improvement, while the second test shows it has no significant effect.
Quoted 8 lines
Quoted 2 lines
I wonder if any test case can be in flavour of my patch. Any suggestion?
I actually wait you to tell me that ;-) Your trick clearly can improve the degree of parallel a bit, but IIUC you need that
srcu_end_gp()runs at the same time with the srcu work function (i.e. pending bit is cleared) to archieve that. And that's really a small time window ;-)
Yes, it is smaller than I expected.
Quoted 9 lines
And there's always the Amdahl's law: do we know whether the state machine processing of SRCU is the bottleneck in any workload?
And are we willing to give CPU time to SRCU state machine processing instead of anything else?
My suggestion is to keep this patch somewhere, and whenever you see a related problem, try the patch and see whether it helps ;-)
Yes. Thanks for your suggestion.
Again, appreciate for your help and advice.
Regards,
Pingfan
Quoted 41 lines
Regards,
BoqunQuoted 38 lines
Thanks,
Pingfan
Quoted 33 lines
Thanks,
Pingfan
Quoted 29 lines
Regards,
BoqunQuoted 26 lines
kernel/rcu/srcutree.c +13
-0rcu_seq_end (&ssp->srcu_gp_seq); gpseq =rcu_seq_current (&ssp->srcu_gp_seq);if (ULONG_CMP_LT (ssp->srcu_gp_seq_needed_exp, gpseq))@@ -1637,6 +1643,19 @@ static void process_srcu(struct work_struct *work)srcu_reschedule (ssp, curdelay); } +/* +* The ssp->work is expected to be concurrent to some extent, but the current +* workqueue does not support the concurrence on the same work. (Refer to the +* section "Non-reentrance Conditions" in the file workqueue.rst) +* Resolving it by changing the work func. +* +* Prevent compilering from optimizing out it. +*/ +static __used void process_srcu_wrap (struct work_struct *work) +{ +process_srcu (work); +} +void srcutorture_get_gp_data (enum rcutorture_type test_type,struct srcu_struct *ssp,int *flags,unsigned long *gp_seq)Signature · 1 line
2.31.1