srcu: Move updating of segcblist from srcu_gp_start() to srcu_might_be_idle()
Subsystems (2)
PATCH 1/1 srcu: Move updating of segcblist from srcu_gp_start() to srcu_might_be_idle()
The pair of segcblist operations: rcu_segcblist_advance() and rcu_segcblist_accelerate() in srcu_gp_start() is needless from two perspectives:
-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.
-2. From the viewpoint of the callback, at the entrance,srcu_gp_start_if_needed() has called that pair operations and attached it with gp_seq. At the exit, srcu_invoke_callbacks() calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair in srcu_gp_start().
But because the updating of RCU_DONE_TAIL by srcu_invoke_callbacks() may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may cause srcu_might_be_idle() not to be real time. To compensate that, supplement that pair just before the calling to rcu_segcblist_pend_cbs() in srcu_might_be_idle().
Test info:
torture test passed using the following command against commit 094226ad94f4 ("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 10h --configs 18*SRCU-P
--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
kernel/rcu/srcutree.c +4 -11
@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */
@@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */
Signature · 1 line
2.31.1
Paul E. McKenney
maintainerQuoted 6 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.
I am not so sure that I agree.
Taking care of all srcu_node structures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where the srcu_struct structure's ->srcu_size_state field is equal to SRCU_SIZE_BIG.
But are you seeing performance issues with that code on real hardware running real workloads when the srcu_struct structure's ->srcu_size_state field does not equal SRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring the srcu_struct structure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)
If you are seeing real slowdowns, then one option is to decrease the default value of big_cpu_lim from 128 to some appropriate smaller value. Maybe 16 or 32?
Alternatively, the code in srcu_gp_start() that this patch removes might be executed only when ssp->srcu_size_state is equal to SRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition of srcu_struct structure's ->lock, right?
Also, if you are seeing real slowdowns, please place that information in the commit log.
Thoughts?
Quoted 11 lines
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().
I agree that srcu_might_be_idle() is not a bad place to add such a check, especially given that it has other reasons to acquire the srcu_data structure's ->lock. Except that this misses both call_srcu() and synchronize_srcu_expedited(), so that if a workload invokes call_srcu() and/or synchronize_srcu_expedited() on a given srcu_struct structure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.
Alternatively, if we have checks in both synchronize_srcu() and srcu_gp_start_if_needed(), we end up duplicating the work in the case where synchronize_srcu() is invoked. This is also not particularly good.
Again, thoughts?
Thanx, Paul
Quoted 56 lines
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1
Quoted 48 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.I am not so sure that I agree.
Taking care of all
srcu_nodestructures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where thesrcu_structstructure's ->srcu_size_state field is equal toSRCU_SIZE_BIG.But are you seeing performance issues with that code on real hardware running real workloads when the
srcu_structstructure's ->srcu_size_state field does not equalSRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring thesrcu_structstructure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)If you are seeing real slowdowns, then one option is to decrease the default value of
big_cpu_limfrom 128 to some appropriate smaller value. Maybe 16 or 32?Alternatively, the code in
srcu_gp_start()that this patch removes might be executed only whenssp->srcu_size_stateis equal toSRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition ofsrcu_structstructure's ->lock, right?Also, if you are seeing real slowdowns, please place that information in the commit log.
Thoughts?
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().
Quoted 8 lines
I agree that
srcu_might_be_idle()is not a bad place to add such a check, especially given that it has other reasons to acquire thesrcu_datastructure's ->lock. Except that this misses bothcall_srcu()andsynchronize_srcu_expedited(), so that if a workload invokescall_srcu()and/orsynchronize_srcu_expedited()on a givensrcu_structstructure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.
Maybe at least we need the following check:
kernel/rcu/srcutree.c +4 -1
@@ -1099,7 +1099,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)check_init_srcu_struct (ssp);/* If the local srcu_data structure has callbacks, not idle. */ - sdp =raw_cpu_ptr (ssp->sda); +if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) + sdp =per_cpu_ptr (ssp->sda,get_boot_cpu_id ()); +else + sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags);if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);
thoughts?
Thanx, Zqiang
Quoted 64 lines
Alternatively, if we have checks in both
synchronize_srcu()andsrcu_gp_start_if_needed(), we end up duplicating the work in the case wheresynchronize_srcu()is invoked. This is also not particularly good.Again, thoughts?
Thanx, Paul
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1
Quoted 65 lines · On Thu, Nov 17, 2022 at 01:39:52AM +0000, Zhang, Qiang1 wrote:
Quoted 48 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.I am not so sure that I agree.
Taking care of all
srcu_nodestructures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where thesrcu_structstructure's ->srcu_size_state field is equal toSRCU_SIZE_BIG.But are you seeing performance issues with that code on real hardware running real workloads when the
srcu_structstructure's ->srcu_size_state field does not equalSRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring thesrcu_structstructure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)If you are seeing real slowdowns, then one option is to decrease the default value of
big_cpu_limfrom 128 to some appropriate smaller value. Maybe 16 or 32?Alternatively, the code in
srcu_gp_start()that this patch removes might be executed only whenssp->srcu_size_stateis equal toSRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition ofsrcu_structstructure's ->lock, right?Also, if you are seeing real slowdowns, please place that information in the commit log.
Thoughts?
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().Quoted 8 lines
I agree that
srcu_might_be_idle()is not a bad place to add such a check, especially given that it has other reasons to acquire thesrcu_datastructure's ->lock. Except that this misses bothcall_srcu()andsynchronize_srcu_expedited(), so that if a workload invokescall_srcu()and/orsynchronize_srcu_expedited()on a givensrcu_structstructure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.Maybe at least we need the following check:
kernel/rcu/srcutree.c +0
-0
kernel/rcu/srcutree.c static bool srcu_might_be_idle(struct srcu_struct *ssp) · line 1105
Quoted 8 lines
kernel/rcu/srcutree.c +4
-1@@ -1099,7 +1099,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)check_init_srcu_struct (ssp);/* If the local srcu_data structure has callbacks, not idle. */ - sdp =raw_cpu_ptr (ssp->sda); +if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) + sdp =per_cpu_ptr (ssp->sda,get_boot_cpu_id ()); +else + sdp =raw_cpu_ptr (ssp->sda);
Consider that srcu_might_be_idle() renders a rough estimation, this is not necessary but better to have.
Also could you pack this pattern in a inline function if my patch is accepted [1]?
Thanks,
Pingfan
Quoted 72 lines
kernel/rcu/srcutree.c +0
-0spin_lock_irqsave_rcu_node (sdp, flags);if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);thoughts?
Thanx, Zqiang
Quoted 64 lines
Alternatively, if we have checks in both
synchronize_srcu()andsrcu_gp_start_if_needed(), we end up duplicating the work in the case wheresynchronize_srcu()is invoked. This is also not particularly good.Again, thoughts?
Thanx, Paul
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1
Quoted 77 lines · On Thu, Nov 17, 2022 at 01:39:52AM +0000, Zhang, Qiang1 wrote:
Quoted 48 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.I am not so sure that I agree.
Taking care of all
srcu_nodestructures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where thesrcu_structstructure's ->srcu_size_state field is equal toSRCU_SIZE_BIG.But are you seeing performance issues with that code on real hardware running real workloads when the
srcu_structstructure's ->srcu_size_state field does not equalSRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring thesrcu_structstructure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)If you are seeing real slowdowns, then one option is to decrease the default value of
big_cpu_limfrom 128 to some appropriate smaller value. Maybe 16 or 32?Alternatively, the code in
srcu_gp_start()that this patch removes might be executed only whenssp->srcu_size_stateis equal toSRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition ofsrcu_structstructure's ->lock, right?Also, if you are seeing real slowdowns, please place that information in the commit log.
Thoughts?
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().Quoted 8 lines
I agree that
srcu_might_be_idle()is not a bad place to add such a check, especially given that it has other reasons to acquire thesrcu_datastructure's ->lock. Except that this misses bothcall_srcu()andsynchronize_srcu_expedited(), so that if a workload invokescall_srcu()and/orsynchronize_srcu_expedited()on a givensrcu_structstructure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.Maybe at least we need the following check:
kernel/rcu/srcutree.c +4
-1@@ -1099,7 +1099,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)check_init_srcu_struct (ssp);/* If the local srcu_data structure has callbacks, not idle. */ - sdp =raw_cpu_ptr (ssp->sda); +if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) + sdp =per_cpu_ptr (ssp->sda,get_boot_cpu_id ()); +else + sdp =raw_cpu_ptr (ssp->sda);Consider that
srcu_might_be_idle()renders a rough estimation, this is not necessary but better to have.
Quoted 3 lines
Also could you pack this pattern in a inline function if my patch is accepted [1]?
Hi, Pingfan
patch already exists
https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/commit/?h=dev&id=1d0f495f69001f0f121aa1f60f114a94b1c052ad
Thanks
Zqiang
Quoted 78 lines
Thanks,
Pingfan
spin_lock_irqsave_rcu_node(sdp, flags); if (rcu_segcblist_pend_cbs(&sdp->srcu_cblist)) { spin_unlock_irqrestore_rcu_node(sdp, flags);thoughts?
Thanx, Zqiang
Quoted 64 lines
Alternatively, if we have checks in both
synchronize_srcu()andsrcu_gp_start_if_needed(), we end up duplicating the work in the case wheresynchronize_srcu()is invoked. This is also not particularly good.Again, thoughts?
Thanx, Paul
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1
Quoted 27 lines · Paul E. McKenney
Quoted 6 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.I am not so sure that I agree.
Taking care of all
srcu_nodestructures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where thesrcu_structstructure's ->srcu_size_state field is equal toSRCU_SIZE_BIG.But are you seeing performance issues with that code on real hardware running real workloads when the
srcu_structstructure's ->srcu_size_state field does not equalSRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring thesrcu_structstructure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)If you are seeing real slowdowns, then one option is to decrease the default value of
big_cpu_limfrom 128 to some appropriate smaller value. Maybe 16 or 32?
Quoted 8 lines
Alternatively, the code in
srcu_gp_start()that this patch removes might be executed only whenssp->srcu_size_stateis equal toSRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition ofsrcu_structstructure's ->lock, right?Also, if you are seeing real slowdowns, please place that information in the commit log.
Thanks for stiring up so much internal information. It is useful to guild me through the code reading.
I have not seen any slowdowns, just try to organize the code. Since the note before srcu_might_be_idle() explains why it only evaluate a local sda, while srcu_gp_start() has not, so I am puzzled about it.
Quoted 14 lines
Thoughts?
Quoted 11 lines
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().
Quoted 8 lines
I agree that
srcu_might_be_idle()is not a bad place to add such a check, especially given that it has other reasons to acquire thesrcu_datastructure's ->lock. Except that this misses bothcall_srcu()andsynchronize_srcu_expedited(), so that if a workload invokescall_srcu()and/orsynchronize_srcu_expedited()on a givensrcu_structstructure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.
srcu_gp_start_if_needed() has rcu_segcblist_{advance/accelerate}() unconditionally, so the beginning of a grace period will be always detected via any path through __call_srcu(). Do I miss anything?
And I think the core issue is if it matters that the updating of segcblist defers from srcu_gp_end()->srcu_gp_start() to srcu_invoke_callbacks().
Quoted 4 lines
Alternatively, if we have checks in both
synchronize_srcu()andsrcu_gp_start_if_needed(), we end up duplicating the work in the case wheresynchronize_srcu()is invoked. This is also not particularly good.
Here srcu_gp_start() has the following caller:
srcu_advance_state()
srcu_funnel_gp_start()
srcu_reschedule()
srcu_gp_end()
Except srcu_gp_end() increase the gp_seq's counter, the rest has not changed the gp_seq's counter. so the calling to rcu_segcblist_{advance/accelerate}() is needless there.
Anyway I tried to understand the code and some internal. If you dislike it, please skip it.
Thanks again for your precious time.
Pingfan
Quoted 60 lines
Again, thoughts?
Thanx, Paul
Quoted 56 lines
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1
Paul E. McKenney
maintainerQuoted 36 lines · Pingfan Liu
Quoted 27 lines · Paul E. McKenney
Quoted 6 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.I am not so sure that I agree.
Taking care of all
srcu_nodestructures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where thesrcu_structstructure's ->srcu_size_state field is equal toSRCU_SIZE_BIG.But are you seeing performance issues with that code on real hardware running real workloads when the
srcu_structstructure's ->srcu_size_state field does not equalSRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring thesrcu_structstructure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)If you are seeing real slowdowns, then one option is to decrease the default value of
big_cpu_limfrom 128 to some appropriate smaller value. Maybe 16 or 32?Quoted 7 lines
Alternatively, the code in
srcu_gp_start()that this patch removes might be executed only whenssp->srcu_size_stateis equal toSRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition ofsrcu_structstructure's ->lock, right?Also, if you are seeing real slowdowns, please place that information in the commit log.
Quoted 6 lines
Thanks for stiring up so much internal information. It is useful to guild me through the code reading.
I have not seen any slowdowns, just try to organize the code. Since the note before
srcu_might_be_idle()explains why it only evaluate a local sda, whilesrcu_gp_start()has not, so I am puzzled about it.
Locality of reference is a good thing in general, so unless it is a strange case, there won't always be a "do this to promote locality of reference" comment.
Quoted 22 lines
Quoted 14 lines
Thoughts?
Quoted 11 lines
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().Quoted 7 lines
I agree that
srcu_might_be_idle()is not a bad place to add such a check, especially given that it has other reasons to acquire thesrcu_datastructure's ->lock. Except that this misses bothcall_srcu()andsynchronize_srcu_expedited(), so that if a workload invokescall_srcu()and/orsynchronize_srcu_expedited()on a givensrcu_structstructure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.
Quoted 3 lines
srcu_gp_start_if_needed()has rcu_segcblist_{advance/accelerate}() unconditionally, so the beginning of a grace period will be always detected via any path through__call_srcu(). Do I miss anything?
Grace periods can also be started by srcu_gp_end().
Quoted 3 lines
And I think the core issue is if it matters that the updating of segcblist defers from srcu_gp_end()->srcu_gp_start() to
srcu_invoke_callbacks().
The main reason for advancing/accelerating from __call_srcu() is to make sure that the newly enqueued callback will be associated with a grace period. Otherwise, the newly enqueued callback might well just sit there for a very long time.
Unlike normal RCU, there is no callback polling mechanism in SRCU. Nor can there reasonably be, due to the potentially very large number of srcu_struct structures.
(Yes, if __call_srcu() was to become a performance bottleneck, there are adjustments that could be made, but let's not increase complexity before we see a clear need to do so.)
Quoted 10 lines
Quoted 3 lines
Alternatively, if we have checks in both
synchronize_srcu()andsrcu_gp_start_if_needed(), we end up duplicating the work in the case wheresynchronize_srcu()is invoked. This is also not particularly good.Here
srcu_gp_start()has the following caller:
srcu_advance_state()
srcu_funnel_gp_start()
srcu_reschedule()
srcu_gp_end()
Quoted 3 lines
Except
srcu_gp_end()increase the gp_seq's counter, the rest has not changed the gp_seq's counter. so the calling to rcu_segcblist_{advance/accelerate}() is needless there.
Suppose that this srcu_struct is in state SRCU_SIZE_BIG, so that there is a callback list for each CPU.
Quoted 2 lines
Anyway I tried to understand the code and some internal. If you dislike it, please skip it.
Let's skip this for the moment.
Quoted 1 line
Thanks again for your precious time.
Please let me know if additional information on locality of reference would be helpful.
Thanx, Paul
Quoted 62 lines
Pingfan
Quoted 60 lines
Again, thoughts?
Thanx, Paul
Quoted 56 lines
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1
Quoted 40 lines · Paul E. McKenney
Quoted 39 lines · Pingfan Liu
Quoted 27 lines · Paul E. McKenney
Quoted 6 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.I am not so sure that I agree.
Taking care of all
srcu_nodestructures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where thesrcu_structstructure's ->srcu_size_state field is equal toSRCU_SIZE_BIG.But are you seeing performance issues with that code on real hardware running real workloads when the
srcu_structstructure's ->srcu_size_state field does not equalSRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring thesrcu_structstructure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)If you are seeing real slowdowns, then one option is to decrease the default value of
big_cpu_limfrom 128 to some appropriate smaller value. Maybe 16 or 32?Quoted 7 lines
Alternatively, the code in
srcu_gp_start()that this patch removes might be executed only whenssp->srcu_size_stateis equal toSRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition ofsrcu_structstructure's ->lock, right?Also, if you are seeing real slowdowns, please place that information in the commit log.
Thanks for stiring up so much internal information. It is useful to guild me through the code reading.
Quoted 8 lines
Quoted 3 lines
I have not seen any slowdowns, just try to organize the code. Since the note before
srcu_might_be_idle()explains why it only evaluate a local sda, whilesrcu_gp_start()has not, so I am puzzled about it.Locality of reference is a good thing in general, so unless it is a strange case, there won't always be a "do this to promote locality of reference" comment.
Learned.
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Thoughts?
Quoted 11 lines
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().I agree that
srcu_might_be_idle()is not a bad place to add such a check, especially given that it has other reasons to acquire thesrcu_datastructure's ->lock. Except that this misses bothcall_srcu()andsynchronize_srcu_expedited(), so that if a workload invokescall_srcu()and/orsynchronize_srcu_expedited()on a givensrcu_structstructure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.
Quoted 6 lines
Quoted 3 lines
srcu_gp_start_if_needed()has rcu_segcblist_{advance/accelerate}() unconditionally, so the beginning of a grace period will be always detected via any path through__call_srcu(). Do I miss anything?Grace periods can also be started by
srcu_gp_end().
But it is helpless for assiciating a gp with the enqueued callback.
Quoted 31 lines
Quoted 3 lines
And I think the core issue is if it matters that the updating of segcblist defers from srcu_gp_end()->srcu_gp_start() to
srcu_invoke_callbacks().The main reason for advancing/accelerating from
__call_srcu()is to make sure that the newly enqueued callback will be associated with a grace period. Otherwise, the newly enqueued callback might well just sit there for a very long time.Unlike normal RCU, there is no callback polling mechanism in SRCU. Nor can there reasonably be, due to the potentially very large number of
srcu_structstructures.(Yes, if
__call_srcu()was to become a performance bottleneck, there are adjustments that could be made, but let's not increase complexity before we see a clear need to do so.)Quoted 10 lines
Quoted 3 lines
Alternatively, if we have checks in both
synchronize_srcu()andsrcu_gp_start_if_needed(), we end up duplicating the work in the case wheresynchronize_srcu()is invoked. This is also not particularly good.Here
srcu_gp_start()has the following caller:
srcu_advance_state()
srcu_funnel_gp_start()
srcu_reschedule()
srcu_gp_end()Quoted 3 lines
Except
srcu_gp_end()increase the gp_seq's counter, the rest has not changed the gp_seq's counter. so the calling to rcu_segcblist_{advance/accelerate}() is needless there.
Quoted 8 lines
Suppose that this
srcu_structis in stateSRCU_SIZE_BIG, so that there is a callback list for each CPU.Quoted 2 lines
Anyway I tried to understand the code and some internal. If you dislike it, please skip it.
Let's skip this for the moment.
Sure. And I have learned a lot from this thread.
Quoted 5 lines
Quoted 1 line
Thanks again for your precious time.
Please let me know if additional information on locality of reference would be helpful.
No. That is enough.
Appreciate for your detailed explanation. It is very helpful for me to understand the code more deeply.
Thanks,
Pingfan
Quoted 64 lines
Thanx, Paul
Quoted 62 lines
Pingfan
Quoted 60 lines
Again, thoughts?
Thanx, Paul
Quoted 56 lines
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1
Paul E. McKenney
maintainerQuoted 73 lines · Pingfan Liu
Quoted 40 lines · Paul E. McKenney
Quoted 39 lines · Pingfan Liu
Quoted 27 lines · Paul E. McKenney
Quoted 6 lines · Pingfan Liu
The pair of segcblist operations:
rcu_segcblist_advance()andrcu_segcblist_accelerate()insrcu_gp_start()is needless from two perspectives:-1. As a part of the SRCU state machine, it should take care of either
all sda or none. But here it only takes care of a single sda.I am not so sure that I agree.
Taking care of all
srcu_nodestructures' callbacks would be extremely expensive, with the expense increasing with the number of CPUs. However, the cost of taking care of the current CPU's callbacks is quite cheap, at least in the case where thesrcu_structstructure's ->srcu_size_state field is equal toSRCU_SIZE_BIG.But are you seeing performance issues with that code on real hardware running real workloads when the
srcu_structstructure's ->srcu_size_state field does not equalSRCU_SIZE_BIG? My guess is "no" given that this code has already paid the cost of acquiring thesrcu_structstructure's ->lock, but I figured I should ask. The opinion of real hardware running real workloads beats any of our opinions, after all. ;-)If you are seeing real slowdowns, then one option is to decrease the default value of
big_cpu_limfrom 128 to some appropriate smaller value. Maybe 16 or 32?Quoted 7 lines
Alternatively, the code in
srcu_gp_start()that this patch removes might be executed only whenssp->srcu_size_stateis equal toSRCU_SIZE_BIG. But this approach would likely still have performance issues due to the acquisition ofsrcu_structstructure's ->lock, right?Also, if you are seeing real slowdowns, please place that information in the commit log.
Thanks for stiring up so much internal information. It is useful to guild me through the code reading.
Quoted 7 lines
Quoted 3 lines
I have not seen any slowdowns, just try to organize the code. Since the note before
srcu_might_be_idle()explains why it only evaluate a local sda, whilesrcu_gp_start()has not, so I am puzzled about it.Locality of reference is a good thing in general, so unless it is a strange case, there won't always be a "do this to promote locality of reference" comment.
Learned.
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Thoughts?
Quoted 11 lines
-2. From the viewpoint of the callback, at the entrance,
srcu_gp_start_if_needed()has called that pair operations and attached it withgp_seq. At the exit,srcu_invoke_callbacks()calls that pair again to extract the done callbacks. So the harvesting of the callback is not affected by the call to that pair insrcu_gp_start().But because the updating of
RCU_DONE_TAILbysrcu_invoke_callbacks()may have some delay than by srcu_gp_end()->srcu_gp_start(), the removal may causesrcu_might_be_idle()not to be real time. To compensate that, supplement that pair just before the calling torcu_segcblist_pend_cbs()insrcu_might_be_idle().I agree that
srcu_might_be_idle()is not a bad place to add such a check, especially given that it has other reasons to acquire thesrcu_datastructure's ->lock. Except that this misses bothcall_srcu()andsynchronize_srcu_expedited(), so that if a workload invokescall_srcu()and/orsynchronize_srcu_expedited()on a givensrcu_structstructure, but never synchronize)srcu(), the opportunity to advance that CPU's callbacks at the beginning of a grace period will always be lost.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
srcu_gp_start_if_needed()has rcu_segcblist_{advance/accelerate}() unconditionally, so the beginning of a grace period will be always detected via any path through__call_srcu(). Do I miss anything?Grace periods can also be started by
srcu_gp_end().But it is helpless for assiciating a gp with the enqueued callback.
Yes, the newly enqueued callbacks should have grace-period numbers associated at call_srcu() time, but the callbacks might be advanced at rcu_gp_end() time. Some of those grace periods might well have completed.
Quoted 43 lines
Quoted 31 lines
Quoted 3 lines
And I think the core issue is if it matters that the updating of segcblist defers from srcu_gp_end()->srcu_gp_start() to
srcu_invoke_callbacks().The main reason for advancing/accelerating from
__call_srcu()is to make sure that the newly enqueued callback will be associated with a grace period. Otherwise, the newly enqueued callback might well just sit there for a very long time.Unlike normal RCU, there is no callback polling mechanism in SRCU. Nor can there reasonably be, due to the potentially very large number of
srcu_structstructures.(Yes, if
__call_srcu()was to become a performance bottleneck, there are adjustments that could be made, but let's not increase complexity before we see a clear need to do so.)Quoted 10 lines
Quoted 3 lines
Alternatively, if we have checks in both
synchronize_srcu()andsrcu_gp_start_if_needed(), we end up duplicating the work in the case wheresynchronize_srcu()is invoked. This is also not particularly good.Here
srcu_gp_start()has the following caller:
srcu_advance_state()
srcu_funnel_gp_start()
srcu_reschedule()
srcu_gp_end()Quoted 3 lines
Except
srcu_gp_end()increase the gp_seq's counter, the rest has not changed the gp_seq's counter. so the calling to rcu_segcblist_{advance/accelerate}() is needless there.Quoted 7 lines
Suppose that this
srcu_structis in stateSRCU_SIZE_BIG, so that there is a callback list for each CPU.Quoted 2 lines
Anyway I tried to understand the code and some internal. If you dislike it, please skip it.
Let's skip this for the moment.
Sure. And I have learned a lot from this thread.
Quoted 2 lines
Quoted 1 line
Thanks again for your precious time.
Quoted 7 lines
Quoted 2 lines
Please let me know if additional information on locality of reference would be helpful.
No. That is enough.
Appreciate for your detailed explanation. It is very helpful for me to understand the code more deeply.
Thank you, and glad that it was helpful.
Thanx, Paul
Quoted 69 lines
Thanks,
Pingfan
Quoted 64 lines
Thanx, Paul
Quoted 62 lines
Pingfan
Quoted 60 lines
Again, thoughts?
Thanx, Paul
Quoted 56 lines
Test info:
torture test passed using the following command against commit094226ad94f4("Linux 6.1-rc5")
tools/testing/selftests/rcutorture/bin/kvm.sh--allcpus --duration 10h --configs 18*SRCU-PSigned-off-by Pingfan Liu <[email protected]>Cc Lai Jiangshan <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>--- kernel/rcu/srcutree.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)kernel/rcu/srcutree.c +4
-11@@ -659,21 +659,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);*/ static void srcu_gp_start (struct srcu_struct *ssp) { -struct srcu_data *sdp;int state; -if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER ) - sdp =per_cpu_ptr (ssp->sda,0 ); -else - sdp =this_cpu_ptr (ssp->sda);lockdep_assert_held (&ACCESS_PRIVATE (ssp, lock));WARN_ON_ONCE (ULONG_CMP_GE (ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); -spin_lock_rcu_node (sdp);/* Interrupts already disabled. */ -rcu_segcblist_advance (&sdp->srcu_cblist, -rcu_seq_current (&ssp->srcu_gp_seq)); - (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, -rcu_seq_snap (&ssp->srcu_gp_seq)); -spin_unlock_rcu_node (sdp);/* Interrupts remain disabled. */ WRITE_ONCE (ssp->srcu_gp_start, jiffies);WRITE_ONCE (ssp->srcu_n_exp_nodelay,0 );smp_mb ();/* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ @@ -1037,6 +1026,10 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp)/* If the local srcu_data structure has callbacks, not idle. */ sdp =raw_cpu_ptr (ssp->sda);spin_lock_irqsave_rcu_node (sdp, flags); +rcu_segcblist_advance (&sdp->srcu_cblist, +rcu_seq_current (&ssp->srcu_gp_seq)); + (void )rcu_segcblist_accelerate (&sdp->srcu_cblist, +rcu_seq_snap (&ssp->srcu_gp_seq));if (rcu_segcblist_pend_cbs (&sdp->srcu_cblist)) {spin_unlock_irqrestore_rcu_node (sdp, flags);return false;/* Callbacks already present, so not idle. */ Signature · 1 line
2.31.1