rcu/tree: Force quiescent state on callback overload
PATCH 1/1 rcu/tree: Force quiescent state on callback overload
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Paul E. McKenney
maintainerQuoted 5 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.Signed-off-by Neeraj Upadhyay <[email protected]>
Good catch!
But what did you do to verify that this change does the right thing?
Thanx, Paul
Quoted 21 lines
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree.c +1
-1@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Hi Paul,
Quoted 5 lines · Paul E. McKenney
Quoted 4 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.
Quoted 8 lines
Quoted 1 line
Signed-off-by Neeraj Upadhyay <[email protected]>Good catch!
But what did you do to verify that this change does the right thing?
Thanx, Paul
I haven't done a runtime verification of this code path; I posted this, based on review of this code.
Thanks
Neeraj
Quoted 21 lines
Quoted 21 lines
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree.c +1
-1@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Signature · 1 line
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Paul E. McKenney
maintainerQuoted 12 lines · Neeraj Upadhyay
Hi Paul,
Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.Signed-off-by Neeraj Upadhyay <[email protected]>Good catch!
Quoted 7 lines
Quoted 4 lines
But what did you do to verify that this change does the right thing?
Thanx, Paul
I haven't done a runtime verification of this code path; I posted this, based on review of this code.
My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.
Or am I missing something subtle that prevents this?
But yes, it does look like the current mainline code fails to do the first scan immediately, so again, good catch!
Thanx, Paul
Quoted 28 lines
Thanks
NeerajQuoted 21 lines
Quoted 21 lines
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree.c +1
-1@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative ProjectSignature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Hi Paul,
Quoted 21 lines · Paul E. McKenney
Quoted 12 lines · Neeraj Upadhyay
Hi Paul,
Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.Signed-off-by Neeraj Upadhyay <[email protected]>Good catch!
Quoted 7 lines
Quoted 4 lines
But what did you do to verify that this change does the right thing?
Thanx, Paul
I haven't done a runtime verification of this code path; I posted this, based on review of this code.
Quoted 6 lines
My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.
Or am I missing something subtle that prevents this?
Looks like under overload, only the first and second scans are accelerated?
gf = 0;
if (first_gp_fqs) {
first_gp_fqs = false;
gf = rcu_state.cbovld ? RCU_GP_FLAG_OVLD : 0;
}Thanks
Neeraj
Quoted 35 lines
But yes, it does look like the current mainline code fails to do the first scan immediately, so again, good catch!
Thanx, Paul
Quoted 28 lines
Thanks
NeerajQuoted 21 lines
Quoted 21 lines
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree.c +1
-1@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative ProjectSignature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Signature · 1 line
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Paul E. McKenney
maintainerQuoted 31 lines · Neeraj Upadhyay
Hi Paul,
Quoted 21 lines · Paul E. McKenney
Quoted 12 lines · Neeraj Upadhyay
Hi Paul,
Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.Signed-off-by Neeraj Upadhyay <[email protected]>Good catch!
Quoted 7 lines
Quoted 4 lines
But what did you do to verify that this change does the right thing?
Thanx, Paul
I haven't done a runtime verification of this code path; I posted this, based on review of this code.
Quoted 6 lines
My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.
Or am I missing something subtle that prevents this?
Quoted 7 lines
Looks like under overload, only the first and second scans are accelerated?
gf = 0; if (first_gp_fqs) { first_gp_fqs = false; gf = rcu_state.cbovld ? RCU_GP_FLAG_OVLD : 0; }
Very good, it does sound like you understand this, and it matches my analysis and passes light testing, so I queued this one. I did improve the commit log, please check below. The added detail is helpful to people (including ourselves, by the way) who might need to look at this commit some time in the future.
If you have an x86 system lying around, running rcutorture is quite straightforward. Non-x86 systems can also run rcutorture, if nothing else by using modprobe and rmmod as described here:
https://paulmck.livejournal.com/57769.html
The scripting described in the latter part of this document has worked on ARMv8 and PowerPC, and might still work for all I know.
Thanx, Paul
------------------------------------------------------------------------
commit 9482524d7dd0aea5d32a6efa2979223eea07c029
Author: Neeraj Upadhyay <[email protected]> Date: Mon Jun 22 00:07:27 2020 +0530
rcu/tree: Force quiescent state on callback overload
On callback overload, it is necessary to quickly detect idle CPUs, and rcu_gp_fqs_check_wake() checks for this condition. Unfortunately, the code following the call to this function does not repeat this check, which means that in reality no actual quiescent-state forcing, instead only a couple of quick and pointless wakeups at the beginning of the grace period.
This commit therefore adds a check for the RCU_GP_FLAG_OVLD flag in the post-wakeup "if" statement in rcu_gp_fqs_loop().
kernel/rcu/tree.c +1 -1
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);
Hi Paul,
Quoted 40 lines · Paul E. McKenney
Quoted 31 lines · Neeraj Upadhyay
Hi Paul,
Quoted 21 lines · Paul E. McKenney
Quoted 12 lines · Neeraj Upadhyay
Hi Paul,
Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.Signed-off-by Neeraj Upadhyay <[email protected]>Good catch!
Quoted 7 lines
Quoted 4 lines
But what did you do to verify that this change does the right thing?
Thanx, Paul
I haven't done a runtime verification of this code path; I posted this, based on review of this code.
Quoted 6 lines
My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.
Or am I missing something subtle that prevents this?
Quoted 7 lines
Looks like under overload, only the first and second scans are accelerated?
gf = 0; if (first_gp_fqs) { first_gp_fqs = false; gf = rcu_state.cbovld ? RCU_GP_FLAG_OVLD : 0; }
Quoted 6 lines
Very good, it does sound like you understand this, and it matches my analysis and passes light testing, so I queued this one. I did improve the commit log, please check below. The added detail is helpful to people (including ourselves, by the way) who might need to look at this commit some time in the future.
Thanks; patch looks good; I will try to put more efforts on commit log for future patches.
Quoted 8 lines
If you have an x86 system lying around, running rcutorture is quite straightforward. Non-x86 systems can also run rcutorture, if nothing else by using modprobe and rmmod as described here:
https://paulmck.livejournal.com/57769.html
The scripting described in the latter part of this document has worked on ARMv8 and PowerPC, and might still work for all I know.
I will set it up at my end; the livejournal is pretty detailed! Thanks for sharing this!
Thanks
Neeraj
Quoted 38 lines
Thanx, Paul
------------------------------------------------------------------------
commit
9482524d7dd0aea5d32a6efa2979223eea07c029Author: Neeraj Upadhyay <[email protected]> Date: Mon Jun 22 00:07:27 2020 +0530rcu/tree: Force quiescent state on callback overload
On callback overload, it is necessary to quickly detect idle CPUs, and
rcu_gp_fqs_check_wake()checks for this condition. Unfortunately, the code following the call to this function does not repeat this check, which means that in reality no actual quiescent-state forcing, instead only a couple of quick and pointless wakeups at the beginning of the grace period.This commit therefore adds a check for the
RCU_GP_FLAG_OVLDflag in the post-wakeup "if" statement inrcu_gp_fqs_loop().Signed-off-by Neeraj Upadhyay <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>kernel/rcu/tree.c +1
-1@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);
Signature · 1 line
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Quoted 6 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.Signed-off-by Neeraj Upadhyay <[email protected]>---
And I think needs fixes tag:
thanks,
- Joel
Quoted 20 lines
kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree.c +1
-1@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Paul E. McKenney
maintainerQuoted 5 lines · Joel Fernandes
Quoted 4 lines · Neeraj Upadhyay
On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including
RCU_GP_FLAG_OVLDflag, in fqsstart check.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Neeraj Upadhyay <[email protected]>---
Reviewed-by Joel Fernandes (Google) <[email protected]>And I think needs fixes tag:
Fixes 1fca4d12f4637 ("rcu: Expedite first two FQS scans under callback-overload conditions")
Excellent, both applied, thank you!
Thanx, Paul
Quoted 25 lines
thanks,
- Joel
Quoted 20 lines
kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree.c +1
-1@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)break ;/* If time for quiescent-state forcing, do it. */ if (!time_after (rcu_state.jiffies_force_qs, jiffies) || - (gf &RCU_GP_FLAG_FQS )) { + (gf & (RCU_GP_FLAG_FQS |RCU_GP_FLAG_OVLD ))) {trace_rcu_grace_period (rcu_state.name, rcu_state.gp_seq,TPS ("fqsstart" ));rcu_gp_fqs (first_gp_fqs);Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project