Skip to contentSkip to conversation

rcu/tree: Force quiescent state on callback overload

PATCH In review Neeraj Upadhyay 3 participants 9 messages Jun 21, 2020 – Jun 23, 2020 RCU

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rcu/tree: Force quiescent state on callback overload

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
 			break;
 		/* If time for quiescent-state forcing, do it. */
 		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
 			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
 					       TPS("fqsstart"));
 			rcu_gp_fqs(first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
maintainer
#1 lore raw
Quoted 5 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

Good catch!

But what did you do to verify that this change does the right thing?

Thanx, Paul

Quoted 21 lines
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
 			break;
 		/* If time for quiescent-state forcing, do it. */
 		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
 			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
 					       TPS("fqsstart"));
 			rcu_gp_fqs(first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#2 lore raw

Hi Paul,

Quoted 5 lines · Paul E. McKenney
Quoted 4 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

Quoted 8 lines
Quoted 1 line

Good catch!

But what did you do to verify that this change does the right thing?

Thanx, Paul

I haven't done a runtime verification of this code path; I posted this, based on review of this code.

Thanks
Neeraj

Quoted 21 lines
Quoted 21 lines
---
  kernel/rcu/tree.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
  			break;
  		/* If time for quiescent-state forcing, do it. */
  		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
  			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
  					       TPS("fqsstart"));
  			rcu_gp_fqs(first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
Signature · 1 line
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
maintainer
#3 lore raw
Quoted 12 lines · Neeraj Upadhyay

Hi Paul,

Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

Good catch!

Quoted 7 lines
Quoted 4 lines

But what did you do to verify that this change does the right thing?

Thanx, Paul

I haven't done a runtime verification of this code path; I posted this, based on review of this code.

My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.

Or am I missing something subtle that prevents this?

But yes, it does look like the current mainline code fails to do the first scan immediately, so again, good catch!

Thanx, Paul

Quoted 28 lines

Thanks
Neeraj

Quoted 21 lines
Quoted 21 lines
---
  kernel/rcu/tree.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
  			break;
  		/* If time for quiescent-state forcing, do it. */
  		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
  			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
  					       TPS("fqsstart"));
  			rcu_gp_fqs(first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of
the Code Aurora Forum, hosted by The Linux Foundation
#4 lore raw

Hi Paul,

Quoted 21 lines · Paul E. McKenney
Quoted 12 lines · Neeraj Upadhyay

Hi Paul,

Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

Good catch!

Quoted 7 lines
Quoted 4 lines

But what did you do to verify that this change does the right thing?

Thanx, Paul

I haven't done a runtime verification of this code path; I posted this, based on review of this code.

Quoted 6 lines

My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.

Or am I missing something subtle that prevents this?

Looks like under overload, only the first and second scans are accelerated?

    gf = 0;
    if (first_gp_fqs) {
         first_gp_fqs = false;
          gf = rcu_state.cbovld ? RCU_GP_FLAG_OVLD : 0;
    }

Thanks
Neeraj

Quoted 35 lines

But yes, it does look like the current mainline code fails to do the first scan immediately, so again, good catch!

Thanx, Paul

Quoted 28 lines

Thanks
Neeraj

Quoted 21 lines
Quoted 21 lines
---
   kernel/rcu/tree.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
   			break;
   		/* If time for quiescent-state forcing, do it. */
   		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
   			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
   					       TPS("fqsstart"));
   			rcu_gp_fqs(first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of
the Code Aurora Forum, hosted by The Linux Foundation
Signature · 1 line
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
maintainer
#5 lore raw
Quoted 31 lines · Neeraj Upadhyay

Hi Paul,

Quoted 21 lines · Paul E. McKenney
Quoted 12 lines · Neeraj Upadhyay

Hi Paul,

Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

Good catch!

Quoted 7 lines
Quoted 4 lines

But what did you do to verify that this change does the right thing?

Thanx, Paul

I haven't done a runtime verification of this code path; I posted this, based on review of this code.

Quoted 6 lines

My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.

Or am I missing something subtle that prevents this?

Quoted 7 lines

Looks like under overload, only the first and second scans are accelerated?

    gf = 0;
    if (first_gp_fqs) {
         first_gp_fqs = false;
          gf = rcu_state.cbovld ? RCU_GP_FLAG_OVLD : 0;
    }

Very good, it does sound like you understand this, and it matches my analysis and passes light testing, so I queued this one. I did improve the commit log, please check below. The added detail is helpful to people (including ourselves, by the way) who might need to look at this commit some time in the future.

If you have an x86 system lying around, running rcutorture is quite straightforward. Non-x86 systems can also run rcutorture, if nothing else by using modprobe and rmmod as described here:

https://paulmck.livejournal.com/57769.html

The scripting described in the latter part of this document has worked on ARMv8 and PowerPC, and might still work for all I know.

Thanx, Paul

------------------------------------------------------------------------

commit 9482524d7dd0aea5d32a6efa2979223eea07c029

Author: Neeraj Upadhyay <[email protected]>
Date:   Mon Jun 22 00:07:27 2020 +0530

rcu/tree: Force quiescent state on callback overload

On callback overload, it is necessary to quickly detect idle CPUs, and rcu_gp_fqs_check_wake() checks for this condition. Unfortunately, the code following the call to this function does not repeat this check, which means that in reality no actual quiescent-state forcing, instead only a couple of quick and pointless wakeups at the beginning of the grace period.

This commit therefore adds a check for the RCU_GP_FLAG_OVLD flag in the post-wakeup "if" statement in rcu_gp_fqs_loop().

kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
 			break;
 		/* If time for quiescent-state forcing, do it. */
 		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
 			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
 					       TPS("fqsstart"));
 			rcu_gp_fqs(first_gp_fqs);
#6 lore raw

Hi Paul,

Quoted 40 lines · Paul E. McKenney
Quoted 31 lines · Neeraj Upadhyay

Hi Paul,

Quoted 21 lines · Paul E. McKenney
Quoted 12 lines · Neeraj Upadhyay

Hi Paul,

Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

Good catch!

Quoted 7 lines
Quoted 4 lines

But what did you do to verify that this change does the right thing?

Thanx, Paul

I haven't done a runtime verification of this code path; I posted this, based on review of this code.

Quoted 6 lines

My concern is that under overload, the FQS scans would happen continuously rather than accelerating only the first such scan in a given grace period. This would of course result in a CPU-bound grace-period kthread, which users might not be all that happy with.

Or am I missing something subtle that prevents this?

Quoted 7 lines

Looks like under overload, only the first and second scans are accelerated?

     gf = 0;
     if (first_gp_fqs) {
          first_gp_fqs = false;
           gf = rcu_state.cbovld ? RCU_GP_FLAG_OVLD : 0;
     }
Quoted 6 lines

Very good, it does sound like you understand this, and it matches my analysis and passes light testing, so I queued this one. I did improve the commit log, please check below. The added detail is helpful to people (including ourselves, by the way) who might need to look at this commit some time in the future.

Thanks; patch looks good; I will try to put more efforts on commit log for future patches.

Quoted 8 lines

If you have an x86 system lying around, running rcutorture is quite straightforward. Non-x86 systems can also run rcutorture, if nothing else by using modprobe and rmmod as described here:

https://paulmck.livejournal.com/57769.html

The scripting described in the latter part of this document has worked on ARMv8 and PowerPC, and might still work for all I know.

I will set it up at my end; the livejournal is pretty detailed! Thanks for sharing this!

Thanks
Neeraj

Quoted 38 lines

Thanx, Paul

------------------------------------------------------------------------

commit 9482524d7dd0aea5d32a6efa2979223eea07c029

Author: Neeraj Upadhyay <[email protected]>
Date:   Mon Jun 22 00:07:27 2020 +0530

rcu/tree: Force quiescent state on callback overload

On callback overload, it is necessary to quickly detect idle CPUs, and rcu_gp_fqs_check_wake() checks for this condition. Unfortunately, the code following the call to this function does not repeat this check, which means that in reality no actual quiescent-state forcing, instead only a couple of quick and pointless wakeups at the beginning of the grace period.

This commit therefore adds a check for the RCU_GP_FLAG_OVLD flag in the post-wakeup "if" statement in rcu_gp_fqs_loop().

kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
  			break;
  		/* If time for quiescent-state forcing, do it. */
  		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
  			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
  					       TPS("fqsstart"));
  			rcu_gp_fqs(first_gp_fqs);
Signature · 1 line
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Reviewed-by on v1 1/1
#7 lore raw
In reply to Neeraj Upadhyay
Quoted 6 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

---

And I think needs fixes tag:

Fixes 1fca4d12f4637 ("rcu: Expedite first two FQS scans under callback-overload conditions")

thanks,

- Joel

Quoted 20 lines
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
 			break;
 		/* If time for quiescent-state forcing, do it. */
 		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
 			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
 					       TPS("fqsstart"));
 			rcu_gp_fqs(first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
maintainer
#8 lore raw
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Neeraj Upadhyay

On callback overload, we want to force quiescent state immediately, for the first and second fqs. Enforce the same, by including RCU_GP_FLAG_OVLD flag, in fqsstart check.

Quoted 7 lines
Quoted 2 lines

---

And I think needs fixes tag:

Fixes 1fca4d12f4637 ("rcu: Expedite first two FQS scans under callback-overload conditions")

Excellent, both applied, thank you!

Thanx, Paul

Quoted 25 lines

thanks,

- Joel

Quoted 20 lines
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index d0988a1..6226bfb 100644
@@ -1865,7 +1865,7 @@ static void rcu_gp_fqs_loop(void)
 			break;
 		/* If time for quiescent-state forcing, do it. */
 		if (!time_after(rcu_state.jiffies_force_qs, jiffies) ||
-		    (gf & RCU_GP_FLAG_FQS)) {
+		    (gf & (RCU_GP_FLAG_FQS | RCU_GP_FLAG_OVLD))) {
 			trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
 					       TPS("fqsstart"));
 			rcu_gp_fqs(first_gp_fqs);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project