Skip to contentSkip to conversation

rcu: simplify the calculation of rcu_state.ncpus

PATCH Superseded Wei Yang 2 participants 5 messages Apr 17, 2020 – Apr 19, 2020 RCU

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Wei Yang

author
#0 lore raw

PATCH 1/1 rcu: simplify the calculation of rcu_state.ncpus

There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.

Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.

BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.

---
 kernel/rcu/tree.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +2 -4
index d91c9156fab2..f0d9251fa663 100644
@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu)
 {
 	unsigned long flags;
 	unsigned long mask;
-	int nbits;
 	unsigned long oldmask;
 	struct rcu_data *rdp;
 	struct rcu_node *rnp;
@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu)
 	rnp->qsmaskinitnext |= mask;
 	oldmask = rnp->expmaskinitnext;
 	rnp->expmaskinitnext |= mask;
-	oldmask ^= rnp->expmaskinitnext;
-	nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
 	/* Allow lockless access for expedited grace periods. */
-	smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
+	if (!(oldmask & mask))
+		smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + 1); /* ^^^ */
 	rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
 	rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
 	rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Signature · 1 line
2.23.0
maintainer
#1 lore raw
Quoted 6 lines · Wei Yang

There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.

Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.

Nice!!! Good eyes!

Quoted 2 lines

BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.

The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like smp_store_release() are supposed to have comments indicating what they are ordering. ;-)

Could you please do the following things and resubmit?

1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

2. Given that oldmask is used only to test to see if a new bit

	was set, why not just replace oldmask with a bool variable
	that is set to "!(rnp->expmaskinitnext & mask)" before the
	bit is ORed into rnp->expmaskinitnext?

3. Put the comment inside the "if" statement with the
smp_store_release().

4. In -rcu, you will find a ASSERT_EXCLUSIVE_WRITER() statement
that should also be placed inside the "if" statement with the smp_store_release().

Thanx, Paul

Quoted 33 lines
---
 kernel/rcu/tree.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +2 -4
index d91c9156fab2..f0d9251fa663 100644
@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu)
 {
 	unsigned long flags;
 	unsigned long mask;
-	int nbits;
 	unsigned long oldmask;
 	struct rcu_data *rdp;
 	struct rcu_node *rnp;
@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu)
 	rnp->qsmaskinitnext |= mask;
 	oldmask = rnp->expmaskinitnext;
 	rnp->expmaskinitnext |= mask;
-	oldmask ^= rnp->expmaskinitnext;
-	nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
 	/* Allow lockless access for expedited grace periods. */
-	smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
+	if (!(oldmask & mask))
+		smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + 1); /* ^^^ */
 	rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
 	rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
 	rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Signature · 1 line
2.23.0
author
#2 lore raw
Quoted 28 lines · Paul E. McKenney
Quoted 6 lines · Wei Yang

There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.

Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.

Nice!!! Good eyes!

Quoted 2 lines

BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.

The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like smp_store_release() are supposed to have comments indicating what they are ordering. ;-)

Could you please do the following things and resubmit?

1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

2. Given that oldmask is used only to test to see if a new bit

        was set, why not just replace oldmask with a bool variable
        that is set to "!(rnp->expmaskinitnext & mask)" before the
        bit is ORed into rnp->expmaskinitnext?
Quoted 7 lines

3. Put the comment inside the "if" statement with the
smp_store_release().

4. In -rcu, you will find a ASSERT_EXCLUSIVE_WRITER() statement
that should also be placed inside the "if" statement with the smp_store_release().

Oops, my email client EAT this mail. Hope this mail will not be banned.

I adjust the code a little according to your suggestion like below. Is this what you expected?

kernel/rcu/tree.c +5 -8
index f288477ee1c2..f01367a80b70 100644
@@ -3732,10 +3732,9 @@ void rcu_cpu_starting(unsigned int cpu)
 {
        unsigned long flags;
        unsigned long mask;
-       int nbits;
-       unsigned long oldmask;
        struct rcu_data *rdp;
        struct rcu_node *rnp;
+       bool has_seen;
 
        if (per_cpu(rcu_cpu_started, cpu))
                return;
@@ -3747,13 +3746,13 @@ void rcu_cpu_starting(unsigned int cpu)
        mask = rdp->grpmask;
        raw_spin_lock_irqsave_rcu_node(rnp, flags);
        WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask);
-       oldmask = rnp->expmaskinitnext;
+       has_seen = rnp->expmaskinitnext & mask;
        rnp->expmaskinitnext |= mask;
-       oldmask ^= rnp->expmaskinitnext;
-       nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
-       /* Allow lockless access for expedited grace periods. */
-       smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
-       ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
+       if (!has_seen) {
+               /* Allow lockless access for expedited grace periods. */
+               smp_store_release(&rcu_state.ncpus, rcu_state.ncpus +

1); /* ^^^ */
+ ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
+ }

        rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
        rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
        rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Quoted 35 lines

Thanx, Paul

Quoted 33 lines
---
 kernel/rcu/tree.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +2 -5
index d91c9156fab2..f0d9251fa663 100644
@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu)
 {
      unsigned long flags;
      unsigned long mask;
-     int nbits;
      unsigned long oldmask;
      struct rcu_data *rdp;
      struct rcu_node *rnp;
@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu)
      rnp->qsmaskinitnext |= mask;
      oldmask = rnp->expmaskinitnext;
      rnp->expmaskinitnext |= mask;
-     oldmask ^= rnp->expmaskinitnext;
-     nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
      /* Allow lockless access for expedited grace periods. */
-     smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
+     if (!(oldmask & mask))
+             smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + 1); /* ^^^ */
      rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
      rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
      rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
--

2.23.0

maintainer
#3 lore raw
Quoted 32 lines · Wei Yang
Quoted 31 lines · Paul E. McKenney
Quoted 6 lines · Wei Yang

There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.

Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.

Nice!!! Good eyes!

Quoted 2 lines

BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.

The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like smp_store_release() are supposed to have comments indicating what they are ordering. ;-)

Could you please do the following things and resubmit?

1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

2. Given that oldmask is used only to test to see if a new bit

        was set, why not just replace oldmask with a bool variable
        that is set to "!(rnp->expmaskinitnext & mask)" before the
        bit is ORed into rnp->expmaskinitnext?

3. Put the comment inside the "if" statement with the
smp_store_release().

Quoted 6 lines
Quoted 4 lines

4. In -rcu, you will find a ASSERT_EXCLUSIVE_WRITER() statement
that should also be placed inside the "if" statement with the smp_store_release().

Oops, my email client EAT this mail. Hope this mail will not be banned.

This one made it. ;-)

Quoted 2 lines

I adjust the code a little according to your suggestion like below. Is this what you expected?

Much better, thank you!

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3737

Quoted 13 lines
kernel/rcu/tree.c +1 -2
index f288477ee1c2..f01367a80b70 100644
@@ -3732,10 +3732,9 @@ void rcu_cpu_starting(unsigned int cpu)
 {
        unsigned long flags;
        unsigned long mask;
-       int nbits;
-       unsigned long oldmask;
        struct rcu_data *rdp;
        struct rcu_node *rnp;
+       bool has_seen;

Could you please use a more descriptive name, perhaps something like "newcpu"? Yes, "has_seen" does make sense, but it leaves the reader the question "has seen what?".

Quoted 17 lines
kernel/rcu/tree.c +4 -6
        if (per_cpu(rcu_cpu_started, cpu))
                return;
@@ -3747,13 +3746,13 @@ void rcu_cpu_starting(unsigned int cpu)
        mask = rdp->grpmask;
        raw_spin_lock_irqsave_rcu_node(rnp, flags);
        WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask);
-       oldmask = rnp->expmaskinitnext;
+       has_seen = rnp->expmaskinitnext & mask;
        rnp->expmaskinitnext |= mask;
-       oldmask ^= rnp->expmaskinitnext;
-       nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
-       /* Allow lockless access for expedited grace periods. */
-       smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
-       ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
+       if (!has_seen) {
+               /* Allow lockless access for expedited grace periods. */
+               smp_store_release(&rcu_state.ncpus, rcu_state.ncpus +
Quoted 2 lines

1); /* ^^^ */
+ ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);

One advantage of flipping the conditional is that it allows dropping the "if" statement, which saves a few lines of code and gets closer to your original suggestion:

	newcpu = !(rnp->expmaskinitnext & mask);
	rnp->expmaskinitnext |= mask;
	/* Allow lockless access for expedited grace periods. */
	smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + newcpu); /* ^^^ */
	ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);

Yes, this does execute a store-release that doesn't change anything, but this is OK because (1) CPU hotplug is infrequent and (2) CPU hotplug has very high overhead. The overhead of the store is thus neglible. (But please do not try this to heavily contended cachelines on fastpaths!

And yes, I should have thought of this in response to your initial patch, but then again, I should have thought of this when writing this code in the first place, shouldn't I have? ;-)

Thanx, Paul

Quoted 40 lines

+ }

        rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
        rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
        rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Quoted 35 lines

Thanx, Paul

Quoted 33 lines
---
 kernel/rcu/tree.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +2 -5
index d91c9156fab2..f0d9251fa663 100644
@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu)
 {
      unsigned long flags;
      unsigned long mask;
-     int nbits;
      unsigned long oldmask;
      struct rcu_data *rdp;
      struct rcu_node *rnp;
@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu)
      rnp->qsmaskinitnext |= mask;
      oldmask = rnp->expmaskinitnext;
      rnp->expmaskinitnext |= mask;
-     oldmask ^= rnp->expmaskinitnext;
-     nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
      /* Allow lockless access for expedited grace periods. */
-     smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
+     if (!(oldmask & mask))
+             smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + 1); /* ^^^ */
      rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
      rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
      rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
--

2.23.0

author
#4 lore raw
Quoted 95 lines · Paul E. McKenney
Quoted 32 lines · Wei Yang
Quoted 31 lines · Paul E. McKenney
Quoted 6 lines · Wei Yang

There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.

Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.

Nice!!! Good eyes!

Quoted 2 lines

BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.

The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like smp_store_release() are supposed to have comments indicating what they are ordering. ;-)

Could you please do the following things and resubmit?

1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

2. Given that oldmask is used only to test to see if a new bit

        was set, why not just replace oldmask with a bool variable
        that is set to "!(rnp->expmaskinitnext & mask)" before the
        bit is ORed into rnp->expmaskinitnext?

3. Put the comment inside the "if" statement with the
smp_store_release().

Quoted 6 lines
Quoted 4 lines

4. In -rcu, you will find a ASSERT_EXCLUSIVE_WRITER() statement
that should also be placed inside the "if" statement with the smp_store_release().

Oops, my email client EAT this mail. Hope this mail will not be banned.

This one made it. ;-)

Quoted 2 lines

I adjust the code a little according to your suggestion like below. Is this what you expected?

Much better, thank you!

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3737

Quoted 13 lines
kernel/rcu/tree.c +1 -2
index f288477ee1c2..f01367a80b70 100644
@@ -3732,10 +3732,9 @@ void rcu_cpu_starting(unsigned int cpu)
 {
        unsigned long flags;
        unsigned long mask;
-       int nbits;
-       unsigned long oldmask;
        struct rcu_data *rdp;
        struct rcu_node *rnp;
+       bool has_seen;

Could you please use a more descriptive name, perhaps something like "newcpu"? Yes, "has_seen" does make sense, but it leaves the reader the question "has seen what?".

Quoted 17 lines
kernel/rcu/tree.c +4 -6
        if (per_cpu(rcu_cpu_started, cpu))
                return;
@@ -3747,13 +3746,13 @@ void rcu_cpu_starting(unsigned int cpu)
        mask = rdp->grpmask;
        raw_spin_lock_irqsave_rcu_node(rnp, flags);
        WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask);
-       oldmask = rnp->expmaskinitnext;
+       has_seen = rnp->expmaskinitnext & mask;
        rnp->expmaskinitnext |= mask;
-       oldmask ^= rnp->expmaskinitnext;
-       nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
-       /* Allow lockless access for expedited grace periods. */
-       smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
-       ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
+       if (!has_seen) {
+               /* Allow lockless access for expedited grace periods. */
+               smp_store_release(&rcu_state.ncpus, rcu_state.ncpus +
Quoted 2 lines

1); /* ^^^ */
+ ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);

One advantage of flipping the conditional is that it allows dropping the "if" statement, which saves a few lines of code and gets closer to your original suggestion:

	newcpu = !(rnp->expmaskinitnext & mask);
	rnp->expmaskinitnext |= mask;
	/* Allow lockless access for expedited grace periods. */
	smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + newcpu); /* ^^^ */
	ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
Quoted 4 lines

Yes, this does execute a store-release that doesn't change anything, but this is OK because (1) CPU hotplug is infrequent and (2) CPU hotplug has very high overhead. The overhead of the store is thus neglible. (But please do not try this to heavily contended cachelines on fastpaths!

I thought I got your point.

Quoted 5 lines

And yes, I should have thought of this in response to your initial patch, but then again, I should have thought of this when writing this code in the first place, shouldn't I have? ;-)

Faire enough.

Quoted 42 lines

Thanx, Paul

Quoted 40 lines

+ }

        rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
        rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
        rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Quoted 35 lines

Thanx, Paul

Quoted 33 lines
---
 kernel/rcu/tree.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +2 -5
index d91c9156fab2..f0d9251fa663 100644
@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu)
 {
      unsigned long flags;
      unsigned long mask;
-     int nbits;
      unsigned long oldmask;
      struct rcu_data *rdp;
      struct rcu_node *rnp;
@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu)
      rnp->qsmaskinitnext |= mask;
      oldmask = rnp->expmaskinitnext;
      rnp->expmaskinitnext |= mask;
-     oldmask ^= rnp->expmaskinitnext;
-     nbits = bitmap_weight(&oldmask, BITS_PER_LONG);
      /* Allow lockless access for expedited grace periods. */
-     smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + nbits); /* ^^^ */
+     if (!(oldmask & mask))
+             smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + 1); /* ^^^ */
      rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
      rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
      rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
--

2.23.0

Signature · 2 lines
Wei Yang
Help you, Help me