rcu: simplify the calculation of rcu_state.ncpus
PATCH 1/1 rcu: simplify the calculation of rcu_state.ncpus
There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.
Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.
BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.
--- kernel/rcu/tree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +2 -4
@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits;unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp;
@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu) rnp->qsmaskinitnext |= mask; oldmask = rnp->expmaskinitnext; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG );/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ +if (!(oldmask & mask)) +smp_store_release (&rcu_state.ncpus, rcu_state.ncpus +1 );/* ^^^ */ rcu_gpnum_ovf (rnp, rdp);/* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq =READ_ONCE (rcu_state.gp_seq); rdp->rcu_onl_gp_flags =READ_ONCE (rcu_state.gp_flags);
Signature · 1 line
2.23.0
Paul E. McKenney
maintainerQuoted 6 lines · Wei Yang
There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.
Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.
Nice!!! Good eyes!
Quoted 2 lines
BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.
The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like smp_store_release() are supposed to have comments indicating what they are ordering. ;-)
Could you please do the following things and resubmit?
1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
2. Given that oldmask is used only to test to see if a new bit
was set, why not just replace oldmask with a bool variable that is set to "!(rnp->expmaskinitnext & mask)" before the bit is ORed into rnp->expmaskinitnext?
3. Put the comment inside the "if" statement with the
smp_store_release().
4. In -rcu, you will find a ASSERT_EXCLUSIVE_WRITER() statement
that should also be placed inside the "if" statement with the smp_store_release().
Thanx, Paul
Quoted 33 lines
Signed-off-by Wei Yang <[email protected]>--- kernel/rcu/tree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)kernel/rcu/tree.c +2
-4@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits;unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp;@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu) rnp->qsmaskinitnext |= mask; oldmask = rnp->expmaskinitnext; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG );/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ +if (!(oldmask & mask)) +smp_store_release (&rcu_state.ncpus, rcu_state.ncpus +1 );/* ^^^ */ rcu_gpnum_ovf (rnp, rdp);/* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq =READ_ONCE (rcu_state.gp_seq); rdp->rcu_onl_gp_flags =READ_ONCE (rcu_state.gp_flags);Signature · 1 line
2.23.0
Quoted 28 lines · Paul E. McKenney
Quoted 6 lines · Wei Yang
There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.
Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.
Nice!!! Good eyes!
Quoted 2 lines
BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.
The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like
smp_store_release()are supposed to have comments indicating what they are ordering. ;-)Could you please do the following things and resubmit?
1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git2. Given that oldmask is used only to test to see if a new bit
was set, why not just replace oldmask with a bool variable that is set to "!(rnp->expmaskinitnext & mask)" before the bit is ORed into rnp->expmaskinitnext?
Quoted 7 lines
3. Put the comment inside the "if" statement with the
smp_store_release().4. In -rcu, you will find a
ASSERT_EXCLUSIVE_WRITER()statement
that should also be placed inside the "if" statement with thesmp_store_release().
Oops, my email client EAT this mail. Hope this mail will not be banned.
I adjust the code a little according to your suggestion like below. Is this what you expected?
kernel/rcu/tree.c +5 -8
@@ -3732,10 +3732,9 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits; -unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp; +bool has_seen;if (per_cpu (rcu_cpu_started, cpu))return ;
@@ -3747,13 +3746,13 @@ void rcu_cpu_starting(unsigned int cpu) mask = rdp->grpmask;raw_spin_lock_irqsave_rcu_node (rnp, flags);WRITE_ONCE (rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask); - oldmask = rnp->expmaskinitnext; + has_seen = rnp->expmaskinitnext & mask; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG ); -/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ -ASSERT_EXCLUSIVE_WRITER (rcu_state.ncpus); +if (!has_seen) { +/* Allow lockless access for expedited grace periods. */ + smp_store_release(&rcu_state.ncpus, rcu_state.ncpus +
1); /* ^^^ */
+ ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
+ }
rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);Quoted 35 lines
Thanx, Paul
Quoted 33 lines
Signed-off-by Wei Yang <[email protected]>--- kernel/rcu/tree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)kernel/rcu/tree.c +2
-5@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits;unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp;@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu) rnp->qsmaskinitnext |= mask; oldmask = rnp->expmaskinitnext; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG );/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ +if (!(oldmask & mask)) +smp_store_release (&rcu_state.ncpus, rcu_state.ncpus +1 );/* ^^^ */ rcu_gpnum_ovf (rnp, rdp);/* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq =READ_ONCE (rcu_state.gp_seq); rdp->rcu_onl_gp_flags =READ_ONCE (rcu_state.gp_flags); --2.23.0
Paul E. McKenney
maintainerQuoted 32 lines · Wei Yang
Quoted 31 lines · Paul E. McKenney
Quoted 6 lines · Wei Yang
There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.
Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.
Nice!!! Good eyes!
Quoted 2 lines
BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.
The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like
smp_store_release()are supposed to have comments indicating what they are ordering. ;-)Could you please do the following things and resubmit?
1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git2. Given that oldmask is used only to test to see if a new bit
was set, why not just replace oldmask with a bool variable that is set to "!(rnp->expmaskinitnext & mask)" before the bit is ORed into rnp->expmaskinitnext?3. Put the comment inside the "if" statement with the
smp_store_release().
Quoted 6 lines
Quoted 4 lines
4. In -rcu, you will find a
ASSERT_EXCLUSIVE_WRITER()statement
that should also be placed inside the "if" statement with thesmp_store_release().Oops, my email client EAT this mail. Hope this mail will not be banned.
This one made it. ;-)
Quoted 2 lines
I adjust the code a little according to your suggestion like below. Is this what you expected?
Much better, thank you!
kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3737
Quoted 13 lines
kernel/rcu/tree.c +1
-2@@ -3732,10 +3732,9 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits; -unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp; +bool has_seen;
Could you please use a more descriptive name, perhaps something like "newcpu"? Yes, "has_seen" does make sense, but it leaves the reader the question "has seen what?".
Quoted 17 lines
kernel/rcu/tree.c +4
-6if (per_cpu (rcu_cpu_started, cpu))return ;@@ -3747,13 +3746,13 @@ void rcu_cpu_starting(unsigned int cpu) mask = rdp->grpmask;raw_spin_lock_irqsave_rcu_node (rnp, flags);WRITE_ONCE (rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask); - oldmask = rnp->expmaskinitnext; + has_seen = rnp->expmaskinitnext & mask; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG ); -/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ -ASSERT_EXCLUSIVE_WRITER (rcu_state.ncpus); +if (!has_seen) { +/* Allow lockless access for expedited grace periods. */ + smp_store_release(&rcu_state.ncpus, rcu_state.ncpus +
Quoted 2 lines
1); /* ^^^ */
+ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
One advantage of flipping the conditional is that it allows dropping the "if" statement, which saves a few lines of code and gets closer to your original suggestion:
newcpu = !(rnp->expmaskinitnext & mask); rnp->expmaskinitnext |= mask; /* Allow lockless access for expedited grace periods. */ smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + newcpu); /* ^^^ */ ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
Yes, this does execute a store-release that doesn't change anything, but this is OK because (1) CPU hotplug is infrequent and (2) CPU hotplug has very high overhead. The overhead of the store is thus neglible. (But please do not try this to heavily contended cachelines on fastpaths!
And yes, I should have thought of this in response to your initial patch, but then again, I should have thought of this when writing this code in the first place, shouldn't I have? ;-)
Thanx, Paul
Quoted 40 lines
+ }
rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq); rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);Quoted 35 lines
Thanx, Paul
Quoted 33 lines
Signed-off-by Wei Yang <[email protected]>--- kernel/rcu/tree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)kernel/rcu/tree.c +2
-5@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits;unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp;@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu) rnp->qsmaskinitnext |= mask; oldmask = rnp->expmaskinitnext; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG );/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ +if (!(oldmask & mask)) +smp_store_release (&rcu_state.ncpus, rcu_state.ncpus +1 );/* ^^^ */ rcu_gpnum_ovf (rnp, rdp);/* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq =READ_ONCE (rcu_state.gp_seq); rdp->rcu_onl_gp_flags =READ_ONCE (rcu_state.gp_flags); --2.23.0
Quoted 95 lines · Paul E. McKenney
Quoted 32 lines · Wei Yang
Quoted 31 lines · Paul E. McKenney
Quoted 6 lines · Wei Yang
There is only 1 bit set in mask, which means the difference between oldmask and the new one would be at the position where the bit is set in mask.
Based on this knowledge, rcu_state.ncpus could be calculated by checking whether mask is already set in oldmask.
Nice!!! Good eyes!
Quoted 2 lines
BTW, the comment at the last of this line is mysterious. Not sure it could be removed or not.
The "^^^" in that comment says to look at the comment on the preceding line. Memory-ordering functions like
smp_store_release()are supposed to have comments indicating what they are ordering. ;-)Could you please do the following things and resubmit?
1. Forward-port to -rcu branch dev? This tree lives here:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git2. Given that oldmask is used only to test to see if a new bit
was set, why not just replace oldmask with a bool variable that is set to "!(rnp->expmaskinitnext & mask)" before the bit is ORed into rnp->expmaskinitnext?3. Put the comment inside the "if" statement with the
smp_store_release().Quoted 6 lines
Quoted 4 lines
4. In -rcu, you will find a
ASSERT_EXCLUSIVE_WRITER()statement
that should also be placed inside the "if" statement with thesmp_store_release().Oops, my email client EAT this mail. Hope this mail will not be banned.
This one made it. ;-)
Quoted 2 lines
I adjust the code a little according to your suggestion like below. Is this what you expected?
Much better, thank you!
kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3737
Quoted 13 lines
kernel/rcu/tree.c +1
-2@@ -3732,10 +3732,9 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits; -unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp; +bool has_seen;Could you please use a more descriptive name, perhaps something like "newcpu"? Yes, "
has_seen" does make sense, but it leaves the reader the question "has seen what?".Quoted 17 lines
kernel/rcu/tree.c +4
-6if (per_cpu (rcu_cpu_started, cpu))return ;@@ -3747,13 +3746,13 @@ void rcu_cpu_starting(unsigned int cpu) mask = rdp->grpmask;raw_spin_lock_irqsave_rcu_node (rnp, flags);WRITE_ONCE (rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask); - oldmask = rnp->expmaskinitnext; + has_seen = rnp->expmaskinitnext & mask; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG ); -/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ -ASSERT_EXCLUSIVE_WRITER (rcu_state.ncpus); +if (!has_seen) { +/* Allow lockless access for expedited grace periods. */ + smp_store_release(&rcu_state.ncpus, rcu_state.ncpus +Quoted 2 lines
1); /* ^^^ */
+ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);One advantage of flipping the conditional is that it allows dropping the "if" statement, which saves a few lines of code and gets closer to your original suggestion:
newcpu = !(rnp->expmaskinitnext & mask); rnp->expmaskinitnext |= mask; /* Allow lockless access for expedited grace periods. */ smp_store_release(&rcu_state.ncpus, rcu_state.ncpus + newcpu); /* ^^^ */ ASSERT_EXCLUSIVE_WRITER(rcu_state.ncpus);
Quoted 4 lines
Yes, this does execute a store-release that doesn't change anything, but this is OK because (1) CPU hotplug is infrequent and (2) CPU hotplug has very high overhead. The overhead of the store is thus neglible. (But please do not try this to heavily contended cachelines on fastpaths!
I thought I got your point.
Quoted 5 lines
And yes, I should have thought of this in response to your initial patch, but then again, I should have thought of this when writing this code in the first place, shouldn't I have? ;-)
Faire enough.
Quoted 42 lines
Thanx, Paul
Quoted 40 lines
+ }
rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq); rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);Quoted 35 lines
Thanx, Paul
Quoted 33 lines
Signed-off-by Wei Yang <[email protected]>--- kernel/rcu/tree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)kernel/rcu/tree.c +2
-5@@ -3364,7 +3364,6 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; -int nbits;unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp;@@ -3381,10 +3380,9 @@ void rcu_cpu_starting(unsigned int cpu) rnp->qsmaskinitnext |= mask; oldmask = rnp->expmaskinitnext; rnp->expmaskinitnext |= mask; - oldmask ^= rnp->expmaskinitnext; - nbits =bitmap_weight (&oldmask,BITS_PER_LONG );/* Allow lockless access for expedited grace periods. */ -smp_store_release (&rcu_state.ncpus, rcu_state.ncpus + nbits);/* ^^^ */ +if (!(oldmask & mask)) +smp_store_release (&rcu_state.ncpus, rcu_state.ncpus +1 );/* ^^^ */ rcu_gpnum_ovf (rnp, rdp);/* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq =READ_ONCE (rcu_state.gp_seq); rdp->rcu_onl_gp_flags =READ_ONCE (rcu_state.gp_flags); --2.23.0
Signature · 2 lines
Wei Yang Help you, Help me