Skip to contentSkip to conversation

percpu_counter: only disable preemption if needed in add_unless_lt()

PATCH In review Alex Elder 2 participants 3 messages Dec 23, 2010 – Dec 29, 2010

Subsystems (2)
lore
Share & tools
lore

Alex Elder

author
#0 lore raw

PATCH 5/5 percpu_counter: only disable preemption if needed in add_unless_lt()

In __percpu_counter_add_unless_lt() we don't need to disable preemption unless we're manipulating a per-cpu variable. That only happens in a limited case, so narrow the scope of that preemption to surround that case. This makes the "out" label rather unnecessary, so replace a couple "goto out" calls to just return.

---
 lib/percpu_counter.c |   21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)
lib/percpu_counter.c +10 -11
Index: b/lib/percpu_counter.c
===================================================================
@@ -232,8 +232,6 @@ int __percpu_counter_add_unless_lt(struc
 	int	cpu;
 	int	ret = -1;
 
-	preempt_disable();
-
 	/*
 	 * Check to see if rough count will be sufficient for
 	 * comparison.  First, if the upper bound is too low,
@@ -241,7 +239,7 @@ int __percpu_counter_add_unless_lt(struc
 	 */
 	count = percpu_counter_read(fbc);
 	if (count + error + amount < threshold)
-		goto out;
+		return -1;
 
 	/*
 	 * Next, if the lower bound is above the threshold, we can
@@ -251,12 +249,15 @@ int __percpu_counter_add_unless_lt(struc
 	if (count - error + amount > threshold) {
 		s32 *pcount = this_cpu_ptr(fbc->counters);
 
+		preempt_disable();
+		pcount = this_cpu_ptr(fbc->counters);
 		count = *pcount + amount;
 		if (abs(count) < batch) {
 			*pcount = count;
-			ret = 1;
-			goto out;
+			preempt_enable();
+			return 1;
 		}
+		preempt_enable();
 	}
 
 	/*
@@ -281,10 +282,9 @@ int __percpu_counter_add_unless_lt(struc
 	}
 
 	/*
-	 * Result is withing the error margin. Run an open-coded sum of the
-	 * per-cpu counters to get the exact value at this point in time,
-	 * and if the result greater than the threshold, add the amount to
-	 * the global counter.
+	 * Now add in all the per-cpu counters to compute the exact
+	 * value at this point in time, and if the result greater
+	 * than the threshold, add the amount to the global counter.
 	 */
 	count = fbc->count;
 	for_each_online_cpu(cpu) {
@@ -301,8 +301,7 @@ int __percpu_counter_add_unless_lt(struc
 	}
 out_unlock:
 	spin_unlock(&fbc->lock);
-out:
-	preempt_enable();
+
 	return ret;
 }
 EXPORT_SYMBOL(__percpu_counter_add_unless_lt);

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 40 lines · Alex Elder

In __percpu_counter_add_unless_lt() we don't need to disable preemption unless we're manipulating a per-cpu variable. That only happens in a limited case, so narrow the scope of that preemption to surround that case. This makes the "out" label rather unnecessary, so replace a couple "goto out" calls to just return.

---
 lib/percpu_counter.c |   21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)

Index: b/lib/percpu_counter.c
===================================================================

lib/percpu_counter.c +3 -3
@@ -232,8 +232,6 @@ int __percpu_counter_add_unless_lt(struc
 	int	cpu;
 	int	ret = -1;
 
-	preempt_disable();
-
 	/*
 	 * Check to see if rough count will be sufficient for
 	 * comparison.  First, if the upper bound is too low,
@@ -241,7 +239,7 @@ int __percpu_counter_add_unless_lt(struc
 	 */
 	count = percpu_counter_read(fbc);
 	if (count + error + amount < threshold)
-		goto out;
+		return -1;
 
 	/*
 	 * Next, if the lower bound is above the threshold, we can
@@ -251,12 +249,15 @@ int __percpu_counter_add_unless_lt(struc
 	if (count - error + amount > threshold) {
 		s32 *pcount = this_cpu_ptr(fbc->counters);
 
+		preempt_disable();
+		pcount = this_cpu_ptr(fbc->counters);

lib/percpu_counter.c int __percpu_counter_add_unless_lt(struc · line 261

Quoted 10 lines
lib/percpu_counter.c +3 -2
 		count = *pcount + amount;
 		if (abs(count) < batch) {
 			*pcount = count;
-			ret = 1;
-			goto out;
+			preempt_enable();
+			return 1;
 		}
+		preempt_enable();
 	}

Regardless of the other changes, this is not valid. That is:

	amount = -1;
	count = fbc->count;
	.....

	<get preempted>

	<other operations may significantly change fbc->count (i.e
	lots more than error will catch), so the current value of
	count in this context is wrong and cannot be trusted>

	<start running again>

	if (count - error + amount > threshold) {
		<not valid to run this lockless optimisation based
		on a stale count value>
		
		....
	}

Effectively, if we want to be able to use lockless optimisations, we need to ensure that the value of the global counter that we read remains within the given error bounds until we have finished making the lockless modification. That is done via disabling preemption across the entire function...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#2 lore raw
Quoted 6 lines · Dave Chinner
Quoted 5 lines · Alex Elder

In __percpu_counter_add_unless_lt() we don't need to disable preemption unless we're manipulating a per-cpu variable. That only happens in a limited case, so narrow the scope of that preemption to surround that case. This makes the "out" label rather unnecessary, so replace a couple "goto out" calls to just return.

. . .

Quoted 2 lines

Regardless of the other changes, this is not valid. That is:

You're right. I was thinking about updates to fbc->count being protected by the spinlock, but that doesn't address the cached value getting stale if this CPU gets preempted and another thread passes through this code before the first one gets resumed.

I'm also looking at the other patches and your responses and will be done with it today. I don't want to hold up your pull request any longer.

If you found anything of value in the little series I posted feel free to incorporate it into your own changes.

-Alex

Quoted 25 lines
	amount = -1;
	count = fbc->count;
	.....

	<get preempted>

	<other operations may significantly change fbc->count (i.e
	lots more than error will catch), so the current value of
	count in this context is wrong and cannot be trusted>

	<start running again>

	if (count - error + amount > threshold) {
		<not valid to run this lockless optimisation based
		on a stale count value>
		
		....
	}

Effectively, if we want to be able to use lockless optimisations, we need to ensure that the value of the global counter that we read remains within the given error bounds until we have finished making the lockless modification. That is done via disabling preemption across the entire function...

Quoted 3 lines

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs