percpu_counter: avoid potential underflow in add_unless_lt
Subsystems (2)
PATCH 2/5 percpu_counter: avoid potential underflow in add_unless_lt
In __percpu_counter_add_unless_lt(), an assumption is made that under certain conditions it's possible to determine that an amount can be safely added to a counter, possibly without having to acquire the lock. This assumption is not valid, however.
These lines encode the assumption:
if (count + amount > threshold + error) {
__percpu_counter_add(fbc, amount, batch);Inside __percpu_counter_add(), the addition is performed without acquiring the lock if the *sum* of the batch size and the CPU-local delta is within the batch size. Otherwise it does the addition after acquiring the lock.
The problem is that *that* sum may actually end up being greater than the batch size, forcing the addition to be performed under protection of the lock. And by the time the lock is acquired, the value of fbc->count may have been updated such that adding the given amount allows the result to go negative.
Fix this by open-coding the portion of the __percpu_counter_add() that avoids the lock.
--- lib/percpu_counter.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
lib/percpu_counter.c +8 -3
@@ -243,9 +243,14 @@ int __percpu_counter_add_unless_lt(struc* we can safely add, and might be able to avoid locking. */ if (count + amount > threshold + error) { -__percpu_counter_add (fbc, amount, batch); - ret =1 ; - goto out; +s32 *pcount =this_cpu_ptr (fbc->counters); + + count = *pcount + amount; +if (abs (count) < batch) { + *pcount = count; + ret =1 ; + goto out; + } }/*
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 39 lines · Alex Elder
In
__percpu_counter_add_unless_lt(), an assumption is made that under certain conditions it's possible to determine that an amount can be safely added to a counter, possibly without having to acquire the lock. This assumption is not valid, however.These lines encode the assumption:
if (count + amount > threshold + error) { __percpu_counter_add(fbc, amount, batch);Inside
__percpu_counter_add(), the addition is performed without acquiring the lock if the *sum* of the batch size and the CPU-local delta is within the batch size. Otherwise it does the addition after acquiring the lock.The problem is that *that* sum may actually end up being greater than the batch size, forcing the addition to be performed under protection of the lock. And by the time the lock is acquired, the value of
fbc->countmay have been updated such that adding the given amount allows the result to go negative.Fix this by open-coding the portion of the
__percpu_counter_add()that avoids the lock.Signed-off-by Alex Elder <[email protected]>--- lib/percpu_counter.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)Index:
b/lib/percpu_counter.c
===================================================================lib/percpu_counter.c +0
-2@@ -243,9 +243,14 @@ int __percpu_counter_add_unless_lt(struc* we can safely add, and might be able to avoid locking. */ if (count + amount > threshold + error) { -__percpu_counter_add (fbc, amount, batch); - ret =1 ;
lib/percpu_counter.c int __percpu_counter_add_unless_lt(struc · line 254
Quoted 10 lines
lib/percpu_counter.c +8
-1- goto out; +s32 *pcount =this_cpu_ptr (fbc->counters); + + count = *pcount + amount; +if (abs (count) < batch) { + *pcount = count; + ret =1 ; + goto out; + } }
The problem with this is that it never zeros pcount. That means after a bunch of increments or decrements, abs(*pcount) == 31, and ever further increment/decrement will drop through to the path that requires locking. Then we simply have a very expensive global counter.
We need to take the lock to zero the pcount value because it has to be added to fbc->count. i.e. if you want this path to remain mostly lockless, then it needs to do exactly what __percpu_counter_add() does....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 48 lines · Dave Chinner
Quoted 47 lines · Alex Elder
In
__percpu_counter_add_unless_lt(), an assumption is made that under certain conditions it's possible to determine that an amount can be safely added to a counter, possibly without having to acquire the lock. This assumption is not valid, however.These lines encode the assumption:
if (count + amount > threshold + error) { __percpu_counter_add(fbc, amount, batch);Inside
__percpu_counter_add(), the addition is performed without acquiring the lock if the *sum* of the batch size and the CPU-local delta is within the batch size. Otherwise it does the addition after acquiring the lock.The problem is that *that* sum may actually end up being greater than the batch size, forcing the addition to be performed under protection of the lock. And by the time the lock is acquired, the value of
fbc->countmay have been updated such that adding the given amount allows the result to go negative.Fix this by open-coding the portion of the
__percpu_counter_add()that avoids the lock.Signed-off-by Alex Elder <[email protected]>--- lib/percpu_counter.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)Index:
b/lib/percpu_counter.c
===================================================================lib/percpu_counter.c +7
-3@@ -243,9 +243,14 @@ int __percpu_counter_add_unless_lt(struc* we can safely add, and might be able to avoid locking. */ if (count + amount > threshold + error) { -__percpu_counter_add (fbc, amount, batch); - ret =1 ; - goto out; +s32 *pcount =this_cpu_ptr (fbc->counters); + + count = *pcount + amount; +if (abs (count) < batch) { + *pcount = count; + ret =1 ; + goto out;
Quoted 8 lines
lib/percpu_counter.c int __percpu_counter_add_unless_lt(struc · line 254
Quoted 2 lines
lib/percpu_counter.c +1
-0+ } }The problem with this is that it never zeros pcount. That means after a bunch of increments or decrements,
abs(*pcount)== 31, and ever further increment/decrement will drop through to the path that requires locking. Then we simply have a very expensive global counter.
I see what you mean.
Perhaps the code (below this) that acquires the lock should zero *pcount while it's updating fbc->count. It's already paying the price of the lock anyway, might as well get the most value out of doing so. Anyway, I have stuff of greater significance in the next note...
Quoted 8 lines
We need to take the lock to zero the pcount value because it has to be added to
fbc->count. i.e. if you want this path to remain mostly lockless, then it needs to do exactly what__percpu_counter_add()does....Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs