lib/percpu_counter: fix data race in __percpu_counter_limited_add()
Subsystems (3)
PATCH 1/1 lib/percpu_counter: fix data race in __percpu_counter_limited_add()
The following data-race was found in __percpu_counter_limited_add():
==================================================================
BUG: KCSAN: data-race in __percpu_counter_limited_add / __percpu_counter_limited_add
write to 0xffff88801f417e50 of 8 bytes by task 6663 on cpu 0:
__percpu_counter_limited_add+0x388/0x4a0 lib/percpu_counter.c:386 percpu_counter_limited_add include/linux/percpu_counter.h:77 [inline] shmem_inode_acct_blocks+0x10e/0x230 mm/shmem.c:233 shmem_alloc_and_add_folio mm/shmem.c:1923 [inline] shmem_get_folio_gfp.constprop.0+0x87f/0xc90 mm/shmem.c:2533 shmem_get_folio mm/shmem.c:2639 [inline] ....
read to 0xffff88801f417e50 of 8 bytes by task 6659 on cpu 1:
__percpu_counter_limited_add+0xc8/0x4a0 lib/percpu_counter.c:344 percpu_counter_limited_add include/linux/percpu_counter.h:77 [inline] shmem_inode_acct_blocks+0x10e/0x230 mm/shmem.c:233 shmem_alloc_and_add_folio mm/shmem.c:1923 [inline] shmem_get_folio_gfp.constprop.0+0x87f/0xc90 mm/shmem.c:2533 shmem_get_folio mm/shmem.c:2639 [inline] ....
value changed: 0x000000000000396d -> 0x000000000000398e
==================================================================
__percpu_counter_limited_add() should protect fbc via raw_spin_lock(), but it calls spinlock in the wrong place. This causes a data-race, so we need to fix it to call raw_spin_lock() a bit earlier.
--- lib/percpu_counter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
lib/percpu_counter.c +3 -3
@@ -336,6 +336,7 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc,return true;local_irq_save (flags); +raw_spin_lock (&fbc->lock); unknown = batch *num_online_cpus (); count =__this_cpu_read (*fbc->counters);
@@ -344,11 +345,10 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, ((amount > 0 && fbc->count + unknown <= limit) || (amount < 0 && fbc->count - unknown >= limit))) {this_cpu_add (*fbc->counters, amount); -local_irq_restore (flags); -return true; + good = true; + goto out; } -raw_spin_lock (&fbc->lock); count = fbc->count + amount;/* Skip percpu_counter_sum() when safe */
--
Quoted 54 lines · Jeongjun Park
The following data-race was found in
__percpu_counter_limited_add():==================================================================
BUG: KCSAN: data-race in __percpu_counter_limited_add / __percpu_counter_limited_addwrite to
0xffff88801f417e50of 8 bytes by task 6663 on cpu 0:__percpu_counter_limited_add+0x388/0x4a0 lib/percpu_counter.c:386 percpu_counter_limited_add include/linux/percpu_counter.h:77 [inline] shmem_inode_acct_blocks+0x10e/0x230 mm/shmem.c:233 shmem_alloc_and_add_folio mm/shmem.c:1923 [inline] shmem_get_folio_gfp.constprop.0+0x87f/0xc90 mm/shmem.c:2533 shmem_get_folio mm/shmem.c:2639 [inline] ....read to
0xffff88801f417e50of 8 bytes by task 6659 on cpu 1:__percpu_counter_limited_add+0xc8/0x4a0 lib/percpu_counter.c:344 percpu_counter_limited_add include/linux/percpu_counter.h:77 [inline] shmem_inode_acct_blocks+0x10e/0x230 mm/shmem.c:233 shmem_alloc_and_add_folio mm/shmem.c:1923 [inline] shmem_get_folio_gfp.constprop.0+0x87f/0xc90 mm/shmem.c:2533 shmem_get_folio mm/shmem.c:2639 [inline] ....value changed:
0x000000000000396d->0x000000000000398e
==================================================================
__percpu_counter_limited_add()should protect fbc viaraw_spin_lock(), but it calls spinlock in the wrong place. This causes a data-race, so we need to fix it to callraw_spin_lock()a bit earlier.Fixes beb986862844 ("shmem,percpu_counter: add _limited_add(fbc, limit, amount)")Signed-off-by Jeongjun Park <[email protected]>--- lib/percpu_counter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)lib/percpu_counter.c +1
-1@@ -336,6 +336,7 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc,return true;local_irq_save (flags); +raw_spin_lock (&fbc->lock); unknown = batch *num_online_cpus (); count =__this_cpu_read (*fbc->counters);@@ -344,11 +345,10 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, ((amount > 0 && fbc->count + unknown <= limit) || (amount < 0 && fbc->count - unknown >= limit))) {this_cpu_add (*fbc->counters, amount); -local_irq_restore (flags);
lib/percpu_counter.c bool __percpu_counter_limited_add(struct percpu_counter *fbc,
Quoted 10 lines
lib/percpu_counter.c +2
-3-return true; + good = true; + goto out; } -raw_spin_lock (&fbc->lock); count = fbc->count + amount;/* Skip percpu_counter_sum() when safe */ --
As this always takes the centralized lock in the fast path this defeats the point of using a per-cpu counter in the first place.
I noted this thing is buggy almost a year ago:
https://lore.kernel.org/linux-mm/5eemkb4lo5eefp7ijgncgogwmadyzmvjfjmmmvfiki6cwdskfs@hi2z4drqeuz6/
per the e-mail I don't believe existence of this routine is warranted.
shmem is still the only consumer.
Andrew Morton
maintainerQuoted 16 lines · On Tue, 6 May 2025 13:56:13 +0200 Mateusz Guzik <[email protected]> wrote:
Quoted 16 lines
lib/percpu_counter.c +2
-3unknown = batch *num_online_cpus (); count =__this_cpu_read (*fbc->counters);@@ -344,11 +345,10 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, ((amount > 0 && fbc->count + unknown <= limit) || (amount < 0 && fbc->count - unknown >= limit))) {this_cpu_add (*fbc->counters, amount); -local_irq_restore (flags); -return true; + good = true; + goto out; } -raw_spin_lock (&fbc->lock); count = fbc->count + amount;
Quoted 6 lines
Review bool __percpu_counter_limited_add(struct percpu_counter *fbc,
Quoted 3 lines
(quoted diff) +0
-1/* Skip percpu_counter_sum() when safe */ --As this always takes the centralized lock in the fast path this defeats the point of using a per-cpu counter in the first place.
Well. It partially "defeats the point" if the client code actually uses percpu_counter_limited_add(). Only shmem.c does that.
Quoted 6 lines
I noted this thing is buggy almost a year ago:
https://lore.kernel.org/linux-mm/5eemkb4lo5eefp7ijgncgogwmadyzmvjfjmmmvfiki6cwdskfs@hi2z4drqeuz6/per the e-mail I don't believe existence of this routine is warranted.
shmem is still the only consumer.
Totally. It would be better to remove percpu_counter_limited_add() and to implement its intended effect within shmem.c