Skip to contentSkip to conversation

lib/percpu_counter: allow concurrent sums with an rwlock

PATCH In review Jiebin Sun 3 participants 6 messages Nov 16 – Nov 18

Subsystems (4)
lore
Share & tools
lore

Jiebin Sun

author
#0 lore raw

PATCH 1/1 lib/percpu_counter: allow concurrent sums with an rwlock

__percpu_counter_sum() holds a raw spinlock while reading the global count and per-CPU deltas, serializing concurrent sums of the same counter. Since a sum changes neither the global count nor the per-CPU deltas, use rwlock_t on !PREEMPT_RT to allow concurrent sums with a read lock. Global updates take the write lock. The per-CPU fast path remains lockless, so sums keep their existing snapshot semantics.

stress-ng dentry reaches this path through ext4_statfs():

statfs -> ext4_statfs -> __percpu_counter_sum

         -> native_queued_spin_lock_slowpath

Keep the raw spinlock on PREEMPT_RT, where rwlock_t can sleep. Global counter updates can run in atomic context and must not sleep.

stress-ng dentry test average throughput change:

  +------------------------+------------+--------------------+
  | Case                   | Unit       | Average change (%) |
  +------------------------+------------+--------------------+
  | Dentry, 64 workers     | bogo ops/s |             +47.34 |
  +------------------------+------------+--------------------+

Dentry ran on ext4 with 64 workers pinned to physical CPUs:

  stress-ng --dentry 64 --dentries 4096 --timeout 20s
---
 include/linux/percpu_counter.h |  4 ++
 lib/percpu_counter.c           | 71 ++++++++++++++++++++++++++--------
 2 files changed, 59 insertions(+), 16 deletions(-)
include/linux/percpu_counter.h +4 -0
index 3a44dd1e33d2..7b28b0e22207 100644
@@ -20,7 +20,11 @@
 #ifdef CONFIG_SMP
 
 struct percpu_counter {
+#ifdef CONFIG_PREEMPT_RT
 	raw_spinlock_t lock;
+#else
+	rwlock_t lock;		/* shared for sums, exclusive for updates */
+#endif
 	s64 count;
 #ifdef CONFIG_HOTPLUG_CPU
 	struct list_head list;	/* All percpu_counters are on a list */
lib/percpu_counter.c +55 -16

155 lines not on this page · raw message

Signature · 1 line
2.52.0
maintainer
#1 lore raw
Quoted 23 lines · On Mon, 16 Nov 2026 09:56:55 +0800 Jiebin Sun <[email protected]> wrote:

__percpu_counter_sum() holds a raw spinlock while reading the global count and per-CPU deltas, serializing concurrent sums of the same counter. Since a sum changes neither the global count nor the per-CPU deltas, use rwlock_t on !PREEMPT_RT to allow concurrent sums with a read lock. Global updates take the write lock. The per-CPU fast path remains lockless, so sums keep their existing snapshot semantics.

stress-ng dentry reaches this path through ext4_statfs():

statfs -> ext4_statfs -> __percpu_counter_sum

         -> native_queued_spin_lock_slowpath

Keep the raw spinlock on PREEMPT_RT, where rwlock_t can sleep. Global counter updates can run in atomic context and must not sleep.

stress-ng dentry test average throughput change:

  +------------------------+------------+--------------------+
  | Case                   | Unit       | Average change (%) |
  +------------------------+------------+--------------------+
  | Dentry, 64 workers     | bogo ops/s |             +47.34 |
  +------------------------+------------+--------------------+
Quoted 3 lines

Dentry ran on ext4 with 64 workers pinned to physical CPUs:

  stress-ng --dentry 64 --dentries 4096 --timeout 20s

lol, this is a fun patch - it reminds me of the old days. All those mission-critical workloads which sit in a loop running statfs will love it!

I had $sidekick do a scan and it told me "the only plausible routine win is dirty throttling: with low dirty limits, concurrent writers sum the same counters. ext4 and XFS sum near ENOSPC, where updates also contend. BFQ's sums are debug-only and serialized. Most other callers report statistics". You might want to add something like this to changelog.

Also mention that percpu_counter_sum_positive callers will benefit.

include/linux/percpu_counter.h line 23

Quoted 7 lines
include/linux/percpu_counter.h +1 -0
@@ -20,7 +20,11 @@
 #ifdef CONFIG_SMP
 
 struct percpu_counter {
+#ifdef CONFIG_PREEMPT_RT

Might be worth a wee comment telling readers why RT is special?

Quoted 25 lines
include/linux/percpu_counter.h +3 -0
 	raw_spinlock_t lock;
+#else
+	rwlock_t lock;		/* shared for sums, exclusive for updates */
+#endif
 	s64 count;
 #ifdef CONFIG_HOTPLUG_CPU
 	struct list_head list;	/* All percpu_counters are on a list */
lib/percpu_counter.c +11 -0
@@ -15,6 +15,46 @@ static LIST_HEAD(percpu_counters);
 static DEFINE_SPINLOCK(percpu_counters_lock);
 #endif
 
+/*
+ * Non-RT sums share the lock; global updates take it exclusively.
+ * Keep raw spinlocks on RT: rwlocks may sleep and their irqsave variants
+ * do not disable interrupts. Updates run in atomic contexts and need
+ * IRQ exclusion for CPU-local read-modify-write sequences.
+ */
+#ifdef CONFIG_PREEMPT_RT
+#define fbc_lock(fbc)			raw_spin_lock(&(fbc)->lock)
+#define fbc_unlock(fbc)			raw_spin_unlock(&(fbc)->lock)
+#define fbc_lock_irqsave(fbc, f)		raw_spin_lock_irqsave(&(fbc)->lock, f)
+#define fbc_unlock_irqrestore(fbc, f)	raw_spin_unlock_irqrestore(&(fbc)->lock, f)

lib/percpu_counter.c static LIST_HEAD(percpu_counters); · line 38

Quoted 10 lines
lib/percpu_counter.c +10 -0
+#define fbc_read_lock_irqsave		fbc_lock_irqsave
+#define fbc_read_unlock_irqrestore	fbc_unlock_irqrestore
+#else
+#define fbc_lock(fbc)			write_lock(&(fbc)->lock)
+#define fbc_unlock(fbc)			write_unlock(&(fbc)->lock)
+#define fbc_lock_irqsave(fbc, f)		write_lock_irqsave(&(fbc)->lock, f)
+#define fbc_unlock_irqrestore(fbc, f)	write_unlock_irqrestore(&(fbc)->lock, f)
+#define fbc_read_lock_irqsave(fbc, f)	read_lock_irqsave(&(fbc)->lock, f)
+#define fbc_read_unlock_irqrestore(fbc, f) read_unlock_irqrestore(&(fbc)->lock, f)
+#endif

Please, no!

https://www.google.com/search?q=why+does+Andrew+Morton+prefer+static+inlines+over+macros

lots of good stuff there ;)

maintainer
#2 lore raw
Quoted 8 lines · On Sun, 15 Nov 2026 19:16:36 -0800 Andrew Morton <[email protected]>…
Quoted 8 lines

stress-ng dentry test average throughput change:

  +------------------------+------------+--------------------+
  | Case                   | Unit       | Average change (%) |
  +------------------------+------------+--------------------+
  | Dentry, 64 workers     | bogo ops/s |             +47.34 |
  +------------------------+------------+--------------------+
Quoted 5 lines
Quoted 3 lines

Dentry ran on ext4 with 64 workers pinned to physical CPUs:

  stress-ng --dentry 64 --dentries 4096 --timeout 20s

lol, this is a fun patch

It's *probably* OK, for sane workloads.

But if some workload is busily running percpu_counter_sum on 1000 CPUs, how is a writer ever to get in there to spill ->counters into ->count?

Modern rwlock_t does implement queuing, so the writer will get in there eventually. But nothing is simple any more - there are complications with rwlocks on PREEMPT_RT and with interrupt-time reads (do we even do that?).

So I think indefinite writer starvation is a risk here?

#3 lore raw
Quoted 2 lines · Andrew Morton

Modern rwlock_t does implement queuing, so the writer will get in there eventually.

On the queued-rwlock implementation, if a writer blocks on the lock then the readers will wait until the writer is done unless the reader is from interrupt.

There is also seqcount_t where the reader would be fine as long as there has been no writer during the read and plenty of readers don't block each other. It might take a while if the writer performs multiple updates and the reader can't query all CPUs in time.
So it is not easy.

Quoted 3 lines

But nothing is simple any more - there are
complications with rwlocks on PREEMPT_RT and with interrupt-time reads (do we even do that?).

There are a few users from preempt/irq disabled section on PREEMPT_RT which mandate something like raw_spinlock_t. The rwlock_t will break.

Sebastian

author
#4 lore raw
In reply to Andrew Morton

Hi Andrew and Sebastian,

Thanks for the comments and the RT review.

I'll replace the macros with static inline helpers, add the RT comment, and clarify the practical callers, including percpu_counter_sum_positive(), in v2.

Interrupt readers can bypass a WAITING writer and delay it if they keep overlapping. I'm considering the impact this could have on counter updates. If this needs addressing, we might take the write lock in __percpu_counter_sum() when called from interrupt context, while keeping the read lock for ordinary sums.

I also tested buffered writes with fio.

Fio (64 jobs, 3 iterations, 60s each):

  +------+------------+-----------------+---------+
  | Test | Throughput | CPU utilization | Cycles  |
  +------+------------+-----------------+---------+
  | Fio  |     +6.91% |         -40.41% | -40.80% |
  +------+------------+-----------------+---------+

Test configuration: ext4, 32 GiB cgroup memory limit.

Commands:

  sysctl -w vm.dirty_bytes=67108864 \
             vm.dirty_background_bytes=33554432

  fio --name=buffered-write --directory="$TEST_DIR" \
      --ioengine=sync --rw=write --bs=64k --size=256m --numjobs=64 \
      --direct=0 --fallocate=none --time_based=1 --runtime=60 \
      --cpus_allowed=0-31,86-117 --cpus_allowed_policy=split \
      --group_reporting --output-format=json

Thanks,
Jiebin

#5 lore raw
In reply to Jiebin Sun
Quoted 23 lines · Jiebin Sun

__percpu_counter_sum() holds a raw spinlock while reading the global count and per-CPU deltas, serializing concurrent sums of the same counter. Since a sum changes neither the global count nor the per-CPU deltas, use rwlock_t on !PREEMPT_RT to allow concurrent sums with a read lock. Global updates take the write lock. The per-CPU fast path remains lockless, so sums keep their existing snapshot semantics.

stress-ng dentry reaches this path through ext4_statfs():

statfs -> ext4_statfs -> __percpu_counter_sum

         -> native_queued_spin_lock_slowpath

Keep the raw spinlock on PREEMPT_RT, where rwlock_t can sleep. Global counter updates can run in atomic context and must not sleep.

stress-ng dentry test average throughput change:

  +------------------------+------------+--------------------+
  | Case                   | Unit       | Average change (%) |
  +------------------------+------------+--------------------+
  | Dentry, 64 workers     | bogo ops/s |             +47.34 |
  +------------------------+------------+--------------------+
Quoted 8 lines

Dentry ran on ext4 with 64 workers pinned to physical CPUs:

  stress-ng --dentry 64 --dentries 4096 --timeout 20s

It looks okay from RT point of view. Is this limited to this benchmark or is there a real workload behind this?

Sebastian