Skip to contentSkip to conversation

percpu_counter: add percpu_counter_sum_all interface

PATCH Superseded Shakeel Butt 2 participants 3 messages Nov 5, 2022 – Nov 7, 2022

Subsystems (6)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 percpu_counter: add percpu_counter_sum_all interface

The percpu_counter is used for scenarios where performance is more important than the accuracy. For percpu_counter users, who want more accurate information in their slowpath, percpu_counter_sum is provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is because percpu_counter does implement CPU offline callback which syncs the local data of the offlined CPU.

However there is a small race window between the online CPUs traversal of percpu_counter_sum and the CPU offline callback. The offline callback has to traverse all the percpu_counters on the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running, percpu_counter_sum can be called for one counter which has some state on the CPU going offline. Since percpu_counter_sum only traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specific percpu_counter on that offlined CPU.

Normally this is not an issue because percpu_counter users can deal with some inaccuracy for small time window. However a new user i.e. mm_struct on the cleanup path wants to check the exact state of the percpu_counter through check_mm(). For such users, this patch introduces percpu_counter_sum_all() which traverses all possible CPUs.

---
 include/linux/percpu_counter.h |  6 ++++++
 kernel/fork.c                  |  2 +-
 lib/percpu_counter.c           | 29 +++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 7 deletions(-)
include/linux/percpu_counter.h +6 -0
index bde6c4c1f405..a3aae8d57a42 100644
@@ -45,6 +45,7 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount);
 void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount,
 			      s32 batch);
 s64 __percpu_counter_sum(struct percpu_counter *fbc);
+s64 percpu_counter_sum_all(struct percpu_counter *fbc);
 int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch);
 void percpu_counter_sync(struct percpu_counter *fbc);
 
@@ -193,6 +194,11 @@ static inline s64 percpu_counter_sum(struct percpu_counter *fbc)
 	return percpu_counter_read(fbc);
 }
 
+static inline s64 percpu_counter_sum_all(struct percpu_counter *fbc)
+{
+	return percpu_counter_read(fbc);
+}
+
 static inline bool percpu_counter_initialized(struct percpu_counter *fbc)
 {
 	return true;
kernel/fork.c +1 -1
index 9c32f593ef11..7d6f510cf397 100644
@@ -756,7 +756,7 @@ static void check_mm(struct mm_struct *mm)
 			 "Please make sure 'struct resident_page_types[]' is updated as well");
 
 	for (i = 0; i < NR_MM_COUNTERS; i++) {
-		long x = percpu_counter_sum(&mm->rss_stat[i]);
+		long x = percpu_counter_sum_all(&mm->rss_stat[i]);
 
 		if (unlikely(x))
 			pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n",
lib/percpu_counter.c +23 -6
index ed610b75dc32..42f729c8e56c 100644
@@ -117,11 +117,8 @@ void percpu_counter_sync(struct percpu_counter *fbc)
 }
 EXPORT_SYMBOL(percpu_counter_sync);
 
-/*
- * Add up all the per-cpu counts, return the result.  This is a more accurate
- * but much slower version of percpu_counter_read_positive()
- */
-s64 __percpu_counter_sum(struct percpu_counter *fbc)
+static s64 __percpu_counter_sum_mask(struct percpu_counter *fbc,
+			      const struct cpumask *cpu_mask)
 {
 	s64 ret;
 	int cpu;
@@ -129,15 +126,35 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc)
 
 	raw_spin_lock_irqsave(&fbc->lock, flags);
 	ret = fbc->count;
-	for_each_online_cpu(cpu) {
+	for_each_cpu(cpu, cpu_mask) {
 		s32 *pcount = per_cpu_ptr(fbc->counters, cpu);
 		ret += *pcount;
 	}
 	raw_spin_unlock_irqrestore(&fbc->lock, flags);
 	return ret;
 }
+
+/*
+ * Add up all the per-cpu counts, return the result.  This is a more accurate
+ * but much slower version of percpu_counter_read_positive()
+ */
+s64 __percpu_counter_sum(struct percpu_counter *fbc)
+{
+	return __percpu_counter_sum_mask(fbc, cpu_online_mask);
+}
 EXPORT_SYMBOL(__percpu_counter_sum);
 
+/*
+ * This is slower version of percpu_counter_sum as it traverses all possible
+ * cpus. Use this only in the cases where accurate data is needed in the
+ * presense of CPUs getting offlined.
+ */
+s64 percpu_counter_sum_all(struct percpu_counter *fbc)
+{
+	return __percpu_counter_sum_mask(fbc, cpu_possible_mask);
+}
+EXPORT_SYMBOL(percpu_counter_sum_all);
+
 int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp,
 			  struct lock_class_key *key)
 {
Signature · 1 line
2.38.1.431.g37b22c650d-goog
maintainer
#1 lore raw
Quoted 8 lines · On Sat, 5 Nov 2022 01:40:13 +0000 Shakeel Butt <[email protected]> wrote:

The percpu_counter is used for scenarios where performance is more important than the accuracy. For percpu_counter users, who want more accurate information in their slowpath, percpu_counter_sum is provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is because percpu_counter does implement CPU offline callback which syncs the local data of the offlined CPU.

Quoted 10 lines

However there is a small race window between the online CPUs traversal of percpu_counter_sum and the CPU offline callback. The offline callback has to traverse all the percpu_counters on the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running, percpu_counter_sum can be called for one counter which has some state on the CPU going offline. Since percpu_counter_sum only traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specific percpu_counter on that offlined CPU.

OK, got it, thanks.

Quoted 5 lines

Normally this is not an issue because percpu_counter users can deal with some inaccuracy for small time window. However a new user i.e. mm_struct on the cleanup path wants to check the exact state of the percpu_counter through check_mm(). For such users, this patch introduces percpu_counter_sum_all() which traverses all possible CPUs.

And uses it in fork.c:check_mm()!

kernel/fork.c static void check_mm(struct mm_struct *mm) · line 759

Quoted 8 lines
kernel/fork.c +1 -1
@@ -756,7 +756,7 @@ static void check_mm(struct mm_struct *mm)
 			 "Please make sure 'struct resident_page_types[]' is updated as well");
 
 	for (i = 0; i < NR_MM_COUNTERS; i++) {
-		long x = percpu_counter_sum(&mm->rss_stat[i]);
+		long x = percpu_counter_sum_all(&mm->rss_stat[i]);

check_mm() just became more expensive in some cases. nr_possible_cpus
* 4. I wonder if this is enough for people to start caring about.

check_mm() is presently non-optional and I'd be reluctant to change this, given how commonly we see the "BUG: Bad rss-counter state" getting reported (22 million hits in a google search!).

We could save a ton of that cost by running percpu_counter_sum() first, then trying percpu_counter_sum_all() if percpu_counter_sum() indicated an error. This is only worth bothering about if the new check_mm() cost is a concern.

#2 lore raw
Quoted 44 lines · Andrew Morton
Quoted 8 lines · On Sat, 5 Nov 2022 01:40:13 +0000 Shakeel Butt <[email protected]> wrote:

The percpu_counter is used for scenarios where performance is more important than the accuracy. For percpu_counter users, who want more accurate information in their slowpath, percpu_counter_sum is provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is because percpu_counter does implement CPU offline callback which syncs the local data of the offlined CPU.

Quoted 10 lines

However there is a small race window between the online CPUs traversal of percpu_counter_sum and the CPU offline callback. The offline callback has to traverse all the percpu_counters on the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running, percpu_counter_sum can be called for one counter which has some state on the CPU going offline. Since percpu_counter_sum only traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specific percpu_counter on that offlined CPU.

OK, got it, thanks.

Quoted 5 lines

Normally this is not an issue because percpu_counter users can deal with some inaccuracy for small time window. However a new user i.e. mm_struct on the cleanup path wants to check the exact state of the percpu_counter through check_mm(). For such users, this patch introduces percpu_counter_sum_all() which traverses all possible CPUs.

And uses it in fork.c:check_mm()!

kernel/fork.c static void check_mm(struct mm_struct *mm) · line 759

Quoted 8 lines
kernel/fork.c +1 -1
@@ -756,7 +756,7 @@ static void check_mm(struct mm_struct *mm)
                       "Please make sure 'struct resident_page_types[]' is updated as well");
 
      for (i = 0; i < NR_MM_COUNTERS; i++) {
-             long x = percpu_counter_sum(&mm->rss_stat[i]);
+             long x = percpu_counter_sum_all(&mm->rss_stat[i]);

check_mm() just became more expensive in some cases. nr_possible_cpus
* 4. I wonder if this is enough for people to start caring about.

Quoted 9 lines

check_mm() is presently non-optional and I'd be reluctant to change this, given how commonly we see the "BUG: Bad rss-counter state" getting reported (22 million hits in a google search!).

We could save a ton of that cost by running percpu_counter_sum() first, then trying percpu_counter_sum_all() if percpu_counter_sum() indicated an error. This is only worth bothering about if the new check_mm() cost is a concern.

Yes, this makes much more sense. I had run hackbench on the original patch and didn't see any significant difference. I will update this and run some more perf benchmarks to make sure there is no regression due to this change.

thanks,
Shakeel