percpu_counter: add percpu_counter_sum_all interface
Subsystems (6)
PATCH 1/1 percpu_counter: add percpu_counter_sum_all interface
The percpu_counter is used for scenarios where performance is more important than the accuracy. For percpu_counter users, who want more accurate information in their slowpath, percpu_counter_sum is provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is because percpu_counter does implement CPU offline callback which syncs the local data of the offlined CPU.
However there is a small race window between the online CPUs traversal of percpu_counter_sum and the CPU offline callback. The offline callback has to traverse all the percpu_counters on the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running, percpu_counter_sum can be called for one counter which has some state on the CPU going offline. Since percpu_counter_sum only traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specific percpu_counter on that offlined CPU.
Normally this is not an issue because percpu_counter users can deal with some inaccuracy for small time window. However a new user i.e. mm_struct on the cleanup path wants to check the exact state of the percpu_counter through check_mm(). For such users, this patch introduces percpu_counter_sum_all() which traverses all possible CPUs.
--- include/linux/percpu_counter.h | 6 ++++++ kernel/fork.c | 2 +- lib/percpu_counter.c | 29 +++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 7 deletions(-)
include/linux/percpu_counter.h +6 -0
@@ -45,6 +45,7 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount);void percpu_counter_add_batch (struct percpu_counter *fbc,s64 amount,s32 batch);s64 __percpu_counter_sum (struct percpu_counter *fbc); +s64 percpu_counter_sum_all (struct percpu_counter *fbc);int __percpu_counter_compare (struct percpu_counter *fbc,s64 rhs,s32 batch);void percpu_counter_sync (struct percpu_counter *fbc);
@@ -193,6 +194,11 @@ static inline s64 percpu_counter_sum(struct percpu_counter *fbc)return percpu_counter_read (fbc); } +static inline s64 percpu_counter_sum_all (struct percpu_counter *fbc) +{ +return percpu_counter_read (fbc); +} +static inline bool percpu_counter_initialized (struct percpu_counter *fbc) {return true;
kernel/fork.c +1 -1
@@ -756,7 +756,7 @@ static void check_mm(struct mm_struct *mm) "Please make sure 'struct resident_page_types[]' is updated as well");for (i =0 ; i <NR_MM_COUNTERS ; i++) { -long x =percpu_counter_sum (&mm->rss_stat[i]); +long x =percpu_counter_sum_all (&mm->rss_stat[i]);if (unlikely (x)) pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n",
lib/percpu_counter.c +23 -6
@@ -117,11 +117,8 @@ void percpu_counter_sync(struct percpu_counter *fbc) }EXPORT_SYMBOL (percpu_counter_sync); -/* -* Add up all the per-cpu counts, return the result. This is a more accurate -* but much slower version of percpu_counter_read_positive() -*/ -s64 __percpu_counter_sum (struct percpu_counter *fbc) +static s64 __percpu_counter_sum_mask (struct percpu_counter *fbc, +const struct cpumask *cpu_mask) {s64 ret;int cpu;
@@ -129,15 +126,35 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc)raw_spin_lock_irqsave (&fbc->lock, flags); ret = fbc->count; -for_each_online_cpu (cpu) { +for_each_cpu (cpu, cpu_mask) {s32 *pcount =per_cpu_ptr (fbc->counters, cpu); ret += *pcount; }raw_spin_unlock_irqrestore (&fbc->lock, flags);return ret; } + +/* +* Add up all the per-cpu counts, return the result. This is a more accurate +* but much slower version of percpu_counter_read_positive() +*/ +s64 __percpu_counter_sum (struct percpu_counter *fbc) +{ +return __percpu_counter_sum_mask (fbc, cpu_online_mask); +}EXPORT_SYMBOL (__percpu_counter_sum); +/* +* This is slower version of percpu_counter_sum as it traverses all possible +* cpus. Use this only in the cases where accurate data is needed in the +* presense of CPUs getting offlined. +*/ +s64 percpu_counter_sum_all (struct percpu_counter *fbc) +{ +return __percpu_counter_sum_mask (fbc, cpu_possible_mask); +} +EXPORT_SYMBOL (percpu_counter_sum_all); +int __percpu_counter_init (struct percpu_counter *fbc,s64 amount,gfp_t gfp,struct lock_class_key *key) {
Signature · 1 line
2.38.1.431.g37b22c650d-goog
Andrew Morton
maintainerQuoted 8 lines · On Sat, 5 Nov 2022 01:40:13 +0000 Shakeel Butt <[email protected]> wrote:
The
percpu_counteris used for scenarios where performance is more important than the accuracy. Forpercpu_counterusers, who want more accurate information in their slowpath,percpu_counter_sumis provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is becausepercpu_counterdoes implement CPU offline callback which syncs the local data of the offlined CPU.
Quoted 10 lines
However there is a small race window between the online CPUs traversal of
percpu_counter_sumand the CPU offline callback. The offline callback has to traverse all thepercpu_counterson the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running,percpu_counter_sumcan be called for one counter which has some state on the CPU going offline. Sincepercpu_counter_sumonly traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specificpercpu_counteron that offlined CPU.
OK, got it, thanks.
Quoted 5 lines
Normally this is not an issue because
percpu_counterusers can deal with some inaccuracy for small time window. However a new user i.e.mm_structon the cleanup path wants to check the exact state of thepercpu_counterthroughcheck_mm(). For such users, this patch introducespercpu_counter_sum_all()which traverses all possible CPUs.
And uses it in fork.c:check_mm()!
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 759
Quoted 8 lines
kernel/fork.c +1
-1@@ -756,7 +756,7 @@ static void check_mm(struct mm_struct *mm) "Please make sure 'struct resident_page_types[]' is updated as well");for (i =0 ; i <NR_MM_COUNTERS ; i++) { -long x =percpu_counter_sum (&mm->rss_stat[i]); +long x =percpu_counter_sum_all (&mm->rss_stat[i]);
check_mm() just became more expensive in some cases. nr_possible_cpus
* 4. I wonder if this is enough for people to start caring about.
check_mm() is presently non-optional and I'd be reluctant to change this, given how commonly we see the "BUG: Bad rss-counter state" getting reported (22 million hits in a google search!).
We could save a ton of that cost by running percpu_counter_sum() first, then trying percpu_counter_sum_all() if percpu_counter_sum() indicated an error. This is only worth bothering about if the new check_mm() cost is a concern.
Quoted 44 lines · Andrew Morton
Quoted 8 lines · On Sat, 5 Nov 2022 01:40:13 +0000 Shakeel Butt <[email protected]> wrote:
The
percpu_counteris used for scenarios where performance is more important than the accuracy. Forpercpu_counterusers, who want more accurate information in their slowpath,percpu_counter_sumis provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is becausepercpu_counterdoes implement CPU offline callback which syncs the local data of the offlined CPU.Quoted 10 lines
However there is a small race window between the online CPUs traversal of
percpu_counter_sumand the CPU offline callback. The offline callback has to traverse all thepercpu_counterson the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running,percpu_counter_sumcan be called for one counter which has some state on the CPU going offline. Sincepercpu_counter_sumonly traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specificpercpu_counteron that offlined CPU.OK, got it, thanks.
Quoted 5 lines
Normally this is not an issue because
percpu_counterusers can deal with some inaccuracy for small time window. However a new user i.e.mm_structon the cleanup path wants to check the exact state of thepercpu_counterthroughcheck_mm(). For such users, this patch introducespercpu_counter_sum_all()which traverses all possible CPUs.And uses it in fork.c:check_mm()!
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 759
Quoted 8 lines
kernel/fork.c +1
-1@@ -756,7 +756,7 @@ static void check_mm(struct mm_struct *mm) "Please make sure 'struct resident_page_types[]' is updated as well");for (i =0 ; i <NR_MM_COUNTERS ; i++) { -long x =percpu_counter_sum (&mm->rss_stat[i]); +long x =percpu_counter_sum_all (&mm->rss_stat[i]);
check_mm()just became more expensive in some cases.nr_possible_cpus
* 4. I wonder if this is enough for people to start caring about.
Quoted 9 lines
check_mm()is presently non-optional and I'd be reluctant to change this, given how commonly we see the "BUG: Bad rss-counter state" getting reported (22 million hits in a google search!).We could save a ton of that cost by running
percpu_counter_sum()first, then tryingpercpu_counter_sum_all()ifpercpu_counter_sum()indicated an error. This is only worth bothering about if the newcheck_mm()cost is a concern.
Yes, this makes much more sense. I had run hackbench on the original patch and didn't see any significant difference. I will update this and run some more perf benchmarks to make sure there is no regression due to this change.
thanks,
Shakeel