percpu_counter-add-percpu_counter_sum_all-interface.patch removed from -mm tree
The quilt patch titled
Subject: percpu_counter: add percpu_counter_sum_all interface
has been removed from the -mm tree. Its filename was
percpu_counter-add-percpu_counter_sum_all-interface.patch
This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: Shakeel Butt <[email protected]>
Subject: percpu_counter: add percpu_counter_sum_all interface
Date: Wed, 9 Nov 2022 01:20:11 +0000
The percpu_counter is used for scenarios where performance is more important than the accuracy. For percpu_counter users, who want more accurate information in their slowpath, percpu_counter_sum is provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is because percpu_counter does implement CPU offline callback which syncs the local data of the offlined CPU.
However there is a small race window between the online CPUs traversal of percpu_counter_sum and the CPU offline callback. The offline callback has to traverse all the percpu_counters on the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running, percpu_counter_sum can be called for one counter which has some state on the CPU going offline. Since percpu_counter_sum only traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specific percpu_counter on that offlined CPU.
Normally this is not an issue because percpu_counter users can deal with some inaccuracy for small time window. However a new user i.e. mm_struct on the cleanup path wants to check the exact state of the percpu_counter through check_mm(). For such users, this patch introduces percpu_counter_sum_all() which traverses all possible CPUs and it is used in fork.c:check_mm() to avoid the potential race.
This issue is exposed by the later patch "mm: convert mm's rss stats into percpu_counter".
---
include/linux/percpu_counter.h | 6 ++++++ kernel/fork.c | 5 +++++ lib/percpu_counter.c | 29 +++++++++++++++++++++++------ 3 files changed, 34 insertions(+), 6 deletions(-)
include/linux/percpu_counter.h renamed from include/linux/percpu_counter.h~percpu_counter-add-percpu_counter_sum_all-interface +6 -0
@@ -45,6 +45,7 @@ void percpu_counter_set(struct percpu_covoid percpu_counter_add_batch (struct percpu_counter *fbc,s64 amount,s32 batch);s64 __percpu_counter_sum (struct percpu_counter *fbc); +s64 percpu_counter_sum_all (struct percpu_counter *fbc);int __percpu_counter_compare (struct percpu_counter *fbc,s64 rhs,s32 batch);void percpu_counter_sync (struct percpu_counter *fbc);
@@ -192,6 +193,11 @@ static inline s64 percpu_counter_sum(str {return percpu_counter_read (fbc); } + +static inline s64 percpu_counter_sum_all (struct percpu_counter *fbc) +{ +return percpu_counter_read (fbc); +}static inline bool percpu_counter_initialized (struct percpu_counter *fbc) {
kernel/fork.c renamed from kernel/fork.c~percpu_counter-add-percpu_counter_sum_all-interface +5 -0
@@ -755,6 +755,11 @@ static void check_mm(struct mm_struct *mfor (i =0 ; i <NR_MM_COUNTERS ; i++) {long x =percpu_counter_sum (&mm->rss_stat[i]); +if (likely (!x)) +continue ; + +/* Making sure this is not due to race with CPU offlining. */ + x =percpu_counter_sum_all (&mm->rss_stat[i]);if (unlikely (x))pr_alert ("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n" , mm, resident_page_types[i], x);
lib/percpu_counter.c renamed from lib/percpu_counter.c~percpu_counter-add-percpu_counter_sum_all-interface +23 -6
@@ -117,11 +117,8 @@ void percpu_counter_sync(struct percpu_c }EXPORT_SYMBOL (percpu_counter_sync); -/* -* Add up all the per-cpu counts, return the result. This is a more accurate -* but much slower version of percpu_counter_read_positive() -*/ -s64 __percpu_counter_sum (struct percpu_counter *fbc) +static s64 __percpu_counter_sum_mask (struct percpu_counter *fbc, +const struct cpumask *cpu_mask) {s64 ret;int cpu;
@@ -129,15 +126,35 @@ s64 __percpu_counter_sum(struct percpu_craw_spin_lock_irqsave (&fbc->lock, flags); ret = fbc->count; -for_each_online_cpu (cpu) { +for_each_cpu (cpu, cpu_mask) {s32 *pcount =per_cpu_ptr (fbc->counters, cpu); ret += *pcount; }raw_spin_unlock_irqrestore (&fbc->lock, flags);return ret; } + +/* +* Add up all the per-cpu counts, return the result. This is a more accurate +* but much slower version of percpu_counter_read_positive() +*/ +s64 __percpu_counter_sum (struct percpu_counter *fbc) +{ +return __percpu_counter_sum_mask (fbc, cpu_online_mask); +}EXPORT_SYMBOL (__percpu_counter_sum); +/* +* This is slower version of percpu_counter_sum as it traverses all possible +* cpus. Use this only in the cases where accurate data is needed in the +* presense of CPUs getting offlined. +*/ +s64 percpu_counter_sum_all (struct percpu_counter *fbc) +{ +return __percpu_counter_sum_mask (fbc, cpu_possible_mask); +} +EXPORT_SYMBOL (percpu_counter_sum_all); +int __percpu_counter_init (struct percpu_counter *fbc,s64 amount,gfp_t gfp,struct lock_class_key *key) {
_
Patches currently in -mm which might be from [email protected] are