mm: Sync percpu mm RSS counters before querying
Subsystems (4)
RFC PATCH 1/1 mm: Sync percpu mm RSS counters before querying
An issue was observed with stats collected in struct rusage on ppc64le with 64kB pages. The percpu counters use batching with
percpu_counter_batch = max(32, nr*2) # in PAGE_SIZE
i.e. with larger pages but similar RSS consumption (bytes), there'll be less flushes and error more noticeable.
In this given case (getting consumption of exited child), we can request percpu counter's flush without worrying about contention with updaters.
Fortunately, the commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter") didn't eradicate all traces of SPLIT_RSS_COUNTING and this mechanism already provided some synchronization points before reading stats.
Therefore, use sync_mm_rss as carrier for percpu counters refreshes and forget SPLIT_RSS_COUNTING macro for good.
--- include/linux/mm.h | 6 ++---- kernel/fork.c | 4 ---- 2 files changed, 2 insertions(+), 8 deletions(-)
include/linux/mm.h +2 -4
@@ -2547,13 +2547,11 @@ static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, *maxrss = hiwater_rss; } -#if defined(SPLIT_RSS_COUNTING ) -void sync_mm_rss (struct mm_struct *mm); -#else static inline void sync_mm_rss (struct mm_struct *mm) { +for (int i =0 ; i <NR_MM_COUNTERS ; ++i) +percpu_counter_sum (&mm->rss_stat[i]); } -#endif #ifndef CONFIG_ARCH_HAS_PTE_SPECIAL static inline int pte_special (pte_t pte)
kernel/fork.c +0 -4
@@ -2412,10 +2412,6 @@ __latent_entropy struct task_struct *copy_process( p->io_uring =NULL ;#endif -#if defined(SPLIT_RSS_COUNTING ) -memset (&p->rss_stat,0 ,sizeof (p->rss_stat)); -#endif - p->default_timer_slack_ns = current->timer_slack_ns;#ifdef CONFIG_PSI
Signature · 1 line
2.40.1
Quoted 16 lines · Michal Koutný
An issue was observed with stats collected in struct rusage on ppc64le with 64kB pages. The percpu counters use batching with
percpu_counter_batch=max(32, nr*2)# inPAGE_SIZE
i.e. with larger pages but similar RSS consumption (bytes), there'll be less flushes and error more noticeable.In this given case (getting consumption of exited child), we can request percpu counter's flush without worrying about contention with updaters.
Fortunately, the commit
f1a7941243c1("mm: convert mm's rss stats intopercpu_counter") didn't eradicate all traces ofSPLIT_RSS_COUNTINGand this mechanism already provided some synchronization points before reading stats.
Therefore, usesync_mm_rssas carrier for percpu counters refreshes and forgetSPLIT_RSS_COUNTINGmacro for good.
Quoted 3 lines
Fixes f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter")Reported-by Adam Majer <[email protected]>Signed-off-by Michal Koutný <[email protected]>
The patch seems reasonable to me. Are any of the callsites of sync_mm_rss performance sensitive?
Quoted 43 lines
--- include/linux/mm.h | 6 ++---- kernel/fork.c | 4 ---- 2 files changed, 2 insertions(+), 8 deletions(-)include/linux/mm.h +2
-4@@ -2547,13 +2547,11 @@ static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, *maxrss = hiwater_rss; } -#if defined(SPLIT_RSS_COUNTING ) -void sync_mm_rss (struct mm_struct *mm); -#else static inline void sync_mm_rss (struct mm_struct *mm) { +for (int i =0 ; i <NR_MM_COUNTERS ; ++i) +percpu_counter_sum (&mm->rss_stat[i]); } -#endif #ifndef CONFIG_ARCH_HAS_PTE_SPECIAL static inline int pte_special (pte_t pte)kernel/fork.c +0
-4@@ -2412,10 +2412,6 @@ __latent_entropy struct task_struct *copy_process( p->io_uring =NULL ;#endif -#if defined(SPLIT_RSS_COUNTING ) -memset (&p->rss_stat,0 ,sizeof (p->rss_stat)); -#endif - p->default_timer_slack_ns = current->timer_slack_ns;#ifdef CONFIG_PSI Signature · 1 line
2.40.1