mm: Fix OOM killer inaccuracy on large many-core systems
Subsystems (9)
Mathieu Desnoyers
reviewerv14 cover mm: Fix OOM killer inaccuracy on large many-core systems
Introduce hierarchical per-cpu counters and use them for RSS tracking to fix the per-mm RSS tracking which has become too inaccurate for OOM killer purposes on large many-core systems.
The following rss tracking issues were noted by Sweet Tea Dorminy [1], which lead to picking wrong tasks as OOM kill target:
Recently, several internal services had an RSS usage regression as part of a kernel upgrade. Previously, they were on a pre-6.2 kernel and were able to read RSS statistics in a backup watchdog process to monitor and decide if they'd overrun their memory budget. Now, however, a representative service with five threads, expected to use about a hundred MB of memory, on a 250-cpu machine had memory usage tens of megabytes different from the expected amount -- this constituted a significant percentage of inaccuracy, causing the watchdog to act.
This was a result of commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter") [1]. Previously, the memory error was bounded by 64*nr_threads pages, a very livable megabyte. Now, however, as a result of scheduler decisions moving the threads around the CPUs, the memory error could be as large as a gigabyte.
This is a really tremendous inaccuracy for any few-threaded program on a large machine and impedes monitoring significantly. These stat counters are also used to make OOM killing decisions, so this additional inaccuracy could make a big difference in OOM situations -- either resulting in the wrong process being killed, or in less memory being returned from an OOM-kill than expected.
The approach proposed here is to replace this by the hierarchical per-cpu counters, which bounds the inaccuracy based on the system topology with O(N*logN).
Notable changes for v14:
- Change check_mm print format from %d to %ld (was folded into the wrong patch).
I've done moderate testing of this series on a 256-core VM with 128GB RAM. Figuring out whether this indeed helps solve issues with real-life workloads will require broader feedback from the community.
This series is based on v6.19-rc4, on top of the following two preparation series:
https://lore.kernel.org/linux-mm/[email protected]/T/#t
https://lore.kernel.org/linux-mm/[email protected]/T/#t
Andrew, this series replaces v13, for testing in mm-new.
Thanks!
Mathieu
To: Andrew Morton <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Mathieu Desnoyers <[email protected]>
Cc: Dennis Zhou <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: Martin Liu <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: [email protected]
Cc: Shakeel Butt <[email protected]>
Cc: SeongJae Park <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Johannes Weiner <[email protected]>
Cc: Sweet Tea Dorminy <[email protected]>
Cc: Lorenzo Stoakes <[email protected]>
Cc: "Liam R . Howlett" <[email protected]>
Cc: Mike Rapoport <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Wei Yang <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Miaohe Lin <[email protected]>
Cc: Al Viro <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: Yu Zhao <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Mateusz Guzik <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Baolin Wang <[email protected]>
Cc: Aboorva Devarajan <[email protected]>
Mathieu Desnoyers (3):
lib: Introduce hierarchical per-cpu counters
mm: Fix OOM killer inaccuracy on large many-core systems
mm: Implement precise OOM killer task selection
fs/proc/base.c | 2 +- include/linux/mm.h | 49 +- include/linux/mm_types.h | 54 ++- include/linux/oom.h | 11 +- include/linux/percpu_counter_tree.h | 344 ++++++++++++++ include/trace/events/kmem.h | 2 +- init/main.c | 2 + kernel/fork.c | 22 +- lib/Makefile | 1 + lib/percpu_counter_tree.c | 702 ++++++++++++++++++++++++++++ mm/oom_kill.c | 84 +++- 11 files changed, 1223 insertions(+), 50 deletions(-) create mode 100644 include/linux/percpu_counter_tree.h create mode 100644 lib/percpu_counter_tree.c
Signature · 1 line
2.39.5
Mathieu Desnoyers
reviewerPATCH v14 1/3 lib: Introduce hierarchical per-cpu counters
* Motivation
The purpose of this hierarchical split-counter scheme is to:
- Minimize contention when incrementing and decrementing counters,
- Provide fast access to a sum approximation,
- Provide a sum approximation with an acceptable accuracy level when scaling to many-core systems.
- Provide approximate and precise comparison of two counters, and between a counter and a value.
It aims at fixing the per-mm RSS tracking which has become too inaccurate for OOM killer purposes on large many-core systems [1].
* Design
The hierarchical per-CPU counters propagate a sum approximation through a N-way tree. When reaching the batch size, the carry is propagated through a binary tree which consists of logN(nr_cpu_ids) levels. The batch size for each level is twice the batch size of the prior level.
Example propagation diagram with 8 cpus through a binary tree:
Level 0: 0 1 2 3 4 5 6 7
| / | / | / | /
| / | / | / | /
| / | / | / | /
Level 1: 0 1 2 3
| / | /
| / | /
| / | /
Level 2: 0 1
| /
| /
| /
Level 3: 0For a binary tree, the maximum inaccuracy is bound by:
batch_size * log2(nr_cpus) * nr_cpus
which evolves with O(n*log(n)) as the number of CPUs increases.
For a N-way tree, the maximum inaccuracy can be pre-calculated based on the the N-arity of each level and the batch size.
---
Changes since v12:
- Use atomic_long_set for percpu_counter_tree_set in UP build.
- percpu_counter_tree_precise_sum returns long type.
Changes since v11:
- Reduce level0 per-cpu memory allocation to the size required by those percpu counters.
- Use unsigned long type rather than unsigned int.
- Introduce functions to return the min/max range of possible precise sums.
Changes since v10:
- Fold "mm: Take into account hierarchical percpu tree items for static mm_struct definitions".
Changes since v9:
- Introduce kerneldoc documentation.
- Document structure fields.
- Remove inline from percpu_counter_tree_add().
- Reject batch_size==1 which makes no sense.
- Fix copy-paste bug in percpu_counter_tree_precise_compare() (wrong inequality comparison).
- Rename "inaccuracy" to "accuracy", which makes it easier to document.
- Track accuracy limits more precisely. In two's complement signed integers, the range before a n-bit underflow is one unit larger than the range before a n-bit overflow. This sums for all the counters within the tree. Therefore the "under" vs "over" accuracy is not symmetrical.
Changes since v8:
- Remove migrate guard from the fast path. It does not matter through which path the carry is propagated up
the tree.
- Rebase on top of v6.18-rc6.
- Introduce percpu_counter_tree_init_many and percpu_counter_tree_destroy_many APIs.
- Move tree items allocation to the caller.
- Introduce percpu_counter_tree_items_size().
- Move percpu_counter_tree_subsystem_init() call before mm_core_init() so percpu_counter_tree_items_size() is initialized before it is used.
Changes since v7:
- Explicitly initialize the subsystem from start_kernel() right after mm_core_init() so it is up and running before the creation of the first mm at boot.
- Remove the module.h include which is not needed with the explicit initialization.
- Only consider levels>0 items for order={0,1} nr_items. No functional change except to allocate only the amount of memory which is strictly needed.
- Introduce positive precise sum API to handle a scenario where an unlucky precise sum iteration would hit negative counter values concurrently with counter updates.
Changes since v5:
- Introduce percpu_counter_tree_approximate_sum_positive.
- Introduce !CONFIG_SMP static inlines for UP build.
- Remove percpu_counter_tree_set_bias from the public API and make it static.
Changes since v3:
- Add gfp flags to init function.
Changes since v2:
- Introduce N-way tree to reduce tree depth on larger systems.
Changes since v1:
- Remove percpu_counter_tree_precise_sum_unbiased from public header, make this function static,
- Introduce precise and approximate comparisons between two counters,
- Reorder the struct percpu_counter_tree fields,
- Introduce approx_sum field, which points to the approximate sum for the percpu_counter_tree_approximate_sum() fast path.
--- include/linux/mm_types.h | 6 +- include/linux/percpu_counter_tree.h | 344 ++++++++++++++ init/main.c | 2 + lib/Makefile | 1 + lib/percpu_counter_tree.c | 702 ++++++++++++++++++++++++++++ 5 files changed, 1052 insertions(+), 3 deletions(-) create mode 100644 include/linux/percpu_counter_tree.h create mode 100644 lib/percpu_counter_tree.c
include/linux/mm_types.h +3 -3
@@ -1366,9 +1366,9 @@ static inline void __mm_flags_set_mask_bits_word(struct mm_struct *mm, MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; -#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \ -{ \ - [0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0 \ +#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \ +{ \ + [0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE + PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE - 1] = 0 \ }/* Pointer magic because the dynamic array size confuses some compilers. */
include/linux/percpu_counter_tree.h new file +344 -0
344 lines not on this page · raw message
init/main.c +2 -0
@@ -104,6 +104,7 @@#include <linux/pidfs.h> #include <linux/ptdump.h> #include <linux/time_namespace.h> +#include <linux/percpu_counter_tree.h> #include <net/net_namespace.h> #include <asm/io.h>
@@ -1063,6 +1064,7 @@ void start_kernel(void)vfs_caches_init_early ();sort_main_extable ();trap_init (); +percpu_counter_tree_subsystem_init ();mm_core_init ();maple_tree_init ();poking_init ();
lib/Makefile +1 -0
@@ -181,6 +181,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o obj-$(CONFIG_SMP) += percpu_counter.o +obj-$(CONFIG_SMP) += percpu_counter_tree.o obj-$(CONFIG_AUDIT_GENERIC) += audit.o obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
lib/percpu_counter_tree.c new file +702 -0
702 lines not on this page · raw message
Signature · 1 line
2.39.5
Mathieu Desnoyers
reviewerPATCH v14 2/3 mm: Fix OOM killer inaccuracy on large many-core systems
In reply to Mathieu DesnoyersUse hierarchical per-cpu counters for rss tracking to fix the per-mm RSS tracking which has become too inaccurate for OOM killer purposes on large many-core systems.
The following rss tracking issues were noted by Sweet Tea Dorminy [1], which lead to picking wrong tasks as OOM kill target:
Recently, several internal services had an RSS usage regression as part of a kernel upgrade. Previously, they were on a pre-6.2 kernel and were able to read RSS statistics in a backup watchdog process to monitor and decide if they'd overrun their memory budget. Now, however, a representative service with five threads, expected to use about a hundred MB of memory, on a 250-cpu machine had memory usage tens of megabytes different from the expected amount -- this constituted a significant percentage of inaccuracy, causing the watchdog to act.
This was a result of commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter") [1]. Previously, the memory error was bounded by 64*nr_threads pages, a very livable megabyte. Now, however, as a result of scheduler decisions moving the threads around the CPUs, the memory error could be as large as a gigabyte.
This is a really tremendous inaccuracy for any few-threaded program on a large machine and impedes monitoring significantly. These stat counters are also used to make OOM killing decisions, so this additional inaccuracy could make a big difference in OOM situations -- either resulting in the wrong process being killed, or in less memory being returned from an OOM-kill than expected.
Here is a (possibly incomplete) list of the prior approaches that were used or proposed, along with their downside:
1) Per-thread rss tracking: large error on many-thread processes.
2) Per-CPU counters: up to 12% slower for short-lived processes and 9% increased system time in make test workloads [1]. Moreover, the inaccuracy increases with O(n^2) with the number of CPUs.
3) Per-NUMA-node counters: requires atomics on fast-path (overhead), error is high with systems that have lots of NUMA nodes (32 times the number of NUMA nodes).
The approach proposed here is to replace this by the hierarchical per-cpu counters, which bounds the inaccuracy based on the system topology with O(N*logN).
commit 82241a83cd15 ("mm: fix the inaccurate memory statistics issue for users") introduced get_mm_counter_sum() for precise /proc memory status queries. Implement it with percpu_counter_tree_precise_sum() since it is not a fast path and precision is preferred over speed.
* Testing results:
Test hardware: 2 sockets AMD EPYC 9654 96-Core Processor (384 logical CPUs total)
Methodology:
Comparing the current upstream implementation with the hierarchical counters is done by keeping both implementations wired up in parallel, and running a single-process, single-threaded program which hops randomly across CPUs in the system, calling mmap(2) and munmap(2) on random CPUs, keeping track of an array of allocated mappings, randomly choosing entries to either map or unmap.
get_mm_counter() is instrumented to compare the upstream counter approximation to the precise value, and print the delta when going over a given threshold. The delta of the hierarchical counter approximation to the precise value is also printed for comparison.
After a few minutes running this test, the upstream implementation counter approximation reaches a 1GB delta from the precise value, compared to 80MB delta with the hierarchical counter. The hierarchical counter provides a guaranteed maximum approximation inaccuracy of 192MB on that hardware topology.
* Fast path implementation comparison
The new inline percpu_counter_tree_add() uses a this_cpu_add_return() for the fast path (under a certain allocation size threshold). Above that, it calls a slow path which "trickles up" the carry to upper level counters with atomic_add_return.
In comparison, the upstream counters implementation calls percpu_counter_add_batch which uses this_cpu_try_cmpxchg() on the fast path, and does a raw_spin_lock_irqsave above a certain threshold.
The hierarchical implementation is therefore expected to have less contention on mid-sized allocations than the upstream counters because the atomic counters tracking those bits are only shared across nearby CPUs. In comparison, the upstream counters immediately use a global spinlock when reaching the threshold.
* Benchmarks
Using will-it-scale page_fault1 benchmarks to compare the upstream counters to the hierarchical counters. This is done with hyperthreading disabled. The speedup is within the standard deviation of the upstream runs, so the overhead is not significant.
upstream hierarchical speedup
page_fault1_processes -s 100 -t 1 614783 615558 +0.1% page_fault1_threads -s 100 -t 1 612788 612447 -0.1% page_fault1_processes -s 100 -t 96 37994977 37932035 -0.2% page_fault1_threads -s 100 -t 96 2484130 2504860 +0.8% page_fault1_processes -s 100 -t 192 71262917 71118830 -0.2% page_fault1_threads -s 100 -t 192 2446437 2469296 +0.1%
---
Changes since v13:
- Change check_mm print format from %d to %ld.
Changes since v10:
- Rebase on top of mm_struct static init fixes.
- Change the alignment of mm_struct flexible array to the alignment of the rss counters items (which are cacheline aligned on SMP).
- Move the rss counters items to first position within the flexible array at the end of the mm_struct to place content in decreasing alignment requirement order.
Changes since v8:
- Use percpu_counter_tree_init_many and percpu_counter_tree_destroy_many APIs.
- Remove percpu tree items allocation. Extend mm_struct size to include rss items. Those are handled through the new helpers get_rss_stat_items() and get_rss_stat_items_size() and passed as parameter to percpu_counter_tree_init_many().
Changes since v7:
- Use precise sum positive API to handle a scenario where an unlucky precise sum iteration would observe negative counter values due to concurrent updates.
Changes since v6:
- Rebased on v6.18-rc3.
- Implement get_mm_counter_sum as percpu_counter_tree_precise_sum for /proc virtual files memory state queries.
Changes since v5:
- Use percpu_counter_tree_approximate_sum_positive.
Change since v4:
- get_mm_counter needs to return 0 or a positive value.
Revert "fixup"
This reverts commit 46f29b72bfaaab4c6ee5a1ac8d29543b230fa293.
--- include/linux/mm.h | 19 ++++++++++---- include/linux/mm_types.h | 50 +++++++++++++++++++++++++++---------- include/trace/events/kmem.h | 2 +- kernel/fork.c | 22 +++++++++------- 4 files changed, 65 insertions(+), 28 deletions(-)
include/linux/mm.h +14 -5
@@ -2843,38 +2843,47 @@ static inline bool get_user_page_fast_only(unsigned long addr, {return get_user_pages_fast_only (addr,1 , gup_flags, pagep) ==1 ; } + +static inline struct percpu_counter_tree_level_item *get_rss_stat_items (struct mm_struct *mm) +{ +unsigned long ptr = (unsigned long )mm; + + ptr += offsetof(struct mm_struct , flexible_array); +return (struct percpu_counter_tree_level_item *)ptr; +} +/* * per-process(per-mm_struct) statistics. */ static inline unsigned long get_mm_counter (struct mm_struct *mm,int member) { -return percpu_counter_read_positive (&mm->rss_stat[member]); +return percpu_counter_tree_approximate_sum_positive (&mm->rss_stat[member]); }static inline unsigned long get_mm_counter_sum (struct mm_struct *mm,int member) { -return percpu_counter_sum_positive (&mm->rss_stat[member]); +return percpu_counter_tree_precise_sum_positive (&mm->rss_stat[member]); }void mm_trace_rss_stat (struct mm_struct *mm,int member);static inline void add_mm_counter (struct mm_struct *mm,int member,long value) { -percpu_counter_add (&mm->rss_stat[member], value); +percpu_counter_tree_add (&mm->rss_stat[member], value);mm_trace_rss_stat (mm, member); }static inline void inc_mm_counter (struct mm_struct *mm,int member) { -percpu_counter_inc (&mm->rss_stat[member]); +percpu_counter_tree_add (&mm->rss_stat[member],1 );mm_trace_rss_stat (mm, member); }static inline void dec_mm_counter (struct mm_struct *mm,int member) { -percpu_counter_dec (&mm->rss_stat[member]); +percpu_counter_tree_add (&mm->rss_stat[member],-1 );mm_trace_rss_stat (mm, member); }
include/linux/mm_types.h +37 -13
95 lines not on this page · raw message
include/trace/events/kmem.h +1 -1
@@ -442,7 +442,7 @@ TRACE_EVENT(rss_stat, __entry->mm_id =mm_ptr_to_hash (mm); __entry->curr = !!(current->mm == mm); __entry->member = member; - __entry->size = (percpu_counter_sum_positive (&mm->rss_stat[member]) + __entry->size = (percpu_counter_tree_approximate_sum_positive (&mm->rss_stat[member]) <<PAGE_SHIFT ); ),
kernel/fork.c +13 -9
@@ -133,6 +133,11 @@*/ #define MAX_THREADS FUTEX_TID_MASK +/* +* Batch size of rss stat approximation +*/ +#define RSS_STAT_BATCH_SIZE 32 +/* * Protected counters by write_lock_irq(&tasklist_lock) */
@@ -626,14 +631,12 @@ static void check_mm(struct mm_struct *mm) "Please make sure 'struct resident_page_types[]' is updated as well");for (i =0 ; i <NR_MM_COUNTERS ; i++) { -long x =percpu_counter_sum (&mm->rss_stat[i]); - -if (unlikely (x)) { +if (unlikely (percpu_counter_tree_precise_compare_value (&mm->rss_stat[i],0 ) !=0 ))pr_alert ("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n" , - mm, resident_page_types[i], x, + mm, resident_page_types[i], +percpu_counter_tree_precise_sum (&mm->rss_stat[i]), current->comm,task_pid_nr (current)); - } }if (mm_pgtables_bytes (mm))
@@ -731,7 +734,7 @@ void __mmdrop(struct mm_struct *mm)put_user_ns (mm->user_ns);mm_pasid_drop (mm);mm_destroy_cid (mm); -percpu_counter_destroy_many (mm->rss_stat,NR_MM_COUNTERS ); +percpu_counter_tree_destroy_many (mm->rss_stat,NR_MM_COUNTERS );free_mm (mm); }
@@ -1123,8 +1126,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,if (mm_alloc_cid (mm, p)) goto fail_cid; -if (percpu_counter_init_many (mm->rss_stat,0 ,GFP_KERNEL_ACCOUNT , -NR_MM_COUNTERS )) +if (percpu_counter_tree_init_many (mm->rss_stat,get_rss_stat_items (mm), +NR_MM_COUNTERS ,RSS_STAT_BATCH_SIZE , +GFP_KERNEL_ACCOUNT )) goto fail_pcpu; mm->user_ns =get_user_ns (user_ns);
@@ -3006,7 +3010,7 @@ void __init mm_cache_init(void)* dynamically sized based on the maximum CPU number this system * can have, taking hotplug into account (nr_cpu_ids). */ - mm_size =sizeof (struct mm_struct ) +cpumask_size () +mm_cid_size (); + mm_size =sizeof (struct mm_struct ) +cpumask_size () +mm_cid_size () +get_rss_stat_items_size (); mm_cachep = kmem_cache_create_usercopy("mm_struct", mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
Signature · 1 line
2.39.5
Mathieu Desnoyers
reviewerPATCH v14 3/3 mm: Implement precise OOM killer task selection
In reply to Mathieu DesnoyersUse the hierarchical tree counter approximation to implement the OOM killer task selection with a 2-pass algorithm. The first pass selects the process that has the highest badness points approximation, and the second pass compares each process using the current max badness points approximation.
The second pass uses an approximate comparison to eliminate all processes which are below the current max badness points approximation accuracy range.
Summing the per-CPU counters to calculate the precise badness of tasks is only required for tasks with an approximate badness within the accuracy range of the current max points value.
Limit to 16 the maximum number of badness sums allowed for an OOM killer task selection before falling back to the approximated comparison. This ensures bounded execution time for scenarios where many tasks have badness within the accuracy of the maximum badness approximation.
Tested with the following script:
#!/bin/sh
for a in $(seq 1 10); do (tail /dev/zero &); done sleep 5 for a in $(seq 1 10); do (tail /dev/zero &); done sleep 2 for a in $(seq 1 10); do (tail /dev/zero &); done echo "Waiting for tasks to finish" wait
Results: OOM kill order on a 128GB memory system
================================================
* systemd and sd-pam are chosen first due to their oom_score_ajd:100:
Out of memory: Killed process 3502 (systemd) total-vm:20096kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:72kB oom_score_adj:100 Out of memory: Killed process 3503 ((sd-pam)) total-vm:21432kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:76kB oom_score_adj:100
* The first batch of 10 processes are gradually killed, consecutively picking the one that uses the most memory. The fact that we are freeing memory from the previous processes increases the threshold at which the remaining processes of that group are killed. Processes from the second and third batches of 10 processes have time to start before we complete killing the first 10 processes:
Out of memory: Killed process 3703 (tail) total-vm:6591280kB, anon-rss:6578176kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12936kB oom_score_adj:0 Out of memory: Killed process 3705 (tail) total-vm:6731716kB, anon-rss:6709248kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13212kB oom_score_adj:0 Out of memory: Killed process 3707 (tail) total-vm:6977216kB, anon-rss:6946816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13692kB oom_score_adj:0 Out of memory: Killed process 3699 (tail) total-vm:7205640kB, anon-rss:7184384kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14136kB oom_score_adj:0 Out of memory: Killed process 3713 (tail) total-vm:7463204kB, anon-rss:7438336kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14644kB oom_score_adj:0 Out of memory: Killed process 3701 (tail) total-vm:7739204kB, anon-rss:7716864kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15180kB oom_score_adj:0 Out of memory: Killed process 3709 (tail) total-vm:8050176kB, anon-rss:8028160kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15792kB oom_score_adj:0 Out of memory: Killed process 3711 (tail) total-vm:8362236kB, anon-rss:8339456kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16404kB oom_score_adj:0 Out of memory: Killed process 3715 (tail) total-vm:8649360kB, anon-rss:8634368kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16972kB oom_score_adj:0 Out of memory: Killed process 3697 (tail) total-vm:8951788kB, anon-rss:8929280kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:17560kB oom_score_adj:0
* Even though there is a 2 seconds delay between the 2nd and 3rd batches those appear to execute in mixed order. Therefore, let's consider them as a single batch of 20 processes. We are hitting oom at a lower memory threshold because at this point the 20 remaining proceses are running rather than the previous 10. The process with highest memory usage is selected for oom, thus making room for the remaining processes so they can use more memory before they fill the available memory, thus explaining why the memory use for selected processes gradually increases, until all system memory is used by the last one:
Out of memory: Killed process 3731 (tail) total-vm:7089868kB, anon-rss:7077888kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13912kB oom_score_adj:0 Out of memory: Killed process 3721 (tail) total-vm:7417248kB, anon-rss:7405568kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14556kB oom_score_adj:0 Out of memory: Killed process 3729 (tail) total-vm:7795864kB, anon-rss:7766016kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15300kB oom_score_adj:0 Out of memory: Killed process 3723 (tail) total-vm:8259620kB, anon-rss:8224768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16208kB oom_score_adj:0 Out of memory: Killed process 3737 (tail) total-vm:8695984kB, anon-rss:8667136kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:17060kB oom_score_adj:0 Out of memory: Killed process 3735 (tail) total-vm:9295980kB, anon-rss:9265152kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:18240kB oom_score_adj:0 Out of memory: Killed process 3727 (tail) total-vm:9907900kB, anon-rss:9895936kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:19428kB oom_score_adj:0 Out of memory: Killed process 3719 (tail) total-vm:10631248kB, anon-rss:10600448kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:20844kB oom_score_adj:0 Out of memory: Killed process 3733 (tail) total-vm:11341720kB, anon-rss:11321344kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22232kB oom_score_adj:0 Out of memory: Killed process 3725 (tail) total-vm:12348124kB, anon-rss:12320768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:24204kB oom_score_adj:0 Out of memory: Killed process 3759 (tail) total-vm:12978888kB, anon-rss:12967936kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:25440kB oom_score_adj:0 Out of memory: Killed process 3751 (tail) total-vm:14386412kB, anon-rss:14352384kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:28196kB oom_score_adj:0 Out of memory: Killed process 3741 (tail) total-vm:16153168kB, anon-rss:16130048kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31652kB oom_score_adj:0 Out of memory: Killed process 3753 (tail) total-vm:18414856kB, anon-rss:18391040kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:36076kB oom_score_adj:0 Out of memory: Killed process 3745 (tail) total-vm:21389456kB, anon-rss:21356544kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:41904kB oom_score_adj:0 Out of memory: Killed process 3747 (tail) total-vm:25659348kB, anon-rss:25632768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:50260kB oom_score_adj:0 Out of memory: Killed process 3755 (tail) total-vm:32030820kB, anon-rss:32006144kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:62720kB oom_score_adj:0 Out of memory: Killed process 3743 (tail) total-vm:42648456kB, anon-rss:42614784kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:83504kB oom_score_adj:0 Out of memory: Killed process 3757 (tail) total-vm:63971028kB, anon-rss:63938560kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:125228kB oom_score_adj:0 Out of memory: Killed process 3749 (tail) total-vm:127799660kB, anon-rss:127778816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:250140kB oom_score_adj:0
---
Changes since v12:
- The oom_task_origin case needs to set points_min rather than points variable now, otherwise points_min is used without being initialized.
Changes since v11:
- get_mm_counter_sum() returns a precise sum.
- Use unsigned long type rather than unsigned int for accuracy.
- Use precise sum min/max calculation to compare the chosen vs current points.
- The first pass finds the maximum task's min points. The second pass eliminates all tasks for which the max points are below the currently chosen min points, and uses a precise sum to validate the candidates which are possibly in range.
--- fs/proc/base.c | 2 +- include/linux/mm.h | 34 +++++++++++++++--- include/linux/oom.h | 11 +++++- mm/oom_kill.c | 84 +++++++++++++++++++++++++++++++++++++-------- 4 files changed, 109 insertions(+), 22 deletions(-)
fs/proc/base.c +1 -1
@@ -589,7 +589,7 @@ static int proc_oom_score(struct seq_file *m, struct pid_namespace *ns,unsigned long points =0 ;long badness; - badness =oom_badness (task, totalpages); + badness =oom_badness (task, totalpages, false,NULL ,NULL );/* * Special case OOM_SCORE_ADJ_MIN for all others scale the * badness value into [0, 2000] range which we have been
include/linux/mm.h +29 -5
@@ -2855,14 +2855,32 @@ static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct m/* * per-process(per-mm_struct) statistics. */ +static inline unsigned long __get_mm_counter (struct mm_struct *mm,int member,bool approximate, +unsigned long *accuracy_under,unsigned long *accuracy_over) +{ +if (approximate) { +if (accuracy_under && accuracy_over) { +unsigned long under, over; + +percpu_counter_tree_approximate_accuracy_range (&mm->rss_stat[member], &under, &over); + *accuracy_under += under; + *accuracy_over += over; + } +return percpu_counter_tree_approximate_sum_positive (&mm->rss_stat[member]); + }else { +return percpu_counter_tree_precise_sum_positive (&mm->rss_stat[member]); + } +} +static inline unsigned long get_mm_counter (struct mm_struct *mm,int member) { -return percpu_counter_tree_approximate_sum_positive (&mm->rss_stat[member]); +return __get_mm_counter (mm, member, true,NULL ,NULL ); } +static inline unsigned long get_mm_counter_sum (struct mm_struct *mm,int member) { -return percpu_counter_tree_precise_sum_positive (&mm->rss_stat[member]); +return __get_mm_counter (mm, member, false,NULL ,NULL ); }void mm_trace_rss_stat (struct mm_struct *mm,int member);
@@ -2903,11 +2921,17 @@ static inline int mm_counter(struct folio *folio)return mm_counter_file (folio); } +static inline unsigned long __get_mm_rss (struct mm_struct *mm,bool approximate, +unsigned long *accuracy_under,unsigned long *accuracy_over) +{ +return __get_mm_counter (mm,MM_FILEPAGES , approximate, accuracy_under, accuracy_over) + +__get_mm_counter (mm,MM_ANONPAGES , approximate, accuracy_under, accuracy_over) + +__get_mm_counter (mm,MM_SHMEMPAGES , approximate, accuracy_under, accuracy_over); +} +static inline unsigned long get_mm_rss (struct mm_struct *mm) { -return get_mm_counter (mm,MM_FILEPAGES ) + -get_mm_counter (mm,MM_ANONPAGES ) + -get_mm_counter (mm,MM_SHMEMPAGES ); +return __get_mm_rss (mm, true,NULL ,NULL ); }static inline unsigned long get_mm_hiwater_rss (struct mm_struct *mm)
include/linux/oom.h +10 -1
@@ -48,6 +48,12 @@ struct oom_control {unsigned long totalpages;struct task_struct *chosen;long chosen_points; +bool approximate; +/* +* Number of precise badness points sums performed by this task +* selection. +*/ +int nr_precise;/* Used to print the constraint info. */ enum oom_constraint constraint;
@@ -97,7 +103,10 @@ static inline vm_fault_t check_stable_address_space(struct mm_struct *mm) }long oom_badness (struct task_struct *p, -unsigned long totalpages); +unsigned long totalpages, +bool approximate, +unsigned long *accuracy_under, +unsigned long *accuracy_over);extern bool out_of_memory (struct oom_control *oc);
mm/oom_kill.c +69 -15
141 lines not on this page · raw message
Signature · 1 line
2.39.5
Andrew Morton
maintainerQuoted 3 lines · On Mon, 12 Jan 2026 15:00:53 -0500 Mathieu Desnoyers…
Introduce hierarchical per-cpu counters and use them for RSS tracking to fix the per-mm RSS tracking which has become too inaccurate for OOM killer purposes on large many-core systems.
Thanks, I'll throw this back into the testing pot while monitoring the discussion with Michal.