memcg: clean up memory.high enforcement
Subsystems (3)
Shakeel Butt
maintainerv2 cover memcg: clean up memory.high enforcement
This series cleans up memory.high enforcement and uses a worker for kernel threads and remote charges.
Kernel threads never return to userspace to run deferred reclaim. Reclaiming and sleeping during a charge can also delay work they do for other tasks.
Remote charges can go to a different cgroup from the caller's mm, for example through filesystem notifications or faults in another process's memory. The high handler uses current->mm, so it can check the caller's limits instead of the limits of the charged cgroup.
Use high_work to reclaim from the charged hierarchy in these cases. mm_match_cgroup() identifies charges outside the current mm's cgroup and its ancestors. Local user charges keep their current handling.
Before that, force charges from PF_MEMALLOC tasks before trying the limits, so they skip memory.high enforcement. Reclaim can charge memory itself, for example when zswap stores a page, and the high_work worker must not queue itself from its own reclaim. These charges still raise MEMCG_MAX for the same cgroup as before.
This is best effort. A fast allocator can keep usage well above memory.high; memory.max stays the hard limit, as it already is for interrupts. The worker paths also skip memory.swap.high throttling and its high-event reporting. memory.swap.high only throttles the charging task, and a worker has no task to throttle. The cgroup's own tasks are still throttled on their own charges, and memory.swap.max still caps swap.
The first patch moves enforcement into memcg_enforce_high(). The second forces charges from reclaim before the limit checks. The next two add kernel thread and remote charge handling. The last simplifies the scan and comments. The first and last patches have no intended behavior change.
Based on mm-everything-2026-10-29-04-31. v1 was tested in an x86-64 VM with loop writes, threaded NAPI, remote faults, notifications, and both vhost worker modes. For v2, each patch was built with GCC and Clang (W=1). In a VM test with incompressible zswap data and remote faults, the high_work worker queued itself from its own reclaim in 118 of 284 runs without the PF_MEMALLOC patch, and never with the series. The runs were counted with bpftrace on queue_work_on() inside high_work_func(). In another test, a parent cgroup was over memory.max while its child was reclaimed. The child's zswap charges raised MEMCG_MAX on the parent 7133 times with the series and 7124 times without it.
Changes in v2:
- Rebased on mm-everything-2026-10-29-04-31.
- Collected Reviewed-by and Acked-by tags.
- Patch 1: add the missing Signed-off-by (Harry).
- Patch 2 (was 4): replaced with Michal's suggestion. Charges from PF_MEMALLOC tasks are now forced before trying the limits, instead of skipping enforcement in memcg_enforce_high(). MEMCG_MAX is still raised for the first cgroup whose limit the charge goes over. Moved before the kernel thread patch, so that no commit lets the worker queue itself. (Michal)
- Patch 4: say more clearly which charges keep the local path.
- Patch 5: drop locals that are no longer needed, and update the comments.
v1: https://lore.kernel.org/[email protected]
Shakeel Butt (5):
memcg: move memory.high enforcement out of try_charge_memcg()
memcg: force charges from reclaim before trying the limits
memcg: use high_work for kernel threads
memcg: use high_work for remote charges
memcg: simplify high limit handling after a charge
mm/memcontrol.c | 157 +++++++++++++++++++++++++++--------------------- 1 file changed, 87 insertions(+), 70 deletions(-)
base-commit: 7e12256cff20c704fd5c0fbbd6a070fda8c341c9
Signature · 1 line
2.53.0-Meta
Shakeel Butt
maintainerPATCH v2 1/5 memcg: move memory.high enforcement out of try_charge_memcg()
try_charge_memcg() does a lot. It charges the page counters, reclaims when a limit is hit, may call the OOM killer, and at the end it checks and enforces memory.high. Move the memory.high part into a new function, memcg_enforce_high(), so that try_charge_memcg() is shorter and the memory.high code is easier to read and change.
The code and its comments move as they are. The only difference is that the new function gets allow_spinning from gfp_mask itself.
No functional change.
--- mm/memcontrol.c | 125 ++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 56 deletions(-)
mm/memcontrol.c +69 -56
@@ -2699,6 +2699,74 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)css_put (&memcg->css); } +/* +* Called after @nr_pages were charged to @memcg. If @memcg or one of its +* ancestors is over memory.high or memory.swap.high, start reclaim or slow +* down the task. +*/ +static void memcg_enforce_high (struct mem_cgroup *memcg,unsigned int nr_pages, +gfp_t gfp_mask) +{ +bool allow_spinning =gfpflags_allow_spinning (gfp_mask); + +/* +* If the hierarchy is above the normal consumption range, schedule +* reclaim on returning to userland. We can perform reclaim here +* if __GFP_RECLAIM but let's always punt for simplicity and so that +* GFP_KERNEL can consistently be used during reclaim. @memcg is +* not recorded as it most likely matches current's and won't +* change in the meantime. As high limit is checked again before +* reclaim, the cost of mismatch is negligible. +*/ +do { +bool mem_high, swap_high; + + mem_high =page_counter_read (&memcg->memory) > +READ_ONCE (memcg->memory.high); + swap_high =page_counter_read (&memcg->swap) > +READ_ONCE (memcg->swap.high); + +/* Don't bother a random interrupted task */ +if (!in_task ()) { +if (mem_high) { +if (allow_spinning) +schedule_work (&memcg->high_work); +else +irq_work_queue (&memcg->high_irq_work); +break ; + } +continue ; + } + +if (mem_high || swap_high) { +/* +* The allocating tasks in this cgroup will need to do +* reclaim or be throttled to prevent further growth +* of the memory or swap footprints. +* +* Target some best-effort fairness between the tasks, +* and distribute reclaim work and delay penalties +* based on how much each task is actually allocating. +*/ + current->memcg_nr_pages_over_high += nr_pages; +set_notify_resume (current); +break ; + } + }while ((memcg =parent_mem_cgroup (memcg))); + +/* +* Reclaim is set up above to be called from the userland +* return path. But also attempt synchronous reclaim to avoid +* excessive overrun while the task is still inside the +* kernel. If this is successful, the return path will see it +* when it rechecks the overage and simply bail out. +*/ +if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH && + !(current->flags &PF_MEMALLOC ) && +gfpflags_allow_blocking (gfp_mask)) +__mem_cgroup_handle_over_high (gfp_mask); +} +static int try_charge_memcg (struct mem_cgroup *memcg,gfp_t gfp_mask,unsigned int nr_pages) {
@@ -2856,62 +2924,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (batch > nr_pages)refill_stock (memcg, batch - nr_pages); -/* -* If the hierarchy is above the normal consumption range, schedule -* reclaim on returning to userland. We can perform reclaim here -* if __GFP_RECLAIM but let's always punt for simplicity and so that -* GFP_KERNEL can consistently be used during reclaim. @memcg is -* not recorded as it most likely matches current's and won't -* change in the meantime. As high limit is checked again before -* reclaim, the cost of mismatch is negligible. -*/ -do { -bool mem_high, swap_high; - - mem_high =page_counter_read (&memcg->memory) > -READ_ONCE (memcg->memory.high); - swap_high =page_counter_read (&memcg->swap) > -READ_ONCE (memcg->swap.high); - -/* Don't bother a random interrupted task */ -if (!in_task ()) { -if (mem_high) { -if (allow_spinning) -schedule_work (&memcg->high_work); -else -irq_work_queue (&memcg->high_irq_work); -break ; - } -continue ; - } - -if (mem_high || swap_high) { -/* -* The allocating tasks in this cgroup will need to do -* reclaim or be throttled to prevent further growth -* of the memory or swap footprints. -* -* Target some best-effort fairness between the tasks, -* and distribute reclaim work and delay penalties -* based on how much each task is actually allocating. -*/ - current->memcg_nr_pages_over_high += batch; -set_notify_resume (current); -break ; - } - }while ((memcg =parent_mem_cgroup (memcg))); - -/* -* Reclaim is set up above to be called from the userland -* return path. But also attempt synchronous reclaim to avoid -* excessive overrun while the task is still inside the -* kernel. If this is successful, the return path will see it -* when it rechecks the overage and simply bail out. -*/ -if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH && - !(current->flags &PF_MEMALLOC ) && -gfpflags_allow_blocking (gfp_mask)) -__mem_cgroup_handle_over_high (gfp_mask); +memcg_enforce_high (memcg, batch, gfp_mask);return ret; }
Signature · 1 line
2.53.0-Meta
Re: [PATCH v2 1/5] memcg: move memory.high enforcement out of try_charge_memcg()
Quoted 10 lines · Shakeel Butt
try_charge_memcg()does a lot. It charges the page counters, reclaims when a limit is hit, may call the OOM killer, and at the end it checks and enforces memory.high. Move the memory.high part into a new function,memcg_enforce_high(), so thattry_charge_memcg()is shorter and the memory.high code is easier to read and change.
Quoted 8 lines
The code and its comments move as they are. The only difference is that the new function gets
allow_spinningfromgfp_maskitself.No functional change.
Signed-off-by Shakeel Butt <[email protected]>Reviewed-by Harry Yoo (Meta) <[email protected]>Acked-by Michal Hocko <[email protected]>
Thanks.
Shakeel Butt
maintainerPATCH v2 2/5 memcg: force charges from reclaim before trying the limits
In reply to Shakeel ButtReclaim can charge memory, for example when zswap stores a page. try_charge_memcg() forces these PF_MEMALLOC charges, but only after they fail at a limit. A charge that fits still has memory.high enforced. That can make a task in direct reclaim do more reclaim and sleep longer later, and kswapd records overage that it never handles.
It is also a problem for the next patches, which make kernel threads queue high_work. The worker charges memory from its own reclaim, so it can queue itself again. With those patches but not this one, the worker queued itself in 118 of 284 runs in a VM test with incompressible zswap data.
Check PF_MEMALLOC before trying the limits. Reclaim charges then skip memory.high, as they can already go over memory.max. Drop the PF_MEMALLOC check in memcg_enforce_high(), which no longer sees them.
Other effects:
- As before, the check reads current, so it also catches interrupts on a PF_MEMALLOC task such as kswapd. Softirqs set PF_MEMALLOC themselves for swap over network, so the check can't be limited to task context.
- These charges no longer charge a batch or refill the per-cpu stock.
- MEMCG_MAX still goes to the first cgroup that the charge puts over its limit. On cgroup v1, memory.failcnt no longer counts them.
- high_work can still run again and again when there is nothing left to reclaim. This only stops reclaim from queuing more reclaim.
--- mm/memcontrol.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
mm/memcontrol.c +17 -12
@@ -2762,7 +2762,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,* when it rechecks the overage and simply bail out. */ if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH && - !(current->flags &PF_MEMALLOC ) &&gfpflags_allow_blocking (gfp_mask))__mem_cgroup_handle_over_high (gfp_mask); }
@@ -2787,6 +2786,21 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (consume_stock (memcg, nr_pages))return ret; +/* +* Reclaim may need memory to free memory. Let it go over the +* limits for a while rather than fail or OOM kill. Doing this +* before trying the limits also skips memory.high, so the +* high_work worker can't queue itself from its own reclaim. +*/ +if (unlikely (current->flags &PF_MEMALLOC )) { +/* Find the cgroup this charge puts over its limit, if any. */ +for (mem_over_limit = memcg; mem_over_limit; + mem_over_limit =parent_mem_cgroup (mem_over_limit)) +if (mem_cgroup_margin (mem_over_limit) < nr_pages) +break ; + goto force; + } +if (!allow_spinning)/* Avoid the refill and flush of the older stock */ batch = nr_pages;
@@ -2809,15 +2823,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, goto retry; } -/* -* Prevent unbounded recursion when reclaim operations need to -* allocate memory. This might exceed the limits temporarily, -* but we prefer facilitating memory reclaim and getting back -* under the limit over triggering OOM kills in these cases. -*/ -if (unlikely (current->flags &PF_MEMALLOC )) - goto force; -if (unlikely (task_in_memcg_oom (current))) goto nomem;
@@ -2913,9 +2918,9 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, out:/* * Don't forget to raise a MEMCG_MAX event for forced or rejected -* requests. +* requests. mem_over_limit is NULL if a PF_MEMALLOC charge fits. */ -if (!raised_max_event) +if (!raised_max_event && mem_over_limit)__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning);return ret;
Signature · 1 line
2.53.0-Meta
Re: [PATCH v2 2/5] memcg: force charges from reclaim before trying the limits
Quoted 31 lines · Shakeel Butt
Reclaim can charge memory, for example when zswap stores a page.
try_charge_memcg()forces thesePF_MEMALLOCcharges, but only after they fail at a limit. A charge that fits still has memory.high enforced. That can make a task in direct reclaim do more reclaim and sleep longer later, and kswapd records overage that it never handles.It is also a problem for the next patches, which make kernel threads queue
high_work. The worker charges memory from its own reclaim, so it can queue itself again. With those patches but not this one, the worker queued itself in 118 of 284 runs in a VM test with incompressible zswap data.Check
PF_MEMALLOCbefore trying the limits. Reclaim charges then skip memory.high, as they can already go over memory.max. Drop thePF_MEMALLOCcheck inmemcg_enforce_high(), which no longer sees them.Other effects:
- As before, the check reads current, so it also catches interrupts on aPF_MEMALLOCtask such as kswapd. Softirqs setPF_MEMALLOCthemselves for swap over network, so the check can't be limited to task context.
- These charges no longer charge a batch or refill the per-cpu stock.
-MEMCG_MAXstill goes to the first cgroup that the charge puts over its limit. On cgroup v1, memory.failcnt no longer counts them.
-high_workcan still run again and again when there is nothing left to reclaim. This only stops reclaim from queuing more reclaim.
Quoted 2 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Shakeel Butt <[email protected]>
Thanks.
Re: [PATCH v2 2/5] memcg: force charges from reclaim before trying the limits
In reply to Shakeel ButtQuoted 53 lines · Shakeel Butt
Reclaim can charge memory, for example when zswap stores a page.
try_charge_memcg()forces thesePF_MEMALLOCcharges, but only after they fail at a limit. A charge that fits still has memory.high enforced. That can make a task in direct reclaim do more reclaim and sleep longer later, and kswapd records overage that it never handles.It is also a problem for the next patches, which make kernel threads queue
high_work. The worker charges memory from its own reclaim, so it can queue itself again. With those patches but not this one, the worker queued itself in 118 of 284 runs in a VM test with incompressible zswap data.Check
PF_MEMALLOCbefore trying the limits. Reclaim charges then skip memory.high, as they can already go over memory.max. Drop thePF_MEMALLOCcheck inmemcg_enforce_high(), which no longer sees them.Other effects:
- As before, the check reads current, so it also catches interrupts on aPF_MEMALLOCtask such as kswapd. Softirqs setPF_MEMALLOCthemselves for swap over network, so the check can't be limited to task context.
- These charges no longer charge a batch or refill the per-cpu stock.
-MEMCG_MAXstill goes to the first cgroup that the charge puts over its limit. On cgroup v1, memory.failcnt no longer counts them.
-high_workcan still run again and again when there is nothing left to reclaim. This only stops reclaim from queuing more reclaim.Suggested-by Michal Hocko <[email protected]>Signed-off-by Shakeel Butt <[email protected]>--- mm/memcontrol.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)mm/memcontrol.c +4
-1@@ -2762,7 +2762,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,* when it rechecks the overage and simply bail out. */ if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH && - !(current->flags &PF_MEMALLOC ) &&gfpflags_allow_blocking (gfp_mask))__mem_cgroup_handle_over_high (gfp_mask); }@@ -2787,6 +2786,21 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (consume_stock (memcg, nr_pages))return ret; +/* +* Reclaim may need memory to free memory. Let it go over the +* limits for a while rather than fail or OOM kill. Doing this +* before trying the limits also skips memory.high, so the
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2802
Quoted 10 lines
mm/memcontrol.c +10
-0+* high_work worker can't queue itself from its own reclaim. +*/ +if (unlikely (current->flags &PF_MEMALLOC )) { +/* Find the cgroup this charge puts over its limit, if any. */ +for (mem_over_limit = memcg; mem_over_limit; + mem_over_limit =parent_mem_cgroup (mem_over_limit)) +if (mem_cgroup_margin (mem_over_limit) < nr_pages) +break ; + goto force; + }
This is racy. Not a huge deal as we would get a false event in the worst case but I think it would make more sense to check the margine after the charge is forced. So this would become
---
mm/memcontrol.c +17 -18
@@ -2653,13 +2653,20 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,bool passed_oom = false;unsigned int reclaim_options;bool drained = false; -bool raised_max_event = false;unsigned long pflags;bool allow_spinning =gfpflags_allow_spinning (gfp_mask); retry:if (consume_stock (memcg, nr_pages))return 0 ; +/* +* Reclaim may need memory to free memory. Let it go over the +* limits for a while rather than fail or OOM kill. Doing this +* before trying the limits also skips memory.high, so the +* high_work worker can't queue itself from its own reclaim. +*/ +if (unlikely (current->flags &PF_MEMALLOC )) + goto force;if (!allow_spinning)/* Avoid the refill and flush of the older stock */
@@ -2683,15 +2690,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, goto retry; } -/* -* Prevent unbounded recursion when reclaim operations need to -* allocate memory. This might exceed the limits temporarily, -* but we prefer facilitating memory reclaim and getting back -* under the limit over triggering OOM kills in these cases. -*/ -if (unlikely (current->flags &PF_MEMALLOC )) - goto force; -if (unlikely (task_in_memcg_oom (current))) goto nomem;
@@ -2712,7 +2710,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, goto nomem;__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); - raised_max_event = true;psi_memstall_enter (&pflags); nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
@@ -2773,13 +2770,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (!(gfp_mask & (__GFP_NOFAIL | __GFP_HIGH)))return -ENOMEM ; force: -/* -* If the allocation has to be enforced, don't forget to raise -* a MEMCG_MAX event. -*/ -if (!raised_max_event) -__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); -/* * The allocation either can't fail or will lead to more memory * being freed very soon. Allow memory usage go over the limit
@@ -2789,6 +2779,15 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (do_memsw_account ())page_counter_charge (&memcg->memsw, nr_pages); +/* +* If the allocation has to be enforced, don't forget to raise +* a MEMCG_MAX event. +*/ +for (mem_over_limit = memcg; mem_over_limit; + mem_over_limit =parent_mem_cgroup (mem_over_limit)) +if (!mem_cgroup_margin (mem_over_limit)) +__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); +return 0 ; done_restock:
Signature · 2 lines
Michal Hocko SUSE Labs
Shakeel Butt
maintainerRe: [PATCH v2 2/5] memcg: force charges from reclaim before trying the limits
Quoted 60 lines · Michal Hocko
Quoted 59 lines · Shakeel Butt
Reclaim can charge memory, for example when zswap stores a page.
try_charge_memcg()forces thesePF_MEMALLOCcharges, but only after they fail at a limit. A charge that fits still has memory.high enforced. That can make a task in direct reclaim do more reclaim and sleep longer later, and kswapd records overage that it never handles.It is also a problem for the next patches, which make kernel threads queue
high_work. The worker charges memory from its own reclaim, so it can queue itself again. With those patches but not this one, the worker queued itself in 118 of 284 runs in a VM test with incompressible zswap data.Check
PF_MEMALLOCbefore trying the limits. Reclaim charges then skip memory.high, as they can already go over memory.max. Drop thePF_MEMALLOCcheck inmemcg_enforce_high(), which no longer sees them.Other effects:
- As before, the check reads current, so it also catches interrupts on aPF_MEMALLOCtask such as kswapd. Softirqs setPF_MEMALLOCthemselves for swap over network, so the check can't be limited to task context.
- These charges no longer charge a batch or refill the per-cpu stock.
-MEMCG_MAXstill goes to the first cgroup that the charge puts over its limit. On cgroup v1, memory.failcnt no longer counts them.
-high_workcan still run again and again when there is nothing left to reclaim. This only stops reclaim from queuing more reclaim.Suggested-by Michal Hocko <[email protected]>Signed-off-by Shakeel Butt <[email protected]>--- mm/memcontrol.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)mm/memcontrol.c +10
-1@@ -2762,7 +2762,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,* when it rechecks the overage and simply bail out. */ if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH && - !(current->flags &PF_MEMALLOC ) &&gfpflags_allow_blocking (gfp_mask))__mem_cgroup_handle_over_high (gfp_mask); }@@ -2787,6 +2786,21 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (consume_stock (memcg, nr_pages))return ret; +/* +* Reclaim may need memory to free memory. Let it go over the +* limits for a while rather than fail or OOM kill. Doing this +* before trying the limits also skips memory.high, so the +* high_work worker can't queue itself from its own reclaim. +*/ +if (unlikely (current->flags &PF_MEMALLOC )) { +/* Find the cgroup this charge puts over its limit, if any. */ +for (mem_over_limit = memcg; mem_over_limit; + mem_over_limit =parent_mem_cgroup (mem_over_limit))
Quoted 8 lines
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2802
Quoted 4 lines
mm/memcontrol.c +4
-0+if (mem_cgroup_margin (mem_over_limit) < nr_pages) +break ; + goto force; + }This is racy. Not a huge deal as we would get a false event in the worst case but I think it would make more sense to check the margine after the charge is forced. So this would become
I actually tested your version as well i.e. after the charge. It is just as racy and it has some additional issues which are inlined.
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2656
Quoted 10 lines
---
mm/memcontrol.c +0
-1@@ -2653,13 +2653,20 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,bool passed_oom = false;unsigned int reclaim_options;bool drained = false; -bool raised_max_event = false;
First, raised_max_event can't go. Since 374f65b14819 rejected charges use it too, and without it forced charges that already raised MEMCG_MAX before reclaim raise it again.
Quoted 56 lines
mm/memcontrol.c +8
-17unsigned long pflags;bool allow_spinning =gfpflags_allow_spinning (gfp_mask); retry:if (consume_stock (memcg, nr_pages))return 0 ; +/* +* Reclaim may need memory to free memory. Let it go over the +* limits for a while rather than fail or OOM kill. Doing this +* before trying the limits also skips memory.high, so the +* high_work worker can't queue itself from its own reclaim. +*/ +if (unlikely (current->flags &PF_MEMALLOC )) + goto force;if (!allow_spinning)/* Avoid the refill and flush of the older stock */ @@ -2683,15 +2690,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, goto retry; } -/* -* Prevent unbounded recursion when reclaim operations need to -* allocate memory. This might exceed the limits temporarily, -* but we prefer facilitating memory reclaim and getting back -* under the limit over triggering OOM kills in these cases. -*/ -if (unlikely (current->flags &PF_MEMALLOC )) - goto force; -if (unlikely (task_in_memcg_oom (current))) goto nomem;@@ -2712,7 +2710,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, goto nomem;__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); - raised_max_event = true;psi_memstall_enter (&pflags); nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,@@ -2773,13 +2770,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (!(gfp_mask & (__GFP_NOFAIL | __GFP_HIGH)))return -ENOMEM ; force: -/* -* If the allocation has to be enforced, don't forget to raise -* a MEMCG_MAX event. -*/ -if (!raised_max_event) -__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning); -/* * The allocation either can't fail or will lead to more memory * being freed very soon. Allow memory usage go over the limit
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2788
Quoted 10 lines
mm/memcontrol.c +7
-0@@ -2789,6 +2779,15 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (do_memsw_account ())page_counter_charge (&memcg->memsw, nr_pages); +/* +* If the allocation has to be enforced, don't forget to raise +* a MEMCG_MAX event. +*/ +for (mem_over_limit = memcg; mem_over_limit; + mem_over_limit =parent_mem_cgroup (mem_over_limit)) +if (!mem_cgroup_margin (mem_over_limit))
Second, !mem_cgroup_margin() is true at usage == max, so a charge that fits exactly raises an event (unlike today).
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2789
Quoted 1 line
mm/memcontrol.c +1
-0+__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning);
Third, we definitely need a break here otherwise every ancestor at its limit gets an event which does not happen today.
I want to keep my current version.
Michal Hocko
Acked-byRe: [PATCH v2 2/5] memcg: force charges from reclaim before trying the limits
Quoted 1 line · Shakeel Butt
On Mon, Nov 02, 2026 at 09:46:38AM +0100, Michal Hocko wrote:
[...]
Quoted 4 lines
Quoted 4 lines
mm/memcontrol.c +0
-0@@ -2789,6 +2779,15 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,if (do_memsw_account ())page_counter_charge (&memcg->memsw, nr_pages);
Quoted 10 lines
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2788
Quoted 7 lines
mm/memcontrol.c +7
-0+/* +* If the allocation has to be enforced, don't forget to raise +* a MEMCG_MAX event. +*/ +for (mem_over_limit = memcg; mem_over_limit; + mem_over_limit =parent_mem_cgroup (mem_over_limit)) +if (!mem_cgroup_margin (mem_over_limit))Second, !mem_cgroup_margin() is true at usage == max, so a charge that fits exactly raises an event (unlike today).
True
Quoted 5 lines
mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2789
Quoted 2 lines
mm/memcontrol.c +1
-0+__memcg_memory_event (mem_over_limit,MEMCG_MAX , allow_spinning);Third, we definitely need a break here otherwise every ancestor at its limit gets an event which does not happen today.
This is a separate thing but shouldn't we actually raise the event on all levels that are over?
Quoted 1 line
I want to keep my current version.
OK, fair enough. My intention was to have those two parts closer together but this is not something I would insist.
Thanks!
Signature · 2 lines
Michal Hocko SUSE Labs
Shakeel Butt
maintainerPATCH v2 3/5 memcg: use high_work for kernel threads
In reply to Shakeel ButtKernel threads never return to userspace, so deferred memory.high reclaim does not run. Reclaim and throttling during a charge can also delay work they do for other tasks.
Treat kernel threads like interrupts. Queue high_work for the first cgroup over memory.high and skip reclaim and throttling in the charging thread.
User workers created by create_io_thread() and vhost_task_create() keep their current behavior. Vhost kernel threads use the new path, even when they borrow the owner's mm.
The worker requests a fixed batch of pages each time. A fast allocator can keep usage well above memory.high. Kernel threads also skip memory.swap.high throttling and its high-event reporting. The memory.max and memory.swap.max checks stay the same.
--- mm/memcontrol.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
mm/memcontrol.c +11 -2
@@ -2708,6 +2708,7 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, gfp_t gfp_mask) {bool allow_spinning =gfpflags_allow_spinning (gfp_mask); +bool use_worker = !in_task () || (current->flags &PF_KTHREAD );/* * If the hierarchy is above the normal consumption range, schedule
@@ -2726,8 +2727,12 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, swap_high =page_counter_read (&memcg->swap) >READ_ONCE (memcg->swap.high); -/* Don't bother a random interrupted task */ -if (!in_task ()) { +/* +* Don't make an unrelated interrupted task handle this charge. +* Kernel threads may be working for other tasks, so let a worker +* reclaim instead. +*/ +if (use_worker) {if (mem_high) {if (allow_spinning)schedule_work (&memcg->high_work);
@@ -2754,6 +2759,10 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, } }while ((memcg =parent_mem_cgroup (memcg))); +/* Interrupts and kernel threads stop here. */ +if (use_worker) +return ; +/* * Reclaim is set up above to be called from the userland * return path. But also attempt synchronous reclaim to avoid
Signature · 1 line
2.53.0-Meta
Re: [PATCH v2 3/5] memcg: use high_work for kernel threads
Quoted 16 lines · Shakeel Butt
Kernel threads never return to userspace, so deferred memory.high reclaim does not run. Reclaim and throttling during a charge can also delay work they do for other tasks.
Treat kernel threads like interrupts. Queue
high_workfor the first cgroup over memory.high and skip reclaim and throttling in the charging thread.User workers created by
create_io_thread()andvhost_task_create()keep their current behavior. Vhost kernel threads use the new path, even when they borrow the owner's mm.
Quoted 8 lines
The worker requests a fixed batch of pages each time. A fast allocator can keep usage well above memory.high. Kernel threads also skip memory.swap.high throttling and its high-event reporting. The memory.max and memory.swap.max checks stay the same.
Suggested-by Tejun Heo <[email protected]>Signed-off-by Shakeel Butt <[email protected]>Acked-by Michal Hocko <[email protected]>
THanks.
Shakeel Butt
maintainerPATCH v2 4/5 memcg: use high_work for remote charges
In reply to Shakeel ButtSome charges go to a cgroup outside the current mm's cgroup and its ancestors. The high handler uses current->mm, so it cannot handle these overages. Filesystem notifications and remote page faults can take this path.
Use mm_match_cgroup() before checking the high limits. If it does not match, use high_work instead of recording task overage or doing synchronous reclaim and throttling. Charges to the current mm's cgroup or one of its ancestors keep their current handling. Tasks without an mm also use the worker.
Like kernel threads, remote chargers skip memory.swap.high throttling and its high-event reporting, and a fast allocator can outrun the worker. The memory.max and memory.swap.max checks stay the same.
--- mm/memcontrol.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
mm/memcontrol.c +4 -2
@@ -2708,7 +2708,8 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, gfp_t gfp_mask) {bool allow_spinning =gfpflags_allow_spinning (gfp_mask); -bool use_worker = !in_task () || (current->flags &PF_KTHREAD ); +bool use_worker = !in_task () || (current->flags &PF_KTHREAD ) || + !current->mm || !mm_match_cgroup (current->mm, memcg);/* * If the hierarchy is above the normal consumption range, schedule
@@ -2731,6 +2732,7 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,* Don't make an unrelated interrupted task handle this charge. * Kernel threads may be working for other tasks, so let a worker * reclaim instead. +* Remote charges need reclaim from their charged hierarchy. */ if (use_worker) {if (mem_high) {
@@ -2759,7 +2761,7 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, } }while ((memcg =parent_mem_cgroup (memcg))); -/* Interrupts and kernel threads stop here. */ +/* Interrupts, kernel threads, and remote chargers stop here. */ if (use_worker)return ;
Signature · 1 line
2.53.0-Meta
Re: [PATCH v2 4/5] memcg: use high_work for remote charges
Quoted 15 lines · Shakeel Butt
Some charges go to a cgroup outside the current mm's cgroup and its ancestors. The high handler uses
current->mm, so it cannot handle these overages. Filesystem notifications and remote page faults can take this path.Use
mm_match_cgroup()before checking the high limits. If it does not match, usehigh_workinstead of recording task overage or doing synchronous reclaim and throttling. Charges to the current mm's cgroup or one of its ancestors keep their current handling. Tasks without an mm also use the worker.
Quoted 7 lines
Like kernel threads, remote chargers skip memory.swap.high throttling and its high-event reporting, and a fast allocator can outrun the worker. The memory.max and memory.swap.max checks stay the same.
Suggested-by Tejun Heo <[email protected]>Signed-off-by Shakeel Butt <[email protected]>Acked-by Michal Hocko <[email protected]>
Thanks.
Shakeel Butt
maintainerPATCH v2 5/5 memcg: simplify high limit handling after a charge
In reply to Shakeel ButtSeparate the hierarchy scan from high-limit enforcement. Find the first cgroup over the applicable limit, then queue high_work or record the task's overage after the scan. Worker contexts check only memory.high; local charges also check memory.swap.high.
Clarify the enforcement comments. Keep the synchronous pending-overage check independent of the scan result so a blocking charge still handles overage left by earlier charges.
No functional change intended.
--- mm/memcontrol.c | 102 +++++++++++++++++++++--------------------------- 1 file changed, 45 insertions(+), 57 deletions(-)
mm/memcontrol.c +45 -57
@@ -2605,9 +2605,8 @@ static unsigned long calculate_high_delay(unsigned int nr_pages, }/* -* Reclaims memory over the high limit. Called directly from -* try_charge() (context permitting), as well as from the userland -* return path where reclaim is always able to block. +* Handle pending high overage during a blocking charge or on return +* to userspace. */ void __mem_cgroup_handle_over_high (gfp_t gfp_mask) {
@@ -2701,76 +2700,65 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)/* * Called after @nr_pages were charged to @memcg. If @memcg or one of its -* ancestors is over memory.high or memory.swap.high, start reclaim or slow -* down the task. +* ancestors is over its high limit, either queue high_work to reclaim from +* that cgroup, or record the overage so the charging task reclaims later +* and sleeps if reclaim can't keep up. high_work only handles memory.high; +* memory.swap.high is only enforced on the charging task. */ static void memcg_enforce_high (struct mem_cgroup *memcg,unsigned int nr_pages,gfp_t gfp_mask) { +struct mem_cgroup *iter;bool allow_spinning =gfpflags_allow_spinning (gfp_mask); -bool use_worker = !in_task () || (current->flags &PF_KTHREAD ) || +bool use_high_work = !in_task () || (current->flags &PF_KTHREAD ) || !current->mm || !mm_match_cgroup (current->mm, memcg);/* -* If the hierarchy is above the normal consumption range, schedule -* reclaim on returning to userland. We can perform reclaim here -* if __GFP_RECLAIM but let's always punt for simplicity and so that -* GFP_KERNEL can consistently be used during reclaim. @memcg is -* not recorded as it most likely matches current's and won't -* change in the meantime. As high limit is checked again before -* reclaim, the cost of mismatch is negligible. +* Check the charged cgroup and its parents. Local charges normally defer +* reclaim until return to userspace so it can use GFP_KERNEL. Large +* pending overages may be handled synchronously below. The handler looks +* up current->mm and rechecks the limits. If the task moved to another +* cgroup in the meantime, the handler works on the new cgroup instead. +* That is rare and costs little. */ -do { +for (iter = memcg; iter; iter =parent_mem_cgroup (iter)) {bool mem_high, swap_high; - mem_high =page_counter_read (&memcg->memory) > -READ_ONCE (memcg->memory.high); - swap_high =page_counter_read (&memcg->swap) > -READ_ONCE (memcg->swap.high); - -/* -* Don't make an unrelated interrupted task handle this charge. -* Kernel threads may be working for other tasks, so let a worker -* reclaim instead. -* Remote charges need reclaim from their charged hierarchy. -*/ -if (use_worker) { -if (mem_high) { -if (allow_spinning) -schedule_work (&memcg->high_work); -else -irq_work_queue (&memcg->high_irq_work); -break ; - } -continue ; - } - -if (mem_high || swap_high) { -/* -* The allocating tasks in this cgroup will need to do -* reclaim or be throttled to prevent further growth -* of the memory or swap footprints. -* -* Target some best-effort fairness between the tasks, -* and distribute reclaim work and delay penalties -* based on how much each task is actually allocating. -*/ - current->memcg_nr_pages_over_high += nr_pages; -set_notify_resume (current); + mem_high =page_counter_read (&iter->memory) > +READ_ONCE (iter->memory.high); + swap_high = !use_high_work && + (page_counter_read (&iter->swap) >READ_ONCE (iter->swap.high)); +if (mem_high || swap_high)break ; - } - }while ((memcg =parent_mem_cgroup (memcg))); + } -/* Interrupts, kernel threads, and remote chargers stop here. */ -if (use_worker) +/* +* Use a worker for interrupts, kernel threads, and tasks whose mm +* does not match the charged hierarchy. +*/ +if (use_high_work) { +if (!iter) +return ; +if (allow_spinning) +schedule_work (&iter->high_work); +else +irq_work_queue (&iter->high_irq_work);return ; + } + +if (iter) { +/* +* Count the full batch so reclaim work and delay penalties scale +* with the pages charged by this task. +*/ + current->memcg_nr_pages_over_high += nr_pages; +set_notify_resume (current); + }/* -* Reclaim is set up above to be called from the userland -* return path. But also attempt synchronous reclaim to avoid -* excessive overrun while the task is still inside the -* kernel. If this is successful, the return path will see it -* when it rechecks the overage and simply bail out. +* Earlier charges may have left high overage pending even if this charge +* found no new overage. Handle it now if this charge can block, to limit +* overrun while the task remains in the kernel. */ if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH &&gfpflags_allow_blocking (gfp_mask))
Signature · 1 line
2.53.0-Meta