mm: memcg: fix memory.high allocator throttling
Subsystems (4)
cover mm: memcg: fix memory.high allocator throttling
From: Hui Zhu <[email protected]>
calculate_high_delay() converts how far a cgroup is over memory.high or memory.swap.high into a sleep penalty. Two defects corrupt that penalty, and since the caller treats "penalty_jiffies <= HZ / 100" as "nothing to do", a small result silently skips both the throttle and the MAX_RECLAIM_RETRIES reclaim loop.
1/2: the u64 product is stored in an unsigned long before being shifted right 34 bits, so on 32-bit kernels it truncates to zero and no overage ever produces a penalty. Broken since v5.6.
2/2: the u64 product itself overflows once a cgroup is about 130x over its limit, leaving an arbitrary result that can be zero. memory.high has no lower bound, so a one page limit with a few MB of usage gets there. Also affects 64-bit kernels.
Both touch calculate_high_delay(), so apply in order. Different Fixes tags, both for stable.
Hui Zhu (2):
mm: memcg: fix zero throttle delay on 32-bit kernels
mm: memcg: fix memory.high throttle lost to u64 overflow
mm/memcontrol.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/2 mm: memcg: fix zero throttle delay on 32-bit kernels
From: Hui Zhu <[email protected]>
calculate_high_delay() stores max_overage * max_overage * HZ, computed as u64, into an unsigned long before shifting it right by 34 bits in total. On 32-bit kernels the store truncates the value first, so the shift always yields zero. Tasks over memory.high or memory.swap.high are then never throttled, and the reclaim retry loop in __mem_cgroup_handle_over_high() is skipped as well.
Keep the intermediate in u64 so the shifts see the full width, and clamp to MEMCG_MAX_HIGH_DELAY_JIFFIES before narrowing to the unsigned long return value: a truncated result would escape the caller's own min(). Nothing changes on 64-bit kernels, where clamping each term is equivalent to clamping their sum, which the caller already does.
--- mm/memcontrol.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
mm/memcontrol.c +8 -2
@@ -2576,7 +2576,7 @@ static u64 swap_find_max_overage(struct mem_cgroup *memcg)static unsigned long calculate_high_delay (unsigned int nr_pages,u64 max_overage) { -unsigned long penalty_jiffies; +u64 penalty_jiffies;if (!max_overage)return 0 ;
@@ -2601,7 +2601,13 @@ static unsigned long calculate_high_delay(unsigned int nr_pages,* MEMCG_CHARGE_BATCH pages is nominal, so work out how much smaller or * larger the current charge patch is than that. */ -return penalty_jiffies * nr_pages /MEMCG_CHARGE_BATCH ; + penalty_jiffies = penalty_jiffies * nr_pages /MEMCG_CHARGE_BATCH ; + +/* +* Clamp before narrowing to unsigned long: a truncated return value +* would escape the caller's min() and look like a small delay. +*/ +return min_t (u64, penalty_jiffies,MEMCG_MAX_HIGH_DELAY_JIFFIES ); }/*
Signature · 1 line
2.53.0
Re: [PATCH 1/2] mm: memcg: fix zero throttle delay on 32-bit kernels
Quoted 1 line · Hui Zhu
From: Hui Zhu <[email protected]>
Hello, thanks for the patch.
Quoted 7 lines
calculate_high_delay()storesmax_overage*max_overage* HZ, computed as u64, into an unsigned long before shifting it right by 34 bits in total. On 32-bit kernels the store truncates the value first, so the shift always yields zero. Tasks over memory.high or memory.swap.high are then never throttled, and the reclaim retry loop in__mem_cgroup_handle_over_high()is skipped as well.
Quoted 7 lines
Keep the intermediate in u64 so the shifts see the full width, and clamp to
MEMCG_MAX_HIGH_DELAY_JIFFIESbefore narrowing to the unsigned long return value: a truncated result would escape the caller's ownmin(). Nothing changes on 64-bit kernels, where clamping each term is equivalent to clamping their sum, which the caller already does.Fixes e26733e0d0ec ("mm, memcg: throttle allocators based on ancestral memory.high")
Is this the right fixes tag? That one doesn't introduce the division by MEMCG_CHARGE_BATCH.
Quoted 21 lines
Signed-off-by Hui Zhu <[email protected]>--- mm/memcontrol.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)mm/memcontrol.c +1
-1@@ -2576,7 +2576,7 @@ static u64 swap_find_max_overage(struct mem_cgroup *memcg)static unsigned long calculate_high_delay (unsigned int nr_pages,u64 max_overage) { -unsigned long penalty_jiffies; +u64 penalty_jiffies;if (!max_overage)return 0 ;
mm/memcontrol.c static unsigned long calculate_high_delay(unsigned int nr_pages, · line 2609
Quoted 10 lines
mm/memcontrol.c +6
-1@@ -2601,7 +2601,13 @@ static unsigned long calculate_high_delay(unsigned int nr_pages,* MEMCG_CHARGE_BATCH pages is nominal, so work out how much smaller or * larger the current charge patch is than that. */ -return penalty_jiffies * nr_pages /MEMCG_CHARGE_BATCH ; + penalty_jiffies = penalty_jiffies * nr_pages /MEMCG_CHARGE_BATCH ; + +/* +* Clamp before narrowing to unsigned long: a truncated return value +* would escape the caller's min() and look like a small delay. +*/
I think this is a bit verbose for what is happening here. Why not:
return min_t(u64, penalty_jiffies * nr_pages / MEMCG_CHARGE_BATCH,
MEMCG_MAX_HIGH_DELAY_JIFFIES);
Or just dropping the comment block seems fine. I think folks should be familiar enough with values wrapping around.
Thanks, have a great day!
Joshua
Quoted 8 lines
mm/memcontrol.c +1
-0+return min_t (u64, penalty_jiffies,MEMCG_MAX_HIGH_DELAY_JIFFIES ); }/* Signature · 1 line
2.53.0
PATCH 2/2 mm: memcg: fix memory.high throttle lost to u64 overflow
In reply to Hui ZhuFrom: Hui Zhu <[email protected]>
calculate_high_delay() squares max_overage and scales it by HZ in u64. max_overage is the ratio ((usage - high) << 20) / high, and memory.high has no lower bound, so the product wraps once a cgroup is about 130x over its limit (at HZ=1000). The wrapped value is arbitrary and can be exactly zero:
with memory.high set to one page and usage at 2049 pages, one page past the 8M mark, max_overage is 2^31, the product wraps to zero and the penalty comes out 0 jiffies instead of the 2 seconds the formula calls for. The caller then takes its "penalty_jiffies <= HZ / 100" exit, skipping both the throttle and the reclaim retry loop.
Clamp the overage to MEMCG_MAX_DELAY_OVERAGE before squaring it. The bound keeps the product within u64 for any supported HZ, and by then the delay has long saturated at MEMCG_MAX_HIGH_DELAY_JIFFIES, so no penalty is shortened.
--- mm/memcontrol.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
mm/memcontrol.c +11 -0
@@ -2521,6 +2521,15 @@ static void high_irq_work_func(struct irq_work *work)#define MEMCG_DELAY_PRECISION_SHIFT 20#define MEMCG_DELAY_SCALING_SHIFT 14 +/* +* Cap for the overage passed to calculate_high_delay(). Squaring a larger +* overage and scaling it by HZ overflows u64, and the wrapped product may +* come out zero, skipping the throttle entirely. At the cap the square +* still fits in u64 for any HZ, and the delay has long saturated at +* MEMCG_MAX_HIGH_DELAY_JIFFIES, so clamping never shortens it. +*/ +#define MEMCG_MAX_DELAY_OVERAGE (1ULL << 26) +static u64 calculate_overage (unsigned long usage,unsigned long high) {u64 overage;
@@ -2581,6 +2590,8 @@ static unsigned long calculate_high_delay(unsigned int nr_pages,if (!max_overage)return 0 ; + max_overage =min_t (u64, max_overage,MEMCG_MAX_DELAY_OVERAGE ); +/* * We use overage compared to memory.high to calculate the number of * jiffies to sleep (penalty_jiffies). Ideally this value should be
Signature · 1 line
2.53.0
Hi Hui,
Thanks for fixing these, I've verified that these fix the issues reported.
That said, Zhang Jiao sent a fix for the same two problems a few weeks ago, and Johannes thought the overage cap there was a bit of a magic value, which we have plenty of in mm already, and I agree we shouldn't add more [1].
Could you please coordinate so we don't have two patchsets doing the same thing?
Chris
2026/11/18 06:17, Chris Down:
Quoted 10 lines
Hi Hui,
Thanks for fixing these, I've verified that these fix the issues reported.
That said, Zhang Jiao sent a fix for the same two problems a few weeks ago, and Johannes thought the overage cap there was a bit of a magic value, which we have plenty of in mm already, and I agree we shouldn't add more [1].
Quoted 6 lines
Could you please coordinate so we don't have two patchsets doing the same thing?
Chris
Hi Chris, Joshua, Zhang Jiao,
Chris: thanks for the pointer, and apologies for the duplicate. I missed Zhang Jiao's patch [1], and Song Hu's earlier report of the overflow, which already proposed check_mul_overflow() [2]. I've added Zhang Jiao to the thread so we can settle the coordination here rather than off-list.
You and Johannes are right about the cap. I've dropped MEMCG_MAX_DELAY_OVERAGE and reworked 2/2 into the shape Johannes suggested, with calculate_high_delay() and the caller's penalty_jiffies both in u64 so the value is narrowed only after the caller's clamp. That also removes the in-function min() from 1/2, which was undoing ff144e69f733 ("mm/memcg: move penalty delay clamping out of calculate_high_delay()"). The series shrinks from 19 insertions to 9.
Joshua: the Fixes tag is the right one, though I can see why the diff suggested otherwise. The division comes from 0e4b01df8659; my patch only touched that line to add the clamp. The bug is the reordering in e26733e0d0ec, which replaced
penalty_jiffies = ((u64)overage * overage * HZ)
>> (MEMCG_DELAY_PRECISION_SHIFT + MEMCG_DELAY_SCALING_SHIFT);
with
penalty_jiffies = max_overage * max_overage * HZ; penalty_jiffies >>= MEMCG_DELAY_PRECISION_SHIFT; penalty_jiffies >>= MEMCG_DELAY_SCALING_SHIFT;
Before that, the shift ran on the u64 temporary and the narrowing into unsigned long happened afterwards. After it, the value is narrowed first, so shifting it right by 34 always yields zero. The division plays no part, and in v2 the diff no longer goes near it. Your other point is settled by the same rework: with the clamp gone, the comment goes too.
How I'd like to proceed:
- Zhang Jiao: would it be OK if I send the combined series? The overflow fix stays under your name and keeps your changelog, in the shape Johannes suggested, and I'd carry the 32-bit truncation alongside it in the same series. If you would rather send it yourself, or split it differently, say so and I'll follow.
- Song Hu and I have been in touch off-list, and they are working in this area as well, so the patches under my name will carry them as Co-developed-by.
- v2 will include what was missing before: a Fixes tag for each of the two regressions (e26733e0d0ec, v5.6, for the 32-bit truncation; 0e4b01df8659, v5.4, for the overflow), Cc: stable, a changelog that says plainly that throttling was completely inert on 32-bit, and the wrap threshold corrected to about 130x at HZ=1000 -- the 4096x figure misses the HZ factor.
Two separate patches rather than one, because they are different regressions with different lifetimes, which matters for the stable backports.
One number that may be worth putting in the changelog: the smallest exact-zero case is memory.high at one page with usage at 2049 pages, where max_overage is exactly 2^31 and the product wraps to zero.
Thanks,
Hui
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/