Skip to contentSkip to conversation

mm, page_alloc: reintroduce page allocation stall warning

RFC In review David Rientjes 6 participants 20 messages Mar 22 – Mar 31 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 mm, page_alloc: reintroduce page allocation stall warning

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it starts with a 10 second floor to begin with. If it does trigger, this reveals an issue that should be
fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Unlike the original implementation, this implementation only reports stalls that are at least a second longer than the longest stall reported thus far.

---
 mm/page_alloc.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
mm/page_alloc.c +32 -0
@@ -4706,6 +4706,36 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static unsigned long max_alloc_stall_warn_msecs = 10 * 1000L;
+
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				   unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(max_alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+	unsigned long flags;
+
+	if (likely(stall_msecs <= READ_ONCE(max_alloc_stall_warn_msecs)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	spin_lock_irqsave(&max_alloc_stall_lock, flags);
+	if (stall_msecs > max_alloc_stall_warn_msecs) {
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+
+		/* Only print future stalls that are more than a second longer */
+		WRITE_ONCE(max_alloc_stall_warn_msecs, stall_msecs + MSEC_PER_SEC);
+	}
+	spin_unlock_irqrestore(&max_alloc_stall_lock, flags);
+}
+
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
@@ -4726,6 +4756,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	int reserve_flags;
 	bool compact_first = false;
 	bool can_retry_reserves = true;
+	unsigned long alloc_start_time = jiffies;
 
 	if (unlikely(nofail)) {
 		/*
@@ -4990,6 +5021,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	warn_alloc(gfp_mask, ac->nodemask,
 			"page allocation failure: order:%u", order);
 got_pg:
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
 	return page;
 }
 
#1 lore raw
Quoted 50 lines · David Rientjes
mm/page_alloc.c +31 -0
@@ -4706,6 +4706,36 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static unsigned long max_alloc_stall_warn_msecs = 10 * 1000L;
+
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				   unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(max_alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+	unsigned long flags;
+
+	if (likely(stall_msecs <= READ_ONCE(max_alloc_stall_warn_msecs)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	spin_lock_irqsave(&max_alloc_stall_lock, flags);
+	if (stall_msecs > max_alloc_stall_warn_msecs) {
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+
+		/* Only print future stalls that are more than a second longer */
+		WRITE_ONCE(max_alloc_stall_warn_msecs, stall_msecs + MSEC_PER_SEC);
+	}
+	spin_unlock_irqrestore(&max_alloc_stall_lock, flags);
+}
+
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
@@ -4726,6 +4756,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	int reserve_flags;
 	bool compact_first = false;
 	bool can_retry_reserves = true;
+	unsigned long alloc_start_time = jiffies;
 
 	if (unlikely(nofail)) {
 		/*

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 5027

Quoted 7 lines
mm/page_alloc.c +1 -0
@@ -4990,6 +5021,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	warn_alloc(gfp_mask, ac->nodemask,
 			"page allocation failure: order:%u", order);
 got_pg:
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
 	return page;
 }
 

Another option here if we're concerned about calling check_alloc_stall_warn() on every slowpath allocation is to check this right after calling into should_reclaim_retry(). We'd normally be looping in the page allocator if a single call is taking >10s. That could output multiple stall warnings for a single page allocation, though, so in this case we'd probably want to (1) increase the amount of time between one warning and another beyond one second and (2) cap the output when some time duration is reached like 60 seconds.

#2 lore raw
In reply to David Rientjes
Quoted 19 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it starts with a 10 second floor to begin with. If it does trigger, this reveals an issue that should be
fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 5 lines

Unlike the original implementation, this implementation only reports stalls that are at least a second longer than the longest stall reported thus far.

I think, why not, if it's useful and we can reintroduce it without the issues it had.
Maybe instead of requiring the stall time to increase by a second, we could just limit the stall reports to once per 10 second. If there are multiple ones in progress, one of them will win that report slot randomly. This would also cover a stall that's so long it reports itself multiple times (as in the original commit).

Quoted 12 lines
---
 mm/page_alloc.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
mm/page_alloc.c +1 -0
@@ -4706,6 +4706,36 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static unsigned long max_alloc_stall_warn_msecs = 10 * 1000L;

mm/page_alloc.c check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac) · line 4719

Quoted 10 lines
mm/page_alloc.c +10 -0
+
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				   unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(max_alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+	unsigned long flags;
+
+	if (likely(stall_msecs <= READ_ONCE(max_alloc_stall_warn_msecs)))
+		return;

This check without lock is while I'm not worried about calling this liberally (as you discuss in your self-reply).

mm/page_alloc.c check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac) · line 4723

Quoted 4 lines
mm/page_alloc.c +4 -0
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	spin_lock_irqsave(&max_alloc_stall_lock, flags);

This could make parallel stallers spin for no good reason while one that has the lock is printing. I think we could use trylock here, and if it fails, do nothing. Then it also shouldn't be necessary to disable irqs.

Quoted 27 lines
mm/page_alloc.c +16 -0
+	if (stall_msecs > max_alloc_stall_warn_msecs) {
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+
+		/* Only print future stalls that are more than a second longer */
+		WRITE_ONCE(max_alloc_stall_warn_msecs, stall_msecs + MSEC_PER_SEC);
+	}
+	spin_unlock_irqrestore(&max_alloc_stall_lock, flags);
+}
+
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
@@ -4726,6 +4756,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	int reserve_flags;
 	bool compact_first = false;
 	bool can_retry_reserves = true;
+	unsigned long alloc_start_time = jiffies;
 
 	if (unlikely(nofail)) {
 		/*

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 5024

Quoted 4 lines
mm/page_alloc.c +1 -0
@@ -4990,6 +5021,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	warn_alloc(gfp_mask, ac->nodemask,
 			"page allocation failure: order:%u", order);
 got_pg:
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);

But placed here it defeats the purpose to some extent, no? We'll only learn about a stall that ended. With the shared 10s rate-limit, we should be able to do it in the repeat loop for stalls in progress, as the original commit did.

Quoted 3 lines
mm/page_alloc.c +0 -0
 	return page;
 }
 
#3 lore raw
Quoted 26 lines · Vlastimil Babka (SUSE)
Quoted 19 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it starts with a 10 second floor to begin with. If it does trigger, this reveals an issue that should be
fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 5 lines

Unlike the original implementation, this implementation only reports stalls that are at least a second longer than the longest stall reported thus far.

Quoted 8 lines

I think, why not, if it's useful and we can reintroduce it without the issues it had.
Maybe instead of requiring the stall time to increase by a second, we could just limit the stall reports to once per 10 second. If there are multiple ones in progress, one of them will win that report slot randomly. This would also cover a stall that's so long it reports itself multiple times (as in the original commit).

I like that a lot, thanks. Since part of the motivation is to correlate userspace unresponsiveness with page allocation stalls in the kernel, we increasingly lack that visiblity if a single long page allocation took 60 seconds a month ago, for example, and we have to reach that threshold to report again.

The original patch ended up at line 4839 here:

4833) }
4834) }
4835)
4836) /* Caller is not willing to reclaim, we can't balance anything */
4837) if (!can_direct_reclaim)
4838) goto nopage;
4839) <===== HERE
4840) /* Avoid recursion of direct reclaim */
4841) if (current->flags & PF_MEMALLOC)
4842) goto nopage;
4843)
4844) /* Try direct reclaim and then allocating */

Which looks like the right place to put it, but probably after the PF_MEMALLOC check.

If we set a minimum reporting threshold of 10 seconds and only report system wide every 10 seconds, I think this will work very well. And, as you mention, this also reports stalls for allocations that never actually return.

I'll implement this and send out a formal patch for it.

reviewer
#4 lore raw
In reply to David Rientjes
Quoted 19 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it starts with a 10 second floor to begin with. If it does trigger, this reveals an issue that should be
fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 3 lines

Unlike the original implementation, this implementation only reports stalls that are at least a second longer than the longest stall reported thus far.

Am all for reintroducing the warning in some shape. The biggest problem back then was printk being too eager to stomp all the work at a single executing context. Not sure this is still the case. Let's add printk maintainers.
Also it makes some sense to differentiate stalled callers and show_mem which is more verbose. The former tells us who is affected and the second will give us more context and we want to get some information about all of them. The latter can be printed much less often as it will describe situation for a batch of concurrent ones.

Quoted 61 lines
---
 mm/page_alloc.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
mm/page_alloc.c +32 -0
@@ -4706,6 +4706,36 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static unsigned long max_alloc_stall_warn_msecs = 10 * 1000L;
+
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				   unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(max_alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+	unsigned long flags;
+
+	if (likely(stall_msecs <= READ_ONCE(max_alloc_stall_warn_msecs)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	spin_lock_irqsave(&max_alloc_stall_lock, flags);
+	if (stall_msecs > max_alloc_stall_warn_msecs) {
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+
+		/* Only print future stalls that are more than a second longer */
+		WRITE_ONCE(max_alloc_stall_warn_msecs, stall_msecs + MSEC_PER_SEC);
+	}
+	spin_unlock_irqrestore(&max_alloc_stall_lock, flags);
+}
+
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
@@ -4726,6 +4756,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	int reserve_flags;
 	bool compact_first = false;
 	bool can_retry_reserves = true;
+	unsigned long alloc_start_time = jiffies;
 
 	if (unlikely(nofail)) {
 		/*
@@ -4990,6 +5021,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	warn_alloc(gfp_mask, ac->nodemask,
 			"page allocation failure: order:%u", order);
 got_pg:
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
 	return page;
 }
 
Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
Quoted 20 lines · Michal Hocko
Quoted 19 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it starts with a 10 second floor to begin with. If it does trigger, this reveals an issue that should be
fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 8 lines
Quoted 3 lines

Unlike the original implementation, this implementation only reports stalls that are at least a second longer than the longest stall reported thus far.

Am all for reintroducing the warning in some shape. The biggest problem back then was printk being too eager to stomp all the work at a single executing context. Not sure this is still the case. Let's add printk maintainers.

Thanks.

Quoted 6 lines

Also it makes some sense to differentiate stalled callers and show_mem which is more verbose. The former tells us who is affected and the second will give us more context and we want to get some information about all of them. The latter can be printed much less often as it will describe situation for a batch of concurrent ones.

Based on Vlastimil's suggestion I think this is trending in the direction of 10-second reporting windows system wide unless that doesn't work for some reason. I do worry about reporting many stalls even without show_mem(), however. In situations where the allocations are unconstained, all userspace goes out to lunch for 10 seconds and that can result in thousands of threads all reporting stalls and spamming the kernel log.

Idea is a 10 second threshold for reporting stalls and then only one stall report across a 10 second sliding window globally.

#6 lore raw
Quoted 22 lines · David Rientjes
Quoted 20 lines · Michal Hocko
Quoted 19 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it starts with a 10 second floor to begin with. If it does trigger, this reveals an issue that should be
fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Unlike the original implementation, this implementation only reports stalls that are at least a second longer than the longest stall reported thus far.

Am all for reintroducing the warning in some shape. The biggest problem back then was printk being too eager to stomp all the work at a single executing context. Not sure this is still the case. Let's add printk maintainers.

printk() is still a constraint. There is an API which allows to offload printk() into a kthread but only few console drivers have been converted so far. Most console drivers, including the most common uart 8250, are still using the legacy loop in printk()/console_unlock().

In addition, the new API introduced an emergency context which forces synchronous flush of printk messages even for the converted console drivers. The emergency context is currently used, for example, by WARN() or RCU stall report. So, there always will be a risk that too many pending messages might cause a stall.

Quoted 10 lines
Quoted 6 lines

Also it makes some sense to differentiate stalled callers and show_mem which is more verbose. The former tells us who is affected and the second will give us more context and we want to get some information about all of them. The latter can be printed much less often as it will describe situation for a batch of concurrent ones.

Based on Vlastimil's suggestion I think this is trending in the direction of 10-second reporting windows system wide unless that doesn't work for some reason.

This is a wise idea.

Quoted 5 lines

I do worry about reporting many stalls even without show_mem(), however. In situations where the allocations are unconstained, all userspace goes out to lunch for 10 seconds and that can result in thousands of threads all reporting stalls and spamming the kernel log.

Yeah, this looks scary even when all console drivers handled messages in kthreads. I believe that printing details about all the stalled tasks is not worth it. It might be enough to add an atomic counter and print the number of stalled tasks within last 10 seconds.

Quoted 2 lines

Idea is a 10 second threshold for reporting stalls and then only one stall report across a 10 second sliding window globally.

I wonder if this even could be added to the existing watchdog, aka report the number of stalled allocations in watchdog_timer_fn().

Best Regards,
Petr

maintainer
#7 lore raw
In reply to David Rientjes
Quoted 19 lines · On Sat, 21 Mar 2026 20:03:16 -0700 (PDT) David Rientjes <[email protected]>…

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it starts with a 10 second floor to begin with. If it does trigger, this reveals an issue that should be
fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 3 lines

Unlike the original implementation, this implementation only reports stalls that are at least a second longer than the longest stall reported thus far.

AI review: https://sashiko.dev/#/patchset/[email protected]

The warn_alloc_show_mem() inside spin_lock_irqsave() does sound problematic.

#8 lore raw

PATCH 1/1 mm, page_alloc: reintroduce page allocation stall warning

In reply to David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

---
 mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
mm/page_alloc.c +46 -0
@@ -316,6 +316,14 @@ EXPORT_SYMBOL(nr_node_ids);
 EXPORT_SYMBOL(nr_online_nodes);
 #endif
 
+/*
+ * When page allocations stall for longer than a threshold,
+ * ALLOC_STALL_WARN_MSECS, leave a warning in the kernel log.  Only one warning
+ * will be printed during this duration for the entire system.
+ */
+#define ALLOC_STALL_WARN_MSECS (10 * 1000UL)
+static unsigned long alloc_stall_warn_jiffies;
+
 static bool page_contains_unaccepted(struct page *page, unsigned int order);
 static bool cond_accept_memory(struct zone *zone, unsigned int order,
 			       int alloc_flags);
@@ -4706,6 +4714,40 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+
+	if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))
+		return;
+	if (time_before(jiffies, READ_ONCE(alloc_stall_warn_jiffies)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	if (!spin_trylock(&alloc_stall_lock))
+		return;
+
+	if (time_after_eq(jiffies, alloc_stall_warn_jiffies)) {
+		WRITE_ONCE(alloc_stall_warn_jiffies,
+			   jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
+		spin_unlock(&alloc_stall_lock);
+
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+		return;
+	}
+
+	spin_unlock(&alloc_stall_lock);
+}
+
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
@@ -4726,6 +4768,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	int reserve_flags;
 	bool compact_first = false;
 	bool can_retry_reserves = true;
+	unsigned long alloc_start_time = jiffies;
 
 	if (unlikely(nofail)) {
 		/*
@@ -4841,6 +4884,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (current->flags & PF_MEMALLOC)
 		goto nopage;
 
+	/* If allocation has taken excessively long, warn about it */
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
maintainer
#9 lore raw
Quoted 41 lines · On Sun, 29 Mar 2026 18:08:52 -0700 (PDT) David Rientjes <[email protected]>…

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

...

mm/page_alloc.c +14 -0
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+
+	if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))
+		return;
+	if (time_before(jiffies, READ_ONCE(alloc_stall_warn_jiffies)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	if (!spin_trylock(&alloc_stall_lock))

mm/page_alloc.c check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac) · line 4740

Quoted 10 lines
mm/page_alloc.c +10 -0
+		return;
+
+	if (time_after_eq(jiffies, alloc_stall_warn_jiffies)) {
+		WRITE_ONCE(alloc_stall_warn_jiffies,
+			   jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
+		spin_unlock(&alloc_stall_lock);
+
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));

Snould we use dump_page() in here? It prints more info, does the snapshotting thing.

Quoted 10 lines
mm/page_alloc.c +10 -0
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+		return;
+	}
+
+	spin_unlock(&alloc_stall_lock);
+}
+
#10 lore raw
Quoted 48 lines · Andrew Morton
Quoted 46 lines · On Sun, 29 Mar 2026 18:08:52 -0700 (PDT) David Rientjes <[email protected]>…

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

...

mm/page_alloc.c +19 -0
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+
+	if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))
+		return;
+	if (time_before(jiffies, READ_ONCE(alloc_stall_warn_jiffies)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	if (!spin_trylock(&alloc_stall_lock))
+		return;
+
+	if (time_after_eq(jiffies, alloc_stall_warn_jiffies)) {
+		WRITE_ONCE(alloc_stall_warn_jiffies,
+			   jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
Quoted 8 lines

mm/page_alloc.c check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac) · line 4740

Quoted 5 lines
mm/page_alloc.c +5 -0
+		spin_unlock(&alloc_stall_lock);
+
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));

Snould we use dump_page() in here? It prints more info, does the snapshotting thing.

But we have no page to dump, or did you mean something else? Maybe some part of warn_alloc() (without its own ratelimit etc) could be extracted end reused.

Quoted 11 lines
Quoted 10 lines
mm/page_alloc.c +10 -0
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+		return;
+	}
+
+	spin_unlock(&alloc_stall_lock);
+}
+
reviewer
#11 lore raw
In reply to David Rientjes
Quoted 7 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

I think it makes sense to summarize reasons for the revert. I would propose to change the above to somehting like
"
The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectivelly stall further progress through printk execution.
"

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4892

Quoted 10 lines
mm/page_alloc.c +3 -0
@@ -4841,6 +4884,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (current->flags & PF_MEMALLOC)
 		goto nopage;
 
+	/* If allocation has taken excessively long, warn about it */
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,

Is there any specific reason for this placement? Compaction can take quite some time as well.

Signature · 2 lines
Michal Hocko
SUSE Labs
#12 lore raw
Quoted 22 lines · Michal Hocko
Quoted 7 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

I think it makes sense to summarize reasons for the revert. I would propose to change the above to somehting like
"
The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectivelly stall further progress through printk execution.
"

Quoted 4 lines
mm/page_alloc.c +0 -0
@@ -4841,6 +4884,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (current->flags & PF_MEMALLOC)
 		goto nopage;
 
Quoted 9 lines

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4892

Quoted 6 lines
mm/page_alloc.c +3 -0
+	/* If allocation has taken excessively long, warn about it */
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,

Is there any specific reason for this placement? Compaction can take quite some time as well.

It seems fine to me - as longs as the slowpath is retrying for 10 seconds and still can't obtain a page, there's a warning.

We don't catch cases when either the get_page_from_freelist() attempt, direct compaction or direct reclaim attempt is what gets us over 10 seconds, and at the same time it results in success. If that's a concern, we should add another check_alloc_stall_warn() call under got_pg label (as the RFC had) - I'm not sure it's all achievable with a single place with the call.

#13 lore raw
Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 8 lines
Quoted 7 lines

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Quoted 9 lines
Quoted 9 lines

I think it makes sense to summarize reasons for the revert. I would propose to change the above to somehting like
"
The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectivelly stall further progress through printk execution.
"

Will do!

Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
mm/page_alloc.c +0 -0
@@ -4841,6 +4884,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (current->flags & PF_MEMALLOC)
 		goto nopage;
 
Quoted 9 lines

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4892

Quoted 6 lines
mm/page_alloc.c +3 -0
+	/* If allocation has taken excessively long, warn about it */
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,

Is there any specific reason for this placement? Compaction can take quite some time as well.

Quoted 9 lines

It seems fine to me - as longs as the slowpath is retrying for 10 seconds and still can't obtain a page, there's a warning.

We don't catch cases when either the get_page_from_freelist() attempt, direct compaction or direct reclaim attempt is what gets us over 10 seconds, and at the same time it results in success. If that's a concern, we should add another check_alloc_stall_warn() call under got_pg label (as the RFC had) - I'm not sure it's all achievable with a single place with the call.

Right, the big idea here is that at least one of the allocations that is looping will run into the check_alloc_stall_warn(). We might miss a borderline case but the 10 seconds is already arbitrary and this is the placement where commit 63f53dea0c98 ("mm: warn about allocations which stall for too long") ended up before it was reverted.

maintainer Acked-by on v1 1/1
#14 lore raw
In reply to David Rientjes
Quoted 20 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") but for reasons unrelated to the warning itself.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 6 lines

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

Nit below:

Quoted 33 lines
---
 mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
mm/page_alloc.c +15 -0
@@ -316,6 +316,14 @@ EXPORT_SYMBOL(nr_node_ids);
 EXPORT_SYMBOL(nr_online_nodes);
 #endif
 
+/*
+ * When page allocations stall for longer than a threshold,
+ * ALLOC_STALL_WARN_MSECS, leave a warning in the kernel log.  Only one warning
+ * will be printed during this duration for the entire system.
+ */
+#define ALLOC_STALL_WARN_MSECS (10 * 1000UL)
+static unsigned long alloc_stall_warn_jiffies;
+
 static bool page_contains_unaccepted(struct page *page, unsigned int order);
 static bool cond_accept_memory(struct zone *zone, unsigned int order,
 			       int alloc_flags);
@@ -4706,6 +4714,40 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+
+	if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))

mm/page_alloc.c check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac) · line 4733

Quoted 10 lines
mm/page_alloc.c +10 -0
+		return;
+	if (time_before(jiffies, READ_ONCE(alloc_stall_warn_jiffies)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	if (!spin_trylock(&alloc_stall_lock))
+		return;
+
+	if (time_after_eq(jiffies, alloc_stall_warn_jiffies)) {

This could also be an unlock+return if the opposite (time_before()) is true, reducing the indentation for the actual warning code.

Quoted 17 lines
mm/page_alloc.c +17 -0
+		WRITE_ONCE(alloc_stall_warn_jiffies,
+			   jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
+		spin_unlock(&alloc_stall_lock);
+
+		pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+			current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+			nodemask_pr_args(nodemask));
+		cpuset_print_current_mems_allowed();
+		pr_cont("\n");
+		dump_stack();
+		warn_alloc_show_mem(gfp_mask, nodemask);
+		return;
+	}
+
+	spin_unlock(&alloc_stall_lock);
+}
+
#15 lore raw

PATCH v2 1/1 mm, page_alloc: reintroduce page allocation stall warning

In reply to David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectively stall further progress through printk execution.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

---
v2:
- commit message update per Michal
- check_alloc_stall_warn() cleanup per Vlastimil

 mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
mm/page_alloc.c +46 -0
@@ -316,6 +316,14 @@ EXPORT_SYMBOL(nr_node_ids);
 EXPORT_SYMBOL(nr_online_nodes);
 #endif
 
+/*
+ * When page allocations stall for longer than a threshold,
+ * ALLOC_STALL_WARN_MSECS, leave a warning in the kernel log.  Only one warning
+ * will be printed during this duration for the entire system.
+ */
+#define ALLOC_STALL_WARN_MSECS (10 * 1000UL)
+static unsigned long alloc_stall_warn_jiffies;
+
 static bool page_contains_unaccepted(struct page *page, unsigned int order);
 static bool cond_accept_memory(struct zone *zone, unsigned int order,
 			       int alloc_flags);
@@ -4706,6 +4714,40 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+
+	if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))
+		return;
+	if (time_before(jiffies, READ_ONCE(alloc_stall_warn_jiffies)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	if (!spin_trylock(&alloc_stall_lock))
+		return;
+
+	/* Check again, this time under the lock */
+	if (time_before(jiffies, alloc_stall_warn_jiffies)) {
+		spin_unlock(&alloc_stall_lock);
+		return;
+	}
+
+	WRITE_ONCE(alloc_stall_warn_jiffies, jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
+	spin_unlock(&alloc_stall_lock);
+
+	pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+		current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+		nodemask_pr_args(nodemask));
+	cpuset_print_current_mems_allowed();
+	pr_cont("\n");
+	dump_stack();
+	warn_alloc_show_mem(gfp_mask, nodemask);
+}
+
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
@@ -4726,6 +4768,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	int reserve_flags;
 	bool compact_first = false;
 	bool can_retry_reserves = true;
+	unsigned long alloc_start_time = jiffies;
 
 	if (unlikely(nofail)) {
 		/*
@@ -4841,6 +4884,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (current->flags & PF_MEMALLOC)
 		goto nopage;
 
+	/* If allocation has taken excessively long, warn about it */
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
#16 lore raw

PATCH v3 1/1 mm, page_alloc: reintroduce page allocation stall warning

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectively stall further progress through printk execution.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

---
v3:
- initialize to INITIAL_JIFFIES per AI review so warnings are not suppressed in the first five minutes after boot
- time_before(jiffies, a) -> time_is_after_jiffies(a)
v2:
- commit message update per Michal
- check_alloc_stall_warn() cleanup per Vlastimil

 mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
mm/page_alloc.c +46 -0
@@ -316,6 +316,14 @@ EXPORT_SYMBOL(nr_node_ids);
 EXPORT_SYMBOL(nr_online_nodes);
 #endif
 
+/*
+ * When page allocations stall for longer than a threshold,
+ * ALLOC_STALL_WARN_MSECS, leave a warning in the kernel log.  Only one warning
+ * will be printed during this duration for the entire system.
+ */
+#define ALLOC_STALL_WARN_MSECS (10 * 1000UL)
+static unsigned long alloc_stall_warn_jiffies = INITIAL_JIFFIES;
+
 static bool page_contains_unaccepted(struct page *page, unsigned int order);
 static bool cond_accept_memory(struct zone *zone, unsigned int order,
 			       int alloc_flags);
@@ -4706,6 +4714,40 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
 	return false;
 }
 
+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+				unsigned int order, unsigned long alloc_start_time)
+{
+	static DEFINE_SPINLOCK(alloc_stall_lock);
+	unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
+
+	if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))
+		return;
+	if (time_is_after_jiffies(READ_ONCE(alloc_stall_warn_jiffies)))
+		return;
+	if (gfp_mask & __GFP_NOWARN)
+		return;
+
+	if (!spin_trylock(&alloc_stall_lock))
+		return;
+
+	/* Check again, this time under the lock */
+	if (time_is_after_jiffies(alloc_stall_warn_jiffies)) {
+		spin_unlock(&alloc_stall_lock);
+		return;
+	}
+
+	WRITE_ONCE(alloc_stall_warn_jiffies, jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
+	spin_unlock(&alloc_stall_lock);
+
+	pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
+		current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
+		nodemask_pr_args(nodemask));
+	cpuset_print_current_mems_allowed();
+	pr_cont("\n");
+	dump_stack();
+	warn_alloc_show_mem(gfp_mask, nodemask);
+}
+
 static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
@@ -4726,6 +4768,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	int reserve_flags;
 	bool compact_first = false;
 	bool can_retry_reserves = true;
+	unsigned long alloc_start_time = jiffies;
 
 	if (unlikely(nofail)) {
 		/*
@@ -4841,6 +4884,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (current->flags & PF_MEMALLOC)
 		goto nopage;
 
+	/* If allocation has taken excessively long, warn about it */
+	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
Reviewed-by on v3 1/1
#17 lore raw
Quoted 21 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectively stall further progress through printk execution.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 7 lines

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

I am hoping that the reason you are reintroducing these warnings is because you already are seeing such cases in your production environment. Do you have anything interesting to share?

reviewer Acked-by on v3 1/1
#18 lore raw
In reply to David Rientjes
Quoted 21 lines · David Rientjes

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectively stall further progress through printk execution.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Quoted 7 lines

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

OK, let's go with this and tune it up as we find a need in the future.

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs

A message that is not in the archive (sent off-list or not yet received): [email protected]

#20 lore raw
Quoted 26 lines · Shakeel Butt
Quoted 26 lines

Previously, we had warnings when a single page allocation took longer than reasonably expected. This was introduced in commit 63f53dea0c98 ("mm: warn about allocations which stall for too long").

The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't warn about allocations which stall for too long") because it was possible to generate memory pressure that would effectively stall further progress through printk execution.

Page allocation stalls in excess of 10 seconds are always useful to debug because they can result in severe userspace unresponsiveness. Adding this artifact can be used to correlate with userspace going out to lunch and to understand the state of memory at the time.

There should be a reasonable expectation that this warning will never trigger given it is very passive, it will only be emitted when a page allocation takes longer than 10 seconds. If it does trigger, this reveals an issue that should be fixed: a single page allocation should never loop for more than 10 seconds without oom killing to make memory available.

Unlike the original implementation, this implementation only reports stalls once for the system every 10 seconds. Otherwise, many concurrent reclaimers could spam the kernel log unnecessarily. Stalls are only reported when calling into direct reclaim.

Quoted 9 lines
Quoted 2 lines

I am hoping that the reason you are reintroducing these warnings is because you already are seeing such cases in your production environment. Do you have anything interesting to share?

We don't have this patch in our production environment (yet). We've been stress testing allocations for page faults with lots of concurrent skb allocations that can keep us persistently below the per-zone min watermarks and hope that this patch will shed some light on some unresponsiveness issues that we've encountered if/when it happens.