INFO: rcu detected stall in khugepaged (3)

Hello,

syzbot found the following issue on:

HEAD commit:    3708dd948844 Merge tag 'pm-7.2-rc6' of git://git.kernel.or..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11ac703e580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=4e38b15c29e6a1d9
dashboard link: https://syzkaller.appspot.com/bug?extid=d2401aeb74cc84adba04
compiler:       Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Quoted 4 lines · On Tue, 04 Aug 2026 17:01:48 -0700 syzbot…

Hello,

syzbot found the following issue on:

Quoted 8 lines

HEAD commit:    3708dd948844 Merge tag 'pm-7.2-rc6' of git://git.kernel.or..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11ac703e580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=4e38b15c29e6a1d9
dashboard link: https://syzkaller.appspot.com/bug?extid=d2401aeb74cc84adba04
compiler:       Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Thanks.

Lazy optimists (ahem) paste this gunk into Gemini and ask "what the heck just happened". The results are often useful, but should be treated with skepticism. In this case I think it came usably close.

https://share.gemini.google/vq4TLhTiLBih

tl;dr: khugepaged's collapse_scan_file() is taking too long and RCU got starved. I don't think khugepaged is doing anything wrong here, per-se. There's a lot of work to do and we're doing it.

An appropriate fix would be to take a break, let RCU do its thing then get back to work. But I don't think RCU offers interfaces for that?

collapse_scan_file()'s main loop has

		if (need_resched()) {
			xas_pause(&xas);
			cond_resched_rcu();
		}

but that won't help with the RCU stall detector(?).

I suggest that a suitable fix here would be to add the analogous

	if (rcu_i_need_to_take_a_break()) {
		rcu_read_unlock();
		rcu_take_a_break())	
		rcu_read_lock();
	}

(iirc rcu_read_unlock() does an rcu run, so rcu_take_a_break() isn't needed here)

Paul, wdyt?

Quoted 17 lines

collapse_scan_file()'s main loop has

		if (need_resched()) {
			xas_pause(&xas);
			cond_resched_rcu();
		}

but that won't help with the RCU stall detector(?).

I suggest that a suitable fix here would be to add the analogous

	if (rcu_i_need_to_take_a_break()) {
		rcu_read_unlock();
		rcu_take_a_break())	
		rcu_read_lock();
	}

Quoted 4 lines

(iirc rcu_read_unlock() does an rcu run, so rcu_take_a_break() isn't needed here)

Paul, wdyt?

Let's see...

The console log says "rcu_preempt detected stalls on CPUs/tasks", which means that cond_resched() is a no-op, but it also means that the rcu_read_unlock() in cond_resched_rcu() will directly take care of informing RCU of the pause.

But that is clearly not happening. Why?

Well, we have this:

rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P37/1:b..l

This means that the task whose RCU read-side critical section is blocking the current RCU grace period isn't even running, and thus cannot invoke cond_resched_rcu(), let alone the rcu_read_unlock() within that function. So an RCU CPU stall warning is expected behavior. Or at least it is not in any way ruled out.

What we need is RCU priority boosting.

Except that the .config file
does not enable this. Not only is there no CONFIG_RCU_BOOST=y, there is also no CONFIG_RCU_EXPERT=y and no CONFIG_PREEMPT_RT=y. But there is CONFIG_RT_MUTEX=y and CONFIG_RCU_EXPERT=y.

Because we don't have RCU priority boosting, if the load on the system is heavy enough to prevent our poor preempted RCU reader (PID 37) from running, the grace period cannot end.

I am not sure why this task is saving its stack, but maybe that is normal for this code path?

My bemusement aside, I recommend running this test either with non-preemptible RCU (CONFIG_PREEMPT_LAZY=y these days) or enabling RCU priority boosting (CONFIG_RCU_EXPERT=y and CONFIG_RCU_BOOST=y).

Maybe RCU_BOOST should no longer depend on RCU_EXPERT? I would of course need ot remove the prompt ("Enable RCU priority boosting") to avoid annoying Linus. Maybe as shown below.

Thoughts?

Quoted 41 lines · On Wed, 5 Aug 2026 13:28:16 -0700 "Paul E. McKenney" <[email protected]> wrote:

Quoted 17 lines

collapse_scan_file()'s main loop has

		if (need_resched()) {
			xas_pause(&xas);
			cond_resched_rcu();
		}

but that won't help with the RCU stall detector(?).

I suggest that a suitable fix here would be to add the analogous

	if (rcu_i_need_to_take_a_break()) {
		rcu_read_unlock();
		rcu_take_a_break())	
		rcu_read_lock();
	}

Quoted 4 lines

(iirc rcu_read_unlock() does an rcu run, so rcu_take_a_break() isn't needed here)

Paul, wdyt?

Let's see...

The console log says "rcu_preempt detected stalls on CPUs/tasks", which means that cond_resched() is a no-op, but it also means that the rcu_read_unlock() in cond_resched_rcu() will directly take care of informing RCU of the pause.

But that is clearly not happening. Why?

Well, we have this:

rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P37/1:b..l

This means that the task whose RCU read-side critical section is blocking the current RCU grace period isn't even running, and thus cannot invoke cond_resched_rcu(), let alone the rcu_read_unlock() within that function. So an RCU CPU stall warning is expected behavior. Or at least it is not in any way ruled out.

Quoted 1 line

What we need is RCU priority boosting.

Do we? I'm suggesting we need need_resched_rcu()!

Quoted 20 lines

Except that the .config file
does not enable this. Not only is there no CONFIG_RCU_BOOST=y, there is also no CONFIG_RCU_EXPERT=y and no CONFIG_PREEMPT_RT=y. But there is CONFIG_RT_MUTEX=y and CONFIG_RCU_EXPERT=y.

Because we don't have RCU priority boosting, if the load on the system is heavy enough to prevent our poor preempted RCU reader (PID 37) from running, the grace period cannot end.

I am not sure why this task is saving its stack, but maybe that is normal for this code path?

My bemusement aside, I recommend running this test either with non-preemptible RCU (CONFIG_PREEMPT_LAZY=y these days) or enabling RCU priority boosting (CONFIG_RCU_EXPERT=y and CONFIG_RCU_BOOST=y).

Maybe RCU_BOOST should no longer depend on RCU_EXPERT? I would of course need ot remove the prompt ("Enable RCU priority boosting") to avoid annoying Linus. Maybe as shown below.

Quoted 1 line

Thoughts?

If I'm understanding correctly, this workload is busted with this config and the proposed fix is to alter the config? Well, why are we permitting that config at all?

Seems to me that a solution to permit this config to work is very simple. Something like:

	time_t start;

	rcu_read_lock();
	start = current_time();

	for (lots of work) {
		...
		if (need_resched_rcu(start)) {
			cond_resched_rcu();
			start = current_time();
	}

Where need_resched_rcu() tests to see if we're getting close to hitting the watchdog timeout.

No?

Quoted 48 lines · Andrew Morton

Quoted 4 lines · On Tue, 04 Aug 2026 17:01:48 -0700 syzbot…

Hello,

syzbot found the following issue on:

Quoted 8 lines

HEAD commit:    3708dd948844 Merge tag 'pm-7.2-rc6' of git://git.kernel.or..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11ac703e580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=4e38b15c29e6a1d9
dashboard link: https://syzkaller.appspot.com/bug?extid=d2401aeb74cc84adba04
compiler:       Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Thanks.

Lazy optimists (ahem) paste this gunk into Gemini and ask "what the heck just happened". The results are often useful, but should be treated with skepticism. In this case I think it came usably close.

https://share.gemini.google/vq4TLhTiLBih

tl;dr: khugepaged's collapse_scan_file() is taking too long and RCU got starved. I don't think khugepaged is doing anything wrong here, per-se. There's a lot of work to do and we're doing it.

An appropriate fix would be to take a break, let RCU do its thing then get back to work. But I don't think RCU offers interfaces for that?

collapse_scan_file()'s main loop has

		if (need_resched()) {
			xas_pause(&xas);
			cond_resched_rcu();
		}

but that won't help with the RCU stall detector(?).

I suggest that a suitable fix here would be to add the analogous

	if (rcu_i_need_to_take_a_break()) {
		rcu_read_unlock();
		rcu_take_a_break())	
		rcu_read_lock();
	}

Quoted 4 lines

(iirc rcu_read_unlock() does an rcu run, so rcu_take_a_break() isn't needed here)

Paul, wdyt?

Let's see...

The console log says "rcu_preempt detected stalls on CPUs/tasks", which means that cond_resched() is a no-op, but it also means that the rcu_read_unlock() in cond_resched_rcu() will directly take care of informing RCU of the pause.

But that is clearly not happening. Why?

Well, we have this:

rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P37/1:b..l

This means that the task whose RCU read-side critical section is blocking the current RCU grace period isn't even running, and thus cannot invoke cond_resched_rcu(), let alone the rcu_read_unlock() within that function. So an RCU CPU stall warning is expected behavior. Or at least it is not in any way ruled out.

What we need is RCU priority boosting. Except that the .config file does not enable this. Not only is there no CONFIG_RCU_BOOST=y, there is also no CONFIG_RCU_EXPERT=y and no CONFIG_PREEMPT_RT=y. But there is CONFIG_RT_MUTEX=y and CONFIG_RCU_EXPERT=y.

Because we don't have RCU priority boosting, if the load on the system is heavy enough to prevent our poor preempted RCU reader (PID 37) from running, the grace period cannot end.

I am not sure why this task is saving its stack, but maybe that is normal for this code path?

My bemusement aside, I recommend running this test either with non-preemptible RCU (CONFIG_PREEMPT_LAZY=y these days) or enabling RCU priority boosting (CONFIG_RCU_EXPERT=y and CONFIG_RCU_BOOST=y).

Maybe RCU_BOOST should no longer depend on RCU_EXPERT? I would of course need ot remove the prompt ("Enable RCU priority boosting") to avoid annoying Linus. Maybe as shown below.

Thoughts?

Thanx, Paul

------------------------------------------------------------------------

kernel/rcu/Kconfig +4 -5

index 1a5fb3156c062a..5141ad8d1cd029 100644
@@ -237,17 +237,16 @@ config RCU_FANOUT_LEAF
 	  Take the default if unsure.
 
 config RCU_BOOST
-	bool "Enable RCU priority boosting"
-	depends on (RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT) || PREEMPT_RT
+	bool
+	depends on (RT_MUTEXES && PREEMPT_RCU) || PREEMPT_RT
 	default y if PREEMPT_RT
 	help
 	  This option boosts the priority of preempted RCU readers that
 	  block the current preemptible RCU grace period for too long.
 	  This option also prevents heavy loads from blocking RCU
-	  callback invocation.
+	  callback invocation.  It is now automatically enabled in
+	  any kernel that can benefit from it and that can support it.
 
-	  Say Y here if you are working with real-time apps or heavy loads
-	  Say N here if you are unsure.
 
 config RCU_BOOST_DELAY
 	int "Milliseconds to delay boosting after RCU grace-period start"

Quoted 73 lines · Paul E. McKenney

Quoted 48 lines · Andrew Morton

Quoted 4 lines · On Tue, 04 Aug 2026 17:01:48 -0700 syzbot…

Hello,

syzbot found the following issue on:

Quoted 8 lines

HEAD commit:    3708dd948844 Merge tag 'pm-7.2-rc6' of git://git.kernel.or..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11ac703e580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=4e38b15c29e6a1d9
dashboard link: https://syzkaller.appspot.com/bug?extid=d2401aeb74cc84adba04
compiler:       Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8

Unfortunately, I don't have any reproducer for this issue yet.

Thanks.

Lazy optimists (ahem) paste this gunk into Gemini and ask "what the heck just happened". The results are often useful, but should be treated with skepticism. In this case I think it came usably close.

https://share.gemini.google/vq4TLhTiLBih

tl;dr: khugepaged's collapse_scan_file() is taking too long and RCU got starved. I don't think khugepaged is doing anything wrong here, per-se. There's a lot of work to do and we're doing it.

An appropriate fix would be to take a break, let RCU do its thing then get back to work. But I don't think RCU offers interfaces for that?

collapse_scan_file()'s main loop has

		if (need_resched()) {
			xas_pause(&xas);
			cond_resched_rcu();
		}

but that won't help with the RCU stall detector(?).

I suggest that a suitable fix here would be to add the analogous

	if (rcu_i_need_to_take_a_break()) {
		rcu_read_unlock();
		rcu_take_a_break())	
		rcu_read_lock();
	}

Quoted 4 lines

(iirc rcu_read_unlock() does an rcu run, so rcu_take_a_break() isn't needed here)

Paul, wdyt?

Let's see...

The console log says "rcu_preempt detected stalls on CPUs/tasks", which means that cond_resched() is a no-op, but it also means that the rcu_read_unlock() in cond_resched_rcu() will directly take care of informing RCU of the pause.

But that is clearly not happening. Why?

Well, we have this:

rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P37/1:b..l

This means that the task whose RCU read-side critical section is blocking the current RCU grace period isn't even running, and thus cannot invoke cond_resched_rcu(), let alone the rcu_read_unlock() within that function. So an RCU CPU stall warning is expected behavior. Or at least it is not in any way ruled out.

Quoted 4 lines

What we need is RCU priority boosting. Except that the .config file does not enable this. Not only is there no CONFIG_RCU_BOOST=y, there is also no CONFIG_RCU_EXPERT=y and no CONFIG_PREEMPT_RT=y. But there is CONFIG_RT_MUTEX=y and CONFIG_RCU_EXPERT=y.

It comes from syzbot so might be likely a randconfig and there's no point in trying to find any sense in that combination :)

Quoted 6 lines

Because we don't have RCU priority boosting, if the load on the system is heavy enough to prevent our poor preempted RCU reader (PID 37) from running, the grace period cannot end.

I am not sure why this task is saving its stack, but maybe that is normal for this code path?

That's because page_owner is also enabled so it's saving the freeing stack for the page it's freeing. That's not a normal production config, only when debugging.

Quoted 7 lines

My bemusement aside, I recommend running this test either with non-preemptible RCU (CONFIG_PREEMPT_LAZY=y these days) or enabling RCU priority boosting (CONFIG_RCU_EXPERT=y and CONFIG_RCU_BOOST=y).

Maybe RCU_BOOST should no longer depend on RCU_EXPERT? I would of course need ot remove the prompt ("Enable RCU priority boosting") to avoid annoying Linus. Maybe as shown below.

The "no longer depend" part alone would make no difference with randconfigs. Removing the prompt too should help indeed.

Maybe a possible strategy in general would be indeed to unconditionally select what's the expected config, like you did below, and only make it possible to override that with RCU_EXPERT. So here with RCU_EXPERT you could disable RCU_BOOST even if it was automatically enabled - assuming this is useful for development or internal rcu testing by people who know what they are doing (not syzbot randconfig) or whatnot.

But then RCU_EXPERT should be excluded from (impossible to be enabled by) randconfig to indicate it's not valid for this kind of testing. I don't know if there's any precedent for such a strategy.

Specifically for the proposal below, could the problem still happen with PREEMPT_RCU without RT_MUTEXES? If yes, it wouldn't be enough?

Quoted 28 lines

------------------------------------------------------------------------

kernel/rcu/Kconfig +4 -5

index 1a5fb3156c062a..5141ad8d1cd029 100644
@@ -237,17 +237,16 @@ config RCU_FANOUT_LEAF
 	  Take the default if unsure.
 
 config RCU_BOOST
-	bool "Enable RCU priority boosting"
-	depends on (RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT) || PREEMPT_RT
+	bool
+	depends on (RT_MUTEXES && PREEMPT_RCU) || PREEMPT_RT
 	default y if PREEMPT_RT
 	help
 	  This option boosts the priority of preempted RCU readers that
 	  block the current preemptible RCU grace period for too long.
 	  This option also prevents heavy loads from blocking RCU
-	  callback invocation.
+	  callback invocation.  It is now automatically enabled in
+	  any kernel that can benefit from it and that can support it.
 
-	  Say Y here if you are working with real-time apps or heavy loads
-	  Say N here if you are unsure.
 
 config RCU_BOOST_DELAY
 	int "Milliseconds to delay boosting after RCU grace-period start"