mm/kmemleak: report RCU-tasks quiescent states during the scan

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

Signed-off-by Breno Leitao <[email protected]>

---
 mm/kmemleak.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

mm/kmemleak.c +7 -8

index 85f18b17e79c4..f63dfacee7ca1 100644
@@ -1586,7 +1586,7 @@ static int scan_large_block(void *start, void *end)
 		if (scan_block(start, next, NULL))
 			return 1;
 		start = next;
-		cond_resched();
+		cond_resched_tasks_rcu_qs();
 	}
 
 	return 0;

@@ -1623,7 +1623,7 @@ static void scan_object(struct kmemleak_object *object)
 			scan_block(start, end, object);
 
 			raw_spin_unlock_irqrestore(&object->lock, flags);
-			cond_resched();
+			cond_resched_tasks_rcu_qs();
 			raw_spin_lock_irqsave(&object->lock, flags);
 			if (!(object->flags & OBJECT_ALLOCATED))
 				break;

@@ -1645,7 +1645,7 @@ static void scan_object(struct kmemleak_object *object)
 				break;
 
 			raw_spin_unlock_irqrestore(&object->lock, flags);
-			cond_resched();
+			cond_resched_tasks_rcu_qs();
 			raw_spin_lock_irqsave(&object->lock, flags);
 		} while (object->flags & OBJECT_ALLOCATED);
 	} else {

@@ -1673,7 +1673,7 @@ static void scan_gray_list(void)
 	 */
 	object = list_entry(gray_list.next, typeof(*object), gray_list);
 	while (&object->gray_list != &gray_list) {
-		cond_resched();
+		cond_resched_tasks_rcu_qs();
 
 		/* may add new objects to the list */
 		if (!scan_should_stop())

@@ -1708,7 +1708,7 @@ static void kmemleak_cond_resched(struct kmemleak_object *object)
 	raw_spin_unlock_irq(&kmemleak_lock);
 
 	rcu_read_unlock();
-	cond_resched();
+	cond_resched_tasks_rcu_qs();
 	rcu_read_lock();
 
 	raw_spin_lock_irq(&kmemleak_lock);

@@ -1753,7 +1753,7 @@ static void kmemleak_scan_task_stacks(void)
 			}
 			put_task_struct(p);
 		}
-		cond_resched();
+		cond_resched_tasks_rcu_qs();
 	} while (pid && !stop);
 }
 

@@ -1935,7 +1935,7 @@ static int __kmemleak_scan(bool full)
 			struct page *page = pfn_to_online_page(pfn);
 
 			if (!(pfn & 63))
-				cond_resched();
+				cond_resched_tasks_rcu_qs();
 
 			if (!page)
 				continue;
 
---

base-commit: 0718283ab28bc3907e10b61a6b4be6fefa1cbb2f
change-id: 20260720-kmemleak_rcu_task-6d3ea111b947

Best regards,

Signature · 1 line

Breno Leitao <[email protected]>

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

Quoted 27 lines · On Mon, 20 Jul 2026 06:23:45 -0700 Breno Leitao <[email protected]> wrote:

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Quoted 2 lines

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

I'll add

Fixes c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning")

Cc <[email protected]>

Quoted 29 lines · Andrew Morton

Quoted 27 lines · On Mon, 20 Jul 2026 06:23:45 -0700 Breno Leitao <[email protected]> wrote:

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Quoted 7 lines

Quoted 2 lines

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

I'll add

Fixes c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning")

Cc <[email protected]>

Thanks to all three of you!

This adds fewer than ten calls to cond_resched_tasks_rcu_qs(), but still more than doubles the number of such calls outside of the RCU subsystem.

Which is most likely just fine, and in any case absolutely should not get in the way of Breno's patch, which after all solves a real problem in the here and now.

Nevertheless, on the off-chance that over the next few months or years we start playing cond_resched_tasks_rcu_qs() whack-a-mole, I figured it would be good to get a head start on writing up alternatives. An initial draft may be found here:

https://docs.google.com/document/d/1s3fn29SCTYVw9jak4iraNIVQR59Wk5C6_I-_eu-MfQA/edit?usp=sharing

TL;DR: Should we get into a rousing game of whack-a-mole, alternatives include continuing as we are, making the existing calls to cond_resched() in turn call cond_resched_tasks_rcu_qs(), decoupling mutex-induced hung-task warnings from synchronize_rcu_tasks(), and various not-so-practical alternatives to RCU Tasks for trampoline synchronization.

Thoughts? Especially thoughts on other schemes?

Hello Paul,

Quoted 60 lines · Paul E. McKenney

Quoted 29 lines · Andrew Morton

Quoted 27 lines · On Mon, 20 Jul 2026 06:23:45 -0700 Breno Leitao <[email protected]> wrote:

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Quoted 7 lines

Quoted 2 lines

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

I'll add

Fixes c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning")

Cc <[email protected]>

Thanks to all three of you!

This adds fewer than ten calls to cond_resched_tasks_rcu_qs(), but still more than doubles the number of such calls outside of the RCU subsystem.

Which is most likely just fine, and in any case absolutely should not get in the way of Breno's patch, which after all solves a real problem in the here and now.

Nevertheless, on the off-chance that over the next few months or years we start playing cond_resched_tasks_rcu_qs() whack-a-mole, I figured it would be good to get a head start on writing up alternatives. An initial draft may be found here:

https://docs.google.com/document/d/1s3fn29SCTYVw9jak4iraNIVQR59Wk5C6_I-_eu-MfQA/edit?usp=sharing

TL;DR: Should we get into a rousing game of whack-a-mole, alternatives include continuing as we are, making the existing calls to cond_resched() in turn call cond_resched_tasks_rcu_qs(), decoupling mutex-induced hung-task warnings from synchronize_rcu_tasks(), and various not-so-practical alternatives to RCU Tasks for trampoline synchronization.

Quoted 1 line

Thoughts? Especially thoughts on other schemes?

While debugging this issue, I was surprised to discover that cond_resched() doesn't provide RCU-tasks quiescent states. That led me to cond_resched_tasks_rcu_qs(), which is the stronger primitive needed for long-running kernel threads like kmemleak_scan(). Is this the common case for cond_resched()? I got the impression that kmemleak is the extreme side, but, I have no data on this.

Worth noting that commit 7dadeaa6e851e7 ("sched: Further restrict the preemption modes") continues to narrow PREEMPT_NONE, so the future of cond_resched() itself may be uncertain, no?

Quoted 63 lines · Breno Leitao

Hello Paul,

Quoted 60 lines · Paul E. McKenney

Quoted 29 lines · Andrew Morton

Quoted 27 lines · On Mon, 20 Jul 2026 06:23:45 -0700 Breno Leitao <[email protected]> wrote:

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Quoted 7 lines

Quoted 2 lines

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

I'll add

Fixes c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning")

Cc <[email protected]>

G
Thanks to all three of you!

This adds fewer than ten calls to cond_resched_tasks_rcu_qs(), but still more than doubles the number of such calls outside of the RCU subsystem.

Which is most likely just fine, and in any case absolutely should not get in the way of Breno's patch, which after all solves a real problem in the here and now.

Nevertheless, on the off-chance that over the next few months or years we start playing cond_resched_tasks_rcu_qs() whack-a-mole, I figured it would be good to get a head start on writing up alternatives. An initial draft may be found here:

https://docs.google.com/document/d/1s3fn29SCTYVw9jak4iraNIVQR59Wk5C6_I-_eu-MfQA/edit?usp=sharing

TL;DR: Should we get into a rousing game of whack-a-mole, alternatives include continuing as we are, making the existing calls to cond_resched() in turn call cond_resched_tasks_rcu_qs(), decoupling mutex-induced hung-task warnings from synchronize_rcu_tasks(), and various not-so-practical alternatives to RCU Tasks for trampoline synchronization.

Quoted 8 lines

Quoted 1 line

Thoughts? Especially thoughts on other schemes?

While debugging this issue, I was surprised to discover that cond_resched() doesn't provide RCU-tasks quiescent states. That led me to cond_resched_tasks_rcu_qs(), which is the stronger primitive needed for long-running kernel threads like kmemleak_scan(). Is this the common case for cond_resched()? I got the impression that kmemleak is the extreme side, but, I have no data on this.

And that discovery did in fact lead you correctly, and hence my giving you a Reviewed-by.

Quoted 3 lines

Worth noting that commit 7dadeaa6e851e7 ("sched: Further restrict the preemption modes") continues to narrow PREEMPT_NONE, so the future of cond_resched() itself may be uncertain, no?

Especially given that one of the explicitly stated goals of lazy preemption was in fact to eliminate cond_resched(). ;-)

Getting back to your point about kmemleak possibly being extreme, as far as I know, none of us have data on how many conversions of cond_resched() to cond_resched_tasks_rcu_qs() will be required, or, for that matter, how many additions of cond_resched_tasks_rcu_qs() above and beyond current cond_resched() instances will be required.

There were concerns raised off-list, so I figured that it was time to lay out what we do know, hence the above gdoc. I very much hope that we don't need many more calls to cond_resched_tasks_rcu_qs(), and thus that the time I spent writing that turns out to be wasted time. But better safe than sorry!

Quoted 80 lines · Paul E. McKenney

Quoted 63 lines · Breno Leitao

Hello Paul,

Quoted 60 lines · Paul E. McKenney

Quoted 29 lines · Andrew Morton

Quoted 27 lines · On Mon, 20 Jul 2026 06:23:45 -0700 Breno Leitao <[email protected]> wrote:

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Quoted 7 lines

Quoted 2 lines

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

I'll add

Fixes c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning")

Cc <[email protected]>

G
Thanks to all three of you!

This adds fewer than ten calls to cond_resched_tasks_rcu_qs(), but still more than doubles the number of such calls outside of the RCU subsystem.

Which is most likely just fine, and in any case absolutely should not get in the way of Breno's patch, which after all solves a real problem in the here and now.

Nevertheless, on the off-chance that over the next few months or years we start playing cond_resched_tasks_rcu_qs() whack-a-mole, I figured it would be good to get a head start on writing up alternatives. An initial draft may be found here:

https://docs.google.com/document/d/1s3fn29SCTYVw9jak4iraNIVQR59Wk5C6_I-_eu-MfQA/edit?usp=sharing

TL;DR: Should we get into a rousing game of whack-a-mole, alternatives include continuing as we are, making the existing calls to cond_resched() in turn call cond_resched_tasks_rcu_qs(), decoupling mutex-induced hung-task warnings from synchronize_rcu_tasks(), and various not-so-practical alternatives to RCU Tasks for trampoline synchronization.

Quoted 8 lines

Quoted 1 line

Thoughts? Especially thoughts on other schemes?

While debugging this issue, I was surprised to discover that cond_resched() doesn't provide RCU-tasks quiescent states. That led me to cond_resched_tasks_rcu_qs(), which is the stronger primitive needed for long-running kernel threads like kmemleak_scan(). Is this the common case for cond_resched()? I got the impression that kmemleak is the extreme side, but, I have no data on this.

And that discovery did in fact lead you correctly, and hence my giving you a Reviewed-by.

Quoted 3 lines

Worth noting that commit 7dadeaa6e851e7 ("sched: Further restrict the preemption modes") continues to narrow PREEMPT_NONE, so the future of cond_resched() itself may be uncertain, no?

Quoted 8 lines

Especially given that one of the explicitly stated goals of lazy preemption was in fact to eliminate cond_resched(). ;-)

Getting back to your point about kmemleak possibly being extreme, as far as I know, none of us have data on how many conversions of cond_resched() to cond_resched_tasks_rcu_qs() will be required, or, for that matter, how many additions of cond_resched_tasks_rcu_qs() above and beyond current cond_resched() instances will be required.

I've searched for "rcu_tasks detected stalls on tasks" on Meta's fleet "dmesg depot" and I was suprised to see we have many of them, in which seems to be on hypervisor workloads (running VMs).

Quoted 5 lines

There were concerns raised off-list, so I figured that it was time to lay out what we do know, hence the above gdoc. I very much hope that we don't need many more calls to cond_resched_tasks_rcu_qs(), and thus that the time I spent writing that turns out to be wasted time. But better safe than sorry!

The document was not a waste of time. it helped people like me to understand what the issue is, and what are the decisions we have ahead of us.

Back to what is the best decision/design, I honestly don't have an opinion, but I am happy to get more data about Meta production servers to help with the decision.

Quoted 7 lines

The document was not a waste of time. it helped people like me to understand what the issue is, and what are the decisions we have ahead of us.

Back to what is the best decision/design, I honestly don't have an opinion, but I am happy to get more data about Meta production servers to help with the decision.

Looking forward to seeing what you come up with!

Quoted 9 lines

Quoted 7 lines

The document was not a waste of time. it helped people like me to understand what the issue is, and what are the decisions we have ahead of us.

Back to what is the best decision/design, I honestly don't have an opinion, but I am happy to get more data about Meta production servers to help with the decision.

Looking forward to seeing what you come up with!

I found 3 different cases on Meta fleet, where rcu task stalls show up:

1) kmemleak -> This patch solves it
2) KVM / kcompactd
* Holdout: kcompactd0 (pid 1216), state:R, nvcsw frozen at 716336/716336 across three reports 10 min apart (stuck ≥20 min in one compaction pass)
* This is coming from: migrate_pages ->

    kvm_mmu_notifier_invalidate_range_start ->
	   tdp_mmu_next_root -> tdp_mmu_zap_leafs

3) Nvidia driver
* stuck in nv_procfs_read_lock_params

Given we can't do much about 3) and 1) is now closed, I will spend some time geting more data and possible a fix for 2.

I found 3 different cases on Meta fleet, where rcu task stalls show up:

3) Nvidia driver
* stuck in nv_procfs_read_lock_params

Quoted 1 line · Breno Leitao

I found 3 different cases on Meta fleet, where rcu task stalls show up:

...

Quoted 2 lines

3) Nvidia driver
* stuck in nv_procfs_read_lock_params

I've investigated this issue. The nvidia driver contains the following locking pattern:

        static inline int nv_down_read_interruptible(struct rw_semaphore *lock)
        {
                while (!down_read_trylock(lock))
                {
                        if (signal_pending(current))
                        return -EINTR;
                        cond_resched();
                }
                return 0;
        }

Code at: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/452cec62d827034798072827d3866d1881662b77/kernel-open/common/inc/nv-lock.h#L54

The lock is never acquired, and cond_resched() provides insufficient scheduling relief.

This triggers a more severe issue with ftrace, which holds ftrace_lock across RCU synchronization:

        mutex_lock(&ftrace_lock);
                ftrace_shutdown(struct ftrace_ops *ops, int command) {
                        synchronize_rcu_tasks_rude();
                        synchronize_rcu_tasks();
                }
        mutex_unlock(&ftrace_lock);

Since nv_down_read_interruptible() spins indefinitely, ftrace_lock now becomes completely wedged.

I suppose we want two things:

1) change cond_resched() with cond_resched_tasks_rcu_qs()
2) Avoiding holding the ftrace lock while calling synchronize_rcu_tasks()? It can take up to 10 minutes on a healthy system to be releasd, right?

Quoted 6 lines

I suppose we want two things:

1) change cond_resched() with cond_resched_tasks_rcu_qs()
2) Avoiding holding the ftrace lock while calling synchronize_rcu_tasks()? It can take up to 10 minutes on a healthy system to be releasd, right?

I do very much like both of those options. ;-)

If #2 is impossible, would it make sense to have a mutex_lock_idle() or similar in order to sanctify the lock synchronize_rcu_tasks() latencies?

Quoted 7 lines · Paul E. McKenney

Quoted 6 lines

I suppose we want two things:

1) change cond_resched() with cond_resched_tasks_rcu_qs()
2) Avoiding holding the ftrace lock while calling synchronize_rcu_tasks()? It can take up to 10 minutes on a healthy system to be releasd, right?

Quoted 4 lines

I do very much like both of those options. ;-)

If #2 is impossible, would it make sense to have a mutex_lock_idle() or similar in order to sanctify the lock synchronize_rcu_tasks() latencies?

How would mutex_lock_idle() differ from a regular mutex_lock() when sanctifying synchronize_rcu_tasks() latencies?

I found 3 different cases on Meta fleet, where rcu task stalls show up:

1) kmemleak -> This patch solves it
2) KVM / kcompactd
* Holdout: kcompactd0 (pid 1216), state:R, nvcsw frozen at 716336/716336 across three reports 10 min apart (stuck ≥20 min in one compaction pass)
* This is coming from: migrate_pages ->

    kvm_mmu_notifier_invalidate_range_start ->
	   tdp_mmu_next_root -> tdp_mmu_zap_leafs

3) Nvidia driver
* stuck in nv_procfs_read_lock_params

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

Signed-off-by Breno Leitao <[email protected]>

kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched().

I've got the fix above deployed in the Meta fleet, and now I am seeing:

  INFO: rcu_tasks detected stalls on tasks:
    task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
     scan_block
     scan_gray_list
     kmemleak_scan

and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check:

  INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
     __wait_rcu_gp
     bpf_struct_ops_map_free

Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives.

Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds.

Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu").

Signed-off-by Breno Leitao <[email protected]>