Skip to contentSkip to conversation

page_pool: avoid infinite loop to schedule delayed worker

PATCH Applied Jason Xing 4 participants 5 messages Feb 14, 2025 – Feb 18, 2025

Subsystems (3)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

PATCH v3 1/1 page_pool: avoid infinite loop to schedule delayed worker

We noticed the kworker in page_pool_release_retry() was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].

Since the inflight value goes negative, it means we should not expect the whole page_pool to get back to work normally.

This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in page_pool_release_retry().

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---

Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of release_dw kworker.

---
v3

1. remove printing warning even when inflight is negative, or else it will cause panic.

v2

1. add more details in commit message.
2. allow printing once when the inflight is negative.
3. correct the position where to stop the reschedule. Suggested by Mina and Jakub.

---
 net/core/page_pool.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
net/core/page_pool.c +7 -1
index 1c6fec08bc43..acef1fcd8ddc 100644
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct work_struct *wq)
 	int inflight;
 
 	inflight = page_pool_release(pool);
-	if (!inflight)
+	/* In rare cases, a driver bug may cause inflight to go negative.
+	 * Don't reschedule release if inflight is 0 or negative.
+	 * - If 0, the page_pool has been destroyed
+	 * - if negative, we will never recover
+	 * in both cases no reschedule is necessary.
+	 */
+	if (inflight <= 0)
 		return;
 
 	/* Periodic warning for page pools the user can't see */
Signature · 1 line
2.43.5
Reviewed-by on v3 1/1
#1 lore raw
Quoted 26 lines · Jason Xing

We noticed the kworker in page_pool_release_retry() was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].

Since the inflight value goes negative, it means we should not expect the whole page_pool to get back to work normally.

This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in page_pool_release_retry().

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
Quoted 4 lines

Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of release_dw kworker.

Thanks Jason,

When you find the root cause of the driver bug, if you can think of ways to catch it sooner in the page_pool or prevent drivers from triggering it, please do consider sending improvements upstream. Thanks!

Signature · 2 lines
Thanks,
Mina
author
#2 lore raw
Quoted 31 lines · Mina Almasry
Quoted 29 lines · Jason Xing

We noticed the kworker in page_pool_release_retry() was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].

Since the inflight value goes negative, it means we should not expect the whole page_pool to get back to work normally.

This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in page_pool_release_retry().

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---

Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of release_dw kworker.

Quoted 5 lines
Quoted 1 line

Thanks Jason,

Thanks for the review.

Quoted 5 lines

When you find the root cause of the driver bug, if you can think of ways to catch it sooner in the page_pool or prevent drivers from triggering it, please do consider sending improvements upstream. Thanks!

Sure, it's exactly what I want to do :)

Thanks,
Jason

Quoted 4 lines

--
Thanks,
Mina

#3 lore raw
Quoted 41 lines · Jason Xing
Quoted 31 lines · Mina Almasry
Quoted 29 lines · Jason Xing

We noticed the kworker in page_pool_release_retry() was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].

Since the inflight value goes negative, it means we should not expect the whole page_pool to get back to work normally.

This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in page_pool_release_retry().

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---

Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of release_dw kworker.

Quoted 5 lines
Quoted 1 line

Thanks Jason,

Thanks for the review.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

When you find the root cause of the driver bug, if you can think of ways to catch it sooner in the page_pool or prevent drivers from triggering it, please do consider sending improvements upstream. Thanks!

Sure, it's exactly what I want to do :)

What driver is this happening in?

--Jesper

#4 lore raw
In reply to Jason Xing
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:

Hello:

This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:

Quoted 9 lines · On Fri, 14 Feb 2025 14:42:50 +0800 you wrote:

We noticed the kworker in page_pool_release_retry() was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].

Since the inflight value goes negative, it means we should not expect the whole page_pool to get back to work normally.

[...]

Here is the summary with links:
- [net-next,v3] page_pool: avoid infinite loop to schedule delayed worker

    https://git.kernel.org/netdev/net-next/c/43130d02baa1

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html