page_pool: avoid infinite loop to schedule delayed worker
Subsystems (3)
PATCH v3 1/1 page_pool: avoid infinite loop to schedule delayed worker
We noticed the kworker in page_pool_release_retry() was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].
Since the inflight value goes negative, it means we should not expect the whole page_pool to get back to work normally.
This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in page_pool_release_retry().
[1]
[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------ [Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages ... [Mon Feb 10 20:36:11 2025] Call Trace: [Mon Feb 10 20:36:11 2025] page_pool_release_retry+0x23/0x70 [Mon Feb 10 20:36:11 2025] process_one_work+0x1b1/0x370 [Mon Feb 10 20:36:11 2025] worker_thread+0x37/0x3a0 [Mon Feb 10 20:36:11 2025] kthread+0x11a/0x140 [Mon Feb 10 20:36:11 2025] ? process_one_work+0x370/0x370 [Mon Feb 10 20:36:11 2025] ? __kthread_cancel_work+0x40/0x40 [Mon Feb 10 20:36:11 2025] ret_from_fork+0x35/0x40 [Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of release_dw kworker.
---
v3
1. remove printing warning even when inflight is negative, or else it will cause panic.
v2
1. add more details in commit message.
2. allow printing once when the inflight is negative.
3. correct the position where to stop the reschedule. Suggested by Mina and Jakub.
--- net/core/page_pool.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
net/core/page_pool.c +7 -1
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct work_struct *wq)int inflight; inflight =page_pool_release (pool); -if (!inflight) +/* In rare cases, a driver bug may cause inflight to go negative. +* Don't reschedule release if inflight is 0 or negative. +* - If 0, the page_pool has been destroyed +* - if negative, we will never recover +* in both cases no reschedule is necessary. +*/ +if (inflight <=0 )return ;/* Periodic warning for page pools the user can't see */
Signature · 1 line
2.43.5
Quoted 26 lines · Jason Xing
We noticed the kworker in
page_pool_release_retry()was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].Since the inflight value goes negative, it means we should not expect the whole
page_poolto get back to work normally.This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in
page_pool_release_retry().[1]
[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------ [Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages ... [Mon Feb 10 20:36:11 2025] Call Trace: [Mon Feb 10 20:36:11 2025] page_pool_release_retry+0x23/0x70 [Mon Feb 10 20:36:11 2025] process_one_work+0x1b1/0x370 [Mon Feb 10 20:36:11 2025] worker_thread+0x37/0x3a0 [Mon Feb 10 20:36:11 2025] kthread+0x11a/0x140 [Mon Feb 10 20:36:11 2025] ? process_one_work+0x370/0x370 [Mon Feb 10 20:36:11 2025] ? __kthread_cancel_work+0x40/0x40 [Mon Feb 10 20:36:11 2025] ret_from_fork+0x35/0x40 [Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
Quoted 4 lines
Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of
release_dwkworker.Signed-off-by Jason Xing <[email protected]>
Thanks Jason,
When you find the root cause of the driver bug, if you can think of ways to catch it sooner in the page_pool or prevent drivers from triggering it, please do consider sending improvements upstream. Thanks!
Signature · 2 lines
Thanks, Mina
Quoted 31 lines · Mina Almasry
Quoted 29 lines · Jason Xing
We noticed the kworker in
page_pool_release_retry()was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].Since the inflight value goes negative, it means we should not expect the whole
page_poolto get back to work normally.This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in
page_pool_release_retry().[1]
[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------ [Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages ... [Mon Feb 10 20:36:11 2025] Call Trace: [Mon Feb 10 20:36:11 2025] page_pool_release_retry+0x23/0x70 [Mon Feb 10 20:36:11 2025] process_one_work+0x1b1/0x370 [Mon Feb 10 20:36:11 2025] worker_thread+0x37/0x3a0 [Mon Feb 10 20:36:11 2025] kthread+0x11a/0x140 [Mon Feb 10 20:36:11 2025] ? process_one_work+0x370/0x370 [Mon Feb 10 20:36:11 2025] ? __kthread_cancel_work+0x40/0x40 [Mon Feb 10 20:36:11 2025] ret_from_fork+0x35/0x40 [Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of
release_dwkworker.
Quoted 5 lines
Quoted 1 line
Signed-off-by Jason Xing <[email protected]>Thanks Jason,
Reviewed-by Mina Almasry <[email protected]>
Thanks for the review.
Quoted 5 lines
When you find the root cause of the driver bug, if you can think of ways to catch it sooner in the
page_poolor prevent drivers from triggering it, please do consider sending improvements upstream. Thanks!
Sure, it's exactly what I want to do :)
Thanks,
Jason
Quoted 4 lines
--
Thanks,
Mina
Jesper Dangaard Brouer
maintainerQuoted 41 lines · Jason Xing
Quoted 31 lines · Mina Almasry
Quoted 29 lines · Jason Xing
We noticed the kworker in
page_pool_release_retry()was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].Since the inflight value goes negative, it means we should not expect the whole
page_poolto get back to work normally.This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in
page_pool_release_retry().[1]
[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------ [Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages ... [Mon Feb 10 20:36:11 2025] Call Trace: [Mon Feb 10 20:36:11 2025] page_pool_release_retry+0x23/0x70 [Mon Feb 10 20:36:11 2025] process_one_work+0x1b1/0x370 [Mon Feb 10 20:36:11 2025] worker_thread+0x37/0x3a0 [Mon Feb 10 20:36:11 2025] kthread+0x11a/0x140 [Mon Feb 10 20:36:11 2025] ? process_one_work+0x370/0x370 [Mon Feb 10 20:36:11 2025] ? __kthread_cancel_work+0x40/0x40 [Mon Feb 10 20:36:11 2025] ret_from_fork+0x35/0x40 [Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of
release_dwkworker.Quoted 5 lines
Quoted 1 line
Signed-off-by Jason Xing <[email protected]>Thanks Jason,
Reviewed-by Mina Almasry <[email protected]>Thanks for the review.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
When you find the root cause of the driver bug, if you can think of ways to catch it sooner in the
page_poolor prevent drivers from triggering it, please do consider sending improvements upstream. Thanks!Sure, it's exactly what I want to do :)
What driver is this happening in?
--Jesper
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Quoted 9 lines · On Fri, 14 Feb 2025 14:42:50 +0800 you wrote:
We noticed the kworker in
page_pool_release_retry()was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].Since the inflight value goes negative, it means we should not expect the whole
page_poolto get back to work normally.[...]
Here is the summary with links:
- [net-next,v3] page_pool: avoid infinite loop to schedule delayed worker
https://git.kernel.org/netdev/net-next/c/43130d02baa1
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html