Skip to contentSkip to conversation

page_pool: avoid infinite loop to schedule delayed worker

PATCH Superseded Jason Xing 3 participants 15 messages Feb 10, 2025 – Feb 13, 2025

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jason Xing

author
#0 lore raw

PATCH 1/1 page_pool: avoid infinite loop to schedule delayed worker

If the buggy driver causes the inflight less than 0 [1] and warns us in page_pool_inflight(), it means we should not expect the whole page_pool to get back to work normally.

We noticed the kworker is waken up repeatedly and infinitely[1] in production. If the page pool detect the error happening, probably letting it go is a better way and do not flood the var log messages. This patch mitigates the adverse effect.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
---
 net/core/page_pool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +1 -1
index 1c6fec08bc43..8e9f5801aabb 100644
@@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
 	page_pool_disable_direct_recycling(pool);
 	page_pool_free_frag(pool);
 
-	if (!page_pool_release(pool))
+	if (page_pool_release(pool) <= 0)
 		return;
 
 	page_pool_detached(pool);
Signature · 1 line
2.43.5
#1 lore raw
Quoted 2 lines · Jason Xing

If the buggy driver causes the inflight less than 0 [1] and warns

How does a buggy driver trigger this?

Quoted 32 lines

us in page_pool_inflight(), it means we should not expect the whole page_pool to get back to work normally.

We noticed the kworker is waken up repeatedly and infinitely[1] in production. If the page pool detect the error happening, probably letting it go is a better way and do not flood the var log messages. This patch mitigates the adverse effect.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
---
 net/core/page_pool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +0 -0
index 1c6fec08bc43..8e9f5801aabb 100644

net/core/page_pool.c void page_pool_destroy(struct page_pool *pool) · line 1171

Quoted 6 lines
net/core/page_pool.c +1 -1
@@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
        page_pool_disable_direct_recycling(pool);
        page_pool_free_frag(pool);
 
-       if (!page_pool_release(pool))
+       if (page_pool_release(pool) <= 0)
                return;

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

I also wonder also whether if the check in page_pool_release() itself needs to be:

if (inflight < 0)
__page_pool_destroy();

otherwise the pool will never be destroyed no?

Signature · 2 lines
Thanks,
Mina
maintainer
#2 lore raw
Quoted 3 lines · On Tue, 11 Feb 2025 18:37:22 -0800 Mina Almasry wrote:

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

+1

A code comment may be useful BTW.

Quoted 7 lines

I also wonder also whether if the check in page_pool_release() itself needs to be:

if (inflight < 0)
__page_pool_destroy();

otherwise the pool will never be destroyed no?

It's probably safer to leak the memory than risk a crash if we undercounted and some page will try to return itself later?

author
#3 lore raw
Quoted 9 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 11 Feb 2025 18:37:22 -0800 Mina Almasry wrote:

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

+1

A code comment may be useful BTW.

I will add it in the next version. Yes, my intention is to avoid initializing the delayed work since we don't expect the worker in page_pool_release_retry() to try over and over again.

Quoted 11 lines
Quoted 8 lines

I also wonder also whether if the check in page_pool_release() itself needs to be:

if (inflight < 0)
__page_pool_destroy();

otherwise the pool will never be destroyed no?

It's probably safer to leak the memory than risk a crash if we undercounted and some page will try to return itself later?

It's also what I'm concerned about.

Thanks,
Jason

maintainer
#4 lore raw
Quoted 7 lines · On Wed, 12 Feb 2025 11:20:16 +0800 Jason Xing wrote:
Quoted 6 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 11 Feb 2025 18:37:22 -0800 Mina Almasry wrote:

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

Quoted 7 lines
Quoted 3 lines

+1

A code comment may be useful BTW.

I will add it in the next version. Yes, my intention is to avoid initializing the delayed work since we don't expect the worker in page_pool_release_retry() to try over and over again.

Initializing a work isn't much cost, is it?

Just to state the obvious the current patch will not catch the situation when there is traffic outstanding (inflight is positive) at the time of detach from the driver. But then the inflight goes negative before the work / time kicks in.

author
#5 lore raw
Quoted 11 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 12 Feb 2025 11:20:16 +0800 Jason Xing wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Tue, 11 Feb 2025 18:37:22 -0800 Mina Almasry wrote:

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

+1

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

A code comment may be useful BTW.

I will add it in the next version. Yes, my intention is to avoid initializing the delayed work since we don't expect the worker in page_pool_release_retry() to try over and over again.

Initializing a work isn't much cost, is it?

Not that much, but it's pointless to start a kworker under this circumstance, right? And it will flood the dmesg.

Quoted 5 lines

Just to state the obvious the current patch will not catch the situation when there is traffic outstanding (inflight is positive) at the time of detach from the driver. But then the inflight goes negative before the work / time kicks in.

Right, only mitigating the side effect. I will add this statement as well while keeping the code itself as-is.

Thanks,
Jason

maintainer
#6 lore raw
Quoted 4 lines · On Wed, 12 Feb 2025 12:38:28 +0800 Jason Xing wrote:
Quoted 1 line

Initializing a work isn't much cost, is it?

Not that much, but it's pointless to start a kworker under this circumstance, right? And it will flood the dmesg.

There's a seriously buggy driver potentially corrupting memory, who cares if we start a kworker. Please don't complicate the code for extremely rare scenarios.

Quoted 7 lines
Quoted 4 lines

Just to state the obvious the current patch will not catch the situation when there is traffic outstanding (inflight is positive) at the time of detach from the driver. But then the inflight goes negative before the work / time kicks in.

Right, only mitigating the side effect. I will add this statement as well while keeping the code itself as-is.

What do you mean by that?! We're telling you your code is wrong.

author
#7 lore raw
Quoted 10 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 12 Feb 2025 12:38:28 +0800 Jason Xing wrote:
Quoted 1 line

Initializing a work isn't much cost, is it?

Not that much, but it's pointless to start a kworker under this circumstance, right? And it will flood the dmesg.

There's a seriously buggy driver potentially corrupting memory, who cares if we start a kworker. Please don't complicate the code for extremely rare scenarios.

My points are:
1) stop the kworker because it's useless.
2) avoid flooding so many warnings and calltraces in the dmesg

The modified code is not complicated.

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

Just to state the obvious the current patch will not catch the situation when there is traffic outstanding (inflight is positive) at the time of detach from the driver. But then the inflight goes negative before the work / time kicks in.

Right, only mitigating the side effect. I will add this statement as well while keeping the code itself as-is.

What do you mean by that?! We're telling you your code is wrong.

Sorry, I misunderstood your suggestion. So the patch I replied yesterday to Mina seems acceptable?

Thanks,
Jason

author
#8 lore raw
In reply to Mina Almasry
Quoted 6 lines · Mina Almasry
Quoted 2 lines · Jason Xing

If the buggy driver causes the inflight less than 0 [1] and warns

How does a buggy driver trigger this?

We're still reproducing and investigating. With a certain version of driver + XDP installed, we have a very slight chance to see this happening.

Quoted 36 lines
Quoted 36 lines

us in page_pool_inflight(), it means we should not expect the whole page_pool to get back to work normally.

We noticed the kworker is waken up repeatedly and infinitely[1] in production. If the page pool detect the error happening, probably letting it go is a better way and do not flood the var log messages. This patch mitigates the adverse effect.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
---
 net/core/page_pool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +0 -0
index 1c6fec08bc43..8e9f5801aabb 100644
@@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
        page_pool_disable_direct_recycling(pool);
        page_pool_free_frag(pool);
 
Quoted 7 lines

net/core/page_pool.c void page_pool_destroy(struct page_pool *pool) · line 1171

Quoted 3 lines
net/core/page_pool.c +1 -1
-       if (!page_pool_release(pool))
+       if (page_pool_release(pool) <= 0)
                return;

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

Right, do you mean this patch?

net/core/page_pool.c +0 -0
index 8e9f5801aabb..7dde3bd5f275 100644
@@ -1112,7 +1112,7 @@ static void page_pool_release_retry(struct

work_struct *wq)

        int inflight;

        inflight = page_pool_release(pool);
-       if (!inflight)
+       if (inflight < 0)
                return;

It has the same behaviour as the current patch does. I thought we could stop it earlier.

Quoted 8 lines

I also wonder also whether if the check in page_pool_release() itself needs to be:

if (inflight < 0)
__page_pool_destroy();

otherwise the pool will never be destroyed no?

I'm worried this would have a more severe impact because it's uncertain if in this case the page pool can be released? :(

Thanks,
Jason

Quoted 4 lines

--
Thanks,
Mina

#9 lore raw
Quoted 71 lines · Jason Xing
Quoted 6 lines · Mina Almasry
Quoted 2 lines · Jason Xing

If the buggy driver causes the inflight less than 0 [1] and warns

How does a buggy driver trigger this?

We're still reproducing and investigating. With a certain version of driver + XDP installed, we have a very slight chance to see this happening.

Quoted 36 lines
Quoted 36 lines

us in page_pool_inflight(), it means we should not expect the whole page_pool to get back to work normally.

We noticed the kworker is waken up repeatedly and infinitely[1] in production. If the page pool detect the error happening, probably letting it go is a better way and do not flood the var log messages. This patch mitigates the adverse effect.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
---
 net/core/page_pool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +0 -0
index 1c6fec08bc43..8e9f5801aabb 100644
@@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
        page_pool_disable_direct_recycling(pool);
        page_pool_free_frag(pool);
 
Quoted 7 lines

net/core/page_pool.c void page_pool_destroy(struct page_pool *pool) · line 1171

Quoted 3 lines
net/core/page_pool.c +1 -1
-       if (!page_pool_release(pool))
+       if (page_pool_release(pool) <= 0)
                return;

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

Right, do you mean this patch?

net/core/page_pool.c +1 -1
index 8e9f5801aabb..7dde3bd5f275 100644
@@ -1112,7 +1112,7 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);
-       if (!inflight)
+       if (inflight < 0)
                return;
 
Quoted 3 lines

It has the same behaviour as the current patch does. I thought we could stop it earlier.

Yes I mean this.

Quoted 4 lines
Quoted 4 lines

I also wonder also whether if the check in page_pool_release() itself needs to be:

Quoted 8 lines
Quoted 4 lines

if (inflight < 0)
__page_pool_destroy();

otherwise the pool will never be destroyed no?

I'm worried this would have a more severe impact because it's uncertain if in this case the page pool can be released? :(

Makes sense indeed. We can't be sure if the page_pool has already been destroyed if inflight < 0. Ignore the earlier suggestion from me, thanks.

Signature · 2 lines
Thanks,
Mina
author
#10 lore raw
Quoted 73 lines · Mina Almasry
Quoted 71 lines · Jason Xing
Quoted 6 lines · Mina Almasry
Quoted 2 lines · Jason Xing

If the buggy driver causes the inflight less than 0 [1] and warns

How does a buggy driver trigger this?

We're still reproducing and investigating. With a certain version of driver + XDP installed, we have a very slight chance to see this happening.

Quoted 36 lines
Quoted 36 lines

us in page_pool_inflight(), it means we should not expect the whole page_pool to get back to work normally.

We noticed the kworker is waken up repeatedly and infinitely[1] in production. If the page pool detect the error happening, probably letting it go is a better way and do not flood the var log messages. This patch mitigates the adverse effect.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
---
 net/core/page_pool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +0 -0
index 1c6fec08bc43..8e9f5801aabb 100644
@@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
        page_pool_disable_direct_recycling(pool);
        page_pool_free_frag(pool);
 
Quoted 7 lines

net/core/page_pool.c void page_pool_destroy(struct page_pool *pool) · line 1171

Quoted 3 lines
net/core/page_pool.c +1 -1
-       if (!page_pool_release(pool))
+       if (page_pool_release(pool) <= 0)
                return;

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

Right, do you mean this patch?

net/core/page_pool.c +1 -1
index 8e9f5801aabb..7dde3bd5f275 100644
@@ -1112,7 +1112,7 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);
-       if (!inflight)
+       if (inflight < 0)
                return;
 
Quoted 5 lines
Quoted 3 lines

It has the same behaviour as the current patch does. I thought we could stop it earlier.

Yes I mean this.

I'm going to post the above diff patch in V2. Am I understanding right?

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

I also wonder also whether if the check in page_pool_release() itself needs to be:

if (inflight < 0)
__page_pool_destroy();

otherwise the pool will never be destroyed no?

Quoted 7 lines
Quoted 3 lines

I'm worried this would have a more severe impact because it's uncertain if in this case the page pool can be released? :(

Makes sense indeed. We can't be sure if the page_pool has already been destroyed if inflight < 0. Ignore the earlier suggestion from me, thanks.

Thanks for the review.

Thanks,
Jason

Quoted 4 lines

--
Thanks,
Mina

#11 lore raw
Quoted 75 lines · Jason Xing
Quoted 73 lines · Mina Almasry
Quoted 71 lines · Jason Xing
Quoted 6 lines · Mina Almasry
Quoted 2 lines · Jason Xing

If the buggy driver causes the inflight less than 0 [1] and warns

How does a buggy driver trigger this?

We're still reproducing and investigating. With a certain version of driver + XDP installed, we have a very slight chance to see this happening.

Quoted 36 lines
Quoted 36 lines

us in page_pool_inflight(), it means we should not expect the whole page_pool to get back to work normally.

We noticed the kworker is waken up repeatedly and infinitely[1] in production. If the page pool detect the error happening, probably letting it go is a better way and do not flood the var log messages. This patch mitigates the adverse effect.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
---
 net/core/page_pool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +0 -0
index 1c6fec08bc43..8e9f5801aabb 100644
@@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
        page_pool_disable_direct_recycling(pool);
        page_pool_free_frag(pool);
 
Quoted 7 lines

net/core/page_pool.c void page_pool_destroy(struct page_pool *pool) · line 1171

Quoted 3 lines
net/core/page_pool.c +1 -1
-       if (!page_pool_release(pool))
+       if (page_pool_release(pool) <= 0)
                return;

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

Right, do you mean this patch?

net/core/page_pool.c +1 -1
index 8e9f5801aabb..7dde3bd5f275 100644
@@ -1112,7 +1112,7 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);
-       if (!inflight)
+       if (inflight < 0)
                return;
 
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

It has the same behaviour as the current patch does. I thought we could stop it earlier.

Yes I mean this.

I'm going to post the above diff patch in V2. Am I understanding right?

Please also add Jakub's request, i.e. a code comment indicating why we return early.

Also, now that I look more closely, we want to make sure we get at least one warning when inflight goes negative, so, maybe something like (untested, may need some iteration):

```
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 2ea8041aba7e..6d62ea45571b 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -1113,13 +1113,12 @@ static void page_pool_release_retry(struct
work_struct *wq)
        int inflight;

        inflight = page_pool_release(pool);
-       if (!inflight)
-               return;

        /* Periodic warning for page pools the user can't see */
        netdev = READ_ONCE(pool->slow.netdev);
        if (time_after_eq(jiffies, pool->defer_warn) &&
-           (!netdev || netdev == NET_PTR_POISON)) {
+           (!netdev || netdev == NET_PTR_POISON) &&
+           inflight != 0) {
                int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;

                pr_warn("%s() stalled pool shutdown: id %u, %d
inflight %d sec devmem=%d\n",
@@ -1128,7 +1127,15 @@ static void page_pool_release_retry(struct
work_struct *wq)
                pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
        }

-       /* Still not ready to be disconnected, retry later */
+       /* In rare cases, a driver bug may cause inflight to go negative. Don't
+        * reschedule release if inflight is 0 or negative.
+        *      - If 0, the page_pool has been destroyed
+        *      - if negative, we will never recover
+        * in both cases no reschedule necessary.
+        */
+       if (inflight < 1)
+               return;
+
        schedule_delayed_work(&pool->release_dw, DEFER_TIME);
 }
```
Signature · 2 lines
Thanks,
Mina
author
#12 lore raw
Quoted 83 lines · Mina Almasry
Quoted 81 lines · Jason Xing
Quoted 78 lines · Mina Almasry
Quoted 71 lines · Jason Xing
Quoted 6 lines · Mina Almasry
Quoted 2 lines · Jason Xing

If the buggy driver causes the inflight less than 0 [1] and warns

How does a buggy driver trigger this?

We're still reproducing and investigating. With a certain version of driver + XDP installed, we have a very slight chance to see this happening.

Quoted 36 lines
Quoted 36 lines

us in page_pool_inflight(), it means we should not expect the whole page_pool to get back to work normally.

We noticed the kworker is waken up repeatedly and infinitely[1] in production. If the page pool detect the error happening, probably letting it go is a better way and do not flood the var log messages. This patch mitigates the adverse effect.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
---
 net/core/page_pool.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +0 -0
index 1c6fec08bc43..8e9f5801aabb 100644
@@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
        page_pool_disable_direct_recycling(pool);
        page_pool_free_frag(pool);
 
Quoted 7 lines

net/core/page_pool.c void page_pool_destroy(struct page_pool *pool) · line 1171

Quoted 3 lines
net/core/page_pool.c +1 -1
-       if (!page_pool_release(pool))
+       if (page_pool_release(pool) <= 0)
                return;

Isn't it the condition in page_pool_release_retry() that you want. to modify? That is the one that handles whether the worker keeps spinning no?

Right, do you mean this patch?

net/core/page_pool.c +1 -1
index 8e9f5801aabb..7dde3bd5f275 100644
@@ -1112,7 +1112,7 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);
-       if (!inflight)
+       if (inflight < 0)
                return;
 
Quoted 3 lines

It has the same behaviour as the current patch does. I thought we could stop it earlier.

Yes I mean this.

Quoted 5 lines
Quoted 2 lines

I'm going to post the above diff patch in V2. Am I understanding right?

Please also add Jakub's request, i.e. a code comment indicating why we return early.

Got it.

Quoted 4 lines

Also, now that I look more closely, we want to make sure we get at least one warning when inflight goes negative, so, maybe something like (untested, may need some iteration):

Good suggestion.

Quoted 27 lines

```

net/core/page_pool.c +2 -3
index 2ea8041aba7e..6d62ea45571b 100644
@@ -1113,13 +1113,12 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);
-       if (!inflight)
-               return;
 
        /* Periodic warning for page pools the user can't see */
        netdev = READ_ONCE(pool->slow.netdev);
        if (time_after_eq(jiffies, pool->defer_warn) &&
-           (!netdev || netdev == NET_PTR_POISON)) {
+           (!netdev || netdev == NET_PTR_POISON) &&
+           inflight != 0) {
                int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;
 

pr_warn("%s() stalled pool shutdown: id %u, %d
inflight %d sec devmem=%d\n",

net/core/page_pool.c +0 -0
@@ -1128,7 +1127,15 @@ static void page_pool_release_retry(struct
 work_struct *wq)
                pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;

net/core/page_pool.c static void page_pool_release_retry(struct · line 1137

Quoted 10 lines
net/core/page_pool.c +7 -1
        }
 
-       /* Still not ready to be disconnected, retry later */
+       /* In rare cases, a driver bug may cause inflight to go negative. Don't
+        * reschedule release if inflight is 0 or negative.
+        *      - If 0, the page_pool has been destroyed
+        *      - if negative, we will never recover
+        * in both cases no reschedule necessary.
+        */
+       if (inflight < 1)

Maybe I would change the above to 'inflight <= 0' which looks more obvious at the first glance? :)

Quoted 5 lines
net/core/page_pool.c +2 -0
+               return;
+
        schedule_delayed_work(&pool->release_dw, DEFER_TIME);
 }

```

I will test it. Thanks, Mina.

Thanks,
Jason

Quoted 4 lines

--
Thanks,
Mina

#13 lore raw
Quoted 29 lines · Jason Xing
Quoted 29 lines

```

net/core/page_pool.c +2 -3
index 2ea8041aba7e..6d62ea45571b 100644
@@ -1113,13 +1113,12 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);
-       if (!inflight)
-               return;
 
        /* Periodic warning for page pools the user can't see */
        netdev = READ_ONCE(pool->slow.netdev);
        if (time_after_eq(jiffies, pool->defer_warn) &&
-           (!netdev || netdev == NET_PTR_POISON)) {
+           (!netdev || netdev == NET_PTR_POISON) &&
+           inflight != 0) {
                int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;
 

pr_warn("%s() stalled pool shutdown: id %u, %d
inflight %d sec devmem=%d\n",

net/core/page_pool.c +0 -0
@@ -1128,7 +1127,15 @@ static void page_pool_release_retry(struct
 work_struct *wq)
                pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
        }
 
Quoted 12 lines

net/core/page_pool.c static void page_pool_release_retry(struct · line 1137

Quoted 8 lines
net/core/page_pool.c +7 -1
-       /* Still not ready to be disconnected, retry later */
+       /* In rare cases, a driver bug may cause inflight to go negative. Don't
+        * reschedule release if inflight is 0 or negative.
+        *      - If 0, the page_pool has been destroyed
+        *      - if negative, we will never recover
+        * in both cases no reschedule necessary.
+        */
+       if (inflight < 1)

Maybe I would change the above to 'inflight <= 0' which looks more obvious at the first glance? :)

Good catch. Yes.

Also, write "no reschedule is necessary" on the last line in the comment (add "is")

Signature · 2 lines
Thanks,
Mina
author
#14 lore raw
Quoted 44 lines · Mina Almasry
Quoted 29 lines · Jason Xing
Quoted 29 lines

```

net/core/page_pool.c +2 -3
index 2ea8041aba7e..6d62ea45571b 100644
@@ -1113,13 +1113,12 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);
-       if (!inflight)
-               return;
 
        /* Periodic warning for page pools the user can't see */
        netdev = READ_ONCE(pool->slow.netdev);
        if (time_after_eq(jiffies, pool->defer_warn) &&
-           (!netdev || netdev == NET_PTR_POISON)) {
+           (!netdev || netdev == NET_PTR_POISON) &&
+           inflight != 0) {
                int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;
 

pr_warn("%s() stalled pool shutdown: id %u, %d
inflight %d sec devmem=%d\n",

net/core/page_pool.c +0 -0
@@ -1128,7 +1127,15 @@ static void page_pool_release_retry(struct
 work_struct *wq)
                pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
        }
 
Quoted 12 lines

net/core/page_pool.c static void page_pool_release_retry(struct · line 1137

Quoted 8 lines
net/core/page_pool.c +7 -1
-       /* Still not ready to be disconnected, retry later */
+       /* In rare cases, a driver bug may cause inflight to go negative. Don't
+        * reschedule release if inflight is 0 or negative.
+        *      - If 0, the page_pool has been destroyed
+        *      - if negative, we will never recover
+        * in both cases no reschedule necessary.
+        */
+       if (inflight < 1)

Maybe I would change the above to 'inflight <= 0' which looks more obvious at the first glance? :)

Quoted 4 lines

Good catch. Yes.

Also, write "no reschedule is necessary" on the last line in the comment (add "is")

Will adjust it. Thanks!

Quoted 4 lines

--
Thanks,
Mina