Skip to contentSkip to conversation

page_pool: avoid infinite loop to schedule delayed worker

PATCH Superseded Jason Xing 3 participants 7 messages Feb 13, 2025 – Feb 14, 2025

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jason Xing

author
#0 lore raw

PATCH v2 1/1 page_pool: avoid infinite loop to schedule delayed worker

We noticed the kworker in page_pool_release_retry() was waken up repeatedly and infinitely in production because of the buggy driver causing the inflight less than 0 and warning us in page_pool_inflight()[1].

Since the inflight value goes negative, it means we should not expect the whole page_pool to get back to work normally.

This patch mitigates the adverse effect by not rescheduling the kworker when detecting the inflight negative in page_pool_release_retry().

This patch allows to pr_warn() once in page_pool_release_retry() in this case.

[1]

[Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
[Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
...
[Mon Feb 10 20:36:11 2025] Call Trace:
[Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
[Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
[Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
[Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
[Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
[Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
[Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
[Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---

Note: before this patch, the above calltrace would flood the dmesg due to repeated reschedule of release_dw kworker.

---
v2

1. add more details in commit message.
2. allow printing once when the inflight is negative.
3. correct the position where to stop the reschedule. Suggested by Mina and Jakub.

---
 net/core/page_pool.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
net/core/page_pool.c +11 -4
index 1c6fec08bc43..e1f89a19a6b6 100644
@@ -1112,13 +1112,12 @@ static void page_pool_release_retry(struct work_struct *wq)
 	int inflight;
 
 	inflight = page_pool_release(pool);
-	if (!inflight)
-		return;
 
 	/* Periodic warning for page pools the user can't see */
 	netdev = READ_ONCE(pool->slow.netdev);
 	if (time_after_eq(jiffies, pool->defer_warn) &&
-	    (!netdev || netdev == NET_PTR_POISON)) {
+	    (!netdev || netdev == NET_PTR_POISON) &&
+	    inflight != 0) {
 		int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;
 
 		pr_warn("%s() stalled pool shutdown: id %u, %d inflight %d sec\n",
@@ -1126,7 +1125,15 @@ static void page_pool_release_retry(struct work_struct *wq)
 		pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
 	}
 
-	/* Still not ready to be disconnected, retry later */
+	/* In rare cases, a driver bug may cause inflight to go negative.
+	 * Don't reschedule release if inflight is 0 or negative.
+	 * - If 0, the page_pool has been destroyed
+	 * - if negative, we will never recover
+	 *   in both cases no reschedule is necessary.
+	 */
+	if (inflight <= 0)
+		return;
+
 	schedule_delayed_work(&pool->release_dw, DEFER_TIME);
 }
 
Signature · 1 line
2.43.5
maintainer
#1 lore raw

net/core/page_pool.c static void page_pool_release_retry(struct work_struct *wq) · line 1117

Quoted 13 lines · Jason Xing
net/core/page_pool.c +0 -2
index 1c6fec08bc43..e1f89a19a6b6 100644
@@ -1112,13 +1112,12 @@ static void page_pool_release_retry(struct work_struct *wq)
 	int inflight;
 
 	inflight = page_pool_release(pool);
-	if (!inflight)
-		return;
 
 	/* Periodic warning for page pools the user can't see */
 	netdev = READ_ONCE(pool->slow.netdev);

This causes UaF, as catched by the CI:

https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/990441/34-udpgro-bench-sh/stderr

at this point 'inflight' could be 0 and 'pool' already freed.

/P

author
#2 lore raw
Quoted 13 lines · Paolo Abeni
Quoted 11 lines · Jason Xing
net/core/page_pool.c +0 -2
index 1c6fec08bc43..e1f89a19a6b6 100644
@@ -1112,13 +1112,12 @@ static void page_pool_release_retry(struct work_struct *wq)
      int inflight;
 
      inflight = page_pool_release(pool);
-     if (!inflight)
-             return;
 
Quoted 8 lines

net/core/page_pool.c static void page_pool_release_retry(struct work_struct *wq) · line 1117

Quoted 2 lines
net/core/page_pool.c +0 -0
      /* Periodic warning for page pools the user can't see */
      netdev = READ_ONCE(pool->slow.netdev);

This causes UaF, as catched by the CI:

https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/990441/34-udpgro-bench-sh/stderr

at this point 'inflight' could be 0 and 'pool' already freed.

Oh, right, thanks for catching that.

I'm going to use the previous approach (one-liner with a few comments):

net/core/page_pool.c +0 -0
index 1c6fec08bc43..209b5028abd7 100644
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct

work_struct *wq)

        int inflight;

        inflight = page_pool_release(pool);
-       if (!inflight)
+       /* In rare cases, a driver bug may cause inflight to go negative.
+        * Don't reschedule release if inflight is 0 or negative.
+        * - If 0, the page_pool has been destroyed
+        * - if negative, we will never recover
+        *   in both cases no reschedule is necessary.
+        */
+       if (inflight <= 0)
                return;

Thanks,
Jason

#3 lore raw
Quoted 36 lines · Jason Xing
Quoted 13 lines · Paolo Abeni
Quoted 11 lines · Jason Xing
net/core/page_pool.c +0 -2
index 1c6fec08bc43..e1f89a19a6b6 100644
@@ -1112,13 +1112,12 @@ static void page_pool_release_retry(struct work_struct *wq)
      int inflight;
 
      inflight = page_pool_release(pool);
-     if (!inflight)
-             return;
 
Quoted 8 lines

net/core/page_pool.c static void page_pool_release_retry(struct work_struct *wq) · line 1117

Quoted 2 lines
net/core/page_pool.c +0 -0
      /* Periodic warning for page pools the user can't see */
      netdev = READ_ONCE(pool->slow.netdev);

This causes UaF, as catched by the CI:

https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/990441/34-udpgro-bench-sh/stderr

at this point 'inflight' could be 0 and 'pool' already freed.

Oh, right, thanks for catching that.

I'm going to use the previous approach (one-liner with a few comments):

net/core/page_pool.c +0 -0
index 1c6fec08bc43..209b5028abd7 100644
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);

net/core/page_pool.c static void page_pool_release_retry(struct · line 1124

Quoted 10 lines
net/core/page_pool.c +7 -1
-       if (!inflight)
+       /* In rare cases, a driver bug may cause inflight to go negative.
+        * Don't reschedule release if inflight is 0 or negative.
+        * - If 0, the page_pool has been destroyed
+        * - if negative, we will never recover
+        *   in both cases no reschedule is necessary.
+        */
+       if (inflight <= 0)
                return;
 

I think it could still be good to have us warn once so that this bug is not silent.

We can return early if page_pool_release(pool) == 0, and then only schedule_delayed_work() after the warning if inflight is positive.

Signature · 2 lines
Thanks,
Mina
author
#4 lore raw
Quoted 37 lines · Mina Almasry
Quoted 35 lines · Jason Xing
Quoted 13 lines · Paolo Abeni
Quoted 11 lines · Jason Xing
net/core/page_pool.c +0 -2
index 1c6fec08bc43..e1f89a19a6b6 100644
@@ -1112,13 +1112,12 @@ static void page_pool_release_retry(struct work_struct *wq)
      int inflight;
 
      inflight = page_pool_release(pool);
-     if (!inflight)
-             return;
 
Quoted 8 lines

net/core/page_pool.c static void page_pool_release_retry(struct work_struct *wq) · line 1117

Quoted 2 lines
net/core/page_pool.c +0 -0
      /* Periodic warning for page pools the user can't see */
      netdev = READ_ONCE(pool->slow.netdev);

This causes UaF, as catched by the CI:

https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/990441/34-udpgro-bench-sh/stderr

at this point 'inflight' could be 0 and 'pool' already freed.

Oh, right, thanks for catching that.

I'm going to use the previous approach (one-liner with a few comments):

net/core/page_pool.c +0 -0
index 1c6fec08bc43..209b5028abd7 100644
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
Quoted 14 lines

net/core/page_pool.c static void page_pool_release_retry(struct · line 1124

Quoted 11 lines
net/core/page_pool.c +7 -1
        inflight = page_pool_release(pool);
-       if (!inflight)
+       /* In rare cases, a driver bug may cause inflight to go negative.
+        * Don't reschedule release if inflight is 0 or negative.
+        * - If 0, the page_pool has been destroyed
+        * - if negative, we will never recover
+        *   in both cases no reschedule is necessary.
+        */
+       if (inflight <= 0)
                return;
 

I think it could still be good to have us warn once so that this bug is not silent.

Allow me to double-check what you meant here. Applying the above patch, we do at least see the warning once in page_pool_release_retry()->page_pool_release()->page_pool_inflight()->WARN() before stopping the reschedule.

Do you expect to see another warning, namely, pr_warn() in page_pool_release_retry()? If so, I assume you expect to only print out the pool->user.id?

Quoted 3 lines

We can return early if page_pool_release(pool) == 0, and then only schedule_delayed_work() after the warning if inflight is positive.

Based on the above analysis, can we adjust in this way:

net/core/page_pool.c +0 -0
index 1c6fec08bc43..e1831cc23d9c 100644
@@ -625,8 +625,8 @@ s32 page_pool_inflight(const struct page_pool

*pool, bool strict)

        if (strict) {
                trace_page_pool_release(pool, inflight, hold_cnt, release_cnt);

- WARN(inflight < 0, "Negative(%d) inflight packet-pages",
- inflight);
+ WARN(inflight < 0, "Pool id(%u): negative(%d) inflight
packet-pages",

+                    pool->user.id, inflight);
        } else {
                inflight = max(0, inflight);
        }
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct
work_struct *wq)
        int inflight;

        inflight = page_pool_release(pool);
-       if (!inflight)
+       /* In rare cases, a driver bug may cause inflight to go negative.
+        * Don't reschedule release if inflight is 0 or negative.
+        * - If 0, the page_pool has been destroyed
+        * - if negative, we will never recover
+        *   in both cases no reschedule is necessary.
+        */
+       if (inflight <= 0)
                return;

Thanks,
Jason

#5 lore raw
Quoted 54 lines · Jason Xing
Quoted 37 lines · Mina Almasry
Quoted 35 lines · Jason Xing
Quoted 13 lines · Paolo Abeni
Quoted 11 lines · Jason Xing
net/core/page_pool.c +0 -2
index 1c6fec08bc43..e1f89a19a6b6 100644
@@ -1112,13 +1112,12 @@ static void page_pool_release_retry(struct work_struct *wq)
      int inflight;
 
      inflight = page_pool_release(pool);
-     if (!inflight)
-             return;
 
Quoted 8 lines

net/core/page_pool.c static void page_pool_release_retry(struct work_struct *wq) · line 1117

Quoted 2 lines
net/core/page_pool.c +0 -0
      /* Periodic warning for page pools the user can't see */
      netdev = READ_ONCE(pool->slow.netdev);

This causes UaF, as catched by the CI:

https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/990441/34-udpgro-bench-sh/stderr

at this point 'inflight' could be 0 and 'pool' already freed.

Oh, right, thanks for catching that.

I'm going to use the previous approach (one-liner with a few comments):

net/core/page_pool.c +0 -0
index 1c6fec08bc43..209b5028abd7 100644
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
Quoted 14 lines

net/core/page_pool.c static void page_pool_release_retry(struct · line 1124

Quoted 11 lines
net/core/page_pool.c +7 -1
        inflight = page_pool_release(pool);
-       if (!inflight)
+       /* In rare cases, a driver bug may cause inflight to go negative.
+        * Don't reschedule release if inflight is 0 or negative.
+        * - If 0, the page_pool has been destroyed
+        * - if negative, we will never recover
+        *   in both cases no reschedule is necessary.
+        */
+       if (inflight <= 0)
                return;
 

I think it could still be good to have us warn once so that this bug is not silent.

Quoted 5 lines

Allow me to double-check what you meant here. Applying the above patch, we do at least see the warning once in page_pool_release_retry()->page_pool_release()->page_pool_inflight()->WARN() before stopping the reschedule.

Ah, I see. I had missed indeed that we warn in page_pool_inflight() if we see anything negative there anyway. Nevermind then. Thanks!

author
#6 lore raw
Quoted 56 lines · Mina Almasry
Quoted 54 lines · Jason Xing
Quoted 51 lines · Mina Almasry
Quoted 36 lines · Jason Xing
Quoted 13 lines · Paolo Abeni
Quoted 11 lines · Jason Xing
net/core/page_pool.c +0 -2
index 1c6fec08bc43..e1f89a19a6b6 100644
@@ -1112,13 +1112,12 @@ static void page_pool_release_retry(struct work_struct *wq)
      int inflight;
 
      inflight = page_pool_release(pool);
-     if (!inflight)
-             return;
 
Quoted 8 lines

net/core/page_pool.c static void page_pool_release_retry(struct work_struct *wq) · line 1117

Quoted 2 lines
net/core/page_pool.c +0 -0
      /* Periodic warning for page pools the user can't see */
      netdev = READ_ONCE(pool->slow.netdev);

This causes UaF, as catched by the CI:

https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/990441/34-udpgro-bench-sh/stderr

at this point 'inflight' could be 0 and 'pool' already freed.

Oh, right, thanks for catching that.

I'm going to use the previous approach (one-liner with a few comments):

net/core/page_pool.c +0 -0
index 1c6fec08bc43..209b5028abd7 100644
@@ -1112,7 +1112,13 @@ static void page_pool_release_retry(struct
 work_struct *wq)
        int inflight;
 
        inflight = page_pool_release(pool);

net/core/page_pool.c static void page_pool_release_retry(struct · line 1124

Quoted 10 lines
net/core/page_pool.c +7 -1
-       if (!inflight)
+       /* In rare cases, a driver bug may cause inflight to go negative.
+        * Don't reschedule release if inflight is 0 or negative.
+        * - If 0, the page_pool has been destroyed
+        * - if negative, we will never recover
+        *   in both cases no reschedule is necessary.
+        */
+       if (inflight <= 0)
                return;
 

I think it could still be good to have us warn once so that this bug is not silent.

Quoted 8 lines
Quoted 5 lines

Allow me to double-check what you meant here. Applying the above patch, we do at least see the warning once in page_pool_release_retry()->page_pool_release()->page_pool_inflight()->WARN() before stopping the reschedule.

Ah, I see. I had missed indeed that we warn in page_pool_inflight() if we see anything negative there anyway. Nevermind then. Thanks!

Okay, then a one-liner patch is enough :)

Thanks for the review.