Skip to contentSkip to conversation

page_pool: add a lockdep check for recycling in hardirq

PATCH In review Jakub Kicinski 3 participants 7 messages Jul 20, 2023 – Jul 22, 2023

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH 1/1 page_pool: add a lockdep check for recycling in hardirq

Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using DEBUG_NET_WARN_ON_ONCE() for this, but there were concerns that people will have DEBUG_NET enabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 include/linux/lockdep.h | 7 +++++++
 net/core/page_pool.c    | 4 ++++
 2 files changed, 11 insertions(+)
include/linux/lockdep.h +7 -0
index 310f85903c91..dc2844b071c2 100644
@@ -625,6 +625,12 @@ do {									\
 	WARN_ON_ONCE(__lockdep_enabled && !this_cpu_read(hardirq_context)); \
 } while (0)
 
+#define lockdep_assert_no_hardirq()					\
+do {									\
+	WARN_ON_ONCE(__lockdep_enabled && (this_cpu_read(hardirq_context) || \
+					   !this_cpu_read(hardirqs_enabled))); \
+} while (0)
+
 #define lockdep_assert_preemption_enabled()				\
 do {									\
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT)	&&		\
@@ -659,6 +665,7 @@ do {									\
 # define lockdep_assert_irqs_enabled() do { } while (0)
 # define lockdep_assert_irqs_disabled() do { } while (0)
 # define lockdep_assert_in_irq() do { } while (0)
+# define lockdep_assert_no_hardirq() do { } while (0)
 
 # define lockdep_assert_preemption_enabled() do { } while (0)
 # define lockdep_assert_preemption_disabled() do { } while (0)
net/core/page_pool.c +4 -0
index a3e12a61d456..3ac760fcdc22 100644
@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 static bool page_pool_recycle_in_ring(struct page_pool *pool, struct page *page)
 {
 	int ret;
+
+	lockdep_assert_no_hardirq();
 	/* BH protection not needed if current is softirq */
 	if (in_softirq())
 		ret = ptr_ring_produce(&pool->ring, page);
@@ -642,6 +644,8 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data,
 	int i, bulk_len = 0;
 	bool in_softirq;
 
+	lockdep_assert_no_hardirq();
+
 	for (i = 0; i < count; i++) {
 		struct page *page = virt_to_head_page(data[i]);
 
Signature · 1 line
2.41.0
#1 lore raw

On 2023/7/21 1:37, Jakub Kicinski wrote:

...

net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 540

Quoted 10 lines
net/core/page_pool.c +2 -0
index a3e12a61d456..3ac760fcdc22 100644
@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 static bool page_pool_recycle_in_ring(struct page_pool *pool, struct page *page)
 {
 	int ret;
+
+	lockdep_assert_no_hardirq();

Is there any reason not to put it in page_pool_put_defragged_page() to catch the case with allow_direct being true when page_pool_recycle_in_ring() may not be called?

Quoted 13 lines
net/core/page_pool.c +2 -0
 	/* BH protection not needed if current is softirq */
 	if (in_softirq())
 		ret = ptr_ring_produce(&pool->ring, page);
@@ -642,6 +644,8 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data,
 	int i, bulk_len = 0;
 	bool in_softirq;
 
+	lockdep_assert_no_hardirq();
+
 	for (i = 0; i < count; i++) {
 		struct page *page = virt_to_head_page(data[i]);
 
author maintainer
#2 lore raw
Quoted 6 lines · On Fri, 21 Jul 2023 19:53:30 +0800 Yunsheng Lin wrote:
Quoted 6 lines
net/core/page_pool.c +0 -0
index a3e12a61d456..3ac760fcdc22 100644
@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 static bool page_pool_recycle_in_ring(struct page_pool *pool, struct page *page)
Quoted 8 lines

net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 540

Quoted 4 lines
net/core/page_pool.c +2 -0
 {
 	int ret;
+
+	lockdep_assert_no_hardirq();  

Is there any reason not to put it in page_pool_put_defragged_page() to catch the case with allow_direct being true when page_pool_recycle_in_ring() may not be called?

I was trying to stick it into places which make an assumption about the calling context, rather than cover the full API.
I don't have a strong preference either way, but I hope it's good enough. The benefit I see is that it should be fairly obvious to a seasoned kernel code reader why this warning is here. A warning that fires from page_pool_put_defragged_page() would need a comment to explain the reason and may go stale.

Quoted 2 lines
Quoted 2 lines
net/core/page_pool.c +0 -0
 	/* BH protection not needed if current is softirq */
 	if (in_softirq())
#3 lore raw
In reply to Jakub Kicinski

From: Jakub Kicinski <[email protected]>
Date: Thu, 20 Jul 2023 10:37:51 -0700

Quoted 6 lines

Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using DEBUG_NET_WARN_ON_ONCE() for this, but there were concerns that people will have DEBUG_NET enabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)

+1 patch to add to my tree to base my current series on... Time to create separate repo named "page-pool-next"? :D

Quoted 45 lines

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 include/linux/lockdep.h | 7 +++++++
 net/core/page_pool.c    | 4 ++++
 2 files changed, 11 insertions(+)
include/linux/lockdep.h +7 -0
index 310f85903c91..dc2844b071c2 100644
@@ -625,6 +625,12 @@ do {									\
 	WARN_ON_ONCE(__lockdep_enabled && !this_cpu_read(hardirq_context)); \
 } while (0)
 
+#define lockdep_assert_no_hardirq()					\
+do {									\
+	WARN_ON_ONCE(__lockdep_enabled && (this_cpu_read(hardirq_context) || \
+					   !this_cpu_read(hardirqs_enabled))); \
+} while (0)
+
 #define lockdep_assert_preemption_enabled()				\
 do {									\
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT)	&&		\
@@ -659,6 +665,7 @@ do {									\
 # define lockdep_assert_irqs_enabled() do { } while (0)
 # define lockdep_assert_irqs_disabled() do { } while (0)
 # define lockdep_assert_in_irq() do { } while (0)
+# define lockdep_assert_no_hardirq() do { } while (0)
 
 # define lockdep_assert_preemption_enabled() do { } while (0)
 # define lockdep_assert_preemption_disabled() do { } while (0)
net/core/page_pool.c +0 -0
index a3e12a61d456..3ac760fcdc22 100644

net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 536

Quoted 1 line
net/core/page_pool.c +0 -0
@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 static bool page_pool_recycle_in_ring(struct page_pool *pool, struct page *page)

Crap can happen earlier. Imagine that some weird code asked for direct recycling with IRQs disabled. Then, we can hit __page_pool_put_page:page_pool_recycle_in_cache and who knows what can happen.
Can't we add this assertion right to the beginning of __page_pool_put_page()? It's reasonable enough, at least for me, and wouldn't require any commentary splats. Unlike put_defragged_page() as Yunsheng proposes :p

Other than that (which is debatable), looks fine to me.

Quoted 8 lines
net/core/page_pool.c +2 -0
 {
 	int ret;
+
+	lockdep_assert_no_hardirq();
 	/* BH protection not needed if current is softirq */
 	if (in_softirq())
 		ret = ptr_ring_produce(&pool->ring, page);

net/core/page_pool.c void page_pool_put_page_bulk(struct page_pool *pool, void **data, · line 651

Quoted 8 lines
net/core/page_pool.c +2 -0
@@ -642,6 +644,8 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data,
 	int i, bulk_len = 0;
 	bool in_softirq;
 
+	lockdep_assert_no_hardirq();
+
 	for (i = 0; i < count; i++) {
 		struct page *page = virt_to_head_page(data[i]);
 

Thanks,
Olek

author maintainer
#4 lore raw
Quoted 9 lines · On Fri, 21 Jul 2023 17:48:25 +0200 Alexander Lobakin wrote:
Quoted 6 lines

Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using DEBUG_NET_WARN_ON_ONCE() for this, but there were concerns that people will have DEBUG_NET enabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)

+1 patch to add to my tree to base my current series on... Time to create separate repo named "page-pool-next"? :D

You joke but I've been scheming how to expose the page pool stats via the netdev netlink family, which would be another conflict to be added to the pile :D When it rains it pours.

You should probably start sending uncontroversial stuff out even if it doesn't have in-tree users yet.

Quoted 18 lines

net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 536

Quoted 8 lines
 # define lockdep_assert_preemption_enabled() do { } while (0)
 # define lockdep_assert_preemption_disabled() do { } while (0)
net/core/page_pool.c +0 -0
index a3e12a61d456..3ac760fcdc22 100644
@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 static bool page_pool_recycle_in_ring(struct page_pool *pool, struct page *page)  

Crap can happen earlier. Imagine that some weird code asked for direct recycling with IRQs disabled. Then, we can hit __page_pool_put_page:page_pool_recycle_in_cache and who knows what can happen.
Can't we add this assertion right to the beginning of __page_pool_put_page()? It's reasonable enough, at least for me, and wouldn't require any commentary splats. Unlike put_defragged_page() as Yunsheng proposes :p

Quoted 1 line

Other than that (which is debatable), looks fine to me.

No strong preference. Would you mind taking over this one? It'd also benefit from testing that the lockdep warning actually fires as expected, I just tested that it doesn't false positive TBH :)

#5 lore raw

From: Jakub Kicinski <[email protected]>
Date: Fri, 21 Jul 2023 09:05:38 -0700

Quoted 36 lines · On Fri, 21 Jul 2023 17:48:25 +0200 Alexander Lobakin wrote:
Quoted 9 lines
Quoted 6 lines

Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using DEBUG_NET_WARN_ON_ONCE() for this, but there were concerns that people will have DEBUG_NET enabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)

+1 patch to add to my tree to base my current series on... Time to create separate repo named "page-pool-next"? :D

You joke but I've been scheming how to expose the page pool stats via the netdev netlink family, which would be another conflict to be added to the pile :D When it rains it pours.

You should probably start sending uncontroversial stuff out even if it doesn't have in-tree users yet.

Quoted 18 lines

net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 536

Quoted 8 lines
 # define lockdep_assert_preemption_enabled() do { } while (0)
 # define lockdep_assert_preemption_disabled() do { } while (0)
net/core/page_pool.c +0 -0
index a3e12a61d456..3ac760fcdc22 100644
@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 static bool page_pool_recycle_in_ring(struct page_pool *pool, struct page *page)  

Crap can happen earlier. Imagine that some weird code asked for direct recycling with IRQs disabled. Then, we can hit __page_pool_put_page:page_pool_recycle_in_cache and who knows what can happen.
Can't we add this assertion right to the beginning of __page_pool_put_page()? It's reasonable enough, at least for me, and wouldn't require any commentary splats. Unlike put_defragged_page() as Yunsheng proposes :p

Quoted 5 lines
Quoted 1 line

Other than that (which is debatable), looks fine to me.

No strong preference. Would you mind taking over this one? It'd also benefit from testing that the lockdep warning actually fires as expected, I just tested that it doesn't false positive TBH :)

Sure! I'll add it to the optimization series as a pre-req to more aggressive direct recycling, would that be fine?

Other than that, it's mostly Yunsheng's 2 submissions (PP header file split and hybrid allocation) I'm basing both of my series on. The optimization series can go without the allocation just fine, but as for the header split, I'd like it to go first, it simplifies things :D

Thanks,
Olek

author maintainer
#6 lore raw
Quoted 6 lines · On Fri, 21 Jul 2023 18:33:39 +0200 Alexander Lobakin wrote:
Quoted 3 lines

No strong preference. Would you mind taking over this one? It'd also benefit from testing that the lockdep warning actually fires as expected, I just tested that it doesn't false positive TBH :)

Sure! I'll add it to the optimization series as a pre-req to more aggressive direct recycling, would that be fine?

SG!

Signature · 1 line
pw-bot: cr