page_pool: add a lockdep check for recycling in hardirq
Subsystems (4)
Jakub Kicinski
maintainerPATCH 1/1 page_pool: add a lockdep check for recycling in hardirq
Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using DEBUG_NET_WARN_ON_ONCE() for this, but there were concerns that people will have DEBUG_NET enabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)
---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
--- include/linux/lockdep.h | 7 +++++++ net/core/page_pool.c | 4 ++++ 2 files changed, 11 insertions(+)
include/linux/lockdep.h +7 -0
@@ -625,6 +625,12 @@ do { \WARN_ON_ONCE (__lockdep_enabled && !this_cpu_read (hardirq_context)); \ } while (0) +#define lockdep_assert_no_hardirq () \ +do { \ + WARN_ON_ONCE(__lockdep_enabled && (this_cpu_read(hardirq_context) || \ + !this_cpu_read(hardirqs_enabled))); \ +} while (0) +#define lockdep_assert_preemption_enabled () \ do { \ WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT) && \
@@ -659,6 +665,7 @@ do { \# define lockdep_assert_irqs_enabled () do { } while (0)# define lockdep_assert_irqs_disabled () do { } while (0)# define lockdep_assert_in_irq () do { } while (0) +# define lockdep_assert_no_hardirq () do { } while (0)# define lockdep_assert_preemption_enabled () do { } while (0)# define lockdep_assert_preemption_disabled () do { } while (0)
net/core/page_pool.c +4 -0
@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)static bool page_pool_recycle_in_ring (struct page_pool *pool,struct page *page) {int ret; + +lockdep_assert_no_hardirq ();/* BH protection not needed if current is softirq */ if (in_softirq ()) ret =ptr_ring_produce (&pool->ring, page);
@@ -642,6 +644,8 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data,int i, bulk_len =0 ;bool in_softirq; +lockdep_assert_no_hardirq (); +for (i =0 ; i < count; i++) {struct page *page =virt_to_head_page (data[i]);
Signature · 1 line
2.41.0
On 2023/7/21 1:37, Jakub Kicinski wrote:
...
net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 540
Quoted 10 lines
net/core/page_pool.c +2
-0@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)static bool page_pool_recycle_in_ring (struct page_pool *pool,struct page *page) {int ret; + +lockdep_assert_no_hardirq ();
Is there any reason not to put it in page_pool_put_defragged_page() to catch the case with allow_direct being true when page_pool_recycle_in_ring() may not be called?
Quoted 13 lines
net/core/page_pool.c +2
-0/* BH protection not needed if current is softirq */ if (in_softirq ()) ret =ptr_ring_produce (&pool->ring, page);@@ -642,6 +644,8 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data,int i, bulk_len =0 ;bool in_softirq; +lockdep_assert_no_hardirq (); +for (i =0 ; i < count; i++) {struct page *page =virt_to_head_page (data[i]);
Jakub Kicinski
maintainerQuoted 6 lines · On Fri, 21 Jul 2023 19:53:30 +0800 Yunsheng Lin wrote:
Quoted 6 lines
net/core/page_pool.c +0
-0@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)static bool page_pool_recycle_in_ring (struct page_pool *pool,struct page *page)
Quoted 8 lines
net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 540
Quoted 4 lines
net/core/page_pool.c +2
-0{int ret; + +lockdep_assert_no_hardirq ();Is there any reason not to put it in
page_pool_put_defragged_page()to catch the case withallow_directbeing true whenpage_pool_recycle_in_ring()may not be called?
I was trying to stick it into places which make an assumption about the calling context, rather than cover the full API.
I don't have a strong preference either way, but I hope it's good enough. The benefit I see is that it should be fairly obvious to a seasoned kernel code reader why this warning is here. A warning that fires from page_pool_put_defragged_page() would need a comment to explain the reason and may go stale.
Quoted 2 lines
Quoted 2 lines
net/core/page_pool.c +0
-0/* BH protection not needed if current is softirq */ if (in_softirq ())
From: Jakub Kicinski <[email protected]>
Date: Thu, 20 Jul 2023 10:37:51 -0700
Quoted 6 lines
Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using
DEBUG_NET_WARN_ON_ONCE()for this, but there were concerns that people will haveDEBUG_NETenabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)
+1 patch to add to my tree to base my current series on... Time to create separate repo named "page-pool-next"? :D
Quoted 45 lines
Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]--- include/linux/lockdep.h | 7 +++++++ net/core/page_pool.c | 4 ++++ 2 files changed, 11 insertions(+)include/linux/lockdep.h +7
-0@@ -625,6 +625,12 @@ do { \WARN_ON_ONCE (__lockdep_enabled && !this_cpu_read (hardirq_context)); \ } while (0) +#define lockdep_assert_no_hardirq () \ +do { \ + WARN_ON_ONCE(__lockdep_enabled && (this_cpu_read(hardirq_context) || \ + !this_cpu_read(hardirqs_enabled))); \ +} while (0) +#define lockdep_assert_preemption_enabled () \ do { \ WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT) && \@@ -659,6 +665,7 @@ do { \# define lockdep_assert_irqs_enabled () do { } while (0)# define lockdep_assert_irqs_disabled () do { } while (0)# define lockdep_assert_in_irq () do { } while (0) +# define lockdep_assert_no_hardirq () do { } while (0)# define lockdep_assert_preemption_enabled () do { } while (0)# define lockdep_assert_preemption_disabled () do { } while (0)net/core/page_pool.c +0
-0
net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 536
Quoted 1 line
net/core/page_pool.c +0
-0@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)static bool page_pool_recycle_in_ring (struct page_pool *pool,struct page *page)
Crap can happen earlier. Imagine that some weird code asked for direct recycling with IRQs disabled. Then, we can hit __page_pool_put_page:page_pool_recycle_in_cache and who knows what can happen.
Can't we add this assertion right to the beginning of __page_pool_put_page()? It's reasonable enough, at least for me, and wouldn't require any commentary splats. Unlike put_defragged_page() as Yunsheng proposes :p
Other than that (which is debatable), looks fine to me.
Quoted 8 lines
net/core/page_pool.c +2
-0{int ret; + +lockdep_assert_no_hardirq ();/* BH protection not needed if current is softirq */ if (in_softirq ()) ret =ptr_ring_produce (&pool->ring, page);
net/core/page_pool.c void page_pool_put_page_bulk(struct page_pool *pool, void **data, · line 651
Quoted 8 lines
net/core/page_pool.c +2
-0@@ -642,6 +644,8 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data,int i, bulk_len =0 ;bool in_softirq; +lockdep_assert_no_hardirq (); +for (i =0 ; i < count; i++) {struct page *page =virt_to_head_page (data[i]);
Thanks,
Olek
Jakub Kicinski
maintainerQuoted 9 lines · On Fri, 21 Jul 2023 17:48:25 +0200 Alexander Lobakin wrote:
Quoted 6 lines
Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using
DEBUG_NET_WARN_ON_ONCE()for this, but there were concerns that people will haveDEBUG_NETenabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)+1 patch to add to my tree to base my current series on... Time to create separate repo named "page-pool-next"? :D
You joke but I've been scheming how to expose the page pool stats via the netdev netlink family, which would be another conflict to be added to the pile :D When it rains it pours.
You should probably start sending uncontroversial stuff out even if it doesn't have in-tree users yet.
Quoted 18 lines
net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 536
Quoted 8 lines
# define lockdep_assert_preemption_enabled() do { } while (0) # define lockdep_assert_preemption_disabled() do { } while (0)net/core/page_pool.c +0
-0@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)static bool page_pool_recycle_in_ring (struct page_pool *pool,struct page *page)Crap can happen earlier. Imagine that some weird code asked for direct recycling with IRQs disabled. Then, we can hit __page_pool_put_page:page_pool_recycle_in_cache and who knows what can happen.
Can't we add this assertion right to the beginning of__page_pool_put_page()? It's reasonable enough, at least for me, and wouldn't require any commentary splats. Unlikeput_defragged_page()as Yunsheng proposes :p
Quoted 1 line
Other than that (which is debatable), looks fine to me.
No strong preference. Would you mind taking over this one? It'd also benefit from testing that the lockdep warning actually fires as expected, I just tested that it doesn't false positive TBH :)
From: Jakub Kicinski <[email protected]>
Date: Fri, 21 Jul 2023 09:05:38 -0700
Quoted 36 lines · On Fri, 21 Jul 2023 17:48:25 +0200 Alexander Lobakin wrote:
Quoted 9 lines
Quoted 6 lines
Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using
DEBUG_NET_WARN_ON_ONCE()for this, but there were concerns that people will haveDEBUG_NETenabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :)+1 patch to add to my tree to base my current series on... Time to create separate repo named "page-pool-next"? :D
You joke but I've been scheming how to expose the page pool stats via the netdev netlink family, which would be another conflict to be added to the pile :D When it rains it pours.
You should probably start sending uncontroversial stuff out even if it doesn't have in-tree users yet.
Quoted 18 lines
net/core/page_pool.c static void page_pool_return_page(struct page_pool *pool, struct page *page) · line 536
Quoted 8 lines
# define lockdep_assert_preemption_enabled() do { } while (0) # define lockdep_assert_preemption_disabled() do { } while (0)net/core/page_pool.c +0
-0@@ -536,6 +536,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)static bool page_pool_recycle_in_ring (struct page_pool *pool,struct page *page)Crap can happen earlier. Imagine that some weird code asked for direct recycling with IRQs disabled. Then, we can hit __page_pool_put_page:page_pool_recycle_in_cache and who knows what can happen.
Can't we add this assertion right to the beginning of__page_pool_put_page()? It's reasonable enough, at least for me, and wouldn't require any commentary splats. Unlikeput_defragged_page()as Yunsheng proposes :p
Quoted 5 lines
Quoted 1 line
Other than that (which is debatable), looks fine to me.
No strong preference. Would you mind taking over this one? It'd also benefit from testing that the lockdep warning actually fires as expected, I just tested that it doesn't false positive TBH :)
Sure! I'll add it to the optimization series as a pre-req to more aggressive direct recycling, would that be fine?
Other than that, it's mostly Yunsheng's 2 submissions (PP header file split and hybrid allocation) I'm basing both of my series on. The optimization series can go without the allocation just fine, but as for the header split, I'd like it to go first, it simplifies things :D
Thanks,
Olek
Jakub Kicinski
maintainerQuoted 6 lines · On Fri, 21 Jul 2023 18:33:39 +0200 Alexander Lobakin wrote:
Quoted 3 lines
No strong preference. Would you mind taking over this one? It'd also benefit from testing that the lockdep warning actually fires as expected, I just tested that it doesn't false positive TBH :)
Sure! I'll add it to the optimization series as a pre-req to more aggressive direct recycling, would that be fine?
SG!
Signature · 1 line
pw-bot: cr