Skip to contentSkip to conversation

page_pool: use irqsave/irqrestore to protect ring access.

PATCH In review Jonathan Lemon 5 participants 10 messages Mar 9, 2020 – Mar 10, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 page_pool: use irqsave/irqrestore to protect ring access.

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161 __local_bh_enable_ip+0x35/0x50

    __page_pool_finish_recycle+0x14f/0x180
    mlx4_en_recycle_tx_desc+0x44/0x50
    mlx4_en_process_tx_cq+0x19f/0x440
    mlx4_en_poll_rx_cq+0xd4/0xf0
    netpoll_poll_dev+0xc2/0x190
    netpoll_send_skb_on_dev+0xf5/0x230
    netpoll_send_udp+0x2b3/0x3cd
    write_ext_msg+0x1be/0x1d0
    console_unlock+0x22e/0x500
    vprintk_emit+0x23a/0x360
    printk+0x48/0x4a
    hpet_rtc_interrupt.cold.17+0xe/0x1a
    __handle_irq_event_percpu+0x43/0x180
    handle_irq_event_percpu+0x20/0x60
    handle_irq_event+0x2a/0x47
    handle_edge_irq+0x8e/0x190
    handle_irq+0xbf/0x100
    do_IRQ+0x41/0xc0
    common_interrupt+0xf/0xf
    </IRQ>
---
 net/core/page_pool.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
net/core/page_pool.c +6 -8
index 626db912fce4..df9804e85a40 100644
@@ -102,6 +102,7 @@ noinline
 static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)
 {
 	struct ptr_ring *r = &pool->ring;
+	unsigned long flags;
 	struct page *page;
 	int pref_nid; /* preferred NUMA node */
 
@@ -120,7 +121,7 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)
 #endif
 
 	/* Slower-path: Get pages from locked ring queue */
-	spin_lock(&r->consumer_lock);
+	spin_lock_irqsave(&r->consumer_lock, flags);
 
 	/* Refill alloc array, but only if NUMA match */
 	do {
@@ -146,7 +147,7 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)
 	if (likely(pool->alloc.count > 0))
 		page = pool->alloc.cache[--pool->alloc.count];
 
-	spin_unlock(&r->consumer_lock);
+	spin_unlock_irqrestore(&r->consumer_lock, flags);
 	return page;
 }
 
@@ -321,11 +322,8 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page)
 static bool page_pool_recycle_in_ring(struct page_pool *pool, struct page *page)
 {
 	int ret;
-	/* BH protection not needed if current is serving softirq */
-	if (in_serving_softirq())
-		ret = ptr_ring_produce(&pool->ring, page);
-	else
-		ret = ptr_ring_produce_bh(&pool->ring, page);
+
+	ret = ptr_ring_produce_any(&pool->ring, page);
 
 	return (ret == 0) ? true : false;
 }
@@ -411,7 +409,7 @@ static void page_pool_empty_ring(struct page_pool *pool)
 	struct page *page;
 
 	/* Empty recycle ring */
-	while ((page = ptr_ring_consume_bh(&pool->ring))) {
+	while ((page = ptr_ring_consume_any(&pool->ring))) {
 		/* Verify the refcnt invariant of cached pages */
 		if (!(page_ref_count(page) == 1))
 			pr_crit("%s() page_pool refcnt %d violation\n",
Signature · 1 line
2.17.1
maintainer
#1 lore raw

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 8 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161 __local_bh_enable_ip+0x35/0x50

...

Quoted 1 line

The netpoll stuff always makes the locking more complicated than it needs to be. I wonder if there is another way around this issue?

Because IRQ save/restore is a high cost to pay in this critical path.

#2 lore raw
Quoted 13 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 8 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161
__local_bh_enable_ip+0x35/0x50
 ...
Quoted 7 lines
Quoted 1 line

The netpoll stuff always makes the locking more complicated than it needs
to be. I wonder if there is another way around this issue?

Because IRQ save/restore is a high cost to pay in this critical path.

a printk inside irq context lead to this, so maybe it can be avoided ..

or instead of checking in_serving_softirq() change page_pool to check in_interrupt() which is more powerful, to avoid ptr_ring locking and the complication with netpoll altogether.

I wonder why Jesper picked in_serving_softirq() in first place, was there a specific reason ? or he just wanted it to be as less strict as possible ?

#3 lore raw
Quoted 10 lines · On Tue, 10 Mar 2020 02:30:34 +0000 Saeed Mahameed <[email protected]> wrote:
Quoted 9 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 6 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

Netpoll calls into drivers are problematic, nasty and annoying. Drivers usually catch netpoll calls via seeing NAPI budget is zero, and handle the situation inside the driver e.g.[1][2]. (even napi_consume_skb catch it this way).

[1] https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#L3179
[2] https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#L1110

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161  
__local_bh_enable_ip+0x35/0x50
 ...  
Quoted 1 line

The netpoll stuff always makes the locking more complicated than it needs to be. I wonder if there is another way around this issue?

Because IRQ save/restore is a high cost to pay in this critical path.

Yes, huge NACK from me, this would kill performance. We need to find another way.

Quoted 9 lines

a printk inside irq context lead to this, so maybe it can be avoided ..

or instead of checking in_serving_softirq() change page_pool to check in_interrupt() which is more powerful, to avoid ptr_ring locking and the complication with netpoll altogether.

I wonder why Jesper picked in_serving_softirq() in first place, was there a specific reason ? or he just wanted it to be as less strict as possible ?

I wanted to make it specific that this is optimized for softirq.

I actually think this is a regression we have introduced recently in combined patches:

304db6cb7610 ("page_pool: refill page when alloc.count of pool is zero") 44768decb7c0 ("page_pool: handle page recycle for NUMA_NO_NODE condition")

I forgot that the in_serving_softirq() check was also protecting us when getting called from netpoll. The general idea before was that if the in_serving_softirq() check failed, then we falled-through to slow-path calling normal alloc_pages_node (regular page allocator).

Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#9 lore raw
In reply to David Miller
Quoted 10 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 7 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

Quoted 8 lines
Quoted 1 line
WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161 __local_bh_enable_ip+0x35/0x50

...

Quoted 1 line

The netpoll stuff always makes the locking more complicated than it needs to be. I wonder if there is another way around this issue?

Because IRQ save/restore is a high cost to pay in this critical path.

I agree with this - the proposed patch is more of a bandaid than anything else. Why is the entire network poll path being called with IRQs off? That seems wrong - while netcons is expected to work in all cases, it does seem like it should be smarter and for the normal case, just queue and raise NET_TX_SOFTIRQ would be a more logical path.

Signature · 1 line
Jonathan
#7 lore raw
In reply to Saeed Mahameed
Quoted 16 lines · Saeed Mahameed
Quoted 15 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 8 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161
__local_bh_enable_ip+0x35/0x50
 ...
Quoted 1 line
Quoted 7 lines
Quoted 5 lines

The netpoll stuff always makes the locking more complicated than it needs
to be. I wonder if there is another way around this issue?

Because IRQ save/restore is a high cost to pay in this critical path.

a printk inside irq context lead to this, so maybe it can be avoided ..

This was caused by a printk in hpet_rtc_timer_reinit() complaining about RTC interrupts being lost. I'm not sure it's practical trying to locate all the printk cases like this.

Quoted 3 lines

or instead of checking in_serving_softirq() change page_pool to check in_interrupt() which is more powerful, to avoid ptr_ring locking and the complication with netpoll altogether.

That's another approach:

    ret = 1;
    if (!in_irq()) {
        if (in_serving_softirq())
            ret = ptr_ring_produce(....
        else
            ret = ptr_ring_produce_bh(....
    }

which would return failure and release the page from the page pool. This doesn't address the allocation or the bulk release path.

Quoted 4 lines

I wonder why Jesper picked in_serving_softirq() in first place, was there a specific reason ? or he just wanted it to be as less strict as possible ?

From the code, it looks like he was optimizing to avoid the _bh variant if possible.

Signature · 1 line
Jonathan
#4 lore raw
In reply to Jesper Dangaard Brouer
Quoted 14 lines · Jesper Dangaard Brouer
Quoted 10 lines · On Tue, 10 Mar 2020 02:30:34 +0000 Saeed Mahameed <[email protected]> wrote:
Quoted 9 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 6 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

Quoted 7 lines

Netpoll calls into drivers are problematic, nasty and annoying. Drivers usually catch netpoll calls via seeing NAPI budget is zero, and handle the situation inside the driver e.g.[1][2]. (even napi_consume_skb catch it this way).

[1] https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#L3179
[2] https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#L1110

This still doesn't seem safe.

netpoll calls napi poll in IRQ context.
Consider the following path:

ixgbe_poll()
ixgbe_for_each_ring()
ixgbe_clean_tx_irq() /* before the L3179 check above */

    xdp_return_frame()
     page_pool_put_page( .. napi=0 )

Which bypasses the cache and tries to return the page to the ring. Since in_serving_softirq() is false, it uses the _bh variant, which blows up.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161
__local_bh_enable_ip+0x35/0x50
 ...
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

The netpoll stuff always makes the locking more complicated than it needs to be. I wonder if there is another way around this issue?

Because IRQ save/restore is a high cost to pay in this critical path.

Yes, huge NACK from me, this would kill performance. We need to find another way.

Sure - checking in_irq() in the free path is another solution.

Quoted 20 lines
Quoted 9 lines

a printk inside irq context lead to this, so maybe it can be avoided ..

or instead of checking in_serving_softirq() change page_pool to check in_interrupt() which is more powerful, to avoid ptr_ring locking and the complication with netpoll altogether.

I wonder why Jesper picked in_serving_softirq() in first place, was there a specific reason ? or he just wanted it to be as less strict as possible ?

I wanted to make it specific that this is optimized for softirq.

I actually think this is a regression we have introduced recently in combined patches:

304db6cb7610 ("page_pool: refill page when alloc.count of pool is
zero")
44768decb7c0 ("page_pool: handle page recycle for NUMA_NO_NODE
condition")

Quoted 4 lines

I forgot that the in_serving_softirq() check was also protecting us when getting called from netpoll. The general idea before was that if the in_serving_softirq() check failed, then we falled-through to slow-path calling normal alloc_pages_node (regular page allocator).

I don't think it ever did that - in all cases, the in_serving_softirq() check guards access to the cache, otherwise it falls through to the ring.

Here, it seems we need a tristate:

   in_irq:  fall through to system page allocator
   napi?    use cache
   else     ring

However, looking at those two patches again, while it doesn't address this current issue, I do agree that they are regressions:

1) Under stress, the system allocator may return pages with a different node id.

2) Since the check is now removed from page_pool_reusable(), the foreign page is placed in the cache and persists there until it is moved to the ring.

3) on refill from ring, if this page is found, processing stops, regardless of the other pages in the ring.

The above regression(s) are introduced by trying to avoid checking the page node_id at free time. Are there any metrics showing that this is a performance issue? Drivers which do page biasing still have to perform this check in the driver (e.g.: mlx4, i40e) regardless.

Signature · 1 line
Jonathan
maintainer
#5 lore raw
In reply to Jesper Dangaard Brouer
Quoted 10 lines · On Tue, 10 Mar 2020 11:04:12 +0100 Jesper Dangaard Brouer wrote:
Quoted 8 lines · On Tue, 10 Mar 2020 02:30:34 +0000 Saeed Mahameed <[email protected]> wrote:
Quoted 7 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 4 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient protection, so use irqsave/restore instead.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Error observed on a modified mlx4 driver, but the code path exists for any driver which calls page_pool_recycle from napi poll.

Netpoll calls into drivers are problematic, nasty and annoying. Drivers usually catch netpoll calls via seeing NAPI budget is zero, and handle the situation inside the driver e.g.[1][2]. (even napi_consume_skb catch it this way).

I'm probably just repeating what you said, but would it be reasonable to expect page_pool users to special-case XDP rings to not be cleaned? netpoll has no use for them.

Perhaps that would not solve the issue for those funky drivers which use the same rings for both XDP and the stack. Sigh. Do we care about them?

#8 lore raw
In reply to Jonathan Lemon
Quoted 58 lines · Jonathan Lemon
Quoted 23 lines · Saeed Mahameed
Quoted 22 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 10 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient
protection, so use irqsave/restore instead.

Error observed on a modified mlx4 driver, but the code path exists
for any driver which calls page_pool_recycle from napi poll.

WARNING: CPU: 34 PID: 550248 at /ro/source/kernel/softirq.c:161
__local_bh_enable_ip+0x35/0x50
 ...
Quoted 1 line

The netpoll stuff always makes the locking more complicated than it
needs
to be. I wonder if there is another way around this issue?

Quoted 5 lines
Quoted 2 lines

Because IRQ save/restore is a high cost to pay in this critical path.

a printk inside irq context lead to this, so maybe it can be avoided ..

This was caused by a printk in hpet_rtc_timer_reinit() complaining about
RTC interrupts being lost. I'm not sure it's practical trying to locate
all the printk cases like this.

Quoted 4 lines

or instead of checking in_serving_softirq() change page_pool to check in_interrupt() which is more powerful, to avoid ptr_ring locking
and the complication with netpoll altogether.

That's another approach:

    ret = 1;
    if (!in_irq()) {
        if (in_serving_softirq())
            ret = ptr_ring_produce(....
        else
            ret = ptr_ring_produce_bh(....
    }

which would return failure and release the page from the page pool. This doesn't address the allocation or the bulk release path.

Quoted 8 lines
Quoted 4 lines

I wonder why Jesper picked in_serving_softirq() in first place, was there a specific reason ? or he just wanted it to be as less strict as
possible ?

From the code, it looks like he was optimizing to avoid the _bh variant
if possible.

I think that if you use another variant this it intorduce a possiblerace bestrewn sofitq and hardirq ..

#6 lore raw
In reply to Jakub Kicinski
Quoted 24 lines · Jakub Kicinski
Quoted 16 lines · On Tue, 10 Mar 2020 11:04:12 +0100 Jesper Dangaard Brouer wrote:
Quoted 9 lines · On Tue, 10 Mar 2020 02:30:34 +0000 Saeed Mahameed <[email protected]> wrote:
Quoted 8 lines · David Miller

From: Jonathan Lemon <[email protected]>
Date: Mon, 9 Mar 2020 12:49:29 -0700

Quoted 5 lines

netpoll may be called from IRQ context, which may access the page pool ring. The current _bh variants do not provide sufficient
protection, so use irqsave/restore instead.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Error observed on a modified mlx4 driver, but the code path exists
for any driver which calls page_pool_recycle from napi poll.

Quoted 6 lines

Netpoll calls into drivers are problematic, nasty and annoying. Drivers
usually catch netpoll calls via seeing NAPI budget is zero, and handle
the situation inside the driver e.g.[1][2]. (even napi_consume_skb catch it this way).

Quoted 5 lines

I'm probably just repeating what you said, but would it be reasonable to expect page_pool users to special-case XDP rings to not be cleaned?
netpoll has no use for them.

sounds like a better band-aid

Quoted 3 lines

Perhaps that would not solve the issue for those funky drivers which use the same rings for both XDP and the stack. Sigh. Do we care about them?

They can special case per-page, if it is an XDP tx page, don't recycle to ptr ring ..