bnx2: Use dev_kfree_skb_any() in bnx2_tx_int()
Subsystems (3)
PATCH 1/1 bnx2: Use dev_kfree_skb_any() in bnx2_tx_int()
Using dev_kfree_skb_any() will resolve the below issue when a netconsole message is transmitted in an irq.
------------[ cut here ]------------ WARNING: at net/core/skbuff.c:451 skb_release_head_state+0x7b/0xe1() ... Pid: 0, comm: swapper/2 Not tainted 3.4.55 #1 Call Trace: <IRQ> [<ffffffff8104934c>] warn_slowpath_common+0x85/0x9d [<ffffffff8104937e>] warn_slowpath_null+0x1a/0x1c [<ffffffff81429aa7>] skb_release_head_state+0x7b/0xe1 [<ffffffff814297e1>] __kfree_skb+0x16/0x81 [<ffffffff814298a0>] consume_skb+0x54/0x69 [<ffffffffa015925b>] bnx2_tx_int.clone.6+0x1b0/0x33e [bnx2] [<ffffffff8129c54d>] ? unmask_msi_irq+0x10/0x12 [<ffffffffa015aa06>] bnx2_poll_work+0x3a/0x73 [bnx2] [<ffffffffa015aa73>] bnx2_poll_msix+0x34/0xb4 [bnx2] [<ffffffff814466a2>] netpoll_poll_dev+0xb9/0x1b7 [<ffffffff814467d7>] ? find_skb+0x37/0x82 [<ffffffff814461ed>] netpoll_send_skb_on_dev+0x117/0x200 [<ffffffff81446a52>] netpoll_send_udp+0x230/0x242 [<ffffffffa0174296>] write_msg+0xa7/0xfb [netconsole] [<ffffffff814258a4>] ? sk_free+0x1c/0x1e [<ffffffff810495ad>] __call_console_drivers+0x7d/0x8f [<ffffffff81049674>] _call_console_drivers+0xb5/0xd0 [<ffffffff8104a134>] console_unlock+0x131/0x219 [<ffffffff8104a7f9>] vprintk+0x3bc/0x405 [<ffffffff81460073>] ? NF_HOOK.clone.1+0x4c/0x53 [<ffffffff81460308>] ? ip_rcv+0x23c/0x268 [<ffffffff814ddd4f>] printk+0x68/0x71 [<ffffffff813315b3>] __dev_printk+0x78/0x7a [<ffffffff813316b2>] dev_warn+0x53/0x55 [<ffffffff8127f181>] ? swiotlb_unmap_sg_attrs+0x47/0x5c [<ffffffffa004f876>] complete_scsi_command+0x28a/0x4a0 [hpsa] [<ffffffffa004fadb>] finish_cmd+0x4f/0x66 [hpsa] [<ffffffffa004fd97>] process_indexed_cmd+0x48/0x54 [hpsa] [<ffffffffa004ff25>] do_hpsa_intr_msi+0x4e/0x77 [hpsa] [<ffffffff810baebb>] handle_irq_event_percpu+0x5e/0x1b6 [<ffffffff81088a0b>] ? timekeeping_update+0x43/0x45 [<ffffffff810bb04b>] handle_irq_event+0x38/0x54 [<ffffffff8102bd1e>] ? ack_apic_edge+0x36/0x3a [<ffffffff810bd762>] handle_edge_irq+0xa5/0xc8 [<ffffffff81010d56>] handle_irq+0x127/0x135 [<ffffffff814e3426>] ? __atomic_notifier_call_chain+0x12/0x14 [<ffffffff814e343c>] ? atomic_notifier_call_chain+0x14/0x16 [<ffffffff814e897d>] do_IRQ+0x4d/0xb4 [<ffffffff814dffea>] common_interrupt+0x6a/0x6a <EOI> [<ffffffff812b7603>] ? intel_idle+0xd8/0x112 [<ffffffff812b7603>] ? intel_idle+0xd8/0x112 [<ffffffff812b75e9>] ? intel_idle+0xbe/0x112 [<ffffffff814012fc>] cpuidle_enter+0x12/0x14 [<ffffffff814019c2>] cpuidle_idle_call+0xd1/0x19b [<ffffffff81016551>] cpu_idle+0xb6/0xff [<ffffffff814d726b>] start_secondary+0xc8/0xca ---[ end trace 3f15cd66441c770d ]---
--- drivers/net/ethernet/broadcom/bnx2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/broadcom/bnx2.c +1 -1
@@ -2868,7 +2868,7 @@ bnx2_tx_int(struct bnx2 *bp, struct bnx2_napi *bnapi, int budget) sw_cons =BNX2_NEXT_TX_BD (sw_cons); tx_bytes += skb->len; -dev_kfree_skb (skb); +dev_kfree_skb_any (skb); tx_pkt++;if (tx_pkt == budget)break ;
Signature · 1 line
1.7.9.5
David Miller
maintainerFrom: David Mackey <[email protected]>
Date: Tue, 29 Oct 2013 15:16:38 -0700
Quoted 2 lines
Using
dev_kfree_skb_any()will resolve the below issue when a netconsole message is transmitted in an irq.
...
Quoted 1 line
Signed-off-by David Mackey <[email protected]>
This is absolutely not the correct fix.
The netpoll facility must invoke ->poll() in an environment which is compatible, locking and interrupt/soft-interrupt wise, as that in which it is normally called.
Therefore, bnx2_tx_int(), which is invoked from the driver's ->poll() method, should not need to use dev_kfree_skb_any(). The real problem is somewhere else.
Quoted 5 lines · David Miller
From: David Mackey <[email protected]>
Date: Tue, 29 Oct 2013 15:16:38 -0700Quoted 2 lines
Using
dev_kfree_skb_any()will resolve the below issue when a netconsole message is transmitted in an irq.
Quoted 9 lines
...
Quoted 1 line
Signed-off-by David Mackey <[email protected]>This is absolutely not the correct fix.
The netpoll facility must invoke ->poll() in an environment which is compatible, locking and interrupt/soft-interrupt wise, as that in which it is normally called.
Normally ->poll() is called in softirq context, while netpoll could be called in any context depending on its caller.
Also, netpoll disables IRQ for tx, which is another difference.
David Miller
maintainerFrom: Cong Wang <[email protected]>
Date: Tue, 29 Oct 2013 20:50:08 -0700
Quoted 2 lines
Normally ->poll() is called in softirq context, while netpoll could be called in any context depending on its caller.
It still makes amends to make the execution context still looks "compatible" as far as locking et al. is concerned.
Quoted 8 lines · David Miller
From: Cong Wang <[email protected]>
Date: Tue, 29 Oct 2013 20:50:08 -0700Quoted 2 lines
Normally ->poll() is called in softirq context, while netpoll could be called in any context depending on its caller.
It still makes amends to make the execution context still looks "compatible" as far as locking et al. is concerned.
Adjusting netpoll code for IRQ context is much harder than just calling dev_kfree_skb_any()...
What's more, we have similar change before:
commit ed79bab847d8e5a2986d8ff43c49c6fb8ee3265f
Author: Eric Dumazet <[email protected]> Date: Wed Oct 14 14:36:43 2009 +0000
virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()
David Miller
maintainerFrom: Cong Wang <[email protected]>
Date: Wed, 30 Oct 2013 12:23:52 -0700
Quoted 13 lines · David Miller
Quoted 8 lines
From: Cong Wang <[email protected]>
Date: Tue, 29 Oct 2013 20:50:08 -0700Quoted 2 lines
Normally ->poll() is called in softirq context, while netpoll could be called in any context depending on its caller.
It still makes amends to make the execution context still looks "compatible" as far as locking et al. is concerned.
Adjusting netpoll code for IRQ context is much harder than just calling
dev_kfree_skb_any()...
Quoted 7 lines
What's more, we have similar change before:
commit
ed79bab847d8e5a2986d8ff43c49c6fb8ee3265fAuthor: Eric Dumazet <[email protected]> Date: Wed Oct 14 14:36:43 2009 +0000
virtio_net: usedev_kfree_skb_any()infree_old_xmit_skbs()
Explain to me then why other ethernet drivers implemented identically, such as tg3, can use plain dev_kfree_skb() just fine?
Quoted 3 lines · David Miller
Explain to me then why other ethernet drivers implemented identically, such as tg3, can use plain
dev_kfree_skb()just fine?
I don't think they are fine, I just don't see bug reports for them. At very least, I saw a same bug report for be2net too.
To reproduce this bug, we need to find some one calling printk() within IRQ handler, which seems rare? It seems there are few people using hpsa driver together with netconsole.
David Miller
maintainerFrom: Cong Wang <[email protected]>
Date: Wed, 30 Oct 2013 15:01:12 -0700
Quoted 11 lines · David Miller
Quoted 3 lines
Explain to me then why other ethernet drivers implemented identically, such as tg3, can use plain
dev_kfree_skb()just fine?I don't think they are fine, I just don't see bug reports for them. At very least, I saw a same bug report for be2net too.
To reproduce this bug, we need to find some one calling
printk()within IRQ handler, which seems rare? It seems there are few people using hpsa driver together with netconsole.
We've had this conversation before I believe, and I absolutely do not want to have to add IRQ safety to every ->poll() implementation.
We have to provide a softint compatible environment for this callback to run in else everything is completely broken.
All these drivers can safely assume softirq safe locking is sufficient, you're suggesting we need to take this hardirq safety and I'm really not willing to allow things to go that far. A lot of effort has been expended precisely to avoid that kind of overhead and cost.
A message that is not in the archive (sent off-list or not yet received): CAM_iQpXPrEj619=PRaKN4PxgOoEB96U0mLFS65AzXOezSWvhxQ@mail.gmail.com
David Miller
maintainerFrom: Cong Wang <[email protected]>
Date: Thu, 31 Oct 2013 21:19:16 -0700
Quoted 5 lines
Quoted 4 lines · 2013年10月30日 下午9:26于 "David Miller" <[email protected]>写道:
We have to provide a softint compatible environment for this callback to run in else everything is completely broken.
Quoted 7 lines
Quoted 5 lines
All these drivers can safely assume softirq safe locking is sufficient, you're suggesting we need to take this hardirq safety and I'm really not willing to allow things to go that far. A lot of effort has been expended precisely to avoid that kind of overhead and cost.
Alright, I am thinking to move
netpoll_poll_dev()to a delayed work.
What if the printk is outputting a message that will help us discover that work queues are deadlocked?
You can't delay the message, because every layer of indirection you add increases the possibility that the message it never seen. You have to do it synchronously.
Quoted 14 lines · David Miller
From: Cong Wang <[email protected]>
Date: Thu, 31 Oct 2013 21:19:16 -0700Quoted 11 lines
Quoted 9 lines · 2013年10月30日 下午9:26于 "David Miller" <[email protected]>写道:
We have to provide a softint compatible environment for this callback to run in else everything is completely broken.
All these drivers can safely assume softirq safe locking is sufficient, you're suggesting we need to take this hardirq safety and I'm really not willing to allow things to go that far. A lot of effort has been expended precisely to avoid that kind of overhead and cost.
Quoted 8 lines
Quoted 1 line
Alright, I am thinking to move
netpoll_poll_dev()to a delayed work.What if the printk is outputting a message that will help us discover that work queues are deadlocked?
You can't delay the message, because every layer of indirection you add increases the possibility that the message it never seen. You have to do it synchronously.
As you've said, the ndo_start_xmit and NAPI poll operations are intended to be called in softirq context, so everything that interlocks with them will use spin_lock_bh(). Calling them from hardirq context obviously opens the possibility of a deadlock. How do you expect anyone to solve that?
I think that most of the time netpoll doesn't actually call the NAPI poll function, and the driver ndo_start_xmit function doesn't take any locks, so we don't actually hit the deadlock in practice (on mainline kernels - RT is a different story).
Obviously, the less machinery netpoll relies on continuing to work, the better, so it should preferably defer to sofirq context rather than workqueue context. I think this means hooking queue_process() into net_tx_action(), and then cutting out much of the rest of netpoll.
Ben.
Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
David Miller
maintainerFrom: Ben Hutchings <[email protected]>
Date: Fri, 1 Nov 2013 23:34:50 +0000
Quoted 5 lines
As you've said, the
ndo_start_xmitand NAPI poll operations are intended to be called in softirq context, so everything that interlocks with them will usespin_lock_bh(). Calling them from hardirq context obviously opens the possibility of a deadlock. How do you expect anyone to solve that?
That's not what I said.
I did not say that it must be invoked in softirq context.
I said that it MUST LOOK like it is being invoked in softirq context as far as the ->poll() code paths can tell.
But yes, that's hard.
The thing is, ->poll() is atomic. You never will have poll calls recurse into eachother. This is why we strongly encourage all driver authors to make their ->poll() implementations lockless.
And, wouldn't you know it, tg3 is a driver that does this properly.
Therefore, there are no netpoll no locking problems, because the poll implementation takes no locks and therefore doesn't care.
That's why tg3 doesn't have any of these netpoll issues.