netpoll: prevent hanging NAPI when netcons gets enabled
Subsystems (2)
Jakub Kicinski
maintainerPATCH v2 1/1 netpoll: prevent hanging NAPI when netcons gets enabled
Paolo spotted hangs in NIPA running driver tests against virtio. The tests hang in virtnet_close() -> virtnet_napi_tx_disable().
The problem is only reproducible if running multiple of our tests in sequence (I used TEST_PROGS="xdp.py ping.py netcons_basic.sh \netpoll_basic.py stats.py"). Initial suspicion was that this is a simple case of double-disable of NAPI, but instrumenting the code reveals:
Deadlocked on NAPI ffff888007cd82c0 (virtnet_poll_tx):
state: 0x37, disabled: false, owner: 0, listed: false, weight: 64
The NAPI was not in fact disabled, owner is 0 (rather than -1), so the NAPI "thinks" it's scheduled for CPU 0 but it's not listed (!list_empty(&n->poll_list) => false). It seems odd that normal NAPI processing would wedge itself like this.
Better suspicion is that netpoll gets enabled while NAPI is polling, and also grabs the NAPI instance. This confuses napi_complete_done():
[netpoll] [normal NAPI]
napi_poll()
have = netpoll_poll_lock()
rcu_access_pointer(dev->npinfo)
return NULL # no netpoll
__napi_poll()
->poll(->weight)
poll_napi()
cmpxchg(->poll_owner, -1, cpu)
poll_one_napi()
set_bit(NAPI_STATE_NPSVC, ->state)
napi_complete_done()
if (NAPIF_STATE_NPSVC)
return false
# exit without clearing SCHEDThis feels very unlikely, but perhaps virtio has some interactions with the hypervisor in the NAPI ->poll that makes the race window larger?
Best I could to to prove the theory was to add and trigger this warning in napi_poll (just before netpoll_poll_unlock()):
WARN_ONCE(!have && rcu_access_pointer(n->dev->npinfo) &&
napi_is_scheduled(n) && list_empty(&n->poll_list),
"NAPI race with netpoll %px", n);If this warning hits the next virtio_close() will hang.
This patch survived 30 test iterations without a hang (without it the longest clean run was around 10). Credit for triggering this goes to Breno's recent netconsole tests.
---
v2:
- move the sync to netpoll_setup()
v1: https://lore.kernel.org/[email protected]
--- net/core/netpoll.c | 7 +++++++ 1 file changed, 7 insertions(+)
net/core/netpoll.c +7 -0
@@ -768,6 +768,13 @@ int netpoll_setup(struct netpoll *np)if (err) goto flush;rtnl_unlock (); + +/* Make sure all NAPI polls which started before dev->npinfo +* was visible have exited before we start calling NAPI poll. +* NAPI skips locking if dev->npinfo is NULL. +*/ +synchronize_rcu (); +return 0 ; flush:
Signature · 1 line
2.50.1
Quoted 37 lines · Jakub Kicinski
Paolo spotted hangs in NIPA running driver tests against virtio. The tests hang in
virtnet_close()->virtnet_napi_tx_disable().The problem is only reproducible if running multiple of our tests in sequence (I used TEST_PROGS="xdp.py
ping.pynetcons_basic.sh\netpoll_basic.pystats.py"). Initial suspicion was that this is a simple case of double-disable of NAPI, but instrumenting the code reveals:Deadlocked on NAPI
ffff888007cd82c0(virtnet_poll_tx):
state:0x37, disabled: false, owner: 0, listed: false, weight: 64The NAPI was not in fact disabled, owner is 0 (rather than -1), so the NAPI "thinks" it's scheduled for CPU 0 but it's not listed (!list_empty(&n->poll_list) => false). It seems odd that normal NAPI processing would wedge itself like this.
Better suspicion is that netpoll gets enabled while NAPI is polling, and also grabs the NAPI instance. This confuses
napi_complete_done():[netpoll] [normal NAPI]
napi_poll() have = netpoll_poll_lock() rcu_access_pointer(dev->npinfo) return NULL # no netpoll __napi_poll() ->poll(->weight) poll_napi() cmpxchg(->poll_owner, -1, cpu) poll_one_napi() set_bit(NAPI_STATE_NPSVC, ->state) napi_complete_done() if (NAPIF_STATE_NPSVC) return false # exit without clearing SCHED
Quoted 3 lines
This feels very unlikely, but perhaps virtio has some interactions with the hypervisor in the NAPI ->poll that makes the race window larger?
Somehow, for example KVM can schedule out the vcpu that runs normal NAPI at this time.
Quoted 54 lines
Best I could to to prove the theory was to add and trigger this warning in
napi_poll(just beforenetpoll_poll_unlock()):WARN_ONCE(!have && rcu_access_pointer(n->dev->npinfo) && napi_is_scheduled(n) && list_empty(&n->poll_list), "NAPI race with netpoll %px", n);If this warning hits the next
virtio_close()will hang.This patch survived 30 test iterations without a hang (without it the longest clean run was around 10). Credit for triggering this goes to Breno's recent netconsole tests.
Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by Paolo Abeni <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>---
v2:
- move the sync tonetpoll_setup()
v1: https://lore.kernel.org/[email protected]CC Jason Wang <[email protected]>CC Zigit Zo <[email protected]>CC "Michael S. Tsirkin" <[email protected]>CC Xuan Zhuo <[email protected]>CC Eugenio Pérez <[email protected]>--- net/core/netpoll.c | 7 +++++++ 1 file changed, 7 insertions(+)net/core/netpoll.c +7
-1@@ -768,6 +768,13 @@ int netpoll_setup(struct netpoll *np)if (err) goto flush;rtnl_unlock (); + +/* Make sure all NAPI polls which started before dev->npinfo +* was visible have exited before we start calling NAPI poll. +* NAPI skips locking if dev->npinfo is NULL. +*/ +synchronize_rcu (); +return 0 ; flush: --
Quoted 2 lines
2.50.1
Thanks
Thanks.
Quoted 95 lines · Jakub Kicinski
Paolo spotted hangs in NIPA running driver tests against virtio. The tests hang in
virtnet_close()->virtnet_napi_tx_disable().The problem is only reproducible if running multiple of our tests in sequence (I used TEST_PROGS="xdp.py
ping.pynetcons_basic.sh\netpoll_basic.pystats.py"). Initial suspicion was that this is a simple case of double-disable of NAPI, but instrumenting the code reveals:Deadlocked on NAPI
ffff888007cd82c0(virtnet_poll_tx):
state:0x37, disabled: false, owner: 0, listed: false, weight: 64The NAPI was not in fact disabled, owner is 0 (rather than -1), so the NAPI "thinks" it's scheduled for CPU 0 but it's not listed (!list_empty(&n->poll_list) => false). It seems odd that normal NAPI processing would wedge itself like this.
Better suspicion is that netpoll gets enabled while NAPI is polling, and also grabs the NAPI instance. This confuses
napi_complete_done():[netpoll] [normal NAPI]
napi_poll() have = netpoll_poll_lock() rcu_access_pointer(dev->npinfo) return NULL # no netpoll __napi_poll() ->poll(->weight) poll_napi() cmpxchg(->poll_owner, -1, cpu) poll_one_napi() set_bit(NAPI_STATE_NPSVC, ->state) napi_complete_done() if (NAPIF_STATE_NPSVC) return false # exit without clearing SCHEDThis feels very unlikely, but perhaps virtio has some interactions with the hypervisor in the NAPI ->poll that makes the race window larger?
Best I could to to prove the theory was to add and trigger this warning in
napi_poll(just beforenetpoll_poll_unlock()):WARN_ONCE(!have && rcu_access_pointer(n->dev->npinfo) && napi_is_scheduled(n) && list_empty(&n->poll_list), "NAPI race with netpoll %px", n);If this warning hits the next
virtio_close()will hang.This patch survived 30 test iterations without a hang (without it the longest clean run was around 10). Credit for triggering this goes to Breno's recent netconsole tests.
Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by Paolo Abeni <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>---
v2:
- move the sync tonetpoll_setup()
v1: https://lore.kernel.org/[email protected]CC Jason Wang <[email protected]>CC Zigit Zo <[email protected]>CC "Michael S. Tsirkin" <[email protected]>CC Xuan Zhuo <[email protected]>CC Eugenio Pérez <[email protected]>--- net/core/netpoll.c | 7 +++++++ 1 file changed, 7 insertions(+)net/core/netpoll.c +7
-1@@ -768,6 +768,13 @@ int netpoll_setup(struct netpoll *np)if (err) goto flush;rtnl_unlock (); + +/* Make sure all NAPI polls which started before dev->npinfo +* was visible have exited before we start calling NAPI poll. +* NAPI skips locking if dev->npinfo is NULL. +*/ +synchronize_rcu (); +return 0 ; flush: --2.50.1
Quoted 49 lines · Jakub Kicinski
Paolo spotted hangs in NIPA running driver tests against virtio. The tests hang in
virtnet_close()->virtnet_napi_tx_disable().The problem is only reproducible if running multiple of our tests in sequence (I used TEST_PROGS="xdp.py
ping.pynetcons_basic.sh\netpoll_basic.pystats.py"). Initial suspicion was that this is a simple case of double-disable of NAPI, but instrumenting the code reveals:Deadlocked on NAPI
ffff888007cd82c0(virtnet_poll_tx):
state:0x37, disabled: false, owner: 0, listed: false, weight: 64The NAPI was not in fact disabled, owner is 0 (rather than -1), so the NAPI "thinks" it's scheduled for CPU 0 but it's not listed (!list_empty(&n->poll_list) => false). It seems odd that normal NAPI processing would wedge itself like this.
Better suspicion is that netpoll gets enabled while NAPI is polling, and also grabs the NAPI instance. This confuses
napi_complete_done():[netpoll] [normal NAPI]
napi_poll() have = netpoll_poll_lock() rcu_access_pointer(dev->npinfo) return NULL # no netpoll __napi_poll() ->poll(->weight) poll_napi() cmpxchg(->poll_owner, -1, cpu) poll_one_napi() set_bit(NAPI_STATE_NPSVC, ->state) napi_complete_done() if (NAPIF_STATE_NPSVC) return false # exit without clearing SCHEDThis feels very unlikely, but perhaps virtio has some interactions with the hypervisor in the NAPI ->poll that makes the race window larger?
Best I could to to prove the theory was to add and trigger this warning in
napi_poll(just beforenetpoll_poll_unlock()):WARN_ONCE(!have && rcu_access_pointer(n->dev->npinfo) && napi_is_scheduled(n) && list_empty(&n->poll_list), "NAPI race with netpoll %px", n);If this warning hits the next
virtio_close()will hang.
Quoted 8 lines
This patch survived 30 test iterations without a hang (without it the longest clean run was around 10). Credit for triggering this goes to Breno's recent netconsole tests.
Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by Paolo Abeni <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 10 lines · On Fri, 25 Jul 2025 18:08:46 -0700 you wrote:
Paolo spotted hangs in NIPA running driver tests against virtio. The tests hang in
virtnet_close()->virtnet_napi_tx_disable().The problem is only reproducible if running multiple of our tests in sequence (I used TEST_PROGS="xdp.py
ping.pynetcons_basic.sh\netpoll_basic.pystats.py"). Initial suspicion was that this is a simple case of double-disable of NAPI, but instrumenting the code reveals:[...]
Here is the summary with links:
- [net,v2] netpoll: prevent hanging NAPI when netcons gets enabled
https://git.kernel.org/netdev/net/c/2da4def0f487
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html