Skip to contentSkip to conversation

hv_netvsc: fix potential deadlock in netvsc_vf_setxdp()

PATCH Superseded Saurabh Sengar 3 participants 5 messages May 21, 2025 – May 23, 2025

Subsystems (6)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 hv_netvsc: fix potential deadlock in netvsc_vf_setxdp()

The MANA driver's probe registers netdevice via the following call chain:

mana_probe()
register_netdev()
register_netdevice()

register_netdevice() calls notifier callback for netvsc driver, holding the netdev mutex via netdev_lock_ops().

Further this netvsc notifier callback end up attempting to acquire the same lock again in dev_xdp_propagate() leading to deadlock.

netvsc_netdev_event()
netvsc_vf_setxdp()
dev_xdp_propagate()

This deadlock was not observed so far because net_shaper_ops was never set, and thus the lock was effectively a no-op in this case. Fix this by using netif_xdp_propagate() instead of dev_xdp_propagate() to avoid recursive locking in this path.

Also, clean up the unregistration path by removing the unnecessary call to netvsc_vf_setxdp(), since unregister_netdevice_many_notify() already performs this cleanup via dev_xdp_uninstall().

Fixes 97246d6d21c2 ("net: hold netdev instance lock during ndo_bpf")

---
[V2]
- Modified commit message

 drivers/net/hyperv/netvsc_bpf.c | 2 +-
 drivers/net/hyperv/netvsc_drv.c | 2 --
 net/core/dev.c                  | 1 +
 3 files changed, 2 insertions(+), 3 deletions(-)
drivers/net/hyperv/netvsc_bpf.c +1 -1
index e01c5997a551..1dd3755d9e6d 100644
@@ -183,7 +183,7 @@ int netvsc_vf_setxdp(struct net_device *vf_netdev, struct bpf_prog *prog)
 	xdp.command = XDP_SETUP_PROG;
 	xdp.prog = prog;
 
-	ret = dev_xdp_propagate(vf_netdev, &xdp);
+	ret = netif_xdp_propagate(vf_netdev, &xdp);
 
 	if (ret && prog)
 		bpf_prog_put(prog);
drivers/net/hyperv/netvsc_drv.c +0 -2
index d8b169ac0343..ee3aaf9c10e6 100644
@@ -2462,8 +2462,6 @@ static int netvsc_unregister_vf(struct net_device *vf_netdev)
 
 	netdev_info(ndev, "VF unregistering: %s\n", vf_netdev->name);
 
-	netvsc_vf_setxdp(vf_netdev, NULL);
-
 	reinit_completion(&net_device_ctx->vf_add);
 	netdev_rx_handler_unregister(vf_netdev);
 	netdev_upper_dev_unlink(vf_netdev, ndev);
net/core/dev.c +1 -0
index fccf2167b235..8c6c9d7fba26 100644
@@ -9953,6 +9953,7 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
 
 	return dev->netdev_ops->ndo_bpf(dev, bpf);
 }
+EXPORT_SYMBOL_GPL(netif_xdp_propagate);
 
 u32 dev_xdp_prog_id(struct net_device *dev, enum bpf_xdp_mode mode)
 {
Signature · 1 line
2.43.0
Reviewed-by on v2 1/1
#1 lore raw
Quoted 25 lines · Saurabh Sengar

The MANA driver's probe registers netdevice via the following call chain:

mana_probe()
register_netdev()
register_netdevice()

register_netdevice() calls notifier callback for netvsc driver, holding the netdev mutex via netdev_lock_ops().

Further this netvsc notifier callback end up attempting to acquire the same lock again in dev_xdp_propagate() leading to deadlock.

netvsc_netdev_event()
netvsc_vf_setxdp()
dev_xdp_propagate()

This deadlock was not observed so far because net_shaper_ops was never set, and thus the lock was effectively a no-op in this case. Fix this by using netif_xdp_propagate() instead of dev_xdp_propagate() to avoid recursive locking in this path.

Also, clean up the unregistration path by removing the unnecessary call to netvsc_vf_setxdp(), since unregister_netdevice_many_notify() already performs this cleanup via dev_xdp_uninstall().

Quoted 5 lines
Fixes 97246d6d21c2 ("net: hold netdev instance lock during ndo_bpf")

Thanks,
Sundeep

Quoted 50 lines

---
[V2]
- Modified commit message

 drivers/net/hyperv/netvsc_bpf.c | 2 +-
 drivers/net/hyperv/netvsc_drv.c | 2 --
 net/core/dev.c                  | 1 +
 3 files changed, 2 insertions(+), 3 deletions(-)
drivers/net/hyperv/netvsc_bpf.c +1 -1
index e01c5997a551..1dd3755d9e6d 100644
@@ -183,7 +183,7 @@ int netvsc_vf_setxdp(struct net_device *vf_netdev, struct bpf_prog *prog)
 	xdp.command = XDP_SETUP_PROG;
 	xdp.prog = prog;
 
-	ret = dev_xdp_propagate(vf_netdev, &xdp);
+	ret = netif_xdp_propagate(vf_netdev, &xdp);
 
 	if (ret && prog)
 		bpf_prog_put(prog);
drivers/net/hyperv/netvsc_drv.c +0 -2
index d8b169ac0343..ee3aaf9c10e6 100644
@@ -2462,8 +2462,6 @@ static int netvsc_unregister_vf(struct net_device *vf_netdev)
 
 	netdev_info(ndev, "VF unregistering: %s\n", vf_netdev->name);
 
-	netvsc_vf_setxdp(vf_netdev, NULL);
-
 	reinit_completion(&net_device_ctx->vf_add);
 	netdev_rx_handler_unregister(vf_netdev);
 	netdev_upper_dev_unlink(vf_netdev, ndev);
net/core/dev.c +1 -0
index fccf2167b235..8c6c9d7fba26 100644
@@ -9953,6 +9953,7 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf)
 
 	return dev->netdev_ops->ndo_bpf(dev, bpf);
 }
+EXPORT_SYMBOL_GPL(netif_xdp_propagate);
 
 u32 dev_xdp_prog_id(struct net_device *dev, enum bpf_xdp_mode mode)
 {
Signature · 1 line
2.43.0
maintainer
#2 lore raw
In reply to Saurabh Sengar
Quoted 9 lines · On Wed, 21 May 2025 03:25:03 -0700 Saurabh Sengar wrote:

The MANA driver's probe registers netdevice via the following call chain:

mana_probe()
register_netdev()
register_netdevice()

register_netdevice() calls notifier callback for netvsc driver, holding the netdev mutex via netdev_lock_ops().

Quoted 8 lines

Further this netvsc notifier callback end up attempting to acquire the same lock again in dev_xdp_propagate() leading to deadlock.

netvsc_netdev_event()
netvsc_vf_setxdp()
dev_xdp_propagate()

This deadlock was not observed so far because net_shaper_ops was never set,

The lock is on the VF, I think you meant to say that no device you use in Azure is ops locked?

There's also the call to netvsc_register_vf() on probe path, please fix or explain why it doesn't need locking in the commit message.

Signature · 1 line
pw-bot: cr
#3 lore raw
Quoted 17 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 21 May 2025 03:25:03 -0700 Saurabh Sengar wrote:

The MANA driver's probe registers netdevice via the following call chain:

mana_probe()
register_netdev()
register_netdevice()

register_netdevice() calls notifier callback for netvsc driver, holding the netdev mutex via netdev_lock_ops().

Further this netvsc notifier callback end up attempting to acquire the same lock again in dev_xdp_propagate() leading to deadlock.

netvsc_netdev_event()
netvsc_vf_setxdp()
dev_xdp_propagate()

Quoted 7 lines
Quoted 1 line

This deadlock was not observed so far because net_shaper_ops was never set,

The lock is on the VF, I think you meant to say that no device you use in Azure is ops locked?

There's also the call to netvsc_register_vf() on probe path, please fix or explain why it doesn't need locking in the commit message.

This patch specifically addresses the netvsc_register_vf() path only. I omitted the mention of netvsc_register_vf() in the commit message to keep the function path shorter. The full stack trace is provided below:

[   92.542180]  dev_xdp_propagate+0x2c/0x1b0
[   92.542185]  netvsc_vf_setxdp+0x10d/0x180 [hv_netvsc]
[   92.542192]  netvsc_register_vf.part.0+0x179/0x200 [hv_netvsc]
[   92.542196]  netvsc_netdev_event+0x267/0x340 [hv_netvsc]
[   92.542200]  notifier_call_chain+0x5f/0xc0
[   92.542203]  raw_notifier_call_chain+0x16/0x20
[   92.542205]  call_netdevice_notifiers_info+0x52/0xa0
[   92.542209]  register_netdevice+0x7c8/0xaa0
[   92.542211]  register_netdev+0x1f/0x40
[   92.542214]  mana_probe+0x6e2/0x8e0 [mana]
[   92.542220]  mana_gd_probe+0x187/0x220 [mana]

If you prefer I can update the stack trace in commit meesage From:

netvsc_netdev_event()
netvsc_vf_setxdp()
dev_xdp_propagate()

To:

netvsc_netdev_event()
netvsc_register_vf()
netvsc_vf_setxdp()
dev_xdp_propagate()

- Saurabh

Quoted 2 lines
Signature · 1 line
pw-bot: cr
#4 lore raw
In reply to Jakub Kicinski
Quoted 13 lines · Jakub Kicinski
Quoted 12 lines · On Wed, 21 May 2025 03:25:03 -0700 Saurabh Sengar wrote:

The MANA driver's probe registers netdevice via the following call chain:

mana_probe()
register_netdev()
register_netdevice()

register_netdevice() calls notifier callback for netvsc driver, holding the netdev mutex via netdev_lock_ops().

Further this netvsc notifier callback end up attempting to acquire the same lock again in dev_xdp_propagate() leading to deadlock.

Quoted 8 lines
Quoted 5 lines

netvsc_netdev_event()
netvsc_vf_setxdp()
dev_xdp_propagate()

This deadlock was not observed so far because net_shaper_ops was never set,

The lock is on the VF, I think you meant to say that no device you use in Azure is ops locked?

That's right.

Quoted 3 lines

There's also the call to netvsc_register_vf() on probe path, please fix or explain why it doesn't need locking in the commit message.

On rethinking I realize you were referring to the netvsc_probe() path not mana_probe(). Since this lock is effectively a no-op, it doesn't really matter whether it's there or not.

However, I think we can revisit this when we add ops for any of the VFs.

- Saurabh