Skip to contentSkip to conversation

Fixing races in probe/remove

PATCH In review Jason Wang 2 participants 13 messages Jun 20, 2022 – Jun 21, 2022

Subsystems (2)
lore
Share & tools
lore

Jason Wang

author
#0 lore raw

cover Fixing races in probe/remove

Hi all:

This series tries to fix races spotted during code review for caif_virtio in probe() and remove().

Compile test only. (I don't have setup to test them).

Please review.

Thanks

Jason Wang (3):
caif_virtio: remove virtqueue_disable_cb() in probe
caif_virtio: fix the race between virtio_device_ready() and ndo_open()
caif_virtio: fix the race between reset and netdev unregister

 drivers/net/caif/caif_virtio.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/3 caif_virtio: remove virtqueue_disable_cb() in probe

This disabling is a just a hint with best effort, there's no guarantee that device doesn't send notification. The driver should survive with that, so let's remove it.

---
 drivers/net/caif/caif_virtio.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/net/caif/caif_virtio.c +0 -3
index 5458f57177a0..c677ded81133 100644
@@ -705,9 +705,6 @@ static int cfv_probe(struct virtio_device *vdev)
 	netdev->needed_headroom = cfv->tx_hr;
 	netdev->needed_tailroom = cfv->tx_tr;
 
-	/* Disable buffer release interrupts unless we have stopped TX queues */
-	virtqueue_disable_cb(cfv->vq_tx);
-
 	netdev->mtu = cfv->mtu - cfv->tx_tr;
 	vdev->priv = cfv;
 
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH 1/3] caif_virtio: remove virtqueue_disable_cb() in probe

Quoted 5 lines · Jason Wang

This disabling is a just a hint with best effort, there's no guarantee that device doesn't send notification. The driver should survive with that, so let's remove it.

I guess, but frankly this change feels gratituous, and just might uncover latent bugs. Which would be fine if we were out to find and fix them, but given this was compile-tested only, I'm not sure that's the case.

Quoted 20 lines
---
 drivers/net/caif/caif_virtio.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/net/caif/caif_virtio.c +0 -3
index 5458f57177a0..c677ded81133 100644
@@ -705,9 +705,6 @@ static int cfv_probe(struct virtio_device *vdev)
 	netdev->needed_headroom = cfv->tx_hr;
 	netdev->needed_tailroom = cfv->tx_tr;
 
-	/* Disable buffer release interrupts unless we have stopped TX queues */
-	virtqueue_disable_cb(cfv->vq_tx);
-
 	netdev->mtu = cfv->mtu - cfv->tx_tr;
 	vdev->priv = cfv;
 
Signature · 1 line
2.25.1
author
#3 lore raw

Re: [PATCH 1/3] caif_virtio: remove virtqueue_disable_cb() in probe

Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang

This disabling is a just a hint with best effort, there's no guarantee that device doesn't send notification. The driver should survive with that, so let's remove it.

Quoted 6 lines
Quoted 1 line

I guess, but frankly this change feels gratituous, and just might uncover latent bugs. Which would be fine if we were out to find and fix them, but given this was compile-tested only, I'm not sure that's the case.

Ok, let me drop this from the next version (no way to test).

Thanks

Quoted 23 lines
Quoted 22 lines

---
 drivers/net/caif/caif_virtio.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/net/caif/caif_virtio.c +0 -4
index 5458f57177a0..c677ded81133 100644
@@ -705,9 +705,6 @@ static int cfv_probe(struct virtio_device *vdev)
      netdev->needed_headroom = cfv->tx_hr;
      netdev->needed_tailroom = cfv->tx_tr;
 
-     /* Disable buffer release interrupts unless we have stopped TX queues */
-     virtqueue_disable_cb(cfv->vq_tx);
-
      netdev->mtu = cfv->mtu - cfv->tx_tr;
      vdev->priv = cfv;
 
--

2.25.1

author
#4 lore raw

PATCH 2/3 caif_virtio: fix the race between virtio_device_ready() and ndo_open()

In reply to Jason Wang

We used to depend on the virtio_device_ready() that is called after probe() by virtio_dev_probe() after netdev registration. This cause a race between ndo_open() and virtio_device_ready(): if ndo_open() is called before virtio_device_ready(), the driver may start to use the device (e.g TX) before DRIVER_OK which violates the spec.

Fixing this by switching to use register_netdevice() and protect the virtio_device_ready() with rtnl_lock() to make sure ndo_open() can only be called after virtio_device_ready().

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/net/caif/caif_virtio.c +9 -1
index c677ded81133..66375bea2fcd 100644
@@ -719,13 +719,21 @@ static int cfv_probe(struct virtio_device *vdev)
 	/* Carrier is off until netdevice is opened */
 	netif_carrier_off(netdev);
 
+	/* serialize netdev register + virtio_device_ready() with ndo_open() */
+	rtnl_lock();
+
 	/* register Netdev */
-	err = register_netdev(netdev);
+	err = register_netdevice(netdev);
 	if (err) {
+		rtnl_unlock();
 		dev_err(&vdev->dev, "Unable to register netdev (%d)\n", err);
 		goto err;
 	}
 
+	virtio_device_ready(vdev);
+
+	rtnl_unlock();
+
 	debugfs_init(cfv);
 
 	return 0;
Signature · 1 line
2.25.1
Acked-by on v1 2/3
#5 lore raw

Re: [PATCH 2/3] caif_virtio: fix the race between virtio_device_ready() and ndo_open()

Quoted 1 line · Jason Wang

We used to depend on the virtio_device_ready() that is called after

"We used to" implies it's not the case currently.
I think you mean "we currently depend".

Quoted 2 lines

probe() by virtio_dev_probe() after netdev registration. This cause

causes

Quoted 6 lines

a race between ndo_open() and virtio_device_ready(): if ndo_open() is called before virtio_device_ready(), the driver may start to use the device (e.g TX) before DRIVER_OK which violates the spec.

Fixing this

Fix this

Quoted 6 lines

by switching to use register_netdevice() and protect the virtio_device_ready() with rtnl_lock() to make sure ndo_open() can only be called after virtio_device_ready().

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
Quoted 33 lines
---
 drivers/net/caif/caif_virtio.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/net/caif/caif_virtio.c +9 -1
index c677ded81133..66375bea2fcd 100644
@@ -719,13 +719,21 @@ static int cfv_probe(struct virtio_device *vdev)
 	/* Carrier is off until netdevice is opened */
 	netif_carrier_off(netdev);
 
+	/* serialize netdev register + virtio_device_ready() with ndo_open() */
+	rtnl_lock();
+
 	/* register Netdev */
-	err = register_netdev(netdev);
+	err = register_netdevice(netdev);
 	if (err) {
+		rtnl_unlock();
 		dev_err(&vdev->dev, "Unable to register netdev (%d)\n", err);
 		goto err;
 	}
 
+	virtio_device_ready(vdev);
+
+	rtnl_unlock();
+
 	debugfs_init(cfv);
 
 	return 0;
Signature · 1 line
2.25.1
author
#6 lore raw

PATCH 3/3 caif_virtio: fix the race between reset and netdev unregister

In reply to Jason Wang

We use to do the following steps during .remove():

static void cfv_remove(struct virtio_device *vdev)

{
	struct cfv_info *cfv = vdev->priv;

	rtnl_lock();
	dev_close(cfv->ndev);
	rtnl_unlock();

	tasklet_kill(&cfv->tx_release_tasklet);
	debugfs_remove_recursive(cfv->debugfs);

	vringh_kiov_cleanup(&cfv->ctx.riov);
	virtio_reset_device(vdev);
	vdev->vringh_config->del_vrhs(cfv->vdev);
	cfv->vr_rx = NULL;
	vdev->config->del_vqs(cfv->vdev);
	unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not device access from both TX and RX side.

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/caif/caif_virtio.c +2 -4
index 66375bea2fcd..a29f9b2df5b1 100644
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
 	struct cfv_info *cfv = vdev->priv;
 
-	rtnl_lock();
-	dev_close(cfv->ndev);
-	rtnl_unlock();
+	/* Make sure NAPI/TX won't try to access the device */
+	unregister_netdev(cfv->ndev);
 
 	tasklet_kill(&cfv->tx_release_tasklet);
 	debugfs_remove_recursive(cfv->debugfs);
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
 	vdev->vringh_config->del_vrhs(cfv->vdev);
 	cfv->vr_rx = NULL;
 	vdev->config->del_vqs(cfv->vdev);
-	unregister_netdev(cfv->ndev);
 }
 
 static struct virtio_device_id id_table[] = {
Signature · 1 line
2.25.1
#7 lore raw

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

Quoted 1 line · Jason Wang

We use to do the following steps during .remove():

We currently do

Quoted 52 lines

static void cfv_remove(struct virtio_device *vdev)

{
	struct cfv_info *cfv = vdev->priv;

	rtnl_lock();
	dev_close(cfv->ndev);
	rtnl_unlock();

	tasklet_kill(&cfv->tx_release_tasklet);
	debugfs_remove_recursive(cfv->debugfs);

	vringh_kiov_cleanup(&cfv->ctx.riov);
	virtio_reset_device(vdev);
	vdev->vringh_config->del_vrhs(cfv->vdev);
	cfv->vr_rx = NULL;
	vdev->config->del_vqs(cfv->vdev);
	unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not device access from both TX and RX side.

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/caif/caif_virtio.c +2 -3
index 66375bea2fcd..a29f9b2df5b1 100644
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
 	struct cfv_info *cfv = vdev->priv;
 
-	rtnl_lock();
-	dev_close(cfv->ndev);
-	rtnl_unlock();
+	/* Make sure NAPI/TX won't try to access the device */
+	unregister_netdev(cfv->ndev);
 
 	tasklet_kill(&cfv->tx_release_tasklet);
 	debugfs_remove_recursive(cfv->debugfs);

drivers/net/caif/caif_virtio.c static void cfv_remove(struct virtio_device *vdev) · line 766

Quoted 5 lines
drivers/net/caif/caif_virtio.c +0 -1
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
 	vdev->vringh_config->del_vrhs(cfv->vdev);
 	cfv->vr_rx = NULL;
 	vdev->config->del_vqs(cfv->vdev);
-	unregister_netdev(cfv->ndev);
 }

This gives me pause, callbacks can now trigger after device has been unregistered. Are we sure this is safe? Won't it be safer to just keep the rtnl_lock around the whole process?

Quoted 3 lines
drivers/net/caif/caif_virtio.c +0 -0
 static struct virtio_device_id id_table[] = {
Signature · 1 line
2.25.1
author
#8 lore raw

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

Quoted 56 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

We use to do the following steps during .remove():

We currently do

Quoted 49 lines

static void cfv_remove(struct virtio_device *vdev)

{
      struct cfv_info *cfv = vdev->priv;

      rtnl_lock();
      dev_close(cfv->ndev);
      rtnl_unlock();

      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);

      vringh_kiov_cleanup(&cfv->ctx.riov);
      virtio_reset_device(vdev);
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
      unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not device access from both TX and RX side.

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/caif/caif_virtio.c +2 -3
index 66375bea2fcd..a29f9b2df5b1 100644
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
      struct cfv_info *cfv = vdev->priv;
 
-     rtnl_lock();
-     dev_close(cfv->ndev);
-     rtnl_unlock();
+     /* Make sure NAPI/TX won't try to access the device */
+     unregister_netdev(cfv->ndev);
 
Quoted 12 lines

drivers/net/caif/caif_virtio.c static void cfv_remove(struct virtio_device *vdev) · line 766

Quoted 8 lines
drivers/net/caif/caif_virtio.c +0 -1
      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
-     unregister_netdev(cfv->ndev);
 }

This gives me pause, callbacks can now trigger after device has been unregistered. Are we sure this is safe?

It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled after tasklet_kill(). I can add a comment to explain this.

Quoted 2 lines

Won't it be safer to just keep the rtnl_lock around the whole process?

It looks to me we rtnl_lock can't help in synchronizing with the callbacks, anything I miss?

Thanks

Quoted 5 lines
Quoted 4 lines
drivers/net/caif/caif_virtio.c +0 -1
 static struct virtio_device_id id_table[] = {
--

2.25.1

#9 lore raw

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

Quoted 70 lines · Jason Wang
Quoted 56 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

We use to do the following steps during .remove():

We currently do

Quoted 49 lines

static void cfv_remove(struct virtio_device *vdev)

{
      struct cfv_info *cfv = vdev->priv;

      rtnl_lock();
      dev_close(cfv->ndev);
      rtnl_unlock();

      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);

      vringh_kiov_cleanup(&cfv->ctx.riov);
      virtio_reset_device(vdev);
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
      unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not device access from both TX and RX side.

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/caif/caif_virtio.c +2 -3
index 66375bea2fcd..a29f9b2df5b1 100644
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
      struct cfv_info *cfv = vdev->priv;
 
-     rtnl_lock();
-     dev_close(cfv->ndev);
-     rtnl_unlock();
+     /* Make sure NAPI/TX won't try to access the device */
+     unregister_netdev(cfv->ndev);
 
Quoted 12 lines

drivers/net/caif/caif_virtio.c static void cfv_remove(struct virtio_device *vdev) · line 766

Quoted 8 lines
drivers/net/caif/caif_virtio.c +0 -1
      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
-     unregister_netdev(cfv->ndev);
 }

This gives me pause, callbacks can now trigger after device has been unregistered. Are we sure this is safe?

Quoted 2 lines

It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled after tasklet_kill(). I can add a comment to explain this.

that waits for outstanding tasklets but does it really prevent future ones?

Quoted 7 lines
Quoted 2 lines

Won't it be safer to just keep the rtnl_lock around the whole process?

It looks to me we rtnl_lock can't help in synchronizing with the callbacks, anything I miss?

Thanks

good point.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
drivers/net/caif/caif_virtio.c +0 -1
 static struct virtio_device_id id_table[] = {
--

2.25.1

author
#10 lore raw

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

Quoted 72 lines · Michael S. Tsirkin
Quoted 70 lines · Jason Wang
Quoted 68 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

We use to do the following steps during .remove():

We currently do

Quoted 52 lines

static void cfv_remove(struct virtio_device *vdev)

{
      struct cfv_info *cfv = vdev->priv;

      rtnl_lock();
      dev_close(cfv->ndev);
      rtnl_unlock();

      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);

      vringh_kiov_cleanup(&cfv->ctx.riov);
      virtio_reset_device(vdev);
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
      unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not device access from both TX and RX side.

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/caif/caif_virtio.c +2 -3
index 66375bea2fcd..a29f9b2df5b1 100644
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
      struct cfv_info *cfv = vdev->priv;
 
-     rtnl_lock();
-     dev_close(cfv->ndev);
-     rtnl_unlock();
+     /* Make sure NAPI/TX won't try to access the device */
+     unregister_netdev(cfv->ndev);
 
      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);

drivers/net/caif/caif_virtio.c static void cfv_remove(struct virtio_device *vdev) · line 766

Quoted 5 lines
drivers/net/caif/caif_virtio.c +0 -1
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
-     unregister_netdev(cfv->ndev);
 }

This gives me pause, callbacks can now trigger after device has been unregistered. Are we sure this is safe?

Quoted 5 lines
Quoted 2 lines

It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled after tasklet_kill(). I can add a comment to explain this.

that waits for outstanding tasklets but does it really prevent future ones?

I think so, it tries to test and set TASKLET_STATE_SCHED which blocks the future scheduling of a tasklet.

Thanks

Quoted 18 lines
Quoted 8 lines
Quoted 3 lines

Won't it be safer to just keep the rtnl_lock around the whole process?

It looks to me we rtnl_lock can't help in synchronizing with the callbacks, anything I miss?

Thanks

good point.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
drivers/net/caif/caif_virtio.c +0 -1
 static struct virtio_device_id id_table[] = {
--

2.25.1

#11 lore raw

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

Quoted 76 lines · Jason Wang
Quoted 75 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 56 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

We use to do the following steps during .remove():

We currently do

Quoted 49 lines

static void cfv_remove(struct virtio_device *vdev)

{
      struct cfv_info *cfv = vdev->priv;

      rtnl_lock();
      dev_close(cfv->ndev);
      rtnl_unlock();

      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);

      vringh_kiov_cleanup(&cfv->ctx.riov);
      virtio_reset_device(vdev);
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
      unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not device access from both TX and RX side.

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/caif/caif_virtio.c +2 -3
index 66375bea2fcd..a29f9b2df5b1 100644
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
      struct cfv_info *cfv = vdev->priv;
 
-     rtnl_lock();
-     dev_close(cfv->ndev);
-     rtnl_unlock();
+     /* Make sure NAPI/TX won't try to access the device */
+     unregister_netdev(cfv->ndev);
 
Quoted 12 lines

drivers/net/caif/caif_virtio.c static void cfv_remove(struct virtio_device *vdev) · line 766

Quoted 8 lines
drivers/net/caif/caif_virtio.c +0 -1
      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
-     unregister_netdev(cfv->ndev);
 }

This gives me pause, callbacks can now trigger after device has been unregistered. Are we sure this is safe?

It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled after tasklet_kill(). I can add a comment to explain this.

Quoted 7 lines
Quoted 2 lines

that waits for outstanding tasklets but does it really prevent future ones?

I think so, it tries to test and set TASKLET_STATE_SCHED which blocks the future scheduling of a tasklet.

Thanks

But then in the end it clears it, does it not?

Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
Quoted 3 lines

Won't it be safer to just keep the rtnl_lock around the whole process?

It looks to me we rtnl_lock can't help in synchronizing with the callbacks, anything I miss?

Thanks

good point.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
drivers/net/caif/caif_virtio.c +0 -1
 static struct virtio_device_id id_table[] = {
--

2.25.1

author
#12 lore raw

Re: [PATCH 3/3] caif_virtio: fix the race between reset and netdev unregister

Quoted 81 lines · Michael S. Tsirkin
Quoted 79 lines · Jason Wang
Quoted 77 lines · Michael S. Tsirkin
Quoted 70 lines · Jason Wang
Quoted 56 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

We use to do the following steps during .remove():

We currently do

Quoted 49 lines

static void cfv_remove(struct virtio_device *vdev)

{
      struct cfv_info *cfv = vdev->priv;

      rtnl_lock();
      dev_close(cfv->ndev);
      rtnl_unlock();

      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);

      vringh_kiov_cleanup(&cfv->ctx.riov);
      virtio_reset_device(vdev);
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
      unregister_netdev(cfv->ndev);
}

This is racy since device could be re-opened after dev_close() but before unregister_netdevice():

1) RX vringh is cleaned before resetting the device, rx callbacks that is called after the vringh_kiov_cleanup() will result a UAF
2) Network stack can still try to use TX virtqueue even if it has been deleted after dev_vqs()

Fixing this by unregistering the network device first to make sure not device access from both TX and RX side.

Fixes 0d2e1a2926b18 ("caif_virtio: Introduce caif over virtio")
---
 drivers/net/caif/caif_virtio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/caif/caif_virtio.c +2 -3
index 66375bea2fcd..a29f9b2df5b1 100644
@@ -752,9 +752,8 @@ static void cfv_remove(struct virtio_device *vdev)
 {
      struct cfv_info *cfv = vdev->priv;
 
-     rtnl_lock();
-     dev_close(cfv->ndev);
-     rtnl_unlock();
+     /* Make sure NAPI/TX won't try to access the device */
+     unregister_netdev(cfv->ndev);
 
Quoted 12 lines

drivers/net/caif/caif_virtio.c static void cfv_remove(struct virtio_device *vdev) · line 766

Quoted 8 lines
drivers/net/caif/caif_virtio.c +0 -1
      tasklet_kill(&cfv->tx_release_tasklet);
      debugfs_remove_recursive(cfv->debugfs);
@@ -764,7 +763,6 @@ static void cfv_remove(struct virtio_device *vdev)
      vdev->vringh_config->del_vrhs(cfv->vdev);
      cfv->vr_rx = NULL;
      vdev->config->del_vqs(cfv->vdev);
-     unregister_netdev(cfv->ndev);
 }

This gives me pause, callbacks can now trigger after device has been unregistered. Are we sure this is safe?

Quoted 2 lines

It looks safe, for RX NAPI is disabled. For TX, tasklet is disabled after tasklet_kill(). I can add a comment to explain this.

that waits for outstanding tasklets but does it really prevent future ones?

Quoted 6 lines
Quoted 4 lines

I think so, it tries to test and set TASKLET_STATE_SCHED which blocks the future scheduling of a tasklet.

Thanks

But then in the end it clears it, does it not?

Right, so we need to reset before taskset_kill().

Thanks

Quoted 20 lines
Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Quoted 4 lines

Won't it be safer to just keep the rtnl_lock around the whole process?

It looks to me we rtnl_lock can't help in synchronizing with the callbacks, anything I miss?

Thanks

good point.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
drivers/net/caif/caif_virtio.c +0 -1
 static struct virtio_device_id id_table[] = {
--

2.25.1