Skip to contentSkip to conversation

virtio-net: don't busy poll for cvq command

PATCH In review Jason Wang 5 participants 42 messages Jul 20, 2023 – Feb 26, 2024

Subsystems (4)
lore
Share & tools
lore

Jason Wang

author
#0 lore raw

v4 cover virtio-net: don't busy poll for cvq command

Hi all:

The code used to busy poll for cvq command which turns out to have several side effects:

1) infinite poll for buggy devices
2) bad interaction with scheduler

So this series tries to use cond_resched() in the waiting loop. Before doing this we need first make sure the cvq command is not executed in atomic environment, so we need first convert rx mode handling to a workqueue.

Note that, this doesn't try to solve the case that a malicous device may block networking stack (RTNL) or break freezer which requries more thought to gracefully exit and resend the commands.

Please review.

Changes since V3:

- Tweak the comments
- Tweak the changelog to explain the rx mode lost after resuming.
- No functional changes

Changes since V2:

- Don't use interrupt but cond_resched()

Changes since V1:

- use RTNL to synchronize rx mode worker
- use completion for simplicity
- don't try to harden CVQ command

Changes since RFC:

- switch to use BAD_RING in virtio_break_device()
- check virtqueue_is_broken() after being woken up
- use more_used() instead of virtqueue_get_buf() to allow caller to get buffers afterwards
- break the virtio-net device when timeout
- get buffer manually since the virtio core check more_used() instead

Jason Wang (2):
virtio-net: convert rx mode setting to use workqueue
virtio-net: add cond_resched() to the command waiting loop

 drivers/net/virtio_net.c | 59 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 55 insertions(+), 4 deletions(-)
Signature · 1 line
2.39.3
author
#1 lore raw

PATCH v4 1/2 virtio-net: convert rx mode setting to use workqueue

This patch convert rx mode setting to be done in a workqueue, this is a must for allow to sleep when waiting for the cvq command to response since current code is executed under addr spin lock.

Note that we need to disable and flush the workqueue during freeze, this means the rx mode setting is lost after resuming. This is not the bug of this patch as we never try to restore rx mode setting during resume.

---
 drivers/net/virtio_net.c | 55 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 52 insertions(+), 3 deletions(-)
drivers/net/virtio_net.c +52 -3
index d67b36fdba0d..9f3b1d6ac33d 100644
@@ -274,6 +274,12 @@ struct virtnet_info {
 	/* Work struct for config space updates */
 	struct work_struct config_work;
 
+	/* Work struct for setting rx mode */
+	struct work_struct rx_mode_work;
+
+	/* OK to queue work setting RX mode? */
+	bool rx_mode_work_enabled;
+
 	/* Does the affinity hint is set for virtqueues? */
 	bool affinity_hint_set;
 
@@ -397,6 +403,20 @@ static void disable_delayed_refill(struct virtnet_info *vi)
 	spin_unlock_bh(&vi->refill_lock);
 }
 
+static void enable_rx_mode_work(struct virtnet_info *vi)
+{
+	rtnl_lock();
+	vi->rx_mode_work_enabled = true;
+	rtnl_unlock();
+}
+
+static void disable_rx_mode_work(struct virtnet_info *vi)
+{
+	rtnl_lock();
+	vi->rx_mode_work_enabled = false;
+	rtnl_unlock();
+}
+
 static void virtqueue_napi_schedule(struct napi_struct *napi,
 				    struct virtqueue *vq)
 {
@@ -2448,9 +2468,11 @@ static int virtnet_close(struct net_device *dev)
 	return 0;
 }
 
-static void virtnet_set_rx_mode(struct net_device *dev)
+static void virtnet_rx_mode_work(struct work_struct *work)
 {
-	struct virtnet_info *vi = netdev_priv(dev);
+	struct virtnet_info *vi =
+		container_of(work, struct virtnet_info, rx_mode_work);
+	struct net_device *dev = vi->dev;
 	struct scatterlist sg[2];
 	struct virtio_net_ctrl_mac *mac_data;
 	struct netdev_hw_addr *ha;
@@ -2463,6 +2485,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)
 	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_RX))
 		return;
 
+	rtnl_lock();
+
 	vi->ctrl->promisc = ((dev->flags & IFF_PROMISC) != 0);
 	vi->ctrl->allmulti = ((dev->flags & IFF_ALLMULTI) != 0);
 
@@ -2480,14 +2504,19 @@ static void virtnet_set_rx_mode(struct net_device *dev)
 		dev_warn(&dev->dev, "Failed to %sable allmulti mode.\n",
 			 vi->ctrl->allmulti ? "en" : "dis");
 
+	netif_addr_lock_bh(dev);
+
 	uc_count = netdev_uc_count(dev);
 	mc_count = netdev_mc_count(dev);
 	/* MAC filter - use one buffer for both lists */
 	buf = kzalloc(((uc_count + mc_count) * ETH_ALEN) +
 		      (2 * sizeof(mac_data->entries)), GFP_ATOMIC);
 	mac_data = buf;
-	if (!buf)
+	if (!buf) {
+		netif_addr_unlock_bh(dev);
+		rtnl_unlock();
 		return;
+	}
 
 	sg_init_table(sg, 2);
 
@@ -2508,6 +2537,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)
 	netdev_for_each_mc_addr(ha, dev)
 		memcpy(&mac_data->macs[i++][0], ha->addr, ETH_ALEN);
 
+	netif_addr_unlock_bh(dev);
+
 	sg_set_buf(&sg[1], mac_data,
 		   sizeof(mac_data->entries) + (mc_count * ETH_ALEN));
 
@@ -2515,9 +2546,19 @@ static void virtnet_set_rx_mode(struct net_device *dev)
 				  VIRTIO_NET_CTRL_MAC_TABLE_SET, sg))
 		dev_warn(&dev->dev, "Failed to set MAC filter table.\n");
 
+	rtnl_unlock();
+
 	kfree(buf);
 }
 
+static void virtnet_set_rx_mode(struct net_device *dev)
+{
+	struct virtnet_info *vi = netdev_priv(dev);
+
+	if (vi->rx_mode_work_enabled)
+		schedule_work(&vi->rx_mode_work);
+}
+
 static int virtnet_vlan_rx_add_vid(struct net_device *dev,
 				   __be16 proto, u16 vid)
 {
@@ -3286,6 +3327,8 @@ static void virtnet_freeze_down(struct virtio_device *vdev)
 
 	/* Make sure no work handler is accessing the device */
 	flush_work(&vi->config_work);
+	disable_rx_mode_work(vi);
+	flush_work(&vi->rx_mode_work);
 
 	netif_tx_lock_bh(vi->dev);
 	netif_device_detach(vi->dev);
@@ -3308,6 +3351,7 @@ static int virtnet_restore_up(struct virtio_device *vdev)
 	virtio_device_ready(vdev);
 
 	enable_delayed_refill(vi);
+	enable_rx_mode_work(vi);
 
 	if (netif_running(vi->dev)) {
 		err = virtnet_open(vi->dev);
@@ -4121,6 +4165,7 @@ static int virtnet_probe(struct virtio_device *vdev)
 	vdev->priv = vi;
 
 	INIT_WORK(&vi->config_work, virtnet_config_changed_work);
+	INIT_WORK(&vi->rx_mode_work, virtnet_rx_mode_work);
 	spin_lock_init(&vi->refill_lock);
 
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) {
@@ -4229,6 +4274,8 @@ static int virtnet_probe(struct virtio_device *vdev)
 	if (vi->has_rss || vi->has_rss_hash_report)
 		virtnet_init_default_rss(vi);
 
+	enable_rx_mode_work(vi);
+
 	/* serialize netdev register + virtio_device_ready() with ndo_open() */
 	rtnl_lock();
 
@@ -4326,6 +4373,8 @@ static void virtnet_remove(struct virtio_device *vdev)
 
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vi->config_work);
+	disable_rx_mode_work(vi);
+	flush_work(&vi->rx_mode_work);
 
 	unregister_netdev(vi->dev);
 
Signature · 1 line
2.39.3
Reviewed-by on v4 1/2
#2 lore raw

Re: [PATCH net-next v4 1/2] virtio-net: convert rx mode setting to use workqueue

Quoted 11 lines · Jason Wang

This patch convert rx mode setting to be done in a workqueue, this is a must for allow to sleep when waiting for the cvq command to response since current code is executed under addr spin lock.

Note that we need to disable and flush the workqueue during freeze, this means the rx mode setting is lost after resuming. This is not the bug of this patch as we never try to restore rx mode setting during resume.

Quoted 165 lines
---
  drivers/net/virtio_net.c | 55 +++++++++++++++++++++++++++++++++++++---
  1 file changed, 52 insertions(+), 3 deletions(-)
drivers/net/virtio_net.c +52 -4
index d67b36fdba0d..9f3b1d6ac33d 100644
@@ -274,6 +274,12 @@ struct virtnet_info {
         /* Work struct for config space updates */
         struct work_struct config_work;
 
+       /* Work struct for setting rx mode */
+       struct work_struct rx_mode_work;
+
+       /* OK to queue work setting RX mode? */
+       bool rx_mode_work_enabled;
+
         /* Does the affinity hint is set for virtqueues? */
         bool affinity_hint_set;
 
@@ -397,6 +403,20 @@ static void disable_delayed_refill(struct virtnet_info *vi)
         spin_unlock_bh(&vi->refill_lock);
  }
 
+static void enable_rx_mode_work(struct virtnet_info *vi)
+{
+       rtnl_lock();
+       vi->rx_mode_work_enabled = true;
+       rtnl_unlock();
+}
+
+static void disable_rx_mode_work(struct virtnet_info *vi)
+{
+       rtnl_lock();
+       vi->rx_mode_work_enabled = false;
+       rtnl_unlock();
+}
+
  static void virtqueue_napi_schedule(struct napi_struct *napi,
                                     struct virtqueue *vq)
  {
@@ -2448,9 +2468,11 @@ static int virtnet_close(struct net_device *dev)
         return 0;
  }
 
-static void virtnet_set_rx_mode(struct net_device *dev)
+static void virtnet_rx_mode_work(struct work_struct *work)
  {
-       struct virtnet_info *vi = netdev_priv(dev);
+       struct virtnet_info *vi =
+               container_of(work, struct virtnet_info, rx_mode_work);
+       struct net_device *dev = vi->dev;
         struct scatterlist sg[2];
         struct virtio_net_ctrl_mac *mac_data;
         struct netdev_hw_addr *ha;
@@ -2463,6 +2485,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)
         if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_RX))
                 return;
 
+       rtnl_lock();
+
         vi->ctrl->promisc = ((dev->flags & IFF_PROMISC) != 0);
         vi->ctrl->allmulti = ((dev->flags & IFF_ALLMULTI) != 0);
 
@@ -2480,14 +2504,19 @@ static void virtnet_set_rx_mode(struct net_device *dev)
                 dev_warn(&dev->dev, "Failed to %sable allmulti mode.\n",
                          vi->ctrl->allmulti ? "en" : "dis");
 
+       netif_addr_lock_bh(dev);
+
         uc_count = netdev_uc_count(dev);
         mc_count = netdev_mc_count(dev);
         /* MAC filter - use one buffer for both lists */
         buf = kzalloc(((uc_count + mc_count) * ETH_ALEN) +
                       (2 * sizeof(mac_data->entries)), GFP_ATOMIC);
         mac_data = buf;
-       if (!buf)
+       if (!buf) {
+               netif_addr_unlock_bh(dev);
+               rtnl_unlock();
                 return;
+       }
 
         sg_init_table(sg, 2);
 
@@ -2508,6 +2537,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)
         netdev_for_each_mc_addr(ha, dev)
                 memcpy(&mac_data->macs[i++][0], ha->addr, ETH_ALEN);
 
+       netif_addr_unlock_bh(dev);
+
         sg_set_buf(&sg[1], mac_data,
                    sizeof(mac_data->entries) + (mc_count * ETH_ALEN));
 
@@ -2515,9 +2546,19 @@ static void virtnet_set_rx_mode(struct net_device *dev)
                                   VIRTIO_NET_CTRL_MAC_TABLE_SET, sg))
                 dev_warn(&dev->dev, "Failed to set MAC filter table.\n");
 
+       rtnl_unlock();
+
         kfree(buf);
  }
 
+static void virtnet_set_rx_mode(struct net_device *dev)
+{
+       struct virtnet_info *vi = netdev_priv(dev);
+
+       if (vi->rx_mode_work_enabled)
+               schedule_work(&vi->rx_mode_work);
+}
+
  static int virtnet_vlan_rx_add_vid(struct net_device *dev,
                                    __be16 proto, u16 vid)
  {
@@ -3286,6 +3327,8 @@ static void virtnet_freeze_down(struct virtio_device *vdev)
 
         /* Make sure no work handler is accessing the device */
         flush_work(&vi->config_work);
+       disable_rx_mode_work(vi);
+       flush_work(&vi->rx_mode_work);
 
         netif_tx_lock_bh(vi->dev);
         netif_device_detach(vi->dev);
@@ -3308,6 +3351,7 @@ static int virtnet_restore_up(struct virtio_device *vdev)
         virtio_device_ready(vdev);
 
         enable_delayed_refill(vi);
+       enable_rx_mode_work(vi);
 
         if (netif_running(vi->dev)) {
                 err = virtnet_open(vi->dev);
@@ -4121,6 +4165,7 @@ static int virtnet_probe(struct virtio_device *vdev)
         vdev->priv = vi;
 
         INIT_WORK(&vi->config_work, virtnet_config_changed_work);
+       INIT_WORK(&vi->rx_mode_work, virtnet_rx_mode_work);
         spin_lock_init(&vi->refill_lock);
 
         if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) {
@@ -4229,6 +4274,8 @@ static int virtnet_probe(struct virtio_device *vdev)
         if (vi->has_rss || vi->has_rss_hash_report)
                 virtnet_init_default_rss(vi);
 
+       enable_rx_mode_work(vi);
+
         /* serialize netdev register + virtio_device_ready() with ndo_open() */
         rtnl_lock();
 
@@ -4326,6 +4373,8 @@ static void virtnet_remove(struct virtio_device *vdev)
 
         /* Make sure no work handler is accessing the device. */
         flush_work(&vi->config_work);
+       disable_rx_mode_work(vi);
+       flush_work(&vi->rx_mode_work);
 
         unregister_netdev(vi->dev);
 
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#3 lore raw

PATCH v4 2/2 virtio-net: add cond_resched() to the command waiting loop

In reply to Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

---
 drivers/net/virtio_net.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -1
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
 	 * into the hypervisor, so the request should be handled immediately.
 	 */
 	while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-	       !virtqueue_is_broken(vi->cvq))
+	       !virtqueue_is_broken(vi->cvq)) {
+		cond_resched();
 		cpu_relax();
+	}
 
 	return vi->ctrl->status == VIRTIO_NET_OK;
 }
Signature · 1 line
2.39.3
#4 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 16 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

---
  drivers/net/virtio_net.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +0 -0
index 9f3b1d6ac33d..e7533f29b219 100644

drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2320

Quoted 8 lines
drivers/net/virtio_net.c +3 -1
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
          * into the hypervisor, so the request should be handled immediately.
          */
         while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                 cpu_relax();
+       }

The cover letter suggests that this addresses the infinite poll for buggy devices, but I don't see how that is resolved here. This should make it a little nicer to the system, but it still is going to poll forever on a device that has gone catatonic. Is there a reason that I'm missing that we don't have a polling limit here?

sln

Quoted 10 lines
drivers/net/virtio_net.c +0 -1
         return vi->ctrl->status == VIRTIO_NET_OK;
  }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#5 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 32 lines · Shannon Nelson
Quoted 16 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

---
  drivers/net/virtio_net.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +0 -0
index 9f3b1d6ac33d..e7533f29b219 100644

drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2320

Quoted 8 lines
drivers/net/virtio_net.c +3 -1
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
          * into the hypervisor, so the request should be handled immediately.
          */
         while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                 cpu_relax();
+       }

The cover letter suggests that this addresses the infinite poll for buggy devices, but I don't see how that is resolved here. This should make it a little nicer to the system, but it still is going to poll forever on a device that has gone catatonic. Is there a reason that I'm missing that we don't have a polling limit here?

Quoted 1 line

sln

we don't know what the limit would be. but given it's a workqueue now, why does it still have to poll as opposed to blocking?

Quoted 10 lines
Quoted 10 lines
drivers/net/virtio_net.c +0 -1
         return vi->ctrl->status == VIRTIO_NET_OK;
  }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Reviewed-by on v4 2/2
#6 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

In reply to Jason Wang
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 27 lines
---
  drivers/net/virtio_net.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
          * into the hypervisor, so the request should be handled immediately.
          */
         while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                 cpu_relax();
+       }
 
         return vi->ctrl->status == VIRTIO_NET_OK;
  }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#7 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 7 lines · Shannon Nelson
Quoted 6 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 6 lines
Quoted 1 line

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt
2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines
---
  drivers/net/virtio_net.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
          * into the hypervisor, so the request should be handled immediately.
          */
         while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                 cpu_relax();
+       }
 
         return vi->ctrl->status == VIRTIO_NET_OK;
  }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#8 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

Quoted 32 lines

2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines
---
   drivers/net/virtio_net.c | 4 +++-
   1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
           * into the hypervisor, so the request should be handled immediately.
           */
          while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                  cpu_relax();
+       }
 
          return vi->ctrl->status == VIRTIO_NET_OK;
   }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#9 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 27 lines
Quoted 27 lines

2- still handle surprise removal correctly by waking in that case

Quoted 23 lines
Quoted 23 lines
---
   drivers/net/virtio_net.c | 4 +++-
   1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
           * into the hypervisor, so the request should be handled immediately.
           */
          while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                  cpu_relax();
+       }
 
          return vi->ctrl->status == VIRTIO_NET_OK;
   }
--

2.39.3

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#10 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines
---
    drivers/net/virtio_net.c | 4 +++-
    1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
            * into the hypervisor, so the request should be handled immediately.
            */
           while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                   cpu_relax();
+       }
 
           return vi->ctrl->status == VIRTIO_NET_OK;
    }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#11 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 28 lines · Maxime Coquelin
Quoted 25 lines · Michael S. Tsirkin
Quoted 23 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

Quoted 7 lines
Quoted 2 lines

virtio spec does not set any limits on the timing of vq processing.

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines

2- still handle surprise removal correctly by waking in that case

Quoted 23 lines
Quoted 23 lines
---
    drivers/net/virtio_net.c | 4 +++-
    1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
            * into the hypervisor, so the request should be handled immediately.
            */
           while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                   cpu_relax();
+       }
 
           return vi->ctrl->status == VIRTIO_NET_OK;
    }
--

2.39.3

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#12 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the
rtnl lock. In this configuration, it is not possible to kill OVS because
it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio-
net.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

2- still handle surprise removal correctly by waking in that case

Quoted 23 lines
Quoted 23 lines
---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#13 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 6 lines

From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 33 lines

2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines
---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#14 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 52 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 6 lines

From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Quoted 2 lines

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.

Thanks

Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 35 lines

2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines
---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#15 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 53 lines · Jason Wang
Quoted 52 lines · Michael S. Tsirkin
Quoted 49 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Quoted 7 lines
Quoted 2 lines

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.

Thanks

since when does device unregister need to send cvq commands?

Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines

2- still handle surprise removal correctly by waking in that case

Quoted 23 lines
Quoted 23 lines
---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#16 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 58 lines · Michael S. Tsirkin
Quoted 56 lines · Jason Wang
Quoted 54 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 6 lines

From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

Quoted 6 lines
Quoted 4 lines

And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.

Thanks

since when does device unregister need to send cvq commands?

It doesn't do this now. But if we don't process the work under rtnl, we need to synchronize with device unregistering.

Thanks

Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 36 lines

2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines
---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#17 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 62 lines · Jason Wang
Quoted 61 lines · Michael S. Tsirkin
Quoted 59 lines · Jason Wang
Quoted 52 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 6 lines

From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Quoted 2 lines

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Thanks

since when does device unregister need to send cvq commands?

It doesn't do this now. But if we don't process the work under rtnl, we need to synchronize with device unregistering.

Thanks

But what's not easy about it?

Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 36 lines

2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines
---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#18 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 66 lines · Michael S. Tsirkin
Quoted 64 lines · Jason Wang
Quoted 63 lines · Michael S. Tsirkin
Quoted 60 lines · Jason Wang
Quoted 52 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.

Signed-off-by: Jason Wang <[email protected]>

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 6 lines

From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Quoted 2 lines

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.

Thanks

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

since when does device unregister need to send cvq commands?

It doesn't do this now. But if we don't process the work under rtnl, we need to synchronize with device unregistering.

Thanks

But what's not easy about it?

Yes, so I'm wondering if we can simply have the cond_resched() merged as the first step.

And leave the indefinite wait for future investigation?

Thanks

Quoted 42 lines
Quoted 41 lines
Quoted 41 lines
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines

2- still handle surprise removal correctly by waking in that case

Quoted 27 lines
Quoted 27 lines

---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }

return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#19 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

In reply to Michael S. Tsirkin
Quoted 54 lines · Michael S. Tsirkin

External email: Use caution opening links or attachments

Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 6 lines

From the tests I have done, I think it is. With OVS, a reconfiguration is
performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Quoted 4 lines

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

Hi Jason

Excuse me, is there any plan for when will v5 patch series be sent out? Will the v5 patches solve the problem of ctrlvq's infinite poll for buggy devices?

Thanks
Feng

Quoted 40 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

2- still handle surprise removal correctly by waking in that case

Quoted 23 lines
Quoted 23 lines
---
      drivers/net/virtio_net.c | 4 +++-
      1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
              * into the hypervisor, so the request should be handled immediately.
              */
             while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                     cpu_relax();
+       }
 
             return vi->ctrl->status == VIRTIO_NET_OK;
      }
--

2.39.3

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#20 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 64 lines · Feng Liu
Quoted 54 lines · Michael S. Tsirkin

External email: Use caution opening links or attachments

Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

Quoted 5 lines
Quoted 2 lines

I'd like to see a full solution
1- block until interrupt

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Quoted 6 lines

From the tests I have done, I think it is. With OVS, a reconfiguration is
performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Quoted 4 lines

So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.

Quoted 5 lines

Hi Jason

Excuse me, is there any plan for when will v5 patch series be sent out? Will the v5 patches solve the problem of ctrlvq's infinite poll for buggy devices?

We agree to harden VDUSE and,

It would be hard if we try to solve it at the virtio-net level, see the discussions before. It might require support from various layers (e.g networking core etc).

We can use workqueue etc as a mitigation. If Michael is fine with this, I can post v5.

Thanks

Quoted 45 lines

Thanks
Feng

Quoted 40 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

2- still handle surprise removal correctly by waking in that case

Quoted 23 lines
Quoted 23 lines
---
      drivers/net/virtio_net.c | 4 +++-
      1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
              * into the hypervisor, so the request should be handled immediately.
              */
             while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                     cpu_relax();
+       }
 
             return vi->ctrl->status == VIRTIO_NET_OK;
      }
--

2.39.3

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#21 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

In reply to Maxime Coquelin
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines


---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#22 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 66 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

Quoted 8 lines

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software. It's mostly trivial to limit memory consumption, vid's is the
only one where it would make sense to have more than
1 command of a given type outstanding. have a tree
or a bitmap with vids to add/remove?

Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines


---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#23 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Quoted 5 lines
Quoted 2 lines

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.

Do you mean something like:

rtnl_lock();
queue up the work
rtnl_unlock();
return success;

?

Quoted 4 lines

It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.

And rx mode so this implies we will fail any command if the previous work is not finished.

Quoted 2 lines

have a tree
or a bitmap with vids to add/remove?

Probably.

Thanks

Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines


---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#24 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 81 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Quoted 5 lines
Quoted 2 lines

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.

Quoted 8 lines

Do you mean something like:

rtnl_lock();
queue up the work
rtnl_unlock();
return success;

?

yes

Quoted 7 lines
Quoted 4 lines

It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.

And rx mode so this implies we will fail any command if the previous work is not finished.

don't fail it, store it.

Quoted 58 lines
Quoted 2 lines

have a tree
or a bitmap with vids to add/remove?

Probably.

Thanks

Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines


---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#25 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 85 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Quoted 5 lines
Quoted 2 lines

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.

Do you mean something like:

Quoted 8 lines
Quoted 6 lines

rtnl_lock();
queue up the work
rtnl_unlock();
return success;

?

yes

We will lose the error reporting, is it a real problem or not?

Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.

And rx mode so this implies we will fail any command if the previous work is not finished.

don't fail it, store it.

Ok.

Thanks

Quoted 60 lines
Quoted 59 lines
Quoted 3 lines

have a tree
or a bitmap with vids to add/remove?

Probably.

Thanks

Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines


---
     drivers/net/virtio_net.c | 4 +++-
     1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -2
index 9f3b1d6ac33d..e7533f29b219 100644
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
             * into the hypervisor, so the request should be handled immediately.
             */
            while (!virtqueue_get_buf(vi->cvq, &tmp) &&
-              !virtqueue_is_broken(vi->cvq))
+              !virtqueue_is_broken(vi->cvq)) {
+               cond_resched();
                    cpu_relax();
+       }
 
            return vi->ctrl->status == VIRTIO_NET_OK;
     }
--

2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#26 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 91 lines · Jason Wang
Quoted 90 lines · Michael S. Tsirkin
Quoted 88 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang

Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.

Signed-off-by: Jason Wang <[email protected]>

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Quoted 5 lines
Quoted 2 lines

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.

Do you mean something like:

rtnl_lock();
queue up the work
rtnl_unlock();
return success;

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

?

yes

We will lose the error reporting, is it a real problem or not?

Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.

Quoted 76 lines
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.

And rx mode so this implies we will fail any command if the previous work is not finished.

don't fail it, store it.

Ok.

Thanks

Quoted 60 lines
Quoted 59 lines
Quoted 3 lines

have a tree
or a bitmap with vids to add/remove?

Probably.

Thanks

Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }

return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#27 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 95 lines · Michael S. Tsirkin
Quoted 93 lines · Jason Wang
Quoted 92 lines · Michael S. Tsirkin
Quoted 89 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson

On 7/20/23 1:38 AM, Jason Wang wrote:

Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.

Signed-off-by: Jason Wang <[email protected]>

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This still leaves hung processes, but at least it doesn't pin the CPU any
more. Thanks.
Reviewed-by: Shannon Nelson <[email protected]>

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Quoted 5 lines
Quoted 2 lines

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.

Do you mean something like:

rtnl_lock();
queue up the work
rtnl_unlock();
return success;

?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

yes

We will lose the error reporting, is it a real problem or not?

Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.

This problem is not VDUSE specific, DPUs/vDPA may suffer from this as well. This might require more thoughts.

Thanks

Quoted 78 lines
Quoted 77 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.

Quoted 4 lines
Quoted 2 lines

And rx mode so this implies we will fail any command if the previous work is not finished.

don't fail it, store it.

Ok.

Thanks

Quoted 60 lines
Quoted 59 lines
Quoted 3 lines

have a tree
or a bitmap with vids to add/remove?

Probably.

Thanks

Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines

---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }

return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#28 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 100 lines · Jason Wang
Quoted 99 lines · Michael S. Tsirkin
Quoted 96 lines · Jason Wang
Quoted 85 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin

On Thu, Jul 20, 2023 at 01:26:20PM -0700, Shannon Nelson wrote:
On 7/20/23 1:38 AM, Jason Wang wrote:

Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.

Signed-off-by: Jason Wang <[email protected]>

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

This still leaves hung processes, but at least it doesn't pin the CPU any
more. Thanks.
Reviewed-by: Shannon Nelson <[email protected]>

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Quoted 5 lines
Quoted 2 lines

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.

Do you mean something like:

Quoted 8 lines
Quoted 6 lines

rtnl_lock();
queue up the work
rtnl_unlock();
return success;

?

yes

We will lose the error reporting, is it a real problem or not?

Quoted 8 lines
Quoted 3 lines

Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.

This problem is not VDUSE specific, DPUs/vDPA may suffer from this as well. This might require more thoughts.

Thanks

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack. But if they do they can always set this flag too.

Quoted 78 lines
Quoted 78 lines
Quoted 77 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.

Quoted 4 lines
Quoted 2 lines

And rx mode so this implies we will fail any command if the previous work is not finished.

don't fail it, store it.

Ok.

Thanks

Quoted 60 lines
Quoted 59 lines
Quoted 3 lines

have a tree
or a bitmap with vids to add/remove?

Probably.

Thanks

Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines

---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }

return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#29 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 106 lines · Michael S. Tsirkin
Quoted 104 lines · Jason Wang
Quoted 102 lines · Michael S. Tsirkin
Quoted 91 lines · Jason Wang
Quoted 90 lines · Michael S. Tsirkin
Quoted 88 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin

On 7/20/23 23:02, Michael S. Tsirkin wrote:
On Thu, Jul 20, 2023 at 01:26:20PM -0700, Shannon Nelson wrote:
On 7/20/23 1:38 AM, Jason Wang wrote:

Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.

Signed-off-by: Jason Wang <[email protected]>

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

This still leaves hung processes, but at least it doesn't pin the CPU any
more. Thanks.
Reviewed-by: Shannon Nelson <[email protected]>

I'd like to see a full solution
1- block until interrupt

I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?

virtio spec does not set any limits on the timing of vq processing.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Indeed, but I thought the driver could decide it is too long for it.

The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.

if this is a problem we should find a way not to keep rtnl locked indefinitely.

Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.

Quoted 7 lines

From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.

Yeah, basically, any RTNL users would be blocked forever.

And the infinite loop has other side effects like it blocks the freezer to work.

To summarize, there are three issues

1) busy polling
2) breaks freezer
3) hold RTNL during the loop

Quoted 5 lines
Quoted 2 lines

Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.

Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.

Do you mean something like:

rtnl_lock();
queue up the work
rtnl_unlock();
return success;

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

?

yes

We will lose the error reporting, is it a real problem or not?

Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.

Quoted 7 lines
Quoted 4 lines

This problem is not VDUSE specific, DPUs/vDPA may suffer from this as well. This might require more thoughts.

Thanks

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.

Quoted 4 lines
Quoted 2 lines

And rx mode so this implies we will fail any command if the previous work is not finished.

don't fail it, store it.

Ok.

Thanks

Quoted 50 lines
Quoted 50 lines
Quoted 3 lines

have a tree
or a bitmap with vids to add/remove?

Probably.

Thanks

Quoted 42 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2- still handle surprise removal correctly by waking in that case

This is basically what version 1 did?

https://lore.kernel.org/lkml/[email protected]/t/

Thanks

Yes - except the timeout part.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }

return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#30 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 8 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

Quoted 7 lines

pre_set_channels
set_channels
post_set_channels

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Signature · 6 lines
MST

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
author
#31 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Thanks

Quoted 5 lines

--
MST

author
#32 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 34 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Quoted 6 lines

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 8 lines

Thanks

Quoted 5 lines

--
MST

#33 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

I propose we fix this for vduse and assume hardware/firmware is well behaved. Or maybe not well behaved firmware will set the flag losing error reporting ability.

Quoted 8 lines
Quoted 8 lines

Thanks

Quoted 5 lines

--
MST

author
#34 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 3 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects.

So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Quoted 2 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Thanks

Quoted 5 lines

--
MST

#35 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 60 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Ugh more mission creep. not at all my point. vduse should cache values in the driver, until someone manages to change net core to be more friendly to userspace devices.

Quoted 8 lines
Quoted 3 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

/me shrugs

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Thanks

Quoted 5 lines

--
MST

author
#36 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 65 lines · Michael S. Tsirkin
Quoted 63 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 3 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

Quoted 7 lines
Quoted 4 lines

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Ugh more mission creep. not at all my point. vduse should cache values in the driver,

What do you mean by values here? The cvq command?

Thanks

Quoted 29 lines

until someone manages to change
net core to be more friendly to userspace devices.

Quoted 8 lines
Quoted 3 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

/me shrugs

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Thanks

Quoted 5 lines

--
MST

#37 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 71 lines · Jason Wang
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 3 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Quoted 6 lines
Quoted 2 lines

Ugh more mission creep. not at all my point. vduse should cache values in the driver,

What do you mean by values here? The cvq command?

Thanks

The card status generally.

Quoted 29 lines
Quoted 29 lines

until someone manages to change
net core to be more friendly to userspace devices.

Quoted 8 lines
Quoted 3 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

/me shrugs

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Thanks

Quoted 5 lines

--
MST

author
#38 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 73 lines · Michael S. Tsirkin
Quoted 71 lines · Jason Wang
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 3 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Ugh more mission creep. not at all my point. vduse should cache values in the driver,

What do you mean by values here? The cvq command?

Thanks

The card status generally.

Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

until someone manages to change
net core to be more friendly to userspace devices.

Quoted 8 lines
Quoted 3 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

/me shrugs

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Thanks

Quoted 5 lines

--
MST

#39 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 79 lines · Jason Wang
Quoted 78 lines · Michael S. Tsirkin
Quoted 76 lines · Jason Wang
Quoted 65 lines · Michael S. Tsirkin
Quoted 63 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 3 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

Quoted 7 lines
Quoted 4 lines

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Ugh more mission creep. not at all my point. vduse should cache values in the driver,

What do you mean by values here? The cvq command?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Thanks

The card status generally.

Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?

Thanks

vduse will have to process it in kernel.

Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

until someone manages to change
net core to be more friendly to userspace devices.

Quoted 8 lines
Quoted 3 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

/me shrugs

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Thanks

Quoted 5 lines

--
MST

author
#40 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 83 lines · Michael S. Tsirkin
Quoted 81 lines · Jason Wang
Quoted 80 lines · Michael S. Tsirkin
Quoted 77 lines · Jason Wang
Quoted 65 lines · Michael S. Tsirkin
Quoted 63 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 3 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

Quoted 7 lines
Quoted 4 lines

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Ugh more mission creep. not at all my point. vduse should cache values in the driver,

What do you mean by values here? The cvq command?

Thanks

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

The card status generally.

Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?

Thanks

vduse will have to process it in kernel.

Right, that's my understanding (trap CVQ).

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines

until someone manages to change
net core to be more friendly to userspace devices.

Quoted 8 lines
Quoted 3 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

/me shrugs

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Thanks

Quoted 5 lines

--
MST

#41 lore raw

Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop

Quoted 89 lines · Jason Wang
Quoted 88 lines · Michael S. Tsirkin
Quoted 86 lines · Jason Wang
Quoted 73 lines · Michael S. Tsirkin
Quoted 71 lines · Jason Wang
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines

They really shouldn't - any NIC that takes forever to
program will create issues in the networking stack.

Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.

Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.

Quoted 12 lines
Quoted 1 line

But if they do they can always set this flag too.

This may have false negatives and may confuse the management.

Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to

pre_set_channels
set_channels
post_set_channels

Quoted 8 lines
Quoted 3 lines

The device specific part could be done in pre and post without a rtnl lock?

Thanks

Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.

Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.

Quoted 6 lines
Quoted 2 lines

I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.

Michael, any more input on this?

Thanks

Quoted 7 lines

I think this is just mission creep. We are trying to fix vduse - let's do that for starters.

Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.

Yes, this might require support not only in the driver

Quoted 3 lines

I propose we fix this for vduse and assume hardware/firmware is well behaved.

One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Ugh more mission creep. not at all my point. vduse should cache values in the driver,

What do you mean by values here? The cvq command?

Thanks

The card status generally.

Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Thanks

vduse will have to process it in kernel.

Right, that's my understanding (trap CVQ).

Thanks

oh this what you mean by trap. ok. I don't see a new for a timeout then though.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

until someone manages to change
net core to be more friendly to userspace devices.

Quoted 8 lines
Quoted 3 lines

Or maybe not well behaved firmware will
set the flag losing error reporting ability.

This might be hard since it means not only the set but also the get is unreliable.

Thanks

/me shrugs

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Thanks

Quoted 2 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

--
MST

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization