virtio-net: don't busy poll for cvq command
Subsystems (4)
v4 cover virtio-net: don't busy poll for cvq command
Hi all:
The code used to busy poll for cvq command which turns out to have several side effects:
1) infinite poll for buggy devices
2) bad interaction with scheduler
So this series tries to use cond_resched() in the waiting loop. Before doing this we need first make sure the cvq command is not executed in atomic environment, so we need first convert rx mode handling to a workqueue.
Note that, this doesn't try to solve the case that a malicous device may block networking stack (RTNL) or break freezer which requries more thought to gracefully exit and resend the commands.
Please review.
Changes since V3:
- Tweak the comments
- Tweak the changelog to explain the rx mode lost after resuming.
- No functional changes
Changes since V2:
- Don't use interrupt but cond_resched()
Changes since V1:
- use RTNL to synchronize rx mode worker
- use completion for simplicity
- don't try to harden CVQ command
Changes since RFC:
- switch to use BAD_RING in virtio_break_device()
- check virtqueue_is_broken() after being woken up
- use more_used() instead of virtqueue_get_buf() to allow caller to get buffers afterwards
- break the virtio-net device when timeout
- get buffer manually since the virtio core check more_used() instead
Jason Wang (2):
virtio-net: convert rx mode setting to use workqueue
virtio-net: add cond_resched() to the command waiting loop
drivers/net/virtio_net.c | 59 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 55 insertions(+), 4 deletions(-)
Signature · 1 line
2.39.3
PATCH v4 1/2 virtio-net: convert rx mode setting to use workqueue
This patch convert rx mode setting to be done in a workqueue, this is a must for allow to sleep when waiting for the cvq command to response since current code is executed under addr spin lock.
Note that we need to disable and flush the workqueue during freeze, this means the rx mode setting is lost after resuming. This is not the bug of this patch as we never try to restore rx mode setting during resume.
--- drivers/net/virtio_net.c | 55 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 3 deletions(-)
drivers/net/virtio_net.c +52 -3
@@ -274,6 +274,12 @@ struct virtnet_info {/* Work struct for config space updates */ struct work_struct config_work; +/* Work struct for setting rx mode */ +struct work_struct rx_mode_work; + +/* OK to queue work setting RX mode? */ +bool rx_mode_work_enabled; +/* Does the affinity hint is set for virtqueues? */ bool affinity_hint_set;
@@ -397,6 +403,20 @@ static void disable_delayed_refill(struct virtnet_info *vi)spin_unlock_bh (&vi->refill_lock); } +static void enable_rx_mode_work (struct virtnet_info *vi) +{ +rtnl_lock (); + vi->rx_mode_work_enabled = true; +rtnl_unlock (); +} + +static void disable_rx_mode_work (struct virtnet_info *vi) +{ +rtnl_lock (); + vi->rx_mode_work_enabled = false; +rtnl_unlock (); +} +static void virtqueue_napi_schedule (struct napi_struct *napi,struct virtqueue *vq) {
@@ -2448,9 +2468,11 @@ static int virtnet_close(struct net_device *dev)return 0 ; } -static void virtnet_set_rx_mode (struct net_device *dev) +static void virtnet_rx_mode_work (struct work_struct *work) { -struct virtnet_info *vi =netdev_priv (dev); +struct virtnet_info *vi = + container_of(work, struct virtnet_info, rx_mode_work); +struct net_device *dev = vi->dev;struct scatterlist sg[2 ];struct virtio_net_ctrl_mac *mac_data;struct netdev_hw_addr *ha;
@@ -2463,6 +2485,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)if (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_RX ))return ; +rtnl_lock (); + vi->ctrl->promisc = ((dev->flags &IFF_PROMISC ) !=0 ); vi->ctrl->allmulti = ((dev->flags &IFF_ALLMULTI ) !=0 );
@@ -2480,14 +2504,19 @@ static void virtnet_set_rx_mode(struct net_device *dev)dev_warn (&dev->dev,"Failed to %sable allmulti mode.\n" , vi->ctrl->allmulti ?"en" :"dis" ); +netif_addr_lock_bh (dev); + uc_count =netdev_uc_count (dev); mc_count =netdev_mc_count (dev);/* MAC filter - use one buffer for both lists */ buf =kzalloc (((uc_count + mc_count) *ETH_ALEN ) + (2 *sizeof (mac_data->entries)),GFP_ATOMIC ); mac_data = buf; -if (!buf) +if (!buf) { +netif_addr_unlock_bh (dev); +rtnl_unlock ();return ; + }sg_init_table (sg,2 );
@@ -2508,6 +2537,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)netdev_for_each_mc_addr (ha, dev)memcpy (&mac_data->macs[i++][0 ], ha->addr,ETH_ALEN ); +netif_addr_unlock_bh (dev); +sg_set_buf (&sg[1 ], mac_data,sizeof (mac_data->entries) + (mc_count *ETH_ALEN ));
@@ -2515,9 +2546,19 @@ static void virtnet_set_rx_mode(struct net_device *dev) VIRTIO_NET_CTRL_MAC_TABLE_SET, sg))dev_warn (&dev->dev,"Failed to set MAC filter table.\n" ); +rtnl_unlock (); +kfree (buf); } +static void virtnet_set_rx_mode (struct net_device *dev) +{ +struct virtnet_info *vi =netdev_priv (dev); + +if (vi->rx_mode_work_enabled) +schedule_work (&vi->rx_mode_work); +} +static int virtnet_vlan_rx_add_vid (struct net_device *dev,__be16 proto,u16 vid) {
@@ -3286,6 +3327,8 @@ static void virtnet_freeze_down(struct virtio_device *vdev)/* Make sure no work handler is accessing the device */ flush_work (&vi->config_work); +disable_rx_mode_work (vi); +flush_work (&vi->rx_mode_work);netif_tx_lock_bh (vi->dev);netif_device_detach (vi->dev);
@@ -3308,6 +3351,7 @@ static int virtnet_restore_up(struct virtio_device *vdev)virtio_device_ready (vdev);enable_delayed_refill (vi); +enable_rx_mode_work (vi);if (netif_running (vi->dev)) { err =virtnet_open (vi->dev);
@@ -4121,6 +4165,7 @@ static int virtnet_probe(struct virtio_device *vdev) vdev->priv = vi;INIT_WORK (&vi->config_work, virtnet_config_changed_work); +INIT_WORK (&vi->rx_mode_work, virtnet_rx_mode_work);spin_lock_init (&vi->refill_lock);if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) {
@@ -4229,6 +4274,8 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->has_rss || vi->has_rss_hash_report)virtnet_init_default_rss (vi); +enable_rx_mode_work (vi); +/* serialize netdev register + virtio_device_ready() with ndo_open() */ rtnl_lock ();
@@ -4326,6 +4373,8 @@ static void virtnet_remove(struct virtio_device *vdev)/* Make sure no work handler is accessing the device. */ flush_work (&vi->config_work); +disable_rx_mode_work (vi); +flush_work (&vi->rx_mode_work);unregister_netdev (vi->dev);
Signature · 1 line
2.39.3
Re: [PATCH net-next v4 1/2] virtio-net: convert rx mode setting to use workqueue
Quoted 11 lines · Jason Wang
This patch convert rx mode setting to be done in a workqueue, this is a must for allow to sleep when waiting for the cvq command to response since current code is executed under addr spin lock.
Note that we need to disable and flush the workqueue during freeze, this means the rx mode setting is lost after resuming. This is not the bug of this patch as we never try to restore rx mode setting during resume.
Signed-off-by Jason Wang <[email protected]>
Quoted 165 lines
--- drivers/net/virtio_net.c | 55 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 3 deletions(-)drivers/net/virtio_net.c +52
-4@@ -274,6 +274,12 @@ struct virtnet_info {/* Work struct for config space updates */ struct work_struct config_work; +/* Work struct for setting rx mode */ +struct work_struct rx_mode_work; + +/* OK to queue work setting RX mode? */ +bool rx_mode_work_enabled; +/* Does the affinity hint is set for virtqueues? */ bool affinity_hint_set;@@ -397,6 +403,20 @@ static void disable_delayed_refill(struct virtnet_info *vi)spin_unlock_bh (&vi->refill_lock); } +static void enable_rx_mode_work (struct virtnet_info *vi) +{ +rtnl_lock (); + vi->rx_mode_work_enabled = true; +rtnl_unlock (); +} + +static void disable_rx_mode_work (struct virtnet_info *vi) +{ +rtnl_lock (); + vi->rx_mode_work_enabled = false; +rtnl_unlock (); +} +static void virtqueue_napi_schedule (struct napi_struct *napi,struct virtqueue *vq) {@@ -2448,9 +2468,11 @@ static int virtnet_close(struct net_device *dev)return 0 ; } -static void virtnet_set_rx_mode (struct net_device *dev) +static void virtnet_rx_mode_work (struct work_struct *work) { -struct virtnet_info *vi =netdev_priv (dev); +struct virtnet_info *vi = + container_of(work, struct virtnet_info, rx_mode_work); +struct net_device *dev = vi->dev;struct scatterlist sg[2 ];struct virtio_net_ctrl_mac *mac_data;struct netdev_hw_addr *ha;@@ -2463,6 +2485,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)if (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_RX ))return ; +rtnl_lock (); + vi->ctrl->promisc = ((dev->flags &IFF_PROMISC ) !=0 ); vi->ctrl->allmulti = ((dev->flags &IFF_ALLMULTI ) !=0 );@@ -2480,14 +2504,19 @@ static void virtnet_set_rx_mode(struct net_device *dev)dev_warn (&dev->dev,"Failed to %sable allmulti mode.\n" , vi->ctrl->allmulti ?"en" :"dis" ); +netif_addr_lock_bh (dev); + uc_count =netdev_uc_count (dev); mc_count =netdev_mc_count (dev);/* MAC filter - use one buffer for both lists */ buf =kzalloc (((uc_count + mc_count) *ETH_ALEN ) + (2 *sizeof (mac_data->entries)),GFP_ATOMIC ); mac_data = buf; -if (!buf) +if (!buf) { +netif_addr_unlock_bh (dev); +rtnl_unlock ();return ; + }sg_init_table (sg,2 );@@ -2508,6 +2537,8 @@ static void virtnet_set_rx_mode(struct net_device *dev)netdev_for_each_mc_addr (ha, dev)memcpy (&mac_data->macs[i++][0 ], ha->addr,ETH_ALEN ); +netif_addr_unlock_bh (dev); +sg_set_buf (&sg[1 ], mac_data,sizeof (mac_data->entries) + (mc_count *ETH_ALEN ));@@ -2515,9 +2546,19 @@ static void virtnet_set_rx_mode(struct net_device *dev) VIRTIO_NET_CTRL_MAC_TABLE_SET, sg))dev_warn (&dev->dev,"Failed to set MAC filter table.\n" ); +rtnl_unlock (); +kfree (buf); } +static void virtnet_set_rx_mode (struct net_device *dev) +{ +struct virtnet_info *vi =netdev_priv (dev); + +if (vi->rx_mode_work_enabled) +schedule_work (&vi->rx_mode_work); +} +static int virtnet_vlan_rx_add_vid (struct net_device *dev,__be16 proto,u16 vid) {@@ -3286,6 +3327,8 @@ static void virtnet_freeze_down(struct virtio_device *vdev)/* Make sure no work handler is accessing the device */ flush_work (&vi->config_work); +disable_rx_mode_work (vi); +flush_work (&vi->rx_mode_work);netif_tx_lock_bh (vi->dev);netif_device_detach (vi->dev);@@ -3308,6 +3351,7 @@ static int virtnet_restore_up(struct virtio_device *vdev)virtio_device_ready (vdev);enable_delayed_refill (vi); +enable_rx_mode_work (vi);if (netif_running (vi->dev)) { err =virtnet_open (vi->dev);@@ -4121,6 +4165,7 @@ static int virtnet_probe(struct virtio_device *vdev) vdev->priv = vi;INIT_WORK (&vi->config_work, virtnet_config_changed_work); +INIT_WORK (&vi->rx_mode_work, virtnet_rx_mode_work);spin_lock_init (&vi->refill_lock);if (virtio_has_feature (vdev,VIRTIO_NET_F_MRG_RXBUF )) {@@ -4229,6 +4274,8 @@ static int virtnet_probe(struct virtio_device *vdev)if (vi->has_rss || vi->has_rss_hash_report)virtnet_init_default_rss (vi); +enable_rx_mode_work (vi); +/* serialize netdev register + virtio_device_ready() with ndo_open() */ rtnl_lock ();@@ -4326,6 +4373,8 @@ static void virtnet_remove(struct virtio_device *vdev)/* Make sure no work handler is accessing the device. */ flush_work (&vi->config_work); +disable_rx_mode_work (vi); +flush_work (&vi->rx_mode_work);unregister_netdev (vi->dev); --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH v4 2/2 virtio-net: add cond_resched() to the command waiting loop
In reply to Jason WangAdding cond_resched() to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/virtio_net.c +3 -1
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; }
Signature · 1 line
2.39.3
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 16 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +0
-0
drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2320
Quoted 8 lines
drivers/net/virtio_net.c +3
-1@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }
The cover letter suggests that this addresses the infinite poll for buggy devices, but I don't see how that is resolved here. This should make it a little nicer to the system, but it still is going to poll forever on a device that has gone catatonic. Is there a reason that I'm missing that we don't have a polling limit here?
sln
Quoted 10 lines
drivers/net/virtio_net.c +0
-1return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 32 lines · Shannon Nelson
Quoted 16 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +0
-0drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2320
Quoted 8 lines
drivers/net/virtio_net.c +3
-1@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }The cover letter suggests that this addresses the infinite poll for buggy devices, but I don't see how that is resolved here. This should make it a little nicer to the system, but it still is going to poll forever on a device that has gone catatonic. Is there a reason that I'm missing that we don't have a polling limit here?
Quoted 1 line
sln
we don't know what the limit would be. but given it's a workqueue now, why does it still have to poll as opposed to blocking?
Quoted 10 lines
Quoted 10 lines
drivers/net/virtio_net.c +0
-1return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
In reply to Jason WangQuoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 7 lines · Shannon Nelson
Quoted 6 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.
Quoted 6 lines
Quoted 1 line
Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>
I'd like to see a full solution
1- block until interrupt
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interrupt
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?
Quoted 32 lines
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>
Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?
virtio spec does not set any limits on the timing of vq processing.
Quoted 27 lines
Quoted 27 lines
2- still handle surprise removal correctly by waking in that case
Quoted 23 lines
Quoted 23 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 28 lines · Maxime Coquelin
Quoted 25 lines · Michael S. Tsirkin
Quoted 23 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?
Quoted 7 lines
Quoted 2 lines
virtio spec does not set any limits on the timing of vq processing.
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
2- still handle surprise removal correctly by waking in that case
Quoted 23 lines
Quoted 23 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an ioctl() that tries to take the
rtnl lock. In this configuration, it is not possible to kill OVS because
it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio-
net.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
2- still handle surprise removal correctly by waking in that case
Quoted 23 lines
Quoted 23 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 6 lines
From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an
ioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.
So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 33 lines
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 52 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 6 lines
From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an
ioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.
Quoted 2 lines
So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.
Thanks
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 35 lines
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 53 lines · Jason Wang
Quoted 52 lines · Michael S. Tsirkin
Quoted 49 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an
ioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.
Quoted 7 lines
Quoted 2 lines
So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.
Thanks
since when does device unregister need to send cvq commands?
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
2- still handle surprise removal correctly by waking in that case
Quoted 23 lines
Quoted 23 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 58 lines · Michael S. Tsirkin
Quoted 56 lines · Jason Wang
Quoted 54 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 6 lines
From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an
ioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
Quoted 6 lines
Quoted 4 lines
And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.
Thanks
since when does device unregister need to send cvq commands?
It doesn't do this now. But if we don't process the work under rtnl, we need to synchronize with device unregistering.
Thanks
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 36 lines
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 62 lines · Jason Wang
Quoted 61 lines · Michael S. Tsirkin
Quoted 59 lines · Jason Wang
Quoted 52 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 6 lines
From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an
ioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Quoted 2 lines
So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Thanks
since when does device unregister need to send cvq commands?
It doesn't do this now. But if we don't process the work under rtnl, we need to synchronize with device unregistering.
Thanks
But what's not easy about it?
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 36 lines
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 66 lines · Michael S. Tsirkin
Quoted 64 lines · Jason Wang
Quoted 63 lines · Michael S. Tsirkin
Quoted 60 lines · Jason Wang
Quoted 52 lines · Michael S. Tsirkin
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.Signed-off-by: Jason Wang <[email protected]>
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 6 lines
From the tests I have done, I think it is. With OVS, a reconfiguration is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing an
ioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Quoted 2 lines
So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
And it needs to sync with rtnl somehow, e.g device unregistering which seems not easy.
Thanks
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
since when does device unregister need to send cvq commands?
It doesn't do this now. But if we don't process the work under rtnl, we need to synchronize with device unregistering.
Thanks
But what's not easy about it?
Yes, so I'm wondering if we can simply have the cond_resched() merged as the first step.
And leave the indefinite wait for future investigation?
Thanks
Quoted 42 lines
Quoted 41 lines
Quoted 41 lines
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
2- still handle surprise removal correctly by waking in that case
Quoted 27 lines
Quoted 27 lines
---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
In reply to Michael S. TsirkinQuoted 54 lines · Michael S. Tsirkin
External email: Use caution opening links or attachments
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 6 lines
From the tests I have done, I think it is. With OVS, a reconfiguration is
performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.
Quoted 4 lines
So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
Hi Jason
Excuse me, is there any plan for when will v5 patch series be sent out? Will the v5 patches solve the problem of ctrlvq's infinite poll for buggy devices?
Thanks
Feng
Quoted 40 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
2- still handle surprise removal correctly by waking in that case
Quoted 23 lines
Quoted 23 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 64 lines · Feng Liu
Quoted 54 lines · Michael S. Tsirkin
External email: Use caution opening links or attachments
Quoted 43 lines · Maxime Coquelin
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Maxime Coquelin
Quoted 27 lines · Michael S. Tsirkin
Quoted 18 lines · Maxime Coquelin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>Quoted 5 lines
Quoted 2 lines
I'd like to see a full solution
1- block until interruptWould it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Quoted 6 lines
From the tests I have done, I think it is. With OVS, a reconfiguration is
performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Quoted 4 lines
So for sure, we can queue up the work and process it later. The somewhat tricky part is limiting the memory consumption.
Quoted 5 lines
Hi Jason
Excuse me, is there any plan for when will v5 patch series be sent out? Will the v5 patches solve the problem of ctrlvq's infinite poll for buggy devices?
We agree to harden VDUSE and,
It would be hard if we try to solve it at the virtio-net level, see the discussions before. It might require support from various layers (e.g networking core etc).
We can use workqueue etc as a mitigation. If Michael is fine with this, I can post v5.
Thanks
Quoted 45 lines
Thanks
FengQuoted 40 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
2- still handle surprise removal correctly by waking in that case
Quoted 23 lines
Quoted 23 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
In reply to Maxime CoquelinQuoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interrupt
I remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.
Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loop
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even virtnet_restore() itself may try to hold the lock.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 66 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
Quoted 8 lines
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loopSolving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.
Yep. So my feeling currently is, the only real fix is to actually queue up the work in software. It's mostly trivial to limit memory consumption, vid's is the
only one where it would make sense to have more than
1 command of a given type outstanding. have a tree
or a bitmap with vids to add/remove?
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loop
Quoted 5 lines
Quoted 2 lines
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.
Do you mean something like:
rtnl_lock();
queue up the workrtnl_unlock();
return success;
?
Quoted 4 lines
It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.
And rx mode so this implies we will fail any command if the previous work is not finished.
Quoted 2 lines
have a tree
or a bitmap with vids to add/remove?
Probably.
Thanks
Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 81 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loopQuoted 5 lines
Quoted 2 lines
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.
Quoted 8 lines
Do you mean something like:
rtnl_lock();
queue up the workrtnl_unlock();
return success;?
yes
Quoted 7 lines
Quoted 4 lines
It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.And rx mode so this implies we will fail any command if the previous work is not finished.
don't fail it, store it.
Quoted 58 lines
Quoted 2 lines
have a tree
or a bitmap with vids to add/remove?Probably.
Thanks
Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 85 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding
cond_resched()to the command waiting loop for a better co-operation with the scheduler. This allows to give CPU a breath to run other task(workqueue) instead of busy looping when preemption is not allowed on a device whose CVQ might be slow.Signed-off-by Jason Wang <[email protected]>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loopQuoted 5 lines
Quoted 2 lines
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.
Do you mean something like:
Quoted 8 lines
Quoted 6 lines
rtnl_lock();
queue up the workrtnl_unlock();
return success;?
yes
We will lose the error reporting, is it a real problem or not?
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines
It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.And rx mode so this implies we will fail any command if the previous work is not finished.
don't fail it, store it.
Ok.
Thanks
Quoted 60 lines
Quoted 59 lines
Quoted 3 lines
have a tree
or a bitmap with vids to add/remove?Probably.
Thanks
Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
--- drivers/net/virtio_net.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/virtio_net.c +3
-2@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +cond_resched ();cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.39.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 91 lines · Jason Wang
Quoted 90 lines · Michael S. Tsirkin
Quoted 88 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
Quoted 7 lines · Jason Wang
Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.Signed-off-by: Jason Wang <[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any more. Thanks.
Reviewed-by Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loopQuoted 5 lines
Quoted 2 lines
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.
Do you mean something like:
rtnl_lock();
queue up the workrtnl_unlock();
return success;
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
?
yes
We will lose the error reporting, is it a real problem or not?
Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.
Quoted 76 lines
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines
It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.And rx mode so this implies we will fail any command if the previous work is not finished.
don't fail it, store it.
Ok.
Thanks
Quoted 60 lines
Quoted 59 lines
Quoted 3 lines
have a tree
or a bitmap with vids to add/remove?Probably.
Thanks
Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 95 lines · Michael S. Tsirkin
Quoted 93 lines · Jason Wang
Quoted 92 lines · Michael S. Tsirkin
Quoted 89 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Shannon Nelson
On 7/20/23 1:38 AM, Jason Wang wrote:
Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.Signed-off-by: Jason Wang <[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
This still leaves hung processes, but at least it doesn't pin the CPU any
more. Thanks.
Reviewed-by: Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loopQuoted 5 lines
Quoted 2 lines
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.
Do you mean something like:
rtnl_lock();
queue up the workrtnl_unlock();
return success;?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
yes
We will lose the error reporting, is it a real problem or not?
Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.
This problem is not VDUSE specific, DPUs/vDPA may suffer from this as well. This might require more thoughts.
Thanks
Quoted 78 lines
Quoted 77 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.Quoted 4 lines
Quoted 2 lines
And rx mode so this implies we will fail any command if the previous work is not finished.
don't fail it, store it.
Ok.
Thanks
Quoted 60 lines
Quoted 59 lines
Quoted 3 lines
have a tree
or a bitmap with vids to add/remove?Probably.
Thanks
Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 100 lines · Jason Wang
Quoted 99 lines · Michael S. Tsirkin
Quoted 96 lines · Jason Wang
Quoted 85 lines · Michael S. Tsirkin
Quoted 83 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
Quoted 10 lines · Michael S. Tsirkin
On Thu, Jul 20, 2023 at 01:26:20PM -0700, Shannon Nelson wrote:
On 7/20/23 1:38 AM, Jason Wang wrote:Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.Signed-off-by: Jason Wang <[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This still leaves hung processes, but at least it doesn't pin the CPU any
more. Thanks.
Reviewed-by: Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loopQuoted 5 lines
Quoted 2 lines
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.
Do you mean something like:
Quoted 8 lines
Quoted 6 lines
rtnl_lock();
queue up the workrtnl_unlock();
return success;?
yes
We will lose the error reporting, is it a real problem or not?
Quoted 8 lines
Quoted 3 lines
Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.This problem is not VDUSE specific, DPUs/vDPA may suffer from this as well. This might require more thoughts.
Thanks
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack. But if they do they can always set this flag too.
Quoted 78 lines
Quoted 78 lines
Quoted 77 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.Quoted 4 lines
Quoted 2 lines
And rx mode so this implies we will fail any command if the previous work is not finished.
don't fail it, store it.
Ok.
Thanks
Quoted 60 lines
Quoted 59 lines
Quoted 3 lines
have a tree
or a bitmap with vids to add/remove?Probably.
Thanks
Quoted 51 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 106 lines · Michael S. Tsirkin
Quoted 104 lines · Jason Wang
Quoted 102 lines · Michael S. Tsirkin
Quoted 91 lines · Jason Wang
Quoted 90 lines · Michael S. Tsirkin
Quoted 88 lines · Jason Wang
Quoted 74 lines · Michael S. Tsirkin
Quoted 72 lines · Jason Wang
Quoted 22 lines · Maxime Coquelin
Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Maxime Coquelin
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Maxime Coquelin
On 7/20/23 23:02, Michael S. Tsirkin wrote:
On Thu, Jul 20, 2023 at 01:26:20PM -0700, Shannon Nelson wrote:
On 7/20/23 1:38 AM, Jason Wang wrote:Adding cond_resched() to the command waiting loop for a better
co-operation with the scheduler. This allows to give CPU a breath to
run other task(workqueue) instead of busy looping when preemption is
not allowed on a device whose CVQ might be slow.Signed-off-by: Jason Wang <[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This still leaves hung processes, but at least it doesn't pin the CPU any
more. Thanks.
Reviewed-by: Shannon Nelson <[email protected]>I'd like to see a full solution
1- block until interruptI remember in previous versions, you worried about the extra MSI vector. (Maybe I was wrong).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Would it make sense to also have a timeout?
And when timeout expires, set FAILED bit in device status?virtio spec does not set any limits on the timing of vq processing.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Indeed, but I thought the driver could decide it is too long for it.
The issue is we keep waiting with rtnl locked, it can quickly make the system unusable.
if this is a problem we should find a way not to keep rtnl locked indefinitely.
Any ideas on this direction? Simply dropping rtnl during the busy loop will result in a lot of races. This seems to require non-trivial changes in the networking core.
Quoted 7 lines
From the tests I have done, I think it is. With OVS, a reconfiguration
is performed when the VDUSE device is added, and when a MLX5 device is in the same bridge, it ends up doing anioctl()that tries to take the rtnl lock. In this configuration, it is not possible to kill OVS because it is stuck trying to acquire rtnl lock for mlx5 that is held by virtio- net.Yeah, basically, any RTNL users would be blocked forever.
And the infinite loop has other side effects like it blocks the freezer to work.
To summarize, there are three issues
1) busy polling
2) breaks freezer
3) hold RTNL during the loopQuoted 5 lines
Quoted 2 lines
Solving 3 may help somehow for 2 e.g some pm routine e.g wireguard or even
virtnet_restore()itself may try to hold the lock.Yep. So my feeling currently is, the only real fix is to actually queue up the work in software.
Do you mean something like:
rtnl_lock();
queue up the workrtnl_unlock();
return success;Quoted 5 lines
Quoted 3 lines
Quoted 1 line
?
yes
We will lose the error reporting, is it a real problem or not?
Fundamental isn't it? Maybe we want a per-device flag for a asynch commands, and vduse will set it while hardware virtio won't.
this way we only lose error reporting for vduse.
Quoted 7 lines
Quoted 4 lines
This problem is not VDUSE specific, DPUs/vDPA may suffer from this as well. This might require more thoughts.
Thanks
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the set_channels to
pre_set_channelsset_channelspost_set_channels
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
It's mostly trivial to limit
memory consumption, vid's is the
only one where it would make sense to have more than 1 command of a given type outstanding.Quoted 4 lines
Quoted 2 lines
And rx mode so this implies we will fail any command if the previous work is not finished.
don't fail it, store it.
Ok.
Thanks
Quoted 50 lines
Quoted 50 lines
Quoted 3 lines
have a tree
or a bitmap with vids to add/remove?Probably.
Thanks
Quoted 42 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
2- still handle surprise removal correctly by waking in that case
This is basically what version 1 did?
https://lore.kernel.org/lkml/[email protected]/t/
Thanks
Yes - except the timeout part.
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
---
drivers/net/virtio_net.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 9f3b1d6ac33d..e7533f29b219 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2314,8 +2314,10 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
* into the hypervisor, so the request should be handled immediately.
*/
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ cond_resched();
cpu_relax();
+ }return vi->ctrl->status == VIRTIO_NET_OK;
}
--
2.39.3
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 8 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
Quoted 7 lines
pre_set_channelsset_channelspost_set_channelsThe device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Signature · 6 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channels
Quoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the virtnet_send_command() and netif_set_real_num_tx_queues() need to appear to be atomic to the networking core.
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Thanks
Quoted 5 lines
--
MST
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 34 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Quoted 6 lines
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 8 lines
Thanks
Quoted 5 lines
--
MST
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.
Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
I propose we fix this for vduse and assume hardware/firmware is well behaved. Or maybe not well behaved firmware will set the flag losing error reporting ability.
Quoted 8 lines
Quoted 8 lines
Thanks
Quoted 5 lines
--
MST
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 3 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects.
So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Quoted 2 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.
This might be hard since it means not only the set but also the get is unreliable.
Thanks
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 5 lines
--
MST
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 60 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Ugh more mission creep. not at all my point. vduse should cache values in the driver, until someone manages to change net core to be more friendly to userspace devices.
Quoted 8 lines
Quoted 3 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.This might be hard since it means not only the set but also the get is unreliable.
Thanks
/me shrugs
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 5 lines
--
MST
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 65 lines · Michael S. Tsirkin
Quoted 63 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 3 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
Quoted 7 lines
Quoted 4 lines
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Ugh more mission creep. not at all my point. vduse should cache values in the driver,
What do you mean by values here? The cvq command?
Thanks
Quoted 29 lines
until someone manages to change
net core to be more friendly to userspace devices.Quoted 8 lines
Quoted 3 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.This might be hard since it means not only the set but also the get is unreliable.
Thanks
/me shrugs
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 5 lines
--
MST
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 71 lines · Jason Wang
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 3 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Quoted 6 lines
Quoted 2 lines
Ugh more mission creep. not at all my point. vduse should cache values in the driver,
What do you mean by values here? The cvq command?
Thanks
The card status generally.
Quoted 29 lines
Quoted 29 lines
until someone manages to change
net core to be more friendly to userspace devices.Quoted 8 lines
Quoted 3 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.This might be hard since it means not only the set but also the get is unreliable.
Thanks
/me shrugs
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 5 lines
--
MST
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 73 lines · Michael S. Tsirkin
Quoted 71 lines · Jason Wang
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 3 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Ugh more mission creep. not at all my point. vduse should cache values in the driver,
What do you mean by values here? The cvq command?
Thanks
The card status generally.
Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?
Thanks
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
until someone manages to change
net core to be more friendly to userspace devices.Quoted 8 lines
Quoted 3 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.This might be hard since it means not only the set but also the get is unreliable.
Thanks
/me shrugs
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 5 lines
--
MST
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 79 lines · Jason Wang
Quoted 78 lines · Michael S. Tsirkin
Quoted 76 lines · Jason Wang
Quoted 65 lines · Michael S. Tsirkin
Quoted 63 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 3 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
Quoted 7 lines
Quoted 4 lines
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Ugh more mission creep. not at all my point. vduse should cache values in the driver,
What do you mean by values here? The cvq command?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Thanks
The card status generally.
Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?
Thanks
vduse will have to process it in kernel.
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
until someone manages to change
net core to be more friendly to userspace devices.Quoted 8 lines
Quoted 3 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.This might be hard since it means not only the set but also the get is unreliable.
Thanks
/me shrugs
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 5 lines
--
MST
Re: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 83 lines · Michael S. Tsirkin
Quoted 81 lines · Jason Wang
Quoted 80 lines · Michael S. Tsirkin
Quoted 77 lines · Jason Wang
Quoted 65 lines · Michael S. Tsirkin
Quoted 63 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to program will create issues in the networking stack.
Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 3 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
Quoted 7 lines
Quoted 4 lines
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Ugh more mission creep. not at all my point. vduse should cache values in the driver,
What do you mean by values here? The cvq command?
Thanks
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
The card status generally.
Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?
Thanks
vduse will have to process it in kernel.
Right, that's my understanding (trap CVQ).
Thanks
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
until someone manages to change
net core to be more friendly to userspace devices.Quoted 8 lines
Quoted 3 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.This might be hard since it means not only the set but also the get is unreliable.
Thanks
/me shrugs
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 5 lines
--
MST
Michael S. Tsirkin
maintainerRe: [PATCH net-next v4 2/2] virtio-net: add cond_resched() to the command waiting loop
Quoted 89 lines · Jason Wang
Quoted 88 lines · Michael S. Tsirkin
Quoted 86 lines · Jason Wang
Quoted 73 lines · Michael S. Tsirkin
Quoted 71 lines · Jason Wang
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 38 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 2 lines
They really shouldn't - any NIC that takes forever to
program will create issues in the networking stack.Unfortunately, it's not rare as the device/cvq could be implemented via firmware or software.
Currently that mean one either has sane firmware with a scheduler that can meet deadlines, or loses ability to report errors back.
Quoted 12 lines
Quoted 1 line
But if they do they can always set this flag too.
This may have false negatives and may confuse the management.
Maybe we can extend the networking core to allow some device specific configurations to be done with device specific lock without rtnl. For example, split the
set_channelsto
pre_set_channelsset_channelspost_set_channelsQuoted 8 lines
Quoted 3 lines
The device specific part could be done in pre and post without a rtnl lock?
Thanks
Would the benefit be that errors can be reported to userspace then? Then maybe. I think you will have to show how this works for at least one card besides virtio.
Even for virtio, this seems not easy, as e.g the
virtnet_send_command()andnetif_set_real_num_tx_queues()need to appear to be atomic to the networking core.Quoted 6 lines
Quoted 2 lines
I wonder if we can re-consider the way of a timeout here and choose a sane value as a start.
Michael, any more input on this?
Thanks
Quoted 7 lines
I think this is just mission creep. We are trying to fix vduse - let's do that for starters.
Recovering from firmware timeouts is far from trivial and just assuming that just because it timed out it will not access memory is just as likely to cause memory corruption with worse results than an infinite spin.
Yes, this might require support not only in the driver
Quoted 3 lines
I propose we fix this for vduse and assume hardware/firmware is well behaved.
One major case is the re-connection, in that case it might take whatever longer that the kernel virito-net driver expects. So we can have a timeout in VDUSE and trap CVQ then VDUSE can return and fail early?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Ugh more mission creep. not at all my point. vduse should cache values in the driver,
What do you mean by values here? The cvq command?
Thanks
The card status generally.
Just to make sure I understand here. The CVQ needs to be processed by the userspace now. How could we cache the status?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Thanks
vduse will have to process it in kernel.
Right, that's my understanding (trap CVQ).
Thanks
oh this what you mean by trap. ok. I don't see a new for a timeout then though.
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
until someone manages to change
net core to be more friendly to userspace devices.Quoted 8 lines
Quoted 3 lines
Or maybe not well behaved firmware will
set the flag losing error reporting ability.This might be hard since it means not only the set but also the get is unreliable.
Thanks
/me shrugs
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks
Quoted 2 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
--
MST
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization