Skip to contentSkip to conversation

virtio-net: fix bottom-half packet TX on asynchronous completion

RFC PATCH Superseded Laurent Vivier 3 participants 5 messages Oct 13, 2022 – Oct 14, 2022

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 virtio-net: fix bottom-half packet TX on asynchronous completion

When virtio-net is used with the socket netdev backend, the backend can be busy and not able to collect new packets.

In this case, net_socket_receive() returns 0 and registers a poll function to detect when the socket is ready again.

In virtio_net_tx_bh(), virtio_net_flush_tx() forwards the 0, the virtio notifications are disabled and the function is not rescheduled, waiting for the backend to be ready.

When the socket netdev backend is again able to send packets, the poll function re-starts to flush remaining packets. This is done by calling virtio_net_tx_complete(). It re-enables notifications and calls again virtio_net_flush_tx().

But it seems if virtio_net_flush_tx() reaches the tx_burst value all the queue is not flushed and no new notification is sent to reschedule virtio_net_tx_bh(). Nothing re-start to flush the queue and remaining packets are stuck in the queue.

To fix that, detect in virtio_net_tx_complete() if virtio_net_flush_tx() has been stopped by tx_burst and if yes reschedule the bottom half function virtio_net_tx_bh() to flush the remaining packets.

This is what is done in virtio_net_tx_bh() when the virtio_net_flush_tx() is synchronous, and completely by-passed when the operation needs to be asynchronous.

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Do we need to reschedule the function in the other case managed in virtio_net_tx_bh() and by-passed when the completion is asynchronous?

    /* If less than a full burst, re-enable notification and flush
     * anything that may have come in while we weren't looking.  If
     * we find something, assume the guest is still active and reschedule */
    virtio_queue_set_notification(q->tx_vq, 1);
    ret = virtio_net_flush_tx(q);
    if (ret == -EINVAL) {
        return;
    } else if (ret > 0) {
        virtio_queue_set_notification(q->tx_vq, 0);
        qemu_bh_schedule(q->tx_bh);
        q->tx_waiting = 1;
    }

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Fixes a697a334b3c4 ("virtio-net: Introduce a new bottom half packet TX")
---
 hw/net/virtio-net.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +12 -1
index e9f696b4cfeb..1fbf2f3e19a7 100644
@@ -2526,6 +2526,7 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
     VirtIONet *n = qemu_get_nic_opaque(nc);
     VirtIONetQueue *q = virtio_net_get_subqueue(nc);
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    int ret;
 
     virtqueue_push(q->tx_vq, q->async_tx.elem, 0);
     virtio_notify(vdev, q->tx_vq);
@@ -2534,7 +2535,17 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
     q->async_tx.elem = NULL;
 
     virtio_queue_set_notification(q->tx_vq, 1);
-    virtio_net_flush_tx(q);
+    ret = virtio_net_flush_tx(q);
+    if (q->tx_bh && ret >= n->tx_burst) {
+        /*
+         * the flush has been stopped by tx_burst
+         * we will not receive notification for the
+         * remainining part, so re-schedule
+         */
+        virtio_queue_set_notification(q->tx_vq, 0);
+        qemu_bh_schedule(q->tx_bh);
+        q->tx_waiting = 1;
+    }
 }
 
 /* TX */
Signature · 1 line
2.37.3
Acked-by on v1 1/1
#1 lore raw
Quoted 24 lines · Laurent Vivier

When virtio-net is used with the socket netdev backend, the backend can be busy and not able to collect new packets.

In this case, net_socket_receive() returns 0 and registers a poll function to detect when the socket is ready again.

In virtio_net_tx_bh(), virtio_net_flush_tx() forwards the 0, the virtio notifications are disabled and the function is not rescheduled, waiting for the backend to be ready.

When the socket netdev backend is again able to send packets, the poll function re-starts to flush remaining packets. This is done by calling virtio_net_tx_complete(). It re-enables notifications and calls again virtio_net_flush_tx().

But it seems if virtio_net_flush_tx() reaches the tx_burst value all the queue is not flushed and no new notification is sent to reschedule virtio_net_tx_bh(). Nothing re-start to flush the queue and remaining packets are stuck in the queue.

To fix that, detect in virtio_net_tx_complete() if virtio_net_flush_tx() has been stopped by tx_burst and if yes reschedule the bottom half function virtio_net_tx_bh() to flush the remaining packets.

Quoted 8 lines

This is what is done in virtio_net_tx_bh() when the virtio_net_flush_tx() is synchronous, and completely by-passed when the operation needs to be asynchronous.

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Do we need to reschedule the function in the other case managed in virtio_net_tx_bh() and by-passed when the completion is asynchronous?

I am guessing no.

Quoted 13 lines
    /* If less than a full burst, re-enable notification and flush
     * anything that may have come in while we weren't looking.  If
     * we find something, assume the guest is still active and reschedule */
    virtio_queue_set_notification(q->tx_vq, 1);
    ret = virtio_net_flush_tx(q);
    if (ret == -EINVAL) {
        return;
    } else if (ret > 0) {
        virtio_queue_set_notification(q->tx_vq, 0);
        qemu_bh_schedule(q->tx_bh);
        q->tx_waiting = 1;
    }
Quoted 5 lines

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Fixes a697a334b3c4 ("virtio-net: Introduce a new bottom half packet TX")

Looks ok superficially

Jason, your area.

Quoted 37 lines
---
 hw/net/virtio-net.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +12 -1
index e9f696b4cfeb..1fbf2f3e19a7 100644
@@ -2526,6 +2526,7 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
     VirtIONet *n = qemu_get_nic_opaque(nc);
     VirtIONetQueue *q = virtio_net_get_subqueue(nc);
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    int ret;
 
     virtqueue_push(q->tx_vq, q->async_tx.elem, 0);
     virtio_notify(vdev, q->tx_vq);
@@ -2534,7 +2535,17 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
     q->async_tx.elem = NULL;
 
     virtio_queue_set_notification(q->tx_vq, 1);
-    virtio_net_flush_tx(q);
+    ret = virtio_net_flush_tx(q);
+    if (q->tx_bh && ret >= n->tx_burst) {
+        /*
+         * the flush has been stopped by tx_burst
+         * we will not receive notification for the
+         * remainining part, so re-schedule
+         */
+        virtio_queue_set_notification(q->tx_vq, 0);
+        qemu_bh_schedule(q->tx_bh);
+        q->tx_waiting = 1;
+    }
 }
 
 /* TX */
Signature · 1 line
2.37.3
#2 lore raw
Quoted 82 lines · Michael S. Tsirkin
Quoted 24 lines · Laurent Vivier

When virtio-net is used with the socket netdev backend, the backend can be busy and not able to collect new packets.

In this case, net_socket_receive() returns 0 and registers a poll function to detect when the socket is ready again.

In virtio_net_tx_bh(), virtio_net_flush_tx() forwards the 0, the virtio notifications are disabled and the function is not rescheduled, waiting for the backend to be ready.

When the socket netdev backend is again able to send packets, the poll function re-starts to flush remaining packets. This is done by calling virtio_net_tx_complete(). It re-enables notifications and calls again virtio_net_flush_tx().

But it seems if virtio_net_flush_tx() reaches the tx_burst value all the queue is not flushed and no new notification is sent to reschedule virtio_net_tx_bh(). Nothing re-start to flush the queue and remaining packets are stuck in the queue.

To fix that, detect in virtio_net_tx_complete() if virtio_net_flush_tx() has been stopped by tx_burst and if yes reschedule the bottom half function virtio_net_tx_bh() to flush the remaining packets.

Quoted 8 lines

This is what is done in virtio_net_tx_bh() when the virtio_net_flush_tx() is synchronous, and completely by-passed when the operation needs to be asynchronous.

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Do we need to reschedule the function in the other case managed in virtio_net_tx_bh() and by-passed when the completion is asynchronous?

I am guessing no.

Quoted 13 lines
    /* If less than a full burst, re-enable notification and flush
     * anything that may have come in while we weren't looking.  If
     * we find something, assume the guest is still active and reschedule */
    virtio_queue_set_notification(q->tx_vq, 1);
    ret = virtio_net_flush_tx(q);
    if (ret == -EINVAL) {
        return;
    } else if (ret > 0) {
        virtio_queue_set_notification(q->tx_vq, 0);
        qemu_bh_schedule(q->tx_bh);
        q->tx_waiting = 1;
    }
Quoted 5 lines

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Fixes a697a334b3c4 ("virtio-net: Introduce a new bottom half packet TX")

Looks ok superficially

Jason, your area.

Quoted 19 lines
---
 hw/net/virtio-net.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -0
index e9f696b4cfeb..1fbf2f3e19a7 100644
@@ -2526,6 +2526,7 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
     VirtIONet *n = qemu_get_nic_opaque(nc);
     VirtIONetQueue *q = virtio_net_get_subqueue(nc);
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    int ret;
 
     virtqueue_push(q->tx_vq, q->async_tx.elem, 0);
     virtio_notify(vdev, q->tx_vq);
@@ -2534,7 +2535,17 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
     q->async_tx.elem = NULL;
 
Quoted 13 lines

hw/net/virtio-net.c static void virtio_net_tx_complete(NetClientState *nc, ssize_t len) · line 2548

Quoted 13 lines
hw/net/virtio-net.c +11 -1
     virtio_queue_set_notification(q->tx_vq, 1);
-    virtio_net_flush_tx(q);
+    ret = virtio_net_flush_tx(q);
+    if (q->tx_bh && ret >= n->tx_burst) {
+        /*
+         * the flush has been stopped by tx_burst
+         * we will not receive notification for the
+         * remainining part, so re-schedule
+         */
+        virtio_queue_set_notification(q->tx_vq, 0);
+        qemu_bh_schedule(q->tx_bh);
+        q->tx_waiting = 1;
+    }

Do we need to fix the case of tx timer or it doesn't suffer from this issue?

Thanks

Quoted 6 lines
Quoted 5 lines
hw/net/virtio-net.c +0 -1
 }
 
 /* TX */
--

2.37.3

#3 lore raw
Quoted 83 lines · Jason Wang
Quoted 82 lines · Michael S. Tsirkin
Quoted 24 lines · Laurent Vivier

When virtio-net is used with the socket netdev backend, the backend can be busy and not able to collect new packets.

In this case, net_socket_receive() returns 0 and registers a poll function to detect when the socket is ready again.

In virtio_net_tx_bh(), virtio_net_flush_tx() forwards the 0, the virtio notifications are disabled and the function is not rescheduled, waiting for the backend to be ready.

When the socket netdev backend is again able to send packets, the poll function re-starts to flush remaining packets. This is done by calling virtio_net_tx_complete(). It re-enables notifications and calls again virtio_net_flush_tx().

But it seems if virtio_net_flush_tx() reaches the tx_burst value all the queue is not flushed and no new notification is sent to reschedule virtio_net_tx_bh(). Nothing re-start to flush the queue and remaining packets are stuck in the queue.

To fix that, detect in virtio_net_tx_complete() if virtio_net_flush_tx() has been stopped by tx_burst and if yes reschedule the bottom half function virtio_net_tx_bh() to flush the remaining packets.

Quoted 8 lines

This is what is done in virtio_net_tx_bh() when the virtio_net_flush_tx() is synchronous, and completely by-passed when the operation needs to be asynchronous.

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Do we need to reschedule the function in the other case managed in virtio_net_tx_bh() and by-passed when the completion is asynchronous?

I am guessing no.

Quoted 13 lines
     /* If less than a full burst, re-enable notification and flush
      * anything that may have come in while we weren't looking.  If
      * we find something, assume the guest is still active and reschedule */
     virtio_queue_set_notification(q->tx_vq, 1);
     ret = virtio_net_flush_tx(q);
     if (ret == -EINVAL) {
         return;
     } else if (ret > 0) {
         virtio_queue_set_notification(q->tx_vq, 0);
         qemu_bh_schedule(q->tx_bh);
         q->tx_waiting = 1;
     }
Quoted 5 lines

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Fixes a697a334b3c4 ("virtio-net: Introduce a new bottom half packet TX")

Looks ok superficially

Jason, your area.

Quoted 19 lines
---
  hw/net/virtio-net.c | 13 ++++++++++++-
  1 file changed, 12 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -0
index e9f696b4cfeb..1fbf2f3e19a7 100644
@@ -2526,6 +2526,7 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
      VirtIONet *n = qemu_get_nic_opaque(nc);
      VirtIONetQueue *q = virtio_net_get_subqueue(nc);
      VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    int ret;
 
      virtqueue_push(q->tx_vq, q->async_tx.elem, 0);
      virtio_notify(vdev, q->tx_vq);
@@ -2534,7 +2535,17 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
      q->async_tx.elem = NULL;
 
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
hw/net/virtio-net.c +3 -1
      virtio_queue_set_notification(q->tx_vq, 1);
-    virtio_net_flush_tx(q);
+    ret = virtio_net_flush_tx(q);
+    if (q->tx_bh && ret >= n->tx_burst) {
+        /*
Quoted 8 lines

hw/net/virtio-net.c static void virtio_net_tx_complete(NetClientState *nc, ssize_t len) · line 2548

Quoted 8 lines
hw/net/virtio-net.c +8 -0
+         * the flush has been stopped by tx_burst
+         * we will not receive notification for the
+         * remainining part, so re-schedule
+         */
+        virtio_queue_set_notification(q->tx_vq, 0);
+        qemu_bh_schedule(q->tx_bh);
+        q->tx_waiting = 1;
+    }

Do we need to fix the case of tx timer or it doesn't suffer from this issue?

I think tx_timer suffers the same issue.

I'm going to have a look to the tx timer fix.

Thanks,
Laurent

#4 lore raw
Quoted 102 lines · Laurent Vivier
Quoted 84 lines · Jason Wang
Quoted 83 lines · Michael S. Tsirkin
Quoted 24 lines · Laurent Vivier

When virtio-net is used with the socket netdev backend, the backend can be busy and not able to collect new packets.

In this case, net_socket_receive() returns 0 and registers a poll function to detect when the socket is ready again.

In virtio_net_tx_bh(), virtio_net_flush_tx() forwards the 0, the virtio notifications are disabled and the function is not rescheduled, waiting for the backend to be ready.

When the socket netdev backend is again able to send packets, the poll function re-starts to flush remaining packets. This is done by calling virtio_net_tx_complete(). It re-enables notifications and calls again virtio_net_flush_tx().

But it seems if virtio_net_flush_tx() reaches the tx_burst value all the queue is not flushed and no new notification is sent to reschedule virtio_net_tx_bh(). Nothing re-start to flush the queue and remaining packets are stuck in the queue.

To fix that, detect in virtio_net_tx_complete() if virtio_net_flush_tx() has been stopped by tx_burst and if yes reschedule the bottom half function virtio_net_tx_bh() to flush the remaining packets.

Quoted 8 lines

This is what is done in virtio_net_tx_bh() when the virtio_net_flush_tx() is synchronous, and completely by-passed when the operation needs to be asynchronous.

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Do we need to reschedule the function in the other case managed in virtio_net_tx_bh() and by-passed when the completion is asynchronous?

I am guessing no.

Quoted 13 lines

     /* If less than a full burst, re-enable notification and flush
      * anything that may have come in while we weren't looking.  If
      * we find something, assume the guest is still active and reschedule */ virtio_queue_set_notification(q->tx_vq, 1);
     ret = virtio_net_flush_tx(q);
     if (ret == -EINVAL) {
         return;
     } else if (ret > 0) { virtio_queue_set_notification(q->tx_vq, 0); qemu_bh_schedule(q->tx_bh);
         q->tx_waiting = 1;
     }

Quoted 5 lines

RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC RFC

Fixes a697a334b3c4 ("virtio-net: Introduce a new bottom half packet TX")

Looks ok superficially

Jason, your area.

Quoted 20 lines

---
  hw/net/virtio-net.c | 13 ++++++++++++-

  1 file changed, 12 insertions(+), 1 deletion(-)
hw/net/virtio-net.c +1 -0
index e9f696b4cfeb..1fbf2f3e19a7 100644
@@ -2526,6 +2526,7 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t len)
       VirtIONet *n = qemu_get_nic_opaque(nc);
       VirtIONetQueue *q = virtio_net_get_subqueue(nc);
       VirtIODevice *vdev = VIRTIO_DEVICE(n);
+    int ret;
 
       virtqueue_push(q->tx_vq, q->async_tx.elem, 0);
       virtio_notify(vdev, q->tx_vq);
@@ -2534,7 +2535,17 @@ static void virtio_net_tx_complete(NetClientState *nc, ssize_t 
 len)
       q->async_tx.elem = NULL;
 
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
hw/net/virtio-net.c +3 -1
       virtio_queue_set_notification(q->tx_vq, 1);
-    virtio_net_flush_tx(q);
+    ret = virtio_net_flush_tx(q);
+    if (q->tx_bh && ret >= n->tx_burst) {
+        /*
Quoted 8 lines

hw/net/virtio-net.c static void virtio_net_tx_complete(NetClientState *nc, ssize_t len) · line 2548

Quoted 8 lines
hw/net/virtio-net.c +8 -0
+         * the flush has been stopped by tx_burst
+         * we will not receive notification for the
+         * remainining part, so re-schedule
+         */
+        virtio_queue_set_notification(q->tx_vq, 0);
+        qemu_bh_schedule(q->tx_bh);
+        q->tx_waiting = 1;
+    }

Do we need to fix the case of tx timer or it doesn't suffer from this issue?

Quoted 3 lines

I think tx_timer suffers the same issue.

I'm going to have a look to the tx timer fix.

It seems worst with tx timer as we have the tx_burst problem even in the synchronous case as the timer is not rearmed.

Thanks,
Laurent