Skip to contentSkip to conversation

virtio_net: Fix queue full check

PATCH In review Krishna Kumar 3 participants 13 messages Oct 28, 2010 – Nov 9, 2010

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 virtio_net: Fix queue full check

I get many queue full errors being wrongly reported when running parallel netperfs:

Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue failure: -28
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue failure: -28
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue failure: -28
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue failure: -28

I initially changed the check from -ENOMEM to -ENOSPC, but virtqueue_add_buf can return only -ENOSPC when it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.

---
 drivers/net/virtio_net.c |   15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)
drivers/net/virtio_net.c +4 -11
diff -ruNp org/drivers/net/virtio_net.c new/drivers/net/virtio_net.c
@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_
 
 	/* This can happen with OOM and indirect buffers. */
 	if (unlikely(capacity < 0)) {
-		if (net_ratelimit()) {
-			if (likely(capacity == -ENOMEM)) {
-				dev_warn(&dev->dev,
-					 "TX queue failure: out of memory\n");
-			} else {
-				dev->stats.tx_fifo_errors++;
-				dev_warn(&dev->dev,
-					 "Unexpected TX queue failure: %d\n",
-					 capacity);
-			}
-		}
+		if (net_ratelimit())
+			dev_warn(&dev->dev,
+				 "TX queue failure (%d): out of memory\n",
+				 capacity); 
 		dev->stats.tx_dropped++;
 		kfree_skb(skb);
 		return NETDEV_TX_OK;
#1 lore raw
Quoted 30 lines · Krishna Kumar

I get many queue full errors being wrongly reported when running parallel netperfs:

Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue failure: -28
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue failure: -28
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue failure: -28
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue failure: -28

I initially changed the check from -ENOMEM to -ENOSPC, but virtqueue_add_buf can return only -ENOSPC when it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.

---
 drivers/net/virtio_net.c |   15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff -ruNp org/drivers/net/virtio_net.c new/drivers/net/virtio_net.c

drivers/net/virtio_net.c +0 -5
@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_
 
 	/* This can happen with OOM and indirect buffers. */
 	if (unlikely(capacity < 0)) {
-		if (net_ratelimit()) {
-			if (likely(capacity == -ENOMEM)) {
-				dev_warn(&dev->dev,
-					 "TX queue failure: out of memory\n");
-			} else {

drivers/net/virtio_net.c static netdev_tx_t start_xmit(struct sk_ · line 576

Quoted 10 lines
drivers/net/virtio_net.c +4 -6
-				dev->stats.tx_fifo_errors++;
-				dev_warn(&dev->dev,
-					 "Unexpected TX queue failure: %d\n",
-					 capacity);
-			}
-		}
+		if (net_ratelimit())
+			dev_warn(&dev->dev,
+				 "TX queue failure (%d): out of memory\n",
+				 capacity); 

Hold on... you were getting -ENOSPC, which shouldn't happen. What makes you think it's out of memory?

Confused,
Rusty.

#2 lore raw

Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:

Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 27 lines
Quoted 27 lines

I initially changed the check from -ENOMEM to -ENOSPC, but virtqueue_add_buf can return only -ENOSPC when it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.

---
 drivers/net/virtio_net.c |   15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff -ruNp org/drivers/net/virtio_net.c new/drivers/net/virtio_net.c

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0 -9
@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_
 
    /* This can happen with OOM and indirect buffers. */
    if (unlikely(capacity < 0)) {
-      if (net_ratelimit()) {
-         if (likely(capacity == -ENOMEM)) {
-            dev_warn(&dev->dev,
-                "TX queue failure: out of memory\n");
-         } else {
-            dev->stats.tx_fifo_errors++;
-            dev_warn(&dev->dev,
-                "Unexpected TX queue failure: %d\n",
-                capacity);
Quoted 8 lines

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576

Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4 -2
-         }
-      }
+      if (net_ratelimit())
+         dev_warn(&dev->dev,
+             "TX queue failure (%d): out of memory\n",
+             capacity);

Hold on... you were getting -ENOSPC, which shouldn't happen. What makes

you

Quoted 1 line

think it's out of memory?

virtqueue_add_buf_gfp returns only -ENOSPC on failure, whether direct or indirect descriptors are used, so isn't -ENOSPC "expected"? (vring_add_indirect returns -ENOMEM on memory failure, but that is masked out and we go direct which is the failure point).

Thanks,

- KK

#3 lore raw
Quoted 45 lines · On Fri, 29 Oct 2010 09:25:09 pm Krishna Kumar2 wrote:

Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:

Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 27 lines
Quoted 27 lines

I initially changed the check from -ENOMEM to -ENOSPC, but virtqueue_add_buf can return only -ENOSPC when it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.

---
 drivers/net/virtio_net.c |   15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff -ruNp org/drivers/net/virtio_net.c new/drivers/net/virtio_net.c

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0 -9
@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_
 
    /* This can happen with OOM and indirect buffers. */
    if (unlikely(capacity < 0)) {
-      if (net_ratelimit()) {
-         if (likely(capacity == -ENOMEM)) {
-            dev_warn(&dev->dev,
-                "TX queue failure: out of memory\n");
-         } else {
-            dev->stats.tx_fifo_errors++;
-            dev_warn(&dev->dev,
-                "Unexpected TX queue failure: %d\n",
-                capacity);
Quoted 8 lines

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576

Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4 -2
-         }
-      }
+      if (net_ratelimit())
+         dev_warn(&dev->dev,
+             "TX queue failure (%d): out of memory\n",
+             capacity);

Hold on... you were getting -ENOSPC, which shouldn't happen. What makes

Quoted 8 lines

you

Quoted 1 line

think it's out of memory?

virtqueue_add_buf_gfp returns only -ENOSPC on failure, whether direct or indirect descriptors are used, so isn't -ENOSPC "expected"? (vring_add_indirect returns -ENOMEM on memory failure, but that is masked out and we go direct which is the failure point).

Ah, OK, gotchya.

I'm not even sure the fallback to linear makes sense; if we're failing kmallocs we should probably just return -ENOMEM. Would mean we can tell the difference between "out of space" (which should never happen since we stop the queue when we have < 2+MAX_SKB_FRAGS slots left) and this case.

Michael, what do you think?

Thanks,
Rusty.

#4 lore raw
Quoted 62 lines · Rusty Russell
Quoted 45 lines · On Fri, 29 Oct 2010 09:25:09 pm Krishna Kumar2 wrote:

Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:

Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 27 lines
Quoted 27 lines

I initially changed the check from -ENOMEM to -ENOSPC, but virtqueue_add_buf can return only -ENOSPC when it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.

---
 drivers/net/virtio_net.c |   15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff -ruNp org/drivers/net/virtio_net.c new/drivers/net/virtio_net.c

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0 -9
@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_
 
    /* This can happen with OOM and indirect buffers. */
    if (unlikely(capacity < 0)) {
-      if (net_ratelimit()) {
-         if (likely(capacity == -ENOMEM)) {
-            dev_warn(&dev->dev,
-                "TX queue failure: out of memory\n");
-         } else {
-            dev->stats.tx_fifo_errors++;
-            dev_warn(&dev->dev,
-                "Unexpected TX queue failure: %d\n",
-                capacity);
Quoted 8 lines

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576

Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4 -2
-         }
-      }
+      if (net_ratelimit())
+         dev_warn(&dev->dev,
+             "TX queue failure (%d): out of memory\n",
+             capacity);

Hold on... you were getting -ENOSPC, which shouldn't happen. What makes

Quoted 8 lines

you

Quoted 1 line

think it's out of memory?

virtqueue_add_buf_gfp returns only -ENOSPC on failure, whether direct or indirect descriptors are used, so isn't -ENOSPC "expected"? (vring_add_indirect returns -ENOMEM on memory failure, but that is masked out and we go direct which is the failure point).

Ah, OK, gotchya.
I'm not even sure the fallback to linear makes sense; if we're failing kmallocs we should probably just return -ENOMEM. Would mean we can tell the difference between "out of space" (which should never happen since we stop the queue when we have < 2+MAX_SKB_FRAGS slots left) and this case.

Quoted 4 lines

Michael, what do you think?

Thanks,
Rusty.

Let's make sure I understand the issue: we use indirect buffers so we assume there's still a lot of place in the ring, then allocation for the indirect fails and so we return -ENOSPC?

So first, I agree it's a bug. But I am not sure killing the fallback is such a good idea: recovering from add buf failure is hard generally, we should try to accomodate if we can. Let's just fix the return code for now?

And generally, we should be smarter: as long as the ring is almost empty, and s/g list is short, it is a waste to use indirect buffers. BTW we have had a FIXME there for a long while, I think Yan suggested increasing that threshold to 3. Yan?

Further, maybe preallocating some memory for the indirect buffers might be a good idea.

In short, lots of good ideas, let's start with the minimal patch that is a good 2.6.37 candidate too. How about the following (untested)?

virtio: fix add_buf return code for OOM

add_buff returned ENOSPC on out of memory: this is a bug as at leats virtio-net expects ENOMEM and handles it specially. Fix that.

---

drivers/virtio/virtio_ring.c +2 -2
index 1475ed6..0a89098 100644
@@ -165,7 +165,7 @@ int virtqueue_add_buf_gfp(struct virtqueue *_vq,
 {
 	struct vring_virtqueue *vq = to_vvq(_vq);
 	unsigned int i, avail, uninitialized_var(prev);
-	int head;
+	int head = -ENOSPC;
 
 	START_USE(vq);
 
@@ -191,7 +191,7 @@ int virtqueue_add_buf_gfp(struct virtqueue *_vq,
 		if (out)
 			vq->notify(&vq->vq);
 		END_USE(vq);
-		return -ENOSPC;
+		return head;
 	}
 
 	/* We're about to use some buffers from the free list. */
Signature · 1 line
MST
#5 lore raw
Quoted 88 lines · Michael S. Tsirkin
Quoted 62 lines · Rusty Russell
Quoted 45 lines · On Fri, 29 Oct 2010 09:25:09 pm Krishna Kumar2 wrote:

Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:

Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue

failure: -28

Quoted 27 lines
Quoted 27 lines

I initially changed the check from -ENOMEM to -ENOSPC, but virtqueue_add_buf can return only -ENOSPC when it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.

---
 drivers/net/virtio_net.c |   15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff -ruNp org/drivers/net/virtio_net.c new/drivers/net/virtio_net.c

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0 -9
@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_
 
    /* This can happen with OOM and indirect buffers. */
    if (unlikely(capacity < 0)) {
-      if (net_ratelimit()) {
-         if (likely(capacity == -ENOMEM)) {
-            dev_warn(&dev->dev,
-                "TX queue failure: out of memory\n");
-         } else {
-            dev->stats.tx_fifo_errors++;
-            dev_warn(&dev->dev,
-                "Unexpected TX queue failure: %d\n",
-                capacity);
Quoted 8 lines

drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576

Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4 -2
-         }
-      }
+      if (net_ratelimit())
+         dev_warn(&dev->dev,
+             "TX queue failure (%d): out of memory\n",
+             capacity);

Hold on... you were getting -ENOSPC, which shouldn't happen. What makes

Quoted 8 lines

you

Quoted 1 line

think it's out of memory?

virtqueue_add_buf_gfp returns only -ENOSPC on failure, whether direct or indirect descriptors are used, so isn't -ENOSPC "expected"? (vring_add_indirect returns -ENOMEM on memory failure, but that is masked out and we go direct which is the failure point).

Ah, OK, gotchya.
I'm not even sure the fallback to linear makes sense; if we're failing kmallocs we should probably just return -ENOMEM. Would mean we can tell the difference between "out of space" (which should never happen since we stop the queue when we have < 2+MAX_SKB_FRAGS slots left) and this case.

Quoted 4 lines

Michael, what do you think?

Thanks,
Rusty.

Let's make sure I understand the issue: we use indirect buffers so we assume there's still a lot of place in the ring, then allocation for the indirect fails and so we return -ENOSPC?

So first, I agree it's a bug. But I am not sure killing the fallback is such a good idea: recovering from add buf failure is hard generally, we should try to accomodate if we can. Let's just fix the return code for now?

And generally, we should be smarter: as long as the ring is almost empty, and s/g list is short, it is a waste to use indirect buffers. BTW we have had a FIXME there for a long while, I think Yan suggested increasing that threshold to 3. Yan?

Further, maybe preallocating some memory for the indirect buffers might be a good idea.

In short, lots of good ideas, let's start with the minimal patch that is a good 2.6.37 candidate too. How about the following (untested)?

Quoted 7 lines

virtio: fix add_buf return code for OOM

add_buff returned ENOSPC on out of memory: this is a bug as at leats virtio-net expects ENOMEM and handles it specially. Fix that.

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

So below is what I propose now - as a replacement for my original patch. Krishna Kumar, Rusty, what do you think?

Separately I'm also considering moving the

	if (vq->num_free < out + in)

check earlier in the function to keep all users honest, but need to check what the implications are for e.g. block. Thoughts on this?

---->

virtio: return correct capacity to users

We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail.

So return the number of buffers we can guarantee users.

drivers/virtio/virtio_ring.c +0 -3
index 1475ed6..cc2f73e 100644
@@ -230,9 +230,6 @@ add_head:
 	pr_debug("Added buffer head %i to %p\n", head, vq);
 	END_USE(vq);
 
-	/* If we're indirect, we can fit many (assuming not OOM). */
-	if (vq->indirect)
-		return vq->num_free ? vq->vring.num : 0;
 	return vq->num_free;
 }
 EXPORT_SYMBOL_GPL(virtqueue_add_buf_gfp);
#6 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 11/04/2010 05:54:24 PM:

Quoted 17 lines
Quoted 1 line

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

Quoted 8 lines

So below is what I propose now - as a replacement for my original patch. Krishna Kumar, Rusty, what do you think?

Separately I'm also considering moving the
if (vq->num_free < out + in)
check earlier in the function to keep all users honest, but need to check what the implications are for e.g. block. Thoughts on this?

This looks like the right thing to do. Besides this, I think virtio-net still needs to remove check for ENOMEM? I will test this patch tomorrow.

Another question about add_recvbuf_small and add_recvbuf_big - both call virtqueue_add_buf_gfp with in+out > 1, and that can fail with -ENOSPC. So try_fill_recv gets -ENOSPC. When that happens, oom is not set to true, I thought it should have got set. Is this a bug?

Thanks,

- KK

#7 lore raw
Quoted 22 lines · On Thu, Nov 04, 2010 at 09:47:04PM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 11/04/2010 05:54:24 PM:

Quoted 20 lines
Quoted 1 line

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

So below is what I propose now - as a replacement for my original patch. Krishna Kumar, Rusty, what do you think?

Quoted 8 lines
Quoted 5 lines

Separately I'm also considering moving the
if (vq->num_free < out + in)
check earlier in the function to keep all users honest, but need to check what the implications are for e.g. block. Thoughts on this?

This looks like the right thing to do. Besides this, I think virtio-net still needs to remove check for ENOMEM?

Yes, the only valid reason for failure would be a unexpected error. No need to special-case ENOMEM anymore.

Quoted 11 lines

I will test this patch tomorrow.

Another question about add_recvbuf_small and add_recvbuf_big - both call virtqueue_add_buf_gfp with in+out > 1, and that can fail with -ENOSPC. So try_fill_recv gets -ENOSPC. When that happens, oom is not set to true, I thought it should have got set. Is this a bug?

Thanks,

- KK

I don't see a bug: on ENOSPC we don't need to (and can't) add any more buffers, we know we will make progress since there must be some buffers in the ring already, ENOMEM makes us try again later with more buffers (and possibly more aggressive GFP flag). What's wrong?

Signature · 1 line
MST
#8 lore raw
In reply to Michael S. Tsirkin
Quoted 11 lines · Michael S. Tsirkin

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

Quoted 3 lines

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room, leading to worst-case assumptions and underutilization of the tx ring.

However, I lost that debate, and so your patch is the way it's supposed to work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.

I enhanced your commentry a little:

Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>

We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular, virtio_net can get into this situation under stress, and it drops packets and performs badly.

So return the number of buffers we can guarantee users.

Thanks!
Rusty.

author Tested-by
#9 lore raw

Rusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:

Quoted 18 lines

Re: [PATCH] virtio_net: Fix queue full check

Quoted 11 lines · Michael S. Tsirkin

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

Quoted 3 lines

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

Quoted 5 lines

I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room, leading to worst-case assumptions and underutilization of the tx ring.

However, I lost that debate, and so your patch is the way it's supposed

to

Quoted 15 lines

work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.

I enhanced your commentry a little:

Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>

We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular, virtio_net can get into this situation under stress, and it drops packets and performs badly.

Quoted 5 lines

So return the number of buffers we can guarantee users.

I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but just in case:

I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of dev->stats.tx_fifo_errors, and reports "memory failure".

Thanks,

- KK

#10 lore raw
Quoted 57 lines · On Tue, Nov 09, 2010 at 09:56:03AM +0530, Krishna Kumar2 wrote:

Rusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:

Quoted 18 lines

Re: [PATCH] virtio_net: Fix queue full check

Quoted 11 lines · Michael S. Tsirkin

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

Quoted 3 lines

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

Quoted 5 lines

I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room, leading to worst-case assumptions and underutilization of the tx ring.

However, I lost that debate, and so your patch is the way it's supposed

to

Quoted 15 lines

work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.

I enhanced your commentry a little:

Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>

We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular, virtio_net can get into this situation under stress, and it drops packets and performs badly.

Quoted 5 lines

So return the number of buffers we can guarantee users.

I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but just in case:

Quoted 7 lines

I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of dev->stats.tx_fifo_errors, and reports "memory failure".

Thanks,

- KK

So I think your patch on top of this one would be wrong:
we actually make sure out of memory does not affect TX path at all, so any error would be unexpected.

Incrementing tx fifo errors is probably also helpful for debugging.

If you like, we could kill the special handling for ENOMEM. Not sure whether it matters.

Signature · 1 line
MST
#11 lore raw

"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:45:55 PM:

Quoted 25 lines

Re: [PATCH] virtio_net: Fix queue full check

Quoted 22 lines · On Tue, Nov 09, 2010 at 09:56:03AM +0530, Krishna Kumar2 wrote:

Rusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:

Quoted 20 lines

Re: [PATCH] virtio_net: Fix queue full check

Quoted 11 lines · Michael S. Tsirkin

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

Quoted 3 lines

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room,

leading to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

worst-case assumptions and underutilization of the tx ring.

However, I lost that debate, and so your patch is the way it's

supposed

Quoted 18 lines
Quoted 18 lines

to

Quoted 17 lines

work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.

I enhanced your commentry a little:

Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>

We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular, virtio_net can get into this situation under stress, and it drops packets and performs badly.

So return the number of buffers we can guarantee users.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but just

in

Quoted 15 lines
Quoted 7 lines

case:

Quoted 7 lines

I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of dev->stats.tx_fifo_errors, and reports "memory failure".

Thanks,

- KK

Quoted 8 lines

So I think your patch on top of this one would be wrong:
we actually make sure out of memory does not affect TX path at all, so any error would be unexpected.

Incrementing tx fifo errors is probably also helpful for debugging.

If you like, we could kill the special handling for ENOMEM. Not sure whether it matters.

Since that is dead code, we could remove it (and the fifo error should disappear too - tx_dropped should be the only counter to be incremented?). Sorry if I misunderstood something.

Thanks,

- KK

#12 lore raw
Quoted 82 lines · On Tue, Nov 09, 2010 at 09:00:58PM +0530, Krishna Kumar2 wrote:

"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:45:55 PM:

Quoted 25 lines

Re: [PATCH] virtio_net: Fix queue full check

Quoted 22 lines · On Tue, Nov 09, 2010 at 09:56:03AM +0530, Krishna Kumar2 wrote:

Rusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:

Quoted 20 lines

Re: [PATCH] virtio_net: Fix queue full check

Quoted 11 lines · Michael S. Tsirkin

I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.

And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.

Quoted 3 lines

So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.

I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room,

leading to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

worst-case assumptions and underutilization of the tx ring.

However, I lost that debate, and so your patch is the way it's

supposed

Quoted 18 lines
Quoted 18 lines

to

Quoted 17 lines

work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.

I enhanced your commentry a little:

Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>

We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular, virtio_net can get into this situation under stress, and it drops packets and performs badly.

So return the number of buffers we can guarantee users.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but just

in

Quoted 15 lines
Quoted 7 lines

case:

Quoted 7 lines

I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of dev->stats.tx_fifo_errors, and reports "memory failure".

Thanks,

- KK

Quoted 8 lines

So I think your patch on top of this one would be wrong:
we actually make sure out of memory does not affect TX path at all, so any error would be unexpected.

Incrementing tx fifo errors is probably also helpful for debugging.

If you like, we could kill the special handling for ENOMEM. Not sure whether it matters.

Quoted 7 lines

Since that is dead code, we could remove it (and the fifo error should disappear too - tx_dropped should be the only counter to be incremented?). Sorry if I misunderstood something.

Thanks,

- KK

It's just a sanity check. The capacity checking is tricky enough that I'm happier with some way to detect overflow both from ifconfig and dmesg.

I don't really care which counter gets incremented really, since this is some TX bug fifo error seems appropriate but I don't really care much.

Signature · 1 line
MST