virtio_net: Fix queue full check
Subsystems (4)
PATCH 1/1 virtio_net: Fix queue full check
I get many queue full errors being wrongly reported when running parallel netperfs:
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue failure: -28 Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue failure: -28 Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue failure: -28 Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue failure: -28
I initially changed the check from -ENOMEM to -ENOSPC, but virtqueue_add_buf can return only -ENOSPC when it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.
--- drivers/net/virtio_net.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
drivers/net/virtio_net.c +4 -11
@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_/* This can happen with OOM and indirect buffers. */ if (unlikely (capacity <0 )) { -if (net_ratelimit ()) { -if (likely (capacity == -ENOMEM )) { -dev_warn (&dev->dev, -"TX queue failure: out of memory\n" ); - }else { - dev->stats.tx_fifo_errors++; -dev_warn (&dev->dev, -"Unexpected TX queue failure: %d\n" , - capacity); - } - } +if (net_ratelimit ()) +dev_warn (&dev->dev, +"TX queue failure (%d): out of memory\n" , + capacity); dev->stats.tx_dropped++;kfree_skb (skb);return NETDEV_TX_OK ;
Quoted 30 lines · Krishna Kumar
I get many queue full errors being wrongly reported when running parallel netperfs:
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue failure: -28 Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue failure: -28 Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue failure: -28 Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue failure: -28I initially changed the check from
-ENOMEMto-ENOSPC, butvirtqueue_add_bufcan return only-ENOSPCwhen it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.Signed-off-by Krishna Kumar <[email protected]>--- drivers/net/virtio_net.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)diff -ruNp
org/drivers/net/virtio_net.cnew/drivers/net/virtio_net.cdrivers/net/virtio_net.c +0
-5@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_/* This can happen with OOM and indirect buffers. */ if (unlikely (capacity <0 )) { -if (net_ratelimit ()) { -if (likely (capacity == -ENOMEM )) { -dev_warn (&dev->dev, -"TX queue failure: out of memory\n" ); - }else {
drivers/net/virtio_net.c static netdev_tx_t start_xmit(struct sk_ · line 576
Quoted 10 lines
drivers/net/virtio_net.c +4
-6- dev->stats.tx_fifo_errors++; -dev_warn (&dev->dev, -"Unexpected TX queue failure: %d\n" , - capacity); - } - } +if (net_ratelimit ()) +dev_warn (&dev->dev, +"TX queue failure (%d): out of memory\n" , + capacity);
Hold on... you were getting -ENOSPC, which shouldn't happen. What makes you think it's out of memory?
Confused,
Rusty.
Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:
Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queue
failure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queue
failure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queue
failure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queue
failure: -28
Quoted 27 lines
Quoted 27 lines
I initially changed the check from
-ENOMEMto-ENOSPC, butvirtqueue_add_bufcan return only-ENOSPCwhen it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.Signed-off-by Krishna Kumar <[email protected]>--- drivers/net/virtio_net.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)diff -ruNp
org/drivers/net/virtio_net.cnew/drivers/net/virtio_net.cdrivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0
-9@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_ /* This can happen with OOM and indirect buffers. */ if (unlikely(capacity < 0)) { - if (net_ratelimit()) { - if (likely(capacity == -ENOMEM)) { - dev_warn(&dev->dev, - "TX queue failure: out of memory\n"); - } else { - dev->stats.tx_fifo_errors++; - dev_warn(&dev->dev, - "Unexpected TX queue failure: %d\n", - capacity);
Quoted 8 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576
Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4
-2- } - } + if (net_ratelimit()) + dev_warn(&dev->dev, + "TX queue failure (%d): out of memory\n", + capacity);Hold on... you were getting
-ENOSPC, which shouldn't happen. What makes
you
Quoted 1 line
think it's out of memory?
virtqueue_add_buf_gfp returns only -ENOSPC on failure, whether direct or indirect descriptors are used, so isn't -ENOSPC "expected"? (vring_add_indirect returns -ENOMEM on memory failure, but that is masked out and we go direct which is the failure point).
Thanks,
- KK
Quoted 45 lines · On Fri, 29 Oct 2010 09:25:09 pm Krishna Kumar2 wrote:
Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:
Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 27 lines
Quoted 27 lines
I initially changed the check from
-ENOMEMto-ENOSPC, butvirtqueue_add_bufcan return only-ENOSPCwhen it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.Signed-off-by Krishna Kumar <[email protected]>--- drivers/net/virtio_net.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)diff -ruNp
org/drivers/net/virtio_net.cnew/drivers/net/virtio_net.cdrivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0
-9@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_ /* This can happen with OOM and indirect buffers. */ if (unlikely(capacity < 0)) { - if (net_ratelimit()) { - if (likely(capacity == -ENOMEM)) { - dev_warn(&dev->dev, - "TX queue failure: out of memory\n"); - } else { - dev->stats.tx_fifo_errors++; - dev_warn(&dev->dev, - "Unexpected TX queue failure: %d\n", - capacity);Quoted 8 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576
Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4
-2- } - } + if (net_ratelimit()) + dev_warn(&dev->dev, + "TX queue failure (%d): out of memory\n", + capacity);Hold on... you were getting
-ENOSPC, which shouldn't happen. What makes
Quoted 8 lines
you
Quoted 1 line
think it's out of memory?
virtqueue_add_buf_gfpreturns only-ENOSPCon failure, whether direct or indirect descriptors are used, so isn't-ENOSPC"expected"? (vring_add_indirectreturns-ENOMEMon memory failure, but that is masked out and we go direct which is the failure point).
Ah, OK, gotchya.
I'm not even sure the fallback to linear makes sense; if we're failing kmallocs we should probably just return -ENOMEM. Would mean we can tell the difference between "out of space" (which should never happen since we stop the queue when we have < 2+MAX_SKB_FRAGS slots left) and this case.
Michael, what do you think?
Thanks,
Rusty.
Michael S. Tsirkin
maintainerQuoted 62 lines · Rusty Russell
Quoted 45 lines · On Fri, 29 Oct 2010 09:25:09 pm Krishna Kumar2 wrote:
Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:
Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 27 lines
Quoted 27 lines
I initially changed the check from
-ENOMEMto-ENOSPC, butvirtqueue_add_bufcan return only-ENOSPCwhen it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.Signed-off-by Krishna Kumar <[email protected]>--- drivers/net/virtio_net.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)diff -ruNp
org/drivers/net/virtio_net.cnew/drivers/net/virtio_net.cdrivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0
-9@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_ /* This can happen with OOM and indirect buffers. */ if (unlikely(capacity < 0)) { - if (net_ratelimit()) { - if (likely(capacity == -ENOMEM)) { - dev_warn(&dev->dev, - "TX queue failure: out of memory\n"); - } else { - dev->stats.tx_fifo_errors++; - dev_warn(&dev->dev, - "Unexpected TX queue failure: %d\n", - capacity);Quoted 8 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576
Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4
-2- } - } + if (net_ratelimit()) + dev_warn(&dev->dev, + "TX queue failure (%d): out of memory\n", + capacity);Hold on... you were getting
-ENOSPC, which shouldn't happen. What makesQuoted 8 lines
you
Quoted 1 line
think it's out of memory?
virtqueue_add_buf_gfpreturns only-ENOSPCon failure, whether direct or indirect descriptors are used, so isn't-ENOSPC"expected"? (vring_add_indirectreturns-ENOMEMon memory failure, but that is masked out and we go direct which is the failure point).Ah, OK, gotchya.
I'm not even sure the fallback to linear makes sense; if we're failing kmallocs we should probably just return-ENOMEM. Would mean we can tell the difference between "out of space" (which should never happen since we stop the queue when we have < 2+MAX_SKB_FRAGS slots left) and this case.
Quoted 4 lines
Michael, what do you think?
Thanks,
Rusty.
Let's make sure I understand the issue: we use indirect buffers so we assume there's still a lot of place in the ring, then allocation for the indirect fails and so we return -ENOSPC?
So first, I agree it's a bug. But I am not sure killing the fallback is such a good idea: recovering from add buf failure is hard generally, we should try to accomodate if we can. Let's just fix the return code for now?
And generally, we should be smarter: as long as the ring is almost empty, and s/g list is short, it is a waste to use indirect buffers. BTW we have had a FIXME there for a long while, I think Yan suggested increasing that threshold to 3. Yan?
Further, maybe preallocating some memory for the indirect buffers might be a good idea.
In short, lots of good ideas, let's start with the minimal patch that is a good 2.6.37 candidate too. How about the following (untested)?
virtio: fix add_buf return code for OOM
add_buff returned ENOSPC on out of memory: this is a bug as at leats virtio-net expects ENOMEM and handles it specially. Fix that.
---
drivers/virtio/virtio_ring.c +2 -2
@@ -165,7 +165,7 @@ int virtqueue_add_buf_gfp(struct virtqueue *_vq, {struct vring_virtqueue *vq =to_vvq (_vq);unsigned int i, avail,uninitialized_var (prev ); -int head; +int head = -ENOSPC ;START_USE (vq);
@@ -191,7 +191,7 @@ int virtqueue_add_buf_gfp(struct virtqueue *_vq,if (out) vq->notify (&vq->vq);END_USE (vq); -return -ENOSPC ; +return head; }/* We're about to use some buffers from the free list. */
Signature · 1 line
MST
Michael S. Tsirkin
maintainerQuoted 88 lines · Michael S. Tsirkin
Quoted 62 lines · Rusty Russell
Quoted 45 lines · On Fri, 29 Oct 2010 09:25:09 pm Krishna Kumar2 wrote:
Rusty Russell <[email protected]> wrote on 10/29/2010 03:17:24 PM:
Quoted 1 line
Quoted 1 line
Oct 17 10:22:40 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:28:22 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:35:58 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 1 line
Quoted 1 line
Oct 17 10:41:06 localhost kernel: net eth0: Unexpected TX queuefailure: -28
Quoted 27 lines
Quoted 27 lines
I initially changed the check from
-ENOMEMto-ENOSPC, butvirtqueue_add_bufcan return only-ENOSPCwhen it doesn't have space for new request. Patch removes redundant checks but displays the failure errno.Signed-off-by Krishna Kumar <[email protected]>--- drivers/net/virtio_net.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)diff -ruNp
org/drivers/net/virtio_net.cnew/drivers/net/virtio_net.cdrivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +0
-9@@ -570,17 +570,10 @@ static netdev_tx_t start_xmit(struct sk_ /* This can happen with OOM and indirect buffers. */ if (unlikely(capacity < 0)) { - if (net_ratelimit()) { - if (likely(capacity == -ENOMEM)) { - dev_warn(&dev->dev, - "TX queue failure: out of memory\n"); - } else { - dev->stats.tx_fifo_errors++; - dev_warn(&dev->dev, - "Unexpected TX queue failure: %d\n", - capacity);Quoted 8 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 static netdev_tx_t start_xmit(struct sk_ · line 576
Quoted 6 lines
drivers/net/virtio_net.c 2010-10-21 17:37:45.000000000 +0530 renamed from drivers/net/virtio_net.c 2010-10-11 10:20:02.000000000 +0530 +4
-2- } - } + if (net_ratelimit()) + dev_warn(&dev->dev, + "TX queue failure (%d): out of memory\n", + capacity);Hold on... you were getting
-ENOSPC, which shouldn't happen. What makesQuoted 8 lines
you
Quoted 1 line
think it's out of memory?
virtqueue_add_buf_gfpreturns only-ENOSPCon failure, whether direct or indirect descriptors are used, so isn't-ENOSPC"expected"? (vring_add_indirectreturns-ENOMEMon memory failure, but that is masked out and we go direct which is the failure point).Ah, OK, gotchya.
I'm not even sure the fallback to linear makes sense; if we're failing kmallocs we should probably just return-ENOMEM. Would mean we can tell the difference between "out of space" (which should never happen since we stop the queue when we have < 2+MAX_SKB_FRAGS slots left) and this case.Quoted 4 lines
Michael, what do you think?
Thanks,
Rusty.Let's make sure I understand the issue: we use indirect buffers so we assume there's still a lot of place in the ring, then allocation for the indirect fails and so we return
-ENOSPC?So first, I agree it's a bug. But I am not sure killing the fallback is such a good idea: recovering from add buf failure is hard generally, we should try to accomodate if we can. Let's just fix the return code for now?
And generally, we should be smarter: as long as the ring is almost empty, and s/g list is short, it is a waste to use indirect buffers. BTW we have had a FIXME there for a long while, I think Yan suggested increasing that threshold to 3. Yan?
Further, maybe preallocating some memory for the indirect buffers might be a good idea.
In short, lots of good ideas, let's start with the minimal patch that is a good 2.6.37 candidate too. How about the following (untested)?
Quoted 7 lines
virtio: fix
add_bufreturn code for OOM
add_buffreturned ENOSPC on out of memory: this is a bug as at leats virtio-net expects ENOMEM and handles it specially. Fix that.Signed-off-by Michael S. Tsirkin <[email protected]>
I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.
And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.
So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
So below is what I propose now - as a replacement for my original patch. Krishna Kumar, Rusty, what do you think?
Separately I'm also considering moving the
if (vq->num_free < out + in)
check earlier in the function to keep all users honest, but need to check what the implications are for e.g. block. Thoughts on this?
---->
virtio: return correct capacity to users
We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail.
So return the number of buffers we can guarantee users.
drivers/virtio/virtio_ring.c +0 -3
@@ -230,9 +230,6 @@ add_head:pr_debug ("Added buffer head %i to %p\n" , head, vq);END_USE (vq); -/* If we're indirect, we can fit many (assuming not OOM). */ -if (vq->indirect) -return vq->num_free ? vq->vring.num :0 ;return vq->num_free; }EXPORT_SYMBOL_GPL (virtqueue_add_buf_gfp);
"Michael S. Tsirkin" <[email protected]> wrote on 11/04/2010 05:54:24 PM:
Quoted 17 lines
Quoted 1 line
Signed-off-by Michael S. Tsirkin <[email protected]>I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
Quoted 8 lines
So below is what I propose now - as a replacement for my original patch. Krishna Kumar, Rusty, what do you think?
Separately I'm also considering moving the
if (vq->num_free< out + in)
check earlier in the function to keep all users honest, but need to check what the implications are for e.g. block. Thoughts on this?
This looks like the right thing to do. Besides this, I think virtio-net still needs to remove check for ENOMEM? I will test this patch tomorrow.
Another question about add_recvbuf_small and add_recvbuf_big - both call virtqueue_add_buf_gfp with in+out > 1, and that can fail with -ENOSPC. So try_fill_recv gets -ENOSPC. When that happens, oom is not set to true, I thought it should have got set. Is this a bug?
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 22 lines · On Thu, Nov 04, 2010 at 09:47:04PM +0530, Krishna Kumar2 wrote:
"Michael S. Tsirkin" <[email protected]> wrote on 11/04/2010 05:54:24 PM:
Quoted 20 lines
Quoted 1 line
Signed-off-by Michael S. Tsirkin <[email protected]>I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
So below is what I propose now - as a replacement for my original patch. Krishna Kumar, Rusty, what do you think?
Quoted 8 lines
Quoted 5 lines
Separately I'm also considering moving the
if (vq->num_free< out + in)
check earlier in the function to keep all users honest, but need to check what the implications are for e.g. block. Thoughts on this?This looks like the right thing to do. Besides this, I think virtio-net still needs to remove check for ENOMEM?
Yes, the only valid reason for failure would be a unexpected error. No need to special-case ENOMEM anymore.
Quoted 11 lines
I will test this patch tomorrow.
Another question about
add_recvbuf_smallandadd_recvbuf_big- both callvirtqueue_add_buf_gfpwith in+out > 1, and that can fail with-ENOSPC. Sotry_fill_recvgets-ENOSPC. When that happens, oom is not set to true, I thought it should have got set. Is this a bug?Thanks,
- KK
I don't see a bug: on ENOSPC we don't need to (and can't) add any more buffers, we know we will make progress since there must be some buffers in the ring already, ENOMEM makes us try again later with more buffers (and possibly more aggressive GFP flag). What's wrong?
Signature · 1 line
MST
Quoted 11 lines · Michael S. Tsirkin
I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.
Quoted 3 lines
So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room, leading to worst-case assumptions and underutilization of the tx ring.
However, I lost that debate, and so your patch is the way it's supposed to work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.
I enhanced your commentry a little:
Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>
We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular, virtio_net can get into this situation under stress, and it drops packets and performs badly.
So return the number of buffers we can guarantee users.
Thanks!
Rusty.
Krishna Kumar2
Tested-byRusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:
Quoted 18 lines
Re: [PATCH]
virtio_net: Fix queue full checkQuoted 11 lines · Michael S. Tsirkin
I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.Quoted 3 lines
So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
Quoted 5 lines
I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room, leading to worst-case assumptions and underutilization of the tx ring.
However, I lost that debate, and so your patch is the way it's supposed
to
Quoted 15 lines
work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.
I enhanced your commentry a little:
Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular,
virtio_netcan get into this situation under stress, and it drops packets and performs badly.
Quoted 5 lines
So return the number of buffers we can guarantee users.
Signed-off-by Michael S. Tsirkin <[email protected]>Signed-off-by Rusty Russell <[email protected]>Reported-By Krishna Kumar2 <[email protected]>
I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but just in case:
I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of dev->stats.tx_fifo_errors, and reports "memory failure".
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 57 lines · On Tue, Nov 09, 2010 at 09:56:03AM +0530, Krishna Kumar2 wrote:
Rusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:
Quoted 18 lines
Re: [PATCH]
virtio_net: Fix queue full checkQuoted 11 lines · Michael S. Tsirkin
I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.Quoted 3 lines
So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
Quoted 5 lines
I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room, leading to worst-case assumptions and underutilization of the tx ring.
However, I lost that debate, and so your patch is the way it's supposed
to
Quoted 15 lines
work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.
I enhanced your commentry a little:
Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular,
virtio_netcan get into this situation under stress, and it drops packets and performs badly.Quoted 5 lines
So return the number of buffers we can guarantee users.
Signed-off-by Michael S. Tsirkin <[email protected]>Signed-off-by Rusty Russell <[email protected]>Reported-By Krishna Kumar2 <[email protected]>I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but just in case:Signed-off-by Michael S. Tsirkin <[email protected]>Signed-off-by Rusty Russell <[email protected]>Reported-By Krishna Kumar2 <[email protected]>Tested-by Krishna Kumar2 <[email protected]>
Quoted 7 lines
I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of
dev->stats.tx_fifo_errors, and reports "memory failure".Thanks,
- KK
So I think your patch on top of this one would be wrong:
we actually make sure out of memory does not affect TX path at all, so any error would be unexpected.
Incrementing tx fifo errors is probably also helpful for debugging.
If you like, we could kill the special handling for ENOMEM. Not sure whether it matters.
Signature · 1 line
MST
"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:45:55 PM:
Quoted 25 lines
Re: [PATCH]
virtio_net: Fix queue full checkQuoted 22 lines · On Tue, Nov 09, 2010 at 09:56:03AM +0530, Krishna Kumar2 wrote:
Rusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:
Quoted 20 lines
Re: [PATCH]
virtio_net: Fix queue full checkQuoted 11 lines · Michael S. Tsirkin
I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.Quoted 3 lines
So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room,
leading to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
worst-case assumptions and underutilization of the tx ring.
However, I lost that debate, and so your patch is the way it's
supposed
Quoted 18 lines
Quoted 18 lines
to
Quoted 17 lines
work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.
I enhanced your commentry a little:
Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular,
virtio_netcan get into this situation under stress, and it drops packets and performs badly.So return the number of buffers we can guarantee users.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Signed-off-by Michael S. Tsirkin <[email protected]>Signed-off-by Rusty Russell <[email protected]>Reported-By Krishna Kumar2 <[email protected]>I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but just
in
Quoted 15 lines
Quoted 7 lines
case:
Signed-off-by Michael S. Tsirkin <[email protected]>Signed-off-by Rusty Russell <[email protected]>Reported-By Krishna Kumar2 <[email protected]>Tested-by Krishna Kumar2 <[email protected]>Quoted 7 lines
I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of
dev->stats.tx_fifo_errors, and reports "memory failure".Thanks,
- KK
Quoted 8 lines
So I think your patch on top of this one would be wrong:
we actually make sure out of memory does not affect TX path at all, so any error would be unexpected.Incrementing tx fifo errors is probably also helpful for debugging.
If you like, we could kill the special handling for ENOMEM. Not sure whether it matters.
Since that is dead code, we could remove it (and the fifo error should disappear too - tx_dropped should be the only counter to be incremented?). Sorry if I misunderstood something.
Thanks,
- KK
Michael S. Tsirkin
maintainerQuoted 82 lines · On Tue, Nov 09, 2010 at 09:00:58PM +0530, Krishna Kumar2 wrote:
"Michael S. Tsirkin" <[email protected]> wrote on 11/09/2010 06:45:55 PM:
Quoted 25 lines
Re: [PATCH]
virtio_net: Fix queue full checkQuoted 22 lines · On Tue, Nov 09, 2010 at 09:56:03AM +0530, Krishna Kumar2 wrote:
Rusty Russell <[email protected]> wrote on 11/08/2010 04:38:47 AM:
Quoted 20 lines
Re: [PATCH]
virtio_net: Fix queue full checkQuoted 11 lines · Michael S. Tsirkin
I thought about this some more. I think the original code is actually correct in returning ENOSPC: indirect buffers are nice, but it's a mistake
to rely on them as a memory allocation might fail.And if you look at virtio-net, it is dropping packets under memory pressure which is not really a happy outcome:
the packet will get freed, reallocated and we get another one, adding pressure on the allocator instead of releasing it until we free up some buffers.Quoted 3 lines
So I now think we should calculate the capacity assuming non-indirect entries, and if we manage to use indirect, all the better.
I've long said it's a weakness in the network stack that it insists drivers stop the tx queue before they *might* run out of room,
leading to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
worst-case assumptions and underutilization of the tx ring.
However, I lost that debate, and so your patch is the way it's
supposed
Quoted 18 lines
Quoted 18 lines
to
Quoted 17 lines
work. The other main indirect user (block) doesn't care as its queue allows for post-attempt blocking.
I enhanced your commentry a little:
Subject: virtio: return correct capacity to users
Date: Thu, 4 Nov 2010 14:24:24 +0200
From: "Michael S. Tsirkin" <[email protected]>We can't rely on indirect buffers for capacity calculations because they need a memory allocation which might fail. In particular,
virtio_netcan get into this situation under stress, and it drops packets and performs badly.So return the number of buffers we can guarantee users.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Signed-off-by Michael S. Tsirkin <[email protected]>Signed-off-by Rusty Russell <[email protected]>Reported-By Krishna Kumar2 <[email protected]>I have tested this patch for 3-4 hours but so far I have not got the tx full
error. I am not sure if "Tested-By" applies to this situation, but justin
Quoted 15 lines
Quoted 7 lines
case:
Signed-off-by Michael S. Tsirkin <[email protected]>Signed-off-by Rusty Russell <[email protected]>Reported-By Krishna Kumar2 <[email protected]>Tested-by Krishna Kumar2 <[email protected]>Quoted 7 lines
I think both this patch and the original patch I submitted are needed? That patch removes ENOMEM check and the increment of
dev->stats.tx_fifo_errors, and reports "memory failure".Thanks,
- KK
Quoted 8 lines
So I think your patch on top of this one would be wrong:
we actually make sure out of memory does not affect TX path at all, so any error would be unexpected.Incrementing tx fifo errors is probably also helpful for debugging.
If you like, we could kill the special handling for ENOMEM. Not sure whether it matters.
Quoted 7 lines
Since that is dead code, we could remove it (and the fifo error should disappear too -
tx_droppedshould be the only counter to be incremented?). Sorry if I misunderstood something.Thanks,
- KK
It's just a sanity check. The capacity checking is tricky enough that I'm happier with some way to detect overflow both from ifconfig and dmesg.
I don't really care which counter gets incremented really, since this is some TX bug fifo error seems appropriate but I don't really care much.
Signature · 1 line
MST