DCCP: handle EAGAIN case
PATCH 1/1 DCCP: handle EAGAIN case
This might be something to help with the poor throughput.
By using this patch, I have observed an increase of tx rates from 600..800kbits/sec (without this modification) up to 94.8 Mbits/sec (which is as good as TCP gets).
Please test - I'd be curious what your experiences are with this one.
--------------------------> Patch <----------------------------------------
[DCCP]: Handle EAGAIN in dccp_write_xmit
If dccp_wait_for_ccid returns -EAGAIN, the packet is silently deqeued and not sent. At the receiver this counts as loss, increases the loss rate and reduces throughput.
This patch handles the EAGAIN case by restarting afresh to peek into the write queue and send the packet.
--- net/dccp/output.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
net/dccp/output.c +4 -1
@@ -255,8 +255,11 @@ void dccp_write_xmit(struct sock *sk, in err =dccp_wait_for_ccid (sk, skb, &timeo); timeo =DCCP_XMIT_TIMEO ; } -if (err) +if (err) { +if (err = -EAGAIN ) +continue ;DCCP_BUG ("err=%d after dccp_wait_for_ccid" , err); + } }skb_dequeue (&sk->sk_write_queue);
Gerrit,
This one doesn't seem right to me because we get EAGAIN in theory only if packet delay is greater than timeo which is 30 seconds, or timeo is 0, or delay < 0 (which shouldn't happen as msecs_to_jiffies returns unsigned). EAGAIN is only set by dccp_wait_for_ccid.
If this was the case this line should have triggered and put info in your logs:
if (err)
DCCP_BUG("err=%d after dccp_wait_for_ccid", err)
Any of these cases indicates a far more fundamental problem and this is masking it.... Can you put some debug in and find out more. I'll add some in too once I finish off another testing cycle I'm working on.
Regards,
Ian
Quoted 36 lines · Gerrit Renker
This might be something to help with the poor throughput.
By using this patch, I have observed an increase of tx rates from 600..800kbits/sec (without this modification) up to 94.8 Mbits/sec (which is as good as TCP gets).
Please test - I'd be curious what your experiences are with this one.--------------------------> Patch <----------------------------------------
[DCCP]: Handle EAGAIN in
dccp_write_xmitIf
dccp_wait_for_ccidreturns-EAGAIN, the packet is silently deqeued and not sent. At the receiver this counts as loss, increases the loss rate and reduces throughput.
This patch handles the EAGAIN case by restarting afresh to peek into the write queue and send the packet.Signed-off-by Gerrit Renker <[email protected]>--- net/dccp/output.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)net/dccp/output.c +4
-1@@ -255,8 +255,11 @@ void dccp_write_xmit(struct sock *sk, in err =dccp_wait_for_ccid (sk, skb, &timeo); timeo =DCCP_XMIT_TIMEO ; } -if (err) +if (err) { +if (err = -EAGAIN ) +continue ;DCCP_BUG ("err=%d after dccp_wait_for_ccid" , err); + } }skb_dequeue (&sk->sk_write_queue);
Signature · 3 lines
Web: http://wand.net.nz/~iam4 Blog: http://imcdnzl.blogspot.com WAND Network Research Group
| This one doesn't seem right to me because we get EAGAIN in theory only | if packet delay is greater than timeo which is 30 seconds, or timeo is | 0, or delay < 0 (which shouldn't happen as msecs_to_jiffies returns | unsigned). EAGAIN is only set by dccp_wait_for_ccid.
That is true, but this patch did not arise from theoretical considerations. Let me break it into:
a) You helped to uncover at least one real bug here:
long delay;
delay = msecs_to_jiffies(rc);
if (delay > *timeo || delay < 0)
I think we should remove the test `delay < 0'. It is either always true (due to the return type of msecs_to_jiffies); but if it is true it is due to a conversion error from unsigned to signed - this is possible since the range of the former is twice the range of the latter.
A similar problem arises in many of the timeval operations, which return suseconds_t (a typedef for signed long) and are stored in u32. I am working on this now.
b) Further test results
Below you asked for more testing: on top of the brandnew davem-2.6 (with the last lot of DCCP patches), I can no longer see a serious performance difference. When I unapply the -EAGAIN patch, the error messages (see below) are also no longer present. Performance on 2.4Ghz P4 uniprocessor is up to 51.6Mbit/sec, on a Xeon 3 Ghz SMP it is up to 92.2 Mbit/sec (4Mbit/sec less without the patch, but there are likely other influences).
c) Why I think that EAGAIN should be considered
The value of `timeo' is an arbitrary value set in dccp_write_xmit, it has nothing to do with CCID 3 internals. When the value of *timeo in dccp_wait_for_ccid has reached 0 or if ccid_hc_tx_send_packet() returns a delay value which is greater than the *timeo, the following will happen:
- the BUG warning is triggered with the message "err=-11 ..." (see below)
- the next step is:
skb_dequeue(&sk->sk_write_queue
if (err = 0) {
/* ... */
} else
kfree(skb); /* packet is thrown into the bin */
==> The packet is silently discarded and the receiver will register it as a lost packet.
I think this is not the `right' way to deal with EAGAIN cases. Again, we can follow your suggestion and keep on testing, but I think this should be handled in a smarter way. In particular, what happens if *timeo = 0 and if the packet (according to CCID 3 calculations) is ready to be sent? Currently it is discarded, which somehow is not right.
==> There is another case: interrupted system call ("err=-4 ...", corresponds to -EINTR).
I get it for instance when I kill the iperf client with CTRL-C. A more graceful handling of this case seems desirable;
|
| If this was the case this line should have triggered and put info in your logs:
| if (err)
| DCCP_BUG("err=%d after dccp_wait_for_ccid", err)Yes, it did: the message was "err=-11 ..." (which corresponds to -EAGAIN). Stumbling over these error messages was the reason I tested/sent this patch.
Quoted 10 lines · Gerrit Renker
| This one doesn't seem right to me because we get EAGAIN in theory only | if packet delay is greater than timeo which is 30 seconds, or timeo is | 0, or delay < 0 (which shouldn't happen as msecs_to_jiffies returnsa) You helped to uncover at least one real bug here:
long delay; delay = msecs_to_jiffies(rc); if (delay > *timeo || delay < 0)
I was thinking of changing the type from long to unsigned long to match and removing the test too. I agree totally.
Quoted 7 lines
b) Further test results
Below you asked for more testing: on top of the brandnew davem-2.6 (with the last lot of DCCP patches), I can no longer see a serious performance difference. When I unapply the-EAGAINpatch, the error messages (see below) are also no longer present. Performance on 2.4Ghz P4 uniprocessor is up to 51.6Mbit/sec, on a Xeon 3 Ghz SMP it is up to 92.2 Mbit/sec (4Mbit/sec less without the patch, but there are likely other influences).
Yes - some of these issues seem intermittent. I was also checking history and there was a bug that timeo wasn't getting reset back to 30 seconds and that occurred fairly recently so you might have missed that patch.
Quoted 22 lines
c) Why I think that EAGAIN should be considered
The value of `timeo' is an arbitrary value set indccp_write_xmit, it has nothing to do with CCID 3 internals. When the value of *timeo indccp_wait_for_ccidhas reached 0 or ifccid_hc_tx_send_packet()returns a delay value which is greater than the *timeo, the following will happen:
- the BUG warning is triggered with the message "err=-11 ..." (see below)
- the next step is:skb_dequeue(&sk->sk_write_queue
if (err = 0) {
/* ... */} else
kfree(skb); /* packet is thrown into the bin */==> The packet is silently discarded and the receiver will register it as a lost packet.
I think this is not the `right' way to deal with EAGAIN cases. Again, we can follow your suggestion and keep on testing, but I think this should be handled in a smarter way. In particular, what happens if *timeo = 0 and if the packet (according to CCID 3 calculations) is ready to be sent? Currently it is discarded, which somehow is not right.
Quoted 4 lines
==> There is another case: interrupted system call ("err=-4 ...", corresponds to
-EINTR).
I get it for instance when I kill the iperf client with CTRL-C. A more graceful handling of this case seems desirable;
Agree on both counts.
Quoted 7 lines
| | If this was the case this line should have triggered and put info in your logs: | if (err) | DCCP_BUG("err=%d after dccp_wait_for_ccid", err)Yes, it did: the message was "err=-11 ..." (which corresponds to
-EAGAIN). Stumbling over these error messages was the reason I tested/sent this patch.
Understand.
Signature · 3 lines
Web: http://wand.net.nz/~iam4 Blog: http://imcdnzl.blogspot.com WAND Network Research Group
Quoting Ian McDonald:
| > a) You helped to uncover at least one real bug here: | > | > long delay; | > | > delay = msecs_to_jiffies(rc); | > if (delay > *timeo || delay < 0) | > | | I was thinking of changing the type from long to unsigned long to | match and removing the test too. I agree totally.
For this case I have prepared a patch. With regard to the EAGAIN case I was half waiting for Arnaldo to say something, for the moment I have put it here:
http://www.erg.abdn.ac.uk/users/gerrit/dccp/experimental_patches/15a_handle_EAGAIN.diff
I think we should fix the EAGAIN case. Maybe also something for EINTR (err=-4).
Quoted 11 lines · Gerrit Renker
Quoting Ian McDonald:
| > a) You helped to uncover at least one real bug here: | > | > long delay; | > | > delay = msecs_to_jiffies(rc); | > if (delay > *timeo || delay < 0) | > | | I was thinking of changing the type from long to unsigned long to | match and removing the test too. I agree totally.
Quoted 6 lines
For this case I have prepared a patch. With regard to the EAGAIN case I was half waiting for Arnaldo to say something, for the moment I have put it here:
http://www.erg.abdn.ac.uk/users/gerrit/dccp/experimental_patches/15a_handle_EAGAIN.diffI think we should fix the EAGAIN case. Maybe also something for EINTR (err=-4).
I was thinking about this some more on the way up to work as I biked...
We will only really have a problem with EAGAIN staying in the loop if we have unbounded tx queues and low x. However I don't think that will normally be the case as that has to be explicity set via sysctl now.
We do need to be careful with changing the type of delay. We should change the type of timeo as well to avoid signed/unsigned issues but then we have the comparison...
And then as I write this realise we are being way too complicated. We should remove timeo altogether!!! As we are doing this in effect by that continue and that is perfectly valid is this is blocking I/O. So we remove timeo, remove DCCP_XMIT_TIMEO and simplify it a fair bit.
Still thinking about why we are getting EAGAIN and suspect it is due to you not having that patch but I'll probably add more debugging in.
Anyway feel free to write a patch or I might when I get home from work or tomorrow.
Ian
Signature · 3 lines
Web: http://wand.net.nz/~iam4 Blog: http://imcdnzl.blogspot.com WAND Network Research Group
Quoting Ian McDonald:
| And then as I write this realise we are being way too complicated. We | should remove timeo altogether!!! As we are doing this in effect by | that continue and that is perfectly valid is this is blocking I/O. So | we remove timeo, remove DCCP_XMIT_TIMEO and simplify it a fair bit.
Your idea seems nice and to me it also seemed that it should be simplified.
| Still thinking about why we are getting EAGAIN and suspect it is due
| to you not having that patch but I'll probably add more debugging in.
No, I have that patch in my code:
if (err > 0) {
if (!block) {
sk_reset_timer(sk, &dp->dccps_xmit_timer,
msecs_to_jiffies(err)+jiffies);
break;
} else {
err = dccp_wait_for_ccid(sk, skb, &timeo);
timeo = DCCP_XMIT_TIMEO; /* <== I think you mean this one */
}| Anyway feel free to write a patch or I might when I get home from work | or tomorrow
I'd be happy to see a patch - maybe several cases can be folded into one.