Skip to contentSkip to conversation

DCCP: handle EAGAIN case

PATCH In review Gerrit Renker 2 participants 7 messages Dec 3, 2006 – Dec 4, 2006 Networking

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 DCCP: handle EAGAIN case

This might be something to help with the poor throughput.
By using this patch, I have observed an increase of tx rates from 600..800kbits/sec (without this modification) up to 94.8 Mbits/sec (which is as good as TCP gets).
Please test - I'd be curious what your experiences are with this one.

--------------------------> Patch <----------------------------------------

[DCCP]: Handle EAGAIN in dccp_write_xmit

If dccp_wait_for_ccid returns -EAGAIN, the packet is silently deqeued and not sent. At the receiver this counts as loss, increases the loss rate and reduces throughput.
This patch handles the EAGAIN case by restarting afresh to peek into the write queue and send the packet.

---
 net/dccp/output.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/dccp/output.c +4 -1
@@ -255,8 +255,11 @@ void dccp_write_xmit(struct sock *sk, in
 				err = dccp_wait_for_ccid(sk, skb, &timeo);
 				timeo = DCCP_XMIT_TIMEO;
 			}
-			if (err)
+			if (err) {
+				if (err = -EAGAIN)
+					continue;
 				DCCP_BUG("err=%d after dccp_wait_for_ccid", err);
+			}
 		}
 
 		skb_dequeue(&sk->sk_write_queue);
#1 lore raw

Gerrit,

This one doesn't seem right to me because we get EAGAIN in theory only if packet delay is greater than timeo which is 30 seconds, or timeo is 0, or delay < 0 (which shouldn't happen as msecs_to_jiffies returns unsigned). EAGAIN is only set by dccp_wait_for_ccid.

If this was the case this line should have triggered and put info in your logs:
if (err)
DCCP_BUG("err=%d after dccp_wait_for_ccid", err)

Any of these cases indicates a far more fundamental problem and this is masking it.... Can you put some debug in and find out more. I'll add some in too once I finish off another testing cycle I'm working on.

Regards,

Ian

Quoted 36 lines · Gerrit Renker

This might be something to help with the poor throughput.
By using this patch, I have observed an increase of tx rates from 600..800kbits/sec (without this modification) up to 94.8 Mbits/sec (which is as good as TCP gets).
Please test - I'd be curious what your experiences are with this one.

--------------------------> Patch <----------------------------------------

[DCCP]: Handle EAGAIN in dccp_write_xmit

If dccp_wait_for_ccid returns -EAGAIN, the packet is silently deqeued and not sent. At the receiver this counts as loss, increases the loss rate and reduces throughput.
This patch handles the EAGAIN case by restarting afresh to peek into the write queue and send the packet.

---
 net/dccp/output.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/dccp/output.c +4 -1
@@ -255,8 +255,11 @@ void dccp_write_xmit(struct sock *sk, in
                                err = dccp_wait_for_ccid(sk, skb, &timeo);
                                timeo = DCCP_XMIT_TIMEO;
                        }
-                       if (err)
+                       if (err) {
+                               if (err = -EAGAIN)
+                                       continue;
                                DCCP_BUG("err=%d after dccp_wait_for_ccid", err);
+                       }
                }
 
                skb_dequeue(&sk->sk_write_queue);
Signature · 3 lines
Web: http://wand.net.nz/~iam4
Blog: http://imcdnzl.blogspot.com
WAND Network Research Group
#2 lore raw
In reply to Gerrit Renker
|  This one doesn't seem right to me because we get EAGAIN in theory only
|  if packet delay is greater than timeo which is 30 seconds, or timeo is
|  0, or delay < 0 (which shouldn't happen as msecs_to_jiffies returns
|  unsigned). EAGAIN is only set by dccp_wait_for_ccid.

That is true, but this patch did not arise from theoretical considerations. Let me break it into:

a) You helped to uncover at least one real bug here:

long delay;

delay = msecs_to_jiffies(rc);
if (delay > *timeo || delay < 0)

I think we should remove the test `delay < 0'. It is either always true (due to the return type of msecs_to_jiffies); but if it is true it is due to a conversion error from unsigned to signed - this is possible since the range of the former is twice the range of the latter.
A similar problem arises in many of the timeval operations, which return suseconds_t (a typedef for signed long) and are stored in u32. I am working on this now.

b) Further test results
Below you asked for more testing: on top of the brandnew davem-2.6 (with the last lot of DCCP patches), I can no longer see a serious performance difference. When I unapply the -EAGAIN patch, the error messages (see below) are also no longer present. Performance on 2.4Ghz P4 uniprocessor is up to 51.6Mbit/sec, on a Xeon 3 Ghz SMP it is up to 92.2 Mbit/sec (4Mbit/sec less without the patch, but there are likely other influences).

c) Why I think that EAGAIN should be considered
The value of `timeo' is an arbitrary value set in dccp_write_xmit, it has nothing to do with CCID 3 internals. When the value of *timeo in dccp_wait_for_ccid has reached 0 or if ccid_hc_tx_send_packet() returns a delay value which is greater than the *timeo, the following will happen:
- the BUG warning is triggered with the message "err=-11 ..." (see below)
- the next step is:

skb_dequeue(&sk->sk_write_queue
if (err = 0) {
/* ... */

} else
kfree(skb); /* packet is thrown into the bin */

==> The packet is silently discarded and the receiver will register it as a lost packet.
I think this is not the `right' way to deal with EAGAIN cases. Again, we can follow your suggestion and keep on testing, but I think this should be handled in a smarter way. In particular, what happens if *timeo = 0 and if the packet (according to CCID 3 calculations) is ready to be sent? Currently it is discarded, which somehow is not right.

==> There is another case: interrupted system call ("err=-4 ...", corresponds to -EINTR).
I get it for instance when I kill the iperf client with CTRL-C. A more graceful handling of this case seems desirable;

|  
|  If this was the case this line should have triggered and put info in your logs:
|  			if (err)
|  				DCCP_BUG("err=%d after dccp_wait_for_ccid", err)

Yes, it did: the message was "err=-11 ..." (which corresponds to -EAGAIN). Stumbling over these error messages was the reason I tested/sent this patch.

#3 lore raw
In reply to Gerrit Renker
Quoted 10 lines · Gerrit Renker
|  This one doesn't seem right to me because we get EAGAIN in theory only
|  if packet delay is greater than timeo which is 30 seconds, or timeo is
|  0, or delay < 0 (which shouldn't happen as msecs_to_jiffies returns

a) You helped to uncover at least one real bug here:

    long delay;

    delay = msecs_to_jiffies(rc);
    if (delay > *timeo || delay < 0)

I was thinking of changing the type from long to unsigned long to match and removing the test too. I agree totally.

Quoted 7 lines

b) Further test results
Below you asked for more testing: on top of the brandnew davem-2.6 (with the last lot of DCCP patches), I can no longer see a serious performance difference. When I unapply the -EAGAIN patch, the error messages (see below) are also no longer present. Performance on 2.4Ghz P4 uniprocessor is up to 51.6Mbit/sec, on a Xeon 3 Ghz SMP it is up to 92.2 Mbit/sec (4Mbit/sec less without the patch, but there are likely other influences).

Yes - some of these issues seem intermittent. I was also checking history and there was a bug that timeo wasn't getting reset back to 30 seconds and that occurred fairly recently so you might have missed that patch.

Quoted 22 lines

c) Why I think that EAGAIN should be considered
The value of `timeo' is an arbitrary value set in dccp_write_xmit, it has nothing to do with CCID 3 internals. When the value of *timeo in dccp_wait_for_ccid has reached 0 or if ccid_hc_tx_send_packet() returns a delay value which is greater than the *timeo, the following will happen:
- the BUG warning is triggered with the message "err=-11 ..." (see below)
- the next step is:

skb_dequeue(&sk->sk_write_queue
if (err = 0) {
/* ... */

} else
kfree(skb); /* packet is thrown into the bin */

==> The packet is silently discarded and the receiver will register it as a lost packet.
I think this is not the `right' way to deal with EAGAIN cases. Again, we can follow your suggestion and keep on testing, but I think this should be handled in a smarter way. In particular, what happens if *timeo = 0 and if the packet (according to CCID 3 calculations) is ready to be sent? Currently it is discarded, which somehow is not right.

Quoted 4 lines

==> There is another case: interrupted system call ("err=-4 ...", corresponds to -EINTR).
I get it for instance when I kill the iperf client with CTRL-C. A more graceful handling of this case seems desirable;

Agree on both counts.

Quoted 7 lines
|
|  If this was the case this line should have triggered and put info in your logs:
|                       if (err)
|                               DCCP_BUG("err=%d after dccp_wait_for_ccid", err)

Yes, it did: the message was "err=-11 ..." (which corresponds to -EAGAIN). Stumbling over these error messages was the reason I tested/sent this patch.

Understand.

Signature · 3 lines
Web: http://wand.net.nz/~iam4
Blog: http://imcdnzl.blogspot.com
WAND Network Research Group
#4 lore raw
In reply to Gerrit Renker

Quoting Ian McDonald:

|  >  a) You helped to uncover at least one real bug here:
|  >
|  >     long delay;
|  >
|  >     delay = msecs_to_jiffies(rc);
|  >     if (delay > *timeo || delay < 0)
|  >
|  
|  I was thinking of changing the type from long to unsigned long to
|  match and removing the test too. I agree totally.

For this case I have prepared a patch. With regard to the EAGAIN case I was half waiting for Arnaldo to say something, for the moment I have put it here:
http://www.erg.abdn.ac.uk/users/gerrit/dccp/experimental_patches/15a_handle_EAGAIN.diff

I think we should fix the EAGAIN case. Maybe also something for EINTR (err=-4).

#5 lore raw
In reply to Gerrit Renker
Quoted 11 lines · Gerrit Renker

Quoting Ian McDonald:

|  > a) You helped to uncover at least one real bug here:
|  >
|  > long delay;
|  >
|  > delay = msecs_to_jiffies(rc);
|  > if (delay > *timeo || delay < 0)
|  >
|
|  I was thinking of changing the type from long to unsigned long to
|  match and removing the test too. I agree totally.
Quoted 6 lines

For this case I have prepared a patch. With regard to the EAGAIN case I was half waiting for Arnaldo to say something, for the moment I have put it here:
http://www.erg.abdn.ac.uk/users/gerrit/dccp/experimental_patches/15a_handle_EAGAIN.diff

I think we should fix the EAGAIN case. Maybe also something for EINTR (err=-4).

I was thinking about this some more on the way up to work as I biked...

We will only really have a problem with EAGAIN staying in the loop if we have unbounded tx queues and low x. However I don't think that will normally be the case as that has to be explicity set via sysctl now.

We do need to be careful with changing the type of delay. We should change the type of timeo as well to avoid signed/unsigned issues but then we have the comparison...

And then as I write this realise we are being way too complicated. We should remove timeo altogether!!! As we are doing this in effect by that continue and that is perfectly valid is this is blocking I/O. So we remove timeo, remove DCCP_XMIT_TIMEO and simplify it a fair bit.

Still thinking about why we are getting EAGAIN and suspect it is due to you not having that patch but I'll probably add more debugging in.

Anyway feel free to write a patch or I might when I get home from work or tomorrow.

Ian

Signature · 3 lines
Web: http://wand.net.nz/~iam4
Blog: http://imcdnzl.blogspot.com
WAND Network Research Group
#6 lore raw
In reply to Gerrit Renker

Quoting Ian McDonald:

|  And then as I write this  realise we are being way too complicated. We
|  should remove timeo altogether!!! As we are doing this in effect by
|  that continue and that is perfectly valid is this is blocking I/O. So
|  we remove timeo, remove DCCP_XMIT_TIMEO and simplify it a fair bit.

Your idea seems nice and to me it also seemed that it should be simplified.

|  Still thinking about why we are getting EAGAIN and suspect it is due
|  to you not having that patch but I'll probably add more debugging in.
No, I have that patch in my code:
	if (err > 0) {
			if (!block) {
				sk_reset_timer(sk, &dp->dccps_xmit_timer,
						msecs_to_jiffies(err)+jiffies);
				break;
			} else {
				err = dccp_wait_for_ccid(sk, skb, &timeo);
				timeo = DCCP_XMIT_TIMEO;          /* <== I think you mean this one */
			}
|  Anyway feel free to write a patch or I might when I get home from work
|  or tomorrow

I'd be happy to see a patch - maybe several cases can be folded into one.