Use skb timestamp for receiver side

This patch replaces calls to do_gettimeofday at the receiver CCID 3 with skb timestamps. The skb timestamps are computed earlier in the receive path, experimental measurements have shown that up to several hundred microseconds can lie between the skb receive timestamp and the timestamp taken when CCID 3 receives the packet. This difference has a negative impact on RTT estimation (reduced accuracy).

---
 net/dccp/ccids/lib/packet_history.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

index b40b9f9..330b58f 100644
@@ -151,7 +151,7 @@ static inline struct dccp_rx_hist_entry *
 		entry->dccphrx_ccval = dh->dccph_ccval;
 		entry->dccphrx_type  = dh->dccph_type;
 		entry->dccphrx_ndp   = ndp;
-		do_gettimeofday(&entry->dccphrx_tstamp);
+		skb_get_timestamp(skb, &entry->dccphrx_tstamp);
 	}
 
 	return entry;

1.5.0.6

From: Arnaldo Carvalho de Melo <[email protected]>
Date: Thu, 12 Apr 2007 18:14:54 -0300

And this creates an even deeper dependency on the global packet timestamping facility. Those are supposed to be enabled only in obscure circumstances because timestamping every packet is incredibly expensive.

Please find a better way.

Well, there seems to be just one better way, to timestamp the packet as soon as it enters the dccp stack, in dccp_v[46]_rcv, and consider the time from the driver -> ip v[46] -> dccp_v[46]_ as "in the wire".

- Arnaldo

I didn't know this time stamping was expensive but I knew the way we were trying to optimise LAN is wrong. I say LAN because a few microseconds or even milliseconds difference on a WAN link makes bugger all difference in throughput. DCCP takes into account operating system granularity etc and if we are running lossless (i.e. receiver can cope with receiving link at full speed) then we can transmit at line speed. I've tested this myself.

This illustrates a trend by you Gerrit to try and micro-optimise what you perceive as bottlenecks and not realise the impact. You have been also promoting possible use of hrtimers which would be even worse for example in far of terms of loading a system.

Gerrit - you send us a missive on how we should humbly accept your patches and stop arguing with you. Obviously Eddie (co-author of DCCP spec) and myself (converted DCCP CCID3 to Linux 2.6) aren't good enough. Maybe now Dave Miller says something you will listen or do you need Linus Torvalds to tell you to stop being a fucking idiot at times? Yes you write lots of good code but you write some bad code at times also. I probably write more bad code than you and less volume of code but I try to listen. Stop pretending to be a deity.

Ian

Web: http://wand.net.nz/~iam4/
Blog: http://iansblog.jandi.co.nz
WAND Network Research Group

From: "Ian McDonald" <[email protected]>
Date: Fri, 13 Apr 2007 09:50:30 +1200

You want the scheduling delays and other issues that can delay DCCP input packet processing to get factored into the RTT, so that the sender will pace properly.

Trying to get perfect timestamping and RTT measurements, and ignoring scheduling delays, is quite foolhardy and shows a lack of understanding of how queueing really works.

Yes exactly and DCCP spec caters for this. Thank you for summing it up succinctly.

Web: http://wand.net.nz/~iam4/
Blog: http://iansblog.jandi.co.nz
WAND Network Research Group

Cool it Ian.

I think you are interpreting something into the email I sent which is not in there. This is exactly the kind of talk I was asking for and I am glad that Dave reviewed this.

Please read that email again.

I am asking for fair technical review here and rather than calling me names you could put some care into what you write.

If for instance you read the documentation accompanying that patch, there is a difference between 100 usec ... 1msec between skb_get_timestamp and taking the timestamp in the CCID3 receiver. It all adds up.

With regard to hrtimers - no patch was ever submitted, for the same reason that you are stating. But with the current schedule_timeout, the controllable limit is at about 12Mbits/sec when HZ00. Above that it is no longer controlled, let alone congestion- controlled.

Quoting Ian McDonald:

|  I didn't know this time stamping was expensive but I knew the way we
|  were trying to optimise LAN is wrong. I say LAN because a few
|  microseconds or even milliseconds difference on a WAN link makes
|  bugger all difference in throughput. DCCP takes into account operating
|  system granularity etc and if we are running lossless (i.e. receiver
|  can cope with receiving link at full speed) then we can transmit at
|  line speed. I've tested this myself.
|  
|  This illustrates a trend by you Gerrit to try and micro-optimise what
|  you perceive as bottlenecks and not realise the impact. You have been
|  also promoting possible use of hrtimers which would be even worse for
|  example in far of terms of loading a system.
| 
|  Gerrit - you send us a missive on how we should humbly accept your
|  patches and stop arguing with you. Obviously Eddie (co-author of DCCP
|  spec) and myself (converted DCCP CCID3 to Linux 2.6) aren't good
|  enough. Maybe now Dave Miller says something you will listen or do you
|  need Linus Torvalds to tell you to stop being a fucking idiot at
|  times? Yes you write lots of good code but you write some bad code at
|  times also. I probably write more bad code than you and less volume of
|  code but I try to listen. Stop pretending to be a deity.
|  

Same comment. Please do read the documentation accompanying the patch.

Quoting David Miller:

|  From: "Ian McDonald" <[email protected]>
|  Date: Fri, 13 Apr 2007 09:50:30 +1200
|  
|  > I didn't know this time stamping was expensive but I knew the way we
|  > were trying to optimise LAN is wrong. I say LAN because a few
|  > microseconds or even milliseconds difference on a WAN link makes
|  > bugger all difference in throughput. DCCP takes into account operating
|  > system granularity etc and if we are running lossless (i.e. receiver
|  > can cope with receiving link at full speed) then we can transmit at
|  > line speed. I've tested this myself.
|  
|  You want the scheduling delays and other issues that can
|  delay DCCP input packet processing to get factored into
|  the RTT, so that the sender will pace properly.
|  
|  Trying to get perfect timestamping and RTT measurements,
|  and ignoring scheduling delays, is quite foolhardy and shows
|  a lack of understanding of how queueing really works.
|  
|  

Sorry, I misread the email. Please see below.

|  You want the scheduling delays and other issues that can
|  delay DCCP input packet processing to get factored into
|  the RTT, so that the sender will pace properly.
|  
|  Trying to get perfect timestamping and RTT measurements,
|  and ignoring scheduling delays, is quite foolhardy and shows
|  a lack of understanding of how queueing really works.

CCID3 uses the sampled RTT value as input into an equation so any inaccuracies will multiply. I observed a difference of about 10.

Currently the performance of CCID3 severely sucks and I tried every bit to make it better. Obviously this is not the right way.

Your comments are helpful, thank you.

I will revert all RX/TX timestamps and I think there is no need to take the timestamp in the dccp receive input function.

By taking the timestamp in the CCID3 handler, it will include all processing overhead, slowing the sending rate further down in proportion to that difference.

From: Gerrit Renker <[email protected]>
Date: Fri, 13 Apr 2007 10:27:16 +0100

This would indicate a bug in the equation if such small variances cause it to explode so easily.

RTTs modulate constantly, that's why they are smoothed out (at least in TCP), to kill the noise and figure out what part of the signal is real :-)

Sarcastic much? :-/

The mistake I made was that I didn't change the subject of the email as it didn't apply to this patch really. You are correct in that.

Well I would have much rather discussed in private but a) you've told Eddie not to reply in public and the way stated seems to imply I shouldn't either b) you said that we were unreasonably holding up your patches in public. It warranted a response but I'm done with my name calling (for now).

I think we need to look at whether the protocol is right for LANs but I think if we carry out Eddie's suggestions it may well be right. Against earlier code a couple of months ago with a couple of my patches I was achieving around 90 Mbits/sec on iperf which matched TCP performance and it was stable. It responded gracefully to loss also.

I have not tested lately and tried to replicate but I know that the state of tree before current patches couldn't do that any more... As time permits I will redo the work.

DCCP CCID3 has proven stable in simulations in ns2 I understand and it has been stable previously as well (though not in Linus' tree ever) so rather than say all these inaccuracies are causing problems we need to work on the solutions.

Ian

Web: http://wand.net.nz/~iam4/
Blog: http://iansblog.jandi.co.nz
WAND Network Research Group

Quoting David Miller:

|  > CCID3 uses the sampled RTT value as input into an equation so any
|  > inaccuracies will multiply. I observed a difference of about 10.
|  
|  This would indicate a bug in the equation if such small variances
|  cause it to explode so easily.
|  
|  RTTs modulate constantly, that's why they are smoothed out (at least
|  in TCP), to kill the noise and figure out what part of the signal
|  is real :-)
|  
|  > By taking the timestamp in the CCID3 handler, it will include all
|  > processing overhead, slowing the sending rate further down in
|  > proportion to that difference.
|  
|  Sarcastic much? :-/

Yeah maybe - it is not always easy to admit being wrong. But even so, I am glad for your comments; all skb_timestamping operations will be reverted without exception.

Thanks again.

|  And this creates an even deeper dependency on the global packet
|  timestamping facility.  Those are supposed to be enabled only
|  in obscure circumstances because timestamping every packet is
|  incredibly expensive.

... these patches will go into the bin as well.

Quoting Ian McDonald:

|  Well I would have much rather discussed in private but a) you've told
|  Eddie not to reply in public and the way stated seems to imply I            
                         ^--- u mean `private' ?
|  shouldn't either b) you said that we were unreasonably holding up your
|  patches in public. It warranted a response but I'm done with my name
|  calling (for now).

If it helps to let off steam ... I am not bearing any grudges. It is good to vent anger (as Metallica would have us believe). The converse is unhealthy (as Karl A. Menninger `Man against himself' makes clear).

With regards to offline discussions - yes, please, I would indeed much rather have them out in the open.

|  > If for instance you read the documentation accompanying that patch, there is a
|  > difference between 100 usec ... 1msec between skb_get_timestamp and taking the timestamp
|  > in the CCID3 receiver. It all adds up.
|  >
|  I think we need to look at whether the protocol is right for LANs but
|  I think if we carry out Eddie's suggestions it may well be right.
|  Against earlier code a couple of months ago with a couple of my
|  patches I was achieving around 90 Mbits/sec on iperf which matched TCP
|  performance and it was stable. It responded gracefully to loss also.

I have two boxes with Gbit cards here and saw speeds up to 750..820 Mbit. It seems that the computation overhead allows it to reach up to 80..90% of the link bandwidth (with coarse-granularity problems still unresolved).

|  I have not tested lately and tried to replicate but I know that the
|  state of tree before current patches couldn't do that any more... As
|  time permits I will redo the work.

Do you remember when the `bidirectional' patch was reverted? - after that the CCID3 sender slowed down again to 75..80 Mbits/sec on a 100Mbit link. This comes from the processing overhead, and was the original motivation for this patch.

That incidentally was the second `moronic' patch I had submitted (as it limited connections to being one-directional). Without Andre's help it would probably still be in. This really is the point I was trying to make in the email - I think we need to focus more on the code. Here are two cases where the missing review/input came from outside.

I am not claiming to be any special. My hacking skills are rather moderate (i.e. you have been warned about them patches :)

Thanks.

How many connections? Up to now, when I was more involved in DCCP development, for the sake of testing the correctness of the protocol I mostly tested with just a few connections, most of the time with just one, which is OK while we're not yet feeling so good about the overal correctness of the implementation, and because I mostly reused the TCP machinery, but for performance we really have to test with many connections, and in fact in combination with TCP connections, so that we can see how DCCP affects overal system performance/stability.

Neither me, and everybody commit mistakes, I should have known better that net_enable_timestamp was not supposed to be enabled for normal connections, just for things like tcpdump, etc, but in the end I pushed for Dave trying to move things a bit forward, my bad, we really have to take into account decisions we make that affects the rest of the system :-\

- Arnaldo

| > Do you remember when the `bidirectional' patch was reverted? - after that the

|  > CCID3 sender slowed down again to 75..80 Mbits/sec on a 100Mbit link.
|  > This comes from the processing overhead, and was the original motivation for
|  > this patch.
|  
|  How many connections? Up to now, when I was more involved in DCCP
|  development, for the sake of testing the correctness of the protocol I
|  mostly tested with just a few connections, most of the time with just
|  one, which is OK while we're not yet feeling so good about the overal
|  correctness of the implementation, and because I mostly reused the TCP
|  machinery, but for performance we really have to test with many
|  connections, and in fact in combination with TCP connections, so that
|  we can see how DCCP affects overal system performance/stability.

This was a single connection, the results that one gets with iperf in byte-blast mode. I am reasonably positive that this is due to processing overhead and actually have a concept how to fix this - by not delivering to the RX CCID when a node has set SHUT_RD. The application can do this via shutdown() when it knows that it is not going to need any data and will only send (e.g. a streaming audio server).

<snip>

|  but in the end I pushed for Dave trying to move things a bit forward, my bad, we really
|  have to take into account decisions we make that affects the rest of
|  the system :-\

I am sorry for the delay that this has caused, I much appreciate the support in getting the patches worked through.