Detecting TCP loss on the receiving side?
Subsystems (2)
We're trying to troubleshoot some problems and we'd like to use the struct tcp_info method to gain some information and attempt to detect TCP loss events on the receiving side of a TCP stream. The problem is struct tcp_info is not well documented and my attempts to trace it seem to reveal that most of the statistics are relevant to the sending side of a TCP connection.
Are there any fields (or any other way) to easily detect loss events? We're mostly concerned with detecting when TCP loss or reordering delays things resulting in additional latency.
Thanks much,
Dan
Signature · 4 lines
Dan Noé Software Engineer Lime Brokerage LLC 781-370-2518
Quoted 3 lines · Dan Noé
We're trying to troubleshoot some problems and we'd like to use the struct
tcp_infomethod to gain some information and attempt to detect TCP loss events on the receiving side of a TCP stream. The problem is structtcp_infois not well documented and my attempts to trace it seem to reveal that most of the statistics are relevant to the sending side of a TCP connection.Are there any fields (or any other way) to easily detect loss events? We're mostly concerned with detecting when TCP loss or reordering delays things resulting in additional latency.
If this is just for troubleshooting, why not just take a tcpdump trace?
rick jones
Quoted 1 line · Rick Jones
If this is just for troubleshooting, why not just take a tcpdump trace?
We're pushing a lot of data.. several Mbps pretty much all day long.. and the (suspected) loss occurs sporadically. Ideally we'd like to be able to easily correlate it with latency seen in our app.
Cheers,
Dan
Signature · 4 lines
Dan Noé Software Engineer Lime Brokerage LLC 781-370-2518
Quoted 7 lines · Dan Noé
Quoted 2 lines · Rick Jones
If this is just for troubleshooting, why not just take a tcpdump trace?
We're pushing a lot of data.. several Mbps pretty much all day long.. and the (suspected) loss occurs sporadically. Ideally we'd like to be able to easily correlate it with latency seen in our app.
Looking for loss at the receiver is a bit tricky. It doesn't look like struct tcp_info has enough information to do this easily. If you are able to install a custom kernel on this machine, the Web100 patch would be able to gather enough information to figure it out. The basic idea would be to look for a difference between RcvNxt and RcvMax.
On the other hand, several Mbps is not that much. It's probably not that hard to take tcpdumps split up every N minutes, and analyze these. One thing to look for would be sack blocks coming from the receiver (assuming sack is enabled.)
-John
Quoted 6 lines · Dan Noé
Quoted 1 line · Rick Jones
If this is just for troubleshooting, why not just take a tcpdump trace?
We're pushing a lot of data.. several Mbps pretty much all day long.. and the (suspected) loss occurs sporadically. Ideally we'd like to be able to easily correlate it with latency seen in our app.
If your app can already log a timestamped "high latency" warning, then it would simply be a matter of a big disc and comparing timestamps in the tcpdump trace :)
Also, it appears that for what you want, you only need to capture up through the TCP header, so while it may be many Mbit/s on the wire, it will be fewer Mbit/s in the trace.
rick jones
Quoted 6 lines · John Heffner
Looking for loss at the receiver is a bit tricky. It doesn't look like struct
tcp_infohas enough information to do this easily. If you are able to install a custom kernel on this machine, the Web100 patch would be able to gather enough information to figure it out. The basic idea would be to look for a difference between RcvNxt and RcvMax.
And even then it depends on the connections having multiple segments in flight at one time. Although I suppose that cuts both ways and affects the tracing too, but perhaps not to the same extent.
Dan - seeing "brokerage" in your email and worries about latency makes me think that your app(s) are pushing around lots of small messages - are those spread-out across lots of connections, or are they consolidated into a rather smaller number of connections? Also, what is the magnitude of the latency in these latency events?
rick jones
Quoted 13 lines · Rick Jones
Quoted 7 lines · John Heffner
Looking for loss at the receiver is a bit tricky. It doesn't look like struct
tcp_infohas enough information to do this easily. If you are able to install a custom kernel on this machine, the Web100 patch would be able to gather enough information to figure it out. The basic idea would be to look for a difference between RcvNxt and RcvMax.And even then it depends on the connections having multiple segments in flight at one time. Although I suppose that cuts both ways and affects the tracing too, but perhaps not to the same extent.
Quoted 5 lines
Dan - seeing "brokerage" in your email and worries about latency makes me think that your app(s) are pushing around lots of small messages - are those spread-out across lots of connections, or are they consolidated into a rather smaller number of connections? Also, what is the magnitude of the latency in these latency events?
Yes, a very slow rate makes things trickier, but transmitting less than one segment per minRTO (200 ms) seems pretty unlikely. Otherwise, the receiver should observe reordering.
-John
Quoted 12 lines · Rick Jones
Quoted 6 lines · John Heffner
Looking for loss at the receiver is a bit tricky. It doesn't look like struct
tcp_infohas enough information to do this easily. If you are able to install a custom kernel on this machine, the Web100 patch would be able to gather enough information to figure it out. The basic idea would be to look for a difference between RcvNxt and RcvMax.And even then it depends on the connections having multiple segments in flight at one time. Although I suppose that cuts both ways and affects the tracing too, but perhaps not to the same extent.
Quoted 5 lines
Dan - seeing "brokerage" in your email and worries about latency makes me think that your app(s) are pushing around lots of small messages - are those spread-out across lots of connections, or are they consolidated into a rather smaller number of connections? Also, what is the magnitude of the latency in these latency events?
Yeah, without going into too much detail: We are getting data from a market via a TCP connection. Usually this is one or two connections. It is dominated by receiving traffic (from peer). The message rates can be very bursty, and the size is small. The links are usually short and fairly fat - metro fast Ethernet, for example, and the geographic latency is low. We believe loss is occurring but we're trying to determine how often and where. The data comes on fairly fast and a loss event (even with fast retransmit) means things piling up in queues and a sudden flood of data coming into userspace (which is a pretty serious latency hit on this scale).
More and more places are moving to multicast (which has its own quirks) but some are still using TCP :)
Cheers,
Dan
Quoted 3 lines · Dan Noé
We're trying to troubleshoot some problems and we'd like to use the struct
tcp_infomethod to gain some information and attempt to detect TCP loss events on the receiving side of a TCP stream.
Receiver Loss can be diagnosed using netstat -s at least. Especially if you can rule out reordering. But only system global currently, although that might change with containers.
-Andi