Skip to contentSkip to conversation

net: TCP thin-stream detection

PATCH Superseded Andreas Petlund 9 participants 16 messages Feb 16, 2010 – Feb 22, 2010

Subsystems (4)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/3 net: TCP thin-stream detection

Major changes: Added thin-stream info in new file:
Documentation/networking/tcp-thin.txt

---
 Documentation/networking/tcp-thin.txt |   47 +++++++++++++++++++++++++++++++++
 include/net/tcp.h                     |    7 +++++
 2 files changed, 54 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/networking/tcp-thin.txt
Documentation/networking/tcp-thin.txt new file +47 -0
new file mode 100644
index 0000000..151e229
@@ -0,0 +1,47 @@
+Thin-streams and TCP
+====================
+A wide range of Internet-based services that use reliable transport
+protocols display what we call thin-stream properties. This means
+that the application sends data with such a low rate that the
+retransmission mechanisms of the transport protocol are not fully
+effective. In time-dependent scenarios (like online games, control
+systems, stock trading etc.) where the user experience depends
+on the data delivery latency, packet loss can be devastating for
+the service quality. Extreme latencies are caused by TCP's
+dependency on the arrival of new data from the application to trigger
+retransmissions effectively through fast retransmit instead of
+waiting for long timeouts.
+
+After analysing a large number of time-dependent interactive
+applications, we have seen that they often produce thin streams
+and also stay with this traffic pattern throughout its entire
+lifespan. The combination of time-dependency and the fact that the
+streams provoke high latencies when using TCP is unfortunate.
+
+In order to reduce application-layer latency when packets are lost,
+a set of mechanisms has been made, which address these latency issues
+for thin streams. In short, if the kernel detects a thin stream,
+the retransmission mechanisms are modified in the following manner:
+
+1) If the stream is thin, fast retransmit on the first dupACK.
+2) If the stream is thin, do not apply exponential backoff.
+
+These enhancements are applied only if the stream is detected as
+thin. This is accomplished by defining a threshold for the number
+of packets in flight. If there are less than 4 packets in flight,
+fast retransmissions can not be triggered, and the stream is prone
+to experience high retransmission latencies.
+
+Since these mechanisms are targeted at time-dependent applications,
+they must be specifically activated by the application using the
+TCP_THIN_LINEAR_TIMEOUTS and TCP_THIN_DUPACK IOCTLS or the
+tcp_thin_linear_timeouts and tcp_thin_dupack sysctls. Both
+modifications are turned off by default.
+
+References
+==========
+More information on the modifications, as well as a wide range of
+experimental data can be found here:
+"Improving latency for interactive, thin-stream applications over
+reliable transport"
+http://simula.no/research/nd/publications/Simula.nd.477/simula_pdf_file
include/net/tcp.h +7 -0
index 87d164b..e5e2056 100644
@@ -1386,6 +1386,13 @@ static inline void tcp_highest_sack_combine(struct sock *sk,
 		tcp_sk(sk)->highest_sack = new;
 }
 
+/* Determines whether this is a thin stream (which may suffer from
+ * increased latency). Used to trigger latency-reducing mechanisms.*/
+static inline unsigned int tcp_stream_is_thin(struct tcp_sock *tp)
+{
+	return tp->packets_out < 4 && !tcp_in_initial_slowstart(tp);
+}
+
 /* /proc */
 enum tcp_seq_states {
 	TCP_SEQ_STATE_LISTENING,
Signature · 1 line
1.6.3.3
maintainer
#1 lore raw

From: Andreas Petlund <[email protected]>
Date: Tue, 16 Feb 2010 15:40:32 +0100

include/net/tcp.h static inline void tcp_highest_sack_combine(struct sock *sk, · line 1396

Quoted 9 lines
include/net/tcp.h +7 -0
+/* Determines whether this is a thin stream (which may suffer from
+ * increased latency). Used to trigger latency-reducing mechanisms.*/
+static inline unsigned int tcp_stream_is_thin(struct tcp_sock *tp)
+{
+	return tp->packets_out < 4 && !tcp_in_initial_slowstart(tp);
+}
+
 /* /proc */

Please format comments:

	/* Like this.  */

or:

	/* Like this.
	 * And this.
	 */

Thanks.

#2 lore raw
In reply to Andreas Petlund

Hi!

Documentation/networking/tcp-thin.txt line 27

Quoted 13 lines
Documentation/networking/tcp-thin.txt new file +13 -0
+After analysing a large number of time-dependent interactive
+applications, we have seen that they often produce thin streams
+and also stay with this traffic pattern throughout its entire
+lifespan. The combination of time-dependency and the fact that the
+streams provoke high latencies when using TCP is unfortunate.
+
+In order to reduce application-layer latency when packets are lost,
+a set of mechanisms has been made, which address these latency issues
+for thin streams. In short, if the kernel detects a thin stream,
+the retransmission mechanisms are modified in the following manner:
+
+1) If the stream is thin, fast retransmit on the first dupACK.
+2) If the stream is thin, do not apply exponential backoff.

2) seems very dangerous/unfair. If network congestion is caused just by thin streams, will the network just fall apart?

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#3 lore raw

Am 21.02.2010 um 11:21 schrieb Pavel Machek:

Quoted 18 lines

Hi!

Documentation/networking/tcp-thin.txt line 27

Quoted 13 lines
Documentation/networking/tcp-thin.txt new file +13 -0
+After analysing a large number of time-dependent interactive
+applications, we have seen that they often produce thin streams
+and also stay with this traffic pattern throughout its entire
+lifespan. The combination of time-dependency and the fact that the
+streams provoke high latencies when using TCP is unfortunate.
+
+In order to reduce application-layer latency when packets are lost,
+a set of mechanisms has been made, which address these latency issues
+for thin streams. In short, if the kernel detects a thin stream,
+the retransmission mechanisms are modified in the following manner:
+
+1) If the stream is thin, fast retransmit on the first dupACK.
+2) If the stream is thin, do not apply exponential backoff.

2) seems very dangerous/unfair. If network congestion is caused just by thin streams, will the network just fall apart?

and 1) can also be dangerous if we have reordering on the path.

I strongly suggest that we discuss Andreas' idea on IETF TCPM *before* we integrate it in the kernel and enable it for everyone

Alex,

as an netdev reader and TCPM member

Quoted 9 lines
Signature · 7 lines
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4 // RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22220 // email: [email protected]
// web: http://www.umic-mesh.net
//

#4 lore raw
Quoted 2 lines · Alexander Zimmermann

I strongly suggest that we discuss Andreas' idea on IETF TCPM *before* we integrate it in the kernel and enable it for everyone

Agree with Alexander. It is not at all clear to me that these changes are safe for non-exerimental use in the wider Internet.

Lars

maintainer
#5 lore raw

From: Lars Eggert <[email protected]>
Date: Sun, 21 Feb 2010 22:04:46 +0200

Quoted 2 lines

Agree with Alexander. It is not at all clear to me that these changes are safe for non-exerimental use in the wider Internet.

It's off by default and we provide all sorts of experimental congestion control algorithms which can cause just as much if not more trouble than these thin-stream bits.

There is no problem integrating these changes.

#6 lore raw

If it's off by default there is no issue.

Quoted 11 lines · David Miller

From: Lars Eggert <[email protected]>
Date: Sun, 21 Feb 2010 22:04:46 +0200

Quoted 2 lines

Agree with Alexander. It is not at all clear to me that these changes are safe for non-exerimental use in the wider Internet.

It's off by default and we provide all sorts of experimental congestion control algorithms which can cause just as much if not more trouble than these thin-stream bits.

There is no problem integrating these changes.

#7 lore raw
Quoted 1 line · Lars Eggert

If it's off by default there is no issue.

Still a sufficient large number of people might turn it on and then the network would melt.

IMHO TCP changes with badly understood congestion behaviour should not be merged until the necessary analysis/simulation etc. is done.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#8 lore raw

Am 22.02.2010 um 16:17 schrieb Andi Kleen:

Quoted 4 lines
Quoted 1 line · Lars Eggert

If it's off by default there is no issue.

Quoted 8 lines

Still a sufficient large number of people might turn it on and then the network would melt.

IMHO TCP changes with badly understood congestion behaviour should not be merged until the necessary analysis/simulation etc. is done.

-Andi

100% ack.

See for example here: http://www.umic-mesh.net/~zimmermann/linux.png

It's a screenshot of the ps3mediaserver. Since it's a streaming server, it's very useful to deactivate Nagle :-)

Alex

Quoted 3 lines
Signature · 2 lines
-- 
[email protected] -- Speaking for myself only.

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4 // RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22220 // email: [email protected]
// web: http://www.umic-mesh.net
//

#9 lore raw
In reply to Andi Kleen
Quoted 4 lines · Andi Kleen
Quoted 1 line · Lars Eggert

If it's off by default there is no issue.

Quoted 8 lines

Still a sufficient large number of people might turn it on and then the network would melt.

IMHO TCP changes with badly understood congestion behaviour should not be merged until the necessary analysis/simulation etc. is done.

-Andi

Graphs from our experiments can be found here:
http://simula.no/research/nd/publications/Simula.nd.477/simula_pdf_file

Section 5.2.5 covers experiments on fairness-issues. We have found no adverse effects of the modifications submitted in this patch set (mFR and LT in the plot), but further testing cannot hurt.

Best regards,
Andreas

#10 lore raw
In reply to Andi Kleen
Quoted 4 lines · Andi Kleen
Quoted 1 line · Lars Eggert

If it's off by default there is no issue.

Quoted 9 lines

Still a sufficient large number of people might turn it on and then the network would melt.

IMHO TCP changes with badly understood congestion behaviour should not be merged until the necessary analysis/simulation etc. is done.

-Andi

I also encourage you to read the discussions from the first posted version of the patch set, since I here reference a lot of the supporting research and related material from other researchers.

http://thread.gmane.org/gmane.linux.network/141394
http://thread.gmane.org/gmane.linux.network/141395
http://thread.gmane.org/gmane.linux.network/141396
http://thread.gmane.org/gmane.linux.network/141397

Cheers,
Andreas

#11 lore raw
In reply to Andi Kleen
Quoted 11 lines · Andi Kleen
Quoted 1 line · Lars Eggert

If it's off by default there is no issue.

Still a sufficient large number of people might turn it on

and then the network would melt.

Then we should apply the same policy as used for the congestion control

algorithms!? Restrict the usage to change the standard behavior via Kconfig

(e.g. menuconfig TCP_CONG_ADVANCED) or even more restricted demand for

CAP_NET_ADMIN.

Hagen

maintainer
#12 lore raw
In reply to Andi Kleen

From: Andi Kleen <[email protected]>
Date: Mon, 22 Feb 2010 16:17:33 +0100

Quoted 6 lines
Quoted 1 line · Lars Eggert

If it's off by default there is no issue.

Still a sufficient large number of people might turn it on and then the network would melt.

And they can do the same by changing the initial congestion window setting, which we also support.

The foundation of your argument is as solid as quicksand.

#13 lore raw
In reply to Alexander Zimmermann
Quoted 22 lines · Alexander Zimmermann

Am 21.02.2010 um 11:21 schrieb Pavel Machek:

Quoted 18 lines

Hi!

Documentation/networking/tcp-thin.txt line 27

Quoted 13 lines
Documentation/networking/tcp-thin.txt new file +13 -0
+After analysing a large number of time-dependent interactive
+applications, we have seen that they often produce thin streams
+and also stay with this traffic pattern throughout its entire
+lifespan. The combination of time-dependency and the fact that the
+streams provoke high latencies when using TCP is unfortunate.
+
+In order to reduce application-layer latency when packets are lost,
+a set of mechanisms has been made, which address these latency issues
+for thin streams. In short, if the kernel detects a thin stream,
+the retransmission mechanisms are modified in the following manner:
+
+1) If the stream is thin, fast retransmit on the first dupACK.
+2) If the stream is thin, do not apply exponential backoff.

2) seems very dangerous/unfair. If network congestion is caused just by thin streams, will the network just fall apart?

Quoted 4 lines

and 1) can also be dangerous if we have reordering on the path.

I strongly suggest that we discuss Andreas' idea on IETF TCPM *before* we integrate it in the kernel and enable it for everyone

What difference you see with 1) and early rexmit when cwnd = 2, the latter being afaict "discussed already" on TCPM?

Signature · 1 line
 i.
#14 lore raw
In reply to Pavel Machek
Quoted 5 lines · * Pavel Machek | 2010-02-21 11:21:03 [+0100]:
Quoted 2 lines

+1) If the stream is thin, fast retransmit on the first dupACK. +2) If the stream is thin, do not apply exponential backoff.

2) seems very dangerous/unfair. If network congestion is caused just by thin streams, will the network just fall apart?

I question the fairness of this modification! The modification sounds _really_ far-reaching. Did you analyse the behaviour of this modification? What about fairness analysis? Great work is available in the network simulator sector for this kind of analysis (e.g. jain's fairness index).

HGN

Signature · 3 lines
Hagen Paul Pfeifer <[email protected]>  ||  http://jauu.net/
Telephone: +49 174 5455209           ||  Key Id: 0x98350C22
Key Fingerprint: 490F 557B 6C48 6D7E 5706 2EA2 4A22 8D45 9835 0C22
#15 lore raw
In reply to Pavel Machek
Quoted 18 lines · Pavel Machek

Hi!

Documentation/networking/tcp-thin.txt line 27

Quoted 13 lines
Documentation/networking/tcp-thin.txt new file +13 -0
+After analysing a large number of time-dependent interactive
+applications, we have seen that they often produce thin streams
+and also stay with this traffic pattern throughout its entire
+lifespan. The combination of time-dependency and the fact that the
+streams provoke high latencies when using TCP is unfortunate.
+
+In order to reduce application-layer latency when packets are lost,
+a set of mechanisms has been made, which address these latency issues
+for thin streams. In short, if the kernel detects a thin stream,
+the retransmission mechanisms are modified in the following manner:
+
+1) If the stream is thin, fast retransmit on the first dupACK.
+2) If the stream is thin, do not apply exponential backoff.

2) seems very dangerous/unfair. If network congestion is caused just by thin streams, will the network just fall apart?

...Network will not fall apart. Two possible cases:

a) cwnd > 1 segment, on RTO the window is reduced, thus the sender backs off regardless of exponential backoff.
b) All flows have window of 1 already... Well, what do you suggest? I'd say that obviously the network is seriously underprovisioned for the workload and since the bottleneck can only serve some of them anyway dropping the rest, no useless work gets done at the bottleneck. RTT estimator then gets a new samples whenever a flow belongs into the lucky group. In case any unnecessary retransmission happened (if there was very dramatic and sudden increase transient in the workload) they won't happen thereafter (unless we increase the workload toward infinity).

...Thus no problem of "falling apart".

Signature · 1 line
 i.