RFC: Latency reducing TCP modifications for thin-stream interactive applications

A wide range of Internet-based services that use reliable transport protocols display what we call thin-stream properties. This means that the application sends data with such a low rate that the retransmission mechanisms of the transport protocol are not fully effective. In time-dependent scenarios (like online games, control systems or some sensor networks) where the user experience depends on the data delivery latency, packet loss can be devastating for the service quality. Extreme latencies are caused by TCP's dependency on the arrival of new data from the application to trigger retransmissions effectively through fast retransmit instead of waiting for long timeouts. After analyzing a large number of time-dependent interactive applications, we have seen that they often produce thin streams (as described above) and also stay with this traffic pattern throughout its entire lifespan. The combination of time-dependency and the fact that the streams provoke high latencies when using TCP is unfortunate.

In order to reduce application-layer latency when packets are lost, we have implemented modifications to the TCP retransmission mechanisms in the Linux kernel. We have also implemented a bundling mechanisms that introduces redundancy in order to preempt the experience of packet loss. In short, if the kernel detects a thin stream, we trade a small amount of bandwidth for latency reduction and apply:

Removal of exponential backoff: To prevent an exponential increase in retransmission delay for a repeatedly lost packet, we remove the exponential factor.

FASTER Fast Retransmit: Instead of waiting for 3 duplicate acknowledgments before sending a fast retransmission, we retransmit after receiving only one.

Redundant Data Bundling: We copy (bundle) data from the unacknowledged packets in the send buffer into the next packet if
space is available.

These enhancements are applied only if the stream is detected as thin.
This is accomplished by defining thresholds for packet size and packets in flight. Also, we consider the redundancy introduced by our mechanisms acceptable because the streams are so thin that normal congestion mechanisms do not come into effect.

We have implemented these changes in the Linux kernel (2.6.23.8), and have tested the modifications on a wide range of different thin-stream applications (Skype, BZFlag, SSH, ...) under varying network conditions. Our results show that applications which use TCP for interactive time-dependent traffic will experience a reduction in both maximum and average latency, giving the users quicker feedback to their interactions.

Availability of this kind of mechanisms will help provide customizability for interactive network services. The quickly growing market for Linux gaming may benefit from lowered latency. As an example, most of the large MMORPG's today use TCP (like World of Warcraft and Age of Conan) and several multimedia applications (like Skype) use TCP fallback if UDP is blocked.

The modifications are all TCP standard compliant and transparent to the receiver. As such, a game server could implement the modifications and get a one-way latency benefit without touching any of the clients.

In the following papers, we discuss the benefits and tradeoffs of the decribed mechanisms:
"The Fun of using TCP for an MMORPG":
http://simula.no/research/networks/publications/Griwodz.2006.1
"TCP Enhancements For Interactive Thin-Stream Applications":
http://simula.no/research/networks/publications/Simula.ND.83
"Improving application layer latency for reliable thin-stream game traffic": http://simula.no/research/networks/publications/Simula.ND.185
"TCP mechanisms for improving the user experience for time-dependent thin-stream applications":
http://simula.no/research/networks/publications/Simula.ND.159
Our presentation from the 2008 Linux-Kongress can be found here:
http://data.guug.de/slides/lk2008/lk-2008-Andreas-Petlund.pdf

We have included a patch for the 2.6.23.8 kernel which implements the modifications. The patch is not properly segmented and formatted, but attached as a reference. We are currently working on an updated patch set which we hopefully will be able to post in a couple of weeks.

This will also give us time to integrate any ideas that may arise from the discussions here.

We are happy for all feedback regarding this:
Is something like this viable to introduce into the kernel?

Is the scheme for thin-stream detection mechanism acceptable.

Any viewpoints on the architecture and design?

include/linux/sysctl.h +0 -0

 	NET_IPV4_TCP_TIMESTAMPS=33,
 	NET_IPV4_TCP_WINDOW_SCALING=34,

diff -Nur linux-2.6.23.8.vanilla/include/linux/tcp.h linux-2.6.23.8-tcp-thin/include/linux/tcp.h

include/linux/tcp.h +4 -0

@@ -97,6 +97,10 @@
 #define TCP_CONGESTION		13	/* Congestion control algorithm */
 #define TCP_MD5SIG		14	/* TCP MD5 Signature (RFC2385) */
 
+#define TCP_THIN_RDB            15      /* Added @ Simula - Enable redundant data bundling  */
+#define TCP_THIN_RM_EXPB        16      /* Added @ Simula - Remove exponential backoff  */
+#define TCP_THIN_DUPACK         17      /* Added @ Simula - Reduce number of dupAcks needed */
+
 #define TCPI_OPT_TIMESTAMPS	1
 #define TCPI_OPT_SACK		2
 #define TCPI_OPT_WSCALE		4

include/linux/tcp.h +0 -0

 /* RTT measurement */
 	u32	srtt;		/* smoothed round trip time << 3	*/
 	u32	mdev;		/* medium deviation			*/

diff -Nur linux-2.6.23.8.vanilla/include/net/sock.h linux-2.6.23.8-tcp-thin/include/net/sock.h

include/net/sock.h +0 -0

 	sock_set_flag(sk, SOCK_QUEUE_SHRUNK);
 	sk->sk_wmem_queued   -= skb->truesize;
 	sk->sk_forward_alloc += skb->truesize;

diff -Nur linux-2.6.23.8.vanilla/include/net/tcp.h linux-2.6.23.8-tcp-thin/include/net/tcp.h

include/net/tcp.h +5 -0

+
 extern struct inet_timewait_death_row tcp_death_row;
 
 /* sysctl variables for tcp */
+extern int sysctl_tcp_force_thin_rdb;         /* Added @ Simula */
+extern int sysctl_tcp_force_thin_rm_expb;     /* Added @ Simula */
+extern int sysctl_tcp_force_thin_dupack;      /* Added @ Simula */
+extern int sysctl_tcp_rdb_max_bundle_bytes;   /* Added @ Simula */
 extern int sysctl_tcp_timestamps;
 extern int sysctl_tcp_window_scaling;
 extern int sysctl_tcp_sack;

include/net/tcp.h +2 -0

+}
+
 /* If cwnd > ssthresh, we may raise ssthresh to be half-way to cwnd.
  * The exception is rate halving phase, when cwnd is decreasing towards
  * ssthresh.

diff -Nur linux-2.6.23.8.vanilla/net/ipv4/sysctl_net_ipv4.c linux-2.6.23.8-tcp-thin/net/ipv4/sysctl_net_ipv4.c

net/ipv4/sysctl_net_ipv4.c +30 -0

+                .procname       = "tcp_force_thin_rdb",
+                .data           = &sysctl_tcp_force_thin_rdb,
+                .maxlen         = sizeof(int),
+                .mode           = 0644,
+                .proc_handler   = &proc_dointvec
+        },
+	{       /* Added @ Simula for thin streams */
+                .ctl_name       = NET_IPV4_TCP_FORCE_THIN_RM_EXPB,
+                .procname       = "tcp_force_thin_rm_expb",
+                .data           = &sysctl_tcp_force_thin_rm_expb,
+                .maxlen         = sizeof(int),
+                .mode           = 0644,
+                .proc_handler   = &proc_dointvec
+        },
+	{       /* Added @ Simula for thin streams */
+                .ctl_name       = NET_IPV4_TCP_FORCE_THIN_DUPACK,
+                .procname       = "tcp_force_thin_dupack",
+                .data           = &sysctl_tcp_force_thin_dupack,
+                .maxlen         = sizeof(int),
+                .mode           = 0644,
+                .proc_handler   = &proc_dointvec
+        },
+	{       /* Added @ Simula for thin streams */
+                .ctl_name       = NET_IPV4_TCP_RDB_MAX_BUNDLE_BYTES,
+                .procname       = "tcp_rdb_max_bundle_bytes",
+                .data           = &sysctl_tcp_rdb_max_bundle_bytes,
+                .maxlen         = sizeof(int),
+                .mode           = 0644,
+                .proc_handler   = &proc_dointvec
+        },
 	{
 		.ctl_name	= NET_IPV4_TCP_TIMESTAMPS,
 		.procname	= "tcp_timestamps",

diff -Nur linux-2.6.23.8.vanilla/net/ipv4/tcp.c linux-2.6.23.8-tcp-thin/net/ipv4/tcp.c

net/ipv4/tcp.c +4 -0

+		struct sk_buff *prev_skb = skb->prev;
+		int skb_size = skb->len;
+		int old_headlen = 0;
+		int ua_data = 0;

net/ipv4/tcp.c +26 -0

+		
+		if(uad_frags > 0){
+			int i = 0;
+			int bytes_frags = 0;
+			
+			if(uad_frags == prev_skb->data_len){
+				ua_nr_frags = skb_shinfo(prev_skb)->nr_frags;
+			} else{
+				for(i=skb_shinfo(prev_skb)->nr_frags - 1; i>=0; i--){
+					if(skb_shinfo(prev_skb)->frags[i].size 
+					   + bytes_frags == uad_frags){
+						ua_nr_frags += 1;
+						break;
+					} 	  
+					ua_nr_frags += 1;
+					bytes_frags += skb_shinfo(prev_skb)->frags[i].size;
+				}
+			}
+		}
+		
+		/*
+		 * Do the diffrenet checks on size and content, and return if
+		 * something will not work.
+		 *
+		 * TODO: Support copying some bytes
+		 *

net/ipv4/tcp.c +11 -0

+			return -1;
+		}
+		
+		if(skb_is_nonlinear(skb) && (uad_frags > 0)){
+			if((ua_nr_frags +
+			    skb_shinfo(skb)->nr_frags) > MAX_SKB_FRAGS){
+				return -1;
+			}
+			
+			if(skb_headlen(skb) > 0){
+				return -1;

net/ipv4/tcp.c +42 -0

+			return -1;
+		}
+		
+		/* Copy linear data */
+		if(uad_head > 0){
+			
+			/* Add required space to the header. Can't use put due to linearity */
+			old_headlen = skb_headlen(skb);
+			skb->tail += uad_head;
+			skb->len += uad_head;
+			
+			if(skb_headlen(skb) > 0){
+				memmove(skb->data + uad_head, skb->data, old_headlen);
+			}
+			
+			skb_copy_to_linear_data(skb, prev_skb->data + (skb_headlen(prev_skb) - uad_head), uad_head);
+		}
+		
+		/*Copy paged data*/
+		if(uad_frags > 0){
+			int i = 0;
+			/*Must move data backwards in the array.*/
+			if(skb_is_nonlinear(skb)){
+				memmove(skb_shinfo(skb)->frags + ua_nr_frags,
+					skb_shinfo(skb)->frags,
+					skb_shinfo(skb)->nr_frags*sizeof(skb_frag_t));
+			}
+			
+			/*Copy info and update pages*/
+			memcpy(skb_shinfo(skb)->frags,
+			       skb_shinfo(prev_skb)->frags + (skb_shinfo(prev_skb)->nr_frags - ua_nr_frags),
+			       ua_nr_frags*sizeof(skb_frag_t));
+			
+			for(i=0; i<ua_nr_frags;i++){
+				get_page(skb_shinfo(skb)->frags[i].page);
+			}
+			
+			skb_shinfo(skb)->nr_frags += ua_nr_frags;
+			skb->data_len += uad_frags;
+			skb->len += uad_frags;
+		}
+		

net/ipv4/tcp.c +1 -0

+
 int tcp_sendmsg(struct kiocb *iocb, struct socket *sock, struct msghdr *msg,
 		size_t size)
 {

net/ipv4/tcp.c +13 -1

+					tcp_trans_merge_prev(sk, skb, mss_now);
+				}
+			} /* End - Simula */
+			
 			if ((seglen -= copy) == 0 && iovlen == 0)
 				goto out;
 

@@ -1870,7 +2045,25 @@
 			tcp_push_pending_frames(sk);
 		}
 		break;
-
+		
+        /* Added @ Simula. Support for thin streams */
+	case TCP_THIN_RDB:
+		if(val)
+			tp->thin_rdb = 1;
+		break;
+		
+        /* Added @ Simula. Support for thin streams */
+	case TCP_THIN_RM_EXPB:

net/ipv4/tcp.c +0 -0

 	case TCP_KEEPIDLE:
 		if (val < 1 || val > MAX_TCP_KEEPIDLE)
 			err = -EINVAL;

diff -Nur linux-2.6.23.8.vanilla/net/ipv4/tcp_input.c linux-2.6.23.8-tcp-thin/net/ipv4/tcp_input.c

net/ipv4/tcp_input.c +3 -0

@@ -89,6 +89,9 @@
 int sysctl_tcp_frto_response __read_mostly;
 int sysctl_tcp_nometrics_save __read_mostly;
 
+/* Added @ Simula */
+int sysctl_tcp_force_thin_dupack __read_mostly = TCP_FORCE_THIN_DUPACK;
+
 int sysctl_tcp_moderate_rcvbuf __read_mostly = 1;
 int sysctl_tcp_abc __read_mostly;
 

net/ipv4/tcp_input.c +11 -0

+
 	__u32 now = tcp_time_stamp;
 	int acked = 0;
 	int prior_packets = tp->packets_out;
+	
+	/*Added at Simula for RDB support*/
+	__u8 done = 0;
+	int remove = 0;
+	int remove_head = 0;
+	int remove_frags = 0;
+	int no_frags;
+	int data_frags;
+	int i;
+		
 	__s32 seq_rtt = -1;
 	ktime_t last_ackt = net_invalid_timestamp();

net/ipv4/tcp_input.c +3 -0

+			break;
+		}
+		
 		/* If our packet is before the ack sequence we can
 		 * discard it as it's confirmed to have arrived at

net/ipv4/tcp_input.c +42 -0

+					remove = tp->snd_una - scb->seq;
+					remove_head = (remove > skb_headlen(skb) ? 
+						       skb_headlen(skb) : remove);
+					remove_frags = (remove > skb_headlen(skb) ? 
+							remove - remove_head : 0);
+					
+					/* Has linear data */
+					if(skb_headlen(skb) > 0 && remove_head > 0){
+						memmove(skb->data,
+							skb->data + remove_head,
+							skb_headlen(skb) - remove_head);
+						
+						skb->tail -= remove_head;
+					}
+					
+					if(skb_is_nonlinear(skb) && remove_frags > 0){
+						no_frags = 0;
+						data_frags = 0;
+						
+						/*Remove unecessary pages*/
+						for(i=0; i<skb_shinfo(skb)->nr_frags; i++){
+							if(data_frags + skb_shinfo(skb)->frags[i].size 
+							   == remove_frags){
+								put_page(skb_shinfo(skb)->frags[i].page);
+								no_frags += 1;
+								break;
+							}
+							put_page(skb_shinfo(skb)->frags[i].page);
+							no_frags += 1;
+							data_frags += skb_shinfo(skb)->frags[i].size;
+						}
+						
+						if(skb_shinfo(skb)->nr_frags > no_frags)
+							memmove(skb_shinfo(skb)->frags,
+								skb_shinfo(skb)->frags + no_frags,
+								(skb_shinfo(skb)->nr_frags 
+								 - no_frags)*sizeof(skb_frag_t));
+						
+						skb->data_len -= remove_frags;
+						skb_shinfo(skb)->nr_frags -= no_frags;
+						
+					}

net/ipv4/tcp_input.c +7 -0

+				
+				/*Only move forward if data could be removed from this packet*/
+				done = 2;
+				
+			}
+			
+			if(done == 1 || tcp_skb_is_last(sk,skb)){

net/ipv4/tcp_input.c +2 -2

 		/* Initial outgoing SYN's get put onto the write_queue
 		 * just like anything else we transmit.  It is not
 		 * true data, and if we misinform our callers that

@@ -2479,14 +2585,14 @@
 			acked |= FLAG_SYN_ACKED;
 			tp->retrans_stamp = 0;
 		}
-
+		
 		/* MTU probing checks */
 		if (icsk->icsk_mtup.probe_size) {
 			if (!after(tp->mtu_probe.probe_seq_end, TCP_SKB_CB(skb)->end_seq)) {
 				tcp_mtup_probe_success(sk, skb);
 			}
 		}
-
+		
 		if (sacked) {
 			if (sacked & TCPCB_RETRANS) {
 				if (sacked & TCPCB_SACKED_RETRANS)

net/ipv4/tcp_input.c +4 -4

 	}
-
+	
 	if (acked&FLAG_ACKED) {
 		u32 pkts_acked = prior_packets - tp->packets_out;
 		const struct tcp_congestion_ops *ca_ops
 			= inet_csk(sk)->icsk_ca_ops;
-
+		
 		tcp_ack_update_rtt(sk, acked, seq_rtt);
 		tcp_ack_packets_out(sk);
-
+		
 		if (ca_ops->pkts_acked) {
 			s32 rtt_us = -1;
-
+			
 			/* Is the ACK triggering packet unambiguous? */
 			if (!(acked & FLAG_RETRANS_DATA_ACKED)) {
 				/* High resolution needed and available? */

diff -Nur linux-2.6.23.8.vanilla/net/ipv4/tcp_output.c linux-2.6.23.8-tcp-thin/net/ipv4/tcp_output.c

net/ipv4/tcp_output.c +17 -1

@@ -1653,7 +1653,7 @@
 
 		BUG_ON(tcp_skb_pcount(skb) != 1 ||
 		       tcp_skb_pcount(next_skb) != 1);
-
+		
 		/* changing transmit queue under us so clear hints */
 		clear_all_retrans_hints(tp);
 

@@ -1702,6 +1702,166 @@
 	}
 }
 
+/* Added at Simula. Variation of the regular collapse,
+   adapted to support RDB  */
+static void tcp_retrans_merge_redundant(struct sock *sk,
+					struct sk_buff *skb,
+					int mss_now)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+	struct sk_buff *next_skb = skb->next;
+	int skb_size = skb->len;
+	int new_data = 0;
+	int new_data_head = 0;
+	int new_data_frags = 0;
+	int new_frags = 0;> +	int old_headlen = 0;
+	
+	int i;
+	int data_frags = 0;

net/ipv4/tcp_output.c +21 -0

+			
+			/*
+			 * 1. Contains the same data
+			 * 2. Size
+			 * 3. Sack
+			 * 4. Window
+			 * 5. Cannot merge with a later packet that has linear data
+			 * 6. The new number of frags will exceed the limit
+			 * 7. Enough tailroom
+			 */
+			
+			if(new_data <= 0){
+				return;
+			}
+			
+			if ((sysctl_tcp_rdb_max_bundle_bytes == 0 && ((skb_size + new_data) > mss_now))
+			    || (sysctl_tcp_rdb_max_bundle_bytes > 0 && ((skb_size + new_data) >
+									sysctl_tcp_rdb_max_bundle_bytes))){
+				return;
+			}
+			

net/ipv4/tcp_output.c +71 -0

+				return;
+			}
+			
+			/* Calculate number of new fragments. Any new data will be 
+			   stored in the back. */
+			if(skb_is_nonlinear(next_skb)){
+				i = (skb_shinfo(next_skb)->nr_frags == 0 ? 
+				     0 : skb_shinfo(next_skb)->nr_frags - 1);
+				for( ; i>=0;i--){
+					if(data_frags + skb_shinfo(next_skb)->frags[i].size == 
+					   new_data_frags){
+						new_frags += 1;
+						break;
+					}
+					
+					data_frags += skb_shinfo(next_skb)->frags[i].size;
+					new_frags += 1;
+				}
+			}
+			
+			/* If dealing with a fragmented skb, only merge 
+			   with an skb that ONLY contain frags */
+			if(skb_is_nonlinear(skb)){
+				
+				/*Due to the way packets are processed, no later data*/
+				if(skb_headlen(next_skb) && new_data_head > 0){
+					return;
+				}
+				
+				if(skb_is_nonlinear(next_skb) && (new_data_frags > 0) && 
+				   ((skb_shinfo(skb)->nr_frags + new_frags) > MAX_SKB_FRAGS)){
+					return;
+				}
+				
+			} else {
+				if(skb_headlen(next_skb) && (new_data_head > (skb->end - skb->tail))){
+					return;
+				}
+			}
+			
+			/*Copy linear data. This will only occur if both are linear, 
+			  or only A is linear*/
+			if(skb_headlen(next_skb) && (new_data_head > 0)){
+				old_headlen = skb_headlen(skb);
+				skb->tail += new_data_head;
+				skb->len += new_data_head;
+				
+				/* The new data starts in the linear area, 
+				   and the correct offset will then be given by 
+				   removing new_data ammount of bytes from length. */
+				skb_copy_to_linear_data_offset(skb, old_headlen, next_skb->tail - 
+							       new_data_head, new_data_head);
+			}
+			
+			if(skb_is_nonlinear(next_skb) && (new_data_frags > 0)){
+				memcpy(skb_shinfo(skb)->frags + skb_shinfo(skb)->nr_frags, 
+				       skb_shinfo(next_skb)->frags + 
+				       (skb_shinfo(next_skb)->nr_frags - new_frags), 
+				       new_frags*sizeof(skb_frag_t));
+				
+				for(i=skb_shinfo(skb)->nr_frags; 
+				    i < skb_shinfo(skb)->nr_frags + new_frags; i++)
+					get_page(skb_shinfo(skb)->frags[i].page);
+				
+				skb_shinfo(skb)->nr_frags += new_frags;
+				skb->data_len += new_data_frags;
+				skb->len += new_data_frags;
+			}
+			
+			TCP_SKB_CB(skb)->end_seq += new_data;
+						

net/ipv4/tcp_output.c +8 -3

+
 /* Do a simple retransmit without using the backoff mechanisms in
  * tcp_timer. This is used for path mtu discovery.
  * The socket is already locked here.

@@ -1756,6 +1916,8 @@
 /* This retransmits one SKB.  Policy decisions and retransmit queue
  * state updates are done by the caller.  Returns non-zero if an
  * error occurred which prevented the send.
+ * Modified at Simula to support thin stream optimizations
+ * TODO: Update to use new helpers (like tcp_write_queue_next())
  */
 int tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb)
 {

@@ -1802,10 +1964,21 @@
 	    (skb->len < (cur_mss >> 1)) &&
 	    (tcp_write_queue_next(sk, skb) != tcp_send_head(sk)) &&
 	    (!tcp_skb_is_last(sk, skb)) &&
-	    (skb_shinfo(skb)->nr_frags == 0 && skb_shinfo(tcp_write_queue_next(sk, skb))->nr_frags == 0) &&
-	    (tcp_skb_pcount(skb) == 1 && tcp_skb_pcount(tcp_write_queue_next(sk, skb)) == 1) &&
-	    (sysctl_tcp_retrans_collapse != 0))
+	    (skb_shinfo(skb)->nr_frags == 0
+	     && skb_shinfo(tcp_write_queue_next(sk, skb))->nr_frags == 0)
+	    && (tcp_skb_pcount(skb) == 1
+		&& tcp_skb_pcount(tcp_write_queue_next(sk, skb)) == 1)
+	    && (sysctl_tcp_retrans_collapse != 0)

net/ipv4/tcp_output.c +0 -0

 	if (inet_csk(sk)->icsk_af_ops->rebuild_header(sk))
 		return -EHOSTUNREACH; /* Routing failure or similar. */

diff -Nur linux-2.6.23.8.vanilla/net/ipv4/tcp_timer.c linux-2.6.23.8-tcp-thin/net/ipv4/tcp_timer.c

net/ipv4/tcp_timer.c +4 -1

@@ -32,6 +32,9 @@
 int sysctl_tcp_retries2 __read_mostly = TCP_RETR2;
 int sysctl_tcp_orphan_retries __read_mostly;
 
+/* Added @ Simula */
+int sysctl_tcp_force_thin_rm_expb __read_mostly = TCP_FORCE_THIN_RM_EXPB;
+
 static void tcp_write_timer(unsigned long);
 static void tcp_delack_timer(unsigned long);
 static void tcp_keepalive_timer (unsigned long data);

@@ -368,13 +371,26 @@
 	 */
 	icsk->icsk_backoff++;
 	icsk->icsk_retransmits++;
-
+	
 out_reset_timer:

                }

net/ipv4/tcp_input.c +3 -3

-
+               
                /* Initial outgoing SYN's get put onto the write_queue
                 * just like anything else we transmit.  It is not
                 * true data, and if we misinform our callers that

@@ -2479,14 +2585,14 @@
                        acked |= FLAG_SYN_ACKED;
                        tp->retrans_stamp = 0;
                }
-
+               
                /* MTU probing checks */
                if (icsk->icsk_mtup.probe_size) {
                        if (!after(tp->mtu_probe.probe_seq_end,
 TCP_SKB_CB(skb)->end_seq)) {
                                tcp_mtup_probe_success(sk, skb);
                        }
                }
-
+               
                if (sacked) {
                        if (sacked & TCPCB_RETRANS) {

if (sacked & TCPCB_SACKED_RETRANS)

(quoted diff) +0 -0

@@ -2510,24 +2616,32 @@