Slow ramp-up for single-stream TCP throughput on 4.2 kernel.

I'm seeing something that looks more dodgy than normal.

Test case id ath10k station uploading to ath10k AP.

AP is always running 4.2 kernel in this case, and both systems are using the same ath10k firmware.

I have tuned the stack:

echo 4000000 > /proc/sys/net/core/wmem_max
echo 4096 87380 50000000 > /proc/sys/net/ipv4/tcp_rmem echo 4096 16384 50000000 > /proc/sys/net/ipv4/tcp_wmem echo 50000000 > /proc/sys/net/core/rmem_max echo 30000 > /proc/sys/net/core/netdev_max_backlog echo 1024000 > /proc/sys/net/ipv4/tcp_limit_output_bytes

On the 3.17.8+ kernel, single stream TCP very quickly (1-2 seconds) reaches about 525Mbps upload throughput (station to AP).

But, when station machine is running the 4.2 kernel, the connection goes to about 30Mbps for 5-10 seconds, then may ramp up to 200-300Mbps, and may plateau at around 400Mbps after another minute or two. Once, I saw it finally reach 500+Mbps after about 3 minutes.

Both behaviors are repeatable in my testing.

For 4.2, I tried setting the send/rcv buffers to 2Mbps,
I tried leaving them at system defaults, same behavior. I tried doubling the tcp_limit_output_bytes to 2048k, and that had no affect.

Netstat shows about 1MB of data setting in the TX queue for for 3.17 and 4.2 kernels when this test is running.

If I start a 50-stream TCP test, then total throughput is 500+Mbps on 4.2, and generally correlates well with whatever UDP can do at that time.

A 50-stream throughput has virtually identical performance to the 1 stream test on the 3.17 kernel.

For the 4.0.4+ kernel, single stream stuck at 30Mbps and would not budge (4.2 does this sometimes too, perhaps it would have gone up if I had waited more than the ~15 seconds that I did) 50 stream stuck at 420Mbps and would not improve, but it ramped to that quickly.
100 stream test ran at 560Mbps throughput, which is about the maximum TCP throughput we normally see for ath10k over-the-air.

I'm interested to know if someone has any suggestions for things to tune in 4.2 or 4.0 that might help this, or any reason why I might be seeing this behaviour.

I'm also interested to know if anyone else sees similar behaviour.

Thanks,
Ben

Gah, seems 'cubic' related. That is the default tcp cong ctrl I was using (same in 3.17, for that matter).

Quoted 2 lines · Ben Greear

Gah, seems 'cubic' related. That is the default tcp cong ctrl I was using (same in 3.17, for that matter).

There have been recent changes to CUBIC that may account for this. If you could repeat your test with more instrumentation, eg "nstat", that would be very helpful.

nstat > /dev/null

# run one test
nstat

Also, if you could take a sender-side tcpdump trace of the test, that would be very useful (default capture length, grabbing just headers, is fine).

Quoted 12 lines · Neal Cardwell

Quoted 3 lines · Ben Greear

Gah, seems 'cubic' related. That is the default tcp cong ctrl I was using (same in 3.17, for that matter).

There have been recent changes to CUBIC that may account for this. If you could repeat your test with more instrumentation, eg "nstat", that would be very helpful.

Quoted 7 lines

nstat > /dev/null

# run one test
nstat

Also, if you could take a sender-side tcpdump trace of the test, that would be very useful (default capture length, grabbing just headers, is fine).

Here is nstat output:

[root@ben-ota-1 ~]# nstat
#kernel

IpInReceives                    14507              0.0
IpInDelivers                    14507              0.0
IpOutRequests                   49531              0.0
TcpActiveOpens                  3                  0.0
TcpPassiveOpens                 2                  0.0
TcpInSegs                       14498              0.0
TcpOutSegs                      50269              0.0
UdpInDatagrams                  9                  0.0
UdpOutDatagrams                 1                  0.0
TcpExtDelayedACKs               43                 0.0
TcpExtDelayedACKLost            5                  0.0
TcpExtTCPHPHits                 483                0.0
TcpExtTCPPureAcks               918                0.0
TcpExtTCPHPAcks                 12758              0.0
TcpExtTCPDSACKOldSent           5                  0.0
TcpExtTCPRcvCoalesce            49                 0.0
TcpExtTCPAutoCorking            3                  0.0
TcpExtTCPOrigDataSent           49776              0.0
TcpExtTCPHystartTrainDetect     1                  0.0
TcpExtTCPHystartTrainCwnd       16                 0.0
IpExtInBcastPkts                8                  0.0
IpExtInOctets                   2934274            0.0
IpExtOutOctets                  74817312           0.0
IpExtInBcastOctets              640                0.0
IpExtInNoECTPkts                14911              0.0

[root@ben-ota-1 ~]#

And, you can find the pcap here:

http://www.candelatech.com/downloads/cubic.pcap.bz2

Let me know if you need anything else.

Quoted 53 lines · Ben Greear

Quoted 12 lines · Neal Cardwell

Quoted 3 lines · Ben Greear

Gah, seems 'cubic' related. That is the default tcp cong ctrl I was using (same in 3.17, for that matter).

There have been recent changes to CUBIC that may account for this. If you could repeat your test with more instrumentation, eg "nstat", that would be very helpful.

Quoted 7 lines

nstat > /dev/null

# run one test
nstat

Also, if you could take a sender-side tcpdump trace of the test, that would be very useful (default capture length, grabbing just headers, is fine).

Here is nstat output:

[root@ben-ota-1 ~]# nstat
#kernel

IpInReceives                    14507              0.0
IpInDelivers                    14507              0.0
IpOutRequests                   49531              0.0
TcpActiveOpens                  3                  0.0
TcpPassiveOpens                 2                  0.0
TcpInSegs                       14498              0.0
TcpOutSegs                      50269              0.0
UdpInDatagrams                  9                  0.0
UdpOutDatagrams                 1                  0.0
TcpExtDelayedACKs               43                 0.0
TcpExtDelayedACKLost            5                  0.0
TcpExtTCPHPHits                 483                0.0
TcpExtTCPPureAcks               918                0.0
TcpExtTCPHPAcks                 12758              0.0
TcpExtTCPDSACKOldSent           5                  0.0
TcpExtTCPRcvCoalesce            49                 0.0
TcpExtTCPAutoCorking            3                  0.0
TcpExtTCPOrigDataSent           49776              0.0
TcpExtTCPHystartTrainDetect     1                  0.0
TcpExtTCPHystartTrainCwnd       16                 0.0
IpExtInBcastPkts                8                  0.0
IpExtInOctets                   2934274            0.0
IpExtOutOctets                  74817312           0.0
IpExtInBcastOctets              640                0.0
IpExtInNoECTPkts                14911              0.0

Quoted 8 lines

[root@ben-ota-1 ~]#

And, you can find the pcap here:

http://www.candelatech.com/downloads/cubic.pcap.bz2

Let me know if you need anything else.

Thanks! This is very useful. It looks like the sender is sending 3 (and later 4) packets every ~1.5ms for the entirety of the trace. 3 packets per burst is usually a hint that this may be related to TSQ.

This slow-and-steady behavior triggers CUBIC's Hystart Train Detection to enter congestion avoidance at a cwnd of 16, which probably in turn leads to slow cwnd growth, since the sending is not cwnd-limited, but probably TSQ-limited, so cwnd does not grow in congestion avoidance mode. Probably most of the other congestion control modules do better because they stay in slow-start, which has a more aggressive criterion for growing cwnd.

So this is probably at root due to the known issue with an interaction between the ath10k driver and the following change in 3.19:

605ad7f tcp: refine TSO autosizing

There has been a lot of discussion about how to address the TSQ-related issues with this driver. For example, you might consider:

   https://patchwork.ozlabs.org/patch/438322/

But I am not sure of the latest status of that effort. Perhaps someone on the ath10k list will know.

I guess I'll just stop using Cubic. Any suggestions for another congestion algorithm to use? I'd prefer something that worked well in pretty much any network condition, of course, and it has to work with ath10k.

We can also run some tests with 1G, 10G, ath10k, ath9k, and in conjunction with network emulators and various congestion control algorithms.

Quoted 7 lines · Ben Greear

I guess I'll just stop using Cubic. Any suggestions for another congestion algorithm to use? I'd prefer something that worked well in pretty much any network condition, of course, and it has to work with ath10k.

We can also run some tests with 1G, 10G, ath10k, ath9k, and in conjunction with network emulators and various congestion control algorithms.

You could use cubic , but disable or tune HyStart, which is known to be problematic anyway.

echo 0 >/sys/module/tcp_cubic/parameters/hystart_detect

Or try

echo 40 >/sys/module/tcp_cubic/parameters/hystart_low_window

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines · Ben Greear

I guess I'll just stop using Cubic. Any suggestions for another congestion algorithm to use? I'd prefer something that worked well in pretty much any network condition, of course, and it has to work with ath10k.

Quoted 2 lines

We can also run some tests with 1G, 10G, ath10k, ath9k, and in conjunction with network emulators and various congestion control algorithms.

You could use cubic , but disable or tune HyStart, which is known to be problematic anyway.

echo 0 >/sys/module/tcp_cubic/parameters/hystart_detect

Or try

echo 40 >/sys/module/tcp_cubic/parameters/hystart_low_window