Make TCP work better with re-ordered frames?

We are investigating a system that has fairly poor TCP throughput with the 3.17 and 4.0 kernels, but evidently it worked pretty well with 3.14 (I should be able to verify 3.14 later today).

One thing I notice is that a UDP download test shows lots of reordered frames, so I am thinking maybe TCP is running slow because of this.

(We see about 800Mbps UDP download, but only 500Mbps TCP, even when
using 100 concurrent TCP streams.)

Is there some way to tune the TCP stack to better handle reordered frames?

Quoted 10 lines · Ben Greear

We are investigating a system that has fairly poor TCP throughput with the 3.17 and 4.0 kernels, but evidently it worked pretty well with 3.14 (I should be able to verify 3.14 later today).

One thing I notice is that a UDP download test shows lots of reordered frames, so I am thinking maybe TCP is running slow because of this.

(We see about 800Mbps UDP download, but only 500Mbps TCP, even when
using 100 concurrent TCP streams.)

Quoted 1 line

Is there some way to tune the TCP stack to better handle reordered frames?

Nothing yet. Are you the sender or the receiver ?

You really want to avoid reorders as much as possible.

Are you telling us something broke in networking layers between 3.14 and 3.17 leadings to reorders ?

Quoted 11 lines · Eric Dumazet

Quoted 10 lines · Ben Greear

We are investigating a system that has fairly poor TCP throughput with the 3.17 and 4.0 kernels, but evidently it worked pretty well with 3.14 (I should be able to verify 3.14 later today).

One thing I notice is that a UDP download test shows lots of reordered frames, so I am thinking maybe TCP is running slow because of this.

(We see about 800Mbps UDP download, but only 500Mbps TCP, even when
using 100 concurrent TCP streams.)

Quoted 8 lines

Quoted 1 line

Is there some way to tune the TCP stack to better handle reordered frames?

Nothing yet. Are you the sender or the receiver ?

You really want to avoid reorders as much as possible.

Are you telling us something broke in networking layers between 3.14 and 3.17 leadings to reorders ?

I am both sender and receiver, through an access-controller and wifi AP as DUT. The sender is Intel 1G NIC, so I suspect it is not causing reordering, which indicates most likely DUT is to blame.

Using several off-the-shelf APs in our lab we do not see this problem.

I am not certain yet what is the difference, but customer reports 600+Mbps with their older code, and best I can get is around 500Mbps with newer stuff.

Lots of stuff changed though (ath10k firmware, user-space at least slightly, kernel, etc), so possibly the regression is elsewhere.

Quoted 26 lines · Ben Greear

Quoted 11 lines · Eric Dumazet

Quoted 10 lines · Ben Greear

We are investigating a system that has fairly poor TCP throughput with the 3.17 and 4.0 kernels, but evidently it worked pretty well with 3.14 (I should be able to verify 3.14 later today).

One thing I notice is that a UDP download test shows lots of reordered frames, so I am thinking maybe TCP is running slow because of this.

(We see about 800Mbps UDP download, but only 500Mbps TCP, even when
using 100 concurrent TCP streams.)

Quoted 8 lines

Quoted 1 line

Is there some way to tune the TCP stack to better handle reordered frames?

Nothing yet. Are you the sender or the receiver ?

You really want to avoid reorders as much as possible.

Are you telling us something broke in networking layers between 3.14 and 3.17 leadings to reorders ?

I am both sender and receiver, through an access-controller and wifi AP as DUT. The sender is Intel 1G NIC, so I suspect it is not causing reordering, which indicates most likely DUT is to blame.

Quoted 8 lines

Using several off-the-shelf APs in our lab we do not see this problem.

I am not certain yet what is the difference, but customer reports 600+Mbps with their older code, and best I can get is around 500Mbps with newer stuff.

Lots of stuff changed though (ath10k firmware, user-space at least slightly, kernel, etc), so possibly the regression is elsewhere.

You possibly could send me some pcap (limited to the headers, using -s 128 for example) and limited to few flows, not the whole of them ;)

TCP reorders are tricky for the receiver : It sends a lot of SACK (one for every incoming packet, instead of the normal rule of sending one ACK for two incoming packets)

Increasing number of ACK might impact half-duplex networks, but also considerably increase cpu processing time.

I will work on captures...do you care if it is from transmitter or receiver's perspective?

Quoted 1 line · Ben Greear

I will work on captures...do you care if it is from transmitter or receiver's perspective?

Receiver would probably be more useful.