vxlan/veth performance issues on net.git + latest kernels

I've been chasing lately a performance issues which come into play when combining veth and vxlan over fast Ethernet NIC.

I came across it while working to enable TCP stateless offloads for vxlan encapsulated traffic in the mlx4 driver, but I can clearly see the issue without any HWoffloads involved, so it would be easier to discuss like that (no offloads involved).

The setup involves a stacked {veth --> bridge --> vlxan --> IP stack --> NIC} or {veth --> ovs+vxlan --> IP stack --> NIC} chain.

Basically, in my testbed which uses iperf over 40Gbs Mellanox NICs, vxlan traffic goes up to 5-7Gbs for single session and up to 14Gbs for multiple sessions, as long as veth isn't involved. Once veth is used I can't get to > 7-8Gbs, no matter how many sessions are used. For the time being, I manually took into account the tunneling overhead and reduced the veth pair MTU by 50 bytes.

Looking on the kernel TCP counters in a {veth --> bridge --> vxlan --> NIC} configuration, on the client side I see lots of hits for the following TCP counters (the numbers are just single sample, I look on the output of iterative sampling every seconds, e.g using "watch -d -n 1 netstat -st"):

67092 segments retransmited

31461 times recovered from packet loss due to SACK data Detected reordering 1045142 times using SACK 436215 fast retransmits
59966 forward retransmits

Also on the passive side I see hits for the "Quick ack mode was activated N times" counter, see below full snapshot of the counters from both sides.

Without using veth, e.g when running in a {vxlan -> NIC} or {bridge --> vxlan --> NIC},I see hits only for the "recovered from packet loss due to SACK data" counter and fastretransmits counter, but not for the forward retransmits or "Detected reordering N timesusing SACK". Also, the quick ack mode counter isn't active on the passive side.

I tested net.git (3.13-rc2+), 3.12.2 and 3.11.9, I see the same problems on all. At this point I don't really see a past point to go and apply bisection. So I hope this counter report can help to shed some light on the nature of the problem and possible solution, so ideas welcome!!

without vxlan, these are the Gbs results for 1/2/4 streams over 3.12.2, the results
for the net.git are pretty much the same.

18/32/38  NIC
17/30/35  bridge --> NIC
14/23/35  veth --> bridge --> NIC

with vxlan, these are the Gbs results for 1/2/4 streams

6/12/14  vxlan --> IP --> NIC
5/10/14  bridge --> vxlan --> IP --> NIC
6/7/7    veth --> bridge --> vxlan --> IP --> NIC

Also, the 3.12.2 number do get any better also when adding a ported version of 82d8189826d5 "veth: extend features to support tunneling" on top of 3.12.2

See @ the end the sequence of commands I use for the environment

Or.

--> TCP counters from active side

# netstat -ts
IcmpMsg:
     InType0: 2
     InType8: 1
     OutType0: 1
     OutType3: 4
     OutType8: 2
Tcp:
     189 active connections openings
     4 passive connection openings
     0 failed connection attempts
     0 connection resets received
     4 connections established
     22403193 segments received
     541234150 segments send out
     14248 segments retransmited
     0 bad segments received.
     5 resets sent
UdpLite:
TcpExt:
     2 invalid SYN cookies received
     178 TCP sockets finished time wait in fast timer
     10 delayed acks sent
     Quick ack mode was activated 1 times
     4 packets directly queued to recvmsg prequeue.
     3728 packets directly received from backlog
     2 packets directly received from prequeue
     2524 packets header predicted
     4 packets header predicted and directly queued to user
     19793310 acknowledgments not containing data received
     1216966 predicted acknowledgments
     2130 times recovered from packet loss due to SACK data
     Detected reordering 73 times using FACK
     Detected reordering 11424 times using SACK
     55 congestion windows partially recovered using Hoe heuristic
     TCPDSACKUndo: 457
     2 congestion windows recovered after partial ack
     11498 fast retransmits
     2748 forward retransmits
     2 other TCP timeouts
     TCPLossProbes: 4
     3 DSACKs sent for old packets
     TCPSackShifted: 1037782
     TCPSackMerged: 332827
     TCPSackShiftFallback: 598055
     TCPRcvCoalesce: 380
     TCPOFOQueue: 463
     TCPSpuriousRtxHostQueues: 192
IpExt:
     InNoRoutes: 1
     InMcastPkts: 191
     OutMcastPkts: 28
     InBcastPkts: 25
     InOctets: 1789360097
     OutOctets: 893757758988
     InMcastOctets: 8152
     OutMcastOctets: 3044
     InBcastOctets: 4259
     InNoECTPkts: 30117553

--> TCP counters from passiveside

netstat -ts
IcmpMsg:

     InType0: 1
     InType8: 2
     OutType0: 2
     OutType3: 5
     OutType8: 1

Tcp:
75 active connections openings
140 passive connection openings
0 failed connection attempts
0 connection resets received
4 connections established
146888643 segments received
27430160 segments send out
0 segments retransmited
0 bad segments received.
6 resets sent
UdpLite:
TcpExt:
3 invalid SYN cookies received
72 TCP sockets finished time wait in fast timer 10 delayed acks sent
3 delayed acks further delayed because of locked socket Quick ack mode was activated 13548 times
4 packets directly queued to recvmsg prequeue. 2 packets directly received from prequeue

139384763 packets header predicted
2 packets header predicted and directly queued to user 671 acknowledgments not containing data received 938 predicted acknowledgments
TCPLossProbes: 2
TCPLossProbeRecovery: 1
14 DSACKs sent for old packets
TCPBacklogDrop: 848

     TCPRcvCoalesce: 118368414

     TCPOFOQueue: 3167879

IpExt:

     InNoRoutes: 1
     InMcastPkts: 184
     OutMcastPkts: 26
     InBcastPkts: 26
     InOctets: 1007364296775
     OutOctets: 2433872888
     InMcastOctets: 6202
     OutMcastOctets: 2888
     InBcastOctets: 4597
     InNoECTPkts: 702313233

client side (node 144)
----------------------

ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN ifconfig vxlan42 192.168.42.144/24 up

brctl addbr br-vx
ip link set br-vx up

ifconfig br-vx 192.168.52.144/24 up
brctl addif br-vx vxlan42

ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.144/24 up
ip link set veth1 up

ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450

server side (node 147)
----------------------

ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN ifconfig vxlan42 192.168.42.147/24 up

brctl addbr br-vx
ip link set br-vx up

ifconfig br-vx 192.168.52.147/24 up
brctl addif br-vx vxlan42

ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.147/24 up
ip link set veth1 up

ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450

Quoted 162 lines · Or Gerlitz

I've been chasing lately a performance issues which come into play when combining veth and vxlan over fast Ethernet NIC.

I came across it while working to enable TCP stateless offloads for vxlan encapsulated traffic in the mlx4 driver, but I can clearly see the issue without any HWoffloads involved, so it would be easier to discuss like that (no offloads involved).

The setup involves a stacked {veth --> bridge --> vlxan --> IP stack --> NIC} or {veth --> ovs+vxlan --> IP stack --> NIC} chain.

Basically, in my testbed which uses iperf over 40Gbs Mellanox NICs, vxlan traffic goes up to 5-7Gbs for single session and up to 14Gbs for multiple sessions, as long as veth isn't involved. Once veth is used I can't get to > 7-8Gbs, no matter how many sessions are used. For the time being, I manually took into account the tunneling overhead and reduced the veth pair MTU by 50 bytes.

Looking on the kernel TCP counters in a {veth --> bridge --> vxlan --> NIC} configuration, on the client side I see lots of hits for the following TCP counters (the numbers are just single sample, I look on the output of iterative sampling every seconds, e.g using "watch -d -n 1 netstat -st"):

67092 segments retransmited

31461 times recovered from packet loss due to SACK data Detected reordering 1045142 times using SACK 436215 fast retransmits
59966 forward retransmits

Also on the passive side I see hits for the "Quick ack mode was activated N times" counter, see below full snapshot of the counters from both sides.

Without using veth, e.g when running in a {vxlan -> NIC} or {bridge --> vxlan --> NIC},I see hits only for the "recovered from packet loss due to SACK data" counter and fastretransmits counter, but not for the forward retransmits or "Detected reordering N timesusing SACK". Also, the quick ack mode counter isn't active on the passive side.

I tested net.git (3.13-rc2+), 3.12.2 and 3.11.9, I see the same problems on all. At this point I don't really see a past point to go and apply bisection. So I hope this counter report can help to shed some light on the nature of the problem and possible solution, so ideas welcome!!

without vxlan, these are the Gbs results for 1/2/4 streams over 3.12.2, the results
for the net.git are pretty much the same.

18/32/38  NIC
17/30/35  bridge --> NIC
14/23/35  veth --> bridge --> NIC

with vxlan, these are the Gbs results for 1/2/4 streams

6/12/14  vxlan --> IP --> NIC
5/10/14  bridge --> vxlan --> IP --> NIC
6/7/7    veth --> bridge --> vxlan --> IP --> NIC

Also, the 3.12.2 number do get any better also when adding a ported version of 82d8189826d5 "veth: extend features to support tunneling" on top of 3.12.2

See @ the end the sequence of commands I use for the environment

Or.

--> TCP counters from active side

# netstat -ts
IcmpMsg:
     InType0: 2
     InType8: 1
     OutType0: 1
     OutType3: 4
     OutType8: 2
Tcp:
     189 active connections openings
     4 passive connection openings
     0 failed connection attempts
     0 connection resets received
     4 connections established
     22403193 segments received
     541234150 segments send out
     14248 segments retransmited
     0 bad segments received.
     5 resets sent
UdpLite:
TcpExt:
     2 invalid SYN cookies received
     178 TCP sockets finished time wait in fast timer
     10 delayed acks sent
     Quick ack mode was activated 1 times
     4 packets directly queued to recvmsg prequeue.
     3728 packets directly received from backlog
     2 packets directly received from prequeue
     2524 packets header predicted
     4 packets header predicted and directly queued to user
     19793310 acknowledgments not containing data received
     1216966 predicted acknowledgments
     2130 times recovered from packet loss due to SACK data
     Detected reordering 73 times using FACK
     Detected reordering 11424 times using SACK
     55 congestion windows partially recovered using Hoe heuristic
     TCPDSACKUndo: 457
     2 congestion windows recovered after partial ack
     11498 fast retransmits
     2748 forward retransmits
     2 other TCP timeouts
     TCPLossProbes: 4
     3 DSACKs sent for old packets
     TCPSackShifted: 1037782
     TCPSackMerged: 332827
     TCPSackShiftFallback: 598055
     TCPRcvCoalesce: 380
     TCPOFOQueue: 463
     TCPSpuriousRtxHostQueues: 192
IpExt:
     InNoRoutes: 1
     InMcastPkts: 191
     OutMcastPkts: 28
     InBcastPkts: 25
     InOctets: 1789360097
     OutOctets: 893757758988
     InMcastOctets: 8152
     OutMcastOctets: 3044
     InBcastOctets: 4259
     InNoECTPkts: 30117553

--> TCP counters from passiveside

netstat -ts
IcmpMsg:

     InType0: 1
     InType8: 2
     OutType0: 2
     OutType3: 5
     OutType8: 1

Tcp:
75 active connections openings
140 passive connection openings
0 failed connection attempts
0 connection resets received
4 connections established
146888643 segments received
27430160 segments send out
0 segments retransmited
0 bad segments received.
6 resets sent
UdpLite:
TcpExt:
3 invalid SYN cookies received
72 TCP sockets finished time wait in fast timer 10 delayed acks sent
3 delayed acks further delayed because of locked socket Quick ack mode was activated 13548 times
4 packets directly queued to recvmsg prequeue. 2 packets directly received from prequeue

Quoted 8 lines

139384763 packets header predicted
2 packets header predicted and directly queued to user 671 acknowledgments not containing data received 938 predicted acknowledgments
TCPLossProbes: 2
TCPLossProbeRecovery: 1
14 DSACKs sent for old packets
TCPBacklogDrop: 848

Thats bad : Dropping packets on receiver.

Check also "ifconfig -a" to see if rxdrop is increasing as well.

Quoted 1 line

     TCPRcvCoalesce: 118368414

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

Quoted 48 lines

IpExt:

     InNoRoutes: 1
     InMcastPkts: 184
     OutMcastPkts: 26
     InBcastPkts: 26
     InOctets: 1007364296775
     OutOctets: 2433872888
     InMcastOctets: 6202
     OutMcastOctets: 2888
     InBcastOctets: 4597
     InNoECTPkts: 702313233

client side (node 144)
----------------------

ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN ifconfig vxlan42 192.168.42.144/24 up

brctl addbr br-vx
ip link set br-vx up

ifconfig br-vx 192.168.52.144/24 up
brctl addif br-vx vxlan42

ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.144/24 up
ip link set veth1 up

ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450

server side (node 147)
----------------------

ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN ifconfig vxlan42 192.168.42.147/24 up

brctl addbr br-vx
ip link set br-vx up

ifconfig br-vx 192.168.52.147/24 up
brctl addif br-vx vxlan42

Quoted 9 lines

ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.147/24 up
ip link set veth1 up

ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 180 lines · Eric Dumazet

Quoted 162 lines · Or Gerlitz

I've been chasing lately a performance issues which come into play when combining veth and vxlan over fast Ethernet NIC.

I came across it while working to enable TCP stateless offloads for vxlan encapsulated traffic in the mlx4 driver, but I can clearly see the issue without any HWoffloads involved, so it would be easier to discuss like that (no offloads involved).

The setup involves a stacked {veth --> bridge --> vlxan --> IP stack --> NIC} or {veth --> ovs+vxlan --> IP stack --> NIC} chain.

Basically, in my testbed which uses iperf over 40Gbs Mellanox NICs, vxlan traffic goes up to 5-7Gbs for single session and up to 14Gbs for multiple sessions, as long as veth isn't involved. Once veth is used I can't get to > 7-8Gbs, no matter how many sessions are used. For the time being, I manually took into account the tunneling overhead and reduced the veth pair MTU by 50 bytes.

Looking on the kernel TCP counters in a {veth --> bridge --> vxlan --> NIC} configuration, on the client side I see lots of hits for the following TCP counters (the numbers are just single sample, I look on the output of iterative sampling every seconds, e.g using "watch -d -n 1 netstat -st"):

67092 segments retransmited

31461 times recovered from packet loss due to SACK data Detected reordering 1045142 times using SACK 436215 fast retransmits
59966 forward retransmits

Also on the passive side I see hits for the "Quick ack mode was activated N times" counter, see below full snapshot of the counters from both sides.

Without using veth, e.g when running in a {vxlan -> NIC} or {bridge --> vxlan --> NIC},I see hits only for the "recovered from packet loss due to SACK data" counter and fastretransmits counter, but not for the forward retransmits or "Detected reordering N timesusing SACK". Also, the quick ack mode counter isn't active on the passive side.

I tested net.git (3.13-rc2+), 3.12.2 and 3.11.9, I see the same problems on all. At this point I don't really see a past point to go and apply bisection. So I hope this counter report can help to shed some light on the nature of the problem and possible solution, so ideas welcome!!

without vxlan, these are the Gbs results for 1/2/4 streams over 3.12.2, the results
for the net.git are pretty much the same.

18/32/38  NIC
17/30/35  bridge --> NIC
14/23/35  veth --> bridge --> NIC

with vxlan, these are the Gbs results for 1/2/4 streams

6/12/14  vxlan --> IP --> NIC
5/10/14  bridge --> vxlan --> IP --> NIC
6/7/7    veth --> bridge --> vxlan --> IP --> NIC

Also, the 3.12.2 number do get any better also when adding a ported version of 82d8189826d5 "veth: extend features to support tunneling" on top of 3.12.2

See @ the end the sequence of commands I use for the environment

Or.

--> TCP counters from active side

# netstat -ts
IcmpMsg:
     InType0: 2
     InType8: 1
     OutType0: 1
     OutType3: 4
     OutType8: 2
Tcp:
     189 active connections openings
     4 passive connection openings
     0 failed connection attempts
     0 connection resets received
     4 connections established
     22403193 segments received
     541234150 segments send out
     14248 segments retransmited
     0 bad segments received.
     5 resets sent
UdpLite:
TcpExt:
     2 invalid SYN cookies received
     178 TCP sockets finished time wait in fast timer
     10 delayed acks sent
     Quick ack mode was activated 1 times
     4 packets directly queued to recvmsg prequeue.
     3728 packets directly received from backlog
     2 packets directly received from prequeue
     2524 packets header predicted
     4 packets header predicted and directly queued to user
     19793310 acknowledgments not containing data received
     1216966 predicted acknowledgments
     2130 times recovered from packet loss due to SACK data
     Detected reordering 73 times using FACK
     Detected reordering 11424 times using SACK
     55 congestion windows partially recovered using Hoe heuristic
     TCPDSACKUndo: 457
     2 congestion windows recovered after partial ack
     11498 fast retransmits
     2748 forward retransmits
     2 other TCP timeouts
     TCPLossProbes: 4
     3 DSACKs sent for old packets
     TCPSackShifted: 1037782
     TCPSackMerged: 332827
     TCPSackShiftFallback: 598055
     TCPRcvCoalesce: 380
     TCPOFOQueue: 463
     TCPSpuriousRtxHostQueues: 192
IpExt:
     InNoRoutes: 1
     InMcastPkts: 191
     OutMcastPkts: 28
     InBcastPkts: 25
     InOctets: 1789360097
     OutOctets: 893757758988
     InMcastOctets: 8152
     OutMcastOctets: 3044
     InBcastOctets: 4259
     InNoECTPkts: 30117553

--> TCP counters from passiveside

netstat -ts
IcmpMsg:

     InType0: 1
     InType8: 2
     OutType0: 2
     OutType3: 5
     OutType8: 1

Tcp:
75 active connections openings
140 passive connection openings
0 failed connection attempts
0 connection resets received
4 connections established
146888643 segments received
27430160 segments send out
0 segments retransmited
0 bad segments received.
6 resets sent
UdpLite:
TcpExt:
3 invalid SYN cookies received
72 TCP sockets finished time wait in fast timer 10 delayed acks sent
3 delayed acks further delayed because of locked socket Quick ack mode was activated 13548 times
4 packets directly queued to recvmsg prequeue. 2 packets directly received from prequeue

Quoted 8 lines

139384763 packets header predicted
2 packets header predicted and directly queued to user 671 acknowledgments not containing data received 938 predicted acknowledgments
TCPLossProbes: 2
TCPLossProbeRecovery: 1
14 DSACKs sent for old packets
TCPBacklogDrop: 848

Thats bad : Dropping packets on receiver.

Check also "ifconfig -a" to see if rxdrop is increasing as well.

Quoted 1 line

     TCPRcvCoalesce: 118368414

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 3 lines

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

Quoted 61 lines

Quoted 50 lines

IpExt:

     InNoRoutes: 1
     InMcastPkts: 184
     OutMcastPkts: 26
     InBcastPkts: 26
     InOctets: 1007364296775
     OutOctets: 2433872888
     InMcastOctets: 6202
     OutMcastOctets: 2888
     InBcastOctets: 4597
     InNoECTPkts: 702313233

client side (node 144)
----------------------

ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN ifconfig vxlan42 192.168.42.144/24 up

brctl addbr br-vx
ip link set br-vx up

ifconfig br-vx 192.168.52.144/24 up
brctl addif br-vx vxlan42

ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.144/24 up
ip link set veth1 up

ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450

server side (node 147)
----------------------

ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN ifconfig vxlan42 192.168.42.147/24 up

brctl addbr br-vx
ip link set br-vx up

ifconfig br-vx 192.168.52.147/24 up
brctl addif br-vx vxlan42

Quoted 9 lines

ip link add type veth
brctl addif br-vx veth1
ifconfig veth0 192.168.62.147/24 up
ip link set veth1 up

ifconfig veth0 mtu 1450
ifconfig veth1 mtu 1450

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 8 lines

Quoted 3 lines

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

I have seen this before. Here are my findings:

The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation.

I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix.

Quoted 18 lines · Joseph Gasparakis

Quoted 8 lines

Quoted 3 lines

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

I have seen this before. Here are my findings:

The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation.

Quoted 3 lines

I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix.

thanks for the heads up, few quick questions/clafications --

-- you are talking on drops done @ the sender side, correct? Eric was saying we have evidences that the drops happen on the receiver.

Quoted 25 lines · Or Gerlitz

Quoted 21 lines · Joseph Gasparakis

Quoted 8 lines

Quoted 3 lines

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

I have seen this before. Here are my findings:

The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation.

I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix.

Quoted 4 lines

thanks for the heads up, few quick questions/clafications --

-- you are talking on drops done @ the sender side, correct? Eric was saying we have evidences that the drops happen on the receiver.

I suggested you take a look at the receiver, like "ifconfig -a"

I suspect one cpu is 100% in sofirq mode draining packets from the NIC and feeding IP / TCP stack.

Because of vxlan encap, all the packets are delivered to a single RX queue (I dont think mlx4 is able to look at inner header to get L4 info)

mpstat -P ALL 10 10

I also thought that under veth there's contention point which could explain why packets are dropped, but haven't found it.

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Quoted 3 lines · David Miller

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Most easy way is to do :

perf record -e skb:kfree_skb -a -g sleep 10

perf report

Quoted 3 lines · David Miller

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Most easy way is to do :

perf record -e skb:kfree_skb -a -g sleep 10

perf report

Quoted 9 lines · Eric Dumazet

Quoted 3 lines · David Miller

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Most easy way is to do :

perf record -e skb:kfree_skb -a -g sleep 10

some typo here? I tried the perf tool that comes with the net git

./perf record -e skb:kfree_skb -a -g sleep 10
invalid or unsupported event: 'skb:kfree_skb'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-e, --event <event> event selector. use 'perf list' to list
available events

Quoted 20 lines · Or Gerlitz

Quoted 9 lines · Eric Dumazet

Quoted 3 lines · David Miller

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Most easy way is to do :

perf record -e skb:kfree_skb -a -g sleep 10

some typo here? I tried the perf tool that comes with the net git

./perf record -e skb:kfree_skb -a -g sleep 10
invalid or unsupported event: 'skb:kfree_skb'
Run 'perf list' for a list of valid events

usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

Quoted 2 lines

-e, --event <event> event selector. use 'perf list' to list
available events

-g takes an optional argument. Reorder the arguments:

perf record -e skb:kfree_skb -g -a sleep 10

Sorry, it doesn't help, I get the same error even with the oerf kernel.org tree which has these events (no sign for skb's)

List of pre-defined events (to be used in -e):

  cpu-cycles OR cycles                               [Hardware event]
  instructions                                       [Hardware event]
  cache-references                                   [Hardware event]
  cache-misses                                       [Hardware event]
  branch-instructions OR branches                    [Hardware event]
  branch-misses                                      [Hardware event]
  bus-cycles                                         [Hardware event]
  stalled-cycles-frontend OR idle-cycles-frontend    [Hardware event]
  stalled-cycles-backend OR idle-cycles-backend      [Hardware event]
  ref-cycles                                         [Hardware event]

  cpu-clock                                          [Software event]
  task-clock                                         [Software event]
  page-faults OR faults                              [Software event]
  context-switches OR cs                             [Software event]
  cpu-migrations OR migrations                       [Software event]
  minor-faults                                       [Software event]
  major-faults                                       [Software event]
  alignment-faults                                   [Software event]
  emulation-faults                                   [Software event]

  L1-dcache-loads                                    [Hardware cache event]
  L1-dcache-load-misses                              [Hardware cache event]
  L1-dcache-stores                                   [Hardware cache event]
  L1-dcache-store-misses                             [Hardware cache event]
  L1-dcache-prefetches                               [Hardware cache event]
  L1-dcache-prefetch-misses                          [Hardware cache event]
  L1-icache-loads                                    [Hardware cache event]
  L1-icache-load-misses                              [Hardware cache event]
  L1-icache-prefetches                               [Hardware cache event]
  L1-icache-prefetch-misses                          [Hardware cache event]
  LLC-loads                                          [Hardware cache event]
  LLC-load-misses                                    [Hardware cache event]
  LLC-stores                                         [Hardware cache event]

  LLC-store-misses                                   [Hardware cache event]
  LLC-prefetches                                     [Hardware cache event]
  LLC-prefetch-misses                                [Hardware cache event]
  dTLB-loads                                         [Hardware cache event]
  dTLB-load-misses                                   [Hardware cache event]
  dTLB-stores                                        [Hardware cache event]
  dTLB-store-misses                                  [Hardware cache event]
  dTLB-prefetches                                    [Hardware cache event]

Quoted 3 lines · David Miller

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Most easy way is to do :

perf record -e skb:kfree_skb -a -g sleep 10
perf report

Quoted 10 lines · Eric Dumazet

Quoted 3 lines · David Miller

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Most easy way is to do :

perf record -e skb:kfree_skb -a -g sleep 10
perf report

$ ./perf record -e skb:kfree_skb -g -a sleep 10
$ ./perf report -i perf.data

Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406

(quoted diff) +1 -0

+  97.13%          swapper  [kernel.kallsyms]  [k] net_tx_action

Quoted 6 lines

Quoted 18 lines · Or Gerlitz

Quoted 10 lines · Eric Dumazet

Quoted 3 lines · David Miller

At this point I would use drop monitor to figure out in what context packets are being dropped on the floor. There are scripts provided with the perf tool to utilize it.

Most easy way is to do :

perf record -e skb:kfree_skb -a -g sleep 10
perf report

$ ./perf record -e skb:kfree_skb -g -a sleep 10
$ ./perf report -i perf.data

Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406

(quoted diff) +1 -0

+  97.13%          swapper  [kernel.kallsyms]  [k] net_tx_action

Review

Quoted 10 lines

(quoted diff) +10 -0

+   1.53%            iperf  [kernel.kallsyms]  [k] net_tx_action
+   1.03%             perf  [kernel.kallsyms]  [k] net_tx_action
+   0.27%      ksoftirqd/7  [kernel.kallsyms]  [k] net_tx_action
+   0.03%      kworker/7:1  [kernel.kallsyms]  [k] net_tx_action
+   0.00%          rpcbind  [kernel.kallsyms]  [k] net_tx_action
+   0.00%          swapper  [kernel.kallsyms]  [k] kfree_skb
+   0.00%            sleep  [kernel.kallsyms]  [k] net_tx_action
+   0.00%  hald-addon-acpi  [kernel.kallsyms]  [k] kfree_skb
+   0.00%            iperf  [kernel.kallsyms]  [k] kfree_skb
+   0.00%             perf  [kernel.kallsyms]  [k] kfree_skb

I added proper sorting (thanks Rick), here's the passive side report

include/linux/netdevice.h +7 -0

+#define SKB_CONSUMED_MAGIC ((void *)0xDEAD0001)
+static inline void dev_consume_skb_any(struct sk_buff *skb)
+{
+       skb->dev = SKB_CONSUMED_MAGIC;
+       dev_kfree_skb_any(skb);
+}
+
 int netif_rx(struct sk_buff *skb);
 int netif_rx_ni(struct sk_buff *skb);
 int netif_receive_skb(struct sk_buff *skb);

net/core/dev.c +0 -0

index ba3b7ea5ebb3..b2b0e5776ce9 100644

Could you use some other way to mark skb ?

In tracing we might want to examine skb more carefully and not being able to see the device
will limit the usability of this tracepoint.

Quoted 2 lines · Alexei Starovoitov

Could you use some other way to mark skb ?

I could ;)

Quoted 3 lines

In tracing we might want to examine skb more carefully and not being able to see the device
will limit the usability of this tracepoint.

Unfortunately, using skb->dev as a pointer to device would be buggy or expensive (you would need to take a reference on device in order not letting it disappear, as we escape RCU protection)

Anyway, this magic is pretty easy to change, I am open to suggestions.

Quoted 7 lines · Eric Dumazet

Quoted 2 lines · Alexei Starovoitov

Could you use some other way to mark skb ?

I could ;)

Quoted 7 lines

Quoted 3 lines

In tracing we might want to examine skb more carefully and not being able to see the device
will limit the usability of this tracepoint.

Unfortunately, using skb->dev as a pointer to device would be buggy or expensive (you would need to take a reference on device in order not letting it disappear, as we escape RCU protection)

well, yes, you might have an skb around when device is already freed when skb_dst_noref.
but I'm not suggesting anything expensive. Tracing definitely should not add overhead by doing rcu_lock() or dev_hold(). Instead it can go through skb, skb->dev, skb->dev->xxx via probe_kernel_read(). If dev is gone, it's still safe.

Quoted 2 lines

Anyway, this magic is pretty easy to change, I am open to suggestions.

you're the expert :) use skb->mark field, since it's unused during freeing path... ?

Quoted 16 lines · Alexei Starovoitov

Quoted 7 lines · Eric Dumazet

Quoted 2 lines · Alexei Starovoitov

Could you use some other way to mark skb ?

I could ;)

Quoted 7 lines

Quoted 3 lines

In tracing we might want to examine skb more carefully and not being able to see the device
will limit the usability of this tracepoint.

Unfortunately, using skb->dev as a pointer to device would be buggy or expensive (you would need to take a reference on device in order not letting it disappear, as we escape RCU protection)

Quoted 6 lines

well, yes, you might have an skb around when device is already freed when skb_dst_noref.
but I'm not suggesting anything expensive. Tracing definitely should not add overhead by doing rcu_lock() or dev_hold(). Instead it can go through skb, skb->dev, skb->dev->xxx via probe_kernel_read(). If dev is gone, it's still safe.

Its certainly not safe to 'probe'.

Its not about faulting inexistent memory, that is the least of the problem.

Any kind of information fetched from skb->dev might have been overwritten.

You could for example fetch security sensitive data and expose it.

Quoted 4 lines

Quoted 1 line

Anyway, this magic is pretty easy to change, I am open to suggestions.

you're the expert :) use skb->mark field, since it's unused during freeing path... ?

cache line miss ;)

skb->dev is in the first cache line, where we access skb->next anyway.

I could use skb->cb[] like the following patch :

include/linux/netdevice.h +16 -1

+struct __dev_kfree_skb_cb {
+       unsigned int reason;
+};
+
+static inline struct __dev_kfree_skb_cb *get_kfree_skb_cb(const struct sk_buff *skb)
+{
+       return (struct __dev_kfree_skb_cb *)skb->cb;
+}
+
+enum {
+       SKB_REASON_CONSUMED,
+       SKB_REASON_DROPPED,
+};
+
 /* Use this variant in places where it could be invoked
  * from either hardware interrupt or other context, with hardware interrupts
  * either disabled or enabled.
+ * Note that TX completion should use dev_consume_skb_any()
  */
-void dev_kfree_skb_any(struct sk_buff *skb);
+static inline void dev_kfree_skb_any(struct sk_buff *skb)

skb->dev is in the first cache line, where we access skb->next anyway. I could use skb->cb[] like the following patch :

Quoted 2 lines · Eric Dumazet

skb->dev is in the first cache line, where we access skb->next anyway. I could use skb->cb[] like the following patch :

Hi Eric, I applied on the net tree the patch you posted yesterday "net: introduce dev_consume_skb_any()" along with the network drivers part of this patch, unless I got it wrong, I assume both pieces are needed?

So I re-run the vxlan/veth test that we suspect goes through packet drops on TX.

With the patches applied I have almost no samples of that event

$ ./perf report -i perf.data

Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89

(quoted diff) +6 -0

+  39.33%  ksoftirqd/2  [kernel.kallsyms]  [k] net_tx_action
 
                    ?
+  28.09%      swapper  [kernel.kallsyms]  [k] net_tx_action
 
                    ?
+  28.09%         sshd  [kernel.kallsyms]  [k] net_tx_action
 
                    ?
+   2.25%      swapper  [kernel.kallsyms]  [k] kfree_skb
 
                    ?
+   1.12%  kworker/2:2  [kernel.kallsyms]  [k] net_tx_action
 
                    ?
+   1.12%        iperf  [kernel.kallsyms]  [k] net_tx_action
 
 ./perf report -i perf.data --sort dso,symbol

Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89

Quoted 9 lines · Or Gerlitz

Quoted 2 lines · Eric Dumazet

skb->dev is in the first cache line, where we access skb->next anyway. I could use skb->cb[] like the following patch :

Hi Eric, I applied on the net tree the patch you posted yesterday "net: introduce dev_consume_skb_any()" along with the network drivers part of this patch, unless I got it wrong, I assume both pieces are needed?

Quoted 5 lines

So I re-run the vxlan/veth test that we suspect goes through packet drops on TX.

With the patches applied I have almost no samples of that event

$ ./perf report -i perf.data

How did you get this perf.data file ? There are a few drops.

Review

Quoted 13 lines

Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89

(quoted diff) +6 -0

+  39.33%  ksoftirqd/2  [kernel.kallsyms]  [k] net_tx_action
+  28.09%      swapper  [kernel.kallsyms]  [k] net_tx_action
+  28.09%         sshd  [kernel.kallsyms]  [k] net_tx_action
+   2.25%      swapper  [kernel.kallsyms]  [k] kfree_skb
+   1.12%  kworker/2:2  [kernel.kallsyms]  [k] net_tx_action
+   1.12%        iperf  [kernel.kallsyms]  [k] net_tx_action
 
 ./perf report -i perf.data --sort dso,symbol

Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89

(quoted diff) +2 -1

+  97.75%  [kernel.kallsyms]  [k] net_tx_action
+   2.25%  [kernel.kallsyms]  [k] kfree_skb
--

OK, this means your driver drops few packets in its ndo_start_xmit() handler.

Could you give us "ifconfig -a" reports as I already asked ?

You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit() to call kfree_skb(skb) instead, to get a stack trace (perf record -a -g -e skb:kfree_skb sleep 20 ; perf report)

drivers/net/ethernet/mellanox/mlx4/en_tx.c +1 -1

index f54ebd5a1702..53130f27dec0 100644
@@ -873,7 +873,7 @@ tx_drop_unmap:
        }
 
 tx_drop:
-       dev_kfree_skb_any(skb);
+       kfree_skb(skb);
        priv->stats.tx_dropped++;
        return NETDEV_TX_OK;
 }

Quoted 8 lines · Or Gerlitz

Quoted 2 lines · Eric Dumazet

skb->dev is in the first cache line, where we access skb->next anyway. I could use skb->cb[] like the following patch :

Hi Eric, I applied on the net tree the patch you posted yesterday "net: introduce dev_consume_skb_any()" along with the network drivers part of this patch, unless I got it wrong, I assume both pieces are needed?

Quoted 5 lines

So I re-run the vxlan/veth test that we suspect goes through packet drops on TX.

With the patches applied I have almost no samples of that event

$ ./perf report -i perf.data

How did you get this perf.data file ? There are a few drops.

Quoted 8 lines

Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89

(quoted diff) +6 -0

+  39.33%  ksoftirqd/2  [kernel.kallsyms]  [k] net_tx_action
+  28.09%      swapper  [kernel.kallsyms]  [k] net_tx_action
+  28.09%         sshd  [kernel.kallsyms]  [k] net_tx_action
+   2.25%      swapper  [kernel.kallsyms]  [k] kfree_skb
+   1.12%  kworker/2:2  [kernel.kallsyms]  [k] net_tx_action
+   1.12%        iperf  [kernel.kallsyms]  [k] net_tx_action

Review

Quoted 5 lines

./perf report -i perf.data --sort dso,symbol
Samples: 89 of event 'skb:kfree_skb', Event count (approx.): 89

(quoted diff) +2 -1

+  97.75%  [kernel.kallsyms]  [k] net_tx_action
+   2.25%  [kernel.kallsyms]  [k] kfree_skb
--

OK, this means your driver drops few packets in its ndo_start_xmit() handler.

Could you give us "ifconfig -a" reports as I already asked ?

You could temporary change the dev_kfree_skb_any() in mlx4_en_xmit() to call kfree_skb(skb) instead, to get a stack trace (perf record -a -g

(quoted diff) +0 -1

-e skb:kfree_skb sleep 20 ; perf report)

drivers/net/ethernet/mellanox/mlx4/en_tx.c +1 -1

index f54ebd5a1702..53130f27dec0 100644
@@ -873,7 +873,7 @@ tx_drop_unmap:
  	}
 
  tx_drop:
-	dev_kfree_skb_any(skb);
+	kfree_skb(skb);
  	priv->stats.tx_dropped++;
  	return NETDEV_TX_OK;
  }

Quoted 4 lines

Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406

(quoted diff) +1 -0

+  97.13%          swapper  [kernel.kallsyms]  [k] net_tx_action

Review

Quoted 10 lines

(quoted diff) +10 -0

+   1.53%            iperf  [kernel.kallsyms]  [k] net_tx_action
+   1.03%             perf  [kernel.kallsyms]  [k] net_tx_action
+   0.27%      ksoftirqd/7  [kernel.kallsyms]  [k] net_tx_action
+   0.03%      kworker/7:1  [kernel.kallsyms]  [k] net_tx_action
+   0.00%          rpcbind  [kernel.kallsyms]  [k] net_tx_action
+   0.00%          swapper  [kernel.kallsyms]  [k] kfree_skb
+   0.00%            sleep  [kernel.kallsyms]  [k] net_tx_action
+   0.00%  hald-addon-acpi  [kernel.kallsyms]  [k] kfree_skb
+   0.00%            iperf  [kernel.kallsyms]  [k] kfree_skb
+   0.00%             perf  [kernel.kallsyms]  [k] kfree_skb

Right, I actually have a patch for that, but was waiting for net-next being re-opened :

commit 9a731d750dd8bf0b8c20fb1ca53c42317fb4dd37

Author: Eric Dumazet <[email protected]>
Date:   Mon Nov 25 13:09:20 2013 -0800

net-fixes: introduce dev_consume_skb_any()

I definitely prefer the control block approach to this.

net/core/dev.c void __netif_schedule(struct Qdisc *q) · line 2162

Quoted 8 lines · Eric Dumazet

net/core/dev.c +2 -6

-		local_irq_save(flags);
-		sd = &__get_cpu_var(softnet_data);
-		skb->next = sd->completion_queue;
-		sd->completion_queue = skb;
-		raise_softirq_irqoff(NET_TX_SOFTIRQ);
-		local_irq_restore(flags);
+	if (likely(atomic_read(&skb->users) == 1)) {
+		smp_rmb();

Could you give me a hint why this barrier is needed? IMHO the volatile access in atomic_read should get rid of the control dependency so I don't see a need for this barrier. Without the volatile access a compiler-barrier would still suffice, I guess?

Quoted 14 lines · Hannes Frederic Sowa

net/core/dev.c void __netif_schedule(struct Qdisc *q) · line 2162

Quoted 8 lines · Eric Dumazet

net/core/dev.c +2 -6

-		local_irq_save(flags);
-		sd = &__get_cpu_var(softnet_data);
-		skb->next = sd->completion_queue;
-		sd->completion_queue = skb;
-		raise_softirq_irqoff(NET_TX_SOFTIRQ);
-		local_irq_restore(flags);
+	if (likely(atomic_read(&skb->users) == 1)) {
+		smp_rmb();

Could you give me a hint why this barrier is needed? IMHO the volatile access in atomic_read should get rid of the control dependency so I don't see a need for this barrier. Without the volatile access a compiler-barrier would still suffice, I guess?

Please take a look at kfree_skb() implementation.

If you think a comment is needed there, please feel free to add it.

Quoted 16 lines · Eric Dumazet

Quoted 14 lines · Hannes Frederic Sowa

net/core/dev.c void __netif_schedule(struct Qdisc *q) · line 2162

Quoted 8 lines · Eric Dumazet

net/core/dev.c +2 -6

-		local_irq_save(flags);
-		sd = &__get_cpu_var(softnet_data);
-		skb->next = sd->completion_queue;
-		sd->completion_queue = skb;
-		raise_softirq_irqoff(NET_TX_SOFTIRQ);
-		local_irq_restore(flags);
+	if (likely(atomic_read(&skb->users) == 1)) {
+		smp_rmb();

Could you give me a hint why this barrier is needed? IMHO the volatile access in atomic_read should get rid of the control dependency so I don't see a need for this barrier. Without the volatile access a compiler-barrier would still suffice, I guess?

Quoted 4 lines

Please take a look at kfree_skb() implementation.

If you think a comment is needed there, please feel free to add it.

My understanding of this (old) barrier here is an implicit wmb in skb_get()

This probably needs something like :

static inline struct sk_buff *skb_get(struct sk_buff *skb)

{
	smp_mb__before_atomic_inc(); /* check {consume|kfree}_skb() */
	atomic_inc(&skb->users);
}

It makes some sense but I did not grasp the whole ->users dependency picture, yet. I guess the barrier is only needed when refcount drops down to 0 and we don't necessarily need one when incrementing ->users.

Quoted 3 lines · Hannes Frederic Sowa

It makes some sense but I did not grasp the whole ->users dependency picture, yet. I guess the barrier is only needed when refcount drops down to 0 and we don't necessarily need one when incrementing ->users.

If you are the only user of this skb, really no smp barrier is needed at all.

The problem comes when another cpu is working on the skb, and finally releases its reference on it.

Before releasing its reference, it must commit all changes it might have done onto skb. Otherwise another cpu might read stale data.

The smp_wmb() is done by the atomic_dec_and_test(), as it contains a full barrier.

So the smp_rmb() pairs with the barrier done in atomic_dec_and_test()

From: Eric Dumazet <[email protected]>

Some network drivers use dev_kfree_skb_any() and dev_kfree_skb_irq() helpers to free skbs, both for dropped packets and TX completed ones.

We need to separate the two causes to get better diagnostics given by dropwatch or "perf record -e skb:kfree_skb"

This patch provides two new helpers, dev_consume_skb_any() and dev_consume_skb_irq() to be used for consumed skbs.

__dev_kfree_skb_irq() is slightly optimized to remove one atomic_dec_and_test() in fast path, and use this_cpu_{r|w} accessors.

Signed-off-by Eric Dumazet <[email protected]>

Quoted 18 lines · Joseph Gasparakis

Quoted 8 lines

Quoted 3 lines

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

I have seen this before. Here are my findings:

The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation.

Quoted 3 lines

I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix.

thanks for the heads up, few quick questions/clafications --

-- you are talking on drops done @ the sender side, correct? Eric was saying we have evidences that the drops happen on the receiver.

-- without the hack you did, still packets are sent/received, so what makes the stack to drop only some of them?

-- why packets coming from veth would have the SKB_GSO_DODGY bit set?

-- so where is now (say net.git or 3.12.x) this one line you commented out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c explicit setting of SKB_GSO_DODGY

Also, I am pretty sure the problem exists also when sending/receiving guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I just sticked to the veth flavour b/c its one (== the hypervisor) network stack to debug and not two (+ the guest one).
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 21 lines · Or Gerlitz

Quoted 18 lines · Joseph Gasparakis

Quoted 8 lines

Quoted 3 lines

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

I have seen this before. Here are my findings:

The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation.

Quoted 8 lines

Quoted 3 lines

I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix.

thanks for the heads up, few quick questions/clafications --

-- you are talking on drops done @ the sender side, correct? Eric was saying we have evidences that the drops happen on the receiver.

I am *guessing* drops on the Rx are due to the drops at the Tx. See my answer to your next question for more info.

Quoted 4 lines

-- without the hack you did, still packets are sent/received, so what makes the stack to drop only some of them?

What I had seen is GSOs getting dropped on the Tx side. Basically the GSOs never made it to the driver, they were broken into non GSO smaller skbs by the stack. I think the stack is not handling well the GSO with the DODGY bit set, and that causes it to maybe partially the packet to be emitted, causing the re-transmits (and maybe the drops on your Rx end)? Of course all this is speculation, the fact that I know is that as soon as I was forcing the gso type I saw offloaded VXLAN encapsulated traffic at decent speeds.

Quoted 1 line

-- why packets coming from veth would have the SKB_GSO_DODGY bit set?

That is something I would love to know too. I am guessing this is a way for the VM to say it is a non-trusted packet? And maybe all this can be fixed by maybe setting something on the VM through a userspace tool that will stop the veth to set the DODGY bit?

Quoted 1 line

Quoted 3 lines

-- so where is now (say net.git or 3.12.x) this one line you commented out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c explicit setting of SKB_GSO_DODGY

I did not commit it, as this was just a workaround to prove to myself that the problem I was seing was due to the gso_type, and it would actually just hide the problem and not give a proper solution to it.

Quoted 5 lines

Also, I am pretty sure the problem exists also when sending/receiving guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I just sticked to the veth flavour b/c its one (== the hypervisor) network stack to debug and not two (+ the guest one).

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 67 lines · Joseph Gasparakis

Quoted 21 lines · Or Gerlitz

Quoted 18 lines · Joseph Gasparakis

Quoted 8 lines

Quoted 3 lines

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

I have seen this before. Here are my findings:

The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation.

Quoted 8 lines

Quoted 3 lines

I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix.

thanks for the heads up, few quick questions/clafications --

-- you are talking on drops done @ the sender side, correct? Eric was saying we have evidences that the drops happen on the receiver.

I am *guessing* drops on the Rx are due to the drops at the Tx. See my answer to your next question for more info.

Quoted 4 lines

-- without the hack you did, still packets are sent/received, so what makes the stack to drop only some of them?

What I had seen is GSOs getting dropped on the Tx side. Basically the GSOs never made it to the driver, they were broken into non GSO smaller skbs by the stack. I think the stack is not handling well the GSO with the DODGY bit set, and that causes it to maybe partially the packet to be emitted, causing the re-transmits (and maybe the drops on your Rx end)? Of course all this is speculation, the fact that I know is that as soon as I was forcing the gso type I saw offloaded VXLAN encapsulated traffic at decent speeds.

Quoted 1 line

-- why packets coming from veth would have the SKB_GSO_DODGY bit set?

That is something I would love to know too. I am guessing this is a way for the VM to say it is a non-trusted packet? And maybe all this can be fixed by maybe setting something on the VM through a userspace tool that will stop the veth to set the DODGY bit?

Quoted 4 lines

-- so where is now (say net.git or 3.12.x) this one line you commented out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c explicit setting of SKB_GSO_DODGY

I did not commit it, as this was just a workaround to prove to myself that the problem I was seing was due to the gso_type, and it would actually just hide the problem and not give a proper solution to it.

Quoted 1 line

Quoted 4 lines

Quoted 4 lines

Also, I am pretty sure the problem exists also when sending/receiving guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I just sticked to the veth flavour b/c its one (== the hypervisor) network stack to debug and not two (+ the guest one).

understood, can you point the line/area you hacked, I'd like to try it too and see the impact

Quoted 6 lines

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I was printing the gso_type in vxlan_xmit_skb(), right before iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the gso_type was different when a VM was involved and when it was not (although I was transmitting exactly the same packet), and then I replaced my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had for non-VM skb> and it all worked.

Then I looked into what was different between the two gso_types and the only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. I am sure I could have been more delicate with the aproach, but hey, it worked for me.

Quoted 8 lines · Joseph Gasparakis

I was printing the gso_type in vxlan_xmit_skb(), right before iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the gso_type was different when a VM was involved and when it was not (although I was transmitting exactly the same packet), and then I replaced my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had for non-VM skb> and it all worked.

Quoted 4 lines

Then I looked into what was different between the two gso_types and the only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. I am sure I could have been more delicate with the aproach, but hey, it worked for me.

hmm. dodgy should be a normal path from vm.
kvm suppose to negotiate vnet_hdr for tap/macvtap and corresponding driver will be remapping virtio_net_gso* flags into skb_gso_* flags plus gso_dodgy.

Then would the fix be as simple as just unsetting the bit in vxlan? Because I am guessing that there is a bug somewhere where the combination of GSO_UDP_TUNNEL and GSO_DODGY. That would solve my problem, hopefully Or's too if it is the same.

Quoted 69 lines · Or Gerlitz

Quoted 67 lines · Joseph Gasparakis

Quoted 21 lines · Or Gerlitz

Quoted 18 lines · Joseph Gasparakis

Quoted 8 lines

Quoted 3 lines

lack of GRO : receiver seems to not be able to receive as fast as you want.

Quoted 1 line

     TCPOFOQueue: 3167879

So many packets are received out of order (because of losses)

I see that there's no GRO also for the non-veth tests which involve vxlan, and over there the receiving side is capable to consume the packets, do you have rough explaination why adding veth to the chain is such game changer which makes things to start falling out?

I have seen this before. Here are my findings:

The gso_type is different if the skb comes from veth or not. From veth, you will see the SKB_GSO_DODGY set. This breaks things as when the skb with DODGY set moves from vxlan to the driver through dev_xmit_hard, the stack drops it silently. I never got the time to find the root cause for this, but I know it causes re-transmissions and big performance degregation.

Quoted 8 lines

Quoted 3 lines

I went as far as just quickly hacking a one liner unsetting the DODGY bit in vxlan.c and that bypassed the issue and recovered the performance problem, but obviously this is not a real fix.

thanks for the heads up, few quick questions/clafications --

-- you are talking on drops done @ the sender side, correct? Eric was saying we have evidences that the drops happen on the receiver.

I am *guessing* drops on the Rx are due to the drops at the Tx. See my answer to your next question for more info.

Quoted 4 lines

-- without the hack you did, still packets are sent/received, so what makes the stack to drop only some of them?

What I had seen is GSOs getting dropped on the Tx side. Basically the GSOs never made it to the driver, they were broken into non GSO smaller skbs by the stack. I think the stack is not handling well the GSO with the DODGY bit set, and that causes it to maybe partially the packet to be emitted, causing the re-transmits (and maybe the drops on your Rx end)? Of course all this is speculation, the fact that I know is that as soon as I was forcing the gso type I saw offloaded VXLAN encapsulated traffic at decent speeds.

Quoted 1 line

-- why packets coming from veth would have the SKB_GSO_DODGY bit set?

That is something I would love to know too. I am guessing this is a way for the VM to say it is a non-trusted packet? And maybe all this can be fixed by maybe setting something on the VM through a userspace tool that will stop the veth to set the DODGY bit?

Quoted 4 lines

-- so where is now (say net.git or 3.12.x) this one line you commented out? I don't see in vxlan.c or in ip_tunnel_core.c / ip_tunnel.c explicit setting of SKB_GSO_DODGY

I did not commit it, as this was just a workaround to prove to myself that the problem I was seing was due to the gso_type, and it would actually just hide the problem and not give a proper solution to it.

Quoted 1 line

Quoted 7 lines

Quoted 4 lines

Quoted 4 lines

Also, I am pretty sure the problem exists also when sending/receiving guest traffic through tap/macvtap <--> vhost/virtio-net and friends, I just sticked to the veth flavour b/c its one (== the hypervisor) network stack to debug and not two (+ the guest one).

understood, can you point the line/area you hacked, I'd like to try it too and see the impact

I was printing the gso_type in vxlan_xmit_skb(), right before iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the gso_type was different when a VM was involved and when it was not (although I was transmitting exactly the same packet), and then I replaced my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had for non-VM skb> and it all worked.

Then I looked into what was different between the two gso_types and the only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. I am sure I could have been more delicate with the aproach, but hey, it worked for me.

I would be curious to see if this is the same issue as mine. It seems like it is.

Quoted 7 lines

Quoted 6 lines

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I was printing the gso_type in vxlan_xmit_skb(), right before iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the gso_type was different when a VM was involved and when it was not (although I was transmitting exactly the same packet), and then I replaced my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had for non-VM skb> and it all worked.

Then I looked into what was different between the two gso_types and the only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. I am sure I could have been more delicate with the aproach, but hey, it worked for me.

I would be curious to see if this is the same issue as mine. It seems like it is.

Quoted 7 lines · Joseph Gasparakis

I was printing the gso_type in vxlan_xmit_skb(), right before iptunnel_xmit() gets called (I was focus UDPv4 encap only). Then I saw the gso_type was different when a VM was involved and when it was not (although I was transmitting exactly the same packet), and then I replaced my printk with something like skb_shinfo(skb)->gso_type = <the gso type I had for non-VM skb> and it all worked.

Quoted 6 lines

Then I looked into what was different between the two gso_types and the only difference was that SKB_GSO_DODGY was set when Tx'ing from the VM. I am sure I could have been more delicate with the aproach, but hey, it worked for me.

I would be curious to see if this is the same issue as mine. It seems like it is.

nope! with the latest net tree, after handle_offloads is called in vxlan_xmit_skb and before iptunnel_xmit is invoked, skb_shinfo(skb)->gso_type is either 0 or 0x201 which is (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and its the same value whether the session runs over veth device or directly over the bridge, where over veth and > 1 stream we see drops, bad perf, etc.

I am very interested in the VM case too, so will check it out and let you know.

Quoted 7 lines · Or Gerlitz

nope! with the latest net tree, after handle_offloads is called in vxlan_xmit_skb and before iptunnel_xmit is invoked, skb_shinfo(skb)->gso_type is either 0 or 0x201 which is (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and its the same value whether the session runs over veth device or directly over the bridge, where over veth and > 1 stream we see drops, bad perf, etc.
I am very interested in the VM case too, so will check it out and let you know.

Ok, I was really hoping that would be the same...

And just for the record,
you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests before it.

Quoted 9 lines · Joseph Gasparakis

Quoted 7 lines · Or Gerlitz

nope! with the latest net tree, after handle_offloads is called in vxlan_xmit_skb and before iptunnel_xmit is invoked, skb_shinfo(skb)->gso_type is either 0 or 0x201 which is (SKB_GSO_UDP_TUNNEL | SKB_GSO_TCPV4) and its the same value whether the session runs over veth device or directly over the bridge, where over veth and > 1 stream we see drops, bad perf, etc.
I am very interested in the VM case too, so will check it out and let you know.

Quoted 1 line

Ok, I was really hoping that would be the same...

So when running traffic from VM I do see SKB_GSO_DODGY bit being set!
my environment for running VMs with sane peformance was screwed a bit, I will bring it up later today and see if unsetting the bit helps.

Quoted 5 lines · Joseph Gasparakis

And just for the record,
you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests before it.

indeed, also, on what kernel did you conducted your tests which you managed to WA the problem with unsetting that bit?

Quoted 1 line · Or Gerlitz

BTW guys, I saw the issues with both bridge/openvswitch configuration
- seems that we might have here somehow large breakage of the system w.r.t vxlan traffic for rates that go over few Gbs -- so would love to get feedback of any kind from the people that were involved with vxlan over the last months/year.

Or.

Quoted 6 lines · Or Gerlitz

Quoted 1 line · Or Gerlitz

BTW guys, I saw the issues with both bridge/openvswitch configuration
- seems that we might have here somehow large breakage of the system w.r.t vxlan traffic for rates that go over few Gbs -- so would love to get feedback of any kind from the people that were involved with vxlan over the last months/year.

I've seen similar problems with vxlan traffic. In our scenario I had two VMs running on the same host and both VMs having the { veth --> bridge --> vlxan --> IP stack --> NIC } chain.

Running iperf on veth showed rate ~6 times slower than direct NIC <-> NIC. With a hack that forces large gso_size in vxlan's handle_offloads, I've got veth performing only slightly slower than NICs ... The explanation I thought of is that performing the split of the packet as late as possible reduces processing overhead and allows more data to be processed.

Quoted 8 lines · Mike Rapoport

Quoted 6 lines · Or Gerlitz

Quoted 1 line · Or Gerlitz

BTW guys, I saw the issues with both bridge/openvswitch configuration
- seems that we might have here somehow large breakage of the system w.r.t vxlan traffic for rates that go over few Gbs -- so would love to get feedback of any kind from the people that were involved with vxlan over the last months/year.

I've seen similar problems with vxlan traffic. In our scenario I had two VMs running on the same host and both VMs having the { veth --> bridge --> vlxan --> IP stack --> NIC } chain.

How the VMs were connected to the veth NICs? what kernel were you using?

Quoted 1 line

Running iperf on veth showed rate ~6 times slower than direct NIC <-> NIC. With a hack that forces large gso_size in vxlan's handle_offloads, I've got veth performing only slightly slower than NICs ... The explanation I thought of is that performing the split of the packet as late as possible reduces processing overhead and allows more data to be processed.

thanks for the tip! few quick clarifications -- so you artificially enlarged the gso_size of the skb? can you provide the line you added here

also, why enlarging the gso size for skb's cause the actual segmentation to come into play lower in the stack?

Or.

It was something *very* hacky:

		if (skb->len < 64000)
			skb_shinfo(skb)->gso_size = skb->len;
		else
			skb_shinfo(skb)->gso_size = 64000;

Quoted 7 lines · Mike Rapoport

It was something *very* hacky:

		if (skb->len < 64000)
			skb_shinfo(skb)->gso_size = skb->len;
		else
			skb_shinfo(skb)->gso_size = 64000;

This sounds like an 16bit overflow somewhere.

This reminds the issue we fix in commit 50bceae9bd356 ("tcp: Reallocate headroom if it would overflow csum_start")

You might try to reduce the 0xFFFF to something smaller.

Quoted 6 lines · Or Gerlitz

Quoted 4 lines · Joseph Gasparakis

Quoted 4 lines

And just for the record,
you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests

Quoted 3 lines

Quoted 1 line

before it.

indeed, also, on what kernel did you conducted your tests which you managed to WA the problem with unsetting that bit?

Hi Joseph,

Really need your response here --

1. on which kernel did you manage to get along fine vxlan performance wise with this hack?

2. did the hack helped for both veth host traffic or only on PV VM traffic?

Currently it doesn't converge with 3.12.x or net.git, with veth/vxlan the DODGE bit isn't set when looking on the skb in the vxlan xmit time, so there's nothing for me to hack there. For VMs without unsetting the bit things don't really work, but unsetting it for itself so far didn't get me far performance wise.

BTW guys, I saw the issues with both bridge/openvswitch configuration - seems that we might have here somehow large breakage of the system w.r.t vxlan traffic for rates that go over few Gbs -- so would love to get feedback of any kind from the people that were involved with vxlan over the last months/year.

Or.

net.git]# grep -rn SKB_GSO_DODGY drivers/net/ net/ipv4 net/core drivers/net/macvtap.c:585: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY;
drivers/net/tun.c:1135: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY;
drivers/net/virtio_net.c:497: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY; drivers/net/xen-netback/netback.c:1146: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY;
drivers/net/xen-netfront.c:823: skb_shinfo(skb)->gso_type |= SKB_GSO_DODGY;
net/ipv4/af_inet.c:1264: SKB_GSO_DODGY |
net/ipv4/tcp_offload.c:56: SKB_GSO_DODGY | net/ipv4/gre_offload.c:40: SKB_GSO_DODGY |
net/ipv4/udp_offload.c:53: if (unlikely(type & ~(SKB_GSO_UDP |
SKB_GSO_DODGY |
net/core/dev.c:2694: if (shinfo->gso_type & SKB_GSO_DODGY)

Quoted 12 lines · Or Gerlitz

Quoted 9 lines · Or Gerlitz

Quoted 5 lines · Joseph Gasparakis

Quoted 4 lines

And just for the record,
you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests

before it.

indeed, also, on what kernel did you conducted your tests which you managed to WA the problem with unsetting that bit?

Quoted 3 lines

Hi Joseph,

Really need your response here --

I'm sorry Or, I managed to miss your original request...

Quoted 2 lines

1. on which kernel did you manage to get along fine vxlan performance wise with this hack?

I was running 3.10.6.

Quoted 1 line

2. did the hack helped for both veth host traffic or only on PV VM traffic?

No, just VM. I haven't tried veth.

If you leave the DODGY bit, does your traffic get droped on Tx, after it leaves vxlan and before it hits your driver, which is what I had seen. Is that right?

If you unset it, do you recover?

What is the output of your ethtool -k on the interface you are transmitting from?

Quoted 14 lines · Joseph Gasparakis

Quoted 12 lines · Or Gerlitz

Quoted 9 lines · Or Gerlitz

Quoted 5 lines · Joseph Gasparakis

Quoted 4 lines

And just for the record,
you are seeing (SKB_UDP_TUNNEL | SKB_GSO_TCPV4) as 0x201 while I was seeing it as 0x81 because commit 61c1db7fae "ipv6: sit: add GSO/TSO support" pushed the SKB_UDP_TUNNEL two bits left, and I had done my tests

before it.

indeed, also, on what kernel did you conducted your tests which you managed to WA the problem with unsetting that bit?

Quoted 5 lines

Quoted 3 lines

Hi Joseph,

Really need your response here --

I'm sorry Or, I managed to miss your original request...

sure.. it happens.

Quoted 4 lines

Quoted 2 lines

1. on which kernel did you manage to get along fine vxlan performance wise with this hack?

I was running 3.10.6.

I see, will try it out, and just for getting closer to your env, what kernel where the guests running? was a bridge / ovs instance involved in the VM PV connectivity?

Quoted 4 lines

Quoted 2 lines

2. did the hack helped for both veth host traffic or only on PV VM traffic?

No, just VM. I haven't tried veth.

I see, earlier I was somehow under the impression you noted the problem for veth too.

Quoted 3 lines

If you leave the DODGY bit, does your traffic get droped on Tx, after it leaves vxlan and before it hits your driver, which is what I had seen. Is that right?

What I saw is that if I leave the DODGY bit set, practically things don't work at all, its not that some packets are dropped, was that what you saw?

Or on your env only **some** or **few** packets were dropped each time but this killed the tcp session performance?

Also, did you hack/modified the VM NIC MTU to take into the account the encapsulation overhead?

Quoted 1 line

If you unset it, do you recover?

let me redo this with your setting and see, please make sure to tell me what kernel the VM was running too (thanks!)

Quoted 2 lines

What is the output of your ethtool -k on the interface you are transmitting from?

will send you tomorrow, but this happens without offloads for encapsulated traffic.

Quoted 3 lines

What I saw is that if I leave the DODGY bit set, practically things don't work at all, its not that some packets are dropped, was that what you saw?

What I saw was gso packets badly segmented, causing many re-transmissions and dropping the performance to a few MB/s.

Quoted 2 lines

Also, did you hack/modified the VM NIC MTU to take into the account the encapsulation overhead?

The virtio interfaces I used had MTU 1500, but the MTU of the physical NIC was increased to 1600.

I have only noticed this with the offloads on. Turning off encapsuation TSO off, would simply make the gso's to get segmented in dev_hard_xmit() as expected.

30Gbs while RX to single VM or even multiple VMs doesn't go beyond

Quoted 3 lines · Or Gerlitz

1. on which kernel did you manage to get along fine vxlan performance wise with this hack?

I was running 3.10.6.

Quoted 2 lines

2. did the hack helped for both veth host traffic or only on PV VM traffic?

No, just VM. I haven't tried veth.

If you leave the DODGY bit, does your traffic get droped on Tx, after it leaves vxlan and before it hits your driver, which is what I had seen. Is that right?

If you unset it, do you recover?

What is the output of your ethtool -k on the interface you are transmitting from?

Quoted 12 lines

Currently it doesn't converge with 3.12.x or net.git, with veth/vxlan the DODGE bit isn't set when looking on the skb in the vxlan xmit time, so there's nothing for me to hack there. For VMs without unsetting the bit things don't really work, but unsetting it for itself so far didn't get me far performance wise.

BTW guys, I saw the issues with both bridge/openvswitch configuration - seems that we might have here somehow large breakage of the system w.r.t vxlan traffic for rates that go over few Gbs -- so would love to get feedback of any kind from the people that were involved with vxlan over the last months/year.

----------------------

ip link add vxlan42 type vxlan id 42 group 239.0.0.42 ttl 10 dev ethN ifconfig vxlan42 192.168.42.144/24 up

brctl addbr br-vx
ip link set br-vx up

ifconfig br-vx 192.168.52.144/24 up
brctl addif br-vx vxlan42

ip link add type veth

brctl addif br-vx veth1

Quoted 6 lines

brctl addbr br-vx
ip link set br-vx up
ifconfig br-vx 192.168.52.144/24 up
brctl addif br-vx vxlan42
ip link add type veth
brctl addif br-vx veth1

Aside question : If I do not have brctl on my host,
how can I use "ip" command to perform this "brctl addif" action ?

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 6 lines · Eric Dumazet

Quoted 6 lines

brctl addbr br-vx
ip link set br-vx up
ifconfig br-vx 192.168.52.144/24 up
brctl addif br-vx vxlan42
ip link add type veth
brctl addif br-vx veth1

Quoted 2 lines

Aside question : If I do not have brctl on my host,
how can I use "ip" command to perform this "brctl addif" action ?

you can get the sources from
git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/bridge-utils.git
or from http://sourceforge.net/projects/bridge/files/bridge/bridge-utils-1.5.tar.gz/download,
should be trivial to configure and build, if I remember correct

you can get the sources from
git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/bridge-utils.git
or from http://sourceforge.net/projects/bridge/files/bridge/bridge-utils-1.5.tar.gz/download,
should be trivial to configure and build, if I remember correct