Skip to contentSkip to conversation

New: Intermittent TCP issues with => 2.6.27

Bugme-new Andrew Morton 7 participants 18 messages Dec 30, 2008 – Jan 9, 2009 1 message not in the archive

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author
#1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 48 lines · On Mon, 29 Dec 2008 18:52:40 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=12327

           Summary: Intermittent TCP issues with => 2.6.27
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.27
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Latest working kernel version: 2.6.26.8
Earliest failing kernel version: 2.6.27
Distribution: Ubuntu
Hardware Environment: amd64, KVM
Software Environment:
Problem Description:

As reported in LP #296767
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/296767) I am experiencing intermittent TCP issues over a PPP ADSL2+ connection with the only change being an upgrade to 2.6.27.

A number of websites, ping, traceroute work correctly but I simply can't connect to several including:

- store.apple.com
- youtube.com
- ANZ internet banking (anz.com.au)
- MSN messenger

I have also tried compiling a generic 2.6.28-rc4 kernel and this still suffers from the same issue, however if I reboot into the previous Ubuntu kernel (2.6.24) or a vanilla 2.6.26 kernel the issue disappears.

Steps to reproduce:

1. Use a KVM guest as a gateway to a PPP internet connection
2. Boot with kernel <= 2.6.26
3. Observe functioning networking
4. Boot into 2.6.27+
5. Observe broken networking

#2 lore raw
Quoted 54 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 48 lines · On Mon, 29 Dec 2008 18:52:40 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=12327

           Summary: Intermittent TCP issues with => 2.6.27
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.27
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Latest working kernel version: 2.6.26.8
Earliest failing kernel version: 2.6.27
Distribution: Ubuntu
Hardware Environment: amd64, KVM
Software Environment:
Problem Description:

As reported in LP #296767
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/296767) I am experiencing intermittent TCP issues over a PPP ADSL2+ connection with the only change being an upgrade to 2.6.27.

A number of websites, ping, traceroute work correctly but I simply can't connect to several including:

- store.apple.com
- youtube.com
- ANZ internet banking (anz.com.au)
- MSN messenger

I have also tried compiling a generic 2.6.28-rc4 kernel and this still suffers from the same issue, however if I reboot into the previous Ubuntu kernel (2.6.24) or a vanilla 2.6.26 kernel the issue disappears.

Steps to reproduce:

1. Use a KVM guest as a gateway to a PPP internet connection
2. Boot with kernel <= 2.6.26
3. Observe functioning networking
4. Boot into 2.6.27+
5. Observe broken networking

Can you please describe the full topology (which is connected to where and using what, and locations of nats, tun/taps, netfilter things, etc.)... There's some contradiction between the ubuntu report description and what you're giving here.

Based on your dumps I find it unlikely that the problem would be in the end host tcp but I'll verify the packets field by field still to be absolutely sure. I'd guess that either the sent packet or reply gets lost somewhere since it never arrives with 2.6.27/2.6.28-rcx.

Signature · 1 line
 i.
#3 lore raw
Quoted 56 lines · Ilpo Järvinen
Quoted 53 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 48 lines · On Mon, 29 Dec 2008 18:52:40 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=12327

           Summary: Intermittent TCP issues with => 2.6.27
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.27
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Latest working kernel version: 2.6.26.8
Earliest failing kernel version: 2.6.27
Distribution: Ubuntu
Hardware Environment: amd64, KVM
Software Environment:
Problem Description:

As reported in LP #296767
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/296767) I am experiencing intermittent TCP issues over a PPP ADSL2+ connection with the only change being an upgrade to 2.6.27.

A number of websites, ping, traceroute work correctly but I simply can't connect to several including:

- store.apple.com
- youtube.com
- ANZ internet banking (anz.com.au)
- MSN messenger

I have also tried compiling a generic 2.6.28-rc4 kernel and this still suffers from the same issue, however if I reboot into the previous Ubuntu kernel (2.6.24) or a vanilla 2.6.26 kernel the issue disappears.

Steps to reproduce:

1. Use a KVM guest as a gateway to a PPP internet connection
2. Boot with kernel <= 2.6.26
3. Observe functioning networking
4. Boot into 2.6.27+
5. Observe broken networking

Quoted 4 lines

Can you please describe the full topology (which is connected to where and using what, and locations of nats, tun/taps, netfilter things, etc.)... There's some contradiction between the ubuntu report description and what you're giving here.

Based on your dumps I find it unlikely that the problem would be in the end host tcp but I'll verify the packets field by field still to be absolutely sure. I'd guess that either the sent packet or reply gets lost somewhere since it never arrives with 2.6.27/2.6.28-rcx.

The gateway machine (whinge) runs as a KVM guest, and shares a physical host with three other guests (one Windows, two Linux). Below are the outputs of bridge topology and VLAN tagging on the physical host.

speedster@whimper:~$ brctl show
bridge name     bridge id               STP enabled     interfaces
dmz             8000.364121864f53       no              vnet0
                                                        vnet4
external                8000.00801e14ffc8       no              vlan50
                                                        vnet3
internal                8000.00801e14ffc8       no              vlan200
                                                        vnet1
                                                        vnet2
                                                        vnet5

-----------------------------------

speedster@whimper:~$ sudo cat /proc/net/vlan/config
VLAN Dev name    | VLAN ID
Name-Type: VLAN_NAME_TYPE_PLUS_VID_NO_PAD
vlan50         | 50  | eth1
vlan200        | 200  | eth1
vlan201        | 201  | eth1
vlan202        | 202  | eth1

-----------------------------------

Whinge (gateway) has three interfaces - one each that connect as taps on the DMZ (vnet4), internal (vnet5) and external (vnet3) bridges. vlan50 is connected to the ADSL2 modem. vlan200 is connected to a physical switch that laptops, access points, other computers connect to.

Ifconfig output from whinge:

speedster@whinge:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:16:3e:24:f7:a1
          inet6 addr: fe80::216:3eff:fe24:f7a1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20302278 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21094867 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13224666138 (13.2 GB)  TX bytes:19329455294 (19.3 GB)

eth0:1 Link encap:Ethernet HWaddr 00:16:3e:24:f7:a1
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth1 Link encap:Ethernet HWaddr 00:16:3e:58:27:c4

          inet addr:203.26.xxx.xxx  Bcast:203.26.xxx.xxx 
Mask:255.255.255.240
          inet6 addr: fe80::216:3eff:fe58:27c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1602092 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1352585 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1078162279 (1.0 GB)  TX bytes:636885252 (636.8 MB)

eth2 Link encap:Ethernet HWaddr 00:16:3e:61:48:91

          inet addr:192.168.200.1  Bcast:192.168.200.255 
Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe61:4891/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:21355818 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20024872 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:18440347046 (18.4 GB)  TX bytes:13623311915 (13.6 GB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:115 errors:0 dropped:0 overruns:0 frame:0 TX packets:115 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:36784 (36.7 KB) TX bytes:36784 (36.7 KB)

ppp0 Link encap:Point-to-Point Protocol

          inet addr:202.72.xxx.xxx  P-t-P:202.72.xxx.xxx 
Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:2053229 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1432508 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:1837869380 (1.8 GB)  TX bytes:637947402 (637.9 MB)

-----------------------------------

eth1 -> ppp0 does not pass through NAT; it is simply routed. eth2 -> ppp0 passes through netfilter MASQUERADE in POSTROUTING

There is a netfilter firewall running on whinge, but I have tried removing all rules, setting policies to ACCEPT and running a simple masquerade rule from eth2 to ppp0

When the issue manifests itself there are connection issues to sites from both physical machines, as well as KVM guests connected to both the DMZ and internal bridges.

I'd like to reiterate that throughout my testing, only whinge had changes made to it (change kernel/reboot). The KVM host and network topology remained unchanged throughout testing.

Please let me know if there is any more information required.

#4 lore raw
Quoted 4 lines · Speedster

The gateway machine (whinge) runs as a KVM guest, and shares a physical host with three other guests (one Windows, two Linux). Below are the outputs of bridge topology and VLAN tagging on the physical host.

What driver are you using on whinge?

When you took the packet dumps, was it on whinge or the physical host? If it was taken on whinge, can you try to take the same dumps on the outgoing physical interface on the host?

Thanks,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#5 lore raw
Quoted 10 lines · Herbert Xu
Quoted 1 line · Speedster

The gateway machine (whinge) runs as a KVM guest, and shares a physical host with three other guests (one Windows, two Linux). Below are the outputs of bridge topology and VLAN tagging on the physical host.

What driver are you using on whinge?

When you took the packet dumps, was it on whinge or the physical host? If it was taken on whinge, can you try to take the same dumps on the outgoing physical interface on the host?

Thanks,

I have tried both virtio and the default (8139too). The original dumps were taken on whinge, I have attached two new dumps showing the functional (2.6.24) and non-functional (2.6.27) states from the host.

Cheers
Dean

#6 lore raw
Quoted 8 lines · Speedster
Quoted 7 lines · Herbert Xu
Quoted 3 lines · Speedster

The gateway machine (whinge) runs as a KVM guest, and shares a physical host with three other guests (one Windows, two Linux). Below are the outputs of bridge topology and VLAN tagging on the physical host.

What driver are you using on whinge?

Quoted 7 lines
Quoted 5 lines

When you took the packet dumps, was it on whinge or the physical host? If it was taken on whinge, can you try to take the same dumps on the outgoing physical interface on the host?

Thanks,

I have tried both virtio and the default (8139too).

And what is the actual eth hw on the host?

Quoted 1 line

The original dumps were taken on whinge, I have attached two new dumps showing the functional (2.6.24) and non-functional (2.6.27) states from the host.

At least to me those didn't reveal anything new :-(.

Signature · 1 line
i.
#7 lore raw
Quoted 5 lines · Ilpo Järvinen
Quoted 2 lines

I have tried both virtio and the default (8139too).

And what is the actual eth hw on the host?

Currently r8169.

speedster@whimper:~$ sudo ethtool -i eth1
driver: r8169
version: 2.3LK-NAPI
firmware-version:
bus-info: 0000:03:01.0

eth0 (currently unused) is a r8168 but uses the same driver.

Quoted 4 lines
Quoted 1 line

The original dumps were taken on whinge, I have attached two new dumps showing the functional (2.6.24) and non-functional (2.6.27) states from the host.

At least to me those didn't reveal anything new :-(.

:-(

Dean

#8 lore raw
In reply to Speedster
Quoted 4 lines · Speedster

I have tried both virtio and the default (8139too). The original dumps were taken on whinge, I have attached two new dumps showing the functional (2.6.24) and non-functional (2.6.27) states from the host.

The MAC on the PPPoE packets appears bogus:

22:07:58.344027 00:16:3e:24:f7:a1 > 00:18:18:54:19:54

Can you also attach a dump for connections that do work under 2.6.27? That should tell us whether this MAC is genuine.

If it is bogus, can you dump on the other interfaces to see who is corrupting it?

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#9 lore raw
Quoted 2 lines · Herbert Xu

Can you also attach a dump for connections that do work under 2.6.27? That should tell us whether this MAC is genuine.

Done. I also realised my wireshark display filter was dst_host rather than simply host - it looks as though replies _do_ come back but for some reason are ignored? My apologies.

The 2.6.27 dump contains a connection to a local file mirror (203.21.20.200) and an attempted connection to store.apple.com (17.149.156.10).

Quoted 3 lines

If it is bogus, can you dump on the other interfaces to see who is corrupting it?

I believe the differing MAC addresses on the PPP remote endpoint are due to my ISP having multiple layer 2 termination routers for load-balancing/redundancy and not a symptom of the issue.

Cheers,
Dean

#10 lore raw
Quoted 4 lines · Speedster

Done. I also realised my wireshark display filter was dst_host rather than simply host - it looks as though replies _do_ come back but for some reason are ignored? My apologies.

Great I think we're getting closer. With your latest dumps it'd appear that there is an odd difference between the replies that get through and the ones that don't. The ones that're somehow dropped have 2 bytes of transport padding at the end. I suspect there's something buggy in your system that's dropping it because of this.

Can you please take a look at /proc/net/snmp on the host and the guest to see if IP InDiscards is non-zero?

Also now that we know the problem is definitely in the host/guest please take another set of dumps on the interfaces leading to and within the guest to see exactly which path of the system is dropping the reply.

Thanks,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#11 lore raw
Quoted 13 lines · Herbert Xu
Quoted 4 lines · Speedster

Done. I also realised my wireshark display filter was dst_host rather than simply host - it looks as though replies _do_ come back but for some reason are ignored? My apologies.

Great I think we're getting closer. With your latest dumps it'd appear that there is an odd difference between the replies that get through and the ones that don't. The ones that're somehow dropped have 2 bytes of transport padding at the end. I suspect there's something buggy in your system that's dropping it because of this.

Quoted 7 lines

Can you please take a look at /proc/net/snmp on the host and the guest to see if IP InDiscards is non-zero?

Also now that we know the problem is definitely in the host/guest please take another set of dumps on the interfaces leading to and within the guest to see exactly which path of the system is dropping the reply.

Another possiblity that comes into my mind is that at TCP side seqnos get messed up because of skb->len getting that padding counted in (end_seq is depending on skb->len) and something don't like encountering such a synack.

Signature · 1 line
 i.
#12 lore raw
In reply to Herbert Xu
Quoted 3 lines · Herbert Xu

Can you please take a look at /proc/net/snmp on the host and the guest to see if IP InDiscards is non-zero?

Both are 0

Quoted 4 lines

Also now that we know the problem is definitely in the host/guest please take another set of dumps on the interfaces leading to and within the guest to see exactly which path of the system is dropping the reply.

Attached (all are the exact same attempted connection), and reveal some interesting information.

The path the inbound traffic should take is
1. vlan50 (host)
2. tap interface vnet3 (host) / eth0 (guest)
3. ppp0 (guest)

It looks as though when it is sent out the tap interface the payload length is incorrect in the PPPoE section of the frame. When it arrives via vlan50 it appears fine. Or at least that's what wireshark highlights for me :)

Dean

#13 lore raw
Quoted 20 lines · On Fri, 09 Jan 2009 00:04:42 +0900 Speedster <[email protected]> wrote:
Quoted 3 lines · Herbert Xu

Can you please take a look at /proc/net/snmp on the host and the guest to see if IP InDiscards is non-zero?

Both are 0

Quoted 4 lines

Also now that we know the problem is definitely in the host/guest please take another set of dumps on the interfaces leading to and within the guest to see exactly which path of the system is dropping the reply.

Attached (all are the exact same attempted connection), and reveal some interesting information.

The path the inbound traffic should take is
1. vlan50 (host)
2. tap interface vnet3 (host) / eth0 (guest)
3. ppp0 (guest)

Quoted 6 lines

It looks as though when it is sent out the tap interface the payload length is incorrect in the PPPoE section of the frame. When it arrives via vlan50 it appears fine. Or at least that's what wireshark highlights for me :)

Dean

Maybe there is an issue that GRO receive isn't handling padding properly?

#14 lore raw
Quoted 23 lines · Stephen Hemminger
Quoted 20 lines · On Fri, 09 Jan 2009 00:04:42 +0900 Speedster <[email protected]> wrote:
Quoted 3 lines · Herbert Xu

Can you please take a look at /proc/net/snmp on the host and the guest to see if IP InDiscards is non-zero?

Both are 0

Quoted 4 lines

Also now that we know the problem is definitely in the host/guest please take another set of dumps on the interfaces leading to and within the guest to see exactly which path of the system is dropping the reply.

Attached (all are the exact same attempted connection), and reveal some interesting information.

The path the inbound traffic should take is
1. vlan50 (host)
2. tap interface vnet3 (host) / eth0 (guest)
3. ppp0 (guest)

Quoted 7 lines
Quoted 4 lines

It looks as though when it is sent out the tap interface the payload length is incorrect in the PPPoE section of the frame. When it arrives via vlan50 it appears fine. Or at least that's what wireshark highlights for me :)

Maybe there is an issue that GRO receive isn't handling padding properly?

Hmm, is gro supposed to have something to do with 2.6.27??? Or are you talking something else than Herbert's recent GRO stuff?

Signature · 1 line
 i.
#15 lore raw
Quoted 30 lines · On Thu, 8 Jan 2009 21:39:30 +0200 (EET) "Ilpo Järvinen"…
Quoted 28 lines · Stephen Hemminger
Quoted 24 lines · On Fri, 09 Jan 2009 00:04:42 +0900 Speedster <[email protected]> wrote:
Quoted 3 lines · Herbert Xu

Can you please take a look at /proc/net/snmp on the host and the guest to see if IP InDiscards is non-zero?

Both are 0

Quoted 4 lines

Also now that we know the problem is definitely in the host/guest please take another set of dumps on the interfaces leading to and within the guest to see exactly which path of the system is dropping the reply.

Attached (all are the exact same attempted connection), and reveal some interesting information.

The path the inbound traffic should take is
1. vlan50 (host)
2. tap interface vnet3 (host) / eth0 (guest)
3. ppp0 (guest)

It looks as though when it is sent out the tap interface the payload length is incorrect in the PPPoE section of the frame. When it arrives via vlan50 it appears fine. Or at least that's what wireshark highlights for me :)

Quoted 5 lines
Quoted 2 lines

Maybe there is an issue that GRO receive isn't handling padding properly?

Hmm, is gro supposed to have something to do with 2.6.27??? Or are you talking something else than Herbert's recent GRO stuff?

Trying to find a common thread of why splice and TCP is having issues in some cases.

#16 lore raw
In reply to Stephen Hemminger
Quoted 3 lines · Stephen Hemminger

Maybe there is an issue that GRO receive isn't handling padding properly?

You are one release early Stephen :) Also GRO can't be enabled without ethtool yet.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#17 lore raw
In reply to Speedster
Quoted 8 lines · Speedster

Attached (all are the exact same attempted connection), and reveal some interesting information.

The path the inbound traffic should take is
1. vlan50 (host)
2. tap interface vnet3 (host) / eth0 (guest)
3. ppp0 (guest)

Quoted 4 lines

It looks as though when it is sent out the tap interface the payload length is incorrect in the PPPoE section of the frame. When it arrives via vlan50 it appears fine. Or at least that's what wireshark highlights for me :)

The length field in the PPPoE header doesn't change.

The packet arrives from the ISP with its PPPoE header length field showing two extra bytes past the IP payload. The bridge or something close to it trims those two bytes from the sk_buff, leaving a packet whose PPPoE header is wrong, but whose upper headers make sense. The PPPoE driver in the virtual machine then drops the packet.

I haven't unscrambled where the drop occurs in the PPPoE driver, but that's presumably what changed in 2.6.27. It seems to me that the drop is proper. What's wrong is trimming the sk_buff to match the IP header while ignoring the L2 header, and that's apparently been that way for a while (if I understand what the reporter is running where).

Or have I screwed up my first posting to netdev?

-- John

#18 lore raw
In reply to Speedster
Quoted 8 lines · Speedster

Attached (all are the exact same attempted connection), and reveal some interesting information.

The path the inbound traffic should take is
1. vlan50 (host)
2. tap interface vnet3 (host) / eth0 (guest)
3. ppp0 (guest)

Quoted 4 lines

It looks as though when it is sent out the tap interface the payload length is incorrect in the PPPoE section of the frame. When it arrives via vlan50 it appears fine. Or at least that's what wireshark highlights for me :)

The length field in the PPPoE header doesn't change.

The packet arrives from the ISP with its PPPoE header length field showing two extra bytes past the IP payload. The bridge or something close to it trims those two bytes from the sk_buff, leaving a packet whose PPPoE header is wrong, but whose upper headers make sense. The PPPoE driver in the virtual machine then drops the packet.

I haven't unscrambled where the drop occurs in the PPPoE driver, but that's presumably what changed in 2.6.27. It seems to me that the drop is proper. What's wrong is trimming the sk_buff to match the IP header while ignoring the L2 header, and that's apparently been that way for a while (if I understand what the reporter is running where).

Or have I screwed up my first posting to netdev?

-- John