Skip to contentSkip to conversation

Constantly varying download rate with a complex xen networking setup, why?

DISCUSSION Raphael Hertzog 3 participants 9 messages Jun 15, 2009 – Jul 22, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

[ Please put me in CC of your answers ]

Hello,

I have a weird problem and I'm not sure if it comes from linux or from Xen. Thus I'd like your opinion and maybe your help so that I can do a proper bugreport where it belongs. I can do supplementary tests on request of course.

I probably should try to reproduce it with kvm and a newer kernel but the hardware where this got tested doesn't have the CPU instructions for virtualization. If you don't have any idea of what can be wrong in this setup, I will try to arrange that test but otherwise I would like your feedback before I do more tests.

(Note the full description below has also been submitted to
http://serverfault.com/questions/22219/)

The network configuration in my Xen setup:
- the dom0 has 3 network cards (eth0, eth1, eth2), 3 brigdes (xenbrE, xenbrI, xenbrD) and each brigde integrates the corresponding network card. Only xenbrD has an IP address configured (192.168.78.2, a private LAN) so that it can discuss with all domU.
- there's a domU that is a firewall/router and it also contains 3 virtual cards (eth0, eth1, eth2). It does masquerading for traffic going out on eth0 (the external interface which is part of xenbrE).

  # grep vif /etc/xen/xm.slis 
  vif = [ 'mac=00:16:3e:14:85:11, bridge=xenbrE', 'mac=00:16:3e:14:85:12, bridge=xenbrI', 'mac=00:16:3e:14:85:13, bridge=xenbrD' ]

My problem is that when I download a big file from the internet by HTTP in the dom0, the download rate is not stable. It goes up progressively and then stalls for a few seconds, and restart again going up progressively (and all this in loop until the download is complete). During the stalls, it looks like all networking is blocked on the machine (noticed on interactive SSH sessions).

    dom0                             │domU
         wget                        │
           ↕                         │
    eth2↔xenbrD(192.168.78.2)↔vif2.2←┼→eth2(192.168.78.1/24)
                                     │   ↕ masquerading
    eth0↔xenbrE↔vif2.0←——————————————┼→eth0(192.168.1.20/24)

↕
internet

If I do the same download but uses a (non-caching) HTTP proxy that runs in the firewall domU, the download rate is stable at its maximum value.

How can I avoid this problem?

I suspect it's a bug in the networking stack but I would like assistance to diagnose it more precisely (and maybe find a work-around).

This is a Debian Etch system with Xen 3.2 and the 2.6.26-xen-686 kernel of Debian Lenny (backports). The bridges are created with /etc/network/interfaces:

auto lo
iface lo inet loopback

auto xenbrE
iface xenbrE inet manual
bridge_ports eth0
bridge_maxwait 0

auto xenbrI
iface xenbrI inet manual
bridge_ports eth1
bridge_maxwait 0

auto xenbrD
iface xenbrD inet static
address 192.168.78.2
netmask 255.255.255.0
gateway 192.168.78.1
bridge_ports eth2
bridge_maxwait 0

The xend configuration is not complicated:

    # grep '^(' /etc/xen/xend-config.sxp 
    (network-script network-dummy)
    (vif-script vif-bridge)
    (dom0-min-mem 150)
    (dom0-cpus 0)
    (vncpasswd '')

And the only routing in dom0 redirects to the domU via xenbrD:

    # route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.78.0    0.0.0.0         255.255.255.0   U     0      0        0 xenbrD
    0.0.0.0         192.168.78.1    0.0.0.0         UG    0      0        0 xenbrD

In the domU, the only iptables configuration done is iptables -t nat -A POSTROUTING -s 192.168.78.0/24 -o eth0 -j MASQUERADE.

Thank you for your help.

Signature · 4 lines
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com
#1 lore raw
Quoted 4 lines · Raphael Hertzog

My problem is that when I download a big file from the internet by HTTP in the dom0, the download rate is not stable. It goes up progressively and then stalls for a few seconds, and restart again going up progressively (and all this in loop until the download is complete).

My guess: TCP sawtooth pattern and the associated loss recovery...? What you describe here seems to match exactly to what is expected to happen because of it (I see this happening quite often actually). What is the size of the bottleneck buffer and base RTTs of the relevant flows (HTTP, ssh)?

Quoted 2 lines

During the stalls, it looks like all networking is blocked on the machine (noticed on interactive SSH sessions).

They too might be recovering from losses if the bottleneck is shared? Please verify "blocking" using tcpdump or so. You may find out e.g. that network is not stalled but that out-of-order arrivals prevent applications from making progress.

If this matches with your problem, deploying AQM at the bottleneck would help.

Signature · 1 line
 i.
#2 lore raw
Quoted 10 lines · Ilpo Järvinen
Quoted 4 lines · Raphael Hertzog

My problem is that when I download a big file from the internet by HTTP in the dom0, the download rate is not stable. It goes up progressively and then stalls for a few seconds, and restart again going up progressively (and all this in loop until the download is complete).

My guess: TCP sawtooth pattern and the associated loss recovery...? What you describe here seems to match exactly to what is expected to happen because of it (I see this happening quite often actually).

Why would this not happen for the same transfer done through a proxy (that runs on the same machine but different domU)?

Quoted 2 lines

What is the size of the bottleneck buffer and base RTTs of the relevant flows (HTTP, ssh)?

How can I collect those information?

Quoted 1 line

Please verify "blocking" using tcpdump or so. You may find out e.g. that

I'll try to do so and follow-up.

Quoted 5 lines

network is not stalled but that out-of-order arrivals prevent applications from making progress.

If this matches with your problem, deploying AQM at the bottleneck would help.

The bottleneck in my test setup is the ADSL connection and I don't control the ADSL router.

Cheers,

Signature · 4 lines
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com
#3 lore raw
Quoted 8 lines · Raphael Hertzog
Quoted 7 lines · Ilpo Järvinen
Quoted 4 lines · Raphael Hertzog

My problem is that when I download a big file from the internet by HTTP in the dom0, the download rate is not stable. It goes up progressively and then stalls for a few seconds, and restart again going up progressively (and all this in loop until the download is complete).

Quoted 6 lines
Quoted 3 lines

My guess: TCP sawtooth pattern and the associated loss recovery...? What you describe here seems to match exactly to what is expected to happen because of it (I see this happening quite often actually).

Why would this not happen for the same transfer done through a proxy (that runs on the same machine but different domU)?

Maybe the proxy interferes there somehow... I don't know enough about the details to say but I suppose the proxy at least breaks your tcp connection to two parts.

It could well be something else. I was just saying that this increasing+stall+repeat pattern itself is quite typical also during normal operation (with short rtts one can't see that though due to humanly limitations in perception :-) and because of the granularity the reporting).

Quoted 4 lines
Quoted 2 lines

What is the size of the bottleneck buffer and base RTTs of the relevant flows (HTTP, ssh)?

How can I collect those information?

Perhaps it's enough to just know if sshs that stall go over the adsl link too or are they local? If they're local it's sort of pointing to something else (of which I probably have no clue).

Quoted 4 lines
Quoted 1 line

Please verify "blocking" using tcpdump or so. You may find out e.g. that

I'll try to do so and follow-up.

Quoted 8 lines
Quoted 5 lines

network is not stalled but that out-of-order arrivals prevent applications from making progress.

If this matches with your problem, deploying AQM at the bottleneck would help.

The bottleneck in my test setup is the ADSL connection and I don't control the ADSL router.

Right, those non-controllable things need a bit more sophisticated control. Lartc has some pre-made script which basically puts an articifial cap on the receiver side slightly below the downlinkg rate to create bottleneck there which in turn allows controlling the queue.

Signature · 1 line
 i.
#4 lore raw
Quoted 3 lines · Ilpo Järvinen

Maybe the proxy interferes there somehow... I don't know enough about the details to say but I suppose the proxy at least breaks your tcp connection to two parts.

Indeed. Is there some processing done in a simple linux bridge where the reapperance of the same TCP packet that has been created and sent on another local interface could create problem?

Quoted 5 lines
Quoted 1 line

How can I collect those information?

Perhaps it's enough to just know if sshs that stall go over the adsl link too or are they local? If they're local it's sort of pointing to something else (of which I probably have no clue).

The ssh connection was local and did not go over the ADSL link.

Cheers,

Signature · 4 lines
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com
#5 lore raw
Quoted 8 lines · Raphael Hertzog
Quoted 3 lines · Ilpo Järvinen

Maybe the proxy interferes there somehow... I don't know enough about the details to say but I suppose the proxy at least breaks your tcp connection to two parts.

Indeed. Is there some processing done in a simple linux bridge where the reapperance of the same TCP packet that has been created and sent on another local interface could create problem?

I thought out had a http proxy in between? I suppose it is certainly doing more than bridging. Anyway, I'll be week away, so no quick responses are to be expected from my part after this mail.

Signature · 1 line
 i.
#6 lore raw
Quoted 7 lines · Ilpo Järvinen
Quoted 5 lines · Raphael Hertzog
Quoted 3 lines · Ilpo Järvinen

Maybe the proxy interferes there somehow... I don't know enough about the details to say but I suppose the proxy at least breaks your tcp connection to two parts.

Quoted 7 lines
Quoted 3 lines

Indeed. Is there some processing done in a simple linux bridge where the reapperance of the same TCP packet that has been created and sent on another local interface could create problem?

I thought out had a http proxy in between? I suppose it is certainly doing more than bridging. Anyway, I'll be week away, so no quick responses are to be expected from my part after this mail.

Well, I have the problem when I don't use the proxy... if I use it, the download is split over two TCP connections and things are fine.

Hence my question was if something could be confused by the fact that the same packet is seen twice on the same machine once (in output) via eth2/xenbrD and once (in forward) via xenbrE (the routing between both bridges is done by the domU independently of the dom0 network config).

Cheers,

Signature · 4 lines
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com
#7 lore raw
Quoted 16 lines · Raphael Hertzog
Quoted 7 lines · Ilpo Järvinen
Quoted 5 lines · Raphael Hertzog
Quoted 3 lines · Ilpo Järvinen

Maybe the proxy interferes there somehow... I don't know enough about the details to say but I suppose the proxy at least breaks your tcp connection to two parts.

Quoted 7 lines
Quoted 3 lines

Indeed. Is there some processing done in a simple linux bridge where the reapperance of the same TCP packet that has been created and sent on another local interface could create problem?

I thought out had a http proxy in between? I suppose it is certainly doing more than bridging. Anyway, I'll be week away, so no quick responses are to be expected from my part after this mail.

Quoted 5 lines

Well, I have the problem when I don't use the proxy... if I use it, the download is split over two TCP connections and things are fine.

Hence my question was if something could be confused by the fact that the
same packet is seen twice on the same machine once (in output) via eth2/xenbrD and once (in forward) via xenbrE (the routing between both bridges is done by the domU independently of the dom0 network config).

Did you ever get tcpdumps btw? Looking into your setup it would probably be useful take it from all the interfaces where the packets are supposed to pass.

Signature · 1 line
 i.
#8 lore raw
Quoted 21 lines · Ilpo Järvinen
Quoted 19 lines · Raphael Hertzog
Quoted 7 lines · Ilpo Järvinen
Quoted 5 lines · Raphael Hertzog
Quoted 3 lines · Ilpo Järvinen

Maybe the proxy interferes there somehow... I don't know enough about the details to say but I suppose the proxy at least breaks your tcp connection to two parts.

Quoted 7 lines
Quoted 3 lines

Indeed. Is there some processing done in a simple linux bridge where the reapperance of the same TCP packet that has been created and sent on another local interface could create problem?

I thought out had a http proxy in between? I suppose it is certainly doing more than bridging. Anyway, I'll be week away, so no quick responses are to be expected from my part after this mail.

Well, I have the problem when I don't use the proxy... if I use it, the download is split over two TCP connections and things are fine.

Quoted 8 lines
Quoted 4 lines

Hence my question was if something could be confused by the fact that the same packet is seen twice on the same machine once (in output) via eth2/xenbrD and once (in forward) via xenbrE (the routing between both bridges is done by the domU independently of the dom0 network config).

Did you ever get tcpdumps btw? Looking into your setup it would probably be useful take it from all the interfaces where the packets are supposed to pass.

I don't know if the following has any applicability to your specific situation, but I thought I'd mention it.

It has to do with what I call the non-transitivity of TCP network performance. If you have something like the following scenario:

	A <----LAN----> B <----WAN----> C
	     low RTT        high RTT

Just because you have good TCP network performance from A to B, and good TCP network performance from B to C, it does not necessarily follow that you will also have good TCP network performance from A to C.

Consider the case where the B<->C path is clean, but there are problems on the A<->B path (causing TCP retransmissions). The low RTT on the A<->B path can mask the problems (basically TCP performs well in spite of the problems because of the low RTT), but a transfer directly from A to C performs poorly because the high RTT results in a very slow ramp up in performance after a loss event.

Again, I have no idea if this applies in any way to your case.

-Bill