Skip to contentSkip to conversation

Life sucks with a saturated link.

LARTC Kevin Turner 2 participants 2 messages Apr 6, 2001

lore
Share & tools
lore
#0 lore raw

Ok, this is where I do some thinking out loud. After I think, I'll continue my research. If and when I find the answer, I'll write it up nice and tidy with a mind toward submitting it to the HOWTO. If anyone out there wants to contribute knowledge or wisdom, then things will go that much faster. :)

The situation is vanilla home-LAN-to-Internet, where I have no influence over my ISP's method of traffic control on the interface I connect to. The majority of the traffic on the link is inbound.

The problem is that when the link is saturated with incoming traffic, the scheduling of things does not appear to be even remotely fair. The usual culprit is a single long-lived TCP (HTTP) connection monopolizing all the bandwidth. Subsequent HTTP connections are then *very* slow to establish and proceed, sometimes taking so long that the kernel's connection tracker forgets they exist and discards replies as 'un-related'. Interactive traffic also lags severly.

So the task is to regulate traffic on the link so it gets handled in a responsive and fair manner. Retransmission cost for packets is relatively expensive, so we would like to accomplish this without randomly dropping packets[1]. The task is complicated by the fact that the link is being saturated by incoming traffic -- traffic which we cannot control directly. Our saviour (I hope) is in the fact that it's an established TCP connection saturating the link, which we should be able to control by regulating ACKs.

One way to obtain control over incoming traffic might be to place a virtual entity between the incoming interface and the local network. Now our "real" host (that is, the one which runs the services which generate and request the traffic) gets its traffic not from the uncontrolled modem interface, but from the virtual interface to the virtual host. This virtual interface is under our control, so we may discipline incoming traffic here, and we control the traffic on the outbound modem interface as well.

[ Sanity check: Will our traffic policies for the virtual interface ] [ be reflected back through the modem interface in a useful way? Or ]

[ will excess traffic just end up piling up in the virtual host's    ]
[ outgoing queue, not affecting upstream's behaviour at all?         ] 
[ There's a 'FIXME: explain congestion windows' in the HOWTO,        ]
[ is that the section I'm looking for?  ;)                           ]

(Note: Perhaps by the time I figure out the solution, the generalization provided by this virtual host will not be necessary, but for now it does make things cleaner conceptually. Never mind the fact that I'm not sure how to implement it yet -- I'm confident that there is a way to do it *somewhere* in this heap of routing power and complexity.)

Now, the problem we have here is one I have not seen addressed in the how-to: The outbound queue and the inbound queue are intimately related to each other. If the outbound queue is empty, the inbound queue is free to use as much bandwidth as it likes. However, if the outbound queue contains low-latency traffic, or packets to establish new connections, the inbound queue needs to back off enough to let this traffic out (and to let the low-latency responses back in, though we'll need the ISP's cooperation if we truly want low-latency responses at the head of our incoming queue).

Put another way:

Outbound and inbound traffic are really classes of a common queue[2], and the outbound class has priority.

I say this is a problem, because my current understanding is that queues are a per-device thing. So how do we get these queues on different devices to play together?

That's about as far as my thoughts run for the time being.

Other helpful things to let me know might be:

1) Am I misdiagnosing the problem? Is this not really a job for TC?

2) Is there some flaw in my reasoning, borne from my ignorance of how modems schedule incoming and outgoing traffic?

3) Am I completely on crack?

Thanks for any light you can shed,

- Kevin Turner

Endnotes:

[1] ...without randomly dropping packets:
I think this makes the current ingress policer a poor choice. However, there is a patch, in iptables's patch-o-matic distribution, which adds a DROPPED table to netfilter. Perhaps that could be used to rescue these packets?
(This relates to section 10.5 "FIXME: instead of dropping, can we assign it to a real queue?")
Working on a patch for ingress might be another option.

[2] outbound and inbound...a common queue:
At least, I believe this is true for modem transports. One can easily imagine this might not be true for all transports.

Signature · 6 lines
Kevin Turner <[email protected]> | OpenPGP encryption welcome here
"Uhh, it's called LART-C.  Is this a bad sign?"

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
#1 lore raw

Hi,

Quoted 2 lines

One way to obtain control over incoming traffic might be to place a virtual entity between the incoming interface and the local network.

I think we are many who wants to do that..

Quoted 4 lines

[ Sanity check: Will our traffic policies for the virtual interface ] [ be reflected back through the modem interface in a useful way? Or ]
[ will excess traffic just end up piling up in the virtual host's ]
[ outgoing queue, not affecting upstream's behaviour at all? ]

No. In theory it will be possible to do this, but I don't know if it is possible in Linux.

I have for almost 2 years been running a seperate Linux box which does incomming scheduleing for a dorm with 500 people in this way with big success. That is, a qdisc is setup on the LAN interface of the machine and all traffic from the internet is passing through this interface. Now, the bandwidth for this interface has been setup to restricting internet-to-the-lan traffic to a rate a bit lower that the capacity of the WAN link. It is described in the documentation for the WRR qdsic. I think it will be very relevant to put some of this stuff in the howto.

Quoted 4 lines

Now, the problem we have here is one I have not seen addressed in the how-to: The outbound queue and the inbound queue are intimately related to each other. If the outbound queue is empty, the inbound queue is free to use as much bandwidth as it likes.

Is the an inbound queue?

Quoted 4 lines

However, if the
outbound queue contains low-latency traffic, or packets to establish new connections, the inbound queue needs to back off enough to let this traffic out

I don't understand this. To me inbound and outbound traffic is completly unrelated. We just want to shape in both directions.

Quoted 2 lines

[1] ...without randomly dropping packets:
I think this makes the current ingress policer a poor choice.

Yes, you won't get any benefit from randomly dropping packets - in effect that will just make your links lower.

Quoted 3 lines

[2] outbound and inbound...a common queue:
At least, I believe this is true for modem transports. One can easily imagine this might not be true for all transports.

Hmm.. You can't control when you receive a packet. At least not on ethernet. I don't imagine you can do that on a modem either - it is up to the sender to send.

Christian

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/