Skip to contentSkip to conversation

Shaping traffic on heavily oversubscribed links?

LARTC Justin Schoeman 6 participants 7 messages Nov 25, 2004 – Jan 2, 2005

lore
Share & tools
lore
#0 lore raw

Hi all,

I am having some fun with traffic shaping, and have run into an interesting situation. Here is South Africa, most internet links are heavily oversubscribed, which means that in most cases the local link is _not_ the bottleneck, and shaping on the local link does not help that much...

Does anybody have some tips on shaping such links? How can you get interractive traffic if you don't know how much bandwidth to reserve for it? How can you give fair access to a link if you don't know what the link capacity is?

Are there perhaps some tools to monitor retransmissions to try and determine congestion levels, and from that adjust shaped bandwidth?

Am I perhaps missing something simple in this scenario?

Thanks!
-justin
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw

Quick answer is: you can't. You need to know the bandwidth so that you can control the queue.

You bring up some interesting theoretical ideas about monitoring the overall congestion level and forcing back-offs when you sense that you're reaching the current level, though. That would sort of involve dynamically adjusting the shaping rules. As far as I know there is nothing in existance that does this, but its an interesting idea to think about. If you were going to try something like this I think you'd need some sort of reliable indicator of what the current congestion is like... perhaps some steady ping to use as your "canary in the coal mine". Then set up several shaping scripts that assume different levels of bandwidth, and depending on the current "canary ping", either upgrade or downgrade your assumed bandwidth by calling the appropriate script. So I guess if you wanted to try this, maybe you could set up three scripts, one set to 80% of your bandwidth, one set to 50%, and one set to 20%. Then set up a cron job that checks the "ping" for your high priority traffic, and calls the appropriate script to adjust. I doubt it'll work, but it could be an interesting experiment.

----- Original Message -----
From: "Justin Schoeman" <[email protected]>
To: <[email protected]>
Sent: Thursday, November 25, 2004 3:17 AM
Subject: [LARTC] Shaping traffic on heavily oversubscribed links?

Quoted 16 lines

Hi all,

I am having some fun with traffic shaping, and have run into an interesting situation. Here is South Africa, most internet links are heavily oversubscribed, which means that in most cases the local link is _not_ the bottleneck, and shaping on the local link does not help that much...

Does anybody have some tips on shaping such links? How can you get interractive traffic if you don't know how much bandwidth to reserve for it? How can you give fair access to a link if you don't know what the link capacity is?

Are there perhaps some tools to monitor retransmissions to try and determine congestion levels, and from that adjust shaped bandwidth?

Quoted 8 lines

Am I perhaps missing something simple in this scenario?

Thanks!
-justin
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Justin Schoeman

it's worse than that. we faced the same problems in china. an oversubscribed adsl system. in fact there is more than one problem with the public internet in these scenarios. the first as you have identified is the lack of capacity. then you can't control the downstream routing - eg traffic from china to australia was going to the west coast us, then to florida and finally back to australia via a different gateway in la! some of the routers along the way have large buffers meaning that they can introduce significant delays into the transmission. and finally when all these things overflow you get large packet loss - up to 50% at some times.

the net result is this: high packet loss means forget about real time services such as voip or video conferencing. they are asynchronous services that don't retransmit and are also not very tolerant of packet loss (over about 10%). long ping times means that synchronous services such as email are very slow, coupled with packet loss you might as well be on a dial up.

our solution in a commercial environment was to keep upgrading until we found a service level that works (more or less). we moved from adsl to fibre. found an isp at each end that we could talk to and who could give assistance with routing issues. eg if they have multiple gateways we experimented until we found the most efficient gateway.

we also know of larger comapnies who pay substantial fees to achieve end to end qos.

all of this means there's not a lot you can do at your end with limited choices except to accept that there is no bandwidth at some times of the day and go have a beer instead :)

regards

rick

Quoted 28 lines · Chris Bennett

Quick answer is: you can't. You need to know the bandwidth so that you can control the queue.

You bring up some interesting theoretical ideas about monitoring the overall congestion level and forcing back-offs when you sense that you're reaching the current level, though. That would sort of involve dynamically adjusting the shaping rules. As far as I know there is nothing in existance that does this, but its an interesting idea to think about. If you were going to try something like this I think you'd need some sort of reliable indicator of what the current congestion is like... perhaps some steady ping to use as your "canary in the coal mine". Then set up several shaping scripts that assume different levels of bandwidth, and depending on the current "canary ping", either upgrade or downgrade your assumed bandwidth by calling the appropriate script. So I guess if you wanted to try this, maybe you could set up three scripts, one set to 80% of your bandwidth, one set to 50%, and one set to 20%. Then set up a cron job that checks the "ping" for your high priority traffic, and calls the appropriate script to adjust. I doubt it'll work, but it could be an interesting experiment.

----- Original Message ----- From: "Justin Schoeman" <[email protected]>
To: <[email protected]>
Sent: Thursday, November 25, 2004 3:17 AM
Subject: [LARTC] Shaping traffic on heavily oversubscribed links?

Quoted 8 lines

Hi all,

I am having some fun with traffic shaping, and have run into an interesting situation. Here is South Africa, most internet links are heavily oversubscribed, which means that in most cases the local link is _not_ the bottleneck, and shaping on the local link does not help that much...

Does anybody have some tips on shaping such links? How can you get interractive traffic if you don't know how much bandwidth to reserve for it? How can you give fair access to a link if you don't know what the link capacity is?

Are there perhaps some tools to monitor retransmissions to try and determine congestion levels, and from that adjust shaped bandwidth?

Quoted 8 lines

Am I perhaps missing something simple in this scenario?

Thanks!
-justin
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Quoted 3 lines

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

begin:vcard
fn:Rick Marshall
n:Marshall;Rick
email;internet:[email protected]
tel;cell:+61 411 287 530
x-mozilla-html:TRUE
version:2.1
end:vcard

#3 lore raw
In reply to Justin Schoeman

Thanks everybody for your advice... This is going to be an interesting one to try and solve ;-).

-justin

Quoted 16 lines · Justin Schoeman

Hi all,

I am having some fun with traffic shaping, and have run into an interesting situation. Here is South Africa, most internet links are heavily oversubscribed, which means that in most cases the local link is _not_ the bottleneck, and shaping on the local link does not help that much...

Does anybody have some tips on shaping such links? How can you get interractive traffic if you don't know how much bandwidth to reserve for it? How can you give fair access to a link if you don't know what the link capacity is?

Are there perhaps some tools to monitor retransmissions to try and determine congestion levels, and from that adjust shaped bandwidth?

Quoted 7 lines

Am I perhaps missing something simple in this scenario?

Thanks!
-justin
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to Justin Schoeman
Quoted 2 lines · Chris Bennett

Quick answer is: you can't. You need to know the bandwidth so that you can control the queue.

Indeed. I suffer from the same problem with my PPPoATM link, where my shaping configuration assumes I'm operating over an Ethernet link when it's really encapsulated over an ATM network. Bandwidth available for Ethernet packets varies depending on how many packets are going out over the link and the size of each packet.

Signature · 8 lines
Jason Boxman
Perl Programmer / *NIX Systems Administrator
Shimberg Center for Affordable Housing | University of Florida
http://edseek.com/ - Linux and FOSS stuff

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#5 lore raw
In reply to Justin Schoeman
Quoted 8 lines · Justin Schoeman

Hi all,

I am having some fun with traffic shaping, and have run into an interesting situation. Here is South Africa, most internet links are heavily oversubscribed, which means that in most cases the local link is _not_ the bottleneck, and shaping on the local link does not help that much...

We have the same problem in DSL lines here in Greece.

I have found that while the average efective speed on such lines varies, tha average rate of packets is more or less constant. I have a theory for this. I believe that the routers that forward the traffic on congested lines - on ISPs and on the ATM circuits at the telcoms - don't take the extra time needed to calculate the size of the packets and distribute the traffic on a per packet basis. This leads to a 'fairness' among the end receivers based on packets/sec instead of bandwidth.

To be more specific. In my ADSL line I usually achieve between 20-30 pps (measured with MRTG). With an average packet size of 1500 this is 20-45 kbytes/sec. But packets sizes close to the MTU are found on single ftp/http connections and pretty much nowehere else. Packet sizes of 400 to 500 are more realistic, especially when p2p programs are involved. 20-30 packets give 8-10kbytes/sec. You can expect even less when using voip programs which utilize smaller packets.

If you find that single a FTP session tends to get more bandwidth thatn p2p programs or multiuser traffic then you have a simillar problem to our own. I would suggest that you setup MRTG to monitor packets to research further into this.

Quoted 5 lines

Does anybody have some tips on shaping such links? How can you get interractive traffic if you don't know how much bandwidth to reserve for it? How can you give fair access to a link if you don't know what the link capacity is?

Well, I am working on one. Since I can't shape bandwidth because it flactuates erratically with time and usage I decided to shape packets. I have created a new queueing discipline based on TBF which uses packets instead of bytes for its tokens and I am allocating a constant packet/sec rate on each user of my ADSL line. A better solution would be to create an HTB alike packet-based qdisc for dynamic shaping.

If you find that you have the some problem as me and you want to experiment with a packet-based TBF qdisc I can send you a patch for linux-2.6.8 and iproute2 in this list.

I would like to here your thought on this anyway ...

Dimitris

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Justin Schoeman
Quoted 29 lines · Dimitris Kotsonis
Quoted 8 lines · Justin Schoeman

Hi all,

I am having some fun with traffic shaping, and have run into an interesting situation. Here is South Africa, most internet links are heavily oversubscribed, which means that in most cases the local link is _not_ the bottleneck, and shaping on the local link does not help that much...

We have the same problem in DSL lines here in Greece.

I have found that while the average efective speed on such lines varies, tha average rate of packets is more or less constant. I have a theory for this. I believe that the routers that forward the traffic on congested lines - on ISPs and on the ATM circuits at the telcoms - don't take the extra time needed to calculate the size of the packets and distribute the traffic on a per packet basis. This leads to a 'fairness' among the end receivers based on packets/sec instead of bandwidth.

To be more specific. In my ADSL line I usually achieve between 20-30 pps (measured with MRTG). With an average packet size of 1500 this is 20-45 kbytes/sec. But packets sizes close to the MTU are found on single ftp/http connections and pretty much nowehere else. Packet sizes of 400 to 500 are more realistic, especially when p2p programs are involved. 20-30 packets give 8-10kbytes/sec. You can expect even less when using voip programs which utilize smaller packets.

Quoted 4 lines

If you find that single a FTP session tends to get more bandwidth thatn p2p programs or multiuser traffic then you have a simillar problem to our own. I would suggest that you setup MRTG to monitor packets to research further into this.

It is normal for an FTP download to take over from p2ps the latter are likely to be higher latency, so TCP will let a lower latency FTP grab more bandwidth.

Try shaping with HTB and sfq - It should help.

Quoted 22 lines
Quoted 8 lines

Does anybody have some tips on shaping such links? How can you get interractive traffic if you don't know how much bandwidth to reserve for it? How can you give fair access to a link if you don't know what the link capacity is?

Well, I am working on one. Since I can't shape bandwidth because it flactuates erratically with time and usage I decided to shape packets. I have created a new queueing discipline based on TBF which uses packets instead of bytes for its tokens and I am allocating a constant packet/sec rate on each user of my ADSL line. A better solution would be to create an HTB alike packet-based qdisc for dynamic shaping.

If you find that you have the some problem as me and you want to experiment with a packet-based TBF qdisc I can send you a patch for linux-2.6.8 and iproute2 in this list.

I would like to here your thought on this anyway ...

Quoted 8 lines

Dimitris

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/