Skip to contentSkip to conversation

Bandwidth Restrictions in Linux

LARTC Intercom - Roberto Ravetti 6 participants 9 messages Jan 23, 2003 – Jan 27, 2003

lore
Share & tools
lore
#0 lore raw

We are ISP and we give Internet Wireless Outdoor Service . The Base Station works in 802.11b and it is connected with a Linux Mandrake Server that make NAT.

Besides the linux Server limit the bandwidth of each Wireless Client, per IP, using an aplication called Traffic Control with CBQ rules. The bandwidth that we are limit is at 32Kb, 48Kb, 64Kb, 72Kb, 96Kb and 128Kb. We have only 26 clients, son we are limiting only 26 Ip numbers at those bandwith.

My topology wireless is: Linux Server connected directly with an UTP CrossOver Cable to the AP, and from the AP to the wireless client.

The problem I have with CBQ is that for moments I have delay of aprox. 2 or 3 seconds with pings from the Linux Server to some client that are not making any traffic and in that moment if I ping from that client to the AP the delay is only 3ms (that is correct, so the problem is not the wireless link). In a normal condition the ping between the server and the Client would must be around 20 ms, as in several times that I ping to a client that besides it is making traffic and the delay is normal. But sometimes it is not.

For example:
The following is a ping to a client that has limited to 72Kbps.. in this moments he is not making any traffic in the internet, the normals delay time of ping would be around 6 ms... but in stead of look that response time.... The pings varies from 4ms up to 2 seconds... and I promisse you that he is not trafficking anything..

64 bytes from 192.168.8.18: icmp_seq=0 ttl8 time=1.479 sec 64 bytes from 192.168.8.18: icmp_seq=1 ttl8 time=1.179 sec 64 bytes from 192.168.8.18: icmp_seq=2 ttl8 time=1.329 sec 64 bytes from 192.168.8.18: icmp_seq=4 ttl8 time=4.078 msec 64 bytes from 192.168.8.18: icmp_seq=6 ttl8 time.622 msec 64 bytes from 192.168.8.18: icmp_seq=7 ttl8 time.002 msec 64 bytes from 192.168.8.18: icmp_seq=8 ttl8 timea0.729 msec 64 bytes from 192.168.8.18: icmp_seq=9 ttl8 timev9.707 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 timex8.533 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time™2.062 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time™9.109 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.184 sec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.009 sec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.359 sec

So I modify the INI files of the CBQ, and I give her 128Kbps and the pings now are normal.... 6ms, 4ms, 10ms... The following is the file I modify...

DEVICE=eth1,10mbit,1mbit
RATE8Kbit
WEIGHTKbit
PRIO=5
BOUNDED=yes
ISOLATED=yes
RULE2.168.8.11
RULE2.168.8.29
RULE2.168.8.15
RULE2.168.8.14
RULE2.168.8.39
RULE2.168.8.23
RULE2.168.8.18
RULE2.168.8.19

You can see I add ip 192.168.8.18 to that list of IP and to that bandwidth...

May the problem is the variety of bandwith I have or with the lowest bandwith ( <128Kbps) ...?
May the problem is the amount of connections that CBQ have to control...? May the problem is saturate of the CBQ's queue ...?
May the problem is some type of hardware problem in the Server (I have a Pentium III 750Mhz - 128 Mb RAM and 20 Gbyte HHD)...?
Someone told me use HTB or TBF which is better.. can the one be in the certain thing?

I will thank any kind of comments.

Bye
------------------------------------
Roberto Ravetti
Intercom I.S.P
Gerente de Servicios
mailto: [email protected]
Te: (54) 3571 427 777
Rio Tercero - Cordoba - Argentina
------------------------------------

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw

Hello

Quoted 5 lines

We are ISP and we give Internet Wireless Outdoor Service . The Base Station
works in 802.11b and it is connected with a Linux Mandrake Server that make
NAT.

802.11b devices are by design experiencing "hidden node" effect.

Quoted 7 lines

Besides the linux Server limit the bandwidth of each Wireless Client, per IP, using an aplication called Traffic Control with CBQ rules. The bandwidth
that we are limit is at 32Kb, 48Kb, 64Kb, 72Kb, 96Kb and 128Kb. We have only
26 clients, son we are limiting only 26 Ip numbers at those bandwith.

If the wireless clients can't "hear" (by radio transmission) then it is quite possible that they want to transmit at the same time resulting in a radio collision at AP. It is a common effect when large distances and large number of 802.11b nodes with directional antenas are involved.

Quoted 10 lines

For example:
The following is a ping to a client that has limited to 72Kbps.. in this moments he is not making any traffic in the internet, the normals delay time
of ping would be around 6 ms... but in stead of look that response time....
The pings varies from 4ms up to 2 seconds... and I promisse you that he is not trafficking anything..

Quoted 14 lines

64 bytes from 192.168.8.18: icmp_seq=0 ttl8 time=1.479 sec 64 bytes from 192.168.8.18: icmp_seq=1 ttl8 time=1.179 sec 64 bytes from 192.168.8.18: icmp_seq=2 ttl8 time=1.329 sec 64 bytes from 192.168.8.18: icmp_seq=4 ttl8 time=4.078 msec 64 bytes from 192.168.8.18: icmp_seq=6 ttl8 time.622 msec 64 bytes from 192.168.8.18: icmp_seq=7 ttl8 time.002 msec 64 bytes from 192.168.8.18: icmp_seq=8 ttl8 timea0.729 msec 64 bytes from 192.168.8.18: icmp_seq=9 ttl8 timev9.707 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 timex8.533 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time™2.062 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time™9.109 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.184 sec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.009 sec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.359 sec

Great chance that it is "hidden node effect" or some mistake in cbq configuration... try not to shape your clients and check if problem arises (check utilization of interface connecting linux router with AP, too) ... check qdisc and classes stats (tc -s -s -d may be of some help) It is also possible that something close to your AP is transmiting at same frequencies, look for some 2.4GHz devices (wireless cameras, other wireless data transmission systems). Try to change channels ..... use at last 20MHz separation between your AP (so use only channels 1,7,13). Those pings may indicate interference.... Try to change polarization at AP and wireless clients... check your cables and antenas... watch AP's stats, check signal level at client antennas with something decent like Lucen Orinoco and NetStumbler. You can find more details on other wireless mailing lists .

The workarounds are....

Limit upload speeds to minimum (it can minimalize hidden node effect) Use some decent 2.4GHz system ... like TDMA systems using karlnet's software (Avaya COR and RORs, AirBus etc.) . they are not vulnerable to this effect.

Quoted 3 lines

So I modify the INI files of the CBQ, and I give her 128Kbps and the pings now are normal.... 6ms, 4ms, 10ms... The following is the file I modify...

Hm, hidden node effect takes place when wireless utilization is high.... but if you are sure that pings are normal for a long period of time with those settings then there are probably some timing issues with CBQ working at low speeds or you classify packets to wrong classes. As far as I remember CBQ during my tests got stuck sometimes while trying to throttle traffic.Try to send more details and perform more tests.

Robert Kryczalo

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Intercom - Roberto Ravetti

Robert:

I thank your comments... We are WISP from just two years... we use AP Lucent Orinoco.. In the begining we use Karlnet software but now we are in mode 802.11b. That change make us see the way to limit the bandwidth... We are using channel 1 with an omni for an PMtP and channel 6 and 11 with two PtP. And we are in a small city.

Fist we make it with CBQ and after that we change the way to manage CBQ and we did it with CBQ.INIT, that is more easy to manage.

I am subscribe to isp-wireless.com and I have discussing this problem several times.

In moments that the delay of the pings are high, the pings to other clients are normal... besides the pings between wireless client are ALWAYS good and normal.. the only delay is SOMETIMES from the linux to the some clients...

The Signal and Noise are very good (between 15 and 25 of SNR in all clients).. it is not so, I would have problem with all clients and I could tell you that I have never not delay with client limit to 128Kbps, and I have sometimes delay with the one who has less bandwidth.

I think that is not a mistake in the configuration of CBQ becouse we are using the INI Files where I put only some values and all the rules are charge in Memory when I execute CBQ.INIT START.

The AP Lucent Orinoco is configure in 2Mg, I will configure at 1Mg to minimalize the hidden node effect.

I think is a problem of CBQ at that lower bandwith.. someone told me to use HTB or TBF.. Can they be better than CBQ..?

We have besides a CISCO 3620 just configures to try and test limit bandwith over PPPoE.. Somebody tell that it is not the best solution... we will try...

Thanks for any comment

Bye
------------------------------------
Roberto Ravetti
Intercom I.S.P
Gerente de Servicios
mailto: [email protected]
Te: (54) 3571 427 777
Rio Tercero - Cordoba - Argentina
------------------------------------

----- Original Message -----
From: "ISC Robert Kryczalo" <[email protected]>
To: "Intercom - Roberto Ravetti" <[email protected]>; <[email protected]>
Sent: Friday, January 24, 2003 6:08 AM
Subject: RE: [LARTC] Bandwidth Restrictions in Linux

Quoted 9 lines

Hello

Quoted 5 lines

We are ISP and we give Internet Wireless Outdoor Service . The Base Station
works in 802.11b and it is connected with a Linux Mandrake Server that make
NAT.

802.11b devices are by design experiencing "hidden node" effect.

Quoted 2 lines

Besides the linux Server limit the bandwidth of each Wireless Client,

per

Quoted 8 lines
Quoted 5 lines

IP, using an aplication called Traffic Control with CBQ rules. The bandwidth
that we are limit is at 32Kb, 48Kb, 64Kb, 72Kb, 96Kb and 128Kb. We have only
26 clients, son we are limiting only 26 Ip numbers at those bandwith.

If the wireless clients can' "hear" (by radio transmission) then it is quite possible that they want to transmit at the same time resulting in a radio collision at AP. It is a common effect when large distances and

large

Quoted 9 lines

number of 802.11b nodes with directional antenas are involved.

Quoted 8 lines

For example:
The following is a ping to a client that has limited to 72Kbps.. in this moments he is not making any traffic in the internet, the normals delay time
of ping would be around 6 ms... but in stead of look that response time....
The pings varies from 4ms up to 2 seconds... and I promisse you that he

is

Quoted 17 lines
Quoted 16 lines

not trafficking anything..

64 bytes from 192.168.8.18: icmp_seq=0 ttl8 time=1.479 sec 64 bytes from 192.168.8.18: icmp_seq=1 ttl8 time=1.179 sec 64 bytes from 192.168.8.18: icmp_seq=2 ttl8 time=1.329 sec 64 bytes from 192.168.8.18: icmp_seq=4 ttl8 time=4.078 msec 64 bytes from 192.168.8.18: icmp_seq=6 ttl8 time.622 msec 64 bytes from 192.168.8.18: icmp_seq=7 ttl8 time.002 msec 64 bytes from 192.168.8.18: icmp_seq=8 ttl8 timea0.729 msec 64 bytes from 192.168.8.18: icmp_seq=9 ttl8 timev9.707 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 timex8.533 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time™2.062 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time™9.109 msec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.184 sec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.009 sec 64 bytes from 192.168.8.18: icmp_seq ttl8 time=1.359 sec

Quoted 6 lines

Great chance that it is "hidden node effect" or some mistake in cbq configuration... try not to shape your clients and check if problem arises (check utilization of interface connecting linux router with AP, too) ... check qdisc and classes stats (tc -s -s -d may be of some help) It is also possible that something close to your AP is transmiting at same frequencies, look for some 2.4GHz devices (wireless cameras, other

wireless

Quoted 11 lines

data transmission systems). Try to change channels ..... use at last 20MHz separation between your AP (so use only channels 1,7,13). Those pings may indicate interference.... Try to change polarization at AP and wireless clients... check your cables and antenas... watch AP's stats, check signal level at client antennas with something decent like Lucen Orinoco and NetStumbler. You can find more details on other wireless mailing lists .

The workarounds are....

Limit upload speeds to minimum (it can minimalize hidden node effect) Use some decent 2.4GHz system ... like TDMA systems using karlnet's

software

Quoted 1 line

(Avaya COR and RORs, AirBus etc.) . they are not vulnerable to this

effect.

Quoted 4 lines
Quoted 4 lines

So I modify the INI files of the CBQ, and I give her 128Kbps and the

pings

Quoted 1 line
Quoted 1 line

now are normal.... 6ms, 4ms, 10ms... The following is the file I

modify...

Quoted 1 line

Hm, hidden node effect takes place when wireless utilization is high....

but

Quoted 2 lines

if you are sure that pings are normal for a long period of time with those settings then there are probably some timing issues with CBQ working at

low

Quoted 2 lines

speeds or you classify packets to wrong classes. As far as I remember CBQ during my tests got stuck sometimes while trying to throttle traffic.Try

to

Quoted 6 lines

send more details and perform more tests.

Robert Kryczalo

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Intercom - Roberto Ravetti

Hello,

Quoted 6 lines

I thank your comments... We are WISP from just two years... we use AP Lucent
Orinoco.. In the begining we use Karlnet software but now we are in mode 802.11b. That change make us see the way to limit the bandwidth... We are using channel 1 with an omni for an PMtP and channel 6 and 11 with two PtP.

It seems ok.

Quoted 5 lines

In moments that the delay of the pings are high, the pings to other clients
are normal... besides the pings between wireless client are ALWAYS good and
normal.. the only delay is SOMETIMES from the linux to the some clients...

Are you 100% sure about it? If so I would bet CBQ is to blame.

Quoted 5 lines

The Signal and Noise are very good (between 15 and 25 of SNR in all clients).. it is not so, I would have problem with all clients and I could tell you that I have never not delay with client limit to 128Kbps, and I have sometimes delay with the one who has less bandwidth.

I always try to be at around -30dB. Anyway 20 is usually good.

Quoted 7 lines

I think that is not a mistake in the configuration of CBQ becouse we are using the INI Files where I put only some values and all the rules are charge in Memory when I execute CBQ.INIT START.

The AP Lucent Orinoco is configure in 2Mg, I will configure at 1Mg to minimalize the hidden node effect.

Well in my lab in perfect conditions I have got problems with 802.11b devices working at 1Mbps.
It resulted in packet loss. Ant it will increase delays for sure.

Quoted 4 lines

I think is a problem of CBQ at that lower bandwith.. someone told me to use
HTB or TBF.. Can they be better than CBQ..?

Yes they can. I have just switched to HTB from CBQ and I am more than pleased with that. Anyway HTB has several drawbacks and its use is sometime not so intuitive. I am still not able to predict its behaviour precisely in many situations.. fortunately I have my setup in one finger;) Look for the messages with "How HTB treats priorities" in subject line. Anyway clever setup can for sure decrese ping times, increase interactivity and limit p2p software. I am currently able to limit my bandwidth usage by 50% and customers are more pleased than ever. And HTB works well with rates of 15kbits/s and with some tweaking 12kbits/s. Lower rates are possible but can produce side effects. And you have burst and cburst... HTB also shapes better and dont get stuck. Sometimes earlier versions of htb work better.

Quoted 5 lines

We have besides a CISCO 3620 just configures to try and test limit bandwith
over PPPoE.. Somebody tell that it is not the best solution... we will try...

Well, I am going to introduce PPPoE with MAC address pairs on karlnet devices but I have to test it first... It seems to be a reasonable solution, especially when billing and history and more security is required.

Robert Kryczalo

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to Intercom - Roberto Ravetti

Hi,

Quoted 1 line

So, you were limiting bandwidth with CBQ... and now you change to HTB...??

Yes.

Quoted 2 lines

What bandwidth you limited with CBQ and to how many clients...??

In most cases 128 kbit/s and for some special classes of trafic 96,80,64,48,40,39,32,24 kbit/s. MInimum guaranted rates of 15kbit/s. Of course there are higher rates... And we serve much, much more customers and have complex classification scheme based on time, port, packet size and type classification scheme... Anyway it take some time to regenerate HTB tree and to create firewall rules.

Quoted 4 lines

Another test I made was in the moment that I have high delays from the Server to a wireless client, I ping from the client to the AP and to others
client.. the result was NORMAL DELAY IN PING... around 4 ms.

So.. really check your CBQ scripts.

Quoted 2 lines

Roberto.

Robert

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Intercom - Roberto Ravetti

Have you tried htb.init (works like cbq.init using htb)? Available in sourceforge, I think. Google would be your friend.

Mohan
-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of ISC Robert Kryczalo
Sent: 25 January 2003 14:13
To: Intercom - Roberto Ravetti; Lartc@Mailman. Ds9a. Nl
Subject: RE: [LARTC] Bandwidth Restrictions in Linux

Hi,

Quoted 1 line

So, you were limiting bandwidth with CBQ... and now you change to HTB...??

Yes.

Quoted 2 lines

What bandwidth you limited with CBQ and to how many clients...??

In most cases 128 kbit/s and for some special classes of trafic 96,80,64,48,40,39,32,24 kbit/s. MInimum guaranted rates of 15kbit/s. Of course there are higher rates... And we serve much, much more customers and have complex classification scheme based on time, port, packet size and type classification scheme... Anyway it take some time to regenerate HTB tree and to create firewall rules.

Quoted 4 lines

Another test I made was in the moment that I have high delays from the Server to a wireless client, I ping from the client to the AP and to others
client.. the result was NORMAL DELAY IN PING... around 4 ms.

So.. really check your CBQ scripts.

Quoted 2 lines

Roberto.

Robert

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Intercom - Roberto Ravetti
Quoted 8 lines

Hello

Quoted 5 lines

We are ISP and we give Internet Wireless Outdoor Service . The Base Station
works in 802.11b and it is connected with a Linux Mandrake Server that make
NAT.

802.11b devices are by design experiencing "hidden node" effect.

"Hidden node" problem can be avoided by the use of RTS-CTS (request to send, clear to send). RTS-CTS are optional but can be enabled. You can also specify a threshold in the size of the packet under which RTS-CTS will not be used, in which case the hidden node problem might occur for those packets. RTS-CTS will solve the hidden node problem, but will consume BW.

Mathieu.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Intercom - Roberto Ravetti

Hi,

We are using squid for caching, with SCSI disk and 512 MB RAM. The cache_mem setting in squid.conf is 64 MB. After running for several hours total free RAM (as seen by top command) reduces to few kilobytes and server response time increases (CPU idle cycles also go to zero), and we need to reboot the server. Though as percentage of total CPU usage squid is usually taking around 15-30%CPU and percentage of RAM as 12-15%MEM . After rebooting the server would again run fine for several hours say half a day, and then RAM would gradually get consumed again.

Any suggestions are welcome.

Thanks
Tushar

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mathieu Deziel
Sent: Monday, January 27, 2003 7:33 PM
To: [email protected]
Cc: [email protected]; [email protected]
Subject: RE: [LARTC] Bandwidth Restrictions in Linux

Quoted 8 lines

Hello

Quoted 5 lines

We are ISP and we give Internet Wireless Outdoor Service . The Base Station
works in 802.11b and it is connected with a Linux Mandrake Server that make
NAT.

802.11b devices are by design experiencing "hidden node" effect.

"Hidden node" problem can be avoided by the use of RTS-CTS (request to send, clear to
send). RTS-CTS are optional but can be enabled. You can also specify a threshold in
the size of the packet under which RTS-CTS will not be used, in which case the hidden
node problem might occur for those packets. RTS-CTS will solve the hidden node
problem, but will consume BW.

Mathieu.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw
In reply to Intercom - Roberto Ravetti

and why are u ask this here?
try here:

http://www.squid-cache.org/mailing-lists.html

Quoted 46 lines · Tushar Gupta

Hi,

We are using squid for caching, with SCSI disk and 512 MB RAM. The cache_mem setting in squid.conf is 64 MB. After running for several hours total free RAM (as seen by top command) reduces to few kilobytes and server response time increases (CPU idle cycles also go to zero), and we need to reboot the server. Though as percentage of total CPU usage squid is usually taking around 15-30%CPU and percentage of RAM as 12-15%MEM . After rebooting the server would again run fine for several hours say half a day, and then RAM would gradually get consumed again.

Any suggestions are welcome.

Thanks
Tushar

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mathieu Deziel
Sent: Monday, January 27, 2003 7:33 PM
To: [email protected]
Cc: [email protected]; [email protected]
Subject: RE: [LARTC] Bandwidth Restrictions in Linux

Quoted 9 lines

Hello

Quoted 5 lines

We are ISP and we give Internet Wireless Outdoor Service . The Base Station
works in 802.11b and it is connected with a Linux Mandrake Server that make
NAT.

802.11b devices are by design experiencing "hidden node" effect.

"Hidden node" problem can be avoided by the use of RTS-CTS (request to send, clear to
send). RTS-CTS are optional but can be enabled. You can also specify a threshold in
the size of the packet under which RTS-CTS will not be used, in which case the hidden
node problem might occur for those packets. RTS-CTS will solve the hidden node
problem, but will consume BW.

Quoted 3 lines

Mathieu.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/