Skip to contentSkip to conversation

how to get the latency down on maxed out classes? + extra question

LARTC sufcrusher 2 participants 4 messages Dec 10, 2002 – Dec 11, 2002

lore
Share & tools
lore

sufcrusher

author
#0 lore raw

I've been using HTB for a while now and been tweaking and experimenting with it a lot. I haven't got it perfect yet, but it's a lot better than nothing (both terms of bandwidth efficiency and latency under load).

Basically what I do is create a few classes with different priorities: 0, 1, 2 and 3 (lower value, higher priority). Then I add a filter rule to each class that sends all packets with a specific iptables MARK to that class. Then I create some iptables rules that put the traffic in the correct class, which means that all game traffic (and for testing all ICMP packets) go into priority 0. Ssh, telnet, ACK packets, etc. go into 1. Http, ftp go into 2 and all other packets (kazaa/direct connect and other unknown stuff) is left with priority 3. I limit the total upstream traffic to 100kbit (of the 128kbit my ISP provides me). I'm pretty sure I could go up a little higher when I get the details tweaked right. I set rates and ceilings for each class, although (in theory) the priorities should take care of it and it should be possible to have each class use what they want.

My question here is: why does HTB not permit me to use more than 4 priorities? Some documentation I've seen says I should be able to go as high as priority 7. Maybe my HTB version is too old? Anyway, my linux box is a mess, I'll re-install it sometime soon and start using 2.4.20.

I use iptables rules like these for the marking:
iptables -I PREROUTING -t mangle -i eth2 --jump MARK --set-mark 1 -p ICMP iptables -I PREROUTING -t mangle -i eth0 --jump MARK --set-mark 1 -p ICMP

Another trick I use is reducing the maximum (tcp) packet size, using (which is good for the latency). I've seen other scripts that do the same using different MTU settings and even custom routing tables with maximum MTU size. I haven't really thought about the differences much, but this trick effectively gives the same results:
iptables -I PREROUTING -t mangle -i eth2 --jump TCPMSS --set-mss 700 -p TCP --tcp-flags SYN,RST SYN
iptables -I PREROUTING -t mangle -i eth0 --jump TCPMSS --set-mss 700 -p TCP --tcp-flags SYN,RST SYN

And another important thing to do is change a simple kernel setting, which improves HTB accuracy enormously, which is nice when you're tweaking the
settings: in /usr/src/linux/include/net/pkt_sched.h change PSCHED_CLOCK_SOURCE to PSCHED_CPU if you have a cpu with timestamp counter (TSC) that will give you Mhz timer granularity.

Don't forget to change the interfacenames in the above lines (and the script below).

After having used a pretty complicated script, I recently started rewriting it, so far I made this (no sfq or other stuff, just bare classes).

#!/bin/sh

INET_INTERFACE="eth0"

PRIOS='0 1 2 3'

TC_UPLINK_RATE="100"
TC_RATE[0]P
TC_CEIL[0]0
TC_RATE[1]
TC_CEIL[1]`
TC_RATE[2]
TC_CEIL[2]`
TC_RATE[3]=5
TC_CEIL[3]0

# Packet marks (which prioritiy is linked to which MARK value) MARK[0]="1"
MARK[1]="2"
MARK[2]="3"
MARK[3]="4"

# Executables and stuff
IPTABLES="iptables"
TC="tc"
IP="ip"
LOG="/dev/null"

# Comment the next two lines to really run the tc commands. TC="echo tc"
LOG="/dev/stdout"

# Find last prio, which will be the default
for PRIO in $PRIOS
do
LAST_PRIO=$PRIO
done

# The TC part
$TC qdisc del dev $INET_INTERFACE root

$TC qdisc add dev $INET_INTERFACE root handle 1:0 htb default $[$LAST_PRIO+10]
$TC class add dev $INET_INTERFACE parent 1:0 classid 1:1 htb rate ${TC_UPLINK_RATE}kbit

for PRIO in $PRIOS
do
echo -e "\n***** Prio $PRIO *****" > $LOG
# Add leaf classes for PRIO traffic
$TC class add dev $INET_INTERFACE parent 1:1 \

   classid 1:$[$PRIO+10] htb                 \
   rate ${TC_RATE[$PRIO]}"kbit"              \
   ceil ${TC_CEIL[$PRIO]}"kbit"              \

prio $PRIO

# Now filter PRIO traffic to this leaf

 $TC filter add dev $INET_INTERFACE parent 1:0         \
     protocol ip prio $PRIO handle ${MARK[$PRIO]}      \

fw flowid 1:$[$PRIO+10]
done

Anyway, hope it helps someone.

Jannes Faber

----- Original Message -----
From: "Don Cohen" <[email protected]>
To: <[email protected]>; <[email protected]>
Sent: Sunday, December 08, 2002 6:30 PM
Subject: [LARTC] how to get the latency down on maxed out classes?

Quoted 1 line
Quoted 1 line

lets say I want to limit traffic to/from client to 64kbit. now, client

opens

Quoted 4 lines
Quoted 4 lines

a tcp connection blasting away at full speed.

If client now pings isp, it gets on average around 7 seconds latency. I tried to improve this by using SFQ on the leaf nodes of my HTB

hierarchy,

Quoted 1 line
Quoted 1 line

but that does not really improve the situation, only makes it much

worse.

Quoted 12 lines
Quoted 1 line

with SFQ I get anything between 250ms and 13 seconds latency.

You understand what's going on here?
As I recall, both pfifo and sfq default to queues of length 128 packets. If you fill that with 1500 byte packets you have ~200Kbytes which is about 1.6Mbits. At 64Kbit/sec that would take ~30 sec to send so your latency could be as high as 30 sec.
You can limit this latency by reducing the queue size.

On the other hand, the application that fills the queue evidently doesn't mind large latency. Otherwise it wouldn't fill the queue.

Quoted 9 lines

I think I posted to this list once a description (maybe even the code?) of another way to limit latency - drop packets that have been in the queue for more than a timeout period (I tend to use 3 sec).

SFQ should have the desirable result that one tcp connection won't slow down another one or a ping.

Quoted 2 lines

I then tried fifos. With small packet fifos the packet loss is just to great to be of any use and even then the latency is quite high

(~200ms).

Quoted 4 lines

You consider 200ms high? One max size packet = 1500 bytes = 12Kbit which is about 200ms on a 64Kbit link. You can't expect to do better.

Quoted 1 line

I'm thinking of using RED, but the number of parameters is daunting and

I

Quoted 1 line
Quoted 1 line

have no idea how the HTB rate correlates to packet size and burst rates

for

Quoted 3 lines
Quoted 1 line

red.

RED should be independent of HTB.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw
Quoted 3 lines

TC_UPLINK_RATE="100"
TC_RATE[0]=50
TC_CEIL[0]=110

So this class has ceil = 110kbit, but the parent class has ceil = 100kbit. The parent ceil is not respected so this class can use 110kbit if it wants. I don't think this is what you want.

Quoted 6 lines

TC_RATE[1]=15
TC_CEIL[1]=60
TC_RATE[2]=10
TC_CEIL[2]=60
TC_RATE[3]=5
TC_CEIL[3]=30

There is also a drawbeck if you use prio to improve latency. Devik did some testing. A lower prio is good for delays IF the class with the lower prio never sends more then it's rate (so it's never overlimited). If it do so, other classes are served first and the delays will be very bad. I haven't test it myself, but you can find it in the user guide of htb chapter 7
(http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio).
You can force a class to never exceeds its rate if you use a policer in the filter so packets that exceeds the rate are dropped by the filter. But again, I haven't test it.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
author
#2 lore raw
In reply to sufcrusher
Quoted 5 lines
Quoted 3 lines

TC_UPLINK_RATE="100"
TC_RATE[0]P
TC_CEIL[0]0

So this class has ceil = 110kbit, but the parent class has ceil = 100kbit. The parent ceil is not respected so this class can use 110kbit if it

wants.

Quoted 1 line

I don't think this is what you want.

Oops, you're right. I had been testing it with parent rate at 110, but obviously forgot to put it back down. Also, I presumed the parent ceil would be respected, so thanks for the tip.

Quoted 7 lines
Quoted 6 lines

TC_RATE[1]
TC_CEIL[1]`
TC_RATE[2]
TC_CEIL[2]`
TC_RATE[3]=5
TC_CEIL[3]0

There is also a drawbeck if you use prio to improve latency. Devik did

some

Quoted 6 lines

testing. A lower prio is good for delays IF the class with the lower prio never sends more then it's rate (so it's never overlimited). If it do so, other classes are served first and the delays will be very bad. I haven't test it myself, but you can find it in the user guide of htb chapter 7
(http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio).
You can force a class to never exceeds its rate if you use a policer in

the

Quoted 2 lines

filter so packets that exceeds the rate are dropped by the filter. But again, I haven't test it.

I need the low-latency class for games (CounterStrike), so I can't afford to drop any packets (I guess). On the other hand it typically doesn't use too much bandwidth, so it can lend the rest.

I had read devik's manual (I've read a lot on htb stuff, but I can't claim to remember it all, let alone put it in practise) but I don't see you statement "A lower prio is good for delays IF the class with the lower prio never sends more then it's rate (so it's never overlimited)." in it. In fact, it says that the higher prio class gets "excess bandwidth first", which is what I want.

But you are probably right that my solution is way too crude for practical purposes. Maybe I should only be using prio for the game-packets and put all other classes in the same prio (which is also mentioned at devik's site).

Like I suspected, my current script is very basic, I need quite some more work on it.

Anyway, thanks again Stef, you're doing a great job!

Jannes Faber

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to sufcrusher
Quoted 10 lines

I need the low-latency class for games (CounterStrike), so I can't afford to drop any packets (I guess). On the other hand it typically doesn't use too much bandwidth, so it can lend the rest.

I had read devik's manual (I've read a lot on htb stuff, but I can't claim to remember it all, let alone put it in practise) but I don't see you statement "A lower prio is good for delays IF the class with the lower prio never sends more then it's rate (so it's never overlimited)." in it. In fact, it says that the higher prio class gets "excess bandwidth first", which is what I want.

http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio
At the end of the paragraph, Devik did some tests. He gave www lower prio and yes, the delays are very low. But www is never sending more then the rate. At point 7, he starts sending more data to the www class then the configured rate. And see what happens, the www delay increases a lot (blue line) and the delay of the other classes drops nearly to zero (yellow line).

Quoted 4 lines

But you are probably right that my solution is way too crude for practical purposes. Maybe I should only be using prio for the game-packets and put all other classes in the same prio (which is also mentioned at devik's site).

If it works, why changing ??

Quoted 4 lines

Like I suspected, my current script is very basic, I need quite some more work on it.

Anyway, thanks again Stef, you're doing a great job!

No problem.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/