Skip to contentSkip to conversation

HTB not that exact

LARTC Stefan Rompf 6 participants 10 messages Dec 6, 2001 – Dec 8, 2001

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I'm experimenting using the HTB queueing discipline for traffic shaping. However, it is not really exact. Currently, I try this setup:

tc qdisc add dev eth0 root handle 1: htb default 3

tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB tc qdisc add dev eth0 parent 1:1 sfq

tc class add dev eth0 parent 1: classid 1:2 htb rate 3Mbit burst 2kB tc qdisc add dev eth0 parent 1:2 sfq

tc class add dev eth0 parent 1: classid 1:3 htb rate 77Mbit burst 150kB tc qdisc add dev eth0 parent 1:3 sfq

tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 1 fw classid 1:1
tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 2 fw classid 1:2

/usr/local/sbin/iptables -t mangle -F OUTPUT /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9021 -j MARK --set-mark 1
/usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9022:9023 -j MARK --set-mark 2

The network adaptor is connected to a 100MBit switch. When testing with netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec via class 1:1, both measured by one/multiple instances of netio and the rate output of "tc -s class dev eth0". This effect occurs with Linux 2.4.16, kernel either compiled with HZ set to 100 or 1024, and of course independant of the filter type used. During the tests, no packets need to be dropped, htb just delays.

Output when two netio instances are sending to class 1:2 for some time:

root@barkeeper /root/ >tc -s class list dev eth0
[...]

class htb 1:2 root leaf 800e: prio 0 rate 3Mbit ceil 3Mbit burst 2Kb cburst 2Kb
Sent 236734498 bytes 156441 pkts (dropped 0, overlimits 301823) rate 378286bps 250pps backlog 44p
lended: 156397 borrowed: 0 giants: 0 injects: 0
tokens: -3714 ctokens: -3714

[...]

tbf seems not to be able to do exact rate limiting, too. Anything wrong in my configuration, or am I just taking false statistics?

Stefan

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#1 lore raw
Quoted 9 lines · Stefan Rompf

Hi,

I'm experimenting using the HTB queueing discipline for traffic shaping. However, it is not really exact. Currently, I try this setup:

tc qdisc add dev eth0 root handle 1: htb default 3

tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB tc qdisc add dev eth0 parent 1:1 sfq

Raise burst, for 20Mbit it needs to be at least 20kbyte!

Quoted 5 lines

The network adaptor is connected to a 100MBit switch. When testing with netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec via class 1:1, both measured by one/multiple instances of netio and the rate output of "tc -s class dev eth0". This effect occurs with Linux 2.4.16, kernel either compiled with HZ set to 100 or 1024, and of

The rate output of tc -s class is not that reliable.

Quoted 2 lines

tbf seems not to be able to do exact rate limiting, too. Anything wrong in my configuration, or am I just taking false statistics?

Both I think. There are some indications that raising HZ may not work, as it may not raise the resolution of the timers in the Linux Kernel - I am investigating this! So try keeping HZ stable, and raising the size of your bucket.

Regards,

bert

Signature · 8 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
#2 lore raw
In reply to Stefan Rompf
Quoted 24 lines · Stefan Rompf

Hi,

tc qdisc add dev eth0 root handle 1: htb default 3

tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB tc qdisc add dev eth0 parent 1:1 sfq

tc class add dev eth0 parent 1: classid 1:2 htb rate 3Mbit burst 2kB tc qdisc add dev eth0 parent 1:2 sfq

tc class add dev eth0 parent 1: classid 1:3 htb rate 77Mbit burst 150kB tc qdisc add dev eth0 parent 1:3 sfq

tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 1 fw classid 1:1
tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 2 fw classid 1:2

/usr/local/sbin/iptables -t mangle -F OUTPUT /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9021 -j MARK --set-mark 1
/usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9022:9023 -j MARK --set-mark 2

Quoted 7 lines

The network adaptor is connected to a 100MBit switch. When testing with netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec via class 1:1, both measured by one/multiple instances of netio and the rate output of "tc -s class dev eth0". This effect occurs with Linux 2.4.16, kernel either compiled with HZ set to 100 or 1024, and of course independant of the filter type used. During the tests, no packets need to be dropped, htb just delays.

I think your results are quite exact. The difference is because of different units.
370kB = 370 * 8 = 2.96 Mbit
2.4MB = 19.2 Mbit

--
Daniel

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#3 lore raw
In reply to Stefan Rompf

Hi,

Quoted 4 lines

I think your results are quite exact. The difference is because of different units.
370kB = 370 * 8 = 2.96 Mbit
2.4MB = 19.2 Mbit

obviously ;-) When calculating, I did remember that one megabit is not 1000000 bits, but forgot about the byte. Thanks for bringing that into my mind again.

Stefan

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#4 lore raw
In reply to Stefan Rompf
Quoted 3 lines

obviously ;-) When calculating, I did remember that one megabit is not 1000000 bits, but forgot about the byte. Thanks for bringing that into my mind again.

One megabit IS actually 10^6 bit - take a look at:

http://physics.nist.gov/cuu/Units/binary.html

What you are thinking about is one mebibit, which is 2^20 bit, take a look at the boot message of a recent kernel and see those prefix'es used for ide disk cache sice.

- David

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#5 lore raw
In reply to Stefan Rompf
Quoted 1 line · bert hubert

On Thu, Dec 06, 2001 at 08:28:07PM +0100, Stefan Rompf wrote:

...

Quoted 7 lines
Quoted 2 lines

tbf seems not to be able to do exact rate limiting, too. Anything wrong in my configuration, or am I just taking false statistics?

Both I think. There are some indications that raising HZ may not work, as it may not raise the resolution of the timers in the Linux Kernel - I am investigating this! So try keeping HZ stable, and raising the size of your bucket.

would it make any difference if you do...

include/net/pkt_sched.h Sun Nov 19 20:20:34 2000 renamed from include/net/pkt_sched.h.bak Sun Nov 19 20:14:10 2000 +1 -1
@@ -5,7 +5,7 @@
 #define PSCHED_JIFFIES                 2
 #define PSCHED_CPU             3
 
-#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
+#define PSCHED_CLOCK_SOURCE    PSCHED_CPU
 
 #include <linux/config.h>
 #include <linux/pkt_sched.h>

I didn't see PSCHED_CLOCK_SOURCE in the 2.4routing docs... Can you tell more about PSCHED_CLOCK_SOURCE ?

Signature · 7 lines
Safari - [email protected] - PGP key 0x427E7914 - http://www.iki.fi/safari/
  The UNIX Guru's View of Sex: "unzip ; strip ; touch ; finger ; \
  mount ; fsck ; more ; yes ; umount ; sleep"

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
#6 lore raw
In reply to Stefan Rompf

Review

Quoted 9 lines
Quoted 4 lines

Both I think. There are some indications that raising HZ may not work, as it may not raise the resolution of the timers in the Linux Kernel - I am investigating this! So try keeping HZ stable, and raising the size of your bucket.

would it make any difference if you do...

(quoted diff) +1 -1
-#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
+#define PSCHED_CLOCK_SOURCE    PSCHED_CPU

by the way if you have pentium with tsc the patch above is stronly recommended. It allows for sub nanosecond precision...
devik

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#7 lore raw
In reply to Stefan Rompf
Quoted 7 lines · Sami Farin
Quoted 4 lines · Ahu

Both I think. There are some indications that raising HZ may not work, as it may not raise the resolution of the timers in the Linux Kernel - I am investigating this! So try keeping HZ stable, and raising the size of your bucket.

would it make any difference if you do...

Ok - I've since investigated this. Shapers use timers internally to block themselves for set periods of time. These timers appear to have HZ resolution, but a comment in the very good Linux Device Drivers book
http://www.xml.com/ldd/chapter/book/
appeared to indicate that raising HZ would not raise their resolution.

I contacted Jonathan Corbet, one of the authors, and he checked and found that the book was incorrect: raising HZ *does* increase timer resolution.

Quoted 5 lines
(quoted diff) +1 -1
-#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
+#define PSCHED_CLOCK_SOURCE    PSCHED_CPU

I didn't see PSCHED_CLOCK_SOURCE in the 2.4routing docs... Can you tell more about PSCHED_CLOCK_SOURCE ?

As far as I can see, this only increases the *resolution*, not *precision*. In other words, you can't specify shorter delays this way but you can measure a delay that already happened far more precisely.

For example, if you decide to delay for 1 jiffy, 10ms, you may in fact have waited 14ms. PSCHED_JIFFIES will not see this - 1.4=1 jiffy has passed. PSCHED_CPU *will* see it, and let the right amount of packets out.

But I'm hoping that Martin Devera can check this for me, devik?

Regards,

bert

Signature · 8 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
#8 lore raw
In reply to Stefan Rompf
Quoted 7 lines · Martin Devera
Quoted 7 lines
Quoted 4 lines

Both I think. There are some indications that raising HZ may not work, as it may not raise the resolution of the timers in the Linux Kernel - I am investigating this! So try keeping HZ stable, and raising the size of your bucket.

would it make any difference if you do...

Quoted 7 lines

Review

Quoted 2 lines
(quoted diff) +1 -1
-#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
+#define PSCHED_CLOCK_SOURCE    PSCHED_CPU

by the way if you have pentium with tsc the patch above is stronly recommended. It allows for sub nanosecond precision...
devik

Is there a reason not to do this? In which case we should probably submit a patch that makes PSCHED_CPU the default if you have a TSC.

Regards,

bert

Signature · 8 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
#9 lore raw
In reply to Stefan Rompf
Quoted 10 lines
Quoted 7 lines

Review

Quoted 2 lines
(quoted diff) +1 -1
-#define PSCHED_CLOCK_SOURCE    PSCHED_JIFFIES
+#define PSCHED_CLOCK_SOURCE    PSCHED_CPU

by the way if you have pentium with tsc the patch above is stronly recommended. It allows for sub nanosecond precision...
devik

Is there a reason not to do this? In which case we should probably submit a patch that makes PSCHED_CPU the default if you have a TSC.

It would be nice - but when ANK does somethink in some way there is reason usualy.
I've already been thinking about it as

#ifdef CONFIG_X86_TSC
#define PSCHED_CLOCK_SOURCE PSCHED_CPU
....

I can found no reason why don't use TSC. With highly loaded interface where eth interrupts are faster than timer ones it should make flow smoother.
Without TSC all packets within single timer tick (10ms) are treated as havin the same arival time.
CBQ's clock integrator based on HW speed computations overcomes this and it might be reason why ANK left JIFFIES here.

If you think that you have such power to convince people about it then go ahead. :)

devik

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/