Skip to contentSkip to conversation

Extremely inaccurate results using either TBF or HTB

LARTC Dan Farino 3 participants 8 messages Nov 12, 2002 – Nov 13, 2002

lore
Share & tools
lore

Dan Farino

author
#0 lore raw

Hi everyone,

I am getting extremely inaccurate results from my setup:

- RedHat 8.0 on a Compaq ProLiant 1600, dual PII/450, Intel Dual 100 NIC

- I've tried both SMP and non-SMP kernels.

- I'm using the updated tc from the HTB home page.

- I'm using the HTB that comes with the RH8 kernel.

Here's what's happening. If I, for instance:

tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8 limit
15Kb

Then my download rate goes at 360,000 bytes/s, which, by my calculations, is 2.77mbit. I am using Win2k perfmon to test the speed. The line is flat and the speed is very consistent (consistently *wrong*, unfortunately...)

Immediately after, I do:

tc qdisc change dev eth2 root tbf rate 1.8mbit buffer 20Kb/8 limit
15Kb

My download speed is then 243,000 bytes/s, or 1.85mbit. Again, very flat, consistent line.

I have played with the parameters on both TBF and HTB in many different configurations. I always seem to have a large jump between 1.8-1.9. I can't find any number between those two that will actually produce a 2mbit limit. I know 1.85 is close to 2.0, but is this the expected margin of error? (Then again, 2.77 isn't close to 1.9 at all, so I hope not!)

Has anyone else experienced anything like this?

Thank you very much for any help you can provide!

Dan Farino

Sr. System Engineer

Stamps.com, Inc.

Santa Monica, CA

#1 lore raw
Quoted 27 lines

I am getting extremely inaccurate results from my setup:

- RedHat 8.0 on a Compaq ProLiant 1600, dual PII/450, Intel Dual 100 NIC
- I've tried both SMP and non-SMP kernels.
- I'm using the updated tc from the HTB home page.
- I'm using the HTB that comes with the RH8 kernel.

Here's what's happening. If I, for instance:

tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8 limit
15Kb

Then my download rate goes at 360,000 bytes/s, which, by my calculations, is 2.77mbit. I am using Win2k perfmon to test the speed. The line is flat and the speed is very consistent (consistently *wrong*, unfortunately...)

Immediately after, I do:

tc qdisc change dev eth2 root tbf rate 1.8mbit buffer 20Kb/8 limit
15Kb

My download speed is then 243,000 bytes/s, or 1.85mbit. Again, very flat, consistent line.

Quoted 8 lines

I have played with the parameters on both TBF and HTB in many different configurations. I always seem to have a large jump between 1.8-1.9. I can't find any number between those two that will actually produce a 2mbit limit. I know 1.85 is close to 2.0, but is this the expected margin of error? (Then again, 2.77 isn't close to 1.9 at all, so I hope not!)

Has anyone else experienced anything like this?

I did some tests with htb and tbf :
http://www.docum.org/stef.coene/qos/tests/tbf/
http://www.docum.org/stef.coene/qos/tests/htb/index.html
And the accuracy was very good.

I did the tests on a debian system with a custom kernel and tc command.

How did you measure the throughput? I use the iptables counters so I know exactly what passes the box. If you download a file from the internet with a ftp client, there can be overhead due to retransmission, control packets, .... So the speed you measure with the ftp client may be lower then the real connection speed.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
author
#2 lore raw
In reply to Dan Farino

Hi Stef,

Thanks for the reply! However, I think the gap is way too large to be explained by overhead, etc.

Here are the results from some other tests. I started with:
tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8 limit 15Kb

Here are the average transfer rates measured over approx 20 seconds (again according to PerfMon) as I change the "1.9mbit" to other values while performing a large download:

Mbit Kb/s
----- ------
2.5 360
2.0 360
1.9 360
1.8 240
1.5 240
1.4 240
1.3 180
1.2 180

The lines really are that flat. There is nothing resembling the graphs on your site. Mine has the low-res, stair-step effect that you can see by graphing the above numbers.

I can't seem to find any "mbit" number that will actually produce a cap at exactly 2mbit.

I will try a both different type NIC and Debian at some point today.

Thanks for your help!

Dan Farino
Sr. System Engineer
Stamps.com, Inc.
Santa Monica, CA

-----Original Message-----
From: Stef Coene [mailto:[email protected]]
Sent: Tuesday, November 12, 2002 8:09 AM
To: Dan Farino; [email protected]
Subject: Re: [LARTC] Extremely inaccurate results using either TBF or HTB

Quoted 4 lines

I am getting extremely inaccurate results from my setup:

- RedHat 8.0 on a Compaq ProLiant 1600, dual PII/450, Intel Dual 100

NIC

Quoted 4 lines

- I've tried both SMP and non-SMP kernels.
- I'm using the updated tc from the HTB home page.
- I'm using the HTB that comes with the RH8 kernel.

Quoted 8 lines

Here's what's happening. If I, for instance:

tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8 limit
15Kb

Then my download rate goes at 360,000 bytes/s, which, by my calculations, is 2.77mbit. I am using Win2k perfmon to test the speed. The line is flat and the speed is very consistent (consistently

*wrong*,

Quoted 5 lines

unfortunately...)

Immediately after, I do:

tc qdisc change dev eth2 root tbf rate 1.8mbit buffer 20Kb/8

limit

Quoted 7 lines

15Kb

My download speed is then 243,000 bytes/s, or 1.85mbit. Again, very flat, consistent line.

I have played with the parameters on both TBF and HTB in many

different

Quoted 4 lines

configurations. I always seem to have a large jump between 1.8-1.9. I can't find any number between those two that will actually produce a 2mbit limit. I know 1.85 is close to 2.0, but is this the expected margin of error? (Then again, 2.77 isn't close to 1.9 at all, so I

hope

Quoted 3 lines

not!)

Has anyone else experienced anything like this?

I did some tests with htb and tbf :
http://www.docum.org/stef.coene/qos/tests/tbf/
http://www.docum.org/stef.coene/qos/tests/htb/index.html
And the accuracy was very good.

I did the tests on a debian system with a custom kernel and tc command.

How did you measure the throughput? I use the iptables counters so I know
exactly what passes the box. If you download a file from the internet with a
ftp client, there can be overhead due to retransmission, control packets,
.... So the speed you measure with the ftp client may be lower then the real
connection speed.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#3 lore raw
In reply to Dan Farino
Quoted 23 lines · Dan Farino

Hi Stef,

Thanks for the reply! However, I think the gap is way too large to be explained by overhead, etc.

Here are the results from some other tests. I started with:
tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8 limit 15Kb

Here are the average transfer rates measured over approx 20 seconds (again according to PerfMon) as I change the "1.9mbit" to other values while performing a large download:

Mbit Kb/s
----- ------
2.5 360
2.0 360
1.9 360
1.8 240
1.5 240
1.4 240
1.3 180
1.2 180

Quoted 8 lines

The lines really are that flat. There is nothing resembling the graphs on your site. Mine has the low-res, stair-step effect that you can see by graphing the above numbers.

I can't seem to find any "mbit" number that will actually produce a cap at exactly 2mbit.

I will try a both different type NIC and Debian at some point today.

What NIC are you using ?

Quoted 1 line

Thanks for your help!

No problem. I can send you my scripts that I use the record the bandwidth usage. Some of them can be found on www.docum.org. Basically, I create a iptables chain for each traffic stream I want to monitor. I record the counters each second and calculates the bandwdith.
I have some scripts that executes a htb script, starts the needed flows, calculates the bandwidth, if the bandwidth is stable : record the bandwidth, restart the proces with a different setting.
After some time, you have it for each rate so you can create a nice graph.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#4 lore raw
In reply to Dan Farino

This worked miracles for me:

change /usr/src/linux/include/net/pkt_sched.h PSCHED_CLOCK_SOURCE to PSCHED_CPU if you have a cpu with timestamp counter (TSC) that will give you Mhz timer granularity.

(don't forget to recompile of course)

Jannes Faber
----- Original Message -----
From: Dan Farino
To: [email protected]
Sent: Tuesday, November 12, 2002 9:57 AM
Subject: [LARTC] Extremely inaccurate results using either TBF or HTB

Hi everyone,

I am getting extremely inaccurate results from my setup:

- RedHat 8.0 on a Compaq ProLiant 1600, dual PII/450, Intel Dual 100 NIC

- I've tried both SMP and non-SMP kernels.

- I'm using the updated tc from the HTB home page.

- I'm using the HTB that comes with the RH8 kernel.

Here's what's happening. If I, for instance:

tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8 limit 15Kb

Then my download rate goes at 360,000 bytes/s, which, by my calculations, is 2.77mbit. I am using Win2k perfmon to test the speed. The line is flat and the speed is very consistent (consistently *wrong*, unfortunately.)

Immediately after, I do:

tc qdisc change dev eth2 root tbf rate 1.8mbit buffer 20Kb/8 limit 15Kb

My download speed is then 243,000 bytes/s, or 1.85mbit. Again, very flat, consistent line.

I have played with the parameters on both TBF and HTB in many different configurations. I always seem to have a large jump between 1.8-1.9. I can't find any number between those two that will actually produce a 2mbit limit. I know 1.85 is close to 2.0, but is this the expected margin of error? (Then again, 2.77 isn't close to 1.9 at all, so I hope not!)

Has anyone else experienced anything like this?

Thank you very much for any help you can provide!

Dan Farino

Sr. System Engineer

Stamps.com, Inc.

Santa Monica, CA

author
#5 lore raw
In reply to Dan Farino

Jannes,

That worked great! Everything is dead-on accurate now.

Thank you very much for your help!

Dan Farino

Sr. System Engineer

Stamps.com, Inc.

Santa Monica, CA

-----Original Message-----
From: sufcrusher [mailto:[email protected]]
Sent: Tuesday, November 12, 2002 3:11 PM
To: Dan Farino; [email protected]
Subject: Re: [LARTC] Extremely inaccurate results using either TBF or HTB

This worked miracles for me:

change /usr/src/linux/include/net/pkt_sched.h PSCHED_CLOCK_SOURCE to PSCHED_CPU if you have a cpu with timestamp counter (TSC) that will give you Mhz timer granularity.

(don't forget to recompile of course)

Jannes Faber

	----- Original Message ----- 

	From: Dan Farino <mailto:[email protected]>  

	To: [email protected] 

	Sent: Tuesday, November 12, 2002 9:57 AM

	Subject: [LARTC] Extremely inaccurate results using either TBF

or HTB

Hi everyone,

I am getting extremely inaccurate results from my setup:

- RedHat 8.0 on a Compaq ProLiant 1600, dual PII/450, Intel Dual
100 NIC

- I've tried both SMP and non-SMP kernels.

- I'm using the updated tc from the HTB home page.

- I'm using the HTB that comes with the RH8 kernel.

Here's what's happening. If I, for instance:

tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8
limit 15Kb

Then my download rate goes at 360,000 bytes/s, which, by my
calculations, is 2.77mbit. I am using Win2k perfmon to test the speed. The line is flat and the speed is very consistent (consistently *wrong*, unfortunately...)

Immediately after, I do:

tc qdisc change dev eth2 root tbf rate 1.8mbit buffer
20Kb/8 limit 15Kb

My download speed is then 243,000 bytes/s, or 1.85mbit. Again,
very flat, consistent line.

I have played with the parameters on both TBF and HTB in many
different configurations. I always seem to have a large jump between 1.8-1.9. I can't find any number between those two that will actually produce a 2mbit limit. I know 1.85 is close to 2.0, but is this the expected margin of error? (Then again, 2.77 isn't close to 1.9 at all, so I hope not!)

Has anyone else experienced anything like this?

Thank you very much for any help you can provide!

Dan Farino

Sr. System Engineer

Stamps.com, Inc.

Santa Monica, CA

author
#6 lore raw
In reply to Dan Farino

Stef,

I solved my problem by changing PSCHED_CLOCK_SOURCE to PSCHED_CPU in /usr/src/linux/include/net/pkt_sched.h

My graph is now a nice smooth diagonal line as I increase the bandwidth in increments as I did before.

Thanks for you help on this one. You site has been a tremendous help in getting me up and running.

Take care,

Dan Farino
Sr. System Engineer
Stamps.com, Inc.
Santa Monica, CA

-----Original Message-----
From: Stef Coene [mailto:[email protected]]
Sent: Tuesday, November 12, 2002 10:07 AM
To: Dan Farino; [email protected]
Subject: Re: [LARTC] Extremely inaccurate results using either TBF or HTB

Quoted 23 lines · Dan Farino

Hi Stef,

Thanks for the reply! However, I think the gap is way too large to be explained by overhead, etc.

Here are the results from some other tests. I started with:
tc qdisc add dev eth2 root tbf rate 1.9mbit buffer 20Kb/8 limit 15Kb

Here are the average transfer rates measured over approx 20 seconds (again according to PerfMon) as I change the "1.9mbit" to other values while performing a large download:

Mbit Kb/s
----- ------
2.5 360
2.0 360
1.9 360
1.8 240
1.5 240
1.4 240
1.3 180
1.2 180

Quoted 5 lines

The lines really are that flat. There is nothing resembling the graphs on your site. Mine has the low-res, stair-step effect that you can see by graphing the above numbers.

I can't seem to find any "mbit" number that will actually produce a

cap

Quoted 3 lines

at exactly 2mbit.

I will try a both different type NIC and Debian at some point today.

What NIC are you using ?

Quoted 1 line

Thanks for your help!

No problem. I can send you my scripts that I use the record the bandwidth
usage. Some of them can be found on www.docum.org. Basically, I create a
iptables chain for each traffic stream I want to monitor. I record the counters each second and calculates the bandwdith.
I have some scripts that executes a htb script, starts the needed flows,

calculates the bandwidth, if the bandwidth is stable : record the bandwidth,
restart the proces with a different setting.
After some time, you have it for each rate so you can create a nice graph.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#7 lore raw
In reply to Dan Farino
Quoted 5 lines · sufcrusher

This worked miracles for me:

change /usr/src/linux/include/net/pkt_sched.h PSCHED_CLOCK_SOURCE to PSCHED_CPU if you have a cpu with timestamp counter (TSC) that will give you Mhz timer granularity.

Is this only needed if you have multiple CPU's ? Or is your setup a 1 CPU setup?

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/