Skip to contentSkip to conversation

Weird traffic shaping behavior after kernel upgrade

DISCUSSION Hoggins! 2 participants 6 messages Jan 5, 2014 – Jan 6, 2014

lore
Share & tools
lore

Hoggins!

author
#0 lore raw

Hello list,

This is my first post, so please forgive me if my ways are wrong, but this is an important request for help.

I was running kernel 3.9.10 (Fedora-flavored), and for several reasons, I needed to upgrade to the last version, 3.12.6. Yeah, it was about time.

Anyway, I really don't know how and where to look for an answer on my problem, and since the kernel is the only thing that changed on this machine, I suspect there has to be something with it.

I have the feeling that my traffic shaping is not working exactly as expected anymore. I use it to "regulate" low-throughput (xDSL) connections, and it was working great until now, but since I upgraded, I feel that the experience is not as "fluid" as it used to be.

And I was surprised to see the following output :

---- SNIP ----

$ tc -s class show dev ppp0

class htb 1:11 parent 1:1 prio 0 rate 200000bit ceil 800000bit burst 1492b cburst 1492b
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 0 borrowed: 0 giants: 0
tokens: 932500 ctokens: 233125

class htb 1:1 root rate 800000bit ceil 800000bit burst 1492b cburst 1492b
Sent 293785084 bytes 341228 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 223981 borrowed: 0 giants: 0
tokens: 26875 ctokens: 26875

class htb 1:13 parent 1:1 prio 1 rate 200000bit ceil 200000bit burst 1492b cburst 1492b
Sent 5746052 bytes 25213 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 25213 borrowed: 0 giants: 0
tokens: 832500 ctokens: 832500

class htb 1:12 parent 1:1 leaf 12: prio 2 rate 100000bit ceil 160000bit burst 1492b cburst 1492b
Sent 66792 bytes 1253 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 1253 borrowed: 0 giants: 0
tokens: 1693898 ctokens: 1067648

class htb 1:15 parent 1:1 prio 1 rate 50000bit ceil 50000bit burst 1492b cburst 1492b
Sent 25153220 bytes 32605 pkt (dropped 6, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 32605 borrowed: 0 giants: 0
tokens: -132903 ctokens: -132903

class htb 1:14 parent 1:1 leaf 14: prio 3 rate 80000bit ceil 400000bit burst 1492b cburst 1492b
Sent 262819020 bytes 282157 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 5p requeues 0
lended: 58178 borrowed: 223981 giants: 0
tokens: -1062034 ctokens: -412442

---- SNIP ----

What worries me is that the "rate" values are always zeroed, even when under heavy traffic. It's not the case with the older version of the kernel.

So I was wondering if there was something I missed, and needed to update in my traffic shaping script. Here it is :

---- SNIP ----

${TC} qdisc del dev ${IF} root
${TC} qdisc add dev ${IF} root handle 1: htb default 14 ${TC} class add dev ${IF} parent 1: classid 1:1 htb rate ${MAX}kbps ceil ${MAX}kbps prio 0 mtu ${MTU}

# Vers TheDude
${TC} class add dev ${IF} parent 1:1 classid 1:11 htb rate 200kbit ceil ${MAX}kbps prio 0 mtu ${MTU}

# Paquets ACK
${TC} class add dev ${IF} parent 1:1 classid 1:12 htb rate 100kbit ceil 160kbit prio 2 mtu ${MTU}

# Téléphone
${TC} class add dev ${IF} parent 1:1 classid 1:13 htb rate 200kbit ceil 200kbit prio 1 mtu ${MTU}

# Traffic général
${TC} class add dev ${IF} parent 1:1 classid 1:14 htb rate 80kbit ceil 400kbit prio 3 mtu ${MTU}

# Vers Duplex
${TC} class add dev ${IF} parent 1:1 classid 1:15 htb rate 50kbit ceil 50kbit prio 1 mtu ${MTU}

${TC} qdisc add dev ${IF} parent 1:12 handle 12: sfq perturb 10 ${TC} qdisc add dev ${IF} parent 1:14 handle 14: sfq perturb 10

${TC} filter add dev ${IF} protocol ip parent 1:0 handle 2 fw classid 1:11 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 3 fw classid 1:12 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 4 fw classid 1:13 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 5 fw classid 1:14 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 6 fw classid 1:15

---- SNIP ----

The packet MARKing in iptables is working as expected, as the counters successfully increase over time.

I'm really stuck, and I need your help.

Happy New Year, and thanks in advance !

Hoggins!

#1 lore raw

Hi Hoggins,

The rate estimator was disabled in kernel 3.12. You can re-enable it by doing the following :

echo 1 > /sys/module/sch_htb/parameters/htb_rate_est

I hope that helps.

Cheers,

Remy

Quoted 122 lines · Hoggins!

Hello list,

This is my first post, so please forgive me if my ways are wrong, but this is an important request for help.

I was running kernel 3.9.10 (Fedora-flavored), and for several reasons, I needed to upgrade to the last version, 3.12.6. Yeah, it was about time.

Anyway, I really don't know how and where to look for an answer on my problem, and since the kernel is the only thing that changed on this machine, I suspect there has to be something with it.

I have the feeling that my traffic shaping is not working exactly as expected anymore. I use it to "regulate" low-throughput (xDSL) connections, and it was working great until now, but since I upgraded, I feel that the experience is not as "fluid" as it used to be.

And I was surprised to see the following output :

---- SNIP ----

$ tc -s class show dev ppp0

class htb 1:11 parent 1:1 prio 0 rate 200000bit ceil 800000bit burst 1492b cburst 1492b
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 0 borrowed: 0 giants: 0
tokens: 932500 ctokens: 233125

class htb 1:1 root rate 800000bit ceil 800000bit burst 1492b cburst 1492b
Sent 293785084 bytes 341228 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 223981 borrowed: 0 giants: 0
tokens: 26875 ctokens: 26875

class htb 1:13 parent 1:1 prio 1 rate 200000bit ceil 200000bit burst 1492b cburst 1492b
Sent 5746052 bytes 25213 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 25213 borrowed: 0 giants: 0
tokens: 832500 ctokens: 832500

class htb 1:12 parent 1:1 leaf 12: prio 2 rate 100000bit ceil 160000bit burst 1492b cburst 1492b
Sent 66792 bytes 1253 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 1253 borrowed: 0 giants: 0
tokens: 1693898 ctokens: 1067648

class htb 1:15 parent 1:1 prio 1 rate 50000bit ceil 50000bit burst 1492b cburst 1492b
Sent 25153220 bytes 32605 pkt (dropped 6, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
lended: 32605 borrowed: 0 giants: 0
tokens: -132903 ctokens: -132903

class htb 1:14 parent 1:1 leaf 14: prio 3 rate 80000bit ceil 400000bit burst 1492b cburst 1492b
Sent 262819020 bytes 282157 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 5p requeues 0
lended: 58178 borrowed: 223981 giants: 0
tokens: -1062034 ctokens: -412442

---- SNIP ----

What worries me is that the "rate" values are always zeroed, even when under heavy traffic. It's not the case with the older version of the kernel.

So I was wondering if there was something I missed, and needed to update in my traffic shaping script. Here it is :

---- SNIP ----

${TC} qdisc del dev ${IF} root
${TC} qdisc add dev ${IF} root handle 1: htb default 14 ${TC} class add dev ${IF} parent 1: classid 1:1 htb rate ${MAX}kbps ceil ${MAX}kbps prio 0 mtu ${MTU}

# Vers TheDude
${TC} class add dev ${IF} parent 1:1 classid 1:11 htb rate 200kbit ceil ${MAX}kbps prio 0 mtu ${MTU}

# Paquets ACK
${TC} class add dev ${IF} parent 1:1 classid 1:12 htb rate 100kbit ceil 160kbit prio 2 mtu ${MTU}

# Téléphone
${TC} class add dev ${IF} parent 1:1 classid 1:13 htb rate 200kbit ceil 200kbit prio 1 mtu ${MTU}

# Traffic général
${TC} class add dev ${IF} parent 1:1 classid 1:14 htb rate 80kbit ceil 400kbit prio 3 mtu ${MTU}

# Vers Duplex
${TC} class add dev ${IF} parent 1:1 classid 1:15 htb rate 50kbit ceil 50kbit prio 1 mtu ${MTU}

${TC} qdisc add dev ${IF} parent 1:12 handle 12: sfq perturb 10 ${TC} qdisc add dev ${IF} parent 1:14 handle 14: sfq perturb 10

${TC} filter add dev ${IF} protocol ip parent 1:0 handle 2 fw classid 1:11 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 3 fw classid 1:12 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 4 fw classid 1:13 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 5 fw classid 1:14 ${TC} filter add dev ${IF} protocol ip parent 1:0 handle 6 fw classid 1:15

---- SNIP ----

The packet MARKing in iptables is working as expected, as the counters successfully increase over time.

I'm really stuck, and I need your help.

Happy New Year, and thanks in advance !

Hoggins!

--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#2 lore raw
In reply to Hoggins!

Hello,

Thanks for the help !
Unfortunately, setting this parameter to 1 does not make the rate indicators change from zero. Strange.
The "Sent" counters do change over time, so I guess the traffic is actually shaped, but maybe in a different way that I was used to. There may also be another variable that I don't master, but I cannot see what for the moment.

How can I track any other change that would have happened on the Linux traffic classifier and HTB between these two kernel versions ?

Le 06/01/2014 02:50, Remy Mudingay a écrit :

Quoted 13 lines

Hi Hoggins,

The rate estimator was disabled in kernel 3.12. You can re-enable it by doing the following :

echo 1 > /sys/module/sch_htb/parameters/htb_rate_est

I hope that helps.

Cheers,

Remy

#3 lore raw
In reply to Hoggins!

Hi again,

Disregard my previous email. It seems that you have to reload your htb qdisc, class and filter rules after updating the kernel htb_rate_est parameter.

Quoted 65 lines · Remy Mudingay

Hi again,

Disregard my previous email. It seems that you have to reload your htb qdisc, class and filter rules after updating the kernel htb_rate_est parameter.

Remy

Quoted 53 lines · Remy Mudingay

Hi Hoggins,

Maybe it's a bug but try setting it a couple of times. Do something like

For i in 1 2 3 4; do
echo 1 > /sys/module/sch_htb/parameters/htb_rate_est done

I've just tested it in a 3.12.1 kernel.

Ref: http://patchwork.ozlabs.org/patch/249551/

Let us know how it goes.

Remy

Sent from my Phone

Quoted 34 lines · Hoggins!

Hello,

Thanks for the help !
Unfortunately, setting this parameter to 1 does not make the rate indicators change from zero. Strange.
The "Sent" counters do change over time, so I guess the traffic is actually shaped, but maybe in a different way that I was used to. There may also be another variable that I don't master, but I cannot see what for the moment.

How can I track any other change that would have happened on the Linux traffic classifier and HTB between these two kernel versions ?

Le 06/01/2014 02:50, Remy Mudingay a écrit :

Quoted 5 lines

Hi Hoggins,

The rate estimator was disabled in kernel 3.12. You can re-enable it by doing the following :

Quoted 8 lines

echo 1 > /sys/module/sch_htb/parameters/htb_rate_est

I hope that helps.

Cheers,

Remy

--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#4 lore raw
In reply to Hoggins!

Hello Dave,

I'll have to try that.
But for the moment, I found a "solution", expecting to use something "cleaner" : I downgraded to kernel 3.10.25, and everything is working as usual.
The machine is a production server for a little student radio in France, so I cannot test a lot without shutting down the entire station.

The downgrade did the trick, so the problem is kinda "solved" for me (although having the latest kernel would have been a better solution for me I guess). Anyway, I'm going to tune these policies a bit, as you are suggesting.

As for the reason I'm not running sfq on all qdiscs, well... as you understand, with my very poor knowledge, I've been working on this little script for a long time to have an acceptable level of service. I had finally found the "perfect" match for the desired performances, and the results of my "researches" showed that I had to use sfq only on the qdiscs that did not require a "guaranteed" (although this is not entirely correct, as we are working on a ADSL connection) throughput :
the classes having sfq are used for TCP operations such as web surfing, and the returning of ACK packets for example. The other ones have fixed limited bandwidth (not borrowable) for UDP (RTP) streams directly to broadcast servers.

Thanks for your help, I'll get back on this list soon.

Hoggins!

Quoted 10 lines · Dave Taht

Is there some reason you aren't running sfq on all qdiscs? That could be a cause.

and:

try

fq_codel target 20ms quantum 300 limit 600

instead of sfq.

author
#5 lore raw
In reply to Hoggins!

Yep. That did the trick for a correct display of the rate stats. But as you see in my previous e-mail, I chose another way to "make things work".

Thanks a lot for your help !

Le 06/01/2014 12:18, Remy Mudingay a écrit :

Quoted 5 lines

Hi again,

Disregard my previous email. It seems that you have to reload your htb qdisc, class and filter rules after updating the kernel htb_rate_est parameter.