Skip to contentSkip to conversation

egress bandwidth not limited / limited extremely

LARTC Roman Skula 3 participants 6 messages Jan 16, 2007 – Jan 17, 2007

lore
Share & tools
lore
#0 lore raw

I'm just-so-fresh to the list, so hello everyone.

I'm having a realy hard time with setting up very simple bandwidth management. What I am trying to do is setup a 10Mbit interface to send at only 1920kbit to most of the network (to make sure it hardly ever tops 2mbit). I did succeed with the ingress traffic... With egress however, I get very odd results at different speeds.
With HTB:
If i try to throttle at 0.5mbit for example, a windows box on 1mbit ADSL line is capable of downloading at ~64k from the box i'm limiting, while a bsd box on a 4mbit DSL line happily downloads at ~200k. Limiting to 2mbit looks like there are no results at all
With TBF:
The dowloads from the limited box are always ~5k, at least to what I've observed with different configurations (even with limits at 2mbit).

I'm doing all of that directly on the Internet with public IPs.

I think I've read all the TLDP and LARTC to no results, so would greatly appreciate any help with strictly cutting the bandwidth ath ~1920kbit. What am I missing?

The scrips I've tried (or tried their variants):

#tc qdisc add dev eth0 root handle 1: htb default 20 #tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit #tc class add dev eth0 parent 1:1 classid 1:10 htb rate 9.5mbit #tc class add dev eth0 parent 1:1 classid 1:20 htb rate 0.5mbit #tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10 #tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10 #tc filter add dev eth0 parent 1: protocol ip u32 match ip dst a.b.c.d/24 flowid 1:10

#tc qdisc add dev eth0 root tbf rate 0.5mbit burst 5kb latency 70ms peakrate 1mbit minburst 1540
(yes that does 0.5mbit limiting, but it would be a start...)

#tc qdisc add dev eth0 root handle 1: htb default 20
#tc class add dev eth0 parent 1: classid 1:1 htb rate 1920kbit #tc class add dev eth0 parent 1:1 classid 1:10 htb rate 128kbit prio 1 #tc class add dev eth0 parent 1:1 classid 1:20 htb rate 1792kbit prio 2 #tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10 #tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10 #tc filter add dev eth0 parent 1:0 protocol ip prio 10 u32 match ip tos 0x10 0xff flowid 1:10
#tc filter add dev eth0 parent 1:0 protocol ip prio 10 u32 match ip protocol 1 0xff flowid 1:10

I use the latest iproute2 from the gentoo portage, marked as stable on amd64:
sys-apps/iproute2-2.6.16.20060323/

I manually setup to NIC at server boot, with:
#!/bin/sh
mii-tool -F 10baseT-FD eth0
(but I had the same problems before I did that when the box was in a private LAN)

The server details are:
(serenity:~) # uname -a89zx
Linux serenity 2.6.18-gentoo-r4 #1 SMP Mon Jan 15 12:33:56 CET 2007 x86_64 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux (this is genkernel generated)

(serenity:~) # grep -i config_hz /etc/kernels/kernel-config-x86_64-2.6.18-gentoo-r4 # CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ%0

(serenity:~) # lspci | grep -i ether
05:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5714 Gigabit Ethernet (rev a3)
05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714 Gigabit Ethernet (rev a3)

(serenity:~) # lsmod

Module                  Size  Used by
sch_sfq                 7424  0
sch_htb                18944  1
act_police              8480  1
cls_u32                 9480  2
sch_ingress             5248  1
sch_tbf                 8320  0
ipt_LOG                 8192  1
xt_limit                4352  1
xt_state                3840  1
ip_conntrack           53092  1 xt_state
xt_tcpudp               4864  5
iptable_filter          4736  1
iptable_mangle          4480  0
ip_tables              22312  2 iptable_filter,iptable_mangle
x_tables               18824  5

ipt_LOG,xt_limit,xt_state,xt_tcpudp,ip_tables

(serenity:~) # iptables -nL
Chain INPUT (policy DROP)

target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:20

flags:0x17/0x02
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:21
flags:0x17/0x02
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
flags:0x17/0x02
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
flags:0x17/0x02
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp
dpts:20000:20100 flags:0x17/0x02
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED

ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           icmp type 8
LOG        all  --  0.0.0.0/0            0.0.0.0/0           limit: avg

3/hour burst 5 LOG flags 0 level 4 prefix `iptables-IN-policy: '

Chain FORWARD (policy DROP)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

(the mangle tables are empty)

Regards,

Signature · 6 lines
Roman Sku³a  (http://roman.skula.com)

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#2 lore raw
In reply to Roman Skula

Andy Furniss napisa³(a):

Quoted 4 lines
Quoted 2 lines · Roman Skula

05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714 Gigabit Ethernet (rev a3)

You may need to turn off segmentation offload with ethtool -k

A huge, wet kiss for you, this turned my sky blue again. :)

Many thanks!

Best of wishes,

Signature · 7 lines
Roman Sku³a  (http://roman.skula.com)


_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#3 lore raw
In reply to Roman Skula

Hi there,

Quoted 4 lines
Quoted 2 lines

05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714 Gigabit Ethernet (rev a3)

You may need to turn off segmentation offload with ethtool -k

That's very interesting. Do you have any more information on this subject? Does tso in general distrupt accuracy of tc ops or is it the case only with Broadcom nics? Any links would be very much appreciated.

Thanks for response in advance!

cheers,
Marek Kierdelewicz
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#4 lore raw
In reply to Roman Skula
Quoted 7 lines · Marek Kierdelewicz

Hi there,

Quoted 2 lines
Quoted 2 lines

05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714 Gigabit Ethernet (rev a3)

Quoted 6 lines
Quoted 1 line

You may need to turn off segmentation offload with ethtool -k

That's very interesting. Do you have any more information on this subject? Does tso in general distrupt accuracy of tc ops or is it the case only with Broadcom nics? Any links would be very much appreciated.

I don't have any gig eth, but it's not just Broadcom.

There have been other posts on here where people have needed to increase
htb's mtu parameter to get htb to handle it - despite the interface
being 1500. For shaping traffic for a slow link that is not going to be nice - you could end up dropping multiple tcp segments at once, and it will hurt jitter. I don't imagine it's right to do it for shaping at gig speed either, maybe the default long queue on eth saves dropping.

I suppose those using gig eth and shaping routed traffic at 1500 MTU will not notice unless they try and shape locally generated aswell.

Andy.

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#5 lore raw

Re: [LARTC] egress bandwidth not limited / limited

In reply to Roman Skula
Quoted 10 lines · Roman Skula

Andy Furniss napisa³(a):

Quoted 6 lines
Quoted 2 lines · Roman Skula

05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714 Gigabit Ethernet (rev a3)

You may need to turn off segmentation offload with ethtool -k

A huge, wet kiss for you, this turned my sky blue again. :)

Lol - I forgot to mention: be carefull about using htb default, your arp/other link layer end up there. If you don't specify a default htb will let them through unshaped, you can make a filter to catch the unclassified IP traffic and send it to the class you want.

Andy.

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc