Skip to contentSkip to conversation

bond + tc regression ?

DISCUSSION Vladimir Ivashchenko 7 participants 27 messages May 5, 2009 – May 18, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I have a traffic policing setup running on Linux, serving about 800 mbps of traffic. Due to the traffic growth I decided to employ network interface bonding to scale over a single GigE.

The Sun X4150 server has 2xIntel E5450 QuadCore CPUs and a total of four built-in e1000e interfaces, which I grouped into two bond interfaces.

With kernel 2.6.23.1, everything works fine, but the system locked up after a few days.

With kernel 2.6.28.7/2.6.29.1, I get 10-20% packet loss. I get packet loss as soon as I put a classful qdisc, even prio, without even having any classes or filters. TC prio statistics report lots of drops, around 10k per sec. With exactly the same setup on 2.6.23, the number of drops is only 50 per sec.

On both kernels, the system is running with at least 70% idle CPU. The network interrupts are distributed accross the cores.

I thought it was a e1000e driver issue, but tweaking e1000e ring buffers didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs, I tried running on a different server with bnx cards, I tried disabling NO_HZ and HRTICK, but still I have the same problem.

However, if I don't utilize bond, but just apply rules on normal ethX interfaces, there is no packet loss with 2.6.28/29.

So, the problem appears only when I use 2.6.28/29 + bond + classful tc combination.

Any ideas ?

Signature · 5 lines
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211
#1 lore raw

Can you show example of rules you are putting?
Probably i can find mistakes and give correct example and i will explain maybe why it is happened.

Quoted 33 lines · On Tuesday 05 May 2009 18:45:58 Vladimir Ivashchenko wrote:

Hi,

I have a traffic policing setup running on Linux, serving about 800 mbps of traffic. Due to the traffic growth I decided to employ network interface bonding to scale over a single GigE.

The Sun X4150 server has 2xIntel E5450 QuadCore CPUs and a total of four built-in e1000e interfaces, which I grouped into two bond interfaces.

With kernel 2.6.23.1, everything works fine, but the system locked up after a few days.

With kernel 2.6.28.7/2.6.29.1, I get 10-20% packet loss. I get packet loss as soon as I put a classful qdisc, even prio, without even having any classes or filters. TC prio statistics report lots of drops, around 10k per sec. With exactly the same setup on 2.6.23, the number of drops is only 50 per sec.

On both kernels, the system is running with at least 70% idle CPU. The network interrupts are distributed accross the cores.

I thought it was a e1000e driver issue, but tweaking e1000e ring buffers didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs, I tried running on a different server with bnx cards, I tried disabling NO_HZ and HRTICK, but still I have the same problem.

However, if I don't utilize bond, but just apply rules on normal ethX interfaces, there is no packet loss with 2.6.28/29.

So, the problem appears only when I use 2.6.28/29 + bond + classful tc combination.

Any ideas ?

#2 lore raw
In reply to Vladimir Ivashchenko
Quoted 12 lines · Vladimir Ivashchenko

Hi,

I have a traffic policing setup running on Linux, serving about 800 mbps of traffic. Due to the traffic growth I decided to employ network interface bonding to scale over a single GigE.

The Sun X4150 server has 2xIntel E5450 QuadCore CPUs and a total of four built-in e1000e interfaces, which I grouped into two bond interfaces.

With kernel 2.6.23.1, everything works fine, but the system locked up after a few days.

Quoted 8 lines

With kernel 2.6.28.7/2.6.29.1, I get 10-20% packet loss. I get packet loss as soon as I put a classful qdisc, even prio, without even having any classes or filters. TC prio statistics report lots of drops, around 10k per sec. With exactly the same setup on 2.6.23, the number of drops is only 50 per sec.

On both kernels, the system is running with at least 70% idle CPU. The network interrupts are distributed accross the cores.

You should not distribute interrupts, but bound a NIC to one CPU

Quoted 6 lines

I thought it was a e1000e driver issue, but tweaking e1000e ring buffers didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs, I tried running on a different server with bnx cards, I tried disabling NO_HZ and HRTICK, but still I have the same problem.

Quoted 8 lines

However, if I don't utilize bond, but just apply rules on normal ethX interfaces, there is no packet loss with 2.6.28/29.

So, the problem appears only when I use 2.6.28/29 + bond + classful tc combination.

Any ideas ?

Yes, we need much more information :)
Is it a forwarding setup only ?

cat /proc/interrupts
cat /proc/net/bonding/bond0
cat /proc/net/bonding/bond1
tc -s -d qdisc
mpstat -P ALL 10
ifconfig -a

and so on ...

#3 lore raw
Quoted 4 lines
Quoted 2 lines

On both kernels, the system is running with at least 70% idle CPU. The network interrupts are distributed accross the cores.

You should not distribute interrupts, but bound a NIC to one CPU

Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct. The defaults are wrong?

I have tried with IRQs bound to one CPU per NIC. Same result.

Quoted 8 lines
Quoted 8 lines

I thought it was a e1000e driver issue, but tweaking e1000e ring buffers didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs, I tried running on a different server with bnx cards, I tried disabling NO_HZ and HRTICK, but still I have the same problem.

However, if I don't utilize bond, but just apply rules on normal ethX interfaces, there is no packet loss with 2.6.28/29.

Quoted 8 lines
Quoted 5 lines

So, the problem appears only when I use 2.6.28/29 + bond + classful tc combination.

Any ideas ?

Yes, we need much more information :)
Is it a forwarding setup only ?

Yes, the server is doing nothing else but forwarding, no iptables.

Quoted 1 line

cat /proc/interrupts

CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7

  0:        130          0          0          0          0          0          0          0   IO-APIC-edge      timer
  1:          2          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  3:          0          0          0          1          0          1          0          0   IO-APIC-edge
  4:          0          0          1          0          0          0          1          0   IO-APIC-edge
  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          4          0          0          0          0          0          0          0   IO-APIC-edge      i8042
 14:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
 15:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
 17:      30901      31910      31446      30655      31618      30550      31543      30958   IO-APIC-fasteoi   aacraid
 20:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 21:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb5, ahci
 22:     298387     297642     295508     294368     295533     295430     295275     296036   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
 23:      10868      10926      10980      10738      10939      10615      10761      10909   IO-APIC-fasteoi   uhci_hcd:usb3
 57: 1486251823 1486835830 1486677250 1487105983 1488000303 1485941815 1487728317 1486624997   PCI-MSI-edge      eth0
 58: 1510676329 1509708161 1510347202 1509969755 1508599471 1511220118 1509094578 1509727616   PCI-MSI-edge      eth1
 59: 1482578890 1483618556 1482963700 1483164528 1484561615 1482130645 1484116749 1483557717   PCI-MSI-edge      eth2
 60: 1507341647 1506685822 1506862759 1506612818 1505689367 1507559672 1505911622 1506940613   PCI-MSI-edge      eth3
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC: 1020533656 1020535165 1020533613 1020534967 1020535173 1020534409 1020534985 1020534220   Local timer interrupts
RES:      18605      21215      15957      18637      22429      19493      16649      15589   Rescheduling interrupts
CAL:        160        214        186        185        199        205        190        180   Function call interrupts
TLB:     259515     264126     309016     312222     263163     265601     306189     305430   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0
Quoted 1 line

tc -s -d qdisc

For test sake, I just put "tc qdisc add dev $IFACE root handle 1: prio" and no filters at all. I get the same with HTB "tc qdisc add dev $IFACE root handle 1: htb default 99" and no subclasses.

qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13287736273644 bytes 1263672018 pkt (dropped 0, overlimits 0 requeues 2928480094) rate 0bit 0pps backlog 0b 0p requeues 2928480094
qdisc pfifo_fast 0: dev eth1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40064376195000 bytes 1747026586 pkt (dropped 0, overlimits 0 requeues 463621814) rate 0bit 0pps backlog 0b 0p requeues 463621814
qdisc pfifo_fast 0: dev eth2 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13350145517965 bytes 1350897201 pkt (dropped 0, overlimits 0 requeues 2930879507) rate 0bit 0pps backlog 0b 0p requeues 2930879507
qdisc pfifo_fast 0: dev eth3 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40193456126884 bytes 1950653764 pkt (dropped 0, overlimits 0 requeues 465511120) rate 0bit 0pps backlog 0b 0p requeues 465511120
qdisc prio 1: dev bond0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 985164834 bytes 2720991 pkt (dropped 241834, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 1: dev bond1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 2347118738 bytes 3089171 pkt (dropped 304601, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0

** Drops on bond0/bond1 are increasing by approximately 5000 per second:

qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13287874353796 bytes 1264050808 pkt (dropped 0, overlimits 0 requeues 2928520779) rate 0bit 0pps backlog 0b 0p requeues 2928520779
qdisc pfifo_fast 0: dev eth1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40064706826018 bytes 1747459793 pkt (dropped 0, overlimits 0 requeues 463669610) rate 0bit 0pps backlog 0b 0p requeues 463669610
qdisc pfifo_fast 0: dev eth2 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13350283202695 bytes 1351277761 pkt (dropped 0, overlimits 0 requeues 2930918488) rate 0bit 0pps backlog 0b 0p requeues 2930918488
qdisc pfifo_fast 0: dev eth3 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40193784868074 bytes 1951084029 pkt (dropped 0, overlimits 0 requeues 465558015) rate 0bit 0pps backlog 0b 0p requeues 465558015
qdisc prio 1: dev bond0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 1260929539 bytes 3480340 pkt (dropped 311145, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 1: dev bond1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 3006490946 bytes 3952643 pkt (dropped 396850, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0

With same setup on 2.6.23, drops are increasing only by 50/sec or so.

As soon as I do "tc qdisc del dev $IFACE root", packet loss stops.

Quoted 1 line

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 80
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 17
Partner Key: 4
Partner Mac Address: 00:19:e7:b2:07:80

Slave Interface: eth0
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:cc
Aggregator ID: 1

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:ce
Aggregator ID: 1

Quoted 1 line

cat /proc/net/bonding/bond1

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 80
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 2
Number of ports: 2
Actor Key: 17
Partner Key: 5
Partner Mac Address: 00:19:e7:b2:07:80

Slave Interface: eth1
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:cd
Aggregator ID: 2

Slave Interface: eth3
MII Status: up
Link Failure Count: 2
Permanent HW addr: 00:1b:24:bd:e9:cf
Aggregator ID: 2

Quoted 1 line

mpstat -P ALL 10

08:04:36 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
08:04:46 PM  all    0.00    0.00    0.01    0.00    0.00    1.05    0.00   98.94  70525.73
08:04:46 PM    0    0.00    0.00    0.00    0.00    0.00    0.70    0.00   99.30   7814.41
08:04:46 PM    1    0.00    0.00    0.00    0.00    0.00    2.10    0.00   97.90   7814.41
08:04:46 PM    2    0.00    0.00    0.00    0.00    0.00    0.20    0.00   99.80   7814.41
08:04:46 PM    3    0.00    0.00    0.10    0.00    0.00    1.30    0.00   98.60   7814.51
08:04:46 PM    4    0.00    0.00    0.00    0.00    0.00    0.50    0.00   99.50   7814.41
08:04:46 PM    5    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7814.41
08:04:46 PM    6    0.00    0.00    0.00    0.00    0.00    0.60    0.00   99.40   7814.41
08:04:46 PM    7    0.00    0.00    0.10    0.00    0.00    0.90    0.00   99.00   7814.51
08:04:46 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
08:04:46 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
08:04:56 PM  all    0.00    0.00    0.01    0.00    0.00    1.49    0.00   98.50  66429.30
08:04:56 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00   7303.50
08:04:56 PM    1    0.00    0.00    0.00    0.00    0.00    1.60    0.00   98.40   7303.50
08:04:56 PM    2    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    3    0.00    0.00    0.00    0.00    0.00    3.20    0.00   96.80   7303.40
08:04:56 PM    4    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7303.60
08:04:56 PM    5    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    6    0.00    0.00    0.10    0.00    0.00    1.80    0.00   98.10   7303.50
08:04:56 PM    7    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Quoted 1 line

ifconfig -a

bond0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)

bond1 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CD

          inet addr:xxx.xxx.70.156  Bcast:xxx.xxx.70.159  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cd/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:239471641 errors:0 dropped:344 overruns:0 frame:0
          TX packets:3704083902 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2488754745 (2.3 GiB)  TX bytes:2685275089 (2.5 GiB)

eth0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:2235085582 errors:0 dropped:353786 overruns:0 frame:0
          TX packets:1266449269 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3768096439 (3.5 GiB)  TX bytes:113363829 (108.1 MiB)
          Memory:fc6e0000-fc700000

eth1 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CD

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:4228974804 errors:0 dropped:344 overruns:0 frame:0
          TX packets:1750216649 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3350270261 (3.1 GiB)  TX bytes:3358220645 (3.1 GiB)
          Memory:fc6c0000-fc6e0000

eth2 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:2495958020 errors:0 dropped:37464 overruns:0 frame:0
          TX packets:1353707165 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:442055526 (421.5 MiB)  TX bytes:2406943933 (2.2 GiB)
          Memory:fcde0000-fce00000

eth3 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CD

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:305464222 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1953867360 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3433479245 (3.1 GiB)  TX bytes:3622113909 (3.3 GiB)
          Memory:fcd80000-fcda0000

lo Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:53537 errors:0 dropped:0 overruns:0 frame:0
          TX packets:53537 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:431006433 (411.0 MiB)  TX bytes:431006433 (411.0 MiB)

NOTE: ifconfig drops on bond0/bond1 are *NOT* increasing. These drops are there from before.

Signature · 4 lines
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com
#4 lore raw
Quoted 6 lines · Vladimir Ivashchenko
Quoted 3 lines
Quoted 2 lines

On both kernels, the system is running with at least 70% idle CPU. The network interrupts are distributed accross the cores.

You should not distribute interrupts, but bound a NIC to one CPU

Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct. The defaults are wrong?

Yes they are, at least for forwarding setups.

Quoted 2 lines

I have tried with IRQs bound to one CPU per NIC. Same result.

Did you check "grep eth /proc/interrupts" that your affinities setup were indeed taken into account ?

You should use same CPU for eth0 and eth2 (bond0),

and another CPU for eth1 and eth3 (bond1)

check how your cpus are setup

egrep 'physical id|core id|processor' /proc/cpuinfo

Because you might play and find best combo

If you use 2.6.29, apply following patch to get better system accounting, to check if your cpu are saturated or not by hard/soft irqs

kernel/sched.c 2009-05-05 20:47:19.000000000 +0200 renamed from kernel/sched.c.orig 2009-05-05 20:46:49.000000000 +0200 +1 -1
@@ -4290,7 +4290,7 @@
 
        if (user_tick)
                account_user_time(p, one_jiffy, one_jiffy_scaled);
-       else if (p != rq->idle)
+       else if ((p != rq->idle) || (irq_count() != HARDIRQ_OFFSET))
                account_system_time(p, HARDIRQ_OFFSET, one_jiffy,
                                    one_jiffy_scaled);
        else
Quoted 255 lines
Quoted 9 lines
Quoted 9 lines

I thought it was a e1000e driver issue, but tweaking e1000e ring buffers didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs, I tried running on a different server with bnx cards, I tried disabling NO_HZ and HRTICK, but still I have the same problem.

However, if I don't utilize bond, but just apply rules on normal ethX interfaces, there is no packet loss with 2.6.28/29.

Quoted 7 lines
Quoted 5 lines

So, the problem appears only when I use 2.6.28/29 + bond + classful tc combination.

Any ideas ?

Yes, we need much more information :)
Is it a forwarding setup only ?

Yes, the server is doing nothing else but forwarding, no iptables.

Quoted 1 line

cat /proc/interrupts

CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7

  0:        130          0          0          0          0          0          0          0   IO-APIC-edge      timer
  1:          2          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  3:          0          0          0          1          0          1          0          0   IO-APIC-edge
  4:          0          0          1          0          0          0          1          0   IO-APIC-edge
  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          4          0          0          0          0          0          0          0   IO-APIC-edge      i8042
 14:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
 15:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
 17:      30901      31910      31446      30655      31618      30550      31543      30958   IO-APIC-fasteoi   aacraid
 20:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 21:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb5, ahci
 22:     298387     297642     295508     294368     295533     295430     295275     296036   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
 23:      10868      10926      10980      10738      10939      10615      10761      10909   IO-APIC-fasteoi   uhci_hcd:usb3
 57: 1486251823 1486835830 1486677250 1487105983 1488000303 1485941815 1487728317 1486624997   PCI-MSI-edge      eth0
 58: 1510676329 1509708161 1510347202 1509969755 1508599471 1511220118 1509094578 1509727616   PCI-MSI-edge      eth1
 59: 1482578890 1483618556 1482963700 1483164528 1484561615 1482130645 1484116749 1483557717   PCI-MSI-edge      eth2
 60: 1507341647 1506685822 1506862759 1506612818 1505689367 1507559672 1505911622 1506940613   PCI-MSI-edge      eth3
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC: 1020533656 1020535165 1020533613 1020534967 1020535173 1020534409 1020534985 1020534220   Local timer interrupts
RES:      18605      21215      15957      18637      22429      19493      16649      15589   Rescheduling interrupts
CAL:        160        214        186        185        199        205        190        180   Function call interrupts
TLB:     259515     264126     309016     312222     263163     265601     306189     305430   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0
Quoted 1 line

tc -s -d qdisc

For test sake, I just put "tc qdisc add dev $IFACE root handle 1: prio" and no filters at all. I get the same with HTB "tc qdisc add dev $IFACE root handle 1: htb default 99" and no subclasses.

qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13287736273644 bytes 1263672018 pkt (dropped 0, overlimits 0 requeues 2928480094) rate 0bit 0pps backlog 0b 0p requeues 2928480094
qdisc pfifo_fast 0: dev eth1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40064376195000 bytes 1747026586 pkt (dropped 0, overlimits 0 requeues 463621814) rate 0bit 0pps backlog 0b 0p requeues 463621814
qdisc pfifo_fast 0: dev eth2 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13350145517965 bytes 1350897201 pkt (dropped 0, overlimits 0 requeues 2930879507) rate 0bit 0pps backlog 0b 0p requeues 2930879507
qdisc pfifo_fast 0: dev eth3 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40193456126884 bytes 1950653764 pkt (dropped 0, overlimits 0 requeues 465511120) rate 0bit 0pps backlog 0b 0p requeues 465511120
qdisc prio 1: dev bond0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 985164834 bytes 2720991 pkt (dropped 241834, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 1: dev bond1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 2347118738 bytes 3089171 pkt (dropped 304601, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0

** Drops on bond0/bond1 are increasing by approximately 5000 per second:

qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13287874353796 bytes 1264050808 pkt (dropped 0, overlimits 0 requeues 2928520779) rate 0bit 0pps backlog 0b 0p requeues 2928520779
qdisc pfifo_fast 0: dev eth1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40064706826018 bytes 1747459793 pkt (dropped 0, overlimits 0 requeues 463669610) rate 0bit 0pps backlog 0b 0p requeues 463669610
qdisc pfifo_fast 0: dev eth2 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 13350283202695 bytes 1351277761 pkt (dropped 0, overlimits 0 requeues 2930918488) rate 0bit 0pps backlog 0b 0p requeues 2930918488
qdisc pfifo_fast 0: dev eth3 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 40193784868074 bytes 1951084029 pkt (dropped 0, overlimits 0 requeues 465558015) rate 0bit 0pps backlog 0b 0p requeues 465558015
qdisc prio 1: dev bond0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 1260929539 bytes 3480340 pkt (dropped 311145, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 1: dev bond1 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 3006490946 bytes 3952643 pkt (dropped 396850, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0

With same setup on 2.6.23, drops are increasing only by 50/sec or so.

As soon as I do "tc qdisc del dev $IFACE root", packet loss stops.

Quoted 1 line

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 80
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 17
Partner Key: 4
Partner Mac Address: 00:19:e7:b2:07:80

Slave Interface: eth0
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:cc
Aggregator ID: 1

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:ce
Aggregator ID: 1

Quoted 1 line

cat /proc/net/bonding/bond1

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 80
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 2
Number of ports: 2
Actor Key: 17
Partner Key: 5
Partner Mac Address: 00:19:e7:b2:07:80

Slave Interface: eth1
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:cd
Aggregator ID: 2

Slave Interface: eth3
MII Status: up
Link Failure Count: 2
Permanent HW addr: 00:1b:24:bd:e9:cf
Aggregator ID: 2

Quoted 1 line

mpstat -P ALL 10

08:04:36 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
08:04:46 PM  all    0.00    0.00    0.01    0.00    0.00    1.05    0.00   98.94  70525.73
08:04:46 PM    0    0.00    0.00    0.00    0.00    0.00    0.70    0.00   99.30   7814.41
08:04:46 PM    1    0.00    0.00    0.00    0.00    0.00    2.10    0.00   97.90   7814.41
08:04:46 PM    2    0.00    0.00    0.00    0.00    0.00    0.20    0.00   99.80   7814.41
08:04:46 PM    3    0.00    0.00    0.10    0.00    0.00    1.30    0.00   98.60   7814.51
08:04:46 PM    4    0.00    0.00    0.00    0.00    0.00    0.50    0.00   99.50   7814.41
08:04:46 PM    5    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7814.41
08:04:46 PM    6    0.00    0.00    0.00    0.00    0.00    0.60    0.00   99.40   7814.41
08:04:46 PM    7    0.00    0.00    0.10    0.00    0.00    0.90    0.00   99.00   7814.51
08:04:46 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
08:04:46 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
08:04:56 PM  all    0.00    0.00    0.01    0.00    0.00    1.49    0.00   98.50  66429.30
08:04:56 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00   7303.50
08:04:56 PM    1    0.00    0.00    0.00    0.00    0.00    1.60    0.00   98.40   7303.50
08:04:56 PM    2    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    3    0.00    0.00    0.00    0.00    0.00    3.20    0.00   96.80   7303.40
08:04:56 PM    4    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7303.60
08:04:56 PM    5    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    6    0.00    0.00    0.10    0.00    0.00    1.80    0.00   98.10   7303.50
08:04:56 PM    7    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
Quoted 1 line

ifconfig -a

bond0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)

bond1 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CD

          inet addr:xxx.xxx.70.156  Bcast:xxx.xxx.70.159  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cd/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:239471641 errors:0 dropped:344 overruns:0 frame:0
          TX packets:3704083902 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2488754745 (2.3 GiB)  TX bytes:2685275089 (2.5 GiB)

eth0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:2235085582 errors:0 dropped:353786 overruns:0 frame:0
          TX packets:1266449269 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3768096439 (3.5 GiB)  TX bytes:113363829 (108.1 MiB)
          Memory:fc6e0000-fc700000

eth1 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CD

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:4228974804 errors:0 dropped:344 overruns:0 frame:0
          TX packets:1750216649 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3350270261 (3.1 GiB)  TX bytes:3358220645 (3.1 GiB)
          Memory:fc6c0000-fc6e0000

eth2 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:2495958020 errors:0 dropped:37464 overruns:0 frame:0
          TX packets:1353707165 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:442055526 (421.5 MiB)  TX bytes:2406943933 (2.2 GiB)
          Memory:fcde0000-fce00000

eth3 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CD

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:305464222 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1953867360 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3433479245 (3.1 GiB)  TX bytes:3622113909 (3.3 GiB)
          Memory:fcd80000-fcda0000

lo Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:53537 errors:0 dropped:0 overruns:0 frame:0
          TX packets:53537 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:431006433 (411.0 MiB)  TX bytes:431006433 (411.0 MiB)

NOTE: ifconfig drops on bond0/bond1 are *NOT* increasing. These drops are there from before.

#5 lore raw
Quoted 8 lines · Eric Dumazet
Quoted 1 line

I have tried with IRQs bound to one CPU per NIC. Same result.

Did you check "grep eth /proc/interrupts" that your affinities setup were indeed taken into account ?

You should use same CPU for eth0 and eth2 (bond0),

and another CPU for eth1 and eth3 (bond1)

Ok, the best result is when assign all IRQs to the same CPU. Zero drops.

When I bind slaves of bond interfaces to the same CPU, I start to get some drops, but much less than before. I didn't play with combinations.

My problem is, after applying your accounting patch below, one of my HTB servers reports only 30-40% CPU idle on one of the cores. That won't take me for very long, load balancing across cores is needed.

Is there any way at least to balance individual NICs on per core basis?

Signature · 4 lines
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com
#6 lore raw
Quoted 13 lines · On Wed, 6 May 2009 02:50:08 +0300 Vladimir Ivashchenko <[email protected]>…
Quoted 8 lines · Eric Dumazet
Quoted 1 line

I have tried with IRQs bound to one CPU per NIC. Same result.

Did you check "grep eth /proc/interrupts" that your affinities setup were indeed taken into account ?

You should use same CPU for eth0 and eth2 (bond0),

and another CPU for eth1 and eth3 (bond1)

Ok, the best result is when assign all IRQs to the same CPU. Zero drops.

Quoted 9 lines

When I bind slaves of bond interfaces to the same CPU, I start to get some drops, but much less than before. I didn't play with combinations.

My problem is, after applying your accounting patch below, one of my HTB servers reports only 30-40% CPU idle on one of the cores. That won't take me for very long, load balancing across cores is needed.

Is there any way at least to balance individual NICs on per core basis?

The user level irqbalance program is a good place to start:
http://www.irqbalance.org/
But it doesn't yet no how to handle multi-queue devices, and it seems to not handle NUMA (like SMP Nehalam) perfectly.

#7 lore raw
In reply to Vladimir Ivashchenko
Quoted 12 lines · Vladimir Ivashchenko
Quoted 7 lines · Eric Dumazet
Quoted 1 line

I have tried with IRQs bound to one CPU per NIC. Same result.

Did you check "grep eth /proc/interrupts" that your affinities setup were indeed taken into account ?

You should use same CPU for eth0 and eth2 (bond0),

and another CPU for eth1 and eth3 (bond1)

Ok, the best result is when assign all IRQs to the same CPU. Zero drops.

Quoted 9 lines

When I bind slaves of bond interfaces to the same CPU, I start to get some drops, but much less than before. I didn't play with combinations.

My problem is, after applying your accounting patch below, one of my HTB servers reports only 30-40% CPU idle on one of the cores. That won't take me for very long, load balancing across cores is needed.

Is there any way at least to balance individual NICs on per core basis?

Problem of this setup is you have four NICS, but two logical devices (bond0 & bond1) and a central HTB thing. This essentialy makes flows go through the same locks (some rwlocks guarding bonding driver, and others guarding HTB structures).

Also when a cpu receives a frame on ethX, it has to forward it on ethY, and another lock guards access to TX queue of ethY device. If another cpus receives a frame on ethZ and want to forward it to ethY device, this other cpu will need same locks and everything slowdown.

I am pretty sure you could get good results choosing two cpus sharing same L2 cache. L2 on your cpu is 6MB. Another point would be to carefuly choose size of RX rings on ethX devices. You could try to *reduce* them so that number of inflight skb is small enough that everything fits in this 6MB cache.

Problem is not really CPU power, but RAM bandwidth. Having two cores instead of one attached to one central memory bank wont increase ram bandwidth, but reduce it.

And making several cores compete for locks on this ram only slows down processing.

Only choice we have is to change bonding so that this driver uses RCU instead of rwlocks, but it is probably a complex task. Multiple cpus accessing bonding structures could share memory structures without dirtying them and ping-pong cache lines.

Ah, I forgot about one patch that could help your setup too (if using more than one cpu on NIC irqs of course), queued for 2.6.31

(commit 6a321cb370ad3db4ba6e405e638b3a42c41089b0)

You could post oprofile results to help us finding other hot spots.

[PATCH] net: netif_tx_queue_stopped too expensive

netif_tx_queue_stopped(txq) is most of the time false.

Yet its cost is very expensive on SMP.

static inline int netif_tx_queue_stopped(const struct netdev_queue *dev_queue)

{
	return test_bit(__QUEUE_STATE_XOFF, &dev_queue->state);
}

I saw this on oprofile hunting and bnx2 driver bnx2_tx_int().

We probably should split "struct netdev_queue" in two parts, one being read mostly.

__netif_tx_lock() touches _xmit_lock & xmit_lock_owner, these deserve a separate cache line.

include/linux/netdevice.h +8 -2
index 2e7783f..1caaebb 100644
@@ -447,12 +447,18 @@ enum netdev_queue_state_t
 };
 
 struct netdev_queue {
+/*
+ * read mostly part
+ */
 	struct net_device	*dev;
 	struct Qdisc		*qdisc;
 	unsigned long		state;
-	spinlock_t		_xmit_lock;
-	int			xmit_lock_owner;
 	struct Qdisc		*qdisc_sleeping;
+/*
+ * write mostly part
+ */
+	spinlock_t		_xmit_lock ____cacheline_aligned_in_smp;
+	int			xmit_lock_owner;
 } ____cacheline_aligned_in_smp;
 
 
#22 lore raw
In reply to Vladimir Ivashchenko
Quoted 7 lines · Vladimir Ivashchenko
Quoted 3 lines
Quoted 2 lines

On both kernels, the system is running with at least 70% idle CPU. The network interrupts are distributed accross the cores.

You should not distribute interrupts, but bound a NIC to one CPU

Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct. The defaults are wrong?

...

Quoted 10 lines
Quoted 1 line

ifconfig -a

bond0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)

Could you try e.g.: ifconfig bond0 txqueuelen 1000 before tc qdisc add?

Jarek P.

#21 lore raw
In reply to Eric Dumazet
Quoted 33 lines · Eric Dumazet
Quoted 6 lines · Vladimir Ivashchenko
Quoted 3 lines
Quoted 2 lines

On both kernels, the system is running with at least 70% idle CPU. The network interrupts are distributed accross the cores.

You should not distribute interrupts, but bound a NIC to one CPU

Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct. The defaults are wrong?

Yes they are, at least for forwarding setups.

Quoted 2 lines

I have tried with IRQs bound to one CPU per NIC. Same result.

Did you check "grep eth /proc/interrupts" that your affinities setup were indeed taken into account ?

You should use same CPU for eth0 and eth2 (bond0),

and another CPU for eth1 and eth3 (bond1)

check how your cpus are setup

egrep 'physical id|core id|processor' /proc/cpuinfo

Because you might play and find best combo

If you use 2.6.29, apply following patch to get better system accounting, to check if your cpu are saturated or not by hard/soft irqs

kernel/sched.c 2009-05-05 20:47:19.000000000 +0200 line 4296

Quoted 8 lines
kernel/sched.c 2009-05-05 20:47:19.000000000 +0200 renamed from kernel/sched.c.orig 2009-05-05 20:46:49.000000000 +0200 +1 -1
@@ -4290,7 +4290,7 @@
 
        if (user_tick)
                account_user_time(p, one_jiffy, one_jiffy_scaled);
-       else if (p != rq->idle)
+       else if ((p != rq->idle) || (irq_count() != HARDIRQ_OFFSET))
                account_system_time(p, HARDIRQ_OFFSET, one_jiffy,
                                    one_jiffy_scaled);
        else

Note, your scheduler fix is upstream now in Linus's tree, as:

f5f293a: sched: account system time properly

"git cherry-pick f5f293a" will apply it to a .29 basis.

Ingo

#8 lore raw
In reply to Eric Dumazet
Quoted 12 lines · Eric Dumazet
Quoted 2 lines

Is there any way at least to balance individual NICs on per core basis?

Problem of this setup is you have four NICS, but two logical devices (bond0 & bond1) and a central HTB thing. This essentialy makes flows go through the same locks (some rwlocks guarding bonding driver, and others guarding HTB structures).

Also when a cpu receives a frame on ethX, it has to forward it on ethY, and another lock guards access to TX queue of ethY device. If another cpus receives a frame on ethZ and want to forward it to ethY device, this other cpu will need same locks and everything slowdown.

Quoted 7 lines

I am pretty sure you could get good results choosing two cpus sharing same L2 cache. L2 on your cpu is 6MB. Another point would be to carefuly choose size of RX rings on ethX devices. You could try to *reduce* them so that number of inflight skb is small enough that everything fits in this 6MB cache.

Problem is not really CPU power, but RAM bandwidth. Having two cores instead of one attached to one central memory bank wont increase ram bandwidth, but reduce it.

Thanks for the detailed explanation.

On the particular server I reported, I worked around the problem by getting rid of classes and switching to ingress policers.

However, I have one central box doing HTB, small amount of classes, but 850 mbps of traffic. The CPU is dual-core 5160 @ 3 Ghz. With 2.6.29 + bond I'm experiencing strange problems with HTB, under high load borrowing doesn't seem to work properly. This box has two BNX2 and two E1000 NICs, and for some reason I cannot force BNX2 to sit on a single IRQ - even though I put only one CPU into smp_affinity, it keeps balancing on both. So I cannot figure out if its related to IRQ balancing or not.

[root@tshape3 tshaper]# cat /proc/irq/63/smp_affinity 01
[root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
63: 44610754 95469129 PCI-MSI-edge eth0
[root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
63: 44614125 95472512 PCI-MSI-edge eth0

lspci -v:

03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Subsystem: Hewlett-Packard Company NC373i Integrated Multifunction Gigabit Server Adapter
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 63
Memory at f8000000 (64-bit, non-prefetchable) [size=32M]
[virtual] Expansion ROM at 88200000 [disabled] [size=2K]
Capabilities: [40] PCI-X non-bridge device
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data <?>
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+ Kernel driver in use: bnx2
Kernel modules: bnx2

Any ideas on how to force it on a single CPU ?

Thanks for the new patch, I will try it and let you know.

Signature · 4 lines
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com
#23 lore raw
In reply to Jarek Poplawski
Quoted 11 lines · Jarek Poplawski
Quoted 10 lines
Quoted 1 line

ifconfig -a

bond0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)
Quoted 2 lines

Could you try e.g.: ifconfig bond0 txqueuelen 1000 before tc qdisc add?

The drops on ifconfig are not increasing - these numbers are there from some tests made before.

Signature · 4 lines
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com
#9 lore raw
In reply to Vladimir Ivashchenko
Quoted 55 lines · Vladimir Ivashchenko
Quoted 11 lines · Eric Dumazet
Quoted 2 lines

Is there any way at least to balance individual NICs on per core basis?

Problem of this setup is you have four NICS, but two logical devices (bond0 & bond1) and a central HTB thing. This essentialy makes flows go through the same locks (some rwlocks guarding bonding driver, and others guarding HTB structures).

Also when a cpu receives a frame on ethX, it has to forward it on ethY, and another lock guards access to TX queue of ethY device. If another cpus receives a frame on ethZ and want to forward it to ethY device, this other cpu will need same locks and everything slowdown.

Quoted 7 lines

I am pretty sure you could get good results choosing two cpus sharing same L2 cache. L2 on your cpu is 6MB. Another point would be to carefuly choose size of RX rings on ethX devices. You could try to *reduce* them so that number of inflight skb is small enough that everything fits in this 6MB cache.

Problem is not really CPU power, but RAM bandwidth. Having two cores instead of one attached to one central memory bank wont increase ram bandwidth, but reduce it.

Thanks for the detailed explanation.

On the particular server I reported, I worked around the problem by getting rid of classes and switching to ingress policers.

However, I have one central box doing HTB, small amount of classes, but 850 mbps of traffic. The CPU is dual-core 5160 @ 3 Ghz. With 2.6.29 + bond I'm experiencing strange problems with HTB, under high load borrowing doesn't seem to work properly. This box has two BNX2 and two E1000 NICs, and for some reason I cannot force BNX2 to sit on a single IRQ - even though I put only one CPU into smp_affinity, it keeps balancing on both. So I cannot figure out if its related to IRQ balancing or not.

[root@tshape3 tshaper]# cat /proc/irq/63/smp_affinity 01
[root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
63: 44610754 95469129 PCI-MSI-edge eth0
[root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
63: 44614125 95472512 PCI-MSI-edge eth0

lspci -v:

03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Subsystem: Hewlett-Packard Company NC373i Integrated Multifunction Gigabit Server Adapter
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 63
Memory at f8000000 (64-bit, non-prefetchable) [size=32M]
[virtual] Expansion ROM at 88200000 [disabled] [size=2K]
Capabilities: [40] PCI-X non-bridge device
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data <?>
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+ Kernel driver in use: bnx2
Kernel modules: bnx2

Quoted 4 lines

Any ideas on how to force it on a single CPU ?

Thanks for the new patch, I will try it and let you know.

Yes, its doable but tricky with bnx2, this is a known problem on recent kernels as well.

You must do for example (to bind on CPU 0)

echo 1 >/proc/irq/default_smp_affinity

ifconfig eth1 down
# IRQ of eth1 handled by CPU0 only
echo 1 >/proc/irq/34/smp_affinity
ifconfig eth1 up

ifconfig eth0 down
# IRQ of eth0 handled by CPU0 only
echo 1 >/proc/irq/36/smp_affinity
ifconfig eth0 up

One thing to consider too is the BIOS option you might have, labeled "Adjacent Sector Prefetch"

This basically tells your cpu to use 128 bytes cache lines, instead of 64

In your forwarding worload, I believe this extra prefetch can slowdown your machine.

#24 lore raw
In reply to Vladimir Ivashchenko
Quoted 13 lines · Vladimir Ivashchenko
Quoted 11 lines · Jarek Poplawski
Quoted 10 lines
Quoted 1 line

ifconfig -a

bond0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)
Quoted 6 lines
Quoted 2 lines

Could you try e.g.: ifconfig bond0 txqueuelen 1000 before tc qdisc add?

The drops on ifconfig are not increasing - these numbers are there from some tests made before.

I'm not sure what do you mean? IMHO you don't use qdiscs properly, so any TX problems end with drops. (Older kernel versions could mask this problem with requeuing.)

Jarek P.

#10 lore raw
In reply to Eric Dumazet
Quoted 9 lines · On Wednesday 06 May 2009 13:41:25 Eric Dumazet wrote:

You must do for example (to bind on CPU 0)

echo 1 >/proc/irq/default_smp_affinity

ifconfig eth1 down
# IRQ of eth1 handled by CPU0 only
echo 1 >/proc/irq/34/smp_affinity
ifconfig eth1 up

Quoted 4 lines

ifconfig eth0 down
# IRQ of eth0 handled by CPU0 only
echo 1 >/proc/irq/36/smp_affinity
ifconfig eth0 up

I think better to use some method over ethtool, that will cause reset. WHen you do down - you will loose default route, beware of that

Quoted 15 lines

One thing to consider too is the BIOS option you might have, labeled "Adjacent Sector Prefetch"

This basically tells your cpu to use 128 bytes cache lines, instead of 64

In your forwarding worload, I believe this extra prefetch can slowdown your machine.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#25 lore raw
In reply to Jarek Poplawski
Quoted 14 lines · Jarek Poplawski
Quoted 14 lines
Quoted 13 lines
Quoted 10 lines
Quoted 1 line

ifconfig -a

bond0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)

Could you try e.g.: ifconfig bond0 txqueuelen 1000 before tc qdisc add?

Quoted 7 lines
Quoted 3 lines

The drops on ifconfig are not increasing - these numbers are there from some tests made before.

I'm not sure what do you mean? IMHO you don't use qdiscs properly, so any TX problems end with drops. (Older kernel versions could mask this problem with requeuing.)

Apologies, my bad, I misread what you wrote.

txqueuelen 1000 fixes the qdisc drops. I didn't notice that bond interfaces have it set to 0 by default.

As suggested by Eric earlier, the drops also disappear if I bind each NIC to a single CPU. That was the default on older kernels and perhaps that's why the issue came up only on 2.6.28.

Signature · 4 lines
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com
#26 lore raw
Quoted 16 lines · Vladimir Ivashchenko
Quoted 14 lines · Jarek Poplawski
Quoted 14 lines
Quoted 11 lines
Quoted 9 lines
Quoted 1 line

ifconfig -a

bond0 Link encap:Ethernet HWaddr 00:1B:24:BD:E9:CC

          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)

Could you try e.g.: ifconfig bond0 txqueuelen 1000 before tc qdisc add?

The drops on ifconfig are not increasing - these numbers are there from some tests made before.

Quoted 7 lines
Quoted 3 lines

I'm not sure what do you mean? IMHO you don't use qdiscs properly, so any TX problems end with drops. (Older kernel versions could mask this problem with requeuing.)

Apologies, my bad, I misread what you wrote.

txqueuelen 1000 fixes the qdisc drops. I didn't notice that bond interfaces have it set to 0 by default.

The fifos use a queue length of 1 when the tx_queue_len is zero (which was added as a workaround for similar problems a long time ago). Perhaps we should instead refuse these broken configurations or at least print a warning.

#11 lore raw
In reply to Eric Dumazet
Quoted 2 lines · Eric Dumazet

Ah, I forgot about one patch that could help your setup too (if using more than one cpu on NIC irqs of course), queued for 2.6.31

I have tried the patch. Didn't make a noticeable difference. Under 850 mbps HTB+sfq load, 2.6.29.1, four NICs / two bond ifaces, IRQ balancing, the dual-core server has only 25% idle on each CPU.

What's interesting, the same 850mbps load, identical machine, but with only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle. 2.5x overhead.

Quoted 52 lines

(commit 6a321cb370ad3db4ba6e405e638b3a42c41089b0)

You could post oprofile results to help us finding other hot spots.

[PATCH] net: netif_tx_queue_stopped too expensive

netif_tx_queue_stopped(txq) is most of the time false.

Yet its cost is very expensive on SMP.

static inline int netif_tx_queue_stopped(const struct netdev_queue *dev_queue)

{
	return test_bit(__QUEUE_STATE_XOFF, &dev_queue->state);
}

I saw this on oprofile hunting and bnx2 driver bnx2_tx_int().

We probably should split "struct netdev_queue" in two parts, one being read mostly.

__netif_tx_lock() touches _xmit_lock & xmit_lock_owner, these deserve a separate cache line.

include/linux/netdevice.h +8 -2
index 2e7783f..1caaebb 100644
@@ -447,12 +447,18 @@ enum netdev_queue_state_t
 };
 
 struct netdev_queue {
+/*
+ * read mostly part
+ */
 	struct net_device	*dev;
 	struct Qdisc		*qdisc;
 	unsigned long		state;
-	spinlock_t		_xmit_lock;
-	int			xmit_lock_owner;
 	struct Qdisc		*qdisc_sleeping;
+/*
+ * write mostly part
+ */
+	spinlock_t		_xmit_lock ____cacheline_aligned_in_smp;
+	int			xmit_lock_owner;
 } ____cacheline_aligned_in_smp;
 
 
Signature · 5 lines
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211
#12 lore raw
Quoted 11 lines · On Wednesday 06 May 2009 21:45:18 Vladimir Ivashchenko wrote:
Quoted 2 lines · Eric Dumazet

Ah, I forgot about one patch that could help your setup too (if using more than one cpu on NIC irqs of course), queued for 2.6.31

I have tried the patch. Didn't make a noticeable difference. Under 850 mbps HTB+sfq load, 2.6.29.1, four NICs / two bond ifaces, IRQ balancing, the dual-core server has only 25% idle on each CPU.

What's interesting, the same 850mbps load, identical machine, but with only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle. 2.5x overhead.

Probably oprofile can sched some light on this.
On my own experience IRQ balancing hurt performance a lot, because of cache misses.

#13 lore raw
Quoted 7 lines · Denys Fedoryschenko
Quoted 3 lines

What's interesting, the same 850mbps load, identical machine, but with only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle. 2.5x overhead.

Probably oprofile can sched some light on this.
On my own experience IRQ balancing hurt performance a lot, because of cache misses.

This is a dual-core machine, isn't cache shared between the cores?

Without IRQ balancing, one of the cores goes around 10% idle and HTB doesn't do its job properly. Actually, in my experience HTB stops working properly after idle goes below 35%.

I'll try gathering some stats using oprofile.

Signature · 4 lines
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com
#14 lore raw
Quoted 9 lines · On Wednesday 06 May 2009 23:47:59 Vladimir Ivashchenko wrote:
Quoted 7 lines · Denys Fedoryschenko
Quoted 3 lines

What's interesting, the same 850mbps load, identical machine, but with only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle. 2.5x overhead.

Probably oprofile can sched some light on this.
On my own experience IRQ balancing hurt performance a lot, because of cache misses.

Quoted 5 lines

This is a dual-core machine, isn't cache shared between the cores?

Without IRQ balancing, one of the cores goes around 10% idle and HTB doesn't do its job properly. Actually, in my experience HTB stops working properly after idle goes below 35%.

It seems they should. No idea, more experienced guys should know more.

Can you show me please
cat /proc/net/psched
If it is highres working, try to add in HTB script, first line

HZ=1000
to set environment variable. Because if clock resolution high, burst calculation going crazy on high speeds.
Maybe it will help.

Also without irq balance, did you try to assign interface to cpu by smp_affinity? (/proc/irq/NN/smp_affinity)

And still i think best thing is oprofile. It can show "hot" places in code, who is spending cpu cycles.

Quoted 2 lines

I'll try gathering some stats using oprofile.

#15 lore raw
Quoted 6 lines
Quoted 4 lines

Without IRQ balancing, one of the cores goes around 10% idle and HTB doesn't do its job properly. Actually, in my experience HTB stops working properly after idle goes below 35%.

It seems they should. No idea, more experienced guys should know more.

Quoted 8 lines

Can you show me please
cat /proc/net/psched
If it is highres working, try to add in HTB script, first line

HZ=1000
to set environment variable. Because if clock resolution high, burst calculation going crazy on high speeds.
Maybe it will help.

Wow, instead of 98425b burst, its calculating 970203b.

Exporting HZ=1000 doesn't help. However, even if I recompile the kernel to 1000 Hz and the burst is calculated correctly, for some reason HTB on 2.6.29 is still worse at rate control than 2.6.21.

With 2.6.21, ceil of 775 mbits, burst 99425b -> actual rate 825 mbits. With 2.6.29, same ceil/burst -> actual rate 890 mbits.

Moreover, after I stop the traffic *COMPLETELY* on 2.6.29, actual rate reported by htb goes ballistic and stays at 1100mbits. Then it drops back to expected value after a minute or so.

Quoted 2 lines

Also without irq balance, did you try to assign interface to cpu by smp_affinity? (/proc/irq/NN/smp_affinity)

Yes, I did, didn't make any difference.

Quoted 2 lines

And still i think best thing is oprofile. It can show "hot" places in code, who is spending cpu cycles.

For some reason I get a hard freeze when I start oprofile daemon, even without traffic. Never used oprofile before, so I'm not sure if I'm doing something wrong ... I'm starting it just with --vmlinux parameter and nothing else. I use vanilla 2.6.29 and oprofile from FC8.

Signature · 5 lines
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211
#16 lore raw
Quoted 10 lines · On Friday 08 May 2009 23:46:11 Vladimir Ivashchenko wrote:
Quoted 10 lines
Quoted 3 lines

Without IRQ balancing, one of the cores goes around 10% idle and HTB doesn't do its job properly. Actually, in my experience HTB stops working properly after idle goes below 35%.

It seems they should. No idea, more experienced guys should know more.

Can you show me please
cat /proc/net/psched
If it is highres working, try to add in HTB script, first line

Quoted 6 lines
Quoted 4 lines

HZ=1000
to set environment variable. Because if clock resolution high, burst calculation going crazy on high speeds.
Maybe it will help.

Wow, instead of 98425b burst, its calculating 970203b.

Kind of strange burst, something wrong there. For 1000HZ and 1 Gbit it should be 126375b. You value is for 8Gbit/s.
What version of iproute2 you are using ( tc -V )?

Quoted 7 lines

Exporting HZ=1000 doesn't help. However, even if I recompile the kernel to 1000 Hz and the burst is calculated correctly, for some reason HTB on 2.6.29 is still worse at rate control than 2.6.21.

With 2.6.21, ceil of 775 mbits, burst 99425b -> actual rate 825 mbits. With 2.6.29, same ceil/burst -> actual rate 890 mbits.

It depends also if there is child classes, what is bursts set for them, and what is ceil/burst set for them.

Quoted 4 lines

Moreover, after I stop the traffic *COMPLETELY* on 2.6.29, actual rate reported by htb goes ballistic and stays at 1100mbits. Then it drops back to expected value after a minute or so.

It is average bandwidth for some period, it is not realtime value.

Quoted 5 lines
Quoted 3 lines

Also without irq balance, did you try to assign interface to cpu by smp_affinity? (/proc/irq/NN/smp_affinity)

Yes, I did, didn't make any difference.

What is a clock source?
cat /sys/devices/system/clocksource/clocksource0/current_clocksource Timer resolution?
cat /proc/net/psched

Quoted 8 lines
Quoted 3 lines

And still i think best thing is oprofile. It can show "hot" places in code, who is spending cpu cycles.

For some reason I get a hard freeze when I start oprofile daemon, even without traffic. Never used oprofile before, so I'm not sure if I'm doing something wrong ... I'm starting it just with --vmlinux parameter and nothing else. I use vanilla 2.6.29 and oprofile from FC8.

#17 lore raw
Quoted 4 lines
Quoted 1 line

Wow, instead of 98425b burst, its calculating 970203b.

Kind of strange burst, something wrong there. For 1000HZ and 1 Gbit it should be 126375b. You value is for 8Gbit/s.
What version of iproute2 you are using ( tc -V )?

That was iproute2-ss080725, I think it is confused by tickless mode. With iproute2-ss090324 I'm getting an opposite: 1589b :)

Quoted 5 lines
Quoted 3 lines

With 2.6.21, ceil of 775 mbits, burst 99425b -> actual rate 825 mbits. With 2.6.29, same ceil/burst -> actual rate 890 mbits.

It depends also if there is child classes, what is bursts set for them, and what is ceil/burst set for them.

All child classes have smaller bursts than the parent. However, there are two sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I don't know HTB internals, perhaps these two classes make the parent class overstretch itself.

By the way, I experience the same "overstretching" with hfsc. In any case, I prefer HTB because it reports statistics of parent classes, unlike hfsc.

Quoted 4 lines
Quoted 3 lines

Moreover, after I stop the traffic *COMPLETELY* on 2.6.29, actual rate reported by htb goes ballistic and stays at 1100mbits. Then it drops back to expected value after a minute or so.

It is average bandwidth for some period, it is not realtime value.

But why it would it jump from 850mbits to 1200mbits *AFTER* I remove all the traffic ?

Quoted 3 lines
Quoted 1 line

Yes, I did, didn't make any difference.

What is a clock source?
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

tsc

Quoted 2 lines

Timer resolution?
cat /proc/net/psched

With tickless kernel:

000003e8 00000400 000f4240 3b9aca00

Signature · 4 lines
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com
#18 lore raw
Quoted 8 lines · On Saturday 09 May 2009 01:07:27 Vladimir Ivashchenko wrote:
Quoted 5 lines
Quoted 1 line

Wow, instead of 98425b burst, its calculating 970203b.

Kind of strange burst, something wrong there. For 1000HZ and 1 Gbit it should be 126375b. You value is for 8Gbit/s.
What version of iproute2 you are using ( tc -V )?

That was iproute2-ss080725, I think it is confused by tickless mode. With iproute2-ss090324 I'm getting an opposite: 1589b :)

And it is too low. Thats why i set HZ=1000

Quoted 6 lines

All child classes have smaller bursts than the parent. However, there are two sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I don't know HTB internals, perhaps these two classes make the parent class overstretch itself.

As i remember important to keep sum of child rates lower or equal parent rate. Sure ceil of childs must not exceed ceil of parent.
Sometimes i had mess, when i tried to play with quantum value. After all that i switched to HFSC which works for me flawlessly. Maybe we should give more attention to HTB problem with high speeds and help kernel developers spot problem, if there is any.

Quoted 3 lines

By the way, I experience the same "overstretching" with hfsc. In any case, I prefer HTB because it reports statistics of parent classes, unlike hfsc.

Sometimes it happen when some offloading enabled on devices. Check ethtool -k device

I think everything except rx/tx checksumming have to be off, at least for test.

Disable them by "ethtool -K device tso off " for example.

Quoted 4 lines

But why it would it jump from 850mbits to 1200mbits *AFTER* I remove all the traffic ?

Well, i dont know how it is doing averaging, even maybe for 1 minute. I dont like it at all, and thats why i prefer HFSC. But HTB work very well in some setups

#19 lore raw
Quoted 11 lines
Quoted 5 lines

All child classes have smaller bursts than the parent. However, there are two sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I don't know HTB internals, perhaps these two classes make the parent class overstretch itself.

As i remember important to keep sum of child rates lower or equal parent rate. Sure ceil of childs must not exceed ceil of parent.
Sometimes i had mess, when i tried to play with quantum value. After all that i switched to HFSC which works for me flawlessly. Maybe we should give more attention to HTB problem with high speeds and help kernel developers spot problem, if there is any.

In case of HFSC my problem is even worse. With 775mbit ceiling configured it is passing over 900mbit in reality. Moreover not having statistics for parent classes makes it difficult to troubleshoot :( I'm 100% sure that it is 900 mbps, I see this on the switch.

Attached is "tc -s -d class show dev bond0" output.

To calculate total traffic rate:

$ cat hfsc-stat.txt | grep rate | grep Kbit | sed 's/Kbit//' | awk
'{ a=a+$2; } END { print a; }'
906955

Did I misconfigure something ?... How can hfsc go above 775mbit when everything goes via class 1:2 with 775mbit rate & ul ?

Quoted 4 lines
Quoted 2 lines

By the way, I experience the same "overstretching" with hfsc. In any case, I prefer HTB because it reports statistics of parent classes, unlike hfsc.

Sometimes it happen when some offloading enabled on devices. Check ethtool -k device

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off

Quoted 4 lines

I think everything except rx/tx checksumming have to be off, at least for test.

Disable them by "ethtool -K device tso off " for example.

Doesn't help.

Signature · 5 lines
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211

class hfsc 1:99 parent 1: leaf 99: sc m1 0bit d 15.0ms m2 542500Kbit
Sent 20385133 bytes 192619 pkt (dropped 7070, overlimits 0 requeues 0) rate 402936bit 432pps backlog 0b 0p requeues 0
period 192581 work 20385133 bytes rtwork 20353003 bytes level 0

class hfsc 1:5005 parent 1:97 leaf 5005: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 2948666978 bytes 4359774 pkt (dropped 50623, overlimits 0 requeues 0)
rate 55946Kbit 10338pps backlog 0b 7p requeues 0
period 1246140 work 2945839727 bytes rtwork 332352472 bytes level 0

class hfsc 1:98 parent 1:2 leaf 98: sc m1 0bit d 500.0ms m2 43400Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 4644738789 bytes 5853965 pkt (dropped 165294, overlimits 0 requeues 0)
rate 85749Kbit 13526pps backlog 0b 0p requeues 0
period 1968169 work 4633993957 bytes rtwork 2322556251 bytes level 0

class hfsc 1:100 parent 1:2 sc m1 0bit d 5.0ms m2 15000Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 290192 work 76779316 bytes level 1

class hfsc 1:10 parent 1:2 sc m1 0bit d 1.0ms m2 542500Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 464809 work 877481554 bytes level 1

class hfsc 1:5004 parent 1:97 leaf 5004: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 3181480387 bytes 4586738 pkt (dropped 50789, overlimits 0 requeues 0)
rate 60049Kbit 10765pps backlog 0b 9p requeues 0
period 1258266 work 3178564479 bytes rtwork 332600104 bytes level 0

class hfsc 1:5006 parent 1:97 leaf 5006: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 842333834 bytes 1339029 pkt (dropped 9194, overlimits 0 requeues 0)
rate 18154Kbit 3349pps backlog 0b 2p requeues 0
period 806244 work 841676259 bytes rtwork 332340905 bytes level 0

class hfsc 1:3e9 parent 1:200 leaf 3e9: sc m1 0bit d 0us m2 256000bit ul m1 0bit d 0us m2 100000Kbit
Sent 23292 bytes 308 pkt (dropped 0, overlimits 0 requeues 0)
rate 488bit 1pps backlog 0b 0p requeues 0
period 298 work 23292 bytes rtwork 21371 bytes level 0

class hfsc 1:5001 parent 1:97 leaf 5001: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 2541073789 bytes 3768515 pkt (dropped 39084, overlimits 0 requeues 0)
rate 52514Kbit 9201pps backlog 0b 4p requeues 0
period 1221187 work 2538919598 bytes rtwork 332543703 bytes level 0

class hfsc 1:3e8 parent 1:200 leaf 3e8: sc m1 0bit d 0us m2 512000bit ul m1 0bit d 0us m2 100000Kbit
Sent 2140 bytes 34 pkt (dropped 0, overlimits 0 requeues 0)
rate 32bit 0pps backlog 0b 0p requeues 0
period 34 work 2140 bytes rtwork 2140 bytes level 0

class hfsc 1:3eb parent 1:300 leaf 3eb: sc m1 0bit d 0us m2 64000bit ul m1 0bit d 0us m2 100000Kbit
Sent 206 bytes 3 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 3 work 206 bytes rtwork 206 bytes level 0

class hfsc 1:5003 parent 1:97 leaf 5003: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 2245687618 bytes 3498522 pkt (dropped 35135, overlimits 0 requeues 0)
rate 42763Kbit 8415pps backlog 0b 5p requeues 0
period 1192583 work 2244028709 bytes rtwork 332304111 bytes level 0

class hfsc 1:3ea parent 1:300 leaf 3ea: sc m1 0bit d 0us m2 1024Kbit ul m1 0bit d 0us m2 100000Kbit
Sent 54214 bytes 442 pkt (dropped 0, overlimits 0 requeues 0)
rate 16bit 0pps backlog 0b 0p requeues 0
period 413 work 54214 bytes rtwork 48427 bytes level 0

class hfsc 1:5002 parent 1:97 leaf 5002: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 2915382517 bytes 4002191 pkt (dropped 35677, overlimits 0 requeues 0)
rate 54029Kbit 9447pps backlog 0b 7p requeues 0
period 1219564 work 2912952580 bytes rtwork 332338435 bytes level 0

class hfsc 1:97 parent 1:2 sc m1 0bit d 200.0ms m2 97650Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 1853176 work 14661981352 bytes level 1

class hfsc 1: root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
period 0 level 3

class hfsc 1:4004 parent 1:79 leaf 4004: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 4401715817 bytes 4690294 pkt (dropped 32975, overlimits 0 requeues 0)
rate 84643Kbit 11103pps backlog 0b 0p requeues 0
period 3255986 work 4402143323 bytes rtwork 1368797472 bytes level 0

class hfsc 1:4005 parent 1:79 leaf 4005: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 4858823203 bytes 5510336 pkt (dropped 45336, overlimits 0 requeues 0)
rate 90141Kbit 12773pps backlog 0b 0p requeues 0
period 3648819 work 4859323215 bytes rtwork 1370067779 bytes level 0

class hfsc 1:2 parent 1: sc m1 0bit d 0us m2 775000Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
period 6902 work 48071507639 bytes level 2

class hfsc 1:4006 parent 1:79 leaf 4006: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 2215296226 bytes 2405111 pkt (dropped 16771, overlimits 0 requeues 0)
rate 41477Kbit 5802pps backlog 0b 0p requeues 0
period 2036400 work 2217428341 bytes rtwork 1369522295 bytes level 0

class hfsc 1:c9 parent 1:200 leaf c9: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit
Sent 240026752 bytes 543130 pkt (dropped 7429, overlimits 0 requeues 0)
rate 4201Kbit 1193pps backlog 0b 0p requeues 0
period 307816 work 240015088 bytes rtwork 1763247 bytes level 0

class hfsc 1:4001 parent 1:79 leaf 4001: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 4432353076 bytes 4969719 pkt (dropped 31037, overlimits 0 requeues 0)
rate 79306Kbit 10880pps backlog 0b 0p requeues 0
period 3452541 work 4431357910 bytes rtwork 1370028629 bytes level 0

class hfsc 1:4002 parent 1:79 leaf 4002: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 4446019233 bytes 4948736 pkt (dropped 42181, overlimits 0 requeues 0)
rate 87046Kbit 11613pps backlog 0b 0p requeues 0
period 3427533 work 4448206779 bytes rtwork 1370171471 bytes level 0

class hfsc 1:4003 parent 1:79 leaf 4003: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 3759018918 bytes 4336293 pkt (dropped 23579, overlimits 0 requeues 0)
rate 75258Kbit 10430pps backlog 0b 0p requeues 0
period 3147915 work 3758684585 bytes rtwork 1368528984 bytes level 0

class hfsc 1:80 parent 1:2 leaf 80: sc m1 0bit d 50.0ms m2 1000bit ul m1 0bit d 0us m2 542500Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 0 level 0

class hfsc 1:b parent 1:10 leaf b: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit
Sent 878491329 bytes 1005975 pkt (dropped 15849, overlimits 0 requeues 0)
rate 14093Kbit 2159pps backlog 0b 0p requeues 0
period 464809 work 877481554 bytes rtwork 1891407 bytes level 0

class hfsc 1:300 parent 1:2 sc m1 0bit d 15.0ms m2 30000Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 742649 work 1854498600 bytes level 1

class hfsc 1:3999 parent 1:79 leaf 3999: sc m1 0bit d 0us m2 1000bit ul m1 0bit d 0us m2 30432Kbit
Sent 1617385819 bytes 1260031 pkt (dropped 110848, overlimits 0 requeues 0)
rate 30581Kbit 2977pps backlog 0b 56p requeues 0
period 1381 work 1609588187 bytes rtwork 58059 bytes level 0

class hfsc 1:12d parent 1:300 leaf 12d: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit
Sent 1860128386 bytes 2152291 pkt (dropped 51268, overlimits 0 requeues 0)
rate 29842Kbit 4647pps backlog 0b 2p requeues 0
period 742499 work 1854444180 bytes rtwork 1775294 bytes level 0

class hfsc 1:79 parent 1:2 sc m1 0bit d 0us m2 401450Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 9008 work 25726746683 bytes level 1

class hfsc 1:2000 parent 1:79 sc m1 0bit d 0us m2 1000bit ul m1 0bit d 0us m2 401450Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 0 level 0

class hfsc 1:200 parent 1:2 sc m1 0bit d 10.0ms m2 35000Kbit ul m1 0bit d 0us m2 542500Kbit
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
period 308015 work 240040520 bytes level 1

class hfsc 1:65 parent 1:100 leaf 65: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit
Sent 76808971 bytes 457236 pkt (dropped 16755, overlimits 0 requeues 0)
rate 1163Kbit 989pps backlog 0b 2p requeues 0
period 290193 work 76779316 bytes rtwork 1743830 bytes level 0

class prio 99:1 parent 99:
Sent 1543 bytes 11 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0
class prio 99:2 parent 99:
Sent 19693553 bytes 185805 pkt (dropped 6999, overlimits 0 requeues 0) backlog 0b 0p requeues 0
class prio 99:3 parent 99:
Sent 690143 bytes 6804 pkt (dropped 71, overlimits 0 requeues 0) backlog 0b 0p requeues 0
class sfq 3999:1a parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 5942

class sfq 3999:2d parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot -1279

class sfq 3999:3e parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 3072

class sfq 3999:46 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot -1035

class sfq 3999:ae parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot -871

class sfq 3999:c4 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot 1429

class sfq 3999:e3 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot 388

class sfq 3999:ee parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot -582

class sfq 3999:155 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot -1409

class sfq 3999:17c parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot -128

class sfq 3999:1c2 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot 9048

class sfq 3999:1ca parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 6p requeues 0
allot -1156

class sfq 3999:217 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot -90

class sfq 3999:229 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 336

class sfq 3999:234 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 4p requeues 0
allot -1248

class sfq 3999:236 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot -350

class sfq 3999:24d parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot 2602

class sfq 3999:259 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot 12595

class sfq 3999:27e parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 15861

class sfq 3999:28e parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 578

class sfq 3999:2af parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 6p requeues 0
allot -280

class sfq 3999:32b parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 109

class sfq 3999:37e parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 6p requeues 0
allot -1355

class sfq 3999:3ed parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot -1080

class sfq 3999:400 parent 3999:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot -293

class sfq 98:9 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 7590

class sfq 98:88 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 10598

class sfq 98:10c parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 16677

class sfq 98:111 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 9084

class sfq 98:22f parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 3028

class sfq 98:294 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 12054

class sfq 98:2f9 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 23634

class sfq 98:322 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot -28605

class sfq 98:364 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 6056

class sfq 98:3b3 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 4546

class sfq 98:3f9 parent 98:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 30100

class sfq 5001:39a parent 5001:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 24540

class sfq 4002:226 parent 4002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 4542

class sfq 5002:49 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 9084

class sfq 5002:a7 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 9084

class sfq 5002:138 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 7570

class sfq 5002:167 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 4542

class sfq 5002:19c parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 10606

class sfq 5002:20b parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot 2974

class sfq 5002:22a parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 12112

class sfq 5002:256 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 25094

class sfq 5002:289 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 9084

class sfq 5002:2f4 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 3028

class sfq 5002:341 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 7570

class sfq 5002:351 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 8974

class sfq 5002:376 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 6056

class sfq 5002:3a6 parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 3028

class sfq 5002:3bc parent 5002:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 6064

class sfq 5003:68 parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot 4542

class sfq 5003:a7 parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot -24050

class sfq 5003:14d parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 2056

class sfq 5003:1f3 parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 6056

class sfq 5003:27f parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 3036

class sfq 5003:2e8 parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 28716

class sfq 5003:311 parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 13572

class sfq 5003:31a parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 13626

class sfq 5003:361 parent 5003:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 3048

class sfq 5004:1e parent 5004:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 10932

class sfq 5004:101 parent 5004:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 4542

class sfq 5004:12e parent 5004:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 9084

class sfq 5004:156 parent 5004:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 19526

class sfq 5004:179 parent 5004:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 671163409p requeues 0
allot 724

class sfq 5004:1a0 parent 5004:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 19682

class sfq 5004:34c parent 5004:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot -30965

class sfq 4005:134 parent 4005:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 1514

class sfq 12d:47 parent 12d:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 4542

class sfq 12d:a8 parent 12d:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 3p requeues 0
allot 1514

class sfq 12d:ef parent 12d:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 1514

class sfq 12d:368 parent 12d:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 3268

class sfq 65:4e parent 65:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 2p requeues 0
allot 1514

class sfq 65:91 parent 65:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 1514

class sfq 65:286 parent 65:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 10530

class sfq c9:357 parent c9:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 1514

class sfq b:72 parent b:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 1514

class sfq b:1ab parent b:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 13560

class sfq b:20b parent b:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 13572

class sfq b:38e parent b:
(dropped 0, overlimits 0 requeues 0)
backlog 0b 1p requeues 0
allot 1514

#20 lore raw
Quoted 18 lines · Vladimir Ivashchenko
Quoted 10 lines
Quoted 4 lines

All child classes have smaller bursts than the parent. However, there are two sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I don't know HTB internals, perhaps these two classes make the parent class overstretch itself.

As i remember important to keep sum of child rates lower or equal parent rate. Sure ceil of childs must not exceed ceil of parent.
Sometimes i had mess, when i tried to play with quantum value. After all that i switched to HFSC which works for me flawlessly. Maybe we should give more attention to HTB problem with high speeds and help kernel developers spot problem, if there is any.

In case of HFSC my problem is even worse. With 775mbit ceiling configured it is passing over 900mbit in reality. Moreover not having statistics for parent classes makes it difficult to troubleshoot :( I'm 100% sure that it is 900 mbps, I see this on the switch.

Attached is "tc -s -d class show dev bond0" output.

Quoted 8 lines

To calculate total traffic rate:

$ cat hfsc-stat.txt | grep rate | grep Kbit | sed 's/Kbit//' | awk
'{ a=a+$2; } END { print a; }'
906955

Did I misconfigure something ?... How can hfsc go above 775mbit when everything goes via class 1:2 with 775mbit rate & ul ?

Maybe... It's a lot of checking - it seems test cases could be simpler to show the real problem. Anyway, it looks like the sum of m2 of 1:2 children is more than 775Mbit.

Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

By the way, I experience the same "overstretching" with hfsc. In any case, I prefer HTB because it reports statistics of parent classes, unlike hfsc.

Sometimes it happen when some offloading enabled on devices. Check ethtool -k device

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off

Current versions of ethtool should show "generic segmentation offload" too.

I hope you've read the nearby thread "HTB accuracy for high speed", which explains at least partially some problems/bugs, and maybe you'll try some patches too (at last one of them addresses the problem you've reported). Anyway, if you don't find hfsc is better for you I'd be more interested in tracking this on htb test cases yet.

Thanks,
Jarek P.