Skip to contentSkip to conversation

Finally, CBQ nearly completely documented

DISCUSSION bert hubert 4 participants 21 messages Dec 1, 2001 – Dec 10, 2001

lore
Share & tools
lore
#0 lore raw

Hi,

After preparing my talk on CBQ/HTB (http://ds9a.nl/cbq-presentation ), I finally understood how CBQ and filters etc truly work. And I wrote it down. Check out the Linux Advanced Routing & Shaping HOWTO, it's changed a lot!

Especially this part is very new, please check it for mistakes and inconsistencies:

http://ds9a.nl/2.4Routing/HOWTO//cvs/2.4routing/output/2.4routing-9.html

I even got 'split' and 'defmap' figured out, which should be a first. There is not a single other page online that tells you correctly what these do.

One thing - does *anybody* understand how hash tables work in tc filter, and what they do? Furthermore, I could use some help with the tc filter police things.

So if you do understand how these work, please drop me a line.

Thanks!

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#1 lore raw

CBQ and all other qdiscs now REALLY completely documented (almost!)

Quoted 3 lines · bert hubert

One thing - does *anybody* understand how hash tables work in tc filter, and what they do? Furthermore, I could use some help with the tc filter police things.

Thanks to Andreas Steinmetz and David Sauer, tc hash tables are now documented as well, thanks!

See:

http://ds9a.nl/2.4Routing/HOWTO//cvs/2.4routing/output/2.4routing-12.html

And then 'Hashing filters for very fast massive filtering'.

I also finished documenting all parameters for TBF, CBQ, SFQ, PRIO, bfifo, pfifo and pfifo_fast. All queues in the Linux kernel are now described in the Linux Advanced Routing & Shaping HOWTO, which can be found on

                          http://ds9a.nl/2.4Routing

I want to send this off to the LDP and Freshmeat somewhere next week, I *would really* like people who are knowledgeable about this subject (this means you, ANK & Jamal 8) ) to read through this.

This HOWTO is rapidly becoming the perceived authoritative source for traffic control in linux (google on 'Linux Routing' finds it), it might as well be right! So if you have any time at all, check the parts you know about. I expect mistakes.

The parts of the table of contents that document stuff in the kernel not documented elsewhere:

9. Queueing Disciplines for Bandwidth Management
9.1 Queues and Queueing Disciplines explained
9.2 Simple, classless Queueing Disciplines
9.2.1 pfifo_fast
9.2.1.1 Parameters & usage
9.2.2 Token Bucket Filter
9.2.2.1 Parameters & usage
9.2.2.2 Sample configuration
9.2.3 Stochastic Fairness Queueing
9.2.3.1 Parameters & usage
9.2.3.2 Sample configuration
9.3 Advice for when to use which queue
9.4 Classful Queueing Disciplines
9.4.1 Flow within classful qdiscs & classes
9.4.2 The qdisc family: roots, handles, siblings and parents
9.4.2.1 How filters are used to classify traffic 9.4.2.2 How packets are dequeued to the hardware
9.4.3 The PRIO qdisc
9.4.3.1 PRIO parameters & usage
9.4.3.2 Sample configuration
9.4.4 The famous CBQ qdisc
9.4.4.1 CBQ shaping in detail
9.4.4.2 CBQ classful behaviour
9.4.4.3 CBQ parameters that determine link sharing & borrowing 9.4.4.4 Sample configuration
9.4.4.5 Other CBQ parameters: split & defmap
9.4.5 Hierarchical Token Bucket
9.4.5.1 Sample configuration
9.5 Classifying packets with filters
9.5.1 Some simple filtering examples
9.5.2 All the filtering commands you will normally need

(...)

12. Advanced filters for (re-)classifying packets 12.1 The "u32" classifier
12.1.1 U32 selector
12.1.2 General selectors
12.1.3 Specific selectors
12.2 The "route" classifier
12.3 Policing filters
12.4 Hashing filters for very fast massive filtering

(...)

14. Advanced & less common queueing disciplines
14.1 bfifo/pfifo

      14.1.1 Parameters & usage

14.2 Clark-Shenker-Zhang algorithm (CSZ)
14.3 DSMARK
14.3.1 Introduction
14.3.2 What is Dsmark related to?
14.3.3 Differentiated Services guidelines
14.3.4 Working with Dsmark
14.3.5 How SCH_DSMARK works.
14.3.6 TC_INDEX Filter
14.4 Ingress policer qdisc
14.5 Random Early Drop (RED)
14.6 VC/ATM emulation
14.7 Weighted Round Robin (WRR)

The only thing left to document are Policing filters.

Regards,

bert hubert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#2 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented (almost!)

Quoted 25 lines · bert hubert
Quoted 3 lines · bert hubert

One thing - does *anybody* understand how hash tables work in tc filter, and what they do? Furthermore, I could use some help with the tc filter police things.

Thanks to Andreas Steinmetz and David Sauer, tc hash tables are now documented as well, thanks!

See:

http://ds9a.nl/2.4Routing/HOWTO//cvs/2.4routing/output/2.4routing-12.html

And then 'Hashing filters for very fast massive filtering'.

I also finished documenting all parameters for TBF, CBQ, SFQ, PRIO, bfifo, pfifo and pfifo_fast. All queues in the Linux kernel are now described in the Linux Advanced Routing & Shaping HOWTO, which can be found on

                          http://ds9a.nl/2.4Routing

I want to send this off to the LDP and Freshmeat somewhere next week, I *would really* like people who are knowledgeable about this subject (this means you, ANK & Jamal 8) ) to read through this.

Quoted 7 lines

This HOWTO is rapidly becoming the perceived authoritative source for traffic control in linux (google on 'Linux Routing' finds it), it might as well be right! So if you have any time at all, check the parts you know about. I expect mistakes.

The parts of the table of contents that document stuff in the kernel not documented elsewhere:

"not documented elsewhere" comes out rude. Werner and I (and even Alexey when he was in the mood -- and i have seen some good documentation by other people as well) have spent numerous hours documenting, presenting and answering questions on mailing lists at times

Sample docs that i was personally involved in:
ftp://icaftp.epfl.ch/pub/linux/diffserv/misc/dsid-01.txt.gz You need to introduce the big picture to the user.
and what is wrong with the definitions used in
http://www.davin.ottawa.on.ca/ols/img10.htm that forced you to introduce your own?
Actually, the big picture is:
http://www.davin.ottawa.on.ca/ols/img9.htm
Also
http://www.linuxjournal.com/article.php?sid=3369
(was written in 98 but got published in 99)

Now despite all the bitching above, i think your efforts are noble.

[My complaints about your style is you often are trying to present facts by using opinions. For example despite a lot of effort in the past to explain ingress qdisc to you in the past and, pointing you to very good documentation from CISCO you still ended using your opinions on what you thought it should be;->
My scanning of the document shows opinions still posing as miscontrued facts. It is improving compared to what i saw last when we discussed ingress. Let me clarify one thing in this email; i'll read what you have later.

Lets start by your description of TC_PRIO and TOS mappings etc:
Your descriptions of these values is insufficient. Consider this a tutorial and reword it as you wish but please avoid opinions. Ok here's clarification, this applies to both prio, default fifo 3 band queueing and CBQ defaultmap classification; applies to both packets being forwarded as well as locally generated:

First Step:
===========

Define TOS: This is a 4 bit value used as defined in RFC 1349.

0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| | | |
| PRECEDENCE | TOS | MBZ |
| | | |
+-----+-----+-----+-----+-----+-----+-----+-----+

Then define the values possible as:

                    1000   --   minimize delay
                    0100   --   maximize throughput
                    0010   --   maximize reliability
                    0001   --   minimize monetary cost
                    0000   --   normal service

Look at RFC 1349 for typical values used by different applications Then of course note that RFC 1349 is obsoleted by RFC 2474 (yes, you can weep);

Having said all that:

Linux remaps packets incoming with different values to some internal value; the colum "mapped to" shows the internal mapping

8value(hex) TOS(dec) mapped to(dec)
----------------------------------

0x0              0      0
                 1      7
                 2      0
                 3      0
                 4      2
                 5      2
                 6      2
                 7      2
0x10             8      6
                 9      6
                10      6
                11      6
                12      2
                13      2
                14      2
                15      2

Fill in the "8value(hex)" column gaps using the bitmap from RFC1349 for the 8 bits; These are the values ou would see with tcpdump -vvv I filled the two easiest ones i could compute in my head.

Second step:

Take the default priority map:
1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
This applies for both default prio and the 3-band FIFO queue. Note the queue map fitted on the last column

8 but value TOS mapped to queue map
---------------------------------------------

0x0              0      0              1
                 1      7              2
                 2      0              2
                 3      0              2
                 4      2              1
                 5      2              2
                 6      2              0
                 7      2              0
0x10             8      6              1
                 9      6              1
                10      6              1
                11      6              1
                12      2              1
                13      2              1
                14      2              1
                15      2              1

Queue 0 gets processed first then queue 1 then queue 2. In the strict priority processing such as in prio or default 3 band sched, queue 0 is processed until no more packets are left, then queue1 etc. This could result in starvation. You could avoid starvation by inserting a TBF in a prio; limit the size of the fifo in a class or use CBQ configured as WRR.
I hope the above explains why you have to recreate the priomap everytime you change the number of bands. You used the word "probably" which is wrong. The proper word is "MUST".
What i think would be useful for you to do is describe some of the vlaues used by some applications (RFC 1349 cut-n-paste job would help).

cheers,
jamal

#3 lore raw

further CBQ/tc documentation ds9a.nl/lartc/manpages

In reply to bert hubert
Quoted 7 lines · jamal
Quoted 2 lines

The parts of the table of contents that document stuff in the kernel not documented elsewhere:

"not documented elsewhere" comes out rude. Werner and I (and even Alexey when he was in the mood -- and i have seen some good documentation by other people as well) have spent numerous hours documenting, presenting and answering questions on mailing lists at times

True. I should have worded that better but I lost sight of politeness due to my great enthusiasm at finally understanding everything. Some parts required literally *hours* of digging through sources and disembodied slides - presentations lose something without a speaker.

Quoted 2 lines

Sample docs that i was personally involved in:
ftp://icaftp.epfl.ch/pub/linux/diffserv/misc/dsid-01.txt.gz

These days I understand this document, but I didn't used to. That might be because I'm thick, though.

Quoted 4 lines

You need to introduce the big picture to the user.
and what is wrong with the definitions used in
http://www.davin.ottawa.on.ca/ols/img10.htm that forced you to introduce your own?

I've since moved to this terminology. Please also see the manpages I'm writing at
http://ds9a.nl/lartc/manpages

Quoted 5 lines

Actually, the big picture is:
http://www.davin.ottawa.on.ca/ols/img9.htm
Also
http://www.linuxjournal.com/article.php?sid=3369
(was written in 98 but got published in 99)

Google is surely to be praised - I had found all these links already. But to
summarize: stuff is out there.

Quoted 5 lines

[My complaints about your style is you often are trying to present facts by using opinions. For example despite a lot of effort in the past to explain ingress qdisc to you in the past and, pointing you to very good documentation from CISCO you still ended using your opinions on what you thought it should be;->

I really didn't understand how everything worked back then, sadly. I do now, hopefully.

Quoted 3 lines

My scanning of the document shows opinions still posing as miscontrued facts. It is improving compared to what i saw last when we discussed ingress. Let me clarify one thing in this email; i'll read what you have later.

Some stuff remains from that time, am working on removing it. My current efforts is writing the manpages and getting them 100% right and devoid of opinion.

Once they are finished & reviewed, I'm 'backporting' the insight to the HOWTO, which will then lose a lot of content and instead refer to the manpages.

Quoted 3 lines

Lets start by your description of TC_PRIO and TOS mappings etc:
Your descriptions of these values is insufficient. Consider this a tutorial and reword it as you wish but please avoid opinions.

Will do, it makes sense now.

Quoted 3 lines

Look at RFC 1349 for typical values used by different applications Then of course note that RFC 1349 is obsoleted by RFC 2474 (yes, you can weep);

That confused me greatly, yes.

Quoted 2 lines

What i think would be useful for you to do is describe some of the vlaues used by some applications (RFC 1349 cut-n-paste job would help).

Thanks. I'm working on making the HOWTO more factual and the manpages 100% factual. I'm always happy with critiques.

Regards,

bert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#4 lore raw

Re: further CBQ/tc documentation ds9a.nl/lartc/manpages

For starters, i think you need a defintions sections. Look at:
http://www.ietf.org/internet-drafts/draft-ietf-diffserv-model-06.txt

(eg what is a shaper etc and how trhings are placed together). At least that will ensure that you dont sday things like "Prio cant shape".

It is a good model but may be insufficient given Linux TCs capabilities. Email me when unsure.

Some other things:
- In your comment "Do not confuse this classless simple qdisc with the classful PRIO one!". This is misleading:
the default 3 band FIFO queue is conceptually the same as the default prio qdisc (the priomaps are identical). 3 bands; same prioritization schemes.
- You really need to fix ingress section:
it works for both forwarding and packets coming in to local sockets.
More importantly, It takes advantages of _all_ filter schemes available for TC as well as the policing functionality (which sadly seemed to have been replicated by someone in netfilter, wrongly if i may add ;->).
- You keep saying "reodering" -- dont know what that means. Reordering is generally considered a Bad Thing(tm).

- your description of the "peakrate" (same in TBF as well as policing)

Well captured. It took ages to get this into peoples heads. This also applies to CBQ.

- your description of "MTU"
Not very good description:
This is just what it literally says; maximum transmit unit; A packet larger than this will be dropped. Default is 2K. For ethernet, MTUs of 1500 bytes, this is fine; however, you should put a cautionary statement here in regards to people having MTUs smaller than 2K (example the lo device); they might find that all their packets greater than 2K being dropped.

More later if dont get distracted.

cheers,
jamal

#5 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented (almost!)

In reply to bert hubert
Quoted 5 lines · jamal

Linux remaps packets incoming with different values to some internal value; the colum "mapped to" shows the internal mapping

8value(hex) TOS(dec) mapped to(dec)
----------------------------------

Quoted 16 lines
0x0              0      0
                 1      7
                 2      0
                 3      0
                 4      2
                 5      2
                 6      2
                 7      2
0x10             8      6
                 9      6
                10      6
                11      6
                12      2
                13      2
                14      2
                15      2

I find this tos2prio table in the kernel (2.5.x), which is somewhat different than your table:

0        TC_PRIO_BESTEFFORT,     0
1        TC_PRIO_(FILLER),       1
2        TC_PRIO_BESTEFFORT,     0
3        TC_PRIO_(BESTEFFORT),   0
4        TC_PRIO_BULK,           2
5        TC_PRIO_(BULK),         2
6        TC_PRIO_BULK,           2
7        TC_PRIO_(BULK),         2
8        TC_PRIO_INTERACTIVE,    6
9        TC_PRIO_(INTERACTIVE),  6
10       TC_PRIO_INTERACTIVE,    6
11       TC_PRIO_(INTERACTIVE),  6
12       TC_PRIO_INTERACTIVE_BULK,       4
13       TC_PRIO_(INTERACTIVE_BULK),     4
14       TC_PRIO_INTERACTIVE_BULK,       4
15       TC_PRIO_(INTERACTIVE_BULK)      4
Quoted 13 lines

Fill in the "8value(hex)" column gaps using the bitmap from RFC1349 for the 8 bits; These are the values ou would see with tcpdump -vvv I filled the two easiest ones i could compute in my head.

Second step:

Take the default priority map:
1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
This applies for both default prio and the 3-band FIFO queue. Note the queue map fitted on the last column

8 but value TOS mapped to queue map
---------------------------------------------

Quoted 16 lines
0x0              0      0              1
                 1      7              2
                 2      0              2
                 3      0              2
                 4      2              1
                 5      2              2
                 6      2              0
                 7      2              0
0x10             8      6              1
                 9      6              1
                10      6              1
                11      6              1
                12      2              1
                13      2              1
                14      2              1
                15      2              1

I've changed this table to:
TOS Bits Means Linux Priority Band
------------------------------------------------------------

0x0     0     Normal Service           0 Best Effort     1
0x2     1     Minimize Monetary Cost   1 Filler          2
0x4     2     Maximize Reliability     0 Best Effort     1
0x6     3     mmc+mr                   0 Best Effort     1
0x8     4     Maximize Throughput      2 Bulk            2
0xa     5     mmc+mt                   2 Bulk            2
0xc     6     mr+mt                    2 Bulk            2
0xe     7     mmc+mr+mt                2 Bulk            2
0x10    8     Minimize Delay           6 Interactive     0
0x12    9     mmc+md                   6 Interactive     0
0x14    10    mr+md                    6 Interactive     0
0x16    11    mmc+mr+md                6 Interactive     0
0x18    12    mt+md                    4 Int. Bulk       1
0x1a    13    mmc+mt+md                4 Int. Bulk       1
0x1c    14    mr+mt+md                 4 Int. Bulk       1
0x1e    15    mmc+mr+mt+md             4 Int. Bulk       1

http://ds9a.nl/lartc/HOWTO/cvs/2.4routing/output/2.4routing-9.html#ss9.2

Your table appears to imply that a Maximum Reliability, Mininum Delay packet, TOS bits=9, gets mapped to band 1, not 0, which would not make sense.

Laying it out like this, which does appear how it works, does mean that you can specify priorities in the priomap which do not correspond to possible TOS values.

Is it possible at all to set skb->priority from userspace without going through the tos2prio mapping?

CBQ can use the skb->priority to classify:

        /*
         *  Step 1. If skb->priority points to one of our classes, use it.
         */
        if (TC_H_MAJ(prio^sch->handle) == 0 &&
            (cl = cbq_class_lookup(q, prio)) != NULL)
                        return cl;

But to do this, you would need to be able to set skb->priority to a 32bit number:

include/linux/pkt_sched.h:#define TC_H_MAJ_MASK (0xFFFF0000U) include/linux/pkt_sched.h:#define TC_H_MAJ(h) ((h)&TC_H_MAJ_MASK)

I can't find where you would do this, any clues?

Thanks again for taking the time to help me.

Regards,

bert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#6 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented (almost!)

Quoted 78 lines · bert hubert
Quoted 5 lines · jamal

Linux remaps packets incoming with different values to some internal value; the colum "mapped to" shows the internal mapping

8value(hex) TOS(dec) mapped to(dec)
----------------------------------

Quoted 16 lines
0x0              0      0
                 1      7
                 2      0
                 3      0
                 4      2
                 5      2
                 6      2
                 7      2
0x10             8      6
                 9      6
                10      6
                11      6
                12      2
                13      2
                14      2
                15      2

I find this tos2prio table in the kernel (2.5.x), which is somewhat different than your table:

0        TC_PRIO_BESTEFFORT,     0
1        TC_PRIO_(FILLER),       1
2        TC_PRIO_BESTEFFORT,     0
3        TC_PRIO_(BESTEFFORT),   0
4        TC_PRIO_BULK,           2
5        TC_PRIO_(BULK),         2
6        TC_PRIO_BULK,           2
7        TC_PRIO_(BULK),         2
8        TC_PRIO_INTERACTIVE,    6
9        TC_PRIO_(INTERACTIVE),  6
10       TC_PRIO_INTERACTIVE,    6
11       TC_PRIO_(INTERACTIVE),  6
12       TC_PRIO_INTERACTIVE_BULK,       4
13       TC_PRIO_(INTERACTIVE_BULK),     4
14       TC_PRIO_INTERACTIVE_BULK,       4
15       TC_PRIO_(INTERACTIVE_BULK)      4
Quoted 13 lines

Fill in the "8value(hex)" column gaps using the bitmap from RFC1349 for the 8 bits; These are the values ou would see with tcpdump -vvv I filled the two easiest ones i could compute in my head.

Second step:

Take the default priority map:
1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
This applies for both default prio and the 3-band FIFO queue. Note the queue map fitted on the last column

8 but value TOS mapped to queue map
---------------------------------------------

Quoted 16 lines
0x0              0      0              1
                 1      7              2
                 2      0              2
                 3      0              2
                 4      2              1
                 5      2              2
                 6      2              0
                 7      2              0
0x10             8      6              1
                 9      6              1
                10      6              1
                11      6              1
                12      2              1
                13      2              1
                14      2              1
                15      2              1

I've changed this table to:
TOS Bits Means Linux Priority Band
------------------------------------------------------------

Quoted 17 lines
0x0     0     Normal Service           0 Best Effort     1
0x2     1     Minimize Monetary Cost   1 Filler          2
0x4     2     Maximize Reliability     0 Best Effort     1
0x6     3     mmc+mr                   0 Best Effort     1
0x8     4     Maximize Throughput      2 Bulk            2
0xa     5     mmc+mt                   2 Bulk            2
0xc     6     mr+mt                    2 Bulk            2
0xe     7     mmc+mr+mt                2 Bulk            2
0x10    8     Minimize Delay           6 Interactive     0
0x12    9     mmc+md                   6 Interactive     0
0x14    10    mr+md                    6 Interactive     0
0x16    11    mmc+mr+md                6 Interactive     0
0x18    12    mt+md                    4 Int. Bulk       1
0x1a    13    mmc+mt+md                4 Int. Bulk       1
0x1c    14    mr+mt+md                 4 Int. Bulk       1
0x1e    15    mmc+mr+mt+md             4 Int. Bulk       1

Yes, sorry the last 4 are int_bulk (value 4) and not just bulk (2). good eye. You are still abusing the word TOS. Thats only 4 bits not 8; Use the terminology from RFC1349 at least.

Quoted 6 lines

http://ds9a.nl/lartc/HOWTO/cvs/2.4routing/output/2.4routing-9.html#ss9.2

Your table appears to imply that a Maximum Reliability, Mininum Delay packet, TOS bits=9, gets mapped to band 1, not 0, which would not make sense.

This is the priomap: 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1 It says 1 is the right value

Quoted 4 lines

Laying it out like this, which does appear how it works, does mean that you can specify priorities in the priomap which do not correspond to possible TOS values.

You cant remap the 3 band scheduler trivially, but you should be able to replace it with a default prio qdisc get exactly the same behavior and use whatever map you want (eg your 0 to 1 substitution for TOS 1001)

Quoted 3 lines

Is it possible at all to set skb->priority from userspace without going through the tos2prio mapping?

SO_PRIORITY socket option is doable; you have to be root.

Quoted 1 line

CBQ can use the skb->priority to classify:

so do prio and pfifo_fast (as i am sure you are aware)

Quoted 10 lines
        /*
         *  Step 1. If skb->priority points to one of our classes, use it.
         */
        if (TC_H_MAJ(prio^sch->handle) == 0 &&
            (cl = cbq_class_lookup(q, prio)) != NULL)
                        return cl;

But to do this, you would need to be able to set skb->priority to a 32bit number:

Cant think of a straight way to do this .... Alexey would know,

cheers,
jamal

#7 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

Hello!

Quoted 5 lines
Quoted 3 lines

But to do this, you would need to be able to set skb->priority to a 32bit number:

Cant think of a straight way to do this .... Alexey would know,

SO_PRIORITY. Or I did not follow you?

Alexey

#8 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

Quoted 10 lines · On Sun, 9 Dec 2001 [email protected] wrote:

Hello!

Quoted 5 lines
Quoted 3 lines

But to do this, you would need to be able to set skb->priority to a 32bit number:

Cant think of a straight way to do this .... Alexey would know,

SO_PRIORITY. Or I did not follow you?

So priority limits the size of skb->priority to be from 0..6; this wont work with that check in cbq.

cheers,
jamal

#9 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

Hello!

Quoted 2 lines

So priority limits the size of skb->priority to be from 0..6; this wont work with that check in cbq.

No, it does not. Values different of "low prio" defaults (0..6) are not allowed to user without privileges by evident reasons. User with correspoding capability may direct traffic to any class.

Alexey

#10 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

In reply to [email protected]
Quoted 6 lines · jamal
Quoted 3 lines
Quoted 1 line

Cant think of a straight way to do this .... Alexey would know,

SO_PRIORITY. Or I did not follow you?

So priority limits the size of skb->priority to be from 0..6; this wont work with that check in cbq.

No, only IP_TOS does so.

Regards,

bert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#11 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

Quoted 10 lines · bert hubert
Quoted 6 lines · jamal
Quoted 3 lines
Quoted 1 line

Cant think of a straight way to do this .... Alexey would know,

SO_PRIORITY. Or I did not follow you?

So priority limits the size of skb->priority to be from 0..6; this wont work with that check in cbq.

No, only IP_TOS does so.

probaly ip precedence. Have you tried this or you are following what the man pages say?

cheers,
jamal

#12 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

In reply to bert hubert
Quoted 7 lines · jamal
Quoted 4 lines
Quoted 2 lines

So priority limits the size of skb->priority to be from 0..6; this wont work with that check in cbq.

No, only IP_TOS does so.

probaly ip precedence. Have you tried this or you are following what the man pages say?

I have been living in the source for quite a while now - see ip_setsockopt() in net/ipv4/ip_sockglue.c.

Regards,

bert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#13 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

Quoted 4 lines · bert hubert
Quoted 3 lines · jamal
Quoted 3 lines
Quoted 2 lines

So priority limits the size of skb->priority to be from 0..6; this wont work with that check in cbq.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

No, only IP_TOS does so.

probaly ip precedence. Have you tried this or you are following what the man pages say?

I have been living in the source for quite a while now - see ip_setsockopt() in net/ipv4/ip_sockglue.c.

Thats the wrong place to look. Look instead at:
net/core/sock.c
I got it; non root is limited to 0..6; root can set the full 32 bit range.

cheers,
jamal

#14 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented

In reply to jamal
Quoted 5 lines · On Sun, Dec 09, 2001 at 09:14:46PM +0300, [email protected] wrote:
Quoted 3 lines
Quoted 2 lines

But to do this, you would need to be able to set skb->priority to a 32bit number:

Cant think of a straight way to do this .... Alexey would know,

SO_PRIORITY. Or I did not follow you?

Ah yes, thanks, that sets sk->priority which later sets skb->priority.

Regards,

bert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#15 lore raw

CBQ MANPAGE: I hear the theme of '2001, A Space Odyssey'

In reply to jamal

... to the sound of 'Also sprach Zarathustra':

After weeks of social deprivation and much digging through heaps of code, I
bring you

tc-cbq.8

The CBQ manpage. Nearly 2500 words, 8 printed pages, of nearly
unintelligible gobledygook, explaining mostly how CBQ works.

It is part of the Linux Advanced Routing & Traffic Control documentation project which contains a HOWTO, a mailinglist, an IRC channel and now manpages:

                            http://ds9a.nl/lartc

I want to thank Jamal for stubbornly straightening me out when I use messy language and explaining how things work. The errors are mine though.

I *implore* ANK and others to read through this. I'm about exhausted and running out of time (need to get on with work), and have a hard time figuring out the exact details of the CBQ link sharing algorithm. I need help, so to speak. The manpage indicates where.

Thanks for your attention. Please find tc-cbq.8 attached.

Regards,

bert hubert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

.TH CBQ 8 "8 December 2001" "iproute2" "Linux"
.SH NAME
CBQ \- Class Based Queueing
.SH SYNOPSIS
.B tc qdisc ... dev
dev
.B ( parent
classid
.B | root) [ handle
major:
.B ] cbq avpkt
bytes
.B bandwidth
rate
.B [ cell
bytes
.B ] [ ewma
log
.B ] [ mpu
bytes
.B ]

.B tc class ... dev
dev
.B parent
major:[minor]
.B [ classid
major:minor
.B ] cbq allot
bytes
.B [ bandwidth
rate
.B ] [ rate
rate
.B ] prio
priority
.B [ weight
weight
.B ] [ minburst
packets
.B ] [ maxburst
packets
.B ] [ ewma
log
.B ] [ cell
bytes
.B ] avpkt
bytes
.B [ mpu
bytes
.B ] [ bounded isolated ] [ split
handle
.B & defmap
defmap
.B ] [ estimator
interval timeconstant
.B ]

.SH DESCRIPTION
Class Based Queueing is a classful qdisc that implements a rich linksharing hierarchy of classes. It contains shaping elements as well as prioritizing capabilities. Shaping is performed using link idle time calculations based on the timing of dequeue events and underlying link bandwidth.

.SH SHAPING ALGORITHM
Shaping is done using link idle time calculations, and actions taken if these calculations deviate from set limits.

When shape a 10mbit/s connection to 1mbit/s, the link will
be idle 90% of the time. If it isn't, it needs to be throttled so that it IS idle 90% of the time.

Quoted 1 line

From the kernel's perspecive, this is hard to measure, so CBQ instead

derives the idle time from the number of microseconds that elapse between requests from the hardware layer for more data. Combined with the knowledge of packet sizes, this is used to approximate how full or empty the link is.

This is rather circumspect and doesn't always arrive at proper results. For example, what is the actual link speed of an interface that is not really able to transmit the full 100mbit/s of data, perhaps because of a badly implemented driver? A PCMCIA network card will also never achieve 100mbit/s because of the way the bus is designed - again, how do we calculate the idle time?

The physical link bandwidth may be ill defined in case of not-quite-real network devices like PPP over Ethernet or PPTP over TCP/IP. The effective bandwidth in that case is probably determined by the efficiency of pipes to userspace - which not defined.

During operations, the effective idletime is measured using an exponential weighted moving average (EWMA), which considers recent packets to be exponentially more important than past ones. The unix loadaverage is calculated in the same way.

The calculated idle time is substracted from the EWMA measured one, the resulting number is called 'avgidle'. A perfectly loaded link has an avgidle of zero: packets arrive exactly at the calculated interval.

An overloaded link has a negative avgidle and if it gets too negative, CBQ throttles and is then 'overlimit'.

Conversely, an idle link might amass a huge avgidle, which would then allow infinite bandwidths after a few hours of silence. To prevent this, avgidle is capped at
.B maxidle.

If overlimit, in theory, the CBQ could throttle itself for exactly the amount of time that was calculated to pass between packets, and then pass one packet, and throttle again. Due to timer resolution constraints, this may not be feasible, see the
.B minburst
parameter below.

.SH CLASSIFICATION
Within the one CBQ instance many classes may exist. Each of these classes contains another qdisc, by default
.BR tc-pfifo (8).

When enqueueing a packet, CBQ starts at the root and uses various methods to determine which class should receive the data. If a verdict is reached, this process is repeated for the recipient class which might have further means of classifying traffic to its children, if any.

CBQ has the following methods available to classify a packet to any child classes.
.TP
(i)
.B skb->priority class encoding.
Can be set from userspace by an application with the
.B IP_PRIO
setsockopt.
The
.B skb->priority class encoding
only applies if the skb->priority holds a major:minor handle of an existing class within this qdisc.
.TP
(ii)
tc filters attached to the class.
.TP
(iii)
The defmap of a class, as set with the
.B split & defmap
parameters. The defmap may contain instructions for each possible Linux packet priority.

.P
Each class also has a
.B level.
Leaf nodes, attached to the bottom of the class hierarchy, have a level of 0. .SH CLASSIFICATION ALGORITHM

Classification is a loop, which terminates when a leaf class is found. At any point the loop may jump to the fallback algorithm.

The loop consists of the following steps:
.TP
(i)
If the packet is generated locally and has a valid classid encoded within its .B skb->priority,
choose it and terminate.

.TP
(ii)
Consult the tc filters, if any, attached to this child. If these return a class which is not a leaf class, restart loop from the class returned. If it is a leaf, choose it and terminate.
.TP
(iii)
If the tc filters did not return a class, but did return a classid, try to find a class with that id within this qdisc.
Check if the found class is of a lower
.B level
than the current class. If so, and the returned class is not a leaf node, restart the loop at the found class. If it is a leaf node, terminate. If we found an upward reference to a higher level, enter the fallback algorithm.
.TP
(iv)
If the tc filters did not return a class, nor a valid reference to one, consider the minor number of the reference to be the priority. Retrieve a class from the defmap of this class for the priority. If this did not contain a class, consult the defmap of this class for the
.B BEST_EFFORT
class. If this is an upward reference, or no
.B BEST_EFFORT
class was defined,
enter the fallback algorithm. If a valid class was found, and it is not a leaf node, restart the loop at this class. If it is a leaf, choose it and terminate. If
neither the priority distilled from the classid, nor the
.B BEST_EFFORT
priority yielded a class, enter the fallback algorithm.
.P
The fallback algorithm resides outside of the loop and is as follows. .TP
(i)
Consult the defmap of the class at which the jump to fallback occured. If the defmap contains a class for the
.B
priority
of the class (which is related to the TOS field), choose this class and terminate.
.TP
(ii)
Consult the map for a class for the
.B BEST_EFFORT
priority. If found, choose it, and terminate.
.TP
(iii)
Choose the class at which breakout to the fallback algorithm occured. Terminate. .P
The packet is enqueued to the class which was chosen when either algorithm terminated. It is therefore possible for a packet to be enqueued *not* at a leaf node, but in the middle of the hierarchy.

.SH LINK SHARING ALGORITHM
When dequeueing for sending to the network device, CBQ decides which of its classes will be allowed to send. It does so with a Weighted Round Robin process in which each class with packets gets a chance to send in turn. The WRR process starts by asking the highest priority classes (lowest numerically - highest semantically) for packets, and will continue to do so until they have no more data to offer, in which case the process repeats for lower priorities.

.B CERTAINTY ENDS HERE, ANK PLEASE HELP

Each class is not allowed to send at length though - they can only dequeue a configurable amount of data during each round.

If a class is about to go overlimit, and it is not
.B bounded
it will try to borrow avgidle from siblings that are not
.B isolated.
This process is repeated from the bottom upwards. If a class is unable to borrow enough avgidle to send a packet, it is throttled and not asked for a packet for enough time for the avgidle to increase above zero.

.B I REALLY NEED HELP FIGURING THIS OUT. REST OF DOCUMENT IS PRETTY CERTAIN .B AGAIN.

.SH QDISC
The root qdisc of a CBQ class tree has the following parameters:

.TP
parent major:minor | root
This mandatory parameter determines the place of the CBQ instance, either at the
.B root
of an interface or within an existing class.
.TP
handle major:
Like all other qdiscs, the CBQ can be assigned a handle. Should consist only
of a major number, followed by a colon. Optional.
.TP
avpkt bytes
For calculations, the average packet size must be known. It is silently capped
at a minimum of 2/3 of the interface MTU. Mandatory.
.TP
bandwidth rate
To determine the idle time, CBQ must know the bandwidth of your underlying
physical interface, or parent qdisc. This is a vital parameter, more about it
later. Mandatory.
.TP
cell
The cell size determines he granularity of packet transmission time calculations. Has a sensible default. .TP
mpu
A zero sized packet may still take time to transmit. This value is the lower
cap for packet transmission time calculations - packets smaller than this value
are still deemed to have this size. Defaults to zero.
.TP
ewma log
When CBQ needs to measure the average idle time, it does so using an
Exponentially Weighted Moving Average which smoothes out measurements into
a moving average. The EWMA LOG determines how much smoothing occurs. Defaults
to 5. Lower values imply greater sensitivity. Must be between 0 and 31.
.P
A CBQ qdisc does not shape out of its own accord. It only needs to know certain parameters about the underlying link. Actual shaping is done in classes.

.SH CLASSES
Classes have a host of parameters to configure their operation.

.TP
parent major:minor
Place of this class within the hierarchy. If attached directly to a qdisc and not to another class, minor can be omitted. Mandatory.
.TP
classid major:minor
Like qdiscs, classes can be named. The major number must be equal to the major number of the qdisc to which it belongs. Optional, but needed if this class is going to have children.
.TP
weight weight
When dequeueing to the interface, classes are tried for traffic in a round-robin fashion. Classes with a higher configured qdisc will generally have more traffic to offer during each round, so it makes sense to allow it to dequeue more traffic. All weights under a class are normalized, so only the ratios matter. Defaults to the configured rate, unless the priority of this class is maximal, in which case it is set to 1.
.TP
allot bytes
Allot specifies how many bytes a qdisc can dequeue
during each round of the process. This parameter is weighted using the renormalized class weight described above.

.TP
priority priority
In the round-robin process, classes with the lowest priority field are tried for packets first. Mandatory.

.TP
rate rate
Maximum rate this class and all its children combined can send at. Mandatory.

.TP
bandwidth rate
This is different from the bandwidth specified when creating a CBQ disc. Only used to determine maxidle and offtime, which are only calculated when specifying maxburst or minburst. Mandatory if specifying maxburst or minburst.

.TP
maxburst
This number of packets is used to calculate maxidle so that when avgidle is at maxidle, this number of average packets can be burst before avgidle drops to 0. Set it higher to be more tolerant of bursts. You can't set maxidle directly, only via this parameter.

.TP
minburst
As mentioned before, CBQ needs to throttle in case of overlimit. The ideal solution is to do so for exactly the calculated idle time, and pass 1 packet. However, Unix kernels generally have a hard time scheduling events shorter than 10ms, so it is better to throttle for a longer period, and then pass minburst packets in one go, and then sleep minburst times longer.

The time to wait is called the offtime. Higher values of minburst lead to more accurate shaping in the long term, but to bigger bursts at millisecond timescales.

.TP
minidle
If avgidle is below 0, we are overlimits and need to wait until avgidle will be big enough to send one packet. To prevent a sudden burst from shutting down the link for a prolonged period of time, avgidle is reset to minidle if it gets too low.

Minidle is specified in negative microseconds, so 10 means that avgidle is capped at -10us.

.TP
bounded
Signifies that this class will not borrow bandwidth from its siblings. .TP
isolated
Means that this class will not borrow bandwidth to its siblings

.TP
split major:minor & defmap bitmap[/bitmap]
If consulting filters attached to a class did not give a verdict, CBQ can also classify based on the packet's priority. There are 16 priorities available, numbered from 0 to 15.

The defmap specifies which priorities this class wants to receive, specified as a bitmap. The Least Significant Bit corresponds to priority zero. The
.B split
parameter tells CBQ at which class the decision must be made, which should be a (grand)parent of the class you are adding.

As an example, 'tc class add ... classid 10:1 cbq .. split 10:0 defmap c0' configures class 10:0 to send packets with priorities 6 and 7 to 10:1.

The complimentary configuration would then
be: 'tc class add ... classid 10:2 cbq ... split 10:0 defmap 3f' Which would send all packets 0, 1, 2, 3, 4 and 5 to 10:1.
.TP
estimator interval timeconstant
CBQ can measure how much bandwidth each class is using, which tc filters can use to classify packets with. In order to determine the bandwidth it uses a very simple estimator that measures once every
.B interval
microseconds how much traffic has passed. This again is a EWMA, for which the time constant can be specified, also in microseconds. The .B time constant
corresponds to the sluggishness of the measurement or, conversely, to the sensitivity of the average to short bursts. Higher values mean less sensitivity.

.SH SOURCES
.TP
o
Sally Floyd and Van Jacobson, "Link-sharing and Resource Management Models for Packet Networks", IEEE/ACM Transactions on Networking, Vol.3, No.4, 1995

.TP
o
Sally Floyd, "Notes on CBQ and Guaranted Service", 1995

.TP
o
Sally Floyd, "Notes on Class-Based Queueing: Setting Parameters", 1996

.TP
o
Sally Floyd and Michael Speer, "Experimental Results for Class-Based Queueing", 1998, not published.

.SH SEE ALSO
.BR tc (8)

.SH AUTHOR
Alexey N. Kuznetsov, <[email protected]>. This manpage maintained by bert hubert <[email protected]>

#16 lore raw

Re: CBQ MANPAGE: I hear the theme of '2001, A Space Odyssey'

Sorry didnt read it; did the 30 sec scan ..
If this is meant to be for users, why are you talking about skb->priority? Isnt it sufficient to just call it prioirity?
Also, if you think that Alexeys imp. is based on Floyd only, you are highly mistaken;
Going back to high latency response mode ...

cheers,
jamal

Quoted 37 lines · bert hubert

... to the sound of 'Also sprach Zarathustra':

After weeks of social deprivation and much digging through heaps of code, I
bring you

tc-cbq.8

The CBQ manpage. Nearly 2500 words, 8 printed pages, of nearly
unintelligible gobledygook, explaining mostly how CBQ works.

It is part of the Linux Advanced Routing & Traffic Control documentation project which contains a HOWTO, a mailinglist, an IRC channel and now manpages:

                            http://ds9a.nl/lartc

I want to thank Jamal for stubbornly straightening me out when I use messy language and explaining how things work. The errors are mine though.

I *implore* ANK and others to read through this. I'm about exhausted and running out of time (need to get on with work), and have a hard time figuring out the exact details of the CBQ link sharing algorithm. I need help, so to speak. The manpage indicates where.

Thanks for your attention. Please find tc-cbq.8 attached.

Regards,

bert hubert

--
http://www.PowerDNS.com Versatile DNS Software & Services
Trilab The Technology People
Netherlabs BV / Rent-a-Nerd.nl - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

#17 lore raw

Re: CBQ MANPAGE: I hear the theme of '2001, A Space Odyssey'

In reply to bert hubert
Quoted 3 lines · jamal

Sorry didnt read it; did the 30 sec scan ..
If this is meant to be for users, why are you talking about skb->priority? Isnt it sufficient to just call it prioirity?

It's not done yet and may need some readability tuning. Note however that skb->priority is a bit overloaded. It can contain a priority, but also a 32bit encoded classid. These are different things, so they deserve different mention.

Quoted 2 lines

Also, if you think that Alexeys imp. is based on Floyd only, you are highly mistaken;

I just copied the attribution from the kernel, am glad to rectify things.

Quoted 1 line

Going back to high latency response mode ...

Thanks for reviewing.

Regards,

bert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#18 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented (almost!)

In reply to bert hubert
Quoted 3 lines · jamal

Yes, sorry the last 4 are int_bulk (value 4) and not just bulk (2). good eye. You are still abusing the word TOS. Thats only 4 bits not 8; Use the terminology from RFC1349 at least.

Will do.

Quoted 8 lines
Quoted 5 lines

http://ds9a.nl/lartc/HOWTO/cvs/2.4routing/output/2.4routing-9.html#ss9.2

Your table appears to imply that a Maximum Reliability, Mininum Delay packet, TOS bits=9, gets mapped to band 1, not 0, which would not make sense.

This is the priomap: 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1 It says 1 is the right value

AFAICT, the priomap maps skb->priority to band. So the translation is as follows:

Type of Service octet, which is fed to:

        skb->priority = rt_tos2priority(iph->tos);

To extract the four TOS bits, and to translate to prio:
static inline char rt_tos2priority(u8 tos)

{
        return ip_tos2prio[IPTOS_TOS(tos)>>1];
}

----

__u8 ip_tos2prio[16] = {
        TC_PRIO_BESTEFFORT,
        ECN_OR_COST(FILLER),
        TC_PRIO_BESTEFFORT,
        ECN_OR_COST(BESTEFFORT),
        TC_PRIO_BULK,
        ECN_OR_COST(BULK),
        TC_PRIO_BULK,
        ECN_OR_COST(BULK),
        TC_PRIO_INTERACTIVE,
        ECN_OR_COST(INTERACTIVE),
        TC_PRIO_INTERACTIVE,
        ECN_OR_COST(INTERACTIVE),
        TC_PRIO_INTERACTIVE_BULK,
        ECN_OR_COST(INTERACTIVE_BULK),
        TC_PRIO_INTERACTIVE_BULK,
        ECN_OR_COST(INTERACTIVE_BULK)
};

---

#define TC_PRIO_BESTEFFORT              0
#define TC_PRIO_FILLER                  1
#define TC_PRIO_BULK                    2
#define TC_PRIO_INTERACTIVE_BULK        4
#define TC_PRIO_INTERACTIVE             6
#define TC_PRIO_CONTROL                 7
#define TC_PRIO_MAX                     15

net/sched/sched_generic.c:

static const u8 prio2band[TC_PRIO_MAX+1] =
{ 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1 };

        list = ((struct sk_buff_head*)qdisc->data) +
                prio2band[skb->priority&TC_PRIO_MAX];
Quoted 3 lines
Quoted 1 line

CBQ can use the skb->priority to classify:

so do prio and pfifo_fast (as i am sure you are aware)

Of course, but only CBQ (& HTB, by the way) can extract a classid directly from it, without a priomap. Devik is planning to learn HTB to extract a classid directly from the fwmark, to skip a layer of indirection.

Regards,

bert hubert

Signature · 4 lines
http://www.PowerDNS.com          Versatile DNS Software & Services
Trilab                                 The Technology People
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
#19 lore raw

Re: CBQ and all other qdiscs now REALLY completely documented (almost!)

Quoted 5 lines · bert hubert

On Sat, Dec 08, 2001 at 08:14:10PM -0500, jamal wrote:

AFAICT, the priomap maps skb->priority to band. So the translation is as follows:

yes ;->

Quoted 7 lines
Quoted 2 lines

so do prio and pfifo_fast (as i am sure you are aware)

Of course, but only CBQ (& HTB, by the way) can extract a classid directly from it, without a priomap. Devik is planning to learn HTB to extract a classid directly from the fwmark, to skip a layer of indirection.

I am not sure if this is such a nice hack. Whats wrong with with using the fwmark classifier to select classes?

cheers,
jamal